EditRank-0.6B (Research Preview)

Status: Research Preview. Not a production model. Do not automatically publish or apply the selected edits. The model selected factually unsafe edits in 12.8% of the evaluated cases. Candidate edits require independent factual validation before use.

Model type	0.6B edit-ranking model (full fine-tune of Qwen/Qwen3-Reranker-0.6B, 28 layers, bf16)
Status	Research Preview
Training objective	Top-1-oriented listwise ranking (softmax cross-entropy against softmax(utility / 0.25) over each query's candidate set)
Intended use	Ranking candidate content edits of one document for one query, for research and human-reviewed workflows
Evaluation	Simulated answer-engine source ranking by a fixed Qwen3.8-27B engine and an independent Gemma-4-31B engine
Evaluated domain	Retail (product pages) only
Language	English

What it does

Given a search query, an original document and a revised version, the model outputs a score logit("yes") - logit("no") estimating whether the revised version would be ranked higher than the original as a source by an AI answer engine for that query, while keeping every fact of the original unchanged. Score each candidate edit and take the highest. See example_inference.py for the exact prompt format; scores are only comparable within one query.

In the bundled example (bf16, single GPU), the model scores the unchanged original +3.625, a rewrite that invents facts ("waterproof", "best boot of 2026") +3.500, and a faithful rewrite +3.125. The fact-inventing rewrite outranks the faithful one. This is the failure mode behind the 12.8% figure below: the model is not a fact checker.

Results (pre-registered, 624 independent Amazon ESCI queries)

Utility = mean rank gain of the selected candidate over the original document under the simulated engine (unsafe candidates receive a penalty below the worst candidate). Original document = 0. The independent Gemma engine was not used to create any training label.

Method	Fixed Qwen3.8-27B engine	Independent Gemma-4-31B engine
EditRank-0.6B	0.388	0.475
Earlier run of the same model family (Run1)	0.370	0.457
Fixed best single strategy ("content improvement")	0.244	0.339
BM25	0.131	0.206
Base model Qwen3-Reranker-0.6B (relevance prompt)	-0.056	0.052
Random candidate	-0.518	-0.367
Oracle (best candidate)	1.193	1.034

Paired bootstrap, mean difference [95% CI]:

Comparison	Qwen engine	Gemma engine
vs BM25	+0.257 [0.157, 0.360]	+0.269 [0.156, 0.392]
vs Run1	+0.018 [-0.033, 0.072]	+0.018 [-0.048, 0.091], not statistically significant
vs fixed best strategy	+0.144 [0.020, 0.268]	+0.135 [0.003, 0.272] (thin margin)
vs base model	+0.444 [0.338, 0.550]	+0.423 [0.307, 0.551]

Pairwise accuracy (pairs with utility gap at least 1): 0.761 / 0.765. ECE: 0.046 / 0.069. Agreement of the two engines on candidate gains: Pearson 0.60.

How to read this. The demonstrated result is a statistically supported top-1 utility advantage over BM25 on held-out retail queries, confirmed by an engine that produced no training labels. It does not show that this model is better than Run1, and it does not show that the listwise objective is better than the earlier one.

Safety and limitations

• Unsafe selections: 12.8%. In 12.8% of the 624 queries the model's top-1 candidate was flagged by an automated judge as changing or adding facts (Run1 13.5%, BM25 19.7%, fixed strategy 11.2%, keeping the original 0%). The judge itself is imperfect (audit on 60 samples: flag precision 0.63, at least 6.7% of unflagged cases missed), so the true rate is uncertain in both directions.
• No guarantee of factual correctness. The model ranks candidates; it does not verify them.
• No proof of improved visibility in live AI search engines (ChatGPT, Perplexity, Google AI Overviews, Gemini or others). All results come from a fixed simulated environment: two fixed open-weight engines, a fixed prompt, and a fixed candidate set. They are not evidence of exposure lift in any production answer engine.
• Evaluated on retail product pages only. No evidence for other domains, other languages, or longer documents (documents are clipped to 1,500 characters, the original to 600).
• Engines change; rankings learned against these engines may not transfer.

Training

• Base: Qwen/Qwen3-Reranker-0.6B, Hub revision e61197ed45024b0ed8a2d74b80b4d909f1255473 (model.safetensors sha256 27cd75a405b9c1b46b59abfd88aaa209e6fed2a1972cde9b70e7659537c5e65b).
• Full fine-tune, 3 epochs, lr 1e-5, gradient accumulation 4, tau 0.25, bf16, epoch selected on a dev split (selected epoch 2 of 3). One B200 GPU, 547 s. Hyperparameters were fixed before the run; no loss or hyperparameter search.
• Data: 768 training query groups (original document plus rewrites per query) from the retail and video-games domains of C-SEO Bench. Candidate rewrites and ranking labels were generated with an in-house deployment of an open-weight Qwen3.8-27B model (5 ranking seeds per candidate). No outputs of commercial LLM APIs were used.
• Training and evaluation data are not released.

Data sources and licenses

Used for	Source	License as published	Notes
Training (retail)	C-SEO Bench retail (Puerto et al., 2025), derived from the Amazon Shopping Queries Dataset (ESCI, Reddy et al., 2022)	C-SEO Bench: Apache-2.0 (Hub metadata); ESCI: Apache-2.0	Product text originates from Amazon listings
Training (video games)	C-SEO Bench videogames, derived from the Steam Games Dataset by FronkonGames (FronkonGames/steam-games-dataset)	CC BY 4.0 for the version C-SEO Bench used (C-SEO Bench was published 2025-06-05; the source carried CC BY 4.0 from 2023-12-03, commit 006aa50f6f, until it was relabeled MIT on 2026-06-19, commit 3f7a8fb501); C-SEO Bench itself: Apache-2.0	Game descriptions originate from Steam store listings and remain the publishers' text; attribution to FronkonGames as required by CC BY 4.0; no text redistributed here
Evaluation only	624 Amazon ESCI queries disjoint from C-SEO Bench	Apache-2.0	Not used for training or model selection
Development check only (cross-domain)	C-SEO Bench web, debate	as published	Not used for training
Excluded	C-SEO Bench news (non-commercial terms), books; published C-SEO results produced with commercial APIs		Not used

The underlying product and game descriptions were written by third parties (sellers, publishers). This repository contains only model weights, configuration, tokenizer files, this card and an example script; it contains no training or evaluation text. The license field (Apache-2.0) covers this repository's contents and follows the base model; it is not a legal determination about the upstream text.

Citations:

@inproceedings{puerto2025cseo,
  title={C-{SEO} Bench: Does Conversational {SEO} Work?},
  author={Puerto, Haritz and Gubri, Martin and Green, Tommaso and Oh, Seong Joon and Yun, Sangdoo},
  booktitle={NeurIPS 2025 Datasets and Benchmarks Track},
  year={2025}
}
@article{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
  author={Reddy, Chandan K. and M{\`a}rquez, Llu{\'i}s and Valero, Fran and Rao, Nikhil and Zaragoza, Hugo and
          Bandyopadhyay, Sambaran and Biswas, Arnab and Xing, Anlu and Subbian, Karthik},
  journal={arXiv preprint arXiv:2206.06588},
  year={2022}
}

Steam Games Dataset: FronkonGames, Hugging Face FronkonGames/steam-games-dataset, doi:10.57967/hf/0511.

Reproducibility

The pre-registered evaluation was re-run from scratch on a fresh B200 pod with sha256-verified weights and inputs; all 76 reported metrics and all per-query utilities were bit-identical. The files in this repository were re-downloaded in a separate process and cache before publication and re-scored on the same 624 queries; see "Release verification" below and eval_manifest.json.

Release verification

Before publication, the weights in this repository were downloaded from a private Hub copy in a separate process with an empty cache and checked offline on one B200 GPU against the original training checkpoint. Every pre-registered criterion passed, both for the private copy and again for an anonymously downloaded public copy:

Check	Result
sha256 of the 6 model files equals the original checkpoint	pass
All 310 tensors bitwise equal; reload with transformers	pass
Top-1 selections on the 624 evaluation queries, example_inference.py scoring vs original code	624/624 identical, max score difference 0.0
Pre-registered evaluation re-run with these files	76/76 metrics identical, per-query utilities identical, unsafe top-1 12.8%
example_inference.py runs	pass
Base model sha256 equals Qwen/Qwen3-Reranker-0.6B@e61197ed	pass

The model files here (weights, config, tokenizer, chat template) are byte-identical to the verified ones; their sha256 values are listed in eval_manifest.json. Only the repository id string in the example and this card's text differ.