EditRank-Multidomain-4B (Research Preview)

Status: Research Preview. Not a production model.

• Do not use this model alone to automatically edit or publish content. Every selected edit needs external fact verification and human approval before use.
• The model can rank a fact-violating edit above the original. In the bundled example, a rewrite that invents a version number, a latency and a statistic scores +3.750, above the unchanged original at +3.312.
• Fact-violating top-1 rate: 10.3% on the documentation test set (automated judge).
• No effect on live AI search has been shown. All results come from simulated answer engines; improved exposure, citation or recommendation in ChatGPT, Gemini, Perplexity, Google AI Overviews or any other product is unverified.

At a glance

Documentation utility (independent Gemma engine, 184 held-out groups)	2.010 (EditRank-0.6B: 1.782)
Difference vs public EditRank-0.6B	+0.228, 95% CI [+0.074, +0.422]
Fact-violating top-1 rate (documentation)	10.3% (EditRank-0.6B: 17.9%)
Over-edit loss (documentation, Gemma)	0.175 (EditRank-0.6B: 0.316)
Video-game independent evaluation	Not done
SaaS / marketing content	Not tested
Live AI search exposure	Not verified
Automatic editing or publishing on its own	Not permitted; requires fact verification and human approval

Model type	4B edit-ranking model: Qwen/Qwen3-Reranker-4B + LoRA (r 16, alpha 32), merged, bf16, 36 layers
Status	Research Preview
Training domains	Retail product pages, video-game store descriptions, technical documentation (Kubernetes, Python)
Main independent evaluation	184 technical-documentation query groups from document families not used in training
Training objective	Listwise top-1 cross-entropy against softmax(utility / 0.25); the unchanged original is always a candidate
Language	English
Smaller sibling	ThakiCloud/EditRank-0.6B (retail-evaluated, lighter)

What it does

Given a search query, an original document and a revised version, the model outputs a score logit("yes") - logit("no") estimating whether the revised version would be ranked higher than the original as a source by an AI answer engine for that query, while keeping every fact of the original unchanged. Score every candidate, including the unchanged original, and take the highest. If the original wins, keep it. See example_inference.py for the exact prompt format; scores are only comparable within one query.

In the bundled example (bf16, one GPU), the model scores a faithful rewrite +3.938, a rewrite that invents a version number, a latency and a statistic +3.750, and the unchanged original +3.312. The fact-inventing rewrite outranks the original. This is the failure mode behind the unsafe rates below: the model is not a fact checker.

Results

Utility = mean rank gain of the selected candidate over the original document under a simulated answer engine (unsafe candidates receive a penalty below the worst candidate in the group). Keeping the original = 0. Two engines are reported separately and never averaged: a fixed in-house Qwen3.8-27B engine (the same model family that produced the training labels) and an independent Gemma-4-31B engine that produced no training label. Decisions used the Gemma engine. Comparisons were pre-registered with fixed thresholds before training; the model was trained once, with no hyperparameter search.

Technical documentation (184 held-out query groups, independent)

Query groups come from Kubernetes and Python documentation pages whose document families (versions, translations, sibling pages of one topic) were kept entirely out of training. Queries are synthetic (see Limitations).

Method	Gemma engine	Qwen engine	Unsafe top-1	Over-edit loss (Gemma)	Original kept
EditRank-Multidomain-4B	2.010	2.273	10.3%	0.175	8.2%
EditRank-0.6B (public, retail-only training)	1.782	2.107	17.9%	0.316	7.6%
Fixed best single strategy ("content improvement")	1.743	2.010	23.4%	0.358	0%
BM25	1.160	1.396	16.3%	0.322	0%
Keep the original	0	0	0%	0	100%
Oracle (best candidate)	2.320	2.676

Paired bootstrap vs EditRank-0.6B, mean difference [95% CI]:

	Gemma engine	Qwen engine
Top-1 utility	+0.228 [+0.074, +0.422]	+0.166 [-0.032, +0.370] (not significant)
Over-edit loss (lower is better)	-0.141 [-0.250, -0.048]	-0.102 [-0.194, -0.017]

Over-edit loss is the mean utility lost when the chosen edit is worse than leaving the document unchanged (max(0, u_original - u_selected)). Unsafe top-1 is the share of queries whose selected candidate an automated judge flagged as changing or adding facts; a stricter numeric/quote judge flags 4.9% (EditRank-0.6B: 6.5%).

Retail (624 query groups, development comparison)

These Amazon ESCI queries were already used in earlier decisions about this model family, so this is a development comparison, not an independent test.

Method	Gemma engine	Qwen engine	Unsafe top-1	Original kept
EditRank-Multidomain-4B	0.467	0.400	12.8%	15.5%
EditRank-0.6B	0.475	0.388	12.8%	19.1%
BM25	0.206	0.131	19.7%	10.4%
Keep the original	0	0	0%	100%

Difference vs EditRank-0.6B on the Gemma engine: -0.007 [-0.092, +0.075]. Retail quality is at the level of the 0.6B model, not above it; an internal 4B variant trained on retail and video games only scored higher on retail (0.530) but did not improve on documentation (1.746) and almost never kept the original (3%).

Video games: not independently evaluated

There is no independent video-game test set. A 37-group exploratory set shares documents with training and has labels from the label engine only; it is not used for any claim. Video-game generalization is unverified.

Limitations

• Simulated engines only. All numbers come from two fixed open-weight engines, a fixed prompt and a fixed candidate set. They do not show improved exposure in ChatGPT, Gemini, Perplexity, Google AI Overviews or any other live product, and the documentation gain above must not be read as a visibility or traffic increase.
• Synthetic documentation queries. The documentation queries were generated from the documentation sections by an in-house open-weight Qwen3.8-27B model. They are not real user queries.
• The decision to keep the original is still weak. On retail groups where the original is strictly the best candidate (25 groups), the model kept it 20% of the time (EditRank-0.6B: 32%). The documentation test set has only 2 such groups, so it barely tests this decision.
• Not a fact checker. 10.3% unsafe top-1 on documentation and 12.8% on retail. The automated judge is itself imperfect, so the true rates are uncertain in both directions.
• Untested domains. SaaS and marketing content, news, other languages and long documents were not tested (documents are clipped to 1,500 characters, the original to 600; inputs truncated at 1,280 tokens). Video games: unverified.
• Some documentation sections start mid-context (for example with a code example) because of chunk boundaries.
• Answer engines change; rankings learned against these engines may not transfer.

Training

• Base: Qwen/Qwen3-Reranker-4B, Hub revision 22e683669bc0f0bd69640a1354a6d0aebcfeede5 (shard sha256 values in eval_manifest.json, checked against the Hub before training).
• LoRA r 16, alpha 32 on q/k/v/o/gate/up/down, lr 1e-4, 3 epochs, gradient accumulation 4, tau 0.25, bf16; epoch selected on dev top-1 utility (epoch 1). Merged into the base weights. One NVIDIA B200 GPU, 50 minutes. One run, hyperparameters fixed before training.
• Training groups: retail 400, video games 368, technical documentation 460 (dev: 45 / 31 / 46). Each group has the original document and 9 rewrites (structure, query alignment, fact-preserving restructuring, and strategies that can introduce unsupported facts). Candidates flagged as fact-violating get the lowest utility, so they are never the training target.
• Rewrites, rankings and documentation queries were produced with an in-house deployment of an open-weight Qwen3.8-27B model. No outputs of commercial LLM APIs were used.
• Training and evaluation data are not released.

Data sources and licenses

Used for	Source	License	Notes
Training (retail)	C-SEO Bench retail (Puerto et al., 2025), derived from Amazon Shopping Queries (ESCI, Reddy et al., 2022)	C-SEO Bench: Apache-2.0 (Hub metadata); ESCI: Apache-2.0	Product text originates from Amazon listings
Training (video games)	C-SEO Bench videogames, derived from the Steam Games Dataset by FronkonGames (FronkonGames/steam-games-dataset, doi:10.57967/hf/0511)	CC BY 4.0 for the version C-SEO Bench used (relabeled MIT on 2026-06-19)	Game descriptions remain the publishers' text
Training and test (technical docs)	Kubernetes documentation, kubernetes/website at commit bc7f3479bc8b0af48097b155c22c8d58e8e1afe4, content/en/docs (concepts, tasks, tutorials, setup)	CC BY 4.0	© The Kubernetes Authors. Pages marked as third-party content and auto-generated reference pages were excluded
Training and test (technical docs)	Python documentation, python/cpython Doc/ at commit 97d361116f27d9a53ff735151ba26f5bcb463b74 (library, tutorial, howto, reference, using, faq)	PSF License Version 2 (code examples: PSF-2.0 or 0BSD)	Copyright © 2001 Python Software Foundation; All Rights Reserved. License, copyright, about, bugs, contents and glossary pages were excluded
Evaluation only	624 Amazon ESCI queries disjoint from C-SEO Bench	Apache-2.0	Not used for training or model selection

Attribution and changes. The Kubernetes and Python documentation was split into sections of 400 to 1,400 characters (long code blocks removed); synthetic queries and rewritten candidates were generated from those sections and used only as training and evaluation signal. Per-file paths, licenses and git blob hashes were recorded for all 844 source pages. No documentation text, query or rewrite is included in this repository, and neither the Kubernetes Authors nor the Python Software Foundation endorses this model.

This repository contains only model weights, configuration, tokenizer files, this card, eval_manifest.json and an example script. The license field (Apache-2.0) covers this repository's contents and follows the base model; it is not a legal determination about the upstream texts.

@inproceedings{puerto2025cseo,
  title={C-{SEO} Bench: Does Conversational {SEO} Work?},
  author={Puerto, Haritz and Gubri, Martin and Green, Tommaso and Oh, Seong Joon and Yun, Sangdoo},
  booktitle={NeurIPS 2025 Datasets and Benchmarks Track},
  year={2025}
}
@article{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
  author={Reddy, Chandan K. and M{\`a}rquez, Llu{\'i}s and Valero, Fran and Rao, Nikhil and Zaragoza, Hugo and
          Bandyopadhyay, Sambaran and Biswas, Arnab and Xing, Anlu and Subbian, Karthik},
  journal={arXiv preprint arXiv:2206.06588},
  year={2022}
}

Inference cost

Retail set, 624 groups of 10 candidates, one B200, bf16: group latency p50 168 ms, p95 257 ms, about 56 candidates/s, peak memory 12.5 GiB. (Timing on the documentation set is not reported because an evaluation-file overwrite in the scoring job made that measurement run over a different group set; documentation scores were unaffected.)

Release verification

Before publication, this repository's files were downloaded from the private Hub revision in a separate process with an empty cache and checked offline on one NVIDIA B200 GPU against the original training checkpoint and the frozen scores behind the pre-registered verdict. The verdict itself was not recomputed. Every check passed:

Check	Result
sha256 of the 6 model files equals the original merged checkpoint	pass
All 398 tensors bitwise equal, all finite (no NaN/Inf); reload with transformers without missing or unexpected keys	pass
Top-1 selections, example_inference.py scoring vs the frozen evaluation scores	documentation 184/184, retail 624/624, video games 37/37 identical; max score difference 0.0
Documentation metrics recomputed from these selections	Gemma 2.0098, Qwen 2.2728, unsafe 10.33%, over-edit loss 0.175, identical to eval_manifest.json
example_inference.py runs	pass
Base shards sha256 equal Qwen/Qwen3-Reranker-4B@22e68366	pass

The sha256 values of the model files are listed in eval_manifest.json.