Daecore/ettin-150m-memory-reranker-ft-v1

Ettin-150M memory reranker v1

Summary

This model reorders the fixed 50-passage pool that a hybrid first stage produces for a memory query. It is a fine-tune of cross-encoder/ettin-reranker-150m-v1 trained with a graded listwise objective on 144,100 unique query-passage pairs across 2,882 queries.

On the 441 development queries whose fixed pool holds at least one useful passage, it raised nDCG@10 from 0.562 for the fused first stage to 0.762 and Hit@1 from 0.644 to 0.807. In the protected promotion read, on the same 865 answerable pools, it scored nDCG@10 0.728 against 0.595 for the previous reranker, +0.133 with a one-sided 95% lower bound of +0.119.

All relevance grades are frontier-model judgments under a frozen protocol, the corpus is mostly generated documents, and the per-query evidence behind the development and gate reads was lost after the aggregates were recorded. The limitations section spells out each.

Model details

Item	Value
Architecture	Ettin-150M (ModernBERT-family) cross-encoder with a trained scoring head; rank-16 LoRA+ adapter merged into the weights
Input	one query and one passage, 1,153 tokens total
Pool	a fixed 50-passage candidate set; the model reorders it and never adds to it
Output	one relevance score per pair, used for ordering
Serving precision	FP16 through ONNX Runtime with the CUDA overlay
Deployment state	protected gate passed; package qualified; not yet published or activated

Intended use

Final ordering inside a local memory-retrieval pipeline: the dense retriever and BM25 each retrieve, reciprocal-rank fusion merges them, and this model scores the top 50. It cannot recover a passage the first stage missed; on the full 553-query development panel the 112 queries with no useful passage in their pool stay at zero before and after reranking.

Training data

Item	Value
Queries	2,882 answerable, plus 30 explicit no-positive rows kept out of the ranking loss
Unique pairs	144,100, every one seen; 184,448 list presentations
Unique passages	35,006
Grade counts 0 / 1 / 2 / 3	32,331 / 37,258 / 43,731 / 30,780
Query mix	1,318 anchor-conditioned, 1,270 unanchored, 294 lexical-style
Protected queries or families in gradients	0

Grades come from final consensus ledgers, so every grade-1 label is usable supervision rather than a masked unknown. The lexical-style queries (terse, exact, pasted, or lightly misspelled) are there so the model learns the lexical lane's cases as well as rich semantic questions.

Recipe

Grade-gain ListNet over lists of 16 with an adjacent-grade RankNet auxiliary at weight 0.2; LoRA+ at rank 16 and alpha 32 on all linear layers, learning rate 5e-5 for the A matrices and the score head and 8e-4 for the B matrices; 2,882 optimizer updates; FP16. Merge and reload changed scores by 0.

Evaluation

Development panel. 553 queries with every one of the 27,650 pool passages graded; 441 have a useful passage in their pool.

Stage	nDCG@10	Hit@1	Hit@5	Hit@10	Hit@20	Hit@50
fused first stage, 441 answerable	0.5617	0.6440	0.8662	0.9206	0.9569	1.0000
this model, 441 answerable	0.7621	0.8073	0.9410	0.9615	0.9841	1.0000
fused first stage, all 553	0.4479	0.5136	0.6908	0.7342	0.7631	0.7975
this model, all 553	0.6078	0.6438	0.7505	0.7667	0.7848	0.7975

Pairwise accuracy on the answerable pools is 0.820. The all-query rows are the honest end-to-end view; Hit@50 is unchanged by design.

How it was chosen. Against the consensus hard-label baseline (0.680 nDCG@10 on this panel), a Lambda-style objective lost, ListNet at width 8 tied, and width 16 without a higher learning rate lost. The adjacent-grade term at 5e-5 reached 0.715. Restarting from the base over every paid pair gave 0.728 with pure ListNet, 0.731 with the auxiliary and standard LoRA, and 0.732 with DoRA. LoRA+ at rank 16 reached 0.762 and was the only large optimization gain; rank 32 tied it (+0.0006) and a further top-10 continuation lost 0.005.

Protected gate, same-pool comparison. On the 865 sealed queries whose candidate pool holds a useful passage, this model scored nDCG@10 0.7280 against 0.5952 for the previous reranker, +0.1328 with a one-sided 95% lower bound of +0.1191 against a registered 0.02 margin. The judge-noise band for this comparison was 0.0036. The full pipeline reached Hit@1 0.7609 and nDCG@10 0.6692 on the 895 union-answerable queries, against 0.6078 and 0.4988 before.

Runtime

The FP16 ONNX export removes CPU-hosted sequence operations from all 22 MLP layers so the whole graph runs on the GPU. It is bounded-quality equivalent rather than bit-exact to the Torch model: as recorded at qualification, +0.0002 nDCG@10 and +0.0023 Hit@1 on the development pools, 4 of 441 top choices and 15 of 441 top-10 memberships changed, and a 95th-percentile score drift of 0.0098. A top-50 pool reranks in 0.664 s median and 1.032 s at the 95th percentile on an RTX 3060 Ti; the one-second target is advisory.

The export regenerated on 2026-09-04 reproduced the qualified graph byte for byte. A parity recheck the same day had to use the training partition (2,882 queries, 144,100 pairs) because the qualification pools were lost, and it fails the frozen policy on two checks: Hit@1 moved by −0.0003, one query, and one passage moved 16 places against a limit of 4. All 10 top-1 changes and all 73 top-10 membership changes on that panel occur between passages whose FP16 scores are exactly tied in one run and one unit apart in the other, so the tie-break decides the order; the largest score change, 0.08, sits at rank 47. On that panel the export reranked a pool in 0.584 s median and 0.912 s at the 95th percentile, and the Torch FP16 model in 0.532 s and 0.777 s.

Limitations, ranked

1. Model-judged relevance. No human grades; session agreement 0.858 and kappa 0.682 on a 419-query audit. Scores measure agreement with that instrument.
2. Generated corpus. Mostly fictional document worlds chosen for breadth; behavior on other real corpora is unmeasured.
3. Lost per-query evidence. The development ledger, gate judgments, pool scores, and gate report were destroyed in a local storage incident after these aggregates were recorded. The numbers stand as recorded and cannot be re-audited; the recipe-search arms survive only as recorded results.
4. One training run per arm. Several selection margins (rank 16 versus 32, width 8 versus 16) are inside a plausible seed effect; compute set that limit.
5. First-stage ceiling. 112 of 553 development queries and 95 of 990 gate queries have no useful passage in any top-50 pool; ranks 51 to 200 were never judged.
6. No CPU or DirectML realization is qualified. Hosts without the CUDA provider serve the fused first-stage order with a typed notice.
7. The reproducible parity read fails the frozen policy. The recorded development-panel pass cannot be re-run, and the 2026-09-04 recheck on the training partition fails two checks at exact FP16 ties. The operator accepted the recorded qualification with this disclosure rather than exporting an FP32 graph or amending the policy; an FP32 export and re-run would settle it.

Reproducibility

• Model tree SHA-256 bd8028b5dbc6ab2cd2ab9a6019346de7b25536f50022617194bcb8503d4467ad; base revision 025501c4e0f9bbeb4c5b198318e0089ff061cc14, tree e0edfb38cc495f23507de50233d0965f4e72ad0872e70e6f4e8fc889d394ea01.
• Training data manifest 286318992c584ae790a72a5a86b0e7d03d8860f8ae93c35f6cead4b08b4d395a, schedule e98d3cec2533b6edcded7c0b5017cba432e4af0e73fd68fca84997f934fa8f63.
• Recipe id final-listnet-adjacent-loraplus16-full-coverage-w16-lr5e5; registry manifest ../model_registry/ettin-150m-memory-reranker-ft-v1.json; experiment log ../history/README.md.
• Serving graph SHA-256 ce05a6a810007372c6de0f4c56dc94f652e1d28f2eeddf1878b64a170b4c48c2.
• Build stack: Transformers 5.2.0, Sentence Transformers 5.5.1, PyTorch 2.10.0; export opset 18.

License

Apache 2.0, following the upstream model; the distribution includes the license, a notice, and a modification disclosure.