44b-reranker-minilm-l6

Method	Precision@1	Precision@3	MRR
44b-reranker-gte-modernbert (149.6M, sibling)	0.929	0.982	0.956
44b-reranker-minilm-l6 (22.7M — this model)	0.865	0.954	0.914
BM25 (keyword search)	0.856	0.940	0.905
Hybrid retrieval (RRF, k=60)	0.728	0.793	0.788
OpenAI text-embedding-3-small	0.681	0.749	0.745

🏆 22.7M model that beats the generalists

A cross-encoder reranker fine-tuned on peer review → paper pairs mined from the 44B corpus.

The query is a peer reviewer's own summary of a paper, written by a domain expert who had never heard of this model, and the positive is the paper that review is attached to.

The relevance label is not a heuristic, not a click, and not LLM-generated. Fine-tuning on it cut the base model's error rate by 62% — and this 22M-parameter model, at 7.4 ms/query, then outscores a 150M-parameter reranker running off the shelf. On a held-out test of 3,157 queries over scientific papers (9-way reranking with hard negatives), this 22.7M-parameter model — small enough to run almost anywhere, at 7.4 ms/query — picks the right document first 87% of the time and lands it in the top 3 over 95% of the time, beating OpenAI's general embeddings and hybrid retrieval and edging keyword search.

What that means, all measured on the same test set:

• 87% vs 68% over OpenAI's general embeddings on top-1 accuracy (+19 points); 91.4% vs 74.5% on MRR.
• Right document in the top 3: 95.4%.
• Beats hybrid retrieval 87% vs 73%, and edges BM25 — at a fraction of the size.
• 62% fewer ranking errors than the base it is fine-tuned from (35.7% → 13.5%). Mean rank 1.322 of 9, 7.4 ms/query.

*Scope: reranking numbers — selecting the right document from a candidate set — not open-corpus search.

• Also see: 44b-reranker-gte-modernbert reaches 93% at 149.6M params.*

---

Results

The benchmark — query and candidate IDs, and every arm's scored output — is public: NYSgpt/44b-eval.

model	P@1	P@3	MRR	mean rank	ms/query
this model	0.8654	0.9541	0.9141	1.322	7.4
ms-marco-MiniLM-L-6-v2 (base)	0.6427	0.8166	0.7500	2.210	9.4
chance	0.1111	0.3333	—	5.000	—

Evaluated on 3,157 held-out queries over 1,534 papers, none of which appear anywhere in training — not as positives and not as negatives. Each query is scored against 9 candidates: 1 positive and 8 mined hard negatives. Random chance is P@1 = 0.1111.

Error rate 35.73% → 13.46%, a 62.3% relative reduction.

At n = 3,157 the gap is far beyond noise (z ≈ 21.3 on an unpaired standard error, which is conservative — these are paired measurements on identical queries).

The size comparison is the interesting one. gte-reranker-modernbert-base — 150M parameters, off the shelf — scores 0.8334 at 53.1 ms/query on this same evaluation. This model beats it on accuracy at roughly 7× the speed and 1/7th the parameters. Domain supervision bought more than 7× the model did.

For reference, the first-stage retrieval arms scored on the same 3,157 queries:

first stage	P@1	MRR
BM25 (lexical)	0.8562	0.9046
Hybrid RRF (k=60)	0.7282	0.7881
Dense (text-embedding-3-small)	0.6807	0.7450

NYSgpt/44b-reranker Sibling model was trained on identical data and reaches P@1 0.9290 at 41.1 ms/query.

What it was trained on

training groups	24,177
negatives per group	7
distinct papers in train	12,058
held out (val + test)	3,099 papers
query	an official review's summary, or a meta-review's metareview
positive	the paper the review is attached to
negatives	mined per query — semantic (HNSW over text-embedding-3-small, ef_search=160) and lexical (Postgres FTS over abstracts)
source venues	ICLR 2024, ICLR 2025, NeurIPS 2024

Splits are disjoint by paper, not by pair. Papers carry ~4.5 reviews each, so splitting by pair would place the same paper in train and test and inflate every number downstream. Proven at build time rather than asserted: |train ∩ val| = 0, |train ∩ test| = 0, |val ∩ test| = 0.

Held-out papers are excluded from the negative pool as well as the positive one. Label leakage was measured after the fact, not assumed: 0 of 1,534 test positives appear as training positives; 23 (1.5%) appear as a negative for some other query, which biases mildly against this model rather than for it.

Training

base	cross-encoder/ms-marco-MiniLM-L-6-v2 (22.7M params)
objective	grouped cross-entropy — one positive against 7 hard negatives per group
epochs	2
max sequence length	256
query / document truncation	600 / 900 characters
groups per batch	8
learning rate	2e-5
hardware	1× NVIDIA A10G (g5.xlarge)
wall clock	621 seconds

Limitations

• Query register is long-form prose. Training and evaluation queries average 700 characters (174 tokens) of fluent reviewer writing. This model has never seen a short keyword query. Do not assume the gains transfer to a two-word search box — that is untested.
• One discipline, one document type. Every review comes from top-tier machine learning conferences (ICLR, NeurIPS; 2024–2025). No other field, venue tier, or document class is represented.
• Short context is a real constraint for this task. Queries are ~180-token review summaries against long abstracts; at 256 tokens most pairs are truncated. The accuracy above is achieved despite that, which is part of why the fine-tune matters so much here.
• Negatives skew semantic. 92% of mined negatives came from the embedding arm and 8% from the lexical arm — the lexical miner conjoined a review's rarest terms and returned nothing on most queries, so the semantic arm backfilled. A balanced mine is the obvious next version.
• Trained on 7 negatives, evaluated against 8. Evaluation is marginally harder than training was.
• The evaluation shares a pipeline with the training data. Same source, same construction, disjoint only by paper. Not an independent measure of end-to-end answer quality in a live system.
• CPU latency is unmeasured. Every timing here is A10G at batch 128. No CPU serving number exists yet.
• False-negative screening may under-catch. Candidate negatives were rejected for sharing the positive's Semantic Scholar id or a near-identical title; the rate was 0.118%, and the S2-duplicate check fired zero times across 28,798 candidates.
• The supervision is not exhausted. This model trained on 30,509 of 77,147 available pairs. V2 pending.

Eval shape

This is a 9-way ranking task, not open-corpus retrieval. A 0.8654 here means "picks the right paper out of nine confusable ones," not "finds the right paper in three million." The negatives are hard by construction — nearest neighbours in embedding space plus lexical near-matches — but the pool is small and the task is closed.

Intended use

Second-stage reranking of scientific-document retrieval — reordering a first-stage candidate list for queries written in the register of a paragraph of technical prose. The speed profile suits a live request path.

Out of scope

Short keyword queries · non-ML disciplines · any setting requiring a measured CPU latency guarantee · any claim of quality on open-corpus retrieval, which this evaluation does not test.

Provenance

Reviews from OpenReview; papers from arXiv, OpenAlex and Semantic Scholar, unified in corpus. Trained 2026-07-31.

The NYSgpt model family

Model	Role	Params
biomed-encoder	Retriever — search over bioRxiv/medRxiv (433,449 preprints)	568M
biomed-reranker	Cross-encoder reranker, second stage over the above	149M
44b-reranker-gte-modernbert	Reranker, scientific-papers corpus (44b), quality tier	149M
44b-reranker-minilm-l6	Reranker, scientific-papers corpus (44b), fast tier	22.7M