Wouze/read-only-reranker-probes

Qwen3-Reranker-4B, twice as fast

Read-only probes for a frozen Qwen3-Reranker-4B. They score each passage part-way through and drop most of the list. The base model is not in this repository. Load Qwen/Qwen3-Reranker-4B separately.

2× speed, measured. On one RTX 3090 the half-latency schedule takes 1,644 ms against a 3,276 ms prefix-cached forward, at an nDCG@10 drop of +0.00048, with the reference top 10 still present on 0.80 of a 400-query check split. The layer-25 freeze moves finalist margins by up to 0.50, and a layer-25 blend (α = 0.15) brings the ranking back.

About 4× less work than a normal transformer forward. A stock Hugging Face forward repeats the query on every passage and projects the full vocabulary, which costs 1.73× the prefix-cached forward in multiply-accumulates. Gathering the projections onto real tokens reaches 1,430 ms (2.29× that forward, drop +0.0012) with the blend removed. 1.73 × 2.29 = 3.96, about 4×. The 2× is the timed speedup. The 4× is that product.

Code, the paper, and the full result tables: Wouze/read-only-reranker-probes.

Files

File	What it is
probes/q4_p1.pt	One-token probe. Exits at layers 9 and 15. Rank-16 LoRA on the probe path only. Used by the half-latency schedule.
probes/q4_full.pt	Nine-token probe. Exit heads at layers 4, 6, 8, and 9 through 16. The cascade reads 9 and 15.
probes/ridge_4b_l21.pt	Layer-21 ridge, beta shape (2560,), λ = 10⁻³, held-out correlation 0.994.
week3_f25_ridge_beta.pt	Layer-25 ridge blend. beta shape (2560,), λ = 10⁻³, α = 0.15, held-out correlation 0.9991 on the 15 finalists.
paper_r1.json	The pre-registered keep rule. SHA-256 750cc3641a728eac51426765e8db15a014ab71e314e0ccf55540ccd45996b43e.
config.json	Lists the checkpoint paths. The Hub counts one download when this file is requested, which hf download does together with the weights.

hf download Wouze/read-only-reranker-probes --local-dir data/fast06

License

Apache 2.0. The probes are a derivative of Qwen3-Reranker-4B, which is also Apache 2.0.