LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference
Authors: Guangyuan Ma
Yongliang Ma
Xuanrui Gou
Zhenpeng Su
Ming Zhou
Songlin Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Information Retrieval
Efficient Deploy
Fast Query Inference
LLM-based Text Retrieval
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded offline, while queries arrive in real-time, necessitating an efficient online query encoder. Although LLMs significantly enhance retrieval capabilities, serving deeply parameterized LLMs slows down query inference throughput and increases demands for online deployment resources. In this paper, we propose LightRetriever, a novel LLM-based retriever with extremely lightweight query encoders. Our method retains a full-sized LLM for document encoding, but reduces the workload of query encoding to no more than an embedding lookup. Compared to serving a full LLM on an A800 GPU, our method achieves over a thousand times of speedup in query encoding and over 10× increase in end-to-end retrieval throughput. Extensive experiments on large-scale retrieval benchmarks show that LightRetriever generalizes well across diverse tasks, maintaining an average of 95% retrieval performance.

Published as a conference paper at ICLR 2026
LIGHTRETRIEVER: A LLM-BASED TEXT RETRIEVAL ARCHITECTURE WITH EXTREMELY FASTER QUERY INFERENCE
Guangyuan Ma Yongliang Ma', Xuanrui Gou', Zhenpeng Su Ming Zhou', Songlin 12+
'Institute Information Engineering, Chinese Academy of Sciences, Beijing, China Schooo of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
'Langboat Technology, Beijing, China
@iie.ac.en, gouxuanrui21 @mails. [mayongliagg.zoumingg/ @ langboat.com
ABSTRACT
Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded of- fline, while queries arrive in real-time, necessitating efficient online query coder. Although LLMs significantly enhance retrieval capabilities, serving deeply parameterized LLMS slows down query inference throughput and increases de- mands for online deployment resources. In this paper, we propose LightRe- triever, novel LLM-based retriever with extremely lightweight query encoders. Our method retains a full-sized LLM for document encoding, but reduces the workload of query encoding no more than an embedding lookup. Compared serving full LLM on an A800 GPU, our method achieves over 1000x speedup in query encoding and over 10x increase in end-to-end retrieval throughput. Ex- tensive experiments on large-scale retrieval benchmarks show that LightRetriever generalizes well across diverse tasks, maintaining an average of 95% retrieval per- formance.
INTRODUCTION
Given search queries and candidate documents, text retrieval employs dual-encoder architectures to encode representation vectors and search documents relevant queries based query-document similarities (Salton et al. 1975). The encoded vectors could be condensed into low-dimensional vectors for dense retrieval or sparsified into high-dimensional vectors for sparse retrieval. Dense retrieval (Karpukhin et al. 2020) compresses textual information into dense vectors with parame- terized language models (L.Ms) and retrieves based on inner product similarity. Meanwhile, sparse retrieval (Robertson et al. 1994) characterizes sparse term frequencies a much broader vocab- ulary space, and then searches based on lexical overlap and term-based relevance metrics, such as direct TF-based search (Inner product) (Bai al. 2020 Formal al. 2021) and BM25 (Robertson cct al. 1994). Additionally, hybrid retrieval enhances retrieval abilities by interpolating their similar- ity scores (Wang al. 2021) ranks (Cormack al. 2009). Text retrieval is broadly applied in various scenarios, such as web search (Lu et al. 2022). semantic textual similarity (Bowman et al., 2013). fact checking (Thorne et al. 2018). and retrieval augmented generation (Lewis et al. (022)).
Deploying retrieval systems requires different processing workflows for queries and documents. As shown in Figure [ search candidates, documents are encoded with document encoder Enca and then indexed into search databases (Johnson et al.] [021]] Robertson et al., 1994). Because doc- uments are obtained before the search operation, they are pre-computable and distributable, which can be indexed offline and distributed anywhere (Yang et al., (2018)). On the contrary, queries are input and encoded in real-time, which are non-pre-computable and non-distributable. The query encoder Enc, needs to be served online. Special GPU accelerators are mandatory if the query en- coder a deeply parameterized LM. Recent sfate-of-the-ar (SOTA) retrievers (Wang et al., 2024b)
"Corresponding author. Code is available at httpp////ttthhbb.c///a/a///////s

---

Published as conference paper at ICLR 2026
Online Serving of Query (Non pre-computable)
Offliie Index of Document (Pre-computable)
1) Full-sized Query Inference (Previous works)
Scarch
Index
Index
Document Text
what froit apple?
Takenize
LM Enc,
Encode
Online
Offline Encodo
En
LM
Tokenize
An apple round, odible fruit produced an apple tree.
Search
Spurse Index
Index
Heavy Onlline
Workllod
2) Extremely Fast Query Inference (LightRetriever)
Sparse Vectors
Lookup
Search
Dense Index
Index
Query
Document Text
fruit apple!
Tokenize
Cached Pinheeddng
Offline Fnoode
LM
Tokenize
An apple round, odible fruit produced an apple tree.
Ency
Directly
Search
Spurse Index
Index
Lightweiggtt
Figure LightRetriever targets extreme query inference speed for LLM-based text retrieval, reducing the workload of query encoding to no more than an embedding lookup.
BehnamGGader al. 2024) employ deeply parameterized LLMS symmetric dual-encoders Enc, and Enea for enhanced retrieval capabilities. However, serving giant I.MSs slows down query in- ference throughput and depletes online deployment resources (Kwon al. 2023). Despite the rapid progress on improving retrieval quality, little prior work has investigated the efficiency challenges of L.L.-bbaee retrievers, necessitating special optimizations for query inference efficiency.
Modern Transformer based LMs empower text retrievers with rich contextual modeling through the attention mechanism (Vaswani et al.] 2017), enabling deep interactions across contextual kens. This capability explains the success of recent LLMM-based dual-encoders (Wang et al. 2024b) BeehnamGhader al,, 2024), but it also means that both queries and documents incur the same com- putational burden. In contrast, traditional statistical methods such as BM25 (Robertson et al. 1994) rely purely on lexical overlap and token frequency, requiring negligible inference cost while still remaining competitive in many retrieval tasks (Thakur et al. 202T.. This contrast highlights prac- tical tension: While document encoders can afford heavy pre pre-computttion offline, query encoders must operate under strict online latency and resource constraints. Therefore, it is natural to ask: do queries truly need the same degree of deep contextual modeling as documents?
To strike balance between performances and efficiencies, propose LightRetriever, brid retrieval architecture that explicitly breaks the symmetry between document and query en- coders. Our key insight is that while documents benefit from the full modeling power of LLMs, queries do not require equally heavy processing to achieve strong retrieval performance. To this end, LighdRetriever preserves a full-sized LLM encoder on the document side, but entirely removes the deep modeling on the query side. 1) For dense retrieval, we end-to-end train cacheable token embeddings from the ducing query inference to single embedding lookup: 2) For sparse retrieval, we directly map token counts to sparse vectors without any encoding. Dense and sparse scores are linearly interpolated Wang et al. 2021) for the final hybrid retrieval.
hy-
query encoder and aggregate them through simple averaging, re-
LightRetriever achieves retrieval quality mostly comparable to full dual-encoder LLMs, but with orders-of-magnin
faster query inference. Compared to serving full-sized LLM on A800 GPU, our method achieves over a thousand times speedup for query encoding, and over 10x total throughput improvements. Importantly,
experiments show that adopting lightweight query coder does not mean queries are deprived of deep semantic understanding. Our work tries to transfer the major modeling cost from the query side to the document side, while still preserving the benefits
2

---

Published as a conference paper at ICLR 2026
Task Instruction
Query
Document Text
Given query, search
what is fruit apple?
An apple round, edible fruit produced by an apple tree.
Tokenize
Takenize
LM
[13396, 6477, ...]
[1. 1535, 29572]
Enc,
Leamingg
Enc,
[1. 1862, 20535, 16513]
Dense Query Vector
Document
Vector
Figure 2:
Contrastive training of full-sized symmetric dense retrieval. A full-sized dual-encoder used to model both queries and documents. Task-specific instructions are added as common practices (Su ct al. 2023 Wang et al. 2024b) to promote better domain adaptation abilities.
of LLM-wowered relevance-based similarity search with query-document interactions. Experiments on large-scale retrieval benchmarks demonstrate that our method generalizes well across diverse LLMs (including Llama- IB, 3B, 8B, and Qwen- 1.50, 3B, 7B) and retrieval benchmarks (including 15 English tasks from BeIR (Thakur et al. (02TT) and 8 Chinese tasks from CMTEB Retrieval (Xiao et al. 2024)), maintaining an average of 95% retrieval performance.
ALGORITHM
2.1 LM-BASED TEXT RETRIEVAL
Definition Given a search query q and candidate documents d {d+ N((q) (including irrelevant documents d' and relevant documents d' retrieved from indexes N. LM-based text retrieval aims to learn optimized encoders Ency and Enca with parameters 0, where the similarity of the query vector f4 = Fnnc, (q) and irrelevant document vectors rag- Erac(dd) are encouraged be smaller than those of relevant documents ex+ Enca(d+). Assuming the dot product used as the similarity metric, the above optimization goals can be described as the following objective,
min max Elg.d', ;0). where Elg, d', ddddddd' ;0) = 0g Uu- - 0g Udtt 0 d-
(1)
which lowers the upper boundary of ranking similarity errors & (Chen al. 2009). The above
objective can be effectively optimized by contrastive loss (Cao et al.) 2007). margin loss (Rosset et al. 2003), etc. Following (Karpukhin et al. 2020), listwise contrastive loss (gOh) is used in our work,
Vat//
CL = =-log 200 + Ed EN(g) e
where is the temperature. The retrieval capacity is strongly correlated with parametrized coders. As discussed before, serving deep parametrized IMSs especially giant LLMs, consumes large amounts of deployment resources. Thus, our work aims address this issue by designing novel asymmetric retrieval architecture with extreme imbalance.
Instructed retrieval Similar to instructed fine-tuning (SFT) (Ouyang et al. 2022). modern LM- based retrievers (Su et al. 2023 Xiao et al., 2024 Wang et al. 2024a). especially LLM retriev- ers (Wang et al. 2024b; BehnamGhader et al.) 2024, Lee et al. 2024). customize task specific instructions for better domain adaption abilities. Specifically, as shown in Figure 2 the task in- struction is prepended to the queries before retrieval. Although (Su et al. 2023 Xiao et al. 2024) use instructions on both query and passage sides, most LLM--bsed retrievers (Wang et al. 2024b
BehnamGhader al. 2024 Lee al. 2024) do not use passage instructions to the rui-b in- dexes across different tasks. In our work, dense retrieval follows the query instructions Mistral-E5
Wang et al. 2024b) and does not involve passage instructions. The sparse retrieval is incompatible
3

---

Published as a conference paper at ICLR 2026
Training
Task Instruction
Query
Document Text
Given query, search
what is troit apple?
apple round, edible fruit produced by an apple rree.
Tokenize
Tokenise
Tokeniæe
[13396, 6477, ...]
(1339, 6477, ...]
[1535]
[1]
LM Enc,
Mean
Contrastive
I Looming
LMM Enca
[1, 1862, 20535, 16513]
[13396, 6477, ...]
(29572)
Desse Query
Document
Token Vectors
Caching [13396, 6477,-. ..]
vocab size
[0]
[13396 6477,
[i]
Enc,
Cached Token Embodkiing Lookup
Index
Online Serving
29572]
Lookup
Cached Embedding Lookup
Mean Avg
Seurch
Dense
Indox
Query
Dense Query
Vector
Figure Dense retrieval of LightRetriever. Three stages are involved for efficient query model- ing. 1) Training: A shared prompt concatenated with a single query token is passed through fill encoder independently to obtain token-level representations. The full query representation is then computed by averaging the token vectors corresponding to each query token. 2) Caching: Prior to serving, the trained encoder is used to precompute token embeddings for the entire LLM's vocabu- lary, which stored in single embedding lookup matrix. 3) Online Serving: At inference time, query embeddings are efficiently generated by lookup and averaging the cached token embeddings. eliminating the need for deep model inference.
with instructions because it does not involve learnable query encoders. Detailed instructions shown in the Appendix A.7
2.2 DENSE RETRIEVAL OF LIGHTRETRIEVER
For LLM-based retrieval, LightRetriever pushes the query inference efficiency to a new extreme by completely removing the deep modeling the query side. For dense retrieval, our work end-to-end trains cacheable token vectors for online serving.
Training Given task instruction Inst and query tokens T, {to, t1, ..., fn-1). 1. each query token t (i {O...., n 13) are prepended with the instruction. Then, the corresponding query token vectors uden are obtained via last token pooling with the query encode Then, the dense query vector u of the whole query text is aggregated by averaging token vectors,
den 4
den where den = Enc, (Inst; t,)
(3)
1=0
Contrastive loss is the main objective for dense training, as defined Equation
Caching Query tokens do not deeply interact with each other. Thus, the query token vectors cacheable. Given a certain vocabulary with size V. all token vectors ofen f E {O.... ..., V 1} are precomputed as in Equation and stored in lookup Embedding F. Typically, caching such lookup Embedding on 8 H800 with Llama-8b model consumes less than 20s, which is a one-go and cheap operation. In Appendix A.4 LightRetriever also shows good ability to balance the serving Embedding size via simple dimension truncations (Kusupati et al. (022).
A customized causal mask is designed to avoid repeated prompt computations, detailed in Appendix A.8
4

---

if ş T Published as a conference paper at ICLR 2026
Training
Query
what fruir apple?
apple round, edible fruit produced by an apple tree.
FL.OPs Regnistor
Tokenize
Tokenize
[1, 1535 29572]
Directly
Project to Vocab
LM Encz
1862, 20535,
Use
Leanung
Spanse
Sparie Document
Hudden
Online Serving
Index
Seorch
Sparse
Index
Figure 4: Sparse retrieval of LightRetriever. The query representation a term-based, unlearn- able sparse vector, which is obtained from tokenization. The document representation is end-to-end trained by projecting last layer hidden states of LLM to vocabulary space, then optimized via con- trastive learning.
Online serving LLM-based encoders are no longer needed during online serving. Instead, only Embedding layer, with single weight matrix shaped [V, HJ. is hosted in RAM. H is the dimen- sion of LLM's hidden states, which is also the dimension dense vectors. The dense query vector obtained with simple bookup-then-avergee operation, which is super efficient with or without GPU acceleration.
den
den
El4
(4)
2.3 SPARSE RETRIEVAL OF LIGHTRETRIEVER
The sparse retrieval of LightRetriever steps towards lightweight even further, by completely remov- ing the need for the sparse query encoder.
Training
The sparse query vector vepr directly maps token and its corresponding quantity,
0 Number of t E T
(5)
where / {O,..., V 1} represents valid token ids within LLM'' vocabulary. We obtain the sparse document vector _pe by first feeding the document d into the pretrained language model (LM), resulting in the final-layer hidden states hast LM LM (d). These hidden states then projected into the vocabulary space to get logits w via the model's own language modeling head, parameterized by projection matrix P. To induce sparsity and suppress dominant term frequencies, we apply ReLU activation followed by a log sog-satttttion function, and then perform a max mlxx-pooling operation of the logits w over the sequence length dimension by following (Formal et al. 2021).
Spr = max(w) (w), where = logits In(max(huast P, 1) 0/
(6)
Here, o RV denotes the final sparse representation vector of the document. Given sparse vectors ni and spc the same contrastive loss in Equation 2
applied for representation learning.
2 is not naturally sparse enough, because LM tends to generate dense distributions. Thus, the spar- sify regulator needed during the training phase. Given batch of documents {do.... d, }. < batch size N. the FLOPs regulator (Paria al. 20200) is used for sparsification. This regulator first averages the logits across different documents within certain batch, then computes the squared sum to get the final regulator.
5

---

Published as a conference paper at ICLR 2026
V-1
N-
FLOOPS
(d+)
N
Online serving The sparse query vector Vi SP
during online serving follows the same rule in Equa- No more query encoder for sparse retrieval is needed.
tion S
EXPERIMENTS
3.1 SETTINGS
Training data Our work finetunes a large existing collection of 20 English and 3 Chinese datasets with 8.38M samples to ensure broad domain coverage and diversity. To ensure reproducibil- ity, work reuses the preprocessed fine-tuning data from tDRO (Ma al. (025) and Sentence Transformers Training Data. English datasets include Amazon Review (Ni et al. 2019). Eli5 (Fan et al. 2019), FEVER (Thorne et al. 2018). FiQA (Maia et al. 2018). GooAQ (Khashabi et al.
2021
HotporQA (Yang et al. 20188), MSMARCO (Nguyen et al. 2016). NFCorpus (Boteva al,
2016 NPR (Lucas et al. 2023), ), NQ Kwiatkowski et al.) 2019). PAQ (Lewis et al. 2021), Quora
Duplicates (Iyer et al. 20. S2ORC et 2020), Scifact Wadden et al.) 2020). Specter (Co- han et al. (020), StackExchange Duplicates (da Silva et al. 2018). E5 synthetic (BeastyZ] 2024).
Trivia (Joshi et al. 2017). WikiHow (Koupaee & Wang 2018), and Yahoo Answers (Zhang et al.
2013]. Chinese datasets include cMedQA2 (Zhang et al. 2018), DuReader (Qiu al. [0022] and
T2Ranking (Xie et al. 2023).
Benchmarks BeIR (Thakur et al. 2021 and CMTEB Retrieval (Xiao et al. 2024) benchmarks used in the evaluation. BelR is massive English retrieval benchmark collection with 15 evalu- ation sets. These heterogeneous datasets cover different domains, tasks, and retrieval granularities. CMTEB Retrieval is massive Chinese retrieval benchmark collection with 8 sets, also covering variety of domains. nDCG@ 10 is reported as the main metric by following (Thakur et al.] [02T]] Xiao et al. 2024), accessing both recall and ranking abilities. Recalls (R @ (20, 50, 100)) also reported to assess retrieval ability at large windows.
Training hyper-parameters Our experiments conducted based the implementation of tDR (Ma et al.) 2025). All experiments are performed with a batch size of 128. 7 hard negatives, max sequence lenghh of 512, a contrastive temperature т of 0.02, and 12k total steps. Following SPLADE (Formal et al. 2021), the coefficient of the FLOPs regulator is 0.001, and the regulator is quadratically increased the maximum the first 4k steps, which helps reduce the side effects of the regulator at initial steps (Paria et al.) 2020). LoRA (Hu et al. 2022) is used to save training GPU memory, where LoRA r 16, alpha is 32, and dropout is 0.1. Different LLMs with different sizes tested as the backbone encoders 3B, Llama-3.1-8B (Touvron et al.) 2023), and Qwen-2.5-15 1.5.1, 3B, 7B (Bai et al., 2023). The hy- brid similarity scores are linearly summed from normalized dense and sparse scores (Wang al.] 2021). All trainings are conducted on 8 NVIDIA H800 or A800 GPUs. Specifically, training a Lighteetriver-Llamaae
verify the generalization abilities, including Llama-3.2-1B,
8 H800 consumes about 15 hours.
Speed comparisons To evaluate the online speedup of LightRetriever, we compare the online time consumptions for retrieving 65536 Bing queries over 1M passages from MSMARCO (Nguyen al. 2016). All speed tests are conducted on single A800 GPU, with a batch size of 256, dimension of IK for Faiss exact search, and 64 threads for Anserini (Lucene) sparse search. Faiss and Anserini search in parallel, where Faiss searches more slowly in our cases.
Performance comparisons To comprehensively evaluate the performance cost of removing deep query modeling, we trained series of fully symmetric retrievers under identical training conditions and compared their retrieval performance degradation. Additionally, we benchmarked several recent dense, sparse, and hybrid retrievers from prior work. For dense retrieval, we report results from Sentence Transformers Static Embedding (Aarsen] (025). distilled USTAD (Kim et al., 2222)). ES- Mistral (Wang et al. 2024b), and L.L22ec (BehnamGhhaerr et al. 2024). For sparse retrieval,

---

Published as a conference paper at ICLR 2026
Comparison of retrieval speed and effectiveness for full symmetric retrievers, one Llama- 8b layer, and the query-lightweight LightRetriever. For retrieval efficiency, we report the consumed time of: query tokenization, model encoding, maximum search time of Faiss/Lucene, and total end- to-end retrieval time. The overall throughput is measured in Queries Per Second (QPS). fUsing the first Transformers layer of Llama-8b. 'Significant speedup (p & 0.01) over baselines.
Table 1:
Benchmark/ nDCG@10
Time consumption /s & Throughput / QPS
Model
BeIR
CMTEB-R
Tokenize
Encode
Search
Total
QPS
Full- balmmLl
56.8
67.6
1.3746
109 4853
8,5133
119.3730
549
Full-Llama'ss
55.6
66.1
1.3250
52.5890
8,5040
62.4180
1050
Full-Llamalb
53.1
63.8
1.2564
19,9030
8.4897
29.6490
2210
Lst-Layer of Llama8bf
50.1
54.4
1,3393
2.3409
8.1634
11.8440
5533
LightRetriever-Llmmas
54.4
63.0
0.8209
0.0412
8,5010
9,3630
6999*
include BM25 and English SPLADE-v3 Lassance el al., 2024). For hybrid retrieval, we evaluate the interpolation of Static Embedding BM25, and BGE m3 dense+sparse (Chen et al. 2024).
Notably, Static Embedding performs retrieval using single shared embedding layer for both queries and documents, offering low retrieval performance but highly efficient baseline. BM25, method, remains a widely adopted sparse retriever. The combination of them thus provides reference for efficient retrieval both sides. In contrast, BGE-m3 is a Transformer-based retriever trained with multi-stage pretraining and fine-tuning, and represents mate-of-theeart performance in hybrid retrieval. USTAD (Kim al., (024) reports small 6-layer DistilBERT retriever, offering a SOTA distillation baseline for small language models (SLM). In Appendix A.5 applications of similar distillation techniques are also discussed in our works.
classical fern-frequency-aaee
Except for Static Embedding and BM25, all compared retrievers, especially LLM retrievers, rely on deep Transformer encoders on both query and document sides. LightRetriever is, to our knowl- edge, the first method to address the online efficiency bottleneck on the query side of LLM-basee retrievers, striking balance between retrieval quality and inference efficiency.
3.2
MAIN RESULTS
Encoding speeds Table presents detailed breakdown of retrieval time for processing 65,536 Bing queries over million MSMARCO passages, highlighting the efficiency advantages of LightRetriever over full symmetric retrievers. Among all time consumptions, model encoding is the dominant contributor to latency in LLM-based retrievers. For example, Full-Llama8b and Full- Qwen7b require over 100 seconds solely for query encoding (109.5s and 100.7s, respectively), mak- ing them bottleneck for large-scale, real-time retrieval.
In contrast, LightRetriever reduces encoding time to below 50 ms 0.0420s) reaching a maximal 2500x speedup of full-sized Llama-Sb in the encoding phase. This drastic reduction is made possible by replacing full forward passes of large Transformer models with a simple embed- ding lookup. And the overall throughput achieves over 10x QPS speedup. Notably, this lightweight encoding does not significantly compromise retrieval performance. LightRetriever maintains com- petitive nDCG@ @ 10 scores both English (BeIR) and Chinese CCMTEB-Retrieval benchmarks.
potent baseline to improve LLM query encoder efficiency is layer reduction. We evaluated by training and serving only the first Transformer layer of the Llama3-88 8b model the query side under the same settings, meanwhile retaining full-sized model on the document side. While this reduces model size during training. it underperforms the LightRetriever (-4.3 on BeIR, -8.6 on CMTEB-R) and introduces inference-time overhead of 2.34s. This proves the need for full-sized query modeling during the training phase.
The above results prove the effectiveness of our extreme decoupled modeling paradigm, which dras- tically improves online query encoding efficiency while preserving retrieval quality, making it suit- able for latency-critical applications.
Overall performance comparisons Table 2 compares retrieval effectiveness on English (BeIR) and Chinese (CMTEB-Retrien) benchmarks. The main focus of this comparison is to evaluate the
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparisons BelR and CMTEB Retrieval (CMTEB-R) benchmarks. The best metrics of full symmetric retrievers and LightRetrievers are marked in bold. Their nDCG@ 10 gaps are also presented.
BEIR (15 datasets)
CMTEB-R (8 datasets)
Benchmark
nDCG@ 10
R@
R@50
R@ 100
nDCG@10
R@20
R@50
R 100
Static Emhed
34.1
44.4
53.1
59.5
31.3
45.5
54.3
60.4
BM25
41.7
48.8
56.5
61.8
50.8
63.9
70.0
74.1
Static Embed BM25
44.7
51.9
59.8
65.1
52.1
65.8
71.9
76.1
USTAD
44.2
65.3
BGE-m3daneee Mimee-ueeee
49.6
56.6
63.6
68.8
65.6
79.7
85.2
88.4
SPLADE-v3
50.2
56.6
63.3
68.6
56.6
62.8
69.8
74.9
54,4
67.0
72.3
75.6
25-Mistral7b
56.9
62.1
68.9
73.6
61.8
75.1
81.6
87.1
Full Symmetric Retrievers
Llamu3.22 2-1b
53.1
60.2
66.9
72.0
63.8
78.1
84.1
87.9
Llamu3.22 2-bb
55.6
62.9
69.7
74.7
66.
81.0
86.6
90.0
Llama3. 1-8b
56.8
64.3
71.1
75.7
67.6
82.0
87 /44
90.9
Owen2.5-1.5
$4.0
60.5
67.5
72.3
65.9
81.0
86.7
90.4
Owen2.5-3h
54.9
62.0
68.7
73.5
69.4
83.8
88.9
91.8
Qwen2.5-7h
56.6
63.6
70.5
75.2
70.1
84.4
89.4
92.4
LightRetriever
Llama3.2-1b
52.0..1
58.1
65.1
70.1
60.8.3.0
74.8
81.2
85.0
Llama3.2-30
53.5..
59.9
66.8
71.7
61.7.44
76.3
Llama3.1-88
54.4.24
60.9
67.7
72.8
63.0.46
77.3
87.4
Qwen2.5-1.50
52.1.19
58.2
70.0
63.8.21
78.3
88.3
Qwen2.5.35
$2.8
59.1
66.0
70.9
65.7
80.2
Owen2.5-7h
53.8 2.8
60.3
67.5
72.5
66.5 3.6
81.1
86.7
89.7
performance degradation introduced by LightRetriever's lightweight query encoding. Overall, by leveraging deep Transformer encoders on both query and document sides, full symmetric retrievers with deep LLM encoders achieve the best scores (e.g., Llama3. 1aama3.1-85 1-8b on BeIR with 56.8 nDCG @ 10, Qwen2.5-7b on CMTEB-R with 70.1 nDCG@ 10), but significant computational cost.
LightRetriever achieves promising effectiveness while drastically reducing query-side computa- tional burden. Across different backbones, incurs only modest degradations (typically around 5% with 1-5 absolute points). For example, LightRetriever-Quee Saaeeer--ee2.55--- achieves 53.8 nDCG@ 10 on BeIR and 66.5 nDCG@n 10 on CMTEB-R, only 2.8 and 3.6 lower than its full counterpart.
Compared with prior LLM-based retrievers, LightRetriever is also competitive: LightRetriever- Llama3.1-8h (54.4 nDCG@ 10 on BeIR) surpasses BGE-m3deeeessppe (49.6 nDCG@ 10) and ap- proaches LLM2Vec and E5- E5-Mistral, despite avoiding full query inference. Moreover, case studies in Appendix A.3 show that LightRetriever is capable of understanding complex queries, such as compositional semantics and negation, rather than simple lexical matches. Overall, LightRetriever balances retrieval quality and efficiency, offering a scalable paradigm for latency-sensitive and high- throughput retrieval scenarios.
Category- performance comparison To provide more fine-grained analysis of our query- lightweight proposal across different retrieval scenarios, we further categorize BeIR datasets into 8 task groups by following the original paper (Thakur et al. 2021) and summarize the results in Table Overall, LightRetriever shows strong robustness on general relevance- based retrieval tasks, including Bio-Medical IR, Argument Retrieval, Fact Checking, Duplicate Question Retrieval, Pas- sage Retrieval, and General QA, where consistently preserves over 93% of the full symmetric retriever's effectiveness, and surpasses it in some 2020h. This indicates that lightweight query encoding is sufficient to capture the semantic signals in typical relevance-based ad-hoe and open-domain retrieval.
(c.g.. TREC-COVID, ArguAna, and
In contrast, moderately larger performance degradations are observed on more challenging tasks, such Domain-specifie QA (HotpotQA, FiQA), Entity Retrieval (DBPedia), and Citation Predic- tion (SCIDOCS), where the retention drops to around 87-89%. Additional benchmarks on CoIR (Li al. 2025) and BRIGHT (Su et al. 2025 in A 10 also show inferior performance of query-
8

---

Published as a conference paper at ICLR 2026
Table 3: Category-wise performance comparison BEIR. We report the average nDCG 10 of
the full symmetric retriever (Llama-3.1-8)) LightRetriever (Llama-3.1-8B), and the performance
retention of LightRetriever [LighiRetiever//ull-yymmee
Task
Datasets
Full
Light
Retention
Bio-Medical IR
TREC-COVID
62.0
71.5
115.3%
Argument Retrieval
ArguAna. Touche2020
39.7
42.5
107.2%
Fact Checking
FEVER, ClimatcFEVER, SciFact
70.8
67.6
95.5%
Duplicate Question Retrieval
CQADupstack, Quora
69.6
65.3
93.8%
Passage Retrieval / General QA
MSMARCO, NFCorpus, NQ
50.1
47.0
93.8%
Domain-specific QA
HoipotQA, FiQA2018
68.4
61.1
89.3%
Entity Retrieval
DBPedia
48.2
42.0
87.1%
Citation Prediction
SCIDOCS
23.8
20.7
87.0%
lightweight design code and reasoning-intensive retrieval. These tasks typically require higher reasoning and query-understanding ability, which is not suitable for an extreme lightweight query encoder. Nevertheless, LightRetriever still maintains competitive absolute performance, suggesting promising generalization with room for further task-specific adaptation.
ABLATIONS
Our core design adopts an asymmetric architecture, with a lightweight query Embedding Bag (Emb) and a full-sized document model for inference. Specifically, we utilize full-sized model on both sides at training time, then cache all query token embeddings in one Embedding Lookup for fast qucry-side inference. ablation studies below (A1-A2), we are interested in other potential sym- metric/asymmettii architectures, which are described in the following Table
Table 4: Symmetry Ablation settings (Al-A2). Emb denotes Embedding Bag. IL-Trans de- notes a single-layer Transformer.
Table 5: Ablations (A1-2) results (nDCG@ 10) of LightRetriever.
Train
Inference
Ablations
BeIR
CMTEB-R
Query
Doc
Query
Doc
Llama-1b
48.7
58.7
Ours
Full
Full
Emb
Full
A1: Both-side Light
34.9-13.6
40.1
A1
Full Emb
Full
Emb
Emb
A2: Enc, use a Emb
37.5-11.2
41.3-17.4
Full
Emb
Full
A1. symmetric lightweight inference effective? To test the necessity of above asymmetry, we evaluate symmetric lightweight setup where both sides use simplified dense encoders. As shown in T5 this leads to severe performance drops on both BelR (-13.8) and CMTEB-R (-18.6), con- firming that full document representations are crucial for maintaining retrieval effectiveness. This symmetric setting is not functional for sparse retrieval due to the absence learnable parameters.
The best symmetric lightweight retriever is Static Embedding (one Embedding Bag) BM25 (term- based) in Table [ with nDCG@ 10 of 44.7 on BelR and 52.1 on CMTEB-R. However, due to the lack of deep modeling documents, such combination is still outperformed. We also tested reversed asymmetry with deep query encoding and lightweight documents. However, we observed training instability and degraded performance, likely caused by embedding collapse.
A2. Can we use only a query Embedding Bag for training? To confirm that a deep query en- coder is still needed during training, we replaced the Transformer-based query encoder with simple Embedding Bag that reuses only the LLM's embedding layer. This setup significantly degrades per- formance (-11.2 BelR, 117.4 CMTEB-RI -R), indicating that although deep token interaction could be limited with a durable cost during query inferencing, the full-sized modeling for query side training remains essential to learn effective representations.

---

Published as a conference paper at ICLR 2026
BEIR
20
CMITEB-R
Liamo3.188
Owen2.5 1.50
Owar2.5
Figure 5: Ablations of retrieval performances (nDCG@ 10) of LirheRerriever on BeIR and CMTEB- R benchmarks. For detailed results, please refer to Table 4 13 and 16 in Appendix.
A3. Breakdown Ablation of Retrieval Performance Figure reports LightRetriever's perfor- mance under dense, sparse, and hybrid configurations. Dense retrieval consistently outperforms sparse across all backbones (e.g.. LLaMA3, Qwen2.5) and model sizes (IB-8B). While sparse re- trieval alone is less effeettve, the hybrid approach recovers most of the performance gap with mini- mal overhead. For instance, LLaMA3-13 achieves 52.0 and 60.8 nDCG@ 10 on BeIR and CMTEB- in the hybrid setting, with only 1.1 and 3.0 points below the full model, but with much smaller query encoder. These results confirm the complementarity of both signals and the effectiveness of LightRetriever in balancing efficiency and accuracy.
CONCLUSION
Existing LLM-based retrievers use symmetric dual-encoders model both queries and documents. While documents could be pre-encoded and indexed offline, queries arrive in real-time and need online encoding. The deployment of LLMs results in inefficient online services. In this paper, we propose LightRetriever, novel LLM-based hybrid retrieval architecture capable of extremely lightweight query encoding. Our approach achieves over a 1000x speedup for query inference and 10x increase in end-to-end retrieval throughput on an A800 GPU. Experiments show great robust- ness and generalization abilities across different foundation LLMSs and retrieval tasks.
ACKNOWLEDGMENTS
This work supported by the National Natural Science Foundation of China (No. U24A20335).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Tom Aarsen. Train 400x faster static embedding models with sentence transformers, January 2025. URL punttps://hugiingee ott.gngc......ccooooo/oooooooooe co/blog//ttteoococcooooooooooooo sttttsttttiii se ngs Hugging Face Blog.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao. Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309. 2023. doi: 10.48550/ARXII.2309... 16609. URL https: doi /ttt:s://oo..org11..oo wtt.:////.....///222/ 2309. 16609
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang. Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. Sparterm: Learning term-based sparse representation for fast text retrieval. CoRR, abs/2010.00768, 2020. URL /ttp:///...ttttt/aaaaaooooo/o//.
00768
BeastyZ. E5-r. htwp:///w//agggigae...ee//attats 2025-05-11.
2024. Accessed:
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Cha- pados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. CoRR, abs/2404.0596. 2024. doi: 10.48500ARXIV.240000555 1.0.4555/arX113 24w.wxwwww..4..... 05961
/Rtpp/ts:://oo.. org/
Andrzej Bialecki, Robert Muir, and Grant Ingersoll. Apache lucene 4. In Andrew Trotman, Charles L. A. Clarke, Iadh Ounis, J. Shane Culpepper, Marc-Allen Cartright, and Shlomo Geva (eds.), Proceedings of the SIGIR 2012 Workshop on Open Source Information Retrieval, SIGIR 2012, Portland, Oregon. USA, 16th August 2012, pp. 17-24. University of Otago, Dunedin, New Zealand, 2012.
Vera Boteva, Demian Gholipour Ghalandari, Artem Sokolov, and Stefan Riezler. full-text learn- ing to rank dataset for medical information retrieval. In Nicola Ferro, Fabio Crestani, Marie- Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello (eds.), Advances in Information Retrieval 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings, volume 9626 of Lecture doi: 10. 00000/997--3319-066611 1007 9788- 3- 319 30671-166111100 58
Notes in Computer Science, pp. 712-722. Springer, 2016.
URL https: /tss:///oi..e http//oi.ooggggg/// wtt.oo/../.....//.......
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher Mamning. A large anno- tated corpus for learning natural language inference. In Lluís Mårquez, Chris Cullison-Burch, and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, pp. 632-642 Lisbon, Portugal, September 2015. Association for Computational Lin- 1/D15-1055
guistics. doi:
Daniel Campos, Alessandro Magnani, and ChengXiang Zhai. Quick dense retrievers consume KALE: post training kullback leibler alignment of embeddings for asymmetrical dual encoders. CoRR, abs/2304.01016 2023. doi: 10.48550/ARXX1.23000000 10. 01.48550/arxie 2304 01016
URL [ttlhttps:///dd org/
Zhe Cao, Tao Qin, Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Zoubin Ghahramani (ed.), Machine Learning, Proceedings of the Twenty-Fourd
International Conference (ICML 2007), Corvallis, Oregon. USA, June 20- 24, 2007, volume 227 of ACM International Conference Proceeding Series, pp. 129-139--6 136. ACM, 2007. doi: 00.1145/1233496.127333 127351
URL https: /ooi org 1ttp:/00.111//110
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality, funtiiiunctionnality multi- granularity text embeddings through self-knowledge distillation. In Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the As- sociation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting,
11

---

Published as a conference paper at ICLR 2026
August 11-16, 2024, volume ACL 2024 of Findings of ACL, pp. Computational Linguistics, 2024. doi: 10. 00.1863/V1/2044FFDIIIGG ACL.137. URL https:
Association for
//doi. 10. 1ww.3/1161vvvi 2024 ftee44..indingg findings-aci.133)
Wei Chen, Tie- Yan Liu, Yanyan Lan, Zhiming Ma, and Hang Li. Ranking measures and loss func- tions in learning to rank. In Yoshua Bengio, Dale Schuurmans, John D. Lafferty, Christopher K. Williams, and Aron Culotta (eds.), Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada, pp. 315-323. Curran Asso-
ciates, Inc., 2009. URL http:::://pooceeeinggneeeee wttp://ppppppppppp//peo/e// htc/pappe//209/aaaa// 2f55707841 110441993cc22 9ww.23dccccc. 7117701180001 9a 9a aalaa19511115 wbt.:ssttt-aaattttttt.o... html
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. SPECTER:
document-level representation learning using citation-informed transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 2270- 2282. Association for Computational Linguistics, 2020, doi: 10. 10.18663/1/20220AA00 207. httpttttp///.i.... /tt////.....111..1133331/////11/
www.111666//000 acl m07
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Anmual Meeting of the Associa- tion for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8440-8451, Asso- ciation for Computational Linguistics, 2020. doi:
URL
https: ///d/i.org//// (tl.o///...1663///// 1ww111661111000 acl main. 747
Gordon V. Cormack, Charles L Clarke, and Stefan Buettcher. Reciprocal rank fusion outper- forms condorcet and individual rank learning methods. In Proceedings of the 32nd Interna- tional ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '09, pp. 758-7878 759, New York, NY, USA, 2009, Association for Computing Machinery. ISBN 9781605584836. doi: 10.1145/1571941115222
URL https:/ http://doooorgg/./../
1571 94 1572114
Rodrigo Fernandes Gomes da Silva, Klérisson Vinícius Ribeiro Paixão, and Marcelo de Almeida Maia. Duplicate question detection in stack overflow: A reproducibility study. In Rocco Oliveto, Massimiliano Di Penta, and David C. Shepherd (eds.). 25th International Con- ference on Software Analysis, Evolution and Reengineering, SANER 2018, Canpobasso, Italy, March 20-23, 2018, pp. 572-581. IEEE Computer Society, 2018. doi: 10. 10911//......011.. 8330262. URI.tttps: //ts:://ooo doi. www.orgg//ooo /00 10..11099 1019///N/NERRR 2018 8 8330262
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:/ https://opppreeee ew. net fformm?id
m2n
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18653/v1/N199112222 1423. URL https:
7/acl ant ogy. .rw. www.or/g///ie..
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. EL15; long form question answering. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics. ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. for Computational Linguistics, 2019. doi: 10.18653/V1/P19-1113 URL Stt.http::22 /ttppp:://do.... /org/
Association
10. wt..165////pl--11111
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant, SPLADE: sparse lexical and expansion model for first stage ranking. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo
12

---

Published as a conference paper at ICLR 2026
Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada. July 11-15, 2021, pp. 2288 2292. ACM, 2021. doi: 10.1145730043533466099 URL https://
doi moi.org//00) 10. 1et..1115344443338 346309
Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval. In Proceed- ings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 981-993, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 1ttSS///122211.mmm-mmaiill 2021 emn 1aannppmaat ma...an.... 75
URL https: httpsss///caaathooooooooo org/
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. In Anna Rogers, lacer Calixto, Ivan Vulic, Naomi Saphra, Nora Kassner, Oana-Maria Camburu, Trapit Bansal, and Vered Shwartz (eds.). Proceedings of the 6th Workshop on Representation Learning for NLP, REpL-ANIP@ACI-LINN
2021, Online, August 6, 2, pp. 316-321. Association for Computational Linguistics, 2021. doi: 10.18653/V1/0011. 1..311111p111111
REPLANLP- 1.31.. URL /ttp://///////.....////....6111.
Pun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song. Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning. and Yanning Chen. Liger kernel: Efficient triton kernels for LLM training. CoRR, abs/2410. 10989, 2024. doi: 10.48550/ARXIV.241000 10999.. URL https://
dow.ddoo../..................... 4555:/1..4555//r113333 2410 10989
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
OpenReview.net, 2022. URL
Shankar Iyer, Nikhil Dandekar. and Kornél
Csernai.
First
quora dataset re- quora. com/
lease:
Question pairs, 2017. URL
Fi
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, I Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.068255 2023. doi: 1044550/ARXXV.2310000832 URL /ttt::///ol..oggg/1.4455/////// 2310.
06825
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Trans. Big Data 7(3):535-547, 2021. doi: 10.1 10.1099BBDAAAA.20992227777 URL https:
//doi.org/ 10. 1109/TBDATA, 2019 2921172
Mandar Joshi, Funsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics, 2017. doi: 10.18653/V1//17--1141 URL https:
//doi. 10. 116533/1//11-11111
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing Online, November 2020. Association for Computational Linguis- TRRUlttps::/aaaaatiolloooo org./2000yyyg//222...
(EMNLP), pp. http:///02200.0m-mmm.. tics. doi: 10.18653/v
emn 1 p- ma n, 550
Daniel Khashabi, Amos Ng. Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. Gooaq: Open question answering with diverse answer types. In Marie-Francing Moens. Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Associa- tion for Computational Linguistics: EMNLP 2021. Virtual Event / Punta Cana, Dominican Re- public, 16-20 November, 2021, pp. 421-433. Association for Computational Linguistics, 2021.
13

---

Published as a conference paper at ICLR 2026
doi: http:///12222111IIIIII EMNLP38. URL/ttpp::/ebb..or//00...1
2021. wwwediiinnssemlll
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 49944. ACM, 2020, doi: (0.1145/3397271.34000 7/doi org/ 10. 1145/ 11211111333222 3397/33977211 34C1075
URL https:
Seungyeon Kim, Ankit Singh Rawat, Manzil Zaheer, Wittawat Jitkrittum, Veeranjaneyulu Sad- hanala, Sadeep Jayasumana, Aditya Krishna Menon, Rob Fergus, and Sanjiv Kumar. USTAD: unified single-model training achieving diverse scores for information retrieval. In Forty-first In- ternational Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
Zqp
Mahnaz. Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. CoRR, abs/1810.09305, 2018. http:httt//aaaiee wtt/ 0110/0900000000
Achintya Kundu, Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti, and Mayank Mishra. Enhancing training efficiency using packing with flash attention. CoRR, abs/2407 09105, 2024. doi: 10. 48550/ARXIV.440..000 0000000............ htLLhhtt:
org/ 10. 485.04555044 arxiv 2407.09105 009105
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William howard-Snyder. Kaifeng Chen, Sham M. Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:/ httt:://papers. www.:rssinsss....ppeoooooo cc/paper files/paper/200 hhahh/ c 32221944886077111774444491188889
AallllAAbssttaac Shg.saaieeeeeaattaaaat--oo-rrrre woo-oooooeeeccc html
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452 466, 2019. doi: N0.1162/TACIILAA,.O00 00276. URL. hittps:/ //tops://od.. //o. .org/ 10.1162/tacl 00276
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flimn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pp. 61 626.. ACM, 2023. doi: 10.1145/ 3600006.3613165.. URL.tttps://dor o / 10. 1145/ 20055/36000005 3613165
Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. Splade-v3: New base- lines for SPLADE. CoRR, abs/2403.06788 2024. doi: 00445500/AXX..240000077 URL huttps://doi. www.d/i...ggg.... 48550/ arwiv. 21..x...2403.065. 06789
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist em- bedding models. CoRR, abs/2405.11428. 2024. doi: 1048550/ARXXV.2405..1444 www.xxiii......... 17428
URL
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeented generation for knowledge-intensive NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual,
htt.::/cc........e.oooo..ee//e/a Sb4493200055 77 bc 2TA.://......0011122555555701110 0r4.5220000771011123335288818823 -
14

---

Published as a conference paper at ICLR 2026
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. PAQ: 65 million probably asked questions and what you can do with them. Trans. Assoc. Comput. Linguistics, 9:1098 1115, 2021. doi: 10.1162/TACL)
A\ 00415. ht.tps::::
0. 11 62/tacll http:/taal___00111___
Shen Li, Pritam Damania, Luca Wehrstedt, Rohan Varma, Omkar Salpekar, Pavel Belevich, Howard Huang, Yanli Zhao, Lucas Hosseini, Wanchao Liang, Hongyi Jia, Shihao Xu, Sa- tendra Gera, Alisson G. Azzolini, Guoqiang Jerry Chen, Zachary Vitoto, Chaoyang He, Amir Ziashahabi, Alban Desmaison, Edward Z. Yang, Gregory Chanan, Brian Vaughan, Manoj Krishnan, Joseph S. Spisak, Salman Avestimehr, and Soumith Chintala. Pytorch RPC: distributed deep learning built tensor-optimized remote procedure calls. In Dawn Song, Michael Carbin, and Tianqi Chen (eds.), Proceedings of the Sixth Conference on Ma- chine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. mlsys.org, 2023. URL
http://sssooooopprrreeeeeeeeeeee fi htt.leespppppee/223.23 html
Xiangyang Li, Kuicai Dong. Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. Coir: A comprehensive benchmark for code information retrieval models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 22074 22091. As- sociation for Computational Linguistics, 2025. doi: 10. ongg 1072
URL
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. S2ORC: the semantic scholar open research corpus. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguisties, ACL 2020, Online, July 5-10, 2020, pp. 4969-4983, Association for Computational Linguisties, 2020. doi: 10. 10.18853//1/20200 A00///2222.AA--AAAAAA7. URL https:
//doi. 10. (01888633//2200
acl- maa-mmain.. 447
Yuxiang Lu, Yiding Liu, Jiaxiang Liu, Yunsheng Shi, Zhengjie Huang, Shikun Feng. Yu Sun, Hao Tian, Hua Wu, Shuaiqiang Wang, Dawei Yin, and Haifeng Wang. Ernie-search: Bridging cross- encoder with dual-encoder via self on-the-fly distillation for dense passage retrieval. CoRR, abs/2205.09153, 2022. doi: 10-85500ARXI.2205.0055 URL (et.hhtts:// htttps://doi.crg/11
48550/arXivg
Joel Pinho Lucas, Joao Felipe Guedes da Silva, and Leticia Freire de Figueiredo. NPR: a news portal recommendations dataset. In Lien Michiels, Johannes Kruse, Jordi Viader Guerrero, and Nava Tintarev (eds.). Proceedings of the First Workshop on the Normative Design and Evalua- tion of Recommender Systems (NORMalize 2023) co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023), Singapore, September 19, 2023, volume 3639 of CEUR Workshop Proceedings. CEUR-WSS WS.org, 2023. URL https:t://///... / ceur -saur-wws.... org. 3639/ paper f
Hans Peter Luhn. A statistical approach to mechanized encoding and searching of literary in- formation. IBM J. Res. Dev. 1(4):309-317, 1957. doi: 11114///..14.00300 URL hhtpss:
//doi.org// 10. 1147
rd. / 0309
Guangyuan Ma, Xing Wu, Zijia Lin, and Songlin Hu. Drop your decoder: Pre-training with bag- of-word prediction for dense passage retrieval. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Wash- ington DC, USA, July 18-18, 2024, pp. 1818 1827. ACM, 2024. doi: 10. 10.1145/3226772365779.. 3657799
URL https:://// /ttp::///oo..ooo org 10.
Guangyuan Ma. Yongliang Ma. Xing Wu, Zhenpeng Su, Ming Zhou, and Songlin Hu. Task- level distributionally robust optimization for large language model-based dense retrieval. In Toby Walsh, Julie Shah, and Zico Kolter (eds.), AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 -March 4, 2025, Philadelphia. PA, USA, pp. 7/doi. (aboi.org//000 /0 1 /609/ aaai v39123. 34657
AAAI Press, 2025. doi: 10.
URL http::
15

---

Published as a conference paper at ICLR 2026
Joel Mackenzie, Shengyao Zhuang, and Guido Zuccon. Exploring the representation power of splade models. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '23, pp. 143-147, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700036. doi: 10. 741144//778333366555222 URL https:
100 100...115550 11t...111//33333111111 3605112
Macedo Maia, André Freitas, Alexandra Balahur, Siegfried Handschuh, Manel Zarrouk, Ross Mc- Dermott, and Brian Davis. Fiqa: Financial opinion mining and question answering. https:
//sites. wwt.w//ww/wwwwaaaaaeeeee 2018, Accessed: 2025-05-11.
Gurmeet Singh Manku, Arvind Jain, and Anish Das Sarma. Detecting near-duplicates for web crawling. In Carey L. Williamson, Mary Ellen Zurko, Peter F. Patel-Schneider, and Prashant J. Shenoy (eds.), Proceedings of the 16th International Conference on World Wide Web, WWW 2007, Banff. Alberta, Canada, May 8- 2007, pp. 141-150. ACM, 2007. doi: 10.1145/1242572. 1242592. URL https://doi org/ 0 1 45/ / 242522222.. 1242592.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: massive text em- hedding benchmark. In Andreas Vlachos and lsabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 2006-200020 2029, Association for Computational Lin- guistics 2023. doi: 221886633112222 EACL-MAIN.LLAEE URL https:://oi. org//...... 1 18653/v1 / eacl 148
Stephen Mussmann and Stefano Ermon. Learning and inference via maximum inner product search. In Maria-Florins Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, vol- ume 48 of JMLR Workshop and Conference Proceedings, pp. 2587-2596. JMLR.org, 2016. URL http: http://proceedinge wwl.iinnnssm.......mmm mlr prwsppeesssseeo v48
wttps///vvmmmsssttnm htt//mussmmaaa /ww.mmmsssaannncaammmmmaammna 6 html
Tri Nguyen, Mir Rosenberg, Xia Song. Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d' Avila Garcez, and Greg Wayne (eds.), Proceed- ings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Anmual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Pro- ceedings. CEUR-WS.org, 2016. URL httph:tt////...... htap:p::///eeueeee wergorrwwsorg/oolloo.. C
2016 paper9.pp....
Jianmo Ni. Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeles reviews and fine-ge mne-grained aspects. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 188-197, Association for Computational Linguistics, 2019. doi: 10.18653/V1/D19-11 1018. URL htttps: http:oo../.....3////
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming- Wei Chang, and Yinfei Yang. Large dual encoders generalizable retrievers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 9844-9855. Association for Computational Btt:://V1/222..MMNPPMMMMM ewe.emmnnlln 669
Linguisties, 2022. doi:
URL https: //do:::///d...o org/
10.18653/v1
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022. URL htRp:ttt::/// http://aapaaaaaa bl (lerfeeffhee33444771114445538333 org.a0a4001.11.00000000000000000
n nips. http:ppss..pppppee.ieeee/aaeeeee htwp://weee.ppaeero....o..eoaaaa
16

---

Published as a conference paper at ICLR 2026
Biswajit Paria, Chih-Kuan Yeh, Ian En-Hsu Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. Minimizing flops to learn efficient sparse representations. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020. URL httpt:://///. pestps:://opereevieee www.//fooommmm Sywprrmmmmm.sSyyoooooooto.
Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, Qiaoqiao She, Jing Liu, Hua Wu, and Haifeng Wang. Dureader-retrieval:: A large-scale chinese benchmark for passage retrieval from web search engine. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 5326-5338, Association for Computational Linguistics, 2022. doi: 2..18653/V120022.00 EMLLL-MAIN......... URL https:
[/doi.org/// 18653//1//02202 emn mommmlp-main.. mammmain.333 357
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline
and Mike Gatford. Okapi at TREC-3. In Donna K. Harman (ed.), Proceedings of The Third Text REtrieval Confer- ence, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication, pp. 109-126. National Institute of Standards and Technology (NIST), 1994.
URL http::/ttees n mist.
Saharon Rosset, Ji Zhu, and Trevor Hastie. Margin maximizing loss functions. In Sebas- tian Thrun, Lawrence K. Saul, and Bernhard Schölkopf (eds.), Advances in Neural Infor- mation Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, De- cember 8-13, 2003, Vancouver and Whistler, British Columbia, Canada), pp. 1237-1244. CC/ htt.pp/s..//.aprr///////aaaas html
MIT Press, 2003. URL ptrpss//ppooceedngg...e....
Gerard Salton and Chris Buckley. Term-weighting approaches in automatic text retrieval. Inf. Process. Manag., 24(5):513-521 1988. doi: 00,100660306-4553388899000022220 URL https:
//doi 10.100000...... 101 000503336-55557 (88) 90021-0
Gerard Salton, Anita Wong, and Chung-Shu Yang. A vector space model for automatic indexing. Commun, ACM, 18(11))613-620, 1975, doi: 10.1145/361211.361222. URL /htpps://ooi.
1t5/100.144//31219..o 361220
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: instruction-iiettun text embeddings. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Find- ings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 1102-1121. Association for Computational Linguistics, 2023. doi: 10.18653/V1/20033. FINDINGS-ACL.77
/ http:///ooooog//////// 18 186333333883 0028886531112222 http://///../2222......gggge/e/ Findings-achh
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang. Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan Ö. Arik, Danqi Chen, and Tao Yu. BRIGHT: A realistic and challenging benchmark for reasoning-intensive retrieval. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: httttp::///peerreieewe net/ Forum?2d=yyucsq3611
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663,, 2021. http:htts:::/asvv.ottt/////
08663
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: large- scale dataset for fact extraction and verification. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies. NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pp. sociation for Computational Linguistics, 2018 doi: 10.18653/V 1074. URL https:
As-
7/doi.orgh 100 httgg//0.11865..00 www.118633v/// 1074
17

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient founda- tion language models. CoRR, abs/2302. 13971, 2023. doi: 10.488550/rxxiv2322.1999 13971. URL
http:://ddi..og///00 48550/ httpvaaiivv///.. 2302. 13971
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL
neur ips. CC httppppaprr/////eeee Abstr--bttttrt html
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang. Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 7534-7550. Association for Computational Linguistics, 2020. doi: httthhht///daae http//oi...g/o/////// 1ttp:/11.1166661. w1w53/6661l11 v02/3//v1222222/// www.eeennnpammee 609
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Multi- lingual E5 text embeddings: A technical report. CoRR, abs/2402.05672, 2024a. doi: 10.48550/
ARXIV.2402.05672. Utt http:htsss//....rggg/g// org http://....050//.....22.000
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Im- proving text embeddings with large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 11897-119 11916. Association for Computational Linguisties. 2024b. doi: 10. (0.18663/V1/2024 ACL-LONG.642. htRts:/ts:://// //doi wotg/orggllo. 10. 111.1....1111110000
acl long- 642
Shuai Wang, Shengyao Zhuang, and Guido Zuccon. Bert-based dense retrievers require inter- polation with BM25 for effective passage retrieval. In Faegheh Hasibi, Yi Fang. and Akiko Aizawa (eds.), ICTIR '21: The 2021 ACM SIGIR International Conference the Theory of Information Retrieval, Virtual Event, Canada, July 11, 2021, pp. 317-324, ACM, 2021. doi: 3111/444111588842233 3472233
00.1145/4471158.442222
URLp:ttss://oo..ooo
Yuxuan Wang and Hong Lyu. Query encoder distillation via embedding alignment is strong baseline method to boost dense retriever online efficiency. In Natise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schuster, and Ameeta Agrawal (eds.), Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Pro- cessing, SustaiNLP 2023, Toronto, Canada (Hybrid), July 13, 2023, pp. 290-298. Associa- tion for Computational Linguistics, 2023. doi: 111166//11/2233SSSSAAANN org. 1.....16633vvl wwwpe///////.////////... sustainlp-1.22
URL
ht
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL
http://ach web. www.oogg//oooo antoo hww.hhhoooyyy///ooo ogyoooyy//oo/o / 1101
Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. Retromae: Pre-training
language models via masked auto-encoder. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empiricul Methods in Natural Language Pro- cessing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 538-548. Association for Computational Linguistics, 2022. doi: 10. 10.18653/V1/2022 EMNLP-MAIN.S. URL /tatttss://oot .wroo.orrg// 10. 18653/ 22022 www.emllllp.....mmm. 1-mmalmmaaa in 35
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C-pack: Packed resources for general chinese embeddings. In Grace Hui Yang. Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM
18

---

Published as a conference paper at ICLR 2026
SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Wash- ingion DC, USA, July 14-18, 2024, pp. 641 649. ACM, 2024. doi: 10.114573626772.567777. URL https: /tt::///dooioo wtw.:oo...gooo.ooooooooooooooo.o 111.////....1112221211111 3677266777226777777777
Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xi- angsheng Li, Haitao Li, Yiqun Liu, and Jin Ma. T2ranking: A large-scale chinese benchmark Huang, Makoto P. Kato, Josiane Mothe, and Barbara Poblete (eds.), Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, doi: (8114//339968.33511938 URL https://doi org.loorggii 11411 1111111333311111 3591874
for passage ranking. In Hsin-Hsi Chen, Wei-Jou (Edward) Duh,
Taiwan, July 23-27, 2023, pp. 2690- ACM, 2023.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. 9th International Conference on Learning Representations, ICLR 2021, Viruall Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
Serumiid=zeFfgyre
Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Reproducible ranking baselines using lucene.
ACM J. Data Inf. Qual.. 10(4):16:1-1:220, 2018a. doi: 10.1145/3239571. URL https://
Goi.orr/// wtt.oogg1...11552399999
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't icchi Tsujii (eds.). Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 2018, pp. 2369-2380. Association for Computational Lin- guistics, 201 90. doi: 10. 186553/V1DD18-1220 URL https://doi.orr//D... 18653//0..633/11111 d18- 1129
Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. Multi-scal attentive interaction networks for chinese medical question answer selection. JEEE Access, 6:74061- 74071, 2018. doi:
SBW.AACCESS.201..8888877. ACCESS 2018 2883637
URL htttss://o// http://ooi..ogg///// ottg/1..110000001
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Confer- ence on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Que- bec, Canada, pp. 649-657, 2015. URL 2015/hash 2500c 815 651.167771111100aa333333.0000000
neurips. cc/ httppppppcccpp/////pp/pppp
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermen Xi- aoguang Li, Qun Liu. Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Trans. Assoc. Comput. Linguistics, 11:1114-1131, 2023. doi: 16.1/62/ A) 00595. URL https: //oi .r.g /00./10..0 21l11112eaaaaaaaa1____
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. SPARTA:
efficient open-domain question swering via sparse transformer matching retrieval. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, 1, Online, June 6-11, 2021, pp. 565-575. Association for Computational Lin- guisties, 2021. doi:
122.55//1/221.NNAACLL 18653/v1/2021 http:///2222...11a1..ae..aa maaaccl-mai.... .4.7
URL httpps://oi..rr////
Shengyao Zhuang. Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Promptreps: Prompting large language models to generate dense and sparse representations for zero-shot doc- ument retrieval. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). P Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 4375 4391. Association for Computational Lin- guistics, 2024. URL httll/ts:/aalaaatllllla 2wwp:ttthhoooooooooo22222 emn wamnnpp-maaii ma mmain.20 250
19

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A. LIMITATIONS
Our work focuses on online efficiency optimizations of recent LLM-based architectures, whose giant parameters pose a greater need for such optimizations. Other readers may be interested in the application of our methods non-LLM architectures, e.g.. BERT. While such combination is possible, due to limited time constraints and lesser needs for non-LLM, we leave this to future work. Moreover, as stated in Section $3.2 our query-lightwiight design is applicable normal relevance- based retrieval tasks. Tasks that require complex query understanding, such as reasoning-intensin retrieval, are not suitable for our work.
A.2 RELATED WORKS
A.2.1
TEXT RETRIEVAL.
Text retrieval builds on top of the vector space model (Salton al. 1975), which encodes query and document representation vectors, and performs relevance search based on vector similarities.
Dense retrieval Dense retrieval (Karpukhin al. 2020) trains PI.M-based encoders condense queries and passages into low-dimensional dense vectors, then performs relevance search based on the Maximum Inner Product Search (MIPS) (Mussmann & Ermon (0016) algorithm. In recent years, dense retrievers, such as BGE (Chen et al. 2024) and E5 (Wang et al. 20.4))) have gained popularity
for strong retrieval abilities across different tasks MMueniihooff et 2023 Thakur et al.
202
languages (Zhang et al. 2023), and context granulaties (Williams et al. 2018 Nguyen et al.
2016
owing diverse traiming data,
pre-training (Gao & Callan 2021]] Xiao al.
2022 Ma et al. 2024), improved negative-mining (Xiong et al. 2021), and so on. Recent SOTA retrievers (Wang et 2024b BeehnmmGaader al. [024]] Lee et al.) 2024) start to utilize LLMS (Jingg al. 2023 Touvron al. (2023) backbone encoders. With larger parameters and pre- training data, LLM-based retrievers enable significantly powerful retrieval capacities and domain adaptation abilities. However, LLMs also incur heavy online workloads when serving query encoders.
Improving the serving efficiency of LL.M-based retrievers is still left for exploration.
Sparse retrieval Traditional statistical sparse retrieval algorithms (c.g., TF-IDF (Salton & Buck- Ley| 1988). and BM25 (Robertson et al. 1994) do not involve Icarnable language models. Instead, they directly analyze lexical frequencies (c.g., TF) and importance (c.g.. IDF) in large vocabulary space. These lexicals are sparsely distributed and require building inverted indexes (Luhn) 1957) for feasible similarity metric calculation, such Apache Lucene (Bialecki et al.] (012) and Tan- tivy. In recent years, sparse retrieval has started to work with pre-trained LMs (PLMs). BGE (Chen) al. 2024), , SPLADE (Formal al. 202T), SparTerm (Bai al. 2020), and SPARTA (Zhao al. (Conneau et al.) 2020) or BERT (Devlin et al. 2019) as backbone encoders learn term frequencies an end-to-end manner and directly perform impact search based on the dot product of sparse vectors. Because of sparsity, these works also search with inverted indexes. Recent works (Zhuang et al. 2024) also try to incorporate LLMMs with sparse retrieval. However, the online efficiency issue still exists. Learning dynamic analysis (Mackenzie cet al.) 2023) on learned sparse retreival shows that semantic information can be fully carried on the document side regardless of token semantics. Motivated by this finding, in this paper we hypothe- size that if the semantic capacity is concentrated on the document side, then the query side can be made extremely lightweight with minimal performance degradation.
2021 use customized XLM-RooERR
Hybrid retreval Hybrid retrieval a technique that combines retrieval scores [Wang et al. 2021 or ranks (Cormack et al. 2009) from multiple retrieval systems, such as dense (Karpukhin et al. sparse (Robertson et al. 1994; Formal et al.] 2021) retrieval, and single (Karpukhin et al. 2020 multiple (Khattab & Zaharia 2020; Chen al. (024)) vector retrieval. It enables better
2020
retrieval abilities by interpolating results from multiple sources. Linear interpolation (Wang et al. 2021 of dense and sparse scores is used in work.
20

---

Published as a conference paper at ICLR 2026
A.2.2 IMPROVING INFERENCE EFFICIENCY
Numerous efforts endeavor improve the inference efficiency of text retrieval.
Distilling smaller encoders Previous works have tried to distill small query encoders from large encoders for better query inference efficiency. KALE (Campos et al.) (023) prunes the BERT-baser query encoder and distills during post-training. However, when the query encoder is pruned to one Transformers layer, its recall @20 on NQ (Kwiatkowski et al.) 2019) decreases by 25-30%. (Wang & Lyu 2023) proposes asymmetric dual-encoders by pruning BERT layers or using smaller BERT models. Its experiments show that 2-layer BERT query encoder retains 92.5% of the full- sized performances via distillation and proper student initialization. USTAD (Kim et al., 2024) also proposes similar asymmetry and distillation strategies, which choose the 12-layer BERT as teachers and the 6-layer DistiBERTT or 2-layer BERT-mini as students. Its experiments show that the distilled BERT-mini query encoder achieves 95-97% of the teacher's performance because of better distillation.
However, these studies have primarily focused on small language models (SLMs), such as BERT (Devlin et al.] 2019), and overlooked feasible approaches for improving online inference efficiency of basd retrievers, where such optimization even more critical. Furthermore, these existing methods typically retain Transformer layers in the query encoders, which limits their efficiency gains. addition, most of these works rely on SLMs and are trained on relatively narrow datasets, such MS-MARCO (Nguyen et al., 2016), thus unable to compete with LLM-based retrievers.
In contrast, with the rapid advancement LLM- text retrievers, driven by larger and more diverse foundation models and datasets, retrieval performance has significantly improved. To ad- dress the growing need for efficient online inference this setting, we propose novel method that, to our knowledge, is the first to target the online efficiency challenge in LLM-bbaed retrievers. Our approach climinates the need for Transformers layers for query encoders and does not rely on mandatory distillation. Yet our work only incurs average performance drop of about 5%. More- over, we evaluate our method across multiple LLMs of varying sizes and diverse retrieval tasks, demonstrating strong generalization capabilities.
Shrinking index sizes Previous works have also been dedicated to reducing index sizes for smaller disk space usage. Matryoshka representation learning (MRL) (Kusupati et al. (022) supports ad- justing the dense vector dimensions by directly taking the top-k dimension of the original vector as shrink vectors. It aligns and trains the corresponding multi-dimmnsiinal vectors in the end-to- end contrastive learning. For example, if k {64, 128, 256}. query- paeeage dense vectors with (64, 128, -dims could be obtained together after one encoder forward operation. For LM-based sparse retrieval, the similarity scores are directly affected by the impacts of corresponding lexi- cals (Formal et al.) (02II. Thus. it naturally supports reducing the index sizes by retaining Top-k terms/lxxccals. To explore the potential of controlling index sizes, our work also explores shrink- ing index sizes with dense MRL and sparse Top-k in the Section A.4 which also shows strong generalization capacities.
A.3
CASE STUDY
Here, we report some hand-crafted ranking cases to show LightRetriever's ability to understand complex queries rather than simple lexical matches, even if the query encoder is simplified and limited for inference efficiency.
Case Study 1: Distinguish compositional semantics. This case tests the ability to search multiple attributes together, capturing fast-moving subjects with minimal motion blur. While both candidate documents discuss camera capabilities, only Doc A contains the correct composition of features. Doc B, although relevant to photography, emphasizes low-light and landscape use cases, which do not align with the compositional semantics. LightRetriever, despite using simplified query encoder, successfully prioritizes Doe A across dense, sparse, and hybrid variants.
Case Study 2: Negation. This case tests whether models can correctly interpret affirma- tion/negation. The query explicitly seeks articles against remote work for improving collaboration.
21

---

Published as a conference paper at ICLR 2026
Table 6: Case Study 1: Distinguish compositional semantics, Texts colored in red important for semantic matching, while the blue ones represent unrelated words or distractions. v means ranking correct, while means rank fail.
Query
Which camera captures fast-moving subjects with minimal motion blur?
(relevant)
The X200 features 1/8000s shutter and phase-detection AF, enabling crisp photos of sports and fast action with minimal motion blur.
Doc B
The Y7 is praised for its wide-angle lens and superior low-light performance, often used for landscapes and night photography.
Full-sized Llama8b
Doc A
Doc B
LightRetriever waaaeerr--LaaBbaarrer Denne
Doc
Doc B
Results
LightRetriever Laaeeverr--Laaaa..--- slama&bSpprree Sparse
Doc Doc
LightRetriever' BaauRetriever-Lllmabbee Llama&b
Doc > Doc B
Table 7: Case Study 2: Negation.
Query
Articles arguing against remote work for improving team collaboration.
Doc A (relevant)
A recent study argues that remote work hinders spontancous collaboration, noting dropoff's in serendipitous conversations and mentoring opportunities among junior staff.
Many experts advocate remote work for flexibility and increased individual produc- tivity, highlighting asynchronous tools that maintain collaboration.
Full-sized Llamm8bb
Doc
LightRetriever-LJJee www.rrieeeee-aaaabaa....o. Jamasb
Doe > Doc B
Results
LightRetriever weweeetereee--aaaaaaeee Llama&b Sparse
Doc A > Doc B
LightRetriever wewaeeer---aabbb-eeeer ghrid
Doc Doc B
Although both candidate documents discuss remote work and collaboration, only Doc aligns with the negative phase by describing how remote work hinders interactions. Doc B, on the contrary, ad- vocates for remote work, which semantically opposite to the query intent. Correctly distinguishing the affirmation/negatio
requires more than lexical overlap with remole work and collaboration. As shown in the results, both the full-sized Llamasb and all LightRetriever variants successfully rank Doc A higher, demonstrating that LightRetriever can preserve such negation discrimination ability.
Table 8: Case Study 3: Complex understanding.
Query
How to revert the last commit without losing changes?
Doc (relevant)
Use git reset -soft HEAD undo the last commit while keeping changes in the working tree and index; this retains your edited files further edits.
Doc
Use git revert HEAD to create new commit that undoes the changes introduced by the previous commit, preserving history and not altering the working tree.
Full-sized Llama8h
Doc > Doc B
LightRetriever- Lta.rrerr-Laaaaaaeeeee..
x Doe A Doc B
Results
Leaeleettrieee--lmaall Sparss
Doc Doc B
Doc
Case Study 3: Complex understanding. This case tests the ability to understand complex tech- nical instructions. The query asks how revert the last Git commit without losing the changes, subtle requirement that distinguishes between keeping edits in the working tree (Doc A) versus cre- ating a new commit that reverts prior changes (Doc B). Despite both documents mentioning ways to undo commits, only Doc A satisfies the instructions. The full-sized Llama8b is able to com- plete such complex technical instructions. However, LightRetriever- Denne and Hybrid fail, likely because they focus the strong lexical signal of revert but neglect the constraint without losing the changes. Interestingly, LightRetriever-Sppe Sparse ranks Doc A correctly, suggesting that token-level sparse matching may emphasize critical keywords such keeping changes and working tree. This
22

---

Published as a conference paper at ICLR 2026
highlights a limitation of lightweight dense encoding: complex compositional understanding may fail, and success may depend on complementary sparse signals.
A.4 ADDITIONAL RESULTS ABOUT CONTROLS OF VECTOR DIMENSION, SPARSITY, AND EMBEDDING SERVING SIZE
Retrieval systems usually involve indexing millions of documents into search databases, which con- sumes large amounts of disk space, especially for dense vectors. For example, indexing 8.8M MS- MARCO passages (Nguyen et al. 2016) with the Llama-8b model consumes 134.92 GB for Faiss dense retrieval and 11.24 GB for Lucene sparse retrieval. Besides index sizes, LightRetriever" dense part also serves as an Embedding matrix in RAM with shape [V, H] in Float16 precision. For the Llama-8b model, if the serving 1.05 GB of RAM.M.
not properly controlled, such online embedding will consume
Thus, it's essential to explore the possibility of controlling index sizes and serving sizes by shrinking the vector dimensions. 1) For dense retrieval, we incorporate Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) with LightRetriever to support flexible controls of dense embedding dimensions. Given dense vector den with dimension H, MRL derives dimension-shrunk vectors by simply cutting out the Top-k dimensions deen(: At]. then applics the same training loss as the original vectors. MRL trains collection of Top-k dimensions together, enabling flexible choosing of a proper k. 2) For sparse retrieval, the sparse vector directly learns the term impact (i.c., lexical frequency) within large sparse dimension. Thus, LightRetriever supports controlling sparsity by directly taking the Top-k terms, where the larger impact terms matter more.
BEIR
CMTEB-R R
60
70
Performance /DDC0@
60
45-
50
Symmetric Baseline:
Symmetric Baseline
40-
LightRetsiever
LightRetriever Dease
LightRetriever Sparse
40
LightRetrieveir Sparse
LightRetriever Densh+
LightRetriever Sparse
1000
2000
2500 Top-k
3000
4000
1500
2500 3500 4000
Top-k
Figure 6: Performances (Llama-3.1-8b) with different Top-k dimensions or sparsities.
Table 9: Performances, dense serving size, and index size of LightRetriever-Llamm88 across differ- ent top-k dimensions (Dim). Note that dim=4096 is the untruncated setting.
Dimension
Performance nDCG@ @
Serving Size (MB)
Index Size (GB)
Top-k
BEIR
CMTEB-R
Embedding
Dense
Sparse
49.3
55.2
32.83
3.03
256
51.6
57.5
65.67
8.43
5.42
512
53.0
58.9
131.33
16.87
8.89
1024
53.1
59.8
262.67
33.73
10.85
2048
53.3
60.4
525.34
67.46
11.22
4096
54.4
63.0
1050.67
134.92
11.24
Index size controls via tuning vector dimension and sparsity We conduct experi- ments of controlling dense vector dimension and sparse vector sparsity with Top-k of {128, 256, 512, 1024, 2048, 4096) on Llama-3.1-8h As presented in Figure/aam Table LightRe- triever has affordable performance cost when truncating the Top-k. The lowest Top-k setting (128)
23

---

Published as a conference paper at ICLR 2026
has 5.1 and 7.8 nDCG@ 10 performance gaps on BelR and CMTEB-R R benchmarks compared the highest Top-k (4096), but its disk usages with MS ASSMARCO are only 4.22 GB for dense trieval and 3.03 GB for sparse retrieval. This shows that our LightRetriever is capable of flexible index controls via simple vector shrinking techniques, with limited effect on retrieval performances.
Dense embedding serving size controls via dimension truncation By applying dimension trun- cation with MRI., LightRetriever also allows serving embeddings flexible sizes proportional the selected Top-k dimensions. As is shown in Table truncating from 4096 to 1024 dimensions re- duces the online embedding size from 1.05 GB only 262 MB, 4x reduction with affordable per- formance loss (53.1 vs. 54.4 nDCG@ 10 on BEIR). Under the extreme truncation to 128 dimensions, the serving size reduced just 32 MB, making it practical to deploy LightRetriever on resource- constrained environments such as memory-limite servers. This demonstrates that LightRetriever supports cost-effective online serving with controllable degradation in retrieval quality, offering a scalable solution for balancing efficiency and effectiveness in real-world deployments.
A.5 ADDITIONAL ABLATION ABOUT THE EFFECT OF AUXILIARY KL LOSS
Table 10: Ablations (nDCG@ 10, w/LLama-3.111 3.1-8b) on BelR and CMTEB-R about auxiliary KI. loss.
w/ KL
wo/ KL
Benchmark
BeIR
CMTEB-R
BeIR
CMTEB-R
Dense
52.0
61.4
50,8-1.2
60.
Sparse
49.5
54.8
48.7
54.4_o.4
Hybrid
54.4
53.9
62.1-0.9
Previous works for BERT-based asymmetric retrievers (Wang & Lyu 2023 Kim et al. 2024) empha- size the importance of alignment between asymmetric students and full symmetric teachers. To plore the effect of such alignment, besides contrastive loss as the main loss function, we also explore adopt the KL loss as an auxiliary align the asymmetric similarity scores 5student den den to full-sized symmetric similarity scores Steacher = Ency(g) (g) Enca(d).
v
Aliggn KL- (enDivvSSttuue -KI-DDvvSSfudeer (Sttudent gteacher)
Results in Table show that removing such loss ineurs limited drops of -0.5 and -0.9 on BelR and CMTEB-R. Thus, such auxiliary KL loss is not mandatory needed in our experiments.
Table 11: Training Dataset informations.
Dulasel
Language
Calegory
Deduped Size
Epech
Ratio
Amazon Review (2018)
(N 2019
English
Amazon
0.01
0.58%
cMeQAA2 (Zhung
Chinese
Chinese Medical
DuReader (Vi a.. 2021
Chinese
Chinese Web Collections
86366
5.05%
Elis Fan L
English
Reddit
325390
0.05
0.95%
Fever Thorie ai
2018
English
Wikipedia QA
109808
0.8
5.13%
FIQA
English
Financial
5498
0.32%
GooA(] Parts [K (ilasllat
2021
English
Web Collections
3012347
0.025
4.40%
HorporQA Ydngg
English
Wikipedia
85000
2.48%
MSMARCO (puyeh pulyen er 21
26
English
Web Collections
502854
29. 39%
NFCogpus (Boxeva
English
Modical
2585
0.15%
NPR (Lacak e 21/23
English
News
594376
0.05
1.74%
NQ [Kwnal
2019
English
Wikipedia QA
58800
3.44%
PQA Pirrs LLowis et ml
2121
English
Wikipedin QA
99999
0.5
2.92%
Quorn Duplcatex Iriplex [yer
2017
English
Forum Duplicates
97011
0.5
2,83%
$20RC S2ORCTHile-Abstnee Title-Abstrai [L 2020
English
Semantic Scholar
99998
0.5
2,92%
SciFact Wauklen el
20/00
English
S2ORC
806
0.05%
SPECTER (Coban al. 20200
English
Semantic Scholar
136642
0.25
2.00%
StackExchangg Body--Bdd) (da Silva al. 2018
English
Forum Duplicatos
250516
0.25
3.66%
ES Synthetic [BeastyZ] 2024
English
GPT
Synthetic G
Data
224791
13.14%
T2Ranking [iv 20353
Chinese
Chinese Web Collections
200362
0.5
5,86%
Trivin [Tosh am 2011
English
Wikipedin QA
0.5
1.76%
Wikihow (Koupass Wang 2018
English
WikiHow
128543
0.25
1.88%
Yahoo Answerk (TRN-A115We Ziag 2015
English
Yahoo
1198018
3.50%
24

---

Published as a conference paper at ICLR 2026
A.6 TRAINING DATASETS
Dataset infos Following the settings of previous LLM-based retrievers (Wang et al. 2024b BehnamGhader al. 2024, Ma et al. 2025). our work uses 23 training sets pre-built by tDRO (Ma et al. 2025) or Sentence Transformers Training Data, including 20 English and Chinese datasets. Detailed dataset information is listed in Table follows.
Many downstream retrieval benchmarks (Thakur et al.) [021]] Xiao et al. 2024) are evaluated out- of-domain, where no training samples from the downstream sources are available. To avoid the overfitting of training sets, is shown in the above Table [1] we slightly tune the epoch used in
the dataset sampling, by following previous works in (Ni et al. 2022 Wang et al. 2024b Ma et al,
2025).
Table 12:
Training set deduplications. We report each dataset with the original size, deduplicated size, and number of duplicates detected in both the train and test sets.
Dataset
Original Size
Deduped Size
Duplicates
Amazon Review (2018)
1000000
999999
cMedQA2
100000
99936
DuReader
86395
86366
29
Eli5
325475
325390
85
Fever
109810
109808
FiQA
5498
5498
GooAQ Pairs
3012496
3012347
149
HotpotQA
85000
85000
MSMARCO
502939
502854
85
NFCorpus
2590
2585
NPR
594384
594376
NQ
58812
58800
12
PQA Pairs
100000
99999
Quora Duplicates Triples
101762
97011
4751
S20RC Title-Abstrad
100000
99998
SciFact
809
806
SPECTER
136645
136642
StackExchange Duplicates (Title-Body)
250519
250516
E5 Synthetic
224791
224791
T2Ranking
200376
200362
14
Trivia
60380
60370
Wikihow
128543
128543
Yahoo Answers (Tille-Answer)
1198260
1198018
242
Table 13: Samples of duplicate queries found both the train and test sets.
Train Set
Train qid
Train Query
Test Set
Test qid
Test Query
Quora
train-164
Is there frame-
Quora
201573
Is there frame- work for auditing social media?
work for auditing social media?
YaswerAnswer
train- 10046
who is sachin ten- dulkar?
Quora
47702
Who is Sachin Ten- dulkar?
MSMARCO
968274
where did abraham lincoln died
DBPedia
QAL.D2 tr-6
Where did Abra- ham Lincoln die?
Fli5
train-3088
How does ployment insurance work?
FIQA
2648
How does playment insurance work?
Training set decontamination Training set decontamination is essential for developing fair, un- biased, and robust retrieval systems. Existing training datasets are derived from multiple sources, whose training data have possible duplicates with the test sets. Following tDRO (Ma et al.] 2025).
25

---

Published as a conference paper at ICLR 2026
perform strict training set decontamination with SimHash (Manku al. 2007 to ensure training queries appear in the BeIR (Thakur et al., 2021 and CMTEB Retrieval (Xiao et al. 2024).
The original sizes, deduplicated sizes, and duplicate numbers listed in the Table
Additionally, we also listed some samples of the duplicated training queries, as well as the corre- sponding test sets in Table 13
A.7 TASK INSTRUCTIONS
As previously mentioned in Section 2.1 modern LLM-based retrievers (Wang et al., 2024b
BehnamGhader et al., (024) introduce task-specific instructions to format input queries, which enables higher retrieval capacities. These LLM-based instruction-tuned retrieval is similar to instruction-base supervised fine-tuning (SFT) in LLM generation (Ouyang et al. (022). Following Mistral-E5 (Wang et al. 2024b), reuse most of its task instructions, and format the instruction and query in the following format:
Instruct: instruction nnQury: query
The detailed instructions are listed below, most of which are directly copied from the previous works M5-Miata (Wang et al. (224b) and tDRO (Ma et al. al.] 2025) for reproducibility. Note that one training set may correspond to multiple instructions to avoid overfitting.
Instructions for training sets
Amazon Review (2018)(1): Given a title, retrieve the corresponding reviews from Amazon
Amazon Review (2):8) Given a title, retrieve Amazon review
cMedQA2: Given a Chinese community medical question, retrieve replies that best answer the question
4. DuReader: Given a Chinese search query, retrieve web passages that answer the question
Eli5: Provided a user question, retrieve the highest voted answers on Reddit ELI5 forum
GooAQ Pairs: Given a web search query, retrieve the corresponding answers from Google
HotpotQA: Given a multi-hop question, retrieve documents that can help answer the question
8. MSMARCO: Given web search query, retrieve relevant passages that answer the query
NQ(1): Given question, retrieve Wikipedia passages that answer the question
10. NQ(2): Retrieve Wikipedia passages that answer the question
11. Quora Duplicates Triples(1): Given a question, retrieve questions that are semantically equiva- lent to given question
12. Quora Duplicates Triples(2): Find questions that have the same meaning as the input question
13. S2ORC Title-Abstraet()) Given a title, retrieve the abstract from scientific papers
14. S2ORC Title-Abstract()): Given title, retrieve abstracts from scientific papers that match the title
15. SciFaet: Given a scientific claim, judge whether the document supports or refutes the claim
16. SPECTER(1): Given a title, retrieve semantic related titles
17. SPECTER(2): Retrieve semantic related titles from scientific publications
18. StackExchange Duplicates (Title-Body): Retrieve duplicate questions and passages from
StackOverflow forum
19. T2Ranking: Given a Chinese search query, retrieve web passages that answer the question
20. Trivia(I): Given a question, retrieve Wikipedia passages that answer the question
21. Trivia(2): Retrieve Wikipedia passages that answer the question
22. Wikihow: Given a summary, retrieve Wikipedia passages that match the summary
23. Yahoo Answers (Title-Answer): Given title, retrieve Yahoo answers that match the title
26

---

Published as a conference paper at ICLR 2026
Instruction for English BeIR benchmarks
1. ArguAna: Given claim, find documents that refute the claim
2. ClimateFEVER: Given claim about climate change, retrieve documents that support refute the claim
3. CQADupStack: Given a question, retrieve detailed question descriptions from Stackexchange that are duplicates to the given question
4. DBPedia: Given query, retrieve relevant entity descriptions from DBPedia
5. FEVER: Given claim, retrieve documents that support or refute the claim
6. FiQA2018: Given financial question, retrieve user replies that best answer the question
HotpotQA: Given a multi-hop question, retrieve documents that can help answer the question
MSMARCO: Given a web search query, retrieve relevant passages that answer the query
9. NFCorpus: Given a question, retrieve relevant documents that best answer the question
10. NQ: Given question, retrieve Wikipedia passages that answer the question
11. Quora: Given question, retrieve questions that semantically equivalent the given question
12. SCIDOCS: Given scientific paper title, retrieve paper abstracts that are eited by the given paper
13. SciFact: Given a scientific claim, retrieve documents that support or refute the claim
14. Touche2020: Given a question, retrieve detailed and persuasive arguments that answer the ques- tion
15. TRECCOVID: Given a query on COVID-19, retrieve documents that answer the query
Instruction for Chinese CMTEB Retrieval benchmarks
1. Cmedqaaetrieeal:: Given a Chinese community medical question, retrieve replics that best an- the question
CovidRetrieval: Given question on COVID-19, retrieve news articles that answer the question
DuRetrieval: Given a Chinese search query, retrieve web passages that answer the question
4. EcomRetrieval: Given a user query from an e-commerce website, retrieve description sentences of relevant products
MMarcoRetrievall: Given a web search query, retrieve relevant passages that answer the query
6. MedicalRetrieval: Given a medical question, retrieve user replies that best answer the question
7. T2Retrieval: Given a Chinese search query, retrieve web passages that answer the question
8. RetioRetrieval: Given video search query, retrieve the titles of relevant videos
A.8 EFFICIENCY OPTIMIZATIONS TECHNIQUES
Customized causal mask As defined in the Equation 3 our work computes dense token vectors for each input token with common prompt. This causes the repeated computation of the common prompt area. We develop a customized causal mask for LLMS to avoid such computation waste. Specifically, given task instruction Inst and query tokens TĄ {to, tn,.... we format the instruction and query tokens as follows,
Input IDs <bos> Inst to <eos> ... fn-1 <eos>
The customized causal mask ensures that the token blocks can attend to the common prompt area, but not attend each other. Let the sequence length be L, prompt length be P, micro block (each token + eos) width be w, and let g E [o, L - 1] denote the query index, E [0, 1] the key index. The attention mask (0 means attend, means masked) is defined as,
27

---

Published as a conference paper at ICLR 2026
Custom Causal Attention Mask
Prompt = 3, Seq Length 11, Block Width
0
2
Query index (q)
3
5
6
7
8
9
2
3
4
8
9
10
Key index (k)
Figure 7: An example of the customized attention mask.
0,
q < and
S
(attn within prompt)
0,
if 2 P and
k < P
(blocks can attend to prompt)
Mask ig, k]
0.
i 2 P and 9 Q-P
k-P and k S (atin within current block)
oc, otherwise
An example of the customized causal mask is shown in Figure
Optimizations of sparse max aggregation As stated in Equation the sparse vector is computed
via
Project to vocab with dim V > ReLU - Log Saturation + 4. Max Aggregation (10)
where the vocabulary dimension is much larger than the hidden dimension H. For example, for Llama3.1-8 (Touvron et al. 2023 model, V == 128256 and H 4096, where [V/H] 31. This means that, given batch size B, sequence length T. hidden size H, vocabulary size V, the inter- mediate tensors with shape [B, T, VJ in the projection, ReLU, and Log 31x larger than dense tensor [B, T, H). which consumes huge amounts of GPU memory during training and inference. However, only a tensor with shape [B, VJ needed for the final aggregated representations. Thus, an optimization for computing the sparse vector is needed urgently.
We design an efficient way of computing the sparse vector. Firstly, because ReLU and Log are monotonically increasing functions, we first move the last max aggregation operations to step 2, right after the projection.
1. Project to vocab with dim V + 2. Max Aggregation + 3. ReLU - 4. Log Saturation (11)
Then we design customized PyTorch Autograd function to fuse steps 1&2. It computes the pro- jection and maximum operations together along the sequence length dimension T. Such sliced computation avoids the creation of large tensor with sequence length dimension. Our prelimi- nary observation shows that this helps save the GPU memory usage of the sparse max aggregation up 100-200x. The forward and backward presented in the Algorithm
28

---

Published as a conference paper at ICLR 2026
Algorithm Forward and Backward Pass of Efficient Projection then Max aggregation
Require:
Input tensor X E I [XXI* weight W E # optional bias b E IRY, optional input attention mask M {0, 1] BxT, where B is batch size, is sequence length, H is hidden size, is vocabulary size.
Ensure: Output tensor E RB
1: function FORWARD(X, W, b, M)
2: Initialize Y <- 00
3: for t = 0 to T do
4: Le + X t
5: if is not None then
6:
L x L b
7:
end if
8:
if M is not None then
9:
for i to B do
10:
if [i. [i 4
= 0 then
12:
end if end for end if
14:
Y + max(Y, Lc)
16:
end for
17:
Save argmax, indices as maxiiddices
18: return Y
19:
end function
20:
function BACKWARD(VV, max_indices, X, W)
21:
Initialize VX + 0, VW + v
22:
for /
0 to 7 do
23:
X, + X :[ BxH
VX, VX[,,t,;] E R
25:
M, < max mraxinddiess
E 10
VT, < VY o M, E BRV
>> Only keep grads where was max
27:
VX, += VL. WT
28:
VW += X, VL.
29:
Vb += E [
end for
31: return VX, VW, Vb
end function
Other important optimizations Proper optimizations are important. For example, we found hard reproduce quickly with some of the open-sourced baselines (BehnamGhader et al.) 2024), which is roughly 6x slower than our implementation when testing with the same size of LLM. Our experiments involve training with 23 train sets and inference with another 23 test sets, which requires proper optimizations urgently. We import other optimizations based on the implementations released by IDRO (Ma al. 2025). These optimizations include 1) distributed multi-node inference framework based on Pyrorc RPC (Li et al., 2023); 2) Sequence packing (Kundu et al. (024)) with Flash Attention 2 (Dao 2024). which gives roughly 50% training speedup due to the removal of the pad tokens.; 3) Fused Kernels from Liger- Kernel (Hsu et al.) (2024), which gives roughly 10% speedup to both training and inferencing; 4) Multi-vector gradient accumulation support with GradCache (Gao et al. 202T) for optionally enlarging batch sizes if necassary.
29

---

Published as conference paper at ICLR 2026
A.9 DETAILED MAIN RESULTS
The detailed main results (nDCG@ 10) on each dataset are shown in the tables below, which are full versions of Table
Table 14: (Partl) Detailed main results (nDCG@ 10) on BelR test sets (except MSMARCO, it uses the dev set).
Task Test, nDCG@10
ArguAna
CQADup
CFEVER
DBPedia
FEVER
FiQA
HotpotQA
MSMARCO
Statie Embedding
44.4
20.4
27.4
46.8
17.9
BM25
31.5
29.9
21.3
31.3
23.6
60.3
22.8
USTAD
34,9
30.6
22.5
35.9
76.9
29.5
56.0
46.6
BGE-m3-dense
26.5
38.5
68.6
37.6
BGF-m33-spase
36.2
30.5
26.7
24.4
85,9
26.9
70.3
17.1
SPLADE-13
50.9
34.7
23.3
45.0
79.6
37.4
69.2
46,8
L.LM
71.8
43,2
E5-Mistral
61.9
42.9
38.3
48.9
87.8
56.6
75.7
43,1
1.1ma3.2-1b
Full Symmetric LightRetriever Dense
58.0
44.0
41.2
43.2
69.3
42.7
37.9
34.2
60.8
39.2
LightRetriever Sparse
49.5
34.5
27.5
35.9
84.7
35.7
65.5
37.2
LightRetriever Hybrid
54.1
39.7
39.3
39.5
42.1
68.2
41.1
1amm332--b
Full Symmetric
58.5
46.8
43.7
46.4
89.9
53.4
73.7
44.3
LightRetriever Dense
55.3
40.3
41.7
36.5
84.0
46.2
64.3
41.6
LightRetriever Sparse
46.5
26.9
38.5
66.3
39,2
LightRetriever Hybrid
56,3
41.4
42.4
41.6
87.6
46.9
69.7
42,9
L-mm
Symmetric
57.1
49.2
44.1
48.2
77.8
46.2
LightRetriever Dense
55.8
41.5
38.7
67.6
43.4
LightRetriever Sparse
46.4
33.6
32.2
38.8
86.1
41.9
67.9
41.0
LirevRetriever Hybrid
57.2
42.4
42.0
71.5
Mistral0.3-7b
Full Symmetric
56.2
49.7
43.6
48.4
91.3
59.6
77.7
46.6
LightRetriever Dense
53.5
41.6
40.9
38.9
85.6
50.9
66.9
44.2
LightRetriever Sparse
44.0
33.6
21.9
37.0
43.6
65.4
40.9
LightRetriever Hyhrid
54.3
42.2
41.4
40.5
86.9
51.7
69.9
44,8
Qwen2.5-1.5b
Symmetric
44.1
40.4
43.9
47.9
42.6
LightRetriever Dense
51.2
38.3
36.1
33.0
82.1
41.2
59.0
39.1
LightRetriover Sparse
46.5
35.0
25.7
36.3
85.1
35.3
63.3
37.3
LightRetriever Hyhrid
Qwen2.5-3b
Full Symmetric LightRetriever Dense
55.5
47.0
44,1
45.8
89.6
52,9
71.6
43.7
49.8
39.6
41.1
35.0
83.9
62.8
41.0
LightRetriever Sparse
45.5
36.2
28.7
36.9
84.9
38.6
64.2
38.7
LightRetriever Hyhrid
52.8
41.4
42.1
39.8
86.9
46.1
68.2
42.3
Qwen2.5-7b
Symmetric
74.7
LighiRetriever Dense
53.3
40.0
39.1
36.8
85,1
48.8
64.4
41.8
LightRetriever Sparse
47.5
35.6
28.2
36.7
85.4
40.2
65.2
40..
LightRetriever Hybrid
54.4
41.0
43.3
30

---

Published as a conference paper at ICLR 2026
Table 15: (Part2) Detailed main results (nDCG@ 10) on BelR test sets.
Task Test, nDCG@10
NFCorpus
NQ
Quora
SCIDOCS
SeiFact
TRECCOVID
Touche2020
Avg (# 15)
Static Embedding
30.0
77.5
13.2
59.3
44.6
22.4
34.1
BM25
32.5
32,9
78.9
15.8
66.5
65.6
36,7
41.7
USTAD
30.7
81.4
14.4
55.5
72.3
24.7
44.2
BGE-m3-dense
31.5
88.4
15.2
63.6
47.0
20.6
47.2
BGE-m3-sparse
28.3
20.3
73.4
12.2
63.3
52.7
27.8
39.7
SPLADDE-13
35.7
71.0
29.3
LLM2Vec-lamaa--8
41.8
64.2
87.2
23.0
78.2
80,3
20.5
56.6
E5-Mistral
38.6
63.5
89.6
16.3
76.4
87.2
26.4
$6.9
1.ama3.2-1b
Full Symmetric
35.5
58.6
89.4
19.8
72.3
66.7
20.9
53.1
LightRetriever Dense
30.1
52.2
87.0
17.2
66.5
68.8
24.7
48.7
LightRetriever Sparse
34.0
82.2
17.2
60.6
27.8
47.6
LightRetriever Hybrid
34.2
56.6
87.4
18.6
71.5
70,5
30,2
52.0
Llama3.2-3b
Symmetric
37.6
89.5
22.5
75.3
21.2
55.6
LighiRetriever Dense
31.7
56.4
87.8
19.0
70.4
68.6
21.6
$1.0
LightRetriever Sparse
33.3
55.3
83.2
17.7
67.3
64.1
28.5
48
LighiRetriever Hybrid
70.9
29.5
1.ama3.1-8b
Full Symmettic
38.6
65.5
89.9
23.8
77.3
62.0
22.2
56.8
LightRetriever Dense
88.5
LightRetriever Sparse
33.6
58.4
84.4
18.9
69.7
63.0
27.1
49.5
LighlRetriever Hybrid
34.6
61.9
88.6
20.7
72.6
71.5
27.8
54.4
Mistral0.3-7b
Full Symmetric
36.8
66.9
90.0
22.7
77.7
71.3
23.6
57.5
LightRetriever Dense
31.9
$9.8
88.7
18.5
70.3
71.5
26.7
52.7
LightRetriever Sparse
33.1
LightRetriever Hybrid
33.8
62.5
$8.8
19.8
71.6
76.7
32,4
54.5
Qwen2.5-1.5b
Symmetric
36.4
89.0
20.4
72.7
73.0
28.4
$4.0
LightRetriever Dease
29.8
50.9
86.7
18.1
66.5
72.6
29.1
48.9
LightRetriever Sparse
32.9
51.6
81.3
17.4
66.4
64.8
31.1
47.3
LightRetriever Hybrid
87.0
19.3
69.2
76.8
32.3
$2.1
Qwen2.5-3h
Full Symmetric LightRetriever
35.9
61.4
89.5
22.3
72.9
65.7
26.0
$4.9
LighiRetriever Sparse
33.2
54.2
82.2
17.6
67.6
57.0
26.4
47.5
LightRetriever Hybrid
34.2
58.5
87.8
19.5
69.1
74.2
29.8
$2.8
Qwen2.5-7b
Symmetric
37.2
89.7
23.7
77.2
25.3
$6.6
LightRetriever Dense
30.7
55.9
87.6
19.5
67.9
64.1
24.1
50.6
LightRetriever Sparse
33.9
78.7
69.0
28.5
48.2
LightRetriever Hybrid
34.4
60.1
87.9
21.1
71.1
73.1
29.4
53.8
31

---

Published as a conference paper at ICLR 2026
Table 16: Detailed main results (nDCG@ @ 10) CMTEB Retrieval dev sets.
Task Dev, aDCG@10
Cmedqu
Covid
DuReader
Ecom
MMarco
Medical
T2
Video
Avg 8)
Static Emheeding
7,5
45.7
37.9
33,9
34.9
15.7
35.8
38.9
31.3
BM25
13.7
86.6
57.1
45.1
48.3
32.1
60.5
62.7
50.8
BGE-m3-dense
77.2
82.9
57.5
76.9
51.5
80.8
BGG--m3-sparse
24.5
76.0
71.4
50.3
59.2
44.0
71.7
58.5
57.0
LLM22ec-llamu3-B
35.2
16.5
80.7
54.4
76.5
54.6
65.2
52.4
54,4
E5-Mistral
34.2
73.1
87.0
46.0
74.8
52.8
80.7
45.4
61.8
Llama3.2-1b
Full Symmetric
32.3
74.2
84.6
53.8
73.5
50.5
78.0
63.1
LightRetriever Dense
27.2
67.6
80.8
$1.0
67.9
43.0
73.4
58.5
LightRetriever Sparse
17.3
64.3
74.4
$1.0
63.3
39.4
69.3
52.2
53.9
LightRetriever Hybrid
27.2
69.7
81.8
55.5
69.8
44.1
75.9
62.3
60.8
Llama3.2-3b
Full Symmetric LightRetriever Dense
36.8
75.6
87.9
57.7
75.7
53.8
81.1
60.5
66.1
29.8
70.2
84.0
51.7
69.2
46.5
75.7
LightRetriever Sparse
17.2
65.4
76.1
47.4
65.0
42.5
68.5
42.0
53.0
LightRetriever Hybrid
29.9
71.5
84.8
54,5
71.2
47.2
77.1
57.7
61.7
Llama3.1-8b
Full Symmetric LightRetriever Dense
39.5
77.1
89.5
58.9
78.1
57.6
84.1
56.2
67.6
32.1
71.0
85.7
53.0
70.2
49.0
78.5
61.4
LightRetriever Sparse
70.4
82.9
47.3
42.2
LightRetriever Hyhrid
32.1
72.8
86.6
55.3
71.6
49.7
80.1
55.8
63.0
Mistral0.3-7b
Full Symmetric
38.5
74.1
88.6
54.7
75.6
56.6
82.3
56.5
65.9
LightRetriever Dense
30.4
64.4
83.6
47.0
64.3
46.3
74.7
48.5
57.4
LightRetriever Sparse
63.2
80.5
43.4
61.7
36.7
71.3
LightRetriever Hybrid
30.3
67.2
84.5
50.7
67.6
46.9
76.4
51.4
59.4
Qwen2.5-1.5b
Full Symmetric LightRetriever Dense LightRetriever Sparse
37.9
78.2
87.6
56.9
75.9
54.7
78.8
57.0
65.9
31.3
73.8
85.3
52.8
71.8
49.5
75.2
55.7
61.9
66.7
78.1
49.3
61.6
39.1
63.5
LightRetriever Hybrid
74.4
56.5
73.4
50.5
76.4
Qwen2.5-3b
Full Symmetric
39.4
80.1
89.5
61.9
77.8
58.7
84.4
63.6
69.4
LightRetriever Dense
32.7
76.0
86.8
56,7
73.2
51.6
80.9
59.5
64.7
LightRetriever Sparse
12.8
70.4
80.2
48.8
61.1
39.4
72.4
47.9
54.1
LightRetriever Hyhrid
74.0
Qwen2.5-7b
Full Symmetric
41.3
81.4
91.0
61.1
80.0
59.8
85.5
60.3
70.1
LightRetriever Dense
34.3
74.6
88.0
55.7
75.4
53.2
81.6
57.2
65.0
LightRetriever Sparse
12.5
72.5
82.4
48.3
65.2
40.4
73.3
45.2
$5.0
LightRetriever Hybrid
34.0
76.7
88.6
58.0
76.2
53.9
83.0
61.4
66.5
32

---

Published as a conference paper at ICLR 2026
A.10 ADDITIONAL RESULTS ON CoIR AND BRIGHT
Benchmarks of our query-lighteeight design on CoIR (Code Retrieval) and BRIGHT (Reasoning- intensive Retrieval) have been conducted using three backbones (Llama-3.2-IB, Llama-3.1-8B, Qwen-2.5-7B). Results are shown below in Table 17 and
As code retrieval and reasoning-intensive -ntenssvve retrieval are more challenging out-of-domain (OOD) tasks requiring deep query or structure understanding, our proposed query-lightwiight encoders are less performant compared to strong LLM embedding models like E5-Mistral (-6.2pp on ColR, -6.3pp on BRIGHT), which is expected in our work. Our method is more suitable for normal relevance- based retrieval. Queries requiring complicated instructions or code understanding are not suitable to be applied to the query-lightweight encoder.
Table 17: Retrieval performance (nDCG@ 10) on the ColR benchmark.
Model
Apps
CosQA Synth-
Code-
Code-
Code-
Code-
Stack-
Code-
Code-
Avg
etic-
Search- SNCC
Trans- Transs-
Over-
Feed-
Feed-
Text2-
Net
Contest DL
Flow-
Back-
Back-
SQL
QA
MT
ST
BM25
1.0
14.0
16.9
26.8
34.7
50.1
8.7
56.8
34.7
54.3
29.8
BGE-M3
7.4
22.7
48.8
43.2
47.6
47.9
31.2
61.0
33.5
49.9
39.3
E5-Mistral
21.3
31.3
66.0
54.3
65.3
82.6
33.2
91.5
33.7
72.7
55.2
LightRetriever
Llama3.2-1B
11.2
30.1
52.0
60.1
58.6
57.7
27.9
77.8
29.6
64.5
47.0
Llama3.1-8B
18.1
29.5
54.5
57.8
59.0
59.7
25.7
82.0
33.8
68.1
48.8
Qwen2.5-7B
30.0
54.0
57.7
61.8
61.5
22.3
84.4
33.1
66.3
49.0
Table 18:
Retrieval performance (nDCG@ 10) on the BRIGHT benchmark.
Model
Bio.
Earth. Econ. Psy.
Rob.
Stack. Sus.
Leet.
Pony
AoPS
TheT. TheQ. Avg
BM25
18.9
27.2
14.9
12.5
13.6
18.4
15.0
24.4
7.9
6.2
10.4
4.9
14.5
BGE-large
11.7
24.6
16.6
17.5
11.7
10.8
13.3
26.7
5.7
6.0
13.0
6.9
13.7
ES-Mistral
18.6
26.0
15.5
15.8
16.3
11.2
18.1
28.7
4.9
7.1
26.1
26.8
17.9
LightRetriever
Llama3.2-11
13.7
21.4
11.2
11.1
18.5
13.0
12.7
15.2
4.0
2.9
6.5
6.0
11.4
Llama3.1 1-8B
1-BB
14.2
22.3
12.6
11.1
17.1
13.9
15.0
16.6
8.1
0.7
4.0
5.7
11.8
Qwen2.5-7B
19.3
24.0
13.8
13.1
14.1
11.3
14.3
11.8
5.2
1.1
5.8
5.7
11.6
33


Reviews:
[{"review_id": "0ZZri2HwIK", "reviewer": "Reviewer_B7um", "summary": "The paper introduces a novel asymmetric methodology that trains the embedding layer of a LLM as efficient token-level retrieval embeddings, which can then be averaged to form a query embedding. During training, each token for a given query is prepended with a task specific instruction before being passed to the encoder. Final query embedding (mean of all instruction + token embeddings) is then used as a query representation for contrastive learning using document representations from the same encoder model.\n\nThis method shows a 1000x speed-up while maintaining quality (up to 95%) for a variety of queries across a suite of datasets.", "questions": "- Is the mean weighted for the results table?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper's primary strength is its direct and effective solution to a practical, real-world bottleneck: the high cost and low throughput of online query encoding in LLM-based retrieval systems. The method reduces query encoding time by orders of magnitude (e.g., ~109s to 0.04s for an 8B model on a test batch), leading to a >10x increase in overall query-per-second (QPS) throughput.\n- The training methodology is simple and effective. \n- The paper's claims are well-supported by its ablations. Ablation A2 confirms that the full-sized query LLM is essential during training and that a simple embedding bag is insufficient, validating the training-time complexity . Ablation A1 confirms the asymmetry is crucial, as a lightweight model on both the query and document sides leads to a severe performance collapse\n- Overall, clear and well written paper.", "weaknesses": "- Evaluations on more challenging IR tasks representing live, real world traffic (maybe include CoIR, BRIGHT).\n- Please add a discussion section (in Appendix if needed) that talks about performance within BeIR splits. For example, in the case of HotpotQA (multi-hop), FiQA, etc the performance drops are significant compared to full baselines. This aspect should be made clearer in the paper write-up (expanded more than L407 - L409). Currently, the paper reads as this approach being a one-solution fits all approach for retrieval.\n\n\n- nit: the citations don't have brackets around them and neither are they highlighted in blue (for hyperlinks) which makes it slightly hard to read. I think it may be a rendering issue. Please fix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3SpexKXxu3", "reviewer": "Reviewer_Zka9", "summary": "LightRetriever proposes an asymmetric LLM-based retrieval architecture that keeps a full-sized LLM for document encoding while making query encoding extremely lightweight. For dense retrieval, the method trains token-level query embeddings end-to-end using a full LLM during training, then caches the entire vocabulary’s token embeddings so that online query vectors are computed by an embedding lookup followed by mean pooling. For sparse retrieval, the query vector is a simple term-count vector and the document vector is learned by projecting LLM hidden states into the vocabulary space with sparsity regularization.", "questions": "(1) How expensive is training the dense token cache compared to a standard symmetric dual-encoder, and can you reduce training cost via distillation from the full query encoder to the token cache or a smaller interim encoder without sacrificing performance?\n\n(2) Can you provide systematic evaluations on compositional and constraint-heavy queries beyond case studies, reporting category-wise results for multi-attribute intents, negation, and temporal constraints, and quantifying how hybrid signals mitigate dense-only failures?\n\n(3) How sensitive are results to instruction templates on the dense side, and can sparse retrieval benefit from instruction conditioning indirectly, for example via vocabulary reweighting derived from instruction tokens?\n\n(4) How robust is LightRetriever to production preprocessing such as truncation, lowercasing, punctuation stripping, query rewriting, and subword-vocabulary mismatches, and what are the measured impacts on speed and accuracy?\n\n(5) Can you propose and evaluate an adaptive serving policy that tunes dense dimension top-k and sparse top-k per query based on estimated difficulty or latency SLAs, reporting accuracy–latency trade-off curves?\n\n(6) How does the method transfer to long-document retrieval, code or log retrieval, and additional languages beyond English and Chinese, and does the sparse projection remain tractable and effective with domain-specific tokenization?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "(1) The paper introduces a clear and practical asymmetric design that eliminates deep query-side inference while preserving full LLM power on the document side, delivering extreme online speedups with modest accuracy trade-offs.\n\n(2) The dense pathway’s cache-and-average mechanism and the sparse pathway’s LM-to-vocabulary projection with FLOPs-based sparsity are well formalized, and the training–caching–serving pipeline is technically sound and reproducible.\n\n(3) Empirical coverage is broad, spanning 23 training datasets and two large benchmark suites in English and Chinese, multiple LLM backbones and sizes, and detailed speed breakdowns on 65k queries over 1M passages.\n\n(4) Ablations substantiate key design choices, showing asymmetry is crucial, full query modeling during training is needed, and hybrid dense+sparse recovers most performance; additional results demonstrate controllable trade-offs via Matryoshka dimension truncation and sparse top-k.", "weaknesses": "(1) The approach still depends on full LLM query modeling during training; the paper does not explore reducing training-time cost through distillation, curricula, or lighter interim encoders while maintaining cached-token quality.\n\n(2) Evaluation focuses on academic text benchmarks; robustness on production-like settings with noisy queries, long or heterogeneous documents, domain shifts, and adversarial inputs is not addressed, limiting external validity.\n\n(3) Instruction conditioning is asymmetric: dense uses query instructions while sparse cannot, and the sensitivity to instruction templates and potential mismatches is underexplored, especially for instruction-heavy tasks.\n\n(4) Mean-pooled token embeddings may miss fine-grained compositional constraints; beyond case studies, more systematic stress tests on compositionality, negation, and multi-attribute intents are needed to quantify failure modes and the extent to which hybrid signals compensate.\n\n(5) Many real pipelines perform truncation, normalization, rewriting, or chunking; the robustness of the embedding-lookup query encoder under such preprocessing is not evaluated.\n\n(6) Baseline breadth could be expanded with stronger asymmetric or distilled LLM retrievers and recent hybrid SOTA across both English and Chinese to contextualize the quality–efficiency trade-offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BeW9pB1u2", "reviewer": "Reviewer_8GXk", "summary": "This paper addresses the issue of excessive inference costs in large language model (LLM)-based retrievers by proposing an asymmetric architecture named LightRetriever. This architecture employs a full LLM for offline document encoding ($Enc_d$), but replaces the online query encoder ($Enc_q$) with a simple embedding lookup. This lookup table is generated through an innovative training process: caching the outputs of the full $Enc_q$ for each instruction-word combination, i.e., $v_{t_i}^{den} = Enc_q(Inst; t_i)$. The final query vector is simply the average of these cached embeddings. This approach claims to achieve >1000x query acceleration and >10x end-to-end throughput improvement while retaining 95% of the performance of the full symmetric model.", "questions": "1. Could the average \"95% performance\" mask catastrophic failures in composite tasks (e.g., Case Study 3)? Can performance breakdowns be provided for word lookup versus composite query tasks?\n2. How does the online memory footprint (RAM/VRAM) of the 1.05 GB embedding cache compare to standard KD student models or the \"first-layer\" baseline? The paper ignores this memory-latency tradeoff.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  This work addresses a critical practical bottleneck in deploying LLM-based retrievers: high online query latency, while achieving substantial throughput gains.\n2.  The \"distill-to-embedding-bag\" approach is novel. It ingeniously caches the word-level understanding of the instruction-aware $Enc_q$ into a simple lookup table, differing from standard knowledge distillation schemes.\n3. The method is validated across multiple LLMs (Llama, Qwen) and benchmark datasets (BeIR, CMTEB-R), with fair comparisons against fully symmetric models trained under identical conditions.", "weaknesses": "1.  The dense query encoder employs a bag-of-words model, where $v_q^{den} = \\frac{1}{n}\\sum E[t_i]$. This model fails to capture query composability, meaning that for the model, the query vectors for \"flights from Beijing to Shanghai\" and \"flights from Shanghai to Beijing\" are identical. This fundamentally limits the model's ability to understand complex queries.\n2.  This approach trades significant online memory consumption (an 8B model requires approximately 1.05GB of embedding tables) for reduced latency, yet never discusses this critical trade-off.\n3. The paper contains inconsistencies. For example, the abstract and experimental settings mention acceleration on A800, but the conclusion refers to H800; the 2500x acceleration cited on line 378 also differs from the 1000x acceleration mentioned elsewhere.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wDBbLbSHBK", "reviewer": "Reviewer_KTB8", "summary": "Proposes and evaluates an LLM-based retriever where the goal it to minimize query encoding time. The retriever has sparse and dense components.  For dense retrieval the query vector consists of the average of token vectors, and for sparse retrieval the query vector is a query term frequency count. Overall, this approach is basically the simplest hybrid approach possible. The evaluation employs BEIR and CMTEB-R. As one might expect, query encoding time is just a milliseconds, resulting in substantial improvements in throughput. NDCG values are comparable to acceptable first-stage baselines, including various query embeddings, BM25, and SPLADE.\n\nI wonder why FIgure 5 (the ablation results) uses barcharts instead of presenting results in as a table, which is the format for the other results. The reason I wonder is because the sparse results look quite good, especially given that the query vector representation is nothing more than query term frequency, the same as what might be used with BM25. As I understand it, the green (hybrid) bars are taken from the bottom part of Table 2, so that the orange (sparse) bars are in the high 40s, maybe hitting even 50. This is a pretty remarkable result, given that BM25 results are in the low 40s. Since these queries are nothing more than BM25-style term frequencies, they would run very fast against the sparse document index, perhaps even faster than BM25 with the right query processing strategy.\n\nThe description of the training for sparse retrieval seems to be SPLADE more-or-less, so this seem unexpected, since splade uses more than term frequencies for the queries. I dug around on the Web a bit, and I think this is a known result, for example, \"Joel Mackenzie, Shengyao Zhuang, and Guido Zuccon. 2023. Exploring the Representation Power of SPLADE Models. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR '23).\"\n\nOn the other hand, the dense retrieval use the average of token vectors, which also seems very simplicist, dating back to the Word2Vec era.", "questions": "Why is this new? I don't see Mackenzie et al. (2023) referenced. The training for document vectors in section 2.3 is basically splade with term frequencies. Token vectors for retrieval were explored back in the Word2Vec era. Convince me there is something new here.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I wrote more than I should have for the summary, so most of what I want to say is there. The paper is readable and complete, the experiments appear properly conducted and (with the exception of Figure 5) properly reported. The focus on efficiency is good.", "weaknesses": "I raised what I consider the main weakness in the summary. The dense part is only an average of token vectors, and the sparse part is only splade with query term frequencies. They are combined with a standard hybrid retrieval formula. Is this enough? It might be good from an engineering standpoint, but I'm not seeing the research-level insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)