Revela: Dense Retriever Learning via Language Modeling
Authors: Fengyu Cai
Tong Chen
Xinran Zhao
Sihao Chen
Hongming Zhang
Tongshuang Wu
Iryna Gurevych
Heinz Koeppl
Venue: ICLR 2026 Oral (2026)
Keywords: Information Retrieval
Unsupervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Dense retrievers play a vital role in accessing external and specialized knowledge to augment language models (LMs). Training dense retrievers typically requires annotated query-document pairs, which are costly to create and scarce in specialized domains (e.g., code) or in complex settings (e.g., requiring reasoning). These practical challenges have sparked growing interest in self-supervised retriever learning.
Since LMs are trained to capture token-level dependencies through a self-supervised learning objective (i.e., next token prediction), we can analogously cast retrieval as learning dependencies among chunks of tokens. This analogy naturally leads to the question: How can we adapt self‑supervised learning objectives in the spirit of language modeling to train retrievers?

To answer this question, we introduce Revela, a unified and scalable training framework for self-supervised retriever learning via language modeling. Revela models semantic dependencies among documents by conditioning next token prediction on local and cross-document context through an in-batch attention mechanism. This attention is weighted by retriever-computed similarity scores, enabling the retriever to be optimized as part of language modeling. We evaluate Revela on domain-specific (CoIR), reasoning-intensive (BRIGHT), and general-domain (BEIR) benchmarks across various retriever backbones. Without annotated or synthetic query-document pairs, Revela surpasses larger supervised models and proprietary APIs on both CoIR and BRIGHT. It achieves BEIR's unsupervised SoTA with ~1000x less training data and 10x less compute. Performance increases with batch size and model size, highlighting Revela's scalability and its promise for self‑supervised retriever learning.

Published as a conference paper at ICLR 2026
REVELA: DENSE RETRIEVER LEARNING VIA LANGUAGE MODELING
Fengyu Cail
Tong Chen
Xinran Zhao
Sihao Chen'
Hongming Zhang
Sherry Tongshuang Wu²
Iryna Gurevych'
Heinz Koeppl*
Technncal University of Darmstadt
?University Washington
*Camegic Mellon University
*Microsoft
TTencent Al Lab
(fengyu.cai, keinz.koepll
ABSTRACT
Dense retrievers play a vital role in accessing external and specialized knowl- edge augment language models (LMs). Training dense retrievers typically re- quires annotated query-document pairs, which are costly to create and scarce in specialized domains (e.g., code) or in complex settings (e.g., requiring reason- ing). These practical challenges have sparked growing interest in self-supervised retriever learning. Since LMs are trained capture token-level dependencies through a self-supervised learning objective (i.e., next token prediction), we can analogously cast retrieval as learning dependencies among chunks of tokens. This analogy naturally leads to the question: How can we adapt self self-supe learn- ing objectives in the spirit of language modeling to train retrievers?
To answer this question, we introduce Revela, a unified and scalable train- ing framework for self-supervised retriever learning via language modeling. Revela a models semantic dependencies among documents by conditioning next token prediction on local and cross-document context through an in-batch at- similar-
tention mechanism. This attention is weighted by cetrieer-compuuee ity scores, enabling the retriever to be optimized as part of language modeling. We evaluate Reve la on domain-specific (CoIR), peasoning-intensing (BRIGH')), and general-domain (BEIR) benchmarks across various retriever backbones. Without annotated synthetic query-document pairs, Revela surpasses larger supervised models and proprictary APIs on both CoIR and BRIGHT. It achieves BEIR's 's unsupervised SoTA with 1000x less training data and 10x less compute. Performance increases with batch size and model size, highlighting Revela's scalability and its promise for self-supervised retriever learning.
INTRODUCTION
Central to information retrieval are dense retrievers (Reimers & Gurevych| 2019 Karpukhin al. 2020 Ma et al., 2024), which map queries and documents into high-dimensional vector spaces and determine relevance through similarity calculations. Typically, these models rely on carefully an- notated query-document pairs and hard negatives for training. However, creating such high-quality training data requires substantial human annotation, which labor-intensive and difficult scale in complex, domain-speciffe scenarios such as law (Feng et al. (0244) and programming (Jimenez al. 2024). This limitation stimulates the interest within the community explore self-supervised approaches for training retrievers directly from unannotated raw texts, i.e., self-supervised retriever learning (Izacard et al. 2022).
Modern LMs (Grattafioi et al. 2024), successful case of self-supervised learning, are typi- cally pretrained with the next-token prediction (NTP) paradigm, modeling dependencies among

---

Published as a conference paper at ICLR 2026
PPL
Lowering red's PPI. puldees the retriever to link it with the related purple.
"Back Progrgation
Batch
Similarity
In-batch Attention
Next Token Prediction
Scuren
Acree "Alorie" proplet high printt
Selff Sellf-Attvation 2
feellior Mases Abollenic
Dense Re
Nambers
Stllf-Attentionn Q
Las Aure roniiva furn clud cumppele coslits
En butch Atteation
XN
couthood bor puls andentt
The Hicrow Bible relaas lut unthee Maso ao tn Fappiin rocal comt...
Atef--nttention O
The bis tril,, Lavite, ubeetiunn respeadiition sifiee the cansuary Mostezeckeed and GLESOaEta Anpes ealka asn
superatitian wath
Monne and
Sclf Attectionn
secsonntal hare and you
Figure I: The framework Revela. The retriever's in-batch similarity scores used as in in-batch attention weights inside transformer blocks. The retriever is trained by optimizing the language modeling objective, i.e., NTP. The related patterns in red and purple sequences are highlighted in bold and underline. An example of training dynamics is illustrated at App.
tokens within a single sequence. Analogously, retriever aims to model relationships among larger units-chunde of tokens capturing more macroscopic dependencies. This motivates us to raise a key question: How can we learn a retriever within a self-supervised learning framework of language modeling? While NTP implicitly identifies the most relevant parts in the context during generation, conditioning language modeling of one sequence on others can serve as effective proxy for mod- eling inter-sequence relationships. This can offer a novel and principled approach to supf-supe retriever learning.
In this work, we introduce a self-supervised retriever learning paradigm Dense Retriever Learning via Language Modeling, abbreviated as Revela. As illustrated in Fig. Revela trains retrievers by simultaneously optimizing retrievers and LMs; different from conventional NTP, Revel la learns the probability of token given both the prefix in this sequence and all other sequences in the batch. Specifically, in addition to classical self-attention which restricts NTP to individual sequences, an in-batch attention mechanism enables sequences to attend their neighbors in the same batch during language modeling. In this process. the retriever provides inter-sequence dependencies that modulate the in-batch attention weights, allowing it be optimized with the LM during training. We split raw texts into chunks within each document and put these chunks into the same batch, motivated by the idea of hard negative samples in contrastive learning (Xiong et al. 2020).
We comprehensively demonstrate the effectiveness of Revela across three benchmarks in $ ColR (Li al.] (2025), benchmark tailored for code retrieval, BRIGHT (Hongjin al.. 2025 aassoning-intensive benchmark across diverse domains, and BEIR (Thakur et al. 2021), a heteroge- neous benchmark covering general domains. To do this, we train Revela on Wikipedia for general retrieval and on code-related corpora (Wang et al., 2025) for code retrieval, using pretrained trans- formers ranging from 135M to 3B parameters as retriever backbones paired with an LMM. On CoIR, Revela a outperforms strong, 7B 7B-parameter supervised retriever EE5-Mistral-7b-Inntrus points (nDCG@ 10) and surpasses the weakly-supervised baseline E5-PT (Wang et af.. (022) by 9.7 points at similar scale. This is particularly noteworthy as both baselines was pre-trained on massive query-document pairs that encompass Revela's training data. Furthermore, Revela lavela outperforms proprietary models on BRIGHT and achieves parity with weakly-supervised E5 model on BEIR while using approximately 1,000x less training data and 10x less compute. These results establish Revela as a highly effective and efficient self superviee solution. In $ we also show that scaling Revela via on larger retriever backbones, larger IMMs, and larger batchs can yield greater gain over baselines. Compared to conventional contrastive learning, Revela exhibits stronger coomin generalization. Moreover, the mixed-data training allows the model to scale across multiple do- mains while guaranteeing high comain-specifte performance. Collectively, the evidence highlights Revela's 's robust scaling behavior and strong generalization across models, data, and domains. To this end, we summarize our contributions as follows:
by 2.8
We introduce Reve la, self-supervised framework that trains retriever via language modeling using an additional in in-batch attention mechanism, where next-token prediction is conditioned on both the input sequence and others within the same batch.
2

---

Published as a conference paper at ICLR 2026
Without query-document pairs, Revela surpasses diiistral-7bInstrruu (nDCG@ 10) and outperforms unsupervised baselines by 9.7% at the comparable scale, while also outperforming proprietary APIs BRIGHT. challenging, reasoning intennsive benchmark. Revel a exhibits robust scalability across larger models, batch sizes, and mixed-domain data; also exhibits stronger cross-domain generalization than unsupervised contrastive methods.
CoIR by 2.8%
RELATED WORKS
Self-supervised Retriever Learning Dense retrievers are typically trained with query-document pairs, requiring extensive human annotation. Given the abundance of unlabeled corpora, a key challenge in the community is: How can we train a dense retriever in self-supervised manner?
Some methods leverage weak supervision from document corpora. Contriever (Izacard et al. 2022) applies contrastive learning, using passages from the same document as positives and in-batch ex- amples negatives. E5 (Wang al. 2022] is trained from numerous sources. The primary drawback of this direction is the risk of overfitting to struc- tural biases present in the training data. Other training strategies include distillation from existing retrievers and autoencoding. Distillation is exemplified by LaPraDoR (Xu et al. 2022), which en- hances dense retrieval by incorporating signals from BM25 (Robertson et al. 2009). Autoencoding methods, such as RetroMAE (Xiao et al., 2022). learn embeddings via sentence reconstruction. A key drawback of autoencoding is the lack of pairwise supervision, which can cause overfitting
massive dataset of query query-document pairs
low-level details (tteck 2020).
Our approach departs from the conventional query-document framework. Drawing inspiration from NTP in LMs, our method adapts the language modeling objective, shifting its focus from predicting adjacent tokens capturing the inherent associations between entire texts (sequences of tokens).
LM-guided Retriever Learning LM-driven query and document expansion, exemplified by Query2Doc (Wang et al. 2023), can be effective but is often computationally costly due to its need for powerful models. Augmenting LMs with relevant information from external knowledge stores not only improves performance in various NLP tasks but also enhances retriever learning. Atlas (Izacard et al. 2023) utilizes cross-attention scores between retrieved documents and the generated output as signal to train the retriever. However, Atlas uses encoder-decoder architecture, which diverges from the prevailing trend of decoder-only models and requires costly periodical reindexing. In contrast, our work leverages standard decoder-only architecture model relationships between text chunks within and across documents, mitigating the need for reindexing.
With the rise of decoder-only LMs, REPLUG (Shi al. 2024) enhances retrieval by prepend- ing retrieved documents to the queries, training retrievers to produce query-document similarity aligned with the LM's perplexity. However, the perplexity frozen LMs is often poorly cali- brated (Geng et al. 2024), resulting in suboptimal retriever learning. This issue can be optimized in Revela where retrivers and LMS are updated jointly during language modeling.
Domain-specific Retrieval In pre- pre-training corpora, domain -speeff knowledge is both scarce and rapidly evolving (Grossmann et al. 2023 Wen et al. 2025). making effective retrieval in special- ized domains critically challenging. To enhance the adaptability of dense retrievers across domains, researchers have explored continual learning (Sachan et al. 2021] Oguz et al.) (002)) and task-aware training (Cheng et al.) 2023). However, these approaches still rely on query-document pairs from domain-specific datasets. Another approach seeks to simplify domain-specific retrieval for general- purpose dense retrievers. Cai et al. (2024) propose divide-and-on aeiid--and-conuue conquer strategy through a mixed- granularity retrieval framework, significantly enhancing dense retriever performance in scientific domains. Our work demonstrates Revela's domain adaptation capability through language mod- eling domain-specific raw texts.
3

---

Published as a conference paper at ICLR 2026
3 REVELA: DENSE RETRIEVER LEARNING VIA LANGUAGE MODELING
3.1 TRAINING OBJECTIVES
LM training typically includes the maximization of NTP for token sequences. Given a batch of documents {Di, D2...... ..... D and an LM parameterized by D classical NTP on the token zj in D, = (c)...., , ). can be calculated as, where '' denotes the tokens preceding z/ in D;
P(z)) = (2f ='2))
(1)
As shown in Fig. 2 in Revela, the NTP for sequence i is conditioned not only on its own preceding context x' but also on every other document in the batch, {Dy} Specifically, we introduce a new attention mechanism in the transformer blocks, i.e., in-batch attention ($ (.2). which is supported by a specific mask design ($ 3.4). The attentions the inter-sequence similarity computed by the retriever parameterized by θ, i.e. Sim Dj) ($ 3.3). where E Sim D; D2) = 1. The re- triever is dynamically optimized as the similar- ity is updated jointly with the NTP objective
Output Probabilities
[Sim(Dr,D))) Dyllar
Softmax
Self-Attention
Similarity Matrix
Linear
XN
Attention Map
Feed Forward
weighted by
Document Embeddings
Layer Norm
In-batch Attention
Retrever
Layer
42 10-50% kg D3-
Tembecklngg
{D) Da)
PR(xf) = Pa.el (1 &'a (Dibsss)
(2) Figure 2: Revela's architecture. With an at-
tention map, the embeddings of in-batch attention [h() B can attend to the self-attention {e(]
3.2 IN-BATCH ATTENTION
Revela augments the transformer block with an in-batch attention mechanism to incorporate context from other sequences. We denote the output of D; at the I-th layer as [c]; ht]. where c hf e R4xd represent the outputs from self-attention and in-batch attention, respectively. For simplicity, omit layer normalization and feed-forward layers, the input to the two attention modules in the (-th layer is (e) (e--t:n!-1)
Standard Sell-Attention In the 1-th layer of the blocks, the self-attention mechanism computes
Qi = W KJ WK Vi - W
(3)
where Wº, W WV E dxd are learnable projection matrices. For multi-head attention with H heads (each of dimension dæ d/ H), the standard causal attention is computed as
Q K VdH
e = softmax
V.
(4)
In-batch attention combines standard self-attention with cross-document attention. The embed- dings of k-thh token in D; is obtained by (1) the prefix of D,, and (2) the other documents (D, Jes based on their similarity to D. This encourages Dj to selectively attend to more relevant documents based on learned retrieval signals.
For the contribution from the prefix of D,, the self-attention outpu the same projection matrices above
s in the in-batch attention uses
Q Q. Vdn
= = Kh=h W WK
V. = WV sf = softmax
Via.
Note that the self-attention output & is distinct from computation of eç. As shown in Fig, e and of correspond to the upper-left and bottom-right corners of the attention map, respectively. Functionally, of captures only sequence-level information, whereas sf contains in-batch information derived from h!-1
4

---

Published as a conference paper at ICLR 2026
Cross-document attention enables D; to attend other documents D, using cached keys Kj and values Vj. With full attention mask, the bottom-left corner in Fig. the output big computed as
Q KTT
softmax
Vj.
(5)
vdn
Weighting by cross-document similarity aggregates the attention outputs byy using the similarity scores Sim(D;, Dý) computed by the retriever
by
Σ J=
Si, Dy) bly-
Combined output integrates the results of self-attention and cross-document attention to form the final output of the in-batch attention
= bb,
3.3 SIMILARITY COMPUTATION
Given a batch B documents (Di) the retrieval mechanism proceeds three steps. First, each document D; encoded into an embedding h, E Rdg using an encoder Ée, such that h, Ee(D,), Second, the embeddings are normalized and pairwise cosine similarities are computed: h; = hh/l// and the similarity score between documents i and is given by Si = h h,. Third, temperature-scail scaled softmax with temperature applied to obtain probabilities across documents
exp(Sig/T)
Sim D2) Exjs exp(Saa/T)
The resulting pairwise weights [Sim( D;, D2)1 RBxB capture cross-document similarities, allowing the model to condition on relevant in-batch documents.
3.4 IMPLEMENTATIO DETAILS
As described earlier, Revela a adapts the classical transformers by additionally including in in-batch attention, which builds upon standard self-attention, shown in Eq. (5) For the minimum modifica- tions to the existing transformer's implementation, we take the computation of and h as duplicating documents and adjusting the attention mask, as illustrated in Fig. 2 In this way. the embeddings { produced by in-batch attention can be obtained by applying full attention over the self- attention outputs {e] and aggregating them. Revela's efficiency is discussed in App. C.7
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUPS
Evaluation Benchmarks To comprehensively evaluate proposed framework, benchmark Revela on three diverse datasets: ColR (Li et al. 2025), a comprehensive benchmark designed for code-specific retrieval tasks, BRIGHT (Hongjin et 2025), retrieval benchmark requiring inten- sive reasoning to retrieve relevant documents spanning diverse domains, and BEIR (Thakur et al., 2021), heterogeneous benchmark covering multiple domains for general information retrieval. A more detailed introduction is listed in App. B.1
Training Data One of the earlier baselines for weakly supervised retrievers is E5 (Wang et al. 2022), which collected 1.3B text pairs from diverse sources such as StackExchange, Wikipedia, Reddit, and scientific papers, and filtered them down to 270M pairs using handcrafted rules. For Revela, We simply convert two E5 pretraining subsets to training corpus: StackOverflow for code-related retrieval (CoIR) and Wikipedia for reasoning-intent intensive and general retrieval (BRIGHT & BEIR). Data preparation and illustrative examples of these subsets are provided in App. B.2

---

Published as a conference paper at ICLR 2026
We segment set of code-related corpora (Wang et al. 2023 into chunks of at most 120 words, each comprising complete sentences, including the posts in the StackOverflow forum (Weber et al. 2024), online tutorials (Overwijk et al. 2022). and library documentations (Zhou et al. 2023). for ColR. The batch size is 16. Overall, there are 358,763 training hatches.
CoIR:
segment the passages in the Wikipedia corpus P Given set of passages {d1, d2...., d where each passage d, is divided into chunks (d11, d:2.... . dim,) ), the chunks are interleaved in the order (d11, d12......... dim d21, d22,...) and then grouped sequen- tially into batches of size 16. In total, we sample 320,000 batches from 339,409 documents for training. Notably, a single batch may contain chunks from different documents, highlighting the flexibility of batch construction in Revela.
BRIGHT & BEIR: Similarly,
Models Revela a jointly trains a retriever and an LM using the NTP objective. We adopt LLaMA- 3.2- IB (Grattafiori al., 2024) as the LM. To ensure a fair comparison across diverse unsupervised baselines, we adopt a range of LMs with parameter sizes from O. IB to 3B as the backbone models for the retrievers: SmolLM2 135M (Allal et al. 2025). Dwen2.5-0.5B Yaang et al. (024). LLaMA-3.2 IB and LLaMA-3 3 2- We follow the approach used in RepLLaMA (Ma et al.) 2024), appending the <eos> token each sentence and use its corresponding embedding as the sentence represen- tation. Additionally, we prepend the prefixes "Query: " and "Passage: " to queries and passages, respectively. For more details about model checkpoints, please refer to App. B.4
Baselines We include representative self-supervised baselines: ES-PT large (E5-PT; Wang et al. 2022] is trained with a contrastive objective, leveraging weak supervision signals from a curated large-scale dataset of text pairs (1.3B raw pairs, filtered to 270M) spanning multiple domains (e.g,, code) and covering Revel. a's training corpus. REPLUG (Shi al. (024) distills supervision from frozen LM into retriever by using LM perplexity to model within-bate chunk-chunk similarity, conditioning one chunk on the other. We adopt REPLUG" as our main baseline because (1) matches Revela's aa's retriever decoder-only LM design, unlike encoder-decoder systems such as Atlas (Izacard et al., 2023), keeping the focus on joint retriever-LM training; (2) this architectural match makes comparisons generalizable across scales; and (3) REPLUG outperforms most prior methods, making representative this line of work (Shi et al. 2024). Consistent with Revela, use LLLMA-3.2-1B the frozen reference LM in REPLUG.
For CoIR, we include several supervised retrievers, as well as API-based models such as -AdanAT-Aa--nn (Ada-2) and Voyage-Code--002 (Voyage-2), following the original setup (Li et al. 2025). Supervised retrievers include UniXcoder (UniX; Guo et al. 2022). which is fine- tuned on code-related datasets; BGE-M3 (BGE; Chen al. 2024 a supervised model pretrained and finetuned on text pairs (including code); and E5-Mistral- Instruct (E5-Mistral; Wang et al.
2024].
For BRIGHT, we use several strong baselines: text settt-ebbeddingg---aar (OpenAD, and
API-based
models from (Cohere),
(VoyageAl), as well as E5-Mistral.
For BEIR, we include Contriever (Izacard et al. 2022). a BERT-based retriever trained via con- trastive learning on unsupervised pairs, and LaPraDor (Xu et al., (022) uses latent-pair contrastive pre-training on C4 (Raffel et al., (020) and fuses dense scores with BM25 via lexicon-enhanced dense retrieval.
For more details about the model access and the Huggingface checkpoints, please refer to App. B.4
Experimental Details For both retrievers and LMs, we apply LoRA with a rank of 256. Train- ing uses the temperature т 1e-4, a learning rate of 1c-4, and 100 warmup steps, following the WarmupDecayLR schedule. We train on 4 A100 80GB GPUs with gradient accumulation step size of 8. Passages are truncated to 160 tokens, and bf16 mixed precision enabled. We finetune the models for one epoch, namely, there are 10,000 steps Wikipedia (- 44 hours) and around 11,000 steps on code-related texts (- 48 hours). During inference, the max token length of queries and documents is 2048. For more details of the experimental setups, please refer to App. B.5
https://uuggingffce http:///wugggnggaa..oootaattttte
For computational efficiency, we set smaller hatch size for the 3B model, c.g., 8. For replication details, App. B.3

---

Published as conference paper at ICLR 2026
Table I: Performance on ColR (nDCG@ @ 10, %). Gray indicates supervised models. Bold marks
the highest score among non-API models in each row. Columns marked used code-related pairs
during pre-training. The results of APIs are collected from et al. (2025). Without query-document
pairs, Reve 333 surpasses larger supervised models and proprietary APIs, averaged across 10 tasks.
Dataset
BM25
UniX Reve_a a
ES-PT*
Revela
BGE"
REPLUG Revela
REPLUG Revels ES-Mistaal
Ada-2 Voyage-2
Model Size
0.IB
0.1B
0.3B
0.5B
0.6B
IB
IB
3B
7B
Apps
4.8
1.4
8.2
10.6
20.5
14.7
13.9
19.4
17.7
26.6
23.5
8.7
26.5
CosQA
15.6
25.
27.1
20.1
30.2
25.2
28.9
ST2SQL.
25.0
50.5
45.7
48.9
53.7
46.9
53.9
55.0
56.8
55.9
68.0
58.3
69.3
SN
40.9
60.2
49.9
35.2
$7.9
58.3
50.8
64.0
53.7
62.6
67.4
74.2
81.8
SNCC
54.0
70.0
62.8
53.3
TransC
47.8
41.8
70.9
56.3
77.6
62.6
61.5
81.1
62.1
83.2
80.6
26.0
27.3
TransL
SOQA
70.2
44.7
69.2
86.9
82.5
80.6
76.3
85.7
78.1
38.3
91.0
74.2
81.8
68.1
74.5
69.3
66.0
76,2
71.7
78.8
76.4
47.1
F.MT
59.2
24.2
51.7
46.2
63.6
47.9
42.9
70.4
49.1
73.0
36.4
17.7
28.7
4.2
EXPERIMENTAL RESULTS
Revela exhibits superior performance on domain-speeific retrieval. Tab. reports the per- formance of Revela and baseline methods on ColR. As an unsupervised model trained with- out query-document pairs, Revelaи surpasses E5-Mistral-7B-Innt In-ssrrcct,,,, much larger supervised model pre-trained and fine-tuned on massive, well-curated query-doe pairs, as well as two propri- etary APIs averaged on tasks. Revela also follows scaling laws: performance consistently improves with larger model sizes, while maintaining superiority every scale over basclines. At O.IB parameters. Revela outperforms the code-specific supervised model UniXCoder by 11.1 points nDCG@ @ 10, At the 0.5B scale, model outperforms ES-PT by nearly 10 points, despite E5-PT being pre-trained on 270 million filtered query-document pairs covering Revela's corpus. Revelag.s surpasses the supervised model BGE-M3, despite the latter being pre-trained on extensive text-code pairs 5 Moreover, at each scale, Revels also surpasses REPLUG, underscoring the effectiveness of the retriever-LM co-Iraining paradigm for retriever learning.
Unsupervised Supervised
3000 GPU hours
Others
20
200 GPU hours
Sparse
41.8
4b.2
oberoge crown Na
Commerdal
145
15.2
15.5
16.6
35.7
30.6
13.1
2355
122
#bCce1a
20
20
REPUI
P5-P7
-0.59
Revalo
Cahera
Ozweod
Vopo
Revela
Comtrier
LaPrater
Revela
8-28
RERLUD
Nevelo
Revale
Revola
Revelo
Figure 3: Performance on BRIGHT (left) and BEIR (right) (nDCG@ 10, %). Results for Revela shown in opaque bars, while all other models are represented by transparent bars. On BRIGHT. Revelagg surpasses E5-Mistral, supervised retriever with more parameters, and properties APIs. On BEIR, Revela achieves similar performance with E5-PT with much less data and compute. Please refer to Tab. and Tab. 8 in App. B.6 the per-task results.
Revela demonstrates strong performance on complex retrieval. Fig. 3 (left) shows the av- erage nDCG@ 10 across the BRIGHT subtasks. Despite being trained unsupervised on only 340K raw Wikipedia documents, Revelase outperforms the supervised E5-Mistral-7BB Instructtt as
Please refer Table 8 in the original paper Chen al. 2024).

---

Published as a conference paper at ICLR 2026
well proprietary embedding models. The scaling trends observed in code retrieval persist here as well: even Revelau.s outperforms REPLUGs3 with much fewer parameters. At comparable scale, Revelag.n exceeds E5-PT by 3.1 points (23.7% relative), noteworthy result given the po- tential advantage from E5-PT"s training corpus overlapping with BRIGHT. These findings highlight Revela a promise for tackling more complex retrieval scenarios.
Revela achieves efficient and robust generalization across tasks. Fig. (right) reports the av- erage nDCG@ 10 across the 13 BEIR tasks, where Revela a demonstrates remarkable efficiency and robustness. At the O.IB scale, Revela outperforms Contriever and LaPraDor by over 3 absolute points. Moreover, Revel 1 a 's consistently surpass REPLUG by a significant margin (3B: 8.9%; 1B: 7.0%), mirroring trends observed on ColR and BRIGHT. Remarkably, despite using approximately 1000x less training data and 10x fewer compute resources, Revelag matches ES-PT's perfor- mance, underscoring its efficiency. Most notably, when trained solely a code-related corpus, Revelaşs performs comparably on the general-domain BEIR benchmark to both its Wikipedia- trained counterpart and E5-PT, demonstrating strong cross-domain generalization.
ANALYSIS
We conduct several targeted analyses to further investigate Revela. First, to isolate its algorith- mic contribution, compare it with Contriever using an identical LM backbone. This comparison reveals that Revela achieves superior performance and exhibits stronger domain-specific robust- ness training data. Second, demonstrate that, consistent with traditional contrastive learn- ing, Revela benefits from larger batch sizes. Finally, we examine the LM's impact on retriever performance within the co-training framework. Additional studies on mixed-domain training, out- of-domain generalization, the LM's Rost-Revela capabilities, and computational efficiency are presented in App.
Controlled experiments under the same LM backbone. As baseline models may use dif- ferent sizes and architectures of LMs as retriever backbones, we implement a classical unsu- pervised retriever learning algorithm, Contriever (Izacard al. (022), using the same model (L.aMAA-3. 2 2B) and the same datasets (Wikipedia and the code-related corpus introduced in $4..1] as training data. To construct pseudo applies two tricks: Inverse Cloze Task, where a sentence is removed from a passage to form query against the remainder, and Independent Cropping, where two spans from the same document form positive pair while spans from different documents serve as negatives. In this way, we generate 500K and 359K query-document pairs, each with 15 negatives, for general and code domains, respectively. The models contrastively trained on them are denoted as Contriever-wikin and Contriever-coden: Please refer to App. C.I for more training details.
pairs from raw text, Contriever mainly
Table 2: Revela vs. Contriever Performance.
As shown in Tab. Revela outperforms Con- triever on both BEIR and CoIR, irrespective of whether training is conducted on general code-specific data. Moreover, the performance disparity becomes distribution domains, underscoring Reve latelaass ro- bustness and its strong capacity for cross-domain generalization over contrastive learning.
Model
BEIR
ColR
AVG
Revela-wiking
42.7
53.2
48.0
Contriever-wiking
42.4
50.3
46.4
pronounced in out-of-
Revela-coders Contriever-codeig
39.6
58.6
49.1
32.3
52.1
42.2
Revela benefits from a larger batch size. The in-batch attention mechanism in Revela is inspired by the concept of in-batch negatives in supervised contrastive learning (Xiong al.) 2020). As described in $ [ the default batch size is set to 16. To analyze the impact of batch size, we construct training data with batch sizes of 4 and 8 using the same batch construction strategy. As
shown in Fig. 4 Revela's performance scales with batch size, suggesting potential for further gains. For more detailed results within CoIR and BEIR, please refer to App. C.2
Larger LMs can help out-of-domain retrieval tasks. As observed in the previous section, triever performance improves with larger backbone models. In this section, we further investigate how the size of the LM, the other key component in training, affects the retriever's effectiveness. In
8

---

Published as conference paper at ICLR 2026
ColR Performance
BEIR Performance
Batchh stze
Betchh
52.5
nDC Coc 10
50.0
67.5
Smo1M2--135M
Swer2.540 Retriever
L1.MA-12--10
Smal SmoLMM-11390
Quwe2 5-0.5-
LLAMM-12-10
Retriever
Figure 4: Performance comparison CoIR and BERR with different batch sizes. For both bench- marks, Revela performance generally scales with batch size.
addition to LLaMA-3.2-IB, we also use SmolLM2- 135M and Qwen2.5-0.5B as LMs, each paired with retrievers of three different sizes. All models are trained using the same experimental setup described in $ on both training corpora.
As shown in Fig. ColR exhibits a clear positive trend: the largest LM delivers best retrieval performance. In contrast, on the general-domain BEIR benchmark, larger LMs do not provide consistent advantage. These findings suggest that incorporating larger LMs will likely enhance Revela's performance on specialized domains while maintaining competitive results general- domain tasks. For per-dataset results, see App.
ColR Performance
Performance
5mal 135M
Qwend 2.58 Retrieerr
LIaMA 3.
SMa 135M
Owan2.5 0.38 Retriever
LLaMASS221B
Figure 5: Performance comparison ColR and BEIR using various combinations of retrievers and LMs. For code retrieval tasks, larger LMs can yield greater gains in retriever performance.
To further investigate Revela, we conducted extra experiments, with the following key findings:
Revela learns efficiently from mixed-domain training corpora. Mixing Wikipedia with the code-related corpora used in maiintaiss or even improves, retrieval performance, demonstrat- ing Revelai a's strong generalization across diverse domains with only texts (See App. C.4 Even when trained on an L.Mtrrining, out-of-domain corpus, Revela still performs competi- tively, underscoring its robustness and confirming the observation in Tab. At similar scale, Revela, trained on out-of-domain data, can still outperform E5-PT on CoIR (See App. C.5)
The co-trained LM's capacity is largely preserved, plausibly due the use of LoRA and retaining the NTP objective while adding only auxiliary in in-batch attention (See App. C.6).
Revel a offers theoretical advantage efficiency over REPLUG, complementing empiri- cally demonstrated performance gains (See App. C.7
FUTURE DIRECTIONS
We train retrievers directly from raw text via self-supervised language modeling, sidestepping query- doc pairs, breakthrough beyond the traditional paradigm. Based on this novel solution, We outline several future directions to suggest follow-up research. (1) Iterative Indexing: While Revela uses document chunking, more general approach would iteratively index documents and group chunks by on-the-fl fly representations. Though explored in prior work (Izacard et al. [023 Shi et al. 2024). the high computational cost of such methods remains key challenge for future work. (2) Scal-

---

Published as a conference paper at ICLR 2026
ing up: We envision several directions scale up Revela, including increasing retriever size in 422, and increasing LM size in $ Additionally, incorporating more advanced attention mech- anisms Yuan et al. 2025) may enhance retriever learning and accelerate the training. (3) Multi- modality: Although Reve a targets text and code retrieval, this verrivee-via--anuugg-moodd paradigm can, in principle, generalize modalities, such images (Tiang et al. 2025).
7 CONCLUSION
Efficiently building information-seeking systems is crucial due to the swiftly evolving world and the wide existence of specific domains, where guery-document curation is one key bottleneck. In this work, we introduce Revela, a self-supervised framework that couples dense retrieval with language modeling through novel in-batch attention mechanism, where a token attends to local context and other sequences in the batch during NTP. By letting the retriever's relevance scores weight cross-sequence attention, Revela transforms NTP into retrieval signal, making use of raw text and eliminating the need for annotation or synthesis. Our experiments on domain-specific, complex, and general benchmarks demonstrate significant performance gains over existing self-supervised methods, with improvements scaling with retriever size. Further analysis on batch size, LM size, and mixed-data composition highlights Revels a as strong and scalable alternative to traditional self-supervised paradigms, paving the way for more general and efficient retriever learning.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility, we provide comprehensive details methods and experiments. The setups for our main results, presented in $4.2 are described in $4..T including hyperparameters,
training resources, training time, etc. Further implementation details are located the appendices, covering our model, Revela (App. B.5). REPLUG implementation (App. [.3) with its checkpoints (App. B.4, evaluation benchmarks (App. B.1 and the training corpus (App.[B.2
The appendices also contain our extended analyses. These include detailed comparison with unsu- pervised contrastive learning methods (App. a study on mixed-domain composition within the training data (App. C.4), and an outline of data usage for out-of- generalization (App. C.5 The code to reproduce our results included the supplementary submission.
ACKNOWLEDGMENTS
We thank Kexin Wang, Sheng Lu, Jan Buchmann, Falko Helm, and Yuhao Zhang for the discussion and comments early draft of this work.
Fengyu Cai is funded by the German Federal Ministry of Education and Research and the Hes- sian Ministry of Higher Education, Research, Science, and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE, and by the Federal Ministry of Edu- cation and Research as part of the Software Campus 3.0 project ETRAG (funding code 16IS23067). This work is also supported by the Distr@144 funding line of the State of Hesse (project number: 493 24 (...001554A1 Xinran Zhao is supported by the ONR Award N0001423122
This work has been supported by the European Union (ERC, InterText, 101054961). Views and opinions expressed are however those of the author(s) only and not necessarily reflect those of the European Union the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. The work is related to WP1 on cross-document modelling.
We gratefully acknowledge support from the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 161S22091) and the hessian.Al Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S- HWW04/001/0003
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big-data-centric training of small language model. arXiv preprint urXiv:2502.02737, 2025.
Yonatan Bisk, Rowan Zellers, Ronan Lebras, Jianfeng Gao, and Yejin Choi. Piqa: physical commonsense natural language. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pp. 7432-7439, 2020.
Reasoning about
Fengyu Cai, Xinran Zhao, Tong Chen, Sihao Chen, Hongming Zhang, Iryna Gurevych, and Heinz Koeppl. MixGR: Enhancing retriever generalization for scientific domain through comple- mentary granularity. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp- 10369-10391, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 1/26663311222222mmpmmmm
URLhttps: /acl:: httaaaclaantologgyyoooooo org/2024.
emm.p ma m.in. 5797
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality
multi-granularity text embeddings through self-knowledge distillation. In Len-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Asso- ciation for Computational Linguistics: ACL 2024, pp. 2318-2335, Bangkok, Thailand, August win.:224.iiinnngaa...... URL
2024. Association for Computational Linguistics. doi: 10. 10.1833311120244 wwg.oogg//2222 wtt.2//......ns...eeeee Findings-acl.1111.
atttss//aclanthoot
Hao Cheng, Hao Fang, Xiaodong Liu, and Jianfeng Gao. Task-aware specialization for efficient and robust dense retrieval for open-domain question answering. Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 1864-1875, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v acl-ssoorttss.S.. URL
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint 0rXiv:1803.055
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop. pp. 177-190. Springer. 2005.
Yi Feng, Chuanyi Li, and Vincent Ng. Legal case retrieval: A survey of the state of the art. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Compututional Linguistics (Volume 1: Long Pupers), pp. 6472-6488 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653v1ll 2024.acl- 1500.000
org 2024 wwwp:////......1.......... aal-aaalloooo 350/
Jiahui Geng, Fengyu Cai, Yuxia Wang. Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6577-6595, Mexico City, Mexico, June 2024, Association for Computational org/
Linguistics. doi: 10. 1e111166/1//22444 2024. www.aaaalll-...m Long. 3667
htw.eenaacllooog.g. long.33.
URL
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.21783, 2024.
Igor Grossmann, Matthew Feinberg, Dawn C Parker, Nicholas A Christakis, Philip E Tetlock, and William A Cunningham. Ai and the transformation of social science research. Science, 380 (6650): 108-1 1109, 2023.
11

---

Published as a conference paper at ICLR 2026
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. UniXcoder: Uni- fied cross-modal pre- -ee-traiinng for code representation. and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7212-7225, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18655/11/0222.cl-lee aot. URL https://aclanthhlogg org/ 2022 acl -ong. 4997
Smaranda Muresan, Preslav Nakov,
SU Hongjn, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, el. Bright: A realistic and challenging bench- mark for maasoning-intensin retrieval. In The Thirteenth International Conference on Learning Representations, 2025.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022, 2022. URL
net Cortmiorum?id=aa jKN pxippxi7 7b0
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini. Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251): 1-43, 2023.
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VIm2vec: Training vision-language models for massive multimodal embedding tasks. The Thirteenth International Conference Learning Representations, 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE:-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https::/
net htth/oorumm?cc VTF 8yNQM66
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domaie question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 18653/ http://1122222mm/l/... emnlp-main.5500 org 2020 wwwnl-mmaiieeememmee 550/]
/tthttts:://aclmee http:///aaaaattaohoooo
Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Hao Zhang, Xinyi Dai, Yasheng Wang, and Ruiming Tang. CoIR: comprehensive benchmark for code information retrieval models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 22074 22091, Vienna, Austria, July 2025. Association for Compu- tational Linguistics. ISBN 979- 81199-888977--5--1
doi: 10. 10.11865/v12200 121/3//1/222.... acl- long.1102 URL https: /tttps:/aaclatthhe antho ogy org. wtw.or///222..1110000.0.a acl long 1072/
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning Ilama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2421 -245, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2381-2391. Brussels, Belgium, October- November 2018 Association for Computational Linguistics. doi: 10.18653/v1/D18 1260. URL /ettps:/aaclaninn loghhhooooyy... www.:lll.....o....ooo.oooo wwg.p1g- 12607
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mehdad. Domain- matched pre-training tasks for dense retrieval. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACI 2022, pp. 1534-1534, Seattle, United States, July 2022. Association for Computational
12

---

Published as a conference paper at ICLR 2026
Linguistics. doi: (0.186633w1/2222fmnnngge
URL bttoa::/aclanthhooogg
orq/2022. Saandings-naacl...n
Arnold Overwijk, Chenyan Xiong, Xiao Liu, Cameron VandenBerg, and Jamie Callan. Clueweb22: 10 billion web documents with visual and semantic information. arXiv preprint MrXiv:2211.15845 2022.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances Neural Information Processing Systems, 37:308 1-30849, 2024.
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for eval- uating context-sensitive meaning representations. In Jill Burstein, Christy Doran, and Thamar Solorio ). Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1267-1273, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guisties. doi: (0.18653/v1/N19. 128. [tthttss:: //acl http//claaaaattaaaae anttoologyycoo org/Ni 9-
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
//jmlr wttp://appp///////00001110 html
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, 2019.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-389 2009.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formaliza- fions of commonsense reasoning, pp. 90-95, 2011.
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamil- ton, and Bryan Catanzaro. End-to-end training of neural retrievers for open-domain question an- swering. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 6648-6662, On- line, August 2021. Association for Computational Linguistics. doi: 10. 519. URL
org
519/
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732-8740, 2020.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmented black-box language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume Long Papers), pp. 8371-8384, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1203/////0224.mmccllloo/
URL.
wwwetthooog....oo org/2222 http:/22222aaacll1001
4637
Harald Steck. Autoencoders that don't overfit towards the identity. Advances in Neural Information Processing Systems, 13:19998-19608 2020.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fift fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. htt.thtts://// fttt:::///operrevieee net wtwt/forummiioo f
13

---

Published as a conference paper at ICLR 2026
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training arXiv preprint arXiv:2212.035533 2022.
Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language mod- els. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9414 4443,, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 112.11666/2223..mm//l/ 585n URL
hog.. /tgg//222333 emnlp-ma n. 5857
Liang Wang, Nan Yang, Xiaolong Huang. Linjun Yang. Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11897-11916, 2024.
Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. CodeRAG-bence Can retrieval augment code generation? In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025. pp. 3199-3214, Albuquerque, New Mexico, April 2025. Association for Computa- tional Linguistics. ISBN 779-8-89176-1988
ac antho logy org.000...........
Firdings-naacl. 1767
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Cha- lamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: open dataset for training large language models. NeurlPS Datasets and Benchmarks Track, 2024.
Haoyang Wen, Jiang Guo, Yi Zhang, Jiarong Jiang, and Zhiguo Wang. On synthetic data strategies for domain-specifie generative retrieval. arXiv preprint 17X2v:250..19957, 2025.
Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. RetroMAE: Pre-training retrieval-oriented language models via masked auto-encoder. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. pp. 538-548, Abu Dhabi, United Arab Emirates, December 2022. Association for Computa- tional Linguistics. doi: 10. 10.18833/v1120222 2ttS5////12222.empmmmmn... URL https: pstts:://allantholoogy orq/2022. emn ww-.eenwnlmmm....mmmm 35/
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu. Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval, 2020. URL /tt//tttss::/xxiv.coo
abs 00808
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. LaPraDoR: Unsupervised pretrained dense retriever for zero-shot text retrieval. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 3557-3569, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/0022. findings-acl.281. URL https: http:/s://aclaahoolog.yooo wttp://hww...o....o22..2..... Ein.iig--iinniggg-aacllaa
org/2022.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511. 2024.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11088 2025.
Shuyan Zhou, Uri Alon, Frank F Xu, Zhengbao Jiang, and Graham Neubig. Docprompting: Gener- ating code by retrieving the docs. In The Eleventh International Conference on Learning Repre- sentations, 2023.
14

---

Published as a conference paper at ICLR 2026
A ILLUSTRATION
To illustrate the training dynamics, we visualize both the in-batch attention computed by the retriever and the LM loss, for example, Fig. 6 We use LLaMA-332-IB for both the retriever and the LM, and compare at the initial checkpoint, 100 steps, and 200 steps. When the retrievers
model the semantics, the NTP loss decreases as shown in Fig. 6 In this example, (Blue, Yellow) and (Red, Purple) semantically relevant pairs, where the former is related to biographical information of Aaron, while the latter is related to Aaron's priesthood and religious duties
Aaron Aaron "Ahirōn") dge of Aaron, along with brother Moses, exclusively religious texts, Bible and Quran.
high priest, and brother of Moses Abruhumic religions. Knowle
The books of Exodus, Levitieus and Numbers maintain Aaron received from God a monopoly gver the peleatho himself and his male descendants, The the altar Yahweh.
had the exclusive right responsibiiay make
Bible that, unlike Moses, grew up Egyptian royal coun, remained their kinsmen in border-land Egypt (Goshen).
The rest his tribe, the Levites, were given subordinate responsibilities within the sanctuury (Numbers 3). M of Exochar
oses
consecrated
priesthood, and arrayed in robes office (Leviticu
Blue
Red
Yellow
Purple
Blue
Red
Yellow
Purple
Blue
Red
Yellow
Purple
Blue
.00
.54
Blue
.00
.57
.35
Blue
.05
.13
Red
.37
.00
.37
266
Red
.00
.00
.00
.99
Red
.02
.00
.00
.98
Yellow
.22
.44
.00
.44
Yellow
.86
.08
.00
.06
Yellow
.96
.01
.00
.02
Purple
34
.3
.43
.00
Purple
.00
.99
.00
.00
Purple
.96
.01
.00
Init / Loss: 4.86
Step 100 LoosL 4.86
Step 200/Loss: 4.52
Figure 6: Example of training dynamics of Revela. The related patterns in red and purple quences are highlighted in bold, underline, and italic.
B EXPERIMENTS
B. EVALUATION BENCHMARKS
CoIR contains four classes of code-related retrieval tasks, including Text-to-Code Retrieval". Code- to-Code Retrieval'. Code-to- Text Retrieval", and Hybrid Code Retrieval Table presents the specific subtasks included in ColR with the corresponding descriptions.
15

---

Published as a conference paper at ICLR 2026
Table ColR Benchmark Tasks. The superscripts present the type of the tasks, The abbreviation of the tasks is noted in the parentheses, presented in Table
Sub-Tasks (Abbr.)
Deseriptions
AppsRetrieval (Apps)"
snippets on natural language queries. code snippets answer search queries. Retrieve SQL queries based on natural languuge questions.
CosQA (CosQA)
SyathetizTex22QQU (ST2SQL)"
COIRCoddsSeacciieeeeeniei (SN)²
Retrieve explanations or summaries code snippets. Identify code snippets that are similar to given one. Retrieve code solutions based on contest problems. Retrieve relevant deep learning code contexts or modules. Handle hybrid code-related QA tasks with both text and code.
CadaSeachNNaCCCerrivvaa (SNCC)"
Code TralsOcsnnnnnn Contest (Trans())
CodeTransOceanD.. (TransDL.)
StackOverflonQA (SOQA)*
CodeeVestbackS (F-ST)"
Retrieve answers single code related questions.
CodeFeddbackMT (F-MTy"
Handle multi-tum question-answer scenarios code.
BRIGHT is benchmark for reasoning-intensive retrieval across diverse domains, including Stack- Exchange forums, coding tasks, and theorem-based math. It emphasizes cases with little lexical overlap between queries and relevant documents, exposing the limitations of existing models. Table presents the specific subtaks included in BRIGHT with the corresponding descriptions.
Table BRIGHT Benchmark Tasks. Abbreviations and descriptions.
Abbrev.
Description
StackExchange Biology questions
evidence
Earth.
StackExchange Earth Science questions with linked resourees
Econ.
StackExchange Economics questions requiring theoretical support
Psy.
StackExchange Psychology questions with cited references
Rob.
StackExchange Robotics questions involving tecinical does
Stack.
StackOverllow programming questions with linked web pages
Sus.
StackExchange Sustainable Living questions with supporting docs
Lect.
LoctCode problems: retrieve similar problens/solutions
Pony:
Pony language coding tasks with syntax dncumentation
AoPS
Math Olympiad problems; retrieve others using skill
TheoQ.
TheoremnA: retrieve problems applying same theorem
TheoT.
TheoremQA: retrieve theorems from ProofWiki relevant to query
B.2 EXAMPLES OF THE TRAINING CORPUS
Fig. 7 presents example batch containing 16 chunks from two topics. The sentences are chunked by NLTK while the maximum length a chunk is limited to 120 words. When training, the chunks will be randomly shuffled within the batch.
B.3 REPLUG
Self-supervised batch loss. In our setup, as there are no external queries, each document plays the role of both query and context. For batch D (Di,..., DR). embed every document with the retriever Ee:
exp(sig)
Sij
Po(j (i, ΣB exp(s1t)
Tr
frozen LM d4 measures how well D, explains De
expl Lig/ Rm))
lj log go Dj),
Polj (1))
ΣB exp( - Lake/ - Rm))
We train the retriever by aligning these two distributions for every anchor document:
hhtps//www.nltk.org/
16

---

Published as a conference paper at ICLR 2026
- properties, and reactions
C-C-hond formation and reductions: Barbier, Nozaki-Hiyama, NaBH,/LiAlH4, Meer-
wein-PPondorff-eriie and Noyori asymmetric hydrogenation.
2. Alcohols have pK₃ 16-19; deprotonation by strong bases yields alkoxides, yet neutral OH a poor leaving group.
3. Protonation (R-OH R-OHz*) activates SNI/SN2 substitution; e.g. terliary alcohol + HCI tert-alkyl chloride, thionyl chloride for primary/secondary cases.
4. Hydrobromic acid or PBr; give alkyl bromides; Barton Mirton--MCCombie deoxygenates alcohols
5. EI elimination: acid-catalysed dehydration follows Zaitsev, fastest for tertiary alcohols;
Fischer esterification and tosylation convert OH into esters.
6. Oxidations: primary aldehyde/carboxxyicc acid, secondary ketone, tertiary inert; pathway proceeds via hydrate (RH)H((H))))
7. Typical oxidants: Collins reagent, Dess-Martin periodinane, KMnO4, Jones reagent; alco-
hol any molecule with a C-OH group.
Achill Island geography, history, and culture
8. Largest Irish island (area 148 knł, pop. 2700); linked to the mainland by Michael Davitt Bridge
9. First settlers circa 3000 BC; 87 % peat bog: late-Neolithic population estimate 500-1 000.
10. Deforestation for cultivation; Iron-Age promontory forts at Slievemore, Atlantic Drive, Achillbeg testify to martial past.
11. Umhall territory ruled by the seafaring O O'Malleys; Butler/de Burgo control after Anglo- Norman invasion; 17th-18th--enntury inward migration.
12. Two Irish dialects co-existed; Carrickläldavnel Castle (15th c.) and the exploits of Gruce O'Malley embody maritime heritage.
13. Grace O' O'Malley met Elizabeth in 1593; Rev. Edward Nangle founded the proselytising Achill Mission ("the Colony") ") 1831.
14. Mission thrived then waned; Westport-Newport railway ... echoing Brian Rua
O' Cearbhain'' 's prophecy of "earts on iron wheels".
15. First train carried victims of the Clew Bay drowning (1894); ..neared with the Kirkintilloch bothy fire (1937).
16. Rail line closed weeks after 1937... St Dymphna's 's holy well stand on the south-east coast.
Figure 7: Example of one batch containing chunks split from Wikipedia: This batch contains chunks from two topics: Alcohol (blue) and Achill Island topic (green).
1 KT(Po(: B
(())
2
|| Po(. Pa(-1i,)))
LM parameters & remain fixed; only θ is optimized.
In our experiments, both of temperatures, 7m and Tr, are set as 0.001. REPLUG training on the same datasets with Revela, with the learning rate 5 and the training steps 4500. All other experimental setups are identical to Revela.
B.4 CHECKPOINTS
We include the off off-the-shelf unsupervised and supervised retrievers and the corresponding hugging- face URLs in Tab. For the prosperity APIs, we list the URLs in Tab. 6
17

---

Published as a conference paper at ICLR 2026
Table Baseline retrievers, LMs (Revela's backbone), CodeRAG-Bench datasets, and evaluation benchmarks with their HuggingFace URLs and licenses.
Name
URL
License
UniXcuder (UniX)
microoofUUnnixccoeee buse
Apache-2.0
Contriever
facehook/contrivver
specified
RetroMAE
ShiaasRetroMAIl
Not specilied
GraphCodeBERT
micmsoffi graphendeheert base
Not specified
LaPraDoR
cunwenxu//aprador
Apacbo-2.0
us-laage-usuueevvieed (ES-PTTsyy)
intloaateS
MIT
BGB-m3
BAAl/go-m3
MIT
BBS-Mistal-7B-nnstrne
Mitoonlcs-misttal----neeee
MIT
SmoLLM2-135M
Apacho-2.0
Qwen 2,5-0.58
Owen/Owen2.55 2-0.8B
Apacho-2.0
I.I.MA-3...1 IB
meta Jamal/ Jama: 3.2.
Llama 3.2 Community License Agreement
LLaMAA3.2-3B
mota-llama/ 3allam/a/aaama32222111 Liama-3.2-3 3.33BB
Llama 3.2 Community License Agreement
LL.MMA3.1-SB
resta-llama/a Llama-3.1-88
Llana 3.1 Community License Agreement
Library Docureetaaiio
code-rag buub-rag-benchblibbrry bww.c//iihacrrrreeooooooaaaaalle htt.:a//bbbncbbbbbbadduceeeaaat documentation
CC BY-SA 4.0 CC BY 4.0
Online Tutorials
code rag benchhoniinee Intorials
StackOverflow Posts
CC BY-SA 4.0
BEIR
BelR/beir
CC BY-SA 4.0 Not specified
ColR
SotR-Retrieval
Table 6: Embedding models and their reference URLs.
OpenAI-Ada 002
platform. openai hwtp:////c//eeeeeeeeeeeeeeeeeiii
Voyage-cude-2
blog, voyageui htw.://wwww////.....o.......e...
Sost-embedding---aaage
voyage-large instruct
does, viccvvvyyggai.co www.ggai..oomdusseeebeddnngggggg
B.5 EXPERIMENTAL SETUPS
Model Architecture When trained on Wikipedia, Revela prevents any single token from dom- inating the attention by scaling the output bly with the norm of V/ thereby encouraging the cross- document attention to focus on sequence-level semantics (Izacard et al. 2023). This operation, referred to as V-normalization, is computed
b9y No +e"
K5 vdh
b/y
where Noj softmax
I1Vill2.
(8)
by
Σ Li
Si,,D;,Dj) D)) b/y-
(9)
where e is a small constant for numerical stability, set as 1c-6 in our experiment. When trained on code-related corpus and evaluated on CoIR, Revela performs better without V-normalization. We will leave the exploration of more variants of the architecture design in the future.
Similarity Calculation within Chunks When calculating the similarity between chunks derived from Wikipedia, we follow REPLUG, using only the first half of the chunk to compute similarity with other chunks. This design is motivated by the inferential semantics inherent in natural language retrieval tasks. In contrast, for code retrieval tasks, we retain the full chunk when computing simi- larity, as these tasks rely more heavily precise semantic matching compared to natural language.
18

---

Published as a conference paper at ICLR 2026
Table 7: Performance of unsuperviseed/sll-suupvvice retriever models on BEIR datasets (nDCG@ 10, %). Bold marks the best score per dataset among unsupervised methods.
BM25 Contriever LaPraDor Revelal
Sevela
REEPLUG
Revela-code
Model Size
0.1B
0.1B
0.1B
0.33
0.5B
0.5B
3B
3B
ArguAna
48.7
44.3
44.6
39.0
44.4
36.4
41.1
39.4
44.6
38.0
45.3
44.3
CEVE
13.6
7.2
12.2
15.3
15.7
13.0
16.6
DBPedia
29.9
27.0
25.0
18.3
57.1
18.0
21.3
19.5
27.6
23.4
27.1
30.9
54.3
RIQAZU18
25.1
12.4
19.8
19.2
43.2
19.9
27.2
22.7
30.4
24.8
34.3
33.2
HorpoiQA
56.9
41.0
30.4
38.5
52.2
35.6
50.6
42.4
56.0
33.9
58.8
57.8
NFCoqus
32.1
27.1
30.4
23.6
33.7
26.9
26.7
27.2
26.9
33,0
32.9
NQ
28.5
18.1
18.0
21.2
41.7
26.7
29.8
27.8
33.9
38.1
40.8
41.0
QuoraRettrivvll
SCIDOCS
15.8
10.9
13.4
12.1
21.8
13.6
14.8
14.5
16.3
15.3
17.6
18.8
49,9
TRECCOVID
62.2
18.2
22.9
60.6
61.8
44.9
58.4
39.4
60.1
46.2
66.7
60.7
Touche2020
33.1
7.2
8.9
12.4
19.8
11.8
19.9
14.7
25.7
1.5
26.9
30.5
Mean
41.8
29.5
29.8
33.3
45.6
34,2
38.8
35.7
42.7
36.3
45.2
45.6
Table 8: Performance BRIGHT (nDCG@ 10, %). Bold marks the best performance. The results of BM25, E5-Mistral and APIs are taken from BRIGHT (Hongjin et al. 2025). )
Dataset
BM25
ES.PT Revela Revala REPLUG Revel 1
Mistral Cohere OpenAl Voyage
Model Size
Biology
18.9
18.5
16.9
16.7
24.9
18.6
18.7
23.3
23.1
Earth Science
27.2
29.5
29.0
29.4
13.0
40.3
26.0
28.4
26.7
25.4
Economics
14.9
10.2
13.9
13.2
12.9
17.2
15.5
20.4
19.5
19.9
Psychology
Robotics
13.6
7.1
9.1
10.5
7.4
15.3
16.3
16.3
12.8
10.8
StackOverlow
8.7
Sustainable
15.0
14,2
13.9
15.4
12.1
19.2
18.1
17.6
20.5
15.4
Pony
7.9
1.8
3.9
3.0
4,9
1.9
2.4
LectCode
24.4
22.5
27.6
26.1
24,8
26.6
28.7
26.8
23.6
30.6
AoPS
6,2
3.3
12.5
8.8
13.3
11.6
7.1
6.3
8.5
7.5
ThmQA-Thm
ThmQA Q
10.4
17.4
24.8
24.1
22.9
27.1
26.1
15.7
23.5
27.4
Mean
14.5
13.1
15.2
15.6
12.2
20.1
17.9
16.6
17.9
17.9
B.6 SUPPLEMENTARY RESULTS
Tab. and Tab. present the per-task results in BEIR and BRIGHT, including Revel and other baselines.
C ANALYSIS
C.I EXPERIMENTAL SETUPS OF CONTRIEVER TRAINING
Our Contriever implementation fine-tunes a LoRA-adapted LLaMA-3.2-IB retriever on a 500k Wikipedia and 359k code dataset, using contrastive learning with a temperature of 0.01, EOS pool- ing, and query/passage prefixes as "Query: and "Passage: " respectively. The LoRA rank is 256, consistent with Revela. The training is optimized with DeepSpeed ZeRO-3, bfloat16 precision, gradient checkpointing, and an effective batch size of 256. The number of negatives for each query 15, which is consistent with the popular contrastive learning setup such MsMarco
The per-task results in Tab. 2 on CoIR and BEIR are displayed in Tab. 9 and Tab, 10 For in- domain evaluation, Revela outperforms the contrastive-leaming counterpart both BEIR and ColR. Though the performance on BEIR is quite close, Revela la demonstrate better generalization: it outperforms Contriever tasks like FiQA, SciFact, and TRECCOVID, while Contriever exhibits better performance on ClimateFEVER or FEVER, which use Wikipedia as the corpus. For out-of-
19

---

Published as a conference paper at ICLR 2026
Table Performance on CoIR (nDCG %). Bold marks the best score among the models.
Dataset
Revela-wiki.s Contreeer-wikig Revela-code
Contrieva--codqss
Apps
9.1
11.3
19.4
24.4
CosQA
22.6
24.3
30.2
32.4
ST2SQI.
55.8
46.3
55.0
44.8
SN
54.2
49.8
64.0
41.5
SNCC
69.2
57.4
70.0
TransC
74.4
76.3
81.1
79.6
TransDL
SOQA
70.9
81.2
85.7
91.0
F-ST
70.6
65.5
76.2
66.0
F.MT
70.7
56.6
70.4
58.5
Table 10: Performance of unsupervised/sel-spperve retriever models on BEIR datasets (nDCG@I 10, %). Bold marks the best score per dataset.
Dataset
Revel a-wikig Contrioecs-wikigs Revele-codejs Contriover-codeus
ArguAna
44.6
48.9
46.9
42,0
ClinateFEVER
14.9
DRPedin
27.6
31.6
21.8
14.6
FEVER
61.7
68.6
51.4
41.2
FIQA2018
HotputQA
56.0
47.1
46.4
17.6
NECorpus
NQ
33.9
28.9
29.5
15.0
QuoraRetrieval
77.5
84.3
SCIDOCS
16.3
21.3
19.9
SciFact
71.9
63.9
66.4
61.5
TRECCOVID
Touche2020
25.7
17.4
21.8
15.2
Mean
42.7
42.4
39.6
32.3
domain tasks, Revels l significantly surpasses Contiever, highlighting the advantage over traditional contrastive learning paradigm.
C.2 ANALYSIS ON BATCH SIZES
Tab. T and Tab. 12 present the performance of Reve la with different batch sizes on ColR and BEIR, respectively. Generally, with larger batch size, Revela's performance will be increased.
Table 11: Retrieval performance (nDCG@ @ 10, %) across the 10 ColR tasks. We report three encoder sizes (135M, 500M, 1B) and three batch sizes (bs4, ba8, bs16) 6).
Revelaa.
Revelaus
Revelain
Dataset
bss
bsd
bs16
bad
bs8
ba16
AppsRetrieval
11.8
20.5
16.2
CosQA
25.0
25.4
26.2
27.5
28.4
27.5
28.0
28.1
30.2
SyatheticTes12S01.
42.8
44.0
45.7
50.9
52.5
53.7
53.4
51.4
55.0
CoIRCnndeSarddNeeReeet
39.5
46.5
49.9
57.0
52.9
57.9
59.3
57.9
64.0
CodlsearrdNalCCCoettiie
56.0
60.4
63.4
62.5
63.7
68.0
69.0
69.2
70.0
ConleTransOcaanConnee
77.6
CoddTransSOeaanD..
34.6
34.9
34.6
33.8
34.0
35.4
34.4
34.3
34,2
StackOverflonOA
62.8
73.8
76.6
82.5
82.5
85.7
CodeFedbackST
59.0
61.2
63.8
74.2
74.9
74.5
74.8
75.5
76.2
FodeFeedbac
50.2
49.5
51.7
61.4
71.4
63.6
69.8
70.7
70.4
Mean
43.2
45.9
48.4
52.4
$4.5
56.1
55.9
56.4
58.6
20

---

Published as a conference paper at ICLR 2026
Retrieval performance (nDCG @ 10, %) across 13 BEIR tasks for three encoder sizes and three batch sizes (os4, bs B, bs 6).
Table 12:
Bevelau.
Bevelam
ba8
bss
bs
ArguAma
36.7
38.5
39.0
35.4
41.0
41.1
43.0
47.9
44,6
CUnnatFEVER
15.5
13.6
13.8
16.4
14.0
15.8
DBPedia
15.5
15.7
18.3
23.1
20.8
21.3
25.5
20.7
27.6
61.7
FiQA2018
17.4
18.2
19.2
24.8
27.8
27.2
29.4
28.8
30.4
HotpotQA
32.7
33.3
38,5
45.7
46.0
50.6
51,0
54.6
56.0
NFCorpus
21.1
22.0
23.6
25.0
27.1
26.8
26.6
28.1
27.2
NQ
16.8
17.3
21.2
23.4
27.9
29.8
34.7
32.9
33.9
Quoraketrieval
SCIDOCS
9.5
10.3
12.1
12.6
14.7
14.8
16.0
15.8
16.3
SciFact
67.9
TREC-COVID
58.8
58.1
60.6
62.3
64.1
58.4
67.1
68.8
60.1
Touche2020
12.7
12.4
21.1
22.4
19.9
26.7
24.7
25.7
Mean
29.6
31.0
33.3
36.8
38.4
38.8
41,7
41.4
42.7
C.3
ANALYSIS ON THE SIZE OF LMS
Tab. 13 and Tab. 14 present an ablation study analyzing the impact of differently sized LMs.
Table 13: nDCG@ 10 (%) on BEIR for three model sizes (Reve 10.18 Reve Levelao.se a0.s. Revelajs), with LM sizes in ascending order. Each triplet of columns highlights the highest value bold.
Bevelagıe
Revelause
Revelaj
Dataset
0.18
0.5B
1B
0.15
0.5B
1B
0.16
0.5B
1B
ArguAna
36.9
39.9
39.0
38.5
41.8
41.1
41.1
43.0
44,6
ClimateFEVER
14.1
16.3
15.3
18.3
20.0
13.8
15.0
15.8
15.8
FEVER
25.5
31.3
33.7
53.0
55.7
51.1
58.7
56.7
61.7
FIQA2018
19.2
26.6
24.9
29.3
28.4
30.4
HotpotQA
41.7
41.0
38.5
54.6
53.9
30.6
58.4
58.7
56.0
NFCorpus
19.7
22.2
23.6
27.4
27.4
26.8
26.4
28.7
27.2
NQ
22.3
21.0
21.2
34.5
32.2
29.8
36.8
34.9
33.9
QuoraRetrieval
79.1
79.7
81.0
83.4
83.5
83.2
82.4
83.3
83.5
SCIDOCS
12.1
SciFact
52.4
56.7
57.9
65.9
65.8
66.0
71.6
72.1
71.9
TREC-COVID
58.6
60.6
60.6
68.7
68.1
58.4
66.4
65.7
60.1
Touche2020
11.2
13.1
12.4
31.1
24.2
19.9
28.7
26.9
25.7
Mean
33.0
33.3
42.7
42.8
42.7
Table 14: nDCG@ 10 (%) on CoIR for three model sizes (Revelãg.1.. Revelan.s. Revelajg). with LM sizes in ascending order. Bold in each row indicates the best performance for each retriever size.
Revelaain
Revelaus
Revelain
Task
0.10
0.50
App=Retrieval
15.0
20.5
19.4
CosQA
24.8
24.6
26.2
26.7
28.3
27.5
28.7
30.2
30.2
SenthetieTxx22QOL
43.9
46.2
45.7
50.8
52.0
53.7
52.1
53.0
55.0
CoiRCCwdeSenchNeeeeee
40.0
40.0
49.9
51.2
48.0
57.9
57.2
55.2
64.0
CodeSearchNeeCCRReeee
$4.7
$6.2
63.4
61.5
61.3
68.0
62.6
64.1
70.0
Code Conteeeennennnne
CodeTransOcaan
34.2
34.4
34.6
34.0
34.8
35.4
33.8
38,0
34.2
StackOverllow
79.5
77.0
82.5
82.2
85.7
CodeFecdbackST
64.8
64.9
63.8
73.3
73,9
74.5
75.2
75.0
76.2
EedeFeeedBac
55.7
54.4
51.7
66.0
62.4
63.6
68.5
69.7
70.4
Mean
45.8
45.9
48.4
53.5
$2.7
56.1
55.5
55.6
58.6
21

---

Published as a conference paper at ICLR 2026
C.4 MIXED-DOMAIN COMPOSITION
Tab. 15 and Tab. 16 report the performance of Revela trained the mixture of the batches structed from Wikipedia and code-related corpus, applied in $ 4 We randomly sample 160,000 batches from both datasets to form the training data, and maintain all other experimental setups. Compared with Tab. and Tab. 7 where Revela is trained separately in each domain, it can largely maintain the original performance when trained the mixed-domain data. These results indicate Revela's potential to generalize to diverse domains.
Table 15: Comparison of Revela's performance (nDCG@ @ 10, across ColR benchmark tasks for three LM sizes trained on the mixture of Wikipedia and code-related corpus. Bold indicates the best performance in each row.
Dataset
Revelag.
Revelag.5
Bevelam
AppsRetrieval
5.9
17.2
17.5
CosQA
25.8
28.0
24.5
SyatheticTex12S0..
46.0
$1.5
55.1
CoIRCadeSuurcheeReeenee
41.1
51.5
58.3
CodleSearchNe
ConleTransCeeenn( Contest
66.3
76.4
78.6
DransOceanDD
34.4
35,0
34.3
StackOverllowQA
66.0
79.4
82.2
CodeFedbackST
64.5
74.6
75.6
52.8
67.7
71.2
46.2
Table 16: Comparison Revela's performance (nDCG@ 10, %) across BEIR benchmark tasks
for three LM sizes trained on the mixture of Wikipedia and code-related corpus. The highest value in each row is highlighted in bold.
Dataset
Reveleuı
Revelagan
Revelam
ArguAna
41.4
43.9
44,2
ClimateFEVER
16.2
16.1
15.3
DBPedia
25.4
FEVER
35.9
44,9
56.8
FIQA2018
21.2
28.3
34.6
HolpolQA
37.0
50.3
59.1
NFCorpus
24.5
28.7
28.7
NQ
19.7
27.4
34,5
QuoraRetrieval
82.2
85.7
85.2
SCIDOCS
SciFact
59.5
63.8
69.2
TRECCOVID
61.8
62.9
67.0
Touche2020
15.7
20.0
23,1
43,0
C.5 OUT-OF-DOMAIN GENERALIZATION
To further validate Revela's generalization, trained it Fineweb-edu (Penedo et al. 2024). widely-used language modeling corpus for 320,000 batches and evaluated the models on the ColR and BEIR benchmarks. The results in and Tab, show that Revela achieves competitive performance despite being trained on out-of-domain data. Notably, the Revel a0.5B model scores 48.6% CoIR, outperforming E5-PT (46.4%), which was trained on 270 million query-document pairs.
8 Following the experimental setup from $ 4.1 we trained
Tab. and
22

---

Published as a conference paper at ICLR 2026
Table 17: Comparison Revela's performance (nDCG@ 10, %) across CoIR benchmark tasks for three LM sizes trained on an out-of-domall corpus, i.e., Fineweb-edu. Bold indicates the best performance in each row.
Dataset
Revelag.
Revelag.n
Bevelam
AppsRetrieval
1.9
8.6
11.3
CosQA
25.8
26.0
26.6
Syathetic
42.0
54.3
53.7
27.8
38,9
46.2
CodeSearchNeeCCC Reitriival
CodleTransOeeannCiii Contett
60.2
68.6
75.2
Do
33.4
33.5
33,8
StackOverflowAA
50.6
63.1
69.0
CodeFeddbuckST
52.7
70.4
70.9
CodeFeedbackMI
38.7
61.8
38.0
Table 18: Comparison of Revela's performance (nDCG@ 10, %) across 13 BEIR tasks for three LM sizes trained on an out-of-domain corpus, i.e., Fineweb-edu. Bold indicates the best perfor- mance in each row.
Dalaset
Revelamm
Revelag.
Revelam
ArguAna
36.4
36.9
38.7
ClimateFEVER
13,9
12.7
12.3
DBPedia
17.2
20.2
24.5
FEVER
FiQA2018
20.1
24.9
30.3
HotpotQA
NFCorpus
26.8
30.1
29.7
NQ
17.4
24.8
31,0
QuoraRetrieval
81.8
83.6
82.4
SCIDOCS
12.6
15.6
16.8
SciFact
TRECCOVID
65.5
66.3
63.3
Toache2020
23.0
Mean
32.9
38.0
40.7
C.6 LM PERFORMANCE AFTER REVELA
We evaluate the original LLaMA-3.2-11 IB and the model co-trained with the retriever (LLaMA- 3.2- IB backboned on Wikipedia in 4.1 on seven tasks, including ARC (Clark et al. 2018)
COPA (Roemmele et al. 201 OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et 2020
RTE Dagan al. 20055. WIC (Pilehvar & Caimacho-Colladae 2019), and Winogrande (Sakaguchi
ct al. 2020). As shown in Tab. 19 LM's performance has been greatly preserved.
Table 19: Performance (Accuracy, %) of the LM before and after Revela various benchmarks.
Model
ARC
COPA
OpenBookQA
PIQA
RTE
WinGGrande
Avg.
L.
28.0
77.0
25.0
73.0
59.5
47.5
57.5
$2.5
52.2
C.7 COMPARISON BETWEEN REVELA AND REPLUG
While both Revela and REPLUG involve retrievers and LMs, their core motivations differ. RE- PLUG distills relevance signals from the perplexity of frozen, off-the-shelf LMs. contrast, Revela proposes broader paradigm: to learn retrievers integral part of language model- ing, rather than as a post hoc add-on. From a technical perspective, Revela also outperforms REPLUG regarding efficiency.
For a batch of documents (each with sequence length L), the cost of modeling inter-document relationships differs significantly between REPLUG and Revela:
23

---

Published as a conference paper at ICLR 2026
To compute pairwise relevance between all document pairs using LM perplexity quires processing each pair independently. Each pair is concatenated (length 2L) and passed through the LM, resulting in forward cost of O (2L)²) for all B 1)/2 pairs. Revela: a: As detailed in $ $3.4. uses in-batch attention with concatenated inputs. By processing all B documents jointly in single forward pass with cross-attention, it computes inter-document relevance with a cost of O(B x (2L)²). Including backward passes (which scale similarly), the total cost remains linear batch size.
REPLUG:
Therefore, when using a reasonably large batch size (e.g., 16 as in our setup), Revel levell a learns inter- document relevance more efficiently than REPLUG, due its linear scaling in batch size compared to REPLUG's s quadratic cost.
D USE OF LLMS
In preparing this manuscript, we employed LLM for general assistance. Its use focuses on improv- ing the clarity and grammar the text and helping generate the code used to create figures (e.g.. matplotlib) or LaTeX tables.
24


Reviews:
[{"review_id": "s9BAHdcfke", "reviewer": "Reviewer_ojFX", "summary": "This paper introduces a self-supervised framework that jointly trains a dense retrieval model along with a LLM using NTP loss and in-batch, cross document attention. The authors posit that NTP loss, when conditioned on in-batch chunks from different documents, weighted by the similarity scores of the retriever, provides strong supervision for measuring similarity among documents. \n\nExperiments demonstrate that a dense retriever trained with <400k batches (~1000x less than modern retriever training datasets) out-performs strong baselines and enterprise API based models for standard tasks (BEIR) and coding (CoIR) among other sets.", "questions": "1. Given the joint training, what are the training dynamics and stability of the system? Does the retriever or the LM converge significantly faster?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The approach of using NTP paradigm to \"distill\" similarity signals into a dense retriever along with he in-batch attention and weighting by similarity scores method, are interesting and novel ideas.\n2. Improved scalability and calibration compared to quadratic baseline (pairwise distillation).\n3. Great experimental design including selection of training data and baselines.\n4. Thorough ablation study showcases generalization, effect of batch size, LLM base performance, mixing training corpora among others. These studies provide a strong baseline for future exploration.\n5. Overall, a very well written paper and easy to understand.", "weaknesses": "1. Training data creation methodology and statistics are underspecified. It would be helpful to understand how the filtering was done and what the \"handcrafted rules\" L249 are. Similarly, while constructing a batch (following example in Appendix B.2), how were the topics chosen?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gc7RFosp7I", "reviewer": "Reviewer_U5J6", "summary": "Revela proposes a novel self-supervised framework that jointly trains a dense retriever and a language model by embedding retriever similarity scores as in-batch attention weights within transformer blocks.\nThis allows the model to learn retrieval ability directly from next-token prediction without any labeled query-document pairs, achieving strong performance on multiple retrieval benchmarks with far less data and computation.", "questions": "The questions have been presented above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Revela proposes a novel self-supervised framework that jointly trains a dense retriever and a language model by embedding retriever similarity scores as in-batch attention weights within transformer blocks.\nThis allows the model to learn retrieval ability directly from next-token prediction without any labeled query-document pairs, achieving strong performance on multiple retrieval benchmarks with far less data and computation.\n- The use of retriever similarity as in-batch attention weights seamlessly embeds retrieval into standard Transformer computation with minimal architectural changes.", "weaknesses": "- The motivation for using next-token prediction is unclear. The authors need to provide a detailed explanation of why this training objective can enhance the retriever’s capability.\n- The In-batch Attention section is somewhat confusing. It states that In-batch Attention consists of two parts — Standard Self-Attention and In-batch Attention — but within In-batch Attention itself, there is another self-attention output s. I suggest the authors restructure this description for greater clarity.\n- Could the authors provide a detailed explanation of the computation steps for In-batch Attention? My understanding is that it involves three internal computations (outputs e, s, and b), yet Figure 2 shows only a single attention map (indicating the computation is fused?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e7Ci93vUQE", "reviewer": "Reviewer_PRP5", "summary": "**Motivation:** Dense retrievers are important, but training them usually needs labeled query–document pairs, which are costly and scarce in specialized domains. The paper asks if we can train a retriever in a self-supervised way by treating retrieval like language modeling, where dependencies among chunks of text play the role that token dependencies play in next token prediction.  \n\n**Approach:** Revela trains a retriever together with a decoder-only LM by adding an in-batch attention path that lets each sequence attend to other sequences, with those cross-sequence attention weights set by the retriever’s similarity scores, and both parts optimized through next token prediction. Documents are split into chunks and batched so the model can learn cross-document links; similarities are temperature-normalized cosine scores that modulate the in-batch attention inside each transformer block.  \n\n**Key Results:** Across CoIR, BRIGHT, and BEIR, Revela beats or matches strong baselines while using only raw text: on CoIR it surpasses E5-Mistral-7B-Instruct by 2.8 points and outperforms E5-PT by 9.7 points at smaller scales. On BRIGHT it is on par with large supervised and API models, and on BEIR it reaches the unsupervised state of the art with about 1000× less data and 10× less compute, with gains that grow with model and batch size.", "questions": "1. Can we keep the LM frozen and train only the retriever while keeping most of the gains?\n2. How did you deal with the causal mask when using LLM as the retriever backbone?\n3. How robust is performance to batch composition choices, such as mixing documents from different topics or changing the interleaving pattern across chunks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "**Unified objective without labels:** The method removes the need for query–document pairs by turning retrieval into language modeling. I like the idea of using NTP for training retriever as it can provide fine-grained supervision compared to InfoNCE. There is a related work of “REPLUG” which also uses supervision from language modeling to train retriever. But the approach from this paper: adding a simple in-batch attention path to a standard decoder-only stack and learning both the retriever and LM with next-token loss has its novelty. \n\n**Strong results across domains:** It beats a larger supervised E5-Mistral-7B model on CoIR, stays competitive with top supervised and API systems on BRIGHT, and matches E5-PT on BEIR while using about 1000× less data and 10× less compute. The wins hold across different topics and corpus types, which suggests good robustness and transfer.\n\n**Good scaling and clear ablations:** Performance improves as you increase batch size, retriever size, and often LM size. Ablations show the cross-sequence attention and similarity weighting matter, and the model outperforms a matched Contriever baseline with the same backbone. Training on mixed domains keeps quality and can even help.", "weaknesses": "**Applicability**: Adoption may be harder than plug-and-play methods like REPLUG. Revela adds an extra in-batch attention path inside decoder blocks and uses retriever similarities as attention weights, which means touching the LM internals and maintaining custom masking. Even if the code changes are “minimal,” such modifications increase maintenance and may impact speed or memory in non-obvious ways.  \n\n**Batch-composition dependence:** The method conditions next-token prediction on *other documents in the same batch*. This can make results sensitive to how batches are formed or interleaved, and the paper does not deeply analyze robustness to batch construction.\n\n**Resources**: Compared to approaches using single retriever with InfoNCE loss, training will probably likely to require high resources compared to existing works, as it requires optimizing both the retrievers and LLMs. This can make the approach use less batch size given same compute budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0QkSq3VALj", "reviewer": "Reviewer_CDCx", "summary": "This paper introduces Revela, a self-supervised retrieval model training method that employs a language modeling task. Revela incorporates in-batch attention, allowing the model to consider both the current sequence and the contexts of other sequences within the same batch when predicting the next token, thereby modeling semantic dependencies between documents. This approach eliminates the need for manually labeled query-document pairs and directly trains the retriever using the modeling task, significantly improving the training efficiency and effectiveness of the model.", "questions": "1. In this paper, it is not explicitly mentioned whether the representation used during inference is from the last token or if mean pooling over all token representations is applied.\n\n2. Is the backbone model used in REPLUG also Llama 3.2?\n\n3. In addition to NDCG@10, could you provide results for other metrics such as MRR and recall?\n\n4. It seems that no ablation experiments are designed in this paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper demonstrates excellent performance across a variety of retrieval tasks with different types, showcasing its versatility.\n2. The method requires a smaller batch size. During training, the batch size used in this paper was 8, which is much smaller than other methods that employ contrastive learning for training. This significantly reduces the forest traversal cost. Additionally, the experiments clearly reveal the scaling law between retrieval performance and batch size.\n3. There is no need for labeled q-d pairs as training data, and the model can be directly trained in an unsupervised manner on large-scale text datasets.", "weaknesses": "1. This paper compares retrieval models that include both encoder-only and decoder-only models, and it also conducts comparisons based on different model sizes. However, there are still some fairness issues in the comparison.  The original RePlug paper uses query-document pairs for KL training in the RAG setting, while this paper only uses document alone in the reproduction and comparison. This might not be a fair comparison. Could training on the q-d pair data mentioned in RePlug, such as the Pile training data or other q-d datasets like MS MARCO, provide a fairer comparison? RePlug is this paper's main baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)