Venue: NAACL 2024

# CELI: A Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders

Xinyu Zhang, Minghan Li, Jimmy Lin

David R. Cheriton School of Computer Science, University of Waterloo, Canada

{x978zhan, m692li, jimmylin}@uwaterloo.ca

 Equal Contribution

###### Abstract

In text ranking, it is generally believed that the cross-encoders already gather sufficient token interaction information via the attention mechanism in the hidden layers. However, our results show that the cross-encoders can consistently benefit from additional token interaction in the similarity computation at the last layer. We introduce **CELI** (**Cross-Encoder with Late Interaction), which incorporates a late interaction layer into the current cross-encoder models. This simple method brings 5% improvement on BEIR without compromising in-domain effectiveness or search latency. Extensive experiments show that this finding is consistent across different sizes of the cross-encoder models and the first-stage retrievers. Our findings suggest that boiling all information into the [CLS] token is a suboptimal use for cross-encoders, and advocate further studies to investigate its relevance score mechanism.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 188–196

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

The two-stage retrieve-then-rerank pipeline has been the _de facto_ design for many information retrieval systems. Recently, both the retriever and the reranker systems have benefited from the advancement in pretrained language models Lin et al. (2022). When using the pretrained models as cross-encoders, the model views the query and document candidate together, allowing rich token interaction via the attention mechanism at all hidden layers. However, all the information then boils down to the representation of [CLS] token at the final stage, during the computation of the relevance score between the query and the document. This raises the concern of whether the single token representation is sufficient to capture all salient information.

Pretrained language models have also been adopted in the retriever stage in various ways. Karpukhin et al. (2020) pioneer in this direction and find that the [CLS] token embedding could well capture query or document representations, whose similarity can be used to indicate the relevance level between the query and document. This line of methods Karpukhin et al. (2020); Xiong et al. (2021), named as _single-vector dense retrievers_, while being effective for the in-domain scenarios, is found to be less robust on the out-of-distribution (OOD) datasets Thakur et al. (2021), possibly due to inadequate information at token-level granularity. Methods such as further pretraining or adding token-level interaction have been applied to improve the OOD generalization, where _multi-vector retrievers_Khattab and Zaharia (2020); Santhanam et al. (2022); Li et al. (2023) perform the best on both the in-domain and OOD effectiveness among neural retrievers Lin et al. (2023a). This ability is usually credited to its design that computes the similarity score based on contextualized embeddings of all tokens, which provides richer token interactions compared to the single-vector dense retrievers.

Inspired by the success of token interaction in the retriever systems, we ask the question: _Can cross-encoder also benefit from additional token interaction when computing the final similarity?_ In this work, we affirm this hypothesis, showing that additional token interaction in the final-stage similarity computation indeed improves the OOD capacity for cross-encoders. We name our method **CELI** (**Cross-Encoder with Late Interaction), which incorporates a late interaction layer into the current

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & **MS MARCO** & **BEIR Avg.** & **Search** \\  & MRR@10 & nDCG@10 & **Latency** \\ \hline monoBERT & 0.390 & 0.467 & 1.18s \\ CELI & 0.392 & 0.491 & 1.28s \\ \hline \hline \end{tabular}
\end{table}
Table 1: A preview of comparison between CELI and monoBERT. Detailed results are shown in Table 2 and 3.



cross-encoder models. As shown in Table 1, CELI improves averaged nDCG@10 on BEIR by 5% (from 0.467 to 0.491), while not sacrificing the in-domain score (0.390 vs. 0.392) and the search latency (1.18s vs. 1.28s). Extensive experiments show that the improvement is consistent over larger-sized models and reranking candidates from various retrievers.

## 2 Methods

**monoBERT**. monoBERT [Nogueira and Cho, 2019] is one of the first cross-encoders [MacAvaney et al., 2019, Nogueira et al., 2020] that apply pretrained transformers in passage retrieval. Given concatenated query \(q\) and document \(d\), the model computes relevance scores \(s_{q,d}\) from the [CLS] representation on the final layer of BERT [Devlin et al., 2019], formulated as follows [Lin et al., 2022, Pradeep et al., 2022]:

\[s_{m}(q,d)=T_{\texttt{[CLS]}}W+b, \tag{1}\]

where \(T_{\texttt{[CLS]}}\in\mathbf{R}^{D}\) is the [CLS] representation on the final layer, and \(W\in\mathbf{R}^{D\times 1}\) and \(b\in\mathbf{R}\) are the weight and bias for classification.

Some of the previous works term the models as "mono[BACKBONE]" when initialized from non-BERT pretrained Transformers, such as monoELECTRA [Pradeep et al., 2022]. However, since the underlying model structure remains the same, we refer to them all as monoBERT while specifying the backbones where the models are initialized.

**Mean-Pooling**. To study whether the improvement of CELI is attributed to the interaction between the query and the documents tokens, or simply the additional token information, we add the Mean-Pooling method as a baseline. Instead of computing the similarity score based solely on the [CLS] representation as in Eq. (1), it uses the mean representation of all the tokens:

\[s_{m}(q,d)=\frac{1}{n}\sum_{i}^{n}\left(T_{tok_{i}}W+b\right), \tag{2}\]

where \(T_{tok_{i}}\) is the final-layer representation of the \(i\)-th token, and \(n\) is the total number of tokens in the input sequence. \(W\in\mathbf{R}^{D\times 1}\) and \(b\in\mathbf{R}\) are the weight and bias for classification, same as Eq. (1).

**CELI**. In this work, we use the simplest version of late interaction proposed by Khattab and Zaharia [2020]. We first obtain the representation of each token in the query \(q\) and document \(d\):

\[v_{q_{i}}=T_{q_{i}}W+b;\quad v_{d_{j}}=T_{d_{j}}W+b, \tag{3}\]

where \(q_{i}\) and \(d_{j}\) represent the \(i\)-th token of query \(q\) and the \(j\)-th token of document \(d\), respectively. Similar to Eq. (1), \(T\in\mathbf{R}^{D}\) refers to each token representation on the final layer. \(W\in\mathbf{R}^{D\times D_{tok}}\) and \(b\in\mathbf{R}^{D_{tok}}\) are the weight and bias in a projection layer, projecting the \(T_{tok}\) to a lower dimension \(D_{tok}<D\).

With token representations \(v_{q_{i}}\) and \(v_{d_{j}}\), the late interaction first computes the inner product scores between all pairs of query and document tokens, then sums up the max similarity score for each query token against all document tokens:

\[s_{l}(q,d)=\sum_{q_{i}}\max_{d_{j}}(v_{q_{i}}^{T}v_{d_{j}}). \tag{4}\]

Eq. (4) shares the same formulation as in the first-stage retrievers, and only differ in that the token representations \(T_{q_{i}}\) and \(T_{d_{j}}\) embed information from both the query and document, whereas in first-stage retrievers, they are computed independently from each other, with \(T_{q_{i}}\) perceiving no information from document \(d\) and vice versa.

During training, we compute the LCE loss on \(s_{m}\) and \(s_{l}\), respectively:

\[\mathcal{L} =lce(s_{m}(q,d^{+}),s_{m}(q,d_{1}^{-}),...,s_{m}(q,d_{n}^{-}))\] \[+lce(s_{l}(q,d^{+}),s_{l}(q,d_{1}^{-}),...,s_{l}(q,d_{n}^{-})),\]

where \(d^{+}\) is the positive document and \(\{d_{i}^{-}\}_{i=1}^{n}\) are the negative documents to the query \(q\).

At inference time, we sum the two scores as the final relevance score, i.e., \(s_{final}=s_{m}+s_{l}\).1

Footnote 1: We have explored adding weighting terms for \(s_{m}\) and \(s_{c}\), but only observed marginal gains. Thus we report the simplest formulation here.

## 3 Experimental Setup

All cross-encoders are trained on MS MARCO [Bajaj et al., 2016], a dataset composed of queries from Bing search log and a collection of passages sourced from the general Web. It contains 8.8M passages, over 500k query-document pairs for training, and 6980 queries for inference.

We implement the model based on Capreolus [Yates et al., 2020a, b], an IR toolkit for end-to-end neural ad hoc retrieval. All training configurations follow Pradeep et al. [2022]: We train MS 

MARCO for 30k steps with a learning rate \(1e-5\) and a batch size 16. We use linear warmup on the first 3k steps, then linearly decay the learning rate on the rest of the steps. Cross-encoders are trained on LCE loss Gao et al. (2021); Pradeep et al. (2022) with 7 negative samples.2 We experimented with three backbones, all available on HuggingFace Wolf et al. (2020): MiniLM Wang et al. (2020),3 ELECTRAbase,4 and ELECTRLarge Clark et al. (2020).5

Footnote 2: We use Quadro RTX 8000 GPUs and A6000 for the experiments. On RTX 8000, the ELECTRAbase models took approximately 8 hours for cross-encoder training.

Footnote 3: microsoft/MiniLM–L12–H384—uncased ‘google/electrica-base-discriminator’google/electrica-large-discriminator’google/electronica-large-discriminator’ [https://github.com/stanford-futuredata/ColBERT](https://github.com/stanford-futuredata/ColBERT)

We use MS MARCO Bajaj et al. (2016) for the in-domain evaluation and 13 datasets from BEIR Thakur et al. (2021) for OOD evaluation, which covers 10 domains including Wikipedia, Finance, Scientific, Quora, and so on.

At the inference stage, we always rerank top-\(1k\) results from the first-stage retrievers. On MS MARCO, we use TCT-ColBERT Lin et al. (2021) as the retriever following Pradeep et al. (2022). On BEIR, we use an extensive list of retrievers that covers the categories of sparse, single- and multi-vector dense retrievers. Retrievers results are produced using Pyserini Lin et al. (2021), BEIR Thakur et al. (2021), or ColBERT Khattab and Zaharia (2020) repository.6 Following the datasets standard, we report MRR@10 on MS MARCO and nDCG@10 on BEIR.

Footnote 4: [https://github.com/stanford-futuredata/ColBERT](https://github.com/stanford-futuredata/ColBERT)

## 4 Results and Analysis

Table 1 provides a preview of the efficacy of CELI. In this section, we first demonstrate that such improvement is consistent over different model sizes and the first-stage retrievers, then analyze how the projected token dimension and the query length impact the improvement.

### Model Size

Previous papers find that models with a larger number of parameters can better generate on unseen distribution Ni et al. (2022). Motivated by this observation, we examine whether the improvement brought by late interaction diminishes with increasing model sizes.

Results show that the contribution of late interaction is consistent over model size. Table 2 shows in-domain and OOD scores with the models initialized from three different sizes of backbones: MiniLM, ELECTRAbase, and ELECTRAlarge.7 The MS MARCO results reranks the top-1k candidates from TCT-ColBERT, and the BEIR results reranks the top-1k candidates from BM25.

Footnote 7: MiniLM, ELECTRAbase, and ELECTRAlarge have 33M, 110M, and 340M parameters respectively.

While we observe higher average scores on BEIR as the model size increases, echoing the previous finding that larger models demonstrate better OOD generalization ability, the improvement brought by token information is consistent across the backbones. On all three models, CELI consistently improves over the two baselines. Additionally, the in-domain scores on the other two backbones are not affected as well, suggesting that the "free" gain is consisten ove 



### First-Stage Retriever

We then extend the experiments into an extensive list of first-stage retrievers, where the retrievers are categorized as sparse, single-vector, and multi-vector dense retrievers.

Table 3 shows the results on BEIR reranking candidates from 9 different retrievers, covering all three categories mentioned above. Looking at the averaged nDCG@10 on BEIR, we find that late interaction consistently improves the OOD capacity when using retrievers of different natures, bringing a similar degree of improvement of 0.02-0.03.

### Token Dimensions

In first-stage retrieval, it is common to project the token representation into lower dimensions as restricted by indexing storage space and search efficiency. However, the representations are computed on the fly for cross-encoders, thus using token representations in higher dimensions brings no additional storage cost and only minor searching latency in the context of cross-encoders. We therefore examine whether using higher token dimensions \(D_{\text{tok}}\) brings additional improvements.

Results are shown in Table 4, where row (2) corresponds to the BM25 results reported in Table 3. Comparing rows (1-4), we find that the token dimensions have little impact on the OOD effectiveness: surprisingly, using \(dim=1\) already obtains \(0.4890\) on average BEIR as shown in on row (1), while increasing the dimension to \(dim=32\) and onwards only provides marginal improvement.

### Query Length

Finally, we present our analysis of how the late interaction improves the OOD capacity of cross-encoders, finding that query length is a prominent indicator of the per-query improvement. Figure 1 plots the distribution of nDCG@10 improvement by late interaction according to the query length on Quora and HotpotQA, two datasets included in BEIR.8 Specifically, each point represents the average of nDCG@10 improvements over the query of the same length (same coordinate on the x-axis). We additionally plot an approximated polynomial line based on the points to better reveal the relationship between the query length and the improvement on nDCG@10.

Footnote 8: Length determined as the number of query tokens delimited by whitespace.

On both datasets, we observe a clear tendency that the late interaction brings higher improvement on longer queries. While Figure 1 is based on results using BM25 as the retriver, we have similar observations when reranking candidates from the other retrievers.

## 5 Related Work

Nogueira and Cho (2019) is one of the first cross-encoders that apply pretrained language models on the passage retrieval task. It considers retrieval as a classification task and uses transformer encoders following the formulation of the next sentence prediction (NSP) pretraining task in BERT, where only the [CLS] vector is used to classify the query-document pair and compute the relevant score. Afterward, CEDR MacAvaney et al. (2019) proposes to incorporate fine-grained token interaction. However, it requires extra complex computations at all layers, which brings difficulty to implementation and adds higher computational overhead.

This line of cross-encoders has been studied and extended to other model architectures: Nogueira et al. (2020) and Zhuang et al. (2023) build cross-encoders on encoder-decoder architecture (e.g., T5, Raffel et al., 2020), and Ma et al. (2023) extend it

Figure 1: nDCG@10 improvement from late interaction on queries over different lengths. Each point represents the average of nDCG@10 improvements over the query of the corresponding length. The line is the least square polynomial fit of the points.



to decoder-only architecture (e.g. LLaMA-2, Touvron et al., 2023). Another line of cross-encoders reranks the document candidates according to the query likelihood given a passage, usually based on generative models (Nogueira dos Santos et al., 2020; Sachan et al., 2022).

Recent works on first-stage retrieval have demonstrated the effectiveness of adding sparse information into dense retrieval Chen et al. (2022). The combination of the token information and dense [CLS] vector could also be done explicitly, by either adding the scores computed from [CLS] and token information or concatenating aggregated token vectors to the [CLS] vector Gao et al. (2021); Lin et al. (2023). The multi-vector dense models could also be viewed under this category, where the token representation vectors jointly contribute to the relevancy computation along with the [CLS] vector Khattab and Zaharia (2020); Li et al. (2023).

Our work is also connected to the interaction-based methods predating pretrained language models, where the text relevance is usually predicted based on the fine-grained similarity matrix between queries and document tokens Socher et al. (2011); Lu and Li (2013); Hu et al. (2014); Pang et al. (2016).

## 6 Conclusion

In this work, we show that adding late interaction to existing cross-encoders brings visible improvement to its OOD capacity without hurting in-domain effectiveness, even though the cross-encoder already processes the token interaction in earlier layers. Extensive experiments on different model sizes and first-stage retrievers show that this improvement is consistent, and according to our analysis, the improvement is more prominent on longer queries. Our findings suggest that boiling all information into the [CLS] token is a suboptimal use for cross-encoders, and further studies are required to better explore their capacities.

## 7 Limitations

While CELI serves as a simple yet effective approach to improve the OOD generalization capacity for cross-encoders, it is not a novel architectural innovation. Instead, it draws inspiration from first-stage retrievers Khattab and Zaharia (2020). That said, We prioritize this simple design approach because we value ease of use and simplicity over novelty in this context.

## 8 Acknowledgement

This research was supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada.

## References

* Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. _ArXiv_, abs/1611.09268.
* Chen et al. (2022) Xilun Chen, Kwshal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliey, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2022. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 250-262, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In _International Conference on Learning Representations_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Gao et al. (2021) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021a. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, Online. Association for Computational Linguistics.
* April 1, 2021, Proceedings, Part II_, page 280-286, Berlin, Heidelberg. Springer-Verlag.
* Hu et al. (2014) Baotiian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. _ArXiv_, abs/1503.03244.
* Karpukhin et al. (2019) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and 

Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781.
* Khattab and Zaharia [2020] Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In _Proceedings of the 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020)_, pages 39-48.
* Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Li et al. [2023] Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. CITADEL: Conditional token interaction via dynamic lexical routing for efficient and effective multi-vector retrieval. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11891-11907, Toronto, Canada. Association for Computational Linguistics.
* Lin et al. [2021a] Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021a. Pysumin a Python toolkit for reproducible information retrieval research with sparse and dense representations. In _Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021)_, pages 2356-2362.
* Lin et al. [2022] Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2022. _Pretrained transformers for text ranking: BERT and beyond_. Springer Nature.
* Lin et al. [2023a] Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023a. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 6385-6400, Singapore. Association for Computational Linguistics.
* Lin et al. [2023b] Sheng-Chieh Lin, Minghan Li, and Jimmy Lin. 2023b. Aggetriever: A simple approach to aggregate textual representations for robust dense passage retrieval. _Transactions of the Association for Computational Linguistics_, 11:436-452.
* Lin et al. [2021b] Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021b. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_, pages 163-173, Online. Association for Computational Linguistics.
* Lu and Li [2013] Zhengdong Lu and Hang Li. 2013. A deep architecture for matching short texts. In _Advances in Neural Information Processing Systems_, volume 26. Curran Associates, Inc.
* Ma et al. [2023] Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-tuning LLaMA for multi-stage text retrieval. _ArXiv_, abs/2310.08319.
* MacAvaney et al. [2019] Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. Cedr: Contextualized embeddings for document ranking. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR'19, page 1101-1104, New York, NY, USA. Association for Computing Machinery.
* Ni et al. [2022] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9844-9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Nogueira and Cho [2019] Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. _ArXiv_, abs/1901.04085.
* Nogueira et al. [2020] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 708-718, Online. Association for Computational Linguistics.
* dos Santos et al. [2020] Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020. Beyond [CLS] through ranking by generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1722-1727, Online. Association for Computational Linguistics.
* Pang et al. [2016] Liang Pang, Yanyan Lan, J Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. _ArXiv_, abs/1602.06359.
* Pradeep et al. [2022] Ronak Pradeep, Yuqi Liu, Xinyu Zhang, Yilin Li, Andrew Yates, and Jimmy Lin. 2022. Squeezing water from a stone: A bag of tricks for further improving cross-encoder effectiveness for reranking. In _Advances in Information Retrieval_, pages 655-670, Cham. Springer International Publishing.
* Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21(1).
* Sachan et al. [2022]Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In _Proceedings of 

the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3781-3797, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Santhanam et al. (2022) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, Seattle, United States. Association for Computational Linguistics.
* Socher et al. (2011) Richard Socher, Eric Hsin-Chun Huang, Jeffrey Pennington, A. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In _Neural Information Processing Systems_.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. LLAMA 2: Open foundation and fine-tuned chat models. _ArXiv_, arXiv:2307.09288.
* Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_, NIPS'20, Red Hook, NY, USA. Curran Associates Inc.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Sheifer, Patrick von Platen, Clara Ma, Yacinne Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariana Drame, Quentin Lohest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online.
* Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _Proceedings of the 9th International Conference on Learning Representations (ICLR 2021)_.
* Yates et al. (2020a) Andrew Yates, Siddharnot Arora, Xinyu Zhang, Wei Yang, Kevin Martin Jose, and Jimmy Lin. 2020a. Capreous: A toolkit for end-to-end neural ad hoc retrieval. In _Proceedings of the 13th International Conference on Web Search and Data Mining_, WSDM '20, page 861-864, New York, NY, USA. Association for Computing Machinery.
* Yates et al. (2020b) Andrew Yates, Kevin Martin Jose, Xinyu Zhang, and Jimmy Lin. 2020b. Flexible ir pipelines with capredus. In _Proceedings of the 29th ACM International Conference on Information Knowledge Management_, CIKM '20, page 3181-3188, New York, NY, USA. Association for Computing Machinery.
* Zhuang et al. (2023) Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. RankT5: Fine-tuning T5 for text ranking with ranking losses. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '23, page 2308-2313, New York, NY, USA. Association for Computing Machinery.

 

Results on BEIR

Due to the space limitation, we only report the averaged scores on BEIR in the main paper. In this section, Table 5 and Table 6 presents the full nDCG@10 scores on each BEIR dataset, corresponding to the Table 2 in Section 4.1 (**Model Size**), and Table 3 in Section 4.2 (**First-Stage Retriever**).

## Appendix B License

The MS MARCO dataset is licensed under Creative Commons Attribution 4.0 International, whereas BEIR datasets and Capreolus toolkit are under Apache License 2.0. The usage of the artifacts in this work is consistent with their intended use. Since our codebase is extended from Capeolus, it would inherit the Apache License 2.0.

 



\begin{table}
\begin{tabular}{l l|c c c c c c c c c c c c c c c} \hline \hline  & & & & & & & & & & & & & & \\
**Fist Stage** & **Model** & **Avg** & \multicolumn{1}{c}{TREC-} & NF & NO & Hotpot & FQA & Arga & Touche- & Qoera & DB & SCI & FEVER & Climate- & Sci \\  & & (MRR@10) & **Avg** & COVID & Corpus & NQ & QA & Aaa & 2020 & Pedia & DOCS & PEVER & Fact \\ \hline \multirow{3}{*}{**BM25**} & monoBERT & 0.467 & 0.699 & 0.355 & 0.504 & 0.620 & 0.359 & 0.335 & 0.308 & 0.722 & 0.426 & 0.151 & 0.754 & 0.164 & 0.679 \\  & mean-Boiling & 0.391 & 0.481 & 0.707 & 0.351 & 0.502 & 0.609 & 0.356 & 0.308 & 0.308 & 0.429 & 0.154 & 0.721 & 0.185 & 0.681 \\  & CEL1 & 0.392 & 0.491 & 0.705 & 0.349 & 0.501 & 0.673 & 0.360 & 0.527 & 0.324 & 0.784 & 0.424 & 0.155 & 0.723 & 0.172 & 0.691 \\ \cline{2-13}  & monoBERT & 0.400 & 0.481 & 0.727 & 0.362 & 0.523 & 0.660 & 0.389 & 0.291 & 0.317 & 0.773 & 0.436 & 0.152 & 0.748 & 0.112 & 0.669 \\ \cline{2-13}  & Mean-Pooling & 0.403 & 0.483 & 0.732 & 0.358 & 0.519 & 0.718 & 0.389 & 0.325 & 0.321 & 0.747 & 0.439 & 0.149 & 0.738 & 0.155 & 0.689 \\  & CEL1 & 0.402 & 0.494 & 0.736 & 0.368 & 0.527 & 0.714 & 0.401 & 0.443 & 0.320 & 0.690 & 0.449 & 0.162 & 0.740 & 0.152 & 0.715 \\ \hline \multirow{3}{*}{**ELECTRAUnce**} & monoBERT & 0.413 & 0.507 & 0.801 & 0.380 & 0.589 & 0.733 & 0.453 & 0.250 & 0.339 & 0.772 & 0.468 & 0.181 & 0.791 & 0.149 & 0.719 \\  & Mean-Pooling & 0.412 & 0.516 & 0.784 & 0.378 & 0.584 & 0.748 & 0.444 & 0.332 & 0.325 & 0.791 & 0.456 & 0.180 & 0.799 & 0.215 & 0.706 \\  & CEL1 & 0.413 & 0.524 & 0.796 & 0.378 & 0.559 & 0.735 & 0.457 & 0.436 & 0.335 & 0.800 & 0.460 & 0.182 & 0.769 & 0.179 & 0.733 \\ \hline \end{tabular}
\end{table}
Table 5: MRR@10 on MS MARCO and nDCG@10 scores on BEIR of CELI and two baselines (i.e., monoBERT and Mean-Pooling). Cross-encoders are initialized from MiniLM, ELECTRAbase, and ELECTRAlarge. Results on BEIR rerank the top-1k passages from BM25.