GradNormIR: When Should We Update the Dense Retriever in Evolving Corpora?
Authors: Dayoon Ko, Jinyoung Kim, Sohyeon Kim, Jinhyuk Kim, Jaehoon Lee, Seonghak Song, Minyoung Lee, Gunhee Kim
Accepted at: acl25

When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR

Dayoon Ko1    Jinyoung Kim1    Sohyeon Kim2    Jinhyuk Kim3
Jaehoon Lee3    Seonghak Song3    Minyoung Lee3    Gunhee Kim1

1Seoul National University   2POSTECH   3Samsung SDS
dayoon.ko@vision.snu.ac.kr   jiny1623@snu.ac.kr   soohyun@postech.ac.kr   gunhee@snu.ac.kr
https://github.com/dayoon-ko/gradnormir

Abstract

Dense retrievers encode texts into embeddings to efficiently retrieve relevant documents from large databases in response to user queries. However, real-world corpora continually evolve, leading to a shift from the original training distribution of the retriever. Without timely updates or retraining, indexing newly emerging documents can degrade retrieval performance for future queries. Thus, identifying when a dense retriever requires an update is critical for maintaining robust retrieval systems. In this paper, we propose a novel task of predicting whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. Addressing this task allows us to proactively manage retriever updates, preventing potential retrieval failures. We introduce GradNormIR, an unsupervised approach that leverages gradient norms to detect OOD corpora effectively. Experiments on the BEIR benchmark demonstrate that GradNormIR enables timely updates of dense retrievers in evolving document collections, significantly enhancing retrieval robustness and efficiency.

1 Introduction

With the exponential growth of digital content, information retrieval (IR) systems have become essential for delivering relevant information from massive document repositories (Bajaj et al., 2016; Kwiatkowski et al., 2019). Unlike traditional sparse retrieval methods (Robertson et al., 2009; Ramos et al., 2003) that rely heavily on lexical overlap, dense retrievers (Karpukhin et al., 2020; Izacard et al., 2022) utilize semantic embeddings to better understand query intent and retrieve documents with similar conceptual meanings, thus overcoming the constraints of exact term matching. Consequently, dense retrievers have gained significant attention in tasks demanding high semantic precision, such as question answering and personalized search. During training, these dense retrievers are optimized to enhance embedding similarity between queries and relevant passages while reducing similarity for irrelevant ones (Karpukhin et al., 2020; Izacard et al., 2022). Document embeddings are then precomputed and stored during indexing, enabling rapid retrieval during inference by identifying documents most similar to the test query embeddings.

In the real world, corpora evolve rapidly due to technological advancements, societal changes, and emerging trends. This continuous evolution poses a substantial challenge for dense retrievers (Chen et al., 2023), which often struggle to generalize effectively to unseen documents in zero-shot

![Motivation diagram showing three evolving corpora and their corresponding dense retriever responses, illustrating the need for detecting OOD corpora.](page_749_355_646_388.png)

Figure 1: Motivation. In evolving corpora, indexing documents that dense retrievers fail to generalize to can severely degrade retrieval performance. Therefore, proactively detecting OOD corpora prior to indexing, without relying on available queries, is essential for maintaining retrieval effectiveness. To address this, we propose GradNormIR, an unsupervised method leveraging gradient norms to identify such OOD corpora.
scenarios (Chen et al., 2022). The problem is particularly critical in retrieval-augmented generation (RAG) systems (Lewis et al., 2020), where retriever performance directly impacts downstream tasks (Petroni et al.; Li et al., 2023a; Ko et al., 2024; Kim et al., 2024). For instance, consider the scenario illustrated in Figure. 1: when a new corpus about Google’s quantum computing chip, Willow, is introduced, a dense retriever trained on existing content, such as the song Willow by Taylor Swift, may erroneously retrieve irrelevant documents. A query like “Tell me how the Willow works!” could mistakenly return information about the song Willow instead of the quantum chip. Anticipating when a retriever might fail due to such distributional shifts is crucial. Also, proactively identifying when to update the retriever ensures robustness and accuracy in dynamically evolving document streams.

This challenge closely relates to the OOD generalization problem. Several approaches in information retrieval (IR) aim to improve a retriever’s performance on queries or documents significantly different from those encountered during training (Izacard et al., 2022; Wang et al., 2021; Chen et al., 2022; Yu et al., 2022; Kasela et al., 2024; Besta et al., 2024; Chen et al., 2023). One prominent approach utilizes a mixture-of-experts framework, employing a gating mechanism to select the most suitable expert retriever for each test query (Kasela et al., 2024; Lee et al., 2024). However, these methods depend heavily on predefined expert retrievers trained offline with explicit domain knowledge and established domain boundaries, making them less flexible for dynamically evolving corpora. Determining the appropriate timing to introduce new experts for emerging content remains challenging.

To address this challenge, we introduce a novel practical task of predicting OOD corpora before indexing for a given dense retriever. Identifying OOD corpora in advance signals when retriever updates are needed. By detecting such corpora before indexing, we can proactively select a more suitable dense retriever or promptly update the current one, safeguarding inference-time performance. To achieve this, we propose GradNormIR, an unsupervised method designed to detect OOD documents within a corpus without relying on queries. Inspired by successful applications in image classification, where gradient norms effectively detect OOD images and estimate test-time accuracy without labeled data (Huang et al., 2021; Xie et al., 2024), GradNormIR leverages gradient norms from the contrastive loss to evaluate a retriever’s generalizability on a given corpus. Specifically, we introduce novel sampling strategies to effectively assign positive and negative instances for computing the contrastive loss.

We assess our method on the BEIR benchmark, which encompasses multiple diverse datasets across various domains. Firstly, we demonstrate that GradNormIR effectively detects OOD documents likely to cause retrieval failures. Subsequently, we show that GradNormIR can select the most suitable retriever using only the corpus, without queries. Finally, we simulate evolving corpora using BEIR by sequentially introducing datasets following Ge et al. (2023), and demonstrate how GradNormIR enables efficient retriever updates while maintaining performance. Our experiments validate both the importance of OOD detection for retrieval systems and GradNormIR’s efficacy in adapting to evolving corpora.

In summary, our contributions are as follows:

1. We introduce a novel task of predicting OOD corpora before indexing, facilitating efficient and effective retriever updates in evolving corpora.

2. We propose GradNormIR, an unsupervised method that leverages gradient norms and novel sampling strategies to detect OOD documents and predict OOD corpora.

3. On the BEIR benchmark, we demonstrate both the necessity of the proposed task for a robust retrieval system and the effectiveness of GradNormIR via three practical use cases.

2 Related Work

Information Retrieval. Recent advancements in text embeddings have significantly transformed the field of IR, particularly with the emergence of dense retrievers. The success of these models has primarily been driven by the availability of large-scale training datasets, including NQ (Kwiatkowski et al., 2019), MS-MARCO (Bajaj et al., 2016), HotpotQA (Yang et al., 2018), and NLI (Gao et al., 2021). A notable example is DPR (Karpukhin et al., 2020), which employs a dual-encoder architecture for open-domain question-answering, independently embedding queries and passages.

Unsupervised approaches have also garnered attention for their ability to enhance the generalization of dense retrievers. Contriever (Izacard
et al., 2022) enlarges pre-training data using unsupervised data augmentation techniques for contrastive learning. Similarly, E5 (Wang et al., 2022) leverages weak supervision to create a large-scale dataset, CCPairs, filtered by consistency criteria. Recently, hybrid methods like BGE-M3 (Chen et al., 2024) have integrated dense, sparse, and multi-vector retrieval strategies via self-knowledge distillation, further advancing retrieval effectiveness.

OOD Robustness. In IR, OOD robustness refers to a model’s capacity to maintain retrieval effectiveness when exposed to documents that deviate significantly from its training distribution. A prominent benchmark to evaluate this robustness is BEIR (Thakur et al., 2021), which encompasses diverse retrieval tasks across multiple domains. Using BEIR, Chen et al. (2022) demonstrated that dense retrievers often underperform on OOD datasets compared to traditional lexical retrievers like BM25. In response, they proposed a hybrid model combining dense and sparse retrieval, achieving robust performance in zero-shot scenarios. Similarly, Yu et al. (2022) showed that distribution shifts lead to a substantial decline in zero-shot accuracy of dense retrievers.

Various strategies have been proposed to improve OOD performance on unseen documents. Data augmentation has yielded promising results (Wang et al., 2021; Izacard et al., 2022). Architectural adaptations have also been explored; mixture-of-experts frameworks (Kasela et al., 2024; Lee et al., 2024) and multi-head RAG models (Besta et al., 2024) adjust retrieval strategies based on domain characteristics. Moreover, Khramtsova et al. (2023) investigated zero-shot retrieval methods for selecting the most suitable retriever, while Khramtsova et al. (2024) proposed leveraging LLM-generated pseudo-queries to rank dense retrievers. Other studies (Cai et al., 2023; Chen et al., 2023) have adopted continual learning methods to handle dynamic corpora without forgetting previously acquired knowledge. For instance, memory-based approaches (Cai et al., 2023) maintain backward compatibility with existing document embeddings, and incremental indexing strategies (Chen et al., 2023) dynamically update document indices in generative retrievers to accommodate new information.

Despite these advancements, few studies have explicitly examined how to identify OOD documents from the perspective of dense retrievers. Layer-wise score aggregation methods, such as those proposed by Darrin et al. (2024), combine anomaly scores from each encoder layer to provide more accurate detection. However, this approach is primarily tailored to text classification tasks, whereas our work explicitly addresses evaluating the generalizability of dense retriever models across evolving document corpora.

3 Problem Statement

3.1 OOD Robustness in IR

OOD robustness refers to the ability of a model to maintain effective performance when encountering data distributions different from the training distribution. Following Liu et al. (2024), we define OOD robustness in IR formally as:

\[
|\mathcal{R}_M(f_\theta; \mathcal{D}_{\text{test}}, K) - \mathcal{R}_M(f_\theta; \tilde{\mathcal{D}}_{\text{test}}, K)| \leq \delta,
\]
where \( \mathcal{D}_{\text{train}}, \mathcal{D}_{\text{test}} \sim \mathcal{G}, \ \tilde{\mathcal{D}}_{\text{test}} \sim \tilde{\mathcal{G}}. \) (1)

Here, \( f_\theta \) is a dense retriever trained on the training set \( \mathcal{D}_{\text{train}} \) drawn from the original distribution \( \mathcal{G} \). \( \mathcal{R}_M(f_\theta; \mathcal{D}, K) \) denotes the retrieval performance metric (e.g., Recall@K) for the top-K retrieved results by \( f_\theta \), and \( \delta \) represents an acceptable error margin. The test set \( \mathcal{D}_{\text{test}} \) is drawn from the original distribution \( \mathcal{G} \), while the new test set \( \tilde{\mathcal{D}}_{\text{test}} \) originates from a distinct distribution \( \tilde{\mathcal{G}} \). A retriever \( f_\theta \) satisfying Eq. (1) is considered \( \delta \)-robust against OOD data under metric \( M \).

3.2 OOD Document

Based on the OOD robustness, we first define an ideal criterion for identifying an OOD document, assuming the availability of queries for evaluation. Given a relevant query-document pair \( (q, d) \) with \( d \in \mathcal{C} \), a document \( d \) is considered OOD if the retriever \( f_\theta \) fails to retrieve \( d \) within the top-K results:

\[
\mathrm{OOD}(d; f_\theta, q, \mathcal{C}) = \mathbb{1}[d \notin f_\theta(q, \mathcal{C})],
\]

where \( \mathbb{1} \) is an indicator function. This definition directly reflects the retriever’s inability to generalize to the document \( d \).

In practice, however, queries are often unavailable when a corpus is introduced. Thus, our proposed method \( \mathcal{M} \) (see Section 4), named GradNormIR, predicts such OOD documents proactively in an unsupervised manner, without relying on labeled query-document pairs. The labeled
query-document pairs are used only for evaluating the effectiveness of our unsupervised predictions.

3.3 OOD Corpus

Based on the predicted OOD documents, we aim to determine whether a given corpus \( \mathcal{C} \) itself is OOD for the retriever \( f_\theta \). The likelihood of \( \mathcal{C} \) being OOD can be quantified by the proportion of predicted OOD documents it contains:

\[
r(\mathcal{C}) = \frac{|\tilde{\mathcal{C}}|}{|\mathcal{C}|}, \quad \text{with } \tilde{\mathcal{C}} = \{ d \in \mathcal{C} \mid \mathcal{M}(d; f_\theta, \mathcal{C}) = 1 \}.
\]

A corpus \( \mathcal{C} \) is classified as OOD if the ratio \( r(\mathcal{C}) \) exceeds a predefined threshold \( \gamma \). This criterion enables proactive detection of corpus-level distributional shifts, facilitating timely retriever updates to maintain retrieval effectiveness.

4 Approach

We first review prior work leveraging gradient norms in image classification. We then introduce GradNormIR, an unsupervised method to proactively detect OOD documents and consequently identify OOD corpora.

4.1 Preliminary of Gradient Norm

Previous studies utilize the gradient norm as an indicator of model performance and uncertainty in image classification. GradNorm (Huang et al., 2021) estimates uncertainty by computing the gradient norm derived from the KL divergence between the softmax output and a uniform distribution, identifying smaller gradient norms with higher uncertainty (OOD images). GDScore (Xie et al., 2024) estimates test-time accuracy in an unsupervised manner by pseudo-labeling the input, computing cross-entropy loss, and measuring the gradient vector norm of the final layer.

Unlike these methods, our work applies gradient norms in information retrieval (IR) to proactively detect OOD documents by employing novel sampling strategies to compute the gradient norm of contrastive loss.

4.2 GradNormIR

We use the gradient norm to detect OOD documents in a corpus. To get the gradient norm, we need to calculate the loss. Dense retrievers are usually trained with InfoNCE loss, defined as

\[
\mathcal{L}_{\text{InfoNCE}} = - \log \frac{e^{s(q, d^+)/\tau}}{e^{s(q, d^+)/\tau} + \sum_{i=1}^N e^{s(q, d_i^-)/\tau}},
\]

where \( s(q, d) = \cos(f_\theta(q), f_\theta(d)) \) is the cosine similarity between query \( q \) and document \( d \). \( f_\theta(\cdot) \) is the last hidden layer’s output, and \( \tau \) is a temperature parameter.

When a new corpus \( \mathcal{C} \) is given, user queries are not yet available, making it challenging to compute the gradients. Therefore, we consider each document \( d \) as a query and assign pseudo-labels of positives and negatives to other documents \( \mathcal{C} \setminus \{d\} \) that are relevant or irrelevant with \( d \), respectively. Instead of using external trained models for such labeling, we obtain pseudo-labels directly from the retriever’s own internal similarity scores.

Query Representation with Dropout. As discussed, every \( d \) is regarded as a document query. To better reflect the retriever’s generalizability in the gradient norm, we introduce perturbations to the representation of \( d \). Following Jeong et al. (2022), we apply stochastic dropout to randomly mask some parts of \( d \)’s representation \( f_\theta(d) \). If \( f_\theta(d) \) generalizes well to \( d \), masking some tokens in its embedding has little impact on selecting its positive and negative samples. Otherwise, if the retriever poorly generalizes to document \( d \), such masking induces significant shifts in the embedding space, causing incorrect positive and negative sample selections, and thereby yielding larger gradient norms.

Specifically, we first encode the document query with the last hidden state \( h = f_\theta(d) \). We then randomly mask the hidden state; the mask \( m \) is sampled from a Bernoulli distribution:

\[
h' = h \odot m, \text{ where } m \sim \text{Bernoulli}(p),
\]

where \( \odot \) denotes element-wise multiplication. Finally, we obtain the perturbed document query \( d' \) by applying pooling on \( h' \).

Positive and Negative Sampling. To effectively compute the gradient norm without actual queries,

![Dropout for the document query representation along with positive and hard negative sampling.](page_820_153_601_181.png)

Figure 2: Dropout for the document query representation along with positive and hard negative sampling.
we introduce an internal pseudo-labeling approach inspired by (Xie et al., 2024). The overall process is illustrated in Figure 2. Specifically, given a perturbed document query representation \( d' \), we first retrieve the top-\( k \) nearest documents from \( \mathcal{C} \setminus \{d\} \) using \( k \)-nearest neighbors (\( k \)-NN). These documents form the positive candidate pool \( D^+(d) \), assuming that documents with higher similarity scores are likely to be relevant.

From this positive pool, we further select the top-\( p \) most similar documents \( \{d_1^+, \ldots, d_p^+\} \) as positives for computing the loss, ensuring precise estimation of the retriever’s generalizability. The negative pool \( D^-(d) \) comprises the remaining documents, assumed to be irrelevant or less relevant to \( d \).

For negative samples, previous work (Zhan et al., 2021) has demonstrated that using hard negatives—documents similar to positives but irrelevant to the query—improves model sensitivity and performance. Therefore, we adopt a hard negative sampling strategy: for each positive \( d_i^+ \), we select its top-\( n \) nearest documents \( \{d_{i1}^-, \ldots, d_{in}^- \} \) from the negative pool \( D^-(d) \). These hard negatives increase gradient sensitivity, enabling effective detection of documents for which the retriever generalizes poorly.

Gradient Norm. We compute the gradient norm of \( d \) based on the InfoNCE loss gradients with respect to the retriever parameters \( \theta \):

\[
\nabla \mathcal{L}_\theta = -\nabla_\theta \log \frac{e^{s(d,d_i^+)/\tau}}{e^{s(d,d_i^+)/\tau} + \sum_{j=1}^n e^{s(d,d_j^-)/\tau}},
\]

where \( d_i^+ \) is a positive sample and \( d_{ij}^- \in D^-(d_i^+) \) are the corresponding hard negatives. The gradient norm measures the sensitivity of the retriever parameters to these samples.

Finally, the average gradient norm across all positive samples \( \{d_i^+\}_{i=1}^p \) is

\[
\text{GradNormIR} = \frac{1}{p} \sum_{i=1}^p \| \nabla_\theta \mathcal{L} \|_2,
\]

where \( \| \cdot \|_2 \) denotes the \( L_2 \)-norm. This average gradient norm serves as a measure of the retriever’s generalizability on \( d \in \mathcal{C} \). A higher value indicates a greater sensitivity and potentially less stability when adapting to \( d \).

Predicting OOD Documents. We classify document \( d \) as OOD if its gradient norm exceeds a threshold derived from the median gradient norm of known in-domain documents. The median provides robustness against outliers, ensuring reliable and stable OOD detection (Leys et al., 2013). We describe threshold selection as well as other hyperparameters in Sec 5.1.

5 Experiments

We conduct three sets of experiments to evaluate our approach. First, Next, we verify that GradNormIR’s OOD detection is effective for selecting the most suitable retriever. Next, we make sure that GradNormIR’s OOD detection is useful in selecting the most suitable retriever, even without any queries. Finally, in evolving corpora, we demonstrate that GradNormIR enables efficient continuous retriever updates by selectively retraining it only on the predicted OOD corpus. We also present an ablation study for several hyperparameters in Appendix C.

5.1 Experimental Setup

Dense Retrievers. We evaluate several state-of-the-art dense retriever models, including BGE (Xiao et al., 2023), Contriever (Izacard et al., 2022), E5 (Wang et al., 2024), and GTE (Li et al., 2023b).

Dataset. The BEIR benchmark (Thakur et al., 2021) provides a diverse collection of datasets for evaluating retriever models across multiple domains. From the 19 available datasets, we exclude those used for fine-tuning the tested retrievers (e.g., MSMARCO, Natural Questions, FEVER, HotpotQA, CQADupStack), as well as those that are no longer accessible (e.g., TREC-News, Robust04, Signal-1M, BioASQ), following Khramtsova et al. (2023). This leaves us with 10 datasets for evaluation. Each dataset consists of a document corpus and query-document pairs. In our experiment, we define the corpus \( \mathcal{C} \) as the set of documents with at least one annotated relevant query, ensuring a quantitative evaluation.

Baselines. We compare our method with three baselines: (i) Layerwise (Izacard et al., 2022): unsupervised textual OOD detection via layerwise anomaly scores (e.g., negative cosine similarity), (ii) IPQ (Chen et al., 2023): incremental production quantization with clustering, and (iii) GenQuery (Khramtsova et al., 2024): zero-shot ranking using pseudo-questions generated by large language models. Implementation details for each baseline are provided within each respective experiment.
<table>
  <tr>
    <th>Retriever</th>
    <th>Documents</th>
    <th>ArguAna</th>
    <th>C-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>COVID</th>
    <th>Touche</th>
    <th>Avg. (\( \downarrow \))</th>
  </tr>
  <tr>
    <td rowspan="4">BGE</td>
    <td>All</td>
    <td>99.68</td>
    <td>79.96</td>
    <td>59.67</td>
    <td>80.25</td>
    <td>21.39</td>
    <td>99.68</td>
    <td>72.33</td>
    <td>99.76</td>
    <td>16.53</td>
    <td>98.45</td>
    <td>73.48</td>
  </tr>
  <tr>
    <td>OOD w/Layerwise</td>
    <td><b>99.01</b></td>
    <td>86.14</td>
    <td>45.48</td>
    <td>79.73</td>
    <td>22.35</td>
    <td>99.78</td>
    <td>61.95</td>
    <td>100.0</td>
    <td>15.34</td>
    <td>93.33</td>
    <td>70.31</td>
  </tr>
  <tr>
    <td>OOD w/IPQ</td>
    <td>100.0</td>
    <td>74.65</td>
    <td>55.02</td>
    <td>81.75</td>
    <td>19.11</td>
    <td>100.0</td>
    <td>82.24</td>
    <td><b>99.72</b></td>
    <td>15.60</td>
    <td>100.0</td>
    <td>72.81</td>
  </tr>
  <tr>
    <td>OOD w/GenQuery</td>
    <td>100.0</td>
    <td>86.87</td>
    <td>75.48</td>
    <td>79.70</td>
    <td>21.42</td>
    <td><b>98.96</b></td>
    <td>62.13</td>
    <td>100.0</td>
    <td>15.97</td>
    <td>97.40</td>
    <td>73.79</td>
  </tr>
  <tr>
    <td>OOD w/Ours</td>
    <td><b>99.01</b></td>
    <td><b>61.14</b></td>
    <td><b>31.49</b></td>
    <td><b>79.16</b></td>
    <td><b>18.36</b></td>
    <td>99.71</td>
    <td><b>56.97</b></td>
    <td>100.0</td>
    <td><b>15.22</b></td>
    <td><b>89.19</b></td>
    <td><b>65.03</b></td>
  </tr>
  <tr>
    <td rowspan="4">Contriever</td>
    <td>All</td>
    <td>96.79</td>
    <td>72.40</td>
    <td>56.76</td>
    <td>59.83</td>
    <td>18.66</td>
    <td>98.83</td>
    <td>55.26</td>
    <td>98.25</td>
    <td>9.14</td>
    <td>96.14</td>
    <td>66.06</td>
  </tr>
  <tr>
    <td>OOD w/Layerwise</td>
    <td>93.83</td>
    <td>68.67</td>
    <td>49.14</td>
    <td>56.61</td>
    <td>17.85</td>
    <td>99.10</td>
    <td>51.75</td>
    <td>98.36</td>
    <td>8.26</td>
    <td>95.34</td>
    <td>63.89</td>
  </tr>
  <tr>
    <td>OOD w/IPQ</td>
    <td>93.83</td>
    <td>69.11</td>
    <td>48.37</td>
    <td>57.72</td>
    <td>18.11</td>
    <td>99.17</td>
    <td>51.39</td>
    <td>98.70</td>
    <td>8.25</td>
    <td>94.04</td>
    <td>63.87</td>
  </tr>
  <tr>
    <td>OOD w/GenQuery</td>
    <td><b>90.12</b></td>
    <td>72.23</td>
    <td>65.87</td>
    <td><b>54.99</b></td>
    <td>18.53</td>
    <td><b>97.28</b></td>
    <td>51.17</td>
    <td>98.65</td>
    <td><b>7.45</b></td>
    <td>93.33</td>
    <td>64.96</td>
  </tr>
  <tr>
    <td>OOD w/Ours</td>
    <td>91.36</td>
    <td><b>63.92</b></td>
    <td><b>40.63</b></td>
    <td>56.12</td>
    <td><b>17.11</b></td>
    <td>98.75</td>
    <td><b>50.64</b></td>
    <td><b>97.78</b></td>
    <td>8.09</td>
    <td><b>90.17</b></td>
    <td><b>61.46</b></td>
  </tr>
  <tr>
    <td rowspan="4">E5</td>
    <td>All</td>
    <td>99.68</td>
    <td>76.42</td>
    <td>55.56</td>
    <td>74.85</td>
    <td>18.03</td>
    <td>99.67</td>
    <td>61.49</td>
    <td>98.49</td>
    <td>15.81</td>
    <td>97.75</td>
    <td>70.00</td>
  </tr>
  <tr>
    <td>OOD w/Layerwise</td>
    <td>100.0</td>
    <td>75.46</td>
    <td>47.01</td>
    <td><b>74.38</b></td>
    <td>19.04</td>
    <td>99.65</td>
    <td>55.53</td>
    <td><b>98.43</b></td>
    <td>15.96</td>
    <td>97.81</td>
    <td>68.33</td>
  </tr>
  <tr>
    <td>OOD w/IPQ</td>
    <td><b>98.91</b></td>
    <td>75.96</td>
    <td>49.54</td>
    <td>74.61</td>
    <td>18.01</td>
    <td>99.83</td>
    <td>58.57</td>
    <td>98.45</td>
    <td><b>15.68</b></td>
    <td>97.24</td>
    <td>68.68</td>
  </tr>
  <tr>
    <td>OOD w/GenQuery</td>
    <td><b>98.91</b></td>
    <td>80.15</td>
    <td>69.33</td>
    <td>74.41</td>
    <td>18.54</td>
    <td><b>99.51</b></td>
    <td>58.05</td>
    <td>98.53</td>
    <td>15.79</td>
    <td>98.03</td>
    <td>71.13</td>
  </tr>
  <tr>
    <td>OOD w/Ours</td>
    <td>99.45</td>
    <td><b>69.19</b></td>
    <td><b>29.74</b></td>
    <td>74.47</td>
    <td><b>17.02</b></td>
    <td>99.68</td>
    <td><b>55.50</b></td>
    <td>98.56</td>
    <td>15.85</td>
    <td><b>96.43</b></td>
    <td><b>65.59</b></td>
  </tr>
  <tr>
    <td rowspan="4">GTE</td>
    <td>All</td>
    <td>99.68</td>
    <td>80.37</td>
    <td>60.85</td>
    <td>75.76</td>
    <td>22.48</td>
    <td>99.57</td>
    <td>72.66</td>
    <td>99.52</td>
    <td>17.53</td>
    <td>99.25</td>
    <td>73.55</td>
  </tr>
  <tr>
    <td>OOD w/Layerwise</td>
    <td>100.0</td>
    <td>82.81</td>
    <td>56.66</td>
    <td>76.17</td>
    <td>21.48</td>
    <td>99.87</td>
    <td>66.14</td>
    <td><b>99.47</b></td>
    <td>14.98</td>
    <td>99.82</td>
    <td>71.74</td>
  </tr>
  <tr>
    <td>OOD w/IPQ</td>
    <td>100.0</td>
    <td>84.59</td>
    <td>66.26</td>
    <td>75.27</td>
    <td>19.82</td>
    <td>99.83</td>
    <td>68.45</td>
    <td>99.50</td>
    <td><b>14.81</b></td>
    <td>99.82</td>
    <td>72.84</td>
  </tr>
  <tr>
    <td>OOD w/GenQuery</td>
    <td>100.0</td>
    <td>84.20</td>
    <td>76.80</td>
    <td><b>70.58</b></td>
    <td>21.33</td>
    <td><b>98.71</b></td>
    <td>65.95</td>
    <td>99.73</td>
    <td>16.16</td>
    <td>99.65</td>
    <td>73.31</td>
  </tr>
  <tr>
    <td>OOD w/Ours</td>
    <td><b>93.75</b></td>
    <td><b>70.83</b></td>
    <td><b>51.24</b></td>
    <td>71.02</td>
    <td><b>19.22</b></td>
    <td>99.60</td>
    <td><b>65.22</b></td>
    <td>100.0</td>
    <td>16.56</td>
    <td><b>98.72</b></td>
    <td><b>68.62</b></td>
  </tr>
</table>

Table 1: Comparison of OOD document detection across different retriever models on the BEIR benchmark. A lower Document Retrieval Rate value, defined in Eq.(5), indicates more accurate OOD detection.

**Hyperparameters.** To calculate the gradient norm for each document \( d \), we set the dropout rate to 0.02 and the number of positives (\( p \)) to 8. We use four negative samples (\( n \)) to reduce the computational cost. For OOD detection, we use the average gradient norm of 3,000 in-domain Natural Questions (NQ) documents (Kwiatkowski et al., 2019) as the reference threshold, since all test retrievers are already trained on NQ. Documents with gradient norms exceeding this average are classified as OOD. We set the OOD corpus prediction threshold (\( \gamma \)) to 0.5. We determine this value empirically based on preliminary experiments. We conduct an extensive ablation study as described in Appendix C, exhibiting the robustness of our approach.

**5.2 Detection of OOD Documents**

This task aims to detect OOD documents from a new document corpus \( \mathcal{C} \). Our method selects OOD documents where GradNormIR exceeds the threshold as described in Section 5.1. For other baselines, we rank the documents in descending order by their OOD scores as described below, and then select the same number of top-ranked documents as GradNormIR for fairness. Finally, we compare these detected OOD documents using their retrieval rate using query-document pairs in the dataset.

**Evaluation Metric.** To evaluate the OOD document detection, we use the document retrieval rate (DRR). As described in Section 3, the effectiveness of an approach can be measured by how poorly detected OOD documents are retrieved by relevant queries. For each dataset, we organize annotations as \( \{d_i, Q_{d_i}\}_{i=1}^N \), where \( Q_{d_i} \) represents the set of relevant queries for each document \( d_i \). DRR is then calculated as

\[
\text{DRR} = \frac{\sum_{d_i \in \mathcal{C}} \sum_{q_{d_i} \in Q_{d_i}} \mathbb{I}\{d_i \in D^+(q_{d_i})\}}{\sum_{d_i \in \mathcal{C}} |Q_{d_i}|},
\]

where \( \mathbb{I} \) is an indicator function that returns 1 if \( d_i \) appears in the top-\( k \) retrieval results \( D^+(q_{d_i}) \) (with \( k = 100 \)), and 0 otherwise. Lower DRR values indicate that the identified OOD documents are indeed retrieved less frequently, validating effective OOD detection.

**Implementations of Baselines.** For each baseline, we first compute the OOD score of each document \( d \) as follows: (i) **Layerwise**: we compute the negative cosine similarity between latent vectors of \( d \) and in-domain documents across all layers and aggregate them to produce a final OOD score for \( d \). (ii) **IPQ** creates quantization codebooks from \( \mathcal{C} \) to get centroids. We quantize all representations to generate centroids and use the average Euclidean distance between the quantized representation and the centroids as the OOD score of \( d \). (iii) **GenQuery**: we generate a pseudo-question \( \hat{q} \) for \( d \) using Llama3.1-8B. We then use the rank of \( d \) in the retrieval results of \( \hat{q} \) as the OOD score.

**5.2.1 Results**

Table 1 presents the results of OOD document detection across 10 datasets of the BEIR benchmark. Our GradNormIR consistently outperforms the baselines, achieving the lowest average DRR on all tested retrievers. Notably, GradNormIR substantially reduces DRR on DBPedia-Entity and Scidocs,
<table>
  <tr>
    <th>Method</th>
    <th>ArguAna</th>
    <th>C-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>COVID</th>
    <th>Touche</th>
  </tr>
  <tr>
    <td>Layerwise</td>
    <td><b>99.36</b></td>
    <td>79.14</td>
    <td>85.45</td>
    <td>77.25</td>
    <td>28.88</td>
    <td><b>99.97</b></td>
    <td>59.06</td>
    <td>98.39</td>
    <td>18.53</td>
    <td>97.16</td>
  </tr>
  <tr>
    <td>IPQ</td>
    <td>99.00</td>
    <td>79.14</td>
    <td>85.45</td>
    <td>77.25</td>
    <td>28.88</td>
    <td><b>99.97</b></td>
    <td>59.06</td>
    <td>98.39</td>
    <td>18.53</td>
    <td>97.16</td>
  </tr>
  <tr>
    <td>GenQuery</td>
    <td>99.00</td>
    <td>79.14</td>
    <td>85.45</td>
    <td>77.25</td>
    <td>28.88</td>
    <td><b>99.97</b></td>
    <td>59.06</td>
    <td>98.15</td>
    <td>18.53</td>
    <td>97.16</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>99.36</b></td>
    <td><b>82.55</b></td>
    <td><b>89.73</b></td>
    <td><b>78.51</b></td>
    <td><b>35.73</b></td>
    <td><b>99.97</b></td>
    <td><b>69.63</b></td>
    <td><b>99.73</b></td>
    <td><b>21.52</b></td>
    <td><b>98.40</b></td>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>99.43</td>
    <td>82.92</td>
    <td>89.73</td>
    <td>83.25</td>
    <td>35.73</td>
    <td>99.97</td>
    <td>69.98</td>
    <td>99.73</td>
    <td>21.52</td>
    <td>99.23</td>
  </tr>
</table>

Table 2: Results of zero-shot retriever selection in terms of Recall@100 scores of the retriever selected by each OOD method. The oracle is the upper bound, indicating the performance of the actual best retriever per dataset.

achieving reductions of 28.18 and 15.36, respectively, for BGE.

In the baselines, the detected OOD documents often show unexpectedly higher retrieval rates than the average DRR of all documents, indicating wrong detection. For instance, GenQuery in DBPedia-Entity shows significant increases across all retrievers, although it achieves the best performance on Quora for Contriever, E5, and GTE. Also, in Climate-FEVER, GenQuery increases for BGE, E5, and GTE. This may be because these documents are also out-of-domain to the LLM. Typically, IPQ and Layerwise baselines show the lowest DRRs in some cases, but their performance fluctuates up and down, indicating low robustness.

Overall, GradNormIR consistently shows lower document retrieval rates for detected OOD documents, demonstrating that it accurately identifies OOD documents across datasets. We further evaluate the OOD documents ratio, \( r(\mathcal{C}) \) in Section 3 in the following experiments.

5.3 Best Retriever Selection

This task predicts the most suitable dense retriever from a set of retrievers given a corpus \( \mathcal{C} \), i.e., it selects the retriever with the highest generalizability for the given \( \mathcal{C} \) using the OOD detection method. This task shows that our approach is helpful for selecting not only when the retrievers are updated, but also which one is the best in the stream of corpora.

Setup. We select one of four retrievers (as described in Section 5.1), choosing the one that has the lowest OOD document ratio, \( r(\mathcal{C}) \). Specifically, given a test dataset including \( \mathcal{C} \) and query-document pairs, we calculate \( r(\mathcal{C}) \) for each retriever. Next, we select the retriever with the lowest \( r(\mathcal{C}) \). Then, we evaluate the selected retriever on the query-document pairs. For each dataset, we report the Recall@100 performance of the retriever selected by each baseline.

Implementations of Baselines. To calculate \( r(\mathcal{C}) \), we first compute the OOD score of the in-domain NQ documents for each baseline in the same way as in Section 5.2. Then, we calculate the ratio of documents with an OOD score greater than the median. In this way, we can compute \( r(\mathcal{C}) \) of each baseline.

5.3.1 Results

Table 2 presents the Recall@100 performance of the selected retriever by each baseline. The oracle row shows the performance of the actual optimal retriever on each dataset. The retriever chosen by GradNormIR consistently achieves the highest performance across datasets. While GradNormIR does not always select the top-performing retriever (e.g., BGE for ArguAna, GTE for FiQA), it consistently identifies the second-best retriever, indicating robust generalizability estimation. These results show that GradNormIR is highly effective in selecting the most appropriate retriever based solely on the given document corpus, even before any queries are introduced.

5.4 Continual Updates

The goal of this task is to update the retriever only when an OOD corpus is given, balancing performance stability and computational cost in evolving corpora.

Setup. We simulate the sequential streaming of a corpus using datasets of the BEIR coming in alphabetical order. This experimental setup mirrors approaches in continual learning research, such as the (Ge et al., 2023), which sequentially combines several heterogeneous image classification datasets to evaluate models’ abilities to adapt to diverse and evolving tasks. Specifically, in session \( S_1 \), the Arguana corpus is given, in session \( S_2 \), the Climate-FEVER corpus is given, and so on. We continually update Contriever using RecAdam optimizer (Chen et al., 2020), widely employed to mitigate the language model’s catastrophic forgetting. In session \( S_t \), we update the current retriever with a given corpus. We then build a retrieval index using corpus from \( S_1 \) to \( S_t \). Finally, we evaluate the retriever with queries from \( S_1 \) to \( S_t \). For training details,
Figure 3: Average of Recall@100 across \( S_1 \) to \( S_{t} \) with respect to the upper bound for each dataset using a single-trained retriever. Although the trend decreases due to the expanding document corpus over sessions, performance remains robust with continual updates.

please refer to Appendix A.

Implementations of Baselines. We test three types of baselines: (i) Zero-shot: the retriever remains fixed with no further updates. (ii) Selective: the retriever is updated only when a newly given corpus is determined as an OOD corpus. (iii) Naïve: the retriever is updated whenever a new corpus is given, common in continual learning. For selective retraining, in each session \( S_t \), we decide whether to update the retriever and use the most recently updated retriever to build an index using corpus from \( S_1 \) to \( S_t \). We evaluate different update strategies from the four baselines. In GradNormIR, we update the retriever when a corpus is OOD, in total \( N \) times (\( N = 6 \)). For the other retraining methods, the retriever undergoes the same \( N \) updates with the corpora of the highest OOD ratios for fairness.

Metrics. We compute the average Recall@100 in each session \( S_t \), computed as the mean Recall@100 of the datasets from \( S_1 \) to \( S_t \). We report the relative performance with respect to an upper bound per dataset, since each dataset has different levels of difficulty. The upper bound of each dataset is the Recall@100 value of the retriever fine-tuned only with the dataset.

5.4.1 Results

Figure 3 illustrates the retrieval results of different continual update strategies over the sessions. Overall, performance degrades as the sessions progress. This occurs because as the number of documents increases, the corpus expands, making it more difficult to retrieve the correct documents. Thus, the performance of the Zero-shot baseline quite drops to around 80. However, with continual updates, the other baselines maintain stable performance around 90, preventing catastrophic forgetting.

Initially, GradNormIR exhibits lower performance in \( S_1 \) and \( S_2 \), since the retriever is not updated. Nonetheless, it does not show significant degradation afterward, maintaining the retriever’s accuracy by retraining in later sessions. Starting from session \( S_6 \), GradNormIR achieves the highest average performance among all baselines. Notably, in \( S_6 \), GradNormIR outperforms even the Naïve baseline, which retrains the retriever in every session. This indicates that unnecessary continual retraining can negatively impact retrieval performance, highlighting the importance of selective retraining. The performance gap persists until the final session, demonstrating the efficiency and effectiveness of GradNormIR’s selective retraining.

Conversely, all other selective baselines exhibit lower performance than Naïve baseline. For instance, Layerwise displays robust performance in the earlier sessions, but it shows persistent performance degradation in later sessions since it is not trained on the OOD corpus in \( S_4 \) and \( S_6 \). This suggests that selective retraining only with the OOD corpus can ensure the maintenance of retriever performance in evolving corpora.

<table>
  <tr>
    <th rowspan="2">Hard Neg</th>
    <th rowspan="2">Dropout</th>
    <th colspan="4">DRR (\%)</th>
  </tr>
  <tr>
    <th>BGE</th>
    <th>Cont</th>
    <th>E5</th>
    <th>GTE</th>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>67.68</td>
    <td>64.19</td>
    <td>67.45</td>
    <td><b>68.07</b></td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>65.79</td>
    <td>62.41</td>
    <td><b>65.58</b></td>
    <td>70.75</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td><b>65.03</b></td>
    <td><b>61.46</b></td>
    <td>65.59</td>
    <td>68.62</td>
  </tr>
</table>

Table 3: Ablation study on the impact of dropout for document queries and the use of hard negatives.

other baselines maintain stable performance around 90, preventing catastrophic forgetting.

Initially, GradNormIR exhibits lower performance in \( S_1 \) and \( S_2 \), since the retriever is not updated. Nonetheless, it does not show significant degradation afterward, maintaining the retriever’s accuracy by retraining in later sessions. Starting from session \( S_6 \), GradNormIR achieves the highest average performance among all baselines. Notably, in \( S_6 \), GradNormIR outperforms even the Naïve baseline, which retrains the retriever in every session. This indicates that unnecessary continual retraining can negatively impact retrieval performance, highlighting the importance of selective retraining. The performance gap persists until the final session, demonstrating the efficiency and effectiveness of GradNormIR’s selective retraining.

Conversely, all other selective baselines exhibit lower performance than Naïve baseline. For instance, Layerwise displays robust performance in the earlier sessions, but it shows persistent performance degradation in later sessions since it is not trained on the OOD corpus in \( S_4 \) and \( S_6 \). This suggests that selective retraining only with the OOD corpus can ensure the maintenance of retriever performance in evolving corpora.

5.5 Ablation Study

We evaluate the impact of the dropout and the use of hard negatives. Table 3 displays the results of the average DRR in OOD document detection. When both dropout and hard negatives are applied, the model achieves the best performance, particularly for the BGE and Contriever. For E5, hard negatives contribute to an increase in DRR, while dropout also proves effective. Conversely, for GTE, hard negatives enhance performance, whereas dropout leads to performance degradation. This suggests that the optimal setting may vary depending on the chosen retriever. Nonetheless, even in these two cases, both hard negatives and dropout yield reasonable performance, indicating the robustness of GradNormIR to hyperparameter choices. Additional ablation experiments are provided in Appendix C.
6 Conclusion

We introduced the novel task of predicting an OOD corpus for dense retrievers before indexing, addressing a critical challenge for maintaining robust retrieval performance in dynamically evolving corpora. To tackle this problem, we proposed GradNormIR, an unsupervised method leveraging gradient norms of the contrastive loss to proactively detect OOD documents. By employing novel sampling strategies, including document-to-document retrieval with carefully selected positive and hard negative samples, GradNormIR can effectively predict whether a corpus might pose retrieval challenges even before query collection. Our method enables timely selection of the most suitable retriever or updates to an existing retriever, thereby ensuring robust performance in evolving document environments.

An interesting direction for future research would involve online OOD document detection, where documents arrive continuously rather than as a complete corpus. Such an extension could enhance the practical applicability of retrievers in real-time retrieval scenarios.

Limitations

While our work effectively demonstrates the importance and efficacy of proactively detecting OOD documents and corpora, there are several limitations that warrant consideration. First, GradNormIR operates solely at the document level without considering potential future queries explicitly. Consequently, retrieval performance may still degrade if unexpected queries significantly differ from the assumed document representations. Incorporating query-aware mechanisms could further enhance robustness against such scenarios. Also, our approach relies on document-to-document similarity computations, which may pose scalability challenges when applied to extremely large corpora. In practical large-scale deployments, it might be necessary to implement efficient indexing strategies or divide corpora into smaller subsets for manageable processing. We validate these strategies in Appendix, so please refer to it.

Ethics Statement

This research does not raise ethical concerns, as it primarily addresses technical advancements in information retrieval models and their evaluation methodologies. Our methods focus exclusively on improving retrieval performance using non-sensitive, general-purpose datasets and do not involve handling any personal, confidential, or ethically sensitive information.

Acknowledgments

We sincerely thank Heeseung Yun, Sangwoo Moon, and other anonymous reviewers for their valuable comments. This work was supported by supported by Samsung SDS, Korea Radio Promotion Association (Development of Intelligent Docent Service for Information-Disadvantaged Groups), Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019-II191082, SW StarLab), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-II220156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573). Gunhee Kim is the corresponding author.

References

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.

Maciej Besta, Ales Kubicek, Roman Niggli, Robert Gerstenberger, Lucas Weitzendorf, Mingyuan Chi, Patrick Iff, Joanna Gajda, Piotr Nyczyk, Jürgen Müller, et al. 2024. Multi-head rag: Solving multi-aspect problems with llms. arXiv preprint arXiv:2406.05085.

Yinqiao Cai, Keping Bi, Yixing Fan, Jiafeng Guo, Wei Chen, and Xueqi Cheng. 2023. L2r: Lifelong learning for first-stage retrieval with backward-compatible representations. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 183–192.

Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Continual learning for generative retrieval over dynamic corpora. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 306–315.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding:
Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.

Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870–7881, Online. Association for Computational Linguistics.

Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, and Marc Najork. 2022. Out-of-domain semantics to the rescue! zero-shot hybrid retrieval models. In European Conference on Information Retrieval, pages 95–110.

Maxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes, Jackie CK Cheung, Pablo Piantanida, and Pierre Colombo. 2024. Unsupervised layer-wise score aggregation for textual ood detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17880–17888.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simce: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.

Yunhao Ge, Yuecheng Li, Di Wu, Ao Xu, Adam M Jones, Amanda Sofie Rios, Iordanis Fostiropoulos, Po-Hsuan Huang, Zachary William Murdock, Gozde Sahin, et al. 2023. Lightweight learner for shared knowledge lifelong learning. In ICML Workshop on Localized Learning (LLW).

Rui Huang, Andrew Geng, and Yixuan Li. 2021. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677–689.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research.

Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2022. Augmenting document representations for dense retrieval with interpolation and perturbation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 442–452.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.

Pranav Kasela, Gabriella Pasi, Raffaele Perego, and Nicola Tonellotto. 2024. Desire-me: Domain-enhanced supervised information retrieval using mixture-of-experts. In European Conference on Information Retrieval, pages 111–125.

Ekaterina Khramtsova, Shengyao Zhuang, Mahsa Bakhtashmotlagh, Xi Wang, and Guido Zuccon. 2023. Selecting which dense retriever to use for zero-shot search. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 223–233.

Ekaterina Khramtsova, Shengyao Zhuang, Mahsa Bakhtashmotlagh, and Guido Zuccon. 2024. Leveraging llms for unsupervised dense retriever ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1307–1317.

Jinyoung Kim, Dayoon Ko, and Gunhee Kim. 2024. DynamicER: Resolving emerging mentions to dynamic entities for RAG. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13752–13770, Miami, Florida, USA. Association for Computational Linguistics.

Dayoon Ko, Jinyoung Kim, Hahyeon Choi, and Gunhee Kim. 2024. GrowOVER: How can LLMs adapt to growing real-world knowledge? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3282–3308, Bangkok, Thailand. Association for Computational Linguistics.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Hyunji Lee, Luca Soldaini, Arman Cohan, Minjoon Seo, and Kyle Lo. 2024. Routerretriever: Exploring the benefits of routing over multiple expert embedding models. arXiv preprint arXiv:2409.02685.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of experimental social psychology, 49(4):764–766.

Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2023a. Large language models with controllable working memory. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1774–1793.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023b. Towards
general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.

Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Robust neural information retrieval: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2407.06992.

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Abrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021. Large dual encoders are generalizable retrievers. Preprint, arXiv:2112.07899.

Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. How context affects language models’ factual predictions. In Automated Knowledge Base Construction.

Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 29–48. Citeseer.

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2021. Gpl: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. arXiv preprint arXiv:2112.07577.

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. Preprint, arXiv:2309.07597.

Renchunzhi Xie, Ambroise Odonnat, Vasilii Feofanov, Ievgen Redko, Jianfeng Zhang, and Bo An. 2024. Characterising gradients for unsupervised accuracy estimation under distribution shift. arXiv preprint arXiv:2401.08909.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380.

Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. Coco-dr: Combating distribution shift in zero-shot dense retrieval with contrastive and distributionally robust learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1462–1479.

Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1503–1512.
A  Experiment Details

Models. In experiments, we use four dense retrievers: BGE-large-en-v1.5, unsupervised Contriever, multilingual E5 large, and GTE-base. In hugging face, the model names are BAAI/bge-large-en-v1.5, facebook/contriever, intfloat/multilingual-e5-large, and thenlper/gte-base, respectively.

GradNormIR. For contrastive loss temperature \( \tau \), we use 0.05 for all baselines but 0.01 for e5. The probability distribution is skewed for E5, as noted in hugging face; setting the temperature to 0.05 does not make the model compute contrastive loss effectively.

For positive and negative sampling, we sample 8 positive samples (\( p \)) and four negatives (\( n \)) for each positive. As sampling 4 negatives per positive is traditional, we follow previous work.

For the dropout, we use 0.02, which means 2 percent of tokens are masked to zero. For the other experimental setups, we follow the default values of BAAI Flagembeddings\(^1\).

We conduct ablation studies on the impact of the number of positive samples and dropout rate in Section C.2.

GenQuery. We use Llama3.1 8B with Q4_0 quantization to generate pseudo queries with the temperature set to 0.5 and the max_new_tokens set to 256. Also, the prompt template is shown in Figure 4.

IPQ. For production quantization, we set the number of quantization groups as 8, which means the last hidden state after pooling (e.g., 1024 dimensions) is divided into 8 groups (e.g., 128 dimensions for each group), and each is clustered using KMeans. We set the number of clusters as 16, which means each 128-dimensional vector becomes an integer between 0 and 15.

Continual Updates. For the training dataset, we use generated queries from the Hugging Face BEIR repository to retrain the retriever, as original test queries in the BEIR are used in the evaluation. Using these queries, we perform supervised fine-tuning. We set epoch to 4, gradient_accumulation_steps to 256, batch_size to 4, learning_rate to 1e-04, lr_scheduler to "Constant", with multi-GPU (4 GPUs) parallelization.

\(^1\)https://github.com/FlagOpen/FlagEmbedding

B  Concerns Regarding Computational Cost and Large Corpora

During the rebuttal period, anonymous reviewers raised insightful and valid concerns regarding computational efficiency and scalability. Here we address these concerns in detail.

B.1  Computational Cost

Gradient computation concern. We emphasize that our GradNormIR method remains practical even for large-scale retriever models exceeding one billion parameters. Specifically, we successfully applied GradNormIR to the GTR-XL retriever model (1.24B parameters) (Ni et al., 2021) and confirmed gradient norm computations were feasible on a single GPU with 48GB of memory. Furthermore, parallelizing computations across four GPUs allowed us to process the entire Arguana corpus (approximately 1.4K documents) in roughly 20 minutes.

It is important to clarify that the computational complexity of GradNormIR primarily depends on the number of negative samples utilized in the InfoNCE loss computation. Crucially, we observed effective GradNorm computations with as few as four negative samples, significantly fewer than the typical 255 negatives per positive used during standard dense retriever training (e.g., Contriever training).

Runtime Cost Analysis. We conducted a detailed runtime analysis of GradNormIR across different retriever models and varying corpus sizes using four GPUs (24GB each) in parallel. The runtime costs are summarized in Table 4.

For large corpora, computational costs can be effectively reduced by randomly sampling documents or partitioning the corpus. To validate this, we conducted additional experiments on large-scale datasets (DBPedia, FiQA, Quora, and COVID) by randomly sampling only 10% of the documents. The results, shown in Table 5, indicate minimal differences in document retrieval rates (DRR), confirming the practicality and scalability of GradNormIR through subset sampling.

C  Ablation Study

We conduct additional ablation study of the impact of (i) the number of documents randomly sampled from the in-domain dataset and (ii) the number of positives in Eq. (4) as well as dropout rate.
<table>
  <tr>
    <th>Retriever (Size)</th>
    <th>Task</th>
    <th>Arguana (1.4K docs)</th>
    <th>Scidocs (4K docs)</th>
    <th>COVID (15K docs)</th>
  </tr>
  <tr>
    <td rowspan="3">Contriever (110M)</td>
    <td>Retrieval</td>
    <td>2m</td>
    <td>3m</td>
    <td>6m</td>
  </tr>
  <tr>
    <td>Gradient norm computation</td>
    <td>8m</td>
    <td>40m</td>
    <td>2h 30m</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>10m</td>
    <td>43m</td>
    <td>2h 45m</td>
  </tr>
  <tr>
    <td rowspan="3">E5 (560M)</td>
    <td>Retrieval</td>
    <td>3m</td>
    <td>5m</td>
    <td>25m</td>
  </tr>
  <tr>
    <td>Gradient norm computation</td>
    <td>12m</td>
    <td>1h</td>
    <td>3h 30m</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>15m</td>
    <td>1h 5m</td>
    <td>3h 55m</td>
  </tr>
</table>

Table 4: Runtime analysis for GradNormIR on different retrievers and corpus sizes.

<table>
  <tr>
    <th>Retriever</th>
    <th>Sampling</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>Quora</th>
    <th>COVID</th>
  </tr>
  <tr>
    <td>BGE</td>
    <td>w/o Sampling</td>
    <td>31.49</td>
    <td>79.16</td>
    <td>99.71</td>
    <td>13.22</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ Sampling</td>
    <td>30.33</td>
    <td>79.26</td>
    <td>98.57</td>
    <td>15.86</td>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>w/o Sampling</td>
    <td>40.63</td>
    <td>56.12</td>
    <td>98.75</td>
    <td>8.09</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ Sampling</td>
    <td>39.89</td>
    <td>52.69</td>
    <td>99.41</td>
    <td>9.33</td>
  </tr>
  <tr>
    <td>E5</td>
    <td>w/o Sampling</td>
    <td>29.74</td>
    <td>74.47</td>
    <td>99.68</td>
    <td>15.85</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ Sampling</td>
    <td>29.78</td>
    <td>75.75</td>
    <td>99.27</td>
    <td>15.54</td>
  </tr>
  <tr>
    <td>GTE</td>
    <td>w/o Sampling</td>
    <td>51.24</td>
    <td>71.02</td>
    <td>99.60</td>
    <td>16.56</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ Sampling</td>
    <td>53.87</td>
    <td>70.20</td>
    <td>99.20</td>
    <td>16.10</td>
  </tr>
</table>

Table 5: Comparison of Document Retrieval Rates (DRR) with and without 10% sampling, demonstrating minimal impact on retrieval performance.

C.1 The Number of In-Domain Documents

Table 6 shows the results of DRR in predicting OOD documents, where the number of documents are determined by randomly selected 1,000 NQ documents, while Table 7 shows the results when 2,000 NQ documents are used for in-domain document samples. In both cases, our method show the lowest average DDR results for all models, indicating the robustness of GradNormIR in predicting OOD documents. Also, the number of documents detected as OOD are presented in Table 11. The number of OOD documents are lowest in GTE, as it can generalize to the datasets of the BEIR benchmark.

C.2 The Number of Positives and Dropout Rate

The DRR results for the number of positives from 1 to 16 are shown in Table 12. As the number of positives increases, the DRR generally decreases because more gradient norm values make the method more robust. Additionally, when comparing the cases with and without dropout, the decrease is significantly higher as the number of positives increases. This is because the lower-ranking positives are more likely to be affected by dropout. However, when the dropout rate increases from 0.02 to 0.05, there are some cases where the filtered documents show higher DRR values, especially increasing 3.79 in an average of 16 samples for E5. This may be because excessive dropout can deteriorate model performance.

D Relevance Gains via Filtering

To evaluate the impact of OOD documents, we also conduct a document filtering experiment. Specifically, we remove OOD documents from the given corpus \( \mathcal{C} \), thereby enhancing retrieval relevance.

Setup. For each dataset, we begin with an evaluation set \( \{(d_i, Q_i)\}_{i=1}^N \). If \( d_i \) is detected as an OOD document, we remove it from the evaluation set, meaning we no longer evaluate \( d_i \) as a gold label for its associated queries \( q_i \in Q_i \). We then evaluate the performance on the test queries \( \{Q_i\}_{i=1}^N \). By removing such OOD documents, we aim to exclude irrelevant or misleading texts that could otherwise confuse the retriever, thereby potentially improving retrieval performance. We measure retrieval performance using Recall@100, following Izacard et al. (2022).

Figure 5 presents the total sum of gains in Recall@100 across 10 datasets of the BEIR after removing OOD documents. Our method, GradNormIR, demonstrates significant performance improvements across all retrievers. Specifically, it achieves gains of 34.73, 62.24, 51.15, and 12.40 points for BGE, Contriever, E5, and GTE, respectively. Even with GTE, where GradNormIR does not yield the best results, the overall retrieval enhancement remains the highest with our method.

E Relation Between OOD Ratio and Performance

Figure 6 shows the relationship between OOD document ratio \( r(\mathcal{C}) \) and retriever performance. The x-axis lists datasets in descending order of performance based on Contriever’s (Izacard et al., 2022) Recall@100. The y-axis represents the Non-OOD
ratio (1 - \( r(\mathcal{C}) \)).

The graph’s descending trend indicates that 1 - \( r(\mathcal{C}) \) is proportional to retriever performance, as datasets with higher retrieval performance show greater Non-OOD ratios. GradNormIR clearly demonstrates this relationship, showing high Non-OOD ratios for Quora, Arguana, and Touché, and low ratios for FiQA, Scidocs, NFCorpus, and COVID. While GenQuery also exhibits a descending trend, it shows minimal variation from Quora to NFCorpus, making OOD corpus detection less effective.

To predict OOD corpora, we set \( \gamma \) to 0.5 based on the average performance across all datasets. With this threshold, we identify SciFact, Touch’e, DBPedia, FiQA, Scidocs, NFCorpus, and COVID as OOD corpora.

F Feasibility of GradNormIR

We aim to validate whether GradNormIR can identify the documents that are difficult for the the models to retrieve. To this end, we inspect if there is a consistent relationship between the computed gradient norm and the likelihood of a document successfully retrieved by its associated queries.

Evaluation Metric. To evaluate the effectiveness, we measure the document-to-query (d2q) as the standard metric. In each dataset, annotations are provided in the form of \( \{ q_i, D_i \}_{i=1}^N \), where \( q_i \) is a query and \( D_i \) is the set of relevant documents. We reorganize these annotations as \( \{ d_i, Q_i \}_{i=1}^N \), where \( Q_i \) represents the set of relevant queries for each document \( d_i \). For a document to be considered effectively retrievable, it should be retrieved for all its relevant queries.

To quantify this, we define the d2q recall as follows:

\[
\text{recall}_{d2q} = \frac{\sum_{q_i \in Q_i} \mathbb{I}\{ d_i \in D^+(q_i) \}}{|Q_i|},
\]

where \( \mathbb{I} \) is an indicator function and \( D^+(q_i) \) represents the top-\( k \) retrieved documents (with \( k = 100 \)).

When the retriever model generalizes well for a document \( d_i \), the d2q recall value will be high. Additionally, if the retriever generalizes effectively on \( d_i \), the gradient norm associated with \( d_i \) will be low, as the retriever does not need to make substantial updates based on the contrastive loss for \( d_i \). Therefore, there should be an inverse relationship: higher the d2q recall values correspond to lower the gradient norms.

Results. Figure 7 illustrates the relationship between GradNormIR and d2q recall. We divide the data points into quartiles based on GradNormIR values, sorted in ascending order and labeled as Q1, Q2, Q3, and Q4. The x-axis represents these quartiles, while the y-axis shows the average d2q recall for each group.

The results reveal a strong inverse correlation between GradNormIR and retrieval performance. As GradNormIR values increase from Q1 to Q4, d2q recall decreases. This indicates that higher GradNormIR values (Q4) are associated with documents that are more challenging for the retriever to retrieve consistently. Conversely, lower GradNormIR values (Q1) correspond to higher recall, indicating better retrieval performance. When d2q recall approaches 1, such as Quora and SciFact, this trend becomes less noticeable. This is likely because the datasets have been trained on; nearly all documents are well generalized and easily retrievable.
Generate one Q&A pair based on a given context, where the context is understood but NOT DIRECTLY VISIBLE to the person answering the question. The question should cover the main focus of the full context.
Assume the person answering the question has common sense and is aware of the details and key points in the sentence(s), but the sentence(s) itself is not quoted or referenced directly.

Sentence(s): {paragraph}
Use the following instructions for generating a Q&A pairs:
1) Provide one {question}{answer}
2) DON'T use phrases such as 'according to the sentence(s)' in your question.
3) DON'T use phrases in the context verbatim.
4) An answer should be an entity or entities.
5) Ensure the question can be answered without referring back to the document, assuming domain knowledge.
6) Ensure the question includes enough context to be understood on its own.
7) The question should be general enough to be answerable by someone familiar with the topic, not requiring specific details from the context.
8) If there is not enough information to generate a question, state 'Not enough information to generate a question.'

Be sure to follow the following format and provide a question and answer pair within curly brackets.

The format is as follows: {Question}{Answer}

Figure 4: The prompt template to create pseudo queries using Llama3.1 8B in zero-shot. We prompt it to generate a question along with a corresponding answer to ensure the question can be answered. We use only generated question for evaluation.

<table>
  <tr>
    <th rowspan="2">Retriever</th>
    <th rowspan="2">Documents</th>
    <th colspan="10">Avg. (%)</th>
  </tr>
  <tr>
    <th>ArguAna</th>
    <th>C-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>COVID</th>
    <th>Touche</th>
    <th>Avg. (%)</th>
  </tr>
  <tr>
    <td rowspan="4">BGE</td>
    <td>ALL</td>
    <td>99.68</td>
    <td>79.96</td>
    <td>59.67</td>
    <td>80.25</td>
    <td>21.39</td>
    <td>99.68</td>
    <td>72.33</td>
    <td>99.76</td>
    <td>16.53</td>
    <td>98.45</td>
    <td>73.48</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td>100.0</td>
    <td>86.87</td>
    <td>75.48</td>
    <td>79.7</td>
    <td>21.42</td>
    <td><b>98.96</b></td>
    <td>62.13</td>
    <td>100.0</td>
    <td>15.97</td>
    <td>97.4</td>
    <td>73.79</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td><b>99.01</b></td>
    <td>86.14</td>
    <td>45.48</td>
    <td>79.73</td>
    <td>22.35</td>
    <td>99.78</td>
    <td>61.95</td>
    <td>100.0</td>
    <td><b>15.34</b></td>
    <td>93.33</td>
    <td>70.31</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td>100.0</td>
    <td>74.65</td>
    <td>55.02</td>
    <td>81.75</td>
    <td>19.11</td>
    <td>100.0</td>
    <td>82.24</td>
    <td><b>99.72</b></td>
    <td>15.6</td>
    <td>100.0</td>
    <td>72.81</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>99.08</td>
    <td><b>63.9</b></td>
    <td><b>32.57</b></td>
    <td><b>79.34</b></td>
    <td><b>18.78</b></td>
    <td>99.74</td>
    <td><b>58.3</b></td>
    <td>100.0</td>
    <td>15.43</td>
    <td><b>89.87</b></td>
    <td><b>65.7</b></td>
  </tr>
  <tr>
    <td rowspan="4">Contriever</td>
    <td>ALL</td>
    <td>96.79</td>
    <td>72.40</td>
    <td>56.76</td>
    <td>59.83</td>
    <td>18.66</td>
    <td>98.83</td>
    <td>55.26</td>
    <td>98.25</td>
    <td>9.14</td>
    <td>96.14</td>
    <td>66.06</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td><b>90.12</b></td>
    <td>72.23</td>
    <td>65.87</td>
    <td><b>54.99</b></td>
    <td>18.53</td>
    <td><b>97.28</b></td>
    <td>51.17</td>
    <td>98.65</td>
    <td><b>7.45</b></td>
    <td>93.33</td>
    <td>64.96</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td>93.83</td>
    <td>68.67</td>
    <td>49.14</td>
    <td>56.61</td>
    <td>17.85</td>
    <td>99.1</td>
    <td>51.75</td>
    <td>98.36</td>
    <td>8.26</td>
    <td>95.34</td>
    <td>63.89</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td>93.83</td>
    <td>69.11</td>
    <td>48.37</td>
    <td>57.72</td>
    <td>18.11</td>
    <td>99.17</td>
    <td>51.39</td>
    <td>98.7</td>
    <td>8.25</td>
    <td>94.04</td>
    <td>63.87</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>91.01</td>
    <td><b>64.45</b></td>
    <td><b>41.38</b></td>
    <td>56.65</td>
    <td><b>17.23</b></td>
    <td>98.75</td>
    <td><b>50.93</b></td>
    <td><b>97.89</b></td>
    <td>8.27</td>
    <td><b>91.04</b></td>
    <td><b>61.76</b></td>
  </tr>
  <tr>
    <td rowspan="4">E5</td>
    <td>ALL</td>
    <td>99.68</td>
    <td>76.42</td>
    <td>55.56</td>
    <td>74.85</td>
    <td>18.03</td>
    <td>99.67</td>
    <td>61.49</td>
    <td>98.49</td>
    <td>15.81</td>
    <td>97.75</td>
    <td>70.00</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td><b>98.91</b></td>
    <td>80.15</td>
    <td>69.33</td>
    <td>74.41</td>
    <td>18.54</td>
    <td><b>99.51</b></td>
    <td>58.05</td>
    <td>98.53</td>
    <td>15.79</td>
    <td>98.03</td>
    <td>71.13</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td>100.0</td>
    <td>75.46</td>
    <td>47.01</td>
    <td><b>74.38</b></td>
    <td>19.04</td>
    <td>99.65</td>
    <td><b>55.53</b></td>
    <td><b>98.43</b></td>
    <td>15.96</td>
    <td>97.81</td>
    <td>68.33</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td><b>98.91</b></td>
    <td>75.96</td>
    <td>49.54</td>
    <td>74.61</td>
    <td>18.01</td>
    <td>99.83</td>
    <td>58.57</td>
    <td>98.45</td>
    <td><b>15.68</b></td>
    <td>97.24</td>
    <td>68.68</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>99.48</td>
    <td><b>69.46</b></td>
    <td><b>30.27</b></td>
    <td>74.53</td>
    <td><b>17.14</b></td>
    <td>99.66</td>
    <td>55.79</td>
    <td>98.59</td>
    <td>15.84</td>
    <td><b>96.2</b></td>
    <td><b>65.7</b></td>
  </tr>
  <tr>
    <td rowspan="4">GTE</td>
    <td>ALL</td>
    <td>99.68</td>
    <td>80.37</td>
    <td>60.85</td>
    <td>75.76</td>
    <td>22.48</td>
    <td>99.57</td>
    <td>72.66</td>
    <td>99.52</td>
    <td>17.53</td>
    <td>99.25</td>
    <td>73.55</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td>100.0</td>
    <td>84.2</td>
    <td>76.8</td>
    <td><b>70.58</b></td>
    <td>21.33</td>
    <td><b>98.71</b></td>
    <td>65.95</td>
    <td>99.73</td>
    <td>16.16</td>
    <td>99.63</td>
    <td>73.31</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td>100.0</td>
    <td>82.81</td>
    <td>56.66</td>
    <td>76.17</td>
    <td>21.48</td>
    <td>99.87</td>
    <td>66.14</td>
    <td><b>99.47</b></td>
    <td>14.98</td>
    <td>99.82</td>
    <td>71.74</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td>100.0</td>
    <td>84.59</td>
    <td>66.26</td>
    <td>75.27</td>
    <td>19.82</td>
    <td>99.83</td>
    <td>68.45</td>
    <td>99.5</td>
    <td><b>14.81</b></td>
    <td>99.82</td>
    <td>72.84</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>93.75</td>
    <td><b>70.73</b></td>
    <td><b>51.49</b></td>
    <td>70.97</td>
    <td><b>19.27</b></td>
    <td>99.59</td>
    <td><b>65.25</b></td>
    <td>100.0</td>
    <td>16.52</td>
    <td><b>98.72</b></td>
    <td><b>68.63</b></td>
  </tr>
</table>

Table 6: Comparison of OOD document detection across different retriever models, with the number of documents selected by **1,000** sampled NQ documents.
<table>
  <tr>
    <th>Retriever</th>
    <th>Documents</th>
    <th>ArguAna</th>
    <th>C-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>COVID</th>
    <th>Touche</th>
    <th>Avg. (±)</th>
  </tr>
  <tr>
    <td rowspan="5">BGE</td>
    <td>ALL</td>
    <td>99.68</td>
    <td>79.96</td>
    <td>59.67</td>
    <td>80.25</td>
    <td>21.39</td>
    <td>99.68</td>
    <td>72.33</td>
    <td>99.76</td>
    <td>16.53</td>
    <td>98.45</td>
    <td>73.48</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td>100.0</td>
    <td>86.87</td>
    <td>75.48</td>
    <td>79.7</td>
    <td>21.42</td>
    <td><b>98.96</b></td>
    <td>62.13</td>
    <td>100.0</td>
    <td>15.97</td>
    <td>97.4</td>
    <td>73.79</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td><b>99.01</b></td>
    <td>86.14</td>
    <td>45.48</td>
    <td>79.73</td>
    <td>22.35</td>
    <td>99.78</td>
    <td>61.95</td>
    <td>100.0</td>
    <td><b>15.34</b></td>
    <td>93.33</td>
    <td>70.31</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td>100.0</td>
    <td>74.65</td>
    <td>55.02</td>
    <td>81.75</td>
    <td>19.11</td>
    <td>100.0</td>
    <td>82.24</td>
    <td><b>99.72</b></td>
    <td>15.6</td>
    <td>100.0</td>
    <td>72.81</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>99.03</td>
    <td><b>84.18</b></td>
    <td>31.97</td>
    <td>79.27</td>
    <td><b>18.42</b></td>
    <td>99.73</td>
    <td><b>57.25</b></td>
    <td>100.0</td>
    <td>13.53</td>
    <td><b>89.47</b></td>
    <td><b>65.47</b></td>
  </tr>
  <tr>
    <td rowspan="5">Contriever</td>
    <td>ALL</td>
    <td>96.79</td>
    <td>72.40</td>
    <td>56.76</td>
    <td>59.83</td>
    <td>18.66</td>
    <td>98.83</td>
    <td>55.26</td>
    <td>98.25</td>
    <td>9.14</td>
    <td>96.14</td>
    <td>66.06</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td><b>90.12</b></td>
    <td>72.23</td>
    <td>65.87</td>
    <td><b>54.99</b></td>
    <td>18.53</td>
    <td><b>97.28</b></td>
    <td>51.17</td>
    <td>98.65</td>
    <td><b>7.45</b></td>
    <td>93.33</td>
    <td>64.96</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td>93.83</td>
    <td>68.67</td>
    <td>49.14</td>
    <td>56.61</td>
    <td>17.85</td>
    <td>99.1</td>
    <td>51.75</td>
    <td>98.36</td>
    <td>8.26</td>
    <td>95.34</td>
    <td>63.89</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td>93.83</td>
    <td>69.11</td>
    <td>48.37</td>
    <td>57.72</td>
    <td>18.11</td>
    <td>99.17</td>
    <td>51.39</td>
    <td>98.7</td>
    <td>8.25</td>
    <td>94.04</td>
    <td>63.87</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>91.76</td>
    <td><b>64.23</b></td>
    <td><b>40.88</b></td>
    <td>56.37</td>
    <td><b>17.17</b></td>
    <td>98.73</td>
    <td><b>50.58</b></td>
    <td><b>97.79</b></td>
    <td>8.17</td>
    <td><b>89.88</b></td>
    <td><b>61.56</b></td>
  </tr>
  <tr>
    <td rowspan="5">E5</td>
    <td>ALL</td>
    <td>99.68</td>
    <td>76.42</td>
    <td>55.56</td>
    <td>74.85</td>
    <td>18.03</td>
    <td>99.67</td>
    <td>61.49</td>
    <td>98.49</td>
    <td>15.81</td>
    <td>97.75</td>
    <td>70.00</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td><b>98.91</b></td>
    <td>80.15</td>
    <td>69.33</td>
    <td>74.41</td>
    <td>18.54</td>
    <td><b>99.51</b></td>
    <td>58.05</td>
    <td>98.53</td>
    <td>15.79</td>
    <td>98.03</td>
    <td>71.13</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td>100.0</td>
    <td>75.46</td>
    <td>47.01</td>
    <td><b>74.38</b></td>
    <td>19.04</td>
    <td>99.65</td>
    <td><b>55.53</b></td>
    <td><b>98.43</b></td>
    <td>15.96</td>
    <td>97.81</td>
    <td>68.33</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td><b>98.91</b></td>
    <td>75.96</td>
    <td>49.54</td>
    <td>74.61</td>
    <td>18.01</td>
    <td>99.83</td>
    <td>58.57</td>
    <td>98.45</td>
    <td><b>15.68</b></td>
    <td>97.24</td>
    <td>68.68</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>99.47</td>
    <td><b>69.1</b></td>
    <td><b>29.97</b></td>
    <td>74.47</td>
    <td><b>17.09</b></td>
    <td>99.67</td>
    <td>55.82</td>
    <td>98.36</td>
    <td>15.83</td>
    <td><b>96.48</b></td>
    <td><b>65.65</b></td>
  </tr>
  <tr>
    <td rowspan="5">GTE</td>
    <td>ALL</td>
    <td>99.68</td>
    <td>80.37</td>
    <td>60.85</td>
    <td>75.76</td>
    <td>22.48</td>
    <td>99.57</td>
    <td>72.66</td>
    <td>99.52</td>
    <td>17.53</td>
    <td>99.25</td>
    <td>73.55</td>
  </tr>
  <tr>
    <td>OOD w/ GenQuery</td>
    <td>100.0</td>
    <td>84.2</td>
    <td>76.8</td>
    <td><b>70.58</b></td>
    <td>21.33</td>
    <td>98.71</td>
    <td>65.95</td>
    <td>99.73</td>
    <td>16.16</td>
    <td>99.63</td>
    <td>73.31</td>
  </tr>
  <tr>
    <td>OOD w/ Layerwise</td>
    <td>100.0</td>
    <td>82.81</td>
    <td>56.66</td>
    <td>76.17</td>
    <td>21.48</td>
    <td>99.87</td>
    <td>66.14</td>
    <td><b>99.47</b></td>
    <td>14.98</td>
    <td>99.82</td>
    <td>71.74</td>
  </tr>
  <tr>
    <td>OOD w/ IPQ</td>
    <td>100.0</td>
    <td>84.59</td>
    <td>66.26</td>
    <td>75.27</td>
    <td>19.82</td>
    <td>99.83</td>
    <td>68.45</td>
    <td>99.5</td>
    <td><b>14.81</b></td>
    <td>99.82</td>
    <td>72.84</td>
  </tr>
  <tr>
    <td>OOD w/ Ours</td>
    <td>93.75</td>
    <td><b>70.71</b></td>
    <td><b>51.17</b></td>
    <td>71.02</td>
    <td><b>19.21</b></td>
    <td>99.6</td>
    <td><b>65.25</b></td>
    <td>100.0</td>
    <td>16.56</td>
    <td><b>98.72</b></td>
    <td><b>68.6</b></td>
  </tr>
</table>

Table 7: Comparison of OOD document detection across different retriever models, with the number of documents selected by **2,000** sampled NQ documents.

<table>
  <tr>
    <th>Retriever</th>
    <th># Samples</th>
    <th>ArguAna</th>
    <th>Climate-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>TREC-COVID</th>
    <th>Touche</th>
  </tr>
  <tr>
    <td rowspan="3">BGE</td>
    <td>1000</td>
    <td>109</td>
    <td>339</td>
    <td>6940</td>
    <td>15686</td>
    <td>1236</td>
    <td>1542</td>
    <td>1204</td>
    <td>221</td>
    <td>10018</td>
    <td>79</td>
  </tr>
  <tr>
    <td>2000</td>
    <td>103</td>
    <td>323</td>
    <td>6795</td>
    <td>15593</td>
    <td>1143</td>
    <td>1458</td>
    <td>1139</td>
    <td>207</td>
    <td>9632</td>
    <td>76</td>
  </tr>
  <tr>
    <td>3000</td>
    <td>101</td>
    <td>306</td>
    <td>6660</td>
    <td>15489</td>
    <td>1063</td>
    <td>1376</td>
    <td>1073</td>
    <td>190</td>
    <td>9194</td>
    <td>74</td>
  </tr>
  <tr>
    <td rowspan="3">Contriever</td>
    <td>1000</td>
    <td>89</td>
    <td>571</td>
    <td>9712</td>
    <td>13201</td>
    <td>2429</td>
    <td>6320</td>
    <td>2926</td>
    <td>505</td>
    <td>10469</td>
    <td>278</td>
  </tr>
  <tr>
    <td>2000</td>
    <td>85</td>
    <td>540</td>
    <td>9492</td>
    <td>12655</td>
    <td>2328</td>
    <td>5922</td>
    <td>2830</td>
    <td>483</td>
    <td>10005</td>
    <td>246</td>
  </tr>
  <tr>
    <td>3000</td>
    <td>81</td>
    <td>520</td>
    <td>9378</td>
    <td>12343</td>
    <td>2271</td>
    <td>5695</td>
    <td>2769</td>
    <td>480</td>
    <td>9742</td>
    <td>233</td>
  </tr>
  <tr>
    <td rowspan="3">E5</td>
    <td>1000</td>
    <td>192</td>
    <td>855</td>
    <td>6035</td>
    <td>16383</td>
    <td>2379</td>
    <td>7347</td>
    <td>2591</td>
    <td>523</td>
    <td>14496</td>
    <td>520</td>
  </tr>
  <tr>
    <td>2000</td>
    <td>187</td>
    <td>836</td>
    <td>5850</td>
    <td>16328</td>
    <td>2319</td>
    <td>7059</td>
    <td>2534</td>
    <td>508</td>
    <td>14464</td>
    <td>505</td>
  </tr>
  <tr>
    <td>3000</td>
    <td>183</td>
    <td>815</td>
    <td>5736</td>
    <td>16297</td>
    <td>2289</td>
    <td>6895</td>
    <td>2500</td>
    <td>506</td>
    <td>14437</td>
    <td>497</td>
  </tr>
  <tr>
    <td rowspan="3">GTE</td>
    <td>1000</td>
    <td>16</td>
    <td>494</td>
    <td>3643</td>
    <td>7095</td>
    <td>469</td>
    <td>2931</td>
    <td>1828</td>
    <td>193</td>
    <td>7977</td>
    <td>536</td>
  </tr>
  <tr>
    <td>2000</td>
    <td>16</td>
    <td>499</td>
    <td>3759</td>
    <td>7259</td>
    <td>486</td>
    <td>2990</td>
    <td>1853</td>
    <td>201</td>
    <td>8113</td>
    <td>537</td>
  </tr>
  <tr>
    <td>3000</td>
    <td>16</td>
    <td>497</td>
    <td>3738</td>
    <td>7219</td>
    <td>478</td>
    <td>2973</td>
    <td>1846</td>
    <td>197</td>
    <td>8082</td>
    <td>537</td>
  </tr>
</table>

Table 8: The number of detected OOD documents for each dataset, determined by the randomly sampled 1000, 2000, and 3000 NQ documents.

<table>
  <tr>
    <th>Retriever</th>
    <th>ArguAna</th>
    <th>Climate-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>TREC-COVID</th>
    <th>Touche</th>
  </tr>
  <tr>
    <td>BGE</td>
    <td>96.83</td>
    <td>24.42</td>
    <td>3.65</td>
    <td>48.4</td>
    <td>5.45</td>
    <td>99.59</td>
    <td>14.04</td>
    <td>100.0</td>
    <td>1.94</td>
    <td>9.38</td>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>81.97</td>
    <td>30.43</td>
    <td>8.0</td>
    <td>26.53</td>
    <td>4.93</td>
    <td>96.06</td>
    <td>23.27</td>
    <td>89.81</td>
    <td>0.76</td>
    <td>23.53</td>
  </tr>
  <tr>
    <td>E5</td>
    <td>95.07</td>
    <td>25.12</td>
    <td>4.67</td>
    <td>43.45</td>
    <td>4.97</td>
    <td>98.72</td>
    <td>21.78</td>
    <td>93.69</td>
    <td>1.9</td>
    <td>24.61</td>
  </tr>
  <tr>
    <td>GTE</td>
    <td>100.0</td>
    <td>39.81</td>
    <td>30.56</td>
    <td>42.88</td>
    <td>7.81</td>
    <td>91.13</td>
    <td>26.87</td>
    <td>81.82</td>
    <td>2.48</td>
    <td>40.53</td>
  </tr>
</table>

Table 9: DRR with Recall@10.
<table>
  <tr>
    <th>Retriever</th>
    <th>ArguAna</th>
    <th>Climate-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>TREC-COVID</th>
    <th>Touche</th>
  </tr>
  <tr>
    <th>BGE</th>
    <td>98.41</td>
    <td>32.26</td>
    <td>9.23</td>
    <td>62.69</td>
    <td>8.53</td>
    <td>99.79</td>
    <td>23.97</td>
    <td>100.0</td>
    <td>4.20</td>
    <td>46.88</td>
  </tr>
  <tr>
    <th>Contriever</th>
    <td>91.80</td>
    <td>44.31</td>
    <td>18.36</td>
    <td>38.19</td>
    <td>8.88</td>
    <td>97.47</td>
    <td>34.08</td>
    <td>93.57</td>
    <td>2.34</td>
    <td>67.65</td>
  </tr>
  <tr>
    <th>E5</th>
    <td>98.03</td>
    <td>34.74</td>
    <td>8.38</td>
    <td>57.64</td>
    <td>8.74</td>
    <td>99.21</td>
    <td>32.46</td>
    <td>97.6</td>
    <td>5.44</td>
    <td>69.11</td>
  </tr>
  <tr>
    <th>GTE</th>
    <td>100.0</td>
    <td>54.37</td>
    <td>56.97</td>
    <td>56.17</td>
    <td>14.51</td>
    <td>95.91</td>
    <td>45.26</td>
    <td>90.91</td>
    <td>5.39</td>
    <td>88.17</td>
  </tr>
</table>

Table 10: DRR with Recall@30.

<table>
  <tr>
    <th rowspan="2">Retriever</th>
    <th rowspan="2"># Samples</th>
    <th colspan="2">ArguAna</th>
    <th colspan="2">Climate-FEVER</th>
    <th colspan="2">DBPedia</th>
    <th colspan="2">FiQA</th>
    <th colspan="2">NFCorpus</th>
    <th colspan="2">Quora</th>
    <th colspan="2">Scidocs</th>
    <th colspan="2">SciFact</th>
    <th colspan="2">TREC-COVID</th>
    <th colspan="2">Touche</th>
  </tr>
  <tr>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
    <th>1000</th>
    <th>2000</th>
  </tr>
  <tr>
    <td>BGE</td>
    <td>1000</td>
    <td>8.71</td>
    <td>10.12</td>
    <td>25.63</td>
    <td>68.71</td>
    <td>5.42</td>
    <td>2.0</td>
    <td>7.39</td>
    <td>6.0</td>
    <td>14.4</td>
    <td>3.15</td>
    <td>2000</td>
    <td>8.71</td>
    <td>10.19</td>
    <td>25.65</td>
    <td>68.82</td>
    <td>5.48</td>
    <td>2.01</td>
    <td>7.41</td>
    <td>6.0</td>
    <td>14.49</td>
    <td>3.15</td>
    <td>3000</td>
    <td>8.99</td>
    <td>10.42</td>
    <td>26.14</td>
    <td>69.96</td>
    <td>5.67</td>
    <td>2.09</td>
    <td>7.81</td>
    <td>6.45</td>
    <td>15.6</td>
    <td>3.48</td>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>1000</td>
    <td>8.21</td>
    <td>22.25</td>
    <td>46.57</td>
    <td>41.71</td>
    <td>38.32</td>
    <td>13.21</td>
    <td>46.89</td>
    <td>49.33</td>
    <td>34.88</td>
    <td>10.33</td>
    <td>2000</td>
    <td>8.42</td>
    <td>22.84</td>
    <td>46.85</td>
    <td>42.6</td>
    <td>39.06</td>
    <td>13.52</td>
    <td>47.56</td>
    <td>50.07</td>
    <td>35.54</td>
    <td>10.65</td>
    <td>3000</td>
    <td>8.71</td>
    <td>22.84</td>
    <td>47.09</td>
    <td>43.25</td>
    <td>39.61</td>
    <td>13.74</td>
    <td>47.99</td>
    <td>50.22</td>
    <td>36.35</td>
    <td>11.09</td>
  </tr>
  <tr>
    <td>E5</td>
    <td>1000</td>
    <td>14.49</td>
    <td>19.12</td>
    <td>10.67</td>
    <td>79.88</td>
    <td>29.51</td>
    <td>8.75</td>
    <td>25.82</td>
    <td>27.74</td>
    <td>84.29</td>
    <td>20.65</td>
    <td>2000</td>
    <td>14.20</td>
    <td>18.68</td>
    <td>10.25</td>
    <td>79.17</td>
    <td>28.57</td>
    <td>8.48</td>
    <td>25.32</td>
    <td>26.84</td>
    <td>83.70</td>
    <td>19.57</td>
    <td>3000</td>
    <td>14.49</td>
    <td>19.12</td>
    <td>10.95</td>
    <td>79.85</td>
    <td>29.48</td>
    <td>8.73</td>
    <td>25.8</td>
    <td>27.74</td>
    <td>84.24</td>
    <td>20.65</td>
  </tr>
  <tr>
    <td>GTE</td>
    <td>1000</td>
    <td>0.21</td>
    <td>9.45</td>
    <td>4.49</td>
    <td>11.96</td>
    <td>2.2</td>
    <td>3.91</td>
    <td>13.73</td>
    <td>1.5</td>
    <td>8.49</td>
    <td>40.43</td>
    <td>2000</td>
    <td>0.21</td>
    <td>8.85</td>
    <td>4.39</td>
    <td>11.63</td>
    <td>2.15</td>
    <td>3.78</td>
    <td>13.23</td>
    <td>1.35</td>
    <td>7.88</td>
    <td>39.78</td>
    <td>3000</td>
    <td>0.07</td>
    <td>6.62</td>
    <td>3.69</td>
    <td>9.89</td>
    <td>1.49</td>
    <td>3.15</td>
    <td>10.92</td>
    <td>0.6</td>
    <td>5.86</td>
    <td>35.87</td>
  </tr>
</table>

Table 11: Ratio of detected OOD documents for each dataset over total documents, determined by the randomly sampled 1000, 2000, and 3000 NQ documents.
<table>
  <tr>
    <th>Retriever</th>
    <th>Dropout</th>
    <th>Num Pos</th>
    <th>ArguAna</th>
    <th>Climate-FEVER</th>
    <th>DBPedia</th>
    <th>FiQA</th>
    <th>NFCorpus</th>
    <th>Quora</th>
    <th>Scidocs</th>
    <th>SciFact</th>
    <th>TREC-COVID</th>
    <th>Touche</th>
    <th>Avg (.)</th>
  </tr>
  <tr>
    <td rowspan="6">BGE</td>
    <td rowspan="6">0.02</td>
    <td>1</td>
    <td>98.99</td>
    <td>68.32</td>
    <td>35.25</td>
    <td>79.48</td>
    <td>19.07</td>
    <td>99.85</td>
    <td>62.51</td>
    <td>100.0</td>
    <td>15.06</td>
    <td>89.19</td>
    <td>66.77</td>
  </tr>
  <tr>
    <td>2</td>
    <td>98.99</td>
    <td>62.57</td>
    <td>33.46</td>
    <td>79.24</td>
    <td>18.59</td>
    <td>99.71</td>
    <td>61.25</td>
    <td>99.7</td>
    <td>15.23</td>
    <td>91.89</td>
    <td>66.99</td>
  </tr>
  <tr>
    <td>4</td>
    <td>98.99</td>
    <td>60.23</td>
    <td>32.66</td>
    <td>79.21</td>
    <td>18.43</td>
    <td>99.71</td>
    <td>60.96</td>
    <td>99.72</td>
    <td>14.89</td>
    <td>89.19</td>
    <td>65.4</td>
  </tr>
  <tr>
    <td>8</td>
    <td>98.99</td>
    <td>62.34</td>
    <td>31.53</td>
    <td>79.13</td>
    <td>18.64</td>
    <td>99.78</td>
    <td>61.06</td>
    <td>99.72</td>
    <td>14.83</td>
    <td>89.19</td>
    <td>65.79</td>
  </tr>
  <tr>
    <td>16</td>
    <td>98.99</td>
    <td>62.34</td>
    <td>31.53</td>
    <td>79.13</td>
    <td>18.64</td>
    <td>99.78</td>
    <td>61.06</td>
    <td>99.72</td>
    <td>14.83</td>
    <td>89.19</td>
    <td>65.79</td>
  </tr>
  <tr>
    <td>1</td>
    <td>99.01</td>
    <td>67.31</td>
    <td>36.78</td>
    <td>79.29</td>
    <td>19.37</td>
    <td>99.35</td>
    <td>59.1</td>
    <td>100.0</td>
    <td>16.17</td>
    <td>89.33</td>
    <td>66.57</td>
  </tr>
  <tr>
    <td rowspan="6">0.05</td>
    <td>2</td>
    <td>99.01</td>
    <td>77.89</td>
    <td>34.66</td>
    <td>79.23</td>
    <td>19.01</td>
    <td>99.42</td>
    <td>57.86</td>
    <td>100.0</td>
    <td>15.84</td>
    <td>88.16</td>
    <td>67.11</td>
  </tr>
  <tr>
    <td>4</td>
    <td>99.01</td>
    <td>78.24</td>
    <td>33.15</td>
    <td>79.22</td>
    <td>18.72</td>
    <td>99.49</td>
    <td>57.77</td>
    <td>100.0</td>
    <td>15.74</td>
    <td>90.54</td>
    <td>67.19</td>
  </tr>
  <tr>
    <td>8</td>
    <td>99.01</td>
    <td>61.14</td>
    <td>31.49</td>
    <td>79.16</td>
    <td>18.36</td>
    <td>99.71</td>
    <td>56.97</td>
    <td>100.0</td>
    <td>15.22</td>
    <td>89.19</td>
    <td>65.03</td>
  </tr>
  <tr>
    <td>16</td>
    <td>99.01</td>
    <td>60.65</td>
    <td>31.57</td>
    <td>79.15</td>
    <td>18.39</td>
    <td>99.85</td>
    <td>60.31</td>
    <td>100.0</td>
    <td>15.0</td>
    <td>89.19</td>
    <td>65.31</td>
  </tr>
  <tr>
    <td>1</td>
    <td>99.01</td>
    <td>61.06</td>
    <td>35.71</td>
    <td>79.45</td>
    <td>19.35</td>
    <td>99.78</td>
    <td>63.91</td>
    <td>100.0</td>
    <td>15.47</td>
    <td>90.54</td>
    <td>66.43</td>
  </tr>
  <tr>
    <td>2</td>
    <td>99.01</td>
    <td>60.3</td>
    <td>33.87</td>
    <td>79.37</td>
    <td>18.92</td>
    <td>99.78</td>
    <td>62.16</td>
    <td>100.0</td>
    <td>15.18</td>
    <td>90.54</td>
    <td>65.88</td>
  </tr>
  <tr>
    <td rowspan="6">Contriever</td>
    <td rowspan="6">0.02</td>
    <td>1</td>
    <td>92.41</td>
    <td>71.08</td>
    <td>45.11</td>
    <td>57.55</td>
    <td>17.54</td>
    <td>99.32</td>
    <td>52.87</td>
    <td>97.79</td>
    <td>8.25</td>
    <td>92.7</td>
    <td>63.36</td>
  </tr>
  <tr>
    <td>2</td>
    <td>92.41</td>
    <td>70.45</td>
    <td>44.47</td>
    <td>57.25</td>
    <td>17.63</td>
    <td>99.24</td>
    <td>52.58</td>
    <td>98.02</td>
    <td>8.04</td>
    <td>91.85</td>
    <td>63.09</td>
  </tr>
  <tr>
    <td>4</td>
    <td>90.12</td>
    <td>71.08</td>
    <td>45.11</td>
    <td>57.55</td>
    <td>17.54</td>
    <td>99.32</td>
    <td>52.87</td>
    <td>97.79</td>
    <td>8.25</td>
    <td>92.7</td>
    <td>63.36</td>
  </tr>
  <tr>
    <td>8</td>
    <td>91.36</td>
    <td>64.35</td>
    <td>43.68</td>
    <td>57.07</td>
    <td>17.42</td>
    <td>99.42</td>
    <td>51.72</td>
    <td>97.79</td>
    <td>7.77</td>
    <td>93.56</td>
    <td>62.41</td>
  </tr>
  <tr>
    <td>16</td>
    <td>92.59</td>
    <td>64.35</td>
    <td>43.68</td>
    <td>57.07</td>
    <td>17.42</td>
    <td>99.42</td>
    <td>51.72</td>
    <td>97.79</td>
    <td>7.77</td>
    <td>93.56</td>
    <td>62.54</td>
  </tr>
  <tr>
    <td>1</td>
    <td>92.59</td>
    <td>71.19</td>
    <td>43.18</td>
    <td>56.32</td>
    <td>17.39</td>
    <td>97.91</td>
    <td>50.97</td>
    <td>97.8</td>
    <td>8.6</td>
    <td>91.88</td>
    <td>62.78</td>
  </tr>
  <tr>
    <td rowspan="6">0.05</td>
    <td>2</td>
    <td>92.59</td>
    <td>69.85</td>
    <td>41.87</td>
    <td>56.18</td>
    <td>17.09</td>
    <td>98.14</td>
    <td>50.01</td>
    <td>97.83</td>
    <td>8.44</td>
    <td>91.03</td>
    <td>62.30</td>
  </tr>
  <tr>
    <td>4</td>
    <td>90.12</td>
    <td>68.72</td>
    <td>40.97</td>
    <td>56.1</td>
    <td>16.99</td>
    <td>98.33</td>
    <td>50.78</td>
    <td>97.8</td>
    <td>8.35</td>
    <td>90.17</td>
    <td>61.83</td>
  </tr>
  <tr>
    <td>8</td>
    <td>91.36</td>
    <td>63.92</td>
    <td>40.63</td>
    <td>56.12</td>
    <td>17.11</td>
    <td>98.75</td>
    <td>50.64</td>
    <td>97.78</td>
    <td>8.09</td>
    <td>90.17</td>
    <td>61.46</td>
  </tr>
  <tr>
    <td>16</td>
    <td>92.59</td>
    <td>64.12</td>
    <td>43.6</td>
    <td>57.12</td>
    <td>17.39</td>
    <td>99.42</td>
    <td>51.68</td>
    <td>97.78</td>
    <td>7.94</td>
    <td>94.02</td>
    <td>62.54</td>
  </tr>
  <tr>
    <td>1</td>
    <td>92.59</td>
    <td>68.93</td>
    <td>41.86</td>
    <td>56.19</td>
    <td>17.11</td>
    <td>98.16</td>
    <td>49.99</td>
    <td>97.83</td>
    <td>8.48</td>
    <td>91.03</td>
    <td>62.75</td>
  </tr>
  <tr>
    <td>2</td>
    <td>92.59</td>
    <td>68.93</td>
    <td>41.86</td>
    <td>56.19</td>
    <td>17.11</td>
    <td>98.16</td>
    <td>49.99</td>
    <td>97.83</td>
    <td>8.48</td>
    <td>91.03</td>
    <td>62.22</td>
  </tr>
  <tr>
    <td rowspan="6">E5</td>
    <td rowspan="6">0.02</td>
    <td>1</td>
    <td>99.45</td>
    <td>75.21</td>
    <td>32.43</td>
    <td>74.54</td>
    <td>16.91</td>
    <td>99.38</td>
    <td>55.11</td>
    <td>98.35</td>
    <td>15.83</td>
    <td>97.03</td>
    <td>66.42</td>
  </tr>
  <tr>
    <td>2</td>
    <td>99.45</td>
    <td>76.79</td>
    <td>30.44</td>
    <td>74.46</td>
    <td>16.99</td>
    <td>99.40</td>
    <td>54.97</td>
    <td>98.65</td>
    <td>15.73</td>
    <td>96.63</td>
    <td>66.35</td>
  </tr>
  <tr>
    <td>4</td>
    <td>99.45</td>
    <td>75.25</td>
    <td>29.84</td>
    <td>74.5</td>
    <td>16.99</td>
    <td>99.61</td>
    <td>55.02</td>
    <td>98.75</td>
    <td>15.84</td>
    <td>96.25</td>
    <td>66.15</td>
  </tr>
  <tr>
    <td>8</td>
    <td>99.45</td>
    <td>69.19</td>
    <td>26.45</td>
    <td>74.47</td>
    <td>17.03</td>
    <td>99.68</td>
    <td>55.52</td>
    <td>98.55</td>
    <td>15.85</td>
    <td>96.43</td>
    <td>65.58</td>
  </tr>
  <tr>
    <td>16</td>
    <td>99.45</td>
    <td>66.97</td>
    <td>31.21</td>
    <td>74.52</td>
    <td>17.05</td>
    <td>99.75</td>
    <td>55.72</td>
    <td>98.35</td>
    <td>15.71</td>
    <td>96.6</td>
    <td>65.53</td>
  </tr>
  <tr>
    <td>1</td>
    <td>99.45</td>
    <td>75.21</td>
    <td>32.47</td>
    <td>74.54</td>
    <td>16.91</td>
    <td>99.38</td>
    <td>55.11</td>
    <td>98.35</td>
    <td>15.83</td>
    <td>97.03</td>
    <td>66.43</td>
  </tr>
  <tr>
    <td rowspan="6">0.05</td>
    <td>2</td>
    <td>99.45</td>
    <td>76.8</td>
    <td>30.44</td>
    <td>74.46</td>
    <td>16.99</td>
    <td>99.40</td>
    <td>54.97</td>
    <td>98.65</td>
    <td>15.73</td>
    <td>96.63</td>
    <td>66.36</td>
  </tr>
  <tr>
    <td>4</td>
    <td>99.45</td>
    <td>75.25</td>
    <td>29.89</td>
    <td>74.5</td>
    <td>16.99</td>
    <td>99.59</td>
    <td>55.0</td>
    <td>98.75</td>
    <td>15.84</td>
    <td>96.25</td>
    <td>66.15</td>
  </tr>
  <tr>
    <td>8</td>
    <td>99.45</td>
    <td>69.19</td>
    <td>29.74</td>
    <td>74.47</td>
    <td>17.02</td>
    <td>99.68</td>
    <td>55.5</td>
    <td>98.56</td>
    <td>15.85</td>
    <td>96.43</td>
    <td>65.59</td>
  </tr>
  <tr>
    <td>16</td>
    <td>99.45</td>
    <td>66.99</td>
    <td>30.99</td>
    <td>74.52</td>
    <td>17.09</td>
    <td>99.77</td>
    <td>55.71</td>
    <td>98.35</td>
    <td>15.72</td>
    <td>96.6</td>
    <td>65.52</td>
  </tr>
  <tr>
    <td>1</td>
    <td>99.51</td>
    <td>64.88</td>
    <td>40.74</td>
    <td>74.41</td>
    <td>16.79</td>
    <td>99.8</td>
    <td>53.42</td>
    <td>98.83</td>
    <td>14.95</td>
    <td>97.4</td>
    <td>66.07</td>
  </tr>
  <tr>
    <td>2</td>
    <td>99.51</td>
    <td>62.81</td>
    <td>40.92</td>
    <td>74.49</td>
    <td>16.59</td>
    <td>99.85</td>
    <td>52.99</td>
    <td>98.21</td>
    <td>15.04</td>
    <td>97.92</td>
    <td>66.85</td>
  </tr>
  <tr>
    <td rowspan="6">GTE</td>
    <td rowspan="6">0.02</td>
    <td>1</td>
    <td>93.75</td>
    <td>80.05</td>
    <td>53.77</td>
    <td>73.15</td>
    <td>21.77</td>
    <td>99.66</td>
    <td>69.18</td>
    <td>99.73</td>
    <td>16.21</td>
    <td>98.9</td>
    <td>70.63</td>
  </tr>
  <tr>
    <td>2</td>
    <td>93.75</td>
    <td>83.22</td>
    <td>54.14</td>
    <td>73.14</td>
    <td>21.14</td>
    <td>99.73</td>
    <td>67.8</td>
    <td>100.0</td>
    <td>15.89</td>
    <td>98.9</td>
    <td>70.77</td>
  </tr>
  <tr>
    <td>4</td>
    <td>93.75</td>
    <td>80.18</td>
    <td>57.45</td>
    <td>72.14</td>
    <td>22.52</td>
    <td>99.66</td>
    <td>67.73</td>
    <td>99.72</td>
    <td>15.89</td>
    <td>98.9</td>
    <td>70.79</td>
  </tr>
  <tr>
    <td>8</td>
    <td>93.75</td>
    <td>78.16</td>
    <td>60.09</td>
    <td>72.09</td>
    <td>21.62</td>
    <td>99.66</td>
    <td>67.42</td>
    <td>99.73</td>
    <td>16.24</td>
    <td>98.72</td>
    <td>70.75</td>
  </tr>
  <tr>
    <td>16</td>
    <td>93.75</td>
    <td>78.16</td>
    <td>60.09</td>
    <td>72.09</td>
    <td>21.62</td>
    <td>99.66</td>
    <td>67.42</td>
    <td>99.73</td>
    <td>16.24</td>
    <td>98.72</td>
    <td>70.75</td>
  </tr>
  <tr>
    <td>1</td>
    <td>93.75</td>
    <td>76.5</td>
    <td>41.71</td>
    <td>69.61</td>
    <td>19.6</td>
    <td>99.63</td>
    <td>64.1</td>
    <td>100.0</td>
    <td>17.2</td>
    <td>99.27</td>
    <td>68.19</td>
  </tr>
  <tr>
    <td rowspan="6">0.05</td>
    <td>2</td>
    <td>93.75</td>
    <td>82.36</td>
    <td>43.65</td>
    <td>69.89</td>
    <td>18.5</td>
    <td>99.66</td>
    <td>64.2</td>
    <td>100.0</td>
    <td>17.32</td>
    <td>99.09</td>
    <td>68.84</td>
  </tr>
  <tr>
    <td>4</td>
    <td>93.75</td>
    <td>75.33</td>
    <td>46.1</td>
    <td>70.26</td>
    <td>19.0</td>
    <td>99.56</td>
    <td>65.08</td>
    <td>100.0</td>
    <td>17.1</td>
    <td>98.72</td>
    <td>68.49</td>
  </tr>
  <tr>
    <td>8</td>
    <td>93.75</td>
    <td>70.83</td>
    <td>51.72</td>
    <td>70.19</td>
    <td>19.22</td>
    <td>99.6</td>
    <td>65.22</td>
    <td>100.0</td>
    <td>16.56</td>
    <td>98.72</td>
    <td>68.62</td>
  </tr>
  <tr>
    <td>16</td>
    <td>93.75</td>
    <td>78.03</td>
    <td>60.04</td>
    <td>72.05</td>
    <td>19.71</td>
    <td>99.66</td>
    <td>67.44</td>
    <td>99.73</td>
    <td>15.87</td>
    <td>98.9</td>
    <td>70.52</td>
  </tr>
  <tr>
    <td>1</td>
    <td>93.75</td>
    <td>76.54</td>
    <td>41.97</td>
    <td>69.7</td>
    <td>19.59</td>
    <td>99.56</td>
    <td>64.2</td>
    <td>100.0</td>
    <td>17.67</td>
    <td>99.27</td>
    <td>68.23</td>
  </tr>
  <tr>
    <td>2</td>
    <td>93.75</td>
    <td>82.41</td>
    <td>43.45</td>
    <td>69.89</td>
    <td>18.44</td>
    <td>99.66</td>
    <td>64.26</td>
    <td>100.0</td>
    <td>17.33</td>
    <td>99.09</td>
    <td>68.83</td>
  </tr>
  <tr>
    <td>4</td>
    <td>93.75</td>
    <td>75.31</td>
    <td>46.52</td>
    <td>70.25</td>
    <td>19.06</td>
    <td>99.56</td>
    <td>65.08</td>
    <td>100.0</td>
    <td>17.12</td>
    <td>98.72</td>
    <td>68.54</td>
  </tr>
  <tr>
    <td>8</td>
    <td>93.75</td>
    <td>70.5</td>
    <td>51.17</td>
    <td>71.09</td>
    <td>19.28</td>
    <td>99.6</td>
    <td>65.28</td>
    <td>100.0</td>
    <td>16.54</td>
    <td>98.72</td>
    <td>68.59</td>
  </tr>
  <tr>
    <td>16</td>
    <td>93.75</td>
    <td>78.41</td>
    <td>60.08</td>
    <td>72.12</td>
    <td>19.61</td>
    <td>99.66</td>
    <td>67.37</td>
    <td>99.73</td>
    <td>16.01</td>
    <td>98.9</td>
    <td>70.56</td>
  </tr>
</table>

Table 12: Ablation study on the number of positives for computing GradNormIR in (p in Eq. (4)). The results present the DRR values of OOD documents prediction, comparing results without dropout and with dropout rates of 0.02 and 0.05 for the document query.
Figure 5: Results of relevance gains via OOD document filtering.

Figure 6: Relation Between OOD Ratio and Performance
Figure 7: Feasibility results of GradNormIR for several recent retrievers on the BEIR benchmark. The x-axis shows quartiles of GradNormIR, sorted in ascending order (Q1 to Q4), while the y-axis represents the d2q recall@100, averaged across documents within each quartile. The results show that GradNormIR can predict retrieval performance; lower GradNormIR values (Q1) generally lead to better retrieval outcomes across most datasets. As GradNormIR increases (Q4), the d2q recall decreases.

Metareview score: 3
Metareview: This paper introduces GradNormIR, an unsupervised method to detect out-of-domain (OOD) documents before indexing in dense retrieval systems. It uses gradient norms from contrastive loss to identify documents that may not align with the existing retriever. The goal is to decide whether to retrain or switch retrievers when the document corpus evolves. The method is tested on the BEIR benchmark using several dense retrievers.
Summary of reasons to publish: The paper tackles a useful and practical problem for real-world retrieval systems: how to handle evolving or shifting corpora.

The idea of using gradient norms for OOD detection is interesting and avoids needing labeled queries.

The method is tested in multiple settings (OOD detection, retriever selection, and continual updates), showing its flexibility.