Venue: ACL 2023

# BERM: Training the Balanced and Extractable Representation for

Matching to Improve Generalization Ability of Dense Retrieval

 Shicheng Xu\({}^{1,2}\), Liang Pang\({}^{1}\), Huawei Shen\({}^{1,2}\), Xueqi Cheng\({}^{1,2}\)

\({}^{1}\)Data Intelligence System Research Center,

Institute of Computing Technology, CAS

\({}^{2}\)University of Chinese Academy of Sciences

xschit@163.com {pangliang,shenhuawei,cxq}@ict.ac.cn

Corresponding authors

###### Abstract

Dense retrieval has shown promise in the first-stage retrieval process when trained on indomain labeled datasets. However, previous studies have found that dense retrieval is hard to generalize to unseen domains due to its weak modeling of domain-invariant and interpretable feature (i.e., matching signal between two texts, which is the essence of information retrieval). In this paper, we propose a novel method to improve the generalization of dense retrieval via capturing matching signal called BERM. Fully fine-grained expression and query-oriented saliency are two properties of the matching signal. Thus, in BERM, a single passage is segmented into multiple units and two unit-level requirements are proposed for representation as the constraint in training to obtain the effective matching signal. One is _semantic unit balance_ and the other is _essential matching unit extractability_. Unit-level view and balanced semantics make representation express the text in a fine-grained manner. Essential matching unit extractability makes passage representation sensitive to the given query to extract the pure matching information from the passage containing complex context. Experiments on BEIR show that our method can be effectively combined with different dense retrieval training methods (vanilla, hard negatives mining and knowledge distillation) to improve its generalization ability without any additional inference overhead and target domain data.


(more fine-grained description for matching) is much more domain-invariant than it from dense retrieval. Match-Prompt Xu et al. (2022), NIRPrompt Xu et al. (2022) and MatchPyramid Pang et al. (2016) point out the positive significance of matching signals for various IR tasks. The challenge of making dense retrieval model learn to capture matching signal is that in many IR tasks such as open-domain question answering Chen et al. (2017) and document retrieval Mitra et al. (2017), the content that matches the query is usually only a unit of the text. The description of matching signal needs to distinguish the matching and not matching information in the text and estimate the overall relevance. This requires the retrieval model to be able to evenly express each unit in the text and dynamically extract matching units through the interaction of the two text representations. However, the requirement on efficiency in first-stage retrieval makes dense retrieval only estimate relevance via vector similarity such as dot product and cosine. Previous training methods based on this architecture lack the above capability because of the coarse-grained training objective and interaction.

In this paper, we propose a novel method called BERM to capture the matching signal between query and passage, which is the domain-invariant feature, to improve the generalization ability of dense retrieval during the training on the single source domain without using the target domain data and other additional modules. First, we introduce a novel concept in dense retrieval, the matching representation. Matching representation is determined by the text representations (output of text encoder) of query and passage, which can reflect the matching information of query and passage. We propose that in the training of dense retrieval models, in addition to using contrastive loss Xiong et al. (2021) to optimize the text representation, the information of the matching representation can be used as a constraint to assist the optimization. Based on this, we divide the single passage into multiple units (each sentence is a unit) and propose two requirements on the generalizable dense retrieval models as the constraint in training (shown in Figure 1). One is _semantic unit balance of text representation_ (**R1**). The other is _essential matching unit extractability of matching representation_ (**R2**). These two requirements can be integrated into different dense retrieval training methods and address the challenge mentioned above. **R1** means the semantics of units in a passage are implicitly aggregated to its text representation and the text representation should evenly and comprehensively express the semantics of each unit. **R2** means that the combination of text representations of query and passage (i.e., matching representation) should extract the information of the matching (i.e, the text chunk in the passage that matches the query and we call it **essential matching unit**) while reducing the overfitting of domain biases. This reflects the ability of the dense retrieval model to determine and score the information that really matches the query in a passage containing complex context, which is the essence of the dense retrieval and domain-invariant. **R1** and **R2** achieve that on the premise that the text representation expresses each unit in a balanced manner, to make essential matching units for different queries be extracted, the semantics of units tend to be orthogonal to each other. In this way, in dot product between representations of query and passage, the semantics of essential matching unit are preserved, while the other units are masked, which is suitable for matching.

Experiments on the standard zero-shot retrieval benchmark (BEIR) show that our method can be effectively combined with different dense retrieval training methods (vanilla, hard negatives mining, and knowledge distillation) to improve the generalization ability without any additional modules, inference overhead, and target domain data. Even in domain adaptation, our method is also effective and performs better than baselines. Code is released at [https://github.com/xsc1234/BERM](https://github.com/xsc1234/BERM).

## 2 Related Work

Dense retrieval estimates the relevance via representations of two texts. DPR Karpukhin et al. (2020) combines dense retrieval with pre-trained models for open-domain question answering Chen et al. (2017). Besides, some methods focus on obtaining more valuable negatives Qu et al. (2021); Xiong et al. (2021); Zhan et al. (2021). Some methods use a more powerful reranker for knowledge distillation Hofstatter et al. (2021); Lin et al. (2021). Recently, the generalization of dense retrieval has received attention. Ren et al. (2022) performs the examination of the generalization of dense retrieval. BEIR Thakur et al. (2021) is proposed as the benchmark to evaluate the zero-shot ability of information retrieval models. MoDIR Xin et al. (2022) uses the data from source and target domains for ad 

versarial training to perform unsupervised domain adaptation. GenQ (Ma et al., 2021) and GPL (Wang et al., 2022) generate queries and pseudo labels for domain adaptation. Contriever Izacard et al. (2021) uses contrastive pre-training on large corpus Wikipedia and CC-Net Wenzek et al. (2020). COCO-DR Yu et al. (2022) performs unsupervised pre-training on target domain and introduces distributional robust optimization. GTR Ni et al. (2021) scales up the model size to improve the generalization. Huebscher et al. (2022); Formal et al. (2022) introduce sparse retrieval to achieve better generalization.

Improvement of generalization of dense retrieval in previous studies comes from the adaptation of the target domain, knowledge from large pre-training corpus, and assistance of sparse retrieval but not dense retrieval itself. They need to obtain the target domain data in the training or increase the complexity of the system. In this paper, we introduce a novel method to improve the generalization of dense retrieval without target domain data and additional modules via learning the generalizable representation for matching.

One thing must be emphasized that the methods of multi-view dense retrieval Zhang et al. (2022); Hong et al. (2022) also divide a passage into multiple units, but our method is essentially a completely different method. Multi-view dense retrieval uses multiple representations to fully express a passage from multiple views, which focuses on in-domain retrieval. Our method uses multiple units to make the model learn to extract essential matching unit from the passage containing complex context, which is domain-invariant for generalization. In our method, multiple units are only used as the constraint for optimization in training and only a single representation is used in inference. Learning-based sparse retrieval such as COIL Gao et al. (2021) and SPLADE Formal et al. (2021) also aim to express fine-grained token-level semantics but they need multiple vectors to represent tokens in passage (COIL) or sparse-vector of vocabulary size (SPLADE) and calculates the score by token-to-token matching, which is not suitable for dense retrieval that uses single dense vector to perform representation and dot product.

## 3 Motivation

Dense retrieval is hard to generalize to unseen domains due to its weak modeling of domain-invariant feature (i.e., matching signal between two texts, which is the essence of information retrieval). Fully fine-grained expression (**P1**) and query-oriented saliency (**P2**) are two properties of the matching signal. These two require the passage representation to be able to evenly express each unit in the text, and dynamically extract matching units according to the interaction with different queries. For example, BM25 uses one-hot to evenly express each word of the text, only scores matching words, and ignores not matching words through word-to-word exact matching of the two texts. Cross-encoder uses word embedding to represent the semantics of each token and uses attention to describe the token-to-token semantic matching between texts in a fine-grained manner.

In this paper, based on the above two properties, for the training of dense retrieval, we segment a single passage into multiple units and propose two requirements as the constraint in training so that dense retrieval can capture the stronger matching signal and produces a suitable representation for matching. One is _semantic unit balance of text representation_ (**R1**), and the other is _essential matching unit extractability of matching representation_ (**R2**). Under **R1**, text representation evenly aggregates semantics of the units in the passage to comprehensively express the passage in a fine-grained manner. Besides, **R1** is the premise of **R2**. It is because that matching representation is composed of text representations from passage and query. Unbalanced semantic expression of different units in text representation will affect the identification of essential matching unit in matching representation because it leads to different preferences for different units. Under **R2**, essential matching unit for the query can be extracted from the passage and reflected in matching representation. Unlike using one-hot or word embedding to explicitly express the semantics of each unit and extract matching

Figure 2: An illustration of the effects of **R1** and **R2**.

 

information through token-to-token interaction, as shown in Figure 2, **R1** makes the model implicitly aggregate the semantics of each unit into the text representation to satisfy **P1**, and **R2** makes the semantics of units tend to be orthogonal to each other (shown in Table 6). In dot product between representations of query and passage, semantics of essential matching unit are preserved, while the other units are masked, which can satisfy **P2**. Our method unlocks the ability of dense retrieval to capture matching signal without additional interaction.

## 4 Our Method

This section introduces the implementation of our method (Figure 3). Our method optimizes the relationship between the representations and the units in the passage. Therefore, before training, we perform unit segmentation and annotate the essential matching unit for the datasets. Then, we design loss functions according to the requirements of **R1** and combine these functions with task loss of dense retrieval (contrastive loss) for joint training.

### Unit Segmentation and Annotation

Given each positive query-passage pair \((q,p_{pos})\) in training data, we segment positive passage into multiple units \(U\) as shown in Figure 3 (a) (We use the sentence as the segmentation granularity to ensure that each unit has complete semantic information.):

\[p_{pos}\stackrel{{Segment}}{{\longrightarrow}}U=\{u_{1},u_{2},... ,u_{n}\}. \tag{1}\]

For \(U\) and \(q\), BM25 is used to compute the word-to-word matching score \(S_{bm25}\) between \(q\) and \(u_{i}\in U\):

\[S_{bm25}=\{bm25(q,u_{1}),...,bm25(q,u_{n})\}.\]

For the datasets for question-answering, a trained reader model is additionally introduced to compute the semantic matching score \(S_{reader}\) between \(q\) and \(u_{i}\in U\). Specifically, reader model computes the probability distribution \(A=\{a_{1},a_{2},..,a_{t}\}\) of the starting positions of the answer in \(p_{pos}\). \(a_{i}\) indicates the probability that the \(i\)-\(th\) token in \(p_{pos}\) is the starting of the answer to \(q\). For each \(u_{i}\in U\), the semantic matching score from the reader model is:

\[r_{i}=max(A[s_{u_{i}}:d_{u_{i}}]), \tag{2}\]

where \([s_{u_{i}}:d_{u_{i}}]\) are the indexes of tokens in \(u_{i}\). The hybrid matching score \(h_{i}\) between \(u_{i}\) and \(q\) is:

\[h_{i}=bm25(q,u_{i})+\delta r_{i},\]

where \(\delta\) is a hyperparameter. We set \(\delta\) as \(0.1\) to give BM25 a higher weight than the reader. It is because the word-to-word exact matching of BM25 is more domain-invariant and conducive to generalization than the semantic matching of reader Thakur et al. (2021). Then we get matching score list \(H=\{h_{1},h_{2},...,h_{n}\}\) for \(U\). The essential matching unit is the unit corresponding to the maximum value in \(H\). For the pair \((q,p_{pos})\), \(y_{i}\) in label list \(Y=\{y_{1},...,y_{n}\}\) for essential matching unit is that if \(i\) is the index corresponding to the maximum value in \(H\), \(y_{i}=1\), otherwise, \(y_{i}=0\).

Figure 3: Overview of BERM. (a) Unit Segmentation and Annotation: Segment passage into multiple units (each sentence is a unit) and use the annotator to identify essential matching unit. (b) Training: Training models under the constraints of two requirements on the generalizable dense retrieval for matching. (c) Inference: Same as the mainstream dense retrieval, without introducing additional inference and storage overhead.

 



### Training for Generalization

Based on the analysis of properties of matching signal in Section 3, we propose two requirements as the constraints in the training of dense retrieval to get a generalizable representation for matching (shown in Figure 3 (b)). These two requirements enable dense retrieval to extract essential matching information under the premise of balanced expression of each unit, so as to learn domain-invariant feature (i.e., matching signal) for generalization.

**Implementation of R1.** The first requirement is _semantic unit balance of text representation_, which means that the text representation of the passage encoder can comprehensively express the semantics of each unit in a balanced manner. Given the passage \(p_{pos}\), the text encoder \(g(\cdot;\theta)\), output hidden states \(\mathbf{Z}=g(p_{pos};\theta)\). Text representation \(\mathbf{t_{p}}\) of \(p_{pos}\) is the embedding of [CLS] token of \(\mathbf{Z}\). The embeddings \(\mathbf{E}\) of units in \(p_{pos}\) can be obtained from \(\mathbf{Z}\) as the segmentation in Equ.(1):

\[\mathbf{E}=\{\mathbf{e_{1}},\mathbf{e_{2}},...,\mathbf{e_{n}}\}, \tag{3}\]

where \(\mathbf{e_{i}}\) is the embedding of the corresponding unit \((u_{i})\) and it is the average pooling of the embeddings of tokens (\(\mathbf{Z}[s_{u_{i}}:d_{u_{i}}]\)) in the unit, where \([s_{u_{i}}:d_{u_{i}}]\) are the indexes of tokens in \(u_{i}\). Under the constraint of **R1**, the relationship between \(\mathbf{t_{p}}\) and \(\mathbf{E}\) is described by the loss function as:

\[\mathcal{L}_{balance}=D_{KL}[\mathbf{b}||sim(\mathbf{t_{p}},\mathbf{E})], \tag{4}\]

where \(D_{KL}[|\cdot|]\) is KL-divergence loss, \(\mathbf{b}=[\frac{1}{n},...,\frac{1}{n}]\) is a uniform distribution with equal values and \(sim(\mathbf{t_{p}},\mathbf{E})=\{dot(\mathbf{t_{p}},\mathbf{e_{i}})|\mathbf{e_{i}}\in\mathbf{E}\}\) is a distribution to represent the semantic similarity between \(\mathbf{t_{p}}\) and \(\mathbf{e_{i}}\in\mathbf{E}\), \(dot(\cdot,\cdot)\) is dot product.

**Implementation of R2.** The second requirement is _essential matching unit extractability of matching representation_, which means that under the premise of **R1**, matching representation can saliently represent the unit where the essential matching block is located in. The motivation for this design is discussed in Section 1 and 3. Given the positive query-passage pair \((q,p_{pos})\), text encoder \(g(\cdot;\theta)\), and the text representations for \(q\) (\(\mathbf{t_{q}}\)) and \(p_{pos}\) (\(\mathbf{t_{p}}\)). Matching representation \(\mathbf{m}\in\mathbb{R}^{v}\) (\(v\) is the dimension of representation) for \(q\) and \(p_{pos}\) can be obtained by the combination of \(\mathbf{t_{q}}\in\mathbb{R}^{v}\) and \(\mathbf{t_{p}}\in\mathbb{R}^{v}\) as:

\[\mathbf{m}=\textit{GELU}(\mathbf{t_{q}}\odot\mathbf{t_{p}}),\]

where \(\odot\) is the element-wise multiplication operator, and \(\textit{GELU}(\cdot)\) is activation function Hendrycks and Gimpel (2016) to introduce stochastic regularization. Under the premise of **R1**, \(\mathbf{t_{p}}\) can express the semantics of the units in \(p_{pos}\) in a balanced manner. In addition, the semantic representation of essential matching unit is more similar to \(\mathbf{t_{q}}\) than other units because it really matches the query \(q\). Based on this, the model can be trained to achieve the goal that element-wise multiplication between \(\mathbf{t_{q}}\) and \(\mathbf{t_{p}}\) can amplify similar patterns (i.e, semantic representation of essential matching unit) and mask the signals of other context units. This design can be supported by convolutional neural networks LeCun et al. (1998) whose convolution operation can amplify similar patterns in tensors Girshick et al. (2014). For the \(p_{pos}\), different \(q\) amplifies different matching units, which makes \(\mathbf{m}\) reflect the semantics of the corresponding essential matching unit. Besides, \(\mathbf{m}\) is obtained by element-wise multiplication between \(\mathbf{t_{q}}\) and \(\mathbf{t_{p}}\), which is an important part of estimating the relevance of two texts because \(dot(\mathbf{t_{q}},\mathbf{t_{p}})=sum(\mathbf{t_{q}}\odot\mathbf{t_{p}})\). Thus, the optimization of \(\mathbf{m}\) can enable the model to obtain the ability to extract essential matching unit according to different queries when estimating relevance. In training, our method utilizes the cross-entropy loss function to optimize the semantic distance between \(\mathbf{m}\) and each unit to identify the corresponding essential matching units. Given query-passage pair (\(q\), \(p_{pos}\)), the embeddings \(\mathbf{E}\) of the units in \(p_{pos}\) as described in Equ. (3), and the label \(Y\) for essential matching unit of (\(q\), \(p_{pos}\)) as described in Sec. 4.1. Loss function for **R2** is:

\[\mathcal{L}_{extract}=-\sum_{i=1}^{n}y_{i}\log(dot(\mathbf{m},\mathbf{e_{ i}})), \tag{5}\]

where \(\mathbf{e_{i}}\in\mathbf{E}\), \(y_{i}\in Y\). \(\mathbf{m}\) is only used as the constraint in training but has important implications for inference. It is because that \(\mathbf{m}\) is the combination of text representations (\(\mathbf{t_{p}}\) and \(\mathbf{t_{q}}\)). The optimization for \(\mathbf{m}\) is training the text encoder to output the text representation that is suitable for matching to improve the generalization ability.

**Effect of R1 and R2.** Table 6 indicates that compared with previous dense retrieval methods, our method makes the semantics of units in text representation tend to be orthogonal to each other. In dot product between two texts, semantics of essential matching unit are preserved, while the other units are masked to capture matching signal.

**Total Loss.** In addition to \(\mathcal{L}_{extract}\) and \(\mathcal{L}_{balance}\), contrastive loss is used to train the dense retrieval

model (Karpukhin et al., 2020) as:

\[{\mathcal{L}_{c}=-\frac{\exp(dot(t_{q},t_{p^{+}}))}{\exp(dot(t_{q},t_{p^{+}}))+ \exp(dot(t_{q},t_{p^{-}}))}.}\]

So the total loss for training in our method is:

\[{\mathcal{L}=\mathcal{L}_{c}+\alpha\mathcal{L}_{extract}+\beta\mathcal{L}_{balance}},\]

where \(\alpha\) and \(\beta\) are the hyperparameters.

## 5 Experiments

This section introduces the experimental setups and analyzes the results.

### Experimental Setups

**Datasets.** We use MS-MARCO (Nguyen et al., 2016) as the training data (source domain) and choose the 14 publicly available datasets from BEIR 1, a heterogeneous benchmark to evaluate the generalization ability of retrieval models. In addition, we also introduce OAG-QA (Tam et al., 2022) to evaluate the topic generalization ability. Details of datasets are in Appendix A.

Footnote 1: The left four are unavailable due to copyright restrictions.

**Baselines.** Our method (BERM) aims to improve the generalization of dense retrieval without any additional modules and target domain data, and it can be combined with different dense retrieval training methods. We select three mainstream dense retrieval training methods including vanilla, hard negatives mining, and knowledge distillation as the baselines. We follow DPR (Karpukhin et al., 2020) to perform vanilla, follow ANCE (Xiong et al., 2021a) to perform hard negatives mining and use a trained cross-encoder as the teacher model to perform knowledge distillation. We compare the change in generalization after combining BERM with these three methods to show the effectiveness of our method. Besides, as previous methods need to obtain target domain data for domain adaptation such as MoDIR (Xin et al., 2022), GenQ (Ma et al., 2021), GPL (Wang et al., 2022) and COCO-DR (Yu et al., 2022), we also compare our method with these methods in domain adaptation setting. Details of baselines are in Appendix B.

**Implementation Details.** To maintain a fair comparison, we follow (Xiong et al., 2021a) to keep all common hyperparameters (learning rate and batch size, etc.) the same as the three dense retrieval training methods in the baselines. The model is initialized by Roberta\({}_{base}\) 12SEM. For the hyperparameters in BERT, \(\delta\) is 0.1, \(\alpha\) is 0.1 and \(\beta\) is 1.0. In domain adaptation, we combine BERM with continuous contrastive pretraining (Yu et al., 2022) to perform unsupervised pre-training on BEIR and use BERM to fine-tune the model on MS-MARCO. We train the model with Pytorch (Paszke et al., 2019) and Hugging Face (Wolf et al., 2020) on 2 Tesla V100 32GB GPUs for about 72 hours.

### Retrieval Performance

**Main Results.** Table 1 shows the main results on BEIR of different dense retrieval training methods. The results indicate that our method (BERM) can

\begin{table}
\begin{tabular}{l|c c|c c|c c|c} \hline \multirow{2}{*}{Datasets} & Jaccard Sim & \multicolumn{2}{c|}{Vanilla} & \multicolumn{2}{c|}{Knowledge Distillation} & \multicolumn{2}{c}{Hard Negatives} \\  & Unigrams & DPR & DPR+BERM & KD & KD+BERTM & ANCE & ANCE+BERM \\ \hline SciFact & 22.16 & 0.478 & **0.495\({}^{\dagger}\)** & 0.481 & **0.504\({}^{\dagger}\)** & 0.507 & **0.511\({}^{\dagger}\)** \\ NFCorpus & 23.45 & 0.208 & **0.234\({}^{\dagger}\)** & 0.205 & **0.242\({}^{\dagger}\)** & 0.237 & **0.248\({}^{\dagger}\)** \\ TREC-COVID & 26.80 & 0.561 & **0.600\({}^{\dagger}\)** & 0.490 & **0.505\({}^{\dagger}\)** & 0.654 & **0.661\({}^{\dagger}\)** \\ SCIDOCS & 27.92 & 0.108 & **0.120\({}^{\dagger}\)** & 0.111 & **0.115\({}^{\dagger}\)** & 0.122 & **0.130\({}^{\dagger}\)** \\ DBPedia & 30.16 & 0.236 & **0.256\({}^{\dagger}\)** & 0.245 & **0.264\({}^{\dagger}\)** & 0.281 & **0.293\({}^{\dagger}\)** \\ CQADupStack & 30.64 & **0.281** & 0.279 & **0.290** & 0.281 & **0.296** & 0.290 \\ HotpotQA & 30.87 & 0.371 & **0.386\({}^{\dagger}\)** & 0.427 & **0.438\({}^{\dagger}\)** & 0.456 & **0.463\({}^{\dagger}\)** \\ ArguAna & 32.92 & 0.414 & **0.435\({}^{\dagger}\)** & 0.435 & **0.437\({}^{\dagger}\)** & 0.415 & **0.428\({}^{\dagger}\)** \\ Climate-FEVER & 34.79 & 0.176 & **0.187\({}^{\dagger}\)** & 0.189 & **0.195\({}^{\dagger}\)** & 0.198 & **0.201\({}^{\dagger}\)** \\ FEVER & 34.79 & **0.589** & 0.585 & 0.633 & **0.664\({}^{\dagger}\)** & 0.669 & **0.674\({}^{\dagger}\)** \\ FiQA-2018 & 35.95 & **0.275** & 0.272 & **0.286** & 0.285 & **0.295** & 0.287 \\ Töuche-2020 & 37.02 & 0.208 & **0.210\({}^{\dagger}\)** & 0.215 & **0.216\({}^{\dagger}\)** & 0.240 & **0.248\({}^{\dagger}\)** \\ Quora & 39.75 & 0.842 & **0.853\({}^{\dagger}\)** & 0.832 & **0.836\({}^{\dagger}\)** & 0.852 & **0.854\({}^{\dagger}\)** \\ NQ & 47.27 & **0.398** & 0.394 & **0.420** & 0.419 & 0.446 & **0.450\({}^{\dagger}\)** \\ Avg & - & 0.368 & **0.379** & 0.376 & **0.386** & 0.405 & **0.410\({}^{\dagger}\)** \\ \hline \end{tabular}
\end{table}
Table 1: Zero-shot performance on BEIR (nDCG@10) without any target domain data. **Bold** indicates the better performance in the same training method. †: results with significant performance improvement with p-value \(\leq 0.05\) compared with baselines. Datasets are ordered by the Jaccard similarity between the source domain (MS-MARCO).

 

be combined with three mainstream dense retrieval training methods (vanilla, knowledge distillation, and hard negatives) to improve the generalization ability without any additional modules and target domain data. For a fair comparison, we combine BERM with the baselines and ensure that their common hyperparameters are consistent. We compute the Jaccard similarity Ioffe (2010) between each dataset and MS-MARCO, which can reflect the domain shift between the source and target domain. Table 1 shows that our method is more effective for the datasets with lower Jaccard similarity between MS-MARCO (i.e., domain shift is more significant). This result reflects the ability of our method to capture domain-invariant feature. DPR+BERM and KD+BERM are better than KD, which shows that BERM more effectively enables dense retrieval to learn to capture matching signal than knowledge distillation from cross-encoder.

**Topic Generalization.** Table 2 shows the generalization performance of DPR and DPR+BERM on different topics of QAG-QA. Topic generalization is important for out-of-domain generalization, which reflects the availability of dense retrieval model for topics with different word distributions. The results show that BERM can significantly improve cross-topic generalization of dense retrieval.

**Domain Adaptation.** Table 3 shows that BERM achieves the best performance in domain adaptation compared with previous baselines. Specifically, BERM achieves the best average out-of-domain adaptation and in-domain performance. Besides, it gets the best dense retrieval results on seven datasets of BEIR, which is the most of all methods. Our method not only learns the word distribution of the target domain, but also learns the representation suitable for matching for the documents in the target corpus during domain adaptation.

### Ablation Study

**Influence of Loss Functions.** Table 4 shows the ablation study on the loss functions constrained by **R1** and **R2** via average performance on BEIR. The results indicate that without \(\mathcal{L}_{balance}\), \(\mathcal{L}_{extract}\) can not improve the generalization, which supports our intuition in Section 3 that only based on the balanced semantic expression of each unit in the text representation, the matching representation is meaningful for extracting the essential semantic unit. This experiment shows that the generalization can be improved significantly when the model is constrained by both **R1** and **R2**.

**Influence of Hyperparameters.** Figure 4 shows the average nDCG@10 performance on BEIR with different \(\alpha\) and \(\beta\) that are used to tune the weights of different loss functions in training. When \(\alpha\) is \(0.1\) and \(\beta\) is \(1.0\), our method can achieve the best performance. When \(\alpha\) and \(\beta\) are too big, they will interfere with the optimization of the contrastive loss leading to performance degradation.

[FIGURE:S5.F4][ENDFIGUR 



### Model Analysis

**Domain-invariant Representation.** Figure 5 shows that our method is effective in capturing the domain-invariant feature of the representation. We utilize T-SNE to visualize the representations of source and target (SciFact) domains encoded by DPR and DPR+BERM respectively. The results indicate that representations of the two domains encoded by DPR are more separable. After combining our method, the two domains become more difficult to separate, which indicates that our method is more invariant to represent the texts in different domains. More datasets are in Appendix C.

**Evaluation of R1 and R2.** Table 5 shows the effectiveness of **R1** and **R2**. We randomly sample 100,000 query-passage pairs from the test set. For each passage \(p\), we compute semantic similarity between text representation and each unit via \(sim({\mathbf{t}_{p}},{\mathbf{E}})=\{dot(dof({\mathbf{t}_{p}},{\mathbf{e}_{i}})|{\mathbf{e}_{i}}\in{ \mathbf{E}}\}\). We compute the variance of \(sim({\mathbf{t}_{p}},{\mathbf{E}})\) and get the average of variance on the sampled set, which can reflect the balance of text representation on expressing the semantics of units. Table 5 shows that BERM has a smaller variance (semantic unit balance of text representation) and is more accurate in identifying the essential matching unit ( essential matching unit extractability of matching representation) than baselines, which indicates the effectiveness of **R1** and **R2**.

**Relationship Between Units.** Table 6 shows that our method makes units in a passage more dispersed (tend to be orthogonal), which is more conducive to determining th

Figure 5: T-SNE of the text and matching representations for source and target domains.

\begin{table}
\begin{tabular}{l|c c|c c c c c c} \hline \hline Datasets & Sparse & Late-Inter. & \multicolumn{8}{c}{Dense} \\  & BM25 & ColBERT & MoDIR & Contriever & GenQ & GPL & COCO-DR & BERM (ours) \\ \hline MS-MARCO & 0.228 & 0.401 & 0.388 & 0.407 & 0.408 & - & 0.419 & 0.421 \\ \hline SciFact & 0.665 & 0.671 & 0.502 & 0.677 & 0.644 & 0.674 & 0.709 & **0.720\({}^{\dagger}\)** \\ NFCorpus & 0.325 & 0.305 & 0.244 & 0.328 & 0.319 & 0.345 & 0.355 & **0.357\({}^{\dagger}\)** \\ TREC-COVID & 0.656 & 0.677 & 0.676 & 0.596 & 0.619 & 0.700 & 0.789 & **0.795\({}^{\dagger}\)** \\ SCIDOCS & 0.158 & 0.145 & 0.124 & 0.165 & 0.143 & **0.169** & 0.160 & 0.161 \\ DBPedia & 0.313 & 0.392 & 0.284 & **0.413** & 0.328 & 0.384 & 0.391 & 0.391 \\ CQADupStack & 0.299 & 0.350 & 0.297 & 0.345 & 0.347 & 0.357 & 0.370 & **0.374\({}^{\dagger}\)** \\ HotpotQA & 0.603 & 0.593 & 0.462 & **0.638** & 0.534 & 0.582 & 0.616 & 0.610 \\ ArguAna & 0.414 & 0.233 & 0.418 & 0.446 & 0.493 & **0.557** & 0.493 & 0.490 \\ Climate-FEVER & 0.213 & 0.184 & 0.206 & **0.237** & 0.175 & 0.235 & 0.211 & 0.220\({}^{\dagger}\) \\ FEVER & 0.753 & 0.771 & 0.680 & 0.758 & 0.669 & 0.759 & 0.751 & **0.760\({}^{\dagger}\)** \\ FiQA-2018 & 0.236 & 0.317 & 0.296 & 0.329 & 0.308 & **0.344** & 0.307 & 0.301 \\ Touche-2020 & 0.367 & 0.202 & **0.315** & 0.230 & 0.182 & 0.255 & 0.238 & 0.235 \\ Quora & 0.789 & 0.854 & 0.856 & 0.865 & 0.830 & 0.836 & 0.867 & **0.870\({}^{\dagger}\)** \\ NQ & 0.329 & 0.524 & 0.442 & 0.498 & 0.358 & 0.483 & 0.505 & **0.506** \\ Avg w/o MS-MARCO & 0.437 & 0.444 & 0.414 & 0.466 & 0.425 & 0.477 & 0.483 & **0.485** \\ \hline \hline \end{tabular}
\end{table}
Table 3: nDGC@10 performance on BEIR. **Gold** indicates the best dense retrieval performance. \(\dagger\): results with significant performance improvement with p-value \(\leq 0.05\) compared with COCO-DR. Dense retrieval models are trained in the domain adaptation setting that unsupervised target domain data can be obtained.



query and masking the signals of other units. Our method makes the representation of the passage more suitable for matching, which is the domain-invariant feature for generalization.

## 6 Conclusion

In this paper, we propose an effective method called BERM to improve the generalization ability of dense retrieval without target domain data and additional modules. The basic idea of BERM is learning the domain-invariant feature, that is, matching signal. To achieve it, we introduce a novel concept of dense retrieval to represent the matching information between two texts, the matching representation. Further, we propose two requirements for matching and text representations as the constraint in the training of dense retrieval to enhance the ability to extract essential matching information from the passage according to different queries under the premise of balanced expression of the text. The two requirements unlock the ability of dense retrieval to capture matching signal without additional interaction. Experimental results show that BERM is a flexible method that can be combined with different dense retrieval training methods without inference overhead to improve the out-of-domain generalization ability. In domain adaptation setting, our method is also effective and performs better than baselines.

### Limitations

In this paper, we propose a novel concept of dense retrieval, the matching representation. Based on this, we introduce a novel generalizable dense retrieval training method via training the balanced and extractable representation for matching (BERM). Despite the strong performance of our method in improving the generalization ability of dense retrieval models, more theoretical proof needs to be researched to gain the deeper understanding of generalization improvement. Especially for matching representation, more theoretical analysis and implementation will be discussed in future work. We believe that the deeper study of matching representation will promote the development of dense retrieval, because it not only alleviates the problem that query and passage cannot interact in depth during training, but also describes the essence of retrieval task.

## Ethics Statement

Our work innovatively proposes the concept of matching representation in dense retrieval and designs a generalization improvement strategy that can be flexibly combined with different dense retrieval training methods. Our work has important implications for improving the performance of neural information retrieval models. We declare that our work complies with the ACL Ethics Policy.2
Footnote 2: [https://www.aclweb.org/portal/content/acl-code-ethics](https://www.aclweb.org/portal/content/acl-code-ethics)

## Acknowledgements

This work was supported by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103704), the National Natural Science Foundation of China (NSFC) under Grants No. 62276248, and the Youth Innovation Promotion Association CAS under Grants No. 2023111.

## References

* Bondarenko et al. (2020) Alexander Bondarenko, Maik Frobe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, and Matthias Hagen. 2020. Overview of Touche 2020: Argument Retrieval. In Working Notes Papers of the CLEF 2020 Evaluation Labs, Vol. 2696 of _CEUR Workshop Proceedings_.
* Boteva et al. (2016) Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In _European Conference on Information Retrieval_, pages 716-722. Springer.
* Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In _Proceedings of the Conference on ACL 2017_, pages 1870-1879. Association for Computational Linguistics.
* Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In _Proceedings of the 2020 Conference on ACL_, pages 2270-2282, Online. Association for Computational Linguistics.
* Coutinho et al. (2015)

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{DPR} & \multicolumn{2}{c}{KD} & ANCE \\ Axis & X & Y & X & Y & X & Y \\ \hline Baseline & 8.1 & 6.5 & 19.5 & 18.5 & 27.2 & 25.9 \\ BERM & 248.5 & 409.5 & 252.3 & 410.9 & 260.4 & 425.4 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Dispersion of T-SNE result of representations of units in a passage (measured by the variance of the coordinates on x-axis and y-axis).

 

Thomas Diggelmann, Jordan Boyd-Graber, Jannis Julian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A dataset for verification of real-world climate claims. _arXiv preprint arXiv:2012.00614_.
* Formal et al. (2022) Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stephane Clinchant. 2022. From distillation to hard negative sampling: Making sparse neural IR models more effective. In _Proceedings of the 2021 Conference on SIGIR_, pages 2353-2359. ACM.
* Formal et al. (2021) Thibault Formal, Benjamin Piwowarski, and Stephane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '21, page 2288-2292, New York, NY, USA. Association for Computing Machinery.
* Gao et al. (2021) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3030-3042, Online. Association for Computational Linguistics.
* Girshick et al. (2014) Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In _2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014_, pages 580-587. IEEE Computer Society.
* Hasibi et al. (2017) Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. DBpedia-Entity v2: A test collection for entity search. In _Proceedings of the 2017 Conference on SIGIR_, SIGIR '17, page 1265-1268, New York, NY, USA. Association for Computing Machinery.
* Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. _CoRR_, abs/1606.08415.
* Hofstatter et al. (2021) Sebastian Hofstatter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In _Proceedings of the 2021 Conference on SIGIR_, pages 113-122. ACM.
* Hong et al. (2022) Wu Hong, Zhuosheng Zhang, Jinyuan Wang, and Hai Zhao. 2022. Sentence-aware contrastive learning for open-domain passage retrieval. In _Proceedings of the 2022 Conference on ACL_, pages 1062-1074. Association for Computational Linguistics.
* Hoogeveen et al. (2015) Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. CQADupStack: A benchmark data set for community question-answering research. In _Proceedings of the 20th Australasian Document Computing Symposium_, ADCS '15, New York, NY, USA. Association for Computing Machinery.
* Huebscher et al. (2022) Michelle Chen Huebscher, Christian Buck, Maximillion Ciaramita, and Sascha Rothe. 2022. Zero-shot retrieval with search agents and hybrid environments. _CoRR_, abs/2209.15469.
* Ioffe (2010) Sergey Ioffe. 2010. Improved consistent sampling, weighted minhash and L1 sketching. In _Proceedings of the 2010 Conference on ICDM_, pages 246-255. IEEE Computer Society.
* Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastiat Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. _CoRR_, abs/2112.09118.
* Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Herve Jegou. 2021. Billion-scale similarity search with gpus. _IEEE Trans. Big Data_, 7(3):535-547.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on EMNLP_, pages 6769-6781. Association for Computational Linguistics.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreti, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* LeCun et al. (1998) Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. _Proc. IEEE_, 86(11):2278-2324.
* Lin et al. (2021) Sheng-Chiieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In _Proceedings of the 6th Workshop on Representation Learning for NLP, RepLANLP@ACL-IJCNLP 2021, Online, August 6, 2021_, pages 163-173. Association for Computational Linguistics.
* Ma et al. (2021) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith B. Hall, and Ryan T. McDonald. 2021. Zero-shot neural passage retrieval via domain-targeted synthetic question generation. In _Proceedings of the 2021 Conference on EACL_, pages 1075-1088. Association for Computational Linguistics.
* Maia et al. (2018) Macedo Maia, Siegfried Handschuh, Andre Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. WWW'18 open challenge: Financial opinion mining and question answering. In _Companion Proceedings of the The Web Conference 

2018_, WWW '18, page 1941-1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to match using local and distributed representations of text for web search. In _Proceedings of the 2017 Conference on WWW_, pages 1291-1299. ACM.
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In _Proceedings of the 2016 Conference on NIPS_, volume 1773 of _CEUR Workshop Proceedings_. CEUR-WS.org.
* Ni et al. (2021) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021. Large dual encoders are generalizable retrievers.
* Pang et al. (2016) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 30.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Proceedings of the 2019 Conference on NeurIPS_, pages 8024-8035.
* Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In _Proceedings of the 2021 Conference on NAACL_, pages 5835-5847. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67.
* Ren et al. (2022) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2022. A thorough examination on zero-shot dense retrieval. _CoRR_, abs/2204.12755.
* Robertson et al. (1995) Stephen E. Robertson, Steve Walker, and Micheline Hancock-Beaulieu. 1995. Large test collection experiments on an operational, interactive system: Okapi at TREC. _Inf. Process. Manag._, 31(3):345-360.
* Tam et al. (2022) Weng Lam Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Xingjian Zhang, Yuxiao Dong, Jiahua Liu, Maodi Hu, and Jie Tang. 2022. Parameter-efficient prompt tuning makes generalized and calibrated neural text retrievers.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In _Proceedings of the 2021 Conference on NeurIPS Datasets and Benchmarks_.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Voorhees et al. (2021) Ellen Voorhees, Tasmeeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. TREC-COVID: Constructing a pandemic information retrieval test collection. _SIGIR Forum_, 54(1).
* Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In _Proceedings of the 2018 Conference on ACL_, pages 241-251, Melbourne, Australia. Association for Computational Linguistics.
* Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeeine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In _Proceedings of the 2020 Conference on EMNLP_, pages 7534-7550, Online. Association for Computational Linguistics.
* Wang et al. (2022) Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In _Proceedings of the 2022 Conference on ACL_, pages 2345-2360, Seattle, United States. Association for Computational Linguistics.
* Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2020. Ccnet: Extracting high quality monolingual datasets from web crawl data. In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_, pages 4003-4012. European Language Resources Association.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shelifer, Patrick von Platen, Clara Ma, Yacine Jenrite, Julien Pu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on EMNLP_, pages 38-45. Association for Computational Linguistics 

Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. 2022. Zero-shot dense retrieval with momentum adversarial domain invariant representations. In _Proceedings of the 2022 Conference on ACL Findings_, pages 4008-4020. Association for Computational Linguistics.
* Xiong et al. (2021a) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021a. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _Proceedings of the 2021 Conference on ICLR_. OpenReview.net.
* Xiong et al. (2021b) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021b. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _Proceedings of the 2021 Conference on ICLR_. OpenReview.net.
* Xu et al. (2022a) Shicheng Xu, Liang Pang, Huawei Shen, and Xueqi Cheng. 2022a. Match-prompt: Improving multi-task generalization ability for neural text matching via prompt learning. In _Proceedings of the 2022 Conference on CIKM_, CIKM '22, page 2290-2300, New York, NY, USA. Association for Computing Machinery.
* Xu et al. (2022b) Shicheng Xu, Liang Pang, Huawei Shen, and Xueqi Cheng. 2022b. Nir-prompt: A multi-task generalized neural information retrieval training framework.
* Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In _Proceedings of the 2018 Conference on EMNLP_, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
* Ye et al. (2021) Haotiian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. 2021. Towards a theoretical framework of out-of-distribution generalization. In _Proceedings of the 2021 Conference on NeurIPS_, pages 23519-23531.
* Yu et al. (2022) Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. COCO-DR: combating distribution shifts in zero-shot dense retrieval with contrastive and distributionally robust learning. _CoRR_, abs/2210.15212.
* Zhan et al. (2021) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. In _Proceedings of the 2021 Conference on SIGIR_, pages 1503-1512. ACM.
* Zhang et al. (2022) Shunyu Zhang, Yaobo Liang, Ming Gong, Davin Jiang, and Nan Duan. 2022. Multi-view document representation learning for open-domain dense retrieval. In _Proceedings of the 2022 Conference on ACL_, pages 5990-6000. Association for Computational Linguistics.

## Appendix A Datasets

In our experiment, source domain datsaet used as training data is MS-MARCO and target domain datasets used as testing data are collected from BEIR Thakur et al. (2021), which is a a heterogeneous benchmark to evaluate the generalization ability of retrieval models. Details of the datasets are shown in Table 7. In addition, we also introduce OAG-QA Tam et al. (2022), which is a fine-grained question-answering retrieval dataset consisting of different topics. We select datasets of different topics from 20 disciplines as the testing data to evaluate the generalization ability to different topics with different word distribution. Details of OAG-QA are shown in Table 8.

## Appendix B Baselines

We introduce the baselines in the main experiment and the domain adaptation experiment respectively.

### Baselines for Main Experiment

In the main experiment, our method is combined with different mainstream dense retrieval training methods to improve its generalization. We consider three training methods including vanilla (DPR Karpukhin et al. (2020)), knowledge distillation (KD) and hard negatives mining ANCE Xiong et al. (2021a)).

* **DPR** trains the dense retrieval model via in-batch negative sampling. Different from Karpukhin et al. (2020), we train DPR on MS-MARCO to achieve a fair comparison.
* **KD** trains the dense retrieval model under the guidance of the soft labels provided by the teacher model. In the experiment, we use a cross-encoder model trained on MS-MARCO as the teacher model.
* **ANCE** trains the dense retrieval model with hard negatives updated in parallel as described in Xiong et al. (2021a).

### Baselines for Domain Adaptation

* **MoDIR** uses the data from source and target domains for adversarial training to perform unsupervised domain adaptation.
* **Contirever** performs unsupervised pretraining on Wikipedia and CC-Net Wenzek et al. (2020) 

* **GenQ** uses T5 Raffel et al. (2020) generates 5 queries for each passage in target domain and fine-tunes TAS-B Hofstatter et al. (2021) on this data.
* **GPL** improves the domain adaptation performance based on GenQ. In addition to generated queries, GPL uses cross-encoder to provide the pseudo-label. GPL fine-tunes multiple backbones on the generated queries and pseudo-labels and we report the best performance that is fine-tuned on TAS-B.
* **COCO-DR** performs unsupervised pretraining on target domain and introduces distributional robust optimization.

## Appendix C Domain-invariant Representation

Visualized results of T-SNE of representations of source and target (SCIDOCS, TREC-COVID, NFCorpus and DBpedia) domains encoded by DPR and DPR+BERM respectively are shown in Figure 6.

\begin{table}
\begin{tabular}{l c l r r r r} \hline \hline Task & Domain & Dataset & \multicolumn{3}{c}{Test} & \multicolumn{3}{c}{Avg. Word Lengths} \\ \cline{4-7}  & & & \#Query & \#Corpus & Query & Document \\ \hline Passage-Retrieval & Misc. & MS-MARCO Nguyen et al. (2016) & 6,980 & 8,841,823 & 5.96 & 55.98 \\ \hline \hline Bio-Medical & Bio-Medical & TREC-COVID Voorhees et al. (2021) & 50 & 171,332 & 10.60 & 160.77 \\ Information Retrieval & Bio-Medical & NFCorpus Boteva et al. (2016) & 323 & 3,633 & 3.30 & 232.26 \\ \hline Open-domain & Wikipedia & NO Kwiatkowski et al. (2019) & 3,452 & 2,681,468 & 9.16 & 78.88 \\ Question & Wikipedia & HotpotQA Yang et al. (2018) & 7,405 & 5,233,329 & 17.61 & 46.30 \\ Answering & Finance & FiQA-2018 Maía et al. (2018) & 648 & 57,638 & 10.77 & 132.32 \\ \hline Argument & Misc. & ArguAna Wachsmutth et al. (2018) & 1,406 & 8,674 & 192.98 & 166.80 \\ Retrieval & Misc. & Touche-2020 Bondarenko et al. (2020) & 49 & 382,545 & 6.55 & 292.37 \\ \hline Duplicate-Question & StackEx. & COADupStack Hoogeveen et al. (2015) & 13,145 & 457,199 & 8.59 & 129.09 \\ Retrieval & Quora & Quora Thakur et al. (2021) & 10,000 & 522,931 & 9.53 & 11.44 \\ \hline Entity-Retrieval & Wikipedia & DBPedia Hasibi et al. (2017) & 400 & 4,635,922 & 5.39 & 49.68 \\ Citation-Prediction & Scientific & SCIDOCS Cohan et al. (2020) & 1,000 & 25,657 & 9.38 & 176.19 \\ \hline Fact Checking & Wikipedia & FEVER Thorne et al. (2018) & 6,666 & 5,416,568 & 8.13 & 84.76 \\  & Wikipedia & Climate-FEVER Diggelmann et al. (2020) & 1,535 & 5,416,568 & 20.13 & 84.76 \\  & Scientific & SciFact Waladen et al. (2020) & 300 & 5,183 & 12.37 & 213.63 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Details of the datasets in BEIR, the table is collected from Thakur et al. (2021).

\begin{table}
\begin{tabular}{l l l l} \hline \hline Discipline & Topic & \#Query \#Corpus \\ \hline Geometry & Geometry & 230 & 10,000 \\ Statistics & Mathematical Statistics & 144 & 10,000 \\ Algebra & Polynomial & 280 & 10,000 \\ Calculus & Calculus & 242 & 10,000 \\ Number theory & Number theory & 274 & 10,000 \\ Linear algebra & Matrix & 130 & 10,000 \\ Astrophysics & Black hole & 160 & 10,000 \\ Physics & Classical mechanics & 115 & 10,000 \\ Chemistry & Physical chemistry & 190 & 10,000 \\ Biochemistry & Biochemistry & 129 & 10,000 \\ Health Care & Health care & 288 & 10,000 \\ Natural Science & Evolutionary biology & 471 & 10,000 \\ Psycology & Cognitive neuroscience & 348 & 10,000 \\ Algorithm & Algorithm & 386 & 10,000 \\ Neural Network & Neural network & 590 & 10,000 \\ Data Mining & Data mining & 131 & 10,000 \\ Computer Graphics & Computer graphics images & 68 & 10,000 \\ Deep Learning & Optimization & 238 & 10,000 \\ Machine Learning & Linear regression & 244 & 10,000 \\ Economics & Economics & 238 & 10,000 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Details of QAG-QA.

 

* [26] M. C. Bader, J. S. Hubert, and A. K. Jain, "A fast algorithm for solving the problem of linear programming problems in a multi-agent system with applications to distributed systems," _IEEE Transactions on Automatic Control_**10**(4):537-589, 2000.
* [27] D. E. Kingma and P. Ba, "Adam: A method for stochastic optimization," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'01)_, pp. 1-10.
* [28] R. L. Bishop, "The alternating 



**ACL 2023 Responsible NLP Checklist+**

A For every submission: A1. Did you describe the limitations of your work? After the Section 6 Conclusion A2. Did you discuss any potential risks of your work? Ethics Statement A3. Do the abstract and introduction summarize the paper's main claims? Section 1 A4. Have you used AI writing assistants when working on this paper? Left blank. B Did you use or create scientific artifacts? Section 5 B1. Did you cite the creators of artifacts you used? Section 5.1 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 5.1 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5.1 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendix (only domain) B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C Did you run computational experiments? Section 5 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.1

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _Section 5_ C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _Section 5_ C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _Section 5_ D **D****Did you use human annotators (e.g., crowdworkers) or research with human participants?**

_Left blank._

* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _No response._

* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _No response._

* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _No response._

* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _No response._

* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _No response._