BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval
Year: 2023

Abstract:
Dense retrieval has shown promise in the firststage retrieval process when trained on indomain labeled datasets. However, previous studies have found that dense retrieval is hard to generalize to unseen domains due to its weak modeling of domain-invariant and interpretable feature (i.e., matching signal between two texts, which is the essence of information retrieval). In this paper, we propose a novel method to improve the generalization of dense retrieval via capturing matching signal called BERM. Fully fine-grained expression and query-oriented saliency are two properties of the matching signal. Thus, in BERM, a single passage is segmented into multiple units and two unit-level requirements are proposed for representation as the constraint in training to obtain the effective matching signal. One is semantic unit balance and the other is essential matching unit extractability. Unit-level view and balanced semantics make representation express the text in a fine-grained manner. Essential matching unit extractability makes passage representation sensitive to the given query to extract the pure matching information from the passage containing complex context. Experiments on BEIR show that our method can be effectively combined with different dense retrieval training methods (vanilla, hard negatives mining and knowledge distillation) to improve its generalization ability without any additional inference overhead and target domain data.

heading: 1 Introduction
text: Dense retrieval encodes the texts to dense embeddings and efficiently gets the target texts via approximate nearest neighbor search (Johnson et al., 2021). Compared with the traditional word-to-word exact matching methods such as BM25 (Robertson et al., 1995), dense retrieval can capture the relevance at the semantic level of two
∗Corresponding authors
texts. Because of the excellent performance in efficiency and effectiveness, dense retrieval has been widely used in first-stage retrieval that efficiently recalls candidate documents from the large corpus (Karpukhin et al., 2020; Xiong et al., 2021a).
However, recent studies show that the excellent performance of dense retrieval relies on the training on large in-domain datasets. When the trained dense retrieval models are applied to the domains that are inconsistent with the training datasets (i.e., zero-shot setting), the performance of the models drops seriously (Ren et al., 2022; Thakur et al., 2021). The poor generalization limits the application scenarios of dense retrieval because it is common that not enough training samples can be obtained in some domains such as medicine, biology and law that have restrictions on data privacy or require professional knowledge to annotate.
In this work, we point out that according to outof-domain generalization learning theory (Ye et al., 2021), making the model capture domain-invariant feature (i.e., essence of tasks) is effective in improving generalization ability. As for dense retrieval, matching signal between query and passage is the important domain-invariant feature and reflects the essence of information retrieval (IR). For example, MoDIR (Xin et al., 2022) shows that representation from the interaction-based cross-encoder
6620
(more fine-grained description for matching) is much more domain-invariant than it from dense retrieval. Match-Prompt (Xu et al., 2022a), NIRPrompt (Xu et al., 2022b) and MatchPyramid (Pang et al., 2016) point out the positive significance of matching signals for various IR tasks. The challenge of making dense retrieval model learn to capture matching signal is that in many IR tasks such as open-domain question answering (Chen et al., 2017) and document retrieval (Mitra et al., 2017), the content that matches the query is usually only a unit of the text. The description of matching signal needs to distinguish the matching and not matching information in the text and estimate the overall relevance. This requires the retrieval model to be able to evenly express each unit in the text and dynamically extract matching units through the interaction of the two text representations. However, the requirement on efficiency in first-stage retrieval makes dense retrieval only estimate relevance via vector similarity such as dot product and cosine. Previous training methods based on this architecture lack the above capability because of the coarse-grained training objective and interaction.
In this paper, we propose a novel method called BERM to capture the matching signal between query and passage, which is the domain-invariant feature, to improve the generalization ability of dense retrieval during the training on the single source domain without using the target domain data and other additional modules. First, we introduce a novel concept in dense retrieval, the matching representation. Matching representation is determined by the text representations (output of text encoder) of query and passage, which can reflect the matching information of query and passage. We propose that in the training of dense retrieval models, in addition to using contrastive loss (Xiong et al., 2021b) to optimize the text representation, the information of the matching representation can be used as a constraint to assist the optimization. Based on this, we divide the single passage into multiple units (each sentence is a unit) and propose two requirements on the generalizable dense retrieval models as the constraint in training (shown in Figure 1). One is semantic unit balance of text representation (R1). The other is essential matching unit extractability of matching representation (R2). These two requirements can be integrated into different dense retrieval training methods and address the challenge mentioned above. R1 means
the semantics of units in a passage are implicitly aggregated to its text representation and the text representation should evenly and comprehensively express the semantics of each unit. R2 means that the combination of text representations of query and passage (i.e., matching representation) should extract the information of the matching (i.e, the text chunk in the passage that matches the query and we call it essential matching unit) while reducing the overfitting of domain biases. This reflects the ability of the dense retrieval model to determine and score the information that really matches the query in a passage containing complex context, which is the essence of the dense retrieval and domaininvariant. R1 and R2 achieve that on the premise that the text representation expresses each unit in a balanced manner, to make essential matching units for different queries be extracted, the semantics of units tend to be orthogonal to each other. In this way, in dot product between representations of query and passage, the semantics of essential matching unit are preserved, while the other units are masked, which is suitable for matching.
Experiments on the standard zero-shot retrieval benchmark (BEIR) show that our method can be effectively combined with different dense retrieval training methods (vanilla, hard negatives mining, and knowledge distillation) to improve the generalization ability without any additional modules, inference overhead, and target domain data. Even in domain adaptation, our method is also effective and performs better than baselines. Code is released at https://github.com/xsc1234/BERM.

heading: 2 Related Work
text: Dense retrieval estimates the relevance via representations of two texts. DPR (Karpukhin et al., 2020) combines dense retrieval with pre-trained models for open-domain question answering (Chen et al., 2017). Besides, some methods focus on obtaining more valuable negatives (Qu et al., 2021; Xiong et al., 2021a; Zhan et al., 2021). Some methods use a more powerful reranker for knowledge distillation (Hofstätter et al., 2021; Lin et al., 2021). Recently, the generalization of dense retrieval has received attention. (Ren et al., 2022) performs the examination of the generalization of dense retrieval. BEIR (Thakur et al., 2021) is proposed as the benchmark to evaluate the zero-shot ability of information retrieval models. MoDIR (Xin et al., 2022) uses the data from source and target domains for ad-
versarial training to perform unsupervised domain adaptation. GenQ (Ma et al., 2021) and GPL (Wang et al., 2022) generate queries and pseudo labels for domain adaptation. Contriever (Izacard et al., 2021) uses contrastive pre-training on large corpus (Wikipedia and CC-Net (Wenzek et al., 2020)). COCO-DR (Yu et al., 2022) performs unsupervised pre-training on target domain and introduces distributional robust optimization. GTR (Ni et al., 2021) scales up the model size to improve the generalization. (Huebscher et al., 2022; Formal et al., 2022) introduce sparse retrieval to achieve better generalization.
Improvement of generalization of dense retrieval in previous studies comes from the adaptation of the target domain, knowledge from large pretraining corpus, and assistance of sparse retrieval but not dense retrieval itself. They need to obtain the target domain data in the training or increase the complexity of the system. In this paper, we introduce a novel method to improve the generalization of dense retrieval without target domain data and additional modules via learning the generalizable representation for matching.
One thing must be emphasized that the methods of multi-view dense retrieval (Zhang et al., 2022; Hong et al., 2022) also divide a passage into multiple units, but our method is essentially a completely different method. Multi-view dense retrieval uses multiple representations to fully express a passage from multiple views, which focuses on in-domain retrieval. Our method uses multiple units to make the model learn to extract essential matching unit from the passage containing complex context, which is domain-invariant for generalization. In our method, multiple units are only used as the constraint for optimization in training and only a single representation is used in inference. Learning-based sparse retrieval such as COIL (Gao et al., 2021) and SPLADE (Formal et al., 2021) also aim to express fine-grained token-level semantics but they need multiple vectors to represent tokens in passage (COIL) or sparse-vector of vocabulary size (SPLADE) and calculates the score by tokento-token matching, which is not suitable for dense retrieval that uses single dense vector to perform representation and dot product.

heading: 3 Motivation
text: Dense retrieval is hard to generalize to unseen domains due to its weak modeling of domain-
invariant feature (i.e., matching signal between two texts, which is the essence of information retrieval). Fully fine-grained expression (P1) and query-oriented saliency (P2) are two properties of the matching signal. These two require the passage representation to be able to evenly express each unit in the text, and dynamically extract matching units according to the interaction with different queries. For example, BM25 uses one-hot to evenly express each word of the text, only scores matching words, and ignores not matching words through word-to-word exact matching of the two texts. Cross-encoder uses word embedding to represent the semantics of each token and uses attention to describe the token-to-token semantic matching between texts in a fine-grained manner.
In this paper, based on the above two properties, for the training of dense retrieval, we segment a single passage into multiple units and propose two requirements as the constraint in training so that dense retrieval can capture the stronger matching signal and produces a suitable representation for matching. One is semantic unit balance of text representation (R1), and the other is essential matching unit extractability of matching representation (R2). Under R1, text representation evenly aggregates semantics of the units in the passage to comprehensively express the passage in a fine-grained manner. Besides, R1 is the premise of R2. It is because that matching representation is composed of text representations from passage and query. Unbalanced semantic expression of different units in text representation will affect the identification of essential matching unit in matching representation because it leads to different preferences for different units. Under R2, essential matching unit for the query can be extracted from the passage and reflected in matching representation. Unlike using one-hot or word embedding to explicitly express the semantics of each unit and extract matching
information through token-to-token interaction, as shown in Figure 2, R1 makes the model implicitly aggregate the semantics of each unit into the text representation to satisfy P1, and R2 makes the semantics of units tend to be orthogonal to each other (shown in Table 6). In dot product between representations of query and passage, semantics of essential matching unit are preserved, while the other units are masked, which can satisfy P2. Our method unlocks the ability of dense retrieval to capture matching signal without additional interaction.

heading: 4 Our Method
text: This section introduces the implementation of our method (Figure 3). Our method optimizes the relationship between the representations and the units in the passage. Therefore, before training, we perform unit segmentation and annotate the essential matching unit for the datasets. Then, we design loss functions according to the requirements of R1 and R2 and combine these functions with task loss of dense retrieval (contrastive loss) for joint training.

heading: 4.1 Unit Segmentation and Annotation
text: Given each positive query-passage pair (q, ppos) in training data, we segment positive passage into multiple units U as shown in Figure 3 (a) (We use the sentence as the segmentation granularity to ensure that each unit has complete semantic information.):
ppos Segment−→ U = {u1, u2, ..., un}. (1)
For U and q, BM25 is used to compute the word-toword matching score Sbm25 between q and ui ∈ U :
Sbm25 = {bm25(q, u1), ..., bm25(q, un)}.
For the datasets for question-answering, a trained reader model is additionally introduced to compute the semantic matching score Sreader between q and ui ∈ U . Specifically, reader model computes the probability distribution A = {a1, a2, .., at} of the starting positions of the answer in ppos. ai indicates the probability that the i-th token in ppos is the starting of the answer to q. For each ui ∈ U , the semantic matching score from the reader model is:
ri = max(A[sui : dui ]), (2)
where [sui : dui ] are the indexes of tokens in ui. The hybrid matching score hi between ui and q is:
hi = bm25(q, ui) + δri,
where δ is a hyperparameter. We set δ as 0.1 to give BM25 a higher weight than the reader. It is because the word-to-word exact matching of BM25 is more domain-invariant and conducive to generalization than the semantic matching of reader (Thakur et al., 2021). Then we get matching score list H = {h1, h2, ..., hn} for U . The essential matching unit is the unit corresponding to the maximum value in H . For the pair (q, ppos), yi in label list Y = {y1, ..., yn} for essential matching unit is that if i is the index corresponding to the maximum value in H , yi = 1, otherwise, yi = 0.

heading: 4.2 Training for Generalization
text: Based on the analysis of properties of matching signal in Section 3, we propose two requirements as the constraints in the training of dense retrieval to get a generalizable representation for matching (shown in Figure 3 (b)). These two requirements enable dense retrieval to extract essential matching information under the premise of balanced expression of each unit, so as to learn domain-invariant feature (i.e., matching signal) for generalization.
Implementation of R1. The first requirement is semantic unit balance of text representation, which means that the text representation of the passage encoder can comprehensively express the semantics of each unit in a balanced manner. Given the passage ppos, the text encoder g(·; θ), output hidden states Z = g(ppos; θ). Text representation tp of ppos is the embedding of [CLS] token of Z. The embeddings E of units in ppos can be obtained from Z as the segmentation in Equ.(1):
E = {e1, e2, ..., en}, (3)
where ei is the embedding of the corresponding unit (ui) and it is the average pooling of the embeddings of tokens (Z[sui : dui ]) in the unit, where [sui : dui ] are the indexes of tokens in ui. Under the constraint of R1, the relationship between tp and E is described by the loss function as:
Lbalance = DKL[b||sim(tp,E)], (4)
where DKL[·||·] is KL-divergence loss, b = [ 1n , ..., 1 n ] is a uniform distribution with equal values and sim(tp,E) = {dot(tp, ei)|ei ∈ E} is a distribution to represent the semantic similarity between tp and ei ∈ E, dot(·, ·) is dot product.
Implementation of R2. The second requirement is essential matching unit extractability of matching representation, which means that under the premise of R1, matching representation can saliently represent the unit where the essential matching block is located in. The motivation for this design is discussed in Section 1 and 3. Given the positive querypassage pair (q, ppos), text encoder g(·; θ), and the text representations for q (tq) and ppos (tp). Matching representation m ∈ Rv (v is the dimension of representation) for q and ppos can be obtained by the combination of tq ∈ Rv and tp ∈ Rv as:
m = GELU(tq ⊙ tp),
where ⊙ is the element-wise multiplication operator, and GELU(·) is activation function (Hendrycks
and Gimpel, 2016) to introduce stochastic regularization. Under the premise of R1, tp can express the semantics of the units in ppos in a balanced manner. In addition, the semantic representation of essential matching unit is more similar to tq than other units because it really matches the query q. Based on this, the model can be trained to achieve the goal that element-wise multiplication between tq and tp can amplify similar patterns (i.e, semantic representation of essential matching unit) and mask the signals of other context units. This design can be supported by convolutional neural networks (LeCun et al., 1998) whose convolution operation can amplify similar patterns in tensors (Girshick et al., 2014). For the ppos, different q amplifies different matching units, which makes m reflect the semantics of the corresponding essential matching unit. Besides, m is obtained by elementwise multiplication between tq and tp, which is an important part of estimating the relevance of two texts because dot(tq, tp) = sum(tq ⊙ tp). Thus, the optimization of m can enable the model to obtain the ability to extract essential matching unit according to different queries when estimating relevance. In training, our method utilizes the cross-entropy loss function to optimize the semantic distance between m and each unit to identify the corresponding essential matching units. Given query-passage pair (q, ppos), the embeddings E of the units in ppos as described in Equ. (3), and the label Y for essential matching unit of (q, ppos) as described in Sec. 4.1. Loss function for R2 is:
Lextract = − n∑
i=1
yi log(dot(m, ei)), (5)
where ei ∈ E, yi ∈ Y . m is only used as the constraint in training but has important implications for inference. It is because that m is the combination of text representations (tp and tq). The optimization for m is training the text encoder to output the text representation that is suitable for matching to improve the generalization ability.
Effect of R1 and R2. Table 6 indicates that compared with previous dense retrieval methods, our method makes the semantics of units in text representation tend to be orthogonal to each other. In dot product between two texts, semantics of essential matching unit are preserved, while the other units are masked to capture matching signal.
Total Loss. In addition to Lextract and Lbalance, contrastive loss is used to train the dense retrieval
model (Karpukhin et al., 2020) as:
Lc = − exp(dot(tq, tp+))
exp(dot(tq, tp+)) + exp(dot(tq, tp−)) .
So the total loss for training in our method is:
L = Lc + αLextract + βLbalance,
where α and β are the hyperparameters.

heading: 5 Experiments
text: This section introduces the experimental setups and analyzes the results.

heading: 5.1 Experimental Setups
text: Datasets. We use MS-MARCO (Nguyen et al., 2016) as the training data (source domain) and choose the 14 publicly available datasets from BEIR 1, a heterogeneous benchmark to evaluate the generalization ability of retrieval models. In addition, we also introduce OAG-QA (Tam et al., 2022) to evaluate the topic generalization ability. Details of datasets are in Appendix A.
Baselines. Our method (BERM) aims to improve the generalization of dense retrieval without any additional modules and target domain data, and it can be combined with different dense retrieval training methods. We select three mainstream dense retrieval training methods including vanilla, hard negatives mining, and knowledge distillation as the baselines. We follow DPR (Karpukhin et al.,
1The left four are unavailable due to copyright restrictions.
2020) to perform vanilla, follow ANCE (Xiong et al., 2021a) to perform hard negatives mining and use a trained cross-encoder as the teacher model to perform knowledge distillation. We compare the change in generalization after combining BERM with these three methods to show the effectiveness of our method. Besides, as previous methods need to obtain target domain data for domain adaptation such as MoDIR (Xin et al., 2022), GenQ (Ma et al., 2021), GPL (Wang et al., 2022) and COCODR (Yu et al., 2022), we also compare our method with these methods in domain adaptation setting. Details of baselines are in Appendix B.
Implementation Details. To maintain a fair comparison, we follow (Xiong et al., 2021a) to keep all common hyperparameters (learning rate and batch size, etc.) the same as the three dense retrieval training methods in the baselines. The model is initialized by Robertabase 125M. For the hyperarameters in BERM, δ is 0.1, α is 0.1 and β is 1.0. In domain adaptation, we combine BERM with continuous contrastive pretraining (Yu et al., 2022) to perform unsupervised pre-training on BEIR and use BERM to fine-tune the model on MS-MARCO. We train the model with Pytorch (Paszke et al., 2019) and Hugging Face (Wolf et al., 2020) on 2 Tesla V100 32GB GPUs for about 72 hours.

heading: 5.2 Retrieval Performance
text: Main Results. Table 1 shows the main results on BEIR of different dense retrieval training methods. The results indicate that our method (BERM) can
be combined with three mainstream dense retrieval training methods (vanilla, knowledge distillation, and hard negatives) to improve the generalization ability without any additional modules and target domain data. For a fair comparison, we combine BERM with the baselines and ensure that their common hyperparameters are consistent. We compute the Jaccard similarity (Ioffe, 2010) between each dataset and MS-MARCO, which can reflect the domain shift between the source and target domain. Table 1 shows that our method is more effective for the datasets with lower Jaccard similarity between MS-MARCO (i.e., domain shift is more significant). This result reflects the ability of our method to capture domain-invariant feature. DPR+BERM and KD+BERM are better than KD, which shows that BERM more effectively enables dense retrieval to learn to capture matching signal than knowledge distillation from cross-encoder.
Topic Generalization. Table 2 shows the generalization performance of DPR and DPR+BERM on different topics of QAG-QA. Topic generalization is important for out-of-domain generalization, which reflects the availability of dense retrieval model for topics with different word distributions. The results show that BERM can significantly improve cross-topic generalization of dense retrieval.
Domain Adaptation. Table 3 shows that BERM
achieves the best performance in domain adaptation compared with previous baselines. Specifically, BERM achieves the best average out-of-domain adaptation and in-domain performance. Besides, it gets the best dense retrieval results on seven datasets of BEIR, which is the most of all methods. Our method not only learns the word distribution of the target domain, but also learns the representation suitable for matching for the documents in the target corpus during domain adaptation.

heading: 5.3 Ablation Study
text: Influence of Loss Functions. Table 4 shows the ablation study on the loss functions constrained by R1 and R2 via average performance on BEIR. The results indicate that without Lbalance, Lextract can not improve the generalization, which supports our intuition in Section 3 that only based on the balanced semantic expression of each unit in the text representation, the matching representation is meaningful for extracting the essential semantic unit. This experiment shows that the generalization can be improved significantly when the model is constrained by both R1 and R2.
Influence of Hyperparameters. Figure 4 shows the average nDCG@10 performance on BEIR with different α and β that are used to tune the weights of different loss functions in training. When α is 0.1 and β is 1.0, our method can achieve the best performance. When α and β are too big, they will interfere with the optimization of the contrastive loss leading to performance degradation.

heading: 5.4 Model Analysis
text: Domain-invariant Representation. Figure 5 shows that our method is effective in capturing the domain-invariant feature of the representation. We utilize T-SNE to visualize the representations of source and target (SciFact) domains encoded by DPR and DPR+BERM respectively. The results indicate that representations of the two domains encoded by DPR are more separable. After combining our method, the two domains become more difficult to separate, which indicates that our method is more invariant to represent the texts in different domains. More datasets are in Appendix C.
Evaluation of R1 and R2. Table 5 shows the effectiveness of R1 and R2. We randomly sample 100,000 query-passage pairs from the test set. For each passage p, we compute semantic similarity between text representation and each unit via sim(tp,E) = {dot(tp, ei)|ei ∈ E}. We compute the variance of sim(tp,E) and get the average of variance on the sampled set, which can
reflect the balance of text representation on expressing the semantics of units. Table 5 shows that BERM has a smaller variance (semantic unit balance of text representation) and is more accurate in identifying the essential matching unit ( essential matching unit extractability of matching representation) than baselines, which indicates the effectiveness of R1 and R2.
Relationship Between Units. Table 6 shows that our method makes units in a passage more dispersed (tend to be orthogonal), which is more conducive to determining the unit that matches the
query and masking the signals of other units. Our method makes the representation of the passage more suitable for matching, which is the domaininvariant feature for generalization.

heading: 6 Conclusion
text: In this paper, we propose an effective method called BERM to improve the generalization ability of dense retrieval without target domain data and additional modules. The basic idea of BERM is learning the domain-invariant feature, that is, matching signal. To achieve it, we introduce a novel concept of dense retrieval to represent the matching information between two texts, the matching representation. Further, we propose two requirements for matching and text representations as the constraint in the training of dense retrieval to enhance the ability to extract essential matching information from the passage according to different queries under the premise of balanced expression of the text. The two requirements unlock the ability of dense retrieval to capture matching signal without additional interaction. Experimental results show that BERM is a flexible method that can be combined with different dense retrieval training methods without inference overhead to improve the out-of-domain generalization ability. In domain adaptation setting, our method is also effective and performs better than baselines.
Limitations
In this paper, we propose a novel concept of dense retrieval, the matching representation. Based on this, we introduce a novel generalizable dense retrieval training method via training the balanced and extractable representation for matching (BERM). Despite the strong performance of our method in improving the generalization ability of dense retrieval models, more theoretical proof needs to be researched to gain the deeper understanding of generalization improvement. Especially for matching representation, more theoretical
analysis and implementation will be discussed in future work. We believe that the deeper study of matching representation will promote the development of dense retrieval, because it not only alleviates the problem that query and passage cannot interact in depth during training, but also describes the essence of retrieval task.
Ethics Statement
Our work innovatively proposes the concept of matching representation in dense retrieval and designs a generalization improvement strategy that can be flexibly combined with different dense retrieval training methods. Our work has important implications for improving the performance of neural information retrieval models. We declare that our work complies with the ACL Ethics Policy.2

heading: Acknowledgements
text: This work was supported by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103704), the National Natural Science Foundation of China (NSFC) under Grants No. 62276248, and the Youth Innovation Promotion Association CAS under Grants No. 2023111.

heading: A Datasets
text: In our experiment, source domain datsaet used as training data is MS-MARCO and target domain datasets used as testing data are collected from BEIR (Thakur et al., 2021), which is a a heterogeneous benchmark to evaluate the generalization ability of retrieval models. Detials of the datasets are shown in Table 7. In addition, we also introduce OAG-QA (Tam et al., 2022), which is a finegrained question-answering retrieval dataset consisting of different topics. We select datasets of different topics from 20 disciplines as the testing data to evaluate the generalization ability to different topics with different word distribution. Details of OAG-QA are shown in Table 8.

heading: B Baselines
text: We introduce the baselines in the main experiment and the domain adaptation experiment respectively.
B.1 Baselines for Main Experiment
In the main experiment, our method is combined with different mainstream dense retrieval training methods to improve its generalization. We consider three training methods including vanilla (DPR (Karpukhin et al., 2020)), knowledge distillation (KD) and hard negatives mining (ANCE (Xiong et al., 2021a)).
• DPR trains the dense retrieval model via in-batch negative sampling. Different from (Karpukhin et al., 2020), we train DPR on MS-MARCO to achieve a fair comparison.
• KD trains the dense retrieval model under the guidance of the soft labels provided by the teacher model. In the experiment, we use a cross-encoder model trained on MS-MARCO as the teacher model.
• ANCE trains the dense retrieval model with hard negatives updated in parallel as described in (Xiong et al., 2021a).
B.2 Baselines for Domain Adaptation
• MoDIR uses the data from source and target domains for adversarial training to perform unsupervised domain adaptation.
• Contriever performs unsupervised pretraining on Wikipedia and CC-Net (Wenzek et al., 2020).
• GenQ uses T5 (Raffel et al., 2020) generates 5 queries for each passage in target domain and fine-tunes TAS-B (Hofstätter et al., 2021) on this data.
• GPL improves the domain adaptation performance based on GenQ. In addition to generated queries, GPL uses cross-encoder to provide the pseudo-label. GPL fine-tunes multiple backbones on the generated queries and pseudo-labels and we report the best performance that is fine-tuned on TAS-B.
• COCO-DR performs unsupervised pre-
training on target domain and introduces distributional robust optimization.

heading: C Domain-invariant Representation
text: Visualized results of T-SNE of representations of source and target (SCIDOCS, TREC-COVID, NFCorpus and DBpedia) domains encoded by DPR and DPR+BERM respectively are shown in Figure 6.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: After the Section 6 Conclusion

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 5.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5.1

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix (only domain)
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Especially for matching representation, more theoretical
analysis and implementation will be discussed in future work. We believe that the deeper study of matching representation will promote the development of dense retrieval, because it not only alleviates the problem that query and passage cannot interact in depth during training, but also describes the essence of retrieval task