Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering
Year: 2023

Abstract:
Given its effectiveness on knowledge-intensive natural language processing tasks, dense retrieval models have become increasingly popular. Specifically, the de-facto architecture for open-domain question answering uses two isomorphic encoders that are initialized from the same pretrained model but separately parameterized for questions and passages. This biencoder architecture is parameter-inefficient in that there is no parameter sharing between encoders. Further, recent studies show that such dense retrievers underperform BM25 in various settings. We thus propose a new architecture, Task-Aware Specialization for dEnse Retrieval (TASER), which enables parameter sharing by interleaving shared and specialized blocks in a single encoder. Our experiments on five question answering datasets show that TASER can achieve superior accuracy, surpassing BM25, while using about 60% of the parameters as bi-encoder dense retrievers. In out-of-domain evaluations, TASER is also empirically more robust than bi-encoder dense retrievers. Our code is available at https: //github.com/microsoft/taser.

heading: 1 Introduction
text: Empowered by learnable neural representations built upon pretrained language models, the dense retrieval framework has become increasingly popular for fetching external knowledge in various natural language processing tasks (Lee et al., 2019; Guu et al., 2020; Lewis et al., 2020). For opendomain question answering (ODQA), the de-facto dense retriever is the bi-encoder architecture (Lee et al., 2019; Karpukhin et al., 2020), consisting of a question encoder and a passage encoder. Typically, the two encoders are isomorphic but separately parameterized, as they are initialized from the same pretrained model and then fine-tuned on the task.
Despite of its popularity, this bi-encoder architecture with fully decoupled parameterization has
some open issues. First, from the efficiency perspective, the bi-encoder parameterization apparently results in scaling bottleneck for both training and inference. Second, empirical results from recent studies show that such bi-encoder dense retrievers underperform its sparse counterpart BM25 (Robertson and Walker, 1994) in various settings. For example, both Lee et al. (2019) and Karpukhin et al. (2020) suggest the inferior performance on SQuAD (Rajpurkar et al., 2016) is partially due to the high lexical overlap between questions and passages, which gives BM25 a clear advantage. Sciavolino et al. (2021) also find that bi-encoder dense retrievers are more sensitive to distribution shift than BM25, resulting in poor generalization on questions with rare entities.
In this paper, we develop Task-Aware Specialization for dEnse Retrieval, TASER, as a more parameter-efficient and robust architecture. Instead of using two isomorphic and fully decoupled Transformer (Vaswani et al., 2017) encoders, TASER interleaves shared encoder blocks with specialized ones in a single encoder, motivated by recent success in using Mixture-of-Experts (MoE) to scale up Transformer (Fedus et al., 2021). For the shared encoder block, the entire network is used to encode both questions and passages. For the specialized encoder block, some sub-networks are task-specific and activated only for certain encoding tasks. To choose among task-specific sub-networks, TASER uses an input-dependent routing mechanism, i.e., questions and passages are passed through separate dedicated sub-networks.
We carry out both in-domain and out-of-domain evaluation for TASER. For the in-domain evaluation, we use five popular ODQA datasets. Our best model outperforms BM25 and existing biencoder dense retrievers, while using much less parameters. It is worth noting that TASER can effectively close the performance gap on SQuAD between dense retrievers and BM25. One interest-
1864
ing finding from our experiments is that excluding SQuAD from the multi-set training is unnecessary, which was a suggestion made by Karpukhin et al. (2020) and adopted by most follow-up work. Our out-of-domain evaluation experiments use EntityQuestions (Sciavolino et al., 2021) and BEIR (Thakur et al., 2021). Consistent improvements over the doubly parameterized bi-encoder dense retriever are observed in these zero-shot evaluations as well. Our code is available at https: //github.com/microsoft/taser.

heading: 2 Background
text: In this section, we provide necessary background about the bi-encoder architecture for dense passage retrieval which is widely used in ODQA (Lee et al., 2019; Karpukhin et al., 2020) and is the primary baseline model in our experiments.
The bi-encoder architecture consists of a question encoder and a passage encoder, both of which are usually Transformer encoders (Vaswani et al., 2017). A Transformer encoder is built up with a stack of Transformer blocks. Each block consists of a multi-head self-attention (MHA) sub-layer and a feed-forward network (FFN) sub-layer, with residual connections (He et al., 2016) and layernormalization (Ba et al., 2016) applied to both sublayers. Given an input vector h ∈ Rd, the FFN sub-layer produces an output vector as following
FFN(h) = W2max{0,W1h+ b1}+ b2, (1)
where W1 ∈ Rm×d,W2 ∈ Rd×m,b1 ∈ Rm, and b2 ∈ Rd are learnable parameters. For a sequence of N tokens, each Transformer block produces N corresponding vectors, together with a vector for the special prefix token [CLS] which can be used as the representation of the sequence. We refer readers to (Vaswani et al., 2017) for other details about Transformer. Typically the question encoder and passage encoder are initialized from a pretrained language model such as BERT (Devlin et al., 2019), but they are parameterized separately, i.e., their parameters would differ after training.
The bi-encoder model independently encodes questions and passages into d-dimension vectors, using the final output vectors for [CLS] from the corresponding encoders, denoted as q ∈ Rd and p ∈ Rd, respectively. The relevance between a question and a passage can then be measured in the vector space using dot product, i.e., sim(q,p) = qTp. During training, the model is
optimized based on a contrastive learning objective,
Lsim = − exp(sim(q,p+))∑
p′∈P∪{p+} exp(sim(q,p′)) , (2)
where p+ is the relevant (positive) passage for the given question, and P is the set of irrelevant (negative) passages. During inference, all passages are pre-converted into vectors using the passage encoder. Then, each incoming question is encoded using the question encoder, and a top-K list of most relevant passages are retrieved based on their relevance scores with respect to the question.
Although the bi-encoder dense retrieval architecture has achieved impressive results in ODQA, few work has attempted to improve its parameter efficiency. Further, compared to the spare vector space model BM25 (Robertson and Walker, 1994), such bi-encoder dense retrievers sometimes suffer from inferior generalization performance, e.g., when the training data is extremely biased (Lebret et al., 2016; Karpukhin et al., 2020) or when there is a distribution shift (Sciavolino et al., 2021). In this paper, we conjecture that the unstable generalization performance is partially related to the unnecessary number of learnable parameters in the model. Therefore, we develop a task-aware specialization architecture for dense retrieval with parameter sharing between the question and passage encoders, which turns out to improve both parameter efficiency and generalization performance.
3 Proposed Model: TASER
As shown in Fig. 1, TASER interleaves shared Transformer blocks with specialized ones. The
shared Transformer block is identical to the Transformer block used in the bi-encoder architecture, but the entire block is shared for both questions and passages. In the specialized block, we apply MoE-style task-aware specialization to the FFN sub-layer, following (Fedus et al., 2021), where the router always routes the input to a single expert FFN sub-layer. In our experiments, we use a simple yet effective routing mechanism, which uses an expert sub-layer (Q-FFN) for questions and another (P-FFN) for passages. The router determines the expert FFN sub-layer based on whether the input is a question or a passage. Other routing mechanisms are discussed in Appendix A. TASER uses one specialized Transformer block after every T shared Transformer blocks in the stack, starting with a shared one at the bottom. Our preliminary study indicates that the model performance is not sensitive to the choice of T , so we use T = 2 for experiments in this paper.
Similar to the bi-encoder architecture, TASER is trained using the contrastive learning objective Lsim defined in Equation 2. Specifically, the objective needs to use a set of negative passages P for each question. Following Xiong et al. (2020) and Qu et al. (2021), we construct P via hard negatives mining (Appendix B). Our experiments use the multi-set training paradigm, i.e., the model is trained by combining data from multiple datasets to obtain a model that works well across the board.

heading: 4 Experiments

heading: 4.1 In-Domain Evaluation
text: We carry out in-domain evaluations on five ODQA datasets: NaturalQuestions (NQ; Kwiatkowski et al., 2019a), TriviaQA (TQ; Joshi et al., 2017), WebQuestions (WQ; Berant et al., 2013), CuratedTrec (CT; Baudiš and Šedivý, 2015), and SQuAD (Rajpurkar et al., 2016). All data splits and the Wikipedia collection for retrieval used in our experiments are the same as Karpukhin et al. (2020). The top-K retrieval accuracy (R@K) is used for evaluation, which evaluates whether any gold answer string is contained in the top K retrieved passages.
Besides BERT-base, coCondenser-Wiki (Gao and Callan, 2022) is also used to initialize TASER models. We further present results of hybrid models that linearly combine the dense retrieval scores with the BM25 scores. See Appendix D for details. Evaluation results are summarized in Ta-
ble 1.1 Note that the last five models Table 1 are trained with the same hard negatives mining.
All prior work excludes SQuAD from the multiset training, as suggested by Karpukhin et al. (2020). We instead train models using all five datasets. Specifically, we observe that this would not hurt the overall performance, and it actually significantly improves the performance on SQuAD, comparing DPR(1) with DPR†.
Comparing models initialized from BERT-base, TASER⋄ significantly outperforms xMoCo (Yang et al., 2018) and is slightly better than DPR⋄, using around 60% parameters. SPAR (Chen et al., 2022) is also initialized from BERT-base, but it augments DPR with another dense lexical model trained on either Wikipedia or PAQ (Lewis et al., 2021), which doubles the model sizes (Table A3). TASER⋄ is mostly on par with SPAR-Wiki and SPAR-PAQ, except on SQuAD, but its model size is about a quarter of SPAR.
Gao and Callan (2022) has shown the coCodenser model outperforms DPR models initialized from BERT-base in the single-set training setting. Here, we show that using coCondenser-Wiki for initialization is also beneficial for TASER under the multi-set setting, especially for SQuAD where
1We also report R@100 scores in Table A2 and corresponding model sizes in Table A3.
R@20 is improved by 3.2 points. Notably, SQuAD is the only dataset among the five where DPR underperforms BM25, due to its higher lexical overlap between questions and passages. Nevertheless, TASER⋆ surpasses BM25 on all five datasets, and they are either on-par or better than state-of-theart dense-only retriever models, demonstrating its superior parameter efficiency.
Consistent with previous work, combining BM25 with dense models can further boost the performance, particularly on SQuAD. However, the improvement is more pronounced on DPR as compared to TASER⋆, indicating that TASER⋆ is able to capture more lexical overlap features. Finally, TASER⋆BM25 sets new state-of-the-art performance on all five ODQA datasets.
We also compare the computation time needed for one epoch of training and validation. The baseline DRP model takes approximately 15 minutes, whereas TASER takes 11 minutes (26% improvement), both measured using 16 V100-32G GPUs.

heading: 4.2 Out-of-Domain Evaluation
text: We use two benchmarks to evaluate the out-ofdomain generalization ability of TASER⋄ and TASER⋆ from Table 1 . EntityQuestions (EQ; Sciavolino et al., 2021) is used to measure the model sensitivity to entity distributions, as DPR is found to perform poorly on entity-centric questions containing rare entities. BEIR (Thakur et al., 2021) is used to study the model generalization ability in other genres of information retrieval tasks. Specifically, we focus on four datasets from BEIR where DPR underperforms BM25, i.e., ArguAna (AA; Wachsmuth et al., 2018), DBPedia (DBP; Hasibi et al., 2017), FEVER (FEV; Thorne et al., 2018), and HotpotQA (HQA; Yang et al., 2018). Results are summarized in Table 2. For EntityQuestions, we report R@20 scores following Sciavolino et al.
(2021).2 For BEIR datasets, nDCG@10 scores are used following Thakur et al. (2021).
On EntityQuestions, both TASER⋄ and TASER⋆
outperform the doubly parameterized DPRMulti (Karpukhin et al., 2020), with TASER⋆ being slightly better. Similar to the in-domain evaluation results, TASER can effectively reduce the performance gap between the dense retrievers and BM25. These results further support our hypothesis that more parameter sharing can improve the model robustness for dense retrievers.
On BEIR datasets, we also observe that TASER models consistently improve over DPRMulti across the board. Notably, TASER⋄ and TASER⋆ can actually match the performance of BM25 on ArguAna and DBpedia. Interestingly, coCondenser pre-training has mixed results here, i.e., TASER⋆ is only better than TASER⋄ on HotpotQA and on par or worse on other datasets.

heading: 5 Related Work
text: Recent seminal work on dense retrieval demonstrates its effectiveness using Transformer-based bi-encoder models by either continual pre-training with an inverse cloze task (Lee et al., 2019) or careful fine-tuning (Karpukhin et al., 2020). One line of follow-up work improves dense retrieval models via various continual pre-training approaches (Guu et al., 2020; Chang et al., 2020; Izacard et al., 2021; Gao and Callan, 2022; Oğuz et al., 2021). Better contrastive learning objectives are also introduced (Xiong et al., 2020; Qu et al., 2021; Yang et al., 2021). Motivated by the success of augmenting dense models with sparse models, Chen et al. (2022) combine the dense retriever with a dense lexical model that mimics sparse retrievers. All above work focus on improving the accuracy of biencoder dense retrievers, whereas our work tackles the parameter efficiency issue.
Unlike most bi-encoder dense retrievers which measure the similarity between a question and a passage using their corresponding [CLS]vectors, ColBERT (Khattab and Zaharia, 2020) develops a late-interaction paradigm and measures the similarity via a MaxSim operator that computes the maximum similarity between a token in a sequence and all tokens in the other sequence. Such architecture has shown promising results in ODQA (Khattab et al., 2021) and the BEIR benchmark (Santhanam
2The R@20 scores are averaged over all relations. More evaluation metrics are reported in Table A4.
et al., 2022). Our work instead focus on the improvement on the underlying text encoders, and the MaxSim operator introduced by ColBERT can be applied on top of TASER.
Xiong et al. (2021) use the BERT-Siamese architecture for dense retrieval, where all Transformer blocks are shared. Compared with this architecture, TASER is a more effective and general way to increase the parameter efficiency, by interleaving specialized Transformer blocks with shared ones.

heading: 6 Conclusion
text: We propose a new parameterization framework, TASER, for improving the efficiency and robustness of dense retrieval for ODQA. It interleaves shared encoder blocks with specialized ones in a single encoder where some sub-networks are task-specific. As the specialized sub-networks are sparsely activated, TASER can provide better parameter efficiency with almost no additional computation cost. Experiments show that TASER substantially outperforms existing fully supervised biencoder dense retrievers on both in-domain and out-of-domain generalization.

heading: 7 Limitations
text: In this section, we point out several limitations in this work.
First, our in-domain evaluation experiments focus on passage retrieval for ODQA. While the dense retriever is mostly successful in ODQA, it can be also used in other types of retrieval tasks which may have different input and output format. For example, the KILT benchmark (Petroni et al., 2021) provides several knowledge-intensive tasks other than ODQA. The performance of TASER models trained on such retrieval tasks remain unknown.
Second, compared with traditional sparse vector models like TF-IDF and BM25, the cost of training is an inherent issue of dense retrievers. Although TASER significantly reduce the number of model parameters, the training cost is still high.
Third, in our experiments, we show that the learned routing does not outperform the deterministic routing. This may suggest a better architecture and/or training algorithms for learned routing is needed to fully unleash the power of MoE.
Last, as observed in §4.2, there is still a gap between TASER and BM25 in out-of-domain evaluation. Therefore, how to close this gap will remain
a critical topic for future work on dense retrievers.

heading: A More Routing Mechanisms
text: In the paper, only input-dependent routing is considered. Here, we provide a more comprehensive study of routing mechanisms. In particular, we introduce three routing mechanisms: the deterministic routing (Det-R) which is used in our main experiments, the sequence-based routing (Seq-R), and the token-based routing (Tok-R). Both Seq-R and Tok-R are learned jointly with the task-specific objective.
Specifically, Det-R is the input-dependent routing studied in the main paper where two expert FFN sub-layers are needed for ODQA retrieval, one for questions and one for passages. In this case, the router determines the expert FFN sub-layer based on whether the input is a question or a passage.
For Seq-R and Tok-R, the router uses a parameterized routing function
R(u) = GumbelSoftmax(Au+ c), (3)
where GumbelSoftmax (Jang et al., 2016) outputs a I-dimensional one-hot vector based on the linear projection parameterized by A ∈ Rd×I and c ∈ RI , I is the number of expert FFN sub-layers in the specialized Transformer block, and u ∈ Rd is the input of the routing function. Here, the routing function is jointly learned with all other parameters using the discrete reparameterization trick. For Seq-R, routing is performed at the sequence level, and all tokens in a sequence share the same u, which is the FFN input vector h[CLS] representing the special prefix token [CLS]. For Tok-R, the router independently routes each token, i.e., for the j-th token in the sequence, u is set to the corresponding FFN input vector hj .
For Seq-R and Tok-R, to avoid routing all inputs to the same expert FFN sub-layer, we further apply the entropic regularization
Lent = − I∑
i=1
P (i) logP (i). (4)
where P (i) = Softmax(Ah+c)i is the probability of the i-th expert FFN sub-layer being selected. Hence, the joint training objective is
Ljoint = Lsim + βLent, (5)
where β is a scalar hyperparameter. In our work, we fix β = 0.01.
Also, all specialized Transformer blocks use the same number of expert FFN sub-layers for simplicity.
Model I # Params Dev Test
DPR - 218M - 78.4 TASERShared 1 109M 78.2 79.3 TASERDet-R 2 128M 79.2 80.7 TASERSeq-R 2 128M 79.2 80.6 TASERSeq-R 4 166M 78.4 80.1 TASERToK-R 2 128M 78.5 79.8 TASERToK-R 4 166M 78.5 79.8 DPR† - 218M - 81.3 TASERDet-R† 2 128M 82.4 83.7
Table A1: R@20 on NQ dev and test sets under the single-set training setting. I is the number of expert FFNs. The # params column shows the number of parameters in the model. † means the model is trained with hard negatives mining described in §B. The results for DPR and DPR† are reported in (Karpukhin et al., 2020) and https://tinyurl.com/yckar3f6, respectively.

heading: B Hard Negative Mining
text: Recall that in Equation 2 the objective Lsim needs to use a set of negative passages P for each question. There are several ways to construct P . In (Karpukhin et al., 2020), the best setting uses two negative passages per question: one is the top passage retrieved by BM25 which does not contain the answer but match most question tokens, and the other is chosen from the gold positive passages for other questions in the same mini-batch. Recent work shows that mining harder negative examples with iterative training can lead to better performance (Xiong et al., 2020; Qu et al., 2021). Hence, in this paper, we also train TASER with hard negatives mining. Specifically, we first train a TASER model with negative passages P1 same as Karpukhin et al. (2020). Then, we use this model to construct P2 by retrieving top-100 ranked passages for each question excluding the gold passage. In the single-set training, we combine P1 and P2 to train the final model. In the multi-set training, only use P2 is used to train the final model for efficiency consideration.
C Comparing TASER Variants
In this part, we compare different TASER variants discussed in §A by evaluating their performance on NQ under the single-set training setting. We use the bi-encoder dense passage retriever (DPR) from (Karpukhin et al., 2020) as our baseline. All models
NQ TriviaQA WebQ TREC SQuAD Model @20 @100 @20 @100 @20 @100 @20 @100 @20 @100
BM25(1) 62.9 78.3 76.4 83.2 62.4 75.5 80.7 89.9 71.1 81.8
Single-Set Training
DPR(2) 78.4 85.4 79.4 85.0 73.2 81.4 79.8 89.1 63.2 77.2 DPR-PAQ(3) 84.7 89.2 - - - - - - - - coCondenser(4) 84.3 89.0 83.2 87.3 - - - - - -
Multi-Set Training (without SQuAD)
DPR(1) 79.5 86.1 78.9 84.8 75.0 83.0 88.8 93.4 52.0 67.7 DPR(1)BM25 82.6 88.6 82.6 86.5 77.3 84.7 90.1 95.0 75.1 84.4 xMoCo(5) 82.5 86.3 80.1 85.7 78.2 84.8 89.4 94.1 55.9 70.1 SPAR-Wiki(6) 83.0 88.8 82.6 86.7 76.0 84.4 89.9 95.2 73.0 83.6 SPAR-PAQ(6) 82.7 88.6 82.5 86.9 76.3 85.2 90.3 95.4 72.9 83.7
Multi-Set Training (with SQuAD)
DPR† 80.9 86.8 79.6 85.0 74.0 83.4 88.0 94.1 63.1 77.2 DPR⋄ 82.5 88.0 81.8 86.4 77.8 84.7 91.2 95.5 67.1 79.8 DPR⋆ 83.7 88.7 82.6 86.7 78.9 85.3 91.6 95.1 68.0 80.2 TASER⋄ 83.6 88.6 82.0 86.6 77.9 85.4 91.1 95.7 69.7 81.2 TASER⋄BM25 83.8 88.6 83.3 87.1 78.7 85.7 91.6 95.8 77.2 86.0 TASER⋆ 84.9 89.2 83.4 87.1 78.9 85.4 90.8 96.0 72.9 83.4 TASER⋆BM25 85.0 89.2 84.0 87.5 79.6 85.8 92.1 96.0 78.0 87.0
Table A2: In-domain evaluation results. Test set R@20 and R100 are reported. (1): (Ma et al., 2021). (2): (Karpukhin et al., 2020). (3): (Oğuz et al., 2021) (4): (Gao and Callan, 2022). (5): (Yang et al., 2021). (6): (Chen et al., 2022). BM25: combined with BM25 scores. †: initialized from BERT-base and without hard negatives mining. ⋄: initialized from BERT-base. ⋆: initialized from coCondenser-Wiki. The last five models are trained with the same hard negatives mining.
Model Num. Parameters
DPR 218M coCodenser 218M xMoCo 218M SPAR-Wiki; SPAR-PAQ 436M DPR-PAQ 710M
TASER⋄; TASER⋆ 128M
Table A3: Number of parameters for models reported in Table A2.
including DPR are initialized from the BERT-base (Devlin et al., 2019).3 All TASER models are finetuned up to 40 epochs with Adam (Kingma and Ba, 2014) using a learning rate chosen from {3e− 5, 5e − 5}. Model selection is performed on the development set following (Karpukhin et al., 2020).
3Without further specification, we only consider the uncased version throughout the paper.
Results are summarized in Table A1.
TASERShared is a variant without any task-aware specialization, i.e., there is a single expert FFN sublayer in the specialized Transformer block and the router is a no-op. As shown in Table A1, it outperforms DPR while using only 50% parameters.
Task-aware specialization brings extra improvements, with little increase in model size. Comparing the two learned routing mechanisms, Seq-R achieves slightly better results than Tok-R, indicating specializing FFNs based on sequence-level features such as sequence types is more effective for ODQA dense retrieval. This is consistent with the positive results for Det-R, which consists of two expert FFNs specialized for questions and passages, respectively. We also find that adding more expert FFNs does not necessarily bring extra gains, and I = 2 is sufficient for NQ. Consistent with the results on DPR, the hard negatives mining described in §B can further boost TASERDet-R per-
Macro R@20 Micro R@20 Micro R@100
BM25 71.2 70.8 79.2
DPRMulti 56.7 56.6 70.1 TASER⋄ 64.7 64.3 76.2 TASER⋆ 66.7 66.2 77.9
Table A4: Out-of-domain evaluation results on EntityQuestions. We report macro R@20 scores which are used in (Sciavolino et al., 2021) as well as micro R@20 and R@100 scores which are used in (Chen et al., 2022). Results for BM25 and DPRMulti are from (Sciavolino et al., 2021) and (Chen et al., 2022).
formance by 3.0 points in test set R@20. Since Det-R achieves the best R@20, our subsequent experiments focus on this simple and effective specialization strategy. In the remainder of the paper, we drop the subscript and simply use TASER to denote models using Det-R.

heading: D Details about In-Domain Evaluations
text: All TASER models are fine-tuned up to 40 epochs with Adam (Kingma and Ba, 2014) using a learning rate chosen from {3e − 5, 5e − 5}. In our experiments, hard negatives are mined from NQ, TriviaQA and WebQ. We combine NQ and TriviaQA development sets for model selection.
We also present results of hybrid models that linearly combine the dense retrieval scores with the BM25 scores,
sim(q,p) + α · BM25(q,p). (6) We search the weight α in the range [0.5, 2.0] with an interval of 0.1 based on the combined development set mentioned above. Unlike (Ma et al., 2021), we use a single α for all five datasets instead of dataset-specified weights so that the resulting hybrid retriever still complies with the multi-set setting in a strict sense. The same normalization techniques described in (Ma et al., 2021) is used. Similar to (Karpukhin et al., 2020; Ma et al., 2021), we separately retrieve K ′ candidates from TASER and BM25, and then retain the top K based on the hybrid scores, though we use a smaller K ′ = 100.
We used 16 V100-32GB GPUs and it took 9 hours to train our models.

heading: E Dataset Licenses and Intended Use
text: All datasets used in our experiments are English datasets. The datasets used in this paper are released under the following licenses.
• NaturalQuestions (Kwiatkowski et al., 2019b): CC-BY-SA 3.0 License
• TriviaQA (Joshi et al., 2017): noncommercial research purposes only
• WebQuestions (Berant et al., 2013): CC-BY 4.0 License.
• SQuAD (Rajpurkar et al., 2016): CC-BY-SA 4.0 License
• EntityQuestions (Sciavolino et al., 2021): MIT License
• ArguAna (Wachsmuth et al., 2018): not specified
• DBPedia (Hasibi et al., 2017): not specified
• FEVER (Thorne et al., 2018): license terms described on the applicable Wikipedia article pages, and CC BY-SA 3.0 License
• HotpotQA (Yang et al., 2018): CC BY-SA 4.0
Our use is consistent with their intended use.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix E.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix E.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? These are widely used datasets for benchmarking.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix E.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We followed the exact same setting as Karpukhin et al., (2020) and explicitly mentioned this in the paper.
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Table A1 and Table A3. Appendix D.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix D.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Our results are from a single run and it is transparent from the description.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We will release the model and code.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Therefore, how to close this gap will remain
a critical topic for future work on dense retrievers