SlowGuess/NAACL_2021_OCR huggingface dataset:
Yingqi  $\mathbf{Q}\mathbf{u}^{1}$ , Yuchen Ding $^{1}$ , Jing Liu $^{1*}$ , Kai Liu $^{1}$ , Ruiyang Ren $^{2\dagger}$ , Wayne Xin Zhao $^{2*}$ , Daxiang Dong $^{1}$ , Hua Wu $^{1}$  and Haifeng Wang $^{1}$
$^{1}$ Baidu Inc.;  $^{2}$ Gaoling School of Artificial Intelligence, Renmin University of China {quyingqi, dingyuchen, liujing46, liukai20, dongdaxiang, wu_hua, wanghaifeng} @baidu.com reyon. ren@ruc.edu.cn, batmanfly@gmail.com
In open-domain question answering, dense passage retrieval has become a new paradigm to retrieve relevant passages for finding answers. Typically, the dual-encoder architecture is adopted to learn dense representations of questions and passages for semantic matching. However, it is difficult to effectively train a dual-encoder due to the challenges including the discrepancy between training and inference, the existence of unlabeled positives and limited training data. To address these challenges, we propose an optimized training approach, called RocketQA, to improving dense passage retrieval. We make three major technical contributions in RocketQA, namely cross-batch negatives, denoised hard negatives and data augmentation. The experiment results show that RocketQA significantly outperforms previous state-of-the-art models on both MS-MARCO and Natural Questions. We also conduct extensive experiments to examine the effectiveness of the three strategies in RocketQA. Besides, we demonstrate that the performance of end-to-end QA can be improved based on our RocketQA retriever<sup>1</sup>.
Open-domain question answering (QA) aims to find the answers to natural language questions from a large collection of documents. Early QA systems (Brill et al., 2002; Dang et al., 2007; Ferrucci et al., 2010) constructed complicated pipelines consisting of multiple components, including question understanding, document retrieval, passage ranking and answer extraction. Recently, inspired by the advancements of machine reading comprehension (MRC), Chen et al. (2017) proposed a simplified two-stage approach, where a traditional IR
(a) A dual-encoder based on pre-trained LMs.
(b) A cross-encoder based on pre-trained LMs.
Figure 1: The comparison of dual-encoder and cross-encoder architectures.
retriever (e.g., TF-IDF or BM25) first selects a few relevant passages as contexts, and then a neural reader reads the contexts and extracts the answers. As the recall component, the first-stage retriever significantly affects the final QA performance. Though efficient with an inverted index, traditional IR retrievers with term-based sparse representations have limited capabilities in matching questions and passages, e.g., term mismatch.
To deal with the issue of term mismatch, the dual-encoder architecture (as shown in Figure 1a) has been widely explored (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020; Luan et al., 2020; Xiong et al., 2020) to learn dense representations of questions and passages in an end-to-end manner, which provides better representations for semantic matching. These studies first separately encode questions and passages to obtain their dense representations, and then compute the similarity between the dense representations using similarity functions such as cosine or dot product. Typically, the dual-encoder is trained by using in-batch random negatives: for each question-positive passage
† The work was done when Ruiyang Ren was doing internship at Baidu.
$^{1}$ Our code is available at https://github.com/PaddlePaddle/Research/tree/master/NLP/NAACL2021-RocketQA
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835-5847 June 6-11, 2021. ©2021 Association for Computational Linguistics
pair in a training batch, the positive passages for the other questions in the batch would be used as negatives. However, it is still difficult to effectively train a dual-encoder for dense passage retrieval due to the following three major challenges.
First, there exists the discrepancy between training and inference for the dual-encoder retriever. During inference, the retriever needs to identify positive (or relevant) passages for each question from a large collection containing millions of candidates. However, during training, the model is learned to estimate the probabilities of positive passages in a small candidate set for each question, due to the limited memory of a single GPU (or other device). To reduce such a discrepancy, previous work tried to design specific mechanisms for selecting a few hard negatives from the top- $k$  retrieved candidates (Gillick et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Luan et al., 2020; Xiong et al., 2020). However, it suffers from the false negative issue due to the following challenge.
Second, there might be a large number of unlabeled positives. Usually, it is infeasible to completely annotate all the candidate passages for one question. By only examining the top- $K$  passages retrieved by a specific retrieval approach (e.g. BM25), the annotators are likely to miss relevant passages to a question. Taking the MSMARCO dataset (Nguyen et al., 2016) as an example, each question has only 1.1 annotated positive passages on average, while there are 8.8M passages in the whole collection. As will be shown in our experiments, we manually examine the top-retrieved passages that were not labeled as positives in the original MSMARCO dataset, and we find that  $70\%$  of them are actually positives. Hence, it is likely to bring false negatives when sampling hard negatives from the top- $k$  retrieved passages.
Third, it is expensive to acquire large-scale training data for open-domain QA. MSMARCO and Natural Questions (Kwiatkowski et al., 2019) are two largest datasets for open-domain QA. They are created from commercial search engines, and have 516K and 300K annotated questions, respectively. However, it is still insufficient to cover all the topics of questions issued by users to search engines.
In this paper, we focus on addressing these challenges so as to effectively train a dual-encoder retriever for open-domain QA. We propose an optimized training approach, called RocketQA, to improving dense passage retrieval. Considering the
above challenges, we make three major technical contributions in RocketQA. First, RocketQA introduces cross-batch negatives. Comparing to in-batch negatives, it increases the number of available negatives for each question during training, and alleviates the discrepancy between training and inference. Second, RocketQA introduces denoised hard negatives. It aims to remove false negatives from the top-ranked results retrieved by a retriever, and derive more reliable hard negatives. Third, RocketQA leverages large-scale unsupervised data "labeled" by a cross-encoder (as shown in Figure 1b) for data augmentation. Though inefficient, the cross-encoder architecture has been found to be more capable than the dual-encoder architecture in both theory and practice (Luan et al., 2020). Therefore, we utilize a cross-encoder to generate high-quality pseudo labels for unlabeled data which are used to train the dual-encoder retriever. The contributions of this paper are as follows:
- The proposed RocketQA introduces three novel training strategies to improve dense passage retrieval for open-domain QA, namely cross-batch negatives, denoised hard negatives, and data augmentation.
- The overall experiments show that our proposed RocketQA significantly outperforms previous state-of-the-art models on both MS-MARCO and Natural Questions datasets.
- We conduct extensive experiments to examine the effectiveness of the above three strategies in RocketQA. Experimental results show that the three strategies are effective to improve the performance of dense passage retrieval.
- We also demonstrate that the performance of end-to-end QA can be improved based on our RocketQA retriever.
Passage retrieval for open-domain QA For open-domain QA, a passage retriever is an important component to identify relevant passages for answer extraction. Traditional approaches (Chen et al., 2017) implemented term-based passage retrievers (e.g. TF-IDF and BM25), which have limited representation capabilities. Recently, researchers have utilized deep learning to improve traditional passage retrievers, including document expansions (Nogueira et al., 2019c), question expansions (Mao et al., 2020) and term weight estimation (Dai and Callan, 2019).
Different from the above term-based approaches, dense passage retrieval has been proposed to represent both questions and documents as dense vectors (i.e., embeddings), typically in a dual-encoder architecture (as shown in Figure 1a). Existing approaches can be divided into two categories: (1) self-supervised pre-training for retrieval (Lee et al., 2019; Guu et al., 2020; Chang et al., 2020) and (2) fine-tuning pre-trained language models on labeled data. Our work follows the second class of approaches, which show better performance with less cost. Although the dual-encoder architecture enables the appealing paradigm of dense retrieval, it is difficult to effectively train a retriever with such an architecture. As discussed in Section 1, it suffers from a number of challenges, including the training and inference discrepancy, a large number of unlabeled positives and limited training data. Several recent studies (Karpukhin et al., 2020; Luan et al., 2020; Chang et al., 2020; Henderson et al., 2017) tried to address the first challenge by designing complicated sampling mechanism to generate hard negatives. However, it still suffers from the issue of false negatives. The later two challenges have seldom been considered for open-domain QA.
Passage re-ranking for open-domain QA Based on the retrieved passages from a first-stage retriever, BERT-based rerankers have recently been applied to retrieval-based question answering and search-related tasks (Wang et al., 2019; Nogueira and Cho, 2019; Nogueira et al., 2019b; Yan et al., 2019), and yield substantial improvements over the traditional methods. Although effective to some extent, these rankers employ the cross-encoder architecture (as shown in Figure 1b) that is impractical to be applied to all passages in a corpus with respect to a question. The re-rankers (Khattab and Zaharia, 2020; Gao et al., 2020) with light weight interaction based on the representations of dense retrievers have been studied. However, these techniques still rely on a separate retriever which provides candidates and representations. As a comparison, we focus on developing dual-encoder based retrievers.
In this section, we propose an optimized training approach to dense passage retrieval for open-domain QA, namely RocketQA. We first introduce the background of the dual-encoder architecture, and then describe the three novel training strategies in RocketQA. Lastly, we present the whole training procedure of RocketQA.
The task of open-domain QA is described as follows. Given a natural language question, a system is required to answer it based on a large collection of documents. Let  $C$  denote the corpus, consisting of  $N$  documents. We split the  $N$  documents into  $M$  passages, denoted by  $p_1, p_2, \ldots, p_M$ , where each passage  $p_i$  can be viewed as an  $l$ -length sequence of tokens  $p_i^{(1)}, p_i^{(2)}, \ldots, p_i^{(l)}$ . Given a question  $q$ , the task is to find a passage  $p_i$  among the  $M$  candidates, and extract a span  $p_i^{(s)}, p_i^{(s+1)}, \ldots, p_i^{(e)}$  from  $p_i$  that can answer the question. In this paper, we mainly focus on developing a dense retriever to retrieve the passages that contain the answer.
3.2 The Dual-Encoder Architecture
We develop our passage retriever based on the typical dual-encoder architecture, as illustrated in Figure 1a. First, a dense passage retriever uses an encoder  $E_{p}(\cdot)$  to obtain the  $d$ -dimensional real-valued vectors (a.k.a., embedding) of passages. Then, an index of passage embeddings is built for retrieval. At query time, another encoder  $E_{q}(\cdot)$  is applied to embed the input question to a  $d$ -dimensional real-valued vector, and  $k$  passages whose embeddings are the closest to the question's will be retrieved. The similarity between the question  $q$  and a candidate passage  $p$  can be computed as the dot product of their vectors:
$$
\operatorname {s i m} (q, p) = E _ {q} (q) \cdot E _ {p} (p). \tag {1}
$$
In practice, the separation of question encoding and passage encoding is desirable, so that the dense representations of all passages can be precomputed for efficient retrieval. Here, we adopt two independent neural networks initialized from pre-trained LMs for the two encoders  $E_{q}(\cdot)$  and  $E_{p}(\cdot)$  separately, and take the representations at the first token (e.g., [CLS] symbol in BERT) as the output for encoding.
Training The training objective is to learn dense representations of questions and passages so that question-positive passage pairs have higher similarity than the question-negative passage pairs in training data. Formally, given a question  $q_{i}$  together with its positive passage  $p_{i}^{+}$ and  $m$  negative passages  $\{p_{i,j}^{-}\}_{j = 1}^{m}$ , we minimize the loss function:
$$
\begin{array}{l} \mathcal {L} \left(q _ {i}, p _ {i} ^ {+}, \left\{p _ {i, j} ^ {-} \right\} _ {j = 1} ^ {m}\right) \\ = - \log \frac {e ^ {\operatorname {s i m} \left(q _ {i} , p _ {i} ^ {+}\right)}}{e ^ {\operatorname {s i m} \left(q _ {i} , p _ {i} ^ {+}\right)} + \sum_ {j = 1} ^ {m} e ^ {\operatorname {s i m} \left(q _ {i} , p _ {i , j} ^ {-}\right)}}, \tag {2} \\ \end{array}
$$
where we aim to optimize the negative log likelihood of the positive passage against a set of  $m$  negative passages. Ideally, we should take all the negative passages in the whole collection into consideration in Equation 2. However, it is computationally infeasible to consider a large number of negative samples for a question, and hence  $m$  is practically set to a small number that is far less than  $M$ . As what will be discussed later, both the number and the quality of negatives affect the final performance of passage retrieval.
Inference In our implementation, we use FAISS (Johnson et al., 2019) to index the dense representations of all passages. Specifically, we use IndexFlatIP for indexing and the exact maximum inner product search for querying.
3.3 Optimized Training Approach
In Section 1, we have discussed three major challenges in training the dual-encoder based retriever, including the training and inference discrepancy, the existence of unlabeled positives, and limited training data. Next, we propose three improved training strategies to address the three challenges.
Cross-batch Negatives When training the dual-encoder, the trick of in-batch negatives has been widely used in previous work (Henderson et al., 2017; Gillick et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Luan et al., 2020). Assume that there are  $B$  questions in a mini-batch on a single GPU, and each question has one positive passage. With the in-batch negative trick, each question can be further paired with  $B - 1$  negatives (i.e., positive passages of the rest questions) without sampling additional negatives. In-batch negative training is a memory-efficient way to reuse the examples already loaded in a mini-batch rather than sampling new negatives, which increases the number of negatives for each question. As illustrated at the top of Figure 2, we present an example for in-batch negatives when training on  $A$  GPUs in a data parallel way. To further optimize the training with more negatives, we propose to use cross-batch negatives when training on multiple GPUs, as illustrated at the bottom of Figure 2. Specifically, we first compute the passage embeddings within each single GPU, and then share these passage embeddings among all the GPUs. Besides the in-batch negatives, we collect all passages (i.e., their dense representations) from other GPUs as the additional negatives for each question. Hence, with  $A$  GPUs
Figure 2: The comparison of traditional in-batch negatives and our cross-batch negatives when trained on multiple GPUs, where  $A$  is the number of GPUs, and  $B$  is the number of questions in each min-batch.
(or mini-batches)  $^2$ , we can indeed obtain  $A \times B - 1$  negatives for a given question, which is approximately  $A$  times as many as the original number of in-batch negatives. In this way, we can use more negatives in the training objective of Equation 2, so that the results are expected to be improved.
Denoised Hard Negatives Although the above strategy can increase the number of negatives, most of negatives are easy ones, which can be easily discriminated. While, hard negatives are shown to be important to train a dual-encoder (Gillick et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Luan et al., 2020; Xiong et al., 2020). To obtain hard negatives, a straightforward method is to select the top-ranked passages (excluding the labeled positive passages) as negative samples. However, it is likely to bring false negatives (i.e., unlabeled positives), since the annotators can only annotate a few top-retrieved passages (as discussed in Section 1). Another note is that previous work mainly focuses on factoid questions, to which the answers are short and concise. Hence, it is not challenging to filter false negatives by using the short answers (Karpukhin et al., 2020). However, it cannot apply to non-factoid questions. In this paper, we aim to learn dense passage retrieval for both factoid questions and non-factoid questions, which needs a more effective way for denoising hard negatives.
Here, our idea is to utilize a well-trained cross-encoder to remove top-retrieved passages that are likely to be false negatives. Because the cross-encoder architecture is more powerful for capturing semantic similarity via deep interaction and shows much better performance than the dual-encoder ar
Note that cross-batch negatives can be applied in both settings of single-GPU and multi-GPUs. When there is only a single GPU available, it can be implemented in an accumulation way while trading off training time.
1. Train a dual-encoder with cross-batch sampling
2. Train a cross-encoder optimized for the output distribution of  $M_D^{(0)}$
3. Train a dual-encoder by sampling hard negatives from the output of  $M_D^{(0)}$  and denosied by  $M_C$
4. Train a dual-encoder with data augmentation by  $M_D^{(l)}$  and  $M_C$
Figure 3: The pipeline of the optimized training approach RocketQA.  $M_D$  and  $M_C$  denote the dual-encoder and cross-encoder, respectively. We use  $M_D^{(0)}$ ,  $M_D^{(1)}$  and  $M_D^{(2)}$  to denote the learned dual-encoders after different steps.
chitecture (Luan et al., 2020). The cross-encoder is more effective and robust, while it is inefficient over a large number of candidates in inference. Hence, we first train a cross-encoder (following the architecture shown in Figure 1b). Then, when sampling hard negatives from the top-ranked passages retrieved by a dense retriever, we select only the passages that are predicted as negatives by the cross-encoder with high confidence scores. The selected top-retrieved passages can be considered as denosied samples that are more reliable to be used as hard negatives.
Data Augmentation The third strategy aims to alleviate the issue of limited training data. Since the cross-encoder is more powerful in measuring the similarity between questions and passages, we utilize it to annotate unlabeled questions for data augmentation. Specifically, we incorporate a new collection of unlabeled questions, while reuse the passage collection. Then, we use the learned cross-encoder to predict the passage labels for the new questions. To ensure the quality of the automatically labeled data, we only select the predicted positive and negative passages with high confidence scores estimated by the cross-encoder. Finally, the automatically labeled data is used as augmented training data to learn the dual encoder. Another view of the data augmentation is knowledge distillation (Hinton et al., 2015), where the cross-encoder is the teacher and the dual-encoder is the student.
3.4 The Training Procedure
As shown in Figure 3, we organize the above three training strategies into an effective training pipeline for the dual-encoder. It makes an analogy to a multi-stage rocket, where the performance of the dual-encoder is consecutively improved at three steps (STEP 1, 3 and 4). That is why we call our
approach RocketQA. Next, we will describe the details of the whole training procedure of RocketQA.
- REQUIRE: Let  $C$  denote a collection of passages.  $Q_{L}$  is a set of questions that have corresponding labeled passages in  $C$ , and  $Q_{U}$  is a set of questions that have no corresponding labeled passages.  $D_{L}$  is a dataset consisting of  $C$  and  $Q_{L}$ , and  $D_{U}$  is a dataset consisting of  $C$  and  $Q_{U}$ .
- STEP 1: Train a dual-encoder  $M_D^{(0)}$  by using cross-batch negatives on  $D_L$ .
- STEP 2: Train a cross-encoder  $M_C$  on  $D_L$ . The positives used for training the cross-encoder are from the original training set  $D_L$ , while the negatives are randomly sampled from the top- $k$  passages (excluding the labeled positive passages) retrieved by  $M_D^{(0)}$  from  $C$  for each question  $q \in Q_L$ . This design is to let the cross-encoder adjust to the distribution of the results retrieved by the dual-encoder, since the cross-encoder will be used in the following two steps for optimizing the dual-encoder. This design is important, and there is similar observation in Facebook Search (Huang et al., 2020).
- STEP 3: Train a dual-encoder  $M_D^{(1)}$  by further introducing denoised hard negative sampling on  $D_L$ . Regarding to each question  $q \in Q_L$ , the hard negatives are sampled from the top passages retrieved by  $M_D^{(0)}$  from  $C$ , and only the passages that are predicted as negatives by the cross-encoder  $M_C$  with high confidence scores will be selected.
- STEP 4: Construct pseudo training data  $D_U$  by using  $M_C$  to label the top- $k$  passages retrieved by  $M_D^{(1)}$  from  $C$  for each question  $q \in Q_U$ , and then train a dual-encoder  $M_D^{(2)}$  on both the manually labeled training data  $D_L$  and the automatically augmented training data  $D_U$ .
Note that the cross-batch negative strategy is applied through all the steps for training the dual
datasets	#q in train	#q in dev	#q in test	#p	ave. q length	ave. p length
MSMARCO	502,939	6,980	6,837	8,841,823	5.97	56.58
NQ	58,812	-	3,610	21,015,324	9.20	100.0
Table 1: The statistics of datasets MSMARCO and Natural Questions. Here, "p" and "q" are the abbreviations of questions and passages, respectively. The length is in tokens.
encoder. The cross-encoder is used both STEP 3 and STEP 4 with different purposes to promote the performance of the dual encoder. The implementation details of denoising hard negatives and data augmentation can be found in Section 4.
We conduct the experiments on two popular QA benchmarks: MSMARCO Passage Ranking (Nguyen et al., 2016) and Natural Questions (NQ) (Kwiatkowski et al., 2019). The statistics of the datasets are listed in Table 1.
MSMARCO Passage Ranking MSMARCO is originally designed for multiple passage MRC, and its questions were sampled from Bing search logs. Based on the questions and passages in MSMARCO Question Answering, a dataset for passage ranking was created, namely MSMARCO Passage Ranking, consisting of about 8.8 million passages. The goal is to find positive passages that answer the questions.
Natural Question (NQ) Kwiatkowski et al. (2019) introduces a large dataset for open-domain QA. The original dataset contains more than 300,000 questions collected from Google search logs. In Karpukhin et al. (2020), around 62,000 factoid questions are selected, and all the Wikipedia articles are processed as the collection of passages. There are more than 21 million passages in the corpus. In our experiments, we reuse the version of NQ created by Karpukhin et al. (2020). Note that the dataset used in DPR contains empty negatives, and we discarded the empty ones.