Adversarial Retriever-Ranker for Dense Text Retrieval
Authors: Hang Zhang
Yeyun Gong
Yelong Shen
Jiancheng Lv
Nan Duan
Weizhu Chen
Venue: ICLR 2022 Poster (2022)

Abstract: Current dense text retrieval models face two typical challenges. First, it adopts a siamese dual-encoder architecture to encode query and document independently for fast indexing and searching, whereas neglecting the finer-grained term-wise interactions. This results in a sub-optimal recall performance. Second, it highly relies on a negative sampling technique to build up the negative documents in its contrastive loss. To address these challenges, we present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker. The two models are jointly optimized according to a minimax adversarial objective: the retriever learns to retrieve negative documents to cheat the ranker, while the ranker learns to rank a collection of candidates including both the ground-truth and the retrieved ones, as well as providing progressive direct feedback to the dual-encoder retriever. Through this adversarial game, the retriever gradually produces harder negative documents to train a better ranker, whereas the cross-encoder ranker provides progressive feedback to improve retriever. We evaluate AR2 on three benchmarks. Experimental results show that AR2 consistently and significantly outperforms existing dense retriever methods and achieves new state-of-the-art results on all of them. This includes the improvements on Natural Questions R@5 to 77.9%(+2.1%), TriviaQA R@5 to 78.2%(+1.4), and MS-MARCO MRR@10 to 39.5%(+1.3%). We will make our code, models, and data publicly available. 

Published as a conference paper at ICLR 2022
ADVERSARIAL
EETRIEVER-RANKER FOR DENSE
TEXT RETRIEVAL
Hang Zhang': Yeyun Gong2! Yelong Shen', Jiancheng Lv Nan Duan', Weizhu Chen 'College of Computer Science, Sichuan University, "iicoosoft Research Asia, Microsoof Azure AI
hangzhang.scuu foxmail 1, com, (yegong, yel.ong.shennjmmcrrooofff com, Lyjiancheng@scu. edu. cn, {nanduan, wrchen/emmicrooofftt
ABSTRACT
Current dense text retrieval models face two typical challenges. First, they adopt a siamese dual-encoder architecture to encode queries and documents indepen- dently for fast indexing and searching, while neglecting the finer-grained term- wise interactions. This results in sub-optimal recall performance. Second, their model training highly relies on negative sampling technique to build up the nega- tive documents in their contrastive losses. To address these challenges, we present Adversarial Retriever- Rannerr (AR2), which consists a dual-encoder retriever plus cross-encoder ranker. The two models jointly optimized according a minimax adversarial objective: the retriever learns to retrieve negative docu- ments to cheat the ranker, while the ranker learns rank collection of candi- dates including both the ground-truth and the retrieved ones, as well as providing progressive direct feedback to the dual-encoder retriever. Through this adversar- ial game, the retriever gradually produces harder negative documents to train a better ranker, whereas the cross-encoder ranker provides progressive feedback improve retriever. We evaluate AR2 on three benchmarks. Experimental results show that AR2 consistently and significantly outperforms existing dense retriever methods and achieves new state-of-the-art results all of them. This includes the improvements on Natural Questions R R@5 to 77.9% (+ (+2.1%) TriviaQA R R@5 5 to 78.2% (+1.4%), and MS-MARCO MRR @ 10 to 59.5% (+1.3% Code and models are available at
com wtt.://....mmmmiiiiioccom http///cccccoooooooooooooooot
INTRODUCTION
Dense text retrieval (Lee et al. 2019 Karpukhin et al. 2020 has achieved great successes in a wide
variety of both research and industrial areas, such as search engines (Brickley et al.) 2019 Shen
let al.
2014).
recommendation system (Hu et al. 2020), open-domain question answering Guo
al. 2018 Liu et al.) 2020), etc. A typical dense retrieval model adopts a dual-encoder (Huang
et al. 2013) architecture to encode queries and documents into -iw-imensional embedding vectors,
with the relevance between query and document being measured by the similarity between embed- dings. In real-world dense text retrieval applications, it pre-computes all the embedding vectors of documents the corpus, and leverages the approximate nearest neighbor (ANN) (Johnson et al.) 2019) technique for efficiency. To train a dense retriever, contrastive loss with negative samples is widely applied in the literature (Xiong et al. 2021 Karpukhin et al. 2020). During training, the model utilizes negative sampling method to obtain negative documents for given query- document pair, and then minimizes the contrastive loss which relies on both the positive document
and the sampled negative ones (Shen et al. 2014 Chen al. 2017 Radford et al. 2021).
Recent studies on contrastive learning (Xiong et al., 2021 Karpukhin et al. 2020) show that the
iterative "hard-negative" sampling technique can significantly improve the performance compared with "random-negative"" sampling approach, as it can pick more representative negative samples to
WWrk is done during internship at Microsoft Research Asia.
1 Cooresponning author

---

Published as a conference paper at ICLR 2022
relevance
relevance
query
document
query
document
Retriever
(b) Ranker
Figure 1:
Illustration of two modules in AR2. (a) Retriever: query and document are encoded independently by dual-encoder. (b) Ranker: concatenated, jointly encoded by cross-encoder.
learn more discriminative retriever. In the work Qu et al. 2021 encoder model to heuristically filter hhard-negative" samples to further improve performance and shows the importance of sampling technique in the contrastive learning.
suggests leveraging cross-
On the other hand, the model architecture of dual-encoders enables the encoding of queries and documents independently which is essential for document indexing and fast retrieval. However, this ignores the modeling of finer-grained interactions between queries and documents which could be sub-optimal solution in terms retrieval accuracy.
Motivated by these phenomena, propose Adversarial Retriever-Ranke (AR2) framework. The intuitive idea AR2 inspired by the "retriever-ranker" architecture the classical informa- tion retrieval systems. AR2 consists of two modules: dual-encoder model served the retrieval module in Figure and a cross-cncee cross-encoder model served as the ranker module in Figure The cross- encoder model takes the concatenation of query and document as input text, and generate more accurate relevance scores compared with the dual-encoder model, since it can fully explore the inter- actions between the query and document through self-attention mechanism using conventional transformer model (Vaswani et al., 2017 Guo et al.) 2020). Instead of training "retriever-ranker" 2008 Mitra Craswell 2017), AR2 constructs unified minimax game for training the retriveer and ranker models interactively, as shown in Figure 2
modules independently in
IR systems (Manning
In particular, AR2 adopts minimax objective from the adversarial game (Goodfellow al. 2014) where the retrieval model is optimized to produce relevant doc- uments fool the ranker model, whereas the ranker model learns to distinguish the ground-truth relevant document and retrieved ones by its opponent retrieval model. Within the adversarial "retriever-ranke" ing framework, the retrieval model receives the smooth training signals from the ranker model which helps alle- viate the harmful effects of "false-negative" issues. For example, a "false-negative" example which is rated as high-relevance by the ranker model, will also be granted with high probability by retrieval model in order fool the ranker, meanwhile the ranker model with better gen- eralization capability is more resistant label noises compared to the retrieval model.
Ranker
train-
Retriever
D
q
Retrieved
Document Pool
In the empirical studies of AR2, we further intro- duce distillation regularization approach to help sta- bilize//iprove the training of the retriever. Intuitively, the retriever would converge to sharp conditional- probabilities over documents given a query within the adversarial training framework, i.e., high retrieval prob-
Figure 2: Illustration of the AR2 train- ing pipeline. q. d, and D, represent the query, positive document, and retrieved documents, respectively.
abilities for the top relevant documents and near-zero retrieval ones for the rest. However, it is not desirable property it might impede exploring dif-
2

---

Published as a conference paper at ICLR 2022
ferent documents during training. Thus, incorporate the distillation loss between the retriever and ranker models as a smooth term for further improvement.
In experiments, evaluate AR2 three widely used benchmarks for dense text retrieval: Natural Questions, Trivia QA and MS-MARCO. Experimental results show that AR2 achieves state-of-the art performance all these datasets. Meanwhile, we provide comprehensive ablation study to demonstrate the advantage of different AR2 components.
PRELIMINARIES
Dense Text Retrieval: We mainly consider a contrastive-leaming paradigm for dense text retrieval
in this work, where the training set consists of a collection of text pairs. C' = {((1, di)...... (fn, d..) In the scenario of open-domain question answering, a text pair (q, d) refers to a question and a corresponding document which contains the answer. A typical dense retrieval model adopts dual encoder architecture, where questions and documents are represented as dense vectors separately and the relevance score so(g, d) between them is measured by the similarity between their embeddings:
so (g. d) (Elq: 0), E(d:
(1)
where E(-:0) denotes the encoder module parameterized with and (-) is the similarity function, e.g., inner-product, Euclidean distance. Based the embeddings, existing solutions generally lever- age on-the-shelf fast ANN-search (Johnson et al. 2019) for efficiency.
A conventional contrastive- contrastive-leariing learning algorithm could be applied for training the dual encoders (Shen et al. 2014 Chen et al. 2017 Liu et al. 2020). For example, given a training instance (q. d), we select n negative irrelevant documents (d, ...., d (denoted as D2y) to optimize the loss function of the negative log likelihood of the positive document:
(oo(qq.)
Le(g, d, 1.7) = -log
(2)
(Toolg.d) + Σm= (reo(9.7))
where 7 is al. [017 a
to control the temperature. Previous works (Shen et al. 2014 Chen [0u 2020) present an effective strategy on negative document sampling, called "In- Batch Negatives" where negative documents are randomly sampled from a collection of documents which are within the same mini-batch as question-documen training pairs.
Recently, some studies e.g., ANCE (Xiong et al. 2021) and Condenser (Gao & Callan 2021b), have shown that selecting "hard-negatives" the training can significantly improve the retrieval perfor- mance in open-domain question answering. For example, instead of sampling negative document randomly, "hard-negatives" are iteratively retrieved through previous checkpoints of the dual en- coder model. However, more recent work RocketQA (Qu al. 2021) continues to point out that the retrieved "hard-negativer could potential be -agat-nnegatives in some cases, which might limit the performance.
Generative Adversarial Network: GANs have been widely studied for generating the realistic- looking images in computation vision (Goodfellow et al.] 2014 Brock et al.) 2018). In the past few years, the idea of GANs has been applied in information retrieval (Wang et al. 2017). For example, IRGAN (Wang al. 2017), proposes minimax retrieval framework which constructs two types of IR models: generative retrieval model and discriminative retrieval model. The two IR models are optimized through a minimax game: the generative retrieval model generates relevant documents that look like ground-truth relevant documents to fool the discriminative retrieval model, whereas the discriminative retrieval model learns to draw clear distinction between the ground- truth relevant documents and the generated ones made by its opponent generative retrieval model. The minimax objective is formulated as:
JG' Ean-max,EEu [l.gD,(d, + Egg ~Go(-19) [log(1 Do(d .9)]
where Ga(-1g) and Do ,q) denote the generative retrieval model and discriminative retrieval model IRGAN, respectively. It is worth noting the original IRGAN model doesn't 't work for dense retrieval tasks it doesn't it contain the dual-encoder model for document indexing or fast retrieval.
3

---

Published as a conference paper at ICLR 2022
3 METHOD
In this section, we introduce the proposed adversarial retriever-ranker (AR2) approach. It consists of two modules: the dual-encoder retriever module Gø as in Figure la and the cross-encoder ranker module D& as in Figure lb Gg and D& computes the relevance score between question and document as follows:
Galg, d) = Eg Ed(d) Do d) = we Es (1g,d)
where E-)-)) and (-) are language model encoders which can be initialized with any pre-trained language model, we the linear projector in Dg. and [g. d] is the concatenation question and document.
In AR2, the retriever and ranker modules are optimized jointly through a contrastive minimax ob- jective:
GGD ming max DD PDy~Gola.) [logpa (dlg: [2y]]
(5)
where [d, 1i=1 the set of n negative documents sampled by Go(q,) :) given q, and p& (dlq: Dg) denotes the probability of selecting the ground-truth document from the document set DD (ID, = {d) U (77) by the ranker module DΦ:
(D..dd)
po(d]q; Dg) Even, eolaa)
According the objective function (Eqn. 5). the dual-encoder retrieval model Gø(g,)) -) would try to sample the high-relevant documents to fool the ranker model, whereas the ranker model D Da(a,)) optimized draw distinctions between ground-truth passage and the ones sampled by Gg(g, We present the illustration of the AR2 framework in Figure - In order to optimize the minimax objective function, we adopt a conventional iterative-leamnng mechanism to optimize the retriever and ranker modules coordinately.
3.1 TRAINING THE RANKER Di
Given the fixed retriever Gg, the ranker model D& is updated by maximizing the log likelihood of selecting ground-truth from set Dr given query q:
urgmax,logpoddegg
where ID, consists of ground-truth document d and negative document set ID 1D7 is sampled by Go according Eqn. [ In experiments, first retrieve top-100 negative 'ocuueetts and then randomly sample examples from them to obtain 17
3.2 TRAINING RETRIEVER Go
With fixing the ranker D3, the model parameters 0 for the retriever Gg is optimized by minimizing the expectation of log likelihood of function. In particular, by isolating 0 from the minimax function (Eqn. 5). the objective for the retriever can be written as:
0t argming.J _ =En, ~Gala)) (d]g; ID))
(8)
However, it intractable optimize 9 directly through Eqn. 8| as the computation of probability D ~ Go(g.) does not follow a close form. Thus, we seek to minimize an alternative upper-bound of the loss criteria:
< j Ex-~pa(la:D)
[logrs {d, dd.
The detailed deviation of Eqn. iis provided in the Appendix A.I eter 0 can be computed as the derivative of jo with respect to
Therefore, the gradient of param-
j Ex-~po(-lydD) (p) Vo Volog (e DD: [log {d, })]
(10)
4

---

Published as a conference paper at ICLR 2022
Algorithm Adversarial Retriever-Ranne (AR2)
Require:
Retriever Gø; Ranker Dæ: Document pool D; Training dataset C.
1: Initialize the retriever Gg and the ranker Da with pre-trained language models.
2: Train the warm-up retriever Gg on training dataset C.
3: Build ANN index on IB
4: Retrieve negative samples on
5: Train the warm-up ranker D
6: while AR2 has not converged do
7: for Retriever training step do
8: Sample n documents {d, In from ANN index.
9: Update parameters of the retriever Gg.
10: end for
11: Refresh ANN Index.
12: for Ranker training step do
Sample n hard negatives {d; Jn from ANN index. 14: Update parameters of the ranker Dg.
end for end while
16:
Here, the same approach is applied to obtain set D,, as in Eqn.
Regularization: we further introduce distillation regularization term Gy's training, which en- courages the retriever model to distill from the ranker model.
J (Pp DD. po(:lq:
(11)
H(-) is the cross entropy function. p- q; D) and po( l:; D) denote the conditional probabilities of document the whole corpus by the ranker and the retriever model, respectively. In practice, we also limit the sampling space over documents to a fixed set, i.e., D2 {d} U D... Thus the regularization loss for the retriever model can rewritten as:
J = H (po(-la: Po).po( p-10: D2))
(12)
3.3 INDEX REFRESH
During each training iteration retriever and ranker models AR2, we refresh the document index to keep the retrieved document set updated. To build the document index, we take the document encoder from the retrieval model to compute the embeddings Edd: 0) for every document d from the corpus: d C, and then build the inner-produc based ANN search index with FAISS tool.
In summary, Algorithm shows the full implementation details of the proposed AR2.
EXPERIMENTS
4.1 DATASETS
We conduct experiments on three popular benchmarks: Natural Questions (Kiiakowski et al., 2019), Trivia QA (Joshi et al. 2017), and MS-MARCO Passage Ranking (Nguyen al. (2016).
Natural Questions (NQ) collects real questions from the Google search engine and each question paired with an answer span and golden passages from the Wikipedia pages. NQ, the goal of the retrieval stage is find positive passages from large passage pool. We report Recall of top-k (R@k), which represents the proportion of top k retrieved passages that contain the answers.
Trivia QA is reading comprehension corpus authored by trivia enthusiasts. Each sample is (question, answer, evidence) triple. In the retrieval stage, the goal is to find passages that contain the answer. We also use Recall of top-k the evaluation metric for Trivia QA.
MMS-MARCO Passage Ranking is widely used in information retrieval. It collects real questions from the Bing search engine. Each question is paired with several web documents. Following previous
5

---

Published as a conference paper at ICLR 2022
Table The comparison the first-stage retrieval performance on Natural Questions test set, Trivia QA test set, and MS-MARCO dev set. The results of the first two blocks are from published papers. the results are not provided, we mark them "-"
Natural Questions
Trivia QA
MS-MARCO
R@5
@@20
R@:100
R@5
R
R 100
MRR@
R@
R@lk
BM25 Yang
2017
59.1
73.7
66.9
76.7
18.7
59.2
85.7
GAR Max mi. 2
60.9
74.4
85.3
73.1
80.4
85.7
dne2query (Nogueira
2019b
DeepCT Dar
2019
24.3
69.0
91.0
dn:TTTT query (Noguetra el 2019a
27.7
75.6
94.7
DPR (Karpuklin
83.3
84.9
ANCE [Xiong et ii 6024
81.9
87.5
80.3
85.3
33,0
95.9
RDR (Yang & 500 20AU
88.2
82.5
87.3
ColBERT (Kharish & Zalarii 2020
36.0
82.9
96.8
RocketQA c 2021
74.0
82.7
88.5
37.0
85.5
97.9
COIL (Gao
20-1
96.3
ME-BEKI Luan o 202]
33.8
Joint Top-k Saadia si 5122a
72.1
81.8
87.8
74.1
81.3
86.3
Individual Top-K (Sachin
a0
75.0
89.2
76.8
83.1
87.0
PAIR (Ren el 2021
74.9
83.5
89.1
37.9
86.4
98.2
DPR-PAD PPRR-PAQPP (Ugiz et
-BERTT..
74.5
83.7
88.6
31.4
-RBBERTapa
74.2
89.2
31.1
Condenser Gao Callan 2021b
88.4
86.2
97.4
coCondenser (Gao & Calaan 402 1
75.8
84.3
89.0
76.8
83.2
87.3
38.2
98.4
AR2-G"
69.7
80.8
87.1
74.4
81.7
86.6
34,8
84.2
98.0
AR2-G
77.9
86.0
90.1
78.2
87.9
39.5
87.8
98.6
works (Ren al. 2021 1: Qu al. 2021). report MRR @ 10, R@ 50, R@ Ik on the dev set. Mean Reciprocal Rank (MRR) is the mean Reciprocal Rank(RR) across questions, calculated as the reciprocal of the rank where the first relevant document retrieved.
4.2 IMPLEMENTATION DETAILS
First step, follow the experiments in Sachan al. (2021b) and Gao & Callan (2021a) tinuous pre-training the ERNIE-2200bass model (Sun et al. (020) with Inverse Cloze Task (ICT) training (Lee al. 2019) for NQ and TriviaQA datasets, and coCondenser training (Gao & Callan 2021a) for MS-MARCO dataset.
Second step, follow the experiment settings of DPR (Karpukhin al. 2020) train warm-up dual-encoder retrieval model G It is initialized with the continuous pretrained ERNIE-2.0-bar model we obtained in step one. Then we train a warm-up cross-encoder model D initialized with the ERNNIE-2.0-Larg DO learns to rank the Top-k documents selected by Go with contrastive learning. The detailed hyper-parameters in training are listed in Appendix A..3
Third step, we iteratively train the ranker (AR2-D) model initialized with ERNIE- ERNIE-2.0-larg. 1.0- and the retriever (AR2-G) initialized with G according to Algorithm | The number of training iterations set to 10. During each iteration of training, the retriever model scheduled to train with 1500 mini- batches, while ranker model is scheduled to train with 500 mini-batches. The document index shown in Appendix A.3
refreshed after each iteration of training. The other hyper-parameee
All the experiments this work run on NVIDIA Tesla A100 GPUs. The implementation code of AR2 is based on Huggingface Transformers (Wolf al. 2020 utilizing gradient checkpoint- ing (Chen et al. 2016), Apex| and gradient accumulation to reduce GPU memory consumption.
4.3 RESULTS
Performance of Retriever AR2-G: The comparison of retrieval performance on NQ, Trivia QA, and MS-MARCO are presented in Table
We compare AR2-G with previous state-of-the toffthheartt art methods, including sparse and dense retrieval models. The top block shows the performance of sparse retrieval methods. BM25 (Yang al. 2017)
a traditional sparse retriever based on the exact term matching. DeepCT' (Dai & Callan 2019 uses
https:: http//ttttttt..co WWw.://pp.//...

---

Published as a conference paper at ICLR 2022
Table Performance of rankers before and after AR2 training on NQ test set.
Table Performance of AR2-G on NQ test set with different negative sample size n.
Retriever
Ranker
R@I
R@5
R@
R@
R@5
R@
R @ 100
Latency
69.7
76.2
56.3
76.4
85.3
89.7
210ms
AR2-G"
AR2-D
60.6
78.7
82.6
n=5
57.8
76.9
85.3
89.7
330ms
AR2-D
64.2
79.0
82.6
n=7
58.0
77.2
85.2
89.7
396ms
58.7
77.9
82.5
58.0
77.1
85.4
89.8
510ms
AR2-G
AR2-D
61.1
80.1
84.3
n=15
57.8
77.3
85.6
90.1
630ms
AR2-D
65.6
81.5
84,9
Table Comparison of AR2 and IRGAN.
Table 5: Effect of regularization in AR2.
R@I
R@S
R@ 20
R@ 100
R@1
R@5
R@200
R@ 100
Entropy
AR2
58.7
77.9
86.0
90.1
AR2-G
58.7
77.9
86.0
90.1
2.10
IRGAN
55.2
75.2
84.5
89.2
w/o R
57.8
77.3
85.6
90.1
1.70
BERT to dynamically generate lexical weights augment BM25 Systems. doc2Query NNgueira et al. 2019b), TTTTTQurrr (Nogueira et al., 2019a), and GAR (Mao et al. 2021a) use text gen- eration expand queries or documents make better use of BM25. The middle block lists the re- sults of strong dense retrieval methods, including DPR (Karpukhin et al. 2020). ANCE (Xiong et al.) 2021. RDR Yang & Seo, 2020), RocketQA (Qu et 2021), Joint and Individual Top-k (Sachan et al. 2021a), PAIR (Ren et al.] (02T). DPR-PAQ (Oguz et al.] (02T). Condenser (Gao & Callan 2021bb. lb). coCondenser (Gao & Callan 2021a). ME-BERT (Luan et al., 202I). CoIL (Gao et al, 2021). These methods improve the performance of dense retrieval by constructing hard negative samples, jointly training the retriever and downstream tasks, pre-training, knowledge distillation, and multi-vector representations.
The bottom block in Table [ shows the results of proposed AR2 models. AR2-G" refers to the warm-up retrieval model in AR2 (details be found in section 4.2) which leverages the existing continuous pre-training technique for dense text retrieval tasks. i.e., it shows a better performance compared with DPR (Karpukhin et al. 20200) and ANCE (Xiong al. 202I). etc approaches that do not adopt the continuous pre-training procedure. We also observed that AR2-G: the retrieval model trained with the adversary framework, significantly outperforms the warm-up AR2-GO model, and achieves new state-of- of-the-art performance on all three datasets.
4.4 ANALYSIS
In this section, we conduct a set of detailed experiments on analyzing the proposed AR2 training framework help understand its pros and cons.
Performance of Ranker AR2-D: To evaluate the performance of ranker AR2-D D on NQ, we first retrieve the top-100 documents for each query in the test set with the help of dual-encoder AR2- G model, and then re-rank them with the scores produced by the AR2-D model. The results are shown in Table 2 "L" represents without ranker. AR2-D refers the warm-up ranker model in AR2. The results show that the ranker obtains better performance compared with only using retriever. It suggests that we could use a two-stage ranking strategy further boost the retrieval performance. Comparing the results of AR2-D and AR2-D we further find that the ranker AR2- D gets a significant gain with adversarial training.
Impact of Negative Sample Size: In the training of AR2, the number of negative documents n would affect both the model performance and training time. In Table 3 we show the performance and the training latency per batch with different negative sample size In this setting, we evaluate AR2 without the regularization term. We observe the improvement over R R@ and @ by increas- ing from 1 to 7, and marginal improvement when keep increasing from 7 to 15. The latency of training per batch is almost línear increased by improving n.
The original IRGAN (Wang al. 2017) doesn't work for dense text retrieval tasks as does not contain the dual-encoder retrieval model for fast document indexing and search. However, it provides an conventional GAN framework for training the generative and discriminative models jointly for IR tasks. To compare the proposed AR2 with IRGAN, we replaced the generative and discriminative models in IRGAN with the retriever and ranker models in AR2,
Comparison with IRGAN:

---

Published as a conference paper at ICLR 2022
0.92
4.2
6.7
5.75
6.71
0.76
6.7
5.7
0.00
20
Earatio
Figure 3: NQ R@5 the number of iteration for both the AR2-retriever and the AR2-ranker.
Figure 4: The comparison of ANCE and AR2 NQ test set.
Table 6: The results of the second-stage ranking Natural Questions test set. Note that copy the numbers of the first block from the RIDER paper (Mao et al. 2021b
Retriever
Ranker
R @
R@5
R@ 10
R@21 20
R@50
R@ 100
GAR
Mao et al.
2021a)
46.8
70.7
77.0
81.5
88.9
GAR +
Mao et al.
2021a
BERT
51.4
67.6
75.7
82.4
88.9
GAR
Mao et
2021a
BART
55.2
73.5
78.5
82.2
88.9
GAR +
Mao et
2021a)
RIDER
53.5
75.2
80.0
83.2
88.9
AR2-G
58.7
77.9
82.5
86.0
88.5
90.1
AR2-G
AR2-D
65.6
81.5
84.9
87.2
89.5
90.1
respectively. Therefore, with the configuration of the same model architectures for generator (re- triever) and discriminator (ranker), The performance of the retriever is shown in Table 4 We see that AR2 outperforms IRGAN significantly.
Effect of Regularization: To study the effectiveness of regularization, we conducted ablation stud- ies by removing the regularization term in the training of retrieval model. In Table [5] "R" refers to the regularization item, it shows that the regularization approach helps improve the R@ and @5 evaluation metrics. In additional, we compute the average entropy of distribution pe (-(g d, D2,) on the NQ test set, where DD is the retrieved top-15 documents. The average entropy measures the sharpness of distribution pe (-1g, d, JDy). ) experiments, the average entropies for with R and w/o R in AR2-G are 2.10 and 1.70 respectively. This indicates that the regularization term could help smooth the prediction of probabilities in the retriever.
Visualization of the Training Procedure: We visualize the changes of R R@5 during the AR2- G training. The result is shown in Figure 3 We see that as adversarial iteration increases, the R@5 of both AR2-retriever and AR2-ranker also gradually increases. AR2-retriever has the most significant improvement (about 4.5%) after the first iteration. While the training advances closer to the convergence, the improvement of @@5 also gradually slows down. In the end, AR2-retriever is improved by approximately 8% and AR2-ranker is improved by approximately 3%.
Adversarial Training versus Iterative Hard-Negative Sampling: To give a fair comparison of AR2 and ANCE (Xiong et al. 2021), we retrain the ANCE model by initializing it with the same warm-up AR2-G" G which leverages the advantage of the continuous pre- pre-training technique. In ex- periments, ANCE trains the retriever with an iterative hard-negative sampling approach instead of adversarial training AR2. In Figure [ we observe that AR2 steadily outperforms ANCE during training terms of R @5 and R @ 10 evaluation metrics with the same that AR2 is a superior training framework compared with ANCE.
It shows
Performance of the Pipeline: We evaluate the performance of the retrieve--teen-rak pipeline on NQ dataset. The results are shown in Table 6 GAR is a sparse retriever which ensembles GAR (Mao et al.) 2021a) and DPR (Karpukhin et al. 2020). BERT (Nogueira & Cho 2019), BART (Nogueira et al. 2020), and RIDER (Mao et al. 2021b) are three ranking methods. BERT ranker is cross-encoder, which makes binary relevance decision for each query-passage pair.
8

---

Published as a conference paper at ICLR 2022
BART ranker generates relevance labels target tokens in seq2seq manner. RIDER re-ranks the retrieved passages based on the lexical overlap with the top predicted answers from the reader. The results show that AR2 pipeline significantly outperforms existing public pipelines.
RELATED WORK
Text retrieval aims find related documents from large corpus given query. Retrieval-then-rank is the widely used pipeline (Huang et al. 2020 Zou et al. (021).
Text Retrieval:
For the first stage retrieval, early researchers used sparse vector space models, e.g., BM25 (Yang et al. 2017p. Recently, some works improve the traditional sparse retriever with neural network, e.g.. Dai & Callan (2019) use BERT dynamically generate term weights, doc2Query (Nogueira) et al. 2019b), docTTTTTQery (Nogueira et al., (019a), and GAR (Mao et al.) 2021a) use text generation to expand queries or documents to make better use of BM25.
Recently, dense retrieval methods have become a new paradigm for the first stage of retrieval. Var- ious methods have been proposed enhance dense retrieval, c.g., DPR (Karpukhin al. 2020)
and ME-BERT (Luan et al.
202T) use in-batch negatives and construct hard negatives by BM25;
ANCE (Xiong et al.j [021]. RocketQA (Qu et al., 202I), and ADORE (Zhan al., (202]) im-
prove the hard negative sampling by iterative replacement, denoising, and dynamic sampling, re- spectively; PAIR (Ren et al.
2021 leverages passage-centric similarity relation into training object; FID-KD (Izacard & Grave 2020 and RDR (Yang & Seo 2020) distill knowledge from reader to retriever; Guu et al. (2020) and Sachan et al. (2021b) enhance retriever by jointly training with
downstream tasks. Some researchers focus the pre-training of dense retrieval, such as ICT (Lee
et al. 2019). Condenser (Gao & Callan 2021b) and Cocondenser (Gao & Callan 2021
For the second stage ranking, previous works typically use cross-encoder based methods. The cross- encoder models which capture the token-level interactions between the query and the document (Guo)
al. 2016 Xiong al. (0077). have shown be more effective. Various methods are proposed to enhance ranker, e.g., NNgueirr & Cho (2019) use BERT to make binary relevance decision for each query-passage pair; Nogueira et al. (2020) adopt BART to generate relevance labels as target tokens in a seg2seq manner, Khattab & Zaharia (2000) and Gao et al. 2020) adopt the lightweight interaction based on the representations of dense retrievers reduce computation. However, nega- tive samples are statically sampled in these works. In AR2, negative samples for training the ranker will be dynamically adjusted with the progressive retriever.
Gao
Generative Adversarial Nets: Generative Adversarial Nets (Goodfellow et al. 2014) have been widely studied in the generation field, i.e., image generation (Brock al. 2018 and text genera- tion Yuu et al. (017). With a minimax game, GAN aims to train generative model to fit the real data distribution under the guidance of a discriminative model. Few works study GAN to text re- trieval. A related work is IRGAN (Wang et al. 2017). It proposes minimax retrieval framework that aims to unify the generative and discriminative retrieval models.
CONCLUSION
In this paper, we introduce AR2, an adversarial retriever- ranker framework to jointly train the end-to- end retrieve-then-ramee pipeline. In AR2, the retriever retrieves hard negatives cheat the ranker, and the ranker learns to rank the collection of positives and hard negatives while providing progressive rewards to the retriever. AR2 can iteratively improve the performance of both the retriever and the ranker because (1) the retriever is guided by the progressive ranker; (2) the ranker learns better through the harder negatives sampled by the progressive retriever. AR2 achieves new state-of-the-a performance on all three competitive benchmarks.
Acknowledgement
This work is supported by the National Natural Science Fund for Distinguished Young Scholar (Grant No. 61625204), and partially supported by the Key Program of National Science Foundation of China (Grant No. 61836006).

---

Published as a conference paper at ICLR 2022
REFERENCES
Dan Brickley, Matthew Burgess, and Natasha Noy. Google dataset search: Building search engine for datasets in an open web ecosystem. In WWW, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2018
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. In ACI., 2017.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604.06174 2016.
Zhuyun Dai and Jamie Callan. Deeper text understanding for ir with contextual neural language modeling. In SIGIR, 2019.
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05544 2021a.
Luyu Gao and Jamie Callan. Is your language model ready for dense representation fine-tuning? arXiv preprint arXiv:2104.0823, 2021b.
Luyu Gao, Zhuyun Dai, and Jamie Callan. Modularized transfomer-based ranking framework. In EMNLP, 2020.
Luyu Gao, Zhuyun Dai, and Jamie Callan. COIL: revisit exact lexical match in information retrieval with contextualized inverted list. In NAACL-HLT, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
Daya Guo. Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. Dialog-to ttalog-o-acction: Conversational question answering over sarge-scale knowledge base. 2018.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009. 08366, 2020.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In CIKM, 2016.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In ICML, 2020.
Shuguang Han, Xuanhui Wang. Mike Bendersky, and Mare Najork. Learning-to-rank with bert in tf-ranking. arXiv preprint (tl:20044.84766 2020.
Linmei Hu, Siyong Xu, Chen Li, Cheng Yang, Chuan Shi, Nan Duan, Xing Xie, and Ming Zhou. Graph neural news recommendation with unsupervised preference disentanglement. In ACL, 2020.
Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padman- abhan, Giuseppe Ottaviano, and Linjun Yang. Embedding- retrieval in facebook search. In KDD, 2020.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In CIKM, 2013.
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:20/2.04584 2020.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scaa similarity search with gpus. JEEE Transactions Big Data, 2019.
10

---

Published as a conference paper at ICLR 2022
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, 2020.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction bert. In SIGIR, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wor Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-460 2019.
Kenton Lee, Ming-Wor Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In ACL, 2019.
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. Rikinet: Reading wikipedia pages for natural question answering. In ACL, 2020.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329-345, 2021.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, Cambridge, UK 2008. ISBN 978-0-521-86571-5..
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation-augmente retrieval for open-domain question answering. In ACL, 2021a.
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Reader- guided passage reranking for open-domain question answering. In Findings of ACL/JCCLP,, 2021b.
Bhaskar Mitra and Nick Craswell. Neural models for information retrieval. CoRR, abs/aaa5.... 10ss/100501509 2017. URL
1170. 01009.5..55
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS, 2016.
Rodrigo Nogueira and Kyunghyun Cho. arXiv:1901.04085, 2019.
Passage re-ranking with bert.
arXiv preprint
Rodrigo Nogueira, Jimmy Lin, and Al Epistemic. From doc2query to doctttttquery. Online preprint, 2019a.
Rodrigo Nogueira, Wei Yang. Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arXiv preprini 1rXXiv:9004.00 2019b.
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence- to-sequence model. arXiv preprint wrXiv:2003.0671: 2020.
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Pik- tus. Xilun Chen. Sebastian Riedel, Wen-tau Yih, Sonal Gupta, et al. Domain-matosn matched pre-training tasks for dense retrieval. arXiv preprint arXiv:2107.13603 2021.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In NAACL-HITT, 2021.

---

Published as a conference paper at ICLR 2022
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021.
Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. PAIR: leveraging massage-centrng similarity relation for improving dense passage retrieval. In Findings of ACL/ICCLL, 2021.
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping. William L. Hamilton, and Bryan Catanzaro. End-to-end training of neural retrievers for open-domain ques- tion answering. In ACL/JCNLP, 2021a.
Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. End-to- end training of multi-document reader and retriever for open-domain question answering. arXiv preprint
2021b.
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. Learning semantic rep- resentations using convolutional neural networks for web search. In WWW, 2014.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernic 2.0: A continual pre-training framework for language understanding. In AAAI, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. Irgan: A minimax game for unifying generative and discriminative information retrieval models. In SIGIR, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In EMNLPP, 2020.
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. End-to-end neural ad-hoc ranking with kernel pooling. In SIGIR, 2017.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In ICLR, 2021.
Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Enabling the use of lucene for information retrieval research. In SIGIR, 2017.
Sohce Yang and Minjoon Sco. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010. 10999, 2020.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. Repbert: Contextualized text embeddings for first-stage retrieval. arXiv preprint arXiv:2006. /5498, 2020.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. In SIGIR, 2021.
Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. Pre-trained language model based ranking in baidu search. In KDD, 2021.
12

---

Published as a conference paper at ICLR 2022
A
APPENDIX
A. PROOF
Proof of Eqn. 9. Suppose di E Dy is sampled by pe(-1q; 1. ). thus
j E D. Galerr
[logoo U UD
(13)
B GGlla))
B (p [logo fd, d.
where mp, indicates the set of negative documents sampled by Go(a,) ) In practice, we approximate Dy by sampling documents from the top-K retrieved negative set. Therefore, could further obtain the following approximately equation in implementation.
E a, -po(ll007)) [logp/[dls {d, d,71]] =
(14)
Proof of Eqn. 10:
Vojo Ng E 47 - (dog (d, dz })]
Vopo l0: D,7)) llogs {d, d; })]
(15)
Σ po 1q: D, Vo log po d7 lq: D,7) logpo (d, D)
Vo
Ed ~pall1933,
Ve log (d, lg: (37)) [log logoo(dla: (d,
A.2 EFFICIENCY REPORT
list the time cost training and inference Table 7 The evaluation is made with NVIDIA A100 GPUs. The max step of ANCE training is from the ANCE's open-source website F We esti- mate the overall training time without taking account the time of continuous pre-training step and warming- up step.
Table 7: Comparison of Efficiency
DPR
ANCE
AR2(n=15)
AR2(n=1)
Training Batch Size
128
128
64
64
Max Step
20k
136k
20k
20k
BP for Retriever
1.8h
11h
2.3h
1h
BP for Ranker
0.75h
0.35h
Iteration Number
10
10
10
Index Refresh
0.5
0.5h
0.5h
0.5h
Overall
1.85h
16h
9.1h
6.4h
Inference
Encoding of Corpus 20min
20min
20min
20min
Query Encoding
40ns
40ns
40ns
40ns
ANN Index Build
2min
2min
2min
2min
ANN Retrieval(Top-100)
2ms
2ms
2ms
2ms
13

---

Published as a conference paper at ICLR 2022
A.3 HYPERPARAMETERS
Table 8: Hyperparameters for AR2 training.
Parameter
NQ
TriviaQA
MS-MARCO
Max query length
32
32
32
Default
Max passage length
128
128
128
Learning rate
le-5
le-5
le-4
Negative size
255
255
127
Batch size
128
128
64
Temperature
1
1
AR2-G
Optimizer
AdamW Linear
A
Adam W
Scheduler
Linear
Linear
Warmup proportion
0.1
0.1
0.1
Training epoch
40
40
3
Learning rate
le-5
le-5
le-5
Negative size
15
15
15
Batch size
64
64
256
Temperature T
1
AR2-Do
Optimizer
AdamW
A
A
Scheduler
Linear
Linear
Linear
Warmup proportion
0.1
0.1
0.1
Training step per iteration
1500
1500
1500
Max step
2000
2000
4000
Learning rate
le-5
1e-5
5e-6
Negative size
15
15
15
Batch size
64
64
64
Temperature т
Optimizer
AdamW
A
A
AR2-G
Scheduler
Linear
Linear
Linear
Warmup proportion
0.1
0.1
0.1
Training step per iteration
1500
1500
1500
Max step
15000
15000
15000
Negative size
15
15
15
Learning rate
le-6
1c-6
5e-7
Batch size
64
64
64
Temperature т
1
AR2-D
Optimizer
AdamW Linear
AdamW
AdamW
Scheduler
Linear
Linear
Warmup proportion
0.1
0.1
0.1
Training step per iteration
500
500
500
Max step
5000
5000
5000
14

---

Published as a conference paper at ICLR 2022
A.4 MODEL CONHIGURATION AND EXPERIMENT SETTINGS
We list the detailed configuration of AR2 and baseline models in Table 9
Table 9: Model configuration and experiment settings.
Further Pretrain
Additional Data
Model
Initial Model
Parameters
DPR (Karpukhin et al. 2020
BERT-Base
110M
ANCE Xtong al. 2021)
BERT/RoBERTa-BBa
110M/125M
ERNIE 2.0 Base
110M
RocketQA Qu al.
2021
1.7 M
ERNIEE2.00-aarg
330M
ERNIE-2.00 2.0--Bse
110M
PATR (Ren et al.
2021
1.7 M
ERNIF-2.0-L.arr Large
330M
ERNIE-2.0 2.0---2..BBass.
110M
Yes
Individual Top-k Sachan al.
2021a
TS-Large
739M
coCondenser (Gao Callan 2021a
BERT-Base
110M
Yes
Our (AR2-G) (Retriever)
ERNIF:-2.0-Baa
110M
Yes
Our (AR2-D) (Ranker)
ERNIE-2.0-Large
330M
A.5 ABLATION STUDY ON DIFFERENT INITIAL MODELS
Table I0 shows the results of our method with different initial models. We see that ERNIE-BAR -Base the initial model achieves little better performance than BERT-Base. And AR2-G using BERT- Base as the initial model still achieves better performance than other methods under the same initial model. Meanwhile, ICT pre-training improves the performance of AR2-G.
Table 10: Performance of AR2-G NQ test with different initial model
Initial Model
R@D
R@5
@ 20
R@ 100
DPR (Karpukhin et al. 2020)
BERT-Base
78.4
85.3
ANCE (Xiong 2021
BERT-Base
81.9
87.5
RocketQA (Qu et al. 2021
ERNIE-Bass -Base
74.0
82.7
88.5
PAIR (Ren 2021
ERNIE- Bass
74.9
83.5
89.1
AR2-G
BERT-BI Base
56.7
76.1
85.0
89.3
AR2-G
ERNIE-Baae
57.2
76.6
85.3
89.8
AR2-G
ERNIE-Baad w/ ICT
58.7
77.9
86.0
90.1
15

---

Published as a conference paper at ICLR 2022
A.6 COMPARISON WITH SEVERAL EXISTING APPROACHES
Table [ shows the comparison of AR2 and several existing retrieval approaches. "Extra Label" refers to whether the answer label is used. AR2 jointly optimizes both the retriever and the ranker
according to a principle adversarial objective, which is the key difference with previous works.
Table 11: Comparison with existing approaches
Surriever-Ranker Retriever- aetriever-Reader
Adversarial Objective
Update
Model
Extra Label
Hard Negatives
FID-KD (Izacard Grave
2020
Yes
Yes
No
No
RDR (Yang & Seo 2020
Yes
Yes
RockeiQA (Qu el al.
2021
No
Yes
No
Yes
ANCE (Xiong al. 2021
No
No
No
Yes
RIDER (Mao et al.
2021b
Yes
Yes
No
No
AR2
No
Yes
Yes
Yes
A.7 PERFORMANCE OF THE PIPELINE
Table 12 shows the performance of the retrieve-then-rane pipeline on Trivia QA and MS-MARCO. From the results of Table 6 and Table 12 find that the ranker AR2-D improves the performance on all three benchmarks including NQ. Trivia QA, and MS-MARCO. Meanwhile, the pipeline based on AR2 achieves state-of-the the-ar art performances on all benchmarks.
Table The results of the second-stage ranking on Trivia QA and MS-MARCO.
Trivia QA
MS-MARCO
Retriever
Ranker
R@1
R@5
R@10
R@20
MRR@ 10
RepBERT Zhan 2020
RepBERT (Zhan al. 2020
37.7
ME-HYBIRD Luan at.
2021
ME-HYBIRD ( [Lann al. 2021
39.4
ME-BERT LLan et a. 2021
ME-BERT LLun ct al. 2021
39.5
BM25
Yane et al. 2017
TFR-BERT [Han et 2020
40.5
GAR
MMo et al. 2021a
RIDER MMa ct al. 2021b
71.9
77.5
79.8
81.8
AR2-G
64.2
78.2
81.8
84.4
39.5
AR2-G
AR2-D
84.1
43.2
A.8 PERFORMANCE OF THE LARGE-SIZE MODEL
Table 13 shows the results of AR2-G (Retriever) initialized with ERNIE-2.00-arge (without tinuous pre-training (ICT)). All baselines are initialized by large-size model, and DPR-PAQ (Oguz al. 2021) utilizes large external corpus (65m question-answer pairs) continue pre-training the model: Individual Top-K (Sachan et al., 2021a) utilizes T5-Large model (739M parameters vs 330M parameters ERNIE-2.0-Large) as reader guide the retriever. Compared with these baseline methods, AR2-G achieves a significant performance improvement, which further demonstrates the effectiveness of AR2-G (Retriever).
Table 13: The performance large-size models Natural Questions test set,
Size
@@D
R@5
R @ 20
R@ 100
DPR- OPR-PAQBEKT Oguz et al. 2021
Large
75.3
84.4
88.9
DPR-PAQReBERT, Oguz et al. 2021
Large
76.9
84.7
89,2
Individual Top-K (Schann et all
2021a)
Large
57.5
76.2
84.8
89.8
AR2-G
Base
58.7
77.9
86.0
90.1
AR2-G
Large
61.1
78.8
86.5
90.4
16


Reviews:
[{"review_id": "RVDbedJ9HB1", "reviewer": "Reviewer_hfGT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* The description of the model is generally well-written and is easy to follow.\n* Experiments are comprehensive --- evaluated on three well-studied datasets, compared with a range of recent prior work, and analysis showing impact of core components like iterative training, effect of regularization and the effect of number of negatives. The empirical improvements over a range of competitive baselines are also very impressive.\n\nWeaknesses:\n* My most critical concern is that there are many components in the paper that are very similar or are already introduced in prior published work, and were not properly credited nor compared.\n     * Adversarially and iteratively obtaining hard negatives, which is the core idea of the paper, is explored in prior work including Xiong et al, Oguz et al and Qu et al.\n        * The paper seems to be aware of some of these works (briefly mentioned in Section 2). However, their differences are not discussed.\n        * I can tell one difference is whether the negatives are coming from its own model (dual encoder) or a separate cross-encoder model, but to me it is a small difference and should have been discussed.\n        * Empirical comparison is not provided either - the paper compared with numbers taken from the original papers, but they use different base models so the comparison is unfair.\n        * Moreover, some descriptions of prior work in Section 2 are incorrect or at least unclear --- for example, Section 2 says prior work takes negatives from the last checkpoint, indicating a one-time update. However, prior work including Xiong et al and Oguz et al clearly describe that they iteratively choose negatives, as the proposed model in this paper does.\n    * Distillation from the ranker to the retriever is identical to methods in prior work including Izacard & Grave and Yang & Seo, and are not cited nor mentioned in the paper.\n* The base model used in the paper is ERNIE 2.0 with Inverse Cloze Task and coCondenser, while most prior work uses BERT base. This makes it hard to compare the performance with prior work in a fair manner. (Also, is there any justification for using a different base model for different datasets? Are the decisions made based on the result on the validation data?)\n* Given that the base model is different from prior work, one important ablation that is missing in the paper is comparison to the method that iteratively chooses negatives from its own dense model, as in Xiong et al.\n\nMinor comments:\n* In Implementation details, it is worth mentioning which text corpus is used for retrieval.\n\n\nReferences:\n- Xiong et al. ICLR 2021. https://arxiv.org/pdf/2007.00808.pdf\n- Oguz et al. 2021. https://arxiv.org/pdf/2012.14610.pdf \n- Qu et al. NAACL 2021. https://arxiv.org/pdf/2010.08191.pdf\n- Izacard & Grave. ICLR 2021. https://openreview.net/forum?id=NTEz-6wysdb \n\n", "summary_of_the_paper": "This paper proposes an adversarial method to jointly train the dual encoder retriever and the cross-attention ranker. The key idea is to obtain harder negatives for the retriever by training a separate ranker with higher capacity and taking passages that the model is confused with. It also proposes a technique like distillation from the ranker to the retriever. Experiments on three datasets demonstrate improvements over a range of baselines.", "summary_of_the_review": "To summarize, this paper includes a thorough exploration of better training of dual encoder model using harder negatives chosen in an interactive manner by training a separate, cross-attention model. There has been a number of prior work exploring similar approaches (which I pointed out that comparisons/discussions are not sufficiently provided), but this work is unique in using a cross-attention model for choosing negatives instead of choosing from its own dense retrieval.\n\nThere are a few critical concerns: (1) discussions of/comparisons to prior work, (2) justification of using a different base model (as it makes the whole comparison with baselines unfair), and (3) empirical comparison to prior work that uses iterative negatives --- this is an important ablation given that this is a core distinction from prior work as well as the main claim in the paper. They may be relatively easy to be added in the rebuttal so I'd love to increase the score based on the author responses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Ii3aMrAQmiD", "reviewer": "Reviewer_idsf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper proposed a (novel) approach for multi-stage training of a retriever and ranker model in an adversarial manner. To the best of my knowledge, previous approaches using a retriever and a ranker were trained in a pipelined manner but not using the minimax adversarial objective. The adversarial training of the model makes the proposed model different from the previous work.\n\n2. The model outperforms a number of recent baselines on three benchmark text retrieval tasks. As these baselines are highly competitive, I believe that the improvements are significant.\n\nWeaknesses:\n\n1. The paper does not discuss the reasons for selecting the large configuration for the ranker and base configuration for the retriever. In my opinion, the large configuration of the ranker contributes a lot to the performance gains. Most of the previous approaches in their Table 2 consist of just the base configuration of models. \nThe large configurations has an increased number of layers due to which the representational capability of the ranker increases and thus  comparisons with the base models is not fair. \nIt would be nice if in the rebuttal the authors can disentangle the effect of configuration choice. For example, having two results tables or sections with just the base configuration and large configuration and discuss their results. For large configurations, the authors should then compare with the respective results of the baseline models.\n\n2. The authors should report the total number of parameters in the model the in the baseline models. This will help in better understanding of the pros and cons of the different approaches. I believe that this should be easy to do.\n\n3. There have been models proposed in previous work that consist of the retriever and ranker components. While it is great that the authors have cited most of the previous work, they should explain in detail the difference between their work and such previous work. For example, there can be a table which contains salient aspects of different models.\n\nQuestions for the authors: Can you include the results of these new experiments in the author's response?\n\n1. Results when just initializing the retriever with either ERNIE or ICT and not applying DPR on them? This will shed light on how much the retriever is robust to initialization parameters. What's the rationale for initializing the model with ERNIE rather than with BERT?\n\n2. How is the D_q^(-) computed for training the retriever? Is it done in the same manner as for the reranker? What happens if we take all the documents in top-100 and don't sample from them?\n\n3. Would it be possible to perform end-to-end training in a single stage instead of the multi-stage step-wise training and then compare their results?\n\n4. I feel that with the current description of the model, it is hard to interpret what the retriever has learned. The weight update step looks similar to the REINFORCE algorithm. Is the retriever training sensitive to hyperparameter choices ?", "summary_of_the_paper": "This paper proposes a minimax multi-stage iterative method for document retrieval. The model consists of two components: dense retriever and ranker. The dense retriever is modeled using a dual-encoder and the ranker is modeled with a cross-encoder, both of which are fairly standard. In their proposed approach, in the first step, the ranker is trained using a contrastive loss defined over the top-K documents from the pre-trained retriever. In the second step, the retriever is trained using an objective similar to REINFORCE to maximize the likelihood of the negative documents. Such stepwise training forces the ranker to improve its predictions because the adversarial training of the retriever leads to more hard-negatives in the top-K documents. The authors perform experiments on three widely used datasets for ranking tasks and compare with a number of recent baselines. Their results show good gains in retrieval recall over the prior models, which can be significant. The authors also include ablation studies to better understand their proposed approach.", "summary_of_the_review": "Summarizing from above:\n\nThe proposed approach has novel aspects that is also demonstrated by its state-of-the-art results on benchmark IR tasks. On the other hand, I expect the authors to conduct more comprehensive experiments to facilitate fair comparisons with baselines, which I have also mentioned under weakness above. The authors should uniformly select the model configurations and compare with the respective baselines from the literature. Other than this, I have several questions regarding the model initialization and training process.\nAnswering them will also help the readers develop a more coherent understanding of the paper.\n\nDepending on the author's response, I am open to reconsidering my scores.\n\n[Update]\nI want to thank the authors for doing additional experiments and answering the questions raised in the reviews. I am happy with the inclusion of the new results and so I am increasing my scores to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G0O-VDI9GFY", "reviewer": "Reviewer_dE9M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a new method to improve deep retrieval performance. The key idea is to use the ranker to help retriever training. The key novelty of this paper is to formulate the problem as an adversarial training procedure. After some approximations, the framework essentially use the ranker as a reward function and a teacher model to distill from. Experiments show better retrieval performance than existing methods on some popular benchmark datasets.\n\nStrength:\n- The reviewer feels the adversarial formulation is interesting and intuitive.\n- The retrieval performance is good and seem to achieve state-of-art numbers.\n\nWeakness:\n- The major concern of this paper is the comprehensiveness of final ranking performance. Though the retrieval numbers are good, it is not clear why the numbers on ranking tasks are significantly lacking. Specifically, only recall numbers on the NQ dataset is reported. Please report reranking numbers with more metrics (such as MRR) on all datasets. For example, in the RocketQA paper, they explicitly focus on the retrieval but still report final ranking performance combined with different re-rankers. This paper focuses a lot of \"retriever-reranker\" joint optimization, so the authors should either 1) Make the contribution clear that this paper mainly focus on retriever. But that will limit the contribution of this paper. Also it is not clear why the \"retriever-reranker\" joint optimization cannot achieve state-of-art ranking performance. If that is the case, the impact of the retriever is not clear in real-world applications. 2) Or stick to the joint optimization framework, in which case user-facing ranking performance is arguably more important than retrieval performance anyway.\n- Novelty may not be very significant. Using ranker to help retriever is not a new idea. This paper's main contribution is a more principled loss. The significance of contribution is subjective and the reviewer is generally ok with it. But still feel it needs support from more comprehensive evaluations to meet the bar of top-tier venues.\n\n ", "summary_of_the_paper": "This paper proposes to use adversarial loss and distillation loss to improve dense retriever performance from the help of the ranker. Experiments show better retrieval performance than existing methods. ", "summary_of_the_review": "The paper proposes a new loss function to leverage ranker to help retriever for text retrieval tasks. Though the idea is interesting, the evaluations now lack important components so the significance of this work is not very convincing. The authors are suggested to provide more details in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EC1evNKXtmU", "reviewer": "Reviewer_ySzP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Compared to the most related work (IRGAN), the main differences are (1) the use of a contrastive loss rather than a cross-entropic one, (2) a different pre-training, inspired by most recent works, (3) regularization through distillation (KL between retriever and ranker), (4) a different sampling procedure. These differences taken together are substantial, and bring a better performance. I think the paper does a good job at showing that all the propositions are complementary.\n\nThe section 3 describes the training procedure. I found the notation $p(d|q,d..)$ a bit misleading because $d$ appears on both side, but besides this, the section reads easily. \n\nExperiments (section 4) are conducted on the MS Marco, Trivia QA, and Natural Questions, showing that the retriever performs better that state-of-the-art baselines. The analysis section is quite interesting, giving insight on various aspects of the model/framework (including the regularization procedure, the improvement brought by the adversarial optimization, and an interesting comparison with other related models e.g. IRGAN and ANCE). It succeeds in showing that all the presented techniques, including the main proposition (adversarial training procedure) do indeed allows the model to perform better than other propositions.\n\nThe experiments conclude with results for the full pipeline in the case of Natural Questions. It would have been interesting to look the results for the other collections (MS Marco and Trivia QA) - especially for MS Marco, since most models are evaluated on the full pipeline.\n\nThe appendix A.3 gives all the hyperparameter settings which is important for reproducibility. \n\nOther remarks\n\n- the English should be revised in many places (improper use of word “it”, “on the other hand” 3rd paragraph 2nd page)\n- Please be clear when you compare your models with 1st stage rankers (e.g. table 2) - since looking at the results from the papers \n- The notation $p(d|q,d,\\mathbb D^-_q)$ is a bit weird: use a different notation for the first $d$\n- Appendix A.1: the first inequality of eq. (13) is not obvious unless the probability is computed through a softmax. The next « approximately equal » is not really proven (is it really approximately equal since documents are independent ?)", "summary_of_the_paper": "This paper proposes to learn a retriever (to find relevant documents in a dataset / dual encoder) and a ranker (to re-rank retrieved documents / cross-encoder) using an adversarial framework (the retriever trying to fool the ranker). This training allows to reach state-of-the-art performance for both the retriever and the ranker. The paper leverages all the recent techniques of neural retrieval models (pre-training, teacher training) and paper mostly focus on the training of the 1st stage (retriever), but also presents result on the full pipeline too (the 2nd stage is more standard, so it is normal the authors did not focus on this part).\n\nOverall, this is a good paper that is inspired by many recent techniques on neural retrievers, bringing back the IRGAN adversarial training but with more success. The experiments are well designed and the analysis interesting. ", "summary_of_the_review": "The paper is an interesting proposition to train retriever-reranker couple for neural information retrieval within a single adversarial framework. The paper demonstrates state-of-the-art results and provide an interesting analysis of the model. There are some minor problems with the paper, but overall I recommend acception. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method for jointly training a dense bi-encoder retriever with a cross-encoder ranker. More precisely, the proposed method is iteratively training the retriever and the ranker, using an objective function inspired by adversarial training. In addition, the authors propose to use a distillation loss from the ranker to the retriever as a regularization term. The proposed method, called AR2, is evaluated on three retrieval benchmarks from question answering: NaturalQuestions, TriviaQA and MS-MARCO. The method obtains state-of-the-art retrieval performance on these three datasets.

Overall, the reviewers agree that the strong performance obtained by the proposed method is a strength of the paper. Regarding novelty, some reviewers argue that the method is a combination of existing techniques, hence lacking novelty, while the others believe that combining these different techniques is novel enough. Regarding the experimental section, some concerns were raised about comparisons with previous work (eg, BERT vs ERNIE) or the fact that it was a bit hard to determine where the improvements come from. I believe that these concerns were well addressed by the authors, and I tend to believe that combining existing techniques to obtain a strong system is novel enough. I thus lean towards accepting this paper to the ICLR conference.
Decision: Accept (Poster)