Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an adversarial method to jointly train the dual encoder retriever and the cross-attention ranker. The key idea is to obtain harder negatives for the retriever by training a separate ranker with higher capacity and taking passages that the model is confused with. It also proposes a technique like distillation from the ranker to the retriever. Experiments on three datasets demonstrate improvements over a range of baselines.
main_review: Strengths
* The description of the model is generally well-written and is easy to follow.
* Experiments are comprehensive --- evaluated on three well-studied datasets, compared with a range of recent prior work, and analysis showing impact of core components like iterative training, effect of regularization and the effect of number of negatives. The empirical improvements over a range of competitive baselines are also very impressive.

Weaknesses:
* My most critical concern is that there are many components in the paper that are very similar or are already introduced in prior published work, and were not properly credited nor compared.
     * Adversarially and iteratively obtaining hard negatives, which is the core idea of the paper, is explored in prior work including Xiong et al, Oguz et al and Qu et al.
        * The paper seems to be aware of some of these works (briefly mentioned in Section 2). However, their differences are not discussed.
        * I can tell one difference is whether the negatives are coming from its own model (dual encoder) or a separate cross-encoder model, but to me it is a small difference and should have been discussed.
        * Empirical comparison is not provided either - the paper compared with numbers taken from the original papers, but they use different base models so the comparison is unfair.
        * Moreover, some descriptions of prior work in Section 2 are incorrect or at least unclear --- for example, Section 2 says prior work takes negatives from the last checkpoint, indicating a one-time update. However, prior work including Xiong et al and Oguz et al clearly describe that they iteratively choose negatives, as the proposed model in this paper does.
    * Distillation from the ranker to the retriever is identical to methods in prior work including Izacard & Grave and Yang & Seo, and are not cited nor mentioned in the paper.
* The base model used in the paper is ERNIE 2.0 with Inverse Cloze Task and coCondenser, while most prior work uses BERT base. This makes it hard to compare the performance with prior work in a fair manner. (Also, is there any justification for using a different base model for different datasets? Are the decisions made based on the result on the validation data?)
* Given that the base model is different from prior work, one important ablation that is missing in the paper is comparison to the method that iteratively chooses negatives from its own dense model, as in Xiong et al.

Minor comments:
* In Implementation details, it is worth mentioning which text corpus is used for retrieval.


References:
- Xiong et al. ICLR 2021. https://arxiv.org/pdf/2007.00808.pdf
- Oguz et al. 2021. https://arxiv.org/pdf/2012.14610.pdf 
- Qu et al. NAACL 2021. https://arxiv.org/pdf/2010.08191.pdf
- Izacard & Grave. ICLR 2021. https://openreview.net/forum?id=NTEz-6wysdb 


summary_of_the_review: To summarize, this paper includes a thorough exploration of better training of dual encoder model using harder negatives chosen in an interactive manner by training a separate, cross-attention model. There has been a number of prior work exploring similar approaches (which I pointed out that comparisons/discussions are not sufficiently provided), but this work is unique in using a cross-attention model for choosing negatives instead of choosing from its own dense retrieval.

There are a few critical concerns: (1) discussions of/comparisons to prior work, (2) justification of using a different base model (as it makes the whole comparison with baselines unfair), and (3) empirical comparison to prior work that uses iterative negatives --- this is an important ablation given that this is a core distinction from prior work as well as the main claim in the paper. They may be relatively easy to be added in the rebuttal so I'd love to increase the score based on the author responses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to learn a retriever (to find relevant documents in a dataset / dual encoder) and a ranker (to re-rank retrieved documents / cross-encoder) using an adversarial framework (the retriever trying to fool the ranker). This training allows to reach state-of-the-art performance for both the retriever and the ranker. The paper leverages all the recent techniques of neural retrieval models (pre-training, teacher training) and paper mostly focus on the training of the 1st stage (retriever), but also presents result on the full pipeline too (the 2nd stage is more standard, so it is normal the authors did not focus on this part).

Overall, this is a good paper that is inspired by many recent techniques on neural retrievers, bringing back the IRGAN adversarial training but with more success. The experiments are well designed and the analysis interesting. 
main_review: Compared to the most related work (IRGAN), the main differences are (1) the use of a contrastive loss rather than a cross-entropic one, (2) a different pre-training, inspired by most recent works, (3) regularization through distillation (KL between retriever and ranker), (4) a different sampling procedure. These differences taken together are substantial, and bring a better performance. I think the paper does a good job at showing that all the propositions are complementary.

The section 3 describes the training procedure. I found the notation $p(d|q,d..)$ a bit misleading because $d$ appears on both side, but besides this, the section reads easily. 

Experiments (section 4) are conducted on the MS Marco, Trivia QA, and Natural Questions, showing that the retriever performs better that state-of-the-art baselines. The analysis section is quite interesting, giving insight on various aspects of the model/framework (including the regularization procedure, the improvement brought by the adversarial optimization, and an interesting comparison with other related models e.g. IRGAN and ANCE). It succeeds in showing that all the presented techniques, including the main proposition (adversarial training procedure) do indeed allows the model to perform better than other propositions.

The experiments conclude with results for the full pipeline in the case of Natural Questions. It would have been interesting to look the results for the other collections (MS Marco and Trivia QA) - especially for MS Marco, since most models are evaluated on the full pipeline.

The appendix A.3 gives all the hyperparameter settings which is important for reproducibility. 

Other remarks

- the English should be revised in many places (improper use of word “it”, “on the other hand” 3rd paragraph 2nd page)
- Please be clear when you compare your models with 1st stage rankers (e.g. table 2) - since looking at the results from the papers 
- The notation $p(d|q,d,\mathbb D^-_q)$ is a bit weird: use a different notation for the first $d$
- Appendix A.1: the first inequality of eq. (13) is not obvious unless the probability is computed through a softmax. The next « approximately equal » is not really proven (is it really approximately equal since documents are independent ?)
summary_of_the_review: The paper is an interesting proposition to train retriever-reranker couple for neural information retrieval within a single adversarial framework. The paper demonstrates state-of-the-art results and provide an interesting analysis of the model. There are some minor problems with the paper, but overall I recommend acception.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a minimax multi-stage iterative method for document retrieval. The model consists of two components: dense retriever and ranker. The dense retriever is modeled using a dual-encoder and the ranker is modeled with a cross-encoder, both of which are fairly standard. In their proposed approach, in the first step, the ranker is trained using a contrastive loss defined over the top-K documents from the pre-trained retriever. In the second step, the retriever is trained using an objective similar to REINFORCE to maximize the likelihood of the negative documents. Such stepwise training forces the ranker to improve its predictions because the adversarial training of the retriever leads to more hard-negatives in the top-K documents. The authors perform experiments on three widely used datasets for ranking tasks and compare with a number of recent baselines. Their results show good gains in retrieval recall over the prior models, which can be significant. The authors also include ablation studies to better understand their proposed approach.
main_review: Strengths:

1. The paper proposed a (novel) approach for multi-stage training of a retriever and ranker model in an adversarial manner. To the best of my knowledge, previous approaches using a retriever and a ranker were trained in a pipelined manner but not using the minimax adversarial objective. The adversarial training of the model makes the proposed model different from the previous work.

2. The model outperforms a number of recent baselines on three benchmark text retrieval tasks. As these baselines are highly competitive, I believe that the improvements are significant.

Weaknesses:

1. The paper does not discuss the reasons for selecting the large configuration for the ranker and base configuration for the retriever. In my opinion, the large configuration of the ranker contributes a lot to the performance gains. Most of the previous approaches in their Table 2 consist of just the base configuration of models. 
The large configurations has an increased number of layers due to which the representational capability of the ranker increases and thus  comparisons with the base models is not fair. 
It would be nice if in the rebuttal the authors can disentangle the effect of configuration choice. For example, having two results tables or sections with just the base configuration and large configuration and discuss their results. For large configurations, the authors should then compare with the respective results of the baseline models.

2. The authors should report the total number of parameters in the model the in the baseline models. This will help in better understanding of the pros and cons of the different approaches. I believe that this should be easy to do.

3. There have been models proposed in previous work that consist of the retriever and ranker components. While it is great that the authors have cited most of the previous work, they should explain in detail the difference between their work and such previous work. For example, there can be a table which contains salient aspects of different models.

Questions for the authors: Can you include the results of these new experiments in the author's response?

1. Results when just initializing the retriever with either ERNIE or ICT and not applying DPR on them? This will shed light on how much the retriever is robust to initialization parameters. What's the rationale for initializing the model with ERNIE rather than with BERT?

2. How is the D_q^(-) computed for training the retriever? Is it done in the same manner as for the reranker? What happens if we take all the documents in top-100 and don't sample from them?

3. Would it be possible to perform end-to-end training in a single stage instead of the multi-stage step-wise training and then compare their results?

4. I feel that with the current description of the model, it is hard to interpret what the retriever has learned. The weight update step looks similar to the REINFORCE algorithm. Is the retriever training sensitive to hyperparameter choices ?
summary_of_the_review: Summarizing from above:

The proposed approach has novel aspects that is also demonstrated by its state-of-the-art results on benchmark IR tasks. On the other hand, I expect the authors to conduct more comprehensive experiments to facilitate fair comparisons with baselines, which I have also mentioned under weakness above. The authors should uniformly select the model configurations and compare with the respective baselines from the literature. Other than this, I have several questions regarding the model initialization and training process.
Answering them will also help the readers develop a more coherent understanding of the paper.

Depending on the author's response, I am open to reconsidering my scores.

[Update]
I want to thank the authors for doing additional experiments and answering the questions raised in the reviews. I am happy with the inclusion of the new results and so I am increasing my scores to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use adversarial loss and distillation loss to improve dense retriever performance from the help of the ranker. Experiments show better retrieval performance than existing methods. 
main_review: This paper proposes a new method to improve deep retrieval performance. The key idea is to use the ranker to help retriever training. The key novelty of this paper is to formulate the problem as an adversarial training procedure. After some approximations, the framework essentially use the ranker as a reward function and a teacher model to distill from. Experiments show better retrieval performance than existing methods on some popular benchmark datasets.

Strength:
- The reviewer feels the adversarial formulation is interesting and intuitive.
- The retrieval performance is good and seem to achieve state-of-art numbers.

Weakness:
- The major concern of this paper is the comprehensiveness of final ranking performance. Though the retrieval numbers are good, it is not clear why the numbers on ranking tasks are significantly lacking. Specifically, only recall numbers on the NQ dataset is reported. Please report reranking numbers with more metrics (such as MRR) on all datasets. For example, in the RocketQA paper, they explicitly focus on the retrieval but still report final ranking performance combined with different re-rankers. This paper focuses a lot of "retriever-reranker" joint optimization, so the authors should either 1) Make the contribution clear that this paper mainly focus on retriever. But that will limit the contribution of this paper. Also it is not clear why the "retriever-reranker" joint optimization cannot achieve state-of-art ranking performance. If that is the case, the impact of the retriever is not clear in real-world applications. 2) Or stick to the joint optimization framework, in which case user-facing ranking performance is arguably more important than retrieval performance anyway.
- Novelty may not be very significant. Using ranker to help retriever is not a new idea. This paper's main contribution is a more principled loss. The significance of contribution is subjective and the reviewer is generally ok with it. But still feel it needs support from more comprehensive evaluations to meet the bar of top-tier venues.

 
summary_of_the_review: The paper proposes a new loss function to leverage ranker to help retriever for text retrieval tasks. Though the idea is interesting, the evaluations now lack important components so the significance of this work is not very convincing. The authors are suggested to provide more details in the rebuttal.
review_title: null

Meta-review: This paper introduces a new method for jointly training a dense bi-encoder retriever with a cross-encoder ranker. More precisely, the proposed method is iteratively training the retriever and the ranker, using an objective function inspired by adversarial training. In addition, the authors propose to use a distillation loss from the ranker to the retriever as a regularization term. The proposed method, called AR2, is evaluated on three retrieval benchmarks from question answering: NaturalQuestions, TriviaQA and MS-MARCO. The method obtains state-of-the-art retrieval performance on these three datasets.

Overall, the reviewers agree that the strong performance obtained by the proposed method is a strength of the paper. Regarding novelty, some reviewers argue that the method is a combination of existing techniques, hence lacking novelty, while the others believe that combining these different techniques is novel enough. Regarding the experimental section, some concerns were raised about comparisons with previous work (eg, BERT vs ERNIE) or the fact that it was a bit hard to determine where the improvements come from. I believe that these concerns were well addressed by the authors, and I tend to believe that combining existing techniques to obtain a strong system is novel enough. I thus lean towards accepting this paper to the ICLR conference.
Decision: Accept (Poster)