Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper explores how to effectively do negative sampling for dense retrieval. The paper shows that negatives sampled locally in batch are not informative, and proposes ANCE, a learning mechanism that selects hard training negatives globally from the entire corpus, using an asynchronously updated ANN index.

Strength:
1) Experiments show that the proposed method significantly outperforms state-of-the-art approaches such as DPR on MSMARCO, TRECDL, NaturalLanguages and TriviaQA.

Weakness:
1) The analysis on why in-batch local negatives are ineffective (in section 3) does not seem to be very insightful. Also, I did not see a big connection between this analysis and the negative sampling technique proposed in the following section.
2) The idea of maintaining a set of global negatives is not new, and refreshing index asynchronously has also been explored in [Guu et al. 2020]

Overall, I think this is a borderline paper. The experiments show improved performance over baseline methods, but the idea seems a bit incremental (a combination of existing tricks for training retrieval model).

review_title: Strong empirical results but idea seems incremental

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ##########################################################################
Summary:
Authors start from an assumption: “local negative sampling is the bottleneck of dense retrieval’s effectiveness”. To overcome this limitation, authors propose ANCE (Approximate nearest neighbour Negative Contrastive Estimation), a new contrastive representation learning mechanism for dense retrieval. The basic idea is that of constructing negatives exploiting the being trained deep retrieval module. The idea is that the model considers as negatives borderline cases. They also show, theoretically, that this improves the variance of the stochastic gradient estimation thus leading to faster convergence.

##########################################################################
Reasons for score: 
I honestly enjoyed reading the paper. It has a theoretical justification that explains the intuition of using hard negatives. Experiments are thorough and they show improvements over the state of the art. The discussion section is thorough. I believe that this research results are very important also in practice
 
 
##########################################################################
Pros: 
 
1. The paper addresses a timely and important problem
2. The paper gives a nice theoretical justification for the reasons why they have to use hard negatives
3. Experiments are thorough and nicely done. Results are very good. I particularly enjoyed seeing that both on public datasets and on real world search systems the novel retrieval mechanism helps greatly.
  
##########################################################################
Cons: 
 
1. The only one thing that I believe might impair the utilisation of this method is that you need to reconstruct the embeddings every m batches. It takes 10h every reconstruction and it is not clear what happens every m batches. Do you start a novel reconstruction? Do you replace the current embedding version with a new one as soon as one finishes training? This aspect, in my opinion, is the weakest of the paper and it would deserve more attention by the authors.
 
##########################################################################
Questions during rebuttal period: 
 
Please address and clarify the cons above 
 
#########################################################################
Some minor issues 
(1) In equation (2) I would call D^+ and D^-, D_q^+ and D_q^- in order to remark that negatives and positives are per-query.
(2) an negative —> a negative
(3) Citation Luan et al. —> It’s Toutanova not Toutanove

review_title: Mining Hard Negatives improves retrieval performance

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:
 
This paper studies the problem of dense text retrieval, which represents texts as dense vectors for approximate nearest neighbors (ANN) search. Dense text retrieval has two phases. The first phase learns a representation model to project semantically similar texts to vectors of large similarity scores (e.g. inner products or cosine similarity scores). The second phase adopts an ANN search algorithm to index these vectors and process queries. The paper claims key contributions at the first phase. Specifically, (1) The paper introduces a better negative sampling method to sample good dissimilar text pairs for training.  (2) The new method enables faster converge of model learning. (3) The new method leads to 100x faster efficiency than a BERT-based baseline, while achieving almost the same accuracy as the baseline. 

##########################################################################

Reasons for score: 
 
Overall, I like the idea of this paper and opt for a weak accept. A carefully designed negative sampling method should be able to outperform baselines that use simple heuristics. The efficiency improvement 100X is very promising. However, the paper can be better in experimental comparison and presentation. For experimental comparison, a stronger baseline using dense vectors should be included to strengthen the performance claim. For presentation, many important terms require clear definitions, without which the performance gain is not understandable. It will be good if the authors can address the above two issues in the rebuttal.
 
##########################################################################

Pros:

1. The paper proposes a novel negative sampling method. Based on the method, the paper proposes a new dense text retrieval framework ANCE. ANCE introduces an asynchronous index refresh to select the most dissimilar text pairs for training in a timely manner. 
 
2. The proposed ANCE achieves faster model training and equally accurate text retrieval when compared with a number of baselines. In a TREC 2019 task, ANCE achieves the best NDCG score against 11 baselines.  
 
3. The authors promise to make code open source. That will greatly improve the reproducibility of the work.  The code, together with its performance, will serve as a new state-of-the-art for future study. 

##########################################################################

Cons: 
 
1.	An important baseline is missing. In section 5, the paper describes Baselines. According to the descriptions, all baseline use BM25 to retrieve samples for training. BM25 may not be the best for a strong baseline since it relies on sparse word tokens. An alternative is to use BERT [CLS] dense vectors of all texts and a similarity search algorithm such as locality sensitive hashing as the retriever. It will be good if the authors can add the baseline to the paper. 

2. The paper does not explain clearly why the proposed method runs faster than baselines. The experimental results support that the proposed method outperforms several baselines. However, the paper does not explain the performance superiority. I am not sure about which of dissimilar text pairs selection or index refresher or others in the proposed negative sampling leads to the superiority. The reason for uncertainty may be due to the lacking of definitions in the paper. For example, “BERT rerank” refers to a baseline but it has not a definition in the paper. The input and output of “BERT rerank” remain unclear. Similarly, “TREC 2019” is an important benchmark but it has not definitions related to the inputs and outputs. It is necessary to explain important concepts for the best readability of the paper.
 
##########################################################################

Questions during rebuttal: 
 
I would like to see some experiments or discussions to clarify the above cons.
 
review_title: The work has values in better performance and open source of the proposed method, though it should justify further the performance gain. I would like to vote for a weak accept. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the problem of representation learning for first-stage retrieval in text ranking/matching. Specifically, it investigates the role of negative instances and how to select them in the quality of representations. The paper is well-written and it does a fair job in motivating the problem and discussing related works.

The main pros are as follows:

* Authors attempt to theoretically study the importance of negative instances and their impact on the gradients of objective function for text retrieval. Specifically, they posit that non-informative negative instances provide small gradients n the course of training, and thus adversely affect the convergence of gradient-based learnings.

* Through various experiments, the authors show that by exploiting the top retrieved texts from the corpus, using the current learned representations, as 'hard' negative instances, the performance of text ranking methods can be significantly improved.

There are however some ambiguities and room for improvement:

* As equation (10) suggests, the optimal distribution over negative instances is the one that minimizes the variance of gradients, or equivalently the norm of loss function. However, at many places it is pointed out that small losses prevent the model from learning and slow down the convergence rate. This is a kind of contradiction, and needs to be clarified,

* Existing results from literature such as equation (11) are leveraged to connect the norm of loss function and the gradient of last layer of representation learning models. However, the arguments here are not quite rigorous. It should be clarified that these results hold for what types of ranking objectives, and what other assumptions such as smoothness, etc. are required.

* The main bottleneck of using top retrieved documents as negative instances is the computational burden of updating the ANN indexing per batch. Therefore, the authors propose to perform this updating less frequently. It would be nice to have more comprehensive experiment to show that how sensitive is the performance vs the frequency of index updates.
review_title: ANCE for Text Retrieval

Meta-review: The paper explores how to effectively conduct negative sampling in learning for text retrieval. The paper shows that negative examples sampled locally are not informative, and proposes ANCE, a new learning mechanism that samples hard negative examples globally, using an asynchronously updated ANN index.

Pros • The problem studied is important. • Paper is generally clearly written. • Solid experimental results. • There is theoretical analysis.

Cons • The idea might not be so new. The contribution is mainly from its empirical part.

During rebuttal, the authors have addressed the clarity issues pointed out by the reviewers. They have also added additional experimental results.
Decision: Accept (Poster)