Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies pre-training of Siamese Transformer encoders for the zero-shot dense retrieval problem. The author consider contrastive learning loss function to optimize the encoders and investigate variants of ICT for data augmentations. While not much novelty in the techincal sides, the author indeed present solid empirical study on zero-shot retrieval benchmark as well as extension to the few-shot evaluation.
main_review: ## Strengths
- the paper writing is clear and easy to follow
- solid empircal comparison

## Weaknesses
- lack of techincal novelty. The contrastive loss for learning Siamese Transformers is not new. Creating pseudo (query, doc) pairs for the constrastive loss is also studied in ICT.  
- the ablation study of truely zero-shot (w/o fine-tuning with MS-MARCO) evaluation on BEIR is missing.
- the author should also compare with other pre-training frameworks in Figure 1, including ICT, REALM, and SimCSE.
- from Figure 1, it seems like the the proposed model is worse than BM25 in terms of the average Recall@100
- for reproducibility, the author should release the experiment code, pre-training datasets and the pre-trained models



summary_of_the_review: This paper studies pre-training of Siamese Transformer encoders for the zero-shot dense retrieval problem. While the empirical results seem promising, there are very little techincal novelty in this work. Since this paper concerns pre-training tasks for retrieval, the author should also show the results of ICT and REALM for the Figure 1 setup. At the moment, I think this paper is marginally below the acceptance threshold. If the author can address my concern about proper ablation comparisons, I am very happly to update the recommendation scores.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use cropping strategy to train a neural model for IR. It is compared to BM25 and several other neural models, including BERT and the ones using inverse cloze task. The goal of cropping is to train a model that fits better the IR tasks. It is hypothesized that this task is more similar to IR than other pre-training strategies.
The method is tested on a set of benchmarks of zero-shot retrieval and MSMARCO with training data. It is shown than the proposed method fine-tuned on MSMARCO can outperform the compared methods, including BM25. When no fine-tuning is used, the method is almost at par with BM25.

main_review: The paper addresses an important problem for neural models for IR: what unsupervised training task is suitable for an IR model? The existing strategies such as masked language model and ICT fit some of the NLP tasks, but not necessarily the IR task. This paper proposes to use cropping to train a model for IR.
The proposed strategy is interesting and intuitively more relevant to IR tasks. The experiments show that this is indeed better than the other training strategies.
The paper is well written, and contains a set of experiments on several datasets. The experimental results are convincing.
One broader question is on the idea of replacing BM25 in the zero-shot learning scenario. As the experiments show, when no training data (aligned query-documents) is available, the proposed method can just catch up with BM25. In other words, the cropping training strategy can just reach the level of BM25, but with heavy pre-training. It does not seem reasonable to try to compete with an inexpensive method such as BM25 by a heavy model. Instead, the neural model should seek to complement BM25, i.e. capture other matching signals.
From this perspective, trying to beat BM25 in a zero-shot learning scenario may not be useful in practice.
summary_of_the_review: The proposed method is interesting and the experiments show it effectiveness in the scenario of zero-shot learning and few-shot learning.
However, the question is about the necessity to build a complexe neural model to compete with much more efficient BM25 model in the zero-shot learning situation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a contrastive learning approach for unsupervised text retrieval. As the approach is unsupervised, similar to the Inverse Cloze Task (ICT) approach, the authors propose to do independent cropping of a paragraph to generate a pair of "pseudo-question" and "pseudo-document", which is used to train the model. The model is trained using contrastive learning following the MoCo algorithm. Experiments are performed on the recent BEIR benchmark for zero-shot and few-shot retrieval evaluation. The proposed approach often performs better than the BM25 algorithm on zero-shot retrieval on the BEIR benchmark. There are also some ablations presented to analyze different aspects of model training such as the effect of number of negative examples, training data, data augmentations etc.
 
main_review: The proposed model has some merits but a lot of weaknesses, which I elaborate next.

**Weaknesses**:
The major weaknesses in the paper are:

Introduction Section:
1. The authors need to assign much more credit to the previous papers on unsupervised retrieval using contrastive methods. It is sad to see that the authors have not properly cited even the original ICT paper which was the first to propose the zero-shot neural text retrieval model and several important follow up works ([1], [2], [3]) in the Introduction section. The authors should pay careful attention to how they are positioning their work in relation to the previous well-known work on this topic. It’s appalling to see this style of writing that leads to improper credit assignment. I am sorry to say but writing the paper in a way where you intentionally or unintentionally hide the contributions of the previous work on the topic misguides the reader towards the contributions of the paper and is not in the spirit of a top-tier conference like ICLR.

2. From the Introduction section, the paper positions itself as a new unsupervised contrastive representation learning algorithm in the field of neural retrieval. My suggestion to you would be to position the paper as a followup to the ICT and subsequent works. And then explain clearly the new features in your proposed model when compared against a strongly trained ICT and followup models in references ([1], [2], [3]). You can make a table in the Introduction section listing the new introduced aspects in the proposed model and compare them in a rigorous manner with the previous work.

Related Work Section:

3. In the 4th paragraph, it is wrongly mentioned that Karpukhin et al. 2020 in their DPR paper introduced the dense retriever trained with BERT initialization. This is not correct. It was first done by the ICT paper and later DPR applied their model for supervised contrastive training.
Minor edits: (Izacard and Grave, 2021) and (Yang and Seo, 2020) appear two times in the same sentence. My suggestion would be to cite all the important work in unsupervised dense retrieval such as [1, 2, 3] etc. and then contrast your work with the results of previous work.

Other issues in writing

4. Section 3.1.2 first paragraph: A method from computer vision literature is presented for the two views of an image but its citation is not provided.

Methods Section 

5. Section 3.1.2: ICT paragraph: There are a couple of errors in how the ICT model is described. First, it should be clearly mentioned that the the view 1 is one sentence while the view 2 is all the remaining sentences in the paragraph. Please refer to Section 4 of the ICT paper for full details (https://arxiv.org/pdf/1906.00300.pdf). Second, to enable lexical matching in ICT, the view 1 is removed with a probability of 0.9 and is kept otherwise. Again, please see Section 4 in the original paper for more details and Section 9.2 for the analysis of why this is important during ICT training.

6. Section 3.1.3: *Negative pairs within a batch*: this method for contrastive learning was first used in ICT and subsequent work such as [2, 3]. *Negative Pairs Across Batches*: This method (MuCo) has also been investigated for dense retriever training by [4]. These papers should be cited here. Also, [4] reports some issues 

Results Section:

7. Section 4.3: (Second last paragraph)  The better performance of contrastively trained models when compared to BERT in both zero-shot and few-shot settings has already been investigated earlier. For example, [3] compares the zero-shot retrieval performance for both BERT and ICT in Table 3 and for few-shot performance in Figure 3. This should be mentioned here.

8. I feel that the inclusion of zero-shot results Figure 1 in the paper doesn’t really add much value until compared to a couple of previous baselines such as that of ICT. As the code is not submitted with the paper, I can't verify if the implementation of ICT is correct or not. My suggestion is that you should first make sure the implementation of ICT is correct, then train it with the same settings as your proposed approach of Independent Cropping. Also include the results of ICT in Table1, Table2, and Figure1. This will help the reader to better appreciate the results in the paper.

**Strengths**

1. The proposed method of Independent Cropping for unsupervised retrieval training seems novel and often outperforms strong BM25 baseline on the BEIR benchmark.

2. The results presented in Table 3 are important as it demonstrates that MoCo can be leveraged to train models instead of SimCLR in a memory efficient manner.

3. If the ICT implementation is correct, then the results in Table 4 would be very insightful. Then, the authors should really dig deep into the reasons why the model performance is better than ICT when keeping all the training settings the same such as the number of training steps, batch size, etc.


**Other Questions**

1. It is mentioned that you averaged the token-level representations to compute query and document representations. Did you compare how much the results differed when you just took their [CLS] token representations?

2. Could you also report the zero-shot Recall@20,100 results in Table 4 without finetuning of MS-MARCO? 

3. For qualitative analysis, can the authors present some examples on NQ when their model shows better top-1 recall than ICT due to a better lexical match? 

References:

[1] Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yim- ing Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In International Conference on Learning Representations.

[2] Luan, Yi, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. "Sparse, Dense, and Attentional Representations for Text Retrieval." Transactions of the Association for Computational Linguistics.

[3] Sachan, Devendra Singh, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, and Bryan Catanzaro. 2021. "End-to-end training of neural retrievers for open-domain question answering." In ACL 2021.

[4] Yang, Nan, Furu Wei, Binxing Jiao, Daxing Jiang, and Linjun Yang. 2021. "xMoCo: Cross momentum contrastive learning for open-domain question answering." In ACL 2021.

summary_of_the_review: The paper proposes the approach of independent cropping of a paragraph into two segments for unsupervised training of the dense retriever. The authors argue that independent cropping enables the model to possess lexical matching ability, a property which improves retrieval performance.  

First, I feel that the writing in the Introduction and Related Work sections needs major improvements, specially that of crediting and citing the previous work in the field of unsupervised contrastive training. I have also provided my suggestions for the same under Weakness section. Second, I feel that the authors should correctly implement a ICT following the original paper and perform training using the same settings as done for their model. Third, instead of comparing with BM25, the emphasis should be on comparing results with ICT and providing an in-depth understanding of why or why not their proposed model works better. 

Due to these shortcomings, in its current form, the paper falls substantially short of the ICLR publication bar. 
However, I would request the authors to provide their responses to the points raised in the Weakness section and under the Other Questions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a contrastive learning-based method for zero-shot dense IR. Based on the contrastive learning framework, 3 positive pair construction methods and 2 negative pair construction methods are used and compared. Evaluations are performed on the BEIR benchmark, with the best average result achieved compared to a list of IR (both sparse and dense) baselines in the zero-shot setting.
main_review: Pros:
- A cropping-based positive pair construction method is proposed and compared with the ICT-based method.
- The zero-shot evaluation and comparison on the BEIR benchmark look good.

Cons:
- Using contrastive learning in dense IR is not new. Most technical pieces mentioned here (SimCLR, MoCo, ICT, etc.) were used somewhere else before. 
- For the fine-tuning setting (e.g. Table 6), it could be better if the baseline dense IR approaches (mentioned in Table 1) can be included as well, instead of just comparing with BERT.
 
summary_of_the_review: Duo to the novelty and insufficient experiment, I think the paper should be further refined and improved.
review_title: null

Meta-review: Good premise:  What unsupervised training supports IR?  This is a key question for IR and is a focus for papers in TREC 2019 Deep Learning Track, for instance.  Also, historically, empirical work in the IR community is a very high standard. 

One reviewer says the contrastive loss for learning Siamese Transformers is not new, and prior experimental work was listed.  Several reviewers suggested extensions to the empirical work, some of which was subsequently done.  Results are "promising" according to one reviewer, but not strong enough.  Another reviewer says a different use context is needed since its hard to compete with efficient BM2t in its own terms.  The authors made some good changes to their paper: updated intro and related work, extended results and discussion, but the 4 reviewers remained in agreement, a reject.  However, some reviewers felt this was a good contribution, so with further empirical work and polish it should be good.
Decision: Reject