Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes a solution to the large scale query-document retrieval problem. The proposed method was shown to be a better alternative to the classic information retrieval approach such as BM-25 (token marching + TF-IDF weights). The proposed method is based on two separate transformer models which has computational benefit over one cross-attention model. For fast training, they have also used the sampled softmax. For pre-training tasks, Inverse Cloze Task (ICT), Body First Selection (BFS), Wiki LinkPrediction (WLP) were studied.


The paper is written well, easy to follow and well-motivated. However, there is a major technical problem in the proposed method. In the proposed approach, the query embedding (q_emb) and the document embedding (d_emb) train separately by two transformer models (two towers --- Query-tower and Doc-tower). After that, the similarity was measured through a dot product. Two embedding models are, therefore, represented by separate vector space representation. Applying dot product to find the similarity does not make much sense to me, as the embedding is not comparable in two different vector representations. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper provides a comprehensive study on the two-tower Transformer models in terms of the impact of its pre-training tasks on large-scale retrieval applications. The studies here show that, pre-training with Inverse Cloze Task (ICT) the two-tower Transformer models significantly outperform the widely used BM-25 algorithm for large-scale information retrieval. The authors also propose two novel pre-training settings which also show improvement over the baseline BM-25. In addition, the authors empirically demonstrate that the token-level masked-LM model used by BERT is not a good choice as pre-training task for the two-tower Transformer when deployed for large-scale information retrieval applications. 

The paper is well written and easy to follow. The Ablation Study of the paper also provides useful insights about the impact of different pre-training schemas on large-scale information retrieval tasks. I think the studies here will benefit the communities where large-scale information retrieval is required such as open-domain question answering. 

The main limitation to me is that, the two novel pre-training tasks proposed in this paper are specific for Wikipedia and they are less effective than the ICT strategy (as shown in Table 5). 

I hope the authors will release the source codes to the community. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies a query-related document retrieval problem using a framework which they call “two-tower retrieval method”. The task is to learn query representation and document representation in order to retrieve query-related documents by the maximum inner product. This is a realistic setting for large-scale retrieval problem since it enables document representations to be computed once regardless of the question, and obtaining query-sensitive document representations is very expensive.
Then, the paper studies three different pretraining methods for this task, ICT (previously proposed by Lee et al 2019), and BFS & WLP (proposed by this paper). 
For evaluation, the paper considers the retrieval task of question answering, based on SQuAD and Natural Questions. The combination of ICT, BFS and WLP achieves remarkable improvement over the number of baselines including BM25 and other neural-based models.

The strength of this paper is that it includes comprehensive studies on the two-tower retrieval problem. In particular, they have conducted extensive ablation studies with different train/test ratios.

However, there are some notable weaknesses of this paper as follows.

First, the benchmark relies on the recall rate instead of the end task (open-domain QA). Recall rate is not a good way to evaluate the retrieval result since a system may retrieve text which contains the answer text but is not semantically related to the question. (I understand that the paper follows Admad et al (2019), but I believe this is not a published paper.) In addition, this paper did not empirically demonstrate the relatedness between the recall rate and the end performance. This makes it very hard to compare with other papers in open-domain QA, which has been extensively studied for a few recent years.

Second, despite comprehensive studies, the fact that ICT+BFS+WLP is almost the same as ICT (93.91 vs. 94.37) means that the method does not give improvement over ICT which was already proposed in the previous study.

Third, the gap between BM25 and ICT+BFS+WLP in Table 3 and 4 are very significant (e.g. 27 vs 94 on Natural Questions), but this doesn't seem to be consistent to Lee et al (2019). (There are some differences: (1) Lee et al (2019) compares BM25 vs. ICT, but according to this paper, ICT and ICT+BFS+WLP are similar. (2) Lee et al (2019) reports the end QA performance while this paper reports the recall rate, but one of the assumptions in this paper is that recall rate and the end performance is related.) What is the explanation for this discrepancy?

(I am happy to increase the rating if my concerns are resolved during rebuttals and/or the paper includes performance on the end QA performance.)

Some questions:
1) Section 4.1 says ICT is sentence-level, BFS is paragraph-level and WLP is document-level. What does it mean? I thought, according to Section 3, all methods are paragraph-level.
2) Section 4.1: it looks like Ahmad et al (2019)’s setting is actually not entirely open-domain. Their candidate sentences/paragraphs are much less than the entire Wikipedia. Did this paper also use the same set of the candidate? In that case, it should be clearly mentioned in the paper. In addition, the data statistics are different across two papers. Did Ahmad et al (2019) include only train set whereas this submission reports train+test? In case there is an official split of train/test, why were different splits used for evaluation?
3) Also regarding the split: for each split, how much was used for development? I believe data used for the development and test should be different. In fact, rather than experimenting on different ratios of train/test, is it possible to report on official test set, while splitting the train set into 90/10 for training and development? Or, split the entire data to 90/5/5 for training/development/test?



Update on Nov 15: The revised paper resolves most of my concerns, so I am updating the score from 3 to 6.
review_title: Official Blind Review #2

Meta-review: This paper conducts a comprehensive study on different retrieval algorithms and show that the two-tower Transformer models with properly designed pre-training tasks can largely improve over the widely used BM-25 algorithm. In fact, the deep learning based two tower retrieval model is already used in the IR field. The main contribution lies in the comprehensive experimental evaluation.

Blind Review #3 has a major misunderstanding of the paper; hence his review will be excluded. The other two reviewers tend to accept the paper with several minor comments.

As the authors promise to release the code as a baseline for further works, I agree to accept the paper.

Decision: Accept (Poster)