Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Summary: This work proposes a new transformer architecture for tasks that involve a query sequence and multiple candidate sequences. The proposed architecture, called poly-encoder, strikes a balance between a dual encoder which independently encodes the query and candidate and combines representations at the top, and a more expressive architecture which does full joint attention over the concatenated query and candidate sequences. Experiments on utterance retrieval tasks for dialog and an information retrieval task show that poly-encoders strike a good trade-off between the inference speed of the dual encoder model and the performance of the full attention model. 

Pros:
- Strong results compared to baselines on multiple dialog and retrieval tasks. 
- Detailed discussion of hyperparameter choices and good ablations.
- Paper is well written and easy to follow.

Cons:
- Limited novelty of methods. Ideas similar to the model variants discussed in this work have been considered in other work (Eg: [1]). It is also known that in-domain pre-training (i.e, pre-training on data close to the downstream task’s data distribution) helps (Eg: [2]). So this work can be considered as an application of existing ideas to dialog tasks. 
- In terms of impact, utterance retrieval has fairly limited applicability in dialog. The dialog tasks considered in this work have a maximum of 100 candidate utterances, whereas in practice, the space of possible responses is much larger. While retrieval models are useful, I am skeptical about the practical value of the improvements shown in the paper (especially the improvements over bi-encoder, which is already a decent model).

Suggestions:
One way to get around the inefficiency of the cross-encoder architecture is to first use an inexpensive scoring mechanism such as TFIDF or bi-encoder to identify a small number of promising candidates from all the possible candidates. We can then use the cross-encoder to do more precise scoring of only the promising candidates. I am curious where a pipelined model such as this compares against the variants discussed in the paper in terms of speed and performance. 

While the paper presents strong results on several dialog utterance retrieval tasks, the methods presented have limited novelty and impact. I am hence leaning towards borderline. 

References

[1] Logeswaran Lajanugen, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
[2] Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.

Edit: I have read the author response. Based on the rebuttal, I am more convinced about the practical impact of the approach. I am raising my score and recommending accept. 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper describes an approach for scoring sentences based on pre-trained transformers. The paper describes two main existing approaches for this task, namely bi-encoders and cross-encoders and then proposes a new formulation called poly-encoders which aims to sit between the existing approaches offering high accuracy -- similarly to cross-encoders -- and high efficiency -- similarly to bi-encoders. The paper is well written and although this is not related to my research I enjoying reading it. The approach proposed seems reasonable to me, and of sufficient novelty while the results presented are impressive. Moreover the paper seems a good fit for ICLR.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents a new neural network architecture based on transformers called poly-encoders. These are compared against many state-of-the-art approaches including cross-encoders and bi-encoders on many large-scale datasets. 
Bi-encoders > Poly-encoders > Cross-encoders in terms of speed and 
Cross-encoders > Poly-encoders > Bi-encoders in terms of accuracy.

I am not an expert in this area. However, to the best of my knowledge I don't see anything immediately wrong with this. The experiments are also comprehensive. Therefore I recommend acceptance.
review_title: Official Blind Review #2

Meta-review: The paper presents a new architecture that achieves the advantages of both Bi-encoder and Cross-encoder architectures. The proposed idea is reasonable and well-motivated, and the paper is clearly written. The experimental results on retrieval and dialog tasks are strong, achieving high accuracy while the computational efficiency is orders of magnitude smaller than Cross-encoder. All reviewers recommend acceptance of the paper and this AC concurs.
Decision: Accept (Poster)