Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a model for multi-hop question answering, specifically in a closed domain setting where the relevant paragraphs are present within a few distractor paragraphs. Their model has the following components — (a) a reader module that reads and collect information from paragraphs, (b) a query reformulation module, that reformulates the query to retrieve the next relevant document required to answer the question. Following the effectiveness of multi-head attention, the reader and reformulation module has K-heads that forms independent reformulations. The k-heads are aggregated via a simple summing and the reformulated query representation is used in the next step of the pre-defined hop. 

Specifically, they start by encoding the question and paragraphs by BERT embeddings. Next they concatenate the fixed set of paragraphs (10 in their experiments) and encode it with a recurrent NN (bi-GRU) to produce token-level recurrent representation. This is followed by a reading module which dies document question attention in BiDAF (Seo et al., 2017) style followed by a self attention module. Next, in the query reformulation phase a convolution style filter is passed across the token embeddings of the document to gather / pool information required for querying. The current query representation is added to the new pooled representation to obtain the updated query representation. This is followed by an answering module that runs 4 layers of Bi-GRU and has an outlet for computing loss for each of the following — supporting facts , start , end of the answer span representation and a classifier to determine if its a yes-no question or a span answer. The network is trained via supervision for all the aforementioned 4 outlets. 

Empricially, this paper shows gain in the distractor setting for HotpotQA dataset. 


Strengths: 
Query reformulation is an important strategy for IR, and multi-hop QA and it is nice to see that this model adapts it. Each module of the network seems to be carefully designed and the ablation results and analysis are helpful.

Weaknesses:
1. Related work: One of the major weakness of the paper is the missing related work. There are two well-established paper Das et al (ICLR 2019) — Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering and Feldman and Yaniv (ACL 2019) — Multi-Hop Paragraph Retrieval for Open-Domain Question Answering that show query reformulation is effective for multi-hop question answering. Das et al 2019 proposes that the query reformulation module is independent of the reader module as long as it has access to the hidden representation of the reader module. This paper builds off from the original papers by carefully designing the reader and the reformulation modules (which is great!), but never mentions any of the above papers. I think that should be fixed.s
2. The setting considered in this paper is closed-domain where the number of paragraphs to be read are fixed and pre-determined. That is a very unrealisitic setting for a general purpose QA system. The paper should consider testing on the open domain setting of the HotpotQA dataset. The other two papers mentioned above (Das et al 2019, Feldman & Yaniv 2019) both test on open-domain setting. Moreover, it has been also shown recently that the distractor setting can easily be fooled and is not a great benchmark for testing the reasoning capabilities.
3. For a real QA system to be deployed in production setting, the system needs to be fast. I am afraid the model proposed by this paper is very computationally expensive. Apart from encoding the question and paragraphs by BERT, they have multiple Bi-GRU encodings (8 additional). That is going to be computationally intensive and the authors should strongly consider other approaches such as replacing bi-grus with transformers that can encode sequences in parallel, and parameter sharing. 

For the above strong weaknesses, I am forced to give a low score to the current paper. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes to iteratively reformulate questions in the latent space for multi-hop question answering. The reformulation of the question depends on the question-aware representation of the documents. 

The authors experiment their model on the HotpotQA dataset and achieve the state of the art performance. But, it's important to experiment with some other datasets. One option could be the Complex WebQuestions [1]. 

Since T is set to 2 in your experiment, is it possible that your model simply predict the intermediate answers and use it to reformulate the question representation? In Talmor et. al paper [1], the SplitQA splits the questions into two subquestions, and appends the answer of the first subquestion to the second subquestion.

Another related work is PullNet by Sun et. al [2].

[1] Talmor, Alon, and Jonathan Berant. "The web as a knowledge-base for answering complex questions." arXiv preprint arXiv:1803.06643 (2018).
[2] Sun, Haitian, Tania Bedrax-Weiss, and William W. Cohen. "PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text." arXiv preprint arXiv:1904.09537 (2019).
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The authors propose a multi-hop latent question reformulation system that performs well in the question-answering setup. The system achieves best current published result on the HotpotQA dataset.

The system achieves that using question-aware representation of the document.

Question: would it be possible to re-generate, at least in an approximate form, one of the reformulations of the question, using decoding? It seems that the visualization of these intermediate forms would allow to understand the model better.
review_title: Official Blind Review #3

Meta-review: This paper proposes a novel approach, Latent Question Reformulation Network (LQR-net), a multi-hop and parallel attentive network designed for question-answering tasks that require multi-hop reasoning capabilities. Experimental results on the HotPotQA dataset achieve competitive results and outperform the top system in terms of exact match and F1 scores. However, reviewers note the limited setting of the experiments on the unrealistic, closed-domain setting of this dataset and suggested experimenting with other data (such as complex WebQuesitons). Reviewers were also concerned about the scalability of the system due to the significant amount of computations. They also noted several previous studies were not included in the paper. Authors acknowledged and made changes according to these suggestions. They also included experiments only on the open-domain subset of the HotPotQA in their rebuttal, unfortunately the results are not as good as before. Hence, I suggest rejecting this paper.
Decision: Reject