Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper studies scaling multi-hop QA to large document collections, rather than working with small candidate lists of  document/paragraphs  (as done in most of the previous work), a very important, practical and challenging direction.

They start with linking mentions to entities in a knowledge base. Every iteration of their mult-hop system produces a set of entities Z_t, relying on entities predicted on the first representation Z_{t-1} and the question representation.   In order to make training tractable, they mask 'attention'  between Z_{t-1}  and Z_t (actually mentions corresponding to Z_t).  They also use top-K relevant mentions at train and test time. As the attention score is based on dot-product, they can plug-in the approximate Maximum Inner Product Search to avoid computing the attention score for every mention in the collection. The architecture is essentially end-to-end trainable (except for specialized pretraining discussed below).   

Whereas itself the architecture is not overly novel (e.g., the architecture does feel a lot similar to models in KB context, and also graph convolution networks applied to QA), there is a lot of clever engineering and the novelty is really in showing that it can work without candidate preselection.

My main worry is pretraining. In both experiments (MetaQA and the new Wikidata Slot Filling task), they pretrain the encoders using a knowledge base, and the knowledge base directly corresponds to the QA task.   E.g., for MetaQA the questions are answerable using the knowledge bases, so relation types in the knowledge base presumably correspond to relations that need to be captured in the QA multihop learning.  This specialized pretraining appears to be crucial (88% for pretraining vs 55% with BER), presumably because of the top K pruning. Though a nice trick, it is likely limiting as a knowledge base needs to be available + it probably constraints the types of mult-hop questions the apporach can handle.  Also, some of the baselines do not benefit from using the KB, and, in principle, if it is used in training, why not use the KB at test time?  (I see though that for the second dataset pretraining seems to be done on a different part of Wikipedia, I guess, to address these concerns).

I was not sure how the number of hops T was selected for the model, it does not seem to be defined in the paper. Do you pretend that you know the true number of hops for each given question?

The authors experiment with reducing the size of a KB for pretraining. It apparently does not harm the first 1 hop questions, but 2 and 3-hop. Do the authors have any explanation for this?  Related to the previous question, does it mean that the model does not learn to exploit the hops for t > 1?

The evaluation is on MetaQA and on the newly introduced Wikidata task, whereas most (?) recent multi-hop QA work has focused on HotpotQA (and to certain degree WikiHop). Is the reason for not using (additionally) HotpotQA?  Is the model suitable for HotpotQA? If not, does this have to do with pretraining or the types of questions in HotpotQA?

The model definition in section 2.1 is not very easy to follow. E.g., it is not immediately clear if the model applied at every hop is the same model, and not clear how the model is made aware of the current search state (e.g., which part of the question has already processed / how the history is encoded) or even of the hop id. 

I would really like to see more analysis of what the model learns at every hop.

-- 
After the rebuttal -- I appreciate the detailed feedback, extra experiments, and analysis. `I increased my score. 

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper proposes a model that can perform multi-hop question-answering based on a textual knowledge base. Results show that the proposed model -- DrKIT -- performed close to or better than the state of the art results over MetaQA and WikiData datasets. Ablation study is offered to show that a few tricks in the model are necessary to make it work, and comparisons with baseline models such as DrQA and PIQA are presented. The paper also provides additional means to speed up the DrKIT model using the hashing trick and approximated top-k methods.

The paper is a good one and I vote for its acceptance. Besides achieving good performance, the proposed DrKIT model makes sense, and all the parts are necessary components based on the ablation study results. In addition, the ablation study and the speed-up methods are great addition to the model to make it work better.

With this generally positive assessment said, I do have a few questions below that I hope the authors could provide some response. These are on top of the high quality of the paper, and should be best regarded as suggestions for future work.

1. In equation (3), do G and F have to be TFIDF features? The likes of word2vec and GloVe (and also pershaps fastText) are trained based on co-occurences of adjacent words, and I would imagine that they will improve over TFIDF. This is just an intuition and I could be wrong, but it would be very helpful to hear the authors' opinions.

2. The ablation study mentioned that the softmax temperature helps with the model. This is a nice observation, but is there any intuition behind why that is the case? I could imagine that it could be because the gradients of a saturated softmax function is small and therefore results in slow training of the model. If this is the case, both low temperature and high temperature will fail to work. It would have been better so show both ends of failing extremes in an ablation study.

3. Can you discuss the similarity between DrKIT and multi-hop End-to-End Memory Networks [1]? It looks very much like an expansion of it with a fixed retrieval mechanism and by expanding the answer to a set rather than a single vector.

[1] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus, End-To-End Memory Networks, NIPS 2015

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper introduces a new architecture for question answering, that can be trained in an end-to-end fashion. The model expands the entities in a given question to relevant mentions, which are in turn aggregated to a new set of entities. The procedure can be repeated for multi-hop questions. The resulting entities are returned as candidate answers. The approach relies on an index of mentions for approximate MIPS, and on sparse matrix-vector products for fast computation. Overall, the model processes queries 10x faster than previous approaches. The method provides state-of-the-art results on the MetaQA benchmark, with significant improvements on 3-hop questions. The experiments are detailed, and the paper is very well written.

A few comments / questions:

1. Do you have any explanation of why taking the max instead of the sum has a significant impact on the 2,3-hop performance, but only gives a small improvement for 1-hop questions?

2. The same observation can be done for the temperature lambda=1 vs lambda=4, so I was wondering about the distribution of the entities you get on the output of the softmax (in Eq 4). Is the distribution very spiky, and Z_t usually only composed of a few entities? In that case, I guess lambda=4 encourages the model to explore/update more relation paths? Something that works very well in text generation is to not just set a temperature, but also to apply a softmax only on the K elements with the highest score (so the softmax is applied on K logits, and everything else is set to 0). Did you consider something like this? It may prevent the model from considering irrelevant entities, but also from considering only a few ones.

3. Given the iterative procedure of the method, I wonder how well the model would generalize to more hops. Did you try for instance to train with 1/2 hops and test whether it can generalize to 3-hop questions?

4. In Section 2.3, you fix the mention encoder because training the encoder would not work with the approximate nearest neighbor search (I assume this is because the index would need to be rebuilt). However, the ablation study suggests that the pretraining is critical, and one could imagine that fine-tuning the mention encoder would improve the performance even further. Instead of considering a mention encoder, could you have a lookup table of mentions (initialized with BERT applied on each mention), where mention embeddings are fine-tuned during training? The problem of the index is still there, but could you consider an exact search on the lookup table (exact search over a few million embeddings is slow, but it should still run in a reasonable amount of time using a framework like FAISS, and it would give an upper-bound of the performance you could achieve by fine-tuning the mention encoder).
review_title: Official Blind Review #2

Meta-review: This paper proposes a novel architecture for question-answering, which is trained in an end-to-end fashion.

The reviewers were unanimous in their vote to accept. Authors are encouraged to revise addressing reviewer comments.
Decision: Accept (Talk)