Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: 

This paper proposes a dynamic memory augmented neural network for question answering. The proposed model iteratively creates a shorter list of relevant entities such that the decoder can look at only a smaller set of entities to answer the given question. Authors show results in bAbi dataset.

My comments:

1. While the proposed model is very interesting, I disagree with the claim that AMN has lower inference times. The memory creation happens only after reading the question and hence the entire process can be considered as part of inference. So it is not clear if there is a huge reduction in the inference time when compared to other models that the authors compare. However, the proposed model looks like a nice piece of interpretable reasoning module. In that sense, it is not any better than EntNet based on the error rate since EntNet is doing better than AMN in 15 out of 20 tasks. So it is not very clear what is the advantage of AMN over EntNet or other MANN architectures.

2. Can you explain equation 9 in detail? What is the input to the softmax function? What is the output size of the softmax? I assume q produces a scalar output. But what is the input size to the q function?

3. In the experiment, when you say “best of 10 runs”, is it based on a separate validation set? Please report the mean and variance of the 10 runs. It is sad that people just report best of multiple runs in the bAbi tasks and not report the variance in the performance. I would like to see the mean and variance in the performance.

4. What happens when number of entities is large? Can you comment about how this model will be useful in situations other than reading comprehension style QA? 

5. Are the authors willing to release the code for reproducing the results?

Minor comments:

1. Page 2, second line: “Networks(AMN)” should be “Networks (AMN).
2. In page 3, first line: “Hazy et al. (2006)” should be “(Hazy et al. 2006)”.
3. In page 3, second para, first line, both references should use \citep instead of \citet.
4. In page 4, fourth para, Vanhoucke et al should also be inside \citep.
5. In page 4, notations paragraph: “a question is a sequence of N_q words” - “of” is missing.
6. In page 5, first paragraph is not clear.
7. In page 6, point 4, 7th line: “nodes in the its path” should be “nodes in its path”.
8. In page 9, section 5.3, multiple questions, 2nd line: “We extend the our model” should be “We extend our model”.

review_title: "faster inference" is not convincing.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a model for QA that given a question and a story adaptively determines the number of  entity groups (banks). The paper is rather hard to follow as many task specific terms are not explained. For instance, it would benefit the paper if the authors introduced the definitions of a bank and a story. This will help the reader have a more comprehensive understanding of their framework.

The paper capitalized on the argument of faster inference and no wall-time for inference is shown. The authors only report the number of used banks. What are the runtime gains compared to Entnet? 
This was the core motivation behind this work and the authors fail to discuss this completely.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper offers a very promising approach to the processing of the type of sequences we find in dialogues, somewhat in between RNNs which have problem modeling memory, and memory networks whose explicit modeling of the memory is too rigid.

To achieve that, the starting point seems to be a strength GRU that has the ability to dynamically add memory banks to the original dialogue and question sentence representations, thanks to the use of imperative DNN programming. The use of the reparametrization trick to enable global differentiability is reminiscent of an ICLR'17 paper "Learning graphical state transitions". Compared to the latter, the current paper seems to offer a more tractable architecture and optimization problem that does not require strong supervision and should be much faster to train.

Unfortunately, this is the best understanding I got from this paper, as it seems to be in such a preliminary stage that the exact operations of the SGRU are not parsable. Maybe the authors have been taken off guard by the new review process where one can no longer improve the manuscript during this 2017 review (something that had enabled a few paper to pass the 2016 review).

After a nice introduction, everything seems to fall apart in section 4, as if the authors did not have time to finish their write-up. 
- N is both the number of sentences and number of word per sentence, which does not make sense.
- i iterates over both the sentences and the words. 

The critical SGRU algorithm is impossible to parse
- The hidden vector sigma, which is usually noted h in the GRU notation, is not even defined
- The critical reset gate operation in Eq.(6) is not even explained, and modified in a way I do not understand compared to standard GRU.
- What is t? From algorithm 1 in Appendix A, it seems to correspond to looping over both sentences and words.
- The most novel and critical operation of this SGRU, to process the entities of the memory bank, is not even explained. All we get at the end of section 4.2 is " After these steps are finished, all entities are passed through the strength modified GRU (4.1) to recompute question relevance."

The algorithm in Appendix A does  not help much.  With PyTorch being so readable, I wish some source code had been made available.

Experiments reporting also contains unacceptable omissions and errors:
- The definition of 'failed task', essential for understanding, is not stated (more than 5% error)
- Reported numbers of failed tasks are erroneous: it should be 1 for DMN+ and 3 for MemN2N.

The reviewers corrections, while significant, do not seem enough to clarify the core of the paper.

Page 3: dynanet -> dynet
review_title: Very promising approach, but it seems the authors were not able to submit a finished manuscript on time.

Meta-review: This paper presents an interesting model which at the time of submission was still quite confusingly described to the reviewers.
A lot of improvements have been made for which I applaud the authors.
However, at this point, the original 20 babi tasks are not quite that exciting and several other models are able to fully solve them as well.
I would encourage the authors to tackle harder datasets that require reasoning or multitask settings that expand beyond babi.


Decision: Invite to Workshop Track