Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an architecture called ReasoNet that reason over the relation. The paper addresses important tasks but there are many other related works. The comparison to other methods are not comprehensive. The Graph Reachability dataset is not a good example to use.
review_title: Review

final_score: rating: null
review_content: 1) One of  the core technical contributions seems to be derive a new baseline for REINFORCE. It would be interesting to compare this with other approaches that learn a baseline like https://www.cs.toronto.edu/~amnih/papers/nvil.pdf and approaches that use multiple-samples (https://arxiv.org/pdf/1602.06725.pdf), particularly because the use of (r/b -1 ) in the paper seems "hacky".
2) It would be interesting to know the difference in results when the maximum number of inference steps is always 5 compared to the proposed adaptive method. Also, what is the average number of inference steps performed by the adaptive model ? Is it much lesser than 5 and hence the model is much faster than fixed number of steps ?
review_title: Questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper aims to consolidate some recent literature in simple types of "reading comprehension" tasks involving matching questions to answers to be found in a passage, and then to explore the types of structure learned by these models and propose modifications. These reading comprehension datasets such as CNN/Daily Mail are on the simpler side because they do not generally involve chains of reasoning over multiple pieces of supporting evidence as can be found in datasets like MCTest. Many models have been proposed for this task, and the paper breaks down these models into "aggregation readers" and "explicit reference readers." The authors show that the aggregation readers organize their hidden states into a predicate structure which allows them to mimic the explicit reference readers. The authors then experiment with adding linguistic features, including reference features, to the existing models to improve performance.

I appreciate the re-naming and re-writing of the paper to make it more clear that the aggregation readers are specifically learning a predicate structure, as well as the inclusion of results about dimensionality of the symbol space. Further, I think the effort to organize and categorize several different reading comprehension models into broader classes is useful, as the field has been producing many such models and the landscape is unclear. 

The concerns with this paper are that the predicate structure demonstrated is fairly simple, and it is not clear that it provides insight towards the development of better models in the future, since the "explicit reference readers" need not learn it, and the CNN/Daily Mail dataset has very little headroom left as demonstrated by Chen et al. 2016. The desire for "dramatic improvements in performance" mentioned in the discussion section probably cannot be achieved on these datasets. More complex datasets would probably involve multi-hop inference which this paper does not discuss. Further, the message of the paper is a bit scattered and hard to parse, and could benefit from a bit more focus.

I think that with the explosion of various competing neural network models for NLP tasks, contributions like this one which attempt to organize and analyze the landscape are valuable, but that this paper might be better suited for an NLP conference or journal such as TACL.

review_title: Simple predicate structure and data sets

final_score: rating: null
review_content: Is there any comparison to existing methods on Graph Reachability dataset? I guess some methods such as Pointer Networks can solve the problem as well. How do you compare?
review_title: Comparison

final_score: rating: null
review_content: The graph reachability task is an interesting contribution to highlight the reasoning ability of the model. Is this the first time such a task has been proposed for highlighting the behaviour of memory network-like models? If not, how have previous models done on this task? If so, would you consider testing out previous CNN/ Daily Mail models on this dataset, or simpler baselines like an LSTM without attention? (to show that it is indeed challenging)
review_title: small question

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new architecture for document comprehension. The main addition to the model, as claimed by the authors, is that the model is able to adaptively determine how many inference ‘hops’ is required in order to solve a particular problem. This is in contrast to previous work, where the number of hops is fixed.

Overall, the change of adding a termination gate is rather modest, and it leads to rather small gains on the tested CNN/ Daily Mail dataset (is it possible to include significance here?). Actually, I’m not sure why having an adaptive number of hops would be better performance-wise than using the maximum number of hops – unless I missed it the authors don’t argue this point well (other than saying it mimics humans). Does the model forget some things if it performs too many hops? The authors do say:
“The results suggest that the termination gate variable in the ReasoNet is helpful when training with sophisticated examples, and makes models converge faster”
but don’t elaborate much beyond this. I could see for example an argument being made that it reduces the amount of computation required per question. The authors do show results comparing the model without a termination gate on the Graph Reachability dataset, and the full model does seem to perform quite a bit better, but I would like this to also be done on the CNN/ Daily Mail datasets, and for there to be more insights into why the performance is improved vs. the ReasoNet-Last model.

One of the contributions I like most from this paper is not the actual model, but the Graph Reachability dataset. It is designed to test the reasoning abilities of the ReasoNet model in more detail. One of the benefits is that the inference procedure necessary to solve the task is very clear, as opposed to the CNN/ Daily Mail dataset, thus it is easier to see what the model is actually doing. I would like to see future models also tested on this dataset.

Overall, I think this is a borderline paper.

Other remarks:

Note that learning a baseline for REINFORCE has previously been studied, see: https://arxiv.org/pdf/1606.01541v4.pdf (although the authors mention only it briefly in the paper)

“ReasoNets are devised to mimic the inference process of human readers.”
I think this is too strong a claim (that humans use the same kind of ‘iterative hop’ method for answering questions), unless it is supported by actual analysis of humans – I think the similarities to humans are more at a surface level. Also, I think it’s unnecessary to actually understanding the model. I would change ‘mimic’ to ‘inspired by’, or something along those lines.


EDIT: I thank the authors for taking the time to reply, and clearing some things up with regards to the idea of multiple hops. I am keeping my score the same for the following reason: in my opinion, accepted papers involving new model architectures should either consist of (1) a small adjustment that leads to a large improvement, or (2) a very innovative idea that leads to comparable performance (or better), but provides many new insights about the problem or can be transferred to many different domains. This paper seems to be a rather small adjustment (allowing multiple hops) which results in small improvements on CNN/Daily Mail (which as Reviewer 3 points out has little headroom). I think this paper would be suitable for a conference such as EMNLP.
review_title: Review

Meta-review: This paper introduces a method for estimating the number of iterations of an attention mechanism in a neural machine reading module using REINFORCE and a custom baseline, which is estimated on the data. Estimating a baseline from data, multi-hop attention, and modelling latent variable models with REINFORCE are not new, so their composition, while sensible, is slightly incremental. There were some concerns from several of the reviewers with the impact the experiments have in terms of validating the model changes. Improvements on "real" tasks such as CNN/DailyMail did not impress reviewers, some of whom believe the benchmarks compared to were not representative of the best comparable architectures tried on these datasets (e.g. pointer networks, which definitely can be used). Overall, I do not find this paper strong enough to recommend acceptance to the main conference in its current state. With better evaluation, it could be a decently strong paper if the results come through.
Decision: Reject