Title: ReasoNet: Learning to Stop Reading in Machine Comprehension
Authors: Yelong Shen
Po-Sen Huang
Jianfeng Gao
Weizhu Chen
Venue: ICLR 2017
Keywords: Deep learning
Natural language processing

Abstract: Teaching a computer to read a document and answer general questions pertaining to the document is a challenging yet unsolved problem. In this paper, we describe a novel neural network architecture called Reasoning Network ({ReasoNet}) for machine comprehension tasks. ReasoNet makes use of multiple turns to effectively exploit and then reason over the relation among queries, documents, and answers. Different from previous approaches using a fixed number of turns during inference, ReasoNet introduces a termination state to relax this constraint on the reasoning depth. With the use of reinforcement learning, ReasoNet can dynamically determine whether to continue the comprehension process after digesting intermediate results, or to terminate reading when it concludes that existing information is adequate to produce an answer. ReasoNet has achieved state-of-the-art performance in machine comprehension datasets, including unstructured CNN and Daily Mail datasets, and a structured Graph Reachability dataset. 


TL;DR: ReasoNet Reader for machine reading and comprehension

Review discussion:
The graph reachability task is an interesting contribution to highlight the reasoning ability of the model. Is this the first time such a task has been proposed for highlighting the behaviour of memory network-like models? If not, how have previous models done on this task? If so, would you consider testing out previous CNN/ Daily Mail models on this dataset, or simpler baselines like an LSTM without attention? (to show that it is indeed challenging)
Thanks for the feedbacks. The graph reachability task is the first time introduced in the paper to verify the reasoning process in the ReasoNet model, and we have shared the data set to public.  This was mentioned in the paper. (An independent work is from Deepmind’s Differentiable Neural Computers (DNC)[1], they applied DNC in graph traversal tasks, but we have different problem settings and use different approaches as well.)
The graph reachability task has a slight difference with the CNN/Daily Mail task. The final prediction of the graph reachability task is doing a binary classification, while the final prediction of CNN/Daily Mail task is detecting the answer entity from the candidate entities.
Hence we cannot directly apply their models to this new task.
However, we are open and would be happy to discuss with you how to produce the models you mentioned as the baseline for this new problem.
In the graph reachability experiment, we have two baselines: “ReasoNet- T_max=2” and “ReasoNet-Last”. “ReasoNet-T_max=2” is similar to single step approaches such as “attention sum reader” [2] in the CNN/Daily Mail task (ReasoNet is trained with RL objectives, and attention sum reader is trained with supervised learning).  The results of “ReasoNet- T_max=2” show the graph reachability task cannot be fully solved if the reasoning step is not large enough for complex instances.
The setup of “ReasoNet- Last” is similar to the multi-step approaches in the CNN/Daily Mail task [3, 4, 5].  Due to variable complexity among instances, “ReasoNet with last step” performs worse than the proposed ReasoNet setup.
[1] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka GrabskaBarwinska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, ´ Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 2016.
[2] Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547, 2016
[3]  Bhuwan Dhingra, Hanxiao Liu, William W. Cohen, and Ruslan Salakhutdinov. Gated-attention readers for text comprehension. CoRR, abs/1606.01549, 2016.
[4] Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. Iterative alternating neural attention for machine reading. CoRR, abs/1606.02245, 2016.
[5] Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. Attention-over-attention neural networks for reading comprehension. CoRR, abs/1607.04423, 2016.
1) One of  the core technical contributions seems to be derive a new baseline for REINFORCE. It would be interesting to compare this with other approaches that learn a baseline like https://www.cs.toronto.edu/~amnih/papers/nvil.pdf and approaches that use multiple-samples (https://arxiv.org/pdf/1602.06725.pdf), particularly because the use of (r/b -1 ) in the paper seems "hacky".
2) It would be interesting to know the difference in results when the maximum number of inference steps is always 5 compared to the proposed adaptive method.
Also, what is the average number of inference steps performed by the adaptive model ? Is it much lesser than 5 and hence the model is much faster than fixed number of steps ?
Thanks for pointing us the related work on REINFORCE algorithms. We also notice that there are a few research work on estimating baseline b in the REINFORCE algorithm. In this work, we empirically found “Contrastive Reward” baseline is a good and simple solution on learning ReasoNet model. It is not the focus of this paper to analyze different variants of REINFORCE algorithms. It is the future work for us to study different optimization algorithms for ReasoNet. We will be happy to discuss with you about more details on this topic.
Explanation of r/b – 1 from the objective function:
The gradient formulation with r/b-1 in section 3.1  can be viewed as the gradient of a function J~(\theta)  = E(r/b – 1).  J~(\theta) can be rewritten as  J~(\theta) = 1/b J(\theta) – 1, where b is larger than zero in our setting and J(\theta) is the expected reward function.
Hence, we are optimizing a weighted expected reward function that puts more focus on training instances with smaller baseline values.
Another intuition behind the meaning of r/b - 1:
Given an training instance x, the model samples K-episodes for x. Each episode performs actions a^i_{1:T} and receives a reward r^i  at the final step (reward is either 1 or 0 in the setting).
Let us define the prior probability of sampling one episode from K episodes as 1/K and the posterior probability of sampling reward from K episodes as (1/K  r_i) / \sum_{j=1}^K  (1/K * r_j ). We set the reward for episode i as the posterior probability minus the prior probability: (r_i / (r_1 + r_2 +…  +r_k)  –  1/K). Since the sum of all the final rewards for K episodes is zero, this indicates some episodes receive positive reward, while the others of them receives negative reward.
Thus, we name it as contrastive reward.  We apply the above reward schema to the expected reward function (J), that is, (r/b – 1), where b =  \sum_{j=1}^K  (1/K * r_j). Please let us know if you have more questions about our explanations.
In the paper, we experimented with the “ReasoNet-Last” model (it performs 15 steps in the small graph dataset, and 25 steps in the large graph dataset) in the synthetic graph reachability task. It shows the “ReasoNet” model is better than “ReasoNet-Last” in experimental results.  We did not run “ReasoNet-Last” on CNN and Daily Mail dataset, since the “ReasoNet-Last” model is similar to one of the baseline models called “Iterative Attention Reader [Sordoni et al., 2016]”.
Hence, we expect the results of “ReasoNet-Last” to be close to the results of “Iterative Attention Reader” on the CNN and Daily Mail dataset.
Following the suggestions, we updated the paper by adding an appendix section to show the distribution of termination steps on test set, and the correlation between termination steps and complexity of test cases. The average number of termination step is 4.07, 8.12, and 16.27 on the CNN dataset (T_max=5), small graph dataset (T_max=15), and large graph dataset (T_max=25), respectively.
Is there any comparison to existing methods on Graph Reachability dataset? I guess some methods such as Pointer Networks can solve the problem as well. How do you compare?
Thanks for your feedbacks. In the graph reachability task, we mainly consider “ReasoNet-T_max=2” model and “ReasoNet-Last” model as baselines, where “ReasoNet-T_max=2” performs single-turn inference, and “ReasoNet-Last” performs fixed-step multi-turn inference. These two types of approaches are analogous to previous work in the CNN/Daily Mail task. The proposed “ReasoNet” shows a better performance than both baseline models. In order to compare ReasoNets with other baselines as you suggested, we add another baseline model called “Deep LSTM Reader” [1] (single-turn inference) in the experiments. (Please see the detail results in our updated version paper draft). Experimental results show “Deep LSTM Reader” performs similar to “ReasoNet-T_max=2”. The reason might be both models have only single-turn inference, which cannot handle sophisticated examples.
[1] Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman and Phil Blunsom , Teaching Machines to Read and Comprehend, NIPS, 2015
This paper proposes a new architecture for document comprehension. The main addition to the model, as claimed by the authors, is that the model is able to adaptively determine how many inference ‘hops’ is required in order to solve a particular problem. This is in contrast to previous work, where the number of hops is fixed.
Overall, the change of adding a termination gate is rather modest, and it leads to rather small gains on the tested CNN/ Daily Mail dataset (is it possible to include significance here?). Actually, I’m not sure why having an adaptive number of hops would be better performance-wise than using the maximum number of hops – unless I missed it the authors don’t argue this point well (other than saying it mimics humans). Does the model forget some things if it performs too many hops? The authors do say:
“The results suggest that the termination gate variable in the ReasoNet is helpful when training with sophisticated examples, and makes models converge faster”
but don’t elaborate much beyond this. I could see for example an argument being made that it reduces the amount of computation required per question. The authors do show results comparing the model without a termination gate on the Graph Reachability dataset, and the full model does seem to perform quite a bit better, but I would like this to also be done on the CNN/ Daily Mail datasets, and for there to be more insights into why the performance is improved vs. the ReasoNet-Last model.
One of the contributions I like most from this paper is not the actual model, but the Graph Reachability dataset. It is designed to test the reasoning abilities of the ReasoNet model in more detail. One of the benefits is that the inference procedure necessary to solve the task is very clear, as opposed to the CNN/ Daily Mail dataset, thus it is easier to see what the model is actually doing. I would like to see future models also tested on this dataset.
Overall, I think this is a borderline paper.
Other remarks:
Note that learning a baseline for REINFORCE has previously been studied, see: https://arxiv.org/pdf/1606.01541v4.pdf (although the authors mention only it briefly in the paper)
“ReasoNets are devised to mimic the inference process of human readers.”
I think this is too strong a claim (that humans use the same kind of ‘iterative hop’ method for answering questions), unless it is supported by actual analysis of humans – I think the similarities to humans are more at a surface level.
Also, I think it’s unnecessary to actually understanding the model. I would change ‘mimic’ to ‘inspired by’, or something along those lines.
EDIT: I thank the authors for taking the time to reply, and clearing some things up with regards to the idea of multiple hops. I am keeping my score the same for the following reason: in my opinion, accepted papers involving new model architectures should either consist of (1) a small adjustment that leads to a large improvement, or (2) a very innovative idea that leads to comparable performance (or better), but provides many new insights about the problem or can be transferred to many different domains. This paper seems to be a rather small adjustment (allowing multiple hops) which results in small improvements on CNN/Daily Mail (which as Reviewer 3 points out has little headroom). I think this paper would be suitable for a conference such as EMNLP.
Thank you for your review. Here are our clarification and we will provide more details later:
1. Why does termination gate help?
Although ReasoNets-Last is a special case of ReasoNets, there are several advantages of ReasoNets compared to the ReasoNet-Last.
First, ReasoNet is much more efficient than ReasoNets-Last in terms of the average number of inference steps at test time.
Second, ReasoNets significantly outperform than ReasoNet-Last on the dataset with complex instances, such as the large graph of the Graph Reachability dataset.
Lastly, the most important advantage of ReasoNet is that the termination gate provides a way to do curriculum learning without human-designed lessons. As reported in the work [1], curriculum Learning is a crucial trick to help models learn difficult tasks.
However, one drawback of curriculum learning is that it requires human-designed lessons (from easy to hard) to iteratively train models. ReasoNet does not need explicit human-designed lessons. With the help of the termination gate, ReasoNet could be able to learn from easy samples to hard samples gradually.
Specifically, given that the termination probability at step i is t_i \Pi_{j=1}^{i-1} (1-t_j), the initial termination probability decreases exponentially. Most samples terminate in the first or two steps after one training epoch on the large graph of Graph Reachability dataset. Along with more training iterations, ReasoNet can gradually allow more reasoning steps for complicated training examples. As we show in the appendix of the paper, the termination steps are strongly correlated with the complexity of test examples. The ReasoNet-Last model does not perform well on the large graph of Graph Reachability dataset. One possible reason is that it does not have the capability to learn from easy to hard.
Hence, it may lead to either over-fitting or a local minimum. We will provide more experimental details and update our claim in the next update.
2. Significance on CNN and Daily Mail?
The CNN/Daily Mail comprehension task is a challenging task. As reported in [2], human could only achieve around 75% accuracy. The 0.5% accuracy improvement can be considered as (empirically) significant improvement in this task, since we are approaching the human-level ceiling. Our proposed model achieves 0.3% accuracy improvement over AoA reader on the CNN dataset, and 1.0% accuracy improvement over GA reader on the Daily Mail dataset. Previous works did not report the detailed results on these two datasets. It will be hard to report significance testing results over baselines.
On the other hand, we have a discussion with AnonReviewer3 regarding the design of baseline and will follow up more for this. For the suggestion of using "inspired by" instead of "mimic", it is a great suggestion and we will revise it in the draft later.
[1] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu & Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. in Nature, 2016.
[2] Danqi Chen, Jason Bolton, and Christopher D Manning. A thorough examination of the CNN / Daily Mail reading comprehension task, in ACL, 2016
The paper aims to consolidate some recent literature in simple types of "reading comprehension" tasks involving matching questions to answers to be found in a passage, and then to explore the types of structure learned by these models and propose modifications. These reading comprehension datasets such as CNN/Daily Mail are on the simpler side because they do not generally involve chains of reasoning over multiple pieces of supporting evidence as can be found in datasets like MCTest. Many models have been proposed for this task, and the paper breaks down these models into "aggregation readers" and "explicit reference readers." The authors show that the aggregation readers organize their hidden states into a predicate structure which allows them to mimic the explicit reference readers. The authors then experiment with adding linguistic features, including reference features, to the existing models to improve performance.
I appreciate the re-naming and re-writing of the paper to make it more clear that the aggregation readers are specifically learning a predicate structure, as well as the inclusion of results about dimensionality of the symbol space.
Further, I think the effort to organize and categorize several different reading comprehension models into broader classes is useful, as the field has been producing many such models and the landscape is unclear.
The concerns with this paper are that the predicate structure demonstrated is fairly simple, and it is not clear that it provides insight towards the development of better models in the future, since the "explicit reference readers" need not learn it, and the CNN/Daily Mail dataset has very little headroom left as demonstrated by Chen et al. 2016. The desire for "dramatic improvements in performance" mentioned in the discussion section probably cannot be achieved on these datasets. More complex datasets would probably involve multi-hop inference which this paper does not discuss.
Further, the message of the paper is a bit scattered and hard to parse, and could benefit from a bit more focus.
I think that with the explosion of various competing neural network models for NLP tasks, contributions like this one which attempt to organize and analyze the landscape are valuable, but that this paper might be better suited for an NLP conference or journal such as TACL.
The paper proposes an architecture called ReasoNet that reason over the relation. The paper addresses important tasks but there are many other related works. The comparison to other methods are not comprehensive. The Graph Reachability dataset is not a good example to use.
1. The Graph reachability problem is an important task to demonstrate the reasoning process of the ReasoNet model. Compared to inference of natural language questions, the proposed graph reachability task provides a way to simulate examples that require multiple-step inference.
Moreover, as pointed out by reviewer 3, it is one of the important contributions of the paper.
2. We have compared with all ten state-of-the-art baselines in "Reading Comprehension" tasks at the time of paper submission,  as shown in Tabel 2 in the paper.  This is a thorough comparison study and we also follow the same approaches adopted by previous works. In the Graph Reachability dataset, we compared it with the baselines "Single-Turn inference model" and "Multi-Turn inference model". Experimental results show the "Graph Reachability task" is challenging, and ReasoNet with adaptive termination steps outperforms previous approaches.
Thus, we already make comprehensive comparison study for both experiments on unstructured and structure data in the paper.  If you think there are many other related works that we should consider as well, would you please let us know about it clearly?
3.
It is worth noting that both "CNN/Daily Mail" and "Graph Reachability" tasks are not a sequence-to-sequence task.
Thus, Pointer Networks cannot be directly applied to them.

Meta-review:
role: PC
data: comment: This paper introduces a method for estimating the number of iterations of an attention mechanism in a neural machine reading module using REINFORCE and a custom baseline, which is estimated on the data. Estimating a baseline from data, multi-hop attention, and modelling latent variable models with REINFORCE are not new, so their composition, while sensible, is slightly incremental. There were some concerns from several of the reviewers with the impact the experiments have in terms of validating the model changes. Improvements on "real" tasks such as CNN/DailyMail did not impress reviewers, some of whom believe the benchmarks compared to were not representative of the best comparable architectures tried on these datasets (e.g. pointer networks, which definitely can be used). Overall, I do not find this paper strong enough to recommend acceptance to the main conference in its current state. With better evaluation, it could be a decently strong paper if the results come through.

Decision: Reject