Title: Reasoning with Memory Augmented Neural Networks for Language Comprehension
Authors: Tsendsuren Munkhdalai
Hong Yu
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Hypothesis testing is an important cognitive process that supports human reasoning. In this paper, we introduce a computational hypothesis testing approach based on memory augmented neural networks. Our approach involves a hypothesis testing loop that reconsiders and progressively refines a previously formed hypothesis in order to generate new hypotheses to test. We apply the proposed approach to language comprehension task by using Neural Semantic Encoders (NSE). Our NSE models achieve the state-of-the-art results showing an absolute improvement of 1.2% to 2.6% accuracy over previous results obtained by single and ensemble systems on standard machine comprehension benchmarks such as the Children's Book Test (CBT) and Who-Did-What (WDW) news article datasets.


Review discussion:
This paper proposes a novel approach at cloze form question answering in which the model iteratively proposes queries to the question until it converges on a query and answer.
Question: what is the relationship between eqn 10 and eqn 13? Is 13 an alternatively way of updating the query? Where is z_t in the query gating approach? For the adaptive computation approach, do you stop based on sampling from the normalized stopping score? Is this stopping decision supervised (it doesn't seem to be)? I would like to see results of T=1 and T=2 of the proposed models to see how important the dynamic aspect of the model is.
Although machine performance on the CNN/Daily News dataset is approaching human performance as the authors noted, I would nevertheless like to see the author demonstrate the effectiveness of this model on the CNN/Daily News dataset, as it has seen more interest in the community and is thus more competitive.
Thank you for the thoughtful comments. Here we address the reviewer's questions below:
> what is the relationship between eqn 10 and eqn 13? Is 13 an alternatively way of updating the query? Where is z_t in the query gating approach?
For the query gating model, the memory is updated twice per step by applying both eqn 10 and 13 sequentially whereas for the adaptive computation model the memory is updated only once with eqn 10. The first update rule defined by eqn 10 involves the information s^d_t retrieved from the document and the memory key z_t. The second update rule of eqn 13 performs gating of the old query information M^q_{t-1} to the current time step. Through this memory gating operation, the write module locks the query content or rolls back the new query updates to prevent the model from forgetting useful information or from making overconfident updates. We showed the effectiveness of the memory gating operation in the query regression visualization of Appendix A.
>For the adaptive computation approach, do you stop based on sampling from the normalized stopping score? Is this stopping decision supervised (it doesn't seem to be)?
For the adaptive computation approach, the write module outputs a halting score ranging from 0 to 1 through its termination head in every step. As the  hypothesis-testing loop proceeds, the scores computed in the previous steps are summed up and once this sum exceeds 1, the model halts the loop and outputs the final answer for a particular query. This stopping decision is not explicitly/directly supervised. The halting score is rather incorporated in the loss function as in eqn 19 for training.
In our approach, no supervision is provided until the models halt the reasoning process, giving the models a freedom to form varying hypotheses. Only a high level supervision to whether the prediction is true or not is given once the models halt the loop. In order to reason about input, the models make multiple intermediate decisions without any explicit supervision, like which memory slot to read new information from, where to write it and when to stop.
>I would like to see results of T=1 and T=2 of the proposed models to see how important the dynamic aspect of the model is.
Although machine performance on the CNN/Daily News dataset is approaching human performance as the authors noted, I would nevertheless like to see the author demonstrate the effectiveness of this model on the CNN/Daily News dataset, as it has seen more interest in the community and is thus more competitive.
We will run the experiments and update the paper when we have the results. Please note that we would train roughly 12 models + CNN/Daily News experiments assuming that all hyper-parameters are available. Each model takes days/weeks to train.
While we agree with the reviewer, we would also like to emphasize the importance of the CBTest and WDW datasets. The CBTest and WDW datasets seem to be gaining popularity quickly as there are multiple submissions at this conference, that are evaluated by using them.
Thanks again for the comments.
Best,
This paper proposed an iterative query updating mechanism for cloze-style QA. The approach is novel and interesting and while it is only verified in the paper for two Cloze-style tasks (CBT and WDW), the concept of read/compose/write operations seem to be more general and can be potentially applied to other reasoning tasks beyond Cloze-style QA.
Another advantage of the proposed model is to learn when to terminate the iteration by the so-called adaptive computation model, such that it avoids the issue of treating the number of iterations as another hyper-parameter, which is a common practice of iterative models/multi-hop reasoning in previous papers.
There are a couple places that this paper can improve.
First, I would like to see the results from CNN/Daily Mail as well to have a more comprehensive comparison.
Secondly, it will be useful to visualize the entire M^q sequence over time t (not just z or the query gating) to help understand better the query regression and if it is human interpretable.
Thank you for the thoughtful review and useful suggestions.
>First, I would like to see the results from CNN/Daily Mail as well to have a more comprehensive comparison.
For the CNN news task, we ran our NSE adaptive model without any hyper-parameter tuning.  It achieved ~74% accuracy (74%~75%) on the CNN test set. The result is not included in our paper as we speculate that the results could be improved further if we have more time. We think that ~74% accuracy is reasonable considering the fact that the human performance on the task is ~75%.
>Secondly, it will be useful to visualize the entire M^q sequence over time t (not just z or the query gating) to help understand better the query regression and if it is human interpretable.
Since the query regression happens in a continuous vector space it is not trivial to directly visualize it.
However, we think that by looking at the z and gating states one could interpret how the query is regressed. As shown in Figure 3, word information provides a cue how answer is kept. As shown in Figure 3, content words (e.g., “Rabbit”) are not overwritten throughout the regression loop while non-content words like '.' and <BOS> are replaced in the very first step.
Thie paper proposed an iterative memory updating model for cloze-style question-answering task. The approach is interesting, and result is good. For the paper, I have some comments:
1. Actually the model in the paper is not single model, it proposed two models. One consists of "reading", "writing", "adaptive computation" and " Answer module 2", the other one is "reading", "composing", "writing", "gate querying" and "Answer module 1". Based on the method section and the experiment, it seems the "adaptive computation" model is simpler and performs better. And without two time memory update in single iteration and composing module, the model is similar to neural turing machine.
2. What is the MLP setting in the composing module?
3. This paper tested different size of hidden state:[256, 368, 436, 512], I do not find any relation between those numbers, how could you find 436? Is there any tricks helping you find those numbers?
4. It needs more ablation study about using different T such as T=1,2..
5. According to my understanding, for the adaptive computation,  it would stop when the P_T <0.
So what is the distribution of T in the testing data?
Thank you for the thoughtful review.
1. Actually the model in the paper is not single model, it proposed two models. One consists of "reading", "writing", "adaptive computation" and " Answer module 2", the other one is "reading", "composing", "writing", "gate querying" and "Answer module 1". Based on the method section and the experiment, it seems the "adaptive computation" model is simpler and performs better. And without two time memory update in single iteration and composing module, the model is similar to neural turing machine.
Comparing NSE to NTM: NTM has a single centralized controller, which is usually an MLP or RNN while NSE takes a modular approach. The main controller in NSE is decomposed into three separate modules, each of which performs for read, compose or write operation. In NSE, the compose module is introduced in addition to the standard memory update operations (i.e. read-write) in order to process the memory entries and input information.
The main advantage of NSE is in its memory update. The NTM controller does not have mechanism to avoid information collision in the memory. Particularly the NTM controller emits two separate set of access weights (i.e. read weight and erase and write weights) that do not explicitly encode the knowledge about where information is read from and written to.
Moreover NTM has a fixed-size memory with no memory allocation or de-allocation protocol.
Therefore unless the controller is intelligent enough to track the previous read/write information, which is hard for an RNN when processing long sequences, the memory content is overlapped and information is overwritten throughout different time scales. We think that this is a potential reason that makes NTM hard to train and makes the training not stable. We also note that the effectiveness of the location based addressing introduced in NTM is unclear.
In NSE, we introduce a systematic memory update approach based on the soft attention mechanism. NSE writes new information to the most recently read memory locations. This is accomplished by sharing the same memory key vector between the read and write modules. The NSE memory update is scalable and potentially more robust to train. NSE is provided with a variable sized memory and thus unlike NTM, the size of the NSE memory is more relaxed. The novel memory update mechanism and the variable sized memory together prevent NSE from the information collision issue and avoid the need of the memory allocation and de-allocation protocols.
Each memory location of the NSE memory stores a token representation in input sequence during encoding. This provides NSE with an anytime-access to the entire input sequence including the tokens from the future time scales, which is not permitted in NTM, RNN and attention-based encoders.
2. What is the MLP setting in the composing module?
We used a single layer MLP with 436 hidden units and ReLU activation for the composition module.
3. This paper tested different size of hidden state:[256, 368, 436, 512], I do not find any relation between those numbers, how could you find 436? Is there any tricks helping you find those numbers?
It is usually not trivial to find those hyper-parameter settings as the search space is huge. Grid search is preferred for small tasks that we can afford multiple runs. For the large scale tasks like ones addressed in this paper, we turn to a random search or rely on previous experiences. Those hidden state sizes mainly came from the previous work, except 436. 436 was selected by a random search and it worked better than 256, 368 and 512 that were taken from the related work.
4. It needs more ablation study about using different T such as T=1,2.. 5. According to my understanding, for the adaptive computation, it would stop when the P_T <0.
So what is the distribution of T in the testing data?
For the given time and resource, we were able to conduct an additional experiment with 12 different runs with T=1,2. In the updated manuscript we also discussed about using different T, maximum number of permitted steps. In general given a small number of permitted steps, we found that our models less likely overfit although the final performance is not high. As the number of permitted steps increases both dev and test accuracy improves yielding an overall higher performance.
However, this holds up to a certain point.
When the permitted steps are large, we no longer observe a significant performance improvement in terms of testing accuracy.
For example, NSE Query Gating model with T=15  achieved 79.2% dev accuracy and 71.4% test accuracy on CBT-NE task, showing the highest accuracy on the dev set yet massively overfitting on the test set.
Furthermore it becomes expensive to train a model with a large number of allowed steps.
During the qualitative analysis, we found out that the distribution of T is mostly concentrated at 2. This is also indicated in the query regression visualizations in Appendix A. As noted in the manuscript, our adaptive computation model adjusts T for a particular task (i.e document and question pair). More recently and following our work, the adaptive computation framework was successfully adapted for image classification and object recognition tasks [1]. Particularly the adaptive computation framework was deployed within residual networks to decide when to halt the computation.
Finally for the CNN news task, we ran our NSE adaptive model without any hyper-parameter tuning.  It achieved ~74% accuracy (74%~75%) on the CNN test set. The result is not included in our paper as we speculate that the results could be improved further if we have more time. We think that ~74% accuracy is reasonable considering the fact that the human performance on the task is ~75%.
Ref:
1. Figurnov, Michael, et al. "Spatially Adaptive Computation Time for Residual Networks." arXiv preprint arXiv:1612.02297 (7 Dec 2016).
First I would like to apologize for the delay in reviewing.
Summary : this work introduces a novel memory based artificial neural network for reading comprehension. Experiments show improvement on state of the art.
The originality of the approach seems to be on the implementation of an iterative procedure with a loop testing that the current answer is the correct one.
In order to get a better sense of the reason for improvement it would be interesting to have a complexity and/or a time analysis of the algorithm. I might be mistaken but I don't see you reporting anything on the actual number of loops necessary in the reported experiments.
The dataset description in section 2.2, should be moved to section 4 where the other datasets are described.
Thank you for the thoughtful review.
>In order to get a better sense of the reason for improvement it would be interesting to have a complexity and/or a time analysis of the algorithm. I might be mistaken but I don't see you reporting anything on the actual number of loops necessary in the reported experiments.
The computational complexity of the model depends on length of the document and query, and the parameter T which is the maximum number of allowed steps. Our ablation study on parameter T (Table 1 and 2 along with the discussion) shows that the larger value of T, the longer the training and the more likely the models overfit.
Therefore T is usually constant and defines the actual number of loops. It is orders of magnitude smaller to compare with the length of the document which is ~500. The main bottleneck is the memory initialization with bi-directional LSTM because it has to sequentially read each document word. The memory read and write is as fast as NSE uses the dot product for content similarity. It is even faster than neural turing machines since NSE calculates the memory key vector z once and uses it for both read and write.
1) We performed query regression analysis and added visualizations in the appendix.
2) We conducted additional experiments of 12 different runs when T=1,2 and included the results and discussion.

Meta-review:
role: PC
data: comment: This paper proposes a memory-enhanced RNN in the vein of NTM, and a novel training method for this architecture of cloze-style QA. The results seem convincing, and the training method is decently novel according to reviewers, although the evaluation seemed somewhat incomplete according to reviewers and my own reading. For instance, it is questionable whether or not the advertised human performance on CNN/DM is accurate (based on 100 samples from a 300k+ dataset), so I'm not sure this warrants not evaluating or reporting performance on it. Overall this looks like an acceptable paper, although there is room for improvement.

Decision: Accept (Poster)