Title: Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes
Authors: Caglar Gulcehre
Sarath Chandar
Kyunghyun Cho
Yoshua Bengio
Venue: ICLR 2017
Keywords: Deep learning
Natural language processing
Reinforcement Learning

Abstract: In this paper, we extend neural Turing machine (NTM) into a dynamic neural Turing machine (D-NTM) by introducing a trainable memory addressing scheme. This addressing scheme maintains for each memory cell two separate vectors, content and address vectors. This allows the D-NTM to learn a wide variety of location-based addressing strategies including both linear and nonlinear ones. We implement the D-NTM with both continuous, differentiable and discrete, non-differentiable read/write mechanisms. We investigate the mechanisms and effects for learning to read and  write to a memory through experiments on Facebook bAbI tasks using both a feedforward and GRU-controller. The D-NTM is evaluated on a set of Facebook bAbI tasks and shown to outperform NTM and LSTM baselines. We also provide further experimental results on sequential MNIST, associative recall and copy tasks.

TL;DR: We propose a new type of Neural Turing Machine, which is simpler than the original model and achieves better results than the baselines on non-trivial tasks. 

Review discussion:
I think the footnote is missing.
Thanks for pointing out our mistake. That footnote was unnecessary, we are going to remove in the next updated version.
You introduce the address matrix, so I expected that you only use the address vectors $a$ for the addressing mechanism but in chapter 3 where you describe the addressing mechanism, you use the whole memory $m$ and not just $a$. Is that right? If so, I don't see where you use the address matrix at all and how this is different to the normal NTM. Can you explain? If you meant $a$ instead of $m$ in chapter 3, the update $\overline{m}$ is still meant to update both address vector and content vector, right?
I think for better understandability, you should state the vector space of all the variables, e.g. $\overline{m}^t \in \R^{d_h}$, etc.
Also, $m_i$ is maybe confusing, why not call it $M_i$ or $M[i]$?
In equation (3), the $h^t$ and $x^t$ are not bold, are that different variables?
For erasing and writing, you use $e^t$ and $u^t_j$.
So it means that the erase is global over all memory cells?
However, in chapter 3, you say that the erase ($e$) vector is just the same as the read ($w$) and write ($u$) vector.
So should it be $e^t_j$ here?
And is it $e^t \in \R^{d_h}$ or is it a scalar?
And is it $u^t_j \in \R^{d_h}$ or is it a scalar?
In chapter 3, you write $w^t_i$ which is a scalar. In equation (5), it is $w^t$ in bold, so that is over all $i$, i.e. $w^t \in \R^N$?
In chapter 7.1.2, what is CBA?
In table 1, what is LBA?
In Table 1 & 2, you call it "Soft D-NTM" and in Table 3 & 4, you call it "D-NTM cont.". Is that the same?
Have you tried to use a NTM with multiple steps? I only see 1-step NTMs.
Why do you think the discrete D-NTM is worse than the cont. D-NTM for pMNIST?
As well as with FF controller (Table 2).
The curriculum strategy of chapter 4 was only applied for bAbI trained with FF controler (Table 2) and not for the other experiments?
Thanks for your valuable comments and questions.
> You introduce the address matrix, so I expected that you only use the address vectors $a$ for the addressing mechanism but in chapter 3 where you describe the addressing mechanism, you use the whole memory $m$ and not just $a$. Is that right?
Addressing for D-NTM involves using both the address vectors $a$ and the content vectors $c$. $m$ in the paper corresponds to the concatenation of $a$ and the $c$ vectors. By using $m$ instead of $a$ in the addressing mechanism our model can combine both the content-based and the location-based addressing together.
> If so, I don't see where you use the address matrix at all and how this is different to the normal NTM. Can you explain? If you meant $a$ instead of $m$ in chapter 3, the update $\overline{m}$ is still meant to update both address vector and content vector, right?
$m$ is being used only when we compute the cosine similarity between the the key generated by the controller and the cells $m$ in the memory.
However, when we update the memory, we do not update/write the address part of the memory.
However, since the $a$ is still being used to compute the addressing weights, it can still be learned with the backpropagation. In the Equation 6, the $m_t$ should have been $c_t$. Thanks for your remark and the correction.
> I think for better understandability, you should state the vector space of all the variables, e.g. $\overline{m}^t \in \R^{d_h}$, etc.
Also, $m_i$ is maybe confusing, why not call it $M_i$ or $M[i]$?
Thanks for pointing this. We are going to fix them in the next update of the paper.
> In equation (3), the $h^t$ and $x^t$ are not bold, are that different variables?
Thanks for pointing out this typo. We are going to fix it.
>For erasing and writing, you use $e^t$ and $u^t_j$.
So it means that the erase is global over all memory cells?
The writing mechanism is very similar to the one used in [1]. $e^t$ is a vector in $R^{d_c}$ where $d_c$ is the number of features in the memory cell. It influences the columns of the memory content and $u^t_j$ is a scalar that influences each memory cell/row.
> However, in chapter 3, you say that the erase ($e$) vector is just the same as the read ($w$) and write ($u$) vector.
So should it be $e^t_j$ here?
Computation of the address vector described in Section 3.1 is only true for "read" and "write" operations. As described above erase vector is shared by all the memory cells and it is just an sigmoid MLP conditioned on the hidden state of the controller as also described in [1]. Thanks for pointing out our mistake in the equations. We are going to fix this.
> And is it $e^t \in \R^{d_h}$ or is it a scalar?
As described earlier $e^t \in \R^{d_c}$ where $d_c$ corresponds to the number of features in $c^t$. We will fix this mistake.
> And is it $u^t_j \in \R^{d_h}$ or is it a scalar?
Yes it is scalar.
> In chapter 3, you write $w^t_i$ which is a scalar. In equation (5), it is $w^t$ in bold, so that is over all $i$, i.e. $w^t \in \R^N$?
Yes that is true.
> In chapter 7.1.2, what is CBA? In table 1, what is LBA?
CBA corresponds to the "Content Based Addressing". LBA  corresponds to the "Location based addressing". As mentioned in Table 1, LBA^$\ast$ combines both location and the content based addressing as described in [1].
> In Table 1 & 2, you call it "Soft D-NTM" and in Table 3 & 4, you call it "D-NTM cont.". Is that the same?
Yes they are same, we will make it consistent.
> Have you tried to use a NTM with multiple steps? I only see 1-step NTMs.
In Table 3, we have shown multi-step results (3-step results) for both NTM and D-NTM.
> Why do you think the discrete D-NTM is worse than the cont. D-NTM for pMNIST?
The length of the sequences on p-mnist is much longer than the bAbI tasks. The variance of the reinforce increases with the length of the sequences as well, which makes it more difficult to train over very long sequences.
> As well as with FF controller (Table 2).
We found training with REINFORCE by just using FF-controller to be very challenging. The main reason is that the model has to learn to read the correct location since the only memory that the controller relies on is the external memory that it uses.
However, in GRU-controller the model can also use the GRU units of the controller as a memory.
> The curriculum strategy of chapter 4 was only applied for bAbI trained with FF controler (Table 2) and not for the other experiments?
The reason why we use curriculum learning for FF-controller using discrete attention was due to the difficulty of training which arises due to the difficulty of learning a proper addressing mechanism as we discussed earlier.
However, for the other experiments we used GRU-controller which we did not experience any difficulty in terms of optimization.
[1] Graves, A., Wayne, G., & Danihelka, I. (2014). Neural turing machines. arXiv preprint arXiv:1410.5401.
Thanks again for your detailed comment and feedbacks for our paper. We have fixed the typos pointed out by you in this comment and added a new revision of the paper.
Is sole FF being compared in table 1?
sec 7.1.2 mentioned that FF controller are harder to train then why is table 2 number are better than table 1?
how comparable are the D-NTM vs MemN2N in table 1 in terms of the number of parameters and representation power?
> Is sole FF being compared in table 1?
We have put the results of FF-controller in Table 2.
On the other hand, our overall best performing model was D-NTM with FF-controller and soft attention. The rest of the results for the same data configurations, but for the GRU controller are presented in Table 1.
> sec 7.1.2 mentioned that FF controller are harder to train then why is table 2 number are better than table 1?
Indeed, FF controller with discrete attention is more difficult to train than the GRU controller using soft/discrete addressing. This phenomenon can be observed from the low performance compared to the results of FF-controller with soft attention and GRU-controller with discrete attention.
However, this does not apply to FF-controller with soft attention. The final generalization performance of the model using FF-controller was better. The main reason of why GRU controller underperforms (in terms of generalization) compared to the FF-controller and sometimes GRU controller tends to learn to ignore the memory and despite that still manages to achieve good training performance just by relying on the memory of the controller.
> how comparable are the D-NTM vs MemN2N in table 1 in terms of the number of parameters and representation power?
D-NTM with FF-controller has almost the same number of parameters with the MemN2N.
However, GRU controller has more parameters (but still comparable to the MemN2N).
The model described in his paper is quite complicated and reproducing the method from its description may be challenging: Are you planing to release the code and what is your estimate release date?
Different cost functions and regularization are introduce in the paper. Would it be possible to summarize the overall cost function minimized by this model?
Some variables seems to have different definition.
In particular w_t and b, would it be possible to clarify this?
“gamma_t is a shallow MLP”: What does it mean? Is gamma_t a function or a variable? It seems from eq. (10) that it is a vector (or a scalar?).
"curriculum learning for the discrete attention": Can you compare this to simpler schemes?
Like rounding the continuous attention?
In your introduction, you state that "it is possible to use the discrete non-differentiable attention mechanism", referencing Zaremba & Sutskever, 2016 that use REINFORCE and a simple controller.
However, in your sec. 4, you are stating that  "Training discrete attention with feed-forward controller and REINFORCE is challenging". It seems that these two statements contradict each other, could you comment on it?
In the introduction, you state that "Memory network (Weston et al. 2015b) [..] uses an attention-based mechanism to index them". To the best of my knowledge, it is actually Sukhbaatar et al., 2015 that has introduce the attention mechanism to memory networks.
In the introduction, the authors state that “memory networks […] [are] used in real tasks (Bordes et al. 2015, Dodge et al. 2015)”.
However in both papers, they use memory network systems different from Weston et al. The current statement is quite misleading, would it be possible to clarify it?
We apologize for the delay and thank you for your comments. We have uploaded a newer version of the paper, fixing the issues you have pointed out in your earlier comment.
> Are you planning to release the code and what is your estimate release date?
Yes, we were planning to release our codes and have released them on github:
https://github.com/caglar/dntm
> summarize the terms in the cost …
In our paper, we have defined our cost with the REINFORCE gradients. We provide the cost which the model is minimizing in Section 4. In that equation, seven different terms appear. One of them is for the cross-entropy cost for predicting the answer to the question. The rest of the six terms are basically defined for the REINFORCE with the read, write and erase head’s terms along with the entropy regularizations. 3 of the 6 terms are basically the entropy regularization terms for the REINFORCE. In a nutshell the cost function, we have defined in section 4, $C^n(\TT)$ is just minimizing the cross-entropy for predicting the answer to the question with REINFORCE and entropy regularization for the read, write and erase heads.
> Some variables seems to have a different definition.
In particular w_t and b, would it be possible to clarify this?
OK thanks, we will clarify those definitions.
> “gamma_t is a shallow MLP”: What does it mean? Is gamma_t a function or a variable? It seems from eq. (10) that it is a vector (or a scalar?).
gamma_t is an output of a single-layer MLP conditioned on the hidden state of the controller. As we mention in the same quoted sentence, $\gamma_t$ is a scalar (since u_t^{\gamma} is a vector and h_t^{\gamma} is also a vector). We will clarify this in the text further.
> "curriculum learning for the discrete attention": Can you compare this to simpler schemes?
Like rounding the continuous attention?
In Appendix E, we have provided additional experimental results on training with soft attention during the training and rounding attention at the test-time (namely using discrete attention).
However, this approach didn’t work very well.
>  In the introduction, you state that "Memory network (Weston et al. 2015b) [..] uses an …
In Memory network (Weston et al. 2015b) paper, authors defined a matching function, s(.) which assigns a weight to each memory location in the memory.
Although they didn’t train their model in an end-to-end manner and s(.) does not use a softmax. It is still possible to think of s(.) as a discrete attention mechanism.
However, indeed the soft-attention mechanism in a more traditional sense (as is done in NMT) is introduced to memory networks in Sukhbaatar et al., 2015.
> ... the authors state that “memory networks […] [are] used in real tasks (Bordes et al. 2015, Dodge et al. 2015)”.
Thanks for your comment, we will further clarify those paragraphs. In that paragraph, by referring to “memory networks”, we do not discuss the particular memory model proposed by Weston et al, 2015.
Rather, we are referring to models using memory without learning to write as opposed to NTM and gave two successful applications of those models on different NLP tasks.
> In your introduction, you state that "it is possible to use the discrete non-differentiable attention mechanism", referencing Zaremba & Sutskever, ...
Zaremba & Sutskever used an LSTM controller and for each timestep, their controller emits the current input tape, the value of the current memory cell, and a representation of all the actions that have been taken in the previous timestep. The addressing mechanism of RL-NTM is very different from D-NTM too. In that sense, it is difficult to compare the both models in an absolute manner.
But our model with FF-controller and the discrete attention is simpler than the RL-NTM’s LSTM controller and D-NTM’s GRU controller, both in terms of "number of parameters" and in terms of "difficulty of implementation". Let us note that our FF-controller is just a simple MLP which receives the current input(representation of the fact) and the memory cell.
On the other hand, FF-controller does not have a memory itself.
Thus it can not keep a history of its previous actions and its only way to access the previously seen facts is the external memory.
However, LSTM/GRU controller has their own memory through the recurrent connections across the cell and the hidden-states.
Thus an LSTM controller with REINFORCE over the external memory can still answer an answer a question correctly, even if the controller fails to read the correct memory location by falling back to the memory of the LSTM controller.
However, feedforward controller can not do that, since the only memory that it relies on to answer the question is the external memory.
However, with soft attention and the feedforward controller, the controller almost never puts 0 weight on the correct memory cell in the external memory which helps the gradients flow through the write memory location and learn the addressing more easily.
The paper extends the NTM by a trainable memory addressing scheme.
The paper also investigates both continuous/differentiable as well as discrete/non-differentiable addressing mechanisms.
Pros:
* Extension to NTM with trainable addressing.
* Experiments with discrete addressing.
* Experiments on bAbI QA tasks.
Cons:
* Big gap to MemN2N and DMN+ in performance.
* Code not available.
* There could be more experiments on other real-world tasks.
Thanks for your valuable feedback and review.
> Big gap to MemN2N and DMN+ in performance.
Both MemN2N and DMN+ computes the attention over the whole input context, such that its memory grows linearly as the input sequence grows. For the bAbI task, very likely that the optimal strategy is to perform attention over all facts in the story.
However, D-NTM keeps a limited-size memory and it learns to write into the memory as well which is much more difficult.
In particular, on tasks which involve ambiguities, it is very easy for the controller to learn a deficient explicit memory usage that does not generalize well for the other tasks. Because the part of the memory to read from or write into becomes ambiguous as well.
Furthermore, reading depends on the writing as well, if the controller fails to write into the correct location in the memory, reading from that part of the memory becomes futile. The advantage of using limited external memory becomes more evident when the input sequence gets very long, such that computation of attention over the whole input sequence would be infeasible.
> Code not available.
We made our code freely available at:
https://github.com/caglar/dntm
> There could be more exp…
Thanks for noticing this we will add more results on more realistic tasks. We plan to add one more experiment on SNLI task to the paper.
This paper introduces a variant of the neural Turing machine (NTM, Graves et al. 2014) where key and values are stored. They try both continuous and discrete mechanisms to control the memory.
The model is quite complicated and seem to require a lot of tricks to work.
Overall it seems that more than 10 different terms appear in the cost function and many different hacks are required to learn the model. It is hard to understand the justification for all of these tricks and sophisticated choices. There is no code available nor plan to release it (afaik).
The model is evaluated on a set of toy problems (the “babi task”) and achieves performance that are only slightly above those of a vanilla LSTM but are much worse than the different memory augmented models proposed in the last few years.
In terms of writing, the description of the model is quite hard to follow, describing different blocks independently, optimization tricks and regularization. The equations are hard to read, using non standard notation (e.g., “softplus”), overloading notations (w_t, b…), or write similar equations in different ways (for example, eq (8-9) compared to (10-11). Why are two equations in scalar and the other in vectors? Why is there an arrow instead of an equal?…).
Overall it is very hard to put together all the pieces of this model(s), there is no code available and I’m afraid there is not enough details to be able to reproduce their numbers.
Finally, the performance on the bAbI tasks are quite poor compared to other memory augmented models.
Thanks for your feedback.
>  Overall it seems that more than 10 different terms appear in the cost function and many different hacks are required to learn the model.
Please note that 6 of the terms of our cost is for the REINFORCE with entropy regularization. REINFORCE with entropy regularization is a very standard way to train neural networks with discrete stochastic decision variables and has been used in many other papers [1,2 ...]. One of the term is the original cross-entropy cost function for predicting the answer to the question about the story. The other two terms are for the regularizations, which are justified in the paper.
Thus if we include the original cost, REINFORCE with entropy regularization and the other two regularizations which we have proposed in this paper (next fact prediction and the read/write consistency) we would have 9 terms.
However, only TWO OF THE TERMS of the cost are introduced in our paper. The rest of the terms are very typical for neural networks trained with REINFORCE. We have justified all the design choices we have taken in our paper and achieved significant improvements over LSTMs despite not using tricks such as linear restart, multi-runs with different seeds (or hyperparameters), joint training, etc … which have been used in most of the recent memory augmented neural networks related papers.
>  slightly above those of a vanilla LSTM
On bAbI task, LSTM got 36.41% and this result is reported in Sukhbaatar 2015 for non-joint training (with joint training it is possible to improve this result, but that would not be a fair comparison). Our model with GRU controller got 21.8 and using feedforward controller we got 12.8 percent error over all tasks. We find the improvement from 36.41% to 12.8% error to a significant improvement.
> There is no code available nor plan to release it (afaik).
We have released our code on github: https://github.com/caglar/dntm . We are improving our codebase and adding more documentation into our repository at the moment.
> The equations are hard to read, using non standard notation (e.g., “softplus”) …
We find “softplus” to be a widely adopted term for log(exp(x)+1) and many papers in machine learning literature has used it [3].
However, we agree that some people still may not be very familiar with it.
We have already improved the readability of our equations in the direction that you have suggested. Please check the new revised version of the paper. We will make further clarifications(mostly in Section 2 and 3) and upload another version of the paper soon as well.
[1] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., ... & Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044, 2(3), 5.
[2] Vezhnevets A, Mnih V, Osindero S, Graves A, Vinyals O, Agapiou J. Strategic attentive writer for learning macro-actions. In Advances in Neural Information Processing Systems 2016 (pp. 3486-3494).
[3] https://scholar.google.ca/scholar?q=softplus+machine+learning&btnG=&hl=en&as_sdt=0%2C5
Dear Reviewers and Readers,
For the codes of the models and the tasks which we have explored/experimented in our paper, please see our repo:
https://github.com/caglar/dntm/
We are still in the process of refactoring and adding more documentation for our code. We also hope that the framework we used in our paper, will be providing an easy framework to implement Memory Augmented Neural Networks(MANNs) in Theano.
The authors proposed a dynamic neural Turing machine (D-NTM) model that overcomes the rigid location-based memory access used in the original NTM model. The paper has two main contributions: 1) introducing a learnable addressing to NTM. 2) curriculum learning using hybrid discrete and continuous attention. The proposed model was empirically evaluated on Facebook bAbI task and has shown improvement over the original NTM.
Pros:
+ Comprehensive comparisons of feed-forward controllers v.s. recurrent controllers
+ Encouraging results on the curriculum learning on hybrid discrete and continuous attentions
Cons:
- Very weak NTM baseline (due to some hyper-parameter engineering?) in Table 1, 31% err. comparing to the NTM 20% err. reported in Table 1 in(Graves et al, 2016, Hybrid computing using a neural network with dynamic external memory).
In fact, the NTM baseline in (Graves et al 2016) is better than the proposed D-NTM with GRU controller. Maybe it is worthwhile to reproduce their results using the hyper-parameter setting in their Table2 which could potentially lead to better D-NTM performance?
- Section 3 of the paper is hard to follow. The overall clarity of the paper needs improvement.
Thanks for your valuable feedback and comments about our paper.
> Very weak NTM baseline ...
The experimental setup followed on the bAbI dataset in DNC paper (Graves et al 2016) and our paper is very different.
Firstly, in Graves et al 2016, they report results only on joint-training.
However, in our paper, we have only trained the models on each task separately. Potentially, the transfer between different tasks in joint training can have a huge impact on the results and it can improve generalization.
Most importantly, 20% error across over all the tasks which "Graves et al 2016" reports is the result obtained with the BEST NETWORK on the validation set after 20 RUNS (please see the caption of Table 1), whereas our results are obtained only from a single-run.
However, in DNC paper (see Table 1), they also report mean results of their networks along with the best network after the multiple runs, their mean result is very close to ours 28.5% with std of +/- 2.9.
Nevertheless, please note that the 31% result which we report in our paper is very close to the mean result of the NTM in the DNC paper (within a single standard deviation of their mean result).
Lastly, we use the representation of GRU over the tokens in each fact of every story in our paper.
However, in Graves et al 2016, their models emit the word embeddings for all the facts in the story.
> Section 3 of the paper is hard to follow…
Thanks for the comment, we have improved the readability of both the Section 2 and Section 3  to fix the issue that you have pointed out and improved the clarity of our paper. We will further improve those two sections and upload a newer version of the paper to openreview again.
If these two concerns are the main reasons for you to give a low score to our paper, we would kindly ask you to re-evaluate your decision.
The reported results of this paper are indeed close to DNC and have demonstrated the effectiveness of the proposed improvements. It is a solid incremental paper and the experimental result is encouraging.  I improved my previous rating of the paper.

Meta-review:
role: PC
data: comment: This paper proposes some novel architectural elements, and the results are not far from published DNC results. However, the main issues of this paper are the complexity of the model, lack of justification for certain architectural choices, gaps with reported DNC numbers on BABI, and also a somewhat toy-ish task.

Decision: Reject