Title: Learning to Remember Rare Events
Authors: Lukasz Kaiser
Ofir Nachum
Aurko Roy
Samy Bengio
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Despite recent advances, memory-augmented deep neural networks are still limited
when it comes to life-long and one-shot learning, especially in remembering rare events.
We present a large-scale life-long memory module for use in deep learning.
The module exploits fast nearest-neighbor algorithms for efficiency and
thus scales to large memory sizes.
Except for the nearest-neighbor query, the module is fully differentiable
and trained end-to-end with no extra supervision.  It operates in
a life-long manner, i.e., without the need to reset it during training.

Our memory module can be easily added to any part of a supervised neural network.
To show its versatility we add it to a number of networks, from simple
convolutional ones tested on image classification to deep sequence-to-sequence
and recurrent-convolutional models.
In all cases, the enhanced network gains the ability to remember
and do life-long one-shot learning.
Our module remembers training examples shown many thousands
of steps in the past and it can successfully generalize from them.
We set new state-of-the-art for one-shot learning on the Omniglot dataset
and demonstrate, for the first time, life-long one-shot learning in
recurrent neural networks on a large-scale machine translation task.

TL;DR: We introduce a memory module for life-long learning that adds one-shot learning capability to any supervised neural network.

Review discussion:
As I understand it, the value $v$ is the supervised desired target value which you use during training.
Then every memory operation will store this supervised value $v$?
So memory updates are only done during training? Or what value $v$ do you use during evaluation / inference?
The output of the memory module $V[n]$ is always only a scalar, or even an scalar integer?
As well as $v$?
In Convolutional Network with Memory, the output is the nearest neighbor from the memory module, i.e. the output of the network is also just a single scalar?
In GNMT with added memory module, how exactly is $\hat{y}_{m+1}$ defined?
For translation if you don't know the target sequence, how would you update and use the memory module?
In our experiments, the values in the memory module V[n] are indeed integer scalars, and during training they are updated using the supervised value v which is also an integer scalar -- the label to be produced by the network at this time-step.
This holds both for RNNs and for the Convolutional Network, exactly as you wrote.
In GNMT with memory, the output y_{m+1} is created as in the pure GNMT, by a softmax layer, but the input to the softmax is created by an additional linear layer that gets two concatenated tensors as input: the output of the last LSTM layer (as in pure GNMT) and the embedded output of the memory module. To embed the scalar, we use the same embedding as the decoder.
For translation experiments with context: the memory is not updated during evaluation, only during training.
But when context is given, we want to update the memory but not the weights. This is equivalent to a training step with learning-rate of 0.
Thank you for a very interesting paper.
I am not sure if I understand the translation experiment correctly.
Can you briefly elaborate on the purpose of even and odd splitting and why it improves performance?
The test set is composed of a few paragraph with sentences coming in order, e.g., say the first 6 sentences are from the same newspaper article in the order they appear in the article.
So when we give odd sentences as context and test the model on even sentences, it will get sentences 1, 3, 5 as context and will be tested on 2, 4, 6. Since they come from the same paragraph, the memory updates from 1, 3, 5 actually help with translating 2, 4, 6.
We re-wrote this paragraph of the experimental section in the new revision and hope that it is clearer now.
1. What is the memory size used in all the three tasks?
2. For the synthetic task, what is the size of the training data?
3. In the synthetic task, the big performance gap between baseline extended neural GPU with and without memory module is surprising. It would be interesting if authors can also report results on seq2seq+attention with and without memory module.
4.
Last few lines in Translation section is not clear. What do you mean by providing the whole test context? Model sees all the test sentences and translations and then asked to translate the same?
5. I do not agree with the comment in the related work section that hard access queries have limited success. See for example [1]
[1] Gulcehre, Caglar, Chandar, Sarath, Cho, Kyunghyun, and Bengio, Yoshua. Dynamic neural turing
machines with soft and hard addressing schemes. CoRR, abs/1607.00036, 2016. URL http:
//arxiv.org/abs/1607.00036.
Great thanks for your questions, we included the answers in the text of the paper in the newly uploaded revision.
1. What is the memory size used in all the three tasks?
We used memory of half a million.
2. For the synthetic task, what is the size of the training data?
It is 40K, so each from the 16K random examples is there 2 or 3 times.
3. In the synthetic task, the big performance gap between baseline extended neural GPU with and without memory module is surprising. It would be interesting if authors can also report results on seq2seq+attention with and without memory module.
We trained a 2-layer 256-unit LSTM seq2seq with attention on this task. It overfits the data, memorizing the traning set but with < 1% accuracy on the test set. The same model with a memory module does much better, accuracy is > 30%. We will train a few more models to see how lower sizes affect the results.
But from our experience with the Extended Neural GPU models on this task, we'd expect the small models to under-fit and not get good results. We'll report a more thorough model-size-vs-accuracy in the final version. In all cases, the models with memory module get better results.
4.
Last few lines in Translation section is not clear. What do you mean by providing the whole test context? Model sees all the test sentences and translations and then asked to translate the same?
This part was re-written to clarify. Yes - the model sees all test sentences, but cannot update its weights, only memory, and is then asked to translate the same. We use it only to assess the headroom that the memory can bring.
5. I do not agree with the comment in the related work section that hard access queries have limited success. See for example [1]
We were not aware of [1] before, the related work section was changed accordingly.
We are very grateful for the reviewers' questions, and we uploaded a new revision that clarifies the paper to answer them.
In addition to the answers, we updated the definition of the memory module for the cases when no positive neighbour is found in the top-k. We now get any vector from memory instead of using all 0s, which prevents the loss from jumping in such cases. This improved our results on Omniglot, they are state-of-the-art now.
The paper proposes a new memory module to be used as an addition to existing neural network models.
Pros:
* Clearly written and original idea.
* Useful memory module, shows nice improvements.
* Tested on some big tasks.
Cons:
* No comparisons to other memory modules such as associative LSTMs etc.
A new memory module based on k-NN is presented.
The paper is very well written and the results are convincing.
Omniglot is a good sanity test and the performance is surprisingly good.
The artificial task shows us that the authors claims hold and highlight the need for better benchmarks in this domain.
And the translation task eventually makes a very strong point on practical usefulness of the proposed model.
I am not a specialist in memory networks so I trust the authors to double-check if all relevant references have been included (another reviewer mentioned associative LSTM).
But besides that I think this is a very nice and useful paper. I hope the authors will publish their code.
This paper proposes a new memory module for large scale life-long and one-shot learning. The module is general enough that the authors apply the module to several neural network architectures and show improvements in performance.
Using k-nearest neighbors for memory access is not completely new. This has been recently explored in Rae et al., 2016 and Chandar et al., 2016. K-nearest neighbors based memory for one-shot learning has also been explored in [R1]. This paper provides experimental evidence that such an approach can be applied to a variety of architectures.
Authors have addressed all my pre-review questions and I am ok with their response.
Are the authors willing to release the source code to reproduce the results?
At least for omniglot experiments and synthetic task experiments?
References:
[R1] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rae, Daan Wierstra, Demis Hassabis: Model-Free Episodic Control. CoRR abs/1606.04460 (2016)
Dear Reviewer,
We are grateful for your review. We will release the source code for the memory module class in TensorFlow that will make it easy to add it to any model, and the code to replicate our results, at least on Omniglot (the GNMT model code is not open-source, so we cannot promise to release code for these experiments, but the module can be added to other TF seq2seq models easily and we might give example code for that).
We will also add [R1] to our related work section.
Are there any other concerns or questions that we could address to improve the rating?

Meta-review:
role: PC
data: comment: The primary contribution of this paper is showing that k-nearest-neighbor method based memory can be usefully incorporated in a variety of architectures and supervised learning tasks. The presentation is clear, and results are good. I like the synthetic task and analysis. For the Omniglot task, running and reporting results on the original splits used by Lake would be good, as the splits used in matching nets are considerably easier and result in ceiling effects.

Decision: Accept (Poster)