Title: Hierarchical Memory Networks
Authors: Sarath Chandar
Sungjin Ahn
Hugo Larochelle
Pascal Vincent
Gerald Tesauro
Yoshua Bengio
Venue: ICLR 2017
Keywords: Deep learning
Natural language processing

Abstract: Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which require the network to read from extremely large memories. On the other hand, it is well known that hard attention mechanisms based on reinforcement learning are challenging to train successfully. In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks. The memory is organized in a hierarchical structure such that reading from it is done with less computation than soft attention over a flat memory, while also being easier to train than hard attention over a flat memory. Specifically, we propose to incorporate Maximum Inner Product Search (MIPS) in the training and inference procedures for our hierarchical memory network. We explore the use of various state-of-the art approximate MIPS techniques and report results on SimpleQuestions, a challenging large scale factoid question answering task.


TL;DR: We propose a hierarchical memory organization strategy for efficient memory access in memory networks with large memory.

Review discussion:
1. Any reason to not learn the memories, as in the case of standard memory networks? Is the model hard to learn, given that it also has to learn MIPS?
2. Why not report the wall clock time in the experiments to get a sense of speedup? Comparing the number of epochs across models is not fair because each epoch could take different amount of time for different models.
>> Any reason to not learn the memories, as in the case of standard memory networks? Is the model hard to learn, given that it also has to learn MIPS?
Yes it is hard to learn both the memory and k-MIPS. Because, all approximate k-MIPS algorithms assume that the memory is fixed. If we start updating the memory, all these algorithms will break.
However, this can be addressed by designing efficient dynamic approximate k-MIPS ('dynamic' since memory is dynamic), which itself is a fundamental and challenging problem by itself.
We think that this might not be necessary for situations where memory is an explicit set of facts for which we can already learn good representations in an unsupervised way (as is the case in the application considered in this paper).
>> Why not report the wall clock time in the experiments to get a sense of speedup? Comparing the number of epochs across models is not fair because each epoch could take different amount of time for different models.
We agree with you that wall clock is a better measure of speedup. We will add wall clock time comparisons in the next update to the paper.
PS: I am sorry for the delay in reply. It was a NIPS week :)
1. It seems the hierarchical memory is not learned, it is obtained by unsupervised method such as clustering.
2. The key in the model is to find the Exact K-Nearest neighbor, because in table 1, the approximate k-nearest neighbor methods are much worse than full-softmax method.
3. In training time, when you run the K-MIPS method and backpropagate the loss based on these K-outputs, but if the groundtruth is not in these set of K-outputs, what is the loss?
>> It seems the hierarchical memory is not learned, it is obtained by unsupervised method such as clustering.
Yes you are right. Since the memory is read only, we obtain the hierarchical memory in an unsupervised way. The main idea is to have a fixed memory access scheme and train the reader to learn this fixed memory access scheme.
In other words, the reader is learning to do search.
While updating the memory hierarchy during training is an interesting problem by itself, it might not be necessary in applications where the external memory is explicit and fact-based rather than implicit memory like in NTMs.
>> The key in the model is to find the Exact K-Nearest neighbor, because in table 1, the approximate k-nearest neighbor methods are much worse than full-softmax method.
The key in the model is to use k-softmax instead of softmax over the entire memory. This can be done using either exact or approximate k-NN.
>> In training time, when you run the K-MIPS method and backpropagate the loss based on these K-outputs, but if the groundtruth is not in these set of K-outputs, what is the loss?
Quoting from section 3: "To avoid this issue, we always include the correct candidate to the top-K candidates retrieved by the K-MIPS algorithm, effectively performing a fully supervised form of learning."
PS: I am sorry for the delay in reply. It was a NIPS week :)
It seems natural to compare against standard fast-nearest neighbor approaches.  How does using exact 10-MIPS with FLANN, say, compare to your approximate K-MIPS approaches?
Thanks for the suggestion. If we use FLANN to compute exact K-MIPS solution (by setting target precision to 1), then the performance should be similar to the exact K-MIPS performance reported in the paper.
However, running time might be slightly better than the linear search based exact K-MIPS used in our experiments. We will add this result in the next update to the paper.
The main message of the paper is to use top-K memory cells based on K-MIPS instead of full softmax and FLANN or any other approximate search methods can be used to do that.
I find this paper not very compelling.  The basic idea seems to be that we can put a fast neighbor searcher into a memory augmented net to make the memory lookups scalable.
However, this was precisely the point of Rae et al.    There are a  number of standardized neighbor searchers; I don't understand why the authors choose to use their own (which they do not benchmark against the standards).
Moreover, they test on a problem where there is no clear need for (vector based) fast-nn, because one can use hashing on the text.     I also find the repeated distinction between "mips" and "nns" distracting; most libraries that can do one can do the other, or inputs can be modified  to switch between the problems;
indeed the authors do this when they convert to the  "mcss" problem.
Thanks for your valuable feedback.
>>> The basic idea seems to be that we can put a fast neighbor searcher into a memory augmented net to make the memory lookups scalable.
However, this was precisely the point of Rae et al.
You are correct that the basic idea is to use fast nearest neighbor search and do k-softmax instead of softmax for memory access. This can be considered as a way of hard attention followed by soft attention and the whole process is still differentiable.
While this is precisely the point of Rae et al., 2016, we would like to highlight the fact that our work and Rae et al.’s work happened around same time independently.
In fact, our arxiv version predates that of Rae et al.
So we feel that it is not fair to discard the main contributions of this paper since it has been proposed in parallel in another independent paper.
>>> There are a number of standardized neighbor searchers; I don't understand why the authors choose to use their own (which they do not benchmark against the standards).
While there are several commonly used nearest neighbor searchers like FLANN, we would like to highlight that we have used recent state-of-the-art methods for maximum inner product search. We chose three algorithms k-means clustering, PCA-Tree, and WTA-Hash each being the current state-of-the-art clustering-based, tree-based, and hashing-based approaches for MIPS respectively. These have already been benchmarked against standard libraries and hence state-of-the-art.
>>> Moreover, they test on a problem where there is no clear need for (vector based) fast-nn, because one can use hashing on the text.
We agree that simple-questions is not a suitable problem for the proposed model. We in fact state this in the paper and mention that we use this task mainly for demonstration purpose and acknowledge that keyword hashing based approach would perform much better in this task. Our goal is to design a general memory access mechanism which does not take into account any dataset specific priors (like keyword hashing).
>>> I also find the repeated distinction between "mips" and "nns" distracting; most libraries that can do one can do the other, or inputs can be modified to switch between the problems;
indeed the authors do this when they convert to the "mcss" problem.
Thanks for this feedback. We will fix this issue.
The paper proposes an algorithm for training memory networks which have very large memories. Training such models in traditional ways, by using soft-attention mechanism over all the memory slots is not only slow, it is also harder to train due to dispersion of gradients. The paper proposes to use the k-mips algorithm over the memories to choose a subset of the memory slots over which the attention is applied. Since the cost of exact k-mips is the same as doing full attention, the authors propose to use approximate k-mips, which while faster to compute, results in inferior performance. An artifact of using k-mips is that one cannot learn the memory slots.
Hence they are pre-trained and kept fixed during entire training. The experimental section shows the efficacy of using k-mips using the SimpleQuestions dataset. The exact k-mips results in the same performance as the full attention. The approximate k-mips results in deterioration in performance. The paper is quite clearly written and easy to understand.
I think the ideas proposed in the paper are not super convincing. I have a number of issues with this paper.
1. The k-mips algorithm forces the memories to be fixed. This to me is a rather limiting constraint, especially on problems/dataset which will require multiple hops of training to do compounded reasoning. As a results I'm not entirely sure about the usefulness of this technique.
2.
Furthermore, the exact k-mips is the sample complexity as the full attention. The only way to achieve speedup is to use approx k-mips. That, as expected, results in a significant drop in performance.
3. The paper motivates the ideas by proposing solutions to eliminate heuristics used to prune the memories.
However in Section 3.1 the authors themselves end up using multiple heuristics to make the training work. Agreed, that the used heuristics are not data dependent, but still, it feels like they are kicking the can down the road as far as heuristics are concerned.
4. The experimental results are not very convincing.
First there is no speed comparison.
Second, the authors do not compare with methods other than k-mips which do fast nearest neighbor search, such as, FLANN.
Thanks for your valuable feedback.
>>> The k-mips algorithm forces the memories to be fixed. This to me is a rather limiting constraint, especially on problems/dataset which will require multiple hops of training to do compounded reasoning. As a results I'm not entirely sure about the usefulness of this technique.
It is true that the k-mips algorithm forces the memories to be fixed.
However, it is not the issue with k-mips algorithm only. We have this issue with nearest neighbor search algorithms and maximum cosine similarity search algorithms as well.
When we need to update the memory, then one can do periodic memory reorganization after every few epochs as done by Vijayanarasimhan et al., 2014 or Rae et al., 2016.
However, this is not a limiting constraint on problems which will require multiple hops. One can design a separate inference module (like the episodic memory in end-to-end memory nets) that will take care of multiple hops and the main memory can still be fixed.
So we believe that this technique is still useful in the setting mentioned by the reviewer.
>>> Furthermore, the exact k-mips is the sample complexity as the full attention. The only way to achieve speedup is to use approx k-mips. That, as expected, results in a significant drop in performance.
Yes true. Exact k-mips has the sample complexity of full attention. Our k-mips experiments serve two purposes: 1. It is a proof of concept that approximate k-mips if done correctly would help us achieve scalability as well as performance. 2. It is suggestive to use k-mips instead of full attention even in situations where full attention is cheap. We think that both are important messages. Our approximate k-mips experiments benchmark the existing state-of-the-art methods for approximate k-mips and show that they are not good enough for this task. This demands more research on better approximate k-mips methods.
>>> The paper motivates the ideas by proposing solutions to eliminate heuristics used to prune the memories.
However in Section 3.1 the authors themselves end up using multiple heuristics to make the training work. Agreed, that the used heuristics are not data dependent, but still, it feels like they are kicking the can down the road as far as heuristics are concerned.
We agree with the reviewer. As the reviewer rightly pointed out, we replaced the dataset-specific heuristics with dataset-independent heuristics which is a research direction worth exploring since it will be applicable to any kind of datasets.
>>> The experimental results are not very convincing.
First there is no speed comparison.
Second, the authors do not compare with methods other than k-mips which do fast nearest neighbor search, such as, FLANN.
FLANN is a standard library that people use for nearest neighbor search.
However, please note that PCA-tree is a better method than FLANN and is a state-of-the-art in tree based methods. We benchmark three search algorithms each state-of-the-art in clustering-based, tree-based, and hashing-based approaches respectively.
Speed comparison of different approximate k-mips algorithms used in this paper has already been done in an extensive setup in Auvolat et al., 2015.
1. The hierarchical memory is fixed, not learned, and there is no hierarchical in the experimental section, only one layer for softmax layer.
2. It shows the 10-mips > 100-mips > 1000-mips, does it mean 1-mips is the best one we should adopt?
3. Approximated k-mips is worse than even original method. Why does it need exact k-mips? It seems the proposed method is not robust.
Thanks for your valuable feedback.
>>> The hierarchical memory is fixed, not learned.
It is true that the hierarchical memory is fixed and not learned. Quoting our reply for the same question in pre-review: “Since the memory is read only, we obtain the hierarchical memory in an unsupervised way. The main idea is to have a fixed memory access scheme and train the reader to learn this fixed memory access scheme.
In other words, the reader is learning to do search.
While updating the memory hierarchy during training is an interesting problem by itself, it might not be necessary in applications where the external memory is explicit and fact-based rather than implicit memory like in NTMs.”
>>> There is no hierarchical in the experimental section, only one layer for softmax layer.
This is not true.
Except for the full-softmax experiments, all other experiments use two levels of hierarchy.
>>> It shows the 10-mips > 100-mips > 1000-mips, does it mean 1-mips is the best one we should adopt?
No. We do not claim that 1-mips is the best one we should adopt. To clarify the definition, “1-mips means a softmax over the correct fact and one incorrect fact who has maximum inner product with the query”. 1-mips can be considered as a probabilistic version of max-margin learning.
While it is tempting to set k=1 seeing that smaller k results in better performance, k=1 will affect the training time since new maximum inner product candidate will pop up every time the old one is pushed down the rank.
However this problem is not significant when k=10 since we already push all top-10 candidates down and hence this will converge faster than k=1 setting.
Thus there is a trade-off in choosing the value of k.
>>> Approximated k-mips is worse than even original method. Why does it need exact k-mips? It seems the proposed method is not robust.
It is true that approximate k-mips is worse than full softmax.
However we would like to highlight that the main message of the paper is to use k-softmax instead of softmax. This can be considered as a way of hard attention followed by soft attention and the whole process is still differentiable. Very similar idea is also proposed in Rae et al., 2016 independently and our arxiv version predates that of Rae et al., 2016. We do not propose a novel approximate k-mips algorithm.
Instead we benchmark all the state-of-the-art approximate k-mips algorithms in this newly proposed setting.

Meta-review:
role: PC
data: comment: This paper was reviewed by three experts. While they find interesting ideas in the manuscript, all three point to deficiencies (unconvincing results, etc) and unanimously recommend rejection.

Decision: Reject