Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1. Any reason to not learn the memories, as in the case of standard memory networks? Is the model hard to learn, given that it also has to learn MIPS? 
2. Why not report the wall clock time in the experiments to get a sense of speedup? Comparing the number of epochs across models is not fair because each epoch could take different amount of time for different models. 
review_title: Model and timing issues

final_score: rating: null
review_content: It seems natural to compare against standard fast-nearest neighbor approaches.  How does using exact 10-MIPS with FLANN, say, compare to your approximate K-MIPS approaches?
review_title: FLANN?  other standard ANN libraries?

final_score: rating: null
review_content: 1. It seems the hierarchical memory is not learned, it is obtained by unsupervised method such as clustering.
2. The key in the model is to find the Exact K-Nearest neighbor, because in table 1, the approximate k-nearest neighbor methods are much worse than full-softmax method.
3. In training time, when you run the K-MIPS method and backpropagate the loss based on these K-outputs, but if the groundtruth is not in these set of K-outputs, what is the loss? 
review_title: Few questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an algorithm for training memory networks which have very large memories. Training such models in traditional ways, by using soft-attention mechanism over all the memory slots is not only slow, it is also harder to train due to dispersion of gradients. The paper proposes to use the k-mips algorithm over the memories to choose a subset of the memory slots over which the attention is applied. Since the cost of exact k-mips is the same as doing full attention, the authors propose to use approximate k-mips, which while faster to compute, results in inferior performance. An artifact of using k-mips is that one cannot learn the memory slots. Hence they are pre-trained and kept fixed during entire training. The experimental section shows the efficacy of using k-mips using the SimpleQuestions dataset. The exact k-mips results in the same performance as the full attention. The approximate k-mips results in deterioration in performance. The paper is quite clearly written and easy to understand. 

I think the ideas proposed in the paper are not super convincing. I have a number of issues with this paper. 

1. The k-mips algorithm forces the memories to be fixed. This to me is a rather limiting constraint, especially on problems/dataset which will require multiple hops of training to do compounded reasoning. As a results I'm not entirely sure about the usefulness of this technique. 
2. Furthermore, the exact k-mips is the sample complexity as the full attention. The only way to achieve speedup is to use approx k-mips. That, as expected, results in a significant drop in performance. 
3. The paper motivates the ideas by proposing solutions to eliminate heuristics used to prune the memories. However in Section 3.1 the authors themselves end up using multiple heuristics to make the training work. Agreed, that the used heuristics are not data dependent, but still, it feels like they are kicking the can down the road as far as heuristics are concerned. 
4. The experimental results are not very convincing. First there is no speed comparison. Second, the authors do not compare with methods other than k-mips which do fast nearest neighbor search, such as, FLANN. 
review_title: Not a very convincing proposal for dealing with very large memories in memory networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1. The hierarchical memory is fixed, not learned, and there is no hierarchical in the experimental section, only one layer for softmax layer.
2. It shows the 10-mips > 100-mips > 1000-mips, does it mean 1-mips is the best one we should adopt?
3. Approximated k-mips is worse than even original method. Why does it need exact k-mips? It seems the proposed method is not robust.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: I find this paper not very compelling.  The basic idea seems to be that we can put a fast neighbor searcher into a memory augmented net to make the memory lookups scalable.  However, this was precisely the point of Rae et al.    There are a  number of standardized neighbor searchers; I don't understand why the authors choose to use their own (which they do not benchmark against the standards).   Moreover, they test on a problem where there is no clear need for (vector based) fast-nn, because one can use hashing on the text.     I also find the repeated distinction between "mips" and "nns" distracting; most libraries that can do one can do the other, or inputs can be modified  to switch between the problems; indeed the authors do this when they convert to the  "mcss" problem.

review_title: null

Meta-review: This paper was reviewed by three experts. While they find interesting ideas in the manuscript, all three point to deficiencies (unconvincing results, etc) and unanimously recommend rejection.
Decision: Reject