Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors present a simple method to affix a cache to neural language models, which provides in effect a copying mechanism from recently used words. Unlike much related work in neural networks with copying mechanisms, this mechanism need not be trained with long-term backpropagation, which makes it efficient and scalable to much larger cache sizes. They demonstrate good improvements on language modeling by adding this cache to RNN baselines.

The main contribution of this paper is the observation that simply using the hidden states h_i as keys for words x_i, and h_t as the query vector, naturally gives a lookup mechanism that works fine without tuning by backprop. This is a simple observation and might already exist as folk knowledge among some people, but it has nice implications for scalability and the experiments are convincing.

The basic idea of repurposing locally-learned representations for large-scale attention where backprop would normally be prohibitively expensive is an interesting one, and could probably be used to improve other types of memory networks.

My main criticism of this work is its simplicity and incrementality when compared to previously existing literature. As a simple modification of existing NLP models, but with good empirical success, simplicity and practicality, it is probably more suitable for an NLP-specific conference. However, I think that approaches that distill recent work into a simple, efficient, applicable form should be rewarded and that this tool will be useful to a large enough portion of the ICLR community to recommend its publication.
review_title: Review

final_score: rating: null
review_content: (1) Could you please comment on why you decided to train without the cache component and apply it only at test time?
(2) On the PTB dataset, do the results get better if you increase the cache size, or is the dataset too small and it starts to overfit?
(3) Similar to the below comment, I also would like to know the accuracy on the LAMBADA dataset.

Thanks!
review_title: comments

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper not only shows that a cache model on top of a pre-trained RNN can improve language modeling, but also illustrates a shortcoming of standard RNN models in that they are unable to capture this information themselves. Regardless of whether this is due to the small BPTT window (35 is standard) or an issue with the capability of the RNN itself, this is a useful insight. This technique is an interesting variation of memory augmented neural networks with a number of advantages to many of the standard memory augmented architectures.

They illustrate the neural cache model on not just the Penn Treebank but also WikiText-2 and WikiText-103, two datasets specifically tailored to illustrating long term dependencies with a more realistic vocabulary size. I have not seen the ability to refer up to 2000 words back previously.
I recommend this paper be accepted. There is additionally extensive analysis of the hyperparameters on these datasets, providing further insight.

I recommend this interesting and well analyzed paper be accepted.
review_title: Review

final_score: rating: null
review_content: Thanks for the paper - it's an intelligent method that can be applied relatively easily to existing RNN models with a good performance boost. I imagine this may find a variety of uses.

Maybe I am missing it, but it doesn't seem immediately clear the cache size used for the Penn Treebank model? The Pointer Sentinel LSTM work used a window history of 100 and Figure 2 notes that the cache size is 500 to get a perplexity of 74.6 - what was the cache size used to produce the PTB number in Table 1? Maybe adding that in the table, as in Table 2, and also a comparable number (100, 2000, X) would be useful for a glance comparison between the tables?

The result comparison between WikiText-2 and WikiText-103 was quite helpful. Citing the importance of testing on relatively large datasets is hugely important given Penn Treebank seems to be the standard dataset for language modeling even though it's quite small in both text and vocabulary. Have you any further insights beyond dataset sizes as to difference in performance improvements? The vocabulary size difference between WikiText-2 and WikiText-103 is quite massive - is it possible that the vocabulary size difference may make the cache memory model less advantageous?

For the Lambada dataset, the insight regarding the lambda parameter for development and control sets were interesting. As you note, it seems the interpolation parameter would need to be flexible to ensure strong generative ability as to beat the LSTM even when the cache memory aspect is not helpful. Might a similar issue have occurred with WikiText-103 in order to decrease the improvement the cache model may give? I don't see a Figure 3 style chart for WikiText-103 - likely as it's very expensive to produce - but it'd be possible that might provide that insight.
review_title: Cache size on PTB, WikiText vocabulary differences, and Lambada question

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a simple extension to a neural network language model by adding a cache component. 
The model stores <previous hidden state, word> pairs in memory cells and uses the current hidden state to control the lookup. 
The final probability of a word is a linear interpolation between a standard language model and the cache language model. 
Additionally, an alternative that uses global normalization instead of linear interpolation is also presented. 
Experiments on PTB, Wikitext, and LAMBADA datasets show that the cache model improves over standard LSTM language model.

There is a lot of similar work on memory-augmented/pointer neural language models, and the main difference is that the proposed method is simple and scales to a large cache size.
However, since the technical contribution is rather limited, the experiments need to be more thorough and conclusive. 
While it is obvious from the results that adding a cache component improves over language models without memory, it is still unclear that this is the best way to do it (instead of, e.g., using pointer networks). 
A side-by-side comparison of models with pointer networks vs. models with cache with roughly the same number of parameters is needed to convincingly argue that the proposed method is a better alternative (either because it achieves lower perplexity, faster to train but similar test perplexity, faster at test time, etc.)

Some questions:
- In the experiment results, for your neural cache model, are those results with linear interpolation or global normalization, or the best model? Can you show results for both? 
- Why is the neural cache model worse than LSTM on Ctrl (Lambada dataset)? Please also show accuracy on this dataset. 
- It is also interesting that the authors mentioned that training the cache component instead of only using it at test time gives little improvements. Are the results about the same or worse?
review_title: review

final_score: rating: null
review_content: The authors mention the similarity of this method to a copying mechanism applied to language modeling (i.e. CopyNet), pointer sentinel mixture models (Merity et al.), and the attention sum reader. Can the authors elaborate more on the difference with Merity et al.? Since both methods require computing attention over the previous timesteps, why is this method more scalable than that one? What performance does the pointer sentinel model obtain with a cache size of 1000?
review_title: Questions

Meta-review: Reviewers agree that this paper is based on a "trick" to build memory without requiring long-distance backprop. This method allows the model to utilize a cache-like mechanism, simply by storing previous states. Everyone agrees that this roughly works (although there could be stronger experimental evidence), and provides long-term memory to simple models. Reviewers/authors also agree that it might not work as well as other pointer-network like method, but there is controversy over whether that is necessary. 
 
 - Further discussion indicated a sense by some reviewers that this method could be quite impactful, even if it was not a huge technical contribution, due to its speed and computational benefits over pointer methods. 
 
 - The clarity of the writing and good use of references was appreciated 
 
 - This paper is a nice complement/rebuttal to "Frustratingly Short Attention Spans in Neural Language Modeling".
 
 Including the discussion about this paper as it might be helpful as it was controversial: 
 
 """
 The technical contribution may appear "limited" but I feel part of that is necessary to ensure the method can scale to both large datasets and long term dependencies. For me, this is similar to simpler machine learning methods being able to scale to more data (though replacing "data" with "timesteps"). More complex methods may do better with a small number of data/timesteps but they won't be able to scale, where other specific advantages may come in to play.
 
 (Timesteps)
 
 Looking back 2000 timesteps is something I've not seen done and speaks to a broader aspect of language modeling - properly capturing recent article level context. Most language models limit BPTT to around 35 timesteps, with some even arguing we don't need that much (i.e. "Frustratingly Short Attention Spans in Neural Language Modeling" that's under review for ICLR). From a general perspective, this is vaguely mad given many sentences are longer than 35 timesteps, yet we know both intuitively and from the evidence they present that the rest of an article is very likely to help modeling the following words, especially for PTB or WikiText.
 
 This paper introduces a technique that not only allows for utilizing dependencies far further back than 35 timesteps but shows it consistently helps, even when thrown against a larger number of timesteps, a larger dataset, or a larger vocabulary. Given it is also a post-processing step that can be applied to any vaguely RNN type model, it's widely applicable and trivial to train in comparison to any more complicated models.
 
 (Data)
 
 Speaking to AnonReviewer1's comment, "A side-by-side comparison of models with pointer networks vs. models with cache with roughly the same number of parameters is needed to convincingly argue that the proposed method is a better alternative (either because it achieves lower perplexity, faster to train but similar test perplexity, faster at test time, etc.)"
 
 Existing pointer network approaches for language modeling are very slow to train - or at least more optimal methods are yet to be discovered - and has such limited the BPTT length they tackle. Merity et al. use 100 at most and that's the only pointer method for language modeling attending to article style text that I am aware of. Merity et al. also have a section of their paper specifically discussing the training speed complications that come from integrating the pointer network. There is a comparison to Merity et al. in Table 1 and Table 2.
 
 The scaling becomes more obvious on the WikiText datasets which have a more realistic long tail vocabulary than PTB's 10k. For WikiText-2, at a cache size of 100, Merity et al. get 80.8 with their pointer network method while the neural cache model get 81.6. Increasing the neural model cache size to 2000 however gives quite a substantial drop to 68.9. They're also able to apply their method to WikiText-103, a far larger dataset than PTB or WikiText-2, and show that it still provides improvements even when there is more data and a larger vocabulary. Scaling to this dataset is only sanely possible as the neural cache model doesn't add to the training time of the base neural model at all - that it's equivalent to training a standard LSTM.
 """
Decision: Accept (Poster)