Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1. How do you compute k,v, and p from h? is it just that you have 3 times hidden units and use each part for k,v, and p respectively? I can guess it. But it is better to make it explicit in the paper.

2. You description about n-gram model in section 2.4 can be even more clear. It is not very clear how you split the LSTM output to N-1 vectors. Also why do you need to do this? Why not simply use previous N-1 LSTM outputs to attend?

3. Miller et al., 2016 is not the only paper that explored key-value memory (as stated in related work section). It has also been explored in Reed and de Freitas 2015 [1] and Gulcehre et al., 2016 [2].

References:

[1] Scott Reed and Nando de Freitas. Neural programmer-interpreters. arXiv preprint arXiv:1511.06279,
2015.

[2] Caglar Gulcehre, Sarath Chandar, Kyunghyun Cho, and Yoshua Bengio. Dynamic neural turing
machines with soft and hard addressing schemes. CoRR, abs/1607.00036, 2016. URL http:
//arxiv.org/abs/1607.00036.

review_title: Few questions

final_score: rating: 7: Good paper, accept
review_content: The paper presents an investigation of various neural language models designed to query context information from their recent history using an attention mechanism. The authors propose to separate the attended vectors into key, value and prediction parts. The results suggest that this helps performance. The authors also found that a simple model which which concatenates recent activation vectors performs at a similar level as the more complicated attention-based models.

The experimental methodology seems sound in general. I do have some issues with the way the dimensionality of the vectors involved in the attention-mechanism is chosen. While it’s good that the hidden layer sizes are adapted to ensure similar numbers of trainable parameters for all the models, this doesn’t control for the fact that key/value/prediction vectors of a higher dimensionality may simply work better regardless of whether their dimensions are dedicated to one particular task or used together. This separation clearly saves parameters but there could also be benefits of having some overlap of information assuming that vectors that lead to similar predictions may also be required in similar contexts for example. Some tasks may also require more dimensions than others and the explicit separation prevents the model from discovering and exploiting this. 

While memory augmented RNNs and RNNs with attention mechanisms are not new, some of these architectures had not yet been applied to language modeling. Similarly (and as acknowledged by the authors), the strategy of separating key and value functionality has been proposed before, but not in the context of natural language modeling. I’m not sure about the novelty of the proposed n-gram RNN because I recall seeing similar architectures before but I understand that novelty was not the point of that architecture as it mainly serves as a proof of the lack of ability of the more complicated architectures to do better. In that sense I do consider it an inventive baseline that could be used in future work to test the ability of other models that claim to exploit long-term dependencies. 

The exact computation of the representation h_t was initially not that clear to me (the terms hidden and output can be ambiguous at times) but besides this, the paper is quite clear and generally well-written.

The results in this paper are important because they show that learning long-term dependencies is not a solved problem by any means. The authors provide a very nice comparison to prior results and the fact that their n-gram RNN is often at least competitive with far more complicated approaches is a clear indication that some of those methods may not capture as much context information as previously thought. The success of the separation of key/value/prediction functionality in attention-based system is also noteworthy although I think this is something that needs to be investigated more thoroughly (i.e., with more control for hyperparameter choices). 


Pros:
Impressive and also interesting results.
Good comparison with earlier work.
The n-gram RNN is an interesting baseline.


Cons:
The relation between the attention-mechanism type and the number of hidden units weakens the claim that the key/value/prediction separation is the reason for the increase in performance somewhat.
The model descriptions are not entirely clear.
I would have liked to have seen what happens when the attention is applied to a much larger context size.

review_title: Review: Frustratingly Short Attention Spans in Neural Language Modeling

final_score: rating: null
review_content: How is h_t exactly computed and did you compare with higher dimensional context vectors? I'm a bit confused about whether these are some affine transformation of the LSTM states or the states themselves. I'd expect the differences between the Key-Value-Predict system and the standard attention-based RNN to be smaller if the standard attention mechanism is allowed to work with vectors that have three times more dimensions. 

Some unrelated detail: In Equation 1 I think that the 1^T should be on the right side of the W^h h_t multiplication such that it leads to an outer product instead of a dot product.
review_title: How is h_t exactly computed and did you compare with higher dimensional context vectors?

final_score: rating: 7: Good paper, accept
review_content: This paper explores a variety of memory augmented architectures (key, key-value, key-predict-value) and additionally simpler near memory-less RNN architectures. Using an attention model that has access to the various decompositions is an interesting idea and one worth future explorations, potentially in different tasks where this type of model could excel even more. The results over the Wikipedia corpus are interesting and feature a wide variety of different model types. This is where the models suggested in the paper are strongest. The same models run over the CBT dataset show a comparable but less convincing demonstration of the variations between the models.

The authors also released their Wikipedia corpus already. Having inspected it I consider it a positive and interesting contribution. I still believe that, if a model was found that could better handle longer term dependencies, it would do better on this Wikipedia dataset, but at least within the realm of what . As an example, the first article in train.txt is about a person named "George Abbot", yet "Abbot" isn't mentioned again until the next sentence 40 tokens later, and then the next "Abbot" is 15 tokens from there. Most gaps between occurrences of "Abbot" are dozens of timesteps. Performing an analysis based upon easily accessed information, such as when the same token reappears again or average sentence length, may be useful as an approximation for the length that an attention window may prefer.

This is a well explained paper that raises interesting questions regarding the spans used in existing language modeling approaches and serves as a potential springboard for future directions.
review_title: Review

final_score: rating: null
review_content: I agree with many of the thoughts in your paper, particularly that long term dependencies are quite difficult and that they've not yet been well solved by any architecture, but have some questions regarding the experiments and resulting conclusions.

Unrolling BPTT for only 20 timesteps during training seems minimal given standard language modeling on word level PTB unrolls for 35 timesteps. The maximum attention window of only length 15 also seems quite small (Figure 2a). Was there a reason you trained for only that many timesteps? Generally I've seen attention mechanisms work better the more history they have access to.

Is it also possible to release your Wikipedia corpus earlier than later? Being able to view that would be useful as there's only so much about the pre-processing and set-up that can be described in the paper. It may also provide an insight into some of the behaviour of the models. The WikiText dataset, used in the Pointer Sentinel Mixture Model and Neural Language Models with a Continuous Cache papers, are based on Wikipedia as well and has shown that models effective in exploiting long range dependencies do better, though those works use a history of hundred or more and access memory using a pointer rather than weighted summation.

For the Wikipedia corpus, you cut the vocabulary to 77k words - was 77k an arbitrary number or a result of a certain frequency cut-off?

You also state in the conclusion that the results on the Wikipedia corpus and CBT show that key-value-predict decomposition outperform simpler attention mechanisms. While the results on your Wikipedia corpus show that, the results on the CBT corpus don't seem as definitive. The results of the attention model for example are quite similar to the key-value-predict model (though slightly better for named entities, slightly worse for verbs). Am I misreading those figures? I'd generally not consider the deltas between the models significant but also don't have substantial experience with the CBT dataset in the past.
review_title: Questions re: BPTT length, Wikipedia corpus, and CBT results

final_score: rating: 7: Good paper, accept
review_content: This paper focusses on attention for neural language modeling and has two major contributions:

1. Authors propose to use separate key, value, and predict vectors for attention mechanism instead of a single vector doing all the 3 functions. This is an interesting extension to standard attention mechanism which can be used in other applications as well.
2. Authors report that very short attention span is sufficient for language models (which is not very surprising) and propose an n-gram RNN which exploits this fact.

The paper has novel models for neural language modeling and some interesting messages. Authors have done a thorough experimental analysis of the proposed ideas on a language modeling task and CBT task.

I am convinced with authors’ responses for my pre-review questions.

Minor comment: Ba et al., Reed & de Freitas, and Gulcehre et al. should be added to the related work section as well.

review_title: Review

Meta-review: Reviewers found this paper to be a rigorous and "thorough experimental analysis" of context-length in language modelingv through the lens of an "interesting extension to standard attention mechanism". The paper reopens and makes more problematic widely accepted but rarely verified claims of the importance of long-term dependency.
 
 Pros:
 - "Well-explained" and clear presentation
 - Use of an "inventive baseline" in the form a ngram rnn 
 - Use a impactful corpus for long-term language modeling
 
 Cons: 
 - Several of the ideas have been explored previously.
 - Some open questions about the soundness of parameters (rev 1)
 - Requests for deeper analysis on data sets released with the paper.
Decision: Accept (Poster)