The paper presents an investigation of various neural language models designed to query context information from their recent history using an attention mechanism. The authors propose to separate the attended vectors into key, value and prediction parts. The results suggest that this helps performance. The authors also found that a simple model which which concatenates recent activation vectors performs at a similar level as the more complicated attention-based models.  The experimental methodology seems sound in general. I do have some issues with the way the dimensionality of the vectors involved in the attention-mechanism is chosen. While it’s good that the hidden layer sizes are adapted to ensure similar numbers of trainable parameters for all the models, this doesn’t control for the fact that key/value/prediction vectors of a higher dimensionality may simply work better regardless of whether their dimensions are dedicated to one particular task or used together. This separation clearly saves parameters but there could also be benefits of having some overlap of information assuming that vectors that lead to similar predictions may also be required in similar contexts for example. Some tasks may also require more dimensions than others and the explicit separation prevents the model from discovering and exploiting this.   While memory augmented RNNs and RNNs with attention mechanisms are not new, some of these architectures had not yet been applied to language modeling. Similarly (and as acknowledged by the authors), the strategy of separating key and value functionality has been proposed before, but not in the context of natural language modeling. I’m not sure about the novelty of the proposed n-gram RNN because I recall seeing similar architectures before but I understand that novelty was not the point of that architecture as it mainly serves as a proof of the lack of ability of the more complicated architectures to do better. In that sense I do consider it an inventive baseline that could be used in future work to test the ability of other models that claim to exploit long-term dependencies.   The exact computation of the representation h_t was initially not that clear to me (the terms hidden and output can be ambiguous at times) but besides this, the paper is quite clear and generally well-written.  The results in this paper are important because they show that learning long-term dependencies is not a solved problem by any means. The authors provide a very nice comparison to prior results and the fact that their n-gram RNN is often at least competitive with far more complicated approaches is a clear indication that some of those methods may not capture as much context information as previously thought. The success of the separation of key/value/prediction functionality in attention-based system is also noteworthy although I think this is something that needs to be investigated more thoroughly (i.e., with more control for hyperparameter choices).    Pros: Impressive and also interesting results. Good comparison with earlier work. The n-gram RNN is an interesting baseline.   Cons: The relation between the attention-mechanism type and the number of hidden units weakens the claim that the key/value/prediction separation is the reason for the increase in performance somewhat. The model descriptions are not entirely clear. I would have liked to have seen what happens when the attention is applied to a much larger context size.
Output: en