Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This is a well-written paper that proposes regularization and optimization strategies for word-based language modeling tasks.   The authors propose the use of DropConnect  on the hidden-hidden connections as a regularization method, in order to take advantage of high-speed LSTM implementations via the cuDNN LSTM libraries from NVIDIA.  The  focus of this work is on the prevention of overfitting on the recurrent connections of the LSTM.  The authors explore a variant of Average-SGD (NT-ASGD) as an optimization strategy which eliminates the need for tuning the average trigger and uses a constant learning rate.  Averaging is triggered when the validation loss worsens or stagnates for a few cycles, leading to two new hyper parameters: logging interval and non-monotone interval.  Other forms of well-know regularization methods were applied to the non-recurrent connections, input, output and embedding matrices.  

As the authors point out, all the methods used in this paper have been proposed before and theoretical convergence explained. The novelty of this work lies in its successful application to the language modeling task achieving state-of-the-art results.

On the PTB task, the proposed AWD-LSTM achieves a perplexity of 57.3 vs 58.3 (Melis et al 2017) and almost the same perplexity as Melis et el. on the Wiki-Text2 task (65.8 vs 65.9).  The addition of a cache model provides significant gains on both tasks.   

It would be useful, if authors had explored the behavior of the  AWD-LSTM algorithm with respect to various hyper parameters  and provided a few insights towards their choices for other large vocabulary language modeling tasks (1 million vocabulary sizes).  

Similarly, the choice of the average trigger and number of cycles seem arbitrary -  it would have been good to see a graph over a range of values, showing their impact on the model's performance.

A 3-layer LSTM has been used for the experiments  - how was this choice made?  What is the impact of this algorithm if the net was a 2-layer net as is typical in most large-scale LMs?

Table 3 is interesting to see how the cache model helps with rare words  and as such has applications in key word spotting tasks. Were the hyper parameters of the cache tuned to perform better on rare words?  More details on the design of the cache model would have been useful.

You state that the gains obtained using the cache model were far less than what was obtained in Graves et al 2016 - what do you attribute this to?

Ablation analysis in Table 4 is very useful - in particular it shows how lack of regularization of the recurrent connections can lead to maximum degradation in performance.

Most of the results in this paper have been based on one choice of various model parameters. Given the emperical nature of this work, it would have made the paper even clearer if an analysis of their choices were presented.  Overall, it would be beneficial to the MLP community to see this paper accepted in the conference.

review_title: This paper proposes regularization strategies for word LSTM-based language models achieving state-of-the-art results on the PennTree Bank and the Wiki-Text2 LM tasks. A variant of  Average-SGD (ASGD) where the threshold for determining which iterates to average is also proposed, yielding better results than SGD.

final_score: rating: 7: Good paper, accept
review_content: The paper sets a new state of the art on word level language modelling on the Penn Treebank and Wikitext-2 datasets using various optimization and regularization techniques. These already very good results are further improved, by a large margin, using a Neural Cache.

The paper is well written, easy to follow and the results speak for themselves. One possible criticism is that the experimental methodology does not allow for reliable conclusions to be drawn about contributions of all different techniques, because they seem to have been evaluated at a single hyperparameter setting (that was hand tuned for the full model?).

A variant on the Averaged SGD method is proposed. This so called NT-ASGD optimizer switches to averaging mode based on recent validation losses. I would have liked to see a more thorough assessment of NT-ASGD, especially against well tuned SGD.

I particularly liked Figure 3 which shows how the Neural Cache makes the model much better at handling rare words and UNK (!) at the expense of very common words. Speaking of the Neural Cache, a natural baseline would have been dynamic evaluation.

All in all, the paper is a solid contribution which deserves to be accepted. It could become even better, were the experiments to tease the various factors apart.

review_title: State of the art on small word level language modelling datasets

final_score: rating: 7: Good paper, accept
review_content: Clearly presented paper, including a number of reasonable techniques to improve LSTM-LMs. The proposed techniques are heuristic, but are reasonable and appear to yield improvements in perplexity. Some specific comments follow.

re. "ASGD" for Averaged SGD: ASGD usually stands for Asynchronous SGD, have the authors considered an alternative acronym? AvSGD?

re. Optimization criterion on page 2, note that SGD is usually taken to minimizing expected loss, not just empirical loss (Bottou thesis 1991).

Is there any theoretical analysis of convergence for Averaged SGD?

re. paragraph starting with "To prevent such inefficient data usage, we randomly select the sequence length for the forward and backward pass in two steps": the explanation is a bit unclear. What is the "base sequence length" exactly? Also, re. the motivation above this paragraph, I'm not sure what "elements" really refers to, though I can guess.

What is the number of training tokens of the datasets used, PTB and WT2?

Can the authors provide more explanation for what "neural cache models" are, and how they relate to "pointer models"?

Why do the sections "Pointer models", "Ablation analysis", and "AWD-QRNN" come after the Experiments section?
review_title: Evaluation of a number of heuristics to improve LSTM language models.

Meta-review: This paper presents a simple yet effective method for weight dropping for an LSTM that requires no modification of an RNN cell's formulation.  Experimental results shows good perplexity results on benchmarks compared to many baselines.  All reviewers agree that the paper will bring good contribution to the conference.
Decision: Accept (Poster)