Title: On the State of the Art of Evaluation in Neural Language Models
Authors: Gábor Melis
Chris Dyer
Phil Blunsom
Venue: ICLR 2018
Keywords: rnn
language modelling

Abstract: Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing codebases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.


TL;DR: Show that LSTMs are as good or better than recent innovations for LM and that model evaluation is often unreliable.

Review discussion:
The submitted manuscript describes an exercise in performance comparison for neural language models under standardization of the hyperparameter tuning and model selection strategies and costs.  This type of study is important to give perspective to non-standardized performance scores reported across separate publications, and indeed the results here are interesting as they favour relatively simpler structures.
I have a favourable impression of this paper but would hope another reviewer is more familiar with the specific application domain than I am.
We thank AnonReviewer1 for their review.
We would like to point out that the state-of-the-art results and model comparisons are only part of the message. More importantly, we argue that the way model evaluation is performed is often unsatisfactory. Evaluation at a single hyperparameter setting, failing to control for dominant sources of variation make results unreliable and slow down progress.
The authors did extensive tuning of the parameters for several recurrent neural architectures. The results are interesting.
However the corpus the authors choose are quite small, the variance of the estimate will be quite high, I suspect whether the same conclusions could be drawn.
It would be more convincing if there are experiments on the billion word corpus or other larger datasets, or at least on a corpus with 50 million tokens. This will use significant resources and is much more difficult, but it's also really valuable, because it's much more close to real world usage of language models. And less tuning is needed for these larger datasets.
Finally it's better to do some experiments on machine translation or speech recognition and see how the improvement on BLEU or WER could get.
PTB is very much a standard baseline in the space that people routinely publish perf-based papers on, so I think it's a little harsh to knock people for publishing on it, especially when they make the effort to get results on multiple larger datasets, such as Wikitext-2.
This is not a paper about the utility of hyperparameter optimization, the fact that that works has been well established. It's a paper about how hyperparameter optimization wasn't properly used on a bunch of standard benchmarks in this space, which has already proven very valuable. I really don't think it's necessary to request results on MT or ASR
We feel that AnonReviewer3 might have missed that the main message of the paper was that evaluation - as it's generally performed - is unreliable. Our results suggest that state-of-the-art results are only superficially considered, and variance and parameter sensitivity are likewise given short shrift.
The main criticism seems to center on evaluating models on datasets that are too small which increases evaluation variance and the results are thus not trustworthy.
That is a very good summary of the main message of the paper! We agree that small datasets are problematic, but one cannot refute previous results that were obtained on small datasets using large datasets.
Furthermore, we do hyperparameter tuning and a careful analysis of the variance.
Furthermore, the third dataset (enwik8) is a large character based corpus and we still improve previously reported LSTM results by a substantial margin.
Finally, to do this kind of study we chose language modelling because of its relevance to all kinds recurrent neural models while being simpler than machine translation and speech recognition models. We have demonstrated evaluation problems in this simple and relevant setting. It is unclear why the reviewer requests results on MT and ASR.
The authors perform a comprehensive validation of LSTM-based word and character language models, establishing that recent claims that other structures can consistently outperform the older stacked LSTM architecture result from failure to fully explore the hyperparameter space.
Instead, with more thorough hyperparameter search, LSTMs are found to achieve state-of-the-art results on many of these language modeling tasks.
This is a significant result in language modeling and a milestone in deep learning reproducibility research. The paper is clearly motivated and authoritative in its conclusions but it's somewhat lacking in detailed model or experiment descriptions.
Some further points:
- There are several hyperparameters set to the "standard" or "default" value, like Adam's beta parameter and the batch size/BPTT length.
Even if it would be prohibitive to include them in the overall hyperparameter search, the community is curious about their effect and it would be interesting to hear if the authors' experience suggests that these choices are indeed reasonably well-justified.
- The description of the model is ambiguous on at least two points.
First, it wasn't completely clear to me what the down-projection is (if it's simply projecting down from the LSTM hidden size to the embedding size, it wouldn't represent a hyperparameter the tuner can set, so I'm assuming it's separate and prior to the conventional output projection).
Second, the phrase "additive skip connections combining outputs of all layers" has a couple possible interpretations (e.g., skip connections that jump from each layer to the last layer or (my assumption) skip connections between every pair of layers?).
- Fully evaluating the "claims of Collins et al. (2016), that capacities of various cells are very similar and their apparent
differences result from trainability and regularisation" would likely involve adding a fourth cell to the hyperparameter sweep, one whose design is more arbitrary and is neither the result of human nor machine optimization.
- The reformulation of the problem of deciding embedding and hidden sizes into one of allocating a fixed parameter budget towards the embedding and recurrent layers represents a significant conceptual step forward in understanding the causes of variation in model performance.
- The plot in Figure 2 is clear and persuasive, but for reproducibility purposes it would also be nice to see an example set of strong hyperparameters in a table. The history of hyperparameter proposals and their perplexities would also make for a fantastic dataset for exploring the structure of RNN hyperparameter spaces.
For instance, it would be helpful for future work to know which hyperparameters' effects are most nearly independent of other hyperparameters.
- The choice between tied and clipped (Sak et al., 2014) LSTM gates, and their comparison to standard untied LSTM gates, is discussed only minimally, although it represents a significant difference between this paper and the most "standard" or "conventional" LSTM implementation (e.g., as provided in optimized GPU libraries).
In addition to further discussion on this point, this result also suggests evaluating other recently proposed "minor changes" to the LSTM architecture such as multiplicative LSTM (Krause et al., 2016)
- It would also have been nice to see a comparison between the variational/recurrent dropout parameterization "in which there is further sharing of masks between gates" and the one with "independent noise for the gates," as described in the footnote. There has been some confusion in the literature as to which of these parameterizations is better or more standard; simply justifying the choice of parameterization a little more would also help.
We thank AnonReviewer2 for the thoughtful and detailed review, let us address the points brought up one by one in the original order (we will likewise clarify these points in the paper):
- Some hyperparameters were indeed left at "default" values because our tuner cannot efficiently tune a large set of hyperparameters.
Still we did tuning studies with lower and higher BPTT lengths, batch sizes and including Adam parameters (beta1, beta2, epsilon) and with other optimizers to make sure that our intuition about what hyperparameters are most important is correct. We did a tuning study with all hyperparameters (about 40 hyperparameters in total) to catch any unexpected parameter combinations even if it was a long shot due to the aforementioned tuner inefficiency.
- Yes, the down-projection is simply projecting down from the LSTM hidden size to the embedding size. The ratio of the embedding size and cell size is a tuneable. The cell and embedding sizes are computed from the budget and this input_embedding_ratio hyperparameter. As the paper puts it: "The tuner is given control over the presence and size of the down-projection, and thus over the tradeoff between the number of embedding vs. recurrent cell parameters.
Consequently, the cells’ hidden size and the embedding size is determined by the actual parameter budget, depth and the input embedding ratio hyperparameter."
- Yes, we didn't find a very different cell with promising results in the literature.
- No comment.
- We are working on factoring out the code from a larger system and providing training scripts with the tuned hyperparameters.
- The Multiplicative LSTM is indeed interesting. We did some preliminary investigation and could not make it perform very well.
In the end, it was excluded to avoid adding further multipliers to our already very high resource consumption.
- We used shared masks because of implementation convenience and for computational considerations.
Changelist:
- Better NAS results with more tuning on Wikitext-2 and Enwik8. The story is the same, still lagging other models.
- Tiny adjustments related to down-projections that hopefully clarify things.

Meta-review:
role: PC
data: comment: this submission demonstrates an existing loop-hole (?) in rushing out new neural language models by carefully (and expensively) running hyperparameter tuning of baseline approaches. i feel this is an important contribution, but as pointed out by some reviewers, i would have liked to see whether the conclusion stands even with a more realistic data (as pointed out by some in the field quite harshly, perplexity on PTB should not be considered seriously, and i believe the same for the other two corpora used in this submission.) that said, it's an important paper in general which will work as an alarm to the current practice in the field, and i recommend it to be accepted.

Decision: Accept (Poster)