Title: DYNAMIC EVALUATION OF NEURAL SEQUENCE MODELS

Abstract: We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.

1. INTRODUCTION:

Sequence generation and prediction tasks span many modes of data, ranging from audio and language modelling, to more general timeseries prediction tasks. Applications of such models include speech recognition, machine translation, dialogue generation, speech synthesis, forecasting, and music generation, among others. Neural networks can be applied to these tasks by predicting sequence elements one-by-one, conditioning on the history of sequence elements, forming an autoregressive model. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs), including long-short term memory (LSTM) networks (Hochreiter & Schmidhuber, 1997) in particular, have achieved many successes at these tasks. However, in their basic form, these models have a limited ability to adapt to recently observed parts of a sequence. Many sequences contain repetition; a pattern that occurs once is more likely to occur again. For instance, a word that occurs once in a document is much more likely to occur again. A sequence of handwriting will generally stay in the same handwriting style. A sequence of speech will generally stay in the same voice. Although RNNs have a hidden state that can summarize the recent past, they are often unable to exploit new patterns that occur repeatedly in a test sequence. This paper concerns dynamic evaluation, which we investigate as a candidate solution to this problem. Our approach adapts models to recent sequences using gradient descent based mechanisms. We show several ways to improve on past dynamic evaluation approaches in Section 5, and use our improved methodology to achieve state-of-the-art results in Section 7. In Section 6 we design a method to dramatically to reduce the number of adaptation parameters in dynamic evaluation, making it practical in a wider range of situations. In Section 7.4 we analyse dynamic evaluation's performance over varying time-scales and distribution shifts, and demonstrate that dynamically evaluated models can generate conditional samples that repeat many patterns from the conditioning data.

2. MOTIVATION:

Generative models can assign probabilities to sequences by modelling each term in the factorization given by the product rule. The probability of a sequence x 1:T = {x 1 , . . . , x T } factorizes as P (x 1:T ) = P (x 1 )P (x 2 |x 1 )P (x 3 |x 2 , x 1 ) • • • P (x T |x 1 . . . x T -1 ).

(1)

Methods that apply this factorization either use a fixed context when predicting P (x t |x 1:t-1 ), for instance as in N-grams or CNNs, or use a recurrent hidden state to summarize the context, as in an RNN. However, for longer sequences, the history x 1:t-1 often contains re-occurring patterns that are difficult to capture using models with fixed parameters (static models).

In many domains, in a dataset of sequences {x 1 1:T , x 2 1:T , ..., x M 1:T }, each sequence x i 1:T is generated from a slightly different distribution P (x i 1:T ). At any point in time t, the history of a sequence x i

1:t-1

contains useful information about the generating distribution for that specific sequence P (x i 1:T ). Therefore adapting the model parameters learned during training θ g is justified. We aim to infer a set of model parameters θ l from x i 1:t-1 that will better approximate P (x i t |x i 1:t-1 ) within sequence i. Many sequence modelling tasks are characterised by sequences generated from slightly different distributions as in the scenario described above. The generating distribution may also change continuously across a single sequence; for instance, a text excerpt may change topic. Furthermore, many machine learning benchmarks do not distinguish between sequence boundaries, and concatenate all sequences into one continuous sequence. Thus, many sequence modelling tasks could be seen as having a local distribution P l (x) as well as a global distribution P g (x) := P (l)P l (x) dl. During training time, the goal is to find the best fixed model possible for P g (x). However, during evaluation time, a model that can infer the current P l (x) from the recent history has an advantage.

3. DYNAMIC EVALUATION:

Dynamic evaluation methods continuously adapt the model parameters θ g , learned at training time, to parts of a sequence during evaluation. The goal is to learn adapted parameters θ l that provide a better model of the local sequence distribution, P l (x). When dynamic evaluation is applied in the present work, a long test sequence x 1:T is divided up into shorter sequences of length n. We define s 1:M to be a sequence of shorter sequence segments s i

s 1:M = {s 1 = x 1:n , s 2 = x n+1:2n , s 3 = x 2n+1:3n , ..., s M }.

(2)

The initial adapted parameters θ 0 l are set to θ g , and used to compute the probability of the first segment, P (s 1 |θ 0 l ). This probability gives a cross entropy loss L(s 1 ), with gradient ∇L(s 1 ), which is computed using truncated back-propagation through time (Werbos, 1990) . The gradient ∇L(s 1 ) is used to update the model, resulting in adapted parameters θ 1 l , before evaluating P (s 2 |θ 1 l ). The same procedure is then repeated for s 2 , and for each s i in the sequence as shown in Figure 1 . Gradients for each loss L(s i ) are only backpropagated to the beginning of s i , so computation is linear in the sequence length. Each update applies one maximum likelihood training step to approximate the current local distribution P l (x). The computational cost of dynamic evaluation is one forward pass and one gradient computation through the data, with some slight overhead to apply the update rule for every sequence segment.

As in all autoregressive models, dynamic evaluation only conditions on sequence elements that it has already predicted, and so evaluates a valid log-probability for each sequence. Dynamic evaluation can also be used while generating sequences. In this case, the model generates each sequence segment s i using fixed weights, and performs a gradient descent based update step on L(s i ). Applying dynamic evaluation for sequence generation could result in generated sequences with more consistent regularities, meaning that patterns that occur in the generated sequence are more likely to occur again.

4. BACKGROUND:



4.1. RELATED APPROACHES:

Adaptive language modelling was first considered for n-grams, adapting to recent history via caching (Jelinek et al., 1991; Kuhn, 1988) , and other methods Bellegarda (2004) . More recently, the neural cache approach (Grave et al., 2017) and the closely related pointer sentinel-LSTM (Merity et al., 2017b) have been used to for adaptive neural language modelling. Neural caching has recently been used to improve the state-of-the-art at word-level language modelling (Merity et al., 2017a) .

The neural cache model learns a type of non-parametric output layer on the fly at test time, which allows the network to adapt to recent observations. Each past hidden state h i is paired with the next input x i+1 , and is stored as a tuple (h i , x i+1 ). When a new hidden state h t is observed, the output probabilities are adjusted to give a higher weight to output words that coincided with past hidden states with a large inner product (h T t h i ). L ( s 2 ) where e (xi+1) is a one hot encoding of x i+1 , and ω is a scaling parameter. The cache probabilities are interpolated with the base network probabilities to adapt the base network at test time.

EQUATION

The neural cache closely relates to dynamic evaluation, as both methods can be added on top of a base model for adaptation at test time. The main difference is the mechanism used to fit to recent history: the neural cache approach uses a non-parametric, nearest neighbours-like method, whereas dynamic evaluation uses a gradient descent based method to change model parameters dynamically. Both methods rely on an autoregressive factorisation, as they depend on observing sequence elements after they are predicted in order to perform adaptation. Dynamic evaluation and neural caching methods are therefore both applicable to sequence prediction and generation tasks, but not directly to more general supervised learning tasks.

One drawback of the neural cache method is that it cannot adjust the recurrent hidden state dynamics. As a result, the neural cache's ability to capture information that occurs jointly between successive sequence elements is limited. This capability is critical for adapting to sequences where each element has very little independent meaning, e.g. character level language modelling. Another related approach is fast weights, (Ba et al., 2016; Schmidhuber, 1992) . Fast weights feature recurrent architectures with dynamically changing weight matrices as a function of recent sequence history. Thus, dynamic evaluation as applied at test time, could be considered a form of fast-weights. In traditional fast weights, the network learns to control changes to the weights during training time, allowing it to be applied to more general sequence problems including sequence labeling. In dynamic evaluation, the procedure to change the weights is automated at test time via gradient descent optimization, making it only directly applicable to autoregressive sequence modelling. As dynamic evaluation leverages gradient descent, it has the potential to generalize better to previously unseen pattern repetitions at test time.

4.2. DYNAMIC EVALUATION IN NEURAL NETWORKS:

Dynamic evaluation of neural language models was proposed by Mikolov et al. (2010) . Their approach simply used stochastic gradient descent (SGD) updates at every time step, computing the gradient with fully truncated backpropagation through time, which is equivalent to setting n = 1 in equation ( 2). Dynamic evaluation has since been applied to character and word-level language models (Graves, 2013; Krause et al., 2017; Ororbia II et al., 2017; Fortunato et al., 2017) . Previous work using dynamic evaluation considered it as an aside, and did not explore it in depth.

5. UPDATE RULE METHODOLOGY FOR DYNAMIC EVALUATION:

We propose several changes to Mikolov et al. (2010) 's dynamic evaluation method with SGD and fully truncated backpropagation, which we refer to as traditional dynamic evaluation. The first modification reduces the update frequency, so that gradients are backpropagated over more timesteps. This change provides more accurate gradient information, and also improves the computational efficiency of dynamic evaluation, since the update rule is applied much less often. We use sequence segments of length 5 for word-level tasks and 20 for character-level tasks.

Next, we add a global decay prior to bias the model towards the parameters θ g learned during training. Our motivation for dynamic evaluation assumes that the local generating distribution P l (x) is constantly changing, so it is potentially desirable to weight recent sequence history higher in adaptation. Adding a global decay prior accomplishes this by causing previous adaptation updates to decay exponentially over time. The use of a decay prior for dynamic evaluation relates to the update rule used for fast weights in Ba et al. (2016) , which decayed fast weights towards zero exponentially over time. For SGD with a global prior, learning rate η and decay rate λ; we form the update rule

EQUATION

We then consider using an RMSprop (Tieleman & Hinton, 2012) derived update rule for the learning rule in place of SGD. RMSprop uses a moving average of recent squared gradients to scale learning rates for each weight. In dynamic evaluation, near the start of a test sequence, RMSprop has had very few gradients to average, and therefore may not be able to leverage its updates as effectively. For this reason, we collect mean squared gradients, MS g , on the training data rather than on recent test data (which is what RMSprop would do). MS g is given by

EQUATION

where N b is the number of training batches and ∇L k is the gradient on the kth training batch. The mini-batch size for this computation becomes a hyper-parameter, as larger mini-batches will result in smaller mean squared gradients. The update rule, which we call RMS with a global prior in our experiments, is then

EQUATION

where is a stabilization parameter. For the decay step of our update rule, we also consider scaling the decay rate for each parameter proportionally to MS g . Parameters with a high RMS gradient affect the dynamics of the network more, so it makes sense to decay them faster. RMS norm is MS g divided by its mean, resulting in a normalized version of MS g with a mean of 1:

EQUATION

)

We clip the values of RMS norm to be no greater than 1 /λ to be sure that the decay rate does not exceed 1 for any parameter. Combining the learning component and the regularization component results in the final update equation, which we refer to as RMS with an RMS global prior

EQUATION

6. SPARSE DYNAMIC EVALUATION:

Mini-batching over sequences is desirable for some test-time sequence modelling applications because it allows faster processing of multiple sequences in parallel. Dynamic evaluation has a high memory cost for mini-batching because it is necessary to store a different set of parameters for each sequence in the mini-batch. Therefore, we consider a sparse dynamic evaluation variant that updates a smaller number of parameters. We introduce a new adaptation matrix M which is initialized to zeros. M multiplies hidden state vector h t of an RNN at every time-step to get a new hidden state h t , via h t = h t + Mh t .

(9) h t then replaces h t and is propagated throughout the network via both recurrent and feed-forward connections. In a stacked RNN, this formulation could be applied to every layer or just one layer. Applying dynamic evaluation to M avoids the need to apply dynamic evaluation to the original parameters of the network, reduces the number of adaptation parameters, and makes mini-batching less memory intensive. We reduce the number of adaptation parameters further by only using M to transform an arbitrary subset of H hidden units. This results in M being an H ×H matrix with d = H2 adaptation parameters. If H is chosen to be much less than the number of hidden units, this reduces the number of adaptation parameters dramatically. In Section 7.3 we experiment with sparse dynamic evaluation for character-level language models.

7. EXPERIMENTS:

We applied dynamic evaluation to word-level and character-level language modelling. In all tasks, we evaluate dynamic evaluation on top of a base model. After training the base model, we tune hyper-parameters for dynamic evaluation on the validation set, and evaluate both the static and dynamic versions of the model on the test set. We also consider follow up experiments that analyse the sequence lengths for which dynamic evaluation is useful.

7.1. SMALL SCALE WORD-LEVEL LANGUAGE MODELLING:

We train base models on the Penn Treebank (PTB, Marcus et al., 1993) , WikiText-2 (Merity et al., 2017b) datasets, and compare the performance of static and dynamic evaluation. These experiments compare dynamic evaluation against past approaches such as the neural cache and measure dynamic evaluation's general performance across different models and datasets.

PTB is derived from articles of the Wall Street Journal. It contains 929k training tokens and a vocab size limited to 10k words. It is one of the most commonly used benchmarks in language modelling. We consider two baseline models on PTB, a standard LSTM implementation with recurrent dropout (Zaremba et al., 2014) , and the recent state-of-the-art averaged SGD (ASGD) weight-dropped LSTM (AWD-LSTM, Merity et al., 2017a) .

Our standard LSTM was taken from the Chainer (Tokui et al., 2015) tutorial on language modelling1 , and used two LSTM layers with 650 units each, trained with SGD and regularized with recurrent dropout. On our standard LSTM, we experiment with traditional dynamic evaluation as applied by Mikolov et al. (2010) , as well as each modification we make building up to our final update rule as described in Section 5. As our final update rule (RMS + RMS global prior) worked best, we use this for all other experiments and use "dynamic eval" by default to refer to this update rule in tables.

We applied dynamic evaluation on an AWD-LSTM (Merity et al., 2017a) . The AWD-LSTM is a vanilla LSTM that combines the use of drop-connect (Wan et al., 2013) on recurrent weights for regularization, and a variant of ASGD (Polyak & Juditsky, 1992) for optimisation. Our model, which used 3 layers and tied input and output embeddings (Press & Wolf, 2017; Inan et al., 2017) , was intended to be a direct replication of AWD-LSTM, using code from their implementation 2 . Results are given in Table 1 .

Dynamic evaluation gives significant overall improvements to both models on this dataset. Dynamic evaluation also achieves better final results than the neural cache on both a standard LSTM and the AWD-LSTM reimplementation, and improves the state-of-the-art on PTB.

model parameters valid test:

RNN+LDA+kN-5+cache (Mikolov & Zweig, 2012) 92.0 CharCNN (Kim et al., 2016) 19M 78.9 LSTM (Zaremba et al., 2014) 66M 82.2 78.4 Variational LSTM (Gal & Ghahramani, 2016) 66M 73.4 Pointer sentinel-LSTM (Merity et al., 2017b) 21M 72.4 70.9 Variational LSTM + augmented loss (Inan et al., 2017) 51M 71.1 68.5 Variational RHN (Zilly et al., 2017) 23M 67.9 65.4 NAS cell (Zoph & Le, 2017) 54M 62.4 Variational LSTM + gradual learning (Aharoni et al., 2017) 105M 61.7 LSTM + BB tuning (Melis et al., 2017) 24M 60.9 58.3 LSTM (Grave et al., 2017) 86.9 82.3 LSTM + neural cache (Grave et al., 2017) 74 (Merity et al., 2017b) 84.8 80.8 LSTM + BB tuning (Melis et al., 2017) 24M 69.1 65.9

LSTM (Grave et al., 2017) 104.2 99.3 LSTM + neural cache (Grave et al., 2017) 72 WikiText-2 is roughly twice the size of PTB, with 2 million training tokens and a vocab size of 33k. It features articles in a non-shuffled order, with dependencies across articles that adaptive methods should be able to exploit. For this dataset, we use the same baseline LSTM implementation and AWD-LSTM re-implementation as on PTB. Results are given in Table 2 .

Dynamic evaluation improves the state-of-the-art perplexity on WikiText-2, and provides a significantly greater improvement than neural caching to both base models. This suggests that dynamic evaluation is effective at exploiting regularities that co-occur across non-shuffled documents. model valid test LSTM (Grave et al., 2017) 121.8 LSTM + neural cache (Grave et al., 2017) 99.9

AWD-LSTM 80.0 87.5 AWD-LSTM + neural cache 67.5 75.1 AWD-LSTM + dynamic eval 63.3 70.3

Table 3 : text8 (word-level) perplexities

7.2. MEDIUM SCALE WORD-LEVEL LANGUAGE MODELLING:

We benchmark the performance of dynamic evaluation against static evaluation and the neural cache on the larger text8 dataset. Like WikiText-2, text8 is derived from Wikipedia text. Text8 was introduced for word level language modelling by Mikolov et al. (2014) , which preprocessed the data by mapping rare words to an '<unk>' token, resulting in a vocab of 44k and 17M training tokens. We use the same test set as in Mikolov et al. (2014) , but also hold out the final 100k training tokens as a validation set to allow for fair hyper-parameter tuning (the original task did not have a validation set). We trained an AWD-LSTM with 52M parameters using the implementation from Merity et al. (2017a) . We then compare the performance of static evaluation, dynamic evaluation, and neural caching at test time.

To ensure a fair comparison between dynamic evaluation and the neural cache, we used robust hyper-parameter tuning on the validation set for both methods. For dynamic evaluation, we used the hyper-parameter settings found on PTB, and only tuned the learning rate (to 2 significant figures). The neural cache uses 3 hyper-parameters: the cache length, a mixing parameter and a flatness parameter.

Starting from a cache size of 3000, we used a series of grid searches to find optimal values for the mixing parameter and flatness parameter (to 2 significant figures). We then varied the cache size in the range of 2000-4000, and found that the affect of this was negligible, so we kept the cache size at 3000. Results are given in table 3 , with the results from Grave et al. (2017) that used the same test set given for context.

Dynamic evaluation soundly outperforms static evaluation and the neural cache method, demonstrating that the benefits of dynamic evaluation do not wash away when using a stronger model with more training data.

7.3. CHARACTER-LEVEL LANGUAGE MODELLING:

We consider dynamic evaluation on the character-level text8, and Hutter Prize (Hutter, 2006) datasets. The Hutter Prize dataset is comprised of Wikipedia text, and includes XML and characters from non-Latin languages. It is 100 million UTF-8 bytes long and contains 205 unique bytes. Similarly to other reported results, we use a 90-5-5 split for training, validation, and testing. The text8 dataset is derived the Hutter Prize dataset, but has all XML removed, and is lower cased to only have 26 characters of English text plus spaces. The character-level text8 task corresponds to the unprocessed version of the text8 data used for our medium-scale word level language modelling experiments. As with Hutter Prize, we use the standard 90-5-5 split for training, validation, and testing for text8. We used a multiplicative LSTM (mLSTM) (Krause et al., 2016) 3 as our base model for both datasets.

The mLSTMs for both tasks used 2800 hidden units, an embedding layer of 400 units, weight normalization (Salimans & Kingma, 2016) , variational dropout (Gal & Ghahramani, 2016) , and ADAM (Kingma & Ba, 2014) for training.

We also consider sparse dynamic evaluation, as described in Section 6, on the Hutter Prize dataset.

For sparse dynamic evaluation, we adapted a subset of 500 hidden units, resulting in a 500×500 adaptation matrix and 250k adaptation parameters. Our mLSTM only contained one recurrent layer, so only one adaptation matrix was used for sparse dynamic evaluation. All of our dynamic evaluation results in this section use the final update rule given in Section 5. Results for Hutter Prize are given in Table 4 , and results for text8 are given in Table 5 .

model parameters test:

Stacked LSTM (Graves, 2013) 21M 1.67 Stacked LSTM + traditional dynamic eval (Graves, 2013) 21M

1.33 Multiplicative integration LSTM (Wu et al., 2016) 17M 1.44 HyperLSTM (Ha et al., 2017) 27M 1.34 Hierarchical multiscale LSTM (Chung et al., 2017) 1.32 Bytenet decoder (Kalchbrenner et al., 2016) 1.31 LSTM + BB tuning (Melis et al., 2017) 46M 1.30 Recurrent highway networks (Zilly et al., 2017) 46M 1.27 Fast-slow LSTM (Mujika et al., 2017 (Wu et al., 2016) 4M 1.44 LSTM (Cooijmans et al., 2017) 1.43 Batch normalised LSTM (Cooijmans et al., 2017) 1.36 Hierarchical multiscale LSTM (Chung et al., 2017) 1.29 Recurrent highway networks (Zilly et al., 2017 Dynamic evaluation achieves large improvements to our base models and state-of-the-art results on both datasets. Sparse dynamic evaluation also achieves significant improvements on Hutter Prize using only 0.5% of the adaptation parameters of regular dynamic evaluation.

7.4. TIME-SCALES OF DYNAMIC EVALUATION:

We measure time-scales at which dynamic evaluation gains an advantage over static evaluation.

Starting from the model trained on Hutter Prize, we plot the performance of static and dynamic evaluation against the number of characters processed on sequences from the Hutter Prize test set, and sequences in Spanish from the European Parliament dataset (Koehn, 2005) .

The Hutter Prize data experiments show the timescales at which dynamic evaluation gained the advantage observed in Table 4 . We divided the Hutter Prize test set into 500 sequences of length 10000, and applied static and dynamic evaluation to these sequences using the same model and methodology used to obtain results in Table 4 . Losses were averaged across these 500 sequences to obtain average losses at each time step. Plots of the average cross-entropy errors against the number of Hutter characters sequenced are given in Figure 2a .

The Spanish experiments measure how dynamic evaluation handles large distribution shifts between training and test time, as Hutter Prize contains very little Spanish. We used the first 5 million characters of the Spanish European Parliament data in place of the Hutter Prize test set. The Spanish experiments used the same base model and dynamic evaluation settings as Hutter Prize. Plots of the average cross-entropy errors against the number of Spanish characters sequenced are given in Figure 2b .

On both datasets, dynamic evaluation gave a very noticeable advantage after a few hundred characters.

For Spanish this advantage continued to grow as more of the sequence was processed, whereas for Hutter, this advantage was maximized after viewing around 2-3k characters. The advantage of dynamic evaluation was also much greater on Spanish sequences than Hutter sequences.

We also drew 300 character conditional samples from the static and dynamic versions of our model after viewing 10k characters of Spanish. For the dynamic model, we continued to apply dynamic evaluation during sampling as well, by the process described in Section 3. The conditional samples are given in the appendix. The static samples quickly switched to English that resembled Hutter Prize data. The dynamic model generated data with some Spanish words and a number of made up words with characteristics of Spanish words for the entirety of the sample. This is an example of the kinds of features that dynamic evaluation was able to learn to model on the fly.

8. CONCLUSION:

This work explores and develops methodology for applying dynamic evaluation to sequence modelling tasks. Experiments show that the proposed dynamic evaluation methodology gives large test time improvements across character and word level language modelling. Our improvements to language modelling have applications to speech recognition and machine translation over longer contexts, including broadcast speech recognition and paragraph level machine translation. Overall, dynamic evaluation is shown to be an effective method for exploiting pattern re-occurrence in sequences.

:

https://github.com/chainer/chainer/tree/master/examples/ptb

https://github.com/salesforce/awd-lstm-lm

https://github.com/benkrause/mLSTM

Figures and Tables:

Figure 1: Illustration of dynamic evaluation. The model evaluates the probability of sequence segments s i . The gradient ∇L(s i ) with respect to the log probability of s i is used to update the model parameters θ i-1 l to θ i l before the model progresses to the next sequence segment. Dashed edges are what distinguish dynamic evaluation from static (normal) evaluation.

Figure 2: Average losses in bits/char of dynamic evaluation and static evaluation plotted against number of characters processed; on sequences from the Hutter Prize test set (left) and European Parliament dataset in Spanish (right), averaged over 500 trials for each. Losses at each data point are averaged over sequence segments of length 100, and are not cumulative. Note the different y-axis scales in the two plots.

Penn Treebank perplexities. bptt refers to sequence segment lengths.

Table content: <table><tr><td>.6 72.1</td></tr></table>

WikiText-2 perplexities.

Table content: <table><tr><td>.1</td><td>68.9</td></tr></table>

Hutter Prize test set error in bits/char.

Table content: <table><tr><td>)</td><td>47M</td><td>1.25</td></tr><tr><td>mLSTM (Krause et al., 2016)</td><td>46M</td><td>1.24</td></tr><tr><td>mLSTM + sparse dynamic eval (d = 250k)</td><td>46M</td><td>1.13</td></tr><tr><td>mLSTM + dynamic eval</td><td>46M</td><td>1.08</td></tr><tr><td>model</td><td colspan="2">parameters test</td></tr><tr><td>Multiplicative RNN (Mikolov et al., 2012)</td><td>5M</td><td>1.54</td></tr><tr><td>Multiplicative integration LSTM</td><td/><td/></tr></table>

Peer review (iclr2018):
 The authors provide an improved implementation of the idea of dynamic evaluation, where the update of the parameters used in the last time step proposed in (Mikolov et al. 2010) is replaced with a back-propagation through the last few time steps, and uses RMSprop rather than vanilla SGD. The method is applied to word level and character level language modeling where it yields some gains in perplexity. The algorithm also appears able to perform domain adaptation, in a setting where a character-level language model trained mostly on English manages to quickly adapt to a Spanish test set. While the general idea is not novel, the implementation choices matter, and the authors provide one which appears to work well with recently proposed models. The character level experiments on the multiplicative LSTM make the most convincing point, providing a significant improvement over already good results on medium size data sets. Figure 2 also makes a strong case for the method*s suitability for applications where domain adaptation is important. The paper*s weakest part is the word level language modeling section. Given the small size of the data sets considered, the results provided are of limited use, especially since the development set is used to fit the RMSprop hyper-parameters. How sensitive are the final results to this choice? Comparing dynamic evaluation to neural cache models is a good idea, given how both depend en medium-term history: (Grave et al. 2017) provide results on the larger text8 and wiki103, it would be useful to see results for dynamic evaluation at least on the former. An indication of the actual additional evaluation time for word-level, char-level and sparse char-level dynamic evaluation would also be welcome. Pros: - Good new implementation of an existing idea - Significant perplexity gains on character level language modeling - Good at domain adaptation Cons: - Memory requirements of the method - Word-level language modeling experiments need to be run on larger data sets (Edit: the authors did respond satisfactorily to the original concern about the size of the word-level data set)