Mogrifier LSTM
Authors: Gábor Melis
Tomáš Kočiský
Phil Blunsom
Venue:  (2020)
Keywords: lstm
language modelling

Abstract: Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur. Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language. In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output. This mechanism affords the modelling of a richer space of interactions between inputs and their context. Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent. Experiments demonstrate markedly improved generalization on language modelling in the range of 3–4 perplexity points on Penn Treebank and Wikitext-2, and 0.01–0.05 bpc on four character-based datasets. We establish a new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.


Published as a conference paper at ICLR 2020
MOGRIFIER LSTM
Gábor Melisi, Tomás Kočiský", v Phil Blunsom it [mel isg1, tkoci sky, , pb pblunsom) @google.com com "DeepMind, London, UK University of Oxford
ABSTRACT
Many advances in Natural Language Processing have been based upon more expres- sive models for how inputs interact with the context in which they occur. Recurrent networks, which have enjoyed modicum of success, still lack the generalization and systematicity ultimately required for modelling language. In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output. This mechanism affords the modelling of richer space of interactions between inputs and their context. Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent. Experiments demonstrate markedly improved generalization on language modelling in the range of 3-4 perplexity points on Penn Treebank and Wikitext-2, and 0.01-0.05 bpc on four character-based datasets. We establish new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.
INTRODUCTION
The domination of Natural Language Processing by neural models is hampered only by their limited ability to generalize and questionable sample complexity (Belinkov and Bisk 2017; Jía and Liang 2017; lyyer et al. 2018; Moosavi and Strube 2017; Agrawal et al. 2016), their poor grasp of grammar (Linzen et al. 2016; Kuncoro al. 2018), and their inability to chunk input sequences into meaningful units (Wang et al. 2017). While direct attacks on the latter are possible, in this paper, we take language-agnostic approach to improving Recurrent Neural Networks (RNN, Rumelhart et al. (1988)), which brought about many advances in tasks such as language modelling, semantic parsing, machine translation, with no shortage of non-NLP applications either (Bakker 2002; Mayer et al. 2008). Many neural models are built from RNNs including the sqquence-to-sequence family (Sutskever al. 2014) and its attention-based branch (Bahdanau et al. 2014). Thus, innovations in RNN architecture tend to have a trickle-down effect from language modelling, where evaluation is often the easiest and data the most readily available, to many other tasks, trend greatly strengthened by ULMFiT (Howard and Ruder 2018), ELMo (Peters et al. 2018) and BERT (Devlin al. 2018), which promote language models from architectural blueprints to pretrained building blocks.
To improve the generalization ability of language models, we propose an extension the LSTM (Hochreiter and Schmidhuber 1997), where the LSTM's input æ is gated conditioned on the output of the previous step hprev- Next, the gated input used in similar manner to gate the output of the previous time step. After couple of rounds of this mutual gating, the last updated x and hrrev are fed to an LSTM. By introducing these additional of gating operations, in one sense, our model joins the long list of recurrent architectures with gating structures of varying complexity which followed the invention of Elman Networks (Elman 1990). Examples include the LSTM, the GRU (Chung et al. 2015), and even designs by Neural Architecture Search (Zoph and Le 2016).
Intuitively, in the lowermost layer, the first gating step scales the input embedding (itself a representa- tion of the average context in which the token occurs) depending on the actual context, resulting in contextualized representation of the input. While intuitive, as Section shows, this interpretation cannot account for all the observed phenomena.
In a more encompassing view, our model can be seen as enriching the mostly additive dynamics of recurrent transitions placing it in the company of the Input Switched Affine Network (Foerster et al.

---

Published as a conference paper at ICLR 2020
ho
h2
h4
LSTM
x1
x
x3
x5
Figure I: Mogrifier with 5 rounds of updates. The previous state h" hyper transformed lincarly (dashed arrows), fed through sigmoid and gates x: linearly transformed x' gates h and produces h² After a number of repetitions this mutual gating cycle, the last values of ht and x= sequences are fed to an LSTM cell. The prev subscript of h omitted to reduce clutter.
in elementwise manner producing x' Conversely, the
2017) with separate transition matrix for each possible input, and the Multiplicative RNN (Sutskever et al. 201 1), which factorizes the three-way tensor of stacked transition matrices. Also following this line of research are the Multiplicative Integration LSTM (Wu al. 2016) and closest to our model in the literature the Multiplicative LSTM (Krause et al. 2016). The results in Section 3.4 demonstrate the utility of our approach, which consistently improves on the LSTM and establishes a new state of the art all but the largest dataset, Enwik8, where match similarly sized transformer models.
MODEL
To allow for ease of subsequent extension, present the standard LSTM update (Sak et al. 2014) with imput and state of size m and n respectively as the following function:
LSTM: RT x R x x Ra x R
LSTM(#, Cprevy hppev) h..
The updated state e and the output are computed as follows:
= o a x + W + b')
i a W W pre + b')
= tanh( w x + Wih hom + b²)
0 = o (Wor x + Woh hpre + b")
c f O Cpev fi
= o @ tanh(c),
where a the logistic sigmoid function, @ the elementwise product, W and b are weight matrices and biases.
While the LSTM is typically presented as a solution to the vanishing gradients problem, its gate can also be interpreted scaling the rows of weight matrices W (ignoring the non-linearity in j). In this sense, the LSTM nudges Elman Networks towards context-dependent transitions and the extreme extreme, we could end up with Hypernetworks (Ha et al. 2016). Here, instead, we take a more cautious step, and equip the LSTM with gates that scale the columns of all its weight matrices W context-dependen manner. The scaling of the matrices Wa= (those that transform the cell input) makes the input embeddings dependent the cell state, while the scaling of W does the reverse.
of Input Switched Affine Networks. If we took another, larger step towards that
The Mogrifier! LSTM is an LSTM where two inputs æ and hpwe modulate one another in an alternating fashion before the usual LSTM computation takes place (see Fig. 1). That is, Mogrify (x, Cprevy hewee) LSTM(x Oprevs prew)) where the modulated inputs øt and h h defined as the highest indexed xt and hiee respectively, from the interleaved sequences (2' prev
f-2
for odd i [1...r)
It's like transmogrifier' without the magic: it can only shrink or expand objects. Transmogrify (verb, 1650s): completely alter the form of something surprising or magical manner.
2

---

Published as a conference paper at ICLR 2020
h prev 20 ( h
prev
for even i [1...r]
(2)
with x= = x and h prrvv Apmy The number of "rounds". E N, is hyperparameter; r 0
recovers the LSTM. Multiplicatiinn with the constant 2 ensures that randomly initialized Q' Rè
matrices result in transformations close to identity, To reduce the number of additional model
parameters, we typically factorize the Q' R matrices products of low-rank matrices: Q'
Qien Oigha with Q' E Rmx" Qien E Rm Qiiht E R where l < min(m, n2) is the rank.
EXPERIMENTS
3.1
THE CASE FOR SMALL-SCALLE
Before describing the details of the data, the experimental setup and the results, we take a short detour to motivate work on smaller-scale datasets. recurring theme in the history of sequence models that the problem of model design is intermingled with optimizability and scalability. Elman Networks are notoriously difficult to optimize, a property that ultimately gave birth to the idea of the LSTM, but also more recent models such as the Unitary Evolution RNN (Arjovsky al. 2016) and fixes like gradient clipping (Pascanu et al. 2013). Still, it is far from clear- - we could optimize these models well how different their biases would turn optimization is fairly evident in these cases.
be. The non-separability of model and
Scalability, the other hand, is often optimized for indirectly. Given the limited ability of current models to generalize, we often compensate by throwing more data at the problem. To fit a larger dataset, model size must be increased. Thus the best performing models are evaluated based on their scalability". Today, scaling up still yields tangible gains on down-stream tasks, and language mod- elling data is abundant. However, we believe that simply scaling up will not solve the generalization problem and better models will be needed. Our hope is that by choosing small enough datasets, so that model size is no longer the limiting factor, we get number of practical advantages:
Generalization ability will be more clearly reflected in evaluations even without domain adaptation.
Turnaround time in experiments will be reduced, and the freed up computational budget can be put to good use by controlling for nuisance factors.
The transient effects of changing hardware performance characteristics are somewhat lessened.
Thus, we develop, analyse and evaluate models primarily on small datasets. Evaluation on larger datasets included learn more about the models' scaling behaviour and because of its relevance for applications, but it is to be understood that these evaluations come with much larger error bars and provide more limited guidance for further research on better models.
3.2 DATASETS
We compare models on both word and character-level language modelling datasets. The two word- level datasets we picked are the Penn Treebank (PTB) corpus by Marcus et al. (1993) with prepro- cessing from Mikolov et (2010) and Wikitext-2 by Merity et al. (2016), which is about twice the size of PTB with larger vocabulary and lighter preprocessing. These datasets are definitely on the small side, but and because of this - they are suitable for exploring different model biases. Their main shortcoming is the small vocabulary size, only in the tens of thousands, which makes them inappropriate for exploring the behaviour of the long tail. For that, open vocabulary language modelling and byte pair encoding (Sennrich al. 2015) would be obvious choice. Still, primary goal here is the comparison of the LSTM and Mogrifier architectures, thus we instead opt for character-based language modelling tasks, where vocabulary size is not an issue, the long tail is not truncated, and there are no additional hyperparameters as in byte pair encoding that make fair comparison harder. The first character-basee based corpus is Enwik8 from the Hutter Prize dataset (Hutter 2012). Following common practice, we use the first 90 million characters for training and the remaining 10 million evenly split between validation and test. The character-level task on the
Note that the focus scalability is not problem per Indeed the unsupervised pretraining methods (Peters al. 2018; Devlin al. 2018) take great advantage of this approach.
3

---

Published as a conference paper at ICLR 2020
Table 1: Word-level perplexities of near state-of-the-ar models, our LSTM baseline and the Mogrifier on PTB Wikitext-2. Models with Mixture of Softmaxes (Yang 2017) are denoted with MoS. depth N with dN. MC stands for Monte- Carlo dropout evaluation. Previous state-of-the-art results in italics. Note the comfortable margin of 2.8-4.3 perplexity points the Mogrifier enjoys over the LSTM.
No Dyneval
Dyneval
Val.
Test
Val.
Test
FRAGE (d3, McS15) (Gong et al. 2018)
22M
54.1
52.4
47.4
46.5
AWD-LSTM (d3, MoS155 (Yang al. 2017)
22M
56.5
54.4
48.3
47.7
Transformer-Xi (Dai et al. 2019)
24M
56.7
54.5
PTB
LSTM (d2)
24M
55.8
54.6
48.9
48.4
Mogrifier (d2)
24M
52.1
51.0
45.1
45.0
LSTM (d2, MC)
24M
55.5
54.1
48.6
48.4
Mogrifier (d2, MC)
24M
51.4
50.1
44.9
44.8
FRAGE (d3, MoS15) (Gong et al. 2018)
35M
60.3
58.0
40.8
39.1
AWD-LSTM (d3, MoS15) (Yang et al. 2017)
35M
63.9
61.2
42.4
40.7
WT2
EN
LSTM (d2, MoS2)
35M
62.6
60.1
43.2
41.5
Mogrifier (d2, MoS2)
35M
58.7
56.6
40.6
39,0
LSTM (d2, MoS2, MC)
35M
61.9
59.4
43.2
41.4
Mogrifier (d2, MoS2, MC)
35M
57.3
55.1
40.2
38.6
Mikolov preprocessed PTB corpus (Merity et al. 2018) unique that it has the disadvantages of closed vocabulary without the advantages of word-level modelling, but we include it for comparison to previous work. The final character-level dataset is the Multilingual Wikipedia Corpus (MWC, Kawakami et al. (2017)), from which we focus on the English and Finnish language subdatasets in the single text, large setting.
3.3 SETUP
We tune hyperparameters following the experimental setup of Melis et al. (2018) using a black-box hyperparameter tuner based on batched Gaussian Process Bandits (Golovin et al. 2017). For the LSTM, the tuned hyperparameters are the same: input_emiiel _eweeeeeneenneeeee ratio, learning_raa rate, 12 12_penalty, imput_ doopo_d_ooutttpp inter laye__ayr__aaa _dropoutr state_ dropout, output_ _drouuut.. For the Mogrifier, the number of rounds r' and the rank of the low-rank approximation also tuned (allowing for full rank, too). For word-level tasks, BPTT (Werbos et al. 1990) window size is set to 70 and batch size to 64. For character-level tasks, BPTT window size is set to 150 and batch size to 128 except for Enwik8 where the window size is 500. Input and output embeddings et al. (2016) and Press and Wolf (2016). Optimization is performed with Adam (Kingma and Ba 2014) with 31 0, setting that resembles R.SPrrp without momentum. Gradients are clipped (Pascanu et al. 2013) to norm 10. We switch to averaging weights similarly to Merity et al. (2017) after certain number of checkpoints with no improvement in validation cross-entropy at 80% of the training time at the latest. We found no benefit to using two-step finetuning.
tied for word-level tasks following Inan
Model evaluation is performed with the standard, deterministic dropout approximation or Monte- Carlo averaging (Gal and Ghahramani 2016) where explicitly noted (MC). In standard dropout evaluation, dropout turned off while in MC dropout predictions are averaged over randomly sampled dropout masks (200 in our experiments). Optimal softmax temperature is determined on the validation set, and in the MC case dropout rates are scaled (Melis al. 2018). Finally, we report results with and without dynamic evaluation (Krause et al. 2017). Hyperparameters for dynamic evaluation are tuned using the same method (see Appendix A for details).
We make the code and the tuner output available at https//gittub.ccmddemmm///aaaaa
3.4 RESULTS
Table I lists our results on word-level datasets. On the PTB and Wikitext-2 datasets, the Mogrifier has lower perplexity than the LSTM by 3-4 perplexity points regardless of whether not dynamic evaluation (Krause et al. 2017) and Monte-Carlo averaging are used. On both datasets, the state of the art is held by the AWD LSTM (Merity al. 2017) extended with Mixture of Softmaxes (Yang
4

---

Published as a conference paper at ICLR 2020
Table 2: Bits per character character-basse datasets of near state-of-the-at models, LSTM baseline and Mogrifier. Previous state-of the-rrr results in italics. Depth denoted dN. MC stands for Monte-Carle Carlo dropout evaluation. Once again the Mogrifier strictly dominates the LSTM and sets a new state of the art on all but the Enwik8 dataset where with dynamic evaluation closes the gap to the Transformer-XI. of similar size Krause et al. (2019), Ben Krause, personal communications, May 17, 2019). On most datasets, model size set large enough for underfitting not be an issue. This was very much the case with Enwik8, so we grouped models of similar sizes together for ease comparison. Unfortunately, couple of dynamic evaluation test rums diverged (NaN) on the test set and some were just too expensive to run (Enwik8, MC).
No Dyneval
Dyneval
Val.
Test
Val.
Test
Trellis Networks (Bai et al. 2018)
13.4M
1.159
AWD-LSTM (d3) (Merity et al. 2017)
13.8M
1.175
PTB
LSTM (d2)
24M
1.163
1.143
1.116
1.103
Mogrifier (d2)
24M
1.149
1.131
1.098
1.088
LSTM (d2, MC)
24M
1.159
1.139
1.115
1.101
Mogrifier (d2, MC)
24M
1.137
1.120
1.094
1.083
HCLM with Cache (Kawakami et al. 2017)
8M
1.591
1.538
LSTM (d1) (Kawakami et al. 2017)
SM
1.793
1.736
MWC
EN
LSTM (d2)
24M
1.353
1.338
1.239
1.225
Mogrifier (d2)
24M
1.319
1.305
1.202
1.188
LSTM (d2, MC)
24M
1.346
1.332
1.238
NaN
Mogrifier (d2, MC)
24M
1.312
1.298
1.200
1.187
HCLM with Cache (Kawakami et al. 2017)
8M
1.754
1.711
LSTM (d1) (Kawakami et al. 2017)
8M
1.943
1.913
MWC
LSTM (d2)
24M
1.382
1.367
1.249
1.237
FI
Mogrifier (d2)
24M
1.338
1.326
1.202
1.191
LSTM (d2, MC)
24M
1.377
1.361
1.247
1.234
Mogrifier (d2, MC)
24M
1.327
1,313
1.198
NaN
Transformer-Xe XL (d24) (Dai et al. 2019)
277M
0.993
0.940
Transformer-Xe (d18) (Dai al. 2019)
88M
1.03
LSTM (d4)
96M
1.145
1.155
1.041
1.020
Mogrifier (d4)
96M
1.110
1.122
1.009
0.988
LSTM (d4, MC)
96M
1.139
1.147
Enwik8
Mogrifier (d4, MC)
96M
1.104
1.116
EN
Transformer-Xt XL (d12) (Dai al. 2019)
41M
1.06
1.01
AWD-LSTM (d3) (Merity et al. 2017)
47M
1.232
mLSTM (d1) (Krause et al. 2016)
46M
1.24
1.08
LSTM (d4)
48M
1.182
1.195
1.073
1.051
Mogrifier (d4)
48M
1.135
1.146
1.035
1.012
LSTM (d4, MC)
48M
1.176
1.188
Mogrifier (d4, MC)
48M
1.130
1.140
et al. 2017) and FRAGE (Gong et al. 2018). The Mogrifier improves the state of the art without either of these methods PTB, and without FRAGE Wikitext-2.
Table lists the character-level modelling results. On all datasets, our baseline LSTM results are much better than those previously reported for LSTMs. highlighting the issue of scalability and experimental controls. In some cases, these unexpectedly large gaps may be down to lack of hyperparameter tuning as in the case Merity et (2017), or others, to using BPTT window size (50) that too small for character-level modelling (Melis et al. 2017) in order to fit the model into memory. The Mogrifier further improves on these baselines by a considerable margin. Even the smallest improvement of 0.012 bpc the highly idiosyncratic, character-basedd Mikolov preprocessed PTB task is equivalent to gaining about 3 perplexity points on word-level PTB. MWC, which was built for open vocabulayy language modelling, is much better smaller-sca scale Character-leve dataset. On the English and the Finnish corpora in MWC, the Mogrifier enjoys a gap of 0.033-0.046 bpc. Finally, on the Enwik8 dataset, the gap is 0009-0.039 bpc in favour of the Mogrifier.
5

---

Published as a conference paper at ICLR 2020
h
h2
h4
ESTM
x-1
x
x3
x5
Figure 2: "No-zigzag" Mogrifier for the ablation study. Gating is always based on the original inputs.
Table PTB ablation study validation perplexities with 24M parameters.
57
56
Mogrifier
54.1
55
Full rank Q', P'
54.6
54
No zigzag
55.0
2
3
4
6
LSTM
57.5
mLSTM
57.8
Figure Perplexity the rounds the PTB ablation study.
Of particular note the comparison to Transformer-Xi XL (Dai al. 2019), state-of-the-ar model on larger datasets such as Wikitext-103 and Enwik8. On PTB, without dynamic evaluation, the Transformer-Xe on par with our LSTM baseline which puts about 3.5 perplexity points behind the Mogrifier On Enwik8, also without dynamic evaluation, the Transformer XL has large, 0.09 bpc advantage similar parameter budgets, but with dynamic evaluation this gap disappears. However, we did not test the Transformer-Xi XL. ourselves, so fair comparison is not possible due to differing experimental setups and the rather sparse result matrix for the Transformer-2
ANALYSIS
4.1
ABLATION STUDY
The Mogrifier consistently outperformed the LSTM in our experiments. The optimal settings were similar across all datasets, with r E (5, 6) and & E [40... 90) (see Appendix for discussion of hyperparameter sensitivity). In this section, we explore the effect of these hyperparameters and show that the proposed model is not unnecessarily complicated. To save computation, we tune all models using shortened schedule with only 145 epochs instead of 964 and a truncated BPTT window size of 35 on the word-level PTB dataset, and evaluate using the standard, deterministic dropout approximation with tuned softmax temperature.
Fig. 3 shows that the number rounds greatly influences the results. Second, we found the low-rank factorization of Q' and R' help bit, but the full-rank variant is close behind which what observed on other datasets, as well. Finally, to verify that the alternating gating scheme is not overly complicated, condition all newly introduced gates the original inputs and hprev (see Fig. 2). That is, instead of Eq. I and Eq. 2 the no-zigzag updates are
x' 20 ) 2 2-2
for odd i E [l...r].
h prev 2x (R'x)) h-2
for even i [1...r].
In experiments, the no-zigzag variant underperformed the baseline Mogrifier by small but significant margin, and was on par with the r = model Fig. suggesting that the Mogrifier's iterative refinement scheme does than simply widen the range of possible gating values of and per to (0, 2[/22) and (0, 21r/21), respectively.
4.2 COMPARISON TO THE MLSTM
The Multiplicative LSTM (Krause et al. 2016), or mLSTM for short, is closest to our model in the literature. is defined as mLSTM(#, Cprev hp LSTM Cpres hm where h

---

Published as a conference paper at ICLR 2020
LSTM Mogrifier
LSTM Mogrifier
1.5
1
0.5
50
100
150
200
50
100
150
200
10M model parameters with vocabulary size lk.
(b) 24M model parameters with vocabulary size 10k.
Cross-entropy vs sequence length the reverse copy task with i.i.d. tokens. Lower better. The Mogrifier is better than the LSTM even in this synthetic task with no resemblance to natural language.
Figure 4:
(W")=x) O (Wihh prev) In this formulation, the differences are readily apparent. First, the mLSTM allows for multiplicative interaction between and hpree but it only overrides hpev. while in the Mogrifier the interaction is two-way, which as the ablation study showed - is important. Second, the mLSTM can change not only the magnitude but also the sign of values prev. something with which we experimented in the Mogrifier, but could not get to work. Furthermore, in the definition of h the unsquashed linearities and their elementwise product make the mLSTM more sensitive to initialization and unstable during optimization.
On the Enwik8 dataset, we greatly improved on the published results the mLSTM (Krause et al. 2016). In fact, experiments on PTB based on our reimplementation the mL.TMM following the same methodology as the ablation study and found that the mLSTM did not improve on the LSTM (see Table 3).
LSTM baseline outperformed the mLSTM by 0.03 bpc. We also conducted
Krause et al. (2016) posit and verify the recovery hypothesis which says that having just suffered large loss, the loss the next time step will be smaller on average for the mLSTM than for the LSTM. This was found not to be the case for the Mogrifier. Neither did we observe a significant change the gap between the LSTM and the Mogrifier the tied and untied embeddings settings, which would be expected if recovery was affected by and hpree being in different domains.
4.3 THE REVERSE COPY TASK
Our original motivation for the Mogrifier was to allow the context to amplify salient and attenuate nuisance features in the input embeddings. We conduct a simple experiment to support this point of view. Consider the reverse copy task where the network reads input sequence of tokens and a marker token after which it has to repeat the input in reverse order. this simple sequence-to sequence learning (Sutskever et al. 2014) setup, the reversal is intended avoid the minimal time lag problem (Hochreiter and Schmidhuber 1997), which is not our focus here.
The experimental setup follows. For the training set, generate 500 examples by uniformly sampling a given number of tokens from a vocabulary of size 1000. The validation and test sets constructed similarly, and contain 10000 examples. The model consists of an independent, unidirectional encoder and decoder, whose total number of parameters is 10 million. The decoder initialized from the last state of the encoder. Since overfitting not an issue here, no dropout is necessary, and we only tune the learning rate, the 12 penalty, and the embedding size for the LSTM. For the Mogrifier, the number of rounds r and the rank it of the low-rank approximation are also tuned.
We compare the case where both the encoder and decoder are L.TTss where both are Mogrifiers. Fig. 4a shows that, for sequences of length 50 and 100, both models can solve the task perfectly. At higher lengths though, the Mogrifier has a considerable advantage. Examining the best hyperparameter settings found, the embedding/hiddes sizes for the LSTM and Mogrifier are 498/787 vs 41/1054 at 150 steps, and 493/790 vs 181/961 at 200 steps. Clearly, the Mogrifier was able to work with a much smaller embedding size than the LSTM. which is in line with our expectations for model with more flexible interaction between the input and recurrent state. We also conducted experiments with larger model and vocabulary size, and found the effect even pronounced (see Fig. 4b).
7

---

Published as a conference paper at ICLR 2020
4.4 WHAT THE MOGRIFIER IS NOT
The results on the reverse copy task support our hypothesis that input embeddings are enriched by the Mogrifier architecture, but that cannot be the full explanation as the results the ablation study indicate. In the following, consider number of hypotheses about where the advantage of the Mogrifier lies and the experiments that provide evidence against them.
1 Hypothesis: the benefit is in scaling æ and hprev. We verified that data dependency is a crucial feature by adding learnable scaling factor the LSTM inputs. We observed no improvement. Also, at extremely low-rank (less than 5) settings where the amount of information in its gating is small, the Mogrifier loses its advantage.
f Hypothesis: the benefit is in making optimization easier. We performed experiments with different optimizers (SGD, RMSProp), with intra-layer batch normalization and layer normalization on the LSTM gates. While we cannot rule out an effect optimization difficulty, in all of these experiments the gap between the LSTM and the Mogrifier was the same.
f Hypothesis: exact rying of embeddings is too constraining, the benefit is in making this rela- tionhhi less strict. Experiments conducted with untied embeddings and character-based models demonstrate improvements of similar magnitude.
Hypothesis: the benefit is in the low-rank factorization of Q' RR implicitly imposing structure on the LSTM weight matrices. We observed that the full-rank Mogrifier also performed better than the plain LSTM. We conducted additional experiments where the LSTM's gate matrices were factorized and observed no improvement.
Hypothesis: the benefit comes fromsfrom from better performance on rare words. The observed advantage on character-based modelling harder to explain based on frequency. Also, in the reverse copy experiments, large number of tokens were sampled uniformly, so there were no rare words at all.
f Hypothesis: the benefit is specific to the English language. This is directly contradicted by the Finnish MWC and the reverse copy experiments.
: Hypothesis: the benefit is in handling long-range dependencies better. Experiments in the episodic setting (i.e. sentence-level language modelling) exhibited the same gap the non-episodic ones.
1 Hypothesis: the scaling up of imputs saturates the downstream LSTM gates. The idea here is that saturated gates may make states more stable over time. We observed the opposite: the means of the standard LSTM gates in the Mogrifier were very close between the two models, but their variance was smaller in the Mogrifier.
CONCLUSIONS AND FUTURE WORK
We presented the Mogrifier LSTM, an extension to the LSTM, with state-of-the-art results on several language modelling tasks. Our original motivation for this work was that the context-free representation of input tokens may be a bottleneck in language models and by conditioning the input embedding on the recurrent state some benefit was indeed derived. While may be part the explanation, this interpretation clearly does not account for the improvements brought by conditioning the recurrent state on the input and especially the applicability to character-level datasets. Positioning our work on the Multiplicative RNN line of research offers a more compelling perspective.
To give more credence this interpretation, in the analysis we highlighted number of possible alternative explanations, and ruled them all out to varying degrees. In particular, the connection to the MLT is weaker than expected as the Mogrifier does not exhibit improved recovery (see Section 4.2), and PTB the mLSTM works only well as the LSTM. At the same time, the evidence against easier optimization is weak, and the Mogrifier establishing some kind of sharing between otherwise independent LSTM weight matrices is distinct possibility.
Finally, note that as shown by Fig. and Eq. 1-2, the Mogrifier is a series of preprocessing steps composed with the LSTM function, but other architectures, such Mogrifier GRU or Mogrifier Elman Network are possible. We also leave investigations into other forms of parameterization of context-dependent transitions for future work.
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We would like thank Ben Krause for the Transformer-XX XL dynamic evaluation results, Laura Rimell, Aida Nematzadeh, Angeliki Lazaridou, Karl Moritz Hermann, Daniel Fried for helping with experiments, Chris Dyer, Sebastian Ruder and Jack Rae for their valuable feedback.
REFERENCES
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. arXiv preprint AXKv::606.003550 2016.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pages 1120-1128, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint 140.::::0000440 2014.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv preprint arXiv:1810.06660 2018.
Bram Bakker.
Reinforcement learning with long short-term memory. In Advances in neural information processing systems, pages 1475-1482, 2002.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. arXiv preprint 11Xiv:1711.02113. 2017.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In International Conference Machine Learning, pages 2067-2075, 2015.
Zihang Dai, Zhilin Yang. Yiming Yang. William W Cohen, Jaime Carbonell, Quoc V and Ruslan Salakhutdinov. Transformer-xi Attentive language models beyond fixed-length context. arXiv preprint arXiv: 028600, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- treitrainin of deep bidirectional transformers for language understanding, arXiv preprint 1rX.v:1810004805, 2018.
Jeffrey I. Elman. Finding structure in time. Cognitive science, 14(2):179-21) 1990.
Jakob N Foerster, Justin Gilmer, Jascha Sohl-Dicksteen, Jan Chorowski, and David Sussillo. Input switched affine networks: An architecture designed for interpretability. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1136-1145 115.. JMLR. org, 2017.
Yarin Gal and Zoubin Ghahramani. theoretically grounded application of dropout recurrent neural networks. In Advances in Neurul Information Processing Systems, pages 1019-1027, 2016.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1487- 1495. ACM, 2017.
Chengyue Gong, Di He, Xu Tan, Tao Qin. Liwei Wang, and Tie-Yan Liu. Frage: frequency-agnostic word representation. In Advances in Neural Information Processing Systems, pages 1334-1345. 2018.
David Ha, Andrew Dai, and Quoe Le. Hypernetoors.. arXiv preprint arXiv:1609.09106 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Lstm can solve hard long time lag problems. In Advances in neural information processing systems, pages 473-479, 1997.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: 06146, 2018
Marcus Hutter. The human knowledge compression contest. URL http://rizee lutterl. net, 6, 2012.
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. CoRR, abs/1611.014 1.01462, 2016. URL http: http://axxvvvv//e abs//bbs///// /161.. 01 462.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv: 1804:0800000053 2018.

---

Published as a conference paper at ICLR 2020
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint 177Ivv1707.07328, 2017.
Kazuya Kawakami. Chris Dyet, and Phil Blunsom. Learning create and words in open open-vocabulary neural language modeling, arXiv preprint arXiv:1704.0690 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Ben Krause, Liang Lu, lain Murray, and Steve Renals. Multiplicative L.TM for sequence modelling. CoRR, abs/1609.07959, 2016. URL http:/ /ariiv...vv. www./bsgg//a.oooooooooooo.. 1.99..0.000 07959.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation neural sequence models. arXiv preprint urXiv:170.004332, 2017.
Ben Krause, Emmanuel Kahembwe, Jain Murray, and Steve Renals. Dynamic evaluation transformer language models, arXiv preprint 1701v:194.00837 2019.
Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. Lstms can leam syntax-sensiin sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426- 1436. 2018.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521 5355, 2016.
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The Penn treebank. Computational linguisticss 19())313-330, 1993.
Hermann Mayer, Faustino Gomez, Daan Wierstra, Istvan Nagy, Alois Knoll, and Jürgen Schmidhuher. A system for robotic heart surgery that learns to tie knots using recurrent neural networks. Advanced Robotics, 22 (13-14):1521-1537, 2008
Gábor Mclis, Chris Dyer, and Phil Blunsom. On the state of the of evaluation in neural language models. arXiv preprint vrXiv:/707.055 700.00589, 2017.
Gábor Melis, Charles Blundell, Tomáš Kočiský, Karl Moritz Hermann, Chris Dyer, and Phil Blunsom. Pushing the bounds of dropout. arXiv preprint / 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, (bbs/1099.08843 2016. URL http: / arwii. org/ abss 00op//105 1669666.... 07843.
Stephen Merity, Nitish Shirish Keskar, Richard Socher. Regularizing and optimizing Istm language models. arXiv preprint arXiv: 7088 02182, 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv: 1803 08240, 2018.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, page 3, 2010.
Nafise Sadat Moosavi and Michael Strube. Lexical features in coreference resolution: To be used with caution. arXiv preprint arXiv: hto::/704.000 7004006779 2017.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1311 1318, 2013.
Matthew E Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1XXi::8802.03666 2018.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. CoRR, abs/1608.05857 2016. URL httt:://aa//t //wp///aavvv///////////// orgg aaaaaba/a/ 608. 05859.
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning representations by back-propagating errors. Cognitive modeling, 5())11, 1988.
Hasim Sak. Andrew W. Senior. and Françoise Beaufays. Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition. CoRR, abs/1402. 1228, 2014. URL http: / //axxivv. www.oorg/bb... abs/ 402.11 128.
10

---

Published as a conference paper at ICLR 2020
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of words with subword units. arXiv preprint arXiv:1508.07909 2015.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference Machine Learning (ICML-11), pages 1017-1024, 2011.
Ily Sutskever, Oriol Vinyals, and Quoc V Le. Sequence sequence learning with neural networks. In Advances neural information processing systems, pages 3104 3112, 2014.
Chong Wang. Yining Wang. Po-Sen Huang, Abdelrahman Mohamed. Dengyong Zhou, and Li Deng. Sequence modeling segmentations. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3674-368 JMLR, org, 2017.
Paul Werbos Backpropagation through time: what does and how to Proceedings of the IEEE, 78 (10):1550-150) 1990.
Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan R Salakhutdinov. multiplicative integration with recurrent neural networks. In Advances in neural information processing systems, pages 2856-2864, 2016.
Zhilin Yang. Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: a high-rank ran language model. arXiv preprint arXiv: 1.09533, 2017.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. CoRR, abs/ 1611.01578 2016. URL http:///p///.. / wwxiv.owooooooooooooeeaooooooo httpa/asss//.111.11

---

Published as a conference paper at ICLR 2020
APPENDIX A HYPERPARAMETER TUNING RANGES
In all experiments, we tuned hyperparameters using Google Vizier (Golovin et al. 2017). The tuning ranges are listed Table 4. Obviously, mogrifier_ rooendss and mogrifier rrank tuned only for the Mogrifier. If seiiuud_emddddinggeeeee sizes are set to equal and the linear projection from the cell output into the output embeddings space is omitted. Similarly, mogri fier_rank & 0 is taken to mean full rank Q", R without factorization. Since Enwik8 is much larger dataset, we don't it tune weweeeeeeeeeneeeeee _ratio and specify tighter tuning ranges for dropout based on preliminary experiments (see Table 5).
> 1, then the input/output embedding sizes and the hidden
Dynamic evaluation hyperparameters were tuned according to Table The highest possible value for max time_tmmeeeee time_steps, the BPTT window size, was 20 for word, and 50 for character-level tasks. The batch size for estimating the mean squared gradients over the training data was set to 1024, gradient clipping was turned off, and the 12 penalty was set to zero.
Table 4: Hyperparameter tuning ranges for all tasks except Enwik8.
Low
High 0.004
Spacing
learning_rai rate
0.001
log
input_embart gouu_eebedding ratio
0.0
2.0
12. 12_penalty
5e-6
le-3
log
input dropout inter layer dropout
0.9
0.0
0.95
state_dropout
0.0
0.8
0.0
0.95
mogrifier_roune rounds
6
mogrifier rank (k:)
Table Hyperparameter tuning ranges for Enwik8.
Low
High
Spacing
learning 0.001
0.004
log
12 2_peenalt
5c-6
lc-3
log
0.0
aier_layer_dropour
0.0
0.2
state state_droppe dropout
0.0
0.25
output dropout
0.0
0.25
mogrifier rouuds (r)
mogrifier_rank (k)
-20
100
Table Hyperparameter tuning ranges dynamic evaluation.
Low
High
Spacing
max_time max_time_stepp
20/50
Leneval_lcaring rate
lc-6
lc-3
log
dyneval decay rate
le-6
le-2
log
dyneval_epsine
le-8
le-2
log
12

---

Published as a conference paper at ICLR 2020
APPENDIX B HYPERPARAMETER SENSITIVITY
The parallel coordinate plots in Fig. 5 and 6, give a rough idea about hyperparameter sensitivity. The red lines correspond to hyperparameter combinations closest to the best solution found. To find the closest combinations, we restricted the range for each hyperparameter separately to about 15% of its entire tuning range.
For both the LSTM and the Mogrifier, the results are at most 1.2 perplexity points off the best result, so our results are somewhat insensitive to jitter in the hyperparameters. Still, in this setup, grid search would require orders of magnitude more trials to find comparable solutions.
On the other hand, the tuner does take advantage of the stochasticity of training, and repeated runs with the same parameters may be give slightly worse results. To gauge the extent of this effect, on PTB we estimated the standard deviation in reruns of the LSTM with the best hyperparameters to be about 0.2 perplexity points, but the mean was about 0.7 perplexity points off the result produced with the weights saved in best tuning run.
objectiveValuee
dropot
inter_ Swwr__lyerrrcoo dropout
12 P_penalty
learning rale
output_ dropout
state dropout 0.8
0.00100
0.0040
4.18-
0.9
0.0038
0.80
0.00090
0.0036
4.10
0.75
0,00080
0.0034
4.14
0.7
0.00070
0.0039
0.30
0,0833
4.12
0.00060
000030
0.65
0.5
0.5
0:0028
0.00050
0.60
4.10
0.0024
0.4
0.00040
4.08
0.00200
0.3
0100030
0.0018
0.50
4.06
0.2
0100200
0.0016
0.2
0.0014
0.45
4.04
0.00010
0.0012
0.1
0.40
0.0
Figure 5: Average word validation cross- eotsspentropies for hyperparameter combinations in the neighbourhood of the best solution for a 2-layer I.S with 24M weights on the Penn Treebank dataset.
objectiveValue
feature mask_ rankleature mask rounds input, dropout
inter_layee Laee__lyerrrrcm www._ayyerddoouut
panattt
leaming_rak rate
output, dropout
state dropout
100
0.00100
0.0040
0.75
4.10
5.5
0.85
0.8
0.00090
0.0038
0.70
0.0066
0.80
4.08
0.7
0. 0.00080
0 0.0004
0.85
0030
0.75
0.00
0.00070
4.06
60
40
00000
0:5
35
0.0028
4.04
30
77
0. 0.00000
0 0006
0.4
4.02
30
25
0.00040
40
20
0.5
3.0090
0.35
4.00
10
1.5
0.60
00018
0.50
0.30
0
1.0
0.2
000g0
0.0016
0.45
3.98
0.55
0.0014
0.1
O 0.00010
0.0012
0.40
3.98
-20
0.0
0.50
0.0010
0.15
Figure 6: Average por-word validation cross-entropies for hyperparameter combinations in the neighbour- hood of the best solution for a 2-layer Mogrifier LSTM with 24M weights on the Penn Treebank dataset. feature_ mask_ rank and saatrremassko_o_e are aliases for mogrifier_rane and mogrifier_roundd
13


Reviews:
[{"review_id": "rJe0f83x9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have read the authors' response. Their points regarding baseline comparisons are sensible in that there isn't a reason to expect the observations to *not* generalization to other datasets. It is odd that mLSTM is outperformed by LSTM in Table 3, but as the authors note in section 4.2 this may be due to instability of mLSTM during training. The results in the paper demonstrate significant improvement over LSTM, and while there are not as many baseline comparison to similar models as I would have liked to see, the quality of this work is sufficiently high that this is not a fatal flaw. In light of the author response and other reviews, I am revising my rating to 6: Weak Accept.\n\n=====\n\nThis paper proposes a modification of LSTM networks in the context of language modeling called Mogrifier LSTM. Ordinary LSTMs are defined as recurrent operations on the current input, previous hidden state, and previous cell state. The proposed Mogrifier LSTM utilizes the same recurrent unit as the LSTM, but the input and previous hidden state are updated with several rounds of mutual gating. In each round, the input  is multiplied elementwise by a gate computed as a function of the hidden state (or vice versa). The authors experiment on word-level and character-level modeling and compare their Mogrifier LSTMs to several state-of-the-art approaches. They also conduct an ablation study to show the effect of various design choices and hyperparameters and experiments on a reverse copy task.\n\nSpecific contributions include:\n* Proposal of a novel approach for modulating inputs to a recurrent unit by mutual gating.\n* Experiments demonstrating strong performance on a number of language modeling tasks.\n  \nThe paper in its current state is borderline, leaning towards weak reject. Points in favor of acceptance include the high clarity of writing, good experiments of the proposed model, and a discussion of possible reasons for why the mogrification operation works well. The main shortcoming of the paper is experimental comparison to baselines.\n\nThe authors were able to train baseline LSTMs to high levels of performance (presumably due to tuning of hyperparameters) and then demonstrate that Mogrifier LSTMs improve upon LSTMs significantly. This is perhaps not entirely surprising, because the hyperparameter range of the Mogrifier LSTM includes zero rounds of updates, which would render it identical to the baseline LSTM. Therefore, if the hyperparameters are tuned sufficiently well, the performance of the Mogrifier LSTM should be at least as good as the LSTM. What the experiments do not show is that the proposed mogrification outperforms other forms of multiplicative interaction and/or gating. The closest that the authors come to this is the single validation perplexity of the Multiplicative LSTM in Table 3. If thorough hyperparameter tuning is applied to the Multiplicative LSTM or the approaches of Wu et al. (2016) and/or Sutskever et al. (2011), does the Mogrifier LSTM still outperform them?\n\nOther than this critical issue of baseline comparison, the experiments are quite informative. The ablation study showing the effect of different design decisions and the hyperparameter visualiztion in Appendix B are particularly useful. The mogrification operation is described precisely enough for other researchers to implement and the arguments made in 4.4 are compelling.\n\nQuestion for the authors:\n* Some qualitative analysis of the learned mogrification operation would be helpful for understanding the nature of the modulation. For example, how do the predictions change depending on the modulation? If x is modulated by different hidden states, is there a noticeable effect on the output?\n* Did you experiment with other forms of modulation before arriving upon the mogrification formulation? There are some naive approaches such as concatenating the hidden state to the input and applying a nonlinear layer, or predicting affine parameters for the input as a function of the hidden state in the style of FiLM [1]. Are there obvious shortcomings in these naive approaches that mogrification handles gracefully?\n\n[1] Perez, E., Strub, F., De Vries, H., Dumoulin, V. and Courville, A., 2018, April. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklaAK10FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper tackles the problem of context modelling within recurrent neural networks (RNNs). The authors propose an interdependent gating mechanism that enriches the coupling between inputs and hidden states. For an input x_0 and hidden state h_0; h_0 gates x_0 to create x_1; x_1 then gates h_0 to create h_1; this cyclical gating operation is applied for several rounds and it's output is fed into a recurrent neural network. For the next time-step, this process is repeated, with h_0 as the final h obtained after the final round of gating in the previous time-step. This results in the RNN processing a more contextualized version of the input tokens x.\n\nMain Contributions:\n1. A simple pre-processing step that contextualizes inputs for recurrent neural networks and significantly improves performance.\n2. An extensive evaluation of the proposed technique against previous works and on all relevant datasets.\n\nPros:\nThe paper is very well-written and clear. It motivates and explores the questions and issues surrounding this topic very well.\n\nCons:\nIt would be good to see how this performance translates to other RNN architectures such as GRUs. \n\n\nFinal notes:\nThis paper raises many interesting question:\n- What is the really going on with the gating mechanism? \nThe authors explore this question but the jury is still out on exactly what is going on here.\n- \"Mogrification\" as a general preprocessing step: could it also improve performance for transformer models?\n- Are there better ways to preprocess and gate the RNN inputs?\n\n--------\n\nReview Decision:\nIt is clear, well motivated, well written and represents a concrete contribution to the language modelling literature. Furthermore, most claims made are substantiated via thorough experimentation. Lastly, this work demonstrates that rather than relying on data and model scaling to improve performance; there is alot left to be done in tackling language modelling on smaller scale datasets. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BylUf0l6YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a novel LSTM architecture that adds several gating mechanism that gates the hidden state and inputs in between the LSTM update. The proposed model shows superior performance on smallish datasets including PTB,  Enwick8 and NWC.\n\nComments on the paper:\n\n1. The paper proposes an interesting architecture and it seems to show significant improvement in terms of performance for some language datasets. \n\n2. The paper is very well written, the motivation and formulation is clear. There are many analysis to understand the model (the strength and weaknesses).\n\n3.. One thing is that since this could take into account more context,  it seems that this model could potentially generate language / tokens with longer time dependencies. I wonder if the authors have performed any experiments on this and if they have seen any improvements on that front.\n\n4. Also, I am curious about the generalization ability of the model. Could the authors train the model on shorter sequences and test for generation with longer sequences and see how this compares with baseline models.\n\n5. The model seems to be related to Adaptive Computation Time (ACT) from Gaves et al. it would be nice to compare to the ACT.\n\n6. Another slight improvement in writing could be to hightlight the intuition (conclusion in page 8) at the beginning of the paper, this could help in better understanding the motivation of the paper.\n\n\nMinor comments on the paper,\n\n1. The link and the self-citations on page 4 are does not seem to be valid links and citations.\n\nOverall, a well-written paper, extensive analysis and good experimental result.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new twist on the typical LSTM that applies several rounds of gating on the history and input, with the end result that the LSTM's transition function is effectively context-dependent. The performance of the model is illustrated on several datasets.

In general, the reviews were positive, with one score being upgraded during the rebuttal period. One of the reviewers complained that the baselines were not adequate, but in the end conceded that the results were still worthy of publication.

One reviewer argued very hard for the acceptance of this paper "Papers that are as clear and informative as this one are few and far between. ... As such, I vehemently argue in favor of this paper being accepted to ICLR."
Decision: Accept (Talk)