Mogrifier LSTM

1 INTRODUCTION . The domination of Natural Language Processing by neural models is hampered only by their limited ability to generalize and questionable sample complexity ( Belinkov and Bisk 2017 ; Jia and Liang 2017 ; Iyyer et al . 2018 ; Moosavi and Strube 2017 ; Agrawal et al . 2016 ) , their poor grasp of grammar ( Linzen et al . 2016 ; Kuncoro et al . 2018 ) , and their inability to chunk input sequences into meaningful units ( Wang et al . 2017 ) . While direct attacks on the latter are possible , in this paper , we take a language-agnostic approach to improving Recurrent Neural Networks ( RNN , Rumelhart et al . ( 1988 ) ) , which brought about many advances in tasks such as language modelling , semantic parsing , machine translation , with no shortage of non-NLP applications either ( Bakker 2002 ; Mayer et al . 2008 ) . Many neural models are built from RNNs including the sequence-to-sequence family ( Sutskever et al . 2014 ) and its attention-based branch ( Bahdanau et al . 2014 ) . Thus , innovations in RNN architecture tend to have a trickle-down effect from language modelling , where evaluation is often the easiest and data the most readily available , to many other tasks , a trend greatly strengthened by ULMFiT ( Howard and Ruder 2018 ) , ELMo ( Peters et al . 2018 ) and BERT ( Devlin et al . 2018 ) , which promote language models from architectural blueprints to pretrained building blocks . To improve the generalization ability of language models , we propose an extension to the LSTM ( Hochreiter and Schmidhuber 1997 ) , where the LSTM ’ s input x is gated conditioned on the output of the previous step hprev . Next , the gated input is used in a similar manner to gate the output of the previous time step . After a couple of rounds of this mutual gating , the last updated x and hprev are fed to an LSTM . By introducing these additional of gating operations , in one sense , our model joins the long list of recurrent architectures with gating structures of varying complexity which followed the invention of Elman Networks ( Elman 1990 ) . Examples include the LSTM , the GRU ( Chung et al . 2015 ) , and even designs by Neural Architecture Search ( Zoph and Le 2016 ) . Intuitively , in the lowermost layer , the first gating step scales the input embedding ( itself a representation of the average context in which the token occurs ) depending on the actual context , resulting in a contextualized representation of the input . While intuitive , as Section 4 shows , this interpretation can not account for all the observed phenomena . In a more encompassing view , our model can be seen as enriching the mostly additive dynamics of recurrent transitions placing it in the company of the Input Switched Affine Network ( Foerster et al . 2017 ) with a separate transition matrix for each possible input , and the Multiplicative RNN ( Sutskever et al . 2011 ) , which factorizes the three-way tensor of stacked transition matrices . Also following this line of research are the Multiplicative Integration LSTM ( Wu et al . 2016 ) and – closest to our model in the literature – the Multiplicative LSTM ( Krause et al . 2016 ) . The results in Section 3.4 demonstrate the utility of our approach , which consistently improves on the LSTM and establishes a new state of the art on all but the largest dataset , Enwik8 , where we match similarly sized transformer models . 2 MODEL . To allow for ease of subsequent extension , we present the standard LSTM update ( Sak et al . 2014 ) with input and state of size m and n respectively as the following function : LSTM : Rm × Rn × Rn → Rn × Rn LSTM ( x , cprev , hprev ) = ( c , h ) . The updated state c and the output h are computed as follows : f = σ ( Wfxx+Wfhhprev + b f ) i = σ ( Wixx+Wihhprev + b i ) j = tanh ( Wjxx+Wjhhprev + b j ) o = σ ( Woxx+Wohhprev + b o ) c = f cprev + i j h = o tanh ( c ) , where σ is the logistic sigmoid function , is the elementwise product , W∗∗ and b∗ are weight matrices and biases . While the LSTM is typically presented as a solution to the vanishing gradients problem , its gate i can also be interpreted as scaling the rows of weight matrices Wj∗ ( ignoring the non-linearity in j ) . In this sense , the LSTM nudges Elman Networks towards context-dependent transitions and the extreme case of Input Switched Affine Networks . If we took another , larger step towards that extreme , we could end up with Hypernetworks ( Ha et al . 2016 ) . Here , instead , we take a more cautious step , and equip the LSTM with gates that scale the columns of all its weight matrices W∗∗ in a context-dependent manner . The scaling of the matrices W∗x ( those that transform the cell input ) makes the input embeddings dependent on the cell state , while the scaling of W∗h does the reverse . The Mogrifier1 LSTM is an LSTM where two inputs x and hprev modulate one another in an alternating fashion before the usual LSTM computation takes place ( see Fig . 1 ) . That is , Mogrify ( x , cprev , hprev ) = LSTM ( x ↑ , cprev , h ↑ prev ) where the modulated inputs x ↑ and h↑prev are defined as the highest indexed xi and hiprev , respectively , from the interleaved sequences xi = 2σ ( Qihi−1prev ) xi−2 , for odd i ∈ [ 1 . . . r ] ( 1 ) 1It ’ s like a transmogrifier2 without the magic : it can only shrink or expand objects . 2Transmogrify ( verb , 1650s ) : to completely alter the form of something in a surprising or magical manner . hiprev = 2σ ( R ixi−1 ) hi−2prev , for even i ∈ [ 1 . . . r ] ( 2 ) with x−1 = x and h0prev = hprev . The number of “ rounds ” , r ∈ N , is a hyperparameter ; r = 0 recovers the LSTM . Multiplication with the constant 2 ensures that randomly initialized Qi , Ri matrices result in transformations close to identity . To reduce the number of additional model parameters , we typically factorize the Qi , Ri matrices as products of low-rank matrices : Qi = QileftQ i right with Q i ∈ Rm×n , Qileft ∈ Rm×k , Qiright ∈ Rk×n , where k < min ( m , n ) is the rank . 3 EXPERIMENTS . 3.1 THE CASE FOR SMALL-SCALE . Before describing the details of the data , the experimental setup and the results , we take a short detour to motivate work on smaller-scale datasets . A recurring theme in the history of sequence models is that the problem of model design is intermingled with optimizability and scalability . Elman Networks are notoriously difficult to optimize , a property that ultimately gave birth to the idea of the LSTM , but also to more recent models such as the Unitary Evolution RNN ( Arjovsky et al . 2016 ) and fixes like gradient clipping ( Pascanu et al . 2013 ) . Still , it is far from clear – if we could optimize these models well – how different their biases would turn out to be . The non-separability of model and optimization is fairly evident in these cases . Scalability , on the other hand , is often optimized for indirectly . Given the limited ability of current models to generalize , we often compensate by throwing more data at the problem . To fit a larger dataset , model size must be increased . Thus the best performing models are evaluated based on their scalability3 . Today , scaling up still yields tangible gains on down-stream tasks , and language modelling data is abundant . However , we believe that simply scaling up will not solve the generalization problem and better models will be needed . Our hope is that by choosing small enough datasets , so that model size is no longer the limiting factor , we get a number of practical advantages : ? Generalization ability will be more clearly reflected in evaluations even without domain adaptation. ? Turnaround time in experiments will be reduced , and the freed up computational budget can be put to good use by controlling for nuisance factors. ? The transient effects of changing hardware performance characteristics are somewhat lessened . Thus , we develop , analyse and evaluate models primarily on small datasets . Evaluation on larger datasets is included to learn more about the models ’ scaling behaviour and because of its relevance for applications , but it is to be understood that these evaluations come with much larger error bars and provide more limited guidance for further research on better models . 3.2 DATASETS . We compare models on both word and character-level language modelling datasets . The two wordlevel datasets we picked are the Penn Treebank ( PTB ) corpus by Marcus et al . ( 1993 ) with preprocessing from Mikolov et al . ( 2010 ) and Wikitext-2 by Merity et al . ( 2016 ) , which is about twice the size of PTB with a larger vocabulary and lighter preprocessing . These datasets are definitely on the small side , but – and because of this – they are suitable for exploring different model biases . Their main shortcoming is the small vocabulary size , only in the tens of thousands , which makes them inappropriate for exploring the behaviour of the long tail . For that , open vocabulary language modelling and byte pair encoding ( Sennrich et al . 2015 ) would be an obvious choice . Still , our primary goal here is the comparison of the LSTM and Mogrifier architectures , thus we instead opt for character-based language modelling tasks , where vocabulary size is not an issue , the long tail is not truncated , and there are no additional hyperparameters as in byte pair encoding that make fair comparison harder . The first character-based corpus is Enwik8 from the Hutter Prize dataset ( Hutter 2012 ) . Following common practice , we use the first 90 million characters for training and the remaining 10 million evenly split between validation and test . The character-level task on the 3Note that the focus on scalability is not a problem per se . Indeed the unsupervised pretraining methods ( Peters et al . 2018 ; Devlin et al . 2018 ) take great advantage of this approach . Mikolov preprocessed PTB corpus ( Merity et al . 2018 ) is unique in that it has the disadvantages of closed vocabulary without the advantages of word-level modelling , but we include it for comparison to previous work . The final character-level dataset is the Multilingual Wikipedia Corpus ( MWC , Kawakami et al . ( 2017 ) ) , from which we focus on the English and Finnish language subdatasets in the single text , large setting . 3.3 SETUP . We tune hyperparameters following the experimental setup of Melis et al . ( 2018 ) using a black-box hyperparameter tuner based on batched Gaussian Process Bandits ( Golovin et al . 2017 ) . For the LSTM , the tuned hyperparameters are the same : input_embedding_ratio , learning_rate , l2_penalty , input_dropout , inter_layer_dropout , state_dropout , output_dropout . For the Mogrifier , the number of rounds r and the rank k of the low-rank approximation is also tuned ( allowing for full rank , too ) . For word-level tasks , BPTT ( Werbos et al . 1990 ) window size is set to 70 and batch size to 64 . For character-level tasks , BPTT window size is set to 150 and batch size to 128 except for Enwik8 where the window size is 500 . Input and output embeddings are tied for word-level tasks following Inan et al . ( 2016 ) and Press and Wolf ( 2016 ) . Optimization is performed with Adam ( Kingma and Ba 2014 ) with β1 = 0 , a setting that resembles RMSProp without momentum . Gradients are clipped ( Pascanu et al . 2013 ) to norm 10 . We switch to averaging weights similarly to Merity et al . ( 2017 ) after a certain number of checkpoints with no improvement in validation cross-entropy or at 80 % of the training time at the latest . We found no benefit to using two-step finetuning . Model evaluation is performed with the standard , deterministic dropout approximation or MonteCarlo averaging ( Gal and Ghahramani 2016 ) where explicitly noted ( MC ) . In standard dropout evaluation , dropout is turned off while in MC dropout predictions are averaged over randomly sampled dropout masks ( 200 in our experiments ) . Optimal softmax temperature is determined on the validation set , and in the MC case dropout rates are scaled ( Melis et al . 2018 ) . Finally , we report results with and without dynamic evaluation ( Krause et al . 2017 ) . Hyperparameters for dynamic evaluation are tuned using the same method ( see Appendix A for details ) . We make the code and the tuner output available at https : //github.com/deepmind/lamb .

Summary: This paper tackles the problem of context modelling within recurrent neural networks (RNNs). The authors propose an interdependent gating mechanism that enriches the coupling between inputs and hidden states. For an input x_0 and hidden state h_0; h_0 gates x_0 to create x_1; x_1 then gates h_0 to create h_1; this cyclical gating operation is applied for several rounds and it's output is fed into a recurrent neural network. For the next time-step, this process is repeated, with h_0 as the final h obtained after the final round of gating in the previous time-step. This results in the RNN processing a more contextualized version of the input tokens x.