The Importance of the Current Input in Sequence Modeling

1 INTRODUCTION . Deep learning models constitute the current state of the art in most artificial intelligence applications , from computer vision to robotics or medicine . When dealing with sequential data , Recurrent Neural Networks ( RNNs ) , specially those architectures with gating mechanisms such as the LSTM ( Hochreiter & Schmidhuber , 1997 ) , the GRU ( Cho et al. , 2014 ) and other variants , are usually the default choice . One of the most interesting applications of RNNs is related to the field of Natural Language Processing , where most tasks , such as machine translation , document summarization or language modeling , involve the manipulation of sequences of textual data . Of these , language modeling has been extensively used to test different innovations in recurrent architectures , mainly due to the ease of obtaining very large datasets that can be used to train neural networks with millions of parameters . Sequence modeling consists of predicting the next element in a sequence given the past history . In language modeling , the sequence is a text , and hence the task is to predict the next word or the next character . In this context , some of the best performing architectures include the Mogrifier LSTM ( Melis et al. , 2020 ) and different variations of the AWD-LSTM ( Merity et al. , 2018 ) , usually combined with dynamic evaluation and mixture of sofmaxes ( MoS ) ( Wang et al. , 2019 ; Gong et al. , 2018 ) . These models obtain the best state-of-the-art performance with moderate size datasets , such as the Penn Treebank ( Mikolov et al. , 2010 ) or the Wikitext-2 ( Merity et al. , 2017 ) corpora , when no additional data are used during training . When larger datasets are considered , or when external data are used to pre-train the networks , attention-based architectures usually outperform other models ( Radford et al. , 2019 ; Brown et al. , 2020 ) . In this work we use moderate-scale language modeling datasets to explore the effect of a mechanism recently proposed by Oliva & Lago-Fernández ( 2021 ) , when combined with different LSTM-based models in the language modeling context . The idea consists of modifying a recurrent architecture by introducing a direct connection between the input and the output of the recurrent module . This has been shown to improve both the model ’ s generalization results and its readability in simple tasks related to the recognition of regular languages . In a standard RNN , the output depends only on the network ’ s hidden state , ht , which in turn depends on both the input , xt , and the recent past , ht−1 . But there is no explicit dependence of the network ’ s output on its input . In some cases this could be a shortcoming , since the transformation of xt needed to compute the network ’ s internal state is not necessarily the most appropriate to compute the output . However , an explicit dependence of the output on xt can be forced by adding a dual connection that skips the recurrent layers . We claim that this strategy may be of general application in RNN models . To test our hypothesis we perform a thorough comparison of several state-of-the-art RNN architectures , with and without the dual connection , on the Penn Treebank and the Wikitext-2 datasets . Our results show that , under all experimental conditions , the dual architectures outperform their nondual counterparts . In addition , the Mogrifier-LSTM enhanced with a dual connection establishes a new state-of-the-art word-level perplexity for the Penn Treebank dataset when no additional data are used to train the models . The remainder of the article is organized as follows . First , in section 2 , we present the different models we have used and the two possible architectures , the standard recurrent architecture and the dual architecture . In section 3 , we describe the datasets and the experimental setup . In section 4 , we present our results . And finally , in section 5 , we extract some conclusions and discuss further lines of research . 2 MODELS . We start by presenting the standard recurrent architecture which is common to all the models . In absence of a dual connection , the basic architecture involves an embedding layer , a recurrent layer and a fully-connected layer with softmax activation : et = W exxt ( 1 ) ht = REC ( et , St−1 ) ( 2 ) yt = softmax ( W yhht + b y ) , ( 3 ) where W ∗∗ and b∗ are weight matrices and biases , respectively , and xt is the input vector at time t. TheREC module represents an arbitrary recurrent layer , with St−1 being a set of vectors describing its internal state at the previous time step . In the most general case , this module will simply be an LSTM cell , but we consider other possibilities as well , as described below . The dual architecture introduces an additional layer , with ReLU activation , which is fed with both the output of the embedding layer and the output of the recurrent module : et = W exxt ( 4 ) ht = REC ( et , St−1 ) ( 5 ) dt = ReLU ( W deet +W dhht + b d ) ( 6 ) yt = softmax ( W yddt + b y ) . ( 7 ) This way the network ’ s input can reach the softmax layer following two different paths , through the recurrent layer and through the dual connection . In the following we consider different forms for the recurrent module in equations 2 and 5 . 2.1 THE LSTM MODULE . In the simplest approach the recurrent module consists of an LSTM cell , where the internal state includes both the output and the memory , St = { ht ; ct } , which are computed as follows : f t = σ ( W feet +W fhht−1 + b f ) ( 8 ) it = σ ( W ieet +W ihht−1 + b i ) ( 9 ) ot = σ ( W oeet +W ohht−1 + b o ) ( 10 ) zt = tanh ( W zeet +W zhht−1 + b z ) ( 11 ) ct = f t ct−1 + it zt ( 12 ) ht = ot tanh ( ct ) , ( 13 ) where , as before , W ∗∗ are weight matrices and b∗ are bias vectors . The operator denotes an element-wise product , and σ is the logistic sigmoid function . For convenience , we summarize the joint effect of equations 8-13 as : ht = LSTM ( et , { ht−1 ; ct−1 } ) . ( 14 ) In the literature it is quite common to stack several LSTM layers . Here we consider a double-layer LSTM , where the output ht of the recurrent module is obtained by the concatenated application of two LSTM layers : h′t = LSTM1 ( et , { h ′ t−1 ; c ′ t−1 } ) ( 15 ) ht = LSTM2 ( h ′ t , { ht−1 ; ct−1 } ) . ( 16 ) We refer to this double LSTM module as dLSTM : ht = dLSTM ( et , { ht−1 ; ct−1 ; h′t−1 ; c′t−1 } ) ( 17 ) = LSTM2 ( LSTM1 ( et , { h′t−1 ; c′t−1 } ) , { ht−1 ; ct−1 } ) . ( 18 ) 2.2 THE MOGRIFIER-LSTM MODULE . The Mogrifier-LSTM ( Melis et al. , 2020 ) is one of the state-of-the-art variations of the standard LSTM architecture achieving the lowest perplexity scores in language modeling tasks . It basically consists of a standard LSTM block , but the input et and the hidden state ht−1 are transformed before entering equations 8-13 . The mogrifier transformation involves several steps where et and ht−1 modulate each other : eit = 2σ ( Q ihi−1t−1 ) ei−2t , for odd i ∈ { 1 , 2 , ... , r } ( 19 ) hit−1 = 2σ ( R iei−1t ) h i−2 t−1 , for even i ∈ { 1 , 2 , ... , r } , ( 20 ) where Qi and Ri are weight matrices and we have e−1t = et and h 0 t−1 = ht−1 . The linear transformations Qihi−1t−1 and R iei−1t can also include the addition of a bias vector , which has been omitted for the sake of clarity . The constant r is a hyperparameter whose value defines the number of rounds of the transformation . We refer to this recurrent module , including the mogrifier transformation and the subsequent application of the LSTM layer , as : ht = mLSTM ( et , { ht−1 ; ct−1 } ) = LSTM ( e∗t , { h ∗ t−1 ; ct−1 } ) , ( 21 ) where e∗t and h ∗ t−1 are the highest indexed e i t and h i t−1 in equations 19 and 20 . Note that the choice r = 0 recovers the standard LSTM model . Melis et al . ( 2020 ) also used a double-layer LSTM enhanced with the mogrifier transformation . This strategy can be summarized as follows : ht = mdLSTM ( et , { ht−1 ; ct−1 ; h′t−1 ; c′t−1 } ) ( 22 ) = mLSTM2 ( mLSTM1 ( et , { h′t−1 ; c′t−1 } ) , { ht−1 ; ct−1 } ) . ( 23 ) 3 EXPERIMENTS . 3.1 DATASETS . We perform experiments on two datasets : the Penn Treebank corpus ( Marcus et al. , 1993 ) , as preprocessed by Mikolov et al . ( 2010 ) , and the WikiText-2 dataset ( Merity et al. , 2017 ) . In both cases , the data are used without any additional preprocessing . The Penn Treebank ( PTB ) dataset has been widely used in the literature to experiment with language modeling . The standard data preprocessing is due to Mikolov et al . ( 2010 ) , and includes transformation of all letters to lower case , elimination of punctuation symbols , and replacement of all numbers with a special token . The vocabulary is limited to the 10,000 most frequent words . The data is split into a training set which contains almost 930,000 tokens , and validation and test sets with around 80,000 words each . The WikiText-2 ( WT2 ) dataset , introduced by Merity et al . ( 2017 ) , is a more realistic benchmark for language modeling tasks . It consists of more than 2 million words extracted from Wikipedia articles . The training , validation and test sets contain around 2,125,000 , 220,000 , and 250,000 words , respectively . The vocabulary includes over 30,000 words , and the data retain capitalization , punctuation , and numbers . 3.2 EXPERIMENTAL SETUP . All the considered models follow one of the two architectures discussed in section 2 , either the Embedding-Recurrent-Softmax ( ERS ) architecture ( equations 1-3 ) or the dual architecture ( equations 4-7 ) . In either case , the recurrent module can be any of LSTM , dLSTM , or mdLSTM . Weight tying ( Inan et al. , 2017 ; Press & Wolf , 2017 ) is used to couple the weight matrices of the embedding and the output layers . This reduces the number of parameters and prevents the model from learning a one-to-one correspondence between the input and the output ( Merity et al. , 2018 ) . We run two different sets of experiments . First , we analyze the effect of the dual connection by comparing the performances of the two architectures ( ERS vs Dual ) , using each of the recurrent modules , on both the PTB and the WT2 datasets . In this setting the hyperparameters are tuned for the ERS architecture , and then transferred to the dual case . Second , we search for the best hyperparameters for the dual architecture using the mdLSTM recurrence , and compare the perplexity score with current state-of-the-art values . All the experiments have been performed using the Keras library ( Chollet et al. , 2015 ) , and the implementation is available in a public Github repository1 . The networks are trained using the Nadam optimizer ( Dozat , 2016 ) , a variation of Adam ( Kingma & Ba , 2015 ) where Nesterov momentum is applied . The number of training epochs is different for each experimental condition . On one hand , when the objective is to perform a pairwise comparison between dual and non-dual architectures , we train the models for 100 epochs . On the other hand , when the goal is to compare the dual network with state of the art approaches , we let the models run for 300 epochs . We use batch sizes of 32 and 128 for the PTB and the WT2 problems , respectively , and set the sequence length to 25 in all cases . The remaining hyperparameters are searched in the ranges described in table 1 . Finally , all the models are run twice , both with and without dynamic evaluation ( Krause et al. , 2018 ) . Dynamic evaluation is a standard method commonly used to adapt the model parameters , learned during training , using also the validation data . This allows the networks to get adapted to the new evaluation conditions , which in general improves their performance . In order to keep the models as simple as possible , no additional modifications have been considered . 1The repository is not yet available to preserve author anonymity . It will be released after the review process .

Summary: This work revisits the LSTM architecture. They propose to modify a recurrent architecture by adding a direct connection between the input and the output of the recurrent module, called "dual". They also consider a double-layer LSTM, where the output of the recurrent module is obtained by the concatenated application of two LSTM layers. In experiments, the dual modification can get consistent improvement.