The Importance of the Current Input in Sequence Modeling

1 INTRODUCTION . Deep learning models constitute the current state of the art in most artificial intelligence applications , from computer vision to robotics or medicine . When dealing with sequential data , Recurrent Neural Networks ( RNNs ) , specially those architectures with gating mechanisms such as the LSTM ( Hochreiter & Schmidhuber , 1997 ) , the GRU ( Cho et al. , 2014 ) and other variants , are usually the default choice . One of the most interesting applications of RNNs is related to the field of Natural Language Processing , where most tasks , such as machine translation , document summarization or language modeling , involve the manipulation of sequences of textual data . Of these , language modeling has been extensively used to test different innovations in recurrent architectures , mainly due to the ease of obtaining very large datasets that can be used to train neural networks with millions of parameters . Sequence modeling consists of predicting the next element in a sequence given the past history . In language modeling , the sequence is a text , and hence the task is to predict the next word or the next character . In this context , some of the best performing architectures include the Mogrifier LSTM ( Melis et al. , 2020 ) and different variations of the AWD-LSTM ( Merity et al. , 2018 ) , usually combined with dynamic evaluation and mixture of sofmaxes ( MoS ) ( Wang et al. , 2019 ; Gong et al. , 2018 ) . These models obtain the best state-of-the-art performance with moderate size datasets , such as the Penn Treebank ( Mikolov et al. , 2010 ) or the Wikitext-2 ( Merity et al. , 2017 ) corpora , when no additional data are used during training . When larger datasets are considered , or when external data are used to pre-train the networks , attention-based architectures usually outperform other models ( Radford et al. , 2019 ; Brown et al. , 2020 ) . In this work we use moderate-scale language modeling datasets to explore the effect of a mechanism recently proposed by Oliva & Lago-Fernández ( 2021 ) , when combined with different LSTM-based models in the language modeling context . The idea consists of modifying a recurrent architecture by introducing a direct connection between the input and the output of the recurrent module . This has been shown to improve both the model ’ s generalization results and its readability in simple tasks related to the recognition of regular languages . In a standard RNN , the output depends only on the network ’ s hidden state , ht , which in turn depends on both the input , xt , and the recent past , ht−1 . But there is no explicit dependence of the network ’ s output on its input . In some cases this could be a shortcoming , since the transformation of xt needed to compute the network ’ s internal state is not necessarily the most appropriate to compute the output . However , an explicit dependence of the output on xt can be forced by adding a dual connection that skips the recurrent layers . We claim that this strategy may be of general application in RNN models . To test our hypothesis we perform a thorough comparison of several state-of-the-art RNN architectures , with and without the dual connection , on the Penn Treebank and the Wikitext-2 datasets . Our results show that , under all experimental conditions , the dual architectures outperform their nondual counterparts . In addition , the Mogrifier-LSTM enhanced with a dual connection establishes a new state-of-the-art word-level perplexity for the Penn Treebank dataset when no additional data are used to train the models . The remainder of the article is organized as follows . First , in section 2 , we present the different models we have used and the two possible architectures , the standard recurrent architecture and the dual architecture . In section 3 , we describe the datasets and the experimental setup . In section 4 , we present our results . And finally , in section 5 , we extract some conclusions and discuss further lines of research . 2 MODELS . We start by presenting the standard recurrent architecture which is common to all the models . In absence of a dual connection , the basic architecture involves an embedding layer , a recurrent layer and a fully-connected layer with softmax activation : et = W exxt ( 1 ) ht = REC ( et , St−1 ) ( 2 ) yt = softmax ( W yhht + b y ) , ( 3 ) where W ∗∗ and b∗ are weight matrices and biases , respectively , and xt is the input vector at time t. TheREC module represents an arbitrary recurrent layer , with St−1 being a set of vectors describing its internal state at the previous time step . In the most general case , this module will simply be an LSTM cell , but we consider other possibilities as well , as described below . The dual architecture introduces an additional layer , with ReLU activation , which is fed with both the output of the embedding layer and the output of the recurrent module : et = W exxt ( 4 ) ht = REC ( et , St−1 ) ( 5 ) dt = ReLU ( W deet +W dhht + b d ) ( 6 ) yt = softmax ( W yddt + b y ) . ( 7 ) This way the network ’ s input can reach the softmax layer following two different paths , through the recurrent layer and through the dual connection . In the following we consider different forms for the recurrent module in equations 2 and 5 . 2.1 THE LSTM MODULE . In the simplest approach the recurrent module consists of an LSTM cell , where the internal state includes both the output and the memory , St = { ht ; ct } , which are computed as follows : f t = σ ( W feet +W fhht−1 + b f ) ( 8 ) it = σ ( W ieet +W ihht−1 + b i ) ( 9 ) ot = σ ( W oeet +W ohht−1 + b o ) ( 10 ) zt = tanh ( W zeet +W zhht−1 + b z ) ( 11 ) ct = f t ct−1 + it zt ( 12 ) ht = ot tanh ( ct ) , ( 13 ) where , as before , W ∗∗ are weight matrices and b∗ are bias vectors . The operator denotes an element-wise product , and σ is the logistic sigmoid function . For convenience , we summarize the joint effect of equations 8-13 as : ht = LSTM ( et , { ht−1 ; ct−1 } ) . ( 14 ) In the literature it is quite common to stack several LSTM layers . Here we consider a double-layer LSTM , where the output ht of the recurrent module is obtained by the concatenated application of two LSTM layers : h′t = LSTM1 ( et , { h ′ t−1 ; c ′ t−1 } ) ( 15 ) ht = LSTM2 ( h ′ t , { ht−1 ; ct−1 } ) . ( 16 ) We refer to this double LSTM module as dLSTM : ht = dLSTM ( et , { ht−1 ; ct−1 ; h′t−1 ; c′t−1 } ) ( 17 ) = LSTM2 ( LSTM1 ( et , { h′t−1 ; c′t−1 } ) , { ht−1 ; ct−1 } ) . ( 18 ) 2.2 THE MOGRIFIER-LSTM MODULE . The Mogrifier-LSTM ( Melis et al. , 2020 ) is one of the state-of-the-art variations of the standard LSTM architecture achieving the lowest perplexity scores in language modeling tasks . It basically consists of a standard LSTM block , but the input et and the hidden state ht−1 are transformed before entering equations 8-13 . The mogrifier transformation involves several steps where et and ht−1 modulate each other : eit = 2σ ( Q ihi−1t−1 ) ei−2t , for odd i ∈ { 1 , 2 , ... , r } ( 19 ) hit−1 = 2σ ( R iei−1t ) h i−2 t−1 , for even i ∈ { 1 , 2 , ... , r } , ( 20 ) where Qi and Ri are weight matrices and we have e−1t = et and h 0 t−1 = ht−1 . The linear transformations Qihi−1t−1 and R iei−1t can also include the addition of a bias vector , which has been omitted for the sake of clarity . The constant r is a hyperparameter whose value defines the number of rounds of the transformation . We refer to this recurrent module , including the mogrifier transformation and the subsequent application of the LSTM layer , as : ht = mLSTM ( et , { ht−1 ; ct−1 } ) = LSTM ( e∗t , { h ∗ t−1 ; ct−1 } ) , ( 21 ) where e∗t and h ∗ t−1 are the highest indexed e i t and h i t−1 in equations 19 and 20 . Note that the choice r = 0 recovers the standard LSTM model . Melis et al . ( 2020 ) also used a double-layer LSTM enhanced with the mogrifier transformation . This strategy can be summarized as follows : ht = mdLSTM ( et , { ht−1 ; ct−1 ; h′t−1 ; c′t−1 } ) ( 22 ) = mLSTM2 ( mLSTM1 ( et , { h′t−1 ; c′t−1 } ) , { ht−1 ; ct−1 } ) . ( 23 ) 3 EXPERIMENTS . 3.1 DATASETS . We perform experiments on two datasets : the Penn Treebank corpus ( Marcus et al. , 1993 ) , as preprocessed by Mikolov et al . ( 2010 ) , and the WikiText-2 dataset ( Merity et al. , 2017 ) . In both cases , the data are used without any additional preprocessing . The Penn Treebank ( PTB ) dataset has been widely used in the literature to experiment with language modeling . The standard data preprocessing is due to Mikolov et al . ( 2010 ) , and includes transformation of all letters to lower case , elimination of punctuation symbols , and replacement of all numbers with a special token . The vocabulary is limited to the 10,000 most frequent words . The data is split into a training set which contains almost 930,000 tokens , and validation and test sets with around 80,000 words each . The WikiText-2 ( WT2 ) dataset , introduced by Merity et al . ( 2017 ) , is a more realistic benchmark for language modeling tasks . It consists of more than 2 million words extracted from Wikipedia articles . The training , validation and test sets contain around 2,125,000 , 220,000 , and 250,000 words , respectively . The vocabulary includes over 30,000 words , and the data retain capitalization , punctuation , and numbers . 3.2 EXPERIMENTAL SETUP . All the considered models follow one of the two architectures discussed in section 2 , either the Embedding-Recurrent-Softmax ( ERS ) architecture ( equations 1-3 ) or the dual architecture ( equations 4-7 ) . In either case , the recurrent module can be any of LSTM , dLSTM , or mdLSTM . Weight tying ( Inan et al. , 2017 ; Press & Wolf , 2017 ) is used to couple the weight matrices of the embedding and the output layers . This reduces the number of parameters and prevents the model from learning a one-to-one correspondence between the input and the output ( Merity et al. , 2018 ) . We run two different sets of experiments . First , we analyze the effect of the dual connection by comparing the performances of the two architectures ( ERS vs Dual ) , using each of the recurrent modules , on both the PTB and the WT2 datasets . In this setting the hyperparameters are tuned for the ERS architecture , and then transferred to the dual case . Second , we search for the best hyperparameters for the dual architecture using the mdLSTM recurrence , and compare the perplexity score with current state-of-the-art values . All the experiments have been performed using the Keras library ( Chollet et al. , 2015 ) , and the implementation is available in a public Github repository1 . The networks are trained using the Nadam optimizer ( Dozat , 2016 ) , a variation of Adam ( Kingma & Ba , 2015 ) where Nesterov momentum is applied . The number of training epochs is different for each experimental condition . On one hand , when the objective is to perform a pairwise comparison between dual and non-dual architectures , we train the models for 100 epochs . On the other hand , when the goal is to compare the dual network with state of the art approaches , we let the models run for 300 epochs . We use batch sizes of 32 and 128 for the PTB and the WT2 problems , respectively , and set the sequence length to 25 in all cases . The remaining hyperparameters are searched in the ranges described in table 1 . Finally , all the models are run twice , both with and without dynamic evaluation ( Krause et al. , 2018 ) . Dynamic evaluation is a standard method commonly used to adapt the model parameters , learned during training , using also the validation data . This allows the networks to get adapted to the new evaluation conditions , which in general improves their performance . In order to keep the models as simple as possible , no additional modifications have been considered . 1The repository is not yet available to preserve author anonymity . It will be released after the review process .

Summary: This paper proposes a simple improvement to recurrent network architectures for language modeling. The idea is to insert a single additional layer into a network with one or more recurrent layers, just before the output layer (Eq. 6). This is termed a "dual connection" and it combines the output of the last recurrent layer directly with the input to the network at the current step. The paper presents the results of adding this modification to both LSTM and Mogrifier-LSTM networks on the Penn Treebank and WikiText-2 datasets.