Gaussian-gated LSTM: Improved convergence by reducing state updates

Recurrent neural networks can be difficult to train on long sequence data due to the well-known vanishing gradient problem . Some architectures incorporate methods to reduce RNN state updates , thereby allowing the network to preserve memory over long temporal intervals . We propose a timing-gated LSTM RNN model , called the Gaussian-gated LSTM ( g-LSTM ) for reducing state updates . The time gate controls when a neuron can be updated during training , enabling longer memory persistence and better error-gradient flow . This model captures long temporal dependencies better than an LSTM on very long sequence tasks and the time gate parameters can be learned even from a non-optimal initialization . Because the time gate limits the updates of the neuron state , the number of computes needed for the network update is also reduced . By adding a computational budget term to the training loss , we obtain a network which further reduces the number of computes by at least 10× . Finally , we propose a temporal curriculum learning schedule for the g-LSTM that helps speed up the convergence time of the equivalent LSTM on long sequences . 1 INTRODUCTION . Numerous methods and architectures have been proposed to mitigate the vanishing gradient problem in RNNs , with LSTMs ( Hochreiter & Schmidhuber , 1997 ) as one of the first prominent solutions doing so by including gating structures in the computation . Although the LSTM has excelled at handling many tasks ( Schmidhuber , 2015 ; Lipton , 2015 ) , it still has difficulties in learning complex and long time dependencies ( Neil et al. , 2016 ; Chang et al. , 2017 ; Trinh et al. , 2018 ) . In the last few years , various methods which reduce the state updates of an RNN ( LSTM ) have been explored to better learn long time dependencies from data . Clockwork RNNs ( Koutnik et al. , 2014 ) group the hidden units of the RNN into “ modules , ” where each module is executed at pre-specified time steps thereby skipping time steps which helps learn longer time dependencies . Recently , various other methods have been proposed which can be characterized by the use of additional “ time gates , ” kn , that control the information flow from one time step to the next ( Krueger et al. , 2016 ; Campos et al. , 2017 ) . Phased LSTM ( PLSTM ) ( Neil et al. , 2016 ) learns a parameterized function , kn , from the the time input of the current state and was proven to be successful at learning over long sequences . The PLSTM time gate , parameterized by period , phase , and ratio parameters for each hidden unit , is defined through a modulo function with an ill-defined gradient . Furthermore , with periodic functions being hard to learn using gradient-based methods ( Shamir , 2016 ) and with kn being periodic , the PLSTM was unable to learn the time gate parameters and hence relied on careful initialization . In order to offset these difficulties , this work proposes a new LSTM variant called the Gaussian-gated LSTM ( g-LSTM ) . Similar to the PLSTM , it is an LSTM model with a parameterized kn but with only two parameters per hidden unit . Unlike the PLSTM which uses a periodic formulation for kn , the g-LSTM uses a Gaussian function . We show in this work that the g-LSTM can provide a number of possible advantages over the LSTM , in particular , on long sequence tasks that pose convergence problems during training : • The g-LSTM network can process very long sequences by reducing the time over which the neurons can be updated . It converges faster than the LSTM , especially on sequences that are over 500 steps . • The “ openness ” of the neuron for an update can be adapted according to the task during training , even for extreme , non-optimal initializations of the time gate parameters . • By introducing a computational budget term into the loss function during training , the “ openness ” of the neuron can be optimized for a reduced computational budget . This reduction can be achieved with little or no degradation to the network performance and is useful for network pruning . • A “ temporal curriculum ” training schedule can be set up for the g-LSTM so that it helps to speed up the convergence of a normal LSTM . The paper is structured as follows : In section 2 , we discuss briefly the related work . Then , in section 3 , we present the formulation of the g-LSTM , the datasets used in this work and details about the experimental hyperparameters . In section 4 , we present experiments demonstrating the usefulness of the g-LSTM with respect to the four claims listed above . We provide gradient analysis in section 5 to further explain the faster convergence results of the g-LSTM . Finally in section 6 , we conclude with a brief discussion of the results . 2 RELATED WORK . There have been a multitude of proposed methods to improve the training of RNNs , especially for long sequences . Apart from incorporating additional gating structures , for example the LSTM and the GRU ( Cho et al. , 2014 ) , more recently various techniques were proposed to further increase the capabilities of recurrent networks to learn on sequences of length over 1000 . Proposed initialization techniques such as the orthogonal initialization of kernel matrices ( Cooijmans et al. , 2016 ) , chrono initialization of the biases ( Tallec & Ollivier , 2018 ) , and diagonal recurrent kernel matrices ( e.g . Li et al . ( 2018 ) ) have demonstrated success . Trinh et al . ( 2018 ) propose using truncated backpropagation with an additional auxiliary loss to reconstruct previous events . Methods that enable more efficient learning on long temporal sequences use solutions that preserve memory over longer timescales . Such solutions were first explored by Koutnik et al . ( 2014 ) in the Clockwork RNN ( CW-RNN ) . This network skips state updates by allowing different neurons to be “ activated ” on different , modulated clock cycles . More recently proposed models for skipping updates include the Phased LSTM ( PLSTM ) ( Neil et al. , 2016 ) which uses a modulo-periodic timing gate to limit state updates ; the Zoneout network ( Krueger et al. , 2016 ) which skips state updates in a random manner ; and the Skip RNN ( Campos et al. , 2017 ) which learns a state skipping scheme from the data to shorten the effective sequence length for the task . Additionally , the LSTM-Jump ( Yu et al. , 2017 ) uses a reinforcement learning algorithm to learn when to skip state updates , showing a method to more quickly process ( long ) sequential data with an RNN while maintaining an accuracy comparable to a baseline LSTM . It has been suggested but not yet demonstrated in the literature that the parameters of the CW-RNN clock cycle and PLSTM timing gate could be learned in training . Currently , the implementation of these networks requires a careful initialization of these parameters . With the Gaussian-gated LSTM ( g-LSTM ) in this work we present a time gated RNN network that converges on long sequence tasks and also has the ability to learn its time gate parameters even when initialized in a nonoptimal way . 3 METHODS . 3.1 G-LSTM . The g-LSTM is an LSTM model with an additional time gate ( Fig . 1 ) . This time gate is used to regulate the information flow in time . Equations 1 - 3 describe the update equations for the hidden and cell states of the LSTM . Equations 4 and 5 describe the gating mechanism of the time gate , kn . in = σ ( xnWxi + hn−1Whi + bi ) , fn = σ ( xnWxf + hn−1Whf + bf ) ( 1 ) c̃n = fn cn−1 + in σ ( xnWxg + hn−1Whg + bg ) ( 2 ) on = σ ( xnWxo + hn−1Who + bo ) , h̃n = on tanh ( c̃n ) ( 3 ) cn = kn c̃n + ( 1− kn ) cn−1 ( 4 ) hn = kn h̃n + ( 1− kn ) hn−1 ( 5 ) In a standard LSTM , the gating functions in , fn , on , represent the input , forget , and output gates respectively at sequence index n. cn is the cell activation vector , and xn and hn represent the input feature vector and the hidden output vector , respectively . The cell state cn is updated with a fraction of the previous cell state that is controlled by fn , and a new input state created from the element-wise ( Hadamard ) product , denoted by , of in and the candidate cell activation as in Eq . 2 . In the g-LSTM , we further control the cell state and the output hidden state through the kn gate which is independent of the input data and hidden states , and is purely dependent on the time input corresponding to the sequence index n. The use of the Hadamard product ensures that each hidden unit is independently controlled by the corresponding time gate unit , thus enabling the different units in the layer to process the input at different time scales . The time gate kn is defined based on a Gaussian function as : kn = e− ( tn−µ ) 2/σ2where the mean parameter , µ , defines the time when the hidden unit is “ open ” and the standard deviation , σ , controls the openness of the time gate for each unit around its corresponding µ . The time inputs t = { t1 , t2 , ... , tn , ... , tN } for the sequence x = { x1 , x2 , ... , xn , ... , xN } can correspond to the physical notion of time at the respective sequence input . In the absence of a standard notion of time , we use the sequence indices as the time input , i.e . t = { 1 , 2 , ... , n , ... , N } . In this work , we assume this notion of time by default . The “ openness ” of kn for a neuron is defined by the parameterization of its Gaussian function . 3.2 BACK PROPAGATION FOR G-LSTM . An important characteristic of the g-LSTM is reduced gradient flow in back propagation training methods . By having the gating structure as in Eqs . 4 and 5 there are fewer gradient product terms , which reduces the likelihood of vanishing or exploding gradients . In a gradient descent learning scheme for a given loss function , L , when training the recurrent parameters , Wh ( from Eqs . 1 - 3 ) , the gradient as in Eq . 6 is used . ∂L ∂Wh = ∂L ∂hN ∂hN ∂Wh ( 6 ) By the chain rule ∂hN∂Wh expands for all time steps of the sequence , n ∈ { 1 , ... , N } . Because each output state is gated by the time gate , kn , the gradient terms in the expansion of ∂hN∂Wh are scaled by kn . When the time gate is open less often , i.e . with a small σ value , then there are fewer influential gradient terms . More details are given in Appendix A . 3.3 DATASETS . The experiments described in the paper are carried out on the adding task and two standard long sequence datasets : the sequential MNIST and the sequential CIFAR-10 datasets . Adding task : In order to test the long sequence learning capability of the g-LSTM , we use the adding task ( Hochreiter & Schmidhuber , 1997 ) . In this task , the network is presented with two sequences of length N , x = { x1 , ... , xN } , xn ∼ U ( 0 , 1 ) ) and m = { m1 , ... , mN } , mn ∈ { 0 , 1 } , ∑Nt=1mn = 2 . The sequence m has exactly two values of 1 and the remaining values of the sequence are 0 . The indices of the “ 1 ” values are chosen at random . For each pair in the sequence ( x , m ) , the associated label value , y , is the sum of the two values in x corresponding to the “ 1 ” values of m. The objective of this task is to minimize the mean squared error between the predicted sum from the network , ŷ , and the labeled sum , y . A new training set of 5000 sequence samples is presented in every epoch during training in order to avoid overfitting . The test set consists of a separate 5000 samples . For N > 1000 , it is known that LSTMs have difficulty learning the task and hence we focus on values of N > 1000 in this work . sMNIST : The sequential MNIST dataset is widely used to analyze the performance of a recurrent model . This dataset consists of 60,000 training samples and 10,000 test samples , each a single vector sequence of length 784 corresponding to the 28 × 28 pixel images in the MNIST dataset ( LeCun et al. , 1998 ) . We also use permuted MNIST ( pMNIST ) , a permuted variant of the sMNIST dataset where the sequences are processed with a fixed random permutation , making the task harder . sCIFAR-10 : The sequential CIFAR-10 dataset is another long sequence dataset based on CIFAR10 ( Krizhevsky et al. , 2014 ) with 10 classes . The 32 × 32 RGB pixel images are reshaped into sequences of length 1024 with 3 dimensional features corresponding the RGB channels at every time step . Like in the sMNIST dataset , the dataset consists of 60,000 training samples and 10,000 test samples .

Summary: This paper focuses on the reduction of training time by various mechanisms. By introducing a time gate during training, it controls when a neuron (weights?) can be updated during training. By introducing and additional budget term in the loss function, training costs (number of computations) are reduced by one order of magnitude.