GATO: Gates Are Not the Only Option

1 INTRODUCTION . RNNs allow us to model structured data like text , image , and sound from domains such as art , healthcare , and environmental sciences . These models and their deep variants form the basis for tasks such as generation and prediction ( Pascanu et al. , 2013 ) . Modeling joint distributions of sequences facilitates the generation of images ( Oord et al. , 2016 ) , videos ( Denton & Fergus , 2018 ) , drawings ( Ha & Eck , 2017 ) , and jazz music ( Huang et al. , 2019 ) . Modeling conditional distributions enables applications like predicting patient outcomes in medical centers with electronic health records ( Che et al. , 2018 ) . In all of these cases , the data generation process may involve dependencies that range over hundreds of time steps ( e.g . number of video frames or health over a lifetime ) where early events may determine the distribution of later events ( Kamiya et al. , 2015 ; Denton & Fergus , 2018 ) . The basic RNN ( Elman , 1990 ) suffers from vanishing and exploding gradient problems ( Bengio et al. , 1994 ) that cause difficulties for learning long-range dependencies . The problems stem from extreme eigenvalues of parameter matrices in the RNN . Such models can also suffer from unbounded growth or shrinkage of the hidden state over long sequence lengths . This can happen even when the hidden-to-hidden Jacobian does not vanish or explode and contributes additional difficulties for learning recurrent and decoder parameters . Careful parameterization and initialization of RNNs is crucial for their success in learning long-range dependencies using gradient-based optimization . The majority of proposed solutions make use of RNNs with gated units , notably LSTMs ( Hochreiter & Schmidhuber , 1997 ) , GRUs ( Cho et al. , 2014 ) and Recurrent Highway Networks ( RHNs ) ( Zilly et al. , 2017 ) . These models and deep variants have been the basis for many state-of-the-art empirical results . Gated architectures address gradient issues by bounding the hidden state and designing the recurrence to copy the state forward in time . The GRU and RHN update their state with a weighted average of bounded updates and the current state , where the weighted average in GRU is a convex combination . The LSTM uses an auxiliary cell state with linear recursive updates to copy the hidden state forward . However , gated models are difficult to optimize in practice , in part due to the saturating gradients of gating non-linearities . An alternative to gated models is to restrict hidden transformations to be unitary ( Arjovsky et al. , 2016 ; Wisdom et al. , 2016 ; Jing et al. , 2017 ; Mhammedi et al. , 2017 ) . Unitary matrices can be safely raised to powers without vanishing gradients since they have modulus 1 eigenvalues . Besides unitary transformations , Zhang et al . ( 2018 ) controls the singular values of the transformation using SVD to avoid vanishing and exploding gradient problems . But , these approaches often impose computational overhead to ensure the constraint and require non-trivial parameterizations . These approaches only address hidden-to-hidden gradients . In this work we discuss other properties , such as exploding hidden states in forward propagation when the input is constant for long periods . We propose a sequence model called GATO . The key insight in GATO is that only part of the hidden state must have non-vanishing gradients for states in the far past to influence the current state . GATO splits its hidden state into two channels . At each time step , each channel is updated by a function of the input and current hidden state . To ensure that one of the channels will not have vanishing gradients , GATO limits the functions for both channels to only depend on one of the channels . This results in an identity-block structure in the hidden-hidden Jacobian . GATO then uses non-saturating functions , such as the cosine , to bound the hidden state upon input to the decoder to stabilize gradients for model parameters . We use a simple version of GATO where the dimensions within the channels do not interact and show that it can model arbitrary conditional distributions through memorization . Partitioned hidden states have been mentioned in ( Oliva et al. , 2017 ) and ( Zhang et al. , 2018 ) . Oliva et al . ( 2017 ) can guarantee the identity Jacobian when α = 1 , but in this case the state stays constant . We study GATO copying and adding on long sequences . GATO solves the copying task while the others do not ( Figure 1 ) . On adding , the GRU experiences difficulty as sequence length increases while GATO does not ( Figure 2 ) . We also test GATO on health and language data . Using a critical care database collected in Boston ( Johnson et al. , 2016 ) , GATO predicts patient length-of-stay in the intensive care unit more accurately than GRUs and LSTMs across seeds and learning rates . GATO achieves lower held-out perplexity on language modeling using the Penn Treebank dataset . We explore this in a setting with no methods such as dropout or gradient clipping . GATO is stable to train across random initializations compared to GRUs , LSTMs , and other recently proposed RNNs such as the SRU ( Oliva et al. , 2017 ) , EURNN ( Jing et al. , 2017 ) , and RHN ( Zilly et al. , 2017 ) . 2 BACKGROUND ON SEQUENCE MODELS . Sequence modeling targets learning the sequence of conditional distributions that constitute the joint . Let x = ( x1 , x2 , . . . , xT ) be a sequence of T observations with joint distribution Q ( x ) . We consider the standard probabilistic model pθ ( x ) with parameter θ , where each sequence element depends on all previous elements . Let Lt at timestep t be the negative conditional log probability of the next sequence element according to the model : pθ ( x ) = pθ ( x1 ) T−1∏ t=1 pθ ( xt+1|x≤t ) ; Lt ( θ ) = − log pθ ( xt+1|x≤t ) ; L ( θ ) = ∑ t Lt ( θ ) . To minimize L ( θ ) on samples from Q , we need expressive pθ to capture complicated conditionals . 2.1 SEQUENCE MODELING WITH NEURAL NETWORKS . Sequence models often represent conditionals using a hidden state that evolves as a sequence of data is observed . Using a function D to map from states to distribution parameters , we define the conditional probability p ( xt+1|x≤t ) with parameters D ( ht ) . To represent the true conditionals of Q , ht must summarize the information in x1 . . .xt useful for predicting x > t . Typically , D is chosen to be a feed-forward neural network . Let U be hidden-to-hidden parameters , W be data-to-hidden parameters , and σ be an element-wise non-linearity . RNNs proceed by parameterizing the dynamics of the hidden state via the following discrete-time recurrence : ht+1 = σ ( Uht +Wxt+1 ) . ( 1 ) Though RNNs can represent a broad class of sequences ( Schäfer & Zimmermann , 2006 ; Siegelmann & Sontag , 1995 ) , they face problems in modeling long sequences where an early part of the sequence influences a later part . On one hand , modeling such data requires the use of powerful functions , but on the other , gradient-based optimization of such functions is unstable and requires manual tuning . 2.2 RNNS HAVE PROBLEMS . Vanishing and Exploding Gradients . Consider two states ht and hT with times t T . Exploding gradients occur when hT changes dramatically for small changes in ht , while vanishing gradients occur when the influence of ht on hT decreases exponentially with gap T − t. Consider the RNN as in Equation ( 1 ) and recall that σ is an element-wise non-linearity . Let LT be the loss at time T . Let Dt = diag ( σ′ ( Uht−1 +Wxt ) ) be the Jacobian for σ . Suppose that λ is a lower-bound on the absolute-value of the Jacobian D ’ s entries . Then if the smallest singular value of U is larger than 1/λ , ∂hT /∂ht blows up exponentially as time gap T − t increases . Suppose instead that λ is an upper-bound on the absolute-value of D ’ s entries . Then if the largest singular value of U is smaller than 1/λ , ∂hT /∂ht decays exponentially with time gap T − t. The Jacobian ∂hT /∂ht involved in LT /∂U is ∂LT ∂U = T∑ t=1 ∂LT ∂hT ∂hT ∂ht ∂ht ∂U , and can stop losses LT from backpropagating to U . These issues cause short-term dependencies to be favored over long-term ones ( Bengio et al. , 1994 ; Chung et al. , 2014 ) . Exploding Forward Computation . The hidden state can also grow without bound or vanish for long sequences . Consider the following recurrence where σ denotes sigmoid : z ( 1 ) = σ ( W ( 1 ) xt+1 +U ( 1 ) ht ) ; z ( 2 ) = σ ( W ( 2 ) xt+1 +U ( 2 ) ht ) ; h̃ = tanh ( W ( 3 ) xt+1 +U ( 3 ) ht ) ; ht+1 = z ( 1 ) ht + z ( 2 ) h̃ . The state h can grow without bound . Two examples of systems with good hidden-hidden gradients but that have unbounded forward propagation are RHN ( Zilly et al. , 2017 ) and ( Efficient ) Unitary RNN ( Arjovsky et al. , 2016 ; Jing et al. , 2017 ) . They are described in Appendix G and Appendix H. These models NAN during forward propagation in our experiments . 2.3 LONG SHORT-TERM MEMORY AND GATED RECURRENT UNITS . To improve gradients , Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) and Gated Recurrent Units ( GRU ) ( Cho et al. , 2014 ) augment RNNs with additional computation that helps bound the state as well as reduce gradient vanishing . The LSTM retains a cell state that copies the hidden state forward in time ( Appendix E ) . The GRU uses a convex combination of the current state and a bounded proposal . Its state is bounded when initialized correctly ( Appendix F ) . This variety of neural sequence models has produced state-of-the-art performance in machine translation ( Sutskever et al. , 2014 ; Luong et al. , 2015 ) and image captioning ( Xu et al. , 2015 ) . However , these models are difficult to optimize . Practitioners must experiment with optimization techinques including gradient clipping , randomly dropping-out hidden state updates ( Krueger et al. , 2016 ) , and even training discriminators to keep generated and data sequences close ( Lamb et al. , 2016 ) . Empirical investigation into the behavior of the LSTM gates has also shown that forget and input gate values tend toward 0.5 , contradicting explicit interpretations as gates ( Li et al. , 2018b ) . Recent analysis has shown that LSTMs still have gradient issues that cause them to favor learning short-term dependencies in data ( Kanuparthi et al. , 2019 ) . See Appendix J for a brief discussion . 3 DESIDERATA FOR SEQUENCE MODELS . We now develop criteria for building sequence models and consider several important gradients . Hidden-to-Hidden gradients . For sequence models to learn long-term dependencies , part of the hidden state at time t needs to have non-decaying influence on states at later times T . Therefore only some , not all , of the hidden units need to avoid vanishing gradients . Our first criterion is : 1 . Part ( a block ) of the Jacobian ∂hT /∂ht does not depend on time gap T − t. Decoder Gradients . Let γ be a non-linear function and consider a loss at time t defined through a linear multi-output decoder D ( ht ) = Wγ ( ht ) . Then the gradient of the loss with respect to W is : ∂Lt/∂W = ( ∂Lt/∂D ) ( ∂D/∂W ) = ( ∂Lt/∂D ) γ ( ht ) > To ensure that ||∂Lt/∂W|| does not become too large , one property that γ needs to have is : 2. γ is bounded to ensure that decoder gradients ∂D/∂W do not grow with time . Loss-to-Hidden Gradients . ∂Lt/∂ht for one time step t also depends on the choice of γ through its gradient γ′ . It is defined through the decoder by ∂Lt/∂ht = ( ∂Lt/∂D ) ( ∂D/∂ht ) = ( W > ( ∂Lt/∂D ) ) γ′ ( ht ) This means that : 3. γ′ ( ht ) should not converge to 0 as ||ht|| grows ( if ht does grow ) . 4 . The magnitude ||γ′|| evaluated at ht is bounded even when ||ht|| increases . These criteria require bounded , differentiable functions whose gradients do not go to 0 as their input increases . There is a relationship between the particular form of h ’ s update and the choice of γ . For example , if the update contracts to a constant c from both the positive and negative direction , then γ should be well-behaved at c ( e.g . non-zero gradients ) . These criteria rule out traditional choices of non-linear functions that saturate , such as sigmoid and tanh , and functions that grow such as SoftPlus and ReLU . Periodic functions such as cos and sin meet these criteria . Forward Computation . In tasks such as language modeling , the hidden state can be processed for thousands of steps through the corpus . Let rt denote the part of ht involved in the recurrence to compute ht+1 . Then if ||rt|| is large at time t during training , ||∂Lt+1/∂U|| will be large and can cause additional issues in learning recurrent parameters U . 5 . ||rt|| in general should not become large as t increases .

Summary: This paper proposes a modification of RNN that does not suffer from vanishing and exploding gradient problems. The proposed model, GATO partitions the RNN hidden state into two channels, and both are updated by the previous state. This model ensures that the state in one of the parts is time-independent by using residual connections. The experiments on the long copy and adding tasks, as well as language modeling on the Penn TreeBank dataset show the performance improvement against the basic LSTM and GRU.