Improving the Gating Mechanism of Recurrent Neural Networks

1 INTRODUCTION . Recurrent neural networks ( RNNs ) have become a standard machine learning tool for learning from sequential data . However , RNNs are prone to the vanishing gradient problem , which occurs when the gradients of the recurrent weights become vanishingly small as they get backpropagated through time ( Hochreiter et al. , 2001 ) . A common approach to alleviate the vanishing gradient problem is to use gating mechanisms , leading to models such as the long short term memory ( Hochreiter & Schmidhuber , 1997 , LSTM ) and gated recurrent units ( Chung et al. , 2014 , GRUs ) . These gated RNNs have been very successful in several different application areas such as in reinforcement learning ( Kapturowski et al. , 2018 ; Espeholt et al. , 2018 ) and natural language processing ( Bahdanau et al. , 2014 ; Kočiskỳ et al. , 2018 ) . At every time step , gated recurrent models form a weighted combination of the history summarized by the previous state , and ( a function of ) the incoming inputs , to create the next state . The values of the gates , which are the coefficients of the combination , control the length of temporal dependencies that can be addressed . This weighted update can be seen as an additive or residual connection on the recurrent state , which helps signals propagate through time without vanishing . However , the gates themselves are prone to a saturating property which can also hamper gradient-based learning . This is particularly troublesome for RNNs , where carrying information for very long time delays requires gates to be very close to their saturated states . We formulate and address two particular problems that arise with the standard gating mechanism of recurrent models . First , typical initialization of the gates is relatively concentrated . This restricts the range of timescales the model can address , as the timescale of a particular unit is dictated by its gates . Our first proposal , which we call uniform gate initialization ( Section 2.2 ) , addresses this by directly initializing the activations of these gates from a distribution that captures a wider spread of dependency lengths . Second , learning when gates are in their saturation regime is difficult because of vanishing gradients through the gates . We derive a modification that uses an auxiliary refine gate to modulate a main gate , which allows it to have a wider range of activations without gradients vanishing as quickly . Combining these two independent modifications yields our main proposal , which we call the URgating mechanism . These changes can be applied to any gate ( i.e . bounded parametrized function ) and have minimal to no overhead in terms of speed , memory , code complexity , and ( hyper- ) parameters . We apply them to the forget gate of recurrent models , and evaluate on many benchmarks including synthetic long-term dependency tasks , sequential pixel-level image classification , language modeling , program execution , and reinforcement learning . Finally , we connect our methods to other proposed gating modifications , introduce a framework that allows each component to be replaced with similar ones , and perform theoretical analysis and extensive ablations of our method . Empirically , the UR- gating mechanism robustly improves on the standard forget and input gates of gated recurrent models . When applied to the LSTM , these simple modifications solve synthetic memory tasks that are pathologically difficult for the standard LSTM , achieve state-of-the-art results on sequential MNIST and CIFAR-10 , and show consistent improvements in language modeling on the WikiText-103 dataset ( Merity et al. , 2016 ) and reinforcement learning tasks ( Hung et al. , 2018 ) . 2 GATED RECURRENT NEURAL NETWORKS . Broadly speaking , RNNs are used to sweep over a sequence of input data xt to produce a sequence of recurrent states ht ∈Rd summarizing information seen so far . At a high level , an RNN is just a parametrized function in which each sequential application of the network computes a state update u : ( xt , ht−1 ) 7→ht . Gating mechanisms were introduced to address the vanishing gradient problem ( Bengio et al. , 1994 ; Hochreiter et al. , 2001 ) , and have proven crucial to the success of RNNs . This mechanism essentially smooths out the update using the following equation , ht=ft ( xt , ht−1 ) ◦ht−1+it ( xt , ht−1 ) ◦u ( xt , ht−1 ) , ( 1 ) where the forget gate ft and input gate it are [ 0,1 ] d-valued functions that control how fast information is forgotten or allowed into the memory state . When the gates are tied , i.e . ft+it=1 as in GRUs , they behave as a low-pass filter , deciding the time-scale on which the unit will respond ( Tallec & Ollivier , 2018 ) . For example , large forget gate activations close to ft=1 are necessary for recurrent models to address long-term dependencies.1 We will introduce our improvements to the gating mechanism primarily in the context of the LSTM , which is the most popular recurrent model . However , these techniques can be used in any model that makes similar use of gates . ft=σ ( Lf ( xt , ht−1 ) ) ( 2 ) it=σ ( Li ( xt , ht−1 ) ) ( 3 ) ut=tanh ( Lu ( xt , ht−1 ) ) ( 4 ) ct=ft◦ct−1+it◦ut ( 5 ) ot=σ ( Lo ( xt , ht−1 ) ) ( 6 ) ht=ottanh ( ct ) ( 7 ) A typical LSTM ( equations ( 2 ) - ( 7 ) ) is an RNN whose state is represented by a tuple ( ht , ct ) consisting of a “ hidden ” state and “ cell ” state . The basic gate equation ( 1 ) is used to create the next cell state ct ( 5 ) . Note that the gate and update activations are a function of the previous hidden state ht−1 instead of ct−1 . Here , L ? stands for a parameterized linear function of its inputs with bias b ? , e.g . Lf ( xt , ht−1 ) =Wfxxt+Wfhht−1+bf , ( 8 ) and σ ( · ) refers to the standard sigmoid activation function which we will assume is used for defining [ 0,1 ] -valued activations in the rest of this paper . The gates of the LSTM were initially motivated as a binary mechanism , switching on or off to allow information and gradients to pass through . However , in reality this fails to happen due to a combination of initialization and saturation . This can be problematic , such as when very long dependencies are present . 2.1 THE UR-LSTM bf ∼σ−1 ( U [ 0,1 ] ) ( 9 ) ft=σ ( Lf ( xt , ht−1 ) ) ( 10 ) rt=σ ( Lr ( xt , ht−1 ) ) ( 11 ) gt=rt · ( 1− ( 1−ft ) 2 ) + ( 1−rt ) ·f2t ( 12 ) ct=gtct−1+ ( 1−gt ) ut ( 13 ) ot=σ ( Lo ( xt , ht−1 ) ) ( 14 ) ht=ottanh ( ct ) ( 15 ) We present two solutions which work in tandem to address the previously described issues . The first ensures a diverse range of gate values at the start of training by sampling the gate ’ s biases so that the activations will be approximately uniformly distributed at initialization . We call this Uniform Gate Initialization ( UGI ) . The second allows better gradient flow by reparameterizing the gate using an auxiliary “ refine ” gate . As our main application is for recurrent models , we present the full UR-LSTM model in equations ( 9 ) - ( 15 ) . However , we note that 1In this work , we use “ gate ” to alternatively refer to a [ 0,1 ] -valued function or the value ( “ activation ” ) of that function . these methods can be used to modify any gate ( or more generally , bounded function ) in any model . In this context the UR-LSTM is simply defined by applying UGI and a refine gate r on the original forget gate f to create an effective forget gate g ( equation ( 12 ) ) . This effective gate is then used in the cell state update ( 13 ) . Empirically , these small modifications to an LSTM are enough to allow it to achieve nearly binary activations and solve difficult memory problems ( Figure 4 ) . In the rest of Section 2 , we provide theoretical justifications for UGI and refine gates . 2.2 UNIFORM GATE INITIALIZATION . Standard initialization schemes for the gates can prevent the learning of long-term temporal correlations ( Tallec & Ollivier , 2018 ) . For example , supposing that a unit in the cell state has constant forget gate value ft , then the contribution of an input xt in k time steps will decay by ( ft ) k. This gives the unit an effective decay period or characteristic timescale ofO ( 11−ft ) . 2 Standard initialization of linear layersL sets the bias term to 0 , which causes the forget gate values ( 2 ) to concentrate around 1/2 . A common trick of setting the forget gate bias to bf =1.0 ( Jozefowicz et al. , 2015 ) does increase the value of the decay period to 11−σ ( 1.0 ) ≈3.7 . However , this is still relatively small , and moreover fixed , hindering the model from easily learning dependencies at varying timescales . We instead propose to directly control the distribution of forget gates , and hence the corresponding distribution of decay periods . In particular , we propose to simply initialize the value of the forget gate activations ft according to a uniform distribution U ( 0,1 ) , as described in Section 2.1 . An important difference between UGI and standard or other ( e.g . Tallec & Ollivier , 2018 ) initializations is that negative forget biases are allowed . The effect of UGI is that all timescales are covered , from units with very high forget activations remembering information ( nearly ) indefinitely , to those with low activations focusing solely on the incoming input . Additionally , it introduces no additional parameters ; it even can have less hyperparameters than the standard gate initialization , which sometimes tunes the forget bias bf . Appendix B.2 and B.3 further discuss the theoretical effects of UGI on timescales . 2.3 THE REFINE GATE . Given a gate f=σ ( Lf ( x ) ) ∈ [ 0,1 ] , the refine gate is an independent gate r=σ ( Lr ( x ) ) , and modulates f to produce a value g∈ [ 0,1 ] that will be used in place of f downstream . It is motivated by considering how to modify the output of a gate f in a way that promotes gradient-based learning , derived below . An additive modification The root of the saturation problem is that the gradient∇f of a gate , which can be written solely as a function of the activation value as f ( 1−f ) , decays rapidly as f approaches 0 or 1 . Thus when the activation f is past a certain upper or lower threshold , learning effectively stops . This problem can not be fully addressed only by modifying the input to the sigmoid , as in UGI and other techniques , as the gradient will still vanish by backpropagating through the activation function . Therefore to better control activations near the saturating regime , instead of changing the input to the sigmoid in f = σ ( L ( x ) ) , we consider modifying the output . In particular , we consider adjusting f with an input-dependent update φ ( f , x ) for some function φ , to create an effective gate g=f+φ ( f , x ) that will be used in place of f downstream such as in the main state update ( 1 ) . This sort of additive ( “ residual ” ) connection is a common technique to increase gradient flow , and indeed was the motivation of the LSTM additive gated update ( 1 ) itself ( Hochreiter & Schmidhuber , 1997 ) . Choosing the adjustment function Although many choices seem plausible for selecting the additive update φ , we reason backwards from necessary properties of the effective activation g to deduce a principled function φ . The refine gate will appear as a result . First , note that ft might need to be increased or decreased , regardless of what its value is . For example , given a large activation ft near saturation , it may need to be even higher to address long-term dependencies in recurrent models ; alternatively , if it is too high by initialization or needs to unlearn previous behavior , it may need to decrease . Therefore , the additive update to f should create an effective activation gt in the range ft±α for some α . Note that the allowed adjustment range α=α ( ft ) needs to be a function of f in order to keep g∈ [ 0,1 ] . 2This corresponds to the number of timesteps it takes to decay by 1/e . In particular , the additive adjustment range α ( f ) should satisfy the following natural properties : Validity : α ( f ) ≤min ( f,1−f ) , to ensure g∈f±α ( f ) ∈ [ 0,1 ] . Symmetry : Since 0 and 1 are completely symmetrical in the gating framework , α ( f ) =α ( 1−f ) . Differentiability : α ( f ) will be used in backpropagation , requiring α∈C1 ( R ) . Figure 2a illustrates the general appearance of α ( f ) based on these properties . In particular , Validity implies that that its derivative satisfies α′ ( 0 ) ≤ 1 and α′ ( 1 ) ≥ −1 , Symmetry implies α′ ( f ) = −α′ ( 1−f ) , and Differentiability implies α′ is continuous . The simplest such function satisfying these is the linear α′ ( f ) =1−2f , yielding α ( f ) =f−f2=f ( 1−f ) . Given such a α ( f ) , recall that the goal is to produce an effective activation g=f+φ ( f , x ) such that g∈f±α ( f ) ( Figure 2b ) . Our final observation is that the simplest such function φ satisfying this is φ ( f , x ) =α ( f ) ψ ( f , x ) for some ψ ( · ) ∈ [ −1,1 ] . Using the standard method for defining [ −1,1 ] -valued functions via a tanh non-linearity leads to φ ( f , x ) =α ( f ) ( 2r−1 ) for another gate r=σ ( L ( x ) ) . The full update is given in Equation ( 16 ) , g=f+α ( f ) ( 2r−1 ) =f+f ( 1−f ) ( 2r−1 ) = ( 1−r ) ·f2+r· ( 1− ( 1−f ) 2 ) ( 16 ) Equation ( 16 ) has the elegant interpretation that the gate r linearly interpolates between the lower band f−α ( f ) =f2 and the symmetric upper band f+α ( f ) =1− ( 1−f ) 2 ( Figure 2b ) . In other words , the original gate f is the coarse-grained determinant of the effective gate g , while the gate r “ refines ” it . This allows the effective gate g to reach much higher and lower activations than the constituent gates f and r ( Figure 2c ) , bypassing the saturating gradient problem . For example , this allows the effective forget gate to reach g=0.99 when the forget gate is only f=0.9 .

Summary: This paper introduces two novel techniques to help long term signal propagation in RNNs. One is an initialization strategy which uses inverse sigmoid function to avoid the decay of the contribution of the input earlier in time and another is a new design of a refine gate which pushes the value of the gate closer to 0 or 1.  The authors conduct exhaustive ablation and empirical studies on copy task, sequential MNIST, language modeling and reinforcement learning.