DecayNet: A Study on the Cell States of Long Short Term Memories

1 INTRODUCTION . Neural networks are powerful universal approximators that are difficult to interpret . This paper presents a numerical study on the forward pass of the long-short term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) recurrent neural networks ( RNNs ) . We treat LSTMs as systems of difference equations and present visualisations to study and to clarify consecutive transitions in the cell state dynamics . We introduce DecayNets as an addition to the LSTM architecture – DecayNets have monotonic decays in forget gates and enjoy smooth transitions in their cell state values . LSTMs are versatile models that have been used to advance the state-of-the-art for a variety of machine learning problems . This covers handwriting recognition ( Graves et al. , 2009 ) , speech recognition ( Sundermeyer et al. , 2012 ) , and text modelling ( Kim et al. , 2016 ) . There are an abundance of work that extends on the basic LSTM architecture . Popular extensions include the attention mechanism ( Bahdanau et al. , 2014 ) and the bidirectional design ( Schuster & Paliwal , 1997 ) . It is also not uncommon to blend LSTMs into vision tasks to expand network practicalities ( Xu et al. , 2015 ) . However , it is unclear whether LSTMs are optimal architectural designs for RNNs ( Jozefowicz et al. , 2015 ) . Additionally , the significance of their individual components are unclear , and empirical explorations are required for understanding their utility in real-world data ( Karpathy et al. , 2015 ) . This lack of interpretability limits our ability in designing better and more transparent networks . RNNs repeatedly integrate new observations into implicit hidden variables to model temporal representations of sequences . Like dynamical systems , the network undergoes iterative computations on a same set of operators . The transitional dynamics can be visualised through cobweb diagrams ( Glendinning , 1994 ) ; and numerical analyses allow us to predict shrinkage and growth in variables . We show that the updating scheme of the LSTM cell state volatilely alternates between a catch and a release phase . The former controls the shrinkage while the latter controls the growth of cell state magnitudes . Our study reveals that the forget gate dictates the dynamical alterations of the cell state . We hence introduce DecayNets , LSTMs with monotonically decreasing forget gates , for stabilising cell state propagation and for making networks more interpretable . With recurrent batch normalisation ( Cooijmans et al. , 2016 ) , DecayNets outperform previous state of the art results ( best : 97.8 % ; mean : 97.4 % ) on the fixed-length permuted pixel-by-pixel ( sequential ) MNIST task ( Le et al. , 2015 ; LeCun et al. , 1998 ) . For the arbitrary-length learning-to-learn task ( Andrychowicz et al. , 2016 ) , LSTM-optimisers with the Decay mechanism decrease multi-layer perceptron ( MLP ) losses more rapidly , and converge to lower and less varied MLP losses . 2 THE VANILLA LSTM . DecayNets will be frequently compared to conventional LSTMs . The most commonly implemented LSTM RNN was proposed by Graves & Schmidhuber ( 2005 ) , and this paper will refer to the architecture therein that paper as the LSTM . LSTMs are recursive systems driven by its input xt , which has the data-size dimensionality M . The network propagates for a data-length of D units of time , with the subscript t as the recursive instance , i.e. , t = 1 · · ·D . Each instantaneous computation ( in abbreviated form ) encompasses System ( 1 ) : the vanilla LSTM the gated variables : ft , it , ot = σ ( W { F , I , O } xt + WR { F , I , O } qt−1 + b { F , I , O } ) , the internal input : at = tanh ( WAxt + WRAqt−1 + bA ) , the cell state : st = ft st−1 + it at , the hidden state : qt = ot tanh ( st ) , with the dimensionality : xt ∈ RM , ft , it , ot , at , qt , st , b { F , I , O , A } ∈ RN , and W { F , I , O , A } ∈ RN×M , WR { F , I , O , A } ∈ RN×N . The input xt and the hidden state qt contribute to other variables via Ws and via WRs . The dimensionality N is user-defined and is the operator for element-wise product . Along with biases bs , the regressive contributions are activated by either the sigmoidal function σ or the hyperbolic tangent function tanh . During the update of the cell state st , the forget gate ft relinquishes fragments of its previous instance st−1 , and the input gate it fine-tunes replenishment from the internal input at . Finally , the variable qt , which also serves as the output of the network , is a transformation of st synchronised with the output gate ot . 3 LSTMS AS DIFFERENCE EQUATIONS . Difference equations are recurrent relations with discrete propagation ; and the term dynamic refers to the dependency between time and geometric space . Dynamical behaviours of a difference equation can be studied to determine whether variables will grow or shrink . This allows us to explain behaviours of individual LSTM variables and clarify network mechanics . This section discusses three key concepts – the importance for studying the LSTM cell state ; the two dynamic regimes for cell state values to grow and to shrink ; and that the forget gate can be controlled to yield interpretable and predictable cell state motions . Cobweb diagrams will be used to support concept visualisations , and a brief introduction to the technique can be found in Appendix A . 3.1 THE CELL STATE REVISITED . This paper postulates the cell state as the source of LSTMs ’ modelling capability . Furthermore , we regard gated variables as secondary variables . Our justifications are as follows . First , gated variables only exist to read and to set the hidden variables of qt and st. That is , they exist to assist LSTMs to memorise better , but are not a part of the memories . Second , tanh is a one-to-one function , thus dynamic properties of qt are inherited from st. Last , at is included in the update of st , and acts as a small component to the grander dynamic entailed in st. For these reasons , we postulate the cell state as the most important variable and study it as a difference equation . 3.2 A PIECEWISE ANALYSIS ON THE CELL STATE . We study the cell state , st = ft st−1 + it at , separately as sp = ft st−1 and sq = it at . Since we have asserted gated variables as secondary variables , we can think of ft and it as constant vectors with elements between 0 and 1 . That is , individual dimensions of sp and sq have cobweb outlines that are analogous to Figures 1 ( a ) and 1 ( b ) respectively . For the rest of this paper , we will refer to the marginal dimensions of ft , it , at , xt , and st as fmt , imt , amt , xmt , and smt , respectively . For instance , each dimension of sq is imt amt and inherits the diverse characteristic outline of 1 ( b ) through tanh from amt . Consecutive transitions in network variables are shown as the dotted trajectories in Figure 1 . They start from the rhombuses and ends on the circles . The characteristic outlines of the sub-dynamics are the solid lines , and the dashed lines represent the 45-degree y = x lines . The two subfigures address qualitatively different dynamic regimes . The origin of Figure 1 ( a ) is a sink that attracts propagation trajectories and decreases variable magnitudes . Conversely , the origin of 1 ( b ) is a source that repels propagation trajectories and increases variable magnitudes . We refer to 1 ( a ) as the dynamic catch regime ; and refer to 1 ( b ) as the dynamic release regime . The update of the LSTM cell state is thus a composition of two opposing modifiers . With one that dedicates to its growth , and the other specialised in its shrinkage . It is thus crucial to understand which one , between sp and sq , serves as the more dominant evolution component . We will then reformulate the more dominant component with deterministic properties to redefine the network with interpretable and affirmative qualities . A key difference between the opposing regimes of Figure 1 is the boundedness of the characteristic outlines . The sub-dynamic of sq is limited between±1 , while that of sp is unbounded . This implies that sq contributes less to the additive relationship of sk , and is less impactful than sp . With the dominant component identified , our next target is to understand how sp affects vanilla LSTMs . The insight will allow us to propose sensible modifications to make LSTM mechanics more interpretable . 3.3 THE FORGET GATE AND THE CATCH-AND-RELEASE DYNAMICS . Based on the two insights of , first , sp = ft st−1 serving as the dominant modifier to st , and second , as each of its dimensional outline sm ( t−1 ) being modified by the gradient fmt , we conjecture that the forget gate controls the overall gradient of the characteristic outline of the cell state . That is , ft predominantly controls the stability of the origin , and dictates the dynamic alterations between the catch and the release phases . As a consequence and regardless of sq , values of the cell state are more likely to grow with large ft , and more likely to shrink with small ft. We use Figure 2 to support these inferences . The first row simulates the near-random dynamics of a vanilla LSTM cell state . We present rows two and three to show stark contrasts , and advantages , of generating predictable cell state propagation with controllable forget gate values . 3.3.1 THE NEAR-RANDOM DYNAMICS OF A VANILLA LSTM CELL STATE . Gated units of LSTMs are not hardwired with deterministic propagation . As a result , they evolve in a near-chaotic fashion . The first row of Figure 2 simulates behaviours of st under unregulated ft . The three subfigures of the row serves as consecutive instances of a propagation , i.e. , t = [ 1 . . . 3 ] . The presentation follows Figure 1 , and dot-dash lines are appended to instances t = 2 and 3 to show the characteristic outlines of their previous instances . Parameters fm ( 1 ... 3 ) = [ 1 , 0.25 , 1 ] and im ( 1 ... 3 ) = [ 1 , 0 , 0.75 ] are carefully chosen to exaggerate impacts brought to the vanilla LSTM cell state by the near-chaotic evolution of gates . Real scenarios and experimental set-ups that justify the choice of these demonstrative parameters can be found in Appendix B . These settings lead to sudden catches and abrupt releases . The disordered dynamics make it difficult to predict growth and shrinkage in cell state magnitudes . It is also unclear how such drastic behaviours can assist us to understand the learned features of a trained LSTM . 3.3.2 INTERPRETABLE CELL STATE DYNAMICS UNDER CONTROLLED FORGET GATES . Let us now consider cell state dynamics under controlled fts . The second row has fmt = 0.9 with im ( 1 ... 3 ) = [ 1 , 0.75 , 0.5 ] ; and the third row has fmt = 0.25 with im ( 1 ... 3 ) = [ 1 , 0.75 , 0.15 ] . Consistent growth and consistent shrinkage are observed in rows two and three respectively . The magnitudes evolve monotonically because the controlled forget gates expose cell states to homogeneous dynamic regimes . Large constant fmts expose the cell state to consistent ( global ) releases ; whereas small constant fmts expose smt to global catches . Dynamic reversals may occur in st through sq , but only under the unlikely incidents where combinations of extreme values of it and at are simultaneously presented . In addition , the dynamic reversals are merely temporary ( local ) due to the boundedness of values in sq . The dynamic regimes of sp and sq behave analogously to the respective difference equations of ( a ) : xt = T ( xt−1 ) = 0.5xt−1 and ( b ) : xt = T ( xt−1 ) = tanh ( 3xt−1 ) . The propagation , in dots , start from rhombuses and end on circles . The dashed lines represent y = x and the solid lines denotes the characteristic outlines of y = T ( x ) . Vertical mappings from y = x to y = T ( x ) reflect instantaneous variable computations of T ( xt−1 ) ; and horizontal mappings from y = T ( x ) to y = x represent the recursive nature of difference equations to re-insert the last output as the new input xt = T ( xt−1 ) . The origin of ( a ) is a sink which attracts propagation and diminishes variable magnitudes ; that of ( b ) is a source which repels propagations and increases variable magnitudes . t =1 t =2 t =3 Three qualitatively different scenarios for cell state propagation are investigated . The subfigures follow the presentation format of Figure 1 ; additional dot-dash lines are appended as the characteristic shape of the t − 1th instance . The characteristic shape of interest is the update of the cell state y = T ( x ) = fmt x + imt tanh ( x ) . From top to bottom , the rows represent for cell state dynamics under an unregulated LSTM , under a large constant forget gate , and under a small constant forget gate , respectively . The latter two scenarios exhibit monotonic propagation because the constant forget gates expose cell states to homogeneous dynamical regimes . The motions are thus more interpretable than those of the unregulated LSTM , where abrupt shrinkage and sudden growth are observed .

Summary: This paper analyses the internal dynamics of an LSTM, focusing on the cell state as being the most important component, and analyses what directly influences the contents of the cell state using difference equations. The authors note that at any timestep, the output cell state is the sum of (previous cell state * forget) and (input gate * input). The former can only shrink or maintain the cell value, which the authors label 'catch' and the latter can increase the magnitude, labelled 'release'.