Title: DECAYNET: A STUDY ON THE CELL STATES OF LONG-SHORT TERM MEMORIES

Abstract: It is unclear whether the extensively applied long-short term memory (LSTM) is an optimised architecture for recurrent neural networks. Its complicated design makes the network hard to analyse and non-immediately clear for its utilities in real-world data. This paper studies LSTMs as systems of difference equations, and takes a theoretical mathematical approach to study consecutive transitions in network variables. Our study shows that the cell state propagation is predominantly controlled by the forget gate. Hence, we introduce DecayNets, LSTMs with monotonically decreasing forget gates, to calibrate cell state dynamics. With recurrent batch normalisation, DecayNet outperforms the previous state of the art for permuted sequential MNIST. The Decay mechanism is also beneficial for LSTMbased optimisers, and decrease optimisee neural network losses more rapidly.

1. INTRODUCTION:

Neural networks are powerful universal approximators that are difficult to interpret. This paper presents a numerical study on the forward pass of the long-short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) recurrent neural networks (RNNs). We treat LSTMs as systems of difference equations and present visualisations to study and to clarify consecutive transitions in the cell state dynamics. We introduce DecayNets as an addition to the LSTM architecture -DecayNets have monotonic decays in forget gates and enjoy smooth transitions in their cell state values.

LSTMs are versatile models that have been used to advance the state-of-the-art for a variety of machine learning problems. This covers handwriting recognition (Graves et al., 2009) , speech recognition (Sundermeyer et al., 2012) , and text modelling (Kim et al., 2016) . There are an abundance of work that extends on the basic LSTM architecture. Popular extensions include the attention mechanism (Bahdanau et al., 2014) and the bidirectional design (Schuster & Paliwal, 1997) . It is also not uncommon to blend LSTMs into vision tasks to expand network practicalities (Xu et al., 2015) . However, it is unclear whether LSTMs are optimal architectural designs for RNNs (Jozefowicz et al., 2015) . Additionally, the significance of their individual components are unclear, and empirical explorations are required for understanding their utility in real-world data (Karpathy et al., 2015) . This lack of interpretability limits our ability in designing better and more transparent networks.

RNNs repeatedly integrate new observations into implicit hidden variables to model temporal representations of sequences. Like dynamical systems, the network undergoes iterative computations on a same set of operators. The transitional dynamics can be visualised through cobweb diagrams (Glendinning, 1994) ; and numerical analyses allow us to predict shrinkage and growth in variables.

We show that the updating scheme of the LSTM cell state volatilely alternates between a catch and a release phase. The former controls the shrinkage while the latter controls the growth of cell state magnitudes. Our study reveals that the forget gate dictates the dynamical alterations of the cell state. We hence introduce DecayNets, LSTMs with monotonically decreasing forget gates, for stabilising cell state propagation and for making networks more interpretable.

With recurrent batch normalisation (Cooijmans et al., 2016) , DecayNets outperform previous state of the art results (best: 97.8%; mean: 97.4%) on the fixed-length permuted pixel-by-pixel (sequential) MNIST task (Le et al., 2015; LeCun et al., 1998) . For the arbitrary-length learning-to-learn task (Andrychowicz et al., 2016) , LSTM-optimisers with the Decay mechanism decrease multi-layer perceptron (MLP) losses more rapidly, and converge to lower and less varied MLP losses.

2. THE VANILLA LSTM:

DecayNets will be frequently compared to conventional LSTMs. The most commonly implemented LSTM RNN was proposed by Graves & Schmidhuber (2005) , and this paper will refer to the architecture therein that paper as the LSTM.

LSTMs are recursive systems driven by its input x t , which has the data-size dimensionality M . The network propagates for a data-length of D units of time, with the subscript t as the recursive instance, i.e., t = 1 • • • D. Each instantaneous computation (in abbreviated form) encompasses System (1): the vanilla LSTM the gated variables:

f t , i t , o t = σ(W {F ,I ,O} x t + W R{F ,I ,O} q t-1 + b {F ,I ,O} ), the internal input: a t = tanh(W A x t + W RA q t-1 + b A ), the cell state: s t = f t s t-1 + i t a t , the hidden state: q t = o t tanh(s t ), with the dimensionality: x t ∈ R M , f t , i t , o t , a t , q t , s t , b {F,I,O,A} ∈ R N , and W {F,I,O,A} ∈ R N ×M , W R{F,I,O,A} ∈ R N ×N .

The input x t and the hidden state q t contribute to other variables via Ws and via W R s. The dimensionality N is user-defined and is the operator for element-wise product. Along with biases bs, the regressive contributions are activated by either the sigmoidal function σ or the hyperbolic tangent function tanh. During the update of the cell state s t , the forget gate f t relinquishes fragments of its previous instance s t-1 , and the input gate i t fine-tunes replenishment from the internal input a t . Finally, the variable q t , which also serves as the output of the network, is a transformation of s t synchronised with the output gate o t .

3. LSTMS AS DIFFERENCE EQUATIONS:

Difference equations are recurrent relations with discrete propagation; and the term dynamic refers to the dependency between time and geometric space. Dynamical behaviours of a difference equation can be studied to determine whether variables will grow or shrink. This allows us to explain behaviours of individual LSTM variables and clarify network mechanics.

This section discusses three key concepts -the importance for studying the LSTM cell state; the two dynamic regimes for cell state values to grow and to shrink; and that the forget gate can be controlled to yield interpretable and predictable cell state motions. Cobweb diagrams will be used to support concept visualisations, and a brief introduction to the technique can be found in Appendix A.

3.1. THE CELL STATE REVISITED:

This paper postulates the cell state as the source of LSTMs' modelling capability. Furthermore, we regard gated variables as secondary variables. Our justifications are as follows.

First, gated variables only exist to read and to set the hidden variables of q t and s t . That is, they exist to assist LSTMs to memorise better, but are not a part of the memories. Second, tanh is a one-to-one function, thus dynamic properties of q t are inherited from s t . Last, a t is included in the update of s t , and acts as a small component to the grander dynamic entailed in s t . For these reasons, we postulate the cell state as the most important variable and study it as a difference equation.

3.2. A PIECEWISE ANALYSIS ON THE CELL STATE:

We study the cell state, s t = f t s t-1 + i t a t , separately as s p = f t s t-1 and s q = i t a t . Since we have asserted gated variables as secondary variables, we can think of f t and i t as constant vectors with elements between 0 and 1. That is, individual dimensions of s p and s q have cobweb outlines that are analogous to Figures 1(a) and 1(b) respectively.

For the rest of this paper, we will refer to the marginal dimensions of f t , i t , a t , x t , and s t as f mt , i mt , a mt , x mt , and s mt , respectively. For instance, each dimension of s q is i mt a mt and inherits the diverse characteristic outline of 1(b) through tanh from a mt .

Consecutive transitions in network variables are shown as the dotted trajectories in Figure 1 . They start from the rhombuses and ends on the circles. The characteristic outlines of the sub-dynamics are the solid lines, and the dashed lines represent the 45-degree y = x lines.

The two subfigures address qualitatively different dynamic regimes. The origin of Figure 1 (a) is a sink that attracts propagation trajectories and decreases variable magnitudes. Conversely, the origin of 1(b) is a source that repels propagation trajectories and increases variable magnitudes. We refer to 1(a) as the dynamic catch regime; and refer to 1(b) as the dynamic release regime.

The update of the LSTM cell state is thus a composition of two opposing modifiers. With one that dedicates to its growth, and the other specialised in its shrinkage. It is thus crucial to understand which one, between s p and s q , serves as the more dominant evolution component. We will then reformulate the more dominant component with deterministic properties to redefine the network with interpretable and affirmative qualities.

A key difference between the opposing regimes of Figure 1 is the boundedness of the characteristic outlines. The sub-dynamic of s q is limited between ±1, while that of s p is unbounded. This implies that s q contributes less to the additive relationship of s k , and is less impactful than s p . With the dominant component identified, our next target is to understand how s p affects vanilla LSTMs. The insight will allow us to propose sensible modifications to make LSTM mechanics more interpretable.

3.3. THE FORGET GATE AND THE CATCH-AND-RELEASE DYNAMICS:

Based on the two insights of, first, s p = f t s t-1 serving as the dominant modifier to s t , and second, as each of its dimensional outline s m(t-1) being modified by the gradient f mt , we conjecture that the forget gate controls the overall gradient of the characteristic outline of the cell state. That is, f t predominantly controls the stability of the origin, and dictates the dynamic alterations between the catch and the release phases. As a consequence and regardless of s q , values of the cell state are more likely to grow with large f t , and more likely to shrink with small f t .

We use Figure 2 to support these inferences. The first row simulates the near-random dynamics of a vanilla LSTM cell state. We present rows two and three to show stark contrasts, and advantages, of generating predictable cell state propagation with controllable forget gate values. These settings lead to sudden catches and abrupt releases. The disordered dynamics make it difficult to predict growth and shrinkage in cell state magnitudes. It is also unclear how such drastic behaviours can assist us to understand the learned features of a trained LSTM.

3.3.2. INTERPRETABLE CELL STATE DYNAMICS UNDER CONTROLLED FORGET GATES:

Let us now consider cell state dynamics under controlled f t s. The second row has f mt = 0.9 with i m(1...3) = [1, 0.75, 0.5]; and the third row has

f mt = 0.25 with i m(1...3) = [1, 0.75, 0.15].

Consistent growth and consistent shrinkage are observed in rows two and three respectively.

The magnitudes evolve monotonically because the controlled forget gates expose cell states to homogeneous dynamic regimes. Large constant f mt s expose the cell state to consistent (global) releases; whereas small constant f mt s expose s mt to global catches.

Dynamic reversals may occur in s t through s q , but only under the unlikely incidents where combinations of extreme values of i t and a t are simultaneously presented. In addition, the dynamic reversals are merely temporary (local) due to the boundedness of values in s q . The subfigures follow the presentation format of Figure 1 ; additional dot-dash lines are appended as the characteristic shape of the t -1th instance. The characteristic shape of interest is the update of the cell state y = T (x ) = fmt x + imt tanh(x ). From top to bottom, the rows represent for cell state dynamics under an unregulated LSTM, under a large constant forget gate, and under a small constant forget gate, respectively. The latter two scenarios exhibit monotonic propagation because the constant forget gates expose cell states to homogeneous dynamical regimes. The motions are thus more interpretable than those of the unregulated LSTM, where abrupt shrinkage and sudden growth are observed.

4. DECAYNETS:

This section presents DecayNets1 , and calibrate cell state dynamics according to mathematical insights therein Subsection 3.3. From Figure 2 , it is evident that forget gates deprived of random evolution yield predictable and stable cell state propagation. For this reason, we propose the idea to hard-wire forget gate outputs with a continuous and monotonic decrease.

The DecayNet modifies the forget gate according to System (2): The Decay mechanism on the LSTM forget gate the forget-polar input:

p t = p t-1 -π 2D 1 6 (W F x t + W RF q t-1 + 3) purposedly presented this way with p 0 = π 2 • 1,

forget gate elements:

f ti = Φ(p ti ) = 1 for p ti > π 2 0 for p ti < 0 sin(p ti ) otherwise for i in 1, 2, . . . N ,

and inherits all remaining variables and their corresponding dimensionalities from System (1). Note, no new learnable parameters are introduced to this system.

The forget gate is initialised as a vector of ones, with equivalent portions of the said initial ones set to be lost over the iterative instances of the LSTM. That is, we set .

f 0i = sin(p 0i ) = sin( π 2 ) = 1, and that, over time, we remove a maximised cumulative amount of 1, and a minimised cumulative amount of 0, to constrain the final value

f t D ∈ [0, 1].

We implement the sinusoidal function as the "activation function" of the forget gate. In addition, we engineer "pre-activated neurons" with values within the bounds of [0, π 2 ]. Let us elaborate. We replace σ with sin to decay equivalent portions over time. The available amount of instantaneous decay needs to be uniform throughout time to ensure that all instantaneous lost in values are equally important. The sigmoidal outline of σ yields less changes when magnitudes of pre-activated values are extreme. We remove this undesirable property with the substitution of sin with radian-like inputs.

Elements of the pre-activated neurons are in the bound of [0, π 2 ]. Empirically, we observed that distributions of W F x t + W RF q t-1 , with entries of W F and W RF initialised with uniform sampling within the interval of ±1, are bounded within the limits of [-3, 3 ].2 That is, we have .

W F x t + W RF q t-1 ∈ [-3, 3] and . π 2 1 6 (W F x t + W RF q t-1 + 3) ∈ [0, π 2 ].

The design in mind is thus, for each instance, a maximum amount of π 2D is lost from p ti '-s initialised π 2 . In other words, over an input-length of D units of time, an accumulation, minimised at 0 and maximum at π 2 , is lost in the forget-polar input. The former induces the final value of p Di = π 2 (equivalently f Di = 1), and the latter gives p Di = 0 (equivalently f Di = 0).

DecayNets learn to dichotomise cell state values conditioned on the input shown. One subset of the cell state maintains large magnitudes, whereas the other decreases to small magnitudes. The former subset corresponds to dimensions of the forget gate that maintain their large magnitude over time; whereas the latter subset corresponds to those that diminish.

More importantly, the monotonic decrease in the forget gate serves as an interpretation to LSTM's inferential decision mechanism. This is because the update of the cell state, s t = s p + s q = f t s t-1 + i t a t , is composed of recurrent memory s p and feedforward memory s q . The irreversible decay indicates that, dimensions of the cell state, that correspond to forget gates that diminish, gradually lose their recurrent nature over time, and effectively function as feedforward neurons. Large forget gate values thus serve as clear indicators for, those neural dimensions, that the DecayNet prefer to reiterate, for the input-driven optimisation.

5. RELATED STUDIES:

The forward pass of LSTMs (and RNNs) consists high dimensional variables with coupled relationships; and is mostly understood with observational studies and search studies. The former explores practical network usages, and the latter finds optimal combinations of network variables.

Observational studies suggest that cell states develop specific functionality. In text (Karpathy et al., 2015) , they are observed to act as line length counters and quotation machines; whereas in speech (Wu & King, 2016) , they are highly correlated to the Mel-Cepsrtal coefficient. Search studies (Greff et al., 2017; Chung et al., 2014; Jozefowicz et al., 2015) found that, no particular combinatory sets of variables yield an architecture, that consistently outperforms LSTM in all experimental conditions.

Though rigorous and informative, these studies are based on exhibited qualities of optimised LSTMs. Karpathy et al. (2015) conjectured the functionality of trained LSTM cell states through examining post-activated cell state values tanh(s t ). Greff et al. (2017) observed that gated recurrent units (GRUs) (Cho et al., 2014) are capable of performing well on tasks unoptimised by those LSTMs deprived of output gates. Hence, they conjectured that GRUs' coupling of the input gate and the forget gate, and the output gate of LSTMs prevent unbounded cell states. In contrast, our theoretical mathematical approach therein Sections 3 and 4 are conducted prior to the training of LSTMs.

Mathematical properties of neural networks are mostly studied through the lens of statistical learning theory. This paper took an alternative approach. We treated neural networks as dynamical systems, and offered the perspective of understanding LSTMs as systems of difference (discrete) equations.

Statistical studies and dynamical studies are complementary. Burger & Neubauer (2003) took an inverse problem approach and integrated Tikhonov regularisations to enhance neural network stability. Tallec & Ollivier (2018) offers another theoretical study on the forward pass. Incremental updates of LSTMs are treated as differential equations, and the effects of multiple scales are analysed.

The mathematical analyses therein this paper differs to existing literature because, we propose modifications on a specific variable, to yield an affirmative / deterministic outcome -DecayNets have monotonically decreasing forget gates. In addition, DecayNets initialise their forget gate values as 1s, which is similar to the effect of the common practice of setting LSTM forget gate biases to 1s (Gers et al., 1999) , for yielding large initial forget values and for "remembering more by default".

A hard regularisation emerges from the reformulation. As mentioned in Section 4, the network learns to irreversibly suppress the recurrent nature of a selection of cell state values conditioned on the input shown. This differs to soft regularisations such as Dropout (Srivastava et al., 2014) and Zoneout (Krueger et al., 2016) , where the respective nullified and frozen activation are recoverable.

6. EXPERIMENTS AND RESULTS:



6.1. PERMUTED SEQUENTIAL MNIST IMAGE CLASSIFICATION:

We apply DecayNets and LSTMs on image classification with sequential inputs. The data of choice is MNIST (LeCun et al., 1998) , a database of handwritten digits of numbers [0-9], with 60K and 10K images for training and testing respectively. MNIST images are in 28 × 28-pixel formats, we present each pixel, one at a time, as a 784(= 28 × 28)-unit pixel-by-pixel sequence to the RNNs therein this subsection. Following Le et al. ( 2015), we apply a fixed random permutation over the pixels for setting up the task commonly known as permuted sequential MNIST (Perm-SeqMNIST).

RNNs of this subsection have a single layer of 100 hidden dimensions and are trained on the RM-SProp optimiser (Tieleman & Hinton, 2012 ) over 150 epoches. The learning rate is 0.001, with 0.9 momentum and no weight decay. Gradient clipping is applied at 1 to avoid exploding gradients (Pascanu et al., 2012) . All weights are initialised with uniform sampling within the interval of ±1, and all biases are initialised as zeros. No normalisation is performed on the data prior to training, and a softmax classifier is attached to produce prediction from the final hidden state q D .

The best prediction accuracy for unregulated RNNs is 92.1% for DecayNet and 89.5% for LSTM (which roughly matches the baseline model of 89.8% in Krueger et al. (2016) ). We then integrated recurrent batch normalisation (RBN) (Cooijmans et al., 2016) to DecayNet and found that both the best performing RBN DecayNet, 97.8%, and the average simulatory performance, 97.4%, yield 2016), we speculate this as a result of differences in the initialisations of weights and biases.

A similar task, of image classification with row-by-row inputs, is presented in Appendix C. There, DecayNets outperform LSTMs with less significant differences, but yield more consistent results.

6.2. LSTMS AS OPTIMISERS:

The second task follows Andrychowicz et al. (2016) and update weights of multi-layer perceptrons (MLPs) with LSTMs. The paper showed that MLP-optimisees trained with LSTM-optimisers yield marginally lower losses than those trained with the hand-crafted optimiser of ADAM (Kingma & Ba, 2014) . Here, we show that the Decay mechanism helps the already successful LSTM-optimiser decrease MLP losses more rapidly, and converge to lower and less varied MLP losses.

We train two-layer MLPs for MNIST classification and update their weights θ with LSTMs. The MNIST images are normalised with mean and standard deviation (MSD) = (0.1037, 0.3081), and are organise as vector inputs of dimensionality 784. The first MLP layer has weights of dimensionality R 784×20 ; the second layer, a softmax layer, has weights of dimensionality R 20×10 . The update . θ t = θ t-1 -α t ∇ θt-1 L t uses learning rate α t and ∇ θt-1 L t , the gradient of the optimisation loss with respect to parameter θ t-1 . We use LSTM-optimisers to infer α t given the element-wise product of θ t and their corresponding gradients G θt-1 . That is, for each instance, in brevity, .

α t , q t , s t = LSTM(θ t-1 G θt-1 , q t-1 , s t-1 ). The optimisers are trained according to Andrychowicz et al. (2016) , which we reiterate in Appendix D. For justifications and details on the choices of the input and the output, see Appendix E.

The Decay mechanism allows LSTM-optimisers to update their hidden variables for multiple steps before they create learning rates. At the beginning of each instance, the forget polar input is reset as the maximal values of π 2 • 1 (effectively f t = 1). For each instance, the input of θ t-1 G θt-1 monotonically decrease the forget gate and update the hidden variables as according to .

q t-1 2 , s t-1 2 = DecayNet(θ t-1 G θt-1 , q t-1 , s t-1 ) and . α t , q t , s t = DecayNet(θ t-1 G θt-1 , q t-1 2 , s t-1 2 ).

That is, the learning rate is created only after the gated units and hidden variables propagated for two times. Such an internal propagation can be pro-longed, for example, to twenty steps.

A comparison of optimiser performances is presented in Figure 4 (a). The figure shows MLP losses after every update of a size 32 mini-batch. The colours of green, grey, blue, yellow, and purple are used to denote stochastic gradient descent with learning rate 0.05, ADAM with learning rate 0.002, RMSProp with learning rate 0.001, a trained LSTM-optimiser, and a trained DecayNet-optimiser, respectively. LSTM-based optimisers outperform hand-crafted rivals by a significant margin, and their MLP losses converge after the 60th update. Hence, we treat losses from the 80th to the 100th update as losses of optimised MLPs. Using these data, two-independent sample t-test found statistical significance that DecayNet-optimised MLPs yield less losses than LSTM-optimised MLPs, with MSD= (-0.13, 0.01) by p-value < 0.001. In addition, we present Figure 4 (b), a sequence of violin-plots, to examine the running performance of the LSTM-based optimisers. With colours inherited from 4(a), each violin-plot aggregates twenty iterative updates of loss data, i.e., the first is of the 1st to the 20th update, and that the second is of the 21st to the 40th update. The figure indicates that DecayNet-optimisers minimise MLP losses more rapidly, and converge to less varied losses.

7. CONCLUSION:

This paper proposes DecayNets, LSTMs with monotonically decreasing forget gates. Motivations and modifications are based on theoretical mathematical clarifications of the LSTM cell state mechanics. The update of the cell state is studied as a difference equation, and we have showen that the update alternates between a catch and a release phase to control shrinkage and growth in its values. We have found that the dynamic alteration is predominately controlled by the forget gate and propose to stabilise the cell state through a monotonically decreasing forget gate. The reformulation is easy to implement and introduces no extra learnable parameters. With RBN, DecayNets achieved better results than previously reported state of the art performance on the Perm-SeqMNIST task. Our experimental results have showen that the Decay mechanism also functions as a promising additional feature for those LSTMs casted as optimisers to update optimisee MLP weights.

However, the most important quality of the Decay mechanism is that it is deterministic. It would be interesting to treat the gradually decaying patterns as features to analyse interactions among speech utterances and other natural language processing tasks (Suominen, 2014) . Such a similar deterministic-ness may be hard-wired to GRUs. However, the update of the GRU cell state is more complicated than its LSTM counterpart, and separate dynamical analyses must be conducted.

LSTMs are still opaque. The authors of this manuscript believe that more mathematical analyses should be focused on component s q of the cell state. We believe this less controllable, and more diverse sub-dynamic of the cell state controls bifurcations (Glendinning, 1994) in LSTM dynamics. Subplot (a) is for y = 0.5x, while (b) is for y = 1.5x. Both scenarios share the origin as an equilibrium. Gradients around the vicinity of the equilibria serve as major influences that dictate their stability. Solutions, the neural trajectories, start from the rhombuses and ends at the circles. The gradients around the origin of (a) is < 1, the equilibrium is hence a sink and attracts neural trajectoreis; its counterpart in (b) is > 1, hence a source, and repels trajectories. The attractiveness and repulsiveness of the dynamic singularities dictate the growth and shrinkage in variables.

This appendix should only serve as a brief introduction to cobweb diagrams. Interested readers should consult Glendinning (1994) for a broad and complete view on cobweb diagrams, and other dynamical techniques / implications.

Cobweb diagram serves as an intuitive technique to visualise dynamical motions for variables in low dimensional difference equations. Note, not differential equations. Let us first revisit the concept of difference equations, and then introduce cobweb diagrams.

The simplest form of a difference equation is x t = T (x t-1 ) with x 0 = α. Difference equations are simple recursive networks where variable x t serves as both the input and the output. Subscript t represents computational instances, and T is some arbitrary function.

A cobweb diagram consists 4 fundamental elements. The 45-degree y = x line, the characteristic shape of the operator y = T (x), an initial point x 0 = α, and a trail of propagation trajectories. Figure 5 will be used to support the narrative therein this appendix. The dashed line is y = x, the solid line is y = T (x), the initial positions are in rhombuses, the solution trajectories are dotted, and the final positions are in circles.

Cobwebs are useful because they visualise repeated insertions of functional outputs. The repetition is visualised by two qualitatively different types of mappings: a vertical mapping, and a horizontal mapping. Vertical mappings initiate from y = x and terminate on y = T (x), whereas horizontal mappings start from y = T (x) and end on y = x. Vertical mappings represent for functional computations, i.e., given the input of x k-1 → compute for the output of T (x k-1 ). Their counterparts of horizontal mappings represent for system updates, i.e., given the post-computational T (x k-1 ) → update the network variable of x k = T (x k-1 ). This is the reason why the variable trajectories shown in Figure 5 are zig-zag-like. Not all trajectories are zig-zagged, interested readers can find descriptions of tent map propagation in Boeing (2016) .

Another important feature to cobwebbing is the labelling of the dynamic singularities known as equilibria. Equilibria are the intersections of y = x and y = T (x). These are the positions where, once a trajectory maps into, or starts on them, the said trajectory will never leave. Trajectories move towards attractive / stable equilibria known as sinks, and moves away from repulsive / unstable equilibria known as sources. For instance, the origin of Figure 5 (a) is a stable equilibria, and that of 5(b) is an unstable equilibria. While travelling towards the sink, the variable of 5(a) decreases in magnitudes, i.e., magnitudes of the y-coordinates of the trajectories decrease; and while travelling away from the source, the variable of 5(b) increases in magnitudes. From these two examples, it is clear that the stability of the equilibria are highly dependent on the gradients around their vicinities. The gradient around the origin of 5(a) and 5(b) are < 1 and > 1 respectively. More interesting dynamics may emerge with more complicated outlines of y = T (x). The binary string additive problem defined:

The vanilla LSTM is provided with two input numbers of x s1 and x s2 in their 16-digit binary form; the network is trained to generate the correct summation output y s , also in the 16-digit binary form. We apply some simple constraints to ensure that values of both the input and the output do not exceed the maximum capacity of the 16-digit binary form. The maximum value of x s1 is limited to the rounded value of 2 16 -1 2 , and that of x s2 is limited to the floored value of 2 16 -1 2 . The LSTM in this appendix is a one layer network with 2 input dimensions, 32 hidden dimensions, and 1 output dimension. The network is set to run for 16 units of time, and during each instance, one digit from each of the paired input is forwarded to the LSTM. The binary digits are presented from back to forth. For every instance, the output of the network is computed as follows. Matrix multiplication of a learnable feedforward regressive weight is applied onto the current hidden state q t , the product is then activated via tanh, and finally rounded.

The network is trained in gradient descent with backpropagation through time. The learning rate is set as 0.1, and the network is not regularised by other means.

Time series of the forget gate::

After the network is optimised, we randomly generated two pairs of x s1 and x s2 and inserted them to the network. Time series of the 2nd, 12th, 22nd, and 32nd dimension of the forget gate are shown in Figure 6 .

Marginal dimensions of the forget gate oscillate in aperiodic and near-random fashions. There are no clear nor repeating patterns, and it is common to have extreme alterations in oscillatory magnitudes. The demonstrative parameters f m(1...3) = [1, 0.25, 1] mimic these behaviours via an initial decrease followed by a sharp increase; the demonstrative parameters are chosen to provide clear illustrations on the impact to s t from f t , thus the three values are chosen to induce extreme alterations. We apply DecayNets and LSTMs for image classification with sequential inputs. The data of choice is, again, MNIST, and Fashion-MNIST (Xiao et al., 2017) . Fashion-MNIST contains images of fashion products from 10 categories; it also has 60K and 10K images for training and testing, and has images in 28 × 28-pixel formats. Comparative results are listed in Table 2 .

This appendix considers the task of processing images row-by-row, sequentially, from top to bottom. There are thus 28 instances and the inputs are vectors of 28 dimensions. Prior to training, no normalisation is applied to MNIST, and a normalisation of mean and standard deviation (MSD) = (0.5, 0.5) is applied to Fashion-MNIST. No other means of regularisation is applied. RNNs of this subsection have one hidden dimension, which we will elaborate later. A softmax classifier is attached to produce prediction from the final hidden state q D ; and all RNNs are trained on the ADAM with learning rate 0.01. Note, the term MSD will reappear later.

We conduct two comparisons on DecayNets and LSTMs. The first under the setting of 32 hidden dimensions (HD) and the second with 64 HD. All networks are trained on MNIST for 1 epoch.

Within each HD-group, the best performing DecayNets outperform LSTMs. The results, in the prediction accuracy, are, 96.4% vs 96.2% for 32 HD, and 97.1% vs 97.0% for 64 HD. Two independent sample t-tests show that DecayNets yield statistically significantly greater accuracy than LSTMs. For 32 HD models, DecayNets give MSD = (95.6%,0.0037) against LSTMs' MSD = (95.1%,0.0078), with p-values < 10 -4 . For 64 HD models, DecayNets give MSD = (96.5,0.0033) against LSTMs' MSD = (96.2%,0.0064), with p-values < 10 -4 . We present back-to-back violin plots with side-by-side box plots in Figure 7 (a). Purple is used to denote DecayNets, and yellow is for LSTMs. The box plots show that the inter-quatile ranges of DecayNets are smaller than those of LSTMs, implying that the DecayNets yield more consistent and more reliable results.

:

Implementation of our algorithm will be available at https://github.com/anonymous author/decaynet/

New/revised text are in brown to assist the reviewers, and will be resumed to black after the revision period.

Figures and Tables:

.1 THE NEAR-RANDOM DYNAMICS OF A VANILLA LSTM CELL STATE Gated units of LSTMs are not hardwired with deterministic propagation. As a result, they evolve in a near-chaotic fashion. The first row of Figure 2 simulates behaviours of s t under unregulated f t . The three subfigures of the row serves as consecutive instances of a propagation, i.e., t =[1 . . . 3]. The presentation follows Figure 1, and dot-dash lines are appended to instances t = 2 and 3 to show the characteristic outlines of their previous instances. Parameters f m(1...3) = [1, 0.25, 1] and i m(1...3) = [1, 0, 0.75] are carefully chosen to exaggerate impacts brought to the vanilla LSTM cell state by the near-chaotic evolution of gates. Real scenarios and experimental set-ups that justify the choice of these demonstrative parameters can be found in Appendix B.

Figure 1: Visualisations of difference equations variable propagation The dynamic regimes of sp and sq behave analogously to the respective difference equations of (a): xt = T (xt-1) = 0.5xt-1 and (b): xt = T (xt-1) = tanh(3xt-1) . The propagation, in dots, start from rhombuses and end on circles. The dashed lines represent y = x and the solid lines denotes the characteristic outlines of y = T (x ). Vertical mappings from y = x to y = T (x ) reflect instantaneous variable computations of T (xt-1); and horizontal mappings from y = T (x ) to y = x represent the recursive nature of difference equations to re-insert the last output as the new input xt = T (xt-1). The origin of (a) is a sink which attracts propagation and diminishes variable magnitudes; that of (b) is a source which repels propagations and increases variable magnitudes.

Figure 2: Cell state propagation of controlled and uncontrolled forget gates Three qualitatively different scenarios for cell state propagation are investigated.The subfigures follow the presentation format of Figure1; additional dot-dash lines are appended as the characteristic shape of the t -1th instance. The characteristic shape of interest is the update of the cell state y = T (x ) = fmt x + imt tanh(x ). From top to bottom, the rows represent for cell state dynamics under an unregulated LSTM, under a large constant forget gate, and under a small constant forget gate, respectively. The latter two scenarios exhibit monotonic propagation because the constant forget gates expose cell states to homogeneous dynamical regimes. The motions are thus more interpretable than those of the unregulated LSTM, where abrupt shrinkage and sudden growth are observed.

Figure 3: Testing accuracy of the RNNs on Perm-SeqMNIST Purple and blue denotes DecayNet, while green and yellow is used for LSTM. Performances of RBN RNNs are thick solid lines over shades. Bottoms and tops of the shades are the worst and the best performance of all simulations; and the thick central line is the average performance.

Figure 4: Performances of optimisers We compare various optimisers to update the weights of MLP-optimisees for learning MNIST classification. Subplot (a) shows that, both LSTM-based optimisers, yellow for LSTMs and purple for DecayNets, yield MLPs with marginally lower losses than their rival hand-crafted optimisational schemes. Subplot (b) shows that the appended Decay mechanism allows LSTMs to decrease MLP losses more rapidly.

Figure 5: Simple examples of cobwebs Two simple examples for cobwebs are presented.Subplot (a) is for y = 0.5x, while (b) is for y = 1.5x. Both scenarios share the origin as an equilibrium. Gradients around the vicinity of the equilibria serve as major influences that dictate their stability. Solutions, the neural trajectories, start from the rhombuses and ends at the circles. The gradients around the origin of (a) is < 1, the equilibrium is hence a sink and attracts neural trajectoreis; its counterpart in (b) is > 1, hence a source, and repels trajectories. The attractiveness and repulsiveness of the dynamic singularities dictate the growth and shrinkage in variables.

Figure 6: Time series of f t in an optimised LSTM

Figure 2 is based on Figure 6. The first two columns of the first row demonstrate an abrupt catch through a large decrease in the forget gate value; whereas the latter two columns of the first row demonstrate a sudden release through a large increase in the forget gate value.

Existing competitive results on Perm-SeqMNIST He

Table content: <table><tr><td>Model</td><td>Accuracy (%)</td><td>Source</td></tr><tr><td>RBN DecayNet-LSTM</td><td>Best: 97.8 ; Mean: 97.4</td><td>This paper</td></tr><tr><td>Temporal Convolutional Network</td><td>97.2</td><td>Bai et al. (2018)</td></tr><tr><td>RBN Zoneout LSTM</td><td>95.9</td><td>Krueger et al. (2016)</td></tr><tr><td>3D tLSTM+CN</td><td>95.7</td><td/></tr></table>



Table content: <table><tr><td>: Results</td></tr></table>

Peer review (iclr2019):
 This paper analyses the internal dynamics of an LSTM, focusing on the cell state as being the most important component, and analyses what directly influences the contents of the cell state using difference equations. The authors note that at any timestep, the output cell state is the sum of (previous cell state * forget) and (input gate * input). The former can only shrink or maintain the cell value, which the authors label *catch* and the latter can increase the magnitude, labelled *release*. The authors show that for a single neuron, with chosen values for the forget gate and inputs, consistent growth or consistent shrinking of the cell state can be observed. When the forget is large, say 0.9, the input gate can be anywhere in the range 0.5, 1.0] and still produce growth in the cell value. For forget gate = 0.25, an even larger range of input gate values all produce a shrinking cell value. Due to the forget gate seemingly having a stronger effect than the (input gate * input) component, the authors propose to hard wire the forget gate to produce a continuous and monotonic decrease, producing the DecayNet. The rate of this decay is controlled by a learned function of the input and previous hidden state, with some shifting in order to maintain a monotonic decrease. Each forget neuron will decrease at a different rate through the processing of a sequence, leading to sections of the cell state which will decay slowly and sections which will decay quickly. The authors perform two sets of experiments. The second is sequence classification with the standard *pixel by pixel* permuted sequential MNIST, in which they show a new SOTA with using Recurrent Batch Norm combined with DecayNet. They also demonstrate a DecayNet with fewer parameters producing roughy the same median performance as a normal LSTM but with lower variance. The first experiment is described as "image classification", with MNIST and Fashion-MNIST. This section is unclear to me, I had initally assumed that the data would be fed in one pixel at a time, but due to the presence of the other experiments I presume this is not the case. It is not clear what the *time* dimension is in how the RNNs are applied here, if not through some ordering of pixels. If the entire image is presented as a flattened input, and the time dimension is iterating through the dataset, then there is no reason to use an RNN here. More detail must be added here to make it clear exactly how these RNNs are being applied to images - the text says the softmax layer is produced from the final hidden state, but without the information about how the different hidden states are produced for a given training example this not meaningful. I can imagine that both tasks are pixel by pixel, and the only difference is whether to apply the permutation.. but that is my guesswork. In general I find this paper an interesting idea, reasonably well communicated but some parts are not clear. All the experiments (as far as I can tell) work on fixed length sequences. One advantage of an LSTM is that can run onnline on arbitrary length data, for example when used in a RL Agent. In those circumstances, does learning a fixed monotonic delay on the forget gate make sense? I would guess not, and therefore I think the paper could be more explicit in indicating when a DecayNet is a good idea. There are definitely tasks in which you want to have the forget gate drop to zero, to reset the state, and then go back up to 1 in subsequent timesteps to remember some new information. Presumably the monotonic delay would perform poorly. Is DecayNet appropriate only when you have fixed length sequences, where the distribution of *when does relevant information appear in the input* is fixed? These questions make me doubt the generality of this approach, whereas "this reformulation increases LSTM modelling power ... and also yields more consistent results" from the abstract reads like this is a strictly better LSTM. A much wider variety of experiments would be required to justify this sentence. It would be interesting to see some diagrams of forget gate / cell state changes throughout a real task, ie a graph with `k` on the x axis. The presentation of the new forget gate in "System 2" is clear in terms of being able to implement it, but it*s not intuitive to me what this actually looks like. The graphs I suggest might go a long way to providing intuition for readers. Overall while I like the spirit of trying to understand and manipulate LSTM learning dynamics I am recommending reject. I do not think the paper sufficiently motivates why a monotonic decay should be good, and while the new SOTA on permuted MNIST is great, I*m concerned that the first experiments are not reproducable, as detailed previously in this review. All hyperparameters appear to be present, so this paper would be reproducable, except for the NIST experiments. General recommendations for a future resubmission: * Clarify description of first MNIST experiments, and how they are different from permuted MNIST. * Experiments on a wider variety of canonical RNN tasks - Penn Treebank is an obvious contender. * Some mention of in what situations this is obviously not a good model to use (RL?) * More intuition / visualisations as to what the internal dynamics inside DecayNet look like, vs normal LSTM. * Devote less space to the initial dynamics analysis, or modify to be representative of a real task. This part was interesting on first read, but the only thing I think it really proves is *when we artificially choose the input values things get bigger or smaller*. The important thing is, what actually happens when training on a task that we care about - if the same catch and release dynamics are observable, then that makes the idea more compelling. Notes and suggestions: I feel the notation would be clearer if instead of k = 1 .. D, this index was t = 1 ... T. This makes it cleare that s_k is not the k*th item in the array, but rather than whole activation array at a specific time. The notation .sigma_t could then be replace with .tanh. "We replace .sigma_t with sin for an ergodic delay over time": as this is a new gate for the forget gate, should this be .sigma_s? One DL rule of thumb heard relatively often is to simply initialise LSTM forget gate biases to 1, to "remember more by default". As this is a (much simpler) way of trying to influence the behaviour of the gate, and it anecdotally improves data efficiency, it is worth mentioning in the paper.