Year: 2018

Abstract:
It is unclear whether the extensively applied long-short term memory (LSTM) is an optimised architecture for recurrent neural networks. Its complicated design makes the network hard to analyse and non-immediately clear for its utilities in real-world data. This paper studies LSTMs as systems of difference equations, and takes a theoretical mathematical approach to study consecutive transitions in network variables. Our study shows that the cell state propagation is predominantly controlled by the forget gate. Hence, we introduce DecayNets, LSTMs with monotonically decreasing forget gates, to calibrate cell state dynamics. With recurrent batch normalisation, DecayNet outperforms the previous state of the art for permuted sequential MNIST. The Decay mechanism is also beneficial for LSTMbased optimisers, and decrease optimisee neural network losses more rapidly.

heading: 1 INTRODUCTION
text: Neural networks are powerful universal approximators that are difficult to interpret. This paper presents a numerical study on the forward pass of the long-short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) recurrent neural networks (RNNs). We treat LSTMs as systems of difference equations and present visualisations to study and to clarify consecutive transitions in the cell state dynamics. We introduce DecayNets as an addition to the LSTM architecture – DecayNets have monotonic decays in forget gates and enjoy smooth transitions in their cell state values.
LSTMs are versatile models that have been used to advance the state-of-the-art for a variety of machine learning problems. This covers handwriting recognition (Graves et al., 2009), speech recognition (Sundermeyer et al., 2012), and text modelling (Kim et al., 2016). There are an abundance of work that extends on the basic LSTM architecture. Popular extensions include the attention mechanism (Bahdanau et al., 2014) and the bidirectional design (Schuster & Paliwal, 1997). It is also not uncommon to blend LSTMs into vision tasks to expand network practicalities (Xu et al., 2015).
However, it is unclear whether LSTMs are optimal architectural designs for RNNs (Jozefowicz et al., 2015). Additionally, the significance of their individual components are unclear, and empirical explorations are required for understanding their utility in real-world data (Karpathy et al., 2015). This lack of interpretability limits our ability in designing better and more transparent networks.
RNNs repeatedly integrate new observations into implicit hidden variables to model temporal representations of sequences. Like dynamical systems, the network undergoes iterative computations on a same set of operators. The transitional dynamics can be visualised through cobweb diagrams (Glendinning, 1994); and numerical analyses allow us to predict shrinkage and growth in variables.
We show that the updating scheme of the LSTM cell state volatilely alternates between a catch and a release phase. The former controls the shrinkage while the latter controls the growth of cell state magnitudes. Our study reveals that the forget gate dictates the dynamical alterations of the cell state. We hence introduce DecayNets, LSTMs with monotonically decreasing forget gates, for stabilising cell state propagation and for making networks more interpretable.
With recurrent batch normalisation (Cooijmans et al., 2016), DecayNets outperform previous state of the art results (best: 97.8%; mean: 97.4%) on the fixed-length permuted pixel-by-pixel (sequential) MNIST task (Le et al., 2015; LeCun et al., 1998). For the arbitrary-length learning-to-learn task (Andrychowicz et al., 2016), LSTM-optimisers with the Decay mechanism decrease multi-layer perceptron (MLP) losses more rapidly, and converge to lower and less varied MLP losses.

heading: 2 THE VANILLA LSTM
text: DecayNets will be frequently compared to conventional LSTMs. The most commonly implemented LSTM RNN was proposed by Graves & Schmidhuber (2005), and this paper will refer to the architecture therein that paper as the LSTM.
LSTMs are recursive systems driven by its input xt, which has the data-size dimensionality M . The network propagates for a data-length of D units of time, with the subscript t as the recursive instance, i.e., t = 1 · · ·D . Each instantaneous computation (in abbreviated form) encompasses
System (1): the vanilla LSTM the gated variables: ft, it,ot = σ(W{F ,I ,O}xt + WR{F ,I ,O}qt−1 + b{F ,I ,O}), the internal input: at = tanh(WAxt + WRAqt−1 + bA), the cell state: st = ft st−1 + it at, the hidden state: qt = ot tanh(st), with the dimensionality: xt ∈ RM , ft, it,ot,at, qt, st, b{F,I,O,A} ∈ RN , and
W{F,I,O,A} ∈ RN×M , WR{F,I,O,A} ∈ RN×N . The input xt and the hidden state qt contribute to other variables via Ws and via WRs. The dimensionality N is user-defined and is the operator for element-wise product. Along with biases bs, the regressive contributions are activated by either the sigmoidal function σ or the hyperbolic tangent function tanh. During the update of the cell state st, the forget gate ft relinquishes fragments of its previous instance st−1, and the input gate it fine-tunes replenishment from the internal input at. Finally, the variable qt, which also serves as the output of the network, is a transformation of st synchronised with the output gate ot.

heading: 3 LSTMS AS DIFFERENCE EQUATIONS
text: Difference equations are recurrent relations with discrete propagation; and the term dynamic refers to the dependency between time and geometric space. Dynamical behaviours of a difference equation can be studied to determine whether variables will grow or shrink. This allows us to explain behaviours of individual LSTM variables and clarify network mechanics.
This section discusses three key concepts – the importance for studying the LSTM cell state; the two dynamic regimes for cell state values to grow and to shrink; and that the forget gate can be controlled to yield interpretable and predictable cell state motions. Cobweb diagrams will be used to support concept visualisations, and a brief introduction to the technique can be found in Appendix A.

heading: 3.1 THE CELL STATE REVISITED
text: This paper postulates the cell state as the source of LSTMs’ modelling capability. Furthermore, we regard gated variables as secondary variables. Our justifications are as follows.
First, gated variables only exist to read and to set the hidden variables of qt and st. That is, they exist to assist LSTMs to memorise better, but are not a part of the memories. Second, tanh is a one-to-one function, thus dynamic properties of qt are inherited from st. Last, at is included in the update of st, and acts as a small component to the grander dynamic entailed in st. For these reasons, we postulate the cell state as the most important variable and study it as a difference equation.

heading: 3.2 A PIECEWISE ANALYSIS ON THE CELL STATE
text: We study the cell state, st = ft st−1 + it at, separately as sp = ft st−1 and sq = it at. Since we have asserted gated variables as secondary variables, we can think of ft and it as constant vectors with elements between 0 and 1. That is, individual dimensions of sp and sq have cobweb outlines that are analogous to Figures 1(a) and 1(b) respectively.
For the rest of this paper, we will refer to the marginal dimensions of ft, it, at, xt, and st as fmt, imt, amt, xmt, and smt, respectively. For instance, each dimension of sq is imt amt and inherits the diverse characteristic outline of 1(b) through tanh from amt.
Consecutive transitions in network variables are shown as the dotted trajectories in Figure 1. They start from the rhombuses and ends on the circles. The characteristic outlines of the sub-dynamics are the solid lines, and the dashed lines represent the 45-degree y = x lines.
The two subfigures address qualitatively different dynamic regimes. The origin of Figure 1(a) is a sink that attracts propagation trajectories and decreases variable magnitudes. Conversely, the origin of 1(b) is a source that repels propagation trajectories and increases variable magnitudes. We refer to 1(a) as the dynamic catch regime; and refer to 1(b) as the dynamic release regime.
The update of the LSTM cell state is thus a composition of two opposing modifiers. With one that dedicates to its growth, and the other specialised in its shrinkage. It is thus crucial to understand which one, between sp and sq , serves as the more dominant evolution component. We will then reformulate the more dominant component with deterministic properties to redefine the network with interpretable and affirmative qualities.
A key difference between the opposing regimes of Figure 1 is the boundedness of the characteristic outlines. The sub-dynamic of sq is limited between±1, while that of sp is unbounded. This implies that sq contributes less to the additive relationship of sk, and is less impactful than sp. With the dominant component identified, our next target is to understand how sp affects vanilla LSTMs. The insight will allow us to propose sensible modifications to make LSTM mechanics more interpretable.

heading: 3.3 THE FORGET GATE AND THE CATCH-AND-RELEASE DYNAMICS
text: Based on the two insights of, first, sp = ft st−1 serving as the dominant modifier to st, and second, as each of its dimensional outline sm(t−1) being modified by the gradient fmt, we conjecture that the forget gate controls the overall gradient of the characteristic outline of the cell state. That is, ft predominantly controls the stability of the origin, and dictates the dynamic alterations between the catch and the release phases. As a consequence and regardless of sq , values of the cell state are more likely to grow with large ft, and more likely to shrink with small ft.
We use Figure 2 to support these inferences. The first row simulates the near-random dynamics of a vanilla LSTM cell state. We present rows two and three to show stark contrasts, and advantages, of generating predictable cell state propagation with controllable forget gate values.

heading: 3.3.1 THE NEAR-RANDOM DYNAMICS OF A VANILLA LSTM CELL STATE
text: Gated units of LSTMs are not hardwired with deterministic propagation. As a result, they evolve in a near-chaotic fashion. The first row of Figure 2 simulates behaviours of st under unregulated ft.
The three subfigures of the row serves as consecutive instances of a propagation, i.e., t =[1 . . . 3]. The presentation follows Figure 1, and dot-dash lines are appended to instances t = 2 and 3 to show the characteristic outlines of their previous instances. Parameters fm(1...3) = [1, 0.25, 1] and im(1...3) = [1, 0, 0.75] are carefully chosen to exaggerate impacts brought to the vanilla LSTM cell state by the near-chaotic evolution of gates. Real scenarios and experimental set-ups that justify the choice of these demonstrative parameters can be found in Appendix B.
These settings lead to sudden catches and abrupt releases. The disordered dynamics make it difficult to predict growth and shrinkage in cell state magnitudes. It is also unclear how such drastic behaviours can assist us to understand the learned features of a trained LSTM.

heading: 3.3.2 INTERPRETABLE CELL STATE DYNAMICS UNDER CONTROLLED FORGET GATES
text: Let us now consider cell state dynamics under controlled fts. The second row has fmt = 0.9 with im(1...3) = [1, 0.75, 0.5]; and the third row has fmt = 0.25 with im(1...3) = [1, 0.75, 0.15]. Consistent growth and consistent shrinkage are observed in rows two and three respectively.
The magnitudes evolve monotonically because the controlled forget gates expose cell states to homogeneous dynamic regimes. Large constant fmts expose the cell state to consistent (global) releases; whereas small constant fmts expose smt to global catches.
Dynamic reversals may occur in st through sq , but only under the unlikely incidents where combinations of extreme values of it and at are simultaneously presented. In addition, the dynamic reversals are merely temporary (local) due to the boundedness of values in sq .
The dynamic regimes of sp and sq behave analogously to the respective difference equations of (a): xt = T (xt−1) = 0.5xt−1 and (b): xt = T (xt−1) = tanh(3xt−1) . The propagation, in dots, start from rhombuses and end on circles. The dashed lines represent y = x and the solid lines denotes the characteristic outlines of y = T (x ). Vertical mappings from y = x to y = T (x ) reflect instantaneous variable computations of T (xt−1); and horizontal mappings from y = T (x ) to y = x represent the recursive nature of difference equations to re-insert the last output as the new input xt = T (xt−1). The origin of (a) is a sink which attracts propagation and diminishes variable magnitudes; that of (b) is a source which repels propagations and increases variable magnitudes.
t =1
t =2
t =3
Three qualitatively different scenarios for cell state propagation are investigated. The subfigures follow the presentation format of Figure 1; additional dot-dash lines are appended as the characteristic shape of the t − 1th instance. The characteristic shape of interest is the update of the cell state y = T (x ) = fmt x + imt tanh(x ). From top to bottom, the rows represent for cell state dynamics under an unregulated LSTM, under a large constant forget gate, and under a small constant forget gate, respectively. The latter two scenarios exhibit monotonic propagation because the constant forget gates expose cell states to homogeneous dynamical regimes. The motions are thus more interpretable than those of the unregulated LSTM, where abrupt shrinkage and sudden growth are observed.

heading: 4 DECAYNETS
text: This section presents DecayNets1, and calibrate cell state dynamics according to mathematical insights therein Subsection 3.3. From Figure 2, it is evident that forget gates deprived of random evolution yield predictable and stable cell state propagation. For this reason, we propose the idea to hard-wire forget gate outputs with a continuous and monotonic decrease.
The DecayNet modifies the forget gate according to
System (2): The Decay mechanism on the LSTM forget gate the forget-polar input: pt = pt−1 − π2D 1 6 (WFxt + WRFqt−1 + 3)
purposedly presented this way
with p0 = π2 · 1̃,
forget gate elements: fti = Φ(pti) =
{ 1 for pti > π 2
0 for pti < 0 sin(pti) otherwise
for i in 1, 2, . . . N ,
and inherits all remaining variables and their corresponding dimensionalities from System (1). Note, no new learnable parameters are introduced to this system.
The forget gate is initialised as a vector of ones, with equivalent portions of the said initial ones set to be lost over the iterative instances of the LSTM. That is, we set . f0i = sin(p0i) = sin( π 2 ) = 1, and that, over time, we remove a maximised cumulative amount of 1, and a minimised cumulative amount of 0, to constrain the final value ftD ∈ [0, 1]. We implement the sinusoidal function as the “activation function” of the forget gate. In addition, we engineer “pre-activated neurons” with values within the bounds of [0, π2 ]. Let us elaborate.
We replace σ with sin to decay equivalent portions over time. The available amount of instantaneous decay needs to be uniform throughout time to ensure that all instantaneous lost in values are equally important. The sigmoidal outline of σ yields less changes when magnitudes of pre-activated values are extreme. We remove this undesirable property with the substitution of sin with radian-like inputs.
Elements of the pre-activated neurons are in the bound of [0, π2 ]. Empirically, we observed that distributions of WFxt+WRFqt−1 , with entries of WF and WRF initialised with uniform sampling within the interval of ±1, are bounded within the limits of [−3, 3].2 That is, we have . WFxt + WRFqt−1 ∈ [−3, 3] and . π2 1 6 (WFxt + WRFqt−1 + 3) ∈ [0, π 2 ].
The design in mind is thus, for each instance, a maximum amount of π2D is lost from pti ’-s initialised π 2 . In other words, over an input-length of D units of time, an accumulation, minimised at 0 and maximum at π2 , is lost in the forget-polar input. The former induces the final value of pDi = π 2 (equivalently fDi = 1), and the latter gives pDi = 0 (equivalently fDi = 0).
DecayNets learn to dichotomise cell state values conditioned on the input shown. One subset of the cell state maintains large magnitudes, whereas the other decreases to small magnitudes. The former subset corresponds to dimensions of the forget gate that maintain their large magnitude over time; whereas the latter subset corresponds to those that diminish.
More importantly, the monotonic decrease in the forget gate serves as an interpretation to LSTM’s inferential decision mechanism. This is because the update of the cell state, st = sp+sq = ft st−1 + it at, is composed of recurrent memory sp and feedforward memory sq . The irreversible decay indicates that, dimensions of the cell state, that correspond to forget gates that diminish, gradually lose their recurrent nature over time, and effectively function as feedforward neurons. Large forget gate values thus serve as clear indicators for, those neural dimensions, that the DecayNet prefer to reiterate, for the input-driven optimisation.
1Implementation of our algorithm will be available at https://github.com/anonymous author/decaynet/ 2New/revised text are in brown to assist the reviewers, and will be resumed to black after the revision period.

heading: 5 RELATED STUDIES
text: The forward pass of LSTMs (and RNNs) consists high dimensional variables with coupled relationships; and is mostly understood with observational studies and search studies. The former explores practical network usages, and the latter finds optimal combinations of network variables.
Observational studies suggest that cell states develop specific functionality. In text (Karpathy et al., 2015), they are observed to act as line length counters and quotation machines; whereas in speech (Wu & King, 2016), they are highly correlated to the Mel-Cepsrtal coefficient. Search studies (Greff et al., 2017; Chung et al., 2014; Jozefowicz et al., 2015) found that, no particular combinatory sets of variables yield an architecture, that consistently outperforms LSTM in all experimental conditions.
Though rigorous and informative, these studies are based on exhibited qualities of optimised LSTMs. Karpathy et al. (2015) conjectured the functionality of trained LSTM cell states through examining post-activated cell state values tanh(st). Greff et al. (2017) observed that gated recurrent units (GRUs) (Cho et al., 2014) are capable of performing well on tasks unoptimised by those LSTMs deprived of output gates. Hence, they conjectured that GRUs’ coupling of the input gate and the forget gate, and the output gate of LSTMs prevent unbounded cell states. In contrast, our theoretical mathematical approach therein Sections 3 and 4 are conducted prior to the training of LSTMs.
Mathematical properties of neural networks are mostly studied through the lens of statistical learning theory. This paper took an alternative approach. We treated neural networks as dynamical systems, and offered the perspective of understanding LSTMs as systems of difference (discrete) equations.
Statistical studies and dynamical studies are complementary. Burger & Neubauer (2003) took an inverse problem approach and integrated Tikhonov regularisations to enhance neural network stability. Tallec & Ollivier (2018) offers another theoretical study on the forward pass. Incremental updates of LSTMs are treated as differential equations, and the effects of multiple scales are analysed.
The mathematical analyses therein this paper differs to existing literature because, we propose modifications on a specific variable, to yield an affirmative / deterministic outcome – DecayNets have monotonically decreasing forget gates. In addition, DecayNets initialise their forget gate values as 1s, which is similar to the effect of the common practice of setting LSTM forget gate biases to 1s (Gers et al., 1999), for yielding large initial forget values and for “remembering more by default”.
A hard regularisation emerges from the reformulation. As mentioned in Section 4, the network learns to irreversibly suppress the recurrent nature of a selection of cell state values conditioned on the input shown. This differs to soft regularisations such as Dropout (Srivastava et al., 2014) and Zoneout (Krueger et al., 2016), where the respective nullified and frozen activation are recoverable.

heading: 6 EXPERIMENTS AND RESULTS

heading: 6.1 PERMUTED SEQUENTIAL MNIST IMAGE CLASSIFICATION
text: We apply DecayNets and LSTMs on image classification with sequential inputs. The data of choice is MNIST (LeCun et al., 1998), a database of handwritten digits of numbers [0-9], with 60K and 10K images for training and testing respectively. MNIST images are in 28 × 28-pixel formats, we present each pixel, one at a time, as a 784(= 28 × 28)-unit pixel-by-pixel sequence to the RNNs therein this subsection. Following Le et al. (2015), we apply a fixed random permutation over the pixels for setting up the task commonly known as permuted sequential MNIST (Perm-SeqMNIST).
RNNs of this subsection have a single layer of 100 hidden dimensions and are trained on the RMSProp optimiser (Tieleman & Hinton, 2012) over 150 epoches. The learning rate is 0.001, with 0.9 momentum and no weight decay. Gradient clipping is applied at 1 to avoid exploding gradients (Pascanu et al., 2012). All weights are initialised with uniform sampling within the interval of ±1, and all biases are initialised as zeros. No normalisation is performed on the data prior to training, and a softmax classifier is attached to produce prediction from the final hidden state qD.
The best prediction accuracy for unregulated RNNs is 92.1% for DecayNet and 89.5% for LSTM (which roughly matches the baseline model of 89.8% in Krueger et al. (2016)). We then integrated recurrent batch normalisation (RBN) (Cooijmans et al., 2016) to DecayNet and found that both the best performing RBN DecayNet, 97.8%, and the average simulatory performance, 97.4%, yield
Purple and blue denotes DecayNet, while green and yellow is used for LSTM. Performances of RBN RNNs are thick solid lines over shades. Bottoms and tops of the shades are the worst and the best performance of all simulations; and the thick central line is the average performance.
better results than the state of the art performance reported in Bai et al. (2018) at 97.2%. Table 1 presents a list of competitive results in existing literature, and Figure 3 shows the test set accuracy. Purple and blue denote DecayNets, while yellow and green are used for LSTMs. The best unregulated networks are dashed and dotted; and 10 simulations of RBN RNNs are presented as thick solid lines on shades. Bottoms and tops of the shades are the worst and best performances of all simulations; the central lines are the mean performances. The RBN LSTM results differ to those reported in Cooijmans et al. (2016) and Krueger et al. (2016), we speculate this as a result of differences in the initialisations of weights and biases.
A similar task, of image classification with row-by-row inputs, is presented in Appendix C. There, DecayNets outperform LSTMs with less significant differences, but yield more consistent results.

heading: 6.2 LSTMS AS OPTIMISERS
text: The second task follows Andrychowicz et al. (2016) and update weights of multi-layer perceptrons (MLPs) with LSTMs. The paper showed that MLP-optimisees trained with LSTM-optimisers yield marginally lower losses than those trained with the hand-crafted optimiser of ADAM (Kingma & Ba, 2014). Here, we show that the Decay mechanism helps the already successful LSTM-optimiser decrease MLP losses more rapidly, and converge to lower and less varied MLP losses.
We train two-layer MLPs for MNIST classification and update their weights θ with LSTMs. The MNIST images are normalised with mean and standard deviation (MSD) = (0.1037, 0.3081), and are organise as vector inputs of dimensionality 784. The first MLP layer has weights of dimensionality R784×20; the second layer, a softmax layer, has weights of dimensionality R20×10. The update . θt = θt−1 −αt∇θt−1Lt uses learning rate αt and ∇θt−1Lt, the gradient of the optimisation loss with respect to parameter θt−1. We use LSTM-optimisers to infer αt given the element-wise product of θt and their corresponding gradients Gθt−1 . That is, for each instance, in brevity, . αt, qt, st = LSTM(θt−1 Gθt−1 , qt−1, st−1). The optimisers are trained according to Andrychowicz et al. (2016), which we reiterate in Appendix D. For justifications and details on the choices of the input and the output, see Appendix E.
The Decay mechanism allows LSTM-optimisers to update their hidden variables for multiple steps before they create learning rates. At the beginning of each instance, the forget polar input is reset as
We compare various optimisers to update the weights of MLP-optimisees for learning MNIST classification. Subplot (a) shows that, both LSTM-based optimisers, yellow for LSTMs and purple for DecayNets, yield MLPs with marginally lower losses than their rival hand-crafted optimisational schemes. Subplot (b) shows that the appended Decay mechanism allows LSTMs to decrease MLP losses more rapidly.
the maximal values of π2 · 1̃ (effectively ft = 1̃). For each instance, the input of θt−1 Gθt−1 monotonically decrease the forget gate and update the hidden variables as according to . qt− 12 , st− 12 = DecayNet(θt−1 Gθt−1 , qt−1, st−1) and . αt, qt, st = DecayNet(θt−1 Gθt−1 , qt− 12 , st− 12 ). That is, the learning rate is created only after the gated units and hidden variables propagated for two times. Such an internal propagation can be pro-longed, for example, to twenty steps.
A comparison of optimiser performances is presented in Figure 4(a). The figure shows MLP losses after every update of a size 32 mini-batch. The colours of green, grey, blue, yellow, and purple are used to denote stochastic gradient descent with learning rate 0.05, ADAM with learning rate 0.002, RMSProp with learning rate 0.001, a trained LSTM-optimiser, and a trained DecayNet-optimiser, respectively. LSTM-based optimisers outperform hand-crafted rivals by a significant margin, and their MLP losses converge after the 60th update. Hence, we treat losses from the 80th to the 100th update as losses of optimised MLPs. Using these data, two-independent sample t-test found statistical significance that DecayNet-optimised MLPs yield less losses than LSTM-optimised MLPs, with MSD= (−0.13, 0.01) by p-value < 0.001. In addition, we present Figure 4(b), a sequence of violin-plots, to examine the running performance of the LSTM-based optimisers. With colours inherited from 4(a), each violin-plot aggregates twenty iterative updates of loss data, i.e., the first is of the 1st to the 20th update, and that the second is of the 21st to the 40th update. The figure indicates that DecayNet-optimisers minimise MLP losses more rapidly, and converge to less varied losses.

heading: 7 CONCLUSION
text: This paper proposes DecayNets, LSTMs with monotonically decreasing forget gates. Motivations and modifications are based on theoretical mathematical clarifications of the LSTM cell state mechanics. The update of the cell state is studied as a difference equation, and we have showen that the update alternates between a catch and a release phase to control shrinkage and growth in its values. We have found that the dynamic alteration is predominately controlled by the forget gate and propose to stabilise the cell state through a monotonically decreasing forget gate. The reformulation is easy to implement and introduces no extra learnable parameters. With RBN, DecayNets achieved better results than previously reported state of the art performance on the Perm-SeqMNIST task. Our experimental results have showen that the Decay mechanism also functions as a promising additional feature for those LSTMs casted as optimisers to update optimisee MLP weights.
However, the most important quality of the Decay mechanism is that it is deterministic. It would be interesting to treat the gradually decaying patterns as features to analyse interactions among speech utterances and other natural language processing tasks (Suominen, 2014). Such a similar deterministic-ness may be hard-wired to GRUs. However, the update of the GRU cell state is more complicated than its LSTM counterpart, and separate dynamical analyses must be conducted.
LSTMs are still opaque. The authors of this manuscript believe that more mathematical analyses should be focused on component sq of the cell state. We believe this less controllable, and more diverse sub-dynamic of the cell state controls bifurcations (Glendinning, 1994) in LSTM dynamics.

heading: D DESCRIPTIONS OF THE LEARNING TO LEARN TASK
text: The learning to learn task follows the paper of Andrychowicz et al. (2016). This section provides some of our own training details, which reiterates some training details provided in the mentioned paper.
Weights of the MLP-optimisee are updated via . θt = θt−1 −αt∇θt−1Lt with LSTM-optimisers used for inferring learning rates αt. We considered two-layer MLPs, with dimensionalities 784× 20× 10; which we denote the overall learnable parametric count as rH . All weights are initialised randomly from a uniform distribution with the bounds of [−0.1, 0.1], and the biases are initialised as zero vectors. This controlled initialisation is to ensure all MLP-optimisees start as if they were empty canvases, and for maximising the utility of the LSTM-optimisers.
The input of the LSTM, θt−1 Gθt−1 , is the element-wise product of the last instance of MLP parameters and their corresponding gradients. That is, . αt, qt, st = LSTM(θt−1 Gθt−1 , qt−1, st−1). The inputs and the outputs for LSTM-based optimisers differ to those given in Andrychowicz et al. (2016). Justifications are provided in Appendix E.
The dimensionality of the vector input θt−1 Gθt−1 is the total count of MLP learnable parameters, rH , which is a large constant. In order to simplify training for LSTM-optimisers, the LSTMs have a small feedforward layer, located prior to the computation of the gated units, with a learnable matrix of dimensionlity RrH×28, for the purpose of down-sizing the input. Thus, the LSTM-based optimisers have a ‘formal’ input dimensions of 28; in addition, the hidden dimensions are chosen to be 64.
Two separate layer normalisations are applied in the LSTM-optimisers. One to post-process the input-to-hidden connections of Wxt, and the other to post-process the hidden-to-hidden connections of WRqt. Trained LSTM-optimisers unfold every instance to create learning rates for the MLP-optimisees. The learning rates αt are of dimensionality RrH , and are created from the matrix multiplication of the LSTM hidden state and a learnable up-size matrix of dimensionality R64×rH .
The LSTM-based optimisers are trained via ADAM with learning rate 0.001. Each newly initialised LSTM-based optimiser learns to update 5 controlled initialised MLP-optimisees. Weights of the MLP-optimisees are updated for 100 consecutive instances. For each update, a size 32 mini-batch is provided to the MLP-optimisees, with records of the differences, between the cross entropy losses, prior and after, the update of MLP-optimisee weights, recorded. For every 20 update instances, of the 100 consecutive instances, from the optimisers to the optimisees, an accumulation of the differences in the recorded losses are used to update weights therein the LSTM-based optimisers.

heading: E JUSTIFICATIONS ON OUR CHOICES OF INPUTS AND OUTPUTS FOR THE LEARNING TO LEARN TASK
text: Justification on inputs:
GradientsGθt−1 of the learning to learn task are the products of the pre-processed gradient provided in Andrychowicz et al. (2016). In that paper, the original gradient is given as∇, and the transformations are given as
. ∇t → { ( log(|∇|)10 , sgn(∇) ) if |∇| ≥ e −10,
( −1, e10∇ ) otherwise.
The authors of this manuscript understood the first pair of gradients ( log(|∇|)10 ,−1) as an emphasis on large magnitude gradients. Its counterpart pair of (sgn(∇), e10∇) provides an emphasis on small magnitude gradients. The first pair consistently produces negative values, whereas the second pair has values of either signs, which mostly dependent on their larger elements. That is, the sign of the second pair could serve as cues, for the directions, to travel sensibly, in search for the solution space for MLPs. In addition, the magnitude of the first pair could serve as cues for the magnitudes of the strides in those directions. Thus, it is not necessary to feed the pre-processed gradients as separate inputs to the LSTM-optimisers. This is because the product of the two retains property of both transformations.
As an additional note, the MLP parameters are element-wise-producted with Gθt−1 , the elementwise product of the pre-processed gradients, to prevent the learning of redundant dependencies between MLP parameters and their gradients during the training of the LSTM-optimisers.
Justification on outputs:
Therein this paper, the outputs of the LSTM-based optimisers are chosen as the learning rates of the MLP weights. This decision is based on two paper – on the original learning to learn paper of Andrychowicz et al. (2016), and on the learning to learn for few-shot learning paper of Ravi & Larochelle (2016).
In Andrychowicz et al. (2016), the authors used LSTM-optimisers to prepare the whole updating content for the weights of the optimisees. However, the output is not simply added towards the existing weights of the optimisees – the LSTM outputs are rescaled by a factor of 0.1. The authors of this manuscript speculate this as a consequence of activation function saturation. That is, the output values are close to extrema of the activation functions. Furthermore, we speculate this as a consequnece of the high dimensionality of rH . This serves as another reason for our productedinput, for avoiding excessive optimiser input contributions that stem from the redundant synaptically connected dimensionality.
In Ravi & Larochelle (2016), the authors used LSTM-optimisers to prepare for a set of learning rates and a set of “anti-learning rates” for the weights of the optimisees. The anti-learning rates first relinquishes fractions of the existing MLP weights prior to updating the remaining said weights.
We have concerns for both implementations. We want to avoid a potentially saturated output, and we want an updating rule for MLPs that are mathematically sound. For this reason, we choose to only produce the learning rates via the LSTM-based optimisers. Our experimental results, shown in Figure 4, closely resemble those in Andrychowicz et al. (2016).

Review summaries:
This paper analyses the internal dynamics of an LSTM, focusing on the cell state as being the most important component, and analyses what directly influences the contents of the cell state using difference equations. The authors note that at any timestep, the output cell state is the sum of (previous cell state * forget) and (input gate * input). The former can only shrink or maintain the cell value, which the authors label 'catch' and the latter can increase the magnitude, labelled 'release'.