Can recurrent neural networks warp time?

Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues. We prove that learnable gates in a recurrent model formally provide quasiinvariance to general time transformations in the input data. We recover part of the LSTM architecture from a simple axiomatic approach. This result leads to a new way of initializing gate biases in LSTMs and GRUs. Experimentally, this new chrono initialization is shown to greatly improve learning of long term dependencies, with minimal implementation effort. Recurrent neural networks (e.g. (Jaeger, 2002)) are a standard machine learning tool to model and represent temporal data; mathematically they amount to learning the parameters of a parameterized dynamical system so that its behavior optimizes some criterion, such as the prediction of the next data in a sequence. Handling long term dependencies in temporal data has been a classical issue in the learning of recurrent networks. Indeed, stability of a dynamical system comes at the price of exponential decay of the gradient signals used for learning, a dilemma known as the vanishing gradient problem (Pascanu et al., 2012; Hochreiter, 1991; Bengio et al., 1994). This has led to the introduction of recurrent models specifically engineered to help with such phenomena. Use of feedback connections (Hochreiter & Schmidhuber, 1997) and control of feedback weights through gating mechanisms (Gers et al., 1999) partly alleviate the vanishing gradient problem. The resulting architectures, namely long short-term memories (LSTMs (Hochreiter & Schmidhuber, 1997; Gers et al., 1999)) and gated recurrent units (GRUs (Chung et al., 2014)) have become a standard for treating sequential data. Using orthogonal weight matrices is another proposed solution to the vanishing gradient problem, thoroughly studied in (Saxe et al., 2013; Le et al., 2015; Arjovsky et al., 2016; Wisdom et al., 2016; Henaff et al., 2016). This comes with either computational overhead, or limitation in representational power. Furthermore, restricting the weight matrices to the set of orthogonal matrices makes forgetting of useless information difficult. The contribution of this paper is threefold: ∙ We show that postulating invariance to time transformations in the data (taking invariance to time warping as an axiom) necessarily leads to a gate-like mechanism in recurrent models (Section 1). This provides a clean derivation of part of the popular LSTM and GRU architectures from first principles. In this framework, gate values appear as time contraction or time dilation coefficients, similar in spirit to the notion of time constant introduced in (Mozer, 1992).  From these insights, we provide precise prescriptions on how to initialize gate biases (Section 2) depending on the range of time dependencies to be captured. It has previously been advocated that setting the bias of the forget gate of LSTMs to 1 or 2 provides overall good performance (Gers & Schmidhuber, 2000; Jozefowicz et al., 2015). The viewpoint here 1 ar X iv :1 80 4. 11 18 8v 1 [ cs .L G ] 2 3 M ar 2 01 8 Published as a conference paper at ICLR 2018 explains why this is reasonable in most cases, when facing medium term dependencies, but fails when facing long to very long term dependencies.  We test the empirical benefits of the new initialization on both synthetic and real world data (Section 3). We observe substantial improvement with long-term dependencies, and slight gains or no change when short-term dependencies dominate. 1 FROM TIME WARPING INVARIANCE TO GATING When tackling sequential learning problems, being resilient to a change in time scale is crucial. Lack of resilience to time rescaling implies that we can make a problem arbitrarily difficult simply by changing the unit of measurement of time. Ordinary recurrent neural networks are highly nonresilient to time rescaling: a task can be rendered impossible for an ordinary recurrent neural network to learn, simply by inserting a fixed, small number of zeros or whitespaces between all elements of the input sequence. An explanation is that, with a given number of recurrent units, the class of functions representable by an ordinary recurrent network is not invariant to time rescaling. Ideally, one would like a recurrent model to be able to learn from time-warped input data x(c(t)) as easily as it learns from data x(t), at least if the time warping c(t) is not overly complex. The change of time c may represent not only time rescalings, but, for instance, accelerations or decelerations of the phenomena in the input data. We call a class of models invariant to time warping, if for any model in the class with input data x(t), and for any time warping c(t), there is another (or the same) model in the class that behaves on data x(c(t)) in the same way the original model behaves on x(t). (In practice, this will only be possible if the warping c is not too complex.) We will show that this is deeply linked to having gating mechanisms in the model. Invariance to time rescaling Let us first discuss the simpler case of a linear time rescaling. Formally, this is a linear transformation of time, that is c : R+ −→ R+ t ↦−→ αt (1) with α > 0. For instance, receiving a new input character every 10 time steps only, would correspond to α = 0.1. Studying time transformations is easier in the continuous-time setting. The discrete time equation of a basic recurrent network with hidden state ht, ht+1 = tanh (Wx xt +Wh ht + b) (2) can be seen as a time-discretized version of the continuous-time equation1 dh(t) dt = tanh (︀ Wx x(t) +Wh h(t) + b )︀ − h(t) (3) namely, (2) is the Taylor expansion h(t+ δt) ≈ h(t) + δt dh(t) dt with discretization step δt = 1. Now imagine that we want to describe time-rescaled data x(αt) with a model from the same class. Substituting t← c(t) = αt, x(t)← x(αt) and h(t)← h(αt) and rewriting (3) in terms of the new variables, the time-rescaled model satisfies2 dh(t) dt = α tanh (︀ Wx x(t) +Wh h(t) + b )︀ − αh(t). (4) However, when translated back to a discrete-time model, this no longer describes an ordinary RNN but a leaky RNN (Jaeger, 2002, §8.1). Indeed, taking the Taylor expansion of h(t+ δt) with δt = 1 in (4) yields the recurrent model ht+1 = α tanh (Wx xt +Wh ht + b) + (1− α)ht (5) We will use indices ht for discrete time and brackets h(t) for continuous time. More precisely, introduce a new time variable T and set the model and data with variable T to H(T ) := h(c(T )) and X(T ) := x(c(T )). Then compute dH(T ) dT . Then rename H to h, X to x and T to t to match the original notation.