Title: THE EXPRESSIVE POWER OF GATED RECURRENT UNITS AS A CONTINUOUS DYNAMICAL SYSTEM

Abstract: Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system. We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction. * The number/dimension of GRUs referenced will indicate the dimension of the latent dynamics of a GRU network.

1. INTRODUCTION:

Recurrent neural networks (RNNs) have been widely used to capture and utilize sequential structure in natural and artificial languages, speech, video, and various other forms of time series. The recurrent information flow within RNN implies that the data seen in the past has influence on the current state of the RNN, forming a mechanism for having memory through (nonlinear) temporal traces. Unfortunately, training vanilla RNNs (which allow input data to directly interact with the hidden state) to capture long-range dependences within a sequence is challenging due to the vanishing gradient problem (Hochreiter, 1991) . Several special RNN architectures have been proposed to mitigate this issue, notably the long short-term memory (LSTM; Hochreiter & Schmidhuber (1997) ) which explicitly guards against unwanted corruption of the information stored in the hidden state until necessary. Recently, a simplification of the LSTM called the gated recurrent unit (GRU; Cho et al. (2014) ) has become wildly popular in the machine learning community thanks to its performance in machine translation (Britz et al., 2017) , speech (Prabhavalkar et al., 2017) , music (Choi et al., 2017) , video (Dwibedi et al., 2018) , and extracting nonlinear dynamics underlying neural data (Pandarinath et al., 2018) . As a variant of the vanilla LSTM, GRUs incorporate the use of forget gates, but lack an output gate (Gers et al., 2000) . While this feature reduces the number of required parameters, LSTM has been shown to outperform GRU on neural machine translation (Britz et al., 2017) . In addition, certain mechanistic tasks, specifically unbounded counting, come easy to LSTM networks but not to GRU networks (Weiss et al., 2018) . Despite these empirical findings, we lack systematic understanding of the internal time evolution of GRU's memory structure and its capability to represent nonlinear temporal dynamics.

In general, a RNN can be written as h t+1 = f (h t , x t ) where x t is the current input in a sequence indexed by t, f is a point-wise nonlinear function, and h t represents the hidden memory state that carries all information responsible for future output. In the absence of input, the hidden state h t can evolve over time on its own:

EQUATION

where f (•) := f (•, 0) for notational simplicity. In other words, we can consider the temporal evolution of memory stored within RNN as a trajectory of a dynamical system defined by (1). Then we can use dynamical systems theory to investigate the fundamental limits in the expressive power of RNNs in terms of their temporal features. We develop a novel theoretical framework to study the dynamical features fundamentally attainable, in particular, given the particular form of GRU. We then validate the theory by training GRUs to predict time series with prescribed dynamics.

2. CONTINUOUS-TIME GATED RECURRENT UNIT:

The GRU uses two internal gating variables: the update gate z t which protects the d-dimensional hidden state h t ∈ R d and the reset gate r t which allows overwriting of the hidden state and controls the interaction with the input

EQUATION

where

W z , W r , W h ∈ R d×p and U z , U r , U h ∈ R d×d are the parameter matrices, b z , b r , b h ∈ R d are bias vectors,

represents the Hadamard product, and σ(z) = 1/(1 + e -z ) is the element-wise logistic sigmoid function. Note that the hidden state is asymptotically contained within [-1, 1] d due to the saturating nonlinearities, implying if the state is initialized outside of this trapping region, it must eventually enter it in finite time and remain in it for all later time.

Note that the update gate z t controls how fast each dimension at the hidden state decays, providing an adaptive time constant for memory. Specifically, as lim zt→0 h t = h t-1 , GRUs can implement perfect memory of the past and ignore x t . Hence, a d-dimensional GRU is capable of keeping a near constant memory through the update gate-near constant since 0 < [z t ] j < 1, where [•] j denotes j-th component of a vector. Moreover, the autoregressive weights (mainly U h and U r ) can support time evolving memory (Laurent & von Brecht (2016) ) considered this a hindrance and proposed removing all complex dynamic behavior in a simplified GRU).

To investigate the memory structure further, let us consider the dynamics of hidden state in the absence of input, i.e.

x t = 0, ∀t, which is of the form (1). To utilize the rich descriptive language of continuous time dynamical system theory, we consider the following continuous time limit of the (autonomous) GRU time evolution:

z(t) = σ(U z h(t) + b z ) (continuous update gate) (5) r(t) = σ(U r h(t) + b r ) (continuous reset gate) (6) ḣ = (z(t) -1) (h(t) -tanh(U h (r(t) h(t)) + b h )) (continuous hidden state dynamics) (7)

where ḣ ≡ dh(t) dt . Since both σ(•) and tanh(•) are smooth, this continuous limit is justified. The update gate z(t) again plays the role of a state-dependent time constant for memory decay. Furthermore, since 1 -z(t) > 0, it does not change the topological structure of the dynamics (only speed). For the following theoretical analysis sections (3 & 4), we can safely ignore the effects of z(t). A derivation of the continuous-time GRU can be found in appendix A.

3. STABILITY ANALYSIS OF A ONE DIMENSIONAL GRU:

For a single GRU * (d = 1), (7) reduces to a one dimensional dynamical system where every variable is a scalar. The expressive power of a single GRU is quite limited, as only three stability structures (topologies) exist (see appendix B): (A) a single stable node, (B) a stable node and a half-stable node, and (C) two stable nodes separated by an unstable node (see Fig. 1 ). The corresponding temporal features are (A) decay to a fixed value, (B) decay to a fixed value, but from one direction halt at an intermediate value until perturbed, or (C) decay to one of two fixed values (bistability). The bistability can be used to capture a binary latent state in the sequence. It should be noted that a one dimensional continuous time autonomous system cannot exhibit oscillatory behavior, as is the case here (Hirsch et al., 2013) . (Meiss, 2007) .

The topology the GRU takes is determined by its parameters. If the GRU begins in a region of the parameter space corresponding to (A), we can smoothly vary the parameters to transverse (B) in the parameter space, and end up at (C). This is commonly known as a saddle-node bifurcation. Speaking generally, a bifurcation is the change in topology of a dynamical system, resulting from a smooth change in parameters. The point in the parameters space at which the bifurcation occurs is called the bifurcation point, and we will refer to the fixed point that changes its stability at the bifurcation point as the bifurcation fixed point. This corresponds to the parameters underlying (B) in our previous example. The codimension of a bifurcation is the number of parameters which must vary in order to achieve the bifurcation. In the case of our example, a saddle-node bifurcation is codimension-1 (Kuznetsov, 1998) . Right before transitioning to (B), from (A), the flow near where the half-stable node would appear can exhibit arbitrarily slow flow.

We will refer to these as slow points (Sussillo & Barak, 2012) .

4. ANALYSIS OF A TWO DIMENSIONAL GRU:

We will see that the addition of a second GRU opens up a substantial variety of possible topological structures when compared with the use of a single GRU. For notational simplicity, we denote the two dimensions of h as x and y. We visualize the flow fields defined by ( 7) in 2-dimension as phase portraits which reveal the topological structures of interest. For starters, the phase portrait of two independent bistable GRUs can be visualized as Figure 2A . It clearly shows 4 stable states as expected, with a total of 9 stable fixed points. This could be thought of as a continuoustime continuous-space implementation of a finite state machine with 4 states (Fig. 2B ). The 3 types of observed fixed points-stable (sinks), unstable (sources), and saddle points-exhibit locally linear dynamics, however, the global geometry is nonlinear and their topological structures can vary depending on their arrangement. We explored stability structures attainable by two GRUs. Due to the relatively large number of observed topologies, this section's main focus will be on demonstrating all observed local dynamical features obtainable by two GRUs.

In addition, existence of two non-local dynamical features will be presented. A complete catalog of all observed topologies can be found in the appendix C, along with the parameters of every phase portrait depicted in this paper.

Before proceeding, let us take this time to describe all the local dynamical features observed. In addition to the previously mentioned three types of fixed points, two GRUs can exhibit a variety of bifurcation fixed points, resulting from regions of parameter space that separate all topologies restricted to simple fixed points (i.e stable, unstable, and saddle points). Behaviorally speaking, these fixed points act as hybrids between the previous three, resulting in a much richer set of obtainable dynamics. These bifurcation fixed points fall into two categories, separated by codimension. More specifically, two GRUs have been seen to feature both codimension-1 and codimension-2 bifurcation (fixed) points. Beginning with codimension-1, we have the saddle-node bifurcation fixed point, as expected from its existence in the single GRU case. We can further classify these points into two types. These can be thought of as both the fusion of a stable fixed point and a saddle point, and the fusion of an unstable fixed point and a saddle point. We will refer to these fixed points as saddle-node bifurcation fixed points of the first kind and second kind respectively.

One type of codimension-2 bifurcation fixed point that has been observed in the two GRU system acts as the fusion of all three simple fixed points. More specifically, these points arise from the fusion of a stable fixed point, unstable fixed point, and two saddle points. All of these local structures are depicted in figure 3 .

While the existence of simple fixed points was already demonstrated (see Fig. 2A ). Figure 3A demonstrates the maximum number of fixed points observed in a two GRU system, for a given set of parameters. A closer look at this system reveals its potential interpretation as a continuous analogue of 5-discrete states with input-driven transitions, similar to that depicted in figure 2 , implying additional GRUs are needed for any Markov process modeled in this manner, requiring more than five discrete states. We conjecture that the system depicted in figure 3A is the only eleven fixed point structure obtainable with two GRUs, as all observed structures containing the same number of fixed points are topologically equivalent to one another.

The addition of bifurcation fixed points opens the door to dynamically realize more sophisticated models. Take for example the four state system depicted in figure 3B . If the hidden state is set to initialize in the first quadrant of phase space, the trajectory will flow towards the codimension-2 bifurcation fixed point at the origin. Introducing noise through the input will stochastically cause the trajectory to approach the stable fixed point at (-1,-1) either directly, or by first flowing into one of the two saddle-node bifurcation fixed points of the first kind. Models of this sort can be used in a variety of applications, such as neural decision making (Wong & Wang (2006) , Churchland & Cunningham (2014)). We will begin our investigation into the non-local dynamics observed with two GRUs by showing the existence of homoclinic orbits. A trajectory initialized on a homoclinic orbit will approach the same fixed point in both forward and backward time. We observe that two GRUs can exhibit one or two bounded planar regions of homoclinic orbits for a given set of parameters, as shown in figure 4A and 4B respectively. Any trajectory initialized in one of these regions will flow into the codimension-2 bifurcation fixed point at the origin, regardless of which direction time flows in. This featured behavior enables the accurate depiction of various models, including neuron spiking (Izhikevich, 2007) .

In regards to the second non-local dynamic feature, it can be shown that two GRUs can exhibit an Andronov-Hopf bifurcation, whereby a stable fixed point bifurcates into an unstable fixed point surrounded by a limit cycle. Behaviorally speaking, a limit cycle is a type of attractor, in the sense that there exists a defined basin of attraction. However, unlike a stable fixed point, where trajectories initialized in the basin of attraction flow towards a single point, a limit cycle pulls trajectories into a stable periodic orbit around the unstable fixed point at its center. To demonstrate this phenomenon, let (8) define the parameters of (7).

EQUATION

where α ∈ R + .

If α = π 3 , the system has a single stable fixed point (stable spiral), as depicted in figure 5A . If we continuously decrease α, the system undergoes an Andronov-Hopf bifurcation approximately about α = π 3.8 . As α continuously decreases, the orbital period increases, and as the nullclines can be made arbitrarily close together, the length of this orbital period can be set arbitrarily, up to machine accuracy. Figure 5B shows an example of a relatively short orbital period, and figure 5C depicts the behavior seen for slower orbits. the system exhibits a single stable fixed point at the origin (Fig. 5A ). If α decreases continuously, a limit cycle emerges around the fixed point, and the fixed point changes stability (Fig. 5B ). Allowing α to decrease further increases the size and orbital period of the limit cycle (Fig. 5C ). The bottom row represents the hidden state as a function of time, for a single trajectory (denoted by black trajectories in each corresponding phase portrait)

With finite-fixed point topologies and global structures out of the way, the next logical question to ask is, can two GRUs exhibit an infinite number of fixed points (countable or uncountable)? Such behavior is often desirable in models that require stationary attraction to non-point structures, such as line attractors and ring attractors. The short answer to this question is no. Lemma 1. Any two dimensional GRU can only have finitely many simple fixed points.

This follows from Lefschetz theory (Guillemin & Pollack, 2010) . The detailed proof can be found in appendix D, and is intended to give the reader intuition behind the result presented in the claim extended to aribitrary dimensional GRU in theorem 1.

Theorem 1. Any finite dimensional GRU can only have finitely many simple fixed points and bifurcation fixed points.

Proof. By definition of simple fixed points, the Jacobian of the dynamics at those fixed points have nonzero real parts, making them Lefschetz fixed points. Since GRU dynamics is asymptotically bounded on the compact set [-1, 1] d , where d is the number of GRUs, it follows from Lefshetz theory (Guillemin & Pollack, 2010 ) that there are finitely many simple fixed points. Furthermore, by construction, a bifurcation fixed point can only exist within a stability structure if and only if there exists a separate topology, such that the simple fixed points making up each bifurcation fixed point exist isolated from one another. Therefore, there can only exist finitely many isolated bifurcation fixed points.

This eliminates the possibility of having countably many fixed points. Next, we show that there cannot be uncountably many non-isolated fixed points.

Theorem 2. Any finite dimensional GRU cannot have a continuous attractor.

Proof. We provide a sketch of a proof. Let h * ∈ R d be a fixed point of a d-dimensional GRU, that is, In each case, it reverts to a 1-dimensional problem where it can be trivially shown to have no continuous attractor around h * along direction k. Thus, we conclude that there is no continuous attractor in any direction.

h * -tanh(U h (r(h * ) h * ) + b h ) = 0.

Despite this limitation, an approximation of a line attractor can be created using two GRUs. This approximation can be made arbitrarily close to an actual line attractor on a finite region in phase space, thereby satisfying computational needs on an arbitrary interval when scaled. We will refer to this phenomenon as a pseudo-line attractor. Figure 6 depicts an example of such an attractor. We conclude this section with a discussion of slow points in the two GRU system. As a logical extension to the single GRU system, slow points occur where the nullclines are sufficiently close together, but do not intersect, as demonstrated in figure 7 . Given the previously discussed classes of dynamic behavior for two GRUs, slow points can only exist so long as the potential for a saddle-node bifurcation fixed point is possible in the location of the desired slow point, given an appropriate change in parameters, as they result from the collision and annihilation of two fixed points. This observation is consistent with the single GRU case, as slow points can only exist for a single fixed point. This would imply that given the one fixed point case, a maximum of five slow points are possible. However, this would imply that there must exist a six fixed point case by which five of the six fixed points exist at saddle-node bifurcation fixed points, which has not been observed (see appendix C). Despite this shortcoming, four simultaneous slow points are obtainable, as shown in figure 5C . 

5. NUMERICAL EXPERIMENTS:

As a means to put our theory to practice, in this section we explore several examples of time series prediction of continuous time planar dynamical systems using two GRUs. Results from the previous section indicate what dynamical features can be learned by this RNN, and suggest cases by which training will fail. All of the following computer experiments consist of an RNN, by which the hidden layer is made up of two GRUs, followed by a linear output layer.

The network is trained to make a 29-step prediction from a given initial observation, and no further input through prediction. As such, to produce accurate predictions, the RNN must rely solely on the hidden layer dynamics.

We train the network to minimize the following multi-step loss function:

L(θ) = 1 T Ntraj i=1 T k=1 ŵi (k; w i (0)) -w i (k) 2 2 (9)

where θ are the parameters of the GRU and linear readout, T = 29 is the prediction horizon, w i (t) is the i-th time series generated by the true system, and ŵ(k; w 0 ) is k-step the prediction given w 0 .

The hidden states are initialized at zero for each trajectory. The RNN is then trained for 4000 epochs, using ADAM (Kingma & Ba, 2014) in whole batch mode to minimize the loss function, i.e., the mean square error between the predicted trajectory and the data. N traj = 667 time series were used for training. Figure 8 depicts the experimental results of the RNN's attempt at learning each dynamical system we describe below.

5.1. LIMIT CYCLE:

To test if two GRUs can learn a limit cycle, we use a simple nonlinear oscillator called the FitzHugh-Nagumo Model.

The FitzHugh-Nagumo model is defined by:

EQUATION

where in this experiment we will chose τ = 12.5, a = 0.7, b = 0.8, and I ext = N (0.7, 0.04). Under this choice of model parameters, the system will exhibit an unstable fixed point (unstable spiral) surrounded by a limit cycle (Fig. 8 ).

As shown in section 4, two GRUs are capable of representing this topology. The results of this experiment verify this claim (Fig. 8 ), as two GRUs can capture topologically equivalent dynamics.

5.2. LINE ATTRACTOR:

As discussed in section 4, two GRUs can exhibit a pseudo-line attractor, by which the system mimics an analytic line attractor. We will use the simplest representation of a planar line attractor: This system will exhibit a line attractor along the y-axis, at x = 0 (Fig. 8 ). Trajectories will flow directly perpendicular towards the attractor. We added white Gaussian noise N (0, 0.1I) in the training data. While the hidden state dynamics of the trained network do not perfectly match that of an analytic line attractor, there exists a subinterval near each of the fixed points acting as a pseudo-line attractor (Fig. 8 ). As such, the added affine transformation (linear readout) can scale and reorient this subinterval as is required by a given problem, thereby mimicking a line attractor.

EQUATION

5.3. RING ATTRACTOR:

For this experiment, a dynamical system representing a standard ring attractor of radius one is used:

EQUATION

This system exhibits an attracting ring, centered around an unstable fixed point. We added Gaussian noise N (0, 0.1I) to the training data.

Two GRUs will not be able to accurately capture the system's continuous attractor dynamics as expected from theorem 3. The results of this experiment are demonstrated in figure 8 . As expected, the RNN fails to capture the proper dynamical features of the ring attractor. Rather, the hidden state dynamics fall into an observed finite fixed point topology (see case xxix in appendix C). In addition, we robustly see this over multiple initializations, and the quality of approximation improves as the dimensionality of GRU increases (Fig. 9 ). Four network sizes (2, 4, 8, 16 dimensional GRU) were trained 3 timeswith different initializations. Initial learning rate for ADAM was tuned using Bayesian optimization procedure.

6. CONCLUSION:

Our analysis shows the rich but limited classes of dynamics the GRU can approximate in one, two, and arbitrary dimensions. We developed a new theoretical framework to analyze GRUs as a continuous dynamical system, and showed that two GRUs can exhibit a variety of expressive dynamic features, such as limit cycles, homoclinic orbits, and a substantial catalog of stability structures and bifurcations. However, we also showed that finitely many GRUs cannot exhibit the dynamics of an arbitrary continuous attractor. These claims were then experimentally verified in two dimensions. We believe these findings also unlock new avenues of research on the trainability of recurrent neural networks. Although we have analyzed GRUs only in 1-and 2-dimensions in near exhaustive, we believe that the insights extends to higher-dimensions. We leave rigorous analysis of higher-dimensional GRUs as future work.

C ALL TOPOLOGICAL STABILITY STRUCTURES OBSERVED WITH TWO GRUS:

Table 2 depicts all observed topologies of multiple-fixed point structures using two GRUs. Figure 10 displays an example of a phase portrait from a two GRU system for each case listed in 2. Note that all fixed points are denoted by a red dot, regardless of classification. Table 3 lists the parameters used for each of the observed cases. Note that all the update gate parameters are set to zero.

Each case in this paper was discovered by hand by considering the geometric constraints on the structure of nullclines for both the decoupled and coupled system (i.e reset gate inactive and active respectively). An exhaustive analysis on the one dimensional GRU allowed for a natural extension into the two dimensional decoupled GRU. Upon establishing a set of base cases (a combinatorial argument regarding all possible ways the decoupled nullclines [topologically conjugate to linear and cubic polynomials] can intersect) From these base cases, the reset gate can be used as a means of bending and manipulating structure of the decoupled nullclines in order to obtain new intersection patterns in the coupled system.

Table 1 : Parameters Used for all Phase Portraits in Section 4 

Figure Uh11 Uh12 Uh21 Uh22 Ur11 Ur12 Ur21 Ur22 bh1 bh2 br1 br2 2 3 0 0 3 0 0 0 0 0 0 0 0 3a 2 0 0 2 5 8 8 5 0 0 5 5 3b 2 0 0 2 -1 0 0 -1 0 0 0 0 3c 2 0 0 2 1 -2 3 1 -0.06 0 0.2 -0.85 4a 2 0 0 2 5 9 5 9 0 0 0 0 4b 2 0 0 2 5 9 9 5 0 0 0 0 5a 1.5 -2.598 2.598 1.5 0 0 0 0 0 0 0 0 5b 2.4271 -1.7634 1.7634 2.4271 0 0 0 0 0 0 0 0 5c 2.9665 -0.4471 0.4471 2.9665 0 0 0 0 0 0 0 0 6a 0.1 -0.1 -1 0 0 0 0 0 0 0 0 0 7a 2 2 2 2 100 100 100 100 0.01 0 0 0 (i) (ii) (iii) (iv) (v) (vi) (vii) (viii) (ix) (x) (xi) (xii)

:



Figures and Tables:

Figure 1: Three possible types of one dimensional flow for a single GRU. When ḣ > 0, h(t) increases. This flow is indicated by a rightward arrow. Nodes ({h | ḣ(h) = 0}) are represented as circles and classified by their stability(Meiss, 2007).

Figure 2: Illustrative example of two independent bistable GRUs. (A) Phase portrait. The flow field ḣ = [ ẋ, ẏ]is decomposed into direction (black arrows) and speed (color). Purple lines represent trajectories of the hidden state which converge to one of the four stable fixed points. Note the four quadrants coincide with the basin of attraction for each of the stable nodes. The fixed points appear when the x-and y-nullclines intersect. (B) The four stable nodes of this system can be interpreted as a continuous analogue of 4-discrete states with input-driven transitions.

Figure 3: Existence of all observed simple fixed points and bifurcation fixed points with two GRUs, depicted in phase space. Orange and pink lines represent the x and y nullclines respectively. Purple lines indicate various trajectories of the hidden state. Direction of the flow is determined by the black arrows, where the colormap underlaying the figure depicts the magnitude of the velocity of the flow in log scale.

Figure4: Two GRUs exhibit bounded regions of homoclinic orbits. 4C and 4D represent the hidden state as a function of time, for a single initial condition within the homoclinic region(s) of the single and double homoclinic region cases respectively (denoted by solid black trajectories in each corresponding phase portrait).

Figure 5: Two GRUs exhibit an Andronov-Hopf bifurcation, where the parameters are defined by (8). When α = π 3

Now for any unit norm vector k ∈ S d-1 , and for any δ > 0, we can show that there exist an > 0 such that, ḣ(h * + δk) -ḣ(h * ) = ḣ(h * + δk) > . This can be argued by taking three cases into consideration, (a) U r k = 0 and U h (σ(b r ) k) = 0, (b) U r k = 0 and U h (σ(b r ) k) = 0, and (c) U r k = 0.

Figure 6: Two GRUs exhibit a pseudo-line attractor. Nullclines intersect at one point, but are close enough on a finite region to mimic an analytic line attractor in practice. 6A and 6B depict the phase portrait, on [-1.5, 1.5] 2 and [-0.2, 0.2] 2 respectively.

Figure 7: An example of a slow point about the origin, obtainable with two GRUs. Initial conditions satisfying y < -x are attracted to the slow point at the origin before a secondary attraction to the sink. 7A depicts the phase portrait of the system, and 7B shows the dynamics of the hidden state for a single initial condition (denoted by a black trajectory on 7A).

Figure 8: Training 2-dim GRUs. (top row) Phase portraits of target dynamical systems. Red solid line represents 1-dimensional attractor.See main text for each system. (middle row) GRU dynamics learned from corresponding 29-step forecasting tasks. Note that the prediction is an affine transformation of the hidden state. (bottom row) An example time series generated through closed-loop prediction of the trained GRU (denoted by a black trajectory). Note that GRU fails to learn the ring attractor.

Figure9: Learning curve (training loss) for ring attractor (left) and the FitzHugh-Nagumo (right) dynamics. Note that the performance of the ring attractor improves as the dimensinoality of the GRU increases unlike the FHN dynamics. Four network sizes (2, 4, 8, 16 dimensional GRU) were trained 3 timeswith different initializations. Initial learning rate for ADAM was tuned using Bayesian optimization procedure.

Figure 10: Thirty six multiple fixed-point topologies obtainable with two GRUs, depicted in phase space. Orange and pink lines represent the x and y nullclines respectively. Red dots indicate fixed points. Each subfigure contains 64 purple lines, indicating trajectories in forward time, whose initial conditions were chosen to be evenly spaced on the vertices of a square grid on [-1.5, 1.5] 2 . Direction of the flow is determined by the black arrows, and the underlaying color map represents the magnitude of the velocity of the flow in log scale.

Parameters of each multiple fixed-point stability structure example

Table content: <table><tr><td>Case</td><td>Uh11</td><td>Uh12</td><td>Uh21</td><td colspan="5">Uh22 Ur11 Ur12 Ur21 Ur22</td><td>bh1</td><td>bh2</td><td>br1</td><td>br2</td></tr><tr><td>i</td><td>2</td><td>2</td><td>2</td><td>2</td><td>100</td><td>100</td><td>100</td><td>100</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>ii</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>iii</td><td>1.2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>12</td><td>8</td><td>5</td><td>0</td><td>0</td><td>-0.22</td><td>-0.5</td></tr><tr><td>iv</td><td>2</td><td>0</td><td>0</td><td>2</td><td>-1</td><td>0</td><td>0</td><td>-1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>v</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>vi</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td>0.2</td><td>-0.85</td></tr><tr><td>vii</td><td>2</td><td>0</td><td>0</td><td>4</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td>-0.3</td><td>-0.42</td></tr><tr><td>viii</td><td>1.2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>20.5</td><td>8</td><td>5</td><td>0</td><td>0</td><td>-0.275</td><td>-3.3</td></tr><tr><td>ix</td><td>3</td><td>3</td><td>0</td><td>3</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>x</td><td>6</td><td>0</td><td>0</td><td>6</td><td>0</td><td>-2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>-1.695</td><td>0</td></tr><tr><td>xi</td><td>6</td><td>0</td><td>0</td><td>6</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>-2.5</td><td>0</td></tr><tr><td>xii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.055</td><td>0</td><td>-0.1</td><td>-0.02</td></tr><tr><td>xiii</td><td>2</td><td>0</td><td>0</td><td>4</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td colspan="2">-0.085 -0.22</td></tr><tr><td>xiv</td><td colspan="4">2.9674 -0.4409 0.4409 2.9674</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xv</td><td>6</td><td>0</td><td>0</td><td>2</td><td>-1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xvi</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td>-0.08</td><td>3</td></tr><tr><td>xvii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1.1</td><td>-0.06</td><td>0</td><td>0.2</td><td>0</td></tr><tr><td>xviii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>3</td><td>2</td><td>2</td><td>3</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xix</td><td>10</td><td>0</td><td>0</td><td>6</td><td>-3</td><td>5</td><td>-5</td><td>3</td><td>0</td><td>0</td><td>-3</td><td>-3</td></tr><tr><td>xx</td><td>1.2</td><td>0</td><td>0</td><td>2</td><td>-17</td><td>35</td><td>10</td><td>-8</td><td>0</td><td>0</td><td>1.2</td><td>-1.2</td></tr><tr><td>xxi</td><td colspan="2">2.9763 -0.376</td><td colspan="2">0.376 2.9763</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0.0315</td><td>0</td><td colspan="2">-0.015 -0.015</td></tr><tr><td>xxii</td><td>6</td><td>0</td><td>0</td><td>6</td><td>0</td><td>-2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xxiii0</td><td>3</td><td>0</td><td>0</td><td>3</td><td>-3</td><td>-5</td><td>-5</td><td>-3</td><td>0</td><td>0</td><td>-2</td><td>-2</td></tr><tr><td>xxiv</td><td>1.5</td><td>0</td><td>0</td><td>2</td><td>-4</td><td>-7</td><td>8</td><td>5</td><td>0</td><td>0</td><td>-0.4</td><td>-1.2</td></tr><tr><td>xxv</td><td>2</td><td>0</td><td>0</td><td>3</td><td>12.4</td><td>11.6</td><td>-8</td><td>-5</td><td>0</td><td>0</td><td>1.8</td><td>7</td></tr><tr><td>xxvi</td><td>3</td><td>0</td><td>0</td><td>3</td><td>5.175</td><td>9</td><td>9</td><td>5.175</td><td>0.3</td><td>0.3</td><td>3.95</td><td>3.95</td></tr><tr><td>xxvii</td><td>7</td><td>0</td><td>0</td><td>3</td><td>6</td><td>3</td><td>9</td><td>6</td><td>0.62</td><td>0.373</td><td>4</td><td>3.4</td></tr><tr><td>xxviii</td><td>6</td><td>0</td><td>0</td><td>10</td><td>0</td><td>0</td><td>-5</td><td>-8</td><td>-0.08</td><td>0</td><td>0</td><td>-2</td></tr><tr><td>xxix</td><td>2</td><td>0</td><td>0</td><td>3</td><td>-10</td><td>-11.6</td><td>8</td><td>5</td><td>0</td><td>0</td><td>4</td><td>4.8</td></tr><tr><td>xxx</td><td>3</td><td>0</td><td>0</td><td>3</td><td>5.26</td><td>9</td><td>9</td><td>5.26</td><td>0.25</td><td>0.25</td><td>3.95</td><td>3.95</td></tr><tr><td>xxxi</td><td>3</td><td>0</td><td>0</td><td>3</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xxxii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>8</td><td>8</td><td>5</td><td>0</td><td>0</td><td>4.4</td><td>4.4</td></tr><tr><td>xxxiii</td><td>3</td><td>0</td><td>0</td><td>3</td><td>6</td><td>9</td><td>9</td><td>6</td><td>0.3</td><td>0.3</td><td>3.75</td><td>3.75</td></tr><tr><td>xxxiv</td><td>1</td><td>0</td><td>0</td><td>1</td><td>5</td><td>8</td><td>8</td><td>5</td><td>0</td><td>0</td><td>4.4</td><td>4.9</td></tr><tr><td>xxxv</td><td>3</td><td>0</td><td>0</td><td>3</td><td>6</td><td>9</td><td>9</td><td>6</td><td>0.24</td><td>0.24</td><td>3.95</td><td>3.95</td></tr><tr><td>xxxvi</td><td>2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>8</td><td>8</td><td>5</td><td>0</td><td>0</td><td>5</td><td>5</td></tr></table>

Peer review (iclr2019):
 The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset. Their analysis shows what kind of hidden state dynamics the GRU can approximate in one and two dimensions. In the experimental part, they show how a GRU with two hidden states trained on a multistep prediction task can learn such dynamics. Although RNNs are important for Machine Learning, the paper seems to contain flaws in the theoretical part, which seem to invalidate some of the claimed results. But we may change our rating in case of a convincing rebuttal. The Proof of Lemma 2 claims that h(t) achieves all values on the real set, which is false (h(t) assumes values in (-1,1)). Nevertheless, the theorem should hold since there is always at least one intersection between h and tanh(f(h)). Lemma 1 claims that for any choice of parameters, there exist only finitely many fixed points. However, in the proof the authors only show that the number of fixed points cannot be uncountable, without taking into consideration the possibility that there are countably many fixed points. The proof also omits steps concerning the Taylor expansion which would make the proof clearer: We suggest adding those steps in the appendix. Furthermore, when equation (12) is Taylor-expanded, the authors do not consider the case where the GRU parameters are such that the argument of function “sech” is outside its convergence radius. These might be parameters for which there are infinitely many fixed points, even if we are unable to provide a Taylor expansion. The Lemma may still be correct, but this does not seem to be a complete proof. The authors claim that an arbitrarily close approximation of a line attractor can be created using two GRUs, but no proof is provided. The experimental part is difficult to evaluate since there are no learning curves for the three tasks. For instance, it is difficult to judge whether the GRUs are unable to learn the dynamics of a ring attractor because of theoretical limitations or because the model has not been properly trained for the specific task. The paper is easy to read, except for certain parts where it is not clear if some of the statements are true in general or just have not been proven false by the authors. It is not clear why Figure 3 is representing all possible simple fixed points and bifurcation fixed points: is there a theoretical result stating that these are the only possible topologies, or are these the only ones found? The same question applies for the 36 images in figure 9. The range of the parameters used for finding these configurations is not specified. Since the hidden state assumes values in (-1,1)^2, why is its range in most of the images (-1.5,1.5)? We are not familiar with related work on transformations from discrete to continuous dynamical systems: are the dynamics of the discrete time GRU model preserved in the transformation? If so, is there a reference for this? Are the phase portraits in the middle row of figure 8 generated by letting the discrete GRU system evolve, or is the continuous system used with the parameters of the trained GRU? We would like to see more explanations of why various topologies are useful for the applications mentioned in the paper. Given a generic dataset, how can these results help to understand how well a GRU will perform? What is the reason behind the belief that the analysis extends to higher dimensions? The effects of a 1D -> 2D extension are far from trivial - why should that be different for higher dimensions? The problem the authors want to solve seems important, and some of the theoretical results are promising, but we think that this paper has to be further polished before acceptance. It is possible that we will increase the score if the authors can provide clarifications on the above questions. Additional comments: Introduction -The vanishing gradient problem was not discovered in 1994, but in 1991 by Hochreiter: Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich, 1991. Advisor J. Schmidhuber. - Make clear that GRU is a variant of vanilla LSTM with forget gates (where one gate is missing): Gers et al. “Learning to Forget: Continual Prediction with LSTM.“ Neural Computation, 12(10):2451-2471, 2000. - The intro says that GRU has become widely popular and cites Britz et al 2017, but Britz et al actually show that LSTM consistently outperforms its variant GRU in Neural Machine Translation. Please clarify this. - Also mention Weiss et al (“On the Practical Computational Power of Finite Precision RNNs for Language Recognition”) who exhibited basic limitations of GRU when compared to LSTM. - Is the result by Weiss et al actually related to the result of the authors who found that 2 GRUS cannot accurately a line attractor without near zero constant curvature in the phase space? Section 2 -Wrong brackets in equation (4) -Missing bracket before citing Laurent & Brecht Section 4 -”We conjecture that the system depicted in figure 2A..” Should be figure 3A - Lemma1: UZ has capital Z subscript Section 5.2 -”The added affine transformation allows for a sufficiently long subinterval”: “sufficiently long” is too vague Section 5.3 “A manifold with without near zero constant curvature”: should be “a manifold without near zero constant curvature” Appendix A -Wrong brackets in equation (20) Appendix B - In the proof of Theorem 1, the derivative is of (29), not of (12) Appendix C Figure 9: “who’s initial conditions” should be “whose initial conditions” After rebuttal: It*s better now. However, the revised introduction still says: "GRU has become wildly popular in the machine learning community thanks to its performance in machine translation (Britz et al., 2017) ... LSTM has been shown to outperform GRU on neural machine translation (Britz et al., 2017).... specifically unbounded counting, come easy to LSTM networks but not to GRU networks (Weiss et al., 2018)." So better remove the first statement on Britz et al: "GRU has become wildly popular ... in machine translation (Britz et al., 2017)" because they actually show why GRU is NOT wildly popular in machine translation, as correctly justified later in the same paragraph. Pending the above revision, we*d like to increase our evaluation by 2 points, up to 6!