Title: THE EXPRESSIVE POWER OF GATED RECURRENT UNITS AS A CONTINUOUS DYNAMICAL SYSTEM

Abstract: Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system. We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction. * The number/dimension of GRUs referenced will indicate the dimension of the latent dynamics of a GRU network.

1. INTRODUCTION:

Recurrent neural networks (RNNs) have been widely used to capture and utilize sequential structure in natural and artificial languages, speech, video, and various other forms of time series. The recurrent information flow within RNN implies that the data seen in the past has influence on the current state of the RNN, forming a mechanism for having memory through (nonlinear) temporal traces. Unfortunately, training vanilla RNNs (which allow input data to directly interact with the hidden state) to capture long-range dependences within a sequence is challenging due to the vanishing gradient problem (Hochreiter, 1991) . Several special RNN architectures have been proposed to mitigate this issue, notably the long short-term memory (LSTM; Hochreiter & Schmidhuber (1997) ) which explicitly guards against unwanted corruption of the information stored in the hidden state until necessary. Recently, a simplification of the LSTM called the gated recurrent unit (GRU; Cho et al. (2014) ) has become wildly popular in the machine learning community thanks to its performance in machine translation (Britz et al., 2017) , speech (Prabhavalkar et al., 2017) , music (Choi et al., 2017) , video (Dwibedi et al., 2018) , and extracting nonlinear dynamics underlying neural data (Pandarinath et al., 2018) . As a variant of the vanilla LSTM, GRUs incorporate the use of forget gates, but lack an output gate (Gers et al., 2000) . While this feature reduces the number of required parameters, LSTM has been shown to outperform GRU on neural machine translation (Britz et al., 2017) . In addition, certain mechanistic tasks, specifically unbounded counting, come easy to LSTM networks but not to GRU networks (Weiss et al., 2018) . Despite these empirical findings, we lack systematic understanding of the internal time evolution of GRU's memory structure and its capability to represent nonlinear temporal dynamics.

In general, a RNN can be written as h t+1 = f (h t , x t ) where x t is the current input in a sequence indexed by t, f is a point-wise nonlinear function, and h t represents the hidden memory state that carries all information responsible for future output. In the absence of input, the hidden state h t can evolve over time on its own:

EQUATION

where f (•) := f (•, 0) for notational simplicity. In other words, we can consider the temporal evolution of memory stored within RNN as a trajectory of a dynamical system defined by (1). Then we can use dynamical systems theory to investigate the fundamental limits in the expressive power of RNNs in terms of their temporal features. We develop a novel theoretical framework to study the dynamical features fundamentally attainable, in particular, given the particular form of GRU. We then validate the theory by training GRUs to predict time series with prescribed dynamics.

2. CONTINUOUS-TIME GATED RECURRENT UNIT:

The GRU uses two internal gating variables: the update gate z t which protects the d-dimensional hidden state h t ∈ R d and the reset gate r t which allows overwriting of the hidden state and controls the interaction with the input

EQUATION

where

W z , W r , W h ∈ R d×p and U z , U r , U h ∈ R d×d are the parameter matrices, b z , b r , b h ∈ R d are bias vectors,

represents the Hadamard product, and σ(z) = 1/(1 + e -z ) is the element-wise logistic sigmoid function. Note that the hidden state is asymptotically contained within [-1, 1] d due to the saturating nonlinearities, implying if the state is initialized outside of this trapping region, it must eventually enter it in finite time and remain in it for all later time.

Note that the update gate z t controls how fast each dimension at the hidden state decays, providing an adaptive time constant for memory. Specifically, as lim zt→0 h t = h t-1 , GRUs can implement perfect memory of the past and ignore x t . Hence, a d-dimensional GRU is capable of keeping a near constant memory through the update gate-near constant since 0 < [z t ] j < 1, where [•] j denotes j-th component of a vector. Moreover, the autoregressive weights (mainly U h and U r ) can support time evolving memory (Laurent & von Brecht (2016) ) considered this a hindrance and proposed removing all complex dynamic behavior in a simplified GRU).

To investigate the memory structure further, let us consider the dynamics of hidden state in the absence of input, i.e.

x t = 0, ∀t, which is of the form (1). To utilize the rich descriptive language of continuous time dynamical system theory, we consider the following continuous time limit of the (autonomous) GRU time evolution:

z(t) = σ(U z h(t) + b z ) (continuous update gate) (5) r(t) = σ(U r h(t) + b r ) (continuous reset gate) (6) ḣ = (z(t) -1) (h(t) -tanh(U h (r(t) h(t)) + b h )) (continuous hidden state dynamics) (7)

where ḣ ≡ dh(t) dt . Since both σ(•) and tanh(•) are smooth, this continuous limit is justified. The update gate z(t) again plays the role of a state-dependent time constant for memory decay. Furthermore, since 1 -z(t) > 0, it does not change the topological structure of the dynamics (only speed). For the following theoretical analysis sections (3 & 4), we can safely ignore the effects of z(t). A derivation of the continuous-time GRU can be found in appendix A.

3. STABILITY ANALYSIS OF A ONE DIMENSIONAL GRU:

For a single GRU * (d = 1), (7) reduces to a one dimensional dynamical system where every variable is a scalar. The expressive power of a single GRU is quite limited, as only three stability structures (topologies) exist (see appendix B): (A) a single stable node, (B) a stable node and a half-stable node, and (C) two stable nodes separated by an unstable node (see Fig. 1 ). The corresponding temporal features are (A) decay to a fixed value, (B) decay to a fixed value, but from one direction halt at an intermediate value until perturbed, or (C) decay to one of two fixed values (bistability). The bistability can be used to capture a binary latent state in the sequence. It should be noted that a one dimensional continuous time autonomous system cannot exhibit oscillatory behavior, as is the case here (Hirsch et al., 2013) . (Meiss, 2007) .

The topology the GRU takes is determined by its parameters. If the GRU begins in a region of the parameter space corresponding to (A), we can smoothly vary the parameters to transverse (B) in the parameter space, and end up at (C). This is commonly known as a saddle-node bifurcation. Speaking generally, a bifurcation is the change in topology of a dynamical system, resulting from a smooth change in parameters. The point in the parameters space at which the bifurcation occurs is called the bifurcation point, and we will refer to the fixed point that changes its stability at the bifurcation point as the bifurcation fixed point. This corresponds to the parameters underlying (B) in our previous example. The codimension of a bifurcation is the number of parameters which must vary in order to achieve the bifurcation. In the case of our example, a saddle-node bifurcation is codimension-1 (Kuznetsov, 1998) . Right before transitioning to (B), from (A), the flow near where the half-stable node would appear can exhibit arbitrarily slow flow.

We will refer to these as slow points (Sussillo & Barak, 2012) .

4. ANALYSIS OF A TWO DIMENSIONAL GRU:

We will see that the addition of a second GRU opens up a substantial variety of possible topological structures when compared with the use of a single GRU. For notational simplicity, we denote the two dimensions of h as x and y. We visualize the flow fields defined by ( 7) in 2-dimension as phase portraits which reveal the topological structures of interest. For starters, the phase portrait of two independent bistable GRUs can be visualized as Figure 2A . It clearly shows 4 stable states as expected, with a total of 9 stable fixed points. This could be thought of as a continuoustime continuous-space implementation of a finite state machine with 4 states (Fig. 2B ). The 3 types of observed fixed points-stable (sinks), unstable (sources), and saddle points-exhibit locally linear dynamics, however, the global geometry is nonlinear and their topological structures can vary depending on their arrangement. We explored stability structures attainable by two GRUs. Due to the relatively large number of observed topologies, this section's main focus will be on demonstrating all observed local dynamical features obtainable by two GRUs.

In addition, existence of two non-local dynamical features will be presented. A complete catalog of all observed topologies can be found in the appendix C, along with the parameters of every phase portrait depicted in this paper.

Before proceeding, let us take this time to describe all the local dynamical features observed. In addition to the previously mentioned three types of fixed points, two GRUs can exhibit a variety of bifurcation fixed points, resulting from regions of parameter space that separate all topologies restricted to simple fixed points (i.e stable, unstable, and saddle points). Behaviorally speaking, these fixed points act as hybrids between the previous three, resulting in a much richer set of obtainable dynamics. These bifurcation fixed points fall into two categories, separated by codimension. More specifically, two GRUs have been seen to feature both codimension-1 and codimension-2 bifurcation (fixed) points. Beginning with codimension-1, we have the saddle-node bifurcation fixed point, as expected from its existence in the single GRU case. We can further classify these points into two types. These can be thought of as both the fusion of a stable fixed point and a saddle point, and the fusion of an unstable fixed point and a saddle point. We will refer to these fixed points as saddle-node bifurcation fixed points of the first kind and second kind respectively.

One type of codimension-2 bifurcation fixed point that has been observed in the two GRU system acts as the fusion of all three simple fixed points. More specifically, these points arise from the fusion of a stable fixed point, unstable fixed point, and two saddle points. All of these local structures are depicted in figure 3 .

While the existence of simple fixed points was already demonstrated (see Fig. 2A ). Figure 3A demonstrates the maximum number of fixed points observed in a two GRU system, for a given set of parameters. A closer look at this system reveals its potential interpretation as a continuous analogue of 5-discrete states with input-driven transitions, similar to that depicted in figure 2 , implying additional GRUs are needed for any Markov process modeled in this manner, requiring more than five discrete states. We conjecture that the system depicted in figure 3A is the only eleven fixed point structure obtainable with two GRUs, as all observed structures containing the same number of fixed points are topologically equivalent to one another.

The addition of bifurcation fixed points opens the door to dynamically realize more sophisticated models. Take for example the four state system depicted in figure 3B . If the hidden state is set to initialize in the first quadrant of phase space, the trajectory will flow towards the codimension-2 bifurcation fixed point at the origin. Introducing noise through the input will stochastically cause the trajectory to approach the stable fixed point at (-1,-1) either directly, or by first flowing into one of the two saddle-node bifurcation fixed points of the first kind. Models of this sort can be used in a variety of applications, such as neural decision making (Wong & Wang (2006) , Churchland & Cunningham (2014)). We will begin our investigation into the non-local dynamics observed with two GRUs by showing the existence of homoclinic orbits. A trajectory initialized on a homoclinic orbit will approach the same fixed point in both forward and backward time. We observe that two GRUs can exhibit one or two bounded planar regions of homoclinic orbits for a given set of parameters, as shown in figure 4A and 4B respectively. Any trajectory initialized in one of these regions will flow into the codimension-2 bifurcation fixed point at the origin, regardless of which direction time flows in. This featured behavior enables the accurate depiction of various models, including neuron spiking (Izhikevich, 2007) .

In regards to the second non-local dynamic feature, it can be shown that two GRUs can exhibit an Andronov-Hopf bifurcation, whereby a stable fixed point bifurcates into an unstable fixed point surrounded by a limit cycle. Behaviorally speaking, a limit cycle is a type of attractor, in the sense that there exists a defined basin of attraction. However, unlike a stable fixed point, where trajectories initialized in the basin of attraction flow towards a single point, a limit cycle pulls trajectories into a stable periodic orbit around the unstable fixed point at its center. To demonstrate this phenomenon, let (8) define the parameters of (7).

EQUATION

where α ∈ R + .

If α = π 3 , the system has a single stable fixed point (stable spiral), as depicted in figure 5A . If we continuously decrease α, the system undergoes an Andronov-Hopf bifurcation approximately about α = π 3.8 . As α continuously decreases, the orbital period increases, and as the nullclines can be made arbitrarily close together, the length of this orbital period can be set arbitrarily, up to machine accuracy. Figure 5B shows an example of a relatively short orbital period, and figure 5C depicts the behavior seen for slower orbits. the system exhibits a single stable fixed point at the origin (Fig. 5A ). If α decreases continuously, a limit cycle emerges around the fixed point, and the fixed point changes stability (Fig. 5B ). Allowing α to decrease further increases the size and orbital period of the limit cycle (Fig. 5C ). The bottom row represents the hidden state as a function of time, for a single trajectory (denoted by black trajectories in each corresponding phase portrait)

With finite-fixed point topologies and global structures out of the way, the next logical question to ask is, can two GRUs exhibit an infinite number of fixed points (countable or uncountable)? Such behavior is often desirable in models that require stationary attraction to non-point structures, such as line attractors and ring attractors. The short answer to this question is no. Lemma 1. Any two dimensional GRU can only have finitely many simple fixed points.

This follows from Lefschetz theory (Guillemin & Pollack, 2010) . The detailed proof can be found in appendix D, and is intended to give the reader intuition behind the result presented in the claim extended to aribitrary dimensional GRU in theorem 1.

Theorem 1. Any finite dimensional GRU can only have finitely many simple fixed points and bifurcation fixed points.

Proof. By definition of simple fixed points, the Jacobian of the dynamics at those fixed points have nonzero real parts, making them Lefschetz fixed points. Since GRU dynamics is asymptotically bounded on the compact set [-1, 1] d , where d is the number of GRUs, it follows from Lefshetz theory (Guillemin & Pollack, 2010 ) that there are finitely many simple fixed points. Furthermore, by construction, a bifurcation fixed point can only exist within a stability structure if and only if there exists a separate topology, such that the simple fixed points making up each bifurcation fixed point exist isolated from one another. Therefore, there can only exist finitely many isolated bifurcation fixed points.

This eliminates the possibility of having countably many fixed points. Next, we show that there cannot be uncountably many non-isolated fixed points.

Theorem 2. Any finite dimensional GRU cannot have a continuous attractor.

Proof. We provide a sketch of a proof. Let h * ∈ R d be a fixed point of a d-dimensional GRU, that is, In each case, it reverts to a 1-dimensional problem where it can be trivially shown to have no continuous attractor around h * along direction k. Thus, we conclude that there is no continuous attractor in any direction.

h * -tanh(U h (r(h * ) h * ) + b h ) = 0.

Despite this limitation, an approximation of a line attractor can be created using two GRUs. This approximation can be made arbitrarily close to an actual line attractor on a finite region in phase space, thereby satisfying computational needs on an arbitrary interval when scaled. We will refer to this phenomenon as a pseudo-line attractor. Figure 6 depicts an example of such an attractor. We conclude this section with a discussion of slow points in the two GRU system. As a logical extension to the single GRU system, slow points occur where the nullclines are sufficiently close together, but do not intersect, as demonstrated in figure 7 . Given the previously discussed classes of dynamic behavior for two GRUs, slow points can only exist so long as the potential for a saddle-node bifurcation fixed point is possible in the location of the desired slow point, given an appropriate change in parameters, as they result from the collision and annihilation of two fixed points. This observation is consistent with the single GRU case, as slow points can only exist for a single fixed point. This would imply that given the one fixed point case, a maximum of five slow points are possible. However, this would imply that there must exist a six fixed point case by which five of the six fixed points exist at saddle-node bifurcation fixed points, which has not been observed (see appendix C). Despite this shortcoming, four simultaneous slow points are obtainable, as shown in figure 5C . 

5. NUMERICAL EXPERIMENTS:

As a means to put our theory to practice, in this section we explore several examples of time series prediction of continuous time planar dynamical systems using two GRUs. Results from the previous section indicate what dynamical features can be learned by this RNN, and suggest cases by which training will fail. All of the following computer experiments consist of an RNN, by which the hidden layer is made up of two GRUs, followed by a linear output layer.

The network is trained to make a 29-step prediction from a given initial observation, and no further input through prediction. As such, to produce accurate predictions, the RNN must rely solely on the hidden layer dynamics.

We train the network to minimize the following multi-step loss function:

L(θ) = 1 T Ntraj i=1 T k=1 ŵi (k; w i (0)) -w i (k) 2 2 (9)

where θ are the parameters of the GRU and linear readout, T = 29 is the prediction horizon, w i (t) is the i-th time series generated by the true system, and ŵ(k; w 0 ) is k-step the prediction given w 0 .

The hidden states are initialized at zero for each trajectory. The RNN is then trained for 4000 epochs, using ADAM (Kingma & Ba, 2014) in whole batch mode to minimize the loss function, i.e., the mean square error between the predicted trajectory and the data. N traj = 667 time series were used for training. Figure 8 depicts the experimental results of the RNN's attempt at learning each dynamical system we describe below.

5.1. LIMIT CYCLE:

To test if two GRUs can learn a limit cycle, we use a simple nonlinear oscillator called the FitzHugh-Nagumo Model.

The FitzHugh-Nagumo model is defined by:

EQUATION

where in this experiment we will chose τ = 12.5, a = 0.7, b = 0.8, and I ext = N (0.7, 0.04). Under this choice of model parameters, the system will exhibit an unstable fixed point (unstable spiral) surrounded by a limit cycle (Fig. 8 ).

As shown in section 4, two GRUs are capable of representing this topology. The results of this experiment verify this claim (Fig. 8 ), as two GRUs can capture topologically equivalent dynamics.

5.2. LINE ATTRACTOR:

As discussed in section 4, two GRUs can exhibit a pseudo-line attractor, by which the system mimics an analytic line attractor. We will use the simplest representation of a planar line attractor: This system will exhibit a line attractor along the y-axis, at x = 0 (Fig. 8 ). Trajectories will flow directly perpendicular towards the attractor. We added white Gaussian noise N (0, 0.1I) in the training data. While the hidden state dynamics of the trained network do not perfectly match that of an analytic line attractor, there exists a subinterval near each of the fixed points acting as a pseudo-line attractor (Fig. 8 ). As such, the added affine transformation (linear readout) can scale and reorient this subinterval as is required by a given problem, thereby mimicking a line attractor.

EQUATION

5.3. RING ATTRACTOR:

For this experiment, a dynamical system representing a standard ring attractor of radius one is used:

EQUATION

This system exhibits an attracting ring, centered around an unstable fixed point. We added Gaussian noise N (0, 0.1I) to the training data.

Two GRUs will not be able to accurately capture the system's continuous attractor dynamics as expected from theorem 3. The results of this experiment are demonstrated in figure 8 . As expected, the RNN fails to capture the proper dynamical features of the ring attractor. Rather, the hidden state dynamics fall into an observed finite fixed point topology (see case xxix in appendix C). In addition, we robustly see this over multiple initializations, and the quality of approximation improves as the dimensionality of GRU increases (Fig. 9 ). Four network sizes (2, 4, 8, 16 dimensional GRU) were trained 3 timeswith different initializations. Initial learning rate for ADAM was tuned using Bayesian optimization procedure.

6. CONCLUSION:

Our analysis shows the rich but limited classes of dynamics the GRU can approximate in one, two, and arbitrary dimensions. We developed a new theoretical framework to analyze GRUs as a continuous dynamical system, and showed that two GRUs can exhibit a variety of expressive dynamic features, such as limit cycles, homoclinic orbits, and a substantial catalog of stability structures and bifurcations. However, we also showed that finitely many GRUs cannot exhibit the dynamics of an arbitrary continuous attractor. These claims were then experimentally verified in two dimensions. We believe these findings also unlock new avenues of research on the trainability of recurrent neural networks. Although we have analyzed GRUs only in 1-and 2-dimensions in near exhaustive, we believe that the insights extends to higher-dimensions. We leave rigorous analysis of higher-dimensional GRUs as future work.

C ALL TOPOLOGICAL STABILITY STRUCTURES OBSERVED WITH TWO GRUS:

Table 2 depicts all observed topologies of multiple-fixed point structures using two GRUs. Figure 10 displays an example of a phase portrait from a two GRU system for each case listed in 2. Note that all fixed points are denoted by a red dot, regardless of classification. Table 3 lists the parameters used for each of the observed cases. Note that all the update gate parameters are set to zero.

Each case in this paper was discovered by hand by considering the geometric constraints on the structure of nullclines for both the decoupled and coupled system (i.e reset gate inactive and active respectively). An exhaustive analysis on the one dimensional GRU allowed for a natural extension into the two dimensional decoupled GRU. Upon establishing a set of base cases (a combinatorial argument regarding all possible ways the decoupled nullclines [topologically conjugate to linear and cubic polynomials] can intersect) From these base cases, the reset gate can be used as a means of bending and manipulating structure of the decoupled nullclines in order to obtain new intersection patterns in the coupled system.

Table 1 : Parameters Used for all Phase Portraits in Section 4 

Figure Uh11 Uh12 Uh21 Uh22 Ur11 Ur12 Ur21 Ur22 bh1 bh2 br1 br2 2 3 0 0 3 0 0 0 0 0 0 0 0 3a 2 0 0 2 5 8 8 5 0 0 5 5 3b 2 0 0 2 -1 0 0 -1 0 0 0 0 3c 2 0 0 2 1 -2 3 1 -0.06 0 0.2 -0.85 4a 2 0 0 2 5 9 5 9 0 0 0 0 4b 2 0 0 2 5 9 9 5 0 0 0 0 5a 1.5 -2.598 2.598 1.5 0 0 0 0 0 0 0 0 5b 2.4271 -1.7634 1.7634 2.4271 0 0 0 0 0 0 0 0 5c 2.9665 -0.4471 0.4471 2.9665 0 0 0 0 0 0 0 0 6a 0.1 -0.1 -1 0 0 0 0 0 0 0 0 0 7a 2 2 2 2 100 100 100 100 0.01 0 0 0 (i) (ii) (iii) (iv) (v) (vi) (vii) (viii) (ix) (x) (xi) (xii)

:



Figures and Tables:

Figure 1: Three possible types of one dimensional flow for a single GRU. When ḣ > 0, h(t) increases. This flow is indicated by a rightward arrow. Nodes ({h | ḣ(h) = 0}) are represented as circles and classified by their stability(Meiss, 2007).

Figure 2: Illustrative example of two independent bistable GRUs. (A) Phase portrait. The flow field ḣ = [ ẋ, ẏ]is decomposed into direction (black arrows) and speed (color). Purple lines represent trajectories of the hidden state which converge to one of the four stable fixed points. Note the four quadrants coincide with the basin of attraction for each of the stable nodes. The fixed points appear when the x-and y-nullclines intersect. (B) The four stable nodes of this system can be interpreted as a continuous analogue of 4-discrete states with input-driven transitions.

Figure 3: Existence of all observed simple fixed points and bifurcation fixed points with two GRUs, depicted in phase space. Orange and pink lines represent the x and y nullclines respectively. Purple lines indicate various trajectories of the hidden state. Direction of the flow is determined by the black arrows, where the colormap underlaying the figure depicts the magnitude of the velocity of the flow in log scale.

Figure4: Two GRUs exhibit bounded regions of homoclinic orbits. 4C and 4D represent the hidden state as a function of time, for a single initial condition within the homoclinic region(s) of the single and double homoclinic region cases respectively (denoted by solid black trajectories in each corresponding phase portrait).

Figure 5: Two GRUs exhibit an Andronov-Hopf bifurcation, where the parameters are defined by (8). When α = π 3

Now for any unit norm vector k ∈ S d-1 , and for any δ > 0, we can show that there exist an > 0 such that, ḣ(h * + δk) -ḣ(h * ) = ḣ(h * + δk) > . This can be argued by taking three cases into consideration, (a) U r k = 0 and U h (σ(b r ) k) = 0, (b) U r k = 0 and U h (σ(b r ) k) = 0, and (c) U r k = 0.

Figure 6: Two GRUs exhibit a pseudo-line attractor. Nullclines intersect at one point, but are close enough on a finite region to mimic an analytic line attractor in practice. 6A and 6B depict the phase portrait, on [-1.5, 1.5] 2 and [-0.2, 0.2] 2 respectively.

Figure 7: An example of a slow point about the origin, obtainable with two GRUs. Initial conditions satisfying y < -x are attracted to the slow point at the origin before a secondary attraction to the sink. 7A depicts the phase portrait of the system, and 7B shows the dynamics of the hidden state for a single initial condition (denoted by a black trajectory on 7A).

Figure 8: Training 2-dim GRUs. (top row) Phase portraits of target dynamical systems. Red solid line represents 1-dimensional attractor.See main text for each system. (middle row) GRU dynamics learned from corresponding 29-step forecasting tasks. Note that the prediction is an affine transformation of the hidden state. (bottom row) An example time series generated through closed-loop prediction of the trained GRU (denoted by a black trajectory). Note that GRU fails to learn the ring attractor.

Figure9: Learning curve (training loss) for ring attractor (left) and the FitzHugh-Nagumo (right) dynamics. Note that the performance of the ring attractor improves as the dimensinoality of the GRU increases unlike the FHN dynamics. Four network sizes (2, 4, 8, 16 dimensional GRU) were trained 3 timeswith different initializations. Initial learning rate for ADAM was tuned using Bayesian optimization procedure.

Figure 10: Thirty six multiple fixed-point topologies obtainable with two GRUs, depicted in phase space. Orange and pink lines represent the x and y nullclines respectively. Red dots indicate fixed points. Each subfigure contains 64 purple lines, indicating trajectories in forward time, whose initial conditions were chosen to be evenly spaced on the vertices of a square grid on [-1.5, 1.5] 2 . Direction of the flow is determined by the black arrows, and the underlaying color map represents the magnitude of the velocity of the flow in log scale.

Parameters of each multiple fixed-point stability structure example

Table content: <table><tr><td>Case</td><td>Uh11</td><td>Uh12</td><td>Uh21</td><td colspan="5">Uh22 Ur11 Ur12 Ur21 Ur22</td><td>bh1</td><td>bh2</td><td>br1</td><td>br2</td></tr><tr><td>i</td><td>2</td><td>2</td><td>2</td><td>2</td><td>100</td><td>100</td><td>100</td><td>100</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>ii</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>iii</td><td>1.2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>12</td><td>8</td><td>5</td><td>0</td><td>0</td><td>-0.22</td><td>-0.5</td></tr><tr><td>iv</td><td>2</td><td>0</td><td>0</td><td>2</td><td>-1</td><td>0</td><td>0</td><td>-1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>v</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>vi</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td>0.2</td><td>-0.85</td></tr><tr><td>vii</td><td>2</td><td>0</td><td>0</td><td>4</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td>-0.3</td><td>-0.42</td></tr><tr><td>viii</td><td>1.2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>20.5</td><td>8</td><td>5</td><td>0</td><td>0</td><td>-0.275</td><td>-3.3</td></tr><tr><td>ix</td><td>3</td><td>3</td><td>0</td><td>3</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>x</td><td>6</td><td>0</td><td>0</td><td>6</td><td>0</td><td>-2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>-1.695</td><td>0</td></tr><tr><td>xi</td><td>6</td><td>0</td><td>0</td><td>6</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>-2.5</td><td>0</td></tr><tr><td>xii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.055</td><td>0</td><td>-0.1</td><td>-0.02</td></tr><tr><td>xiii</td><td>2</td><td>0</td><td>0</td><td>4</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td colspan="2">-0.085 -0.22</td></tr><tr><td>xiv</td><td colspan="4">2.9674 -0.4409 0.4409 2.9674</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xv</td><td>6</td><td>0</td><td>0</td><td>2</td><td>-1</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xvi</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1</td><td>-0.06</td><td>0</td><td>-0.08</td><td>3</td></tr><tr><td>xvii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>1</td><td>-2</td><td>3</td><td>1.1</td><td>-0.06</td><td>0</td><td>0.2</td><td>0</td></tr><tr><td>xviii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>3</td><td>2</td><td>2</td><td>3</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xix</td><td>10</td><td>0</td><td>0</td><td>6</td><td>-3</td><td>5</td><td>-5</td><td>3</td><td>0</td><td>0</td><td>-3</td><td>-3</td></tr><tr><td>xx</td><td>1.2</td><td>0</td><td>0</td><td>2</td><td>-17</td><td>35</td><td>10</td><td>-8</td><td>0</td><td>0</td><td>1.2</td><td>-1.2</td></tr><tr><td>xxi</td><td colspan="2">2.9763 -0.376</td><td colspan="2">0.376 2.9763</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0.0315</td><td>0</td><td colspan="2">-0.015 -0.015</td></tr><tr><td>xxii</td><td>6</td><td>0</td><td>0</td><td>6</td><td>0</td><td>-2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xxiii0</td><td>3</td><td>0</td><td>0</td><td>3</td><td>-3</td><td>-5</td><td>-5</td><td>-3</td><td>0</td><td>0</td><td>-2</td><td>-2</td></tr><tr><td>xxiv</td><td>1.5</td><td>0</td><td>0</td><td>2</td><td>-4</td><td>-7</td><td>8</td><td>5</td><td>0</td><td>0</td><td>-0.4</td><td>-1.2</td></tr><tr><td>xxv</td><td>2</td><td>0</td><td>0</td><td>3</td><td>12.4</td><td>11.6</td><td>-8</td><td>-5</td><td>0</td><td>0</td><td>1.8</td><td>7</td></tr><tr><td>xxvi</td><td>3</td><td>0</td><td>0</td><td>3</td><td>5.175</td><td>9</td><td>9</td><td>5.175</td><td>0.3</td><td>0.3</td><td>3.95</td><td>3.95</td></tr><tr><td>xxvii</td><td>7</td><td>0</td><td>0</td><td>3</td><td>6</td><td>3</td><td>9</td><td>6</td><td>0.62</td><td>0.373</td><td>4</td><td>3.4</td></tr><tr><td>xxviii</td><td>6</td><td>0</td><td>0</td><td>10</td><td>0</td><td>0</td><td>-5</td><td>-8</td><td>-0.08</td><td>0</td><td>0</td><td>-2</td></tr><tr><td>xxix</td><td>2</td><td>0</td><td>0</td><td>3</td><td>-10</td><td>-11.6</td><td>8</td><td>5</td><td>0</td><td>0</td><td>4</td><td>4.8</td></tr><tr><td>xxx</td><td>3</td><td>0</td><td>0</td><td>3</td><td>5.26</td><td>9</td><td>9</td><td>5.26</td><td>0.25</td><td>0.25</td><td>3.95</td><td>3.95</td></tr><tr><td>xxxi</td><td>3</td><td>0</td><td>0</td><td>3</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>xxxii</td><td>2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>8</td><td>8</td><td>5</td><td>0</td><td>0</td><td>4.4</td><td>4.4</td></tr><tr><td>xxxiii</td><td>3</td><td>0</td><td>0</td><td>3</td><td>6</td><td>9</td><td>9</td><td>6</td><td>0.3</td><td>0.3</td><td>3.75</td><td>3.75</td></tr><tr><td>xxxiv</td><td>1</td><td>0</td><td>0</td><td>1</td><td>5</td><td>8</td><td>8</td><td>5</td><td>0</td><td>0</td><td>4.4</td><td>4.9</td></tr><tr><td>xxxv</td><td>3</td><td>0</td><td>0</td><td>3</td><td>6</td><td>9</td><td>9</td><td>6</td><td>0.24</td><td>0.24</td><td>3.95</td><td>3.95</td></tr><tr><td>xxxvi</td><td>2</td><td>0</td><td>0</td><td>2</td><td>5</td><td>8</td><td>8</td><td>5</td><td>0</td><td>0</td><td>5</td><td>5</td></tr></table>

Peer review (iclr2019):
 This paper analyzes GRUs from a dynamical systems perspective, i.e. phase diagrams, fixed points, and bifurcations. The abstract and intro are well written and motivate the need for more theoretical framework to understand RNNs, especially how well they are able to represent and express temporal features in the training data. The dynamical systems analysis is well presented and visualized nicely. Most of the paper concentrates on the 1d (one single GRU) and 2d (two GRU*s) case. They show that 2d GRUs can be trained to adopt a variety of fixed points, can approximate a line attractors (an important feature for short-term memory), but cannot mimic a ring attractor. My concerns are: - The derivation of the continuous time dynamical system (Appendix A) is confusing to me. Unless I*m not following the derivation correctly, should there be another .Delta t in the denominator of the right-hand side of (23), from (22)? It*s confusing to me that the continuous-time version in (26) has essentially the same form as the discrete-time version in (22). - The applicability of this analysis to RNNs of even modest size is unclear. Generically, there*s no reason to believe the intuitions from 2d should necessarily generalize to higher dimensions, and rigorous analysis of higher dimensional systems of this kind can be fairly challenging, even if one starts from a continuation analysis. - Small typo: top of Page 4, figure should refer to 3A, not 2A.