Year: 2018

Abstract:
Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRURNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system. We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction.

text: Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRURNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system. We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction.

heading: 1 INTRODUCTION
text: Recurrent neural networks (RNNs) have been widely used to capture and utilize sequential structure in natural and artificial languages, speech, video, and various other forms of time series. The recurrent information flow within RNN implies that the data seen in the past has influence on the current state of the RNN, forming a mechanism for having memory through (nonlinear) temporal traces. Unfortunately, training vanilla RNNs (which allow input data to directly interact with the hidden state) to capture long-range dependences within a sequence is challenging due to the vanishing gradient problem (Hochreiter, 1991). Several special RNN architectures have been proposed to mitigate this issue, notably the long short-term memory (LSTM; Hochreiter & Schmidhuber (1997)) which explicitly guards against unwanted corruption of the information stored in the hidden state until necessary. Recently, a simplification of the LSTM called the gated recurrent unit (GRU; Cho et al. (2014)) has become wildly popular in the machine learning community thanks to its performance in machine translation (Britz et al., 2017), speech (Prabhavalkar et al., 2017), music (Choi et al., 2017), video (Dwibedi et al., 2018), and extracting nonlinear dynamics underlying neural data (Pandarinath et al., 2018). As a variant of the vanilla LSTM, GRUs incorporate the use of forget gates, but lack an output gate (Gers et al., 2000). While this feature reduces the number of required parameters, LSTM has been shown to outperform GRU on neural machine translation (Britz et al., 2017). In addition, certain mechanistic tasks, specifically unbounded counting, come easy to LSTM networks but not to GRU networks (Weiss et al., 2018). Despite these empirical findings, we lack systematic understanding of the internal time evolution of GRU’s memory structure and its capability to represent nonlinear temporal dynamics.
In general, a RNN can be written as ht+1 = f(ht,xt) where xt is the current input in a sequence indexed by t, f is a point-wise nonlinear function, and ht represents the hidden memory state that carries all information responsible for future output. In the absence of input, the hidden state ht can evolve over time on its own:
ht+1 = f(ht) (1)
where f(·) := f(·,0) for notational simplicity. In other words, we can consider the temporal evolution of memory stored within RNN as a trajectory of a dynamical system defined by (1). Then we can use dynamical systems theory to investigate the fundamental limits in the expressive power of RNNs in terms of their temporal features. We develop a novel theoretical framework to study the dynamical features fundamentally attainable, in particular, given the particular form of GRU. We then validate the theory by training GRUs to predict time series with prescribed dynamics.

heading: 2 CONTINUOUS-TIME GATED RECURRENT UNIT
text: The GRU uses two internal gating variables: the update gate zt which protects the d-dimensional hidden state ht ∈ Rd and the reset gate rt which allows overwriting of the hidden state and controls the interaction with the input xt ∈ Rp.
zt = σ(Wzxt+Uzht−1 + bz) (update gate) (2) rt = σ(Wrxt+Urht−1 + br) (reset gate) (3) ht = (1− zt) tanh(Whxt+Uh(rt ht−1 + bh)) + zt ht−1 (hidden state) (4)
where Wz,Wr,Wh ∈ Rd×p and Uz,Ur,Uh ∈ Rd×d are the parameter matrices, bz,br,bh ∈ Rd are bias vectors, represents the Hadamard product, and σ(z) = 1/(1 + e−z) is the element-wise logistic sigmoid function. Note that the hidden state is asymptotically contained within [−1, 1]d due to the saturating nonlinearities, implying if the state is initialized outside of this trapping region, it must eventually enter it in finite time and remain in it for all later time.
Note that the update gate zt controls how fast each dimension at the hidden state decays, providing an adaptive time constant for memory. Specifically, as limzt→0 ht = ht−1, GRUs can implement perfect memory of the past and ignore xt. Hence, a d-dimensional GRU is capable of keeping a near constant memory through the update gate—near constant since 0 < [zt]j < 1, where [·]j denotes j-th component of a vector. Moreover, the autoregressive weights (mainly Uh and Ur) can support time evolving memory (Laurent & von Brecht (2016)) considered this a hindrance and proposed removing all complex dynamic behavior in a simplified GRU).
To investigate the memory structure further, let us consider the dynamics of hidden state in the absence of input, i.e. xt = 0,∀t, which is of the form (1). To utilize the rich descriptive language of continuous time dynamical system theory, we consider the following continuous time limit of the (autonomous) GRU time evolution:
z(t) = σ(Uzh(t) + bz) (continuous update gate) (5) r(t) = σ(Urh(t) + br) (continuous reset gate) (6)
ḣ = (z(t)− 1) (h(t)− tanh(Uh(r(t) h(t)) + bh)) (continuous hidden state dynamics) (7)
where ḣ ≡ dh(t)dt . Since both σ(·) and tanh(·) are smooth, this continuous limit is justified. The update gate z(t) again plays the role of a state-dependent time constant for memory decay. Furthermore, since 1 − z(t) > 0, it does not change the topological structure of the dynamics (only speed). For the following theoretical analysis sections (3 & 4), we can safely ignore the effects of z(t). A derivation of the continuous-time GRU can be found in appendix A.

heading: 3 STABILITY ANALYSIS OF A ONE DIMENSIONAL GRU
text: For a single GRU∗ (d = 1), (7) reduces to a one dimensional dynamical system where every variable is a scalar. The expressive power of a single GRU is quite limited, as only three stability structures (topologies) exist (see appendix B): (A) a single stable node, (B) a stable node and a half-stable node, and (C) two stable nodes separated by an unstable node (see Fig. 1). The corresponding temporal features are (A) decay to a fixed value, (B) decay to a fixed value, but from one direction halt at an intermediate value until perturbed, or (C) decay to one of two fixed values (bistability). The bistability can be used to capture a binary latent state in the sequence. It should be noted that a one dimensional continuous time autonomous system cannot exhibit oscillatory behavior, as is the case here (Hirsch et al., 2013).
The topology the GRU takes is determined by its parameters. If the GRU begins in a region of the parameter space corresponding to (A), we can smoothly vary the parameters to transverse (B) in the parameter space, and end up at (C).
∗The number/dimension of GRUs referenced will indicate the dimension of the latent dynamics of a GRU network.
This is commonly known as a saddle-node bifurcation. Speaking generally, a bifurcation is the change in topology of a dynamical system, resulting from a smooth change in parameters. The point in the parameters space at which the bifurcation occurs is called the bifurcation point, and we will refer to the fixed point that changes its stability at the bifurcation point as the bifurcation fixed point. This corresponds to the parameters underlying (B) in our previous example. The codimension of a bifurcation is the number of parameters which must vary in order to achieve the bifurcation. In the case of our example, a saddle-node bifurcation is codimension-1 (Kuznetsov, 1998). Right before transitioning to (B), from (A), the flow near where the half-stable node would appear can exhibit arbitrarily slow flow. We will refer to these as slow points (Sussillo & Barak, 2012).

heading: 4 ANALYSIS OF A TWO DIMENSIONAL GRU
text: We will see that the addition of a second GRU opens up a substantial variety of possible topological structures when compared with the use of a single GRU. For notational simplicity, we denote the two dimensions of h as x and y. We visualize the flow fields defined by (7) in 2-dimension as phase portraits which reveal the topological structures of interest. For starters, the phase portrait of two independent bistable GRUs can be visualized as Figure 2A. It clearly shows 4 stable states as expected, with a total of 9 stable fixed points. This could be thought of as a continuoustime continuous-space implementation of a finite state machine with 4 states (Fig. 2B). The 3 types of observed fixed points—stable (sinks), unstable (sources), and saddle points—exhibit locally linear dynamics, however, the global geometry is nonlinear and their topological structures can vary depending on their arrangement.
We explored stability structures attainable by two GRUs. Due to the relatively large number of observed topologies, this section’s main focus will be on demonstrating all observed local dynamical features obtainable by two GRUs. In addition, existence of two non-local dynamical features will be presented. A complete catalog of all observed topologies can be found in the appendix C, along with the parameters of every phase portrait depicted in this paper.
Before proceeding, let us take this time to describe all the local dynamical features observed. In addition to the previously mentioned three types of fixed points, two GRUs can exhibit a variety of bifurcation fixed points, resulting from regions of parameter space that separate all topologies restricted to simple fixed points (i.e stable, unstable, and saddle points). Behaviorally speaking, these fixed points act as hybrids between the previous three, resulting in a much richer set of obtainable dynamics. These bifurcation fixed points fall into two categories, separated by codimension. More specifically, two GRUs have been seen to feature both codimension-1 and codimension-2 bifurcation (fixed) points. Beginning with codimension-1, we have the saddle-node bifurcation fixed point, as expected from its existence in the single GRU case. We can further classify these points into two types. These can be thought of as both the fusion
of a stable fixed point and a saddle point, and the fusion of an unstable fixed point and a saddle point. We will refer to these fixed points as saddle-node bifurcation fixed points of the first kind and second kind respectively.
One type of codimension-2 bifurcation fixed point that has been observed in the two GRU system acts as the fusion of all three simple fixed points. More specifically, these points arise from the fusion of a stable fixed point, unstable fixed point, and two saddle points. All of these local structures are depicted in figure 3.
While the existence of simple fixed points was already demonstrated (see Fig. 2A). Figure 3A demonstrates the maximum number of fixed points observed in a two GRU system, for a given set of parameters. A closer look at this system reveals its potential interpretation as a continuous analogue of 5-discrete states with input-driven transitions, similar to that depicted in figure 2, implying additional GRUs are needed for any Markov process modeled in this manner, requiring more than five discrete states. We conjecture that the system depicted in figure 3A is the only eleven fixed point structure obtainable with two GRUs, as all observed structures containing the same number of fixed points are topologically equivalent to one another.
The addition of bifurcation fixed points opens the door to dynamically realize more sophisticated models. Take for example the four state system depicted in figure 3B. If the hidden state is set to initialize in the first quadrant of phase space, the trajectory will flow towards the codimension-2 bifurcation fixed point at the origin. Introducing noise through the input will stochastically cause the trajectory to approach the stable fixed point at (-1,-1) either directly, or by first flowing into one of the two saddle-node bifurcation fixed points of the first kind. Models of this sort can be used in a variety of applications, such as neural decision making (Wong & Wang (2006), Churchland & Cunningham (2014)).
We will begin our investigation into the non-local dynamics observed with two GRUs by showing the existence of homoclinic orbits. A trajectory initialized on a homoclinic orbit will approach the same fixed point in both forward and backward time. We observe that two GRUs can exhibit one or two bounded planar regions of homoclinic orbits for a given set of parameters, as shown in figure 4A and 4B respectively. Any trajectory initialized in one of these regions will flow into the codimension-2 bifurcation fixed point at the origin, regardless of which direction time flows in. This featured behavior enables the accurate depiction of various models, including neuron spiking (Izhikevich, 2007).
In regards to the second non-local dynamic feature, it can be shown that two GRUs can exhibit an Andronov-Hopf bifurcation, whereby a stable fixed point bifurcates into an unstable fixed point surrounded by a limit cycle. Behaviorally speaking, a limit cycle is a type of attractor, in the sense that there exists a defined basin of attraction. However, unlike a stable fixed point, where trajectories initialized in the basin of attraction flow towards a single point, a limit cycle pulls trajectories into a stable periodic orbit around the unstable fixed point at its center. To demonstrate this phenomenon, let (8) define the parameters of (7).
Uz, Ur, bz, br, bh = 0, Uh = 3
2
[ cosα − sinα sinα cosα ] (8)
where α ∈ R+. If α = π3 , the system has a single stable fixed point (stable spiral), as depicted in figure 5A. If we continuously decrease α, the system undergoes an Andronov-Hopf bifurcation approximately about α = π3.8 . As α continuously decreases, the orbital period increases, and as the nullclines can be made arbitrarily close together, the length of this
orbital period can be set arbitrarily, up to machine accuracy. Figure 5B shows an example of a relatively short orbital period, and figure 5C depicts the behavior seen for slower orbits.
With finite-fixed point topologies and global structures out of the way, the next logical question to ask is, can two GRUs exhibit an infinite number of fixed points (countable or uncountable)? Such behavior is often desirable in models that require stationary attraction to non-point structures, such as line attractors and ring attractors. The short answer to this question is no. Lemma 1. Any two dimensional GRU can only have finitely many simple fixed points.
This follows from Lefschetz theory (Guillemin & Pollack, 2010). The detailed proof can be found in appendix D, and is intended to give the reader intuition behind the result presented in the claim extended to aribitrary dimensional GRU in theorem 1.
Theorem 1. Any finite dimensional GRU can only have finitely many simple fixed points and bifurcation fixed points.
Proof. By definition of simple fixed points, the Jacobian of the dynamics at those fixed points have nonzero real parts, making them Lefschetz fixed points. Since GRU dynamics is asymptotically bounded on the compact set [−1, 1]d, where d is the number of GRUs, it follows from Lefshetz theory (Guillemin & Pollack, 2010) that there are finitely many simple fixed points. Furthermore, by construction, a bifurcation fixed point can only exist within a stability structure if and only if there exists a separate topology, such that the simple fixed points making up each bifurcation fixed point exist isolated from one another. Therefore, there can only exist finitely many isolated bifurcation fixed points.
This eliminates the possibility of having countably many fixed points. Next, we show that there cannot be uncountably many non-isolated fixed points.
Theorem 2. Any finite dimensional GRU cannot have a continuous attractor.
Proof. We provide a sketch of a proof. Let h∗ ∈ Rd be a fixed point of a d-dimensional GRU, that is, h∗ − tanh(Uh(r(h∗) h∗) + bh) = 0. Now for any unit norm vector k ∈ Sd−1, and for any δ > 0, we can show that there exist an > 0 such that, ‖ḣ(h∗ + δk)− ḣ(h∗)‖ = ‖ḣ(h∗ + δk)‖ > . This can be argued by taking three cases into consideration, (a) Urk = 0 and Uh(σ(br) k) 6= 0, (b) Urk = 0 and Uh(σ(br) k) = 0, and (c) Urk 6= 0. In each case, it reverts to a 1-dimensional problem where it can be trivially shown to have no continuous attractor around h∗ along direction k. Thus, we conclude that there is no continuous attractor in any direction.
Despite this limitation, an approximation of a line attractor can be created using two GRUs. This approximation can be made arbitrarily close to an actual line attractor on a finite region in phase space, thereby satisfying computational needs on an arbitrary interval when scaled. We will refer to this phenomenon as a pseudo-line attractor. Figure 6 depicts an example of such an attractor.
We conclude this section with a discussion of slow points in the two GRU system. As a logical extension to the single GRU system, slow points occur where the nullclines are sufficiently close together, but do not intersect, as demonstrated in figure 7. Given the previously discussed classes of dynamic behavior for two GRUs, slow points can only exist so long as the potential for a saddle-node bifurcation fixed point is possible in the location of the desired slow point, given an appropriate change in parameters, as they result from the collision and annihilation of two fixed points. This observation is consistent with the single GRU case, as slow points can only exist for a single fixed point. This would imply that given the one fixed point case, a maximum of five slow points are possible. However, this would imply that there must exist a six fixed point case by which five of the six fixed points exist at saddle-node bifurcation fixed points, which has not been observed (see appendix C). Despite this shortcoming, four simultaneous slow points are obtainable, as shown in figure 5C.

heading: 5 NUMERICAL EXPERIMENTS
text: As a means to put our theory to practice, in this section we explore several examples of time series prediction of continuous time planar dynamical systems using two GRUs. Results from the previous section indicate what dynamical features can be learned by this RNN, and suggest cases by which training will fail. All of the following computer experiments consist of an RNN, by which the hidden layer is made up of two GRUs, followed by a linear output layer. The network is trained to make a 29-step prediction from a given initial observation, and no further input through prediction. As such, to produce accurate predictions, the RNN must rely solely on the hidden layer dynamics.
We train the network to minimize the following multi-step loss function:
L(θ) = 1 T Ntraj∑ i=1 T∑ k=1 ‖ŵi(k;wi(0))−wi(k)‖22 (9)
where θ are the parameters of the GRU and linear readout, T = 29 is the prediction horizon, wi(t) is the i-th time series generated by the true system, and ŵ(k;w0) is k-step the prediction given w0.
The hidden states are initialized at zero for each trajectory. The RNN is then trained for 4000 epochs, using ADAM (Kingma & Ba, 2014) in whole batch mode to minimize the loss function, i.e., the mean square error between the predicted trajectory and the data. Ntraj = 667 time series were used for training. Figure 8 depicts the experimental results of the RNN’s attempt at learning each dynamical system we describe below.

heading: 5.1 LIMIT CYCLE
text: To test if two GRUs can learn a limit cycle, we use a simple nonlinear oscillator called the FitzHugh-Nagumo Model. The FitzHugh-Nagumo model is defined by:
ẋ = x− x 3
3 − y + Iext, τ ẏ = x+ a− by (10)
where in this experiment we will chose τ = 12.5, a = 0.7, b = 0.8, and Iext = N (0.7, 0.04). Under this choice of model parameters, the system will exhibit an unstable fixed point (unstable spiral) surrounded by a limit cycle (Fig. 8). As shown in section 4, two GRUs are capable of representing this topology. The results of this experiment verify this claim (Fig. 8), as two GRUs can capture topologically equivalent dynamics.

heading: 5.2 LINE ATTRACTOR
text: As discussed in section 4, two GRUs can exhibit a pseudo-line attractor, by which the system mimics an analytic line attractor. We will use the simplest representation of a planar line attractor:
ẋ = −x, ẏ = 0 (11)
This system will exhibit a line attractor along the y-axis, at x = 0 (Fig. 8). Trajectories will flow directly perpendicular towards the attractor. We added white Gaussian noiseN (0, 0.1I) in the training data. While the hidden state dynamics of the trained network do not perfectly match that of an analytic line attractor, there exists a subinterval near each of the fixed points acting as a pseudo-line attractor (Fig. 8). As such, the added affine transformation (linear readout) can scale and reorient this subinterval as is required by a given problem, thereby mimicking a line attractor.

heading: 5.3 RING ATTRACTOR
text: For this experiment, a dynamical system representing a standard ring attractor of radius one is used:
ẋ = −(x2 + y2 − 1)x, ẏ = −(x2 + y2 − 1)y (12)
This system exhibits an attracting ring, centered around an unstable fixed point. We added Gaussian noiseN (0, 0.1I) to the training data.
Two GRUs will not be able to accurately capture the system’s continuous attractor dynamics as expected from theorem 3. The results of this experiment are demonstrated in figure 8. As expected, the RNN fails to capture the proper dynamical features of the ring attractor. Rather, the hidden state dynamics fall into an observed finite fixed point topology (see case xxix in appendix C). In addition, we robustly see this over multiple initializations, and the quality of approximation improves as the dimensionality of GRU increases (Fig. 9).

heading: 6 CONCLUSION
text: Our analysis shows the rich but limited classes of dynamics the GRU can approximate in one, two, and arbitrary dimensions. We developed a new theoretical framework to analyze GRUs as a continuous dynamical system, and showed that two GRUs can exhibit a variety of expressive dynamic features, such as limit cycles, homoclinic orbits, and a substantial catalog of stability structures and bifurcations. However, we also showed that finitely many GRUs cannot exhibit the dynamics of an arbitrary continuous attractor. These claims were then experimentally verified in two dimensions. We believe these findings also unlock new avenues of research on the trainability of recurrent neural networks. Although we have analyzed GRUs only in 1- and 2- dimensions in near exhaustive, we believe that the insights extends to higher-dimensions. We leave rigorous analysis of higher-dimensional GRUs as future work.

heading: A CONTINUOUS TIME SYSTEM DERIVATION
text: We begin with the fully gated GRU as a discrete time system, where the input vector xt has been set equal to zero, as depicted in (13) - (15), where is the Hadamard product, and σ is the sigmoid function.
zt = σ(Uzht−1 + bz) (13) rt = σ(Urht−1 + br) (14)
ht = zt ht−1 + (1− zt) tanh (Uh(rt ht−1) + bh) (15) We recognize that (15) is a forward Euler discretization of a continuous time dynamical system. This allows us to consider the underlying continuous time dynamics on the basis of the discretization. The following steps are a walk through of the derivation:
Since zt is a bounded function on R ∀t, there exists a function z̃t, such that zt + z̃t = 1 at each time step (due to the symmetry of zt, z̃t is the result of vertically flipping zt about 0.5, the midpoint of its range). As such, we can rewrite (15) with z̃t as depicted in (16).
ht = (1− z̃t) ht−1 + z̃t tanh (Uh(rt ht−1) + bh) (16) where,
z̃t = σ(Ũzht−1 + b̃z) (17) ht = ht−1 − z̃t ht−1 + z̃t tanh (Uh(rt ht−1) + bh) (18) ht − ht−1 = −z̃t (ht−1 − tanh (Uh(rt ht−1) + bh)) (19)
Let h(t) ≡ ht−1. As a result, we can say z̃t ≡ z̃(t) and rt ≡ r(t), as depicted in (20). h(t+ 1)− h(t) = −z̃(t) (h(t)− tanh (Uh(r(t) h(t)) + bh)) (20)
where, z̃(t) = σ(Ũzh(t) + b̃z) (21) r(t) = σ(Urh(t) + br) (22)
Let ∆t define an arbitrary time interval. Then (20) becomes, h(t+ ∆t)− h(t) = −z̃(t) (h(t)− tanhUh(r(t) h(t)) + bh)∆t (23)
Dividing both sides of the equation by ∆t yields (24). h(t+ ∆t)− h(t)
∆t = −z̃(t) (h(t)− tanhUh(r(t) h(t)) + bh) (24)
If we take the limit as ∆t→ 0, we get the analogous continuous time system to (13) - (15), ḣ = −z̃(t) (h(t)− tanh (Uh(r(t) h(t)) + bh)) (25)
where ḣ ≡ dh(t)dt Finally, we can rewrite (25) as follows:
ḣ = (z(t)− 1) (h(t)− tanh (Uh(r(t) h(t)) + bh)) (26) where
z(t) = σ(Uzh(t) + bz) (27)

heading: B SINGLE GRU FIXED POINT PROOFS
text: The fixed points of our continuous time system (25) exist where the derivative ḣ = 0. In the single GRU case, the Hadamard product reduces to standard scalar multiplication, yielding,
0 = (z(t)− 1)[h∗ − tanh (Uhr(t)h∗ + bh)] (28) where z(t) and r(t) are defined by (27) and (22) respectively, and h∗ ∈ R represents a solution of (28). We can divide out z(t) − 1, indicating that the update gate does not play a part in the stability of the system. For simplicity, lets expand r(t) in (28) by its definition (22).
0 = tanh (Uhσ(Urh ∗ + br)h ∗ + bh)− h∗ (29) where Uh, bh, Ur, br ∈ R.
Lemma 2. for all Uh, bh, Ur, br, there exists h∗ such that (29) is satisfied.
Proof. The hyperbolic tangent function is continuous and bounded on R, having a range of (−1, 1). −h(t) ≡ −h is monotonic and achieves all values on R, as −h is bijective on R. Thus, their sum is unbounded and obtains every value on R at least once. By the intermediate value theorem, there is at least one point h∗ such that (29) is satisfied, regardless of the choice of parameters Uh, bh, Ur, br.
Lemma 3. There exists a set of parameters Uh, bh, Ur, br such that there exists one, two, or three solutions to (29).
Proof. We will prove this lemma by showing the existence of each case. Let Ur = 80, br = 40, and Uh = −60. We then allow bh to vary. The existence of each of the three cases are shown in figure 1.
If bh = −1, there exists a single solution to (29). If bh decreases continuously, a second root appears and splits in two. Analogously, the system (25) goes through a saddle-node bifurcation, where a half-stable fixed point appears, and splits into a stable/unstable fixed point pair.
Theorem 3. For any choice of parameters Ur, br, Uh, bh, there can only exist one, two, or three solutions to (29), and all solutions exist on the interval (-1,1).
Proof. We begin with the argument of the hyperbolic tangent function in (29),
Uhσ(Urh+ br)h+ bh (30)
Taking the derivative of (30) yields,
Uh(1 + e −Urh−br ) + UhUrhe −Urh−br
(1 + e−Urh−br )2 (31)
Setting (31) to zero and simplifying will allow us to find the nontrivial critical points of (30), as shown in (32). Note that if Uh = 0, (30) is equal to bh ∀h, yielding no critical points.
1 + e−Urh−br (1 + Urh) = 0 (32)
Let x ≡ e−br and ĥ ≡ Urh, and solve (32) for ĥ.
ĥ = −W ( 1 xe )− 1 (33)
where W is principal branch of the Lambert W function. Therefore, (30) has exactly one local maximum or minimum, so long as Uh 6= 0.
Now consider, tanh (Uhσ(Urh+ br)h+ bh) (34)
The hyperbolic tangent function preserves intervals of monotonic behavior in its argument. Therefore, (34) has at most one local maximum or minimum.
We take into account the fact that the hyperbolic tangent function bounds its argument on the interval (-1,1). If there exists a subset S = [a, 1], for some a ∈ [−1, 1) such that (34) is increasing, then there exists a k ∈ S such that when h = k (35) and (36) hold.
d
dh (tanh (Uhσ(Urh+ br)h+ bh)− h) = 0 (35)
d
dh (tanh (Uhσ(Urh+ br)h+ bh)− h) < 0,∀h > k (36)
This result in conjunction with the previous two lemmas completes the proof.

heading: C ALL TOPOLOGICAL STABILITY STRUCTURES OBSERVED WITH TWO GRUS
text: Table 2 depicts all observed topologies of multiple-fixed point structures using two GRUs. Figure 10 displays an example of a phase portrait from a two GRU system for each case listed in 2. Note that all fixed points are denoted by a red dot, regardless of classification. Table 3 lists the parameters used for each of the observed cases. Note that all the update gate parameters are set to zero.
Each case in this paper was discovered by hand by considering the geometric constraints on the structure of nullclines for both the decoupled and coupled system (i.e reset gate inactive and active respectively). An exhaustive analysis on the one dimensional GRU allowed for a natural extension into the two dimensional decoupled GRU. Upon establishing a set of base cases (a combinatorial argument regarding all possible ways the decoupled nullclines [topologically conjugate to linear and cubic polynomials] can intersect) From these base cases, the reset gate can be used as a means of bending and manipulating structure of the decoupled nullclines in order to obtain new intersection patterns in the coupled system.

heading: D PROOF OF LEMMA 1
text: We begin this proof by showing that all fixed points obtainable with two GRUs are Lefschetz fixed points. To show this is the case let (37) expand our previous notation. We set all elements in Uz and bz to zero, as the update gate plays no part in the topology of (7) (shown in appendix B).
Uh = [ Uh11 Uh12 Uh21 Uh22 ] , Ur = [ Ur11 Ur12 Ur21 Ur22 ] , bh = [ bh1 bh2 ] , br = [ br1 br2 ] (37)
We can now rewrite (7) expanded in terms of the individual elements of Uh, Ur, bh, and br, as shown in (38) and (39).
ẋ = −1 2 [x− tanh( Uh11x 1 + e−(Ur11x+Ur12y+br1) + Uh12y 1 + e−(Ur21x+Ur22y+br2) + bh1)] (38)
ẏ = −1 2 [y − tanh( Uh21x 1 + e−(Ur11x+Ur12y+br1) + Uh22y 1 + e−(Ur21x+Ur22y+br2) + bh2)] (39)
If 0 is not an eigenvalue of the Jacobian matrix of (38) and (39) at a fixed point, the fixed point is said to be Lefshetz. Let Fx = dẋdx , Fy = dẋ dy , Gx = dẏ dx , and Fy = dẏ dy .
where
Fx = − 1
2 (1− sech2
( Uh11x
e−Ur11x−Ur12y−br1 + 1 +
Uh12y
e−Ur21x−Ur22y−br2 + 1 + bh1
) γFx) (40)
γFx = Uh11Ur11xe
−Ur11x−Ur12y−br1
(e−Ur11x−Ur12y−br1 + 1) 2 + Uh11 e−Ur11x−Ur12y−br1 + 1 + Uh12Ur21ye
−Ur21x−Ur22y−br2
(e−Ur21x−Ur22y−br2 + 1) 2 (41)
Fy = 1
2 sech2
( Uh11x
e−Ur11x−Ur12y−br1 + 1 +
Uh12y
e−Ur21x−Ur22y−br2 + 1 + bh1
) γFy (42)
γFy = Uh11Ur11xe
−Ur11x−Ur12y−br1
(e−Ur11x−Ur12y−br1 + 1) 2 + Uh12 e−Ur21x−Ur22y−br2 + 1 + Uh12Ur21ye
−Ur21x−Ur22y−br2
(e−Ur21x−Ur22y−br2 + 1) 2 (43)
Gx = 1
2 sech2
( Uh21x
e−Ur11x−Ur12y−br1 + 1 +
Uh22y
e−Ur21x−Ur22y−br2 + 1 + bh2
) γGx (44)
γGx = Uh21Ur11xe
−Ur11x−Ur12y−br1
(e−Ur11x−Ur12y−br1 + 1) 2 + Uh21 e−Ur11x−Ur12y−br1 + 1 + Uh22Ur21ye
−Ur21x−Ur22y−br2
(e−Ur21x−Ur22y−br2 + 1) 2 (45)
Gy = − 1
2 (1− sech2
( Uh21x
e−Ur11x−Ur12y−br1 + 1 +
Uh22y
e−Ur21x−Ur22y−br2 + 1 + bh2
) γGy) (46)
γGy = Uh21Ur11xe
−Ur11x−Ur12y−br1
(e−Ur11x−Ur12y−br1 + 1) 2 + Uh22 e−Ur21x−Ur22y−br2 + 1 + Uh22Ur21ye
−Ur21x−Ur22y−br2
(e−Ur21x−Ur22y−br2 + 1) 2 (47)
Let J denote the Jacobian matrix of (38) and (39).
J = [ Fx Fy Gx Gy ] (48)
Note that we can rewrite (38) and (39) as follows:
ẋ = −1 2 [x− tanh(f(x, y, θ))] (49)
ẏ = −1 2 [y − tanh(g(x, y, θ))] (50)
where θ represents the set of parameters, f(x, y, θ) = Uh11x 1+e−(Ur11x+Ur12y+br1) + Uh12y 1+e−(Ur21x+Ur22y+br2) + bh1, and g(x, y, θ) = Uh21x 1+e−(Ur11x+Ur12y+br1) + Uh22y 1+e−(Ur21x+Ur22y+br2) + bh2
An ordered pair (x, y) is a fixed point of (49) and (50) if and only if (51) and (52) hold.
0 = −1 2 [x− tanh(f(x, y, θ))] (51)
0 = −1 2 [y − tanh(g(x, y, θ))] (52)
As such, we can say the following: x = tanh(f(x, y, θ)) = u(x, y, θ) (53) y = tanh(g(x, y, θ)) = v(x, y, θ) (54)
If we let λ represent the eigenvalues of (48), the characteristic equation of (48) is as follows:
λ2 +λ(−1− sech2(f(x, y, θ))∂f ∂x − sech2(g(x, y, θ))∂g ∂y ) + 1 4 (1− sech2(f(x, y, θ))sech2(g(x, y, θ))∂f ∂y ∂g ∂x ) (55)
We can rewrite (55) in terms of u(x, y, θ) and v(x, y, θ) as shown in (56)
λ2 + λ(−1− ux − vy) + 1
4 (1− uyvx) (56)
where ux ≡ ∂u∂x , uy ≡ ∂u ∂y , vx ≡ ∂v ∂x , vy ≡ ∂v ∂y . We can use the quadratic formula to solve for λ.
λ = 1 + ux + vy
2 ±
√ u2x + 2ux + 2uxvy + 2vy + v 2 y + uyvx
2 (57)
Setting λ = 0 and simplifying yields the following constraint:
|uyvx| = 1 (58)
which can be realized as follows:
sech2(f(x, y, θ))sech2(g(x, y, θ)) = 1
∂f ∂y ∂g ∂x
(59)
We observe that sech2(f(x, y, θ))sech2(g(x, y, θ)) ∈ (0, 1), which implies that ∂f∂y ∂g ∂x ∈ (1,∞) However, from (53) and (54), we see that f(x, y, θ) = tanh−1(x) and g(x, y, θ) = tanh−1(y) at a critical point. Which implies ∂f ∂y ∂g ∂x = 0 /∈ (1,∞). Therefore λ 6= 0 ∀θ.
This implies that (38) and (39) is a Lefschetz map. Since (38) and (39) are asymptotically bound to (−1, 1)2, we can always find a finite time t0 such that x, y ∈ (−1, 1)2 ∀t > t0. Therefore, for every trajectory initialized outside of the trapping region, we can always find a point on [−1, 1]2 that arises as the transition of that initial condition flowing into the trapping region. This implies that (38) and (39) can be thought of as existing on a compact set, and therefore has a finite number of simple fixed points Guillemin & Pollack (2010).

Review summaries:
The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset. Their analysis shows what kind of hidden state dynamics the GRU can approximate in one and two dimensions. In the experimental part, they show how a GRU with two hidden states trained on a multistep prediction task can learn such dynamics.