The Expressive Power of Gated Recurrent Units as a Continuous Dynamical System

Gated recurrent units ( GRUs ) were inspired by the common gated recurrent unit , long short-term memory ( LSTM ) , as a means of capturing temporal structure with less complex memory unit architecture . Despite their incredible success in tasks such as natural and artificial language processing , speech , video , and polyphonic music , very little is understood about the specific dynamic features representable in a GRU network . As a result , it is difficult to know a priori how successful a GRURNN will perform on a given data set . In this paper , we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system , and classify the dynamical features obtainable with such system . We found rich repertoire that includes stable limit cycles over time ( nonlinear oscillations ) , multi-stable state transitions with various topologies , and homoclinic orbits . In addition , we show that any finite dimensional GRU can not precisely replicate the dynamics of a ring attractor , or more generally , any continuous attractor , and is limited to finitely many isolated fixed points in theory . These findings were then experimentally verified in two dimensions by means of time series prediction . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) have been widely used to capture and utilize sequential structure in natural and artificial languages , speech , video , and various other forms of time series . The recurrent information flow within RNN implies that the data seen in the past has influence on the current state of the RNN , forming a mechanism for having memory through ( nonlinear ) temporal traces . Unfortunately , training vanilla RNNs ( which allow input data to directly interact with the hidden state ) to capture long-range dependences within a sequence is challenging due to the vanishing gradient problem ( Hochreiter , 1991 ) . Several special RNN architectures have been proposed to mitigate this issue , notably the long short-term memory ( LSTM ; Hochreiter & Schmidhuber ( 1997 ) ) which explicitly guards against unwanted corruption of the information stored in the hidden state until necessary . Recently , a simplification of the LSTM called the gated recurrent unit ( GRU ; Cho et al . ( 2014 ) ) has become wildly popular in the machine learning community thanks to its performance in machine translation ( Britz et al. , 2017 ) , speech ( Prabhavalkar et al. , 2017 ) , music ( Choi et al. , 2017 ) , video ( Dwibedi et al. , 2018 ) , and extracting nonlinear dynamics underlying neural data ( Pandarinath et al. , 2018 ) . As a variant of the vanilla LSTM , GRUs incorporate the use of forget gates , but lack an output gate ( Gers et al. , 2000 ) . While this feature reduces the number of required parameters , LSTM has been shown to outperform GRU on neural machine translation ( Britz et al. , 2017 ) . In addition , certain mechanistic tasks , specifically unbounded counting , come easy to LSTM networks but not to GRU networks ( Weiss et al. , 2018 ) . Despite these empirical findings , we lack systematic understanding of the internal time evolution of GRU ’ s memory structure and its capability to represent nonlinear temporal dynamics . In general , a RNN can be written as ht+1 = f ( ht , xt ) where xt is the current input in a sequence indexed by t , f is a point-wise nonlinear function , and ht represents the hidden memory state that carries all information responsible for future output . In the absence of input , the hidden state ht can evolve over time on its own : ht+1 = f ( ht ) ( 1 ) where f ( · ) : = f ( ·,0 ) for notational simplicity . In other words , we can consider the temporal evolution of memory stored within RNN as a trajectory of a dynamical system defined by ( 1 ) . Then we can use dynamical systems theory to investigate the fundamental limits in the expressive power of RNNs in terms of their temporal features . We develop a novel theoretical framework to study the dynamical features fundamentally attainable , in particular , given the particular form of GRU . We then validate the theory by training GRUs to predict time series with prescribed dynamics . 2 CONTINUOUS-TIME GATED RECURRENT UNIT . The GRU uses two internal gating variables : the update gate zt which protects the d-dimensional hidden state ht ∈ Rd and the reset gate rt which allows overwriting of the hidden state and controls the interaction with the input xt ∈ Rp . zt = σ ( Wzxt+Uzht−1 + bz ) ( update gate ) ( 2 ) rt = σ ( Wrxt+Urht−1 + br ) ( reset gate ) ( 3 ) ht = ( 1− zt ) tanh ( Whxt+Uh ( rt ht−1 + bh ) ) + zt ht−1 ( hidden state ) ( 4 ) where Wz , Wr , Wh ∈ Rd×p and Uz , Ur , Uh ∈ Rd×d are the parameter matrices , bz , br , bh ∈ Rd are bias vectors , represents the Hadamard product , and σ ( z ) = 1/ ( 1 + e−z ) is the element-wise logistic sigmoid function . Note that the hidden state is asymptotically contained within [ −1 , 1 ] d due to the saturating nonlinearities , implying if the state is initialized outside of this trapping region , it must eventually enter it in finite time and remain in it for all later time . Note that the update gate zt controls how fast each dimension at the hidden state decays , providing an adaptive time constant for memory . Specifically , as limzt→0 ht = ht−1 , GRUs can implement perfect memory of the past and ignore xt . Hence , a d-dimensional GRU is capable of keeping a near constant memory through the update gate—near constant since 0 < [ zt ] j < 1 , where [ · ] j denotes j-th component of a vector . Moreover , the autoregressive weights ( mainly Uh and Ur ) can support time evolving memory ( Laurent & von Brecht ( 2016 ) ) considered this a hindrance and proposed removing all complex dynamic behavior in a simplified GRU ) . To investigate the memory structure further , let us consider the dynamics of hidden state in the absence of input , i.e . xt = 0 , ∀t , which is of the form ( 1 ) . To utilize the rich descriptive language of continuous time dynamical system theory , we consider the following continuous time limit of the ( autonomous ) GRU time evolution : z ( t ) = σ ( Uzh ( t ) + bz ) ( continuous update gate ) ( 5 ) r ( t ) = σ ( Urh ( t ) + br ) ( continuous reset gate ) ( 6 ) ḣ = ( z ( t ) − 1 ) ( h ( t ) − tanh ( Uh ( r ( t ) h ( t ) ) + bh ) ) ( continuous hidden state dynamics ) ( 7 ) where ḣ ≡ dh ( t ) dt . Since both σ ( · ) and tanh ( · ) are smooth , this continuous limit is justified . The update gate z ( t ) again plays the role of a state-dependent time constant for memory decay . Furthermore , since 1 − z ( t ) > 0 , it does not change the topological structure of the dynamics ( only speed ) . For the following theoretical analysis sections ( 3 & 4 ) , we can safely ignore the effects of z ( t ) . A derivation of the continuous-time GRU can be found in appendix A . 3 STABILITY ANALYSIS OF A ONE DIMENSIONAL GRU . For a single GRU∗ ( d = 1 ) , ( 7 ) reduces to a one dimensional dynamical system where every variable is a scalar . The expressive power of a single GRU is quite limited , as only three stability structures ( topologies ) exist ( see appendix B ) : ( A ) a single stable node , ( B ) a stable node and a half-stable node , and ( C ) two stable nodes separated by an unstable node ( see Fig . 1 ) . The corresponding temporal features are ( A ) decay to a fixed value , ( B ) decay to a fixed value , but from one direction halt at an intermediate value until perturbed , or ( C ) decay to one of two fixed values ( bistability ) . The bistability can be used to capture a binary latent state in the sequence . It should be noted that a one dimensional continuous time autonomous system can not exhibit oscillatory behavior , as is the case here ( Hirsch et al. , 2013 ) . The topology the GRU takes is determined by its parameters . If the GRU begins in a region of the parameter space corresponding to ( A ) , we can smoothly vary the parameters to transverse ( B ) in the parameter space , and end up at ( C ) . ∗The number/dimension of GRUs referenced will indicate the dimension of the latent dynamics of a GRU network . This is commonly known as a saddle-node bifurcation . Speaking generally , a bifurcation is the change in topology of a dynamical system , resulting from a smooth change in parameters . The point in the parameters space at which the bifurcation occurs is called the bifurcation point , and we will refer to the fixed point that changes its stability at the bifurcation point as the bifurcation fixed point . This corresponds to the parameters underlying ( B ) in our previous example . The codimension of a bifurcation is the number of parameters which must vary in order to achieve the bifurcation . In the case of our example , a saddle-node bifurcation is codimension-1 ( Kuznetsov , 1998 ) . Right before transitioning to ( B ) , from ( A ) , the flow near where the half-stable node would appear can exhibit arbitrarily slow flow . We will refer to these as slow points ( Sussillo & Barak , 2012 ) . 4 ANALYSIS OF A TWO DIMENSIONAL GRU . We will see that the addition of a second GRU opens up a substantial variety of possible topological structures when compared with the use of a single GRU . For notational simplicity , we denote the two dimensions of h as x and y . We visualize the flow fields defined by ( 7 ) in 2-dimension as phase portraits which reveal the topological structures of interest . For starters , the phase portrait of two independent bistable GRUs can be visualized as Figure 2A . It clearly shows 4 stable states as expected , with a total of 9 stable fixed points . This could be thought of as a continuoustime continuous-space implementation of a finite state machine with 4 states ( Fig . 2B ) . The 3 types of observed fixed points—stable ( sinks ) , unstable ( sources ) , and saddle points—exhibit locally linear dynamics , however , the global geometry is nonlinear and their topological structures can vary depending on their arrangement . We explored stability structures attainable by two GRUs . Due to the relatively large number of observed topologies , this section ’ s main focus will be on demonstrating all observed local dynamical features obtainable by two GRUs . In addition , existence of two non-local dynamical features will be presented . A complete catalog of all observed topologies can be found in the appendix C , along with the parameters of every phase portrait depicted in this paper . Before proceeding , let us take this time to describe all the local dynamical features observed . In addition to the previously mentioned three types of fixed points , two GRUs can exhibit a variety of bifurcation fixed points , resulting from regions of parameter space that separate all topologies restricted to simple fixed points ( i.e stable , unstable , and saddle points ) . Behaviorally speaking , these fixed points act as hybrids between the previous three , resulting in a much richer set of obtainable dynamics . These bifurcation fixed points fall into two categories , separated by codimension . More specifically , two GRUs have been seen to feature both codimension-1 and codimension-2 bifurcation ( fixed ) points . Beginning with codimension-1 , we have the saddle-node bifurcation fixed point , as expected from its existence in the single GRU case . We can further classify these points into two types . These can be thought of as both the fusion of a stable fixed point and a saddle point , and the fusion of an unstable fixed point and a saddle point . We will refer to these fixed points as saddle-node bifurcation fixed points of the first kind and second kind respectively . One type of codimension-2 bifurcation fixed point that has been observed in the two GRU system acts as the fusion of all three simple fixed points . More specifically , these points arise from the fusion of a stable fixed point , unstable fixed point , and two saddle points . All of these local structures are depicted in figure 3 . While the existence of simple fixed points was already demonstrated ( see Fig . 2A ) . Figure 3A demonstrates the maximum number of fixed points observed in a two GRU system , for a given set of parameters . A closer look at this system reveals its potential interpretation as a continuous analogue of 5-discrete states with input-driven transitions , similar to that depicted in figure 2 , implying additional GRUs are needed for any Markov process modeled in this manner , requiring more than five discrete states . We conjecture that the system depicted in figure 3A is the only eleven fixed point structure obtainable with two GRUs , as all observed structures containing the same number of fixed points are topologically equivalent to one another . The addition of bifurcation fixed points opens the door to dynamically realize more sophisticated models . Take for example the four state system depicted in figure 3B . If the hidden state is set to initialize in the first quadrant of phase space , the trajectory will flow towards the codimension-2 bifurcation fixed point at the origin . Introducing noise through the input will stochastically cause the trajectory to approach the stable fixed point at ( -1 , -1 ) either directly , or by first flowing into one of the two saddle-node bifurcation fixed points of the first kind . Models of this sort can be used in a variety of applications , such as neural decision making ( Wong & Wang ( 2006 ) , Churchland & Cunningham ( 2014 ) ) . We will begin our investigation into the non-local dynamics observed with two GRUs by showing the existence of homoclinic orbits . A trajectory initialized on a homoclinic orbit will approach the same fixed point in both forward and backward time . We observe that two GRUs can exhibit one or two bounded planar regions of homoclinic orbits for a given set of parameters , as shown in figure 4A and 4B respectively . Any trajectory initialized in one of these regions will flow into the codimension-2 bifurcation fixed point at the origin , regardless of which direction time flows in . This featured behavior enables the accurate depiction of various models , including neuron spiking ( Izhikevich , 2007 ) . In regards to the second non-local dynamic feature , it can be shown that two GRUs can exhibit an Andronov-Hopf bifurcation , whereby a stable fixed point bifurcates into an unstable fixed point surrounded by a limit cycle . Behaviorally speaking , a limit cycle is a type of attractor , in the sense that there exists a defined basin of attraction . However , unlike a stable fixed point , where trajectories initialized in the basin of attraction flow towards a single point , a limit cycle pulls trajectories into a stable periodic orbit around the unstable fixed point at its center . To demonstrate this phenomenon , let ( 8 ) define the parameters of ( 7 ) . Uz , Ur , bz , br , bh = 0 , Uh = 3 2 [ cosα − sinα sinα cosα ] ( 8 ) where α ∈ R+ . If α = π3 , the system has a single stable fixed point ( stable spiral ) , as depicted in figure 5A . If we continuously decrease α , the system undergoes an Andronov-Hopf bifurcation approximately about α = π3.8 . As α continuously decreases , the orbital period increases , and as the nullclines can be made arbitrarily close together , the length of this orbital period can be set arbitrarily , up to machine accuracy . Figure 5B shows an example of a relatively short orbital period , and figure 5C depicts the behavior seen for slower orbits . With finite-fixed point topologies and global structures out of the way , the next logical question to ask is , can two GRUs exhibit an infinite number of fixed points ( countable or uncountable ) ? Such behavior is often desirable in models that require stationary attraction to non-point structures , such as line attractors and ring attractors . The short answer to this question is no . Lemma 1 . Any two dimensional GRU can only have finitely many simple fixed points . This follows from Lefschetz theory ( Guillemin & Pollack , 2010 ) . The detailed proof can be found in appendix D , and is intended to give the reader intuition behind the result presented in the claim extended to aribitrary dimensional GRU in theorem 1 . Theorem 1 . Any finite dimensional GRU can only have finitely many simple fixed points and bifurcation fixed points . Proof . By definition of simple fixed points , the Jacobian of the dynamics at those fixed points have nonzero real parts , making them Lefschetz fixed points . Since GRU dynamics is asymptotically bounded on the compact set [ −1 , 1 ] d , where d is the number of GRUs , it follows from Lefshetz theory ( Guillemin & Pollack , 2010 ) that there are finitely many simple fixed points . Furthermore , by construction , a bifurcation fixed point can only exist within a stability structure if and only if there exists a separate topology , such that the simple fixed points making up each bifurcation fixed point exist isolated from one another . Therefore , there can only exist finitely many isolated bifurcation fixed points . This eliminates the possibility of having countably many fixed points . Next , we show that there can not be uncountably many non-isolated fixed points . Theorem 2 . Any finite dimensional GRU can not have a continuous attractor . Proof . We provide a sketch of a proof . Let h∗ ∈ Rd be a fixed point of a d-dimensional GRU , that is , h∗ − tanh ( Uh ( r ( h∗ ) h∗ ) + bh ) = 0 . Now for any unit norm vector k ∈ Sd−1 , and for any δ > 0 , we can show that there exist an > 0 such that , ‖ḣ ( h∗ + δk ) − ḣ ( h∗ ) ‖ = ‖ḣ ( h∗ + δk ) ‖ > . This can be argued by taking three cases into consideration , ( a ) Urk = 0 and Uh ( σ ( br ) k ) 6= 0 , ( b ) Urk = 0 and Uh ( σ ( br ) k ) = 0 , and ( c ) Urk 6= 0 . In each case , it reverts to a 1-dimensional problem where it can be trivially shown to have no continuous attractor around h∗ along direction k. Thus , we conclude that there is no continuous attractor in any direction . Despite this limitation , an approximation of a line attractor can be created using two GRUs . This approximation can be made arbitrarily close to an actual line attractor on a finite region in phase space , thereby satisfying computational needs on an arbitrary interval when scaled . We will refer to this phenomenon as a pseudo-line attractor . Figure 6 depicts an example of such an attractor . We conclude this section with a discussion of slow points in the two GRU system . As a logical extension to the single GRU system , slow points occur where the nullclines are sufficiently close together , but do not intersect , as demonstrated in figure 7 . Given the previously discussed classes of dynamic behavior for two GRUs , slow points can only exist so long as the potential for a saddle-node bifurcation fixed point is possible in the location of the desired slow point , given an appropriate change in parameters , as they result from the collision and annihilation of two fixed points . This observation is consistent with the single GRU case , as slow points can only exist for a single fixed point . This would imply that given the one fixed point case , a maximum of five slow points are possible . However , this would imply that there must exist a six fixed point case by which five of the six fixed points exist at saddle-node bifurcation fixed points , which has not been observed ( see appendix C ) . Despite this shortcoming , four simultaneous slow points are obtainable , as shown in figure 5C .

Summary: The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset. Their analysis shows what kind of hidden state dynamics the GRU can approximate in one and two dimensions. In the experimental part, they show how a GRU with two hidden states trained on a multistep prediction task can learn such dynamics.