Title: Bifurcations and loss jumps in RNN training
Venue: NeurIPS 2023
Accepted: yes
Keywords: dynamical systems

bifurcations

Recurrent Neural Networks

attractors

training algorithm

BPTT

exploding and vanishing gradient problem

nonlinear dynamics

time series
Abstract: Recurrent neural networks (RNNs) are popular machine learning tools for modeling and forecasting sequential data and for inferring dynamical systems (DS) from observed time series. Concepts from DS theory (DST) have variously been used to further our understanding of both, how trained RNNs solve complex tasks, and the training process itself. Bifurcations are particularly important phenomena in DS, including RNNs, that refer to topological (qualitative) changes in a system's dynamical behavior as one or more of its parameters are varied. Knowing the bifurcation structure of an RNN will thus allow to deduce many of its computational and dynamical properties, like its sensitivity to parameter variations or its behavior during training. In particular, bifurcations may account for sudden loss jumps observed in RNN training that could severely impede the training process. Here we first mathematically prove for a particular class of ReLU-based RNNs that certain bifurcations are indeed associated with loss gradients tending toward infinity or zero. We then introduce a novel heuristic algorithm for detecting all fixed points and $k$-cycles in ReLU-based RNNs and their existence and stability regions, hence bifurcation manifolds in parameter space. In contrast to previous numerical algorithms for finding fixed points and common continuation methods, our algorithm provides $\textit{exact}$ results and returns fixed points and cycles up to high orders with surprisingly good scaling behavior. We exemplify the algorithm on the analysis of the training process of RNNs, and find that the recently introduced technique of generalized teacher forcing completely avoids certain types of bifurcations in training. Thus, besides facilitating the DST analysis of trained RNNs, our algorithm provides a powerful instrument for analyzing the training process itself.

Bifurcations and loss jumps in RNN training

Abstract

Recurrent neural networks (RNNs) are popular machine learning tools for modeling and forecasting sequential data and for inferring dynamical systems (DS) from observed time series. Concepts from DS theory (DST) have variously been used to further our understanding of both, how trained RNNs solve complex tasks, and the training process itself. Bifurcations are particularly important phenomena in DS, including RNNs, that refer to topological (qualitative) changes in a system's dynamical behavior as one or more of its parameters are varied. Knowing the bifurcation structure of an RNN will thus allow to deduce many of its computational and dynamical properties, like its sensitivity to parameter variations or its behavior during training. In particular, bifurcations may account for sudden loss jumps observed in RNN training that could severely impede the training process. Here we first mathematically prove for a particular class of ReLU-based RNNs that certain bifurcations are indeed associated with loss gradients tending toward infinity or zero. We then introduce a novel heuristic algorithm for detecting all fixed points and k -cycles in ReLU-based RNNs and their existence and stability regions, hence bifurcation manifolds in parameter space. In contrast to previous numerical algorithms for finding fixed points and common continuation methods, our algorithm provides exact results and returns fixed points and cycles up to high orders with surprisingly good scaling behavior. We exemplify the algorithm on the analysis of the training process of RNNs, and find that the recently introduced technique of generalized teacher forcing completely avoids certain types of bifurcations in training. Thus, besides facilitating the DST analysis of trained RNNs, our algorithm provides a powerful instrument for analyzing the training process itself.

1 Introduction

Recurrent neural networks (RNNs) are common and powerful tools for learning sequential tasks or modeling and forecasting time series data [31, 65, 2, 43, 9, 26]. Typically, RNNs are "black boxes," whose inner workings are hard to dissect. Techniques from dynamical system theory (DST) can significantly aid in this effort, as RNNs are formally discrete-time dynamical systems (DS) [37]. A better understanding of how RNNs solve their tasks is important for detecting failure modes and designing better architectures and training algorithms. In scientific machine learning, on the other hand, RNNs are often employed for reconstructing unknown DS from sets of time series observations [10, 30, 64, 63, 23, 55], e.g. in climate or disease modeling. In this case, the RNN is supposed to provide an approximation to the flow of the observed system that reproduces all its dynamical properties, e.g., cyclic behavior in climate or epidemiological systems. In such scientific or medical

settings, a detailed understanding of the RNN's dynamics and its sensitivity to parameter variations is in fact often crucial.

Beyond understanding the dynamical properties of a once trained RNN, it may also be of interest to know how its dynamical repertoire changes with changes in its parameters. The parameter space of any DS is partitioned into regions (or sets) with topologically different dynamical behaviors by bifurcation curves (or, more generally, manifolds). Such bifurcations, qualitative changes in the system dynamics due to parameter variations, may not only be crucial in scientific applications where we use RNNs, for instance, to predict tipping points in climate systems or medical scenarios like sepsis detection [45]. They also pose severe challenges for the training process itself as qualitative changes in RNN dynamics may go in hand with sudden jumps in the loss landscape [16, 44]. Although methods from DST have significantly advanced the field in recent years, especially with regards to algorithms for reconstructing nonlinear DS from data [55, 46, 67], progress is still hampered by the lack of efficient tools for analysing the dynamics of higher-dimensional RNNs and their bifurcations. In particular, methods are needed for exactly locating geometrical objects like fixed points or cycles in an RNN's state space, but current numerical techniques do not scale well to higher-dimensional scenarios and provide only approximate solutions [29, 21, 61].

The contributions of this work are threefold: After first providing an introduction into bifurcations of piecewise-linear (ReLU-based) RNNs (PLRNNs), which have been extensively used for reconstructing DS from empirical data [10, 30], we mathematically prove that certain bifurcations during the training process will indeed cause loss gradients to diverge to infinity, resulting in abrupt jumps in the loss, while others will cause them to vanish. As RNNs are likely to undergo several bifurcations during training on their way from some initial parameter configuration toward a dynamics that successfully implements any given task, this poses severe challenges for RNN training and may be one of the reasons for exploding and vanishing gradients [16, 44, 8, 25]. We then create a novel, efficient heuristic algorithm for exactly locating all fixed points and k -cycles in PLRNNs, which can be used to delineate bifurcation manifolds in higher-dimensional systems. Our algorithm finds these dynamical objects in many orders of magnitude less time than an exhaustive search would take. Using this algorithm, we demonstrate empirically that steep cliffs in loss landscapes and bifurcation curves indeed tightly overlap, and that bifurcations in the system dynamics are accompanied by sudden loss jumps. Finally, we prove and demonstrate that the recently introduced technique of generalized teacher forcing (GTF) [24] completely eliminates certain types of bifurcation in training, providing an explanation for its efficiency.

2 Related Work

DS analysis of trained RNNs In many areas of science one is interested in identifying the nonlinear DS that explains a set of observed time series [12, 23]. A variety of purely data-driven machine learning approaches have been developed for this purpose [11, 51, 15], but mostly RNNs are used for the goal of reconstructing DS from measured time series [10, 55, 30, 64, 63, 46, 45]. In scientific settings in particular, but also often in engineering applications, we seek a detailed understanding of the system dynamics captured - or the dynamical repertoire produced - by a trained RNN [62, 33, 61, 14, 13]. To analyze an RNN's dynamics, typically its fixed points are determined numerically, for instance by optimizing some cost function [21, 34, 61], by numerically traversing directional fibers [29], or by co-training a switching linear dynamical system [56]. These techniques, however, often scale poorly with dimensionality, provide only approximate solutions, and are not designed for detecting other dynamical objects like k -cycles. This is, however, crucial for understanding the complete dynamical repertoire of a trained RNN. PLRNNs are of particular interest in this context [10, 55, 30, 38], because their piecewise linear nature makes some of their properties analytically accessible [55, 10], which is a tremendous advantage from the perspective of DST. In this work, we develop an efficient heuristic algorithm for locating a PLRNN's fixed points exactly, as well as its k -cycles.

Bifurcations and loss jumps in RNN training The idea that bifurcations in RNN dynamics could impede the training process is not new [16, 44]. Doya [16], to our knowledge, was the first to point out that even in simple single-unit RNNs with sigmoid activation function (saddle-node) bifurcations may occur as an RNN parameter is adapted during training. This may not only cause an abrupt jump in training loss, but could lead to situations where it is impossible, even in principle, to reach the training objective (the desired target output), as across the bifurcation point there is a discrete change

in network behavior [16, 68, 4]. Pascanu et al. [44] discussed similar associations between steep cliffs in RNN loss functions and bifurcations. Although profound for training success, this topic received surprisingly little attention over the years. Haschke and Steil [22] extended previous work by a more formal treatment of bifurcation boundaries in RNNs, and Marichal et al. [35] examined fold bifurcations in RNNs. The effects of bifurcations and their relation to exploding gradients in gated recurrent units (GRUs) was investigated in Kanai et al. [27]. Ribeiro et al. [50] looked at the connection between dynamics and smoothness of the cost function, but failed to find a link between bifurcations and jumps in performance. In contrast, Rehmer and Kroll [49] observed large gradients at bifurcation boundaries and concluded that bifurcations can indeed cause problems in gradient-based optimization. To the best of our knowledge, we are, however, the first to formally prove a direct link between bifurcations and the behavior of loss gradients, and to derive a systematic and efficient algorithmic procedure for identifying bifurcation manifolds for a class of ReLU-based RNNs.

3 Theoretical analysis

In this paper we will focus on PLRNNs as one representative of the wider class of ReLU-based RNNs, but similar derivations and algorithmic procedures could be devised for any type of ReLU-based RNN (in fact, many other types of ReLU-based RNNs could be brought into the same functional form as PLRNNs; e.g. [10]). We will first, in sect. 3.1, provide some theoretical background on bifurcations in PLRNNs, and illustrate how existence and stability regions of fixed points and cycles could be analytically computed for low dimensional ( 2 d ) PLRNNs. In sect. 3.2 we will then state two theorems regarding the association of bifurcations and loss gradients in training. It turns out that for certain types of bifurcations exploding or vanishing gradients are inevitable in gradient-based training procedures like Back-Propagation Through Time (BPTT).

3.1 Bifurcation curves in PLRNN parameter space

The PLRNN, originally introduced as a kind of discrete time neural population model [18], has the general form

where z t ∈ R M is the latent state vector and θ are system parameters consisting of diagonal matrix A ∈ R M × M (auto-regression weights), off-diagonal matrix W ∈ R M × M (coupling weights), ϕ ( z t -1 ) = max( z t -1 , 0) is the element-wise rectified linear unit ( ReLU ) function, h ∈ R M a constant bias term, and s t ∈ R K represents external inputs weighted by C ∈ R M × K . The original formulation of the PLRNN is stochastic [18, 30], with a Gaussian noise term added to eq. (1), but here we will consider the deterministic variant.

Formally, like other ReLU-based RNNs, PLRNNs constitute piecewise linear (PWL) maps, a subclass of piecewise smooth (PWS) discrete-time DS. Define D Ω( t ) := diag( d Ω( t ) ) as a diagonal matrix with indicator vector d Ω( t ) := ( d 1 , d 2 , · · · , d M ) such that d m ( z m,t ) =: d m = 1 whenever z m,t > 0 , and zero otherwise. Then (1) can be rewritten as

where we have ignored external inputs s t for simplicity. There are in general 2 M different configurations for matrix D Ω( t -1) and hence for matrix W Ω( t -1) , dividing the phase space into 2 M sub-regions separated by switching manifolds (see Appx. A.1.1 for more details).

Recall that fixed points of a map z t = F θ ( z t -1 ) are defined as the set of points for which we have z ∗ = F θ ( z ∗ ) , and that the type (node, saddle, spiral) and stability of a fixed point can be read off from the eigenvalues of the Jacobian J t := ∂F θ ( z t -1 ) ∂ z t -1 = ∂ z t ∂ z t -1 evaluated at z ∗ [3, 47]. Similarly, a k -cycle of map F θ is a periodic orbit { z ∗ 1 , z ∗ 2 , . . . , z ∗ k } such that each of the periodic points z ∗ i , i = 1 . . . k , is distinct, and is a solution to the equation z ∗ i = F k θ ( z ∗ i ) , i.e. the k times iterated map F θ . Type and stability of a k -cycle are then determined via the Jacobian ∏ k r =1 J t + k -r = ∏ k r =1 ∂ z t + k -r ∂ z t + k -r -1 = ∂ z t + k -1 ∂ z t -1 . Solving these equations and computing the corresponding Jacobians thus allows to determine all existence and stability regions of fixed points and cycles, where the latter are a subset of the former, bounded by bifurcation curves (see Appx. A.1 for more formal details).

To provide a specific example, assume M = 2 and fix - for the purpose of this exposition - parameters w 12 = w 22 = 0 , such that we have

i.e. only one border which divides the phase space into two distinct sub-regions (see Appx. A.1.1). For this setup, Fig. 1A provides examples of analytically determined stability regions for two low order cycles in the ( a 11 , a 11 + w 11 ) -parameter plane (see Appx. A.1). Note that there are regions in parameter space where two or more stability regions overlap: In these regions we have multi-stability , the co-existence of different attractor states in the PLRNN's state space.

As noted above, bifurcation curves delimit the different stability regions in parameter space and are hence associated with abrupt changes in the topological structure of a system's state space. In general, there are many different types of bifurcations through which dynamical objects can come into existence, disappear, or change stability (see, e.g., [3, 40, 47]), the most common ones being saddle node, transcritical, pitchfork, homoclinic, and Hopf bifurcations. In comparison with smooth systems, bifurcation theory of PWS (or PWL) maps includes additional dynamical phenomena related to the existence of borders in the phase space [6]. Border-collision bifurcations (BCBs) arise when for a PWS map a specific point of an invariant set collides with a border and this collision leads to a qualitative change of dynamics [5, 6, 42]. More specifically, a BCB occurs, if for a PWS map z t = F θ ( z t -1 ) a fixed point or k -cycle either crosses the switching manifold ∑ i := { z ∈ R n : e T i z = 0 } transversely at θ = θ ∗ and its qualitative behavior changes in the event, or if it collides on the border with another fixed point or k -cycle and both objects disappear [7, 41]. Degenerate transcritical bifurcations (DTBs) occur when a fixed point or a periodic point of a cycle tends to infinity and one of its eigenvalues tends to 1 by variation of a parameter. Specifically, let Γ k , k ≥ 1 , be a fixed point or a k -cycle with the periodic points { z ∗ 1 , z ∗ 2 , . . . , z ∗ k } , and assume λ i denotes an eigenvalue of the Jacobian matrix at the periodic point z ∗ i , i ∈ { 1 , 2 , · · · , k } . Then Γ k undergoes a DTB at θ = θ ∗ , if λ i ( θ ∗ ) → +1 and ∥ z ∗ i ∥ → ∞ . Γ k undergoes a degenerate flip bifurcation (DFB) , iff λ i ( θ ∗ ) = -1 and the map F k has locally, in some neighborhood of z ∗ i , infinitely many 2 -cycles at θ = θ ∗ . A center bifurcation (CB) occurs, if Γ k has a pair of complex conjugate eigenvalues λ 1 , 2 and locally becomes a center at the bifurcation value θ = θ ∗ , i.e. if its eigenvalues are complex and lie on the unit circle ( | λ 1 , 2 ( θ ∗ ) | = 1 ). Another important class of bifurcations are multiple attractor bifurcations (MABs) , discussed in more detail in Appx. A.1.5 (see Fig. S3). In addition to existence and stability regions of fixed points and cycles, in Appx. A.1.2-A.1.4 we also illustrate how to analytically determine the types of bifurcation curves bounding the existence and stability regions of k -cycles (DTB, DFB, CB and BCB curves; see also Fig. S6 for a 1 d example).

3.2 Bifurcations and loss jumps in training

Here we will prove that major types of bifurcations discussed above are always associated with exploding or vanishing gradients in PLRNNs during training, and hence often with abrupt jumps in the loss. For this we may assume any generic loss function L ( θ ) , like a negative log-likelihood or a mean-squared-error (MSE) loss, and a gradient-based training technique like BPTT [52] or Real-Time-Recurrent-Learning (RTRL) that involves a recursion (via chain rule) through loss terms across time. The first theorem establishes that a DTB inevitably causes exploding gradients.

Theorem 1. Consider a PLRNN of the form (2) with parameters θ = { A , W , h } . Assume that it has a stable fixed point or k -cycle Γ k ( k ≥ 1) with B Γ k as its basin of attraction. If Γ k undergoes a degenerate transcritical bifurcation (DTB) for some parameter value θ = θ 0 ∈ θ , then the norm of the PLRNN loss gradient, ∥ ∥ ∂ L t ∂θ ∥ ∥ , tends to infinity at θ = θ 0 for every z 1 ∈ B Γ k , i.e. lim θ → θ 0 ∥ ∥ ∂ L t ∂θ ∥ ∥ = ∞ .

However, bifurcations may also cause gradients to suddenly vanish, as it is the case for a BCB as established by our second theorem:

Theorem 2. Consider a PLRNN of the form (2) with parameters θ = { A , W , h } . Assume that it has a stable fixed point or k -cycle Γ k ( k ≥ 1) with B Γ k as its basin of attraction. If Γ k undergoes a border collision bifurcation (BCB) for some parameter value θ = θ 0 ∈ θ , then the gradient of the loss function, ∂ L t ∂θ , vanishes at θ = θ 0 for every z 1 ∈ B Γ k , i.e. lim θ → θ 0 ∥ ∥ ∂ L t ∂θ ∥ ∥ = 0 .

A

0

B

0

DFB+BCB

-1

SRC

-1

DFB

3

1

-2

3

1

-2

DTB

BCB

DFB

十

ILD

-3

D

ILp

-3

SRC2

-4

-4

DTB

-1.5

-1

-0.5

0

0.5

1

1.5

-1.5

-1

-0.5

0

a11

0.5

1

1.5

c

DTB

iBCBi

iDFB

DFB

D

4

2.0

2

1.5

0

21.0

2

0.5

-2

0.0

-4

-0.5

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

-4

-2

0

2

a11

21

Corollary 1. Assume that the PLRNN (2) has a stable fixed point Γ 1 with B Γ 1 as its basin of attraction. If Γ 1 undergoes a degenerate flip bifurcation (DFB) for some parameter value θ = θ 0 ∈ θ , then this will always coincide with a BCB of a 2 -cycle, and as a result lim θ → θ 0 ∥ ∥ ∂ L t ∂θ ∥ ∥ = 0 for every z 1 ∈ B Γ 1 .

Hence, certain bifurcations will inevitably cause gradients to suddenly explode or vanish, and often induce abrupt jumps in the loss (see Appx. A.3.2 for when this will happen for a BCB). We emphasize that these results are general and hold for systems of any dimension , as well as in the presence of inputs . Since inputs do not affect the Jacobians in eqn. (70), (71) and (76), they do not change the theorems (even if they would affect the Jacobians, Theorem 2 would be unaltered, and Theorem 1 could be amended in a straightforward way). Furthermore, since we are addressing bifurcations that occur during model training, from this angle inputs may simply be treated as either additional parameters (if piecewise constant) or states of the system (without changing any of the mathematical derivations). In fact, mathematically, any non-autonomous dynamical system (RNN with inputs) can always and strictly be reformulated as an autonomous system (RNN without inputs), see [3, 47, 66].

4 Heuristic algorithm for finding PLRNN bifurcation manifolds

4.1 Searcher for fixed points and cycles (SCYFI): motivation and validation

In sect. 3.1 and Appx. A.1.2-A.1.4 we derived existence and stability regions for fixed points and low order ( k ≤ 3 ) cycles in 2 d PLRNNs with specific parameter constraints analytically. For higher-order cycles and higher-dimensional PLRNNs (or any other ReLU-type RNN) this is no longer feasible due to the combinatorial explosion in the number of subregions that need to be considered as M and k increase. Here we therefore introduce an efficient search algorithm for finding all k -cycles of a given PLRNN, which we call S earcher for Cy cles and Fi xed points : SCYFI (Algorithm 1). Once all k -cycles ( k ≥ 1 ) have been detected on some parameter grid, the stability-/existence regions of these objects and thereby the bifurcation manifolds can be determined. k -cycles were defined in sect. 3.1,

Algorithm 1 SCYFI

The algorithm is iteratively run with k = 1 · · · K max , with K max the max. order of cycles tested Input: PLRNN parameters A , W , h ;

L = {L n } k -1 n =1 : collection of all sets L n = {{ z ( m ) l } n l =1 } M n m =1 of all lower order n -cycles discovered so far, where M n is the number of found cycles { z ( m ) l } n l =1 of order n , with corresponding ReLU derivative matrices { D ( m ) } n .

and for the PLRNN, eq. (2), are given by the set of k -periodic points { z ∗ 1 , . . . , z ∗ l , . . . , z ∗ k } , where

if ( 1 -∏ k -1 r =0 W Ω( k -r ) ) is invertible (if not, we are dealing with a bifurcation or a continuous set of fixed points). The other periodic points are z l = F l ( z ∗ k ) , l = 1 , · · · , k -1 , with corresponding matrices W Ω( l ) = A + WD l . Now, if the diagonal entries in D l are consistent with the signs of the corresponding states z ∗ ml , i.e. if d ( l ) mm = 1 if z ∗ ml > 0 and d ( l ) mm = 0 otherwise for all l , { z ∗ 1 , . . . , z ∗ k } is a true cycle of eq. (2), otherwise we call it virtual . To find a k -cycle, since an M -dimensional PLRNN harbors 2 M different linear sub-regions, there are approximately 2 Mk different combinations of configurations of the matrices D l , l = 1 . . . k , to consider (strictly, mathematical constraints rule out some of these possibilities, e.g. not all periodic points can lie within the same sub-region/orthant).

Clearly, for higher-dimensional PLRNNs and higher cycle orders exhaustively searching this space becomes unfeasible. Instead, we found that the following heuristic works surprisingly well: First, for some order k and a random initialization of the matrices D l , l = 1 . . . k , generate a cycle candidate by solving eq. (4). If each of the points z ∗ l , l = 1 . . . k , is consistent with the diagonal entries in the corresponding matrix D l , l = 1 . . . k , and none of them is already in the current library of cyclic points, then a true k -cycle has been identified, otherwise the cycle is virtual (or a super-set of lower-order cycles). We discovered that the search becomes extremely efficient, without the need to exhaustively consider all configurations, if a new search loop is re-initialized at the last visited virtual cyclic point (i.e., all inconsistent entries d ( l ) mm in the matrices D l , l = 1 . . . k , are flipped, d ( l ) mm → 1 -d ( l ) mm , to bring them into agreement with the signs of the solution points, z ∗ l = [ z ∗ ml ] , of eq.

(4), thus yielding the next initial configuration). It is straightforward to see that this procedure almost surely converges if N out is chosen large enough, see Appx. A.2.4. The whole procedure is formalized in Algorithm 1, and the code is available at https://github.com/DurstewitzLab/SCYFI .

To validate the algorithm, we can compare analytical solutions as derived in sect. 3.1 to the output of the algorithm. To delineate all existence and stability regions, the algorithm searches for all k -cycles up to some maximum order K along a fine grid across the ( a 11 , a 11 + w 11 )-parameter plane. A bifurcation happens whenever between two grid points a cycle appears, disappears, or changes stability (as determined from the eigenvalue spectrum of the respective k th -order Jacobian). The results of this procedure are shown in Fig. 1B, illustrating that the analytical solutions for existence and stability regions precisely overlap with those identified by Algorithm 1 (see also Fig. S7).

4.2 Numerical and theoretical results on SCYFI's scaling behavior

Because of the combinatorial nature of the problem, it is generally not feasible to obtain ground truth settings in higher dimensions for SCYFI to compare to. To nevertheless assess its scaling behavior, we therefore studied two specific scenarios. For an exhaustive search, the expected and median numbers of linear subregions n until an object of interest (fixed point or cycle) is found, i.e. the number of { D 1: k } constellations that need to be inspected until the first hit, are given by

with m being the number of existing k -cycles and N the total number of combinations, as shown in Ahlgren [1] (assuming no prior knowledge about the mathematical limitations when drawing regions). The median n as well as the actual median number of to-be-searched combinations required by SCYFI to find at least one k -cycle is given for low-dimensional systems in Fig. 2A as a function of cycle order k , and can be seen to be surprisingly linear as confirmed by linear regression fits to the data (see Fig. 2 legend for details). To assess scaling as a function of dimensionality M , we explicitly constructed systems with one known fixed point (see Appx. A.3.1 for details) and determined the number n of subregions required by SCYFI to detect this embedded fixed point (Fig. 2B). In general, the scaling depended on the system's eigenspectrum, but for reasonable scenarios was polynomial or even sublinear (Fig. 2B, see also Fig. S8). In either case, the number of required SCYFI iterations scaled much more favorably than would be expected from an exhaustive search.

A

B107

20

log-log

linearsubregions

=

linear subregions

quadratic

15

105

expon.

E[n]

(max) ≈ 0.71

10

103

(max) ≥ 1.25

(max) = 2.33

5

101

0

0

10

20

30

40

50

Q

20

40

60

80

100

120

cycle orderk

M

How could this surprisingly good scaling behavior be explained? As shown numerically in Fig. S9, when we initiate SCYFI in different randomly selected linear subregions, it converges to the subregions including the dynamical objects of interest exponentially fast, offsetting the combinatorial explosion. A more specific and stronger theoretical result about SCYFI's convergence speed can be obtained under certain conditions on the parameters (which agrees nicely with the numerical results in Fig. 2). It rests on the observation that SCYFI is designed to move only among subregions containing virtual or actual fixed points or cycles , based on the fact that it is always reinitialized with the next virtual fixed (cyclic) point in case the consistency check fails. The result can be stated as follows:

Proof. See Appx. A.2.5

5 Loss landscapes and bifurcation curves

5.1 Bifurcations and loss jumps in training

A

1.0

4.5

4.0

B

C

80

D

80

0.5

3.5

3.0

100

60

60

2.5

5

50

1

0.0

2.0

☆C

5

40

40

1.5

-0.5

1.0

1

20

20

C

0.5

-1.0

-1.0-0.50.0

a00

0.51.0

0

a11

0

-1 -1

△aoo

0

1

0

0

500

Epoch

1000

1500

0

Q

500

Epoch

1000

1500

Fig. 3 provides a 2 d toy example illustrating the tight association between the loss landscape and bifurcation curves, as determined through SCYFI, for a PLRNN trained by BPTT on reproducing a specific 16 -cycle. Fig. 3A depicts a contour plot of the gradient norms with overlaid bifurcation curves in yellow, while Fig. 3B shows the MSE loss landscape as a relief for better appreciation of the sharp changes in loss height associated with the bifurcation curves. Shown in green are two trajectories from two different parameter initializations traced out during PLRNN training in parameter space, where training was confined to only those two parameters given in the graphs (i.e., all other PLRNN parameters were kept fixed during training for the purpose of this illustration). As confirmed in Fig. 3C & D, as soon as the training trajectory crosses the bifurcation curves in parameter space, a huge jump in the loss associated with a sudden increase in the gradient norm occurs. This illustrates empirically and graphically the theoretical results derived in sect. 3.

Next we illustrate the application of SCYFI on a real-world example, learning the behavior of a rodent spiking cortical neuron observed through time series measurements of its membrane potential (note that spiking is a highly nonlinear behavior involving fast within-spike and much slower between-spike time scales). For this, we constructed a 6 -dimensional delay embedding of the membrane voltage [53, 28], and trained a PLRNN with one hidden layer (cf. eq. 6) using BPTT with sparse teacher forcing (STF) [37] to approximate the dynamics of the spiking neuron (see Appx. A.3.2 for a similar analysis on a biophysical neuron model). With M = 6 latent states and H = 20 hidden dimensions, the trained PLRNN comprises 2 20 different linear subregions and | θ | = 272 parameters, much higher-dimensional than the toy example considered above. Fig. 4A gives the MAE loss as a function of training epoch (i.e., single SGD updates), while Figs. 4B & C illustrate the well-trained behavior in time (Fig. 4B) and in a 2 -dimensional projection of the model's state space obtained by PCA (Fig. 4C). The loss curve exhibits several steep jumps. Zooming into one of these regions (Fig. 4A; indicated by the red box) and examining the transitions in parameter space using SCYFI, we find they are indeed produced by bifurcations, with an example given in Fig. 4D. Note that we are now dealing with high-dimensional state and parameter spaces, such that visualization of results becomes tricky. For the bifurcation diagram in Fig. 4D we therefore projected all extracted k -cycles ( k ≥ 1 ) onto a line given by the PCA-derived maximum eigenvalue component, and plotted this as a function of training epoch. 1 Since SCYFI extracts all k -cycles and their eigenvalue spectrum, we can also determine the type of bifurcation that caused the jump. While before the loss jump the PLRNN already produced time series quite similar to those of the physiologically recorded cell (Fig. 4E), a

A

10

8

8

B

[mV]

-10

0

generated

data

C

2.0

1.5

cycleorder

。

1

9

1

品：

SSOl

6

potential[

-20

PC1

1.0

.

39

39

8

4

0.5

0

4000

6000

-40

2

0.0

B/C

-50

0.5

0

0

1.0×104

3.0×104

5.0×104

0

50

time

100

150

200

-5

-4

-3

PC2

-2

-1

0

D

E

F

0

0

2

0

9

6

[mV]

-10

[mV]

-5.00×103

-2

20

-1.00×104

一

potential[

potential

P

-6

-30

-1.50×104

37.5

40

-2.00×104

-40.0

42.5

-50

-2.50×104

generated

data

45.0

4750

4800

4850

4900

0

50

100

150

200

0

50

100

150

200

time

time

5.2 Implications for designing training algorithms

What are potential take-homes of the results in sects. 3.2 & 5.1 for designing RNN training algorithms? One possibility is to design smart initialization or training procedures that aim to place or push an RNN into the right topological regime by taking big leaps in parameter space whenever the current regime is not fit for the data, rather than dwelling within a wrong regime for too long. These ideas are discussed in a bit more depth in Appx.A.3.3, with a proof of concept in Fig. S12.

1.00

α=0

α=0.1

0.75

DSS

0.50

0.25

0.00

0

50

100

150

200

More importantly, however, we discovered that the recently proposed technique of 'generalized teacher forcing (GTF)' [24] tends to circumvent bifurcations in RNN training altogether, leading to much faster convergence as illustrated in Fig. 5. The way this works is that GTF, by trading off forward-iterated RNN latent states with data-inferred states according to a specific annealing schedule during training (see Appx. A.2.6), tends to pull the RNN directly into the right dynamical regime. In fact, for DTBs we can strictly prove these will never occur in PLRNN training with the right adjustment of the GTF parameter:

Theorem 4. Consider a PLRNN of the form (2) with parameters θ = { A , W , h } . Assume that it has a stable fixed point or k -cycle Γ k ( k ≥ 1 ) that undergoes a degenerate transcritical bifurcation (DTB) for some parameter value θ = θ 0 ∈ θ .

(i) If ∥ A ∥ + ∥ W ∥ ≤ 1 , then for any GTF parameter 0 < α < 1 , GTF controls the system, avoiding a DTB and, hence, gradient divergence at θ 0 .

(ii) If ∥ A ∥ + ∥ W ∥ = r > 1 , then for any 1 -1 r < α < 1 , GTF prevents a DTB and, hence, gradient divergence at θ 0 .

As this example illustrates, we may be able to amend training procedures such as to avoid specific types of bifurcations.

6 Discussion

DS theory [3, 47, 58] is increasingly appreciated in the ML/AI community as a powerful mathematical framework for understanding both the training process of ML models [49, 54, 44, 16] as well as the behavior of trained models [62, 33, 61]. While the latter is generally useful for understanding how a trained RNN performs a given ML task, with prospects of improving found solutions, it is in fact imperative in areas like science or medicine where excavating the dynamical behavior and repertoire of trained models yields direct insight into the underlying physical, biological, or medical processes the model is supposed to capture. However, application of DS theory is often not straightforward, especially when dealing with higher-dimensional systems, and commonly requires numerical routines that may only find some of the dynamical objects of interest, and also only approximate solutions. One central contribution of the present work therefore was the design of a novel algorithm, SCYFI, that can exactly locate fixed points and cycles of a wide class of ReLU-based RNNs. This provides an efficient instrument for the DS analysis of trained models, supporting their interpretability and explainability.

A surprising observation was that SCYFI often finds cycles in only linear time, despite the combinatorial nature of the problem, a feature shared with the famous Simplex algorithm for solving linear programming tasks [36, 32, 57]. While we discovered numerically that SCYFI for empirically relevant scenarios converges surprisingly fast, deriving strict theoretical guarantees is hard, and so far we could establish stronger theoretical results on its convergence properties only under specific assumptions on the RNN parameters. Further theoretical work is therefore necessary to precisely understand why the algorithm works so effectively.

In this work we applied SCYFI to illuminate the training process itself. Since RNNs are themselves DS, they are subject to different forms of bifurcations during training as their parameters are varied under the action of a training algorithm (similar considerations may apply to very deep NNs). It has been recognized for some time that bifurcations in RNN training may give rise to sudden jumps in the loss [44, 16], but the phenomenon has rarely been treated more systematically and mathematically. Another major contribution of this work thus was to formally prove a strict connection between three types of bifurcations and abrupt changes in the gradient norms, and to use SCYFI to further reveal such events during PLRNN training on various example systems. There are numerous other types of bifurcations (e.g., center bifurcations, Hopf bifurcations etc.) that are likely to impact gradients during training, only for a subset of which we could provide formal proofs here. As we have demonstrated, understanding the topological and bifurcation landscape of RNNs could help improve training algorithms and provide insights into their working. Hence, a more general understanding of how various types of bifurcation affect the training process in a diverse range of RNN architectures is a promising future avenue not only for our theoretical understandings of RNNs, but also for guiding future algorithm design.

Acknowledgements This work was supported by the German Research Foundation (DFG) through individual grants Du 354/10-1 & Du 354/15-1 to DD, within research cluster FOR-5159 ('Resolving prefrontal flexibility'; Du 354/14-1), and through the Excellence Strategy EXC 2181/1 - 390900948 (STRUCTURES). We also thank Mahasheta Patra for lending us code for graphing analytically derived bifurcation diagrams and state spaces.

References

J. Ahlgren. The probability distribution for draws until first success without replacement. arXiv preprint arXiv:1404.1161 , 2014.

A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Kathleen T. Alligood, Tim D. Sauer, and James A. Yorke. Chaos: An Introduction to Dynamical Systems . Springer, New York, NY, 1996.

S. Arora, Z. Li, and A. Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning , pages 948-1024. PMLR, 2022.

V. Avrutin, M. Schanz, and S. Banerjee. Occurrence of multiple attractor bifurcations in the two-dimensional piecewise linear normal form map. Nonlinear Dynamics , 67:293-307, 2012.

V. Avrutin, L. Gardini, I. Sushko, and F. Tramontana. Continuous and Discontinuous Piecewise Smooth One-Dimensional Maps: Invariant Sets and Bifurcation structures , volume 1. World Scientific, Singapore, 2019.

S. Banerjee, MS. Karthik, G. Yuan, and J. A. Yorke. Bifurcations in one-dimensional piecewise smooth maps-theory and applications in switching circuits. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications , 47(3):389-394, 2000.

Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5(2):157-166, 1994. doi: 10.1109/72.279181.

S.h Bouktif, A. Fiaz, A. Ouni, and M. A. Serhani. Multi-sequence lstm-rnn deep learning and metaheuristics for electric load forecasting. Energies , 13(2), 2020. ISSN 1996-1073. doi: 10.3390/en13020391.

M. Brenner, F. Hess, J. M. Mikhaeil, L. F. Bereska, Z. Monfared, P. Kuo, and D. Durstewitz. Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems. In Proceedings of the 39th International Conference on Machine Learning , pages 2292-2320. PMLR, June 2022. ISSN: 2640-3498.

S. L. Brunton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences , 113(15):3932-3937, 2016.

K. Champion, B. Lusch, J. N. Kutz, and S. L. Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences , 116(45):22445-22451, 2019. doi: 10.1073/pnas.1906995116.

V. Chatziafratis, S. G. Nagarajan, and I. Panageas. Better depth-width trade-offs for neural networks through the lens of dynamical systems. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1469-1478. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/chatziafratis20a.html .

V. Chatziafratis, I. Panageas, and S. Sanford, C.and Stavroulakis. On scrambling phenomena for randomly initialized recurrent networks. Advances in Neural Information Processing Systems , 35:18501-18513, 2022.

B. M. de Silva, K. Champion, M. Quade, J. Loiseau, J. N. Kutz, and . L. Brunton. Pysindy: A python package for the sparse identification of nonlinear dynamics from data, 2020. URL https://arxiv.org/abs/2004.08424 .

K. Doya. Bifurcations in the learning of recurrent neural networks. Proceedings 1992 IEEE International Symposium on Circuits and Systems , 6:2777-2780, 1992.

D. Durstewitz. Implications of synaptic biophysics for recurrent network dynamics and active memory. Neural Networks , 22(8):1189-1200, 2009. ISSN 0893-6080. doi: https://doi.org/ 10.1016/j.neunet.2009.07.016. URL https://www.sciencedirect.com/science/ article/pii/S0893608009001622 . Cortical Microcircuits.

D. Durstewitz. A state space approach for piecewiselinear recurrent neural networks for reconstructing nonlinear dynamics from neural measurements. PLoS Computational Biology , 13(6), 2017.

A. Ganguli and S. Banerjee. Dangerous bifurcation at border collision: When does it occur? Physical Review E , 71, 2005.

L. Gardini, D. Fournier-Prunaret, and P. Chargé. Border collision bifurcations in a twodimensional piecewise smooth map from a simple switching circuit. Chaos: An Interdisciplinary Journal of Nonlinear Science , 21, 2011.

M. Golub and D. Sussillo. FixedPointFinder: A Tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks. Journal of Open Source Software , 3 (31):1003, November 2018. ISSN 2475-9066.

R. Haschke and J. J. Steil. Input space bifurcation manifolds of recurrent neural networks. Neurocomputing , 64:25-38, March 2005. ISSN 0925-2312. doi: 10.1016/j.neucom.2004.11. 030.

D.l Hernandez, A. K. Moretti, Z.g Wei, S. Saxena, J. Cunningham, and L. Paninski. Nonlinear Evolution via Spatially-Dependent Linear Dynamics for Electrophysiology and Calcium Data, June 2020. arXiv:1811.02459 [cs, q-bio, stat].

F. Hess, Z. Monfared, M. Brenner, and D. Durstewitz. Generalized teacher forcing for learning chaotic dynamics. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 13017-13049. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/hess23a.html .

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation , 9(8): 1735-1780, 1997.

T. Hossen, A. S. Nair, R. A. Chinnathambi, and P. Ranganathan. Residential load forecasting using deep neural networks (dnn). In 2018 North American Power Symposium (NAPS) , pages 1-5, 2018. doi: 10.1109/NAPS.2018.8600549.

S. Kanai, Y. Fujiwara, and S. Iwamura. Preventing Gradient Explosions in Gated Recurrent Units. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

H. Kantz and T. Schreiber. Nonlinear time series analysis , volume 7. Cambridge university press, 2004.

G. E. Katz and J. A. Reggia. Using Directional Fibers to Locate Fixed Points of Recurrent Neural Networks. IEEE Transactions on Neural Networks and Learning Systems , 29(8):36363646, August 2018. ISSN 2162-2388. doi: 10.1109/TNNLS.2017.2733544. Conference Name: IEEE Transactions on Neural Networks and Learning Systems.

G. Koppe, H . Toutounji, P. Kirsch, S. Lis, and D. Durstewitz. Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fmri. PLOS Computational Biology , 15(8):1-35, 2019.

T. Lang and M. Rettenmeier. Understanding consumer behavior with recurrent neural networks. 2017.

D. G. Luenberger and Y. Ye. Linear and Nonlinear Programming . Springer International Publishing, 2016. doi: doi:10.1007/978-3-319-18842-3.

N. Maheswaranathan and D. Sussillo. How recurrent networks implement contextual processing in sentiment analysis. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6608-6619. PMLR, 13-18 Jul 2020.

N. Maheswaranathan, AH. Williams, MD. Golub, S. Ganguli, and D. Sussillo. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. Advances in neural information processing systemsv , 32:15696-15705, 2019.

R Marichal, JD Piñeiro, and E González. Study of fold bifurcation in a discrete recurrent neural network. In Proceedings of the World Congress on Engineering and Computer Science , volume 2, 2009.

N. Megiddo. On the complexity of linear programming , page 225-268. Econometric Society Monographs. Cambridge University Press, 1987. doi: 10.1017/CCOL0521340446.006.

J. M. Mikhaeil, Z. Monfared, and D. Durstewitz. On the difficulty of learning chaotic dynamics with RNNs. In Advances in Neural Information Processing Systems , 2022.

Z. Monfared and D. Durstewitz. Existence of n-cycles and border-collision bifurcations in piecewise-linear continuous maps with applications to recurrent neural networks. Nonlinear Dynamics , 101(2):1037-1052, 2020.

Z. Monfared and D. Durstewitz. Transformation of ReLU-based recurrent neural networks from discrete-time to continuous-time. In International Conference on Machine Learning , pages 6999-7009. PMLR, 2020. ISSN: 2640-3498.

Z. Monfared, Z. Dadi, N. Miladi Lari, and Z. Afsharnezhad. Existence and nonexistence of periodic solution and hopf bifurcation of a tourism-based social-ecological system. Optik , 127 (22):10908-10918, 2016.

H. E. Nusse and J. A. Yorke. Border-collision bifurcations including 'period two to period three' for piecewise smooth systems. Physica D: Nonlinear Phenomena , 57(1-2):39-57, 1992.

H.E. Nusse and J.A. Yorke. Border-collision bifurcations for piecewise smooth one dimensional maps. Int. J. Bifurc. Chaos , 5(1):189-207, 1995.

S. H. Park, B. Kim, C. M. Kang, C. C. Chung, and J. W. Choi. Sequence-to-sequence prediction of vehicle trajectory via lstm encoder-decoder architecture. In 2018 IEEE Intelligent Vehicles Symposium (IV) , pages 1672-1678, 2018. doi: 10.1109/IVS.2018.8500658.

R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning , volume 28 of Proceedings of Machine Learning Research , pages 1310-1318, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.

D. Patel and E. Ott. Using machine learning to anticipate tipping points and extrapolate to post-tipping dynamics of non-stationary dynamical systems, 2022. URL https://arxiv. org/abs/2207.00521 .

J. Pathak, B. Hunt, M. Girvan, Z.n Lu, and E. Ott. Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach. Phys. Rev. Lett. , 120:024102, Jan 2018. doi: 10.1103/PhysRevLett.120.024102.

L. Perko. Differential Equations and Dynamical Systems . Springer New York, NY, 2001.

P. Razvan. On Recurrent and Deep Neural Networks . PhD thesis, 2014. Université de Montréal.

A. Rehmer and A. Kroll. The effect of the forget gate on bifurcation boundaries and dynamics in Recurrent Neural Networks and its implications for gradient-based optimization. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 01-08, July 2022. doi: 10.1109/IJCNN55064.2022.9892458. ISSN: 2161-4407.

A. H. Ribeiro, K. Tiels, Lu. A. Aguirre, and T. Schön. Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages 2370-2380. PMLR, June 2020. URL https://proceedings.mlr.press/v108/ribeiro20a.html . ISSN: 2640-3498.

S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz. Data-driven discovery of partial differential equations. Science Advances , 3(4):e1602614, 2017. doi: 10.1126/sciadv.1602614.

D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. nature , 323(6088):533-536, 1986.

T. Sauer, J. A. Yorke, and M. Casdagli. Embedology. Journal of statistical Physics , 65:579-616, 1991.

A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2013. URL https://arxiv.org/abs/1312. 6120 .

D. Schmidt, G. Koppe, Z. Monfared, M. Beutelspacher, and D. Durstewitz. Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies. In proceedings of the Ninth International Conference on Learning Representations, ICLR , 2021.

J. Smith, S. Linderman, and D. Sussillo. Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systems. In Advances in Neural Information Processing Systems , volume 34, pages 16700-16713. Curran Associates, Inc., 2021.

D. A. Spielman and S. Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time, 2001. URL https://arxiv.org/abs/cs/0111050 .

S. H. Strogatz. Nonlinear Dynamics and Chaos: Applications to Physics, Biology, Chemistry, and Engineering: With Applications to Physics, Biology, Chemistry and Engineering . CRC Press, 2015.

I. Sushko and L. Gardini. Center bifurcation for a two-dimensional piecewise linear map, in business cycles dynamics. Models and Tools, eds. Puu, T. & Sushko, I. (Springer-Verlag, NY) , pages 49--78, 2006.

I. Sushko and L. Gardini. Center bifurcation for two-dimensional border-collision normal form. International Journal of Bifurcation and Chaos , 18:1029-1050, 2008.

D. Sussillo and O. Barak. Opening the Black Box: Low-Dimensional Dynamics in HighDimensional Recurrent Neural Networks. Neural Computation , 25(3):626-649, March 2013. ISSN 0899-7667, 1530-888X. doi: 10.1162/NECO_a_00409.

E. Turner, K. V. Dabholkar, and O. Barak. Charting and navigating the space of solutions for recurrent neural networks. In Advances in Neural Information Processing Systems , volume 34, pages 25320-25333. Curran Associates, Inc., 2021.

P. R. Vlachas, W. Byeon, Z. Y. Wan, T. P. Sapsis, and P. Koumoutsakos. Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 474(2213):20170844, 2018. doi: 10.1098/rspa.2017.0844.

P.R. Vlachas, J. Pathak, B.R. Hunt, T.P. Sapsis, M. Girvan, E. Ott, and P. Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Networks , 126:191-217, 2020.

C. Wang, D. Han, Q. Liu, and S. Luo. A deep learning approach for credit scoring of peerto-peer lending using attention mechanism lstm. IEEE Access , 7:2161-2168, 2019. doi: 10.1109/ACCESS.2018.2887138.

D. Zhang, H.and Liu and Z. Wang. Controlling chaos: suppression, synchronization and chaotification . Springer Science & Business Media, 2009.

Xun Zheng, Manzil Zaheer, Amr Ahmed, Yuan Wang, Eric P. Xing, and Alexander J. Smola. State space LSTM models with particle MCMC inference. CoRR , abs/1711.11179, 2017. URL http://arxiv.org/abs/1711.11179 .

X. Zhu, Z. Wang, X. Wang, M. Zhou, and R. Ge. Understanding edge-of-stability training dynamics with a minimalist example. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=p7EagBsMAEO .

A Appendix

A.1 Analysis of bifurcation curves

A.1.1 PLRNNs

The standard PLRNN [18], given in eq. (1) in sect. 3.1, was defined by

where ϕ ( z t -1 ) = max( z t -1 , 0) . There are various extensions of this basic architecture like the dendPLRNN [10] or the 'shallow PLRNN' (shPLRNN) [24], as used in sect. 5.1 for training on single cell membrane potentials. The latter is essentially a 1-hidden-layer version of the form

with W 1 ∈ R M × L and W 2 ∈ R L × M , L ≥ M , connectivity matrices, h 1 ∈ R M , h 2 ∈ R L bias terms, and all other parameters and variables as in eq. (1). While this formulation is beneficial for training, the shPLRNN can essentially be rewritten in standard PLRNN form (see [24]).

Assume that D Ω( t ) := diag( d Ω( t ) ) is a diagonal matrix with an indicator vector d Ω( t ) := ( d 1 , d 2 , · · · , d M ) such that d m ( z m,t ) =: d m = 1 whenever z m,t > 0 , and zero otherwise. Then eq. (1) can be rewritten as

Let us ignore the inputs for simplicity. There are 2 M different configurations for matrix D Ω( t -1) and so 2 M different forms for matrix W Ω( t -1) in the system

Thus, the phase space of the system is divided into 2 M sub-regions corresponding to the indexed matrices

see [38, 39] for more details. For M = 2 , assuming

Hence, for this parameter constellation, the map simplifies as there exists only one border which divides the phase space into two distinct sub-regions, such that (7) can be rewritten as a map of the form

in (8), we have

with a l = a 11 , a r = a 11 + w 11 , b r = w 21 , d = a 22 , b l = c = 0 . The map (11) is a PWL dynamical system whose phase space is split into left and right half-planes (sub-regions) by the borderline Σ ( z 2 -axis). Note that bifurcation curves of the 2 d PLRNN (7) in the ( a 11 , a 11 + w 11 ) -parameter space can be determined analogous to those of the PWL map (11) in the ( a l , a r ) -parameter space.

Another way to simplify the PLRNN to a 2 d ( M = 2 ) PWL map with just a single border is to remove one of the ReLU nonlinearities and define ϕ ( z t -1 ) = ( ϕ 1 ( z 1 ,t -1 ) , β z 2 ,t -1 ) T , where β ∈ R and ϕ 1 is some variant of the ReLU function such as the leaky or parametric ReLU given by

Then D Ω( t ) := diag( d 1 , β ) such that

and so

This gives another example of a representative of 2 d PWL maps with only one border defined in eq. (11). We are pointing this out because eq. (11) is a generic system considered more widely in the discrete dynamical systems literature [5, 6], and also was the basis for the analyses below and in Fig. 1.

A.1.2 Fixed points of the map (11) and their bifurcations

For a l , a r , b l , b r , c, d, h 1 , h 2 ∈ R , the map (11) has the following two fixed points

The fixed points O L and O R exist iff z L 1 < 0 and z R 1 > 0 respectively; otherwise they are virtual. Hence, the existence regions of admissible fixed points are

Let D L / R be the determinant and T L / R the trace of A L / R , and

its characteristic polynomial. The corresponding eigenvalues are given by

which are always real for b l/r c ≥ 0 , while for b l/r c < 0 they are real provided that | a l/r -d | > 2 √ -b l/r c . For complex conjugate eigenvalues of A L / R obviously | λ | 2 = D L / R . Thus computing the real eigenvalues, the stability condition for the fixed points is determined as

Accordingly, the stability region of the fixed points O L and O R can be obtained by P L / R ( ± 1) = 1 ∓ ( a l/r + d ) + a l/r d -b l/r c > 0 and D L / R < 1 as

Note that when D L / R < 0 , all the eigenvalues are real and so there cannot be any spiralling orbit. Remark 1. Consider the PLRNN (2) with M = 2 . For the parameter setting (3), i.e.

Hence, the existence regions of admissible fixed points are

and their stability regions can be obtained as

Remark 2. If b l/r c = 0 , then λ 1 , 2 ( O L / R ) are real and the stability regions S L / R become

The fixed points are regular saddles for all parameters that belong to

and in this case λ 1 ( O L / R ) > 1 , 0 < λ 2 ( O L / R ) < 1 . Furthermore, they are flip saddles (i.e., with one negative eigenvalue) if parameters are in

such that 0 < λ 1 ( O L / R ) < 1 , λ 2 ( O L / R ) < -1 .

When b l/r c < 0 and | a l/r -d | < 2 √ -b l/r c , the eigenvalues are complex conjugates and both O L and O R are spirally attracting (attracting focus) if a l/r d -b l/r c < 1 . In this case, if a l/r + d > 0 then they are clockwise spiral, while for a l/r + d < 0 the spiralling motion will be counterclockwise. Moreover, for a l/r d -b l/r c > 1 they are repelling foci. Finally, for a l/r d -b l/r c = 1 , the fixed points are locally centers and they undergo a CB at the following boundaries:

̸

At these boundaries, the fixed points lose their stability with a pair of complex conjugate eigenvalues crossing the unit circle. For the parameters belonging to C L / R , the Jacobian J L / R is a rotation matrix whose determinant is equal to 1 . In this case, J L / R can be determined by a rotation number which is either rational ( p q ) or irrational ( ρ ). Therefore, in some neighborhood of O L / R , there is a region filled with invariant ellipses such that they are periodic with period p (if the rotation number is a rational number p q ) or quasiperiodic (if the rotation number is an irrational number ρ ); for more information see [60, 59]. For (1 -d ) h 1 + c h 2 = 0 , at the boundary

̸

the fixed point O L undergoes a DTB, since, if the parameters tend to τ L , then O L → ±∞ and λ ( O L ) → 1 . Similarly, for (1 -d ) h 1 + c h 2 = 0 , a DTB occurs for the fixed point O R at the boundary

A DTB of a fixed point results in its disappearance, as in this case the fixed point becomes virtual which may lead to changes in the global dynamics [6]. Furthermore, the BCB curves are given by

̸

and

̸

In addition, the DFB curves for the fixed points O L and O R are

Remark 3. The existence regions E O L and E O R are bounded by the BCB curves ξ L and ξ R .

Remark 4. The stability regions S L / R of fixed points (eq. (20)) are bounded by the DTB curves τ L / R (eqn. (29) and (30)), the DFB curves F L / R (eq. (33)), and the CB curves a l/r d -b l/r c = 1 . For instance, for d = 1 , S L / R are illustrated in Fig. S1 ( a ) . In this case, the stability regions only exist for b l/r c < 0 and a l/r -d < -2 √ -b l/r c . Moreover, as shown in Fig. S1 ( b ) , for d = 0 . 01 , these stability regions can exist for both cases b l/r c < 0 , a l/r -d < -2 √ -b l/r c (in blue), and b l/r c < 0 , a l/r -d > 2 √ -b l/r c (in green), but not for b l/r c ≥ 0 . Furthermore, if c = 1 , there are stability regions S L / R for the two cases b l/r c < 0 , a l/r -d < -2 √ -b l/r c (in blue), and b l/r c > 0 (in purple); see Fig. S2 ( a ) . Finally, when c = 0 , as explained in Remark 2, the stability regions have the form (24), i.e.

2

0

-2

2

2

0

0

-2

-2

bu/r

al/r

2

0

-2

2

0

-2

-2

-1

0

bu/r

al/r

2

1

d

0

0

-1

-1

0

-2

-2

bu/r

al/r

2

d

0

-2

10

0

0

2

-10

-2

bl/r

al/r

Since the system (11) is a linear map in each sub-region L and R , there cannot be any n -cycle, n ≥ 2 , with all periodic points on only one linear side. So, all periodn orbits have both letters L and R in their symbolic sequence.

A.1.3 2-cycles of the map (11) and their bifurcations

The 2-cycle O RL of the map (11) is determined by solving the equation T L ◦ T R ( z 1 , z 2 ) = ( z 1 , z 2 ) T where

In this case if ( I -J L J R ) is invertible, then the solution ( z 1 , z 2 ) T = ( z (1) , z (1) ) T is given by

Hence, the existence region of the 2-cycle O RL is

The characteristic polynomial of J RL = J L J R = ( a l a r + b r c a l c + c d a r b l + b r d d 2 + b l c ) is given by

and

Thus, the stability region of O RL is

̸

In addition, for ( (1 -d ) h 1 + c h 2 )( a l + d + a l d -b l c +1 ) = 0 , the set

̸

is the DTB curve for the 2 -cycle O RL . As in this case, for the parameter values belonging to τ RL , the points of the 2 -cycle O RL tend to ±∞ , and the corresponding eigenvalue tends to one. Moreover, for (1 -d ) h 1 + c h 2 = 0 , the BCB curves of O RL can be computed as

̸

Note that here the condition (1 -d ) h 1 + c h 2 = 0 guarantees a regular BCB in the sense that only one periodic point of O RL collides with the switching boundary; for more details see [6]. Besides,

is the DFB curve of the 2-cycle O RL .

Remark 5. One can see that for (1 -d ) h 1 + c h 2 = 0 the DFB curves of the fixed points O L and O R ( F L and F R ) and the BCB boundaries of the 2-cycle O RL ( ξ 1 RL and ξ 2 RL ) are the same. In this case, the DFB of the fixed points can lead to the (attracting) 2-cycle O RL .

̸

A.1.4 3-cycles of the map (11) and their bifurcations

Here, we investigate the existence, stability and bifurcation structure of maximal or basic 3-cycles. Note that for the continuous map (11), basic n -cycles O RL n -1 ( n ≥ 3 ) exist in pairs with their complementary cycles ( O RL n -2 R ), and they appear via BCBs such that one of them may be attracting and the other repelling [6, 20, 38]. In this case, a BCB of basic cycles demonstrates a non-smooth fold bifurcation which includes a stable basic orbit and an unstable nonbasic orbit [6, 5, 19]. Furthermore, the complementary orbits can have nonempty stability regions such that, similar to the basic orbits, they are bounded by curves of BCBs, DTBs and DFBs [6, 5].

Basic 3-cycles O RL 2 and their complementary cycles O R 2 L . The basic 3-cycle O RL 2 can be obtained from the equation T L ◦ T L ◦ T R ( z 1 , z 2 ) = ( z 1 , z 2 ) T where

If ( I -J 2 L J R ) is invertible, then the solution ( z 1 , z 2 ) T = ( z (1) 1 , z (1) 2 ) T is

where

Further where

Therefore, the existence region of the 3-cycle O RL 2 is given by

where G,G 1 , K 1 and H 1 are defined in (47) and (49). On the other hand, the characteristic polynomial of

is

According to

the stability region of the 3-cycle O RL 2 is given by

̸

Furthermore, for (1 -d ) h 1 + c h 2 = 0 and G 1 , G 2 , K 1 , K 2 , H 1 , H 2 = 0 , the DTB curve for the 3-cycle O RL 2 is

̸

̸

are (regular) BCB curves of O RL 2 . Furthermore, the set

is the DFB curve of O RL 2 . As noted, the basic 3-cycle O RL 2 exists in a pair with its complementary cycle O R 2 L . Moreover, the existence region of O R 2 L can easily be found by interchanging the letters L and R in all notations of the equations (45)- (49) and considering

Further, the stability region of the 3-cycle O R 2 L for the parameter values satisfying (57) is given by

̸

Notice that whenever the stable 3-cycle O RL 2 exists, its complementary orbit O R 2 L also exists, but it is unstable. Furthermore, for (1 -d ) h 1 + c h 2 = 0 both the 3-cycles O RL 2 and O R 2 L appear at the same BCB curves (55). On the other hand, the DTB and DFB curves of the 3-cycle O R 2 L are given by

and

respectively.

A.1.5 Multiple attractor bifurcations (MABs) of the map (11)

To detect multiple attractor bifurcations for the map (11), a straightforward way is to determine the overlapping stability regions of different periodic orbits. For instance, as shown in Fig. 1A, for c = 0 . 8 , d = 0 . 2 , b l = -0 . 4 , b r = 0 . 5 , two stability regions S RL and S RL 2 overlap in the ( a l , a r ) -parameter plane (or in the ( a 11 , a 11 + w 11 ) -parameter space for the 2 d PLRNN (7)). Their overlapping region, displayed in yellow, reveals the structure of the ( a l , a r ) -parameter plane. This helps us to find various MABs. Assuming h 2 = 0 and varying h 1 from a negative value to a positive one, an MAB of the form

occurs in the overlapping region. An example of this kind of bifurcation is illustrated in Fig. S3A.

Moreover, Fig. S4 indicates, for c = 0 . 9 , d = 0 . 3 , b l = -0 . 6 , b r = -1 . 54 , there are nonempty overlapping regions S RL ∩ S RL 2 and S RL 2 ∩ S R . This leads to the occurrence of two different MABs given by (61) and

for h 2 = 0 and h 1 changing from negative to positive values. Both of these bifurcations are shown in Fig. S3A and Fig. S3B, associated with the points P 1 and P 2 in Fig. S4. Note that in Fig. S4, all the points P 2 , P 3 and P 4 belong to the overlapping region S RL 2 ∩ S R (in sky blue). These points are related to the parameter values c = 0 . 9 , d = 0 . 3 , b l = -0 . 6 , b r = -1 . 54 , a r = -1 . 8 , h 2 = 0 , and they only differ in the parameter a l . In this case, one can see that fixing all parameters and changing only the parameter a l , from P 2 to P 4 , the basins of attraction change. The corresponding basins of attraction for these three points are demonstrated in Fig. S3B (right) and Fig. S5 for h 1 = 0 . 5 (after the bifurcation).

A2

1

0

×

-1

-2

-0.5

0

0.5

1

h1

B

0.4

0.2

×

0

-0.2

-0.4

-0.3

3-0.2-0.1

0

0.1

0.2

0.3

0.4

0.5

h1

4

2

0

-2

-3

-2

-1

0

1

2

0

2

-1.5

-1

-0.5

0

0.5

1

X

P2

P3

P4

-1

CR

ER.C2

（—FRC2

m+

1

-2

m+

1

-2

FR=RL

FR

ILp

-3

-3

TRC

P

TRC2

-1.5

-1

-0.5

0

0.5

1

-1.5

-1

-0.5

0

0.5

1

2

1

0

-1

-2

-3

-2

-1.5

-1

-0.5

0

0.5

1

1.5

X

2

1

y

0

-2

-2

-1

0

1

2

X

A.2 Proofs of theorems

A.2.1 Proof of theorem 1

Proof. Let L ( θ ) be some loss function employed for PLRNN training that decomposes in time as L = ∑ T t =1 L t . Then

Denoting the Jacobian of system (2) at time t by

we have

where ∂ + denotes the immediate partial derivative (see [48] for more details). Assume that Γ k is a k -cycle ( k ≥ 1) of (2). Thus, Γ k is a set of temporally successive periodic points

such that all of them are fixed points of

and k is the smallest such positive integer (for k = 1 , Γ 1 is a fixed point of F θ ). Similar to (65), the tangent vector ∂ z t + k ∂θ can be computed as

Thus, for z t ∗ k = F k θ ( z t ∗ k ) we have

Accordingly

where P ∏ k -1 r =0 J t ∗ k -r (1) = det ( I -∏ k -1 r =0 J t ∗ k -r ) . Moreover, from (63) and (70) we have

Now, suppose that Γ k undergoes a DTB, such that the fixed or cyclic points given by (66) tend to infinity and one of their eigenvalues tends to 1 for some parameter value θ = θ 0 . This implies P ∏ k -1 r =0 J t ∗ k -r (1) becomes zero at θ = θ 0 and so, due to (70), ∥ ∥ ∥ ∂ z t ∗ k ∂θ ∥ ∥ ∥ goes to infinity. Therefore the norm of the loss gradient, ∥ ∥ ∂ L t ∂θ ∥ ∥ , tends to infinity at θ = θ 0 which results in a abrupt jump in the loss function.

Let { z t 1 , z t 2 , z t 3 , . . . } be an orbit which converges to Γ k , i.e.

Then there exists a neighborhood U of Γ k and k sub-sequences { z t km } ∞ m =1 , { z t km +1 } ∞ m =1 , · · · , { z t km +( k -1) } ∞ m =1 of the sequence { z t n } ∞ n =1 such that all these sub-sequences belong to U and

for every z t n ∈ U there is some s ∈ { 0 , 1 , 2 , · · · , k -1 } such that z t n ∈ { z t km + s } ∞ m =1 .

This implies for every z t n ∈ U with z t n ∈ { z t km + s } ∞ m =1 , there exists some ˜ n ∈ N such that z t n = z t k ˜ n + s and lim ˜ n →∞ z t k ˜ n + s = z t ∗ k -s . Consequently, there exists some ˜ N ∈ N such that for every ˜ n ≥ ˜ N both z t k ˜ n + s and z t ∗ k -s belong to the same sub-region and so the matrices W Ω( t k ˜ n + s ) and W Ω( t ∗ k -s ) ( s ∈ { 0 , 1 , 2 , · · · , k -1 } ) are identical. Without loss of generality, let s = 0 . Since z t k (˜ n +1) = F k ( z t k ˜ n ) , so

On the other hand, lim ¯ n →∞ ∂ z t k (˜ n +1) ∂θ = lim ¯ n →∞ ∂ z t k ˜ n ∂θ , which results in

This means as ¯ n →∞ , for any orbit converging to Γ k the norm of the loss gradient tends to infinity at θ = θ 0 which completes the proof.

A.2.2 Proof of theorem 2

Proof. Let Γ k be a k -cycle ( k ≥ 1) of (2) defined by periodic points (66). Suppose further that Γ k undergoes a BCB for some parameter value θ = θ 0 . Hence, one of its periodic points, e.g.

z t ∗ k , collides with one border. Therefore, z mt ∗ k = 0 for some 1 ≤ m ≤ M by the definition of discontinuity boundaries in [38, 39]. Similar to the proof of Theorem 1, for θ = A , W we have

where 1 ( n,m ) is an M × M indicator matrix with a 1 for the ( n, m ) 'th entry and 0 everywhere else. Since z mt ∗ k = 0 at θ = θ 0 , due to (76) ∂ + z t ∗ k -1 ∂θ becomes the zero vector at θ = θ 0 . Consequently, ∥ ∥ ∥ ∂ z t ∗ k -1 ∂θ ∥ ∥ ∥ and so ∥ ∥ ∂ L t ∂θ ∥ ∥ vanishes at θ = θ 0 . Now it can be shown that at θ = θ 0 the loss gradient goes to zero for every z 1 ∈ B Γ k (the proof is similar to the last part of the proof of Theorem 1).

A.2.3 Proof of corollary 1

̸

Proof. For M = 2 , let h 1 = 0 . Then the DFB curves of the fixed point Γ 1 coincide with the BCB curves of the 2-cycle O RL of the form

or

For M > 2 , assume that Γ 1 = { z ∗ 1 } is a fixed point of the system, i.e.

̸

where P I -W Ω( t ∗ 1 ) (1) is the characteristic polynomial of I -W Ω( t ∗ 1 ) at 1 . Let us denote the first row of the adjoint matrix of I -W Ω( t ∗ 1 ) by adj ( I -W Ω( t ∗ 1 ) ) 1 . If adj ( I -W Ω( t ∗ 1 ) ) 1 h = 0 , then we can analogously demonstrate that the DFB curves of the fixed point align with the BCB curves of the 2 -cycles. This implies that, in accordance with Theorem 2, DFBs of fixed points will also lead to vanishing gradients in the loss function.

A.2.4 Convergence of SCYFI

To ensure that SCYFI almost surely converges, we can simply choose the number of random initializations (i.e., N out in algorithm 1) large enough such that every linear subregion will have been sampled with probability almost 1 .More precisely, drawing uniformly from the 2 M different D Ω -matrices (linear subregions) for initialization, the probability that a particular subregion has not been drawn after r repetitions is p = (1 -1 2 M ) r . Hence, in order to ensure that all 2 M subregions have been visited with probability 1 -ϵ we need r ≥ ⌈ ln( ϵ ) ln(1 -1 2 M ) ⌉ iterations. Choosing N out = r , we can thus ensure that SCYFI was initialized in each subregion with probability almost 1 , and thus, in the limit, will have probed all subregions for dynamical objects. This argument extends to k -cycles by replacing 2 M by 2 kM above (strictly, a more precise bound for k ≥ 2 is given by 2 M ( k -1) × (2 M -1) = 2 Mk -2 M ( k -1) , due to the fact that the PLRNN (2) is a linear map in each subregion and, hence, cannot have any k -cycles with all periodic points in only one subregion).

in which

A.2.5 Proof of theorem 3

Proof. We examine the convergence and scaling behavior of SCYFI for fixed points. A similar argument applies to k -cycles where k > 1 .

Let z ∗ 1 be a fixed point of the system, i.e.

z ∗ 1 is a true fixed point iff

where D Ω( t ∗ 1 ) = diag ( d 1 ( t ∗ 1 ) , d 2 ( t ∗ 1 ) , · · · , d M ( t ∗ 1 )) and 0 < a < 1 is a positive real constant.

For examining SCYFI's efficiency, here we focus on two scenarios that impose specific constraints on parameters θ ; other cases remain to be investigated.

̸

Case (I) : Let R be a randomly generated matrix with uniformly distributed entries in the interval [0 , 1) , and h = 0 be a random vector with all its components being non-negative. For an arbitrary ϵ > 0 , we set

and so ∥ A ∥ + ∥ W ∥ < 1 . Therefore

and so

Then

In this case, for any n ∈ N , we also have

This ensures the stability of all fixed points and k -cycles of the system.

According to (85), we have

Hence, all the elements of ( I -W Ω( t ∗ 1 ) ) -1 are positive, and so z m,t ∗ 1 > 0 for every t ∗ 1 . This implies that all true and virtual fixed points exist within a singular sub-region. Additionally, only one fixed point is true, while all the other fixed points are virtual.

Case (II) : Let h = ( h 1 , h 2 , · · · , h M ) T be a random vector with all h m uniformly distributed in (0 , 1] and

Assume further that R 1 is a randomly generated matrix with uniformly distributed entries in the interval ( -1 , 0] , and for M ≥ 2

Consider

and S ⊂ { 1 , 2 , · · · , M } = I such that K = 2 M -card ( S ) ≪ 2 M . Suppose that R 2 = diag ( r 1 , · · · , r M ) is a randomly chosen diagonal matrix with r m uniformly distributed in ( -1 , 1) for m ∈ I \ S , and the other elements ( m ∈ S ) uniformly distributed in ( r ∗ -1 , 0) where r ∗ = ( M -1) α max β max β min . Since

If then

which implies ∥ A ∥ + ∥ W ∥ < 1 . We set ϵ > 0 large enough to satisfy the condition

On the other hand, for any t we have

Hence

Since for every t ∗ 1

so

̸

̸

̸

̸

̸

Moreover a ss ∈ ( r ∗ -1 , 0) , for every s ∈ S , and thus

̸

̸

Therefore, due to (98) and (99), z s,t ∗ 1 > 0 for every t ∗ 1 and s ∈ S . This means that all true and virtual fixed points only exist within a relatively small number of sub-regions, denoted as K = 2 M -card ( S ) ≪ 2 M . Given our specific initalization of θ , in both cases (I) and (II) there is a set of K different sub-regions, each associated with a unique D Ω( t ) matrix. We refer to the entire set of these matrices as

SCYFI, by its definition, only moves within the sub-regions that have virtual and true fixed points, continuing until it discovers a true fixed point (or gets stuck in a virtual cycle). Thus, it can iterate between J ≤ K sub-regions

or within the set of virtual fixed points

In case (I) , there is only one true fixed point. Since all virtual fixed points are located within the same single sub-region, SCYFI's initialization will naturally position it within the correct linear region, requiring no more than 1 iteration. Hence, it needs at most 2 iterations to find the true fixed point. Consequently, SCYFI's scaling is constant.

For case (II) , if we suppose that SCYFI follows the virtual/true fixed point structure of the underlying system in these K sub-regions, the necessity for the probability of discovering the fixed point to be close to 1, specifically 1 -ϵ , is to have

iterations. Since 1 ≤ card ( S ) ≤ M -1 , so K ≥ 2 and ln(1 -1 K ) ≈ -1 K . For ϵ ∗ ≥ ϵ , let N = ⌈ ln( ϵ ∗ ) ln(1 -1 K ) ⌉ ≥ ⌈ ln( ϵ ) ln(1 -1 K ) ⌉ , then

which implies the number of iterations is bounded from above. If, for every M , we choose K small enough, then the upper bound will stay within a linear growth.

A.2.6 Proof of theorem 4

In GTF [24], during training RNN latent states are replaced by a weighted sum of forward propagated states z t = F θ ( z t -1 ) and data-inferred states ¯ z t = G -1 ϕ ( x t ) (obtained by inversion of the decoder model G ϕ ):

where 0 ≤ α ≤ 1 is the GTF parameter (usually adaptively regulated in training, see [24]). This leads to the following factorization of Jacobians in PLRNN (2) training:

Proof. ( i ) Since ∥ A ∥ + ∥ W ∥ ≤ 1 , we have ∀ t ∥ ∥ W Ω( t ) ∥ ∥ = ∥ ∥ A + WD Ω( t ) ∥ ∥ ≤ ∥ A ∥ + ∥ W ∥ ∥ ∥ D Ω( t ) ∥ ∥ ≤ ∥ A ∥ + ∥ W ∥ ≤ 1 , (107)

and so and thus

Since 0 < 1 -1 r < 1 , inserting 1 -1 r < α = α ∗ < 1 into the r.h.s. of (113) again gives ρ ( ∏ n i =1 J GTF t i ) < 1 for any n ∈ N , implying that no DTB can occur.

where ρ denotes the spectral radius of a matrix. In this case, for any n ∈ N , we also have

Now, for any 0 < α < 1 , the product of Jacobians under GTF is

and

Hence ρ ( ∏ n i =1 J GTF t i ) < 1 which implies for any n ∈ N and 0 < α < 1 , the product ∏ n i =1 J GTF t i has no eigenvalue equal to 1 and so no DTB can occur (see definition of DTB in sect. 3).

( ii ) Let ∥ A ∥ + ∥ W ∥ = r > 1 , then for any n ∈ N we have

A.3 Additional results

A

2

DTB

DFB+BCB

1

DFB

0

SRC

-1

DTB

S

-2

一

2

-1

0

1

2

ar

B

5

DFB+BCB

alzn+1if zn≤0

DTB

４３２１０

Zn+1 =

arZn+1if zn≥0

2

1

一

2

-1

0

1

2

a

E

1.0

F

5.0

0.5

2.5

0.0

2

0.0

-0.5

2.5

-1.0

-0.4

-0.2

0.0

0.2

0.4

-5.0

-0.4

-0.2

0.0

0.2

0.4

a11

a11

A.3.1 Scaling analysis

Although the results presented in Fig. 2 suggest that SCYFI's scaling behavior is much better than theoretically expected, the fact that it is hard to obtain ground truth comparisons for high-dimensional systems (because of the combinatorial explosion) generally makes an extensive empirical analysis difficult. For Fig. 2 we therefore focused on scenarios for which we can also provide analytical curves for an exhaustive search strategy (eq. (5)) and where we then either examined scaling with cycle-order k for rather low-dimensional systems, or where we explicitly embedded fixed points to search for which allowed us to move to very high dimensionality M . In general we observed that the scaling behavior also depended on the PLRNN's matrix norms and the eigenspectrum of the embedded fixed points, so we constructed different scenarios where we varied these factors as well.

To construct a fairly well behaved case with low matrix norms, we randomly generated matrices R with uniformly distributed entries in the interval [ -1 , 1] and then normalized by its maximal eigenvalue: We set PLRNN parameters A = 1 λ max diag ( R ) and W = 1 λ max ( R -diag ( R )) , and chose h uniformly in the interval [ -50 , 50] . For each of 10 different such systems, we fixed the number of outer loops and inner loops ( N out , N in in Algorithm 1) such that a fixed point would be detected in at least 50 / 75 independent runs of the algorithm, and then determined the total number n of linear regions (i.e., across all N out different initializations) the algorithm needed to cycle through

to detect a stable fixed point. We also ensured that across all different runs this stable fixed point would be the same, in accordance with our assumptions. The resulting scaling behavior was well fitted by a doubly-logarithmic curve of the form c 1 ln(ln( M )) + c 2 ( R 2 ≈ 0 . 913 , p < 10 -4 ). This low-matrix norm scenario with a stable fixed point may be seen as a kind of lower bound on the scaling.

To embed a specific fixed point z ∗ , we again start with a matrix R as described above and take A = diag ( R ) and W = ( R -diag ( R )) . We then minimize

subject to A staying diagonal and W off-diagonal (we observed that adding a small Gaussian noise term to the right appearance of z ∗ in eq. 114 which decayed proportionally to the learning rate improved numerical stability in the optimization process). The such constructed PLRNNs generally have several fixed points, but to compute n we only search for the inserted fixed point z ∗ (making eq. (5) directly applicable). This way we produced 5 -10 systems, initializing R with values in [ -0 . 2 , 0 . 2] (orange curve in Fig. 2B) or [ -1 . 0 , 1 . 0] (blue curve in Fig. 2B), thus effectively restricting the eigenspectrum of the fixed point as well as the matrix norms of the PLRNN to a certain range. However, since matrix norms may change during optimization, eq. (114), our procedure is not strictly guaranteed to produce eigenspectra and matrix norms within a desired range, which is crucial especially for the first scenario where we wanted to keep norms within a 'typical range' (see below). So here, to ensure consistency among drawn systems and with our assumptions, the mean absolute eigenvalue of the embedded fixed points was kept close to 0 . 31 ± 0 . 05 and the mean maximum absolute eigenvalue close to 1 . 25 ± 0 . 13 . For > 75% of the resulting systems spectral matrix norms were within the range [1 . 0 , 3 . 0] . While this produced matrix spectra typical for trained PLRNNs ( > 95% out of 361 PLRNNs trained on various benchmarks and data had spectral matrix norms within [1 . 0 , 3 . 0] ), the second initialization range resulted in unnaturally large matrix norms and hence may be seen as providing a kind of upper bound on SCYFI's scaling behavior. Fig. S8 shows the best case (left; purple curve in Fig. 2B) and typical (right; orange curve in Fig. 2B) scaling scenarios on linear scale to better expose the scaling behavior and function fits.

n

6

n

300

linear subregions

linear subregions

5

200

4

100

3

乙

0

0

25

50

75

100

125

50

100

M

M

A

D

1000

Linear region

750

500

250

0

0.0

2.5

5.0

7.5

10.0

SCYFIiterations

B

Mean #linear regions

1000

750

500

250

0

0.0

2.5

5.0

7.5

10.0

SCYFI iterations

A.3.2 Loss jumps & bifurcations in PLRNN training on biophysical model simulations

Here we provide an additional illustration of how SCYFI can be used to dissect bifurcations in model training. For this, we produced time series of membrane voltage and a gating variable from a biophysical neuron model [17], on which we trained a dendPLRNN [10] using BPTT [52] with sparse teacher forcing (STF) [37]. The dendPLRNN used ( M = 9 latent states, B = 2 bases) has 2 18 different linear subregions and | θ | = 124 parameters. Fig. S10A gives the MSE loss as a function of training epoch (i.e., single SGD updates). The loss curve exhibits several steep jumps. Zooming into these points and examining the transitions in parameter space using SCYFI, we find they are indeed produced by bifurcations, with an example given in Fig. S10B. As we had done for Fig. 4 in the main text, since the state and parameter spaces are very high dimensional, for the bifurcation diagram in Fig. S10B all extracted k -cycles ( k ≥ 1 ), including fixed points, were projected onto a line given by the PCA-derived maximum eigenvalue component, and plotted as a function of training epoch. For the example in Fig. S10B, we found that a BCB (Theorem 2) underlies the transition in the qualitative dynamics of the PLRNN as training progresses. Fig. S10C illustrates the dendPLRNN dynamics just before (left) and right after (right) the bifurcation point highlighted in Fig. S10B, together with time series from the true system.

More generally, whether a bifurcation associated with vanishing gradients produces a loss jump depends on the system's dynamics before and after the bifurcation point. In the case of BCBs, one possible scenario involves a change in stability, as illustrated in Fig . S10. During a BCB, a stable fixed point (or cycle) can loose stability as it passes through the bifurcation point. The maximum Lyapunov exponent of an unstable fixed point (or cycle) is positive, resulting in exploding gradients right after the bifurcation point [37], and consequently to a very steep slope in the loss function near the bifurcation point as in Fig . S10.

A

B

0.3

0.08

0.2

0.06

0.1

Loss

PC1

0.04

0.0

D

-0.1

0.02

-0.2

0

2000

4000

24002450250025502600

C

D

0.0

0.0

-0.5

8-0.5

-1.0L

-1.0L

0

50

100

150

200

0

50

100

150

200

time time

A

0.175

S

0.150

S

0.125

0.100

0.075

170

180

190

200

210

220

B

0.3

DFB+BCB

0.2

2

0.1

0.0

-0.1

170

180

190

200

210

220

A.3.3 Dealing with bifurcations in RNN training

Here are some additional thoughts on how RNN training algorithms could possibly be modified to deal with bifurcations. If the algorithm finds itself during training in a parameter regime which does not exhibit the right topological structure, it does not make sense to further dwell within that regime, or possibly anywhere within the vicinity of the current parameter estimate. Unlike standard SGD, the algorithm should therefore perhaps take large leaps in parameter space as soon as it gets stuck in a non-suitable dynamical regime. One possibility to implement this is through a 'look-ahead' mechanism that probes for topological properties of regions not visited so far. While fully fleshing out this idea is beyond this paper, a proof of concept that this may speed up convergence is provided in Fig. S12. Along similar lines, if we knew the model's full bifurcation structure in parameter space ahead of time, we could simply pick a parameter set which corresponds to the right dynamics

describing patterns in the data best. While of course it will in general not be feasible to chart the whole bifurcation structure before training (this is in a sense the whole point of a training algorithm), it may be possible to design smart initialization procedures based on this insight, e.g. probing topological regimes at randomly selected points in parameter space before starting training and initializing with parameters that produce a desired type of dynamics (e.g., cyclic behavior) to begin with.

A1.0

B 1.0

nolook-ahead

nolook-ahead

look-ahead

look-ahead

0.9

0.9

S

S

S

5

5

0.8

0.8

0.7

0.7

0

250

500

750

1000

0

250

500

750

1000

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the dynamical systems properties of piecewise-linear recurrent neural networks (PLRNNs). Specifically:\n- Section 3.1 (and its accompanying Appendix A.1-A.5) analytically categorizes the fixed points of 2-dimensional PLRNNs, and analyzes its bifurcations by explaining how the fixed points (and cycles) change as a function of the network parameters. The analysis involves computing the Jacobian of the recurrent unit and classifying fixed points by using its spectrum. The paper specifically identifies instances of multi-stability, where where multiple attractor regions exist.\n- Theorems 1 and 2 in Section 3.2 show that a PLRNN whose unit undergoes a degenerate transcritical bifurcation (DTB) or a border collision bifurcation (BCB) has gradients trending towards infinite and zero respectively as the bifurcation occurs in parameter space. \n- Section 4 introduces the SCYFI algorithm for identifying fixed points and cycles in higher dimensional PLRNNs. SCYFI is a heuristic algorithm that finds k-cycles by using exploring the exponentially large space of linear regions from a randomly initialized space of regions. They find empirically that the algorithm finds fixed points much faster than brute-force algorithms that search naively search each subregion.\n- Section 5 applies SCYFI and their bifurcation analysis to the prediction of a neuron spiking with a PLRNN and the resulting detection of a stable 39-cycle that predicts the spike. They observe a bifurcation in the training procedure where the stable 39-cycle is eradicated due to a DTB that causes the loss to spike (as predicted by Theorem 1's analysis of exploding gradients.)\n\nSTRENGTHS:\nAs far as the reviewer is aware, the paper provides novel analysis of bifurcations on PLRNNs and identifies innovative applications of their techniques through their neuroscience application. The analysis appears to be correct. The theoretical results effectively draw connections between bifurcation theory and gradient-based training and are validated experimentally with the loss jump coinciding the the DTB bifurcation in Section 5.\n\nWEAKNESSES:\nAs of now, the SCYFI algorithm as written is highly ambiguous, and I cannot recommend acceptance as long as that holds. (These ambiguities are noted in the next part of the review.) I am happy to reconsider if these concerns and concerns about presentation are satisfactorily addressed, given the innovative nature of the results.\n\n**Note: the score of the paper was increased from 4 to 6, because the authors rewrote the algorithm more precisely.**\n\nThe results of the paper appear to be novel from a dynamical systems perspective, but their current presentation of the results may be inaccessible for the ML community of NeurIPS. Given the relative novelty of work on the intersection of dynamical systems and ML, clear explanation of dynamical systems concepts and their relevance to ML is key.\n- In particular, the paper could more explicitly trace the importance of being able to identify k-cycles and bifurcations when training neural networks. The application in Section 5 is interesting and it could be discussed more prominently in the intro to motivate the study of k-cycles for ML researchers looking for more justification.\n- The paper would benefit from more rigorously defining the bifurcations under consideration, like BCBs and DTBs, as well as having a visualization in the parameter space of a toy example. \n- The plots in Figure 1 may be unintuitive to NeurIPS readers with a limited dynamical systems background, especially as Figure 1B is not discussed in the text. This applies more broadly to figures in the main body: including more discussion of what concretely is represented in each plot (while potentially pushing some plots to the appendix) would make for a clearer read.\n\nMinor typos and notes: \n- $\\Gamma$ is notationally overloaded: In 3.1, it refers the spectrum of a fixed point. In 3.2, it is a $k$-cycle.\n- On l238, the code's location is referred to by \"[PLACEHOLDER].\" \n- The first paragraph of Section 5 refers to Figure 4, but likely means Figure 3.\n\nQUESTIONS:\nThe paper chooses to ignore external inputs $s_t$ to its RNNs, which seems like a very significant omission, given that the primary applications of RNNs in the past have been to NLP and other sequence-learning tasks. Since [some results](http://proceedings.mlr.press/v97/hanin19a/hanin19a.pdf) suggest that adding random noise RNNs can dramatically change the structure of fixed points, are the results expected to hold in regimes with noisy or structured inputs $s_t$ included? \n\nAs written, the SCYFI algorithm is somewhat ambiguous to the reviewer:\n- $z_n$ and $D_n$ terms are ambiguous. $z_n $ appears to be a known cycle ahead of time (line 1), component obtained by solving equation (7) (line 8), and a cycle already stored in $\\mathcal{L}_{k - m}$ (line 9). Which of these sets are being compared in the RHS of line 9? \n- Why does the algorithm start out by presuming to know some $z_n$ and $D_n$, or is that just setting up notation for future lines? (line 1) \n- When solving equation (7), what precisely is being solved for?  The equation relates $z_k^*$, $W_{\\Omega(k-r)}$, and $h$. What is known ahead of time, and what must be solved for? Also, how do we solve the equation? If we are solving for $D_n$ by solving for $W_{\\Omega(n)}$, then it appears that the equation is not linear.\n- Does the algorithm only work if every $\\mathcal{L}_n$ for $n < k$ is exactly correct? While it's difficult to prove convergence results due to the searching of an exponential space, some kind of theoretical guarantee about what is necessary for the algorithm to be successful would be useful, even if just in an appendix.\n\nLIMITATIONS:\nBesides the omission of $s_t$ from the analysis, limitations of the paper appear to be well-documented by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyses a class of ReLU-RNNs (PLRNNs) from the perspective of bifurcations in dynamical systems. This gives rise to new insights into the training process of RNNs (as specific bifurcations are associated with exploding or vanishing gradients -- a major issue in training RNNs on long-term dependency data), but also into the behavior of the trained RNN. The paper further proposes a novel heuristic algorithm to detect bifurcation manifolds in PLRNNs. Finally, the paper provides empirical evidence demonstrating the theoretical findings.\n\nSTRENGTHS:\nThe paper is very well written and easy to follow. The idea of applying bifurcation analysis in such a rigorous and broad manner is novel up to my knowledge. Moreover, the authors manage to leverage this perspective to gain new and interesting insights into the training behavior of a broad class of RNNs. The reviewer further appreciates that the paper does not over-claim but rather truthfully demonstrates and discusses its contributions.\n\nWEAKNESSES:\nThere are not many weaknesses. The paper is novel and interesting. However, one might argue that besides a novel theoretical perspective into the training process (restricted to the exploding and vanishing gradient problem, which has been studied in RNNs for several decades now), there is very little significance of this work. Although one can find some theoretical justification using the proposed SCYFI algorithm on why the training exhibits loss jumps, this has no influence on the training itself. More concretely, even though one might know a-priori that the RNN will run into a bifurcation associated with exploding or vanishing gradients, this does not change the training procedure. \nTherefore, in order to argue that the proposed method is not only novel, interesting and well explained, the reviewer would highly appreciate if the authors can provide an experiment or application, where the newly gained insights into the training process has an influence on how the RNN is trained in the end.\n\nQUESTIONS:\nAs already stated in the weaknesses: is there a scenario, where the gained insights can positively change/influence the training procedure of the PLRNN?\n\nSince deep feedforward neural networks are also dynamical systems (simply without forcing), can the provided theory also be applied to (residual) feedforward neural networks with ReLU activations?\n\nLIMITATIONS:\nAs already explained above, the only limitation/weakness is the limited significance of this work. If the authors can convince me of its significance, I'm happy to further increase my score.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main focus of the paper is piecewise-linear RNNs, that is ReLU-based RNNs called PLRNNs, and how certain bifurcations that take place during their training phase will have an adverse effect on their gradients and hence will cause sudden jumps in the loss functions. This helps shed light onto the difficulty of training RNNs and yields new perspectives for the problem of exploding/vanishing gradients.\n\nMoreover, the paper offers a novel way for locating fixed points and k-cycles in PLRNNs which can be helpful in visualizing and exploring the properties of these networks.\n\nThe main technical contributions are Theorem 1 and Theorem 2 which provide sufficient conditions under which the loss gradient will go to infinity or will vanish. These sufficient conditions are directly linked to the bifurcation properties of the PLRNN.\n\nSTRENGTHS:\n+the paper presents much-needed theory on a topic that is interesting both for theoreticians and practitioners\n+conceptually, the work is very interesting and helps clarify an important problem in the literature of RNNs\n+the distinction between the two cases of exploding and vanishing gradient through the lens of bifurcation diagrams is interesting\n+their algorithm for discovering fixed points and k-cycles, though heuristic in nature, seems to be performing well.\n\nWEAKNESSES:\n-the algorithm for finding fixed points doesn't come with guarantees but that's perhaps not the most important weakness.\n\nThe main weakness is in presentation and in the literature review that make the paper hard to read in certain parts. The topic on training RNNs and the difficulty that arises has been studied a lot, but more recently there have been efforts to explain certain phenomena through the lens of dynamical systems.\n\nIn terms of presentation, I believe the paper would benefit by instantiating the theorems 1 and 2, on specific simple examples. You do sth related in lines 142-184 but I would create a small subsection \"Simple Examples\" to help illustrate the points. I believe Figure 1 is hard to understand and would perhaps break it into 2 different figures.\n\nIn terms of comparison to related works, recently we have seen several works that analyze behavior of the networks at the edge of stability or using the notion of k-cycles and chaos to illustrate why RNNs are hard to train. See papers below:\n\n[1] On Scrambling Phenomena for Randomly Initialized Recurrent Networks by Chatziafratis, Panageas, Sanford, Stavroulakis\n\n[2] \"Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems\" by Chatziafratis, Nagarajan, Panageas,\n\n[3] Understanding Edge-of-Stability Training Dynamics with a Minimalist Example by Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, Rong Ge\n\n[4] Understanding Gradient Descent on Edge of Stability in Deep Learning by Sanjeev Arora, Zhiyuan Li, Abhishek Panigrahi\n\nYour paper studies low-dimensional settings, especially in Fig.4 you provided a 2d toy example that helps illustrating the tight association between the loss landscape and bifurcation curves, whereas [1] and [2] studies the 1-dimensional setting giving characterizations from Li-Yorke chaotic systems, a standard notion from the dynamical systems literature. Papers [3] and [4] study the dynamics of GD and establish new connections with the notion of \"sharpness\"; while the emphasis is more on deep learning, the papers are very related as they analyze the dynamics through eigenvalues analyses.\n\nQUESTIONS:\nQ1: Is there a simple 1-d example that is even simpler that the 2d example you provide? And that helps illustrate your two main theorems?\nQ2: In more applied settings, RNNs often operate in large dimensions. Could you comment on how your results can be helpful in these setting as well?\n\nLIMITATIONS:\nNo significant limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors provide a thorough analysis of the bifurcation structure of RNNs and its relationship with discontinuities of the loss function during the training process. The authors also present a novel algorithm for detecting fixed points and k-cycles in ReLU-based RNNs, providing a new tool for understanding the computational and dynamical properties of these networks.\n\nSTRENGTHS:\nThis paper nicely connects dynamical systems theory (DST) and machine learning by studying the problem of RNN training. This subject is known to be a challenging one, and the authors apply DST techniques to prove a rigorous result about the connection between bifurcations and jumps in the loss function during training, a question I would see as quite natural but rather unexplored in this context.\n\nThe paper is nicely written and accessible to the broad audience. Results are clearly stated and novel.\n\nWEAKNESSES:\nFrom the theoretical standpoint, the authors limit their discussions to two types of bifurcations. A more general and comprehensive discussion would be a nice addition to the paper.\n\nThe proposed algorithm appears to be purely heuristic and clashes a bit with the more rigorous first part of the paper.\n\nThe numerical experiments are a bit limited. The 2d example only qualitatively fits with the stated results: the bifurcation points do not seem to be exactly the points where the phase transition occurs. Is this due to numerical approximation?\n\nQUESTIONS:\nCan the authors say something about other kinds of bifurcations and their connection to loss function behavior during training?\n\nOther questions are listed in the \"weaknesses\" sections\n\nLIMITATIONS:\nThe authors should address the limitations more openly, both in terms of the presented algorithm and in terms of the scope of their theoretical results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-02"}