SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lukas Eisenmann $^{1,2,*}$ , Zahra Monfared $^{1,*}$ , Niclas Goring $^{1,2}$ , and Daniel Durstewitz $^{1,2,3}$
{lukas.eisenmann,zahra.monfared,daniel.durstwitz}@zi-mannheim.de
$^{1}$ Department of Theoretical Neuroscience, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
$^{2}$ Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany
$^{3}$ Interdisciplinary Center for Scientific Computing, Heidelberg University *These authors contributed equally
Recurrent neural networks (RNNs) are popular machine learning tools for modeling and forecasting sequential data and for inferring dynamical systems (DS) from observed time series. Concepts from DS theory (DST) have variously been used to further our understanding of both, how trained RNNs solve complex tasks, and the training process itself. Bifurcations are particularly important phenomena in DS, including RNNs, that refer to topological (qualitative) changes in a system's dynamical behavior as one or more of its parameters are varied. Knowing the bifurcation structure of an RNN will thus allow to deduce many of its computational and dynamical properties, like its sensitivity to parameter variations or its behavior during training. In particular, bifurcations may account for sudden loss jumps observed in RNN training that could severely impede the training process. Here we first mathematically prove for a particular class of ReLU-based RNNs that certain bifurcations are indeed associated with loss gradients tending toward infinity or zero. We then introduce a novel heuristic algorithm for detecting all fixed points and  $k$ -cycles in ReLU-based RNNs and their existence and stability regions, hence bifurcation manifolds in parameter space. In contrast to previous numerical algorithms for finding fixed points and common continuation methods, our algorithm provides exact results and returns fixed points and cycles up to high orders with surprisingly good scaling behavior. We exemplify the algorithm on the analysis of the training process of RNNs, and find that the recently introduced technique of generalized teacher forcing completely avoids certain types of bifurcations in training. Thus, besides facilitating the DST analysis of trained RNNs, our algorithm provides a powerful instrument for analyzing the training process itself.
Recurrent neural networks (RNNs) are common and powerful tools for learning sequential tasks or modeling and forecasting time series data [31, 65, 2, 43, 9, 26]. Typically, RNNs are "black boxes," whose inner workings are hard to dissect. Techniques from dynamical system theory (DST) can significantly aid in this effort, as RNNs are formally discrete-time dynamical systems (DS) [37]. A better understanding of how RNNs solve their tasks is important for detecting failure modes and designing better architectures and training algorithms. In scientific machine learning, on the other hand, RNNs are often employed for reconstructing unknown DS from sets of time series observations [10, 30, 64, 63, 23, 55], e.g. in climate or disease modeling. In this case, the RNN is supposed to provide an approximation to the flow of the observed system that reproduces all its dynamical properties, e.g., cyclic behavior in climate or epidemiological systems. In such scientific or medical
settings, a detailed understanding of the RNN's dynamics and its sensitivity to parameter variations is in fact often crucial.
Beyond understanding the dynamical properties of a once trained RNN, it may also be of interest to know how its dynamical repertoire changes with changes in its parameters. The parameter space of any DS is partitioned into regions (or sets) with topologically different dynamical behaviors by bifurcation curves (or, more generally, manifolds). Such bifurcations, qualitative changes in the system dynamics due to parameter variations, may not only be crucial in scientific applications where we use RNNs, for instance, to predict tipping points in climate systems or medical scenarios like sepsis detection [45]. They also pose severe challenges for the training process itself as qualitative changes in RNN dynamics may go in hand with sudden jumps in the loss landscape [16, 44]. Although methods from DST have significantly advanced the field in recent years, especially with regards to algorithms for reconstructing nonlinear DS from data [55, 46, 67], progress is still hampered by the lack of efficient tools for analysing the dynamics of higher-dimensional RNNs and their bifurcations. In particular, methods are needed for exactly locating geometrical objects like fixed points or cycles in an RNN's state space, but current numerical techniques do not scale well to higher-dimensional scenarios and provide only approximate solutions [29, 21, 61].
The contributions of this work are threefold: After first providing an introduction into bifurcations of piecewise-linear (ReLU-based) RNNs (PLRNNs), which have been extensively used for reconstructing DS from empirical data [10, 30], we mathematically prove that certain bifurcations during the training process will indeed cause loss gradients to diverge to infinity, resulting in abrupt jumps in the loss, while others will cause them to vanish. As RNNs are likely to undergo several bifurcations during training on their way from some initial parameter configuration toward a dynamics that successfully implements any given task, this poses severe challenges for RNN training and may be one of the reasons for exploding and vanishing gradients [16, 44, 8, 25]. We then create a novel, efficient heuristic algorithm for exactly locating all fixed points and  $k$ -cycles in PLRNNs, which can be used to delineate bifurcation manifolds in higher-dimensional systems. Our algorithm finds these dynamical objects in many orders of magnitude less time than an exhaustive search would take. Using this algorithm, we demonstrate empirically that steep cliffs in loss landscapes and bifurcation curves indeed tightly overlap, and that bifurcations in the system dynamics are accompanied by sudden loss jumps. Finally, we prove and demonstrate that the recently introduced technique of generalized teacher forcing (GTF) [24] completely eliminates certain types of bifurcation in training, providing an explanation for its efficiency.
DS analysis of trained RNNs In many areas of science one is interested in identifying the nonlinear DS that explains a set of observed time series [12, 23]. A variety of purely data-driven machine learning approaches have been developed for this purpose [11, 51, 15], but mostly RNNs are used for the goal of reconstructing DS from measured time series [10, 55, 30, 64, 63, 46, 45]. In scientific settings in particular, but also often in engineering applications, we seek a detailed understanding of the system dynamics captured – or the dynamical repertoire produced – by a trained RNN [62, 33, 61, 14, 13]. To analyze an RNN's dynamics, typically its fixed points are determined numerically, for instance by optimizing some cost function [21, 34, 61], by numerically traversing directional fibers [29], or by co-training a switching linear dynamical system [56]. These techniques, however, often scale poorly with dimensionality, provide only approximate solutions, and are not designed for detecting other dynamical objects like  $k$ -cycles. This is, however, crucial for understanding the complete dynamical repertoire of a trained RNN. PLRNNs are of particular interest in this context [10, 55, 30, 38], because their piecewise linear nature makes some of their properties analytically accessible [55, 10], which is a tremendous advantage from the perspective of DST. In this work, we develop an efficient heuristic algorithm for locating a PLRNN's fixed points exactly, as well as its  $k$ -cycles.
Bifurcations and loss jumps in RNN training The idea that bifurcations in RNN dynamics could impede the training process is not new [16, 44]. Doya [16], to our knowledge, was the first to point out that even in simple single-unit RNNs with sigmoid activation function (saddle-node) bifurcations may occur as an RNN parameter is adapted during training. This may not only cause an abrupt jump in training loss, but could lead to situations where it is impossible, even in principle, to reach the training objective (the desired target output), as across the bifurcation point there is a discrete change
in network behavior [16, 68, 4]. Pascanu et al. [44] discussed similar associations between steep cliffs in RNN loss functions and bifurcations. Although profound for training success, this topic received surprisingly little attention over the years. Haschke and Steil [22] extended previous work by a more formal treatment of bifurcation boundaries in RNNs, and Marichal et al. [35] examined fold bifurcations in RNNs. The effects of bifurcations and their relation to exploding gradients in gated recurrent units (GRUs) was investigated in Kanai et al. [27]. Ribeiro et al. [50] looked at the connection between dynamics and smoothness of the cost function, but failed to find a link between bifurcations and jumps in performance. In contrast, Rehmer and Kroll [49] observed large gradients at bifurcation boundaries and concluded that bifurcations can indeed cause problems in gradient-based optimization. To the best of our knowledge, we are, however, the first to formally prove a direct link between bifurcations and the behavior of loss gradients, and to derive a systematic and efficient algorithmic procedure for identifying bifurcation manifolds for a class of ReLU-based RNNs.
In this paper we will focus on PLRNNs as one representative of the wider class of ReLU-based RNNs, but similar derivations and algorithmic procedures could be devised for any type of ReLU-based RNN (in fact, many other types of ReLU-based RNNs could be brought into the same functional form as PLRNNs; e.g. [10]). We will first, in sect. 3.1, provide some theoretical background on bifurcations in PLRNNs, and illustrate how existence and stability regions of fixed points and cycles could be analytically computed for low dimensional  $(2d)$  PLRNNs. In sect. 3.2 we will then state two theorems regarding the association of bifurcations and loss gradients in training. It turns out that for certain types of bifurcations exploding or vanishing gradients are inevitable in gradient-based training procedures like Back-Propagation Through Time (BPTT).
3.1 Bifurcation curves in PLRNN parameter space
The PLRNN, originally introduced as a kind of discrete time neural population model [18], has the general form
$$
\boldsymbol {z} _ {t} = F _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t - 1}, \boldsymbol {s} _ {t}\right) = \boldsymbol {A} \boldsymbol {z} _ {t - 1} + \boldsymbol {W} \phi \left(\boldsymbol {z} _ {t - 1}\right) + \boldsymbol {C} \boldsymbol {s} _ {t} + \boldsymbol {h}, \tag {1}
$$
where  $\pmb{z}_t \in \mathbb{R}^M$  is the latent state vector and  $\pmb{\theta}$  are system parameters consisting of diagonal matrix  $\pmb{A} \in \mathbb{R}^{M \times M}$  (auto-regression weights), off-diagonal matrix  $\pmb{W} \in \mathbb{R}^{M \times M}$  (coupling weights),  $\phi(\pmb{z}_{t-1}) = \max(\pmb{z}_{t-1}, 0)$  is the element-wise rectified linear unit (ReLU) function,  $\pmb{h} \in \mathbb{R}^M$  a constant bias term, and  $\pmb{s}_t \in \mathbb{R}^K$  represents external inputs weighted by  $\pmb{C} \in \mathbb{R}^{M \times K}$ . The original formulation of the PLRNN is stochastic [18, 30], with a Gaussian noise term added to eq. (1), but here we will consider the deterministic variant.
Formally, like other ReLU-based RNNs, PLRNNs constitute piecewise linear (PWL) maps, a subclass of piecewise smooth (PWS) discrete-time DS. Define  $D_{\Omega(t)} \coloneqq \mathrm{diag}(d_{\Omega(t)})$  as a diagonal matrix with indicator vector  $d_{\Omega(t)} \coloneqq (d_1, d_2, \dots, d_M)$  such that  $d_m(z_{m,t}) \eqqcolon d_m = 1$  whenever  $z_{m,t} > 0$ , and zero otherwise. Then (1) can be rewritten as
$$
\boldsymbol {z} _ {t} = F _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t - 1}\right) = \left(\boldsymbol {A} + \boldsymbol {W} \boldsymbol {D} _ {\Omega (t - 1)}\right) \boldsymbol {z} _ {t - 1} + \boldsymbol {h} =: \boldsymbol {W} _ {\Omega (t - 1)} \boldsymbol {z} _ {t - 1} + \boldsymbol {h}, \tag {2}
$$
where we have ignored external inputs  $s_t$  for simplicity. There are in general  $2^M$  different configurations for matrix  $D_{\Omega(t-1)}$  and hence for matrix  $W_{\Omega(t-1)}$ , dividing the phase space into  $2^M$  sub-regions separated by switching manifolds (see Appx. A.1.1 for more details).
Recall that fixed points of a map  $z_{t} = F_{\theta}(z_{t - 1})$  are defined as the set of points for which we have  $z^{*} = F_{\theta}(z^{*})$ , and that the type (node, saddle, spiral) and stability of a fixed point can be read off from the eigenvalues of the Jacobian  $J_{t} := \frac{\partial F_{\theta}(z_{t - 1})}{\partial z_{t - 1}} = \frac{\partial z_{t}}{\partial z_{t - 1}}$  evaluated at  $z^{*}$  [3, 47]. Similarly, a  $k$ -cycle of map  $F_{\theta}$  is a periodic orbit  $\{z_1^*, z_2^*, \ldots, z_k^*\}$  such that each of the periodic points  $z_{i}^{*}, i = 1 \ldots k$ , is distinct, and is a solution to the equation  $z_{i}^{*} = F_{\theta}^{k}(z_{i}^{*})$ , i.e. the  $k$  times iterated map  $F_{\theta}$ . Type and stability of a  $k$ -cycle are then determined via the Jacobian  $\prod_{r=1}^{k} J_{t+k-r} = \prod_{r=1}^{k} \frac{\partial z_{t+k-r}}{\partial z_{t+k-r-1}} = \frac{\partial z_{t+k-1}}{\partial z_{t-1}}$ . Solving these equations and computing the corresponding Jacobians thus allows to determine all existence and stability regions of fixed points and cycles, where the latter are a subset of the former, bounded by bifurcation curves (see Appx. A.1 for more formal details).
To provide a specific example, assume  $M = 2$  and fix - for the purpose of this exposition - parameters  $w_{12} = w_{22} = 0$ , such that we have
$$
\boldsymbol {W} _ {\Omega^ {1}} = \boldsymbol {W} _ {\Omega^ {3}} = \left( \begin{array}{c c} a _ {1 1} & 0 \\ 0 & a _ {2 2} \end{array} \right), \quad \boldsymbol {W} _ {\Omega^ {2}} = \boldsymbol {W} _ {\Omega^ {4}} = \left( \begin{array}{c c} a _ {1 1} + w _ {1 1} & 0 \\ w _ {2 1} & a _ {2 2} \end{array} \right), \tag {3}
$$
i.e. only one border which divides the phase space into two distinct sub-regions (see Appx. A.1.1). For this setup, Fig. 1A provides examples of analytically determined stability regions for two low order cycles in the  $(a_{11}, a_{11} + w_{11})$ -parameter plane (see Appx. A.1). Note that there are regions in parameter space where two or more stability regions overlap: In these regions we have multi-stability, the co-existence of different attractor states in the PLRNN's state space.
As noted above, bifurcation curves delimit the different stability regions in parameter space and are hence associated with abrupt changes in the topological structure of a system's state space. In general, there are many different types of bifurcations through which dynamical objects can come into existence, disappear, or change stability (see, e.g., [3, 40, 47]), the most common ones being saddle node, transcritical, pitchfork, homoclinic, and Hopf bifurcations. In comparison with smooth systems, bifurcation theory of PWS (or PWL) maps includes additional dynamical phenomena related to the existence of borders in the phase space [6]. Border-collision bifurcations (BCBs) arise when for a PWS map a specific point of an invariant set collides with a border and this collision leads to a qualitative change of dynamics [5, 6, 42]. More specifically, a CCB occurs, if for a PWS map  $z_{t} = F_{\theta}(z_{t - 1})$  a fixed point or  $k$ -cycle either crosses the switching manifold  $\sum_{i} \coloneqq \{z \in \mathbb{R}^{n} : e_{i}^{\top}z = 0\}$  transversely at  $\theta = \theta^{*}$  and its qualitative behavior changes in the event, or if it collides on the border with another fixed point or  $k$ -cycle and both objects disappear [7, 41]. Degenerate transcritical bifurcations (DTBs) occur when a fixed point or a periodic point of a cycle tends to infinity and one of its eigenvalues tends to 1 by variation of a parameter. Specifically, let  $\Gamma_{k}, k \geq 1$ , be a fixed point or a  $k$ -cycle with the periodic points  $\{z_1^*, z_2^*, \ldots, z_k^*\}$ , and assume  $\lambda^i$  denotes an eigenvalue of the Jacobian matrix at the periodic point  $z_i^*, i \in \{1, 2, \dots, k\}$ . Then  $\Gamma_{k}$  undergoes a DTB at  $\theta = \theta^{*}$ , if  $\lambda^i(\theta^*) \to +1$  and  $\| z_i^* \| \to \infty$ .  $\Gamma_{k}$  undergoes a degenerate flip bifurcation (DFB), iff  $\lambda^i(\theta^*) = -1$  and the map  $F^k$  has locally, in some neighborhood of  $z_i^*$ , infinitely many 2-cycles at  $\theta = \theta^{*}$ . A center bifurcation (CB) occurs, if  $\Gamma_{k}$  has a pair of complex conjugate eigenvalues  $\lambda_{1,2}$  and locally becomes a center at the bifurcation value  $\theta = \theta^{*}$ , i.e. if its eigenvalues are complex and lie on the unit circle  $(|\lambda_{1,2}(\theta^{*})| = 1)$ . Another important class of bifurcations are multiple attractor bifurcations (MABs), discussed in more detail in Appx. A.1.5 (see Fig. S3). In addition to existence and stability regions of fixed points and cycles, in Appx. A.1.2-A.1.4 we also illustrate how to analytically determine the types of bifurcation curves bounding the existence and stability regions of  $k$ -cycles (DTB, DFB, CB and CCB curves; see also Fig. S6 for a 1d example).
3.2 Bifurcations and loss jumps in training
Here we will prove that major types of bifurcations discussed above are always associated with exploding or vanishing gradients in PLRNNs during training, and hence often with abrupt jumps in the loss. For this we may assume any generic loss function  $\mathcal{L}(\theta)$ , like a negative log-likelihood or a mean-squared-error (MSE) loss, and a gradient-based training technique like BPTT [52] or Real-Time-Recurrent-Learning (RTRL) that involves a recursion (via chain rule) through loss terms across time. The first theorem establishes that a DTB inevitably causes exploding gradients.
Theorem 1. Consider a PLRNN of the form (2) with parameters  $\pmb{\theta} = \{\pmb{A},\pmb{W},\pmb{h}\}$ . Assume that it has a stable fixed point or  $k$ -cycle  $\Gamma_{k}$  ( $k\geq 1$ ) with  $\mathcal{B}_{\Gamma_k}$  as its basin of attraction. If  $\Gamma_{k}$  undergoes a degenerate transcritical bifurcation (DTB) for some parameter value  $\theta = \theta_0\in \theta$  then the norm of the PLRNN loss gradient,  $\left\| \frac{\partial\mathcal{L}_t}{\partial\theta}\right\|$ , tends to infinity at  $\theta = \theta_0$  for every  $z_{1}\in \mathcal{B}_{\Gamma_{k}}$ , i.e.  $\lim_{\theta \to \theta_0}\left\| \frac{\partial\mathcal{L}_t}{\partial\theta}\right\| = \infty$ .
Proof. See Appx. A.2.1
However, bifurcations may also cause gradients to suddenly vanish, as it is the case for a BCB as established by our second theorem:
Theorem 2. Consider a PLRNN of the form (2) with parameters  $\theta = \{A, W, h\}$ . Assume that it has a stable fixed point or  $k$ -cycle  $\Gamma_k$  ( $k \geq 1$ ) with  $\mathcal{B}_{\Gamma_k}$  as its basin of attraction. If  $\Gamma_k$  undergoes a border collision bifurcation (BCB) for some parameter value  $\theta = \theta_0 \in \theta$ , then the gradient of the loss function,  $\frac{\partial \mathcal{L}_t}{\partial \theta}$ , vanishes at  $\theta = \theta_0$  for every  $z_1 \in \mathcal{B}_{\Gamma_k}$ , i.e.  $\lim_{\theta \to \theta_0} \left\| \frac{\partial \mathcal{L}_t}{\partial \theta} \right\| = 0$ .
Figure 1: A) Analytically calculated stability regions for a 2-cycle  $(\mathcal{S}_{\mathcal{R}\mathcal{L}},$  red), a 3-cycle  $(\mathcal{S}_{\mathcal{R}\mathcal{L}^2},$  blue), and their intersection (yellow) in the  $(a_{11},a_{11} + w_{11})$  -parameter plane for the system eq. (3) with  $a_{22} = 0.2$ $w_{21} = 0.5$  . B) Same as determined by SCYFI, with bifurcation curves bordering the stability regions labeled by the type of bifurcation (DTB  $=$  Degenerate Transcritical Bifurcation,  $\mathrm{BCB} =$  Border Collision Bifurcation, DFB  $=$  Degenerate Flip Bifurcation). C) Bifurcation graph (showing the stable cyclic points in the  $z_{1}$  coordinate) along the cross-section in A indicated by the gray line, illustrating the different types of bifurcation encountered when moving in and out of the various stability regions in A. D) State space at the point denoted  ${}^{*}\mathrm{D}^{\prime}$  in A (for  $a_{11} = 0.253,$ $a_{11} + w_{11} = -2.83)$  , where the 2-cycle (red) and 3-cycle (blue) co-exist for the same parameter settings, with their corresponding basins of attraction indicated by lighter colors.
Proof. See Appx. A.2.2.
Corollary 1. Assume that the PLRNN (2) has a stable fixed point  $\Gamma_{1}$  with  $\mathcal{B}_{\Gamma_1}$  as its basin of attraction. If  $\Gamma_{1}$  undergoes a degenerate flip bifurcation (DFB) for some parameter value  $\theta = \theta_0\in \pmb{\theta}$ , then this will always coincide with a BCB of a 2-cycle, and as a result  $\lim_{\theta \to \theta_0}\left\| \frac{\partial\mathcal{L}_t}{\partial\theta}\right\| = 0$  for every  $z_{1}\in \mathcal{B}_{\Gamma_{1}}$ .
Proof. See Appx. A.2.3.
Hence, certain bifurcations will inevitably cause gradients to suddenly explode or vanish, and often induce abrupt jumps in the loss (see Appx. A.3.2 for when this will happen for a BCB). We emphasize that these results are general and hold for systems of any dimension, as well as in the presence of inputs. Since inputs do not affect the Jacobians in eqn. (70), (71) and (76), they do not change the theorems (even if they would affect the Jacobians, Theorem 2 would be unaltered, and Theorem 1 could be amended in a straightforward way). Furthermore, since we are addressing bifurcations that occur during model training, from this angle inputs may simply be treated as either additional parameters (if piecewise constant) or states of the system (without changing any of the mathematical derivations). In fact, mathematically, any non-autonomous dynamical system (RNN with inputs) can always and strictly be reformulated as an autonomous system (RNN without inputs), see [3, 47, 66].
4 Heuristic algorithm for finding PLRNN bifurcation manifolds
4.1 Searcher for fixed points and cycles (SCYFI): motivation and validation
In sect. 3.1 and Appx. A.1.2-A.1.4 we derived existence and stability regions for fixed points and low order  $(k\leq 3)$  cycles in  $2d$  PLRNNs with specific parameter constraints analytically. For higher-order cycles and higher-dimensional PLRNNs (or any other ReLU-type RNN) this is no longer feasible due to the combinatorial explosion in the number of subregions that need to be considered as  $M$  and  $k$  increase. Here we therefore introduce an efficient search algorithm for finding all  $k$ -cycles of a given PLRNN, which we call Searcher for Cycles and Fixed points: SCYFI (Algorithm 1). Once all  $k$ -cycles  $(k\geq 1)$  have been detected on some parameter grid, the stability-/existence regions of these objects and thereby the bifurcation manifolds can be determined.  $k$ -cycles were defined in sect. 3.1,
Algorithm 1 SCYFI
The algorithm is iteratively run with  $k = 1 \cdots K_{max}$ , with  $K_{max}$  the max. order of cycles tested  
Input: PLRNN parameters  $A, W, h$ ;
$\mathcal{L} = \{\mathcal{L}_n\}_{n=1}^{k-1}$ : collection of all sets  $\mathcal{L}_n = \{\{z_l^{(m)}\}_{l=1}^n\}_{m=1}^{M_n}$  of all lower order  $n$ -cycles discovered so far, where  $M_n$  is the number of found cycles  $\{z_l^{(m)}\}_{l=1}^n$  of order  $n$ , with corresponding ReLU derivative matrices  $\{D_l^{(m)}\}_{l=1}^n$ .
Parameters:
$N_{out}$ : max. number of random initialisations;
$N_{in}$ : max. number of iterations
Output:  $\mathcal{L} \cup \mathcal{L}_k$ ;  $\mathcal{L}_k$ : set of all discovered  $k$ -cycles