SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1:  $\mathcal{L}_k = \{\}$
2:  $i\to 0$
3: while  $i < N_{out} \ldots$  do
4: Select  $k$  subregions  $\pmb{D}_{init} = \{\pmb{D}_1, \pmb{D}_2, \dots, \pmb{D}_k\}$  at random with replacement
5:  $c\to 0$
6: while  $c < N_{in} \ldots$  do
7: Solve eq. (4) for a cycle candidate  $\{z_l^*\}_{l=1}^k$  with  $W_{\Omega(k-r)}$  as defined in eq. (2) based on  $D_{init}$
8: Determine  $\{D_l^*\}_{l = 1}^k$  based on the signs of the corresponding components of  $\{z_l^*\}_{l = 1}^k$
9: if  $D_{init} = \{D_l^*\}_{l=1}^k$  (self-consistency) and  $\forall 1 \leq s \leq k, \forall \{z_l^{(m)}\}_{l=1}^{k-s+1} \in \mathcal{L}_{k-s+1}$ :  $\{z_l^{(m)}\}_{l=1}^{k-s+1} \not\subseteq \{z_l^*\}_{l=1}^k$  then
10:  $\mathcal{L}_k\to \mathcal{L}_k\cup \{\{\pmb {z}_l^*\}_{l = 1}^k\}$
11:  $i\to c\to 0$
12: else
13:  $D_{init} \to \{D_l^*\}_{l=1}^k$
14: end if
15:  $c\to c + 1$
16: end while
17:  $i\to i + 1$
18: end while
and for the PLRNN, eq. (2), are given by the set of  $k$ -periodic points  $\{z_1^*, \ldots, z_l^*, \ldots, z_k^*\}$ , where
$$
\boldsymbol {z} _ {k} ^ {*} = \left(\mathbb {1} - \prod_ {r = 0} ^ {k - 1} \boldsymbol {W} _ {\Omega (k - r)}\right) ^ {- 1} \left[ \sum_ {j = 2} ^ {k - 1} \prod_ {r = 0} ^ {k - j} \boldsymbol {W} _ {\Omega (k - r)} + \mathbb {1} \right] \boldsymbol {h}, \tag {4}
$$
if  $\left(1 - \prod_{r=0}^{k-1} W_{\Omega(k-r)}\right)$  is invertible (if not, we are dealing with a bifurcation or a continuous set of fixed points). The other periodic points are  $z_l = F^l(z_k^*)$ ,  $l = 1, \dots, k-1$ , with corresponding matrices  $W_{\Omega(l)} = A + WD_l$ . Now, if the diagonal entries in  $D_l$  are consistent with the signs of the corresponding states  $z_{ml}^*$ , i.e. if  $d_{mm}^{(l)} = 1$  if  $z_{ml}^* > 0$  and  $d_{mm}^{(l)} = 0$  otherwise for all  $l$ ,  $\{z_1^*, \dots, z_k^*\}$  is a true cycle of eq. (2), otherwise we call it virtual. To find a  $k$ -cycle, since an  $M$ -dimensional PLRNN harbors  $2^M$  different linear sub-regions, there are approximately  $2^{Mk}$  different combinations of configurations of the matrices  $D_l$ ,  $l = 1 \dots k$ , to consider (strictly, mathematical constraints rule out some of these possibilities, e.g. not all periodic points can lie within the same sub-region/orthant).
Clearly, for higher-dimensional PLRNNs and higher cycle orders exhaustively searching this space becomes unfeasible. Instead, we found that the following heuristic works surprisingly well: First, for some order  $k$  and a random initialization of the matrices  $D_{l}, l = 1 \ldots k$ , generate a cycle candidate by solving eq. (4). If each of the points  $z_{l}^{*}, l = 1 \ldots k$ , is consistent with the diagonal entries in the corresponding matrix  $D_{l}, l = 1 \ldots k$ , and none of them is already in the current library of cyclic points, then a true  $k$ -cycle has been identified, otherwise the cycle is virtual (or a super-set of lower-order cycles). We discovered that the search becomes extremely efficient, without the need to exhaustively consider all configurations, if a new search loop is re-initialized at the last visited virtual cyclic point (i.e., all inconsistent entries  $d_{mm}^{(l)}$  in the matrices  $D_{l}, l = 1 \ldots k$ , are flipped,  $d_{mm}^{(l)} \to 1 - d_{mm}^{(l)}$ , to bring them into agreement with the signs of the solution points,  $z_{l}^{*} = [z_{ml}^{*}]$ , of eq.
(4), thus yielding the next initial configuration). It is straightforward to see that this procedure almost surely converges if  $N_{out}$  is chosen large enough, see Appx. A.2.4. The whole procedure is formalized in Algorithm 1, and the code is available at https://github.com/DurstewitzLab/SCYFI.
To validate the algorithm, we can compare analytical solutions as derived in sect. 3.1 to the output of the algorithm. To delineate all existence and stability regions, the algorithm searches for all  $k$ -cycles up to some maximum order  $K$  along a fine grid across the  $(a_{11}, a_{11} + w_{11})$ -parameter plane. A bifurcation happens whenever between two grid points a cycle appears, disappears, or changes stability (as determined from the eigenvalue spectrum of the respective  $k^{th}$ -order Jacobian). The results of this procedure are shown in Fig. 1B, illustrating that the analytical solutions for existence and stability regions precisely overlap with those identified by Algorithm 1 (see also Fig. S7).
4.2 Numerical and theoretical results on SCYFI's scaling behavior
Because of the combinatorial nature of the problem, it is generally not feasible to obtain ground truth settings in higher dimensions for SCYFI to compare to. To nevertheless assess its scaling behavior, we therefore studied two specific scenarios. For an exhaustive search, the expected and median numbers of linear subregions  $n$  until an object of interest (fixed point or cycle) is found, i.e. the number of  $\{D_{1:k}\}$  constellations that need to be inspected until the first hit, are given by
$$
E [ n ] = \frac {N + 1}{m + 1} = \frac {2 ^ {M k} + 1}{m + 1}, \quad \bar {n} = \min  \left\{n \in \mathbb {N} \left| \binom {2 ^ {M k} - n} {m} \leq \frac {1}{2} \binom {2 ^ {M k}} {m} \right. \right\} \tag {5}
$$
with  $m$  being the number of existing  $k$ -cycles and  $N$  the total number of combinations, as shown in Ahlgren [1] (assuming no prior knowledge about the mathematical limitations when drawing regions). The median  $\overline{n}$  as well as the actual median number of to-be-searched combinations required by SCYFI to find at least one  $k$ -cycle is given for low-dimensional systems in Fig. 2A as a function of cycle order  $k$ , and can be seen to be surprisingly linear as confirmed by linear regression fits to the data (see Fig. 2 legend for details). To assess scaling as a function of dimensionality  $M$ , we explicitly constructed systems with one known fixed point (see Appx. A.3.1 for details) and determined the number  $n$  of subregions required by SCYFI to detect this embedded fixed point (Fig. 2B). In general, the scaling depended on the system's eigenspectrum, but for reasonable scenarios was polynomial or even sublinear (Fig. 2B, see also Fig. S8). In either case, the number of required SCYFI iterations scaled much more favorably than would be expected from an exhaustive search.
Figure 2: A) Number of linear subregions  $n$  searched until at least one cycle of order  $k$  was found by SCYFI (blue) vs. the median number  $\overline{n}$  an exhaustive search would take by randomly drawing combinations without replacement (black) as a function of cycle order ( $M = 2$  fixed). Each data point represents the median of 50 different initializations across 5 different PLRNN models. Error bars = median absolute deviation. Linear regression fit using weighted least-squares ( $R^2 \approx 0.998, p < 10^{-30}$ ). B) Number of linear subregions  $n$  searched until a specific fixed point was found as function of dimensionality  $M$  for different eigenvalue spectra (see Appx. A.3.1 for details).
How could this surprisingly good scaling behavior be explained? As shown numerically in Fig. S9, when we initiate SCYFI in different randomly selected linear subregions, it converges to the subregions including the dynamical objects of interest exponentially fast, offsetting the combinatorial explosion. A more specific and stronger theoretical result about SCYFI's convergence speed can be obtained under certain conditions on the parameters (which agrees nicely with the numerical results in Fig. 2). It rests on the observation that SCYFI is designed to move only among subregions containing virtual or actual fixed points or cycles, based on the fact that it is always reinitialized with the next virtual fixed (cyclic) point in case the consistency check fails. The result can be stated as follows:
Theorem 3. Consider a PLRNN of the form (2) with parameters  $\theta = \{A, W, h\}$ . Under certain conditions on  $\theta$  (for which  $\| A \| + \| W \| < 1$ ), SCYFI will converge in at most linear time.
Proof. See Appx. A.2.5
5 Loss landscapes and bifurcation curves
5.1 Bifurcations and loss jumps in training
Figure 3: A) Logarithm of gradient norm of the loss in PLRNN parameter space, with ground truth parameters centered at  $(0,0)$ . Superimposed in yellow are bifurcation curves computed by SCYFI, and in green two examples of training trajectories from different parameter initial conditions (indicated by the stars). Red dots indicate the bifurcation crossing time points shown in C & D. B) Relief plot of the loss landscape from A to highlight the differences in loss altitude associated with the bifurcations. C) Loss during the training run represented by the green trajectory labeled C in A and B. Red dot indicates the time point of bifurcation crossing corresponding to the red dot in A and B. D) Same for trajectory labeled D in A and B.
Fig. 3 provides a  $2d$  toy example illustrating the tight association between the loss landscape and bifurcation curves, as determined through SCYFI, for a PLRNN trained by BPTT on reproducing a specific 16-cycle. Fig. 3A depicts a contour plot of the gradient norms with overlaid bifurcation curves in yellow, while Fig. 3B shows the MSE loss landscape as a relief for better appreciation of the sharp changes in loss height associated with the bifurcation curves. Shown in green are two trajectories from two different parameter initializations traced out during PLRNN training in parameter space, where training was confined to only those two parameters given in the graphs (i.e., all other PLRNN parameters were kept fixed during training for the purpose of this illustration). As confirmed in Fig. 3C & D, as soon as the training trajectory crosses the bifurcation curves in parameter space, a huge jump in the loss associated with a sudden increase in the gradient norm occurs. This illustrates empirically and graphically the theoretical results derived in sect. 3.
Next we illustrate the application of SCYFI on a real-world example, learning the behavior of a rodent spiking cortical neuron observed through time series measurements of its membrane potential (note that spiking is a highly nonlinear behavior involving fast within-spike and much slower between-spike time scales). For this, we constructed a 6-dimensional delay embedding of the membrane voltage [53, 28], and trained a PLRNN with one hidden layer (cf. eq. 6) using BPTT with sparse teacher forcing (STF) [37] to approximate the dynamics of the spiking neuron (see Appx. A.3.2 for a similar analysis on a biophysical neuron model). With  $M = 6$  latent states and  $H = 20$  hidden dimensions, the trained PLRNN comprises  $2^{20}$  different linear subregions and  $|\theta| = 272$  parameters, much higher-dimensional than the toy example considered above. Fig. 4A gives the MAE loss as a function of training epoch (i.e., single SGD updates), while Figs. 4B & C illustrate the well-trained behavior in time (Fig. 4B) and in a 2-dimensional projection of the model's state space obtained by PCA (Fig. 4C). The loss curve exhibits several steep jumps. Zooming into one of these regions (Fig. 4A; indicated by the red box) and examining the transitions in parameter space using SCYFI, we find they are indeed produced by bifurcations, with an example given in Fig. 4D. Note that we are now dealing with high-dimensional state and parameter spaces, such that visualization of results becomes tricky. For the bifurcation diagram in Fig. 4D we therefore projected all extracted  $k$ -cycles ( $k \geq 1$ ) onto a line given by the PCA-derived maximum eigenvalue component, and plotted this as a function of training epoch.<sup>1</sup> Since SCYFI extracts all  $k$ -cycles and their eigenvalue spectrum, we can also determine the type of bifurcation that caused the jump. While before the loss jump the PLRNN already produced time series quite similar to those of the physiologically recorded cell (Fig. 4E), a
<sup>1</sup>Of course, very many of the PLRNN parameters may change from one epoch to the next.
DTB (cf. Theorem 1) produced catastrophic forgetting of the learned behavior with the PLRNN's states suddenly diverging to minus infinity (Fig. 4F; Fig. S10 also provides an example of a CCB during PLRNN training, and Fig. S11 an example of a DFB). This illustrates how SCYFI can be used to analyze the training process with respect to bifurcation events also for high-dimensional real-world examples, as well as the behavior of the trained model (Fig. 4C).
Figure 4: A) Loss across training epochs for a PLRNN with one hidden layer trained on electrophysiological recordings from a cortical neuron. Red box zooms in on one of the training phases with a huge loss jump, caused by a DTB. Letters refer to selected training epochs in other subpanels. B) Time series of true (gray) and PLRNN-simulated (black) membrane potential in the well-trained regime (see A). C) All fixed points and cycles discovered by SCYFI for the well-trained model in state space projected onto the first two principle components using PCA. Filled circles represent stable and open circles unstable objects. The stable 39-cycle corresponds to the spiking behavior. D) Bifurcation diagram of the PLRNN as a function of training epoch around the loss peak in A. Locations of stable (filled circles) and unstable (open circles) objects projected onto the first principle component. E) Model behavior as in B shortly before the DTB and associated loss jump (from the epoch indicated in A, D). F) Model behavior as in B right around the DTB (diverging to  $-\infty$ ).
5.2 Implications for designing training algorithms
What are potential take-homes of the results in sects. 3.2 & 5.1 for designing RNN training algorithms? One possibility is to design smart initialization or training procedures that aim to place or push an RNN into the right topological regime by taking big leaps in parameter space whenever the current regime is not fit for the data, rather than dwelling within a wrong regime for too long. These ideas are discussed in a bit more depth in Appx.A.3.3, with a proof of concept in Fig. S12.
Figure 5: Example loss curves during training a PLRNN ( $M = 10$ ) on a  $2d$  cycle using gradient descent, once without GTF ( $\alpha = 0$ , blue curve) but gradient clipping, and once with GTF ( $\alpha = 0.1$ ). Note that without GTF there are several sharp loss jumps associated with bifurcations in the PLRNN parameters, while activating GTF leads to a smooth loss curve avoiding bifurcations. Note: For direct comparability both loss curves were cut off at 4 and then scaled to  $[0, 1]$ . The absolute loss is much lower for GTF.
More importantly, however, we discovered that the recently proposed technique of 'generalized teacher forcing (GTF)' [24] tends to circumvent bifurcations in RNN training altogether, leading to much faster convergence as illustrated in Fig. 5. The way this works is that GTF, by trading off forward-iterated RNN latent states with data-inferred states according to a specific annealing schedule during training (see Appx. A.2.6), tends to pull the RNN directly into the right dynamical regime. In fact, for DTBs we can strictly prove these will never occur in PLRNN training with the right adjustment of the GTF parameter:
Theorem 4. Consider a PLRNN of the form (2) with parameters  $\theta = \{A, W, h\}$ . Assume that it has a stable fixed point or  $k$ -cycle  $\Gamma_k$  ( $k \geq 1$ ) that undergoes a degenerate transcritical bifurcation (DTB) for some parameter value  $\theta = \theta_0 \in \theta$ .
(i) If  $\| \mathbf{A}\| +\| \mathbf{W}\| \leq 1$  , then for any GTF parameter  $0 <   \alpha <  1$  , GTF controls the system, avoiding a DTB and, hence, gradient divergence at  $\theta_0$
(ii) If  $\| \mathbf{A}\| +\| \mathbf{W}\| = r > 1$  , then for any  $1 - \frac{1}{r} <  \alpha <  1$  GTF prevents a DTB and, hence, gradient divergence at  $\theta_0$
Proof. See Appx. A.2.6.
As this example illustrates, we may be able to amend training procedures such as to avoid specific types of bifurcations.
DS theory [3, 47, 58] is increasingly appreciated in the ML.AI community as a powerful mathematical framework for understanding both the training process of ML models [49, 54, 44, 16] as well as the behavior of trained models [62, 33, 61]. While the latter is generally useful for understanding how a trained RNN performs a given ML task, with prospects of improving found solutions, it is in fact imperative in areas like science or medicine where excavating the dynamical behavior and repertoire of trained models yields direct insight into the underlying physical, biological, or medical processes the model is supposed to capture. However, application of DS theory is often not straightforward, especially when dealing with higher-dimensional systems, and commonly requires numerical routines that may only find some of the dynamical objects of interest, and also only approximate solutions. One central contribution of the present work therefore was the design of a novel algorithm, SCYFI, that can exactly locate fixed points and cycles of a wide class of ReLU-based RNNs. This provides an efficient instrument for the DS analysis of trained models, supporting their interpretability and explainability.
A surprising observation was that SCYFI often finds cycles in only linear time, despite the combinatorial nature of the problem, a feature shared with the famous Simplex algorithm for solving linear programming tasks [36, 32, 57]. While we discovered numerically that SCYFI for empirically relevant scenarios converges surprisingly fast, deriving strict theoretical guarantees is hard, and so far we could establish stronger theoretical results on its convergence properties only under specific assumptions on the RNN parameters. Further theoretical work is therefore necessary to precisely understand why the algorithm works so effectively.
In this work we applied SCYFI to illuminate the training process itself. Since RNNs are themselves DS, they are subject to different forms of bifurcations during training as their parameters are varied under the action of a training algorithm (similar considerations may apply to very deep NNs). It has been recognized for some time that bifurcations in RNN training may give rise to sudden jumps in the loss [44, 16], but the phenomenon has rarely been treated more systematically and mathematically. Another major contribution of this work thus was to formally prove a strict connection between three types of bifurcations and abrupt changes in the gradient norms, and to use SCYFI to further reveal such events during PLRNN training on various example systems. There are numerous other types of bifurcations (e.g., center bifurcations, Hopf bifurcations etc.) that are likely to impact gradients during training, only for a subset of which we could provide formal proofs here. As we have demonstrated, understanding the topological and bifurcation landscape of RNNs could help improve training algorithms and provide insights into their working. Hence, a more general understanding of how various types of bifurcation affect the training process in a diverse range of RNN architectures is a promising future avenue not only for our theoretical understandings of RNNs, but also for guiding future algorithm design.
Acknowledgements This work was supported by the German Research Foundation (DFG) through individual grants Du 354/10-1 & Du 354/15-1 to DD, within research cluster FOR-5159 ("Resolving prefrontal flexibility"; Du 354/14-1), and through the Excellence Strategy EXC 2181/1 - 390900948 (STRUCTURES). We also thank Mahasheta Patra for lending us code for graphing analytically derived bifurcation diagrams and state spaces.
[1] J. Ahlgren. The probability distribution for draws until first success without replacement. arXiv preprint arXiv:1404.1161, 2014.
[2] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[3] Kathleen T. Alligood, Tim D. Sauer, and James A. Yorke. *Chaos: An Introduction to Dynamical Systems*. Springer, New York, NY, 1996.
[4] S. Arora, Z. Li, and A. Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning, pages 948-1024. PMLR, 2022.
[5] V. Avrutin, M. Schanz, and S. Banerjee. Occurrence of multiple attractor bifurcations in the two-dimensional piecewise linear normal form map. Nonlinear Dynamics, 67:293-307, 2012.
[6] V. Avrutin, L. Gardini, I. Sushko, and F. Tramontana. Continuous and Discontinuous Piecewise Smooth One-Dimensional Maps: Invariant Sets and Bifurcation structures, volume 1. World Scientific, Singapore, 2019.
[7] S. Banerjee, MS. Karthik, G. Yuan, and J. A. Yorke. Bifurcations in one-dimensional piecewise smooth maps-theory and applications in switching circuits. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 47(3):389-394, 2000.
[8] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994. doi: 10.1109/72.279181.
[9] S.h Bouktif, A. Fiaz, A. Ouni, and M. A. Serhani. Multi-sequence LSTM deep learning and metaheuristics for electric load forecasting. *Energies*, 13(2), 2020. ISSN 1996-1073. doi: 10.3390/en13020391.
[10] M. Brenner, F. Hess, J. M. Mikhaeil, L. F. Bereska, Z. Monfared, P. Kuo, and D. Durstewitz. Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems. In Proceedings of the 39th International Conference on Machine Learning, pages 2292-2320. PMLR, June 2022. ISSN: 2640-3498.
[11] S. L. Brunton, J. L. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932-3937, 2016.
[12] K. Champion, B. Lusch, J. N. Kutz, and S. L. Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45):22445-22451, 2019. doi: 10.1073/pnas.1906995116.
[13] V. Chatziafratis, S. G. Nagarajan, and I. Panageas. Better depth-width trade-offs for neural networks through the lens of dynamical systems. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1469-1478. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/chatziafratis20a.html.
[14] V. Chatziafratis, I. Panageas, and S. Sanford, C.and Stavroulakis. On scrambling phenomena for randomly initialized recurrent networks. Advances in Neural Information Processing Systems, 35:18501-18513, 2022.
[65] C. Wang, D. Han, Q. Liu, and S. Luo. A deep learning approach for credit scoring of peerto-peer lending using attention mechanism LSTM. IEEE Access, 7:2161-2168, 2019. doi: 10.1109/ACCESS.2018.2887138.
[66] D. Zhang, H.and Liu and Z. Wang. Controlling chaos: suppression, synchronization and chaotification. Springer Science & Business Media, 2009.
[67] Xun Zheng, Manzil Zaheer, Amr Ahmed, Yuan Wang, Eric P. Xing, and Alexander J. Smola. State space LSTM models with particle MCMC inference. CoRR, abs/1711.11179, 2017. URL http://arxiv.org/abs/1711.11179.
[68] X. Zhu, Z. Wang, X. Wang, M. Zhou, and R. Ge. Understanding edge-of-stability training dynamics with a minimalist example. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=p7EagBsMAEO.
A.1 Analysis of bifurcation curves
A.1.1 PLRNNs
The standard PLRNN [18], given in eq. (1) in sect. 3.1, was defined by
$$
\boldsymbol {z} _ {t} = F _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t - 1}, \boldsymbol {s} _ {t}\right) = \boldsymbol {A} \boldsymbol {z} _ {t - 1} + \boldsymbol {W} \phi \left(\boldsymbol {z} _ {t - 1}\right) + \boldsymbol {C} \boldsymbol {s} _ {t} + \boldsymbol {h},
$$
where  $\phi(z_{t-1}) = \max(z_{t-1}, 0)$ . There are various extensions of this basic architecture like the dendPLRNN [10] or the 'shallow PLRNN' (shPLRNN) [24], as used in sect. 5.1 for training on single cell membrane potentials. The latter is essentially a 1-hidden-layer version of the form