SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although the results presented in Fig. 2 suggest that SCYFI's scaling behavior is much better than theoretically expected, the fact that it is hard to obtain ground truth comparisons for high-dimensional systems (because of the combinatorial explosion) generally makes an extensive empirical analysis difficult. For Fig. 2 we therefore focused on scenarios for which we can also provide analytical curves for an exhaustive search strategy (eq. (5)) and where we then either examined scaling with cycle-order  $k$  for rather low-dimensional systems, or where we explicitly embedded fixed points to search for which allowed us to move to very high dimensionality  $M$ . In general we observed that the scaling behavior also depended on the PLRNN's matrix norms and the eigenspectrum of the embedded fixed points, so we constructed different scenarios where we varied these factors as well.
To construct a fairly well behaved case with low matrix norms, we randomly generated matrices  $\pmb{R}$  with uniformly distributed entries in the interval  $[-1, 1]$  and then normalized by its maximal eigenvalue: We set PLRNN parameters  $\pmb{A} = \frac{1}{\lambda_{max}} \mathrm{diag}(\pmb{R})$  and  $\pmb{W} = \frac{1}{\lambda_{max}} (\pmb{R} - \mathrm{diag}(\pmb{R}))$ , and chose  $\pmb{h}$  uniformly in the interval  $[-50, 50]$ . For each of 10 different such systems, we fixed the number of outer loops and inner loops ( $N_{out}, N_{in}$  in Algorithm 1) such that a fixed point would be detected in at least  $50/75$  independent runs of the algorithm, and then determined the total number  $n$  of linear regions (i.e., across all  $N_{out}$  different initializations) the algorithm needed to cycle through
to detect a stable fixed point. We also ensured that across all different runs this stable fixed point would be the same, in accordance with our assumptions. The resulting scaling behavior was well fitted by a doubly-logarithmic curve of the form  $c_{1}\ln (\ln (M)) + c_{2}(R^{2}\approx 0.913,p < 10^{-4})$ . This low-matrix norm scenario with a stable fixed point may be seen as a kind of lower bound on the scaling.
To embed a specific fixed point  $z^{*}$ , we again start with a matrix  $\mathbf{R}$  as described above and take  $\mathbf{A} = \mathrm{diag}(\mathbf{R})$  and  $\mathbf{W} = (\mathbf{R} - \mathrm{diag}(\mathbf{R}))$ . We then minimize
$$
\min  _ {\boldsymbol {A}, \boldsymbol {W}, \boldsymbol {h}} | \boldsymbol {z} ^ {*} - ((\boldsymbol {A} + \boldsymbol {W} \cdot \boldsymbol {D} _ {\Omega (t ^ {*})}) \cdot \boldsymbol {z} ^ {*} + \boldsymbol {h}) |, \tag {114}
$$
subject to  $\mathbf{A}$  staying diagonal and  $\mathbf{W}$  off-diagonal (we observed that adding a small Gaussian noise term to the right appearance of  $z^{*}$  in eq. 114 which decayed proportionally to the learning rate improved numerical stability in the optimization process). The such constructed PLRNNs generally have several fixed points, but to compute  $n$  we only search for the inserted fixed point  $z^{*}$  (making eq. (5) directly applicable). This way we produced  $5 - 10$  systems, initializing  $\mathbf{R}$  with values in  $[-0.2, 0.2]$  (orange curve in Fig. 2B) or  $[-1.0, 1.0]$  (blue curve in Fig. 2B), thus effectively restricting the eigenspectrum of the fixed point as well as the matrix norms of the PLRNN to a certain range. However, since matrix norms may change during optimization, eq. (114), our procedure is not strictly guaranteed to produce eigenspectra and matrix norms within a desired range, which is crucial especially for the first scenario where we wanted to keep norms within a 'typical range' (see below). So here, to ensure consistency among drawn systems and with our assumptions, the mean absolute eigenvalue of the embedded fixed points was kept close to  $0.31 \pm 0.05$  and the mean maximum absolute eigenvalue close to  $1.25 \pm 0.13$ . For  $>75\%$  of the resulting systems spectral matrix norms were within the range [1.0, 3.0]. While this produced matrix spectra typical for trained PLRNNs ( $>95\%$  out of 361 PLRNNs trained on various benchmarks and data had spectral matrix norms within [1.0, 3.0]), the second initialization range resulted in unnaturally large matrix norms and hence may be seen as providing a kind of upper bound on SCYFI's scaling behavior. Fig. S8 shows the best case (left; purple curve in Fig. 2B) and typical (right; orange curve in Fig. 2B) scaling scenarios on linear scale to better expose the scaling behavior and function fits.
Figure S8: Zoom-ins on linear axes of the scenarios with doubly-logarithmic (left;  $R^2 \approx 0.913$ ,  $p < 10^{-4}$ ) and quadratic (right;  $R^2 \approx 0.925$ ,  $p < 10^{-5}$ ) scaling behaviors from Fig. 2B.
Figure S9: A) Initializing SCYFI in a wide array of different subregions (different colors), it quickly converges – within just a few iterations – to the same set of linear subregions which contain the dynamical objects of interest (fixed points in this case). B) The number of different subregions explored by SCYFI when started from different initializations shrinks exponentially fast with the number of iterations. Shown are means  $(\pm \mathrm{stdv})$  from 10 different systems with  $M = 10$ .
A.3.2 Loss jumps & bifurcations in PLRNN training on biophysical model simulations
Here we provide an additional illustration of how SCYFI can be used to dissect bifurcations in model training. For this, we produced time series of membrane voltage and a gating variable from a biophysical neuron model [17], on which we trained a dendPLRNN [10] using BPTT [52] with sparse teacher forcing (STF) [37]. The dendPLRNN used ( $M = 9$  latent states,  $B = 2$  bases) has  $2^{18}$  different linear subregions and  $|\theta| = 124$  parameters. Fig. S10A gives the MSE loss as a function of training epoch (i.e., single SGD updates). The loss curve exhibits several steep jumps. Zooming into these points and examining the transitions in parameter space using SCYFI, we find they are indeed produced by bifurcations, with an example given in Fig. S10B. As we had done for Fig. 4 in the main text, since the state and parameter spaces are very high dimensional, for the bifurcation diagram in Fig. S10B all extracted  $k$ -cycles ( $k \geq 1$ ), including fixed points, were projected onto a line given by the PCA-derived maximum eigenvalue component, and plotted as a function of training epoch. For the example in Fig. S10B, we found that a BCB (Theorem 2) underlies the transition in the qualitative dynamics of the PLRNN as training progresses. Fig. S10C illustrates the dendPLRNN dynamics just before (left) and right after (right) the bifurcation point highlighted in Fig. S10B, together with time series from the true system.
More generally, whether a bifurcation associated with vanishing gradients produces a loss jump depends on the system's dynamics before and after the bifurcation point. In the case of BCBs, one possible scenario involves a change in stability, as illustrated in Fig . S10. During a CCB, a stable fixed point (or cycle) can lose stability as it passes through the bifurcation point. The maximum Lyapunov exponent of an unstable fixed point (or cycle) is positive, resulting in exploding gradients right after the bifurcation point [37], and consequently to a very steep slope in the loss function near the bifurcation point as in Fig . S10.
Figure S10: A) Loss across training epochs for a dendPLRNN ( $M = 9$  states,  $B = 2$  bases) trained on a biophysical neuron model in a limit cycle (spiking) regime. Red dots indicate training epochs just before and after a loss jump for which time graphs are given in C and D. B) Bifurcation diagram of the dendPLRNN as a function of training epoch, with all state space locations of stable (filled circles) and unstable (open circles) objects projected onto the first principle component. The loss jump in A is accompanied by a bifurcation from fixed point to cyclic behavior. C) Time series of the voltage variable ( $x_{1}$ ) of the biophysical model (gray) and that predicted by the dendPLRNN (black) before the bifurcation event indicated in B. D) Same directly after the bifurcation event.
Figure S11: Loss jump induced by a degenerate flip bifurcation (DFB). A) Loss during a training run of a PLRNN  $(M = 5)$  on a 2-cycle. The gray line indicates a loss jump corresponding to a DFB and a simultaneously occurring border collision bifurcation (BCB). B) Bifurcation diagram of the PLRNN, with the DFB and CCB leading to the destruction of the fixed point and the emergence of a 2-cycle as indicated by the gray line.
A.3.3 Dealing with bifurcations in RNN training
Here are some additional thoughts on how RNN training algorithms could possibly be modified to deal with bifurcations. If the algorithm finds itself during training in a parameter regime which does not exhibit the right topological structure, it does not make sense to further dwell within that regime, or possibly anywhere within the vicinity of the current parameter estimate. Unlike standard SGD, the algorithm should therefore perhaps take large leaps in parameter space as soon as it gets stuck in a non-suitable dynamical regime. One possibility to implement this is through a 'look-ahead' mechanism that probes for topological properties of regions not visited so far. While fully fleshing out this idea is beyond this paper, a proof of concept that this may speed up convergence is provided in Fig. S12. Along similar lines, if we knew the model's full bifurcation structure in parameter space ahead of time, we could simply pick a parameter set which corresponds to the right dynamics
describing patterns in the data best. While of course it will in general not be feasible to chart the whole bifurcation structure before training (this is in a sense the whole point of a training algorithm), it may be possible to design smart initialization procedures based on this insight, e.g. probing topological regimes at randomly selected points in parameter space before starting training and initializing with parameters that produce a desired type of dynamics (e.g., cyclic behavior) to begin with.
Figure S12: A) Example loss curves for RNNs trained on electrophysiological recordings by BPTT without (blue) vs. with (black) 'look-ahead' (the look-ahead function checks whether there would be a bifurcation away from a stable fixed point when taking  $10 \times$  the current gradient step). Dashed yellow line indicates the epoch at which the look-ahead step was executed. B) Average across 6 loss curves of RNNs trained without (blue) vs. with (black) look-ahead. Error bands = SEM.