SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial \boldsymbol {z} _ {t ^ {*} k}}{\partial \theta} = \left(\boldsymbol {I} - \prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {* k} - r}\right) ^ {- 1} \frac {\partial^ {+} \boldsymbol {z} _ {t ^ {* k}}}{\partial \theta} = \frac {\operatorname {a d j} \left(\boldsymbol {I} - \prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {* k} - r}\right)}{P _ {\prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {* k} - r}} (1)} \frac {\partial^ {+} \boldsymbol {z} _ {t ^ {* k}}}{\partial \theta}, \tag {70}
$$
where  $P_{\prod_{r=0}^{k-1} J_{t^{*k-r}}}(1) = \det \left(I - \prod_{r=0}^{k-1} J_{t^{*k-r}}\right)$ . Moreover, from (63) and (70) we have
$$
\left\| \frac {\partial \mathcal {L} _ {t}}{\partial \theta} \right\| = \frac {1}{P _ {\prod_ {r = 0} ^ {k - 1} J _ {t ^ {*} k - r}} (1)} \left\| \frac {\partial \mathcal {L} _ {t}}{\partial z _ {t ^ {* k}}} a d j \left(\boldsymbol {I} - \prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {*} k - r}\right) \frac {\partial^ {+} \boldsymbol {z} _ {t ^ {*} k}}{\partial \theta} \right\|. \tag {71}
$$
Now, suppose that  $\Gamma_{k}$  undergoes a DTB, such that the fixed or cyclic points given by (66) tend to infinity and one of their eigenvalues tends to 1 for some parameter value  $\theta = \theta_0$ . This implies  $P_{\prod_{r=0}^{k-1} J_{t^{*}k-r}}(1)$  becomes zero at  $\theta = \theta_0$  and so, due to (70),  $\left\| \frac{\partial z_{t^{*}k}}{\partial \theta} \right\|$  goes to infinity. Therefore the norm of the loss gradient,  $\left\| \frac{\partial L_t}{\partial \theta} \right\|$ , tends to infinity at  $\theta = \theta_0$  which results in an abrupt jump in the loss function.
Let  $\{z_{t_1}, z_{t_2}, z_{t_3}, \ldots\}$  be an orbit which converges to  $\Gamma_k$ , i.e.
$$
\lim  _ {n \rightarrow \infty} d \left(\boldsymbol {z} _ {t _ {n}}, \Gamma_ {k}\right) = 0. \tag {72}
$$
Then there exists a neighborhood  $U$  of  $\Gamma_{k}$  and  $k$  sub-sequences  $\{z_{t_{k_m}}\}_{m = 1}^{\infty},\{z_{t_{k_{m + 1}}}\}_{m = 1}^{\infty},\dots ,$ $\{z_{t_{k_m + (k - 1)}}\}_{m = 1}^{\infty}$  of the sequence  $\{z_{t_n}\}_{n = 1}^{\infty}$  such that all these sub-sequences belong to  $U$  and
a)  $\pmb{z}_{t_{km + s}} = F^k (\pmb{z}_{t_{k(m - 1) + s}}), s = 0,1,2,\dots ,k - 1,$
b)  $\lim_{m\to \infty}\pmb{z}_{t_{km + s}} = \pmb{z}_{t^{*k} - s},s = 0,1,2,\dots ,k - 1,$
c) for every  $\mathbf{z}_{t_n} \in U$  there is some  $s \in \{0, 1, 2, \dots, k - 1\}$  such that  $\mathbf{z}_{t_n} \in \{\mathbf{z}_{t_{k_m + s}}\}_{m = 1}^{\infty}$ .
This implies for every  $z_{t_n} \in U$  with  $z_{t_n} \in \{z_{t_{k_m + s}}\}_{m = 1}^{\infty}$ , there exists some  $\tilde{n} \in \mathbb{N}$  such that  $z_{t_n} = z_{t_{k\bar{n} +s}}$  and  $\lim_{\tilde{n}\to \infty}z_{t_k\tilde{n} +s} = z_{t^{*k} - s}$ . Consequently, there exists some  $\tilde{N} \in \mathbb{N}$  such that for every  $\tilde{n} \geq \tilde{N}$  both  $z_{t_{k\bar{n} +s}}$  and  $z_{t^{*k} - s}$  belong to the same sub-region and so the matrices  $W_{\Omega (t_{k\bar{n} +s})}$  and  $W_{\Omega (t^{*k} - s)}$  ( $s \in \{0,1,2,\dots ,k - 1\}$ ) are identical. Without loss of generality, let  $s = 0$ . Since  $z_{t_{k(\tilde{n} +1)}} = F^{k}(z_{t_{k\tilde{n}}})$ , so
$$
\frac {\partial \boldsymbol {z} _ {t _ {k (\bar {n} + 1)}}}{\partial \theta} = \prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {*} k - r} \frac {\partial \boldsymbol {z} _ {t _ {k \bar {n}}}}{\partial \theta} + \frac {\partial^ {+} \boldsymbol {z} _ {t _ {k (\bar {n} + 1)}}}{\partial \theta}. \tag {73}
$$
On the other hand,  $\lim_{\bar{n}\to \infty}\frac{\partial\pmb{z}_{t_k(\bar{n} + 1)}}{\partial\theta} = \lim_{\bar{n}\to \infty}\frac{\partial\pmb{z}_{t_k\bar{n}}}{\partial\theta}$ , which results in
$$
\begin{array}{l} l i m _ {\bar {n} \rightarrow \infty} \frac {\partial \boldsymbol {z} _ {t _ {k (\bar {n} + 1)}}}{\partial \theta} = \left(\boldsymbol {I} - \prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {*} k - r}\right) ^ {- 1} l i m _ {\bar {n} \rightarrow \infty} \frac {\partial^ {+} \boldsymbol {z} _ {t _ {k (\bar {n} + 1)}}}{\partial \theta} \\ = \frac {\operatorname {a d j} \left(\boldsymbol {I} - \prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {*} k - r}\right)}{P _ {\prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {*} k - r}} (1)} \lim  _ {\bar {n} \rightarrow \infty} \frac {\partial^ {+} \boldsymbol {z} _ {t _ {k (\bar {n} + 1)}}}{\partial \theta}. \tag {74} \\ \end{array}
$$
This means as  $\bar{n} \to \infty$ , for any orbit converging to  $\Gamma_k$  the norm of the loss gradient tends to infinity at  $\theta = \theta_0$  which completes the proof.
A.2.2 Proof of theorem 2
Proof. Let  $\Gamma_k$  be a  $k$ -cycle ( $k \geq 1$ ) of (2) defined by periodic points (66). Suppose further that  $\Gamma_k$  undergoes a BCB for some parameter value  $\theta = \theta_0$ . Hence, one of its periodic points, e.g.
$z_{t^{*k}}$ , collides with one border. Therefore,  $z_{m t^{*k}} = 0$  for some  $1 \leq m \leq M$  by the definition of discontinuity boundaries in [38, 39]. Similar to the proof of Theorem 1, for  $\pmb{\theta} = \pmb{A}$ ,  $\pmb{W}$  we have
$$
\frac {\partial \boldsymbol {z} _ {t ^ {* k} - 1}}{\partial \theta} = \frac {\operatorname {a d j} \left(\boldsymbol {I} - \prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {* k} - 1 - r}\right)}{P _ {\prod_ {r = 0} ^ {k - 1} \boldsymbol {J} _ {t ^ {* k} - 1 - r}} (1)} \frac {\partial^ {+} \boldsymbol {z} _ {t ^ {* k} - 1}}{\partial \theta}, \tag {75}
$$
$$
\frac {\partial^ {+} \pmb {z} _ {t ^ {* k} - 1}}{\partial w _ {n m}} = \mathbf {1} _ {(n, m)} D _ {\Omega (t ^ {* k})} \pmb {z} _ {t ^ {* k}},
$$
$$
\frac {\partial^ {+} \boldsymbol {z} _ {t ^ {*} k - 1}}{\partial a _ {m m}} = \mathbf {1} _ {(m, m)} \boldsymbol {z} _ {t ^ {* k}}, \tag {76}
$$
where  $\mathbf{1}_{(n,m)}$  is an  $M\times M$  indicator matrix with a 1 for the  $(n,m)$  'th entry and 0 everywhere else. Since  $z_{mt^{*k}} = 0$  at  $\theta = \theta_0$ , due to (76)  $\frac{\partial^{+}\boldsymbol{z}_{t^{*}k - 1}}{\partial\theta}$  becomes the zero vector at  $\theta = \theta_0$ . Consequently,  $\left\| \frac{\partial\boldsymbol{z}_{t^{*}k - 1}}{\partial\theta}\right\|$  and so  $\left\| \frac{\partial\mathcal{L}_t}{\partial\theta}\right\|$  vanishes at  $\theta = \theta_0$ . Now it can be shown that at  $\theta = \theta_0$  the loss gradient goes to zero for every  $\boldsymbol {z}_1\in \mathcal{B}_{\Gamma_k}$  (the proof is similar to the last part of the proof of Theorem 1).
A.2.3 Proof of corollary 1
Proof. For  $M = 2$ , let  $h_1 \neq 0$ . Then the DFB curves of the fixed point  $\Gamma_1$  coincide with the BCB curves of the 2-cycle  $\mathcal{O}_{\mathcal{R}\mathcal{L}}$  of the form
$$
\mathcal {F} _ {1} = \xi_ {\mathcal {R L}} ^ {1} = \left\{\left(h _ {1}, h _ {2}, a _ {1 1}, a _ {2 2}\right) \mid 1 + a _ {1 1} + a _ {2 2} + a _ {1 1} a _ {2 2} = 0 \right\}, \tag {77}
$$
$$
\mathcal {F} _ {2} = \xi_ {\mathcal {R L}} ^ {2} = \left\{\left(h _ {1}, h _ {2}, a _ {1 1}, w _ {1 1}, w _ {2 1}, a _ {2 2}\right) \mid 1 + a _ {1 1} + w _ {1 1} + a _ {2 2} + \left(a _ {1 1} + w _ {1 1}\right) a _ {2 2} = 0 \right\}. \tag {78}
$$
For  $M > 2$ , assume that  $\Gamma_1 = \{z_1^*\}$  is a fixed point of the system, i.e.
$$
\boldsymbol {z} _ {1} ^ {*} = \left(\boldsymbol {I} - \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)}\right) ^ {- 1} \boldsymbol {h} = \frac {\operatorname {a d j} \left(\boldsymbol {I} - \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)}\right)}{P _ {\boldsymbol {I} - \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)}} (1)} \boldsymbol {h}, \tag {79}
$$
where  $P_{I - W_{\Omega(t_1^*)}}(1)$  is the characteristic polynomial of  $I - W_{\Omega(t_1^*)}$  at 1. Let us denote the first row of the adjoint matrix of  $I - W_{\Omega(t_1^*)}$  by  $adj(I - W_{\Omega(t_1^*)})_1$ . If  $adj(I - W_{\Omega(t_1^*)})_1 h \neq 0$ , then we can analogously demonstrate that the DFB curves of the fixed point align with the BCB curves of the 2-cycles. This implies that, in accordance with Theorem 2, DFBs of fixed points will also lead to vanishing gradients in the loss function.
A.2.4 Convergence of SCYFI
To ensure that SCYFI almost surely converges, we can simply choose the number of random initializations (i.e.,  $N_{out}$  in algorithm 1) large enough such that every linear subregion will have been sampled with probability almost 1. More precisely, drawing uniformly from the  $2^{M}$  different  $D_{\Omega}$ -matrices (linear subregions) for initialization, the probability that a particular subregion has not been drawn after  $r$  repetitions is  $p = \left(1 - \frac{1}{2^M}\right)^r$ . Hence, in order to ensure that all  $2^{M}$  subregions have been visited with probability  $1 - \epsilon$  we need  $r \geq \left\lceil \frac{\ln(\epsilon)}{\ln(1 - \frac{1}{2^M})} \right\rceil$  iterations. Choosing  $N_{out} = r$ , we can thus ensure that SCYFI was initialized in each subregion with probability almost 1, and thus, in the limit, will have probed all subregions for dynamical objects. This argument extends to  $k$ -cycles by replacing  $2^{M}$  by  $2^{kM}$  above (strictly, a more precise bound for  $k \geq 2$  is given by  $2^{M(k-1)} \times (2^{M} - 1) = 2^{Mk} - 2^{M(k-1)}$ , due to the fact that the PLRNN (2) is a linear map in each subregion and, hence, cannot have any  $k$ -cycles with all periodic points in only one subregion).
A.2.5 Proof of theorem 3
Proof. We examine the convergence and scaling behavior of SCYFI for fixed points. A similar argument applies to  $k$ -cycles where  $k > 1$ .
Let  $z_1^*$  be a fixed point of the system, i.e.
$$
\boldsymbol {z} _ {1} ^ {*} = \left(\boldsymbol {I} - \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)}\right) ^ {- 1} \boldsymbol {h}. \tag {80}
$$
$z_{1}^{*}$  is a true fixed point iff
$$
\left(d _ {m} \left(t _ {1} ^ {*}\right) - a\right) \cdot z _ {m, t _ {1} ^ {*}} > 0, \quad \forall m \in \{1, 2, \dots , M \}, \tag {81}
$$
where  $D_{\Omega (t_1^*)} = diag(d_1(t_1^*),d_2(t_1^*),\dots ,d_M(t_1^*))$  and  $0 < a < 1$  is a positive real constant.
For examining SCYFI's efficiency, here we focus on two scenarios that impose specific constraints on parameters  $\theta$ ; other cases remain to be investigated.
Case (I): Let  $\pmb{R}$  be a randomly generated matrix with uniformly distributed entries in the interval [0, 1), and  $h \neq 0$  be a random vector with all its components being non-negative. For an arbitrary  $\epsilon > 0$ , we set
$$
\boldsymbol {A} = \frac {1}{2 + \| \boldsymbol {R} \| + \epsilon} \operatorname {d i a g} (\boldsymbol {R}),
$$
$$
\boldsymbol {W} = \frac {1}{2 + \| \boldsymbol {R} \| + \epsilon} \left(\boldsymbol {R} - \operatorname {d i a g} (\boldsymbol {R})\right). \tag {82}
$$
$$
\left\| \boldsymbol {A} \right\| = \frac {\left\| \operatorname {d i a g} (\boldsymbol {R}) \right\|}{2 + \left\| \boldsymbol {R} \right\| + \epsilon} <   \frac {1}{2 + \left\| \boldsymbol {R} \right\| + \epsilon},
$$
$$
\| \boldsymbol {W} \| = \frac {\| \boldsymbol {R} - \operatorname {d i a g} (\boldsymbol {R}) \|}{2 + \| \boldsymbol {R} \| + \epsilon} \leq \frac {\| \boldsymbol {R} \| + \| \operatorname {d i a g} (\boldsymbol {R}) \|}{2 + \| \boldsymbol {R} \| + \epsilon} <   \frac {1 + \| \boldsymbol {R} \|}{2 + \| \boldsymbol {R} \| + \epsilon}, \tag {83}
$$
and so  $\| A\| +\| W\| <  1$  . Therefore
$$
\forall t \left\| \boldsymbol {W} _ {\Omega (t)} \right\| = \left\| \boldsymbol {A} + \boldsymbol {W} \boldsymbol {D} _ {\Omega (t)} \right\| \leq \| A \| + \| W \| \| \boldsymbol {D} _ {\Omega (t)} \| \leq \| A \| + \| W \| <   1, \tag {84}
$$
$$
\forall t \rho \left(\boldsymbol {W} _ {\Omega (t)}\right) \leq \left\| \boldsymbol {W} _ {\Omega (t)} \right\| <   1. \tag {85}
$$
In this case, for any  $n\in \mathbb{N}$  , we also have
$$
\left\| \prod_ {i = 1} ^ {n} \boldsymbol {W} _ {\Omega (t _ {i})} \right\| \leq \prod_ {i = 1} ^ {n} \| \boldsymbol {W} _ {\Omega (t _ {i})} \| \leq \left(\| A \| + \| W \|\right) ^ {n} <   1. \tag {86}
$$
This ensures the stability of all fixed points and  $k$ -cycles of the system.
According to (85), we have
$$
\left(I - \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)}\right) ^ {- 1} = \sum_ {n = 0} ^ {\infty} \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)} ^ {n} = \boldsymbol {I} + \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)} + \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)} ^ {2} + \dots . \tag {87}
$$
Hence, all the elements of  $\left(I - W_{\Omega(t_1^*)}\right)^{-1}$  are positive, and so  $z_{m,t_1^*} > 0$  for every  $t_1^*$ . This implies that all true and virtual fixed points exist within a singular sub-region. Additionally, only one fixed point is true, while all the other fixed points are virtual.
Case (II): Let  $\pmb{h} = (h_1, h_2, \dots, h_M)^\top$  be a random vector with all  $h_m$  uniformly distributed in  $(0, 1]$  and
$$
\beta_ {m i n} = \min  \left\{h _ {m}: h _ {m} \in \boldsymbol {h}, 1 \leq m \leq M \right\} > 0, \quad 0 <   \beta_ {m i n} \leq 1,
$$
$$
\beta_ {m a x} = \max  \left\{h _ {m}: h _ {m} \in \mathbf {h}, 1 \leq m \leq M \right\} > 0, \quad 0 <   \beta_ {m a x} \leq 1. \tag {88}
$$
Assume further that  $R_{1}$  is a randomly generated matrix with uniformly distributed entries in the interval  $(-1,0]$ , and for  $M \geq 2$
$$
\boldsymbol {W} = \frac {\beta_ {\text {m i n}}}{M + \| R _ {1} \| + \epsilon} \left(\boldsymbol {R} _ {1} - \operatorname {d i a g} \left(\boldsymbol {R} _ {1}\right)\right). \tag {89}
$$
$$
\alpha_ {m a x} = \max  \left\{\left| w _ {i j} \right|: w _ {i j} \in \boldsymbol {W} \right\}, \quad 0 \leq \alpha_ {m a x} <   \frac {\beta_ {m i n}}{M + \| R \| + \epsilon} \tag {90}
$$
and  $S \subset \{1,2,\dots ,M\} = I$  such that  $K = 2^{M - card(S)} \ll 2^{M}$ . Suppose that  $R_{2} = diag(r_{1},\dots ,r_{M})$  is a randomly chosen diagonal matrix with  $r_m$  uniformly distributed in  $(-1,1)$  for  $m \in I \setminus S$ , and the other elements  $(m \in S)$  uniformly distributed in  $(r^* - 1,0)$  where  $r^* = \frac{(M - 1)\alpha_{max}\beta_{max}}{\beta_{min}}$ . Since
$$
0 \leq \frac {(M - 1) \alpha_ {m a x} \beta_ {m a x}}{\beta_ {m i n}} <   \frac {(M - 1) \beta_ {m a x}}{M + \| R \| + \epsilon} \leq \frac {(M - 1)}{M + \| R \| + \epsilon} <   \frac {(M - 1)}{M} <   1, \tag {91}
$$
so  $-1\leq r^{*} - 1 <   0$
$$
\boldsymbol {A} = \frac {1}{2 + \| R _ {1} \| + \epsilon} \boldsymbol {R} _ {2}, \tag {92}
$$
$$
\left\| \boldsymbol {A} \right\| = \frac {\left\| \boldsymbol {R} _ {2} \right\|}{2 + \left\| \boldsymbol {R} _ {1} \right\| + \epsilon} <   \frac {1}{2 + \left\| \boldsymbol {R} _ {1} \right\| + \epsilon},
$$
$$
\left\| \boldsymbol {W} \right\| = \frac {\beta_ {\text {m i n}} \left\| \boldsymbol {R} _ {1} - \operatorname {d i a g} \left(\boldsymbol {R} _ {1}\right) \right\|}{M + \left\| \boldsymbol {R} _ {1} \right\| + \epsilon} \leq \frac {\left\| \boldsymbol {R} _ {1} \right\| + \left\| \operatorname {d i a g} \left(\boldsymbol {R} _ {1}\right) \right\|}{M + \left\| \boldsymbol {R} _ {1} \right\| + \epsilon} <   \frac {1 + \left\| \boldsymbol {R} _ {1} \right\|}{2 + \left\| \boldsymbol {R} _ {1} \right\| + \epsilon}, \tag {93}
$$
which implies  $\| \mathbf{A}\| +\| \mathbf{W}\| <  1$  . We set  $\epsilon >0$  large enough to satisfy the condition
$$
\left(I - \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)}\right) ^ {- 1} = \sum_ {n = 0} ^ {\infty} \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)} ^ {n} \approx \boldsymbol {I} + \boldsymbol {W} _ {\Omega \left(t _ {1} ^ {*}\right)} \quad \forall t _ {1} ^ {*}. \tag {94}
$$
On the other hand, for any  $t$  we have
$$
\boldsymbol {W} _ {\Omega (t)} = \boldsymbol {A} + \boldsymbol {W} \boldsymbol {D} _ {\Omega (t)} = \left( \begin{array}{c c c c c} a _ {1 1} & w _ {1 2} d _ {2} (t) & w _ {1 3} d _ {3} (t) & \dots & w _ {1 M} d _ {M} (t) \\ w _ {2 1} d _ {1} (t) & a _ {2 2} & w _ {2 3} d _ {3} (t) & \dots & w _ {2 M} d _ {M} (t) \\ w _ {3 1} d _ {1} (t) & w _ {3 2} d _ {2} (t) & a _ {3 3} & \dots & w _ {3 M} d _ {M} (t) \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ w _ {M 1} d _ {1} (t) & w _ {M 2} d _ {2} (t) & w _ {M 3} d _ {3} (t) & \dots & a _ {M M} \end{array} \right). \tag {95}
$$
$$
z _ {m, t _ {1} ^ {*}} = \left(1 + a _ {m m}\right) h _ {m} + \sum_ {\substack {j = 1 \\ j \neq m}} ^ {M} w _ {m j} d _ {j} \left(t _ {1} ^ {*}\right) h _ {j} = \left(1 + a _ {m m}\right) h _ {m} - \sum_ {\substack {j = 1 \\ j \neq m}} ^ {M} \left| w _ {m j} \right| d _ {j} \left(t _ {1} ^ {*}\right) h _ {j}. \tag{96}
$$
Since for every  $t_1^*$
$$
\sum_ {\substack {j = 1 \\ j \neq m}} ^ {M} | w _ {m j} | d _ {j} \left(t _ {1} ^ {*}\right) h _ {j} \leq \sum_ {\substack {j = 1 \\ j \neq m}} ^ {M} | w _ {m j} | h _ {j}, \tag{97}
$$
$$
z _ {m, t _ {1} ^ {*}} \geq (1 + a _ {m m}) h _ {m} - \sum_ {\substack {j = 1 \\ j \neq m}} ^ {M} | w _ {m j} | h _ {j} \quad \forall m \in I. \tag{98}
$$
Moreover  $a_{ss} \in (r^* - 1, 0)$ , for every  $s \in S$ , and thus
$$
a _ {s s} + 1 > \frac {(M - 1) \alpha_ {m a x} \beta_ {m a x}}{\beta_ {m i n}} = \frac {\sum_ {j = 1} ^ {M} \alpha_ {m a x} \beta_ {m a x}}{\beta_ {m i n}} \geq \frac {\sum_ {j = 1} ^ {M} \left| w _ {s j} \right| h _ {j}}{h _ {s}}. \tag {99}
$$
Therefore, due to (98) and (99),  $z_{s,t_1^*} > 0$  for every  $t_1^*$  and  $s \in S$ . This means that all true and virtual fixed points only exist within a relatively small number of sub-regions, denoted as  $K = 2^{M - \text{card}(S)} \ll 2^M$ . Given our specific initialization of  $\theta$ , in both cases (I) and (II) there is a set of  $K$  different sub-regions, each associated with a unique  $D_{\Omega(t)}$  matrix. We refer to the entire set of these matrices as
$$
\mathcal {D} _ {K} = \left\{\boldsymbol {D} _ {1}, \dots , \boldsymbol {D} _ {K} \right\}. \tag {100}
$$
SCYFI, by its definition, only moves within the sub-regions that have virtual and true fixed points, continuing until it discovers a true fixed point (or gets stuck in a virtual cycle). Thus, it can iterate between  $J \leq K$  sub-regions
$$
\mathcal {D} _ {J} = \left\{\boldsymbol {D} _ {1}, \dots , \boldsymbol {D} _ {J} \right\} \subseteq \mathcal {D} _ {K}, \tag {101}
$$
or within the set of virtual fixed points
$$
\mathcal {Z} _ {L} = \left\{\boldsymbol {z} _ {1}, \dots , \boldsymbol {z} _ {L} \right\}. \tag {102}
$$
In case (I), there is only one true fixed point. Since all virtual fixed points are located within the same single sub-region, SCYFI's initialization will naturally position it within the correct linear region, requiring no more than 1 iteration. Hence, it needs at most 2 iterations to find the true fixed point. Consequently, SCYFI's scaling is constant.
For case (II), if we suppose that SCYFI follows the virtual/true fixed point structure of the underlying system in these  $K$  sub-regions, the necessity for the probability of discovering the fixed point to be close to 1, specifically  $1 - \epsilon$ , is to have
$$
N \geq \lceil \frac {\ln (\epsilon)}{\ln \left(1 - \frac {1}{2 ^ {M - c a r d (S)}}\right)} \rceil = \lceil \frac {\ln (\epsilon)}{\ln \left(1 - \frac {1}{K}\right)} \rceil , \tag {103}
$$
iterations. Since  $1 \leq \text{card}(S) \leq M - 1$ , so  $K \geq 2$  and  $\ln(1 - \frac{1}{K}) \approx \frac{-1}{K}$ . For  $\epsilon^* \geq \epsilon$ , let  $N = \lceil \frac{\ln(\epsilon^*)}{\ln(1 - \frac{1}{K})} \rceil \geq \lceil \frac{\ln(\epsilon)}{\ln(1 - \frac{1}{K})} \rceil$ , then
$$
N = \left\lceil \frac {\ln \left(\epsilon^ {*}\right)}{\ln \left(1 - \frac {1}{K}\right)} \right\rceil \leq \frac {\ln \left(\epsilon^ {*}\right)}{\ln \left(1 - \frac {1}{K}\right)} \approx \ln \left(\frac {1}{\epsilon^ {*}}\right) K := c K, \tag {104}
$$
which implies the number of iterations is bounded from above. If, for every  $M$ , we choose  $K$  small enough, then the upper bound will stay within a linear growth.
A.2.6 Proof of theorem 4
In GTF [24], during training RNN latent states are replaced by a weighted sum of forward propagated states  $\boldsymbol{z}_t = F_{\boldsymbol{\theta}}(\boldsymbol{z}_{t-1})$  and data-inferred states  $\bar{\boldsymbol{z}}_t = G_{\phi}^{-1}(\boldsymbol{x}_t)$  (obtained by inversion of the decoder model  $G_{\phi}$ ):
$$
\tilde {z} _ {t} := (1 - \alpha) z _ {t} + \alpha \bar {z} _ {t}, \tag {105}
$$
where  $0 \leq \alpha \leq 1$  is the GTF parameter (usually adaptively regulated in training, see [24]). This leads to the following factorization of Jacobians in PLRNN (2) training:
$$
\boldsymbol {J} _ {t} ^ {G T F} = \frac {\partial \boldsymbol {z} _ {t}}{\partial \boldsymbol {z} _ {t - 1}} = \frac {\partial \boldsymbol {z} _ {t}}{\partial \tilde {\boldsymbol {z}} _ {t - 1}} \frac {\partial \tilde {\boldsymbol {z}} _ {t - 1}}{\partial \boldsymbol {z} _ {t - 1}} = \frac {\partial \boldsymbol {F} _ {\boldsymbol {\theta}} (\tilde {\boldsymbol {z}} _ {t - 1})}{\partial \tilde {\boldsymbol {z}} _ {t - 1}} \frac {\partial \tilde {\boldsymbol {z}} _ {t - 1}}{\partial \boldsymbol {z} _ {t - 1}} = (1 - \alpha) \tilde {\boldsymbol {J}} _ {t} = (1 - \alpha) \boldsymbol {W} _ {\Omega (t)}. (1 0 6)
$$
Proof. (i) Since  $\| \mathbf{A} \| + \| \mathbf{W} \| \leq 1$ , we have
$$
\forall t \left\| \boldsymbol {W} _ {\Omega (t)} \right\| = \left\| \boldsymbol {A} + \boldsymbol {W} \boldsymbol {D} _ {\Omega (t)} \right\| \leq \| A \| + \| W \| \left\| \boldsymbol {D} _ {\Omega (t)} \right\| \leq \| A \| + \| W \| \leq 1, \tag {107}
$$
$$
\forall t \rho \left(\boldsymbol {W} _ {\Omega (t)}\right) \leq \left\| \boldsymbol {W} _ {\Omega (t)} \right\| \leq 1, \tag {108}
$$
where  $\rho$  denotes the spectral radius of a matrix. In this case, for any  $n\in \mathbb{N}$ , we also have
$$
\rho \left(\prod_ {i = 1} ^ {n} \boldsymbol {W} _ {\Omega (t _ {i})}\right) \leq \left\| \prod_ {i = 1} ^ {n} \boldsymbol {W} _ {\Omega (t _ {i})} \right\| \leq \prod_ {i = 1} ^ {n} \left\| \boldsymbol {W} _ {\Omega (t _ {i})} \right\| \leq \left(\| A \| + \| W \|\right) ^ {n} \leq 1. \tag {109}
$$
Now, for any  $0 < \alpha < 1$ , the product of Jacobians under GTF is
$$
\prod_ {i = 1} ^ {n} \boldsymbol {J} _ {t _ {i}} ^ {G T F} = (1 - \alpha) ^ {n} \prod_ {i = 1} ^ {n} \tilde {\boldsymbol {J}} _ {t _ {i}} = (1 - \alpha) ^ {n} \prod_ {i = 1} ^ {n} \boldsymbol {W} _ {\Omega \left(t _ {i}\right)}, \tag {110}
$$
$$
\begin{array}{l} \rho \left(\prod_ {i = 1} ^ {n} \boldsymbol {J} _ {t _ {i}} ^ {G T F}\right) = \rho \left((1 - \alpha) ^ {n} \prod_ {i = 1} ^ {n} \boldsymbol {W} _ {\Omega (t _ {i})}\right) = (1 - \alpha) ^ {n} \rho \left(\prod_ {i = 1} ^ {n} \boldsymbol {W} _ {\Omega (t _ {i})}\right) \\ \leq (1 - \alpha) ^ {n} <   1. \tag {111} \\ \end{array}
$$
Hence  $\rho\left(\prod_{i=1}^{n} J_{t_i}^{GTF}\right) < 1$  which implies for any  $n \in \mathbb{N}$  and  $0 < \alpha < 1$ , the product  $\prod_{i=1}^{n} J_{t_i}^{GTF}$  has no eigenvalue equal to 1 and so no DTB can occur (see definition of DTB in sect. 3).
(ii) Let  $\| \mathbf{A}\| +\| \mathbf{W}\| = r > 1$  , then for any  $n\in \mathbb{N}$  we have
$$
\rho \left(\prod_ {i = 1} ^ {n} \tilde {\boldsymbol {J}} _ {t _ {i}}\right) \leq r ^ {n}, \tag {112}
$$
$$
\rho \left(\prod_ {i = 1} ^ {n} \boldsymbol {J} _ {t _ {i}} ^ {\text {G T F}}\right) = (1 - \alpha) ^ {n} \rho \left(\prod_ {i = 1} ^ {n} \boldsymbol {W} _ {\Omega \left(t _ {i}\right)}\right) \leq [ (1 - \alpha) r ] ^ {n}. \tag {113}
$$
Since  $0 < 1 - \frac{1}{r} < 1$ , inserting  $1 - \frac{1}{r} < \alpha = \alpha^{*} < 1$  into the r.h.s. of (113) again gives  $\rho \left( \prod_{i=1}^{n} J_{t_i}^{GTF} \right) < 1$  for any  $n \in \mathbb{N}$ , implying that no DTB can occur.
A.3 Additional results
Figure S6: A) Analytically calculated stability regions for a 2-cycle  $S_{\mathcal{R}\mathcal{L}}$  (red), and a fixed point  $S_{\mathcal{R}}$  (green) for the  $1d$  skew tent map, as defined in the figure, in the parameter plane given by  $(a_r,a_l)$ . B) Bifurcation diagram along the cross section indicated by the gray line in A, showing a BCB and DFB occurring simultaneously at  $a_r = -1$  and a DTB occurring at  $a_r = 1$ .
Figure S7: Results from SCYFI (blue) versus analytical results (orange) for the 2-cycle in Fig. 1  $(a_{11} + w_{11} = -2)$ . Eigenvalues (left) and location in state space (right) for one of the cyclic points. This confirms that fixed point locations and eigenvalues computed in closed-form and via SCYFI exactly agree, as they should.
A.3.1 Scaling analysis