SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sigma_ {i} ^ {B} \leq \min  \left\{\sigma_ {1} ^ {\mathbf {a} _ {\mathbf {x}}} \sigma_ {i} ^ {\mathbf {u}}, \sigma_ {1} ^ {\mathbf {u}} \sigma_ {i} ^ {\mathbf {a} _ {\mathbf {x}}} \right\}. \tag {E.3.6}
$$
By a similar argument as for the derivation of Theorem 2,  $B_i^{(k)} \in \operatorname{span}\{\alpha_i\} \cup \{\beta_i\} \cup \{B_i\} \cup \{D_i\}$ . Second, for  $D$ , since  $\frac{df}{dD} = 0$ , that is  $\dot{\mathbf{a}}_D = 0$
$$
\mathbf {a} _ {D} (T) = \mathbf {a} _ {D} (0) = \nabla_ {D} L = \frac {d L (T)}{d D}. \tag {E.3.7}
$$
In other words, the gradient of the loss with respect to the decoder weights have zero dynamics.
The gradient of a functional w.r.t. a given parameter is independent of the gradient of that functional with respect to another parameter if these two parameters do not depend on one another. Therefore these results hold regardless of which combination of  $W$ ,  $B$  or  $D$  is optimized.
Batched updates. Most often, the weights are updated in batches. That is  $\Delta W^{(k)} = Q^{-1}\sum_q^Q\nabla_WL^{(k,q)}$  where  $q$  is the index over the batched dimension. Since the column and row spaces of  $\nabla_W L^{(k,q)}$  remain unchanged, Theorem 2 2-3. remain unchanged, while 1. becomes rank  $\Delta W^{(0)}\le R + \min \{d,m\}$ . For Theorem 1, the common singular value identities [56]  $\sigma_{i + j - 1}(A + B)\leq \sigma_i(A) + \sigma_j(B)$  and  $\sigma_{i}(cA) = c\sigma_{i}(A)$  for  $c\in \mathbb{R}_0^+$  can be used. Then,  $\sigma_{\sum i_q - Q + 1}(\Delta W^{(k)})\leq Q^{-1}\sum_q^Q\sigma_{i_q}(\nabla_WL^{(k,q)})$ .
Momentum-based optimizers. Momentum-based optimizers such as Adam [53] are commonly used to train RNNs on behavioural tasks. Here we focus on the first moment, a similar derivation can be undertaken for higher moments. In that case, a momentum variable is introduced, which is updated as  $M^{(k + 1)} = \beta M^{(k)} + (1 - \beta)\nabla_{W}L^{(k)}$ , where  $\beta$  determines the speed of the exponential decay. The weights are then updated as  $W^{(k + 1)} = W^{(k)} - \alpha M^{(k + 1)}$ , where  $\alpha$  is the learning rate. Which implies,  $\Delta W^{(k)} = W^{(k + 1)} - W^{(k)} = -\alpha \sum_{j}^{k}(1 - \beta)^{j}\nabla_{W}L^{(j)}$ . Using the same identities are for batched updates,  $\sigma_{\sum i_q - k + 1}(\Delta W^{(k)})\leq \sum_j^k (1 - \beta)^j\sigma_{i_q}(\nabla_WL^{(j)})$ .
E.4 Numerical simulations
Similarly to [14], we illustrate our mathematical results on random RNNs. Since constant inputs are one dimensional ( $B\mathbf{u} = \sum B_i u_i$ ), we instead use time-varying inputs parameterized with LDS:
$$
\dot {\mathbf {u}} = M \mathbf {u} - \mathbf {u} \quad \mathbf {u} (0) = \mathbf {u} _ {0} \tag {E.4.1}
$$
where  $\mathbf{u}(t) \in \mathbb{R}^m$ ,  $M_{ij} \sim \mathcal{N}(0,1\sqrt{m})$ ,  $\mathbf{u}_0 \sim \mathcal{N}(0,\mathbf{1}/2)$ . The target outputs are set as  $\mathbf{y} \in \mathbb{R}^l$ ,  $\mathbf{y}_i \sim \mathcal{U}(-1,1)$ . The loss is defined as,
$$
L (W) = \left| \left| D \phi (\mathbf {x}) (T) - \mathbf {y} \right| \right| ^ {2} \tag {E.4.2}
$$
where  $\mathbf{x}$  is the solution of an RNN as considered thus far.
In Sup. Fig. 7 we show the effect of varying  $\phi$ , the rank  $R$  of the initial weights as well as the standard deviation (or strength)  $g$  of the initial weights such that  $W_{ij}^{(0)} \sim \mathcal{N}(0, g^2)$ .
Supplementary Figure 7: Singular values of adjoints and gradients. a. Loss over training. Inset: activation function. b. Activity during the last trial (black). Inset: activity of two example neurons over training. c. Singular values of the firing rate and adjoint. Inset: additional singular values. d. Singular values of the gradient, weights, and the bound we derive. e. Variance explained per rank of the tensor decomposition of weight tensor  $\mathbf{W} - W^{(0)}\otimes \mathbf{1}$ . We additionally plot the variance explained over performing matrix decomposition on all possible unfoldings of the weight tensor. Architectures. Unless noted, the initial weight std was  $g = 1.5$ , the input and output dimensions  $m = d = 2$ . i. Rectified tanh. We found that non-smooth activation functions seem to give the slowest decay of the singular values of the firing rate and adjoint. ii-iii. Softplus and tanh, which are the most common activation functions in neuroscience, have exponentially decaying firing rate and adjoint singular values, and therefore (by Theorem 1) exponentially decaying gradient singular values. iv. Low rank ( $R = 3$ ) linear RNN. As per Theorem 2, the first gradient step is of rank  $R + 1 = 4$ . v. Tanh in a chaotic regime ( $g = 2.1$ ). Despite being in a chaotic regime, the firing rate, the adjoint, and therefore (by Theorem 1) the gradient have exponentially decaying singular values.