SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Analysis of the weights. We found that, consistent with the literature [17], the changes in weights resulting from the initial training were much larger than those of motor perturbation learning. PARAFAC on the full tensor of updates  $\mathbf{W} - W_0\otimes \mathbf{1}$  captured the weight tensor in 3 components (not shown), whose columns and rows were essentially those of performing SVD on  $W^{*}$ . Fitting additional PARAFAC components revealed that residual variability in the updates of pretraining was larger than the motor perturbation learning variability. Furthermore, unlike SVD, there is no guarantee that the components of fitting a rank  $k + 1$  PARAFAC model will be related to those of fitting a rank  $k$  model. Nevertheless, motor perturbation learning had a significant change on the eigenvalues and activity of the RNN (Fig. 5f,g)
To uncover an upper bound on rank of the weight tensor, we split the analysis into the pre-training and motor perturbation learning. We first performed SVD on the change in weights matrix  $W^{*} - W_{0}$ , where  $W^{*}$  are the weights of the network post-training, but pre-motor perturbation learning (Fig. 5b.). We found that the changes in weights were well captured by a rank-3 decomposition. Then, we performed PARAFAC on the change in weights tensor  $\mathbf{W} - W^{*} \otimes \mathbf{1}$  of the motor perturbation learning. We found that this change of weight tensor was well approximated by a tensor rank 2 decomposition. The combination of these results upper-bounds the tensor rank of the overall changes in weights to 5. Finally, we compared the subspace spanned by the columns of the SVD and PARAFAC decomposition by projecting the weight tensor  $\mathbf{W} - W^{*} \otimes \mathbf{1}$  on the first three column and row singular vectors of  $W^{*} - W_{0}$  and found approximately a remaining 0.2 unexplained variance, suggesting that the columns of  $\mathbf{W}_{0} - W^{*}$  and  $W^{*}$  were not orthogonal, but did not span the same subspace. Combined, these results suggest that the numerical tensor rank is at least 4 and at most 5, consistent with the results uncovered by ltrRNN from the RNN activity (Fig. 5e).
D Task-trained RNN models of additional neuroscience tasks
To investigate the generality of the low tensor rank framework, we additionally trained RNNs on three non-motor tasks commonly used in neuroscience.
Sensory evidence accumulation task [54] (Fig. 6i). The RNN receives a one-dimensional Ornstein-Uhlenbeck (OU) process input whose expected steady-state is either positive or negative. The target output is either  $+1$  or  $-1$  if the mean of the input is positive or negative, respectively.
Contextual decision making task [10] (Fig. 6ii). The RNN receives a three-dimensional input. The first two inputs are independent OU processes as in the previous task. The third input is binary and constant, and determines which of the two stochastic inputs must be integrated. The target output is +1 (or -1) if the mean of the input of the OU process indicated by the contextual input is positive (or negative).
Working memory task [14] (Fig. 6iii). The RNN receives a 1-dimensional input consisting of two stimuli of different amplitudes separated by a delay in time. The target output is the identity of the stimulus (1 or 2) which had larger amplitude.
Note that, in contrast to the motor adaptation task, these tasks do not contain any baseline period. Therefore, as we could not compare weight updates to a pre-trained solution, we simply analyzed the tensor rank of the weights starting from random initialization. That is, we first trained an RNN on each of these tasks using gradient descent, then used PARAFAC on  $\mathbf{W} - W^{(0)}\otimes \mathbf{1}$  to determine the tensor rank of the resulting neuron  $\times$  neuron  $\times$  iteration tensor of weights over training. In each of these tasks we found that the variance explained indeed saturated at low tensor ranks (at  $R = 1,4,$  and 3; Fig. 6e).
Supplementary Figure 6: Validation of low-tensor-rank framework on RNNs trained on additional tasks. Each row indicates a different task. Row i. Sensory evidence accumulation. a. Example inputs. Each line represents a different trial; red/blue indicate target identity. b. Example outputs after training (for the inputs shown in a). c. Loss curve over iterations. d. Eigenspectrum of  $W^{(k)} - W^{(0)}$ . Color gradient indicates training iterations  $k$  (with  $k = 1$  in yellow). e. After training, we perform PARAFAC on  $\mathbf{W} - W^{(0)} \otimes \mathbf{1}$  to estimate the rank of learning dynamics. Row ii. Contextual decision making. Row iii. Working memory task.
E Mathematical results
E.1 Adjoint derivation
In this section, we present a derivation of the adjoint mainly following [38]. Then, we derive the adjoint of recurrent neural networks.
E.1.1 State adjoint
Consider the dynamical system  $\dot{\mathbf{x}} = f(\mathbf{x},\pmb {\theta})\in \mathbb{R}^n$  where  $\pmb {\theta}\in \mathbb{R}^k$  is a set of parameters. Furthermore, let the functional  $L:\mathbb{R}^n\to \mathbb{R}$  such that  $L(\mathbf{x}(T))$  is our loss. First, define the state adjoint,
$$
\mathbf {a} (t) = \frac {d L (\mathbf {x} (T))}{d \mathbf {x} (t)}. \tag {E.1.1}
$$
Notice that since  $\mathbf{x}(t + \Delta t)$  is a function of  $\mathbf{x}(t)$ ,
$$
\mathbf {a} (t) = \frac {d L (\mathbf {x} (T))}{d \mathbf {x} (t + \Delta t)} \frac {d \mathbf {x} (t + \Delta t)}{d \mathbf {x} (t)} = \mathbf {a} (t + \Delta t) \frac {d \mathbf {x} (t + \Delta t)}{d \mathbf {x} (t)}. \tag {E.1.2}
$$
By Taylor expanding  $\mathbf{x}(t + \Delta t) = \mathbf{x}(t) + \Delta tf(\mathbf{x}(t)) + O(\Delta t^2)$ , we get,
$$
\mathbf {a} (t) = \mathbf {a} (t + \Delta t) \left(I + \frac {d}{d \mathbf {x} (t)} f (\mathbf {x} (t)) + O \left(\Delta t ^ {2}\right)\right) \tag {E.1.3}
$$
where  $I$  is the identity matrix. By rearranging,
$$
\frac {\mathbf {a} (t + \Delta t) - \mathbf {a} (t)}{\Delta t} = \mathbf {a} (t + \Delta t) \frac {d f (\mathbf {x} (t))}{d \mathbf {x} (t)} + O (\Delta t). \tag {E.1.4}
$$
Taking the limit as  $\Delta t\to 0$
$$
\frac {d \mathbf {a} (t)}{d t} = \mathbf {a} (t) \frac {d f (\mathbf {x} (t))}{d \mathbf {x} (t)}. \tag {E.1.5}
$$
We now have the dynamics of the adjoint; all that remains is that we find an initial (or rather terminal) condition. For this, notice that
$$
\mathbf {a} (T) = \frac {d L (\mathbf {x} (T))}{d \mathbf {x} (T)} \tag {E.1.6}
$$
is the usual gradient of  $L$  w.r.t. to its argument.
E.1.2 Parameter adjoint and gradient
In the above we derived the state adjoint  $\mathbf{a}$ . However, for our purposes we also require the parameter adjoint  $dL(\mathbf{x}(T)) / d\theta$ . For this, it suffices to augment the original dynamical system with its parameters<sup>5</sup>  $\dot{\mathbf{z}} = [f(\mathbf{x},\pmb {\theta}),\mathbf{0}]$  and initial (later terminal) condition  $\mathbf{z}(0) = [\mathbf{x}(0),\pmb {\theta}]$ . Defining the loss  $\bar{L} (\mathbf{z}(T)) = L(\mathbf{x}(T))$ , the adjoint of this augmented system is,
$$
\mathbf {a} _ {\mathbf {z}} (t) = \frac {d \bar {L} (\mathbf {z} (T))}{d \mathbf {z} (t)} = \left[ \frac {d \bar {L} (\mathbf {z} (T))}{d \mathbf {x} (t)}, \frac {d \bar {L} (\mathbf {z} (T))}{d \boldsymbol {\theta}} \right] = \left[ \frac {d L (\mathbf {x} (T))}{d \mathbf {x} (t)}, \frac {d L (\mathbf {x} (T))}{d \boldsymbol {\theta}} \right] = \left[ \mathbf {a} _ {\mathbf {x}} (t), \frac {d L (\mathbf {x} (T))}{d \boldsymbol {\theta}} \right], \tag {E.1.7}
$$
which contains the desired term  $dL(\mathbf{x}(T)) / d\pmb{\theta}$ . It now remains to describe the dynamics of the augmented system. By the same argument as for the state adjoint (E.1.5),
$$
\frac {d \mathbf {a} _ {\mathbf {z}} (t)}{d t} = \mathbf {a} _ {\mathbf {z}} (t) \frac {d \dot {\mathbf {z}}}{d \mathbf {z}}. \tag {E.1.8}
$$
By (E.1.7) and by unconcatenating  $\mathbf{z}$ ,
$$
= \left[ \mathbf {a} _ {\mathbf {x}} (t), \frac {d L (\mathbf {x} (T))}{d \theta} \right] \left[ \begin{array}{c c} \frac {d f (\mathbf {x} (t))}{d \mathbf {x} (t)} & \frac {d \mathbf {x} (t)}{d \theta} \\ \frac {d \mathbf {0}}{d \mathbf {x} (t)} & \frac {d \mathbf {0}}{d \theta} \end{array} \right] \tag {E.1.9}
$$
$$
= \left[ \mathbf {a} (t) \frac {d f (\mathbf {x} (t))}{\mathbf {x} (t)}, \mathbf {a} (t) \frac {d f (\mathbf {x} (t))}{d \theta} \right]. \tag {E.1.10}
$$
Here  $[\cdot ,\cdot ]$  denotes row concatenation.
Hence the following dynamical system can be evaluated,
$$
\frac {d}{d t} \left[ \mathbf {x} (t), \mathbf {a} (t), \frac {d L (\mathbf {x} (T))}{d \boldsymbol {\theta}} \right] = \left[ f (\mathbf {x} (t), \boldsymbol {\theta}), \mathbf {a} (t) \frac {d f (\mathbf {x} (t))}{\mathbf {x} (t)}, \mathbf {a} (t) \frac {d f (\mathbf {x} (t))}{d \boldsymbol {\theta}} \right], \tag {E.1.11}
$$
with terminal condition
$$
\left[ \mathbf {x} (T), \mathbf {a} (T), \frac {d L (\mathbf {x} (T))}{d \boldsymbol {\theta}} \right] = \left[ \mathbf {x} (T), \frac {d L (\mathbf {x} (T))}{d \mathbf {x} (T)}, \mathbf {0} \right]. \tag {E.1.12}
$$
Notice that to obtain the terminal condition, since it depends on  $\mathbf{x}(T)$ , the original dynamical system must be evaluated forward once.
E.1.3 RNN adjoint
Let  $\dot{\mathbf{x}} = f(\mathbf{x},W) = W\phi (\mathbf{x}) - \mathbf{x} + B\mathbf{u}(t)$  and  $L(\mathbf{x}(T)) = ||D\phi (\mathbf{x}(T)) - \mathbf{y}||^2$  for  $y\in \mathbb{R}^d$ . Furthermore, as it will be convenient, let  $W = \sum_{i}^{R}\alpha_{i}\otimes \beta_{i}$ . We will derive one by one the terms needed to characterize the parameter adjoint. First the Jacobian is
$$
\frac {d f (\mathbf {x} (t) , \boldsymbol {\theta})}{d \mathbf {x} (t)} = \sum_ {i} ^ {R} \boldsymbol {\alpha} _ {i} \otimes \left(\boldsymbol {\beta} _ {i} \odot \phi^ {\prime} (\mathbf {x} (t))\right) - I \tag {E.1.13}
$$
where  $\odot$  denotes the element-wise product,  $I$  the identity matrix, and  $\phi'$  the derivative of  $\phi$ . Next,
$$
\frac {d f (\mathbf {x} (t) , \boldsymbol {\theta}) _ {i}}{d W _ {j k}} = \left\{ \begin{array}{l l} 0 & i \neq j \\ \phi (\mathbf {x}) _ {k} & i = j \end{array} \right., \tag {E.1.14}
$$
$$
\frac {d f (\mathbf {x} (t) , \boldsymbol {\theta})}{d W} = I \otimes \phi (\mathbf {x}). \tag {E.1.15}
$$
Finally the terminal condition,
$$
\frac {d L (\mathbf {x} (T))}{d \mathbf {x} (T) _ {i}} = \frac {d}{d x _ {i}} \sum_ {j} ^ {d} \left(D _ {j} \cdot \phi (\mathbf {x}) - y _ {j}\right) ^ {2} = \sum_ {j} ^ {d} \left(D _ {j} \cdot \phi (\mathbf {x}) - y _ {j}\right) \left(D _ {i j} \phi^ {\prime} (\mathbf {x}) _ {i}\right) \tag {E.1.16}
$$
$$
\frac {d L (\mathbf {x} (T))}{d \mathbf {x} (T)} = \sum_ {j} ^ {d} \left(D _ {j} \cdot \phi (\mathbf {x}) - y _ {j}\right) \left(D _ {j} \odot \phi^ {\prime} (\mathbf {x})\right) = \phi^ {\prime} (\mathbf {x}) \odot \sum_ {j} ^ {d} \left(D _ {j} \cdot \phi (\mathbf {x}) - y _ {j}\right) D _ {j} \tag {E.1.17}
$$
Or more explicitly,  $\dot{\mathbf{a}}_{\mathbf{z}} =$
$$
\left\{ \begin{array}{l l} \dot {\mathbf {a}} _ {\mathbf {x}} = \left(\sum_ {i} ^ {R} \boldsymbol {\alpha} _ {i} \otimes \boldsymbol {\beta} _ {i} \odot \phi^ {\prime} (\mathbf {x})\right) ^ {T} \mathbf {a} _ {\mathbf {x}} - \mathbf {a} _ {\mathbf {x}}, & \mathbf {a} _ {\mathbf {x}} (T) = \phi^ {\prime} (\mathbf {x} (T)) \odot \sum_ {j} ^ {d} \left(D _ {j} \cdot \phi (\mathbf {x} (T)) - y _ {j}\right) D _ {j} \\ \dot {\mathbf {a}} _ {W} = \mathbf {a} _ {\mathbf {x}} \otimes \phi (\mathbf {x}), & \mathbf {a} _ {W} (T) = 0 \end{array} \right. \tag {E.1.18}
$$
E.2 Rank of the gradient
E.2.1 The gradient as a composition of operators
In this section, we prove Theorems 1. In order to derive bounds on the singular values of  $\nabla_W L = \mathbf{a}_W(0)$ , we shall now consider  $\mathbf{a}_{\mathbf{x}}$  and  $\phi(\mathbf{x})$  as linear operators with integration. Namely,
$$
\mathbf {a} _ {\mathbf {x}} \mathbf {y} := \int_ {0} ^ {T} \mathbf {a} _ {\mathbf {x}} (t) y (t) d t \tag {E.2.1}
$$
where  $\mathbf{y} \in \mathcal{H}$  for  $\mathcal{H}$  some suitable Hilbert space, such as  $L^2$  for our case.
More formally, let  $\mathbf{a}_{\mathbf{x}}, \phi(\mathbf{x}) \in \mathcal{B}_{0,0}$ , where  $\mathcal{B}_{0,0}$  is the Banach space of i) compact ii) bounded operators from  $\mathcal{H}$  to  $\mathbb{R}^n$ , such that  $\mathbf{a}_{\mathbf{x}}, \phi(\mathbf{x}): \mathcal{H} \to \mathbb{R}^n$ . In particular, we note that compactness follows from the image of  $\mathbf{a}_{\mathbf{x}}$ , that is  $\mathbf{a}_{\mathbf{x}}(\mathcal{H})$ , to be a vector subspace of  $\mathbb{R}^n$  and therefore be of finite rank. By the same argument,  $\phi(\mathbf{x})$  is compact, and therefore so is  $\phi(\mathbf{x})^*$  by Schauder's theorem [55], where  $^*$  is adjunction. Furthermore, notice that  $\phi(\mathbf{x}), \mathbf{a}_{\mathbf{x}}$  are solutions of dynamical systems with differentiable right hand side and therefore bounded (in  $\mathbb{R}^n$ ) if they are evaluated for finite time, and therefore bounded when seen as operators. The following result can now be applied:
Lemma 2 ([55]). Let  $T \in \mathcal{B}_{0,0}$ , then  $T$  admits a singular value decomposition. Furthermore, this singular value decomposition is of finite rank.
We can now prove the main theorem.
Proof of Theorem 1. Notice that the composition of the two operators is:  $\mathbf{a}_{\mathbf{x}}\circ \phi (\mathbf{x})^{*} = \nabla_{W}L$  Furthermore, the singular values of  $\nabla_W L$  are,
$$
\sigma_ {i} ^ {\nabla_ {W} L} = \min  _ { \begin{array}{c} U \subseteq \mathbb {R} ^ {n}, \\ \dim U = \\ n - i - 1 \end{array} } \max  _ {| | \mathbf {y} | | = 1} | | \nabla_ {W} L \mathbf {y} | | = \min  _ { \begin{array}{c} U \subseteq \mathbb {R} ^ {n}, \\ \dim U = \\ n - i - 1 \end{array} } \max  _ {| | \mathbf {y} | | = 1} | | \mathbf {a} _ {\mathbf {x}} \phi (\mathbf {x}) ^ {*} \mathbf {y} | | \tag {E.2.2}
$$
which can be bounded as,
$$
\begin{array}{l} \min  _ { \begin{array}{l} U \subseteq \mathbb {R} ^ {n}, \\ \dim U = \\ n - i - 1 \end{array} } \max  _ {\mathbf {y} \in U,} \| \mathbf {a} _ {\mathbf {x}} \phi (\mathbf {x}) ^ {*} \mathbf {y} \| \leq \min  _ { \begin{array}{l} U \subseteq \mathbb {R} ^ {n}, \\ \dim U = \\ n - i - 1 \end{array} } \max  _ {\mathbf {y} \in U,} \| \mathbf {a} _ {\mathbf {x}} \| | | \phi (\mathbf {x}) ^ {*} \mathbf {y} | | (E.2.3) \\ = \sigma_{1}^{\mathbf{a}_{\mathbf{x}}}\min_{\substack{U\subseteq \mathbb{R}^{n},\\ \dim U = ||\mathbf{y}|| = 1}}\max_{\mathbf{y}\in U,}||\phi (\mathbf{x})^{*}\mathbf{y}|| (E.2.4) \\ \end{array}
$$
(E.2.5)
$$
\sigma_ {i} ^ {\nabla_ {W} L} \leq \sigma_ {1} ^ {\mathbf {a} _ {\mathbf {x}}} \sigma_ {i} ^ {\phi (\mathbf {x}) ^ {*}} \tag {E.2.6}
$$
Now notice that, for any operator, akin to the matrix case,  $T_{1}, T_{2}$ , the adjoint of their composition is  $(T_{1}T_{2})^{*} = T_{2}^{*}T_{1}^{*}$ . Furthermore,
$$
\begin{array}{l} \sigma_ {i} ^ {\left(\nabla_ {W} L\right) ^ {T}} = \min  _ { \begin{array}{l} U \subseteq \mathbb {R} ^ {n}, \\ \dim U = | | \mathbf {y} | | = 1 \\ n - i - 1 \end{array} } \max  _ {\mathbf {y} \in U,} \left| \left| (\mathbf {a} _ {\mathbf {x}} \phi (\mathbf {x}) ^ {*}) ^ {*} \mathbf {y} \right| \right| = \min  _ { \begin{array}{l} U \subseteq \mathbb {R} ^ {n}, \\ \dim U = | | \mathbf {y} | | = 1 \\ n - i - 1 \end{array} } \max  _ {\mathbf {y} \in U,} \left| \left| \phi (\mathbf {x}) \mathbf {a} _ {\mathbf {x}} ^ {*} \mathbf {y} \right| \right| (E.2.7) \\ \leq \sigma_ {1} ^ {\phi (\mathbf {x})} \min  _ { \begin{array}{c} U \subseteq \mathbb {R} ^ {n}, \\ \dim U = \\ n - i - 1 \end{array} } \max  _ {\mathbf {y} \in U,} \| \mathbf {a} _ {\mathbf {x}} ^ {*} \mathbf {y} \| (E.2.8) \\ = \sigma_ {1} ^ {\phi (\mathbf {x})} \sigma_ {i} ^ {\mathbf {a} _ {\mathbf {x}} ^ {*}} (E.2.9) \\ \end{array}
$$
Noticing that  $\sigma_{i}^{\mathbf{a}_{\mathbf{x}}^{*}} = \sigma_{i}^{\mathbf{a}_{\mathbf{x}}}$  and  $\sigma_{i}^{\nabla_{W}L} = \sigma_{i}^{(\nabla_{W}L)^{T}}$
$$
\sigma_ {i} ^ {\nabla_ {W} L} \leq \sigma_ {1} ^ {\phi (\mathbf {x})} \sigma_ {i} ^ {\mathbf {a} _ {\mathbf {x}}} \tag {E.2.10}
$$
Combining E.2.6 and E.2.10 we obtain the sought upper bound of Theorem 1,
$$
\sigma_ {i} ^ {\nabla_ {W} L} \leq \min  \left\{\sigma_ {1} ^ {\mathbf {a} _ {\mathbf {x}}} \sigma_ {i} ^ {\phi (\mathbf {x})}, \sigma_ {1} ^ {\phi (\mathbf {x})} \sigma_ {i} ^ {\mathbf {a} _ {\mathbf {x}}} \right\} \tag {E.2.11}
$$
Similar steps can be used to derive the lower bound of Theorem 1, using instead the identity  $\sigma_{n}^{T_{1}}||T_{2}\mathbf{y}||\leq ||T_{1}T_{2}\mathbf{y}||$  where  $T_{1},T_{2}\in \mathcal{B}_{0,0}$  and  $\sigma_n^{T_1}$  is the smallest non-zero singular value of  $T_{1}$ .
We however note that, unlike the upper bound, the lower bound we provide does not have any numerical use, as the smallest singular value of the adjoint or of the firing rate is practically 0 (for example, well bellow machine precision).
We further point out that a more explicit characterization of the singular values of the gradient can be obtained. Let  $U_{i}^{\mathbf{a}_{\mathbf{x}}}, U_{i}^{\phi(\mathbf{x})}$  be the right singular vectors of respectively  $\mathbf{a}_{\mathbf{x}}$  and  $\phi(\mathbf{x})$ , so that  $V_{i}^{\mathbf{a}_{\mathbf{x}}} (t), V_{i}^{\phi(\mathbf{x})} (t)$  are the left singular vectors and  $\sigma_{i}^{\mathbf{a}_{\mathbf{x}}}, \sigma_{j}^{\phi(\mathbf{x})}$  the singular values.
$$
\nabla_ {W} L = \sum_ {i, j = 1} ^ {n} \left(V _ {i} ^ {\mathbf {a} _ {\mathbf {x}}} \otimes V _ {j} ^ {\phi (\mathbf {x})}\right) \sigma_ {i} ^ {\mathbf {a} _ {\mathbf {x}}} \sigma_ {j} ^ {\phi (\mathbf {x})} \int_ {0} ^ {T} U _ {i} ^ {\mathbf {a} _ {\mathbf {x}}} (t) U _ {j} ^ {\phi (\mathbf {x})} (t) d t \tag {E.2.12}
$$
The characterization of the rank of the gradient has thus shifted to the firing rate and adjoint spaces. Furthermore, the integral is just the inner product between their right singular vectors and therefore of magnitude bounded by 1. Hence, for the dynamics of the weights to be large in a given direction in weight space  $V_{i}^{\mathbf{a}_{\mathbf{x}}} \otimes V_{j}^{\phi(\mathbf{x})}$ , all three of: the singular values of the firing rate, the state adjoint, as well as their cofluctuation in time, must not be small.
E.2.2 Weight gradient for time discretizations
Finally, we mention that our detour through functional analysis was for the sake of mathematical rigour, and that in practical applications, the RNN and its adjoint are evaluated at discrete time steps  $0, \Delta t\ldots, q\Delta t$ . In that case, the gradient can be estimated as a simple matrix-matrix product. Let  $A = [\mathbf{a}_{\mathbf{x}}(0), \dots, \mathbf{a}_{\mathbf{x}}(q\Delta t)]$  and  $B = [\phi(\mathbf{x})(0), \dots, \phi(\mathbf{x})(q\Delta t)]$ . Then  $\nabla_{W}L = \Delta tAB^{T}$ , and the bounds  $E.2.6$  and  $E.2.10$  follow from classic matrix-matrix product bounds [56]. In particular, given that  $\phi(\mathbf{x})$  and  $\mathbf{a}_{\mathbf{x}}$  are smooth, without loss of generality,
$$
\sigma_ {i} ^ {\nabla W L} = \lim  _ {\min  _ {j} \left(t _ {j + 1} - t _ {j}\right)\rightarrow 0} \sigma_ {i} ^ {A B ^ {T}} \tag {E.2.13}
$$
This simple matrix-matrix product opens up the possibility of fast RNN adjoint implementations as, unlike computing  $\mathbf{a}_{\mathbf{x}}\frac{df}{dW}$  in general adjoint solvers, which requires  $O(qn^3)$  time and  $O(n^{2})$  memory complexity for an  $n$ -dimensional RNN and  $q$  time steps, here the time complexity drops to  $O(qn^{2})$ .
E.2.3 Rank of linear RNNs
In this section we prove Theorem 2.
Proof of claim 1. If  $W = \sum_{i}^{R}\alpha_{i}\otimes \beta_{i}$ , then by Lemma 1, the dynamics of the adjoint are constrained to the span of the rows of  $W$ , namely,  $\dot{\mathbf{a}}_{\mathbf{x}}\in \operatorname {span}\{\beta_i\}$ . Therefore,  $\mathbf{a}_{\mathbf{x}}\in \operatorname {span}\{\beta_i\} \cup \{\mathbf{a}_{\mathbf{x}}(T)\}$  which is a at most  $R + 1$  dimensional subspace. If  $\mathbf{a}_{\mathbf{x}}$  is constrained to a at most  $R + 1$  dimensional subspace, then  $\dot{\mathbf{a}}_W = \mathbf{a}_{\mathbf{x}}\otimes \mathbf{x}$  is also constrained to a at most  $R + 1$  dimensional subspace. Since  $\mathbf{a}_W(T) = 0$ ,  $\mathbf{a}_W$  is constrained to the same subspace as its dynamics, and in particular, rank  $\mathbf{a}_W(0) = \mathrm{rank}\nabla_W L^{(0)}\leq R + 1$ .
Proof of claim 2. Suppose  $W^{(k)} = \sum_{i}^{2R + m + d}\pmb{\alpha}_{i}^{(k)}\otimes \pmb{\beta}_{j}^{(k)}$ , where  $\pmb{\alpha}_{i}^{(k)},\pmb{\beta}_{i}^{(k)}\in \operatorname {span}\{\pmb{\alpha}_{i}^{(0)}\} \cup \{\pmb{\beta}_{i}^{(0)}\} \cup \{B_{i}\} \cup \{D_{i}\} := V$ . In particular, notice that  $V$  is only dependent on the initial weight. Then by a similar argument as above,  $\mathbf{a}_{\mathbf{x}}^{(k)}\in V$ , which implies  $\mathbf{a}_W(0)^{(k)} = \nabla_W L^{(k)}\in V$ . Therefore  $W^{(k + 1)} = W^{(k)} + \gamma \nabla_W L^{(k)} = \sum_i^{2R + m + d}\pmb{\alpha}_i^{(k + 1)}\otimes \pmb{\beta}_j^{(k + 1)}$  with  $\pmb{\alpha}_{i}^{(k + 1)}\in V$ . That is rank  $W^{(k + 1)}\leq 2R + m + d$ .
Proof of claim 3. Mutatis mutandis  $\beta^{(k + 1)}\in V$  , that is  $\mathbf{x}\in V$  . Therefore,  $\mathbf{W}^{(k)} = \sum_{ij}^{2R + m + d}c_{ij}^{(k)}\pmb{\alpha}_i\otimes \pmb{\alpha}_j$  for some  $c_{ij}^{(k)}$  s, where  $\pmb{\alpha}_{i}\in V$  . Or equivalently,  $\mathbf{W} = \sum_{ij}^{2R + m + d}\pmb{\alpha}_i\otimes$ $\pmb{\alpha}_{j}\otimes \mathbf{c}_{ij}$  . That is, rank  $\mathbf{W}\leq (2R + m + d)^2$
E.3 Extensions of our results
Loss integrated over time. Commonly, the loss considered might be integrated,
$$
\mathcal {L} (T) := \int_ {0} ^ {T} L (\mathbf {x} (t), \mathbf {y} (t)) d t \tag {E.3.1}
$$
The parameter adjoint is dependent only linearly on the state adjoint, we may therefore integrate the state adjoint for all initial conditions.
$$
\mathcal {L} (T) = \left(\int_ {T} ^ {0} \mathbf {a} _ {\mathbf {x}} (t) + \int_ {t} ^ {0} \dot {\mathbf {a}} _ {\mathbf {x}} \left(t ^ {\prime}\right) d t ^ {\prime}\right) d t \tag {E.3.2}
$$
The term inside the first integral is just the solution of time-varying autonomous LDS, therefore,
$$
\int_ {T} ^ {0} \mathbf {a} _ {\mathbf {x}} (t) d t = \int_ {T} ^ {0} \Phi (0, t) \frac {d L (\mathbf {x} (t) , \mathbf {y} (t))}{d \mathbf {x} (t)} d t \tag {E.3.3}
$$
Where  $\Phi$  is the linear dynamical system state transition matrix [57]. But notice that this is the solution to the controlled LDS,
$$
\dot {\mathbf {a}} _ {\mathbf {x}} = \mathbf {a} _ {\mathbf {x}} \frac {d f (\mathbf {x} (t))}{d \mathbf {x} (t)} + \frac {d L (\mathbf {x} (t) , \mathbf {y} (t))}{d \mathbf {x} (t)}, \quad \dot {\mathbf {a}} _ {\mathbf {x}} (T) = 0 \tag {E.3.4}
$$
In the specific case of an RNN,
$$
\dot {\mathbf {a}} _ {\mathbf {x}} = \left(\sum_ {i} ^ {R} \alpha_ {i} \otimes \beta_ {i} \odot \phi^ {\prime} (\mathbf {x})\right) ^ {T} \mathbf {a} _ {\mathbf {x}} - \mathbf {a} _ {\mathbf {x}} + \phi^ {\prime} (\mathbf {x} (t)) \odot \sum_ {j} ^ {d} \left(D _ {j} \cdot \phi (\mathbf {x} (t)) - \mathbf {y} _ {j}\right) D _ {j}, \quad \dot {\mathbf {a}} _ {\mathbf {x}} (T) = 0 \tag {E.3.5}
$$
Therefore, Theorem 1 remains unchanged for a loss integrated over time. For Theorem 2, Claims 2-3 remain unchanged, while Claim 1 becomes  $\mathrm{rank}\nabla_W L^{(0)}\leq R + \min \{d,m\}$ .
Gradient with respect to other parameters. So far we have focused on the gradient of the weights of the RNN. As we have seen, the space over which the state adjoint  $\mathbf{a}_{\mathbf{x}}(t)$  evolves as well as the trajectories of the system itself  $\mathbf{x}(t)$  determine the space over which the gradient evolves. But those are respectively dependent on the decoder  $D$  and encoder  $B$  of the system. If all parameters  $D, B, W$  of the system are optimized simultaneously, as is most often the case, we may wonder how our bounds hold.
First, for  $B$ , notice that by a similar argument as for  $W$ ,  $\frac{df(\mathbf{x}, B\mathbf{u}(t))}{dB} = I \otimes \mathbf{u}(t)$ , so that  $\dot{\mathbf{a}}_B = \mathbf{a}_{\mathbf{x}} \otimes \mathbf{u}(t)$ . Therefore, following essentially the same derivation as that of Theorem 1, the following bound can be derived,