SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Smoothness constraint over trials. We further constrain the initial covariance in trial of the trial factors by parameterizing them as  $\mathbf{c}_r = (L + \sigma I)\overline{\mathbf{c}}_r$  where  $LL^T = A$  is the Cholesky decomposition of the smooth covariance matrix  $A$ , and  $\overline{\mathbf{c}}_r$  is initialized as  $\overline{\mathbf{c}}_r \sim \mathcal{N}(\mathbf{0}, I)$ . In particular, we use a rational quadratic kernel  $s^2(1 + (2l)^{-1}(k_i - k_j)^2)^{-1}$ , where  $k_i$  is the  $i$ th trial index. This is equivalent to performing a 3-mode matrix-tensor product on the weight tensor itself  $(L + \sigma I) \times_3 \mathbf{W}$  so that its entries over trials are linear combinations of smooth functions up to observation noise.
This parameterization is similar to that of Gaussian process regression, except no probabilistic objective is set (kernel regression). In particular, given that  $(L + \sigma I)$  is invertible, any possible  $\mathbf{c}_r$  can in theory be obtained upon optimization of the  $\bar{\mathbf{c}}_r$ . By additionally setting a regularization on  $\bar{\mathbf{c}}_r$ , we penalize the smoothness of  $\mathbf{c}_r$  as non-smooth solutions have diverging  $\bar{\mathbf{c}}_r$ . In this way, we bias the optimization process towards smoother  $\mathbf{c}_r$ 's. As illustrated in Fig. 4b, the cross-validated loss remains similar to the non-smooth, full-rank, case.
A key advantage of having smooth trial factors is that missing trials can be easily accounted for. Indeed, in most large-scale neural datasets, such as the one explored in the present work, potentially many trials may have been discarded, for example due to behavioural performance being outside the range set by the experimentalist. The assumption being made here is that the across-trial covariance is preserved when such trials occur. That is, we assume that a failure of the animal to perform a given trial does not imply that a trial wasn't informative, or that learning did not occur.
Condition-wise inputs. We parameterize the condition-wise inputs to the network with neural ordinary differential equations [38], i.e. as a dynamical system whose right hand side is parameterized by a deep neural network (DNN)
$$
\dot {\mathbf {v}} ^ {(i)} = D N N (\mathbf {v} ^ {(i)}) \quad \mathbf {u} ^ {(i)} (t) = \phi (D \mathbf {v} ^ {(i)} (t)),
$$
where  $\mathbf{v}^{(i)}(t) \in \mathbb{R}^{N_{\mathrm{NODE}}}$  and  $D \in \mathbb{R}^{N_{\mathrm{input}} \times N_{\mathrm{NODE}}}$ . Throughout this work, we used a fully-connected 3-layer DNN with layers of size 150 and ReLU nonlinearities. This provides inputs whose dynamics are considerably less constrained than those generated by low-rank RNNs. Thus, we do not make any assumption on the activity of upstream brain regions which drive the activity of the brain region being recorded. While here we chose to model the inputs using autonomous neural ODEs, one might imagine feeding the neural ODE with behavioural or task covariates to relax the condition specificity assumption, or fitting residual inputs to capture trial-by-trial variability arising from unmeasured variables [21]. This could account for some of the variability that can neither be explained by the task condition, nor by changes in the dynamics due to learning.
Loss. In sections 4 and 5 we focus on optimizing for the mean squared error (MSE)
$$
L (\mathbf {a}, \mathbf {b}, \mathbf {c}, B, M) = \sum_ {k} ^ {K} \sum_ {q} ^ {T} \left| \left| M \phi \left(\mathbf {x} ^ {(k)} \left(t _ {q}\right)\right) - \mathbf {y} ^ {(k)} \left(t _ {q}\right) \right| \right| ^ {2}, \tag {A.1.1}
$$
where  $\mathbf{y}$  are firing rate estimates data and  $M\in \mathbb{R}^{N\times N_{\mathrm{data}}}$ . In section B we present supplementary results on optimizing the Poisson log-likelihood with respect to spike data,
$$
L (\mathbf {a}, \mathbf {b}, \mathbf {c}, B, M) = - \sum_ {k} ^ {K} \sum_ {q} ^ {T} \log \left(\operatorname {P o i s s o n} \left(\mathbf {y} ^ {(k)} \left(t _ {q}\right) \mid M \phi \left(\mathbf {x} ^ {(k)} \left(t _ {q}\right)\right)\right)\right) \tag {A.1.2}
$$
where  $\mathbf{y}$  are binned spike data.
Pseudocode. The following pseudocode summarizes the steps of fitting an ltrRNN to neural data. For the sake of clarity, we present the simplest case: an autonomous ltrRNN with fixed initial state. The neural ODE-driven case can be achieved by coupling  $f$  with a deep neural net. Additionally, to allow for the initial state (w.r.t. the data) of the RNN to change over conditions and trials, the evaluation of the dynamical system can be done over  $\{-T_0\Delta t,\dots,0,\dots,T\Delta t\}$ , where  $\mathbf{x}_0$  is now the state at  $-T_0\Delta t$ , and the fit to data is still done on the non-negative time states.
Algorithm 1 Low tensor rank recurrent neural network fit to data
inputs:  $\mathbf{Y}\in \mathbb{R}^{N_{\mathrm{data}}\times T\times K}$  Neural data tensor of shape neuron  $\times$  time  $\times$  trial  $\mathbf{t} = \{0,\Delta t,\dots,T\Delta t\}$  Time points of evaluation of the dynamical system   
initializations: randomly initialize  $\mathbf{a}_r,\mathbf{b}_r\in \mathbb{R}^N$  for  $r = 1,\ldots ,R$  randomly initialize  $\overline{\mathbf{c}}_r\in \mathbb{R}^N$  for  $r = 1,\ldots ,R$  randomly initialize  $\mathbf{q}\in \mathbb{R}^R,M\in \mathbb{R}^{N_{\mathrm{data}}\times N}$    
definitions:  $f_{W}(\mathbf{x}) = W\phi (\mathbf{x}) - \mathbf{x}$  RNN dynamics for a given weight matrix  $A\in \mathbb{R}^{K\times K}$  such that  $A_{ij}\gets \kappa (i,j)$  Trial covariance matrix defined by a smooth kernel  $\kappa$ $L\leftarrow$  Cholesky  $(A + \sigma^2 I)$  Cholesky decomposition   
while the loss  $l$  hasn't converged do.   
 $l\gets 0$ $\mathbf{c}_r\gets L\bar{\mathbf{c}}$ $\mathbf{W}\gets \sum_{r = 1}^{R}\mathbf{a}_{r}\otimes \mathbf{b}_{r}\otimes \mathbf{c}_{r}$ ${\bf x}_0\gets \sum_{r = 1}^{R}q_{r}{\bf a}_{r}$  The initial state   
for  $k = 1,\dots,K$  do For all trials (parallelizable)   
 $X^{(k)}\gets \mathrm{ODESolve}(f_{W^{(k)}},\mathbf{x}_0,\mathbf{t})$  Matrix of activity of the RNN during trial  $k$ $l\gets l + \underbrace{|M\phi(X^{(k)}) - Y^{(k)}|_2^2}_{\mathrm{Fit~to~data}} + \underbrace{\alpha||X^{(k)}|_2^2}_{\mathrm{Regularization}}$    
end for   
SGDUpdate(a_r,br,cr,M,q) E.g. ar- ar- n d X dX dW da r   
end while
Code availability. The ltrRNN implementation can be found at https://github.com/ arthur-pe/LltrRNN.
A.2 The dynamics of ltrRNNs
Rich changes of dynamics through oblique columns. Unlike for a matrix rank decomposition, a tensor rank decomposition can be unique even for non-orthogonal factors. A sufficient condition for uniqueness is that  $r_{\mathbf{a}} + r_{\mathbf{b}} + r_{\mathbf{c}} \leq R - 2$  where, without loss of generality,  $r_{\mathbf{a}}$  denotes the maximum number of linearly dependent columns of a [50]. In other words, fitting the changes of dynamics over trials as opposed to a single low rank RNN shared over all trials gives additional information regarding the columns and rows. In the case where the  $\mathbf{a}_j$ 's are not orthogonal, non-trivial qualitative changes in the vector fields can occur. Since for any given trial  $k$ , an ltrRNN is simply a low rank RNN [15],
$$
\dot {\mathbf {x}} ^ {(k)} = \sum_ {j} \mathbf {a} _ {j} \left(c _ {j} ^ {(k)} \mathbf {b} _ {j} \cdot \phi \left(\mathbf {x} ^ {(k)}\right)\right) - \mathbf {x} ^ {(k)} + B \mathbf {u} ^ {(k)} (t) \tag {A.2.1}
$$
Table 1: Hyperparameters of the ltrRNN models. Bold indicates values specific to section 4 and 5, * indicates cross-validated hyperparameters, other hyperparameters were tuned by hand.
Neural data (S4)	Simulated data (S5)
LtrRNN
R	5*	5*
n	200*	200*
φ	tanh	tanh
Smoothness
l	50	15
s	0.1	0.1
σ	0.1	0.1
Neural ODE
Layers	150 × 150 × 150	N/A
φ	ReLU	N/A
Regularization
α	0.01	0
Cross-validation
Train blocks	1 × 10 × 20	1 × 10 × 10
Test blocks	1 × 5 × 10	1 × 5 × 5
neuron × time × trial
Table 2: Approximate training time of ltrRNNs. Here on the neural data of section 4. The variables which impact the most training time are the trial and neuron dimensions of the ltrRNN (not the rank or data time steps), as well as the neural ODE architecture. Hardware : desktop with an RTX 3090 Nvidia GPU and i7-12700K Intel CPU. * indicates the one used in section 4.
Neurons
200	400
Trials	370	12min	22min
740	16min*	40min
so that the dynamics of  $\mathbf{x}^{(k)}$  are constrained to span  $\{\mathbf{a}_j\} \cup \{B_j\}$ . Unlike low rank RNNs, ltrRNN are not necessarily invariant under changes of bases of  $\mathbf{a}_j$ 's. Nevertheless, we can introduce an orthonormal basis  $\{\tilde{\mathbf{a}}_j\}$  so that,
$$
W ^ {(k)} \phi (\mathbf {x} ^ {(k)}) = \sum_ {i} ^ {R} \tilde {\mathbf {a}} _ {i} \sum_ {j} \left(\tilde {\mathbf {a}} _ {i} \cdot \mathbf {a} _ {j}\right) \left(\mathbf {b} _ {j} \cdot \phi (\mathbf {x})\right) c _ {j} ^ {(k)} \tag {A.2.2}
$$
In particular, notice that the dynamics along all  $\tilde{\mathbf{a}}_i$  could potentially be affected by varying  $c_{j}^{(k)}$ .
Conversely, the  $\mathbf{a}_i$ 's and  $\mathbf{b}_i$ 's being respectively orthonormal — e.g. as in a singular value decomposition — is not a sufficient condition for uniqueness of the tensor rank decomposition [51]. Nevertheless, constraining the  $\mathbf{a}_i$  to be orthogonal and ensuring that the Kruskal constraint is satisfied, the vector fields are then orthogonal. In that case, varying  $c_i^{(k)}$  for some  $i$  corresponds to rescaling the vector field along  $\mathbf{a}_i$ . Nevertheless, as is illustrated below with a single component, the leak term allows the system to display typical properties of linear and nonlinear dynamical systems.
Bifurcation in a tensor rank-one RNN. A classical example of bifurcation in a two-neuron system is that of the pitchfork supercritical bifurcation. Here, we show that it is essentially a tensor rank one RNN. We also illustrate how the corresponding linear RNN bifurcates. Let  $\mathbf{a} = \mathbf{b} = [-1/\sqrt{2}, 1/\sqrt{2}]^T$  so that,
$$
W ^ {(k)} = \frac {c}{2} \left[ \begin{array}{l l} 1 & - 1 \\ - 1 & 1 \end{array} \right] \xrightarrow {\dot {\mathbf {x}} = 0} \left[ \begin{array}{l} \mathbf {x} _ {1} \\ \mathbf {x} _ {2} \end{array} \right] = \frac {c}{2} \left[ \begin{array}{l} \phi (\mathbf {x} _ {1}) - \phi (\mathbf {x} _ {2}) \\ - \phi (\mathbf {x} _ {1}) + \phi (\mathbf {x} _ {2}) \end{array} \right] \tag {A.2.3}
$$
That is  $\mathbf{x}_1 = -\mathbf{x}_2$ . We consider two cases,  $\phi = \tanh$  and  $\phi = \mathrm{id}$ , both odd functions. Introducing the two in the previous equation,  $-\phi (\mathbf{x}_2) = \phi (\mathbf{x}_1)$ . So that  $\mathbf{x}_i = c\phi (\mathbf{x}_i)$ . Now considering each activation function separately,
- tanh: i)  $c > 1$  has two solutions. ii)  $c \leq 1$  one solution (the origin). At  $c = 1$  the origin is a non-hyperbolic fixed point.
- id: i)  $c = 1$  for any  $\mathbf{x}_1$ . The non-zero eigenvalue of the Jacobian of the system is negative, therefore it is a line attractor. ii)  $\mathbf{x}_1 = 0$  for any  $c$ . Then the Jacobian of the system at the origin has both positive and negative eigenvalues for  $c > 1$  and only negative for  $c \leq 1$ .
Supplementary Figure 1: Bifurcation in a tensor rank one RNN.
More general changes in vector fields. At the cost of increasing the rank, an ltrRNN can possibly transition between two arbitrary vector fields of ranks  $R_{1}$  and  $R_{2}$ . For example, let  $\mathbf{c}_j = [1,0.9,\dots,0]$  for  $j \in \{1,\dots,R_1\}$  and  $\mathbf{c}_j = [0,\dots,0.9,1]$  for  $j \in \{R_1 + 1,\dots,R_1 + R_2\}$ . There might however be multiple bifurcations between the first and last trial. More generally, given that any tensor has a (possibly high rank) tensor decomposition, any weight tensor can in theory be captured by an ltrRNN. This further illustrates the relevance of the result we found that ltrRNN of very low ranks fit data as well as full tensor RNNs.
A.3 Relationship between rank decomposition and eigendecomposition
In this section, we investigate the relationship between an arbitrary low rank decomposition of a matrix and the eigendecomposition. We assume that, on a given trial  $k$ ,  $W^{(k)}$  has the rank decomposition
$$
W ^ {(k)} = \mathbf {a} \operatorname {d i a g} \left(\mathbf {c} ^ {(k)}\right) \mathbf {b} ^ {T} = \sum_ {i = 1} ^ {R ^ {\prime}} c _ {i} ^ {(k)} \mathbf {a} _ {i} \mathbf {b} _ {i} ^ {T}. \tag {A.3.1}
$$
where  $\mathbf{a} \in \mathbb{C}^{N \times R'}$ ,  $\mathbf{b} \in \mathbb{C}^{R' \times N}$ ,  $\mathbf{c}^{(k)} \in \mathbb{R}^{R'}$ , and  $\mathbf{a}_i, \mathbf{b}_i$  are the rows/columns of  $\mathbf{a}$ ,  $\mathbf{b}$  respectively. One such low rank decomposition is the eigendecomposition<sup>34</sup>
$$
W ^ {(k)} = V \Lambda V ^ {- 1} = V \operatorname {d i a g} (\boldsymbol {\lambda}) V ^ {- 1} = \sum_ {i = 1} ^ {R} \lambda_ {i} \mathbf {v} _ {i} \tilde {\mathbf {v}} _ {i} ^ {T}.
$$
By the rank-nullity theorem,  $R$  is the rank of  $W^{(k)}$ , so that  $R' \geq R$ , with  $R = R'$  when Equation (A.3.1) is a minimal rank decomposition. Equating the two decompositions gives a general expression for the eigenvalues:
$$
\Lambda = V ^ {- 1} \mathbf {a} \operatorname {d i a g} (\mathbf {c}) \mathbf {b} ^ {T} V \Longrightarrow \lambda_ {i} = (V ^ {- 1} \mathbf {a} \operatorname {d i a g} (\mathbf {c}) \mathbf {b} ^ {T} V) _ {i i} = \sum_ {j} c _ {j} (\tilde {\mathbf {v}} _ {i} \cdot \mathbf {a} _ {j}) (\mathbf {v} _ {i} \cdot \mathbf {b} _ {j}).
$$
3Assuming  $W^{(k)}$  is diagonalisable. A similar argument using the Jordan normal form holds for defective matrices.
4Here, we write the left eigenvectors (rows of  $V^{-1}$ , transposed into column vectors) as  $\tilde{\mathbf{v}}_r$ . We can assume without loss of generality that the right eigenvectors  $\mathbf{v}_i$  are normalised to unit length, in which case the orthonormality of left and right eigenvectors gives  $\tilde{\mathbf{v}}_i \cdot \mathbf{v}_j = \delta_{ij} = \| \tilde{\mathbf{v}}_i \| \| \mathbf{v}_j \| \cos (\theta_{ij}^{\tilde{\mathbf{v}},\mathbf{v}}) \Rightarrow \| \tilde{\mathbf{v}}_i \| = 1 / \cos (\theta_{ii}^{\tilde{\mathbf{v}},\mathbf{v}})$ , where  $\theta_{ii}^{\tilde{\mathbf{v}},\mathbf{v}}$  is the angle between the  $i$ th left and right eigenvector.
If  $\lambda_i, \mathbf{a}, \mathbf{b}, \mathbf{c}^{(k)} \in \mathbb{R}$ , this gives rise to the result stated in the main text:
$$
\begin{array}{l} \lambda_ {i} = \sum_ {j} c _ {j} \left(\| \tilde {\mathbf {v}} _ {i} \| \| \mathbf {a} _ {j} \| \cos \theta_ {i j} ^ {\tilde {\mathbf {v}}, \mathbf {a}}\right) \left(\| \mathbf {v} _ {i} \| \| \mathbf {b} _ {j} \| \cos \theta_ {i j} ^ {\mathbf {v}, \mathbf {b}}\right) (A.3.2) \\ = \sum_ {j} c _ {j} \left(\frac {\| \mathbf {a} _ {j} \|}{\cos \left(\theta_ {i i} ^ {\tilde {\mathbf {v}}, \mathbf {v}}\right)} \cos \theta_ {i j} ^ {\tilde {\mathbf {v}}, \mathbf {a}}\right) \left(\| \mathbf {b} _ {j} \| \cos \theta_ {i j} ^ {\mathbf {v}, \mathbf {b}}\right) (A.3.3) \\ = \sum_ {j} c _ {j} \frac {\left\| \mathbf {a} _ {j} \right\| \left\| \mathbf {b} _ {j} \right\|}{\cos \left(\theta_ {i i} ^ {\tilde {\mathbf {v}}, \mathbf {v}}\right)} \cos \theta_ {i j} ^ {\tilde {\mathbf {v}}, \mathbf {a}} \cos \theta_ {i j} ^ {\mathbf {v}, \mathbf {b}} (A.3.4) \\ = \frac {1}{2} \sum_ {j} c _ {j} \frac {\| \mathbf {a} _ {j} \| \| \mathbf {b} _ {j} \|}{\cos \left(\theta_ {i n} ^ {\tilde {\mathbf {v}}, \mathbf {v}}\right)} \left(\cos \left(\theta_ {i j} ^ {\tilde {\mathbf {v}}, \mathbf {a}} + \theta_ {i j} ^ {\mathbf {v}, \mathbf {b}}\right) + \cos \left(\theta_ {i j} ^ {\tilde {\mathbf {v}}, \mathbf {a}} - \theta_ {i j} ^ {\mathbf {v}, \mathbf {b}}\right)\right). (A.3.5) \\ \end{array}
$$
Note that the above derivation makes no assumptions about the form of the low rank decomposition, other than that it is real-valued. Low rank decompositions commonly set  $\| \mathbf{a}_i\| = \| \mathbf{b}_i\| = 1$ , and often enforce orthogonality on the  $\mathbf{a}_i$  and/or  $\mathbf{b}_i$ , thereby introducing additional constraints on the relationship between the  $c$ 's and  $\lambda$ 's. Normal matrices have  $\cos (\theta_{ij}^{\tilde{\mathbf{v}},\mathbf{v}}) = \delta_{ij}$  and  $\tilde{\mathbf{v}}_i = \mathbf{v}_i$ , in which case further simplifications can be made.
B Motor learning
Pre-processing. Motor  $(n = 72$  for Fig. 4,  $n = 70$  for Sup. Fig. 4) and premotor  $(n = 231$  for Fig. 4,  $n = 137$  for Sup. Fig. 4) cortical neurons were used. The data were Gaussian filtered with a standard deviation of  $40~\mathrm{ms}$  (4 time bins). It was then centered by its baseline activity through subtracting neuron-wise the mean activity from around target onset to go-cue, and rescaled by dividing neuron-wise by the standard deviation of the execution period. Namely, the activity of a neuron  $\bar{\mathbf{y}}_i(t)$  was given by
$$
\bar {\mathbf {y}} _ {i} (t) = \frac {\mathbf {y} _ {i} (t) - \langle \mathbf {y} _ {i} (t) \rangle_ {t \leq 1 0 0}}{\langle (\mathbf {y} _ {i} (t) - \langle \mathbf {y} _ {i} (t) \rangle_ {t > 1 0 0}) ^ {2} \rangle_ {t > 1 0 0}} \tag {B.0.1}
$$
where  $t = 0$  is the go cue. Example of activity upon this pre-processing is given in Supp. Fig. 2.
Modeling assumptions. We assumed that motor and premotor cortex was driven into an initial state by inputs from upstream regions during the preparatory period, after which the input shuts off so that the resulting activity during the reach evolves autonomously via the recurrent dynamics dynamics from that initial state. We therefore set  $\mathbf{u}(t) = 0$  for  $t > 100$  where  $t = 0$  is the time of the go cue. Where the  $100\mathrm{ms}$  account for a sensory delay.
Cross-validation procedure. We cross-validated the optimal rank and number of neurons of the ltrRNN. Low matrix or tensor rank models can be cross-validated by holding out specific entries of the matrix or tensor for training, and then used for testing. However, neural data has temporal correlation, such that the entry of the time-by-trial-by-neuron data tensor  $T_{ijk}$  is strongly correlated with  $T_{i-1,jk}$  and  $T_{i+1,jk}$ . For example, assuming the data are continuous, a simple average of these entries will give an optimal estimate  $T_{ijk}$  in the limit of small time bins. Thus, the test set can be trivially inferred from the train set. We validated this intuition by performing the same cross-validation as section 4 but with  $1 \times 1 \times 1$  blocks, and found the test loss was similar as using the train loss over the whole dataset (Sup. Fig. 3).
To counter this effect, sets of contiguous entries  $[T_{ijk},\dots,T_{i + n,jk}]$  can be held out of training, and the interior of these blocks  $[T_{i + q,jk},\dots,T_{i + n - q,jk}]$  used for testing [30]. Here, given that we are interested in uncovering smooth changes in neural activity over slow timescales, we hold out  $n$  -by-  $m$  matrices  $\left[[T_{ijk},\dots,T_{i + n,jk}],\dots,[T_{i,j + m,k},\dots,T_{i + n,j + m,k}]\right]$  (Fig. 4b. inset).
As mentioned in Supplementary Material A, our method allows inferring the dynamics of held-out trials. We found that using the cross-validation procedure from [30] infers similar ranks as holding out entire trials (Sup. Fig. 3). We nevertheless applied this procedure for the sake of being able to compare different classes of models.
Poisson log likelihood. Another method for fitting firing-rate models to spike data is to use the negative Poisson log likelihood loss [11, 10]. We fitted a ltrRNN  $(R = 5, n = 200$  as in the MSE
loss case) with softplus activation using negative Poisson log likelihood loss. We found similar but overall noisier results (Supp. Fig. 5). This may be due to the presence of high firing rate neurons which are normalized by the preprocessing procedure but not likelihood fitting.
Supplementary Figure 2: Persistent effects of motor learning. Top: session used in the main text (Fig. 4). Bottom: session used in supplementary material (Sup. Fig. 4). a. Hand movement during the first and last 80 trials of perturbation learning and washout. The hand trajectories of some reach directions do not revert back post-washout (e.g. light green for top; dark green for bottom) b. Single neuron activity averaged within each condition. c. State at go cue  $+100\mathrm{ms}$ . Larger full color marker are median within a condition. For some reach directions, the washout tends to be more similar to the perturbed state than the baseline.
C Task-trained RNN model of motor learning
Task design. A trial is split into a preparatory period  $t \in [0, T_{go})$  and execution period  $[T_{go}, T_{end}]$ . Here we set  $T_{go} = 2$ ,  $T_{end} = 4$ . To model a ballistic reach, the RNN receives input the target information and a hold cue during the preparatory period. During the execution period it evolves
Supplementary Figure 3: Comparison of cross-validation procedures. Applied to the neural data of the session used in the main text.
Supplementary Figure 4: LtrRNN applied to an additional recording session. a. Cross-validated loss with held out blocks of size 100ms by 20 trials. b. PCA on preprocessed data. c. State of the ltrRNN at go cue  $+100\mathrm{ms}$ . f. Projection on first three  $\mathbf{a}_j$ . e. Projection of the vector field along  $\mathbf{a}_j$  (see main text). f. Eigenspectrum of  $W^{(k)}$  over trials. g. First three  $\mathbf{c}_j$ . h. Trial factors  $\mathbf{c}_j$ . i. Projection of  $\mathbf{x}^{(k)}(t)$  on the corresponding  $\mathbf{a}_j$ .
autonomously.
$$
\mathbf {d} \mathbf {x} ^ {(k)} = \left[ W \phi \left(\mathbf {x} ^ {(k)}\right) - \mathbf {x} ^ {(k)} + \mathbb {1} _ {t <   T _ {g o}} B _ {\text {t a r g e t}} \mathbf {u} _ {\text {t a r g e t}} ^ {(k)} + \mathbb {1} _ {t <   T _ {g o}} B _ {\text {h o l d}} \mathbf {u} _ {\text {h o l d}} ^ {(k)} \right] d t + \sigma \mathbf {d} \mathbf {W} \tag {C.0.1}
$$
where  $B_{target} \in \mathbb{R}^{n \times 2}$ ,  $\mathbf{u}_{target}^{(k)} = [\cos(\theta^{(k)}), \sin(\theta^{(k)})]$  is a static vector representing the position of the target,  $B_{hold} \in \mathbb{R}^{n \times 1}$ ,  $\mathbf{u}_{hold}^{(k)} = 1$  a cue indicating to hold movement, and  $\mathbf{dW}$  the infinitesimal increments of a Wiener process [52]. The dynamics of the hand are given in section 5 of the main text. The loss is taken to minimize the distance between the hand  $\mathbf{y}^{(k)}(t)$  and the target  $\mathbf{v}^{(k)}$  throughout the execution period, while keeping the hand still during the preparatory period, that is  $L(W, B_{target}, B_{hold}, D) =$
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \left(\frac {1}{T _ {g o}} \int_ {0} ^ {T _ {g o}} | | \mathbf {y} ^ {(k)} (t) | | ^ {2} d t + \frac {1}{T _ {e n d} - T _ {g o}} \int_ {T _ {g o}} ^ {T _ {e n d}} | | \mathbf {y} ^ {(k)} (t) - \mathbf {v} ^ {(k)} | | ^ {2} d t\right). \tag {C.0.2}
$$
Supplementary Figure 5: Poisson log-likelihood fitting. a. Projection on first three  $\mathbf{a}_j$ . b. State of the ltrRNN at go cue  $+100\mathrm{ms}$ . c. First three  $\mathbf{c}_j$ . d. Eigenspectrum of  $W^{(k)}$  over trials. e. Trial factors  $\mathbf{c}_j$ . f. Projection of  $\mathbf{x}^{(k)}(t)$  on the corresponding  $\mathbf{a}_j$ .
In particular, the speed of the reach is only constrained by the noise of the RNN and the hand. The dynamical system as a whole is evaluated with a differentiable adaptive step SDE solver [40] and trained with ADAM [53] during initial training, and SGD during motor perturbation learning.