SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $n_{\mathrm{tot}} = n_{\mathrm{rec}} + n_{\mathrm{in}}$ . We note that for a given  $i$ , the functions  $\mathcal{L}_i$  depend only on  $\hat{\beta}^{(i)} \coloneqq \hat{\theta}_{;i}$ , i.e., the  $i$ th column of the  $\hat{\theta}$  matrix. In other words, the lack of cross-talk between the columns of  $\hat{\theta}$  in the loss function means each output can be regressed independently with respect to the corresponding part of the loss function, i.e.,  $\mathcal{L}_i(\hat{\theta}_i)$ . Thus, for the rest of this section, without loss of generality, we simply considered one instance of the regression, i.e., fix  $i$ , derived the learning rule, and later generalized to the full problem.
The loss function of interest for the subproblem is:
$$
\mathcal {L} (\hat {\beta}) = \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \mathrm {C E} \left(\frac {1 + \hat {d} _ {t}}{2}, \frac {1 + d _ {t}}{2}\right) + \lambda \sum_ {j = 1} ^ {n _ {\mathrm {t o t}}} \frac {(\hat {\beta} _ {j}) ^ {2}}{2}. \tag {S2}
$$
The gradient can be computed as:
$$
\begin{array}{l} (\nabla_ {\hat {\beta}} \mathcal {L} (\hat {\beta})) _ {k} = \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \left[ \nabla_ {\hat {\beta}} \mathbf {C E} \left(\frac {1 + \hat {d} _ {t}}{2}, \frac {1 + d _ {t}}{2}\right) \right] _ {k} + \lambda \hat {\beta} _ {k}, \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {c _ {t}}{2} \left(1 - \hat {d} _ {t}\right) \left(1 + \hat {d} _ {t}\right) \left[ \frac {1 - d _ {t}}{1 - \hat {d} _ {t}} - \frac {1 + d _ {t}}{1 + \hat {d} _ {t}} \right] x _ {t, k} + \lambda \hat {\beta} _ {k}, \tag {S3} \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \left(\hat {d} _ {t} - d _ {t}\right) x _ {t, k} + \lambda \hat {\beta} _ {k} \\ \end{array}
$$
The Hessian follows directly from the gradient and the fact that  $\tanh^{\prime}(x) = (1 - \tanh^{2}(x))$ :
$$
\left(\nabla_ {\hat {\beta}} ^ {2} \mathcal {L} (\hat {\beta})\right) _ {k l} = \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \left(1 - \hat {d} _ {t} ^ {2}\right) x _ {t, k} x _ {t, l} + \lambda \delta_ {k l}. \tag {S4}
$$
At this point, it is worth emphasizing that while  $x_{t,i}$  are shared across all subproblems,  $d_t$  are not. In fact, for any neuron  $i$ , a different set of  $d_{t,i}$  would contribute to the Hessian of the corresponding subproblem. Thus, in its current form, using the exact Hessian would require repeating matrix inversions for each of the  $N_{\mathrm{rec}}$  neurons. We performed benchmarking with the exact Newton method, See Supplementary Section S2, though the repeated matrix inversion was no longer feasible for large networks (Fig. 3). To mitigate this, we took a fixed-point solver approach in CORNN, described below.
First, we assumed that one can initialize the full problem with an initial point, call it  $\bar{d}_t = \tanh (\hat{\beta}_{\mathrm{fp}}^T x_t)$  such that  $\bar{d}_t\approx d_t$ . We discussed in Supplementary Section S1.2 on how to estimate  $\hat{\beta}_{\mathrm{fp}}$ . Then, we replaced the Hessian with a local approximation  $\nabla_{\hat{\beta}}^{2}\mathcal{L}(\hat{\beta})\approx \nabla_{\hat{\beta}}^{2}\mathcal{L}(\hat{\beta}_{\mathrm{fp}})$  within the vicinity of  $\bar{d}_t\approx d_t$  leading to the local approximate Hessian:
$$
\left(\nabla_ {\hat {\beta}} ^ {2} \mathcal {L} \left(\hat {\beta} _ {\mathrm {f p}}\right)\right) _ {i j} = \frac {1}{T} \sum_ {t = 1} ^ {T} \underbrace {\frac {1 - \hat {d} _ {t} ^ {2}}{1 - \bar {d} _ {t} ^ {2}}} _ {\approx 1} x _ {t, i} x _ {t, j} + \lambda \delta_ {i j} \approx \frac {1}{T} X ^ {T} X + \lambda I = H _ {\mathrm {f p}}. \tag {S5}
$$
From this equation, we can infer why the choice of  $c_{t}$  in the main text was scalable. The local Hessian depends only on the inputs, which are shared across all the  $n_{\mathrm{rec}}$  independent subproblems, and not on
the predicted or true outputs  $(\hat{d}_t$  or  $d_{t})$ . Moreover, it needs to be computed once, and at the beginning of the optimization only.
Using the approximate Hessian and gradient, the parameter update rule simply follows:
$$
\Delta_ {\mathrm {f p}} = - H _ {\mathrm {f p}} ^ {- 1} \nabla_ {\hat {\beta}} \mathcal {L} (\hat {\beta}). \tag {S6}
$$
Even though  $\nabla_{\hat{\beta}}\mathcal{L}(\hat{\beta})$  was a vector for the subproblem, given that the (approximate) Hessians for all subproblems are aligned to the correlation matrix, we can promote the gradient to a matrix  $\nabla_{\hat{\theta}}\mathcal{L}(\hat{\theta})$  to solve all subproblems at once. Thus, one can vectorize the computation for each subproblem, i.e., minimization of  $\mathcal{L}_i(\hat{\beta}^{(i)})$  for  $i = 1,\dots ,N_{\mathrm{rec}}$ , without the need for an external for loop.
At this point, we note that since  $H_{\mathrm{fp}} \succeq 0$  is positive semi-definite (PSD), the direction of the parameter updates is guaranteed to be a descent direction. This is because the update direction has a negative angle with the gradient following  $\Delta_{\mathrm{fp}}^{T}\nabla_{\hat{\theta}}\mathcal{L}(\hat{\theta}) = -\nabla_{\hat{\theta}}\mathcal{L}(\hat{\theta})^{T}H_{\mathrm{fp}}^{-1}\nabla_{\hat{\theta}}\mathcal{L}(\theta) \leq 0$  by positive semi-definiteness. Thus, even if the error between  $\bar{d}_t$  and  $d_t$  is large for an instance, e.g., fixed-point is incorrectly chosen, it can only hurt the speed of the optimization. Bringing all together, fixed-point updates can be written as:
$$
\theta^ {(k + 1)} := \theta^ {(k)} - \gamma_ {k} H _ {\mathrm {f p}} ^ {- 1} \nabla_ {\theta} \mathcal {L} \left(\theta^ {(k)}\right), \tag {S7}
$$
where  $\gamma_{k}$  is the user-controlled learning parameter chosen either as 1 (to bypass the line search if fixed-point was close enough) or through back-tracking line search. For this work, we simply picked  $\gamma_{k} = 1$  without any line search as we did not observe any convergence issues. We will simplify Eq. (S7) to its final form when discussing the ADMM solver in Supplementary Section S1.4.
S1.2 Fixed point initialization
To derive the fixed point update in Eq. (11), we performed a Taylor approximation of the gradient and carried out a single update near the vicinity of the putative fixed-point:
$$
\begin{array}{l} \hat {\beta} ^ {(t + 1)} := \hat {\beta} ^ {(t)} - H _ {\mathrm {f p}} ^ {- 1} \nabla_ {\hat {\beta}} \mathcal {L} (\hat {\beta} ^ {(t)}), \\ \approx \hat {\beta} ^ {(t)} - \left(\frac {1}{T} X ^ {T} X + \lambda I\right) ^ {- 1} \left[ \left(\frac {1}{T} X ^ {T} X + \lambda I\right) \hat {\beta} ^ {(t)} - \frac {1}{T} X ^ {T} Z \right], \tag {S8} \\ = \hat {\beta} ^ {(t)} - \hat {\beta} ^ {(t)} + \left(\frac {1}{T} X ^ {T} X + \lambda I\right) ^ {- 1} \frac {1}{T} X ^ {T} Z, \\ = \hat {\beta} _ {\mathrm {l s}}, \\ \end{array}
$$
where a single-step converges to  $\hat{\beta}_{\mathrm{ls}} = (X^T X + T\lambda I)^{-1}X^T Z$ , i.e., the solution to the least squares problem on the currents  $Z$ . Thus,  $\hat{\beta}_{\mathrm{ls}}$  is expected to be near to both the solution  $\beta^{*}$  and the regime described by the fixed-point Hessian  $H_{\mathrm{fp}}$ ; thus a suitable candidate to be the fixed-point itself. As an added bonus, this choice did not increase the complexity of the solver; since  $(X^T X + T\lambda I)^{-1}X^T$  has already been pre-computed for the iterative updates.
S1.3 Interpretation of the weighted convex loss
To interpret CORNN's loss function, we consider the low error limit of the cross entropy  $\mathrm{CE}(\hat{p}, p)$  around  $\hat{p} \approx p$ :
$$
\begin{array}{l} \mathrm {C E} (\hat {p}, p) = - p \log (\hat {p}) - (1 - p) \log (1 - \hat {p}), \\ = - H (p) - \frac {\mathrm {d}}{\mathrm {d} \hat {p}} [ p \log (\hat {p}) + (1 - p) \log (1 - \hat {p}) ] \Big | _ {\hat {p} = p} (\hat {p} - p) \\ - \frac {\mathrm {d} ^ {2}}{2 \mathrm {d} \hat {p} ^ {2} [ t ]} \left[ p \log (\hat {p}) + (1 - p) \log (1 - \hat {p}) \right] \Big | _ {\hat {p} = p} (\hat {p} - p) ^ {2}, \\ = - H (p) + \frac {(\hat {p} - p) ^ {2}}{2 p (1 - p)} + O ((\hat {p} - p) ^ {3}), \\ \end{array}
$$
where  $-H(p)$  is a constant independent of the optimization variables, and thus can be ignored. Then, the low-error limit of the CORNN loss is:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \mathrm {C E} \left(\frac {1 + \hat {d} _ {t}}{2}, \frac {1 + d _ {t}}{2}\right) \approx \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {\left(d _ {t} - \hat {d} _ {t}\right) ^ {2}}{2 \left(1 - d _ {t} ^ {2}\right) ^ {2}}, \\ \approx \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {\left(d _ {t} - d _ {t} - \left(1 - d _ {t} ^ {2}\right) [ \hat {z} - f ^ {- 1} \left(d _ {t}\right) ]\right) ^ {2}}{2 \left(1 - d _ {t} ^ {2}\right) ^ {2}}, \tag {S9} \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} \frac {\left[ \hat {z} - f ^ {- 1} \left(d _ {t}\right) \right] ^ {2}}{2}. \\ \end{array}
$$
Here, in the second row, we performed a Taylor approximation around the optimal value  $\hat{d}_t \approx d_t + (\hat{d}_t')|_{\hat{d}_t = d_t}(\hat{z} - f^{-1}(d_t))$ . In this low error limit, minimizing the scalable weighted loss approximately accounts to minimizing the  $\mathcal{L}_2$  loss on the currents. The main difference is, since the original weighted loss is on the firing rates and not currents, the weighted loss can account for the conversion noise, or non-linearity mismatches, that would induce bias in the explicit conversion  $z = f^{-1}(d)$ . The observation that low-error limit of the CORNN loss approximates the least-squares provided additional support to the claim that least-squares solution serves as a suitable initialization.
S1.4 Derivation of the ADMM updates
So far, we discussed the fixed-point solver for minimizing the unconstrained CORNN loss with L2 regularization. In this section, we discuss how to add constraints on weights. Specifically, in a general scenario, it is unexpected that all neurons would be connected to each other. Rather, in many cases, one might be interested in introducing a trainable parameter mask such that  $\forall (i,j) \in B(i,j) W_{ij} = 0$ . To account for this scenario, we incorporated our fast solver into the ADMM framework.
S1.4.1 Setting up the full problem
In this section, we will first setup the full problem of interest with constraints, and then review the ADMM equations [51]. To start with, we considered a problem of the following form:
$$
\text {m i n i m i z e} \quad \mathcal {L} (\hat {\beta}) = \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \mathrm {C E} \left(\frac {1 + \hat {d} _ {t}}{2}, \frac {1 + d _ {t}}{2}\right) + \lambda \sum_ {j = 1} ^ {n _ {\mathrm {t o t}}} \frac {(\hat {\beta} _ {j}) ^ {2}}{2}, \tag {S10}
$$
$$
\text {s u b j e c t} \quad \forall k \in B (k), \hat {\beta} _ {k} = 0.
$$
We transformed this problem by adding a new variable  $\hat{\chi}$ , which was constrained to be equal to  $\hat{\beta}$ :
$$
\text {m i n i m i z e} \quad \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \mathrm {C E} \left(\frac {1 + \hat {d} _ {t}}{2}, \frac {1 + d _ {t}}{2}\right) + \lambda \sum_ {j = 1} ^ {n _ {\mathrm {t o t}}} \frac {(\hat {\beta} _ {j}) ^ {2}}{2}, \tag {S11}
$$
$$
\text {s u b j e c t} \quad \forall k \in B (k), \hat {\chi} _ {k} = 0, \quad \hat {\chi} = \hat {\beta}.
$$
Defining the indicator function  $1(x)$  as
$$
1 (x) = \left\{ \begin{array}{l l} 0 & \text {i f x i s t r u e ,} \\ \infty & \text {i f x i s f a l s e} \end{array} \right. \tag {S12}
$$
The ADMM loss function became:
$$
\text {m i n i m i z e} \quad \mathcal {L} _ {\mathrm {A D M M}} (\hat {\beta}, \hat {\chi}) = \underbrace {\frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \mathrm {C E} \left(\frac {1 + \hat {d} _ {t}}{2} , \frac {1 + d _ {t}}{2}\right) + \lambda \sum_ {j = 1} ^ {n _ {\mathrm {t o t}}} \frac {(\hat {\beta} _ {j}) ^ {2}}{2}} _ {\mathcal {L} _ {\mathrm {C O R N N}} (\hat {\beta})} + \underbrace {\sum_ {k \in B (k)} 1 (\hat {\chi} _ {k} = 0)} _ {\mathcal {L} _ {\mathrm {c o n s t r a i n t}} (\hat {\chi})},
$$
$$
\text {s u b j e c t} \quad \hat {\chi} = \hat {\beta}. \tag {S13}
$$
ADMM Lagrangian has two separable loss functions of independent variables, connected through a linear equality constraint, which can be augmented to obtain:
$$
\mathcal {L} _ {\rho} (\hat {\beta}, \hat {\chi}, \kappa) = \mathcal {L} _ {\mathrm {A D M M}} (\hat {\beta}, \hat {\chi}) + \kappa^ {T} (\hat {\beta} - \chi) + \frac {\rho}{2} \| \hat {\beta} - \hat {\chi} \| _ {2} ^ {2}, \tag {S14}
$$
where  $\kappa$  is the dual variable. The ADMM steps (after re-defining  $\kappa \rightarrow (1 / \rho)\kappa$ ) became (See Eqs. (3.5-3.7) in [51]):
$$
\hat {\beta} ^ {k + 1} := \underset {\hat {\beta}} {\arg \min } \left(\mathcal {L} _ {\text {C O R N N}} (\hat {\beta}) + (\rho / 2) \| \hat {\beta} - \hat {\chi} ^ {k} + \kappa^ {k} \| _ {2} ^ {2}\right), \tag {S15a}
$$
$$
\hat {\chi} ^ {k + 1} := \underset {\hat {\chi}} {\arg \min } \left(\mathcal {L} _ {\text {c o n s t r a i n t}} (\hat {\chi}) + (\rho / 2) \| \hat {\beta} ^ {k + 1} - \hat {\chi} + \kappa^ {k} \| _ {2} ^ {2}\right), \tag {S15b}
$$
$$
\kappa^ {k + 1} := \kappa^ {k} + \hat {\beta} ^ {k + 1} - \hat {\chi} ^ {k + 1}. \tag {S15c}
$$
S1.4.2 Solving first ADMM iteration via fixed-point updates
Similar to the unconstrained case, we started by finding the gradient and the fixed-point Hessian:
$$
\nabla_ {\hat {\beta}} \left[ \mathcal {L} _ {\text {C O R N N}} (\hat {\beta}) + (\rho / 2) | | \hat {\beta} - \hat {\chi} ^ {k} + \kappa^ {k} | | _ {2} ^ {2} \right] = - \frac {1}{T} X ^ {T} E + (\lambda + \rho) \hat {\beta} + \rho \left(\kappa^ {k} - \hat {\chi} ^ {k}\right), \tag {S16a}
$$
$$
\nabla_ {\hat {\beta} _ {f p}} ^ {2} \left[ \mathcal {L} _ {\text {C O R N N}} (\hat {\beta}) + (\rho / 2) | | \hat {\beta} - \hat {\chi} ^ {k} + \kappa^ {k} | | _ {2} ^ {2} \right] \approx \frac {1}{T} X ^ {T} X + (\lambda + \rho) I, \tag {S16b}
$$
where we recall the prediction error matrix:
$$
E _ {t, i} = \frac {d _ {t , i} - \hat {d} _ {t , i}}{1 - d _ {t , i} ^ {2}}. \tag {S17}
$$
Then, a single fixed-point update step became:
$$
\begin{array}{l} \Delta_ {\mathrm {f p}} = - \left[ \frac {1}{T} X ^ {T} X + (\lambda + \rho) I \right] ^ {- 1} \left[ - \frac {1}{T} X ^ {T} E + (\lambda + \rho) \hat {\beta} + \rho (\kappa^ {k} - \hat {\chi} ^ {k}) \right], \\ = \left[ X ^ {T} X + T (\lambda + \rho) I \right] ^ {- 1} \left[ X ^ {T} E - T (\lambda + \rho) \hat {\beta} + T \rho \left(\hat {\chi} ^ {k} - \kappa^ {k}\right) \right], \tag {S18} \\ = A ^ {+} X ^ {T} E - (\tilde {\lambda} + \tilde {\rho}) A ^ {+} \hat {\beta} + \tilde {\rho} A ^ {+} \left(\hat {\chi} ^ {k} - \kappa^ {k}\right), \\ \end{array}
$$
with  $\tilde{\lambda} = T\lambda$  and  $\tilde{\rho} = T\rho$ . Here, we defined the short-hand notation for the inverse matrix:
$$
A ^ {+} = \left[ X ^ {T} X + (\tilde {\lambda} + \tilde {\rho}) I \right] ^ {- 1}. \tag {S19}
$$
Consequently, the update rule for  $\gamma_{k} = 1$  became:
$$
\begin{array}{l} \hat {\beta} ^ {k + 1} = \hat {\beta} ^ {k} + A ^ {+} X ^ {T} E ^ {k} - (\tilde {\lambda} + \tilde {\rho}) A ^ {+} \hat {\beta} ^ {k} + \tilde {\rho} A ^ {+} \left(\hat {\chi} ^ {k} - \kappa^ {k}\right), \tag {S20} \\ = A ^ {+} X ^ {T} X \hat {\beta} ^ {k} + A ^ {+} X ^ {T} E ^ {k} + \bar {\rho} A ^ {+} \left(\hat {\chi} ^ {k} - \kappa^ {k}\right), \\ \end{array}
$$
where we highlighted all quantities that can be pre-computed with blue. For  $\rho = 0$ , this reproduces the update rule for the unconstrained problem in Eq. (10). As before, we promoted the vector  $\hat{\beta}$  to a matrix  $\hat{\theta}$ , hence vectorized across the full problem.
S1.4.3 Solving second ADMM iteration via projection
For this step, the goal is to minimize
$$
\text {m i n i m i z e} \sum_ {j = 1} ^ {n _ {\text {t o t}}} (\rho / 2) \left(\hat {\beta} _ {j} ^ {k + 1} - \hat {\chi} _ {j} + \kappa_ {j} ^ {k}\right) ^ {2}, \tag {S21}
$$
$$
\text {s u b j e c t} \quad \forall k \in B (k), \hat {\chi} _ {k} = 0.
$$
Fortunately, this problem was perfectly separable across the vector entries of  $\hat{\chi}_j$ . For cases where  $\chi_k = 0$  was constrained, this was the only feasible point and thus the answer to that particular problem. For when  $j \notin B(j)$ , we obtained the minimum loss value by simply picking  $\hat{\chi}_j = \hat{\beta}_j^{k+1} + \kappa_j^k$ . In short notation, the update rule for this subproblem became:
$$
\hat {\chi} ^ {k + 1} := \underset {\hat {\chi}} {\arg \min } \left(\mathcal {L} _ {\text {c o n s t r a i n t}} (\hat {\chi}) + (\rho / 2) \| \hat {\beta} ^ {k + 1} - \hat {\chi} + \kappa^ {k} \| _ {2} ^ {2}\right) = \Pi_ {B} \left(\hat {\beta} ^ {k + 1} + \kappa^ {k}\right), \tag {S22}
$$
where we defined  $\Pi_B$  as the projection operator to the constraint satisfying subspace (i.e.,  $\forall k\in B(k)$ ,  $\hat{\chi}_k = 0$ ). Similar to before, we promoted the vector  $\hat{\chi}$  to a matrix to vectorize the full problem.
S1.4.4 Variable initialization and choice of  $\rho$
A priori, it was not clear how  $\rho$  should be chosen. However, looking at the Hessian inverse  $A^{+}$ , we observed that  $\rho \geq \lambda$  was an appropriate choice for the step size  $\rho$  to have a reasonable impact on the optimization problem. Empirically, we observed that  $\rho \gg \lambda$  lead to faster convergence.
We initialized the primal and dual variables with  $\chi^0 = \hat{\beta}^0 = \hat{\beta}_{\mathrm{fp}}$ , i.e. the pre-computed least-squares solution,  $\hat{\beta}_{\mathrm{fp}} = A^{+}X^{T}Z$ , without additional computational complexity. Moreover, given that ADMM ensured convergence  $\hat{\chi} = \hat{\theta}$ , the dual variable  $\kappa$  was bound to converge to non-zero values only at the diagonal such that the 2nd ADMM step would converge.
To sum up, without any additional computational cost, we defined the initial conditions for all three variables as:
$$
\hat {\beta} ^ {0} = A ^ {+} X ^ {T} Z, \tag {S23a}
$$
$$
\hat {\chi} ^ {0} = A ^ {+} X ^ {T} Z, \tag {S23b}
$$
$$
\kappa^ {0} = 0. \tag {S23c}
$$
S1.5 Automated outlier detection and the pseudo-code for the full solver
Upon close inspection of the prediction error, we observed that the extremely large error values stemming from  $d_{t} \approx 1$  would amplify the conversion noise and carry little-to-no learning signal. Thus, we implemented a simple outlier detection by ignoring errors larger than some value such that  $|E_{t,i}| > v \Rightarrow E_{t,i} = 0$ . In the CPU solver, we scaled the gradient to account for the missing data points. Since these data points constituted small percentages, mostly  $< 1\%$  of the full dataset, we did not perform the scaling for the GPU based solver to prevent unnecessary overhead.
It is worth emphasizing that this choice of outlier detection takes away the convexity of the problem since zeroing out part of the prediction error leads to a non-convex global loss. However, the outliers tended to be ignored throughout the optimization process, thus the problem remained convex in remaining data points. Future work can consider a Huber-like [58] global convex robust loss that would automatically detect outliers. For the purpose of this work, we observed that the version of outlier detection we employed here performed well on empirical benchmarks.
For all solvers, we also defined a working precision  $\delta$  ( $= 10^{-6}$ ) and whenever  $|d_t| \geq |1 - \delta|$ , we projected  $d_t$  back to  $\pm (1 - \delta)$  to prevent numerical overflow errors. For experiments in Figs. S1 and S6 with BPTT, we projected the firing rates  $|r_t| \geq |1 - \delta|$  to the boundary  $r_t = \pm (1 - \delta)$  for the same reasons. Empirically, this stabilized all solvers and was performed to ensure a fair comparison.
Bringing all together, we state the full CORNN solver in Algorithm 1. We note that for the figures in this work, we simply used a maximum number of iterations (30-100 depending on experiment) instead of monitoring the convergence of CORNN; as CORNN runtimes remained sub-minute and a few more iterations than ideal were perfectly tolerable.
S2 Implementation details for other solvers
S2.1 Back-propagation through time
We performed back-propagation through time for the experiments in Figs. S1 and S6. In our experiments, the teacher forcing means that the neural activities at time  $t + 1$  is computed via Eq. (2), but we replace  $\hat{x}_{t,i}$  in  $\hat{z}_{t,i} = \sum_{j} \hat{\theta}_{t,j} \hat{x}_{t,j}$  with the data  $x_{t,i}$ , and  $\hat{r}_{t,i}$  with  $r_{t,i}$ . Initially,  $\hat{x}_{t,i}$  depends on  $\hat{\theta}$  implicitly, so this replacement cuts off the time-dependent computation graph for the gradient. The probability of teacher forcing is the probability with which the replacement is performed at every single time point during the forward propagation of the network. Specifically, we use the following forward-propagation equations:
$$
\hat {r} _ {t + 1, i} = \left\{ \begin{array}{l l} (1 - \alpha) r _ {t, i} + \alpha f \left(z _ {t, i}\right), & u <   p, \\ (1 - \alpha) \hat {r} _ {t, i} + \alpha f \left(\hat {z} _ {t, i}\right), & u \geq p, \end{array} \right. \tag {S24}
$$
Algorithm 1: Convex and scalable solver for CORNN via ADMM framework
CORNN.fit  $(X,D,\lambda ,\mathrm{nIter} = 30,\rho_{\mathrm{in}} = 100)$ $\% X$  contains the aggregated input, e.g. firing rates and input units. It is of size  $[T\times N_{\mathrm{tot}}]$ $\% D$  contains the targets. It is of size  $[T\times N_{\mathrm{rec}}]$ $\%$  To start, rescale the regularization parameter  $\lambda \coloneqq T\lambda$ $\rho \coloneqq \rho_{\mathrm{in}}\lambda \% \rho$  is picked to be comparable to entries of  $X^T X$  and rescaled  $\lambda$ $\%$  Pre-compute the inverses once during initialization;   
 $A^{+}\coloneqq (X^{T}X + (\rho +\lambda)I)^{-1};$ $X^{+}\coloneqq A^{+}X^{T}$ $X^{-}\coloneqq A^{+}X^{T}X = X^{+}X$ $Z\coloneqq f^{-1}(D)\%$  Compute the currents from the targets ;   
 $\theta_{\mathrm{fp}}\coloneqq X^{+}Z\%$  Initialize the fixed-point to the ls solution;   
 $\%$  Initialize the primary and dual variables;   
 $\theta \coloneqq \theta_{\mathrm{fp}}$  %Initialize to the least-squares solution;   
 $\chi \coloneqq \theta_{\mathrm{fp}}$  %Initialize to the least-squares solution;   
 $u\coloneqq 0*\theta_{\mathrm{fp}}$  %Initialize the dual variable;   
for  $i = 1:$  nIter; Predict the targets  $\hat{D} = f(X\theta^k)$  . Compute the prediction error  $E^{k} = (D - \hat{D})\oslash (1 - D\odot D)$  and perform outlier detection  $\% \oslash /\odot$  stand for element-wise division/product ; Update first primal variable  $\theta^{k + 1}\coloneqq X^{-}\theta^{k} + X^{+}E^{k} + \rho A^{+}(\chi^{k} - \kappa^{k})$  Update second primal variable  $\chi^{k + 1}\coloneqq \Pi_B(\theta^{k + 1} + u^k)$  Update the dual variable  $u^{k + 1}\coloneqq \kappa^k +\theta^{k + 1} - \chi^{k + 1}$  Probe convergence by checking  $||\chi^{k + 1} - \theta^{k + 1}||$  , continue if not converged;   
end   
return  $\chi$
where  $u$  is a uniform random number generated between 0 and 1. It is worth emphasizing that the teacher forcing in this context is distinct from [59], where the teacher signal is acquired not from the real-data but from a second network, though the overall idea is similar.
S2.2 Current and firing rate based FORCE
We followed the approach by [25] for training the recurrent weights to reproduce target neural activities via FORCE. For both network types, we defined the prediction errors as:
$$
e [ t ] = (\hat {r} [ t ] - r [ t ]) / \alpha , \quad \text {o r} \quad e [ t ] = \hat {x} [ t ]) - x [ t ]. \tag {S25}
$$
Here,  $\hat{r}[t]$  is the predicted firing rate at time point  $t$ , vice versa for  $\hat{x}[t]$ . We no longer performed any teacher forcing, thus  $r[t - 1]$  and  $x[t - 1]$  needed to predict  $\hat{r}[t]$  came from the previous prediction step of the network, and were not the ground truth values.
The recurrent weights and the corresponding inverse covariance matrix  $P[t]$  were updated by the following rule:
$$
P [ t ] = P [ t - 1 ] - \frac {P [ t - 1 ] \hat {r} [ t - 1 ] \hat {r} ^ {T} [ t - 1 ] P [ t - 1 ]}{1 + \hat {r} ^ {T} [ t - 1 ] P [ t - 1 ] \hat {r} [ t - 1 ]}, \tag {S26a}
$$
$$
W _ {\text {r e c}} [ t ] = W _ {\text {r e c}} [ t - 1 ] - e [ t ] \left(P [ t ] \hat {r} [ t - 1 ]\right) ^ {T}. \tag {S26b}
$$