Error Forcing in Recurrent Neural Networks
Authors: A Erdem Sağtekin
Colin Bredenberg
Cristina Savin
Venue: NeurIPS 2025 spotlight ()
Keywords: recurrent neural networks
bio-plausible learning
Primary area: neuroscience_and_cognitive_science
TL;DR: A more effective way of forcing RNN dynamics during supervised learning.

Abstract: How should feedback influence recurrent neural network (RNN) learning? One way to address the known limitations of backpropagation through time is to directly adjust neural activities during the learning process. However, it remains unclear how to effectively use feedback to shape RNN dynamics. Here, we introduce error forcing (EF), where the network activity is guided orthogonally toward the zero-error manifold during learning. This method contrasts with alternatives like teaching forcing, which impose stronger constraints on neural activity and thus induce larger feedback influence on circuit dynamics. Furthermore, EF can be understood from a Bayesian perspective as a form of approximate dynamic inference.  Empirically, EF consistently outperforms other learning algorithms across several tasks and its benefits persist when additional biological constraints are taken into account. Overall, EF is a powerful temporal credit assignment mechanism and a promising candidate model for learning in biological systems.

Error Forcing in Recurrent Neural Networks
A. Erdem Sağtekin New York University University of Tübingen aes10217 00002211@@yueemm edu
Colin Bredenberg
University of Montreal
Mila Quebec Al Institute colint in. bredenberg@mila quebec
Cristina Savin New York University
csavin@nyu.cm
edu edu
Abstract
How should feedback influence recurrent neural network (RNN) learning? One way to address the known limitations of backpropagation through time is to directly adjust neural activities during the learning process. However, it remains unclear how to effectively use feedback to shape RNN dynamics. Here, we introduce error foreing (EF), where the network activity is guided orthogonally toward the zero-eree pero-error manifold during learning. This method contrasts with alternatives like teaching forcing, which impose stronger constraints on neural activity and thus induce larger feedback influence on circuit dynamics. Furthermore, EF can understood from a Bayesian perspective as a form of approximate dynamic inference. Empirically, EF consistently outperforms other learning algorithms across several tasks and its benefits persist when additional biological constraints are taken into account. Overall, EF is powerful temporal credit assignment mechanism and promising candidate model for learning in biological systems.
Introduction
Most theoretical neuroscience models posit that learning in the brain occurs via feedback projections that guide local synaptic plasticity (i-12). Such models-based on approximations of gradient descent-assume that feedback signals are delivered without interfering with network activity, for example by multiplexing feedforward and feedback signals [33, via global neuromodulatory
signals [14], by only infinitesimally perturbing the network state [15 However, circuit-level evidence of a crisp separation between learning and dynamics is less clear. Moreover, there is substantial evidence that humans are able to learn both cognitive and motor skills with very few feedback examples [8], and that they can rapidly correct for systematic errors induced by various manipulations (e-g. external force fields [9] or visuomotor manipulations (20)) This suggests that the brain may use dynamic error-correctig mechanisms to affect within- neural activity in order to rapidly improve performance before longer-tine timescale synaptic learning and consolidation 22). However, previous models of dynamic error correction-based synaptic plasticity in the brain have either been used in exclusively feedforward neural network architectures (3 24), or have been
used in recurrent neural networks provided with dense feedback signals 25/ without rigorous quantification of the temporal credit assignment capabilities of the proposed learning algorithms.
Here, we introduce error forcing (EF), an algorithm designed to exploit both greedy error-based guidance of cireuit dynamics and temporal credit assignment stabilize learning of long-term temporal dependencies (Fig. []p.. Improving on generalized teacher forcing (28), we show that our algorithm can better stabilize learning for supervised tasks with sparse feedback and long temporal horizons. Further, we justify our approach through theoretically principled connection to Bayesian
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

a
b
yt
X
Yt et
W
W
(1-a)
a) Illustration of the eπrо foreing mechanism. Errors are used to update the synaptic weights given by any gradient-based algorithm (blue dashed-lines), well to update the activity (red dashed lines). b) Discrete-time computational graph of the EF, showing only the forward changes. Red dashed lines depict the error to activity projection weights. Notice we used stop-gradients (small red line), which prevents gradients flowing backwards from activity to errors. See Section 2.3 for mathematical justification of this step.
Figure 1:
demonstrating that dynamic error correction has a sensible connection to existing optimization algorithms. Lastly, we demonstrate that EF provides significant benefits for temporal credit assignment across variety of tasks, and that its biologically plausible extensions preserve similar properties. These results collectively show that EF has promise both an improvement for temporal credit assignment machine learning methods, and as a model of rapid adaptation in the brain.
inference and variational expectation maximization 29
2 Learning by error forcing
2.1 Background
Consider the general formulation of a discrete-time RNN with a linear readout:
= Fo(r..,,x)).
(1)
Y: rr.
(2)
where F, E RV is the network state, X, N= the input, and Yt E NV the output. Parameters and á define the recurrent dynamics and readout, respectively. Minimizing the error between network outputs and the target yř requires computing the derivative of the loss with respect network parameters 0, typically achieved by backpropagation through time (BPTT). Following [00, given loss function C. = Σimr. (y),,. where T. < l < T is defined as response period, the BPTT
equations are given by:
0c
dL,
dC. Σ Uh, Ör, Ör, 08,
Σ
with
(3)
00,
t=1 00,
Òr δry 00,
Ør. Ør, Ør
0r
Or, II J CiSt
(4)
Or:
dr.-1 òr,-2
òr
Ōr
12
Depending the spectral properties of the Jacobians, the above product can lead to instabilities. where error signals either decay or grow without bounds [30]. For example, this occurs when the network state changes little over time (i.e. J; are very small across steps, when the network is visiting a stable attractor). Such conditions are of practical importance in many neuroscience tasks with explicit memory component (31). and here the Jacobian product exponentially decay, leading to vanishing gradient problem. Conversely, when the goal of the RNN training is to generate chaotic dynamics, the gradients inevitably explode (32
This well-known issue of vanishing and exploding gradients motivated the development of alternative approaches such teacher forcing (and its variations) [8 37, especially for the self-supervised

---

a
neuron activity 2
b
neuron activity 2
neuron activity 2
r2
r1
zero error manifold
To r1=1 12
Fo r2
ro
F1
neuron activity
neuron activity
neuron activity
Figure 2: Geometric explanation of differences between no forcing (a), teacher forcing with the minimum-norm solution (b), and error forcing (c); a = a) Each circle denotes the RNN state re at given time, and arrows denote the application of the forward function (Eq. [T. The RNN freely runs during training when no forcing mechanism is used. The zero-error manifold is the set points where, for a given set of readout weights (which do not change within trial), the network produces the target outputs Y (assumed constant for visual simplicity). b) The teacher forcing mechanism results in choosing the target state the minimum-nowe norm solution (white circle, the closest point to the origin on the zero error manifold). Beginning from rg. we apply forcing to get Tog via Eq. 5 Then, apply the forward function via Eq. which results in r1- The same procedure will be applied over the trial, and since the teacher state is not a function of the RNN state, it will remain the same, causing the RNN repeat the same activity sequence. Since ig = ij r2g. illustrate these points with one circle. c) When error forcing is used instead, the RNN can explore the phase-space during training.
setting. In its generalized version 38 28 teacher forcing (TF) pushes neuron activity towards states that would correspond correct outputs:
r, (1 - a)r; + ar,
(5)
rg Fe (F
(6)
where r, denotes the natural RNN state, rī is the target (teacher) that satisfies ył W Wor and their linear interpolation leads to the forced dynamics F,, with 0 S a S 1. When 0. the method reduces to BPTT. When states are partially forced, the computational graph of the network dynamics changes, which in turn changes the BPTT Jacobians:
Ōr.
Ör, 0i.-1
(F 07.-1 (1 aa)
J. 0r:-1 0 0r.-1
0r,-1
dr:-1
Well-behaved gradients can thus be ensured by a judicious selection of a 28
2.2 Geometric perspective
The question at hand becomes: How should determine the target states rt? When the readout is invertible, simply use its inverse to go from the output space the neural space: rt yy Similar approaches have been shown to be beneficial for RNN training when the output is higher dimensional than the hidden neurons, or when the effective latent dimensionality is lower than the output dimensionality (28 33) However, in most practical scenarios, and, arguably, in the brain [39,, the output is dindemennional but the readout pools over a larger group of neurons in the circuit. Mathematically. it follows that there is entire manifold of neural states with zero error. In this case, it is not priori clear towards which target state we should force neural activity.
In one version of TF. the activity target is defined as the minimum-norm mapping from the low- dimensional output space to activity space, given by the readout pseudoinverse, r min W y To illustrate the potential issue with this strategy and a better way forward, we will start by simple toy example. Consider the scenario of a network with 2 neurons and a 1-dimensional output. Considering the separation of time-scales between neuron activities and synaptic updates, we can think of mapping between activity and output as constant within trials. The only factor that changes the zero- error manifold within trials the output target (hence the offset of the manifold). For visualization purposes only, let assume constant target states y" (Fig. 2 cyan line), that the zero error
3

---

manifold does not move from time step t to t ++ 1 within a trial. When states are not forced. the RNN runs freely, finding itself outside of the zero-error manifold (Fgg.2)). With full foreing (a = 1). using the minimum-nome solution, the RNN is driven to the teacher state, and the next step follows Eq. 6 For stationary target output, the same teacher state is used at each step, causing the RNN to repeat the same activity pattern (Fig.Zb). limiting the exploration of the neural space during learning. Even when the output is time-varying. the teacher states will always lie on the one-dimensional subspace orthogonal to the null space of the readout (therefore orthogonal to the zero error manifold). Thus, traditional TF enforces unnecessarily strong constraints on the network's 's activity, which, as we will show empirically, can slow down the learning process and damage test performance.
In contrast, our proposed EF selects the optimal state such that r is the orthogonal projection of onto the manifold of possible optimal responses (Fig. [ The idea is that instead of enforcing a minimal norm solution, we select the minimal intervention to the preexisting network state to minimize the error, which should result in broader exploration of the zero-error manifold over learning and generally less forcing of the dynamics away from their natural state.
More precisely. the EF target state is given by r rG + g( (W a ex), where g(-) is a "stop gradient' operation. We will justify the use of the stop gradient here in terms of our Bayesian perspective on EF (Section 2.3, Appendix [B). but note that the empirical performance of methods with and without
the stop gradient was similar (A (Appendix(e Ch. The resulting computational graph is shown in Fig. Tb.
Substituting this term into Eq. 5
F. - (I-r)ri + (re (0o+o(W3e (WJei))
(8)
rr + ag (W e));
where e, - yi - Yz. Given this form of the full dynamics (including the feedback loop), we can use BPTT (or any of its bio-plausible approximations, as explained in Section (2.4 for the synaptic weight updates. When the EF mechanism is used with BPTT, we called the resulting learning procedure EF-BPTT.
To see how EF changes the BPTT gradient computation and how it is different from the changes induced by teacher forcing, we can write Eq. by rearranging the terms as (see Appendix A
F Ir; ag( W W Waxr) + a aa y2
(10)
This form illustrates the differences between how EF and TF modify dynamics and gradient updates.
EF and TF become dynamically equivalent when Ng 2 N as the W W. term becomes an identity matrix (see Appendix [)]. but the stop gradient operation prevents the TF gradient dampening phenomenon (Eq. 7 because the two rightmost terms of Eq. 110have zero gradient. When N, < N, EF only immediately affects activity in the row space of the readout. Since it doesn't it restrict activity in the readout nullspace (see Fig. 2b), neuron activities are free to move along the zero error manifold. These observations imply that the empirical performance improvements produced by EF relative to TF and BPTT are not due to dampening of Jacobian magnitudes, but are rather due to minimally guiding network activity towards an optimal activity regime.
2.3 Bayesian perspective
Although we have provided geometric interpretation of why EF may be expected to improve performance relative to TF, question remains: is there a way to connect EF to theoretically principled optimization methods? The need for such a link is underscored by well-known limitation of teacher forcing: it is unclear why TF should work at test time, since the training objective (an RNN with forced states) differs from the one we ultimately care about (unforced states). In fact, there has been targeted work on this train-test mismatch (c.g.. 40 4T). For GTF in particular, annealing the parameter during training has been will see that error forcing does not introduce this train-ter test mismatch, because it can be linked to principled optimization method in which training- time forcing can be expected to generalize to unforced dynamics at test time.
way to get around this discrepancy. In contrast, we
One way construct this connection is by viewing EF as a mechanism to train an input-conditioned latent generative model of the output targets, where the error corrections e, correspond to hidden latent variables. In this view, latent error corrections are inferred during training, conditioned on

---

both input stimuli x and output targets subsequently, in an approximation to the variational Expectation
Expecation-Maximizatton Maximization algorithm (29). network parameters are updated via gradient descent. Under this interpretation, test-time performance can be viewed as stimulus-conditionss generative process, where both latent errors and output targets are sampled via a stochastic process.
To illustrate how this works, we first replace the deterministic RNN dynamics (Eq. D with a Gaussian state-space model (by the addition of Gaussian noise e, onto the hidden state and n, onto the output):
Ty Cu:s, Xo:, 8) = F6 (r-- (re-1(Eo:t-1 Xu. X0 xx) + Et
(11)
Wari (E0.2, Xoa, 0) + +),-
(12)
where, as in the deterministic case r; refers to the network hidden state, Y2 refers to the output targets, and here we are treating E as an additive latent variable. There are two factors that differentiate this model from the more traditional Kalman Filter: 1) the transition function (-)-) is nonlinear: and 2) the error E is treated as the random variable for inference, whereas the hidden state variable r, is treated as deterministic function that depends on all network parameters 8, as well as preceding errors Coot and stimuli Xo:--1- We show (Appendix B that greedily inferring the optimal state space correction e given current and preceding targets Yo:t, stimuli Xu:t, and corrections c via maximum a posteriori filtering produces nearly identical state space dynamics as the deterministic EF case, while still allowing for BPTT learning (via the reparameterization trick [42). In this case, "reparameterization" refers to viewing the errors 6, as stochastic latent variables, which allows for the use of backpropagation through time through the deterministic hidden state update Fe(.). The stochastic EF dynamics are given by:
oe I o²
F. = F. WT
W W
W
e
(13)
where Ft is the "uncorrected" state driven by network inputs and recurrent corrections, O₁ is the error in output space, and F, is the "corrected" state. The state space error correction
E = WT WW + II e, corresponds to the output space error e, multiplied by a regular- ized pseudoinverse of the decoder, much we have used in original deterministic model. The main difference is that instead of controlling the magnitude of forcing via o, the ratio between latent noise and observation noise variances of of/42 controls the magnitude of forcing 33 (A//02 0 gives full forcing and o//02 + 0C gives no forcing).
This perspective connects EF to inference in state-space models (e.g.. the Kalman filter) and provides theoretically grounded basis for stochastic EF training. In particular, it explains why backpropagating through the error signal that modulates the RNN dynamics is unnecessary- gradient operator because the variational EM algorithm performs coordinate descent: it first infers the errors given the current parameters and then updates the network parameters while treating the inferred errors as constants (see Appendix for details). Furthermore, clarifies why applying error forcing only during training is natural (as opposed also forcing at test time): during training, error corrections correspond to targef-conditioned latent states inferred via approximate MAP estimation under the generative model, whereas test time both the latent errors (zero-mean, low-variance Gaussian noise) and the targets are generated by the model (Eq.
the use of the stop-
2.4 Bio-plausible error forcing
In this section we describe additional approximations that we explored to increase the biological plausibility of EF, which in its most basic form is unrealistic duc to its dependence on BPTT 43) and violations of weight transport [44] in its feedback signals used for both learning and error correction.
To alleviate issues associated with BPTT, we use RFLO 45 for temporal credit assignment in place of BPTT. RFLO provides local, online eligibility trace, by ignoring non-local interactions when computing gradients. We allow weight transport when computing synaptic updates for two reasons: the original work reported similar performance with and without random feedback weights, and recent studies show that local learning rules can adapt feedback pathways to approximate the transpose or pseudoinverse of the feedforward weights 46 47.. We refer to this variant-using our EF with
RFLO- based temporal credit assignment-
EF RFLO.
5

---

Less explored in previous work is the role played by feedback state-forcing weights. In particular, EF introduces an additional weight transport problem because the optimal forcing weights are constrained to be the pseudoinverse of the feedforward decoder weights, and it is unclear how this type of parameter information could be shared between physiologically separate synapses. To investigate this issue, we explored random fixed forcing weights and random learned forcing weights, pairing these feedback mechanisms with RFLO synaptic learning. These experiments probe how essential the exact pseudoinverse feedback matrix is to produce the benefits afforded by EF.
3 Numerical results
In this section, we evaluate the effectiveness of the eπror forcing (EF) mechanism and compare it with competing algorithms. We consider a diverse battery of tasks: some are ethologically relevant and widely used in computational neuroscience (evidence integration), and some are difficult long-time credit assignment benchmarks (nonlinear working memory). Each task family includes hyperparameters controlling the task difficulty to further probe the computational efficiency of EF. We first present results for EF-BPTT, followed by EF-RFLO and its variants.
3.1 EF-BPT
First, we tested the algorithms on the delayed XOR task (Fig (l. This is a nonlinear working memory task in which two input channels deliver rectangular pulses of amplitude +1 or -1, separated by fixed inter-stimulus delay at the start of each trial (Fig. 3h.. The network must compute the XOR of the two stimuli and report the answer after second delay. Crucially, this second delay varies across trials; consequently, the response time also varies across trials. To mark the start of the response window, we provide an explicit cue signal-if the second delay were fixed, such a cue would be unnecessary. Task difficulty is controlled by the mean of the second delay (not by its trial-to-trial variability): for each task condition, the second delay is sampled from distribution with specified mean and limited jitter, and increasing this mean increases the memory demand and thus the task difficulty. The network loss is assessed only during the response time, and so the RNN readout is unconstrained during non-response periods. Therefore, the forcing mechanism is also only active during the response period.
For every algorithm, and for every task difficulty, we trained 20 networks (continuous time RNNs, details in Appendix (D) and monitored their convergence. Specifically, a network is said to have converged its mean squared error (MSE) remains under 0.1 for at least 10 gradient steps, within a maximum of 200 training epochs. While the exact setting of this convergence threshold is somewhat arbitrary, it does not qualitatively affect the nature of our results. The optimal degree of forcing a was determined by grid search (note that a 0 reduces EF-BPTT to vanilla BPTT).
While task performance decreased sharply for standard BPTT with increasing task difficulty, EF-BPTT maintained a high success rate over the task range, outperforming GTF at very long delays (Fig. Bp). When results are aggregated across all delays, EF-BPTT proved not only systematically better when compared to TF-BPTT, but it was also more robust to variations in (Fig. 33)). This implies that the directionality of the feedback w.r.t. the zero-error manifold is more important for learning in this task than the precise strength of the forcing.
Next, we focused on input-dependent periodic signal generation, another task for which RNN training is known to be nontrivial 48). Concretely, the network received one analog input (one out of 7 possible values, selected uniformly at random in each trial), constant throughout the trial, and had to generate two sine waves its two output channels, with frequency proportional to the input and a fixed phase difference between them (Fig. 3). The multidimensional output space allows us to test the generality of EF improvements to variations in Ny. Moreover, unlike the previous task, feedback is persistent throughout the trial rather than sparse. As before, convergence is defined by MSE; visually accurate reconstruction corresponds to MSE of order 0.01, but we report the distribution of test MSEs for the full picture
We varied task difficulty by shifting the set of 7 frequencies upward. BPTT MSE increased substan- tially with increasing mean frequency within the simulated range (Fig. 3p) Surprisingly, teacher forcing destabilized learning, causing very high test MSE values for the entire frequency range.

---

a
delayed XOR
d
sine wave gen.
g
evidence int.
stimulus
input
stimulus
cue
target
target
deay response
dely resporse
% networks learned
b
% networks learned
h
100
10
100-
75
EF
test MSE
75
TF
50
BPTT
10"
50
25
25
10
1000
2000
4.8 4.9 5.0 5.1 5.2
1000 2000 3000
delay timesteps
mean frequency
delay timesteps
% networks learned
c
f
% networks learned
100-
100
75
test MSE
75
10"
50
50
25
25
10"
T
0
2
4
6
8
1.0
100
200
0
2
.4
66
8
1.0
epoch
Figure 3: a) Delayed XOR schematic. b) Proportion of networks that learn the delayed XOR task up to criterion as a function of delay (task difficulty);; see main text for details. EF-BPTT and TF-BPTT shown EF and TF, respectively, for brevity. e) Percentage of networks that learn XOR for different a values, aggregated over every task difficulty. We use values as small as 0.01. Note that a 0 is equivalent BPTT for both algorithms. d) Schematic of sine generation task: higher inputs (dashed line) require producing sines with higher frequency and the same relative phase. e) Mean- suuurdd error violin plots calculated for the sine wave generation task, estimated across 50 different networks for each task setting (x (x-axis) and for each algorithm (colors as in b). The x-axis denotes the mean frequency of the signals that the network has to generate. f) Test MSE during sine generation training for EF-,FF, and BPTT. Bold lines denote mean over networks. g) Schematic of evidence integration task. h) As b) for evidence integration. As in e) for evidence integration.
Unlike delayed XOR, this task proved more sensitive to judicious selection of a, but error forcing still systematically improved test performance over BPTT.
Lastly, we trained RNNs (40 networks for each setting) on variant of evidence integration motivated
by experiments rodents [49] (Fig. Fg). this task, two input channels provide a total of temporally sparse inputs on the 'left" or 'right' side. After an additional delay period (fixed across trials), the network has to output a persistent I or -1 during the response period depending on which of the two input channels had more signals. What makes this task an interesting case is the fact that its solution requires a different dynamical systems structure to emerge for efficcent learning (line attractor, as opposed to fixed points for XOR, set of limit cycles for sine generation) [50] 51]. Similar to the delayed XOR task, forcing is applied only during the response period, with the output well EF systematically improves over BPTT,
of the network unconstrained otherwise. In this although the differences relative to TF are more modest.
To conclude, across tasks and manipulations, EF-BPTT consistently outperformed TF-BPTT and BPTT, and proved much more robust to the degree of forcing a, consistently outperforming BPTT unlike TF-BPTT.
7

---

a delayed XOR
b evidence int.
delayed XOR
% networks learned
c
100
EF-RFLO TE-RFLO RFLO
100
100
EF-BPTT
50
50
50
rand. fixed
0
50 delay timesteps
100
150
100
200
delay timesteps
a
% networks learned
d delayed XOR
a ===0. 0.1
f
a. = 1
1.0
1.0
cos. cos. similarity
50
0.5
0.5
25
rand. learned rand. fixed
0.0
0.0
EF-RFLO rand. learned
rand. fixed
RFLO
1500
3000
0
1500
3000
gradient updates
gradient updates
Figure 4: a) Percentage of networks learned for set of networks with increasing delay timesteps (task difficulty), using RFLO. b) As in a), but for evidence integration. e) Comparison percentage of networks that learn (-BPTT), for delayed XOR task; with delay of 1500 time steps. Red denotes BPTT, green denotes EF-BPTT, blue denotes random-fixed projections instead of pseudoinverse. d) Comparison of percentage of networks that learn (-RFLO), for delayed XOR task; with delay of 95 time steps. Red denotes RFLO, green denotes EF-RFLO, blue denotes random-fixe green denotes random-learne projections. e) Blue: cosine similarity during gradient updates between random-fixed connections and readout weights. Teal-green: between random-learned connections and readout weights. 0.1. As in e), but a 1.0.
projections, teal
3.2 EF-RFLO
To assess the efficiency of the EF in biologically plausible settings, we equipped RFLO with EF and evaluated learning performance EF-RFLO, TF-RFLO, and RFLO. Despite the demands of temporal credit assignment, RFLO still learned within a limited range of delays (Fig, (..)). In the delayed XOR task, adding EF consistently improved performance relative to TF-RFLO and RFLO, though the gains were smaller than with BPTT. All three algorithms performed comparably the evidence integration task. These results suggest that EF's computational benefits persist under biologically plausible synaptic learning rules.
the delayed XOR and evidence integration tasks, comparing
As explained in Section 2.4 separate issue is the learning of the feedback forcing weights themselves. For this we consider two different approximations to EF-RFLO that avoid weight transport issues for the error forcing feedback synapses. Instead of using the correct projections (pseudoinverse)) we used either random-fixed connections, drawn from the same distribution as the initial readout or mandom-learned connections, where the learning is achieved by RFLO used on the same loss for all other network parameters. First, we chose an casy task difficulty for the delayed XOR task, and compared EF-BPTT with random-fixed projections to the idealized version using the pseudoinverse (Fig. 4p). As expected, using random projections severely decreased the performance, supporting the idea that the feedback needs to push the dynamics towards the zero error manifold as opposed to in some arbitrary fixed direction.
When testing the role of adaptive projections for RFLO (Fig. 4d), we found an overall degradation of performance relative to the idealized case but similar or slightly better performance compared the no feedback (RFLO) scenario.
8

---

Investigating the alignment between the readout and enror forcing weights during learning reveals an effect similar to feedback alignment in static multilayer perceptron networks [52]. Even with fixed random projections. the cosine similarity between feedback and readout weights steadily increases over time (Fig. f..f) because the readout weights become aligned with the fixed random projections. Increasing the a value negatively affects the cosine similarity between the weights; however, alignment is not directly predictive of task performance (Fig. 4 d.e.I). This somewhat paradoxical effect could be explained by the fact that alignment is primarily due to modification of the decoder weights, not the feedback weights. Under such conditions, EF suboptimal feedback weights could be effectively constraining the decoder to be in turn suboptimal with respect to the intrinsic network dynamics at initialization, impairing performance.
Overall, we have shown that the eπrrr forcing mechanism works in concert with bio-plausible gradient approximation methods (RFLO in particular), although additional work might be needed to fully address the weight transport problem for forcing synapses.
4 Discussion
In this study we have introduced error forcing (EF), method capable of reaping the benefits of dynamic error corrections for stabilizing neural dynamics during learning, while still preserving powerful temporal credit assignment capabilities. We have demonstrated empirical improvements of our method over both BPTT and TF-BPTT on several supervised tasks, and were able to justify these improvements in terms of both geometric and Bayesian interpretations of EF. Specifically, error forcing-in contrast to i'-iuduces a minimally invasive intervention on neural dynamics by orthogonally projecting neural activity onto the manifold of optimal outputs. We justified this projection by establishing a connection between EF and variational EM [29). showing that our error corrections can be interpreted as a form of online filtering, where network states are dynamically adjusted based on incoming readout error information. Lastly, while the basic version of EF uses the biologically implausible BPTT algorithm to adjust synaptic weights and suffers from the weight transport problem, we showed that these plausibility issues can be fixed, at the cost of some perfor- mance, by a combination of using random or learned error feedback projections in conjunction with the substitution of RFLO for BPTT.
EF can be viewed as hybridization of Kalman filter-based control theoretic error correction learning algorithms 25 26 53 and TF, theoretically unified by the use of the reparameterizztion trick. A similar algorithm using combination of RFLO and driving error feedback has recently been used to model sensorimotor adaptation in response to a virtual reality visuomotor rotation in macaques (27): our work extends this approach through our theoretical justification for EF, well as our empirical demonstrations of performance improvements in sparse-feedback supervised learning regimes.
While our analysis has been conducted assuming a linear decoder, EF could be extended to nonlinear (and potentially multilayer) output decoders by using Taylor expansions to locally linearize the output, in analogy to the extended Kalman filter [54] (see Appendix B.4.. Similar approaches have been used for feedforward neural networks [21 24 but it remains empirical open question whether linear Taylor expansions prove a sufficient approximation for stable use in conjunction with error forcing mechanisms. Such nonlinear extension could be useful for stabilizing training dynamics in modern state-space models 155 56], with potential applications to training large-language models (57).
Because of its relationship to Bayesian inference, EF has potential applications for both in-context learning (58) and few-shot learning [59 60]: namely, the rapid inference of 'errors' allows for EF networks to reach good training performance very rapidly, while test performance is consolidated on longer timescale through synaptic plasticity. Similarly, though we only used additive error corrections in this study, EF could be combined with hierarchical, compositional Bayesian models to both enable rapid compositional generalization 61 62 and potentially ameliorate catastrophic forgetting 633
Lastly, though we have only explored supervised learning in this study, one might expect that most biological motor learning occurs via reinforcement. This disconnect could resolved by using a critic network's 's expected future reward as a form of short-term supervised objective, as used in active inference (64] and deep Q-learning [65] (66 approaches; this context, the rapid error correction mechanism provided by error forcing could prove faster and more stable than the Markov-chain

---

Monte Carlo approach taken by [67 while eliminating the need for the separate parameterized variational posterior distributions used in 68 69
To conclude, this study we have shown that HF a theoretically justified approach stabilizing temporal credit assignment in sparse supervised feedback regimes; our work opens promising avenues for future research in improving machine learning algorithms, as well as for modeling learning in the brain.
This study is foundational work devoted to improving optimization of recurrent neural networks, with the goal of furnishing computationally powerful models of learning in the brain. While there are variety of potentially negative applications of effectively trained recurrent neural network models, extensive modification by third parties would be required to produce such negative impacts on the basis of the insights drawn here.
Broader Impacts:
Acknowledgments and Disclosure of Funding
We would like to thank members of Savin lab and Neslihan Serap Şengör for insightful discussions and feedback. This work was supported by the National Science Foundation under NSF Award No. 1922658, the Simons Foundation, and a Google faculty award. [CB] is supported part by the FRONT Strategic Clusters Program (Centre UNIQUE- Quebec Neuro-Al Research Center).
10

---

References
[1] Lukasz Kušmierz, Takuya Isomura, and Taro Toyoizumi. Learning with three factors: modu- lating Hebbian plasticity with errors. Current Opinion in Neurobiology, 46:170-177, October 2017.
[2] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated Spike-Timing-Deeeee and Theory of Three-Factor Learning Rules. Frontiers Neural Circuits, 9:85, January 2016.
Plasticity,
[3] Spike- Baike-Baseddde Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail PLOS Computational Biology.
[4] Peter L. Bartlett and Jonathan Baxter. Hebbian Synaptic Modifications in Spiking Neurons that Learn, November 2019. arXiv:1911.07244 [cs, stat].
[5] R.V. Florian. A reinforcement learning algorithm for spiking neural networks. In Seventh International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC'05), pages 8 pp.-, September 2005.
[6] Michael A. Farries and Adrienne L. Fairhall. Reinforcement learning with modulated spike timing dependent synaptic plasticity. Journal of Neurophysiolooy, 9866):3648-3665, December 2007.
[7] A Learning Theory for Reward-Modulated Spike- BoakkeTTiiin--Deenddeer Plasticity with Application Biofeedback PLOS Computational Biology.
[8] Dorit Baras and Ron Meir. Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural Computation, 19(8):2255-2779, August 2007.
[9] Frontiers Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules.
[10] Wulfram Gerstner, Marco Lehmann, Vasiliki Liakoni, Dane Corneil, and Johanni Brea. Eligi- bility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules. Frontiers in Neural Circuits, 12, July 2018. Publisher: Frontiers.
[11] James M Murray. Local online learning in recurrent networks with random feedback. eLife, 8:e43299, May 2019, Publisher: eLife Sciences Publications, Ltd.
[12] A solution to the learning dilemma for recurrent networks of spiking neurons Nature Commu- nications.
[13] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, 1010)1000-,111, 2021.
[14] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike-timing -eeeeideddeeendeentt plasticity, and theory of three-factor learning rules. Frontiers in neural circuits, 9:85, 2016.
[15] Xiaohui Xie and H Sebastian Seung. Equivalence of backpropagation and contrastive hebbian learning in a layered network. Neural computation, 15(2):441 14044444454,,, 2003.
[16] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between meergy-based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
[17] Daniel M Wolpert, Jörn Diedrichsen, and J Randall Flanagan. Principles of sensorimotor learning. Nature reviews neuroscience, 12(12):739-751 2011.
machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017. [18] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building
[19] Maurice Smith, Ali Ghazizadeh, and Reza Shadmehr. Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS biology, ((6):e179, 2006.
11

---

[20] Reza Shadmehr, Maurice A Smith, and John W Krakauer. Error correction, sensory prediction, and adaptation in motor control. Annual review of neuroscience, 33(1):89-108, 2010.
[21] Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi, Johannes Von Oswald, and João Sacramento. The least-control principle for local learning at equilibrium. Advances in Neural Information Processing Systems, 35:33603-33617, 2022.
[22] Walter Senn, Dominik Dold, Akos F Kungl, Benjamin Ellenberger, Jakob Jordan, Yoshua Bengio, João Sacramento, and Mihai A Petrovici. A neuronal least-action principle for real-time learning in cortical circuits. ELife. 12:RP89674, 2024.
[23] Bill Podlaski and Christian K Machens. Biological credit assignment through dynamic inversion of feedforward networks. Advances in Neural Information Processing Systems, 33: 130655001007 2020.
[24] Alexander Meulemans, Matilde Tristany Farinha, Javier García Ordóñez, Pau Vil- imelis Aceituno, João Sacramento, and Benjamin F Grewe. Credit assignment in neural networks through deep feedback control. Advances in Neural Information Processing Systems, 34:4674-4687, 2021.
[25] Aditya Gilra and Wulfram Gerstner. Predicting non-linear dynamics by stable local learning in a recurrent spiking neural network. Elife, 6:e28295, 2017.
[26] Alireza Alemi, Christian Machens, Sophie Deneve, and Jean-Jacques Slotine. Learning nonlin- ear dynamics in efficient, balanced spiking networks using local plasticity rules. In Proceedings of the AAAI conference on artificial intelligence. volume 32, 2018.
[27] Barbara Feulner, Matthew G Perich, Lee E Miller, Claudia Clopath, and Juan Gallego. A neural implementation model of feedback-based motor learning. Nature communications 16(1):1805, 2025.
[28] Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized teacher forcing for learning chaotic dynamics. arXiv preprint arXiv:2306.04440 2023.
[29] Radford M Neal and Geoffrey E Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pages 355-368. Springer, 1998.
[30] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference machine learning, pages 1310-1318. Pmlr, 2013.
[31] Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. Advances in neural information processing systems, 32, 2019.
[32] Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 11297-11312. Curran Associates, Inc., 2022.
[33] Matthijs Pals, A Erdem Sağtekin, Felix C Pei, Manuel Gloeckler, and Jakob H. Macke. Inferring stochastic low-rank recurrent neural networks from neural data. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[34] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-280) 1989.
[35] A Learning Algorithm for Continually Running Fully Recurrent Neural Networks MIT Press Journals & Magazine IEEE Xplore.
[36] Manuel Brenner, Florian Hess, Georgia Koppe, and Daniel Durstewitz. Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics, December 2022. arXiv:2212.07892 [nlin] version: 1.
12

---

[37] Jonas M. Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with RNNs, October 2022. urXiv:2110.07238 [es, math, stat].
[38] Kenji Doya. Bifurcations in the learning of recurrent neural networks. In Proceedings of the 1992 IEEE International Symposium on Circuits and Systems (ISCAS), pages 2777-2780 IEEE, 1992.
[39] Adam Kohn, Anna Jasper, João Semedo, Evren Gokcen, Christian K Machens, and Byron M Yu. Principles of corticocortical communication: proposed schemes and design considerations. Trends in Neurosciences, 43(9):725- 73777, 2020.
[40] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks, 2015.
[41] P.R. Vlachas and P. Koumoutsakos. Learning on predictions: Fusing training and autore- gressive inference for long-term spatiotemporal forecasts. Physica D: Nonlinear Phenomena, 470:134371, 2024.
[42] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[43] Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. Journal of Machine Learning Research, 21 11(135):1334, 2020.
[44] Stephen Grossberg, Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23-63, 1987.
[45] James M Murray. Local online learning in recurrent networks with random feedback. Elife, 8:e43299, 2019.
[46] Dong- Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propa- gation. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings. Part I 15, pages 498-515. Springer, 2015.
[47] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances in neural information processing systems, 32, 2019.
[48] David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Neural computation, 25(3):626-649, 2013.
[49] Bingni W Brunton, Matthew M Botvinick, and Carlos D Brody. Rats and humans can optimally accumulate evidence for decision making.. Science, 340(6128):95-98, 2013.
[50] Colin Bredenberg, Cristina Savin, and Roozbeh Kiani. Recurrent neural circuits overeome partial inactivation by compensation and re-learning Journal of Neuroscience, 44(16), 2024.
[51] Klavdia Zemlianova, Amitabha Bose, and John Rinzel. Dynamical mechanisms of how an rnn keeps a beat, uncovered with a low-dimensional reduced model. Scientific Reports, 14(1):26388, 2024.
[52] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 2016.
[53] Karl Friston and Stefan Kiebel. Predictive coding under the free-energy principle. Philosophical transactions of the Royal Society B: Biological sciences, 364( 364(15211:1111-222 2009.
[54] Maria Isabel Ribeiro. Kalman and extended kalman filters: Concept, derivation and properties. Institute for Systems and Robotics, 43(46))3736-3741 2004.
[55] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585, 2021.
13

---

[56] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
[57] Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2025.
[58] Sarthak Mittal, Niels Leif Bracher, Guillaume Lajoie, Priyank Jaini, and Mareus Brubaker. Amortized in-context bayesian posterior estimation. arXiv preprint arXiv:2502.06001, 2025.
[59] Brenden M Lake, Russ R Salakhutdinov, and Josh Tenenbaum. One-shot learning by inverting a compositional causal process. Advances in neural information processing systems, 26, 2013.
[60] Reuben Feinman and Brenden M Lake. Learning task general representations with generative neuro-symbolic modeling- arXiv preprint arXiv:2006. /4448, 2020.
[61] Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach learning physical dynamics. arXiv preprint arXiv: 1612.00341, 2016.
[62] Nan Liu, Yilun Du, Shuang Li, Joshua Tenenbaum, and Antonio Torralba. Unsupervised compositional concepts discovery with text-to-image generative models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2085-2095, 2023.
[63] Lea Duncker, Laura Driscoll, Krishna V Shenoy, Maneesh Sahani, and David Sussillo. Organiz- ing recurrent network dynamics by task-computation to enable continual learning. Advances in neural information processing systems, 33: 33:14387-14337. 2020.
[64] Noor Sajid, Philip Ball, Thomas Parr, and Karl Friston. Active inference: demystified and compared. Neural computation, 33(3):674-712, 2021.
[65] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[66] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 12XXvv15090009999 2015.
[67] Alexander Ororbia and Ankur Mali. Active predictive coding: Brain-inspired reinforcement learning for sparse reward robotic control problems. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3015-3021. IEEE, 2023.
[68] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 191iv:191..0/1 2019.
[69] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0417 2023.
[70] Rudolph Emil Kalman. new approach linear filtering and prediction problems. Transactions of the ASME-Journal of Basic Engineering, 82(Series D):35 45, 1960.
[71] Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin F. Grewe. A Thcoretical Framework for Target Propagation. In Advances in Neural Information Processing Systems, volume 33, pages 20024-20036. Curran Associates, Inc., 2020.
[72] Colin Bredenberg, Hzekiel Williams, Cristina Savin, Blake Richards, and Guillaume Lajoie. Formalizing locality for normative synaptic plasticity models. Advances in Neural Information Processing Systems, 36:5653-5684, 2023.
[73] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286 PMLR, 2014.
[74] Kenneth Lange. gradient algorithm locally equivalent to the em algorithm. Journal of the Royal Statistical Society: Series B (Methodologicall) 57(2):425-433 1995.
14

---

[75] Sam Roweis and Zoubin Ghahramani. A unifying review of linear gaussian models. Neural computation, 1122:305--345, 1999.
[76] Thomas Page Jr. Multivariate statistics: vector space approach. Journal of Marketing Research, 21(2):236-266, 1984.
[77] P. J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings the IEEE, 15500) 1990.
[78] David Sussillo and Larry F Abbott. Generating coherent patterns activity from chaotic neural networks. Neuron, 6344)5544-557, 2009.
[79] Yuhang Song, Beren Millidge, Tommaso Salvatori, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Inferring neural activity before plasticity as a foundation for learning beyond backpropagation. Nature Neuroscience, 27(2):348-355 2024.
[80] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint 1255,,,, 2014.
15

---

A Analyzing the hidden state impact of Error Forcing
In this section, for completeness, we give step by step derivation explaining how error foreing orthogonally projects the RNN hidden state onto the optimal readout manifold. First, notice that there are infinitely many viable teacher states r due to the dimensions of the decoder we consider W d, all
of which will produce the optimal readout y. We can take:
r
t
*min
+ 2
(14)
smin W Ft
(15)
V1 + 02V2
N,
(16)
where v, E O, NNN,,NNNNN N2 form basis for the null space of W d. and coefficients a, are arbitrary scalar coefficients. This solution works for arbitrary coefficients aj because by definition Waz = 0 Va, and W o y7. We will find the teacher state by orthogonally projecting the current state I to the manifold of possible optimal responses. To do so, we can think of the orthogonally projected vector consisting of two parts: the vector denoting the minimum-norm solution and a second vector denoting the vector needed to get from the minimum-norm solution to the projected solution. The second term can be found by taking the difference vector between the current state and minimum-norm state, and projecting it onto the nullspace of the decoder W.: This can be done by setting the in particular way:
P nulll
I - P Prow:
(17)
P ruw
W &
W
o
(18)
P
P (Fe- r.min)..
(19)
One can think of this choice of rz + z as the point closest to r, that lies on the hyperplane of optimal responses: it is the orthogonal projection of r, onto this manifold. Putting it all together:
r
r min
+ Pmmll (T: r!
(20)
r min + W W Wol(re - min
(21)
Rearranging the terms and using Eq.
15 we have:
r = [I - W W (r. wxvz) + W
(22)
w w. a r. Whyz - + w W W.
(23)
First, notice that,
W = W
(24)
Therefore,
r = [I W Wor. W W w
(25)
= [ W Wolr W.y.
(26)
Now we plug this teacher state to the generalized teacher dynamics (Eq. 5
F (1 x)) + ar; r. W
(27)
(28)
Lastly, one can arrange terms to recover the EF equations we used in the main text:
F; rr + a W (y: Wars)
(29)
= = +r e
(30)
Aside from the 'stop gradient' operation, these equations are equivalent to the EF dynamics, which shows that error foreing pushes network activity orthogonally towards the optimal readout manifold. This equation also makes it clear why EF and TF are dynamically same when Ny > N, because under this condition
In the main text, we apply a 'stop gradient' to e,, which we justify in Appendix B
16

---

W I. It is also worth noting that this approach is similar to how the Kalman Filter was first derived in the seminal paper of Kalman (70), using orthogonal projections.
Here we have compared EF to TF in terms of their dynamics. However, since it has been shown 28 that TF can solve the vanishing/exploding gradient problem by damping Jacobians, it is also useful to investigate the differences between EF and TF in terms of their effects on parameter gradients during optimization. Because of its use of stop gradients, EF does not reduce the magnitude of the network dynamics Jacobian in the same way as TF, implying that its performance improvements are due its guiding influence on network states. However, it is also worthwhile investigating what happens if we do let gradients flow through e; (explored empirically in Appendix [ When let the gradients flow, EF only dampens the dimensions the current state that are in the rowspace of the decoder, replacing this activity with the target (optimal) network state. In this case, the Jacobians become (see Eq. 28):
Ör:
Ōr 0r
(Fo(T:-1)
0T J, - W
J. 0r,-1 1 0r,-1
(31)
0r:-1
Ōr,-1
This is in contrast to TF, which dampens the activity state in every dimension (decaying activity by a multiplicative factor of (1 o.I.).
A.1 Network state updates induced by forcing versus gradient updates
The error forcing mechanism involves an optimization process in terms of both network activity states and network parameters; in fact, the error driving mechanism be viewed as form of second order optimization derived from Bayesian principles [71] (Appendix [). However, the question stands whether forcing via errors derived from first-order loss gradients could be equally more effective than using the approach we have taken in this study. In this section we derive a first-order alternative to error forcing and provide empirical comparisons our approach.
First, consider the loss function:
C
L
(32)
where L. E the instantaneous loss at each time step As commonly done, we assume that the instantancous loss does not depend the past activities, given the current activity. The RTRL factorization makes use of the instantancous loss:
OC
aC(t)
T aL, Ör. Òr, 00,
Σ 1=1
Σ
(33)
08;
09:
Where the term
0r.-1 from the previous time step. Much work has dr. is computed iteratively, using 800 år: and it's approximations. However, here we will focus on the first
80, been done in the iterative term 09;
DC because this term defines how network activity r, should change to locally improve the
term, Dr. loss. Consider instantaneous MSE loss: L, = = e,, where e, = yi- Using the chain rule,
aL. OL, Se: Qy:
(34)
Ör oc, dy: Ör
(e:) IN, W e e, W
(35)
aC. Which means that every 36, term is in the direction of the vector given by Equation 35 Consequently, activity changes induced by synaptic weight changes are in the direction of this vector. One could imagine that nudging network activity states r, in the direction ac," T W e; could improve performance during training; however, error forcing updates the dynamics via W which is different from Eq. 35 since WT 0 # W One caveat of using the transpose rather than the pseudoinverse is that parameter a can no longer be interpreted as an interpolation between the current state and the target state; only the pseudoinverse guarantees the exact target when 1. We
Dr.
17

---

empirically compared EF and this first-order error forcing approach by sweeping a over a range of values for both methods. We found that EF outperforms the first-order method (Fig. [STp). although for more complex readouts (e.g., multilayer perceptrons, nonlinear functions), first-order gradients may provide more stable solution for error foreing. In that case, we would use the first-order forcing using the following equation:
tLi Ör.
F2 r.
119
(36)
which is equal to F. = Fr- (W ex) when linear decoder combined with a MSE loss is used.
B A Bayesian Interpretation of Error Forcing
B.1 Model definition
To start, consider a noisy dynamical system, with transition function Fo(-): RN + RV parameterized by 0:
Fe (Eort, No.t, 8) = Fo r 1 0), + Ez:
(37)
where E, ~ (0,00 Notice that here we define the network dynamics treating the errors e random variables, opposed to treating r, as a random variable. This is an example of the reparameterization trick [42) which will ultimately allow us to do long-term maddient-based credit assignment, as opposed the traditional EM algorithm for Markovian state-space models, which typically uses temporally 'local' updates 722, relying on the inference distribution to implicitly perform temporal credit assignment. The network predicts targets yt E TR-, via:
Yt Wor (East, No.t, 0) + +i:
(38)
where nr NN(, 1) and We & is N₂ x N matrix, that we have:
ply Xo.) = N/ (Wor (Cu:s, No.1, 02,
(39)
These equations collectively define a stimulus-conditionee generative model for the output:
p(yor (o:T; 0) = / ply:, 1,
0) p(yo, eo xo: f)deo:r,
(40)
where p(yo, co Xx; f) specifies the network's 's initial conditions and output predictions, and of and of are free hyperparameters that will ultimately control-via -va the ratio o² are corrected by output errors, very similar in function to the Kalman gain 54
how much network states
B.2 A review of variational model fitting
Naively, to fit this generative model to distribution of outputs, would perform maximum likelihood estimation by minimizing the Hellbackk-eibbee divergence between the data distribution and the generative model, taking the loss to be:
(M) KL
(41)
For latent variable models, this loss difficult to evaluate analytically, because of the intractable integral in Eq. 40 42 73). To resolve this issue, variational methods minimize upper bound on (M) called the variational free energy (29], which has better computational properties:
(MLE(0) SKL polyorr X
Ep.(sor KL g(EorT xo.r vorr)ll(eopperoreeoooo (Xo.T, yo:r: 0)]
(42)
=KL polvo =KL[pelvorr xo. (o.r))(eo:r) xww.rloooo..rrrroloo (xo:
(43)
-LF
(44)
18

---

where (<Eorr Xo.T. Norr) is variational inference distribution over the enrors CD:T that we also optimize, and the inequality is simply due the fact that the KL divergence is strictly positive. The advantage of this approach is that (0) is function of the joint distribution p(Yo.T, 60.r Xo:7: 0), which unlike the marginal distribution (Lyy:T/XooTT 0) does not require computing an intractable integral. Practically, one selects an appropriate variational inference distribution q, and approximates this loss with Monte Carlo samples from the joint distribution pa(yo:r Xo:T (o.Eo.Tlyoo Xo:T), updating the generative model p(yo.T, E0.T Xo:T; 0) via maximum likelihood estimation.
To select a distribution 9, one typically minimizes LFe with respect some parametric family of distributions (as is the case for variational autoencoders (42), or one simply notes that LF is minimized when one takes q = p(Eo:r] No:T. Yo.r; Bold). where Bold treated as a constant with respect to subsequent optimization (as is done in variational Expectation Expectation-Muxiiizziit Maxxmmzzatiin [29] and gradient EM (74). This results in a coordinate descent procedure, where one alternates between optimizing q and 0.
B.3 Error Forcing as approximate gradient EM
In this section, we will show that Error Forcing can be viewed as an approximation of this latter approach. While it would be convenient to take q p(Cu:r(Xo:rr No:r: Oold). inference for arbitrary dynamical systems typically requires a temporally causal filternng procedure, followed by temporally acausal smoothing procedure, as required by the prevents online learning, because it requires storing a record of past network states. This is much the same problem that BPTT faces, making it unclear how either algorithm could be implemented by a biological system (in contrast to RFL.O, for instance). Furthermore, optimal filtering itself for Gaussian state space models requires tracking large covariance matrices for the inferred 6, variables; it is again unclear how a biological system could store and perform nonlocal computations (like matrix inversion) on 2 state variables.
algorithm [75] smoothing
As an approximation, we will instead do only filtering of maximum posteriori (MAP) estimates. In variational inference terms, this amounts to taking:
T
q( (CCrrIyooT, ) = II 5(c)
(45)
t=0
= =rgmax p(Erlyo:r, Xo:1, 1i)-1;0)
(46)
where 8(-) is a Dirac delta distribution. Thus, at each time step, we will infer the appropriate error correction e, given the previous network state, errors, stimuli, and observations, but we will not adjust our error estimate based on future information. After computing the estimated error correction terms G.r for a sequence, we then update parameters for the recurrent network via gradient descent on Eq. 52 It is important to stress that this approach is an approximation: because our maximization with respect to e, is performed in temporally greedy way, it cannot be viewed as global optimization of LFF as would be required for proper MAP estimation. However, empirically our approach is still quite effecttvv, and the absence of smoothing allows for exclusively online error correction.
Given the overall framework, the question now becomes: how do we calculate the filtered posterior p(celyu.t, Xu:t, c f), and maximize it with respect to c₂? Since, within our generative model, conditioned on - 1: Xu.r, and Y0---1, 1, the current time step's 's target output y, a linear function of e, the joint distribution p(c,, y: No: Xo.t, 1 0) is a multivariate Gaussian, with mean and covariance given by:
Yu:s,
Me (1y)
(wo))
) E
1 o.W,
d of w + o
#
(47)
where F, = fr-r 15 ,), x,; Now the filtered posterior amounts to conditioning this multivariate distribution on the observed output yr. For multivariate Gaussian the conditioned distribution has an analytic formula [6]:
p(E 1 XO.1 10 0) (M Eely),
(48)
19

---

where Mely and Jrly are given by:
of
Mey W WW +
= de
(y: -WoFt)
(49)
o +
Exy = of o. WW W
7 oe
W
(50)
Since the filtered posterior is a Gaussian distribution, the maximum posteriori estimate is equal to the mean. Thus we use Eq. 49 as our point estimate E of the posterior. This has the functional form of feedback error multiphed by regularized pseudoinverse of the decoder WG. that the latent network state is corrected, at each time point, by error feedback delivered from the output. These equations provide all of the information we need to perform our optimization procedure; while the pseudoinverse calculation may appear biologically implausible, previous work has shown that it can easily be estimated by regression using local 'delta' learning rule [46]. For sequence of inputs Xp:T and outputs yo:r, we iteratively compute network states using the equation:
Fr (E Xo:t; 0) [( (f X0.-- 0), xx; 0) + +
(51)
After computing errors c and network states Fo:r for a full sequence, we update parameters using backpropagation through time (77) on the loss (Eq. (52), or for additional biological realism, RFLO 45). To see how Eq. produces standard mean-squared error loss, we can substitute in our choice of variational distribution q and remove terms that do not depend on the parameters 0:
(F) =KL pa(yo:r Xo:r ) Currrrrrrroooo Xo:r)) co.r (x:r1x 8)
(52)
=I =Ppa (x)r) [- p(g. c Xo.r) + log (Yo.TT Xo.T))
(53)
-Epatrorr [- log p(Vo.T corrl [x:)]]
(54)
Here the first equivalency follows from the fact that the data distribution pa does not depend on the parameters and thus functions as an additive constant, and the second equivalence follows from the fact that the entropy of q, product of Dirac delta distributions, has constant (negative infinity) entropy that does not depend on the value of Gu:T or any network parameters (if one does not want to worry about negative infinities, distributions with arbitrarily small variance, where the same principles apply). This loss now just a negative log-likelihood over targets Yo.7 and inferred errors 60.7. Substituting in the log-likelihood of a Gaussian distribution and discarding additive constants, we have:
may simply consider q to be a product of Gaussian
T
(FE(O) = Pp
Σ
(y:
Wofi Xo.2, 0)) (y: W No.r, 0)) +
20 (Ei) (Ei) (E?)
20 t=
(55)
Σ
T
1
Nn.t T W (Ca:e, Xo.t, 01)
Epp (Xoir)
(y.
20
1=
(56)
where we may discard the final term because has no dependence on 0. Therefore, despite its apparent complexity, for our choice of the loss ( except that the errors G are used to correct internal network dynamics online.
(0) is no different from standard mean-squared error,
This approach provides useful balance: we are able to use inferred errors et to drive the network into dynamical regime that is close optimal performance before updating parameters, which has been shown to provide benefits for learning in chaotic recurrent networks [28 78 and in deep networks [79; simultaneously, we preserve temporal dependencies between parameters recurrent network. enabling the use of BPTT and its variants, which have been shown to be very powerful for fitting complex time series data distributions 80 69 conclude, we can now see that Error Forcing functions an optimization algorithm (and generalizes to the unforced test-time condition) because approximately optimizing an upper bound on LMLE(()) where the driving error signals c function as inferred error corrections similar to those observed in the Kalman Filter.
our
20

---

B.4 Error forcing with an extended Kalman Filter
In the preceding section we used greedy MAP estimates for cu:r, however these greedy estimates do not account for uncertainty regarding the proper enror forcing term at previous time steps. To resolve this issue, we can instead take:
T
( xo.)) II
((
(57)
1=0
c = =rgmaax p(e, You, Xox; 0)
(58)
G1
which is to say we now take for our filtered posterior. This approach differs from the previous greedy MAP approach (Eq. 45) in that we do not condition our probability distribution on 60:--1: our maximization is instead performed by marginalizing over uncertainty in our previous state estimates. This is still an approximation of variational EM, because we not conditioning E, targets and stimuli from future time points, but could theoretically improve performance by taking into account previous eπrrr forcing terms C0:--1-
uncertainty
In order to perform this filtered maximization approach, we need, for each timestep, access to p(Ex)) (o.,, Xo::; f). If our latent transition dynamics (Eq. 37 were linear, this filtered posterior could be computed analytically via Kalman Filtering. However, because the transition dynamics for our noisy RNN are nonlinear, we can only perform approximate inference procedure using the Extended Kalman Filter [54]. For simplicity and consistency with the literature, we will slightly adapt the Extended Kalman Filter equations by writing in terms of e, (as opposed to r,).
Under the Extended Kalman Filter, the filtered posterior pVoe Xo::: 8) [ Egje) nor- mally distributed, with mean 6, and covariance Σnj,, where the subscript t/t is used note that the state estimate time is conditioned on inputs and targets up to time t, whereas the subscript t|t 1 will be used to note that the state estimate at time is conditioned on inputs and targets up to time t t- 1. Estimates are computed iteratively, based estimates obtained at the previous time t 1. First, we define the predicted state estimates given previous state information by rolling forward the network dynamics by one time step:
Fijr-1 = Fo + 61 xx)
(59)
Σ J, 1 I²
(60)
where E gives the covariance distribution over latent states before conditioning on targets y,, and we use J, to denote the Jacobian of the transition function evaluated at r2-1/e-1 rz-1 + 1(-1/(-1. ie. we have:
J. = (T,, 0r,
(61)
P1-185-1
This Jacobian is used in order to linearize network dynamics around the mean estimate of the previous state, in order to preserve the Gaussianity of the inference distribution and the analytic tractability provides. In terms of these variables, we can then define εijt and Eget as:
t K, (y WoF(--
(62)
E. = (I - K. W E
(63)
K, E (W E + +0,
(64)
where K, is the Kalman gain. Since the maximum likelihood estimate of normal distribution is simply its mean, filtered MAP estimate becomes: E; Espt- Comparing Eqs. 62 and 49 we see that our error estimates in the two cases differ because Σet--1, through the Kalman gain K,, increases error corrections along dimensions with high latent uncertainty inherited from previous time steps. In the event that we ignore latent uncertainty inherited from previous timesteps, by taking Eqj-11 I these equations identical thereforre the error forcing method employed in this paper can be thought of as a simplified approximation of error forcing with an extended Kalman Filter.
Because this latter method requires tracking errors and covariance matrices, it is more memory- intensive, requiring records of (NN + 1) state variables, and it is unclear how single neurons
21

---

could be expected to store such variables in biologically plausible way. Therefore, as a softer approximation, one could also imagine taking Σnjt-1 diag (J+ o so that only 2NN variables would be required, with each neuron being required to track its own state and its own error-uncertainty. To our knowledge, there is little evidence that cortical neurons maintain or use such an error-uncertainty estimate. All the same, these methods may provide empirical performance improvements relative to Error Forcing in its basic form.
C Additional results
We examined the difference between using or not using the stop-gradient in Eq. The principal theoretical difference between using not using stop gradients is whether the Jacobians damped in the rowspace of the readout weights (see Eq. 31). Notice this doesn't mean that the Jacobians are always damped in one particular direction during training. because the readout weights trained through learning and hence they change over learning. We compared the performance of these two approaches the delayed XOR task (Fig. (STp), where found that
using stop gradients marginally improved performance. This experiment shows that the benefits of EF mostly come from dynamically guiding the network towards optimal state, and has less to do with Jacobian damping effects.
Lastly, we trained the networks on the sine wave generation task to examine the effects of choosing different forcing feedback weights: particular, we compared the pseudoinverse and transpose of the decoder weights (Fig. (S We used the sine wave generation task because the differences between using the pscudoinverse and transpose of the readout weights is only evident the case of multidimensional outputs. Here, found that using W produces improvements in performance relative to using the W justifying our choice for the feedback weights.
% networks learned
a
delayed XOR
b
delayed XOR
c
sine wave gen.
100
100
100
75
EF
75
TF
75
I
BPTT
50
50
50
25
25
25
0
0
1000
2000
W
BPTT
delay timesteps
al algorithm
Figure S1: a) As in Figure 3p, but without the stop gradient operation. b) As in Figure but without the stop gradient operation. e) We trained the sine wave generation task for a specific task difficulty of mean frequency 5.0, using either the pseudoinverse (using 0.01) of the readout or the transpose.
D Details task and training setup
We used leaky RNNs in the form:
A r = (1 r
A W. tanh(r,-1) W. e. x,
A
A
(65)
where Wo. for recurrent connections, Wa, for input to hidden neurons, b is the bias, which can be different for each neuron and is trained, and we used a linear decoder as in the main text. For every experiment, we used T = 10 ms, and = 50 neurons. We never trained the input-to-hidden connections, and always trained the recurrent and output connections with learning rate 0.001,
22

---

using the Adam optimizer, except for the delayed XOR task with -PPT, where we used learning rate 0.0003. For initialization, we used:
w N 0,
g N
(66)
where g is the spectral radius, which was set at 1.5, The number of input units N₂ depended on the task: was 3 for delayed XOR (two stimuli, one cue), 2 for evidence integral (two stimuli, no cue), and for sine wave generation (1 stimulus). Inputs were initialized with:
NN(O 1),
(67)
and finally the decoder weights were initialized with:
w NO
(68)
As initial values, we used
b 0
(69)
0.
(70)
We did not treat ro as trainable parameter. We used L2 regularization for neuron activities with a 0.00001 scalar, for evidence integration and the sine wave generation task, which was arbitrarily chosen and was not optimized. For every task, we trained the networks for at most 200 epochs with batch size of 128. During each epoch, 12 batches were used for testing. i.e., without forcing. We neither clipped the gradients nor used weight decay. Each batch included 512 trials for delayed XOR, 1024 trials for evidence integration, and 2048 trials for sine wave generation. The network was said to have converged if the MSE loss was under 0.1 for delayed XOR and evidence integration, and if it was under 0.01 for sine wave generation, for 10 consecutive epochs. We stopped the training when the network reached convergence according to these criteria.
For delayed XOR, the delay between the stimuli and the stimulus duration was chosen as 10 ms. For evidence integration, it was 5 ms. The delay between the last stimulus and the response period was varied to change the task difficulty; exact values are shown in the main text figures. For the delayed XOR task, the second delay perios was decreased or increased by maximum of 5 ms, chosen uniformly a given task difficulty, and the beginning of the response period was indicated with cue signal. We did not vary the last delay period for the evidence integration task, hence, the task did not require a cue signal. The distribution of stimuli the evidence integration task for left and right stimuli was chosen uniformly, with maximum of 7 signals in total. For delayed XOR, input configurations were also chosen uniformly.
For sine wave generation, for a given task difficulty, we randomly (uniformly) picked from 7 discrete frequencies for each trial, and provided the chosen frequency value to the network as a constant input. The networks were required to generate 2 sine waves (output to two different output channels) depending on the given input. These two sine waves were separated by 45 degree phase difference, which was kept fixed for every task difficulty.
The experiments were conducted using an AMD Rome CPU. Each setting was trained using 20 seeds for the delayed XOR and sine wave generation tasks, and 40 seeds for the evidence integration task.
com Our code is available at betth:t///// //tgthuu www.e/oo.//aaaeeeeee Swt///aviiiiieii Lab Code/ eeeeeeerror--re forcing
23

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes] All claims made are supported by both empirical and theoretical results discussed within the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed at length in the discussion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]

---

Justification:
Yes, proofs (along with corresponding assumptions) for all mathematical claims made in the paper are supported in the Appendices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Full descriptions of experiments run (and the network architecture used) are provided in the results section of the paper and supplemental.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code will be provided with the camera ready draft for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee httae/gggiieee/eeeeebbsssesssss /Cc////oddiiisssseeeeeee ssiiiisiicyiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Procedures used for hyperparameter and optimizer selection are described in the results section, and further information is provided in the Appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars across multiple random seeds are provided for all figures produced.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Data pertaining to the compute resources used and execution time are provided in the Supplemental.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The work in this study basic science research on synthetic datasets, and as such does not have issues pertaining human subjects research data; negative societal impacts caused by this study would require extensive additional work with the explicit intention cause harm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Both positive and negative societal impacts are described in the discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work done in this study poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Code packages used in the paper are credited in the supplemental.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There are no new assets released with this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
The method development in this research does not involve LLMs as any important, original non-standard component.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
30


Reviews:
[{"review_id": "Kvbtkx2Naw", "reviewer": "Reviewer_j5Yw", "summary": "This paper proposes a variation on teacher forcing for RNNs, which moves the internal state on each timestep toward a target state on the solution manifold (the set of states yielding zero error). The proposed Error Forcing modifies teacher forcing by picking the target state that minimizes the norm of the adjustment rather than the norm of the target state itself.", "questions": "EF and TF both train the model for a different task than it will be tested on. That is, its weights are optimized for a setting where its state is nudged by an oracle on each timestep. How does this affect generalization to the test scenario where there is no nudging? Would the model benefit from being weaned off EF in the latter phases of training (by reducing alpha toward zero)?\n\nThe greedy MAP estimation of epsilon makes the Bayesian connection somewhat superficial; it’s just the classic idea that delta-rule learning can be interpreted as the mean update of a conjugate Gaussian prior. If you wanted to more seriously pursue the Bayesian connection you could do a proper (extended) KF and carry along uncertainty in r_t (probably using a diagonal covariance matrix for computation efficiency).\n\nIt’s unclear whether the metrics in the experiments are from training or testing — specifically the % networks learned in figs 3bchi and 4abd and MSE in 3e (whereas the caption for 3f says test MSE).\n\nA similar issue comes up in Transformers, specifically causal language modeling where teacher forcing entails that LLMs are typically trained only on one step ahead prediction (e.g., arxiv:2403.06963). This goes beyond the present paper’s scope but I wonder whether EF would be helpful there.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Well motivated variation on the existing teacher forcing; retrospectively this is clearly the right way to do TF\n- Principled Bayesian motivation\n- Clear experimental advantage over TF and baselines\n\nWeaknesses\n\n- Experiments are in toy domains. Scaling up is especially important for this project because of the nature of the problem it aims to solve (vanishing gradients over long sequences).", "was_revised": true, "final_justification": "Theoretically strong and conceptually well motivated method. Experiment results are convincing for toy domains.", "rebuttal": "We thank the reviewer for highlighting both its merits (“clearly the right way to do TF”, “principled Bayesian motivation”) and the areas that would benefit from additional clarification or empirical evidence. Detailed answers below.\n\n**Toy tasks:**  While we acknowledge that the tasks we use are purposefully simple one should note that we deliberately used extremely long delays (more than twice that commonly used in past work, e.g. [1, 2]). Under these harder regimes BPTT succeeds on only ≈25 % of random seeds, while EF still solves >85 % (Figure 3). Since the task is so simple and only time delays increase, it is unequivocally clear that the BPTT failures are directly due to vanishing gradients and so EF owes its performance improvement from mitigating that. To push this point further, we have run further numerical experiments expanding this temporal horizon on delayed XOR task (delay=2500) with BPTT performance %12 +/-4 vs. EF performance %52 +-5, while TF-BPTT is %25 +/-4. This reinforces the idea of EF substantially improving on standard approaches for low dimensional tasks with very long temporal horizons.  While it would be great to expand these results to high dimensional real world applications, we feel that the main goal of this paper is conceptual, with scalability questions best left for a separate publication. \n\n[1] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. \"A solution to the learning dilemma for recurrent networks of spiking neurons.\" Nature Communications 11(1): 3625, 2020. doi:10.1038/s41467-020-17236-y\n\n[2] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown, and Uygar Sümbül. \"Cell-type–specific neuromodulation guides synaptic credit assignment in a spiking neural network.\" Proceedings of the National Academy of Sciences 118(51): e2111821118, 2021. doi:10.1073/pnas.2111821118\n\n**Training vs test in TF/EF:** While Teacher Forcing (TF) indeed introduces a train–test mismatch, EF maximizes a lower bound on the true log-likelihood by jointly adapting hidden states (corresponding to the E step) and weights (M step); nudging is therefore part of the exact training objective, not an external oracle: the same generative objective is optimised in training and evaluated in testing. The Bayesian perspective on EF thus provides a formal justification for the train-test difference in dynamics.\n\n**Annealing EF:** We did not try it for two reasons: 1) In the original GTF paper annealing always performed worse or the same as the GTF. 2) Given the Bayesian perspective on EF,  $\\alpha$ value should not be a hyperparameter but the ratio between the latent noise and the observation noise (i.e. constant over time). Nonetheless, due to Error Forcing’s approximate (variational EM) nature, reducing alpha over the training might still prove practically helpful. This potentially introduces two different hyperparameters, one for the rate of (exponential) decay, the other for the lower bound of the $\\alpha$ (potentially zero). In new numerical experiments we found that there always exists a constant $\\alpha$ value that performs better than the best performing annealing process. We therefore believe that if necessary, optimizing the $\\alpha$ value might be a better approach than optimizing the annealing process. We will add these additional experiments in the revised version of the paper.\n\n**The use of uncertainty in EF updates:** While the MAP approximation is already useful in formally justifying the form of the EF update and training/test distinction as EM, it is quite correct that discarding uncertainty information potentially comes with performance costs. \n\nAs the reviewer has noticed, tracking uncertainty through time could be accomplished via an explicit extended Kalman Filter approach, or a variational approximation of EKF that uses diagonal precision matrices for increased computational efficiency. Formally, this translates into latent dimension-specific forcing $\\alpha$ values that change dynamically through time (though practically Kalman gains stabilize relatively quickly so one could remove the temporal dependence). This would produce less forcing occurring along axes with low latent uncertainty and more along axes with higher latent uncertainty, at the cost of having to track extra state variables through time. To provide guidance for future work, we will include in the Appendix the full EKM derivation under our error reparameterization, as well as a variational approximation of the extended Kalman Filter using diagonal precision matrices for increased computational efficiency.\n\n**Metrics in the experiments:** All quantitative results in Figs 3 and 4 are always on a test set. We will add “test” to every y-axis label that lacks it, and state explicitly in the captions.\n\nThe reference on TF in Transformers is interesting, we will mention it in discussion."}, {"review_id": "o3jANQYLC2", "reviewer": "Reviewer_XnST", "summary": "The authors introduce Error Forcing, a departure from Teacher Forcing (TF) that focuses on minimal intervention in including error-based learning, while also adding gradient stopping to deal with long-term temporal dependencies.  The authors provide geometric and Bayesian perspectives to explain the methodology, as well as consider more biologically realistic alternatives. They demonstrate the learning model on a variety of tasks, constrasting it against classical BPTT (a subcase of their method) and TF.", "questions": "- Could you explain further the distinction between training and testing alluded to at the end of Section 2.3? \n- Is there a mismatch at $\\alpha = 0$ in Fig 3c with EF-BPTT and BPTT not coinciding?\n- What is going on in early training in Fig. 3F, with EF-BPTT being a bit worst at first than BPTT? What explains this difference in behavior?", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The motivation for Error forcing as minimal intervention is well-framed. \n- The geometric and Bayesian perspectives are complementary and both useful to provide intuition. \n- The authors demonstrate their inference methodology across a variety of tasks with varying difficulties. \n\nWeaknesses:\n- The argument that less forcing implies more exploration could be made stronger. Theoretical derivations, providing geometric intuition beyond full forcing or incorporating numerical experiments to corroborate would all strengthen the claim. \n- The work is fairly skim on the inclusion of other relevant works, especially on online learning from feedback errors.\n- The Bayesian perspective is insightful but would benefit from being more carefully detailed. I gather the EF forward pass emulates Kalman filtering (or potentially recursive least squares for static targets) and the variational EM comes from the alternation with the backward pass. These connections feel high-level, however, when in fact they are quite exact. I believe fleshing out these ideas even further and bringing in details from the Appendix could help the reader and strengthen the argument.", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I believe the paper is in good enough standing for acceptance. I maintain however my confidence level.", "rebuttal": "We thank the reviewer for appreciating that the provided \"perspectives are complementary and both useful to provide intuition” and that we evaluated Error Forcing “across a variety of tasks with varying difficulties”. We addressed the raised issues below:\n\n**On the ”more exploration during learning” advantage of EF vs. TF:** We agree that the demonstration of the effect could be more precise; to address this, we performed new numerical tests to empirically estimate the dimensionality of neural activity over learning, directly comparing EF and TF. The new experimental results will be added in the revised version.\n\n**On the inclusion of other relevant works:** We would be happy to discuss the broader online learning from feedback work, however, we are uncertain which specific works the reviewer has in mind. Could you please clarify the works you are referring to?\n\n**More on Bayesian perspective:** The connections to Kalman and EM are indeed formally tight and detailed in the Appendix; we can revise the text to make that even clearer in the main text. We are also adding more information on how to generalize the approach for a tighter treatment of uncertainty and the EF implications of that at the prompting of reviewer j5Yw.\n\n**Train/test:** During training, we apply error forcing to adjust network dynamics, whereas during testing, we do not employ error forcing at all. Naively, one might assume that a network trained in the presence of error forcing might come to ‘rely’ on that error signal, and would not generalize well to unforced conditions. This is also a concern for teacher forcing, which has led to approaches that gradually reduce the amount of forcing required over training [1]. We did not find such annealing empirically necessary to reach good performance for our networks. More importantly, our results formally justify the generalization performance of our trained networks: according to the principles of the variational EM algorithm, Error Forcing maximizes a lower bound on the true log-likelihood by jointly adapting hidden states (corresponding to the E step) and weights (M step); nudging is therefore part of the exact training objective, not an external oracle: the same generative objective is optimised in training and evaluated in testing. The Bayesian perspective on EF thus provides a formal justification for the train-test difference in dynamics.\n\n[1] Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 11297–11312. Curran Associates, Inc., 2022.\n\n**Mismatch in Fig 3c:**  We plot EF‑BPTT only for  $\\alpha$ ≥ 0.01; at $\\alpha$ = 0 the update is mathematically identical to plain BPTT, so the two curves coincide and we skipped the duplicate point. Even a tiny $\\alpha$ (0.01) already gives a clear boost over vanilla BPTT. Note that small $\\alpha$ values being already very effective is also noted in the appendix of GTF paper. We will make this point visually clearer in the revised version of Figure 3 and its caption to ensure there is no ambiguity.\n\n**Early training in Fig 3f:** Thank you for pointing this out. When investigating the loss curves for other tasks, we did not see the same pattern, so we believe this is an idiosyncratic behavior specific to that numerical experiment."}, {"review_id": "f3JxfzMts3", "reviewer": "Reviewer_SdUL", "summary": "To enable learning long-range dependencies in RNNs, this paper proposed Error Forcing (EF) as an algorithm to provide explicit greedy error-based guidance of circuit dynamics along with temporal credit assignment. It builds on top of teacher forcing scheme wherein the updated RNN state is a linear interpolation between with the teacher guided state ($r^*_t$) and the natural RNN state ($r_t$), i.e., the updated RNN state $\\tilde{r}_t = (1-\\alpha) r_t + \\alpha r^*_t$. The RNN non-linear hidden state recursion uses the updated hidden state, i.e., $r_t = F_\\theta(r^*_{t-1}, x_t)$. Since the output $y = W_\\phi r_t$, for teacher forcing, $r^*_t = W^{-1}_\\phi y^*_t$, if the inverse does not exist, one can opt for pseudo inverse $W^{+}_\\phi$. However, the issue with teacher forcing is that it constraints the RNN state update and activity heavily towards the teacher guidance.  \n\nIn this proposed work, Error Forcing uses the target state $r^*_t = r_t + g(W^{+}_\\phi  (y^*_t - y_t) )$, this enables exploration of the RNN hidden state along with relaxed guidance from teacher. This work analyzes the proposed error forcing mechanism through the geometric and Bayesian lens. Various empirical experiments have been done to validate the efficacy of the proposed error forcing mechanism on diverse set of tasks. When combined with BPTT, EF-BPTT was able to outperform TF-BPTT and BPTT, for delayed XOR, Sine Wave Generation and evidence integration tasks. Similar observations were achieved when replacing BPTT with RFLO paradigm.", "questions": "- In Sec. 3.1, how were the $\\alpha$ values chosen for GTF-BPTT and EF-BPTT?\n- In Fig 3c, since EF-BPTT reduces to BPTT for $\\alpha=0$, why doesn’t the percentage of network learned the same for both these curves at $\\alpha=0$ ? \n- What is the significance of $\\alpha=1$ in the update Eq. (10)? This does not translate to the equivalent teacher forcing update due to the stop gradient operation. Similarly, shouldn’t teacher forcing and error forcing achieve similar performance for $\\alpha=1$ regime?\n- In evidence int. , BPTT seems to be performing similar to GTF and EF, it’s unclear what advantage these two methods bring over the traditional BPTT on this task. \n- In supplementary (C additional results), it seems the experiments without stopping gradients seem to be obtaining decent results. If so, is there any reason to keep the stop-gradient operation in the hidden state update?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Since the teacher guidance is more relaxed compared to the teacher forcing update, Error Forcing (EF) allows the RNN to explore a better hidden state trajectory while still guided with the teacher update. \n- Error Forcing outperforms Teacher Forcing in both BPTT and RFLO optimization paradigms in the empirical experiments\n\nWeaknesses:\n- The proposed method has only been evaluated with toy experimental setups (such as delayed XOR, sine wave generation, evidence int.). It is unclear how the error forcing would perform on tasks with more complexity and higher dimension input / outputs.\n-  In the update equation, Error Forcing proposes to use the stop gradient operation to remove feedback loop back to the teacher guidance. This proposed update seems a bit ad-hoc and has not been explored in detail.", "was_revised": true, "final_justification": "Having read author rebuttal and other reviewers' feedback, I believe the proposed method has some merits which allows the RNNs to explore more relaxed trajectory. I agree with some reviewers that the tasks considered in this work for evaluation are toy examples and could be improved to showcase the benefits of the proposed work. I agree that this work has merits and I'll increase my scores to reflect leaning on acceptance on this work.", "rebuttal": "We thank the reviewer for the feedback.\n\n**Toy tasks:**  While we acknowledge that the tasks we use are purposefully simple one should note that we deliberately used extremely long delays (more than twice that commonly used in past work, e.g. [1, 2]). Under these harder regimes BPTT succeeds on only ≈25 % of random seeds, while EF still solves >85 % (Figure 1). Since the task is so simple and only time delays increase, it is unequivocally clear that the BPTT failures are directly due to vanishing gradients and so EF owes its performance improvement from mitigating that. To push this point further, we have run further numerical experiments expanding this temporal horizon on delayed XOR task (delay=2500) with BPTT performance %12 +/-4 vs. EF performance %52 +-5, while TF-BPTT is %25 +/-4. This reinforces the idea of EF substantially improving on standard approaches for low dimensional tasks with very long temporal horizons.  While it would be great to expand these results to high dimensional real world applications, we feel that the main goal of this paper is conceptual, with scalability questions best left for a separate publication. \n\n[1] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. \"A solution to the learning dilemma for recurrent networks of spiking neurons.\" Nature Communications 11(1): 3625, 2020. doi:10.1038/s41467-020-17236-y\n\n[2] Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brown, and Uygar Sümbül. \"Cell-type–specific neuromodulation guides synaptic credit assignment in a spiking neural network.\" Proceedings of the National Academy of Sciences 118(51): e2111821118, 2021. doi:10.1073/pnas.2111821118\n\n**Stop gradients:** It is important to note that the stop gradient solution is not an ad-hoc choice but rather the formal solution implied by the Bayesian perspective of EF: Because error corrections are dynamically inferred via a coordinate descent procedure on the variational free energy, they are effectively constants with respect to network parameters (hence the use of stop gradients). We do show empirically that EF works with or without the stop gradient. Nonetheless, dropping it comes at the cost of non-local gradient calculations, which makes including the stop gradient still the preferable option from a biological modeling perspective. \n\n**Hyperparameter $\\boldsymbol{\\alpha}$ selection:** Please see lines 198 and 199 of the text. “The optimal degree of forcing $\\alpha$ was determined by a grid search (note that \\alpha = 0 reduces EF-BPTT to vanilla BPTT).” This holds for both TF-BPTT and EF-BPTT and for every different task and task difficulty.\n\n**Mismatch in Fig 3c:**  We plot EF‑BPTT only for  $\\alpha$ ≥ 0.01; at $\\alpha$ = 0 the update is mathematically identical to plain BPTT, so the two curves coincide and we skipped the duplicate point. Even a tiny $\\alpha$ (0.01) already gives a clear boost over vanilla BPTT. Note that small $\\alpha$ values being already very effective is also noted in the appendix of GTF paper. We will make this point visually clearer in the revised version of Figure 3 and its caption to ensure there is no ambiguity.\n\n**$\\boldsymbol{\\alpha}$ = 1 limit:** In TF, if pseudoinverse is used, $\\alpha$ = 1 gives\n\n$r^{\\sim}\\_{t}= W^{+}\\_{\\phi} y^{*}\\_{t}$ while for EF, $\\alpha = 1$ gives (even without the stop-gradient operation):\n\n$r^{\\sim}\\_{t} = r_{t} + W^{+}\\_{\\phi} e_t = r_{t} + W^{+}\\_{\\phi} (y^{*}\\_{t} - y_{t})$ (see equation 9 in the main text)\n\nSo these two algorithms are different even when we set $\\alpha = 1$, and even when we don’t use the stop-gradient operation.\n\n**Benefits over BPTT in evidence integration task:** We have conducted additional experiments where we trained the BPTT, TF, and EF on evidence integration tasks with limited resources (less trials per epoch). In this regime we saw the importance of the EF algorithm over the other two more prominently: delay timesteps: 1000, EF-BPTT: %84 +/-8, TF-BPTT: %73 +/-8, BPTT: %52 +/-4\n\nWe believe that for this task if enough resources are provided the learning algorithms perform similarly, while in the less resource regime EF performs better."}]

Author rebuttal:

Meta-review: The paper studies learning (non-linear) RNNs with a teacher forcing variant. The paper studies this setting theoretically and empirically and shows strong performance against other RNN learning algorithms. 

Strengths: All reviewers and me unambiguously find the paper well written and the results thorough and convincing. We agree that the presented method is a well-motivated extension of teacher forcing, with obvious advantages in hindsight. 

Weaknesses: The paper is verified only on toy-ish problems empirically. 

I, and all reviewers, find the paper to present strong evidence for their claims and the paper clear and insightful. 
Furthermore, in the discussion period, the authors were able to address all open concerns.
Decision: Accept (spotlight)