Feedback control guides credit assignment in recurrent neural networks
Authors: Klara Kaleb
Barbara Feulner
Juan A. Gallego
Claudia Clopath
Venue: NeurIPS 2024 poster (2024)
Keywords: biologically-plausible learning
RNNs
motor control
feedback control
Primary area: neuroscience_and_cognitive_science
TL;DR: Feedback control may enable biological recurrent neural networks to achieve accurate and efficient credit assignment, facilitating real-time learning and adaptation in behavior generation.

Abstract: How do brain circuits learn to generate behaviour?
  While significant strides have been made in understanding learning in artificial neural networks, applying this knowledge to biological networks remains challenging.
  For instance, while backpropagation is known to perform accurate credit assignment of error in artificial neural networks, how a similarly powerful process can be realized within the constraints of biological circuits remains largely unclear.
  One of the major challenges is that the brain's extensive recurrent connectivity requires the propagation of error through both space and time, a problem that is notoriously difficult to solve in vanilla recurrent neural networks.
  Moreover, the extensive feedback connections in the brain are known to influence forward network activity, but the interaction between feedback-driven activity changes and local, synaptic plasticity-based learning is not fully understood.
  Building on our previous work modelling motor learning, this work investigates the mechanistic properties of pre-trained networks with feedback control on a standard motor task.
  We show that feedback control of the ongoing recurrent network dynamics approximates the optimal first-order gradient with respect to the network activities, allowing for rapid, ongoing movement correction.
  Moreover, we show that trial-by-trial adaptation to a persistent perturbation using a local, biologically plausible learning rule that integrates recent activity and error feedback is both more accurate and more efficient with feedback control during learning, due to the decoupling of the recurrent network dynamics and the injection of an adaptive, second-order gradient into the network dynamics.
  Thus, our results suggest that feedback control may guide credit assignment in biological recurrent neural networks, enabling both rapid and efficient learning in the brain.

Feedback control guides credit assignment in recurrent neural networks
Klara Kaleb
Barbara Feulner Department of Bioengineering Imperial College London London, UK
Department of Bioengineering Imperial College London
London, UK klaaa.aalebt@@immrrial. ac.uk
Juan A. Gallego Department of Bioengineering Imperial College London London, UK
Claudia Clopath Department of Bioengineering Imperial College London London, UK
Abstract
How do brain circuits learn to generate behaviour? While significant strides have been made in understanding learning in artificial neural networks, applying this knowledge to biological networks remains challenging. For instance, while back- propagation known perform accurate credit assignment of error in artificial neural networks, how a similarly powerful process can be realized within the con- straints of biological circuits remains largely unclear. One of the major challenges is that the brain's 's extensive recurrent connectivity requires the propagation of error through both space and time, problem that is notoriously difficult to solve in vanilla recurrent neural networks. Moreover, the extensive feedback connections in the brain are known to influence forward network activity, but the interaction between feedback-driven activity changes and local, synaptic plasticity-basse fully understood. Building on our previous work modelling motor learning, this work investigates the mechanistic properties of pre- pre-trained networks with feedback control on standard motor task. We show that feedback control of the ongoing recurrent network dynamics approximates the optimal first-order gradient with respect to the network activities, allowing for rapid, ongoing move- show that trial-by-triae adaptation to persistent
learning
ment correction. Moreover, perturbation using local, biologically plausible learning rule that integrates recent activity and eтror feedback is both more accurate and more efficient with feedback control during learning, due to the decoupling of the recurrent network dynamics and the injection of adaptive, second-order gradient into the network dynamics. Thus, our results suggest that feedback control may guide credit assignment biological recurrent neural networks, enabling both rapid and efficient learning in the brain.
Introduction
Despite the "unreasonable effectiveness" of the backpropagation (BP) algorithm (Rumelhart et al. 1986), learning in the brain must leverage a different solution (Lillicrap & Santoro, 2011 Lillicrap
al. 2020p. For instance, the brain's extensive recurrent connectivity (Felleman & Van Essen
1991 Douglas & Martin 2004) requires propagation of error through both space and time. Learning
vanilla recurrent neural networks (RNNs) with Backpropagation through time (BPTT) (Werbos)
1990,
Mozer et al. 1995 Robinson & Fallside, 1987) is known to be difficult due to the vanishing
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and exploding gradient problem (Pascanu et al. 2013 Hochreiter & Schmidhuber 1997p. This challenge is excarbated under biological constraints. which require learning rules with significant approximations of the full temporal memory (Marschall et al. 2020 Bellec et al. 2020; Murray
2019 Liu et al. 202T), and often lead to an undesirable generalization gap (Liu et al. (022).
In addition to intra-layer recurrence, the brain also features extensive feedback connections from top- down sources (Peters Payne 1993: Peters al.
1994) that can modulate network activity in lower
layers (Przybyszewski 1998 Girard et al.
2001
De Pasquale& & Sherman 2011; Briggs & Usrey
2011; Jordan & Keller 2020; Keller et al.
2020
and improve bottom-up up information processing
(Gilbert & 2 Manita et al. [2015] Fyall et al.
2017). This multiplicity of simultaneously active
pathways contrasts vanilla RNNs, where feedback connections are used exclusively for gradient
propagation.
It is precisely this simultaneous integration of several streams of information that is thought to mediate ongoing movement correction in both humans and non-human primates (Thoroughman & Shadmehr 2000; Perich al., 2018 Krakauer et al.) 2000). For example, when subjects receive distorted feedback during a learned reaching movement, such as an unexpected force acting on the limb, they are able to make movement corrections already the first trial (Thoroughman & Shadmehr 2000). Moreover, they rapidly adapt to the mismatch between the observed and expected movement trajectory upon further perturbed trials, requiring no more online corrections (Krakauer it al.] 2000). Recent dataenriven modelling work (Feulner et al. (022) proposed an elegant solution to this problem by integrating feedback control into the recurrent network dynamics, and pre-training the whole network on stereotypical motor task. The resulting networks showed rapid adaptation to task perturbation during ongoing movement and further improvement in the task performance upon persistent perturbation using a feedback-driven local learning rule, and thus reproduce key behavioural and neurophysiological findings. However, the relationship between this local feedback- driven learning and gradient descent, or any of its approximations, remains unclear.
In this work, we investigate the mechanistic properties of such recurrent networks pre-trained with feedback control on stereotypical motor task. Our key findings are listed as follows:
1. Feedback control allows for approximate learning in the activity space 3.2
2. Feedback control enables increased accuracy of approximate, local learning rules in the recurrent layer due to the "decoupling" of the network from its past activity 3.4
3. Feedhack control enables more efficient weight updates during task adaptation due the implicit incorporation of adaptive, second-order gradient into the network dynamics 3.5
Related work
In most modelling studies, the role of feedback connections is restricted to gradient signal propagation only (Bellee et al. 2200 MMrray
[urryMuurryy [019 Payeur et al. [02] Liu et al. 202I). The need for separation seems clear: in the studies where this restriction is lifted, the plasticity mechanisms involved often require tight coordination (Scellier & Bengio 2017 7 Whittington & Bogacz 2017 Sacramento et al., 2018 Payeur et al., [021 Podlaski & Machens 2020) that may be hard to implement and maintain in the brain. Furthermore, the relative influence of feedback may still be kept small to avoid interference with the forward inference.
Nevertheless, several other computational studies have shown that stronger influence of feedback on the network activities may actually aid network training (Gilra & Gerstner] 2017 Deneve et al.
2017 Alemi et al., 2018 Bourdoukan & Deneve 2015 Meulemans al. [021] 2022 a 2020). For
example, Meulemans et al. (2020) formally characterize the link between previously proposed Target Propagation (Bengio 2014 Lee al. 2015) to more efficient learning with second-order optimization methods (Gauss) 1877.. We also highlight Meulemans et al. (2022b), where the authors formally derive the relationship between gradient- based learning and optimal control through feedback activity influence for equilibrium systems. However, despite the success on a number of difficult problems, it remains unknown how well these findings extend to out-of-equilibrium dynamics commonly found in the brain. In contrast, Feulner et al. (2022) takes a more agnostic approach by pre-training recurrent networks with feedback control of their output -aquo---quilibrumm and under non-stationary conditions. However, while the resulting networks and the local learning algorithm reproduce key behavioural

---

and neurophysiological findings from both humans and non non- human primates, their relationship with gradient descent remains unclear.
Methods
2.1
Training stages
In this work, we used a modelling setup built to reproduce motor adaption experiments in monkeys (Perich et al. 2018) and similar to that used in Feulner al. (2022) (see Appendix A.6 for key differences). We construct a model that mimics both the movement repertoire and the flexibilityy of the motor cortex by applying two-stage training strategy: pre-training followed by fine-tuning.
Pre-training. To obtain stereotypical motor movement dynamics, we first train the recurrent networks with feedback control on synthetic instructed delay centre-oul reaching task using BPTT (Werbos) T990). Here, the task is to produce broad set of two-dimensional reaching velocity trajectories of varying lengths, following an instructed delay phase (see Section A. for more details). A visualization of a sub-task with 8 equidistant targets can be seen in Figure I-
Fine-tuning. To probe the recurrent network's ability to adapt to task perturbations following initial pre-training, replicate a classic visuomotor (VR) perturbation paradigm (Perich et al. 2018 Krakauer et al., 2000) in which the model output is rotated around the centre by a certain fixed angle. Motivated by progressive adaptations such persistent perturbations in the experimental studies (Perich et al. 2018), in our work perturbation fine-tuning occurs through a local, biologically plausible learning rule.
2.2 Network architecture
We train the recurrent neural networks with single recurrent hidden layer whose activity is described by the dynamics Equation [ previous network activity post-nonlincarity ((h;) and the positional error e, from the previous time step i.c. 1).
Here, hidden layer unit hy receives the stimulus signal s² the
The (-h, Σ Win +
Σ Wha((h) Ew +
In Equation 1 + represents the ReLU function, and l are the backward projecting feedback weights. The 2D output velocity of the recurrent network is obtained through linear readout of its hidden layer activity post-nonlinearity, where a, = 4(h)):
W2ya; aj + D%
2.3 Pre-training procedure
We pre-train W Wh, / 8 5² and b" using BPTT (Werbos 1990) with mean squared error (MSE) over the integrated position space using Adam (Kingma Baj 2014). To encourage the networks to learn to use the feedback layer to correct their output online, we use randomly distributed velocity perturbations (see Section A.3 for more details). The 2D error feedback signal en projected back to the network with feedback weights S the derivative of MSE of the integrated position pl = Pk + dt ui. i.e. of = rl ph. where pf the task target.
2.4 Adaptation and feedback-driver plasticity rule
To assess how the feedback integration may enable biologically plausible learning in the hidden recurrent layer during persistent task perturbation, we implement previously proposed, temporally and spatially local learning rule, Random Feedback Local Online (RFLO) learning (Murray 2019

---

BBz Bji 2 (h))aap
50 Wh: 1/2 W2 Bx
(4)
where 772 is the adaptation learning rate and a deres is hidden activity from the previous timestep i.e. (t -t-32 1). Thus, the change in weights is proportional to the eligibility trace Bjs and the feedback signal received w j&: <R- Note that when the same weight matrix W is used for both control and learning, denote this by adding +c the respective learning rule. For further biological realism, the learning batch size is fixed to and the weight updates are applied online i.c. after every presented example.
3 Results
3.1 Recurrent neural networks with feedback control adapt well to task perturbation using feedback control and local learning rules
First, we pre-train the networks with feedback control on the instructed delay menter-out-reaching task, where the networks need to produce sigmoidal velocity profiles based on the target and the go signal input given (see Section 2 for details). Once trained, we test their ability to adapt to task perturbations using the visuo-motor rotation (VR) paradigm. Here, we rotate the target positions by 30 around the centre axis, creating mismatch between the learned relationship between the input and the target. As previously shown in Feulner et al. (2022), the networks with feedback control show online adaptation to such perturbation during ongoing movement in the first trial (Figure [T]. With persistent perturbation, the networks with feedback control show further improvement in the task performance, where the adapted network readout is independent of feedback control (Figure Tb).
30
30
E
go h
(a) before plasticity
(b) after plasticity
Figure Recurrent neural networks with feedback control feature rapid adaptation to acute task perturbation and can learn the persistent task perturbation using local learning rule. A recurrent neural network with feedback control is first pre-trained produce 2D sigmoidal velocity profile based target go cue input input, then tested on a 302 target rotation. (a) The pre-trained network adapts the perturbation during the first trial with feedback control (full-line). The dashed line denotes the same network without feedback control. (b) The network further improves its performance during persistent perturbation with feedback-driven local learning (Feulner et al. 2022)
3.2 Feedback control approximates the true gradient w.rt. networks activations during task perturbation
What happens during task perturbation in the networks with feedback control that allows for such rapid adaptation? In Figure we show in networks with feedback control, the feedback contribution the overall network output increases during perturbation (see for full analysis details). Thus, during the task perturbation, the network activity is increasingly driven by the feedback signal. The average magnitude of this drive proportional to the movement perturbation magnitude (Figure (D), as the feedback drive is used to correct the recurrent network activity during perturbed trials. To further understand the mechanism behind this rapid movement correction, we investigate the alignment of the online feedback signal ! with the true first-order gradient w.r.t. the network

---

activations. We show that during predicted movement (between 50 and t = 100), the online feedback signal injected into the ongoing network dynamics upproximates well the optimal, global trial gradient Fri This adaptive behaviour is not typically observed in vanilla recurrent neural networks, where any performance feedback "locked" i.e. not available to the network activity until the end of the whole trial, and even then only indirectly so through an update in the network weights. Thus, with feedback control, the feedback is "unlocked" and corrects the network activity in real-time.
35
0"
120
stare
peok
Readout contribution
30
30
25-
feedback
100
20
155
start
punk
20-
2
%
120
30 40 perturbation
60
100
timesteps (t)
timestops (8)
(a)
(c)
Figure 2: Feedback control modulates rapid adaptation during task perturbation through approximate
learning in the activity space. (a) Increasing task perturhations degree () + 30 - 60) increases the relative feedback contribution the network readout during single task trial, which included a complete reaching movement. start denotes the start
velocity peak. The shaded region denotes the standard error of the mean over 20 network seeds. (b) The average magnitude of the feedback drive is proportional to the movement perturbation magnitude. (c) The cosine similarity (reported in degrees) of the online feedback signal injected into the ongoing network dynamics and the optimal, global trial gradient w.r.t. activations during single task trial. The shaded region denotes the standard error of the mean over 20 network seeds.
movement, and peak denotes
3.3 Feedback control and local, feedback-driven learning enable rapid network adaptation during persistent task perturbation
We next focus on network adaptation during persistent perturbation with feedback-driven plasticity only in the recurrent layer weights. Here, plasticity is governed by the temporally and spatially local learning rule that uses the learned feedback weights to project errors from the readout layer back to the recurrent layer. In addition to guiding credit assignment for the recurrent network weights, recurrent network activations are also concurrently influenced by feedback control. Thus, we refer to this as RFLO with control, i.e. RFLO+c (for more details, see Section 2.4b. Such plasticity further improves the network performance during persistent adaptation (Figure 3a. blue line, Feulner et al.) )2022)). ). Moreover, the network becomes independent of feedback control after limited number of trials (Figure 3a, grey line). In Figure Bb, we show the final test performance without feedback control (i.c. control OFF) for the different learning algorithms at varying perturbation degree magnitudes. Using this simple learning rule with online feedback control during learning (RFLO+c), networks show performance similar to those trained with online BPTT minor perturbations (Figure (3)). However, with larger perturbations, networks trained using local learning rules show an increasing performance gap with online BPTT (Figure 3)).
5

---

10-1
10a
control ON
RFLO+<
BPTT BPTT+C
control OFF
RFLO
10-1
LMSE
L
f
10-2
10-2
10-3
0
100
200
300
400
40
trials
perturbation
(a)
(b)
Figure 3: Feedback control aids local learning in the recurrent network layer. (a) Train loss as function trial number during persistent perturbation with feedhack-driven local learning. Grey line denotes the performance of the same network tested without feedhack control, showing that the network relies less on feedback control with learning. The shaded regions denote the standard error of the mean over 10 random network seeds. (b) The final performance (after 1000 trials, without feedback control) of recurrent networks trained with RFLO with (+c, blue) or without (yellow) feedback control during learning with varying degrees of persistent perturbation. The performance of same networks trained with online BPTT with (+c, orange) or without (red) feodback. The error bars denote the standard error of the mcan over 10 random network seeds.
3.4 Feedback control enables accurate recurrent weight adaptation during task perturbation
What makes online recurrent network adaptation so hard the first place, and how might feedback control help to guide learning? To answer this question, we outline the computational requirements of the true online gradient w.r.t. the network activations and the feedback-driven local learning rule (Equation 4 using the notation from Marschall et al. (2020). Using online BPTT (Werbos) [990), i.e. Real-Time Recurrent Learning (RTRL) (Williams & Zipser 1989,, one can decompose the total loss L, w.r.t. parameters W using the chain rule as follows:
Σ dh ow
dc, da, Da, OW
Σ
(5)
Oc. where 3g. C3 is the credit assignment vector and 0a. Mt is the influence matrix Marschall aw
cct al. 2020). Using RTRL, the credit assignment vector č is immediate as it is simply calculated by backpropagating the immediate eπrоr e' through the derivative of the output function F However, the influence matrix M requires the recursive computation of the past network states using the network Jacobian J+ 0a./da,-1 (for details, see Marschall et al. 2020
In this work, similar to previous work (Feulner et al., 2022 Bellec et al., 2020 Murray 2019). M is approximated with an eligibility trace B (see Equation 4). However, shown in Marschall ett al. (2020), such a severe approximation of the Jacobian can fail to capture the true temporal dependencies, instead biasing the learning trajectory towards capturing only short-term dependencies present in the neural activity at the point of learning. Depending on the task and the recurrent matrix structure, this can lead to the capturing of spurious correlations and significant generalization gap (Liu et al. 2022). However, in networks with feedback control during task perturbation, the network activities increasingly driven by the present feedback signal (Figure 24), and thus less by their past, recurrent state. We can quantify this further by calculating the norm of the Jacobian of the current network hidden activity w.r.t. past hidden activity with and without feedback control during task perturbation, which gives us an indication of current's networks sensitivity to its past activity vs the present, immediate feedback error. This is shown in Figure 4h. where we see that there is drop in sensitivity to the past activity in the networks with feedback control during increasing task perturbation. Thus, the network relies less the past and more the present, and is therefore "uncoupled", we expect the Jacobian approximation in the eligibility trace to the more accurate.
In this work, T is approximated with a pre-learned feedback matrix W which through pre- pre-training approximately aligns with the transpose of W

---

start
peok
1.0
local gradients
global gradients
0.8
1.25
0.8
0.7
|| 10a
cosine similarity
cosine similarity
0.6-
1.20
0.6
0.5
1.15
0.4
0
0.4
30
f
RFLD+c RFLO
0.3
1.10
60
0.2
0.2
20
40 timesteps (U)
60
80
100
120
30
40
50
60
60
perturbation
perturbation
(b)
(c)
Figure 4: Feedback increases the accuracy of local learning. (a) The norm of the Jacobians of the network activities at time t w.r.t to activitics at previous timestep during single trial. The shaded regions denote the standard deviation over 10 random network seeds. (b) The mean local alignment of the feedback-driven local learning rule with the true local gradient during task adaptation. The error hars denote the standard deviation over 10 seeds. The error bars denote the standard deviation over 10 random network seeds. (c) The mean global alignment of the feedback-drive driven local learning rule with the true global gradient during task adaptation. The error bars denote the standard deviation over 10 random network seeds.
We empirically validate this hypothesis by comparing the gradients of the local learning rule with that of online BPTT during task adaptation, with (RFLO+c) and without (RFLO) feedback control. Note that we calculate both the local, i.e. at each learning step, and global, i.e. after all the learning steps, gradient alignments. We show that while there is only a minor difference in the local gradient alignements, this compounds at the global level, where the local learning rule in the networks with feedback control aligns better with the true, global gradient than without feedback control (Figure (f-c). This may explain the better performance of the networks with feedback control during task adaptation (Figure as the local learning rule is more accurate.
3.5 Feedback control enables efficient recurrent weight adaptation during task perturbation
To further understand the impact of feedback control on weight-based learning, we next focus on the relative magnitude of weight updates during task adaptation. Specifically, for each learning episode consisting of a limited number of trials, we can calculate the magnitude of final, global weight update || globaill, as well the total combined magnitude of the online, local weight updates > W The ratio of the two quantities allows us to quantify the efficiency of the online weight updates during task adaptation. For instance, the less erratic the online weights updates are, the more efficient the learning trajectory, and our calculated ration will be closer to 1. Despite the relatively minor difference in local gradient accuracy (Figure 4p), we show that our pre-defined "efficiency ratio" is significantly different between the network with and without feedback control during learning, all tested perturbation magnitudes (Figure 5b)). Note that we don't 't observe this difference with more accurate gradients i.c. with BPTT (Figure 10b), but we do also observe it on an additional toy task (see Figure TT). Thus, with online feedback control, local weight updates are more efficient and less sensitive to the noise inherent to online learning using pseudo -ggdddnnsss
What could be the mechanism behind this marked increase in efficiency of the online weights updates with feedback control? In our networks, the feedback-controlled network activities are more constrained during task perturbation, where the network is initially more driven by the feedback signal that approximately aligns with their first-order gradient (see Figure 2c), irrespective of any weight based learning. This can be seen as a form of local adaptive inference in the activity space, or "prospective configuration" (Song et al. (002),, that can guide subsequent update steps in the weight space. Furthermore, the efficiency of learning often linked to the network's 's sensitivity to its higher-order gradients, which equips it with better navigation of pathological curvatures in the objective function (Martens et al.) 2010). To assess this our networks, we calculate the aligmment of the online feedback control signal injected into the network dynamics with the exact gradient used in the canonical 2nd order optimization scheme, Newton's 's method. Here, the gradient defined as (H - - where H is the Hessian w.r.t. the network activations, 7 is some small constant that re-conditions and g is the 1st order gradient. We show that during peaks of movement error, the feedback control signal injected into the ongoing network dynamics is also sensitive to their 2nd
7

---

order gradient (Figurdp,, see also Figure 10ap. Thus, feedback control also implicitly injects adaptive, second-order gradient information into the network dynamics, which may explain the magnitude of the increased efficiency observed in online weight updates during task adaptation.
0.30
RFLO+C
RFLO
start
peak
0.25
100
Σ A Wioaat
0.20
AW global
80
0.15
60
0.10
40-
0.05
20
ac
0o
0.00
40
50
60
0
20
40 60
80
100
120
perturbation
timesteps
(a)
(b)
Figure 5: Feedback increases the efficiency of local learning. (a) The mcan norm or efficiency ratio of the total, global weight update with that of the online, local weight update during task adaptation at various perturbation degrees. The error bars denote the standard deviation over 10 random network seeds. (b) The cosine similarity between the online feedback control signal injected into the ongoing network dynamics and the second-order gradient (calculated using the Newton method) w.r.t. activations (blue) during a single trial. The grey line denotes the first-order gradient as in Figure Zp. The shaded region denotes the standard error of the mean over 10 network seeds.
4 Discussion
Summary. Although feedback connections are known to influence forward network activity in biological networks, the interaction of such feedback-driven activity for control with weight-based learning is less understood. In this work, we investigated the mechanistic properties of recurrent neural networks pre-trained with online feedback control using BPTT and tested on a task perturbation, which can also be seen as task modification, originally proposed in Feulner et al. (2022). We show that the online adaptation of such networks during task perturbations feedback drive of the network and (2) the approximate alignment of the feedback signal with the true first and second-order gradient w.r.t. the network activations. Furthermore, we show that the consequence of such control is that the temporally and spatially local weight updates during task adaptation are both more accurate, robust and efficient.
due to (1) the increased
Feedback control and learning. Our results are in line with recent theoretical work showing that injecting control signals into network activity may aid local, biologically plausible learning in systems at equilibrium (Meulemans et al.] 2022b). Here, we build on this work by showing that such strict equilibrium conditions can be relaxed in the presence of learned online feedback control during an ongoing network trial. Moreover, we show that the feedback control integration in the activity space also implicitly injects adaptive, second-order gradient information into the network dynamics, which may explain the magnitude of the increased efficiency observed in online weight updates during task adaptation. This is also line with recent theoretical (Innocenti et al., 2023 Alonso al. 2022) work linking network inference in predictive coding networks with the second-orde trust region optimization methods (Conn al., 2000), Dauphin al. 2014 Yuan 2013). Here, unlike in standard line-searce methods, "safe" region in the loss landscape is first determined, before the actual weight updates within this region are calculated and applied. As in Innocenti et al.) (2023). we show that in our networks this "safe" region could be implicitly calculated through adaptive, feedback- inference in the activity space. Importantly, in our networks this is done without an additional relaxation phase nor any explicit, second-order gradient calculation, and thus without the additional, often prohibitive, computational cost associated with such methods compared to a standard forward and backward network pass in BPTT.
8

---

Limitations & future work. In this work, we demonstrate the benefits of feedback control on simple, biologically relevant task in single layer RNN. Further work is needed investigate the generality of our findings to wider range of tasks and network architectures. Moreover, we remain agnostic to the exact biological implementation of the feedhack signal (probably via cerehellum, see Pemberton et al. (2022)) and the local learning rule (but see Aceituno et al. (2023)). Finally, the feedback form used in this work is a simple, linear projection of the signed error signal back to the network, and anticipate that this may not hold in many brain regions. We believe it would be interesting both for computational neuroscience and machine learning to investigate the impact of less explicit feedback mechanism. Thus, extending our results to learning conditions with increased generality is an important direction for future work.
Impact statement
This paper presents work whose goal advance our understanding of fundamental principles of biological motor control. We recognize that this could have far-reaching implications in the future, particularly in the fields of robotics and prosthetics. Beyond these applications, this research may also enhance our understanding of motor disorders, potentially leading to improved rehabilitation strategies. Careful ethical considerations throughout our research are vital to proactively navigate the benefits and potential challenges these advancements may introduce to society.
6 Acknowledgements
This work was supported by the Wellcome Trust [grants 200790/2/16/Z and 219995/2/19/2J, the Simons Foundation [grant 564408), and EPSRC [grant EP/R035806/1), We gratefully acknowledge the computational resources and support provided by the Imperial College Research Computing Service (ttp://doi.orr//101444////))) 4469/hpc/2232), We also thank Dr. Tudor Berariu for the insightful discussions throughout this project and the reviewers for their constructive feedback, both which greatly improved the quality of this paper.

---

References
Aceituno, V., Farinha, M. T., Loidl, R., and Grewe, B. F. Learning cortical hierarchies with temporal hebbian updates. Frontiers in Computational Neuroscience, 17: 136010, 2023.
Alemi, A., Machens, C., Deneve, S., and Slotine, J.-J. Learning nonlinear dynamics in efficient, balanced spiking networks using local plasticity rules. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Alonso, N., Millidge, B., Krichmar, J., and Neftci, E. O. A theoretical framework for inference learning. Advances Neural Information Processing Systems, 55:37335-377353 2022.
Bellec, G., Scherr, F., Subramoney, A., Hajek, E., Salaj. D., Legenstein, R., and Maass, W. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1): 3625, 2020.
Bengio, How auto-encoders could provide credit assignment deep networks via target propaga- tion. arXiv preprint arXiv:1407.7999 2014.
Bourdoukan, R. and Deneve, S.
Enforcing balance allows local supervised learning in spiking recurrent networks. Advances in Neural Information Processing Systems, 28, 2015.
Briggs, F. and Usrey, W. M. Corticogeniculate feedback and visual processing in the primate. The Journal of physiology, 589(1):33 11:3):333303 40, 2011.
Conn, A. R., Gould, N. I., and Toint, L. Trust region methods. SIAM, 2000.
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high- high-dimensiioal non-convex optimization. Advances in neural information processing systems, 27, 2014.
De Pasquale, R. and Sherman, S. M. Synaptic properties of corticocortical connections between the primary and secondary visual cortical areas in the mouse. Journal of Neuroscience, 3 (46): 16494-16506, 2011.
Denève, S., Alemi, A., and Bourdoukan, The brain as an efficient and robust adaptive learner. Neuron, 94(5):96-9977, 2017.
Douglas, R. J. and Martin, K. A. Neuronal circuits of the neocortex. Annu. Rev. Neurosci., 27: 419-451, 2004.
Felleman, D. and Van Essen, D. C. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 199/), 1(1):1-47, 1991.
Feulner, B., Perich, M. G., Miller, L. E., Clopath, C., and Gallego, J. A. Feedback- based motor control can guide plasticity and drive rapid learning. bioRxiv, pp. 2022-10, 2022.
Fyall, A. M., El-Shamayleh, Y., Choi, H., Shea-Brown, E., and Pasupathy, A. Dynamic representation of partially occluded objects in primate prefrontal and visual cortex. Elife, 6:e25784, 2017.
Gauss, C. F. Theoria motus corporum coelestium in sectionibus conicis solem ambientium, volume 7.
FA Perthes, 1877.
Gilbert, C. D. and Li, W. Tow-down influences visual processing. Nature Reviews Neuroscience, 14(5):350-36322)) 2013.
Gilra, A. and Gerstner, W. Predicting non-linear dynamics by stable local learning in a recurrent spiking neural network. Elife, 6xe28295, 2017.
Girard, P., Hupé, J., and Bullier, J. Feedforward and feedback connections between areas v1 and v2 of the monkey have similar rapid conduction velocities. Journal of neurophysiology. 85(3): 2001.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
10

---

Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 98))11335-1788 1997.
Innocenti, F., Singh, R., and Buckley, C. Understanding predictive coding as second-order trust- region method. In JCML Workshop on Localized Learning (LLW), 2023.
Jordan, R. and Keller, G. B. Opposing influence of top-down and bottom-up input on excitatory layer 2/3 neurons in mouse primary visual cortex. Neuron, (08(6):114-11000 2020.
Keller, A. J., Roth, M. M., and Scanziani, M. Feedback generates a second receptive field in neurons of the visual cortex. Nature, 582(7813):5445-555 2020.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint 14X2.6:412.694 2014.
Krakauer, J. W., Pine, M., Ghilardi, M.-F. and Ghez, C. Learning of visuomotor transformations for vectorial planning of reaching trajectories. Journal of neuroscience, 20(23):8916 6888999446944444 2000.
Lee, D.H.., Zhang. S., Fischer, A., and Bengio. Y. Difference target propagation. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-1 2015, Proceedings, Part 155, pp. 498-515. Springer, 2015.
Lillicrap. T. P. and Santoro, A. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82 89, 2019.
Lillicrap. T. Santoro, A., Marris, L., Akerman, C. J., and Hinton, G. Backpropagation and the brain. Nature Reviews Neuroscience, 21 26)633335-346 2020.
Liu, Y. H., Smith, S., Mihalas, S.. tion guides synaptic credit assignment spiking neural network. Proceedings of the National Academy of Sciences, 118(51):e211822)) 1112))11111 2021.
E., and Sümbül, U. Cell-tyee-speiific neuromodula-
Liu, Y. H., Ghosh, A., Richards, B., Brown, E.,, and Lajoie, G. Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules. Advances Neural Information Processing Systems, 35:23077-2300 2022.
Manita, S., Suzuki, T., Homma, C., Matsumoto, T., Odagawa, M., Yamada, K., Ota, K., Matsubara, C., Inutsuka, A., Sato, M., al. A top-down cortical circuit for accurate sensory perception. Neuron, (6(5):1304-1111
2015.
Marschall, O., Cho, K., and Savin, C. A unified framework of online learning algorithms for training recurrent neural networks. The Journal of Machine Learning Research, 21(1):5320-533 2020.
Martens, J. et al. Deep learning via hessian-free optimization. In lcml, volume 27, pp. 735-742, 2010.
Meulemans, A., Carzaniga, F., Suykens, J., Sacramento. J., and Grewe, B. F. A theoretical framework for target propagation. Advances Neural Information Processing Systems. 33:20024-20036. 2020.
Meulemans, A., Tristany Farinha, M., Gareía Ordóñez, J.. Vilimelis Aceituno, P., Sacramento, J., and Grewe, B. Credit assignment in neural networks through deep feedback control. Advances in Neural Information Processing Systems, 34:4674 2021.
Meulemans, A., Farinha, M. T., Cervera, M. R., Sacramento, J., and Grewe, B. F. Minimizing control for credit assignment with strong feedback. In International Conference on Machine Learning, pp- 15458-15488. PMLR, 2022a.
Meulemans, A., Zucchet, N., Kobayashi, S., Von Oswald, J., and Sacramento, J. The least-control principle for local learning equilibrium, Advances Neural Information Processing Systems, 35:33603-33617, 2022b.
Mozer, M., Chauvin, Y., and Rumelhart, D. Backpropagation: Theory, architectures. and applications. Chapter A Focused Backpropagation Algorithm for Temporal Pattern Recognition, Erlbaum Associates Inc., Hillsdale, NJ, USA. pp. 137-169, 1995.

---

Murray, J. M. Local online learning in recurrent networks with random feedback. Elife, 8:e43299, 2019.
Pascanu, R., Mikolov, T., and Bengio, On the difficulty training recurrent neural networks. In International conference machine learning, pp. 1310-1318, Pmir, 2013.
Payeur, A., Guerguiev, J., Zenke, F., Richards, B. A., and Naud, R. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, 24(7):0010-1099 2021.
Pemberton, J., Chadderton, P., and Costa, R. P. Cerehellar-driven cortical dynamics enable task
acquisition, switching and consolidation. bioRxiv, pp. 2022-11, 2022.
Perich, M. G., Gallego, A., and Miller, L. E. neural population mechanism for rapid learning. Neuron, 100(4):964-977) 2018.
Peters, A. and Payne, B. R. Numerical relationships between geniculocortical afferents and pyramidal cell modules in cat primary visual cortex. Cerebral cortex, 3(1):69-78, 1993.
Peters, A., Payne, B. R., and Budd, J. A numerical analysis the geniculocortical input to striate cortex in the monkey. Cerebral Cortex, 4(3):215-229, 1994.
Podlaski, B. and Machens, C. K. Biological credit assignment through dynamic inversion of feedforward networks. Advances in Neural Information Processing Systems, 33:10065-10076, 2020.
Przybyszewski, A. W. Vision: Does top-down processing help us see? Current Biology. 8(4): R135-R139, 1998.
Robinson, A. J. and Fallside, F. The utility driven dynamic error propagation network, volume 11.
University of Cambridge Department of Engineering Cambridge, 1987.
Rumelhart, D. E., Hinton, G. E., and Williams, R. Learning representations by back- back-propagating errors. nature, (6688))) 1986.
Sacramento, J., Ponte Costa, Bengio, Y., and Senn, W. Dendritic cortical microcireuits approximate the backpropagation algorithm. Advances in neural information processing systems, 31, 2018.
Scellier, B. and Bengio, Y. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers computational neuroscience, 11:24, 2017.
Song, Y., Millidge, B., Salvatori, T., Lukasiewicz, T., Xu, Z., and Bogacz, R. Inferring neural activity before plasticity: A foundation for learning beyond backpropagation. bioRxiv, pp. 2022-05, 2022.
Thoroughman, K. A. and Shadmehr, R. Learning of action through adaptive combination of motor primitives. Nature, 407(6805)::42-777,, 2000.
Werbos, P. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 15510)0 1990.
Whittington, J. C. and Bogacz, R. An approximation of the error backpropagation algorithm in predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5): 1229-1266 1262, 2017.
Williams, and Zipser, D. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280 1989.
Yuan, Y.-x. Recent advances in trust region algorithms. Mathematical Programming, 151:249-221 2811, 2015.
12

---

A Detailed methods
A.1 Task
To create a synthetic instructed delay centre-out reaching task as experimentally performed in monkeys (Perich al. 2018), we set both the trajectory start (p*tart R²) and end point (pond IR?) by randomly sampling each coordinate independently from a uniform distribution U U'(66, 6) and interpolate the corresponding velocity trajectories [Uz, Byy with a sigmoid function parametrized by 10/s (Equation 6).
f(t) 1 exp(-tx)
(6)
This creates natural, bell-shaped reaching velocity profiles. Each trial lasts 1.25 (125 steps) and features (fixed) instructed delay period eho of 0.2 s, included in the network as Heaviside step function that is non-zero during the delay period. Thus, the input and output task dimensionalities 3 and 2 respectively. The radius of the circle for the center-out-reac sub-task used in visualisations set to 5 em.
A.2 Network
Here, we use recurrent neural networks with 400 hidden units (but also see Figure The weights (W Wh W W and biases (b*. b²) of the networks are initialized from a uniform distri- bution U U-11/Vi, 1/vl) (He et al. 2015) where I either the number of layer parameters (for " Wh W b²) or the dimensionality of the output (for bo)."). The hidden network activity is integrated using Equation I, with di 10 and 50 ms.
A.3 Pre-training
We pre-train the network parameters using the Adam optimizer (Kingma & Ba, 2014) with learning 0.9, 83 0.999). batch size of 256 and L.2 regularization for network parameters (3 = le 3 and activations (? = 23-3) for 5000 independently drawn data batches i.e. trials. The total norm of the gradients clipped 0.2. Here, we eliminate the separate parameter training phases used in Feulner et al. (2022) and keep all network parameters plastic throughout the initial training. We also perturb the network output during the entire pre-training stage by adding brief (0.1 s) velocity pulses with an amplitude of 10 cm/s at random in 25% of the trials.
rate 7/1 0.001
A.4 Fine-tuning
During persistent perturbation, weight updates after each trial timestep as per Equation 4 As per the learning rate sweep shown in Figure 8 the fine-tuning learning rates 1/2 used for each learning algorithm are: 1e-4 (BPTT+c), 5e (BPTT) and 5e (RFLO, RFLO+c). Note that for the analysis in the Figure 4 we use f/2 = le
the SGD optimizer and the batch size of and modify the
A.5
Computational requirements
We ran all our experiments using NVIDIA GeForce GPUs (RTX 2080 Ti). The code, adapted from (Feulner et al. 2022), is available at: http://ggthb...omkkkknnbbbllleee
A.6
Key differences from the modelling setup in Feulner et (002
Task Simplification: We simplify the task by fixing the go signal timing, facilitating clearer analysis of within-trial adaptation mechanisms.
Curriculum Streamlining: We single-phase training curriculum instead of the three-phase approach, again focusing on the core mechanisms of interest.
Pre-training batch size increase: We use larger batch size than in Feulner et al. (022) (20 246), for faster convergence of networks with and without feedback.
13

---

Online feedback: We provide truly online feedback, removing the moologicall-mottvvan feedback delay used in Feulner et (2022).
We shorten the individual trial lengths (300 to 125) to improve the distribution target values Feulner et al. (2022), most of the target values are set to 0 due to the steepness of the velocity sigmoid.
Shorter Individual Trials:
Local Learning Rule: We improve the biological plausibility of the learning rule by making it fully online (instead of accumulated) with continuous eligibility trace (see Equation
4h. We also add @' (a,) to the eligibility trace, which makes equivalent to that of RFLO learning (Murray 2019) and increases its stability with larger perturbations (see Figure 7
(a) 30"
(b) 40"
(c) 50
(d) 60
Figure Performance of networks trained using the local rule from Feulner al. 2022 with increasing degrees
of perturbation.
(a) 30
(b) 40
(c) 50%
(d) 60
Figure 7: Performance of networks trained using RFLO learning from Murray 2019 with increasing degrees of perturbation.
A.7 Analysis details
To quantify the percentage readout contribution of the immediate feedback control signal in Figure
2h. we revisit Equations [ [nddd] that govern the network hidden and readout dynamics, respectively. Combining the two into single equation, we get:
0k Σ W&ya aj + b4
= Σ We, ((hs) + bi
(8)
Σ We $ dt
ff D Win
Σ
Σ who(h)) + Σ W cs
+b²
+ b%
(9)
Then, the percentage of the immediate feedback control signal % contribution to the network output calculated as:
dt Σ
100
percentage feedback contribution Σ1h, Iby
we,
x
(10)
Σ, Wg
Note that for Figure 2 report average value over k.
14

---

OFTT
0.033
0.011
0.033
OPTT
0.037
0,032
0.041
Cysse
Cyss
Leamiing algarthmm
BFTT-c
0.57
0.022
0.013
Leaminn algarthm
BPTT-c
0.55
0.039
0.016
AFLD
0.051
0.029
0.023
0.046
AFLD
0.37
0.13
0.05
0.031
0.097
RFLO
0.42
0.05
0,012
0.76
0,026
0,016
-10-2
55.05 0.0002
0.0001
Learning rate
Learning rate
(a) 30
(b) 40"
BPTT
0,23
0.04
0.111
0.024
0:23
BPTT
0.039
0.03
0.009
0.26
LNGE
CASE
Learino algoortbi
Leamine algorithi
1.8
0.089
0.008
0.012
2.5
0.15
0.12
0,034
0.65
0.076
n.022
0.0K5
0.094
1.1
0.098
0.31
0.074
0.37
IFLO-E
2.1
1:2
0.051
02222
IFLLOH
3.1
1.9
0.043
-10-2
-10-2
So-07
So:07
Leariin rate
Learning rate
50"
(d) 60
Figure 8: Adaptation learning rate sweep using different learning algorithms with increasing degrees of persistent perturbation.
109
100
BPTT
0.012
0.01
0.013
BPTT
0.016
0.012
0.017
LASE
LusE
Learning algorithm
Learning algorithm
BPTT+c
0.0091
0.0095
0.0092
BPTT+C
0.01
0.011
0.01
-10-1
-10-
RFLO
0.03
0.023
0.02
RFLO
0.038
0.031
0.023
RFLO+C
0.11
0.012
0.0099
RFLO+C
1.4
0.016
0.013
-10-2
-10-2
200
400
800
200
400
800
hidden size
hidden sire
(a) 30
(b) 40"
Figure 9: Hidden network size sweep for the different learning algorithms with increasing degrees of persistent perturbation. The adaptation learning rates for each learning algorithm le (BPTT+c), 5e (BPTT) and 5e (RFLO, RFLO+c), as per Figure
15

---

120
0.30
100
BPTT+c
0.25
alggmment
WWwolll
80
A globai
0.20
60
0.15
40
UE ZWP--1
0.10
20
Ba 4-
8C
90
0.05
0
0.00
20
40
60
80
100 120
30
40
50
60
timesteps
perturbation
(a)
(b)
Figure 10: (a) Baseline measured alignment between the 1st and 2nd order gradient (yellow line), as in Figure
2c and 5b (b) Learning efficiency for BPTT with (orange) and without (red) control, in Figure 5a
1.0
0.8
RFLO
AW stciel v [toesll
0.6
0.4
0.2
0.5 freqshift
(a)
(b)
Figure 11: (a) Sine and Cosine wave task: given some input frequency f sampled uniformly from U(1, 4), network has to generate a 2D output containing both the Sine and the Cosine with the same given frequency. Note we mask the first quarter of the period for the Cosine wave to ensure task smoothness. The corresponding perturbation in this task is a frequency shift here, the desired generated frequency is increased by some constant. (b) Learning efficiency during persistent perturbation (+0.5 Hz) for the and cosine wave task for RFLO with (blue) and without (yellow) control, Figure Here, 1/2 is set 1e-5.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract effectively sets up the problem, outlines the specific approach taken, and clearly states the key findings and implications.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper does discuss the limitations of the work performed by the authors. mentions the need to investigate the generality of findings across different tasks and network architectures, remains uncertain about the exact biological implementation of the feedback signal, and acknowledges that the simple, linear projection used in the study may apply to many brain regions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
The paper does not provide theoretical proof for its results because the focus is primarily empirical findings. Instead of theoretical results, the paper presents empirical evidence to support the benefits of feedback control in single-layer RNNs on a specific task. The assumptions and methodologies for these empirical investigations clearly outlined in the methods section.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results. It provides detailed descriptions of the experimental setup, including the tasks, network architectures, feedback control mechanisms, and learning rules used. We also provide the code used.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
18

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide access to the (anonymised) code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
cotpsicynnoooyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. httpp/cccppblcc/uuuuueue httpc//gguddeeeooeouuuuumbibea wttpee///oooeeeuueeiiiiioomee
asionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All the details are included in the supplementary information section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Wherever error bars are reported, they are clearly denoted in figure captions.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is included in the supplementary information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: There were human subjects involved in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The clearly state the potential societal implications of our work the end of the discussion section.
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: As our work focuses on the mechanistic interpretability of the model from Feulner et al. (2022). we clearly cite it when required and include clear checklist of any methodology changes made in the supplementary information.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package dataset.
The answer NA means that the paper does
The authors should state which version of the asset used and, possible, include URL.
21

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the balaaets.ppaerssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only asset we provide is the code, which is documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "mwxnT896na", "reviewer": "Reviewer_mASo", "summary": "The paper studies the effect of feedback control on motor learning in recurrent neural networks, finding that feedback control improves learning performance and better aligns with the true gradient w.r.t. the task.", "questions": "What is the difference between RFLO and RFLO+c? Does the first lack the feedback term in Eq. 1? This should be clearly stated within Sections 2.2-2.4.\n\nLine 141: “outout”\nLine 141: “ is increasingly by”", "limitations": "The authors have addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Alignment with the true gradient is an interesting result and helps explain why feedback works\n- The authors study alignment from different perspectives (e.g. step-wise/full gradients, Newton method)\n- The task the authors consider is widely used in monkey experiments, therefore it should be possible to adapt the conclusions to real data or use them to guide new experiments", "weaknesses": "- The training setup is rather limited; it would be interesting to see training done for other tasks and architectures (or RNN sizes).\n- The paper might benefit from some theoretical analysis of why the feedback signal alings with the true gradient, although it’s not clear if that can be easily done.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.\n\n> The training setup is rather limited; it would be interesting to see training done for other tasks and architectures (or RNN sizes).\n\nThank you for raising this important point. We also trained RNNs on an additional, commonly used task, where the network has to generate both the sine and cosine wave given some input frequency (see cartoon in extra figure E). Here, the relevant perturbation is a frequency shift. Our adaptation to perturbation results remain qualitatively the same (extra figure F and G), expanding the generality of our method.  We also show results for different RNN sizes (extra figure C, final adaptation loss as a function of learning algorithm and hidden layer size) - where we mostly see the expected increase in accuracy post adaptation with increased network capacity. We will include both of these results as an additional supplementary figure in the final manuscript.\n\n> The paper might benefit from some theoretical analysis of why the feedback signal alings with the true gradient, although it’s not clear if that can be easily done.\n\nGiven that the feedback weights approximately align with the transpose of the forward weights and the feedback is a linear projection of the loss function derivative (the error), the (approximate) alignment of the feedback with the gradient of the network activities is somewhat expected. However, why this would aid local learning is not entirely clear. In Section 3.4 and Figure 4 of our work, we address this question empirically by demonstrating that the output of the networks with feedback control relies less on past and more on present states. This reliance reduces the bias introduced by the severe Jacobian approximation by the eligibility trace, as studied by Marchall et al. (2020), making it more accurate. We acknowledge that further theoretical analysis could provide additional insights and leave this exploration for future work.\n\n> What is the difference between RFLO and RFLO+c? Does the first lack the feedback term in Eq. 1? This should be clearly stated within Sections 2.2-2.4.\n\nThank you for highlighting the lack of clarity in Figure 3(b). You are correct in that +c indicates the presence of feedback in Eq.1 during adaptation to persistent perturbation via local or non-local learning. We now clarify this in both the figure caption and methods section 2.4:\n\n\"Note that when the same weight matrix $\\boldsymbol{W^{fb}}$ is used for both control and learning, we denote this by adding $+c$ to the respective learning rule.\"\n\n> Line 141: “outout” Line 141: “ is increasingly by”\n\nThank you for catching these typos!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "THjocgjjx9", "reviewer": "Reviewer_GeF2", "summary": "Recent work has shown that feedback signals can be critical to rapid adaptation in control tasks, and may explain how biological intelligence can make rapid adjustments when solving such tasks. This paper studies how feedback control achieves this. To do so, the authors train an RNN enhanced with feedback control on a common control task, and study how the feedback signal lead the network to achieve more rapid adjustments when perturbations are introduced. The 3 main findings are that the feedback signals align well with the optimal global gradient of the error, that they help the network better weigh current information (vs. less relevant past information) during perturbations, and that they indirectly inject second-order information to the RNN.", "questions": "- How does this work relate to hierarchical predictive coding, and to the feedback connections introduced by Hinton in [1] (and further explored by [2])?\n- The learning setting presented in this work seem very similar to the setting of reinforcement learning, which also deals with control tasks and shifting distributions. Do you foresee these same results (i.e. feedback control improves performance) to carry over to some RL tasks? If not, what are the differences that limit these results from applying there?\n\n[1] Hinton, G. (2022). The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345.\n\n[2] Ororbia, A., & Mali, A. (2023). The predictive forward-forward algorithm. arXiv preprint arXiv:2301.01452.", "limitations": "I'd like to see an expanded discussion in the limitations section regarding the remaining aspects of the feedback-enhanced RNN that remain biologically implausible. Particularly, the usage of BPTT in a work that aims to explain how biological credit assignment is performed is quite troubling for me, given its significant biological implausibility. Ideally, I think the authors should show that their results hold on a network trained using a biologically-plausible learning rule enhanced with feedback control.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- This work focuses on improving the theoretical understanding of an important method. Given that our understanding of many deep learning methods are woefully inadequate, such work is critically important for the field's development.\n- The method and the results are clearly presented, the figures are excellent, and the writing is easy to follow.", "weaknesses": "I am not familiar with feedback control and motor tasks; hence, I ask the AC to please take this into consideration to appropriately weigh my review. My remarks on the methods could be wrong or trivial. That said, I'll do my best to provide feedback.\n\n- Several sections of the paper seem to just present results from previous work, including section 3.1 and the entirety of the methods section. This makes the contributions of this paper seem rather thin.\n\n- I may be missing something, but some of the results seem minimally surprising. For example, in section 3.2, the authors state \"...the feedback contribution to the overall network output increases during perturbation.\" But how could it not increase during perturbation? Isn't the network explicitly trained to use the feedback information to make corrections during perturbation? The same goes for the alignment between the feedback signal and the optimal global gradient, and the indirect introduction of second-order information-- is it not by design that the network use feedback to make corrections, and thus the larger the correction needed (i.e. the larger the optimal gradient) the larger the feedback signal? And is it not by design that second-order information gets introduced via the recurrent connections that enables the network to \"save\" information from previous timesteps in the hidden state?\n\n- The authors claim that feedback control guides credit assignment in biological circuits, but uses BPTT during the pretraining phase of the RNN, which they acknowledge is not biologically plausible. \nIt seems to me that backprop is still doing much of the heavy lifting in terms of solving credit assignment, thus I'm not sure this claim is sufficiently justifiable. A more defensible claim given the current results may be that feedback control may guide motor adaptation in biological circuits.\nSimilarly, some parts of the intro and abstract strongly suggest that the presented method would perform credit assignment without suffering from the biological implausibilities of backpropagation (e.g. the abstract sets up the problem as \"backpropagation is known to perform accurate credit assignment of error, how a similarly powerful process can be realized within the constraints of biological circuits remains largely unclear\"), yet the actual method relies heavily on backpropagation.\n\n- The experiments are performed on a single task, using a small single layer RNN with 400 hidden units, and therefore it's unclear whether the findings would scale to other tasks and larger architectures. Given that the primary goal of this paper is to improve understanding of an existing learning algorithm, and most of the analysis are performed via empirical testing, I believe it's important for the authors to demonstrate that their conclusions are robust over a wider range of tasks and hyperparameters/architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. \n\n> Several sections of the paper seem to just present results from previous work ...\n\nWhile our work builds upon the task adaptation findings of previous work (Feulner et al. 2022), it offers a substantially distinct focus and set of contributions, as outlined in our introduction. Our primary goal is to investigate the mechanistic interpretability of the model published by Feulner et al. 2022., and to connect their results to the broader literature of online learning in recurrent neural networks.  Specifically, we address why rapid adaptation occurs without any plasticity and how this benefits approximate, local learning algorithms, whose limitations have been previously studied by Marschall et al. 2020). Beyond learning accuracy, we also analyse learning efficiency and show significant differences between learning with and without feedback during the fine-tuning stage. Thus, despite the similarity in the problem setup and the methods presented in the main text, our analysis is distinct from Feulner et al. (2022), where the emphasis was on modeling primate observations and data. Here, we ask a different question: based on what we know about local learning in recurrent neural networks, why does this model work so well?\n\n> I may be missing something, but ...\n\nThank you for your insightful comments. While it may seem intuitive that the feedback contribution to the overall network output would increase during perturbation, it is important to note that this behaviour is not explicitly designed in our networks. The same applies to the incorporation of second-order information.  The key contribution of our work lies in empirically showing that these mechanisms naturally emerge from our model. Even if some aspects may seem anticipated, demonstrating this behaviour in a practical and previously unexplored context is a significant step forward in understanding online learning in recurrent neural networks, and specifically how it relates to biologically plausible learning. \n\n> The authors claim that feedback control guides credit assignment in biological circuits ...\n\nThank you for raising this important question. Yes, this approach reflects principles commonly observed in biological learning systems, specifically the different time scales of learning:\n- Foundational Development: Certain core circuits and functions may develop on a slower timescale, guided by an innate basis shaped over evolutionary time. This provides a foundational framework for subsequent learning (analogous to BPTT pretaining).\n- Rapid, Contextual Adaptation: Building upon this foundation, biological systems exhibit rapid adaptability to specific environments and tasks. This occurs on a much faster timescale, and is thought to occur through fine-tuning of pre-existing circuits (analogous to fine-tuning with a local learning rule).\n\nFor example, infants possess a general capacity for movement, learn to walk relatively slowly, but quickly adapt to different terrains like ice. While BPTT plays a significant role in the credit assignment during pretraining, the rest of this work focuses on alternative, biologically plausible credit assignment during rapid, contextual adaptation, on perturbation not seen during pretraining.\n\n> Given that the primary goal of this paper is to improve understanding of an existing learning algorithm ...\n\nThank you for raising this important point.  We now include extra figures with this rebuttal that include the final adaptation to perturbation for different learning algorithms as a function of learning rate and hidden size (extra figures B+C). We also include results from an additional, commonly used synthetic task (where the network has to produce a sine and a cosine wave given some frequency, see cartoon in extra figure E), showing that our results remain qualitatively the same (extra figure F+G), expanding the generality of our method. We will add these results to the supplementary figures of the final paper. \n\n> How does this work relate to hierarchical predictive coding, and to the feedback connections introduced by Hinton in [1] (and further explored by [2])?\n\nIn the FF network architecture, the activity vectors from the layer l+1 at previous timestep contribute to the activity vectors at layer l, which is a common feature with our work. However, nature of and reason for this contribution is different. In our case, the information communicated between the layers is a global error to control the activity space online. In FF, the information communicated uses layer activities themselves and is done to sync up local learning between different layers. \nThe follow up work links FF to the predictive coding framework, where the error neurons are explicitly represented. We link our work to other relevant literature in predictive coding in our discussion.\n\n> The learning setting presented in this work seem very similar to the setting of reinforcement learning...\n\nEven though, in principle, feedback control studied here resembles a reinforcement learning (RL) problem due to the interaction with the environment, there are some major differences from classic RL problems. For instance, unlike many RL problems, the feedback here is an explicit function of the network's prediction. Moreover, during pretraining, we are backpropagating through the environment itself, which is generally assumed to be impossible in most RL problems. However, our results may be relevant to certain RL subfields, such as movement control in robotics, and we briefly mention this in the \"Impact Statement\" section of our manuscript. Moreover, it may be interesting to expand our work to settings where feedback is less explicit (as we note in our limitations section), which would more closely link it to RL problems.\n\n>I'd like to see an expanded discussion ...\n\nThank you for this suggestion. As explained above, here we use BPTT for pre-training of the networks only.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xJ8VELqo6j", "reviewer": "Reviewer_9rgZ", "summary": "Feedback controllers are ubiquitous in neuroscience but their functions are not fully understood. This paper studies how feedback control interplays with biologically plausible online learning on a standard motor control task. The authors show that:\n- feedback control enables to adapt to task variations without any plasticity, by approximating the gradient of the loss with respect to the hidden states.\n- it makes tractable approximations to RTRL more reliable by shrinking the recurrent Jacobian eigenvalues.\n- it incorporates some second-order information in the weight updates, leading to faster learning.", "questions": "- l98-99: can the author clarify the link between the use of a local learning rule and the rapid adaptations shown in neuroscience studies?\n- Fig1: a, b, c legends are missing in the figure.\n- l140-141: \"that\" missing after \"feedback control,\"? typo \"outout\".\n- Fig2: \"approximate inference learning\": what do the authors mean by inference learning? I could not find any definition.\n- l167-168: \"does\" missing + typo for \"adaptatation\".\n- Appendix A.3: the authors mention that they use Adam with weight decay. The standard practice is to use AdamW instead (c.f. the AdamW paper for more detail). Can the author confirm that they are using AdamW?", "limitations": "The paper is theoretical and its limitations have been properly addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper studies an important understudied question, that is the interplay between feedback and learning.\nThe paper is overall well-written and is easy to read. The message is clearly delivered.\nThe experiments are carefully designed and well executed, and support the claims of the paper.\n\nOverall, the paper will be an insightful read to the community.", "weaknesses": "While the experiments are overall well executed, there are a few points that should be improved to make the paper's claims more robust:\n- In the appendix, it is written that the learning rate is taken to be constant. To make claims about e.g. learning speed, the optimizer, in particular its learning rate, has to be tuned.\n- Figure 5b: it is not clear from this experiment that RFLO-c contains some second-order information. The alignment with the 2nd-order gradient result is not convincing as the estimated gradient is more aligned to the first-order gradient than to the second-order one. This experiment needs to be improved for it to support its claim. The BPTT-c baseline that I mention below may be a good starting point for further analysis as the gradient of a \"controlled loss\" (which is not the case for RFLO).\n- A BPTT-c/RTRL-c baseline would be an interesting add to disambiguate between the role of feedback control and approximate gradient estimation through RFLO. This baseline would include feedback control in the recurrent neural network dynamics and optimize for the MSE loss at the output. This would be useful in e.g. Fig3b and Fig5b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.\n\n\n> In the appendix, it is written that the learning rate is taken to be constant. To make claims about e.g. learning speed, the optimizer, in particular its learning rate, has to be tuned.\n\nThank you for pointing this out. To address your concern, we have included a learning rate sweep for Figure 3b in the extra figures page (see figure B).\n\n> Figure 5b: it is not clear from this experiment that RFLO-c contains some second-order information.\n\nThank you for raising this important point. To clarify, we are not claiming that our learning method performs exact 2nd-order optimization. Instead, it effectively interpolates between a 1st and 2nd order method, with the gradient showing a significant projection in both directions. In the extra figure A, we show the measured alignment between the 1st and 2nd order gradient as a baseline (yellow line). This baseline alignment is lower than the overlap of our gradient with both 1st and 2nd order directions, indicating that the observed alignment is not accidental. Additionally, having any curvature information allows the optimizer to better navigate the learning landscape and improve learning efficiency. \n\n> A BPTT-c/RTRL-c baseline would be an interesting add to disambiguate between the role of feedback control and approximate gradient estimation through RFLO...\n\nThank you for the suggestion. We have added the BPTT-c baseline to Figure 3b in the extra figures page (see D) to help disambiguate the roles of feedback control and approx gradient through RFLO. It seems that BPTT+c performs worse than BPTT (remember that the test loss is reported in the open-loop setting, i.e. no feedback is provided to the network regardless of whether it was trained with it or not). This suggests that BPTT+c overfits to feedback control here, unlike RFLO+c. However, note extra figure B, which suggests that our choice of learning rate for BPTT+c may simpy not be ideal. For Figure 5b, however, this baseline is not applicable because in it, there is no plasticity in the weight matrix yet, and both gradients are computed in the activity space. While we could perform the same analysis in the weight space, calculating the Hessians of the full weight matrix is intractable due to poor scaling.\n\n> l98-99: can the author clarify the link between the use of a local learning rule and the rapid adaptations shown in neuroscience studies?\n\nLong-term skill learning is known to involve synaptic changes in the motor cortices (e.g. Roth et al. Neuron 2020). However, we don’t know how rapid learning is implemented in the brain, nor where it happens. In the neuroscience study that motivated our work (Perich et al. 2018), plasticity was not directly measured, due to the extreme difficulty of such measurements. Nevertheless, animals progressively learn to adapt better to persistent perturbations, which requires some form of rapid learning along the sensorimotor pathways. Thus, in the fine-tuning phase of this work, we enable plasticity in the recurrent layer of our RNNs, which serve as a model of a nonlinear dynamical system doing closed-loop control. Additionally, we use a learning rule that could feasibly be implemented in the brain, as it respects the locality constraint. Thank you for flagging this important point - we have now edited the relevant section in the main text to clarify this reasoning better. \n\n> Fig1: a, b, c legends are missing in the figure.\n> l140-141: \"that\" missing after \"feedback control,\"? typo \"outout\".\n> l167-168: \"does\" missing + typo for \"adaptatation\".\n\nThank you for catching these errors!\n\n> Fig2: \"approximate inference learning\": what do the authors mean by inference learning? I could not find any definition.\n\nThank you for flagging this. We are referring to the inference learning (IL) definition from Alonso et al. 2022, which is based on the predictive coding (PC) literature. In IL, neuron activities are modified first using some gradient, and only after this step are the weights modified. This contrasts with backpropagation, where weights are modified first and only. As defining this term in Figure 2 may not be ideal, we have omitted it in our updated manuscript and instead included the relevant citation in the discussion.\n\nAlonso, Nick, et al. \"A theoretical framework for inference learning.\" Advances in Neural Information Processing Systems 35 (2022): 37335-37348.\n\n> Appendix A.3: the authors mention that they use Adam with weight decay. The standard practice is to use AdamW instead (c.f. the AdamW paper for more detail). Can the author confirm that they are using AdamW?\n\nWe use Adam, and we explicitly add the regularization terms (i.e., weight decay) directly to the cost function. This approach is functionally different from the weight decay correction in AdamW. While we did not use AdamW, it does not impact our results as we do not report any performance gains with Adam and instead use SGD during fine-tuning. We now make this explicit in the detailed methods section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTTpQzwRlO", "reviewer": "Reviewer_HTCn", "summary": "The authors explore the relationship between feedback control and learning with recurrent neural networks (RNN). Specifically, they enforce a control signal onto a RNN that is used to generate a trajectory for a outreaching task, and then propose to use local learning rules on the neurons in the RNN. They show that with feedback control the network can adapt faster to perturbations, of the task and show that the local (in time) gradients are better aligned with the global ones.", "questions": "I am not 100%, but I think that the work of Gilra and Gerstner had a very similar architecture. Could you mention what are the main differences?", "limitations": "They did mention some limitations, but the key issue of whether this is general motor control (or shaping attractors) is not addressed.\nAlso, if the paper is about how feedback control guides credit assignment on biological circuits, being agnostic to the biological circuit is a problem, rather than a strength. To make a biological statement there should be some non trivial biological predictions, or some mention of what exactly does this bring to neuroscience that wasn't already known.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The claims are all very reasonable and well illustrated. I this is the first time such feedback-based learning used in proper control settings, which is surprising given that it is based on control theory.", "weaknesses": "Main problem:\nMy main concern is that I the task chosen consists on bringing a system to a desired static target, so it is possible that there is no \"out of equilibrium dynamics\", rather the learning simply consists on bringing the \"arm\" to the required target and it just so happens that the shortest trajectory aligns with the velocity profile. While it could be that the trajectory is indeed learned (and with some implicit or explicit regularization it should be the case), the current task is not conclusive. If the point is to really learn a trajectory, the authors should have picked a task where the trajectory is a bit more complex than going to equilibrium. Maybe a limit cycle? Otherwise the work might be a minor modification of Meulemans et al.\nAlso, I fail to see the \"biological circuits\". If we are talking about recurrent neural networks, this is fine, but usually when we talk about circuits in biology we would refer to cell types (and this has a lot of constraints). In fact the authors themselves state that they are agnostic to the biological implementation, which is hardly in compatible with the title. I would replace it by recurrent neural networks.\n\nOther issues:\n- The key findings are not clear in the introduction. The term \"inference learning\" is only used there and in one of the figure, but it is not clearly defined. If the authors mean that feedback control can train an RNN then this has already been shown. For the second finding, \"increased accuracy of approximate local learning rules\" it would be better stated as increased accuracy WITH local learning rules (or something similar). For the third, the second order gradient is not really injected (this would suggest that the gradient is imposed on purpose); rather, the feedback control is implicitly related to second order optimization methods.\n- Line 142: it seems natural that if the network is perturbed from its trajectory the feedback would be stronger to compensate for the perturbation. I don't see why this is \"suggested\". Also, the sentence is badly written \"suggest that the during task... activity is increasingly by feedback\").\n- LInes 164 and 165. The authors say that \" using a local learning rule without feedback control show an increasing performance gap compared to those trained with feedback control and BPTT\". The sentence could be interpreted as if the network is trained with feedback control AND BPTT (combined). A better wording would replace AND by OR \n- In 3.4 it is a bit hard to follow. It seems as if the authors are using an eligibility trace to train the RNN through BPTT. But this intermediate step might not be real BPTT as it is commonly used. \n\n\nLiterature issues:\n- The work of Meulemans et al 2022b is credited with alleviating the temporal storage required by BPTT. While they did do that (and it is a good paper), I think that they based the memory load decrease on previous work (Bai et al., Deep Equilibrium models 2019), which if memory serves does use BP. The logic of my comment is that by training the equlibrium point of the network one can avoid the memory load, regardless of the training method.\n- The connection between feedback-based learning and second order optimization has been is very closely related to Meulemans, et al. \"A theoretical framework for target propagation\" 2020. That paper mentions target propagation, but it is very similar to feedback based learning (as the authors probably can infer).\n- This is a personal opinion, the authors do not need to take it into consideration: The biological plausibility claims seem to rely on the locality of the learning rules. While it's a requirement that learning rules should not break the laws of physics (or in this case basic biological knowledge), learning rules should at least have some basis on biology, which I am missing here. A brief mention of why would one think that the learning rules are close to biological ones would be welcome. My guess for this feedback-based work would be something with a temporal component such as  temporal hebbian rules (ex: Aceituno et al. \"Learning cortical hierarchies with temporal Hebbian updates.\" 2020)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work.\n\n> My main concern is that I the task chosen consists on bringing a system to a desired static target...\n\nThank you for flagging this - we also trained RNNs on an additional, commonly used task, where the network has to generate both the sine and cosine wave given some input frequency (see cartoon in extra figure E). Here, the relevant perturbation is a frequency shift. Our adaptation to perturbation results remain qualitatively the same (extra figure F and G), expanding the generality of our method. We will include these results as an additional supplementary figure in the final manuscript.\n\n> Also, I fail to see the \"biological circuits\".\n\nThank you for flagging this. In the discussion, we state that we are agnostic to the exact source and implementation of the feedback signal. For the RNN itself, while it does not directly correspond 1-1 to a single brain region, it has been used in the literature as a useful model of a non-linear dynamical system such as the brain. Hence, our study is more about the brain as an integrated system rather than a single brain region or circuit per say. \n\n>  .. The term \"inference learning\" is only used there and in one of the figure, but it is not clearly defined. \n\nThank you for flagging this. We are referring to the inference learning (IL) definition from Alonso et al. 2022, which is based on the predictive coding (PC) literature. In IL, neuron activities are modified first using some gradient, and only after this step are the weights modified. This contrasts with backpropagation, where weights are modified first and only. As defining this term in Figure 2/introduction may not be ideal, we have omitted it in our updated manuscript and instead included the relevant citation in the discussion. We now instead refer to simply \"approximate learning in the activity space\", due to approximate gradient alignment (which was not explicitly shown in previous work).\n\nAlonso, Nick, et al. \"A theoretical framework for inference learning.\" Advances in Neural Information Processing Systems 35 (2022): 37335-37348.\n\n>  For the second finding ...\n\nThank you for flagging this - here, we refer to accuracy in terms of approximation of the gradient, not performance accuracy, as the referenced section 3.4 outlines.\n\n> For the third, the second order gradient is not really injected ...\n\nThank you for flagging this - you are right! We now changed this to read: \"Feedback control enables more efficient weight updates during task adaptation due to the (implicit) incorporation of adaptive, second-order gradient into the network dynamics \"\n\n> Line 142: it seems natural that ...\n\nThank you for flagging this - we now changed this to simply read: \"Thus, during the task perturbation, the network activity is increasingly driven by the feedback signal.\"\n\n> LInes 164 and 165. The authors say that \"using a local learning rule ...\n\nThank you for flagging this - we corrected this and we thank the reviewer again for these important detailed comments!\n\n> In 3.4 it is a bit hard to follow. It seems as if the authors are using an eligibility trace to train the RNN through BPTT. But this intermediate step might not be real BPTT as it is commonly used.\n\nThank you for flagging this! In the beginning of section 3.4, our main goal is to briefly outline why biologically plausible learning is particularly hard in RNNs, due to the need for recursive computation of the past network states using the network Jacobian. This and the problems with the currently proposed biologically plausible solutions was explored in more detail in Marchall et al. 2020., where RFLO was shown to be quite a severe approximation of RTRL. This mainly serves as context for our main contribution of this section - that with feedback control, RNNs are more driven by their present than their past states, which simplifies the learning problem and allows for more accurate local learning. \n\n> The work of Meulemans et al 2022b is credited with alleviating the temporal storage required by BPTT.\n\nThank you for flagging this. For simplification, as we are not exploring equilibrium systems in this work, we removed this specific claim in the updated manuscript.\n\n> The connection between feedback-based learning and second order optimization ...\n\nThank you for flagging this citation omission! We now cite it in our updated section on related work. \n\n> A brief mention of why would one think that the learning rules are close to biological ones would be welcome. \n\nThank you for this suggestion and citation, which has now been added to the discussion section of our updated manuscript.\n\n> I am not 100%, but I think that the work of Gilra and Gerstner had a very similar architecture. Could you mention what are the main differences?\n\nThank you for flagging this! Our work is indeed similar to that of Gilra and Gerstner 2017 (eLife), with the main differences being 1) we use a rate instead of a spiking network, which simplified our analysis, such as gradients + Hessians 2) in their work, the feedback loop is trained separately from the the recurrent network, while we pretrain them together 3) in their work, in contrast to ours, the network output is close to reference from the very start of learning due to \"tighter\" feedback control, which further eases the learning problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their careful review of our manuscript and all the insightful comments!
Here, we attach a single page with extra figures to support our individual rebuttals below.

Meta-review: This paper presents a recurrent neural network model focused on feedback control-based motor learning. The authors use a motor learning task commonly used in monkey electrophysiology to investigate the role of feedback control in learning. The results are well-presented, the paper is well-written, and the figures are clear and effective.

The reviewers unanimously support the paper's acceptance, although their levels of enthusiasm vary. Nonetheless, they all acknowledge the paper’s significant contributions. Based on my reading of the discussions, most of the reviewers' comments have either been addressed or are not seen as barriers to the paper's acceptance at NeurIPS. The comments and discussions also raise several interesting points that I encourage readers to consider.

Given these factors, I recommend this paper for acceptance.
Decision: Accept (poster)