Credit Assignment in Neural Networks through Deep Feedback Control
Authors: Alexander Meulemans
Matilde Tristany Farinha
Javier Garcia Ordonez
Pau Vilimelis Aceituno
Joao Sacramento
Benjamin F Grewe
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Biologically-plausible Deep Learning
Neuroscience
Deep Learning
Control theory
Optimization theory

Abstract: The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output. However, the majority of current attempts at biologically-plausible learning methods are either non-local in time, require highly specific connectivity motifs, or have no clear link to any known mathematical optimization method. Here, we introduce Deep Feedback Control (DFC), a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose control signal can be used for credit assignment. The resulting learning rule is fully local in space and time and approximates Gauss-Newton optimization for a wide range of feedback connectivity patterns. To further underline its biological plausibility, we relate DFC to a multi-compartment model of cortical pyramidal neurons with a local voltage-dependent synaptic plasticity rule, consistent with recent theories of dendritic processing. By combining dynamical system theory with mathematical optimization theory, we provide a strong theoretical foundation for DFC that we corroborate with detailed results on toy experiments and standard computer-vision benchmarks.

Credit Assignment in Neural Networks through Deep Feedback Control
Alexander Meulemans; Matilde Tristany parinhaa,,Jvieen García Ordóñez,, Pau Vilimelis Aceituno, João Sacramento, Benjamin F. Grewe
Institute of Neuroinformatics, University of Zürich and ETH Zürich ameulema@ethz.cd
Abstract
The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output. However, the majority of current attempts biologically- plausible learning methods are either non-local in time, require highly specific connectivity motifs, or have no clear link to any known mathematical optimization method. Here, introduce Deep Feedback Control (DFC), a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose control signal can be used for credit assignment. The resulting learning rule is fully local in space and time and approximates Gauss- Newton optimization for wide range of feedback connectivity patterns. To further underline its biological plausibility, we relate DFC to a coltii-oompartment model of cortical pyramidal neurons with local voltage-dependent synaptic plasticity rule, consistent with recent theories of dendritic processing. By combining dynam- ical system theory with mathematical optimization theory, provide strong theoretical foundation for DFC that we corroborate with detailed results on toy experiments and standard computer-vision benchmarks.
1 Introduction
The error backpropagation (BP) algorithm [1, 2, 3] currently the gold standard perform credit assignment (CA) in deep neural networks. Although deep learning was inspired by biological neural networks, an exact mapping of BP onto biology to explain learning in the brain leads to several inconsistencies with experimental results that are not yet fully addressed [4. 5, 6]. First, BP requires an exact symmetry between the weights the forward and feedback pathways [5, 6], also called weight transport problem. Another issue of relevance is that, in biological networks, feedback also changes each neuron's activation and thus its immediate output [7, 8]. which does not occur BP.
Lillicrap et al. [9] convincingly showed that the weight transport problem can be sidestepped in modest supervised learning problems by using random feedback connections. However, follow-up studies indicated that random feedback paths cannot provide precise CA in more complex problems [10, 11, 12, 13], which can be mitigated by learning feedback weights that align with the forward pathway [14, 15, 16. 17, 18] or approximate its inverse [19, 20, 21, 22). However, this precise alignment imposes strict constraints on the feedback weights, whereas more flexible constraints could provide the freedom to use feedback also for other purposes besides learning, such attention and prediction [8].
A complementary line of research proposes models of cortical microcireuits which propagate CA signals through the network using dynamic feedback [23, 24, 25] or multiplexed neural codes [26]. thereby directly influencing neural activations with feedback. However, these models introduce
"quual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

highly specific connectivity motifs and tightly coordinated plasticity mechanisms. Whether these constraints can be fulfilled by cortical networks an interesting experimental question. Another line of work uses adaptive control theory [27] to derive learning rules for non non-hierarchical recurrent neural networks (RNNs) based on error feedback, which drives neural activity to track a reference output [28, 29, 30, 31]. These methods have so far only been used to train single-layer RNNs with fixed output and feedback weights, making unclear whether they can be extended to deep neural networks. Finally, two recent studies [32, 33] use enror feedback in a dynamical setting to invert the forward pathway, thereby enabling errors to flow backward. These approaches rely on a learning rule that is non-local in time and it remains unclear whether they approximate any known optimization method. Addressing the latter, two recent studies take a first step by relating learned (non-dynamical) inverses of the forward pathway [21] and iterative inverses restricted to invertible networks [22] to approximate Mauss-Newton optimization.
Inspired by the Dynamic Inversion method [32], we introduce Deep Feedback Control (DFC), new aalogicclly-pluusbble CA method that addresses the above-mentioned limitations and extends the control theory approach to learning [28, 29, 30, 31] to deep neural networks. DFC uses a feedback controller that drives a deep neural network to match a desired output target. For learning, DFC then simply uses the dynamic change in the neuron activations to update their synaptic weights, resulting in a learning rule fully local in space and time. We show that DFC approximates Gauss-Newton (GN) optimization and therefore provides fundamentally different approach to CA compared to BP. Furthermore, DFC does not require precise alignment between forward and feedback weights, nor does it rely on highly specific connectivity motifs. Interestingly, the neuron model used by DFC can be closely connected to recent multi-compartment models of cortical pyramidal neurons. Finally, we provide detailed experimental results, corroborating our theoretical contributions and showing that DFC does principled CA on standard computer-visioe benchmarks in a way that fundamentally differs from standard BP.
2 The Deep Feedback Control method
Here, we introduce the core parts of DFC. In contrast to conventional feedforward neural network models, DFC makes use of dynamical neuron model (Section 2.1). We use a feedback controller to drive the neurons of the network to match a desired output target (Section 2.2), while simultancously updating the synaptic weights using the change in neuronal activities (Section 2.3). This combination of dynamical neurons and controller leads to simple but powerful learning method, that is linked to GN optimization and offers flexible range of feedback connectivity (see Section 3).
2.1 Neuron and network dynamics
The first main component of DFC a dynamical multilayer network, in which every neuron integrates its forward and feedback inputs according to the following dynamics:
d (t) = (t) + (io(v=-(tt)+ + Qau(t) 1SisL. To dt
(1)
with v, a vector containing the pre nrelinonlyyyrtty activations of the neurons in layer i, W; the forward weight matrix, ó smooth nonlinearity, a feedback input, Q, the feedback weight matrix, and T a time constant. See Fig. 1B for a schematic representation of the network. To simplify notation, we define r; ((v:) the post-nonlinearity activations of layer i. The input ro remains fixed throughout the dynamics (1). Note that in the absence of feedback, i.e., u = O. the equilibrium state of the network dynamics (1) corresponds to conventional multilayer feedforward network state, which we denote with superscript :-:
r = o(v (Wrr IsiS L, with ry r--
2.2 Feedback controller
The second core component of DFC is a feedback controller, which is only active during learning. Instead of a single backward pass for providing feedback, DFC uses a feedback controller to continuously drive the network to an output target rž (see Fig. ID). Following the Target Propagation framework [20, 21, 22], we define rū as the feedforward output nudged towards lower loss:

---

([(r):y) drl
ri - rz
=I E1:
with C(rL, y) supervised loss function defining the task, y the label of the training sample, À stepsize, and on shorthand notation. Note that (3) only needs the easily obtained loss gradient w.r.t. the output, e.g., for an I.2 output loss, obtains the convex combination rž (1 2 2Ay.
The feedback controller produces a feedback signal u(t) to drive the network output (t) towards its target r using the control error e(t) _ rž rz (t). A standard approach in designing a feedback controller is the inpportional-Integrn Derivva-Deevvvveeee (PID) framework [34]. While DFC is compatible
with various controller types, such as a full PID controller or pure proportional controller (sce Appendix A, in the following controller dynamics (see also Fig. 1A):
Pl controller for combination of simplicity and good performance, resulting
d
n (1) = Kyu'" (1) + Kre((),
Tu 1n (i(((1) = e(t) an"
(4)
dt
where leakage term is added to constrain the magnitude of u'" For mathematical simplicity, take the control matrices equal to K, = I and Kp = / with kp 2 0 the proportional control constant. This controller adds a leaky integration of the error u to a scaled version of the error kye which could be implemented by dedicated neural microcircuit (for discussion see App. I). Drawing inspiration from the Target Propagation framework [19, 20, 21, 22] and the Dynamic Inversion framework [32], one can think of the controller and network dynamics as performing dynamic inversion of the output target rž towards the hidden layers, as the controller dynamically changes the activation of the hidden layers until the output target reached.
D
(()
controller
Qau apical
ri
e(1)
H 1OO O
rt
tt
W
(t)
[e(r]dr
TO 100 Qi
T's
soma Ut
u(t)
W E0( O00
r
35
Wir-1
basal u.
Figure 1: (A) block diagram of the controller, where we omitted the leakage term of the integral controller. (B) Schematic illustration of DFC. (C) Schematic illustration of the multi-compartment neuron used by DFC, compared to a cortical pyramidal neuron sketch (see also Discussion). (D) Illustration of the output ry. (() and the controller dynamics u(l) in DFC.
2.3 Forward weight updates
The update rule for the feedforward weights has the form:
d -
TW di
(5)
This learning rule simply compares the neuron's controlled activation input and is thus local in space and time. Furthermore, it can be interpreted most naturally by compartmentalizing the neuron into the central compartment v, from (1) and feedforward compart- ment v A Wir,-1 that integrates the feedforward input. Now, the forward weight dynamics (5) represents delta rule using the difference between the actual firing rate of the neuron, @(v:). and its estimated firing rate, @(v!), based on the feedforward inputs. Note that we assume Tw to large time constant, such that the network (1) and controller dynamics (4) are not influenced by the weight dynamics, i.c., the weights are considered fixed in the timescale of the controller and network dynamics.
current feedforward

---

In Section 5, we show how the feedback weights Qi can also be learned locally in time and space for supporting the stability of the network dynamics and the learning of W. This feedback learning rule needs feedback compartment with & Q,u, u, leading the ciree-compartment neuron schematized in Fig. IC, inspired by recent multi-compartment models of the pyramidal neuron (see Discussion). Now, that we introduced the DFC model, we will show that (i) the weight updates (5) can properly optimize a loss function (Section 3), (ii) the resulting dynamical system is stable under certain conditions (Section 4), and (iii) learning the feedback weights facilitates (i) and (ii) (Section 5).
3 Learning theory
To understand how DFC optimizes the feedforward mapping (2) on given loss function, we link the weight updates (5) to mathematical optimization theory. We start by showing that DFC dynamically inverts the output error to the hidden layers (Section 3. which we link to GN optimization under flexible constraints on the feedback weights Qi and on layer activations (Section 3.2). In Section 3.3, we relax some of these constraints, and show that DFC still does principled optimization by using minimum norm (MN) updates for W. During this learning theory section, we assume stable dynamics, which we investigate more detail in Section 4. All theoretical results of this section are tailored towards PI controller, and they can be easily extended to pure proportional or integral control (see App. A.8).
3.1 DFC dynamically inverts the output error
To understand how the weight update (5) can access error information, we start by investigating the steady state of the network dynamics (1) and the controller dynamics (4), assuming that all weights are fixed (hence, a separation of timescales). As the feedback controller controls all layers simultaneously,
we introduce a compact notation for: concatenated neuron activations - [vT y..., vEI" feedforward
compartments " é [v vL 1T, and feedback weights Q & DITT..IIT. Lemma shows first-order Taylor approximation of the steady-state solution (full proof in App. A.1).
Lemma 1. Assuming stable dynamics, small target stepsize x, and W; and Q; fixed, the steady-state solutions of the dynamical systems (1) and (4) can be approximated by:
U (JQ + +l) 82 +(()), V === + QQQ + aT) 6 + 0(22),
(6)
with J Oor,
Dr,
the Jacobian of the network output w.r.t all vs, evaluated at the network equilibrium without feedback, 85, the output error as defined in (3), v!Il 555( and ā = a/ + akp).
8vv ..... TVL
To get a better intuition of what this steady state represents, consider the scenario where we want to nudge the network activation with Av, i.e., Vas v + Av, such that the steady-state network output equals its target r7,- With linearized network dynamics, this results in solving the linear system JÁv L. As Av is of much higher dimension than 8z, this is an underdetermined system with infinitely many solutions. Constraining the solution to the column space of Q leads to the unique solution Av = Q(Q)) (((JQ)-18 18,, corresponding to the steady-state solution in Lemma minus a small damping constant ã. Hence, similar to Podlaski and Machens [32], through an interplay between the network and controller dynamics, the controller dynamically inverts the output error ô, to produce feedback that exactly drives the network output to its desired target.
3.2 DFC approximates Gauss-Newton optimization
To understand the optimization characteristics of DFC, we show that under flexible conditions on Q and the layer activations, DFC approximates GN optimization. We first briefly review GN optimization and introduce two conditions needed for the main theorem.
Gauss-Newton optimization [35] is an approximate second-order optimization method used in nonlinear least-squares regression. The GN update for the model parameters is computed as:
A = Jie..
with Jo the Jacobian of the model output w.r.t. 0 concatenated for all minibatch samples, J its Moore-Penrose pscudoinverse, and e₁ the output errors.

---

Condition 1. Each layer of the network, except from the output layer, has the same activation norm:
||ool2 =|rrll ...||/r--1112 _ ||1||2
(8)
Note that the latter condition considers a statistic ||r: l|2 of a whole layer and does not impose specific constraints on single neural firing rates. This condition can be interpreted as each layer, except the output layer, having the same 'energy budget' for firing.
Condition 2. The column space of Q is equal to the row space of J.
This more abstract condition imposes flexible constraint on the feedback weights Qi, that generalizes common learning rules with direct feedback connections [16, 21J. For instance, besides Q J (BP: [16]) and Q = JJ [21], many other instances of Q which have not yet been explored in the literature fulfiil Condition (see Fig. 2). hence leading to principled optimization (see Theorem 2). With these conditions in place, we are ready to state the main theorem of this section (full proof in App. A).
Theorem 2. Assuming Conditions and hold, J is full rank, the task loss C is a L loss, and a - 0, then the following steady-state (ss) updates for the forward weights,
(9)
with n a stepsize parameter, align with the weight updates for W for the feedforward network (2) prescribed by the GN optimization method with a minibatch size of I.
In this theorem, we need Condition such that the dynam- ical inversion Q(JQ)-1 (6) equals the pseudoinverse of and we need Condition to extend this pseudoinverse to the Jacobian of the output w.r.t the network weights, as in eq. (7). Theorem 2 links the DFC method to GN optimization, thereby showing that it does principled op- timization, while being fundamentally different from BP. In contrast to recent work that connects target propagation to GN [21, 22], we do not need to approximate the GN curvature matrix by block-diagonal matrix but use the full curvature instead. Hence, one can use Theorem 2 in Cai et al. [36] to obtain convergence results for this setting of GN with a minibatch size of I, in highly overparame- terized networks. Strikingly, the feedback path of DFC does not need to align with the forward path or its inverse to provide optimally aligned weight updates with GN, as long as it satisfies the flexible Condition 2 (see Fig. 2).
0 '''' +1--
0.6
0.5
0.5
Probable density
0.4
pC2
0.1
-0.5
-0.5
0.5
0.0
PC1
Figure 2: Randomly generated feedback matrices Q (blue) that satisfy Conditions 2 and 3, and have unity norm, visualized by a principal component analysis, with density contours added for visual clar- ity. J ,Jt, and J" +-))- E [10-5 102]. added, highlighting that the optimal feedback configurations for DFC (blue) span a much wider space compared to conventional CA methods.
The steady state updates (9) used in Theorem 2 differ from the actual updates (5) in two nuanced ways. First, the plas- ticity rule (5) uses nonlinearity, Ø, of the compartment activations, whereas in Theorem this nonlinearity is not included. There are two reasons for this: (i) the use of & in (5) can be linked to specific biophysical mechanisms in the pyramidal cell [37] (see Discussion), and (ii) using makes sure that saturated neurons do not update their forward weights, which leads to better performance (see App. A.6). Second, in Theorem 2, the weights are only updated at steady state, whereas in (5) they are continuously updated during the dynamics of the network and controller. Before settling rapidly, the dynamics oscillate around the steady-state value (see Fig. 1D), and hence, the accumulated continuous updates (5) will be approximately equal to its steady-state equivalent, since the oscillations approximately cancel each other out and the steady state quickly reached (see Section 6.1 and App. A.7). Theorem 2 needs a 2 loss function and Condition I and 2 to hold for linking DFC with GN. In the following subsection, we relax these assumptions and show that DFC still does principled optimization.
3.3 DFC uses weighted minimum norm updates
GN optimization with a minibatch size is equivalent to MN updates [21], i.c., it computes the smallest possible weight update such that the network exactly reaches the current output target after
5

---

the update. These MN updates can be generalized to weighted MN updates for targets using arbitrary loss functions. The following theorem shows the connection between DFC and these weighted MN updates, while removing the need for Condition and an L loss (full proof in App. A).
Theorem 3. Assuming stable dynamics, Condition 2 holds and X, a + O, the steady-state weight updates (9) are proportional to the weighted MN updates of W for letting the feedforward oulput L reach Fi i.e., the solution to the following optimization problem:
arg min AW,
Σ1 || 1
(m)
S.F.
TL
(m+11 (m)
= L
(10)
im1
with m the iteration and I
(m)+1) the network outpu without feedback after the weight update.
Theorem 3 shows that Condition 2 enables the controller to drive the network towards its target r with MN activation changes, Av = v - which combined with the steady-state weight update (9) result in weighted MN updates AW; (see also App. A.4). When the feedback weights do not have the correct column space, the weight updates will not be MN. Nevertheless, the following proposition shows that the weight updates still follow a descent direction given arbitrary feedback weights.
Proposition 4. Assuming stable dynamics and X, + O, the steady- state weight updates (9) with a layer-specific learning rate T7 1//1|:-1111 lie within 90 degrees of the loss gradient direction.
4 Stability of DFC
Until now, we assumed that the network dynamics are stable, which is necessary for DFC, as an unstable network will diverge, making learning impossible. In this section, we investigate the conditions on the feedback weights Q, necessary for stability. To gain intuition, we linearize the network around its feedforward values, assume a separation of timescales between the controller and the network (Tu > Ta), and only consider integrative control (kp = 0). This results in the following dynamics (see App. B for the derivation):
Tu di u() = (JQ + aa))(()))))))) + 8r.
(11)
Hence, in this simplified case, the local stability of the network around the equilibrium point depends on the eigenvalues of JQ, which is formalized in the following condition and proposition.
Condition 3. Given the network Jacobian evaluated at the steady state, Jus
A Tor
ar
Tvi
DVL
the real parts of the eigenvalues of Q are all greater than -a. Proposition 5. Assuming Tu >> T, and k = the network and controller dynamics are locally asympiotically stable around its equilibrium iff Condition 3 holds.
This proposition follows directly from Lyapunov's Indirect Method [38]. When assuming the more general case where Ty is not negligible and kp > O, the stability criteria quickly become less interpretable (see App. B). However, experimentally, condition for guaranteeing stability in the general case where Tu is not negligible and kp > (see Section 6 and App. B).
see that Condition 3 is good proxy
5 Learning the feedback weights
Condition 2 and 3 emphasize the importance of the feedback weights for enabling efficient learning and ensuring stability of the network dynamics, respectively. As the forward weights, and hence the network Jacobian, J, change during training, the set of feedback configurations that satisfy Conditions 2 and also change. This creates the need to adapt the feedback weights accordingly efficient learning and network stability. We solve this challenge by learning the feedback weights, such that they can adapt to the changing network during training. We separate forward and feedback weight training in alternating wake-sleep phases [39]. Note that in practice, a fast alternation between the two phases is not required (see Section 6).
ensure
Inspired by the Weight Mirror method [14], we learn the feedback weights by inserting independent zero-mean noise in the system dynamics:
d vt(()) (t) + (io(vi-1(t) + Q.u(t) + te, dt
(12)

---

The noise fluctuations propagated to the output carry information from the network Jacobian, J. To let e, and hence u, incorporate this noise information, we set the output target rī to the average network output ri. As the network is continuously perturbed by noise, the controller will try to counteract the noise and regulate the network towards the output target r The feedback weights can then be trained with simple space and time:
plasticity rule with weight decay, which is local in
d TQ di Q:(1) - (v(0))vv(())())) 3Q:
(13)
where 33 is the scale factor of the weight decay term and where we assume that a subset of the noise imput E, enters through the feedback compartment, i..., vf = Q + one," The correlation between the noise in v and noise fluctuations in provides the teaching signal for Q; Theorem 6 shows under simplifying assumptions that the feedback learning rule (13) drives Q2 to satisfy Condition and 3 (see App. C for the full theorem and its proof).
Theorem 6 (Short version). Assume a separation of timescales To & TQ. a big. kp = 0. r = and Condition 3 holds. Then, for fixed input sample and a - 0, the first moment of Q converges approximately to:
lim E|Qud R JJJ + a=
(14)
for some y > O. Furthermore, E(Qss) satisfies Conditions? 2 and 3, even if a 0 in the latter.
Theorem 6 shows that under simplifying assumptions, Q converges towards a damped pseudoinverse of J, which satisfies Conditions 2 and 3. Empirically, we see that this also approximately holds for more general settings where T, is not negligible, hp > 0, and small (see Section 6 and App. C).
The above theorem leaves two questions unanswered. First, it assumes that Condition 3 holds, however, the task of the feedback weight training is to make unstable network dynamics stable, resulting in a thicken-and-oon network stable during early training, after which the feedback weights align according to (14) and a can be decreased. Second, Theorem 6 considers training the feedback weights to convergence over one fixed input sample. However, in reality many different input samples will be considered during learning. When the network is linear, J is the same for each input sample and eq. (14) holds exactly. However, for nonlinear networks, J will be different for each sample, causing the feedback weights to align with an average of J J (JJ" + 7/) over many samples.
problem. The solution we use is to take a big enough to make the
6 Experiments
We evaluate DFC in detail on toy experiments to showcase that our theoretical results translate to practice (Section 6.1) and modest range of computer vision benchmarks MNIST classification and autoencoding [40], and Fashion MNIST classification [41] - to show that DFC can do precise CA in more challenging settings (Section 6.2). Alongside DFC, we test two variants: (i) DFC-SS which only updates its feedforward weights W; after the steady state (SS) of (1) and (4) is reached; and (ii) DFC-SSA which analytically computes the linearized steady state of (1) and (4) according to Lemma 1. To investigate whether learning the feedback weights is crucial for DFC, we compare for all three settings: (i) learning the feedback weights Q, according to (13); and (ii) fixing the feedback weights to the initialization Qi = 1 WT,, which approximately satisfies Condition and at the beginning of training (see App. F), denoted with suffix (fixed). For the former, we pre-train the feedback weights according to (13) to ensure stability. During training. we iterate between I epoch of forward weight training and X epochs of feedback weight training (if applicable), where X E 2,3) is a hyperparameter. We compare all variants to Direct Feedback Alignment (DFA) [42] as a control for direct feedback connectivity. DFC is simulated with the Fuler-Maruyama method, which is the equivalent of forward Fuler for stochastic differential equations [43]. We initialize the network to its feedforward activations (2) for each datasample and, for computational efficiency, we buffer the weight updates (5) and (13) and apply them once at the end of the simulation for the considered datasample. App. E and F provide further details on the implementation of all experiments.
PPyorch implementation of all methods is available at https: //github. thub. com www.cmewmmmmeeeecceeeeeoooeeoweo wnw.salee/eea.p.pp.. feedback_comtrol.
7

---

6.1 Empirical verification of the theory
Figure 3 visualizes the theoretical results of Theorems 2 and 3 and Conditions 1, and 3, in an empirical setting of nonlinear student teacher regression, where a randomly initialized teacher network generates synthetic training data for a student network. We see that Condition 2 is approximately satisfied for all DFC variants that learn their feedback weights (Fig. 3A), leading to close alignment with the ideal weighted MN updates of Theorem 3 (Fig. 3B). For nonlinear networks and linear direct feedback, it is in general not possible to perfectly satisfy Condition 2 as the network Jacobian J varies for each datasample, while Q, remains the same. However, the results indicate that feedback learning finds configuration for Q; that approximately satisfies Condition 2 for all datasamples. When the feedback weights are fixed, Condition is approximately satisfied in the heginning of training due to good initialization. However, as the network changes during training, Condition 2 degrades modestly, which results in worse alignment compared to DFC with trained feedback weights (Fig. 3B).
For having GN updates, both Conditions and 2 need to be satisfied. Although we do not enforce Condition 1 during training, we see in Fig. 3C that it is crudely satisfied, which can be explained by the saturating properties of the tanh nonlinearity. This is reflected in the alignment with the ideal GN updates in Fig. 3D that follows the same trend the alignment with the MN updates. Fig. 3E shows that all DFC variants remain stable throughout training, even when the feedback weights are fixed. In App. B, we indicate that Condition 3 is good proxy for the stability shown in Fig. 3E, Finally, we see in Fig. 3F that the weight updates of DFC and DFC-SS align well with the analytical steady-state solution of Lemma I, confirming that our learning theory of Section 3 applies to the continuous weight updates (5) of DFC.
A
C
1.0
0.45
Condition 2
0.9
Condition
0.30
-0.475
-0.550
0.8
0.15
-0.625
0.7
0.00
-0.700
B
F
80
DFC
DFC (fixed)
DFC-SS
DFC-S5 (fixed
AW; Aww
60-
BW, L AWW
60-
awV L Awys
DFC-SSA
DFC SSA (fixed)
DFA
20-
20
100
200
300
100
200
300
100
200
300
Epoch
Epoch
tpoch
Figure 3: Results for nonlinear student-teacher regression task with layer sizes (15-10-10-5), tanh nonlinearities, a linear output layer, kp 1.5, À 0.05, and 0.0015. (A) Ratio between the norms of Q projected into the row space of J, and Q, with a value of 1 indicating perfect compliance of Condition 2. (B.D.F) Angle in degrees between the concatenated parameter updates of the whole network and: (B) the ideal weighted MN parameter updates (Theorem 3); (D) the ideal GN parameter updates (Theorem 2); and (F) the DFC-SSA parameter updates (see App. F.1 for all definitions). (C) The standard deviation of the layer norms ||r; l|2, divided by the average layer norm, with value of zero indicating perfect compliance to Condition 1. (E) The maximum real part of the eigenvalues of the total system dynamics matrix evaluated at equilibrium (see App. F.I), with negative real parts indicating local stability. For all measures, a window-average is plotted together with the window-std (shade). Stars indicate overlapping plots.
In Fig. 4, we show that the alignment with MN updates remains robust for X E [10-3, 10-'1 and E [10-4 : 10-'], highlighting that our theory explains the behavior of DFC robustly when the limit of and a to zero does not hold. When we clamp the output target to the label (A 0.5), the alignment with the MN updates decreases as expected (see Fig. 4), because the linearization of Lemma 1 becomes less accurate and the strong feedback changes the neural activations more significantly, thereby changing the pre-synaptic factor of the update rules (c.f. eq. 9). However, performance results on MNIST, provided in Table 2, show that the performance of DFC remains robust for wide range of As and as, including λ= 0.5, suggesting that DFC can also provide principled
8

---

CA in this setting of strong feedback, which motivates future work to design complementary theory for DFC focused on this extreme case.
Figure 4: Comparison of the alignment between the DFC weight updates and the MN updates for variable values of À (A) and a (B), when performing the nonlinear cuudent-teacher regression task described in Fig. 3. Stars indicate overlapping plots.
A 60
À 3
B
a= 1e-e
À 1
a= 3
==le-1 1 À = 0.5
a= 1e-el-2
40
a= 10-1
AW, LAW!
100
200
300
200
Epoch
Epoch
6.2 Performance of DFC on computer vision benchmarks
The classification results on MNIST and Rishion-MNISS (Table 1) show that the performances of DFC and its variants, but also its controls, lie close to the performance of BP, indicating that they perform proper CA these tasks. To see significant differences between the methods, we consider the more challenging task of training an autoencoder on MNIST, where it is known that DFA fails to provide precise CA [9, 16, 32]. The results in Table I show that the DFC variants with trained feedback weights clearly outperform DFA and have close performance to BP. The low performance of the DFC variants with fixed feedback weights show the importance of learning the feedback weights continuously during training satisfy Condition 2. Finally, to disentangle optimization performance from implicit regularization mechanisms, which both influence the test performance, we investigate the performance of all methods in minimizing the training loss of MNIST.² The results in Table 1 show improved performance of the DFC method with trained feedback weights compared to BP and controls, suggesting that the approximate MN updates of DFC can faster descend the loss landscape for this simple dataset.
Table 1: Test errors (classification) and test loss (autoencoder) corresponding to the epoch with the best validation result (for 5000 validation samples) over a training of 100 epochs (classification) or 25 epochs (autoencoder). Training loss after 100 epochs (MNIST train loss). We use the Adam optimizer [44]. Architectures: 3x256 fully connected (FC) tanh hidden layers and softmax output (classification), 256-32-256 FC hidden layers for autoencoder MNIST with tanb-linear-tanh nonlinearities, and linear output. Mean + std (5 random seeds). Best results (except BP) are displayed bold.
MNIST
Fashion-MNIS MNIST-autoonncoder MNIST (train loss)
BP
10.1%%
10.60%
9.42 10
1.53
10 7
DFC
2.25#0.004%
11.17#0.27%
11.28+0.18
10
7.61
10 8
DFC-SSA
0.18
11.28
11.27
10-2
4.89+1.26
10
DFC-SS
0.09%%
10.15+
11.21
10-2
4.80+0.70
10-
DFC (fixed)
2.47#0.12%
11 11.62+0.309)
33.37+0.60
10-2
1.30
10-6
DFC-SSA (fixed)
2.46 10.11%
10.14%
31.90
-0
1.73
10
DFC-SS (fixed)
2.39-1.22gg
11.35.40.420
32.31
10-2
1.6740.70
10
DFA
2.69
11.38+0.250%
20.95
10-2
7.09
10-7
Table 2: Test errors MNIST with variable À values and fixed a 0.0015 (left), and with variable
a values and fixed À = 0.08 (right). Same experimental setting as in Table
A
DFC-SS
DFC
DFC-SS
DFC
le 3 2.26#0.11% 1%
2.29
le 4
2.31 12%
2.28%%
le 2 2.2040
2.31
20.44%
2.31 10.11% % le 3 1.2810.15%
-1
2.2710.07g
2.30 0.06%
le
2.32 10.12% 2.26 10.05%
0.5
10.1%
2.34=0.15%
le-1 2.28 11%
20.10.166
We used separate hyperparameter configurations, selected for minimizing the training loss.

---

7 Discussion
We introduced DFC as an alternative pooogically-plassibbe learning method for deep neural networks. DFC uses error feedback to drive the network activations to a desired output target. This process generates a neuron-specific learning signal which can be used to learn both forward and feedback weights locally in time and space. In contrast to other recent methods that learn the feedback weights and aim to approximate BP [14, 15, 16, 17, 26], we show that DFC approximates GN optimization, making it fundamentally different from BP approximations.
DFC is optimal i.e., Conditions and are satisfied for wide range of feedback connectivity strengths. Thus, we prove that principled learning can be achieved with local rules and without symmetric feedforward and feedback connectivity by leveraging the network dynamics. This finding has interesting implications for experimental neuroscientific research looking for precise patterns of symmetric connectivity in the brain. Moreover, from a computational standpoint, the flexibility that stems from Conditions 2 and 3 might be relevant for other mechanisms besides learning, such as attention and prediction [8].
To present DFC in its simplest form, we used direct feedback mappings from the output controller to all hidden layers. Although numerous anatomical studies of the mammalian neocortex reported the occurrence of such direct feedback connections [45, 46], it is unlikely that all feedback pathways are direct. We note that DFC is also compatible with other feedback mappings, such as layerwise connections or separate feedback pathways with multiple layers of neurons (see App. H).
Interestingly. the paree-compartment neuron is closely linked to recent compaicommtr models of the cortical pyramidal neuron [23, 25, 26, 47]. In the terminology of these models, our central, feedforward, and feedback compartments, correspond to the somatic, basal dendritic, and apical den- dritic compartments of pyramidal neurons, respectively (see Fig. 1C). In line with DFC, experimental observations [48, 49] suggest that feedforward connections converge onto the basal compartment and feedback connections onto the apical compartment. Moreover, our plasticity rule for the for- ward weights (5) belongs to a class of dendritic predictive plasticity rules for which a biological implementation based on backpropagating action potentials has been put forward [37].
Limitations and future work. In practice, the forward weight updates are not exactly equal to GN or MN updates (Theorems 2 and 3). due to (i) the nonlinearity & in the weight update rule 5, (ii) non- infinitesimal values for and X, (iii) limited training iterations for the feedback weights, and (iv) the limited capacity of linear feedback mappings to satisfy Condition 2 for each datasample. Figs. 3 and 4, and Table show that DFC approximates the theory well in practice and has robust performance, however, future work can improve the results further by investigating new feedback architectures (see App. H). We note that, even though GN optimization has desirable approximate second-order optimization properties, it is presently unclear whether these second-orde characteristics translate to our setting with a minibatch size of 1. Currently, our proposed feedback learning rule (13) aims to approximate one specific configuration and hence does not capitalize on the increased flexibility of DFC and Condition 2. Therefore, an interesting future direction is to design more flexible feedback learning rules that aim to satisfy Conditions 2 and 3 without targeting Furthermore, DFC needs two separate phases for training the forward weights and feedback weights Interestingly, if the feedback plasticity rule (13) uses high-passed filtered version of the presynaptic input u, both phases can be merged into one, with plasticity always on for both forward and feedback weights (see App. C.3). Finally, as DFC is dynamical in nature, it is costly simulate commonly used hardware for deep learning, prohibiting us from testing DFC on large-scale problems such as those considered by Bartunov et al. [10]. A promising alternative is implement DFC on analog hardware, where the dynamics of DFC can correspond to real physical processes on a chip. This would not only make DFC resource-efficient, but also position DFC as an interesting training method for analog implementations of deep neural networks, commonly used in Edge Al and other applications where low energy consumption key [50, 51].
specific configuration.
To conclude, we show that DFC can provide principled CA in deep neural networks by actively using error feedback drive neural activations. The flexible requirements for feedback mappings combined with the strong link between DFC and GN, underline that it is possible to do principled CA in neural networks without adhering to the symmetrie layer-wise feedback structure imposed by BP.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Swiss National Science Foundation (B.FG. CRSI15-17372 and 315230 189251), ) ETH project funding (B.FG. ETH-20 19-01), the Human Frontiers Science Program (RGY0072/2019) and funding from the Swiss Data Science Center (B.F.G, C17-18, J. v. P18-03). João Sacramento was supported by Ambizione grant (PZ00P3_1866)
from the Swiss National Science Foundation. Pau Vilimelis Aceituno was supported by an ETH Zürich Postdoc fellowship. Javier García Ordóñez received support from La Caixa Foundation through the Postgraduate Studies in Europe scholarship. We would like to thank Anh Duong Vo and Nicolas Zucchet for feedback, William Podlaski, Jean-Pascat Pfister and Aditya Gilra for insightful discussions, and Simone Surace for his detailed feedback on Appendix C.I.
References
[1] David E Rumelhart, Geoffrey E. Hinton, and Ronald Williams. Learning representations by back-propagating errors. Nature, 323(6088):533,, 1986.
[2] Paul Werbos. Applications of advances in nonlinear sensitivity analysis. In System modeling and optimization, pages 762-770. Springer, 1982.
[3] Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm as taylor expansion of the local rounding errors. Master's 's Thesis (in Fimnish), Univ. Helsinki, pages 6-7, 1970.
[4] Francis Crick. The recent excitement about neural networks. Nature, (5776603):129-111 1989.
[5] Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11(1):23-63 63, 1987.
[6] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, pages 1-12, 2020.
[7] Matthew E Larkum, Thomas Nevian, Maya Sandler, Alon Polsky, and Jackie Schiller. Synaptic integration in tuft dendrites of layer 5 pyramidal neurons: new unifying principle. Science, 325(5941):756-766 2009.
[8] Charles D Gilbert and Wu Li. Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5):350- 063,,3333333 2013.
[9] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. Nature Communications, 7:13276, 2016.
[10] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of booogiccall-mooivatee deep learning algorithms and architectures. In Advances in Neural Information Processing Systems 31, pages 9368-99989 9378, 2018.
[11] Julien Launay, lacopo Poli, and Florent Krzakala. Principled training of neural networks with direct feedback alignment. arXiv preprint wwXXiv/906..4550 945544 2019.
[12] Theodore H Moskovitz, Ashok Litwin-Kumar, and LF Abbott. Feedback alignment in deep convolutional networks. arXiv preprint arXiv:
2018
[13] Brian Alexander Crafton, Abhinav Parihar, Evan Gebhardt, and Arijit Raychowdhury. Direct feedback alignment with sparse connections for local learning. Frontiers in Neuroscience, 13: 525, 2019.
[14] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. In Advances in Neural Information Processing Systems 32, pages 974-982, 2019.
[15] Daniel Kunin, Aran Nayebi, Javier Sagastuy- Brena, Surya Ganguli, Jonathan Bloom, and Daniel Yamins. Two routes scalable credit assignment without weight symmetry. In International Conference on Machine Learning, pages 5511-5521. PMLR, 2020.
[16] Benjamin James Lansdell, Prashanth Prakash, and Konrad Paul Kording. Learning to solve the credit assignment problem. In International Conference on Learning Representations, 2020.
11

---

[17] Jordan Guerguiev, Konrad Kording, and Blake Richards. Spike-based causal inference for weight alignment. In International Conference on Learning Representations, 2020.
[18] Siavash Golkar, David Lipshutz, Yanis Bahroun, Anirvan M. Sengupta, and Dmitri B. Chklovskii. A biologically plausible neural network for local supervision in cortical mi- crocircuits, 2020.
[19] Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint Ie:::790077 2014.
[20] Hyyn Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propaga- tion. Joint european conference on machine learning and knowledge discovery in databases, pages 498-515. Springer, 2015.
[21] Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin F. Grewe. A theoretical framework for target propagation. Advances in Neural Information Processing Systems, 33:20024-200366 2020.
[22] Yoshua Bengio. Deriving differential target propagation from iterating approximate inverses. arXiv preprint arXiv:2007. 15139, 2020.
[23] João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems 31, pages 8721-8732, 2018.
[24] James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algo- rithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5):1229-13 1262, 2017.
[25] Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. ELife, 6:e22901, 2017.
[26] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, 24(5):156,, 2021.
[27] Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
[28] Aditya Gilra and Wulfram Gerstner. Predicting non-linear dynamics by stable local learning in a recurrent spiking neural network. Elife, 6:e28295, 2017.
[29] Sophie Denève, Alireza Alemi, and Ralph Bourdoukan. The brain as an efficient and robust adaptive learner. Neuron, 94(5):969-99)) 977999999777999 2017.
[30] Alireza Alemi, Christian Machens, Sophie Denève, and Jean-Jacques Slotine. Learning arbitrary dynamics in efficient, balanced spiking networks using local plasticity rules. AAAI Conference on Artificial Intelligence (AAAI), 2018.
[31] Ralph Bourdoukan and Sophie Deneve. Enforeing balance allows local supervised learning in spiking recurrent networks, Advances in Neural Information Processing Systems, 28:982-990, 2015.
[32] William F Podlaski and Christian K Machens. Biological credit assignment through dynamic inversion of feedforward networks. Advances in Neural Information Processing Systems 33, 2020.
[33] Adam A Kohan, Edward A Rietman, and Hava T Siegelmann. Error forward-propagation: Reusing feedforward connections to propagate enrors in deep learning. arXiv preprint NKKV:18808.0357 2018
[34] Gene F Franklin, David Powell, and Abbas Emami-Naeini. Feedback control of dynamic systems. Pearson London, 2015.
[35] Carl Friedrich Gauss. Theoria motus corporum coelestium in sectionibus conicis solem ambien- fium, volume 7. Perthes et Besser, 1809.
[36] Tianle Cai, Ruiqi Gao, Jikai Hou, Siyu Chen, Dong Wang, Di He, Zhihua Zhang, and Liwei Wang.
method learning overparameterized deep neural networks for regression problems. arXiv preprint arXiv: 1905. 1675, 2019.
[37] Robert Urbanczik and Walter Senn. Learning by the dendritic prediction of somatic spiking. Newron, 81(3)5221-528, 2014.
12

---

[38] A. M. Lyapunov. The general problem of the stability of motion. International Journal of Control, 55(3)531-533, 1992. doi: 10.1080/0020711920999444499
[39] Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The" wake-sleep" algorithm for unsupervised neural networks. Science, 2 1995.
[40] Yann LeCun. The mnist database of handwritten digits. httpp://aamn. lecun. com/exdb/mmist,, 1998.
[41] Han Xiao, Kashif Rasul, and Roland Vollgraf. fashion-mnist novel image dataset for benchmarking machine learning algorithms. arXiv preprint (t0.::770.77444 2017.
[42] Arild Nøkland. Direct feedback alignment provides learning deep neural networks. In Advances in neural information processing systems, pages 1037-1045, 2016.
[43] Simo Särkkä and Amo Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
[44] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, 3rd Interna- tional Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2014.
[45] Leslie G Ungerleider, Thelma W Galkin, Robert Desimone, and Ricardo Gattass. Cortical connections of area v4 in the macaque. Cerebral Cortex, 18(3):477-499 2008.
[46] Kathleen Rockland and Gary W Van Hoesen. Direct temporal-occipital feedback connections striate cortex (v1) in the macaque monkey. Cerebral cortex, 4(3):300-313, 1994.
[47] Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current opinion in neurobiology, 14:28-36,12 2019.
[48] Matthew Larkum. A cellular mechanism for cortical associations: an organizing principle for the cerebral cortex. Trends in neurosciences, 36(3):141-151) 2013.
[49] Nelson Spruston. Pyramidal neurons: dendritic structure and synaptic integration. Nature Reviews Neuroscience, 93)::06-2211 2008
[50] Patrick Xiao, Christopher H Bennett, Ben Feinberg, Sapan Agarwal, and Matthew Marinella. Analog architectures for neural network accelcration based on non-volatile memory. Applied Physics Reviews, 7(3)x031301, 2020.
[51] Janardan Misra and Indranil Saha. Artificial neural networks in hardware: A survey of two decades of progress. Neurocomputing, 74(1-3):239-25,, 2010.
[52] Eliakim Moore. On the reciprocal of the general algebraic matrix. Bull. Am. Math. Soc., 26: 394-395, 1920.
[53] Roger Penrose. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society, volume 51, pages 406-413, Cambridge University Press, 1955.
[54] Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics, 2(2): 1944.
[55] Stephen L Campbell and Carl D Meyer. Generalized inverses of linear transformations. SIAM, 2009.
[56] Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7):1723-173: 2002.
[57] Guodong Zhang, James Martens, and Roger Grosse. Fast convergence of natural gradient descent for over-parameterizee neural networks. In Advances in Neural Information Processing Systems 32, pages 5080-8091, 2019.
[58] H Sebastian Seung. How the brain keeps the eyes still. Proceedings of the National Academy of Sciences, (3339):13339-1334 1996.
[59] Alexei A Koulakov, Sridhar Raghavachari, Adam Kepecs, and John E Lisman. Model for a robust neural integrator. Nature neuroscience, 588)7775-72,, 2002.
[60] Mark S Goldman, Joseph H Levine, Guy Major, David W Tank, and HS Seung. Robust persistent neural activity in a model integrator with multiple hysteretic dendrites per neuron. Cerebral cortex, (185))1185-1199 2003.
13

---

[61] Mark S Goldman, A Compte, and Xiao-Jing Wang. Neural integrator models. Encyclopedia of neuroscience, pages 165-178, 2010.
[62] Sukbin Lim and Mark S Goldman. Balanced cortical microcircuitry for maintaining information in working memory. Nature neuroscience, 16(9): (19):1006-131, 2013.
[63] Bejarano, Eduardo
-oggtenn-oondaaóón, and Enith Amanda Hennzz-Hrráádez. A stability test for non linear systems of ordinary differential equations based on the gershgorin circles. Contemporary Engineering Sciences, 11(91))4541-435 14548, 2018.
[64] James Martens and Roger Grosse. Optimizing neural networks with monecker-factored approx- imate curvature. In Proceedings of the 32nd International Conference on Machine Learning. pages 2408-2482200 2417, 2015.
[65] Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. In Proceedings of the 34th International Conference on Machine Learning. pages 557-565. JMLR. org. 2017.
[66] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[67] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in pytorch. 2017.
[68] James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In Advances in neural information processing systems, pages 2546- 2554, 2011.
[69] James Bergstra, Dan Yamins, and David Cox. Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In Proceedings of the 12th Python in science conference, pages 13-20. Citeseer, 2013.
[70] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph Gonzalez, and lon Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05// 18, 2018.
[71] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[72] R Angus Silver. Neuronal arithmetic. Nature Reviews Neuroscience, 11(7):474-48,, 2010.
[73] Katie A Ferguson and Jessica A Cardin. Mechanisms underlying gain modulation in the cortex. Nature Reviews Neuroxcience, (2):800-9, 2020.
[74] Matthew E Larkum, Walter Senn, and Hans-R Lüscher. Top-down dendritic input increases the gain of layer 5 pyramidal neurons. Cerebral cortex, 14(10);1099-1070 2004.
[75] Richard Naud and Henning Sprekeler. Burst ensemble multiplexing: A neural code connecting dendritic spikes with microcircuits. bioRxiv, page 143636, 2017.
[76] Yoshua Bengio, Dong-Hyun Jorg Bornschein, Thomas Mesnard, and Zhouhan Lin. Towards biologically plausible deep learning. arXiv preprint arXiv: 550.004/56 2015.
14


Reviews:
[{"review_id": "sZMH-xfTJsq", "reviewer": "Reviewer_nh8b", "summary": "The authors introduce Deep Feedback Control, an algorithm for training deep neural networks where credit assignment is done with the help of a proportional-integral-derivative (PID or rather PI in this case) controller. \n\nTheoretically, the authors show that a learning rule similar to what they propose (but not the same), under certain assumptions, gives the Gauss-Newton update for an L2 loss, and under milder assumptions gives the minimum norm updates of W. The authors show that the PI controller helps with stability and perform some experiments to show that the assumptions remain more or less valid during training.\n\n### After rebuttal\n\nBecause of the increased quality of the work stemming from more theoretical arguments and more experiments, I have increased my score from 5 to 8.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nThe novelty of the method is in the use of the PI controller which helps stabilize the learning dynamics. Otherwise, proving theorems in the limit of small nudges, similarly showing a connection with GN is not new. \n\n## Quality\nOverall this is a good paper. It is grounded in theory, guaranteeing performance in certain limits. The assumptions (but not the limits) of the theorems are more or less well explored. However it suffers from some weaknesses that if addressed can drastically improve the quality and the impact of the work. Here are a number of shortcomings in no particular order.\n\n* The exact influence of the PI controller is not clear. It seems like Theorems 2 and 3 would hold also for purely proportional control (setting $\\tau_u=0$), therefore the only effect of the integrative part is in stabilizing dynamics in stabilizing the dynamics such that $Q$ can be learned (sections 4 and 5). Given that without the integral part - that is just having just proportional feedback - brings the submission much closer to prior work, the authors should further clarify exactly where the integral part is essential, and what parts of their arguments would also hold for prior work. (The section in the appendix on related works is an attempt in this direction but still does not clarify where the differences are coming from. Is it in the architecture? In the controller? In the update rules?)\n\n* There are some recent developments missing from the related work section and should be included (e.g  arxiv: 2011.15031).\n\n* In lines (350-351) is is said that forward weight updates are not exactly GN etc. Similarly in Fig. 3 it is said that DFC-SSA is the ideal _damped_ MN update. First, the damped MN or GN update is nowhere defined, but I assume it refers to the fact that theorems 2,3 apply for update in Eq. (9) not the proposed update of Eq. (5). The effect of this seemingly adhoc dampening is mentioned in passing in lines 189 and expanded a little for MN optimization in the appendix, but never verified experimentally. This can be fixed by included MN and GN in Fig 3 and Tab 1.\n\n* Similar to above, theorems 2,3 apply in the limit of $\\alpha,\\lambda\\to0$. Unfortunately this aspect of the algorithm is never explored. In fact, the actual value of the $\\lambda$ (is it actually small?) in experiments is not even found in the 45 page appendix! In biological networks, neither of these parameters can be too small. Because of noise and the limited dynamic range of neurons the nudging would have to be finite. Overall, the authors spend more time examining whether or not the conditions of their theorems are satisfied, than convincing the reader that the theorems are still approximately valid in realistic situations. If the authors spend a little bit of time exploring this, the potential impact of the paper would increase dramatically.\n\n* Comparison to prior work (TP, DTP,  the hybrid GN-SGD of [20] etc) and non-damped GN and MN update. The authors do not answer the question of 'Does DFV perform better than the many (or even any) prior incarnations of TP?'\n\n## Clarity\n\nOverall the paper is clear. There are some small points.should be clarified). \n\n* As mentioned above, the differences with prior work (especially [20]) should be further clarified. For experts actively working on this problem, the submission might look like a big departure from prior work, but for general computational neuroscience audience, it looks like a small evolution of [20,21], with many arguments seemingly applicable to it as well. A clear section, mentioning 1. what exactly is different (architecture, learning rule etc) and 2. what is the effect of each departure from previous work (whether it improves stability or is required for relationship to GN etc.).\n\n* It seems the influence of the controller is a little exaggerated. For example in line 142-143, it is mentioned that the controller dynamically inverts the output error. Judging by Fig. 1, the controller (the box at the top of the circuit) just integrates the error. The dynamical inversion is done in the body of the circuit (even if the controller was just spitting out the error itself).\n\n\n## Significance\n\nThe paper is an evolution of the TP framework. With added clarity and extra experiments it can have a not-insignificant impact.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "qB4Frhivy8g", "reviewer": "Reviewer_Syop", "summary": "This paper builds on recent insights in applying adaptive control theory, Gauss-Newton optimization, target propagation, and dynamical inversion methods to the issue of local and bio-plausible credit assignment in neural networks. The authors propose deep feedback control (DFC) as a principled approach to learning feedforward and feedback weights via local plasticity rules to approximate Gauss-Newton optimization of an output loss function for a neural network.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a masterful paper that lucidly combines several recent insights towards solving the issue of credit assignment via local plasticity rules in the brain and neuromorphic hardware. A number of theorems/propositions are proved along the way to showing the conditions and approximations needed. (I have not checked the proofs, but the theorems and the conditions appear plausible.) A bio-plausible three-compartment neural implementation is also presented. The method is tested on simple benchmarks and shows good performance compared to backpropagation. I expect this work to open new directions as an alternative to approximating backpropagation to obtain brain-like learning rules.\n \nMinor:\nlines 5 and 56: motives -> motifs.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "ovsRBhKSHT", "reviewer": "Reviewer_f7V6", "summary": "This paper proposes a deep feedback control (DFC) method for deep networks, which is biological plausibility, and the control signal can be used for credit assignment. Experiments on MNIST and Fashion MNIST dataset demonstrate the effectiveness of DFC, and the performances is competitive to back-propagation based learning algorithms.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed deep feedback control approach for network learning is interesting and conveys innovate ideas both theoretically and algorithmically. The DFC method can be implemented with local learning, which is biological plausibility. It is also interesting to ask what are the possible applications of DFC based networks? Can the proposed model help explain biological mechanisms in the brain?\n\n=============================\n\n\nPost-rebuttal: \n\nThe authors have addressed most of my concerns. I have increased my score from 6 to 7.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new framework for biological plausible credit assignment algorithms in neural networks. The main idea consists in sending a top-down 'feedback' signal that drives the network to output the desired values, and update weights to make the feedback correction unneeded. They show a connection between the resulting algorithm and Gauss-Newton optimization. The work has connection to previous biological CA literature, in particular work on target propagation and feedback alignment.
Reviewers thought the contribution was insightful, and well written. Initial reviews brought about some concerns regarding novelty and what the precise contributions of some components were (whether they were important or needed at all, like the PI controller), which were extensively discussed in the rebuttal period. Please adjust the final version of the paper by taking these discussions into account.
A final improvement could include a more formal, clear theorem connecting DFC to GN: at the moment, it feels that one needs to connect theorem 1 and 2 to fully get the connection between the two. The relation between DFC and GN would ideally be made clear and rigorous in a single theorem, without having to connect dots.

Decision: Accept (Spotlight)