Learning to solve the credit assignment problem
Authors: Benjamin James Lansdell
Prashanth Ravi Prakash
Konrad Paul Kording
Venue:  (2020)
Keywords: biologically plausible deep learning
node perturbation
REINFORCE
synthetic gradients
feedback alignment

Abstract: Backpropagation is driving today's artificial neural networks (ANNs). However, despite extensive research, it remains unclear if the brain implements this algorithm. Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient. However, the convergence rate of such learning scales poorly with the number of involved neurons. Here we propose a hybrid learning approach. Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide. We provide proof that our approach converges to the true gradient for certain classes of networks. In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match the performance of gradient-based learning. Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.

Published as a conference paper at ICLR 2020
LEARNING TO SOLVE THE CREDIT ASSIGNMENT PROBLEM
Benjamin James Lansdell Department of Bioengineering University of Pennsylvania Pennsylvania, PA 19104 Mansdell@sees upenn edu
Prashanth Ravi Prakash
Department Bioengineering University of Pennsylvania Pennsylvania, PA 19104
Konrad Paul Kording Department of Bioengineering University of Pennsylvania Pennsylvania, PA 19104
ABSTRACT
Backpropagation is driving today's artificial neural networks (ANNs). However, despite extensive research, it remains unclear if the brain implements this algo- rithm. Among ncuroscientists, reinforcement learning (RL) algorithms are often realistic alternative: neurons can randomly introduce change, and use specific feedback signals to observe their effect on the cost and thus approximate their gradient. However, the convergence rate of such learning scales poorly with the number of involved neurons. Here we propose a hybrid learning approach. Each neuron uses an RL-type strategy learn how to approximate the gradients that backpropagation would provide. We provide proof that our approach con- verges the true gradient for certain classes of networks. In both feedforward and convolutional networks, we empirically show that our approach learns ap- proximate the gradient, and can match or the performance of exact gradient- bradient-based learning. Learning feedback weights provides biologically plausible mechanism achieving good performance, without the need for precise, pre-specified learn- ing rules.
INTRODUCTION
It is unknown how the brain solves the credit assignment problem when learning: how does each neuron know its role in a positive (or negative) outcome, and thus know how to change its activity perform better next time? This is challenge for models of learning in the brain.
Biologically plausible solutions to credit assignment include those based on reinforcement learn- ing (RL) algorithms and reward-modulated STDP (Bouvier al.) 2016 Ficte et al. 2007 Fiete & Seung 2006; Legenstein et al., 2010 Miconi 2017). In these approaches globally distributed reward signal provides feedback all neurons network. Essentially, changes in rewards from a baseline, or expected, level are correlated with noise in neural activity, allowing a stochastic ap- proximation of the gradient be computed. However these methods have not been demonstrated to operate at scale. For instance, variance in the REINFORCE estimator (Williams) 1992) scales with the number units in the network (Rezende et al.] 2014). This drives hypothesis that learning in the brain must rely additional structures beyond global reward signal.
In artificial neural networks (ANNs), credit assignment performed with gradient-based methods computed through backpropagation (Rumelhart et al. 1986 Werbos 1982 Linnainmaa 1976). This significantly more efficient than RL-based algorithms, with ANNs now matching or surpassing human- performance in number of domains (Mnih et al.) 2015 Silver et al. 2017 LeCun et al. 2013). He et al.] 2013), Haaenslee et al.] 22188] [Russakovsky et al. 2013). However there are well known problems with implementing backpropagation in biologically realistic neural networks.

---

Published as a conference paper at ICLR 2020
One problem is known as weight transport (Grossberg 1987): propagation requires a feedback structure with the same weights as the feedforward network to communicate gradients. Such a symmetric feedback structure has not been observed in biological neural circuits. Despite such issues, backpropagation the only method known to solve supervised and reinforcement learning problems at scale. Thus modifications approximations to backpropa- gation that are more plausible have been the focus of significant recent attention (Scellier & Bengio
exact implementation of back-
2016 Lillicrap al. 2016 Lee al. 2015 Lansdell & Kording 2018 Ororbia al. 2018)).
These efforts do show some ways forward. Synthetic gradients demonstrate that learning can be based on approximate gradients, and need not be temporally locked (Jaderberg et al. 2016 Czar- necki et al. 2017b). In small feedforward networks, somewhat surprisinly, fixed random feedback matrices in fact suffice for learning (Lillicrap et al., 20T6) (a phenomenon known as feedback align- ment). But still issues remain: feedback alignment does not work in CNNs, very deep networks, or networks with tight bottleneck layers. Regardless, these results show that rough approximations of a gradient signal can be used to learn; even relatively inefficient methods of approximating the gradient may be good enough.
On this basis, here we propose an RL algorithm to train a feedback system to enable learning. Recent work has explored similar ideas, but not with the explicit goal of approximating backpropagation (Miconi 2017 Miconi et al. 2018 Song et al. 2017). RL-based methods like REINFORCE may be inefficient when used as a base learner, but they may be sufficient when used to train a system that itself instructs base learner. We propose to REINFORCE-style perturbation approach to train feedback signals to approximate what would have been provided by backpropagation.
This sort of two-learner system, where one network helps the other learn more efficiently, may in fact align well with cortical neuron physiology. For instance, the dendritic trees of pyramidal neurons consist of an apical and basal component. Such setup has been shown to support supervised learning in feedforward networks (Guergiuev et al., 2017 Kording & Konig (001). Similarly, climbing fibers and Purkinje cells may define a learner/teacher system in the cerebellum (Marr) 1969). These components allow for independent integration of two different signals, and may thus provide a realistic solution to the credit assignment problem.
Thus we implement a network that learns to use feedback signals trained with reinforcement learn- ing via a global reward signal. We mathematically analyze the model, and compare its capabilities to other methods for learning in ANNs. We prove consistency of the estimator in particular cases, extending the theory of synthetic gradient-like approaches (Jaderberg et al. 2016 Czarnecki et al.) 2017b. Werbbos 1992; Schmidhuber) 1990). We demonstrate that our model learns as well as reg- ular backpropagation in small models, overcomes the limitations of feedback alignment on more complicated feedforward networks, and can be used in convolutional networks. Thus, by combining local and global feedback signals, this method points to more plausible ways the brain could solve the credit assignment problem.
2 LEARNING FEEDBACK WEIGHTS THROUGH PERTURBATIONS
We use the following notation. Let x E Rm represent an input vector. Let an hidden-layer network be given by ý f(x) R". This is composed of set of layer-wise summation and non-linear activations
h = ('hh"tt-- (w'h-t)
for hidden layer states h' E RR. non-linearity σ, weight matrices W E R and denoting hd x and N+1 G. Some loss function defined in terms of the network output: L(y, ÿ). Let C denote the loss as function of y): L(x, y) L(y, f(x).. Let data y) E D be drawn from a distribution p. We aim to minimize: Ep [C(x,y]]
Backpropagation relies the error signal et, computed in top-down fashion:
Jac/oý o (W"hi- ((W++))Te ( ( (W))))) ISigN
i= ++
where denotes element- multiplication.
2

---

Published as a conference paper at ICLR 2020
A)
B)
C
L
W
x
B
h
h2
W
B
h
h
Figure 1: Learning feedback weights through perturbations. (A) Backpropagation sends error infor- mation from an output loss function, L. through each layer from top to bottom via the same matrices W used in the feedforward network. (B) Node perturbation introduces noise each layer, &,, that perturbs that layer's output and resulting loss function. The perturbed loss function, L. correlated with the noise give an estimate of the enror current. This estimate is used to update feedback matrices B to better approximate the error signal.
2.1
BASIC SETUP
Let the loss gradient term be denoted as
OC oh
X
(Wit
In this work replace X' with approximation with its parameters to be learned (known synthetic gradient, or conspiring network, (Jaderberg et al. 2016 Czarneck et al.) 2017b), or error critic (Werbos 1992)):
x g(' e(+1;0),
for parameters Note that we must distinguish true loss gradients from their synthetic estimates. Let é' be loss gradients computed by backpropagating the synthetic gradients
SaC/0y o (W'h (g(h', el o (W'h')) ISiSN
i=
For the final layer the synthetic gradient matches the true gradient: N+1 = N+1 This setup can accommodate both top-down and bottom-up information, and encompasses number of published
models (Jaderberg el al. 2016; Czarnecki et al., 2017b Lillicrap et al. 2016 Nøkland 2016 Liao
al. Xiao al. 2018).
2.2
STOCHASTIC NETWORKS AND GRADIENT DESCENT
To learn a synthetic gradient we utilze the stochasticity inherent to biological neural networks. A number of biologically plausible learning rules exploit random perturbations in neural activity (Xie & Seung 2004; Seung 2003; Ficte Seung, 20066) Fiete et al. 2007 Song et al. 2017). Here, at
each time each unit produces noisy response:
hj Σ With + ca
for independent Gaussian noise E¹ == - 1) and standard deviation ca 0. This generates noisy loss L(x,y, y, { and baseline loss L(x,y) y) L(x, y. 0). We will use the noisy response to estimate gradients that then allow us to optimize the baseline C the gradients used for weight updates are computed using the deterministic baseline.

---

Published as a conference paper at ICLR 2020
2.3 SYNTHETIC GRADIENTS VIA PERTURBATION
For Gaussian white noise, the well-known REINFORCE algorithm (Williams) 1992) coincides with
the node perturbation method (Fiete Seung 2006 Fiete al. 2007). Node perturbation works by linearizing the loss:
dc RxL+ Thi
cnCj.
such that
0c ch Oh
I ((C- D).xE:
x.y
with expectation taken over the noise distribution (E). This provides estimator of the loss gradi- ent
X = ( L(x,
(2)
This approximation made more precise in Theorem
(Supplementary material).
2.4 TRAINING A FEEDBACK NETWORK
There are many possible sensible choices of g(-). For example, taking g as simply a function of each layer's activations: X' g(h)) ) is in fact sufficient parameterization to express the true gradient function (Jaderberg et al. 2016). We may expect, however, that the gradient estimation problem be simpler if each layer provided with some propagated in a top-down fashion. Symmetric feedback weights may not be biologically plausible, and random fixed weights may only solve certain problems of limited size complexity (Lillierap et al., 2016). However, system that can learn to appropriate feedback weights B may be able to align the feedforward and feedback weights as much as needed successfully learn.
information obtained from the loss function and
We investigate various choices of g(' Bit1) outlined in the applications below. Parameters Bi+1 are estimated by solving the least squares problem:
g('', e X$ 1
Bi+1 arg min E B
(3)
Unless otherwise noted this solved by gradient-descent, updating parameters once with each minibatch. Refer to the supplementary material for additional experimental descriptions and param- eters.
THEORETICAL RESULTS
We can prove the estimator (3) is consistent the noise variance G₁. - in some particular cases. We state the results informally here, and give the exact details in the supplementary materials. Consider first convergence of the final layer feedback matrix, BN1
Theorem 1. (Informal) For gFA(h',
(h',e e Bit1) git1 then the least squares estimator
(c
solves 3) and converges to the true feedback matrix, in the sense that: lime, +0 plimp WN+ N +1 where plim indicates convergence in probability.
BN+1
Theorem thus establishes convergence of B in a shallow (1 hidden layer) non-linear network. In deep, linear network
also use Theorem to establish convergence over the rest of the layers.
Theorem 2. (Informal) For gFA Bi+1) Bitløt+1 and o o(x) = the least squares estimator
T (o(())))
I
(5)
solves and converges to the true feedback matrix, the sense that: lim p oo W
B
Si +1.

---

Published as a conference paper at ICLR 2020
A)
B)
G)
D)
1
relative ortor
alignment gradient
100
sign congruence
10
error
80
[Ma///]]
60
layer
10
angle
40
28
88 10
4.3%
20
50
3.5%
10-1
0-
10
2.6%
0
1000 noration (x10%)
2000
1y=10
wyye2
[w-B)WW]
10
60
40
8
75
9,-10%
ch 10
node perturbation
C)=10
20
50
feodback alignment
10-1
back prop
1000
2000
1000
2000
0
1000
2000
Aaration (xt0²)
Iteration (x10²)
Iteration (x10²)
Figure 2: Node perturbation in small 4-layer network (784-50-20-10 neurons), for varying noise levels c, compared to feedback alignment and backpropagation. (A) Relative error between feedfor- ward and feedback matrix. (B) Angle between true gradient and synthetic gradient estimate for each layer. (C) Percentage of signs W and B' that are agreement. (D) Test error for node perturba- tion, backpropagation and feedback alignment. Curves show mean plus/minus standard error over 5 runs.
Given these results we can establish consistency for the 'direct feedback alignment" (DFA; Nøkland (2016)) estimator: gDFA (
@N+1; B(+1) B(+2) = (Bi+))T@N++, Theorem applies triv-
ially since for the final layer, the two approximations have the same form: gF [N+1:0N)
ADFA(h' N NN+1 (x)). Theorem 2 can be casily extended according to the following:
Corollary 1. (Informal) For ADPA(h', N41 Bi+l) Bi+le and (x) = I, the least squares
estimator
(B") = =
15n
(6)
solves 3 and converges to the true feedback matrix, in the sense that: lime, plimy Bi
- WD,
+iiSN+1.
Thus for non-linear shallow network deep linear network, for both 9FA and GDFA. have the result that, for sufficiently small cj. if we fix the network weights W and train B through node perturbation then converge W. Validation that the method learns approximate W, for fixed W. is provided in the supplementary material. In practice, we update B and W simultaneously.
Some convergence theory established for this case in (Jaderberg et al. 2016 Czarnecki et al., 2017b).
4 APPLICATIONS
4.1 FULLY CONNECTED NETWORKS SOLVING MNIST
First we investigate g(h Tit1; 1)+ = (Bi+))Te"* which describes a non-symmetric feedback network (Figure D.. To demonstrate the method can be used to solve simple supervised learning problems we use node perturbation with a four-layer network and MSE loss to solve MNIST (Figure 2}. Updates to W' are made using the synthetic gradients AW' = n for learning rate 1). The feedback network needs to co-adapt with the feedforward network in order to continue to provide a useful error signal. We observed that the system is able to adjust to provide close correspondence between the feedforward and feedback matrices relative error between B and W is lower than what is observed for feedback alignment, suggesting that this oo-adaptation of both 4 and B' is indeed beneficial. The relative error depends on the amount of noise used in node perturbation lower variance doesn't necessarily imply the lowest
both layers of the network (Figure 2 The
5

---

Published as a conference paper at ICLR 2020
A)
backpropagation
1000
feedback alignment
ADAM
backpropagation
ADAM
backpropagation
node perturbation backprop (ADAM)
9
3 6
cutput
matched
9
9 0 3 0
800
DAE
input
2
Y
4
& 2
loss
9
4
i
700
feedback alignment
node
feedback alignment
node
perturbation
perturbation
600
? 5 $
4
g 2 5
500
output
g Q 8 8
2
2 4 $ 2 4
5
6
200
400
600
800
1000
input
9 6
lo
2 o
F.
iteration (x10")
Figure 3: Results with five-layer MNIST autoencoder network. (A) Mean loss plus/minus standard error over 10 runs. Dashed lines represent training loss, solid lines represent test loss. (B) Latent space activations, colored by input label for each method. (C) Sample outputs for each method.
error between W and B, suggesting there is an optimal noise level that balances bias in the estimate and the ability to co-adapt to the changing feedforward weights
Consistent with the low relative error in both layers, we observe that the alignment (the angle be- tween the estimated gradient and the true gradient proportional to e W B ē) is low in each layer much lower for node perturbation than for feedback alignment. again suggesting that the method is much better at communicating eπгоr signals between layers (Figure 2 In fact, recent studies have shown that sign congruence the feedforward and feedback matrices all that required to achieve good performance (Liao et al.) [2066] Xiao et al., 2018). Here the sign congruence is also higher in node perturbation, again depending somewhat the variance. The amount congruence is comparable between layers (Figure 2C). Finally, the learning performance of node perturbation is comparable to backpropagation (Figure[D) and better than feedback alignment in this case, though not by much. Note that by setting the feedback learning rate to zero, recover the feedback alignment algorithm. So we should expect to be always able to do at least as well as feedback align- ment. These results instead highlight the qualitative differences between the methods, and suggest that node perturbation for learning feedback weights can be used to approximate gradients in deep networks.
4.2 AUTO-ENCODING MNIST
The above results demonstrate node perturbation provides error signals closely aligned with the true gradients. However, performance-wise they do not demonstrate any clear advantage over feed- back alignment backpropagation. A known shortcoming of feedback alignment is in very deep networks and in autoencoding networks with tight bottleneck layers (Lillicrap et al. 2016). To if node perturbation has the same shortcoming, we test performance of g(h', ē e,,,, (8++1 model on a simple auto-encoding network with MNIST input data (size 784-200-2- 200-784). In this more challenging case we also compare the method to the 'matching' learning rule (Rombouts et al. 2015; Martinolli et al. 2018), in which updates to B match updates to W and weight decay is added, denoising autoencoder (DAE) (Vincent et al. 2008), and the ADAM Kingma & Ba 2015) optimizer (with backprop gradients).
As expected, feedback alignment performs poorly, while node perturbation performs better than backpropagation (Figure 3). The increased performance relative to backpropagation may seem surprising. A possible reason is the addition of noise our method encourages learning more robust latent factors (Alain & Bengio, 2015). The DAE also improves the loss over vanilla back- propagation (Figure (FigureBAA). And, in line with these ideas, the latent space learnt by node perturbation shows more uniform separation between the digits, compared to the networks trained by backprop- agation. Feedback alignment, in contrast, does not learn to separate digits in the bottleneck layer all (Figure B), resulting in scrambled output (Figure 3C). The matched learning rule performs similarly to backpropagation. These possible explanations are investigated more below. Regardless,
!Code to reproduce these results can be found at: https: thub. cwm...com/oooc..cccooooooc...tot booo//ben/ ansdel1 c
synthfeedback

---

Published as a conference paper at ICLR 2020
these results show that node perturbation is able successfully communicate signals through thin layers of a network as needed.
4.3 CONVOLUTIONAL NEURAL NETWORKS SOLVING CIFAR
Convolutional networks are another known shortcoming of feedback aligmment. Here we test the method convolutional neural network (CNN) solving CIFAR (Krizhevsky) (009). Refer to the supplementary material for architecture and parameter details. For this network we learn feedback weights direct from the output layer each earlier layer: g(h', ; B++1)= (Bi+1)T& (sim- ilar to *direct feedback alignment" (Nokland, 2016)). Here this was solved by gradient-descent, On CIFARIO we obtain a test accuracy of 75%. When compared with fixed feedback weights and backpropagation, we see it is advantageous to learn feedback weights on CIFAR10 10 and marginally advantageous on CIFARI00 100 (Table 1). This shows the method can be used in a CNN, and can solve challenging computer vision problems without weight transport.
Table 1: Mean test accuracy of CNN over 5 runs trained with backpropagation, node perturbation and direct feedback alignment (DFA) (Nøkland 2016 Crafton et al. 2019).
dataset
backpropagation
node perturbation
DFA
CIFARIO
76.910.1
74.8
72.41:.2
CIFAR100
51.2++0.1
48.1+0..2
47.3+0.1
4.4 WHAT IS HELPING, NOISY ACTIVATIONS APPROXIMATING THE GRADIENT?
To solve the credit assignment problem, our method utilizes two well-explored strategies in deep learning: adding noise (generally used to regularize (Bengio et al. 2013: Gulcchre et al. 2016 Neelakantan et al. 2015 Bishop 1995)), and approximating the true gradients (Jaderberg et al, 2016). To determine which of these features are responsible for the improvement in performance over fixed weights, in the autoencoding and CIFARIO cases, study the performance while varying where noise is added to the models (Table 2). Noise can be added to the activations (BP and FA w. noise, Table 2), or the inputs, in a denoising autoencoder (DAE, Table 2). Or, noise can be used only in obtaining an estimator of the true gradients (as in our method; NP, Table 2). For comparison, noiseless version of our method must instead assume access to the true gradients, and use this to learn feedback weights (i.e. synthetic gradients (Jaderberg et al. 2016);; SG, Table 2). Each of these models is tested on the autoencoding and CIFARTO tasks, allowing us better understand the performance of the node perturbation method.
Table 2: Mean loss (plus/minus standard error) autoencoding MNIST task (left) and mean accu-
racy on CIFARIO 10 task (right). Shaded cells indicate methods which do not use weight transport or exact gradient supervision. Best performance indicated in boldface. Implementation details of each method is provided in the supplementary material.
(a) MNIST autoencoder
(b) CIFARIO classification
method
noise
noise
method
noise
noise
BP(SGD)
536.8:2.1
609.8:11.4
BP
76.8
76.910.1
BP(ADAM)
522.3++0.4
533.3:+2.2
DFA
72.4+002
72.3+0.1
FA
768.2+2.7
759.1=3.3
NP (ours)
74.8+0.2
DAE
539.8
SG
75.
NP (ours)
SG 515.3
521.6442.3
Matched
629.9E1.1
615,0
In the autoencoding task, both noise (either in the inputs the activations) and using approxima- tor to the gradient improve performance (Table 2, left). Noise benefits performance for both SGD optimization and ADAM (Kingma & Ba 2015). In fact in this task, the combination of both of
7

---

Published as a conference paper at ICLR 2020
these factors (i.e. our method) results in better performance over either alone. Yet, the addition of noise to the activations does not help feedback alignment. This suggests that our method is indeed learning useful approximations of the enror signals, and is not merely improving due to the addition of noise to the system. In the CIFAR10 10 task (Table 2, right), the addition of noise to the activations has minimal effect performance, while having access to the true gradients (SG) does result in improved performance over fixed feedback weights. Thus in these tasks it appears that noise does not always help, but using a less-based gradient estimator does, and noisy activations are one way of obtaining an unbiased gradient estimator. Our method also is the best performing method that does not require either weight transport or access to the true gradients as supervisory signal.
5 DISCUSSION
Here we implement a perturbation-base synthetic gradient method to train neural networks. We show that this hybrid approach can be used in both fully connected and convolutional networks. By removing the symmetric feedforward/feedbace weight requirement imposed by backpropagation, this approach is a step towards more biologically-plausbb deep learning. By reaching compara- ble performance to backpropagation on MNIST, the method able to solve larger problems than perturbation-only methods (Xie & Seung, 2004 Fiete et al., 2007 Werfel et al. 2005). By working in cases that feedback alignment fails, the method can provide learning without weight transport in more diverse set of network architectures. We thus believe the idea of integrating both local and global feedback signals a promising direction towards biologically plausible learning algorithms.
Of course, the method does not solve all issues with implementing gradient-based learning in biologically plausible manner. For instance, in the current implementation, the forward and the backwards passes are locked. Here just focus on the weight transport problem. A current draw- back is that the method does not reach state-of-the-an performance on more challenging datasets like CIFAR. We focused on demonstrating that it is advantageous to learn feedback weights, when compared with fixed weights, and successfully did so in a number of cases. However, we did not use any additional data augmentation and regularization methods often employed reach state-of- the-art performance. Thus fully characterizing the performance of this method remains important future work. The method also does not tackle the temporal credit assignment problem, which has also seen recent progress in biologically plausible implementation Ororbia et al. (2019bJa).
However the method does has a number of computational advantages. First, without weight trans- port the method has better data-movement performance (Crafton et al., 2019; Akrout et al., 2019), meaning may be more efficiently implemented than backpropagation on specialized hardware. Second, by relying random perturbations to measure gradients. the method does not rely on the environment to provide gradients (compared with e.g. CZarneeki et al. (2017a); Jaderberg et al. 2016)). Our theoretical results are somewhat similar to that of AllAn & Bengio 2015), who demon- strate that a denoising autoencoder converges to the unperturbed solution as Gaussian noise goes to zero. However our results apply to subgaussian noise more generally.
While previous research has provided some insight and theory for how feedback alignment works (Lillicrap al. 2016 Ororbia et al. 2018 Moskovitz et al. 2018: Bartunov et al. 2018 Baldi al., 2018) the effect remains somewhat mysterious, and not applicable in some network architectures. Recent studies have shown that some of these weaknesses can be addressed by instead imposing sign congruent feedforward and feedback matrices (Xiao et al. 2018). Yet what mechanism may produce congruence biological networks unknown. Here we show that the shortcomings of feedback alignment can be addressed in another way: the system can learn to adjust weights as needed to provide a useful error signal. Our work is closely related Akrout et al. (2019). which also uses perturbations to learn feedback weights. However our approach does not divide learning into two phases, and training of the feedback weights does not occur in a layer-wise fashion, assuming only one layer is noisy at time, which is strong assumption. Here instead we focus on combining global and local learning signals.
Here we tested our method in an idealized setting. However the method is consistent with neuro- biology in two important ways. First, it involves separate learning of feedforward and feedback weights. This is possible in cortical networks, where complex feedback connections exist between layers (Lacefied et al. Richards & Lillicrap, 2019) compartments that allow for separate integration of feedback and feedforward signals (Guerguiev
(0019) and pyramidal cells have apical and basal
8

---

Published as a conference paper at ICLR 2020
et al. 2017 Körding & König 2001). A recent finding that apical dendrites receive reward informa-
tion is particularly interesting (Lacefield et al. 2019). Models like Guerguiev et al. (2017) show how the ideas in this paper may be implemented in spiking neural networks. We believe such models can be augmented with a perturbation-ban rule like ours to provide a better learning system.
The second feature is that perturbations used to learn the feedback weights. How can ron measure these perturbations? There are many plausible mechanisms (Seung) 2003 Xie & Se- ung 2004, Fiete & Seung 2006, Fiete al. 2007). For instance, birdsong learning uses empiric synapses from area LMAN (Fiete el al. 2007). others proposed it is approximated (Legenstein et al.)
2010 Hoerzer et al., 2014). neurons could use a learning rule that does not require knowing the
noise [Lanselll & Kording 2018). Further, our model involves the subtraction of a baseline loss to reduce the variance of the estimator. This does not affect the expected value of the estimator technically the baseline could be removed or replaced with an approximation (Legenstein et al,, Loewenstein & Seung, 2006). Thus both separation of feedforward and feedback systems and perturbation-baseed estimators can be implemented by neurons.
2010
As RL-based methods do not scale by themselves, and exact gradient signals are infeasible, the brain may well use a feedback system trained through reinforcement signals to usefully approximate gradients. There a large space of plausible learning rules that can learn to use feedback signals in order to more efficiently learn, and these promise inform both models of learning in the brain and learning algorithms artificial networks. Here we take an early step in this direction.
REFERENCES
Mohamed Akrout, Collin Wilson, Peter C Humphreys, Timothy Lillierap, and Douglas Tweed. Deep Learning without Weight Transport. ArXiv e-prints, 2019.
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. Journal of Machine Learning Research, 15:3563-3593, 2015. ISSN 15337928.
Pierre Baldi, Peter Sadowski, and Zhiqin Lu. Learning in the Machine: Random Backpropagation and the Deep Learning Channel. Artificial Intelligence, 260: 1-3,, 2018. ISSN 00043702. doi: 10.1016/j. 2018201113300 8.03.003. URL http:://axii.coooo//// org. abs////b///// http::/://arrtv..rrraa//////////
Sergey Bartunov, Adam Santoro, Blake Richard, Geoffrey Hinton, and Timothy Lillicrap. Assessing the scalability of betooiicall-mmoivvani deep learning algorithms and architectures. ArXiv e- prints, 2018. ISSN 18979483. doi: 10.20452/pamw.328.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. Advances in Neural Information Processing Systems, pp. 1-9, 2013. ISSN 10495258.
Chris M. Bishop. Training with Noise is Equivalent Tikhonov Regularization. Neural Computa- tion, 7(1):108-116, 1995. ISSN 0899-7667. doi: 10.1162/neco.19/0 1995.7.1.108.
Guy Bouvier, Claudia Clopath, Célian Bimbard, Jean-Pierne Nadal, Nicolas Brunel, Vincent Hakim, and Boris Barbour. Cerebellar learning using perturbations. bioRxiv, pp. 053785, 2016. doi: 10.1101/053785. URL
Brian Crafton, Abhinav Parihar, Evan Gebhardt, and Arijit Raychowdhury. Direct Feedback Align- ment with Sparse Connections for Local Learning. ArXiv e-prints, pp. 1-13, 2019.
Wojeiech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pas- canu. Sobolev training for neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Curran Associates, Inc., 2017a. http::/pppp: http://apppp.ee/ee/ nips.
Systems 30, pp. cc/ htc/paper/7015 soboleve seteev-trriiinng for neural wwwworks..oo.oooomommm. foresna....e..ttrrrrssss..
Wojeiech Marian Czamecki. Grzegorz Šwirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding Synthetic Gradients and Decoupled Neural Interfaces. ArXiv e-prints, 2017b. ISSN 1938-7228. Utt.http::/arxxiv.oo
00522

---

Published as a conference paper at ICLR 2020
lla R Ficte and H Sebastian Seung. Gradient learning in spiking neural networks by dynamic pertur- bation of conductances. Physical Review Letters, 97, 2006. doi: 10.1105/hhyRRewLettt.s 97.048 104.
lla R Fiete, Michale S Fee, and H Sebastian Seung. Model of Birdsong Learning Based on Gradient Estimation by Dynamic Perturbation of Neural Conductances. Journal of neurophysiology, 98:
2038-2057 2007. doi: 10.1152//jn011. 01.1...000.........1
Stephen Grossberg. Competitive learning: From interactive activation to adaptive reso- nance. Cognitive Science, 11(1):23 63, 1987. ISSN 0364-0213. doi: htts:///oi.org/
0010166$00644-001334777888222333 URL http:://////tt enceencedireeee www.coooccscce..mo scc.//cc../ ence/ article/p11/0006402222
Jordan Guergiuev, Timothy P. Lillicrap, and Blake A. Richards. Towards deep learning with seg-
regated dendrites. eliife, 6:1-37, 2017. ISSN 2050-084X. doi: N0.7554/el.ife.22901 URL
http://arxiv/ org/ abs/ www.aals.................aam...a 111600000611
Jordan Guerguiev, Timothy P Lilliccapp, and Blake A Richards. Towards deep learning with segre- gated dendrites. Elife. 6, December 2017.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. 33rd International Conference on Machine Learning, ICML 2016, 6:4457-4466, 2016.
H A Haenssle, Fink, R Schneiderbauer, F Toberer, T Buhl, A Blum, A Kalloo, Ben Hadj Hassen, L Thomas, A Enk, L Uhlmann, and Reader study level-I and level-II Groups. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol., 29(8): 1836-1842, August 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing Human-Level performance on ImageNet classification. In 2015 IEEE International Conference Computer Vision (ICCV), 2015.
Gregor M. Hoerzer, Robert Legenstein, and Wolfgang Maass. Emergence of complex computational structures from chaotic neural networks through seward-modulated hebbian learning. Cerebral Cortex, 24(3):677-690, 2014. ISSN 10473211, doi: 10. 10.10///erroo//bhs34....
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled Neural Interfaces using Synthetic Gradients. ArXiv e-prints, 2016. ISSN 1938-7228. URL
05343
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR 2015, pp. 1-15, 2015. ISSN 09252312. doi: http://doi.acmmorgg wwt.:/wwww.....ooo.............. 1830503. URL
abs/////bs//// htt.aa////11...1. 6980
Konrad Kording and Peter Konig. Supervised and Unsupervised Leanning with Two Sites of Synap- tic Integration. Journal of Computational Neuroscience, 1 11:207-215. 2001.
Konrad P Körding and Peter König. Supervised and unsupervised learning with two sites of synaptic integration. Journal of computational neuroscience, 11(3):207-215,22) 2001.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. ISSN 00012475.
Clay Lacefield, Eftychios Pnevmatikakis, Liam Paninski, and Randy M Bruno. Reinforcement Learning Recruits Somata and Apical Dendrites across Layers of Primary Sensory Cortex. Cell Reports,
2008 2019. ISSN 2 doi: 10.
URL
hittps://dos .rgg 0. 1016/1 rep. 2019 9.199... 01 093.
Benjamin James Lansdell and Konrad Paul Kording. Spiking allows neurons to estimate their causal effect. bioRxiv, pp. 1-19, 2018.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, May 2015.
10

---

Published as a conference paper at ICLR 2020
Dong Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Biotnformatics), 9284:498-511 2015. ISSN 16113349. doi: 10.1007/ 978-3-319-23522 31.
Robert Legenstein, Steven M. Chase, Andrew B. Schwartz, Wolfgang Maas, and W. Maass. A Neward-Modulater Hebbian Learning Rule Can Explain Experimentally Observed Network Re- organization in Brain Control Task. Journal of Neuroscience, 30(25):8400-83 8410, 2010. ISSN
0270-6474. doi: 10. 1522/IEEEOOSTIIASS------- 2010. URL http:/ /ttp:://www.// /www. httt::/wwwwjjeerross.. ineurosci. org/ /doi /1000111000 22R33/JNNUROSSC 42281-44.84111111111
Qianli Liao, Joel Z. Leibo, and Tomaso Poggio. How Important is Weight Symmetry in Backprop- agation? AAAI, 1, 2016. URL htth:tp:/// //axxiv org/ abs/1510. 05067
Timothy Lillicrap. Daniel Cownden, Douglas Tweed, and Colin J Akerman. Random feed- back weights support learning deep neural networks. Nature Communications, 7:13276, 2016. ISSN 2041-1723. doi: 00.1038/ncomms1320
URL /tttp://dx.oo. http:///ddd......g///.././ nature com/ 1tt.:/mmeeee....100oonoooommmms.
ncomms http:m111166hht////m
Seppo Linnainmaa. Taylor expansion the accumulated rounding error. BIT., 16(2):146,160) 1976. ISSN 0006-3835.
Y. Loewenstein and H. S. Seung. Operant matching is generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proceedings of the National Academy of Sciences, 103(41):15224-1522, 2006. ISSN 0027-8424. doi: 10. 11w.////..00000200002
URL
http:// www. pnas. org httpr///////////////////ooo//o// httii//oo//iie 1073// ppw./////p/p////////.a//..s/asa/ 0505220103
David Marr. A theory of cerebellar cortex. J. Physiol, 202:437-470, 1969. ISSN 0022-3751. doi:
10.2307/1776957.
Marco Martinolli, Wulfram Gerstner, and Aditya Gilra. Task Repertoire in Reinforcement Learning Network With Attention- Gated Memory. Front. Comput. Neurosci..... 120ull)):-15, 2018. doi: 10.3389/finom..200 0tt99//fncom2288.000003. 8.00050.
Memory Dynamics Extend
Thomas Miconi. Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks, eLife, 6:1-24, 2017. ISSN 2050084X. doi: 10.7554/ eLife.20899
Thomas Miconi, Jeff Clune, and Kenneth O. Stanley. Differentiable plasticity: training plastic neural networks with backpropagation. ArXiv e-prints, 2018. ISSN 1938-7228. doi: arXiv: 1804.02464v2. URL httpttt//aaxxv..ooo
abs 1804. 02464
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529- 533, February 2015.
Theodore H. Moskovitz, Ashok Litwin-kumar, and L.f.. Abbott. Feedback alignment deep convolutional networks. arXiv Neural and Evolutionary Computing, pp. 1-10, 2018. doi:
arXiv:1812.06488VV. /tLp:ttp://arx.t www.://www..g/a/////a//aa/ 06488
Arvind Neelakantan, Luke Vilnis, Quoc V. Lee, llya Sutskever, Lukasz. Kaiser, Karol Kurach, and James Martens. Adding Gradient Noise Improves Leaming for Very Deep Networks. pp. 1-11, 2015. URL
Arild Nøkland. Direct Feedback Alignment Provides Learning in Deep Neural Networks. Advances in neural information processing systems, 2016.
Alexander Ororbia, Ankur Mali, C. Lee Giles, and Daniel Kifer. Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations, IEEE Transactions Neural Networks and Learning Systems, pp. 1-13, 2019a. URL
obtp:////r//////////ooo/.ooss//s /88C..
07411
11

---

Published as a conference paper at ICLR 2020
Alexander Ororbia, Ankur Mali, Daniel Kifer, and C. Lee Giles. Lifelong Neural Predictive Coding: Sparsity Yields Less Forgetting when Learning Cumulatively. Arxiv e-prints, pp. 1-11, 2019b. URL httt:://aaxxi... org/ /905//995.... 10696
Alexander G. Ororbia, Ankur Mali, Daniel Kifer, and C. Lee Giles. Conducting Credit Assignment by Aligning Local Representations. ArXiv e-prints, pp. 1-27, 2018. URL http:://rxiv.
org BTg/aba/188 1103. 834
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the 3 International Conference on Machine Learning, PMLR, 32(2):1778-1286, 2014. ISSN 10495258. doi: 4082
http/riiiiiie..i. URL http:://aaxx....rrra/b//////////
Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem.
Current Opinion in Neurobiology, 54:28-36, 2019. ISSN 0959-4388. doi: 10.10166j.conb.001. 08.003.
2018 08. 003
Jaldert O Rombouts, Sander M Bohte, and Pieter R Roelfsema. How Attention Can Create Synaptic Tags for the Learning of Working Memories in Sequential Tasks. PLoS Computational Biology, 11(3):133,, 2015. ISSN 15537358. doi: 10. 37. pcbi. 1006060000
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating emrors. Nature, 93333) 1986 URL hatp:: google. com/
( oaw.anaaeeerrraaaaat.ooooooooooo
dg
2.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis., 1 115(3):111-55, 2015.
Benjamin Scellier and Yoshua Bengio. Equilibrium Propagation: Bridging the Gap Between Energy-Basee Models and Backpropagation. arXiv, 11(1987):1-13, 2016. ISSN 1662-5188. doi: 10.3388/fncom..201..000 .00024. htll:tttp//ee
/602. 05179
Jürgen Schmidhuber. Networks Adjusting Networks. In Proceedings of 'Distributed Adaptive Neu- ral Information Processing", St.Augustin, pp. 24-25. Oldenbourg, 1990.
Sebastian Seung. Learning in Spiking Neural Networks by Reinforcement of Stochastics Transmission. Neuron, 40:1063-1073, 2003. URL papprss:://publlcaiiionn
5D6B29BF 1380- 4D78 A152 AF8F233DE7F9
David Silver, Julian Schritwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen. Timothy Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676) October 2017.
H Francis Song, Guangyu R Yang. and Xiao Jing Wang. Reward-based training of recurrent neural networks for cognitive and value-based tasks. eLife, 6:1-24, 2017. ISSN 2050084X. doi: 10. 7f54/2.
composing robust features with denoising autoencoders. ICML 2008, 2008. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Mazagol. Extracting and
Paul Werbos. Applications of advances in nonlinear sensitivity analysis. Springer, Berlin, 1982.
Paul Werbos. Approximate dynamic programming for real-time control and neural modeling. In Handbook of Intelligent Control: Neural, Fuzzy and Adaptive Approaches, chapter 13, Multi- science Press, Inc., New York, 1992.
Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning Curves for Stochastic Gradient De- scent in Linear Feedforward Networks. Neural Computation, 17(12):2699 2718, 2005. ISSN 0899-7667. doi: 10. 101116220997660577442299999 URL www mi wwt.presssssournaaass.. journals.
12

---

Published as a conference paper at ICLR 2020
Ronald Williams. Simple Statistical Gradient-Followwii Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, 8:299-256, 1992.
Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-Plausibee Learning Algo- rithms Can Scale Large Datasets. ArXiv e-prints, 92, 2018
Xiaohui Xie and H. Sebastian Seung. Learning in neural networks by reinforcement of irregular spiking. Physical Review E, 69, 2004. ISSN 08966273. doi: 10.1066688896-222333377771.
13

---

Published as a conference paper at ICLR 2020
A
PROOFS
We review the key components of the model. Data (x E are drawn from a distribution p. The loss function is linearized:
dc Ōhg
C+
CbEjt
such that
0L ch Th x.y
E ((C- C).,E/xx.)
with expectation taken over the noise distribution (E). This suggests a good estimator of the loss gradient is
X = (É(x.y, ((xxy)) y(x
S
(8)
Let è' be the error signal computed by backpropagating the synthetic gradients:
(ac/0ý o
i= N +1;
= ( ) I
Then parameters B*+1 are estimated by solving the least squares problem:
Bi+1 arg min E
BT -
B
Note that the matrix-vector form of backpropagation given here setup so that we can think each term either vector for single input, matrices corresponding set of T inputs. Here focus the question, under what conditions
show that Bi+1 Wi+1 T oo?
One way to find an answer to define the synthetic gradient in terms of the system without noise added. Then _ is deterministic with respect to x, y and, assuming Ć has convergent power series around & = O, we can write
(m))
E(ixx,y) E
1
0C (onE))²
2
m! (cn
ij
xx,y
ch
Oh
m=
2
(m)
=(Wit))T
i+1 +
(cht)) mm xxy
m!
Taken together these suggest we can prove Bi+1 - W in the same way we prove consistency of the linear least squares estimator.
For this to work we must show the expectation of the Taylor series approximation 1 is well behaved. That is, we must show the expected remainder term of the expansion:
(m)
(cw)
E
m!!
m=2
finite and goes to zero as c + This requires some additional assumptions on the problem. We make the following assumptions:
AI: the noise ş subgaussian,
A2: the loss function C(x,y) is analytic on
A3: the enror matrices ह (Te)T are full rank, for 1 sis N + 1, with probability 1,
A4: the mean of the remainder and error terms is bounded:
E E (cx))(( < co,
for SiS N.
14

---

Published as a conference paper at ICLR 2020
Consider first convergence of the final layer feedback matrix, BN+1 In the final layer it is true that eN+1 = -N++1
Theorem 1. Assume A1-4. For gF (h', cit1, B B o then the least squares estimator =
(cN*I(e
(10)
solves 33) and converges to the true feedback matrix, in the sense that:
lim plim BN+1 WN+1
Cy 0--0 10-0-- T-
Proof.
Let c.
8mh Bhg
We first show that, under A1-2, the conditional expectation of the esti-
mator 2 converges the gradient [/ oh - 0. For each A5", by A2, have the following series expanded around { 0:
30
(m)
ij
(ch
)
mmt1
m!
Taking conditional expectation gives:
Ix)) (WWN+]]
Nj
(cnE) eN y mm+ 1x,
m!!
We must show the remainder term
[m) Nj
(cx) =E E
(cnE) N
|x,y 1x
ch
m!
m=
goes zero as CY O. This is true provided each moment B((()) )r |x, y) is sufficiently well- behaved. Using Jensen's ' inequality and the triangle inequality in the first line, we have that
((m)
( <
E
Nj
Jen5y
mm+ |x,
V(x,)) E
m!!
m==
[(m)
Nj
(ch) ( LE [m pmt1]
[monotone convergence]
m!!
(m) NN; (cn)m=I(Vme (vm++1 1) m+
[subgaussian]
K
Σ
m!!
m=
(cr))
as
Ch + 0.
(11)
With this in place, we have that the problem close to a linear least squares problem, since
N = NWITT N+1 + EN (cx) + n²",
(12)
with residual 17 N - E(ί y) The residual satisfies
E (e N+1 (n (T) E(ee N+1 (ÁNIT - N+ *+IB((AN) (x,,))
= - ((
-
0.
(13)
This follows since e N+1 is defined in relation to the baseline loss. not the stochastic loss, meaning is measurable with respect to (x, y) and can be moved into the conditional expectation.
From 12 and A3, we have that the least squares estimator T0
satisfies
(B-N+1)T= = NT)+ ( (ch) (e (e
15

---

Published as a conference paper at ICLR 2020
Thus, using the continuous mapping theorem
plim (BN+1)T= = (W+++))+++ + plim + (8 (cn) + (
plim 1 e (e
T T
7-400
=WN++) + [(((n) + 17 ) ( [()
[WLLN]
[Eq. 13)
= = + + {E(en))
[B ((
[A4 and Eq. -WWN+)) + (ct).
Then we have:
lim plim BN+1 WN+1
Go 9--0 T-
We can use Theorem to establish convergence over the rest of the layers of the network when the activation function is the identity.
Theorem 2. Assume AI-4. For gF (h', e Bi+1) Bi+1 and (x) I, the least squares
estimator
(o)()))))
I +1,
(14)
solves (
and converges to the true feedback matrix, in the sense that:
lim plim B = W
1
T-000
Proof. Define
('(c) plim B T-x
assuming this limit exists. From Theorem I the top layer estimate fN+1 converges in probability Wc)-
We can then use induction to establish that B in the remaining layers also converges in probability
to (c). That is, assume that B² converge in probability to (c)) in higher layers N + 2 j > Then we must establish that B' also converges in probability.
To proceed is useful to also define
i= N (0))+))) o (W'hh") isisM N,
(BC/0 o (W'h"")),
ē(c)²
as the error signal backpropagated through the converged (but biased) weight matrices W()). Again is true that N+1 N+1.
As in Theorem 1, the least squares estimator has the form:
( = (e)) (é' (e)
Thus, again by the continuous mapping theorem:
plim x plim (
plim =
10 T
T-20
1 T-x _
plim
Bi+1
plim (Et))
In this case continuity again allows us separate convergence of each term in the product:
plim
No 0it1
plim
{ 3i (c
plim BN+1 T-x
plim Bi+1
T-xx T
T-x
T
(15)
-(()))
16

---

Published as a conference paper at ICLR 2020
using the weak law of large numbers in the first term, and the induction assumption for the remaining terms. In the same way
plim = ((ē
T-
Note that the induction assumption also implies lime- (c) c¹. Thus, putting it together, by A3, A4 and the same reasoning as in Theorem
| we have the result:
lim plim (B')²= lim (ww) T(e 6k +0 T-000
E(E (0( [
[+(())((0)]]]
c00
= T
Corollary 1. Assume A1-4. For gDFA e Bit)) BitlEN-1 and oxx) the least
squares estimator
= ((N+)()
IsisN+1,
(16)
solves (3) and converges to the true feedback matrix, in the sense that:
lim plim T-000
II W3 j=++1
1
Proof. For a deep linear network notice that the node perturbation estimator be expressed as:
X (wi+I... N+I Te (cn) + n
(17)
where the first term represents the true gradient, given by the simple linear backpropagation, the second and third terms are the remainder and a noise term, as in Theorem I. Define
II W j=N+1
Then following the same reasoning as the proof of Theorem 1, we have:
plim (B++1)T= = (Vi+I)T+ + plim - (cx + (e T-400
7-000 7 - (o
plim 1
7-x0
=(Vi+I)T + (E((e)) + (e) N+1 B(e (oN+))))
= =VV+I)T + (e [B(e (ett)-)
-VV++)) + Occ.).
Then we have:
lim plim Bi+l V++1
Ga -0 7-000-000 T-400
A.1 DISCUSSION OF ASSUMPTIONS
is worth making the following points on each of the assumptions:
A1. In the paper we assume ş is Gaussian. Here we prove the more general result of convergence for any subgaussian random variable.
A2. In practice this may be a fairly restrictive assumption, since it precludes using relu non- linearities. Other common choices, such as hyperbolic tangent and sigmoid ooneircasss with an analytic cost function do satisfy this assumption, however.
A3. It is hard to establish general conditions under which é'(e") will be full rank. While it may be reasonable assumption in some cases.
17

---

Published as a conference paper at ICLR 2020
Noise variance
Number of neurons
A]
B)
101
100
301
relative eiror
degrees
101
sin cangrueence
90
relafivee enrrr
10P
degroes
107
10-1
109
10-1
30
10P
5.70.01
5.10.1
10-1
50
10-2
100
500
1000
10-1
10-2
10-2
Layer
101
Sinn congruence (%6)
100
101
relative error
degrees
101
90
degrees
101
80
10-1
100
70
relative
10-1
100
10-2
10-1
10
10-1
10
40
10
2
250 500
750 1000
250
500
750 1000
250 500
750 1000
250 500 750 1000
250
500 750 1000
Iteration (x10')
Iteration (x10")
Iteration (x10")
Iteration (x110))
iteration (x10")
Layer
Figure 4: Convergence of node perturbation method in two hidden layer neural network (784-50- 20-10) with MSE loss, for varying noise levels c. Node perturbation used to estimate feedback matrices that provide gradient estimates for fixed W. (A) Relative error W B | // ([x) for each layer. (B) Angle between true gradient and synthetic gradient estimate at each layer. (C)
Percentage of signs in W and B' that are in agreement. (D) Relative error when number of neurons is varied (784-N-50-10). (E) Angle between true gradient and synthetic gradient estimate at each
layer.
Extensions of Theorem non-linear network may be possible. However, the method of proof used here is not immediately applicable because the continuous mapping theorem can not be applied in such a straightforward fashion as in Equation (T5). In the non-linear case the resulting sums over all observations are neither independent or identically distributed, which makes applying any law of large numbers complicated.
B VALIDATION WITH FIXED W
We demonstrate the method's convergence in small non-linear network solving MNIST for dif- ferent noise levels, Ch, and layer widths (Figure 4). As basic validation of the method, in this experiment the feedback matrices updated while the feedforward weights W are held fixed. We should expect the feedback matrices B' to converge to the feedforward matrices W Here different noise variance does results equally accurate estimators (Figure 4A). The estimator correctly esti- mates the true feedback matrix W to a relative error of 0.8%. The convergence is layer dependent, with the second hidden layer matrix, W being accurately estimated, and the convergence of the first hidden layer matrix, W being less accurately estimated. Despite this, the angles between the estimated gradient and the true gradient (proportional to c T are very close zero for both layers (Figure 43 (less than 90 degrees corresponds to descent direction). Thus the estimated gradients strongly align with true gradients both layers. Recent studies have shown that sign con- gruence of the feedforward and feedback matrices is all that is required to achieve good performance Liao et al. (2016); Xiao et al. (2018). Here significant sign congruence is achieved in both layers (Figure 4 ). despite the matrices themselves being quite different in the first layer. The number of neurons has an effect on both the relative error in each layer and the extent of alignment between true and synthetic gradient (Figure ED).. The method provides useful error signals for variety of sized networks, and can provide useful error information to layers through a deep network.
C EXPERIMENT DETAILS
Details of each task and parameters are provided here. All code is implemented in TensorFlow.
18

---

Published as a conference paper at ICLR 2020
C.I FIGURE 2
Networks are 784-50-20-10 with an MSE loss function. A sigmoid non-linearity is used. A batch size of 32 is used. B is updated using synthetic gradient updates with learning rate v) = 0.0005, W is updated with learning rate 0.0004, standard deviation of noise is 0.01. Same step size is used for feedback alignment, backpropagation and node perturbation. An initial warm-up period of 1000 iterations is used, in which the feedforward weights are frozen but the feedback weights are adjusted.
C.2 FIGURE 3
Network has dimensions 784-200-2-200--1 Activation functions are, in order: tanh, identity, tanh, relu. MNIST imput data with MSE reconstruction loss is used. A batch size of 32 was used. In this case stochastic gradient descent was used to update B. Values for W step size, noise variance and B step size were found by random hyperparameter search for each method. The denoising autoencoder used Gaussian noise with zero mean and standard deviation o = 0.3 added to the input training data.
C.3 FIGURE 4
Networks are 784-50-20-10 (noise variance) or 784-N-50-10 (number of neurons) solving MNIST with an MSE loss function. A sigmoid non-linearity is used. A batch size of 32 is used. Here W is fixed, and is updated according to an online ridge regression least-squares solution. This was used becase it converges faster than the gradient-descent based optimization used for learning B throughout the rest of the text, so is a better test of consistency. A regularization parameter of y 0.1 was used for the ridge regression. That is, for each update, B' was set to the exact solution of the following:
Bi+l arg min E g('' ett B
Xo $1 + BIB
(18)
C.4 CNN ARCHITECTURE AND IMPLEMENTATION
Code and CNN architecture are based on the direct feedback alignment implementation of Crafton al. 2019 Specifically, for both CIFARIO and CIFAR100, the CNN has the architecture Conv (3x3, Ixl, 32), MaxPool(3x3, 2x2), Conv(5x5, 1x1, 128), MaxPool(333, 2x2), Conv(5x5, 1x1, 256), MaxPool(3x3, 2x2). FC 2048, FC 2048, Softmax(10), Hyperparameters (learning rate, feedback learning rate, and perturbation noise level) were found through random search. All other parameters are the same as Crafton et al. (2019). In particular, ADAM optimizer was used, and dropout with probability 0.5 was used.
C.5 NOISE ABLATION STUDY
The methods listed in Table 2 are implemented as follows. For the autoencoding task: Through hyperparameter search, noise standard deviation of 6f 0.02 was found to give optimal perfor- mance for our method. For BP(SGD). BP(ADAM), FA, the 'noise' results in the Table are obtained by adding zero-mean Gaussian noise the activations with the same standard deviation, c₁. For the DAE, noise standard deviation of c, = 0.3 was added to the inputs of the network. Implementa- tion of the synthetic gradient method here takes the same form as our method: (h, y; B) = Be (this contrasts with the form used in Jaderberg et al. (016): 9(h y; B, c) h + c). But the matrices are trained by providing true gradients X, instead of noisy estimators based on node per- turbation. This is not biologically plausible, but provides a useful baseline to determine the source of good performance. The other co-adapting baseline we investigate is the 'matching' rule (similar to (Akrout al. 2019: Rombouts et al., 2015) Martinolli et al. 2018)): the updates to B match those of W. and weight decay is used to drive the feedforward and feedback matrices to be similar.
For the CIFAR10 10 results, hyperparameter search identified noise standard deviation of Ca 0.067 to be optimal. This was added to the activations. : The synthetic gradients took the same form as above.
19


Reviews:
[{"review_id": "rylITWMqKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method that addresses the \"weight transport\" problem [1] (not cited by the authors) emphasizing the biological infeasibility of artificial neural networks (ANNs) which are trained by gradients computed by the backpropagation algorithm [2a, 2b]. It is arguably the most eminent criticism (among many many others) when learning in the brain is modelled by such ANNs.\n\nThe authors cite only Rumelhart et al. (1986) for backpropagation (which is the same as the reverse mode of automatic differentiation), although Rumelhart cited neither Linnainmaa, the inventor of the method [2a], nor Werbos, who first applied it to ANNs [2b]. \n\nThe authors propose to estimate the weights for the backward pass using a noise-based estimator and provide theoretical and empirical arguments. The proposed method is compared with backpropagation (the ground truth) and direct feedback alignment (DFA; which resorts to random matrices for the backward pass) on small fully-connected, convolutional, and fully-connected auto-encoder networks for MNIST, CIFAR10, and CIFAR100 and offers analysis based on insights from recent work.\n\nGenerally, this work is well-placed and the method straightforward and novel. That said, we found the paper to be difficult to parse with some questionable claims which we don't believe are sufficiently backed up by experiments. Due to the following issues, we think that the paper is not yet quite ready for acceptance.\n\n1.) As the authors point out, the recent work by Akrout et al. [3] is very closely related but the paper is lacking a significant discussion let alone an empirical comparison. Extrapolating from the experiments in this paper we think that the presented method is likely going to underperform since Akrout et al. show on-par performance with backprop on imagenet.\n\n2.) The paper appears to blend synthetic gradients [4a, 4b] and feedback alignment to a somewhat questionable degree. A connection that is usually not done in other recent papers about biologically feasible ANNs. And that for good reasons: synthetic gradients do not address the weight transport problem. They backprop the synthetic gradients in order to train the gradient estimators of the previous layer.\n\nAlso, the paper cites only the recent reference [4b] (2016) on synthetic gradients but not the original work [4a] (1990).\n\n3.) The authors claim \"Thus our method illustrates a biologically realistic way by which the brain could perform gradient descent learning\". Even though their method is a more plausible way by which the brain could perform gradient descent learning, it is still far from realistic. The proposed method still implicitly \"transports\" a lot of information from the forward pass to the backward pass (such as a. topology or b. the derivative of the activation function or c. the activations of the forward pass (h^i) that are necessary to compute the gradient estimate in either case).\n\n4.) The experimental results of the proposed method but also the baseline are simply too far from the state of the art. This is partially dismissed in section 5 due to regularization and data augmentation \"tricks\". We'd like to point out that feedback alignment has been reported to achieve <2% error on the MNIST dataset [6] which is significantly better than the results reported in this work without any tricks. Similar seems to be the case for CIFAR10 and CIFAR100. The reported results on CIFAR 10 and 100 are over 20% below the state of the art. Furthermore, many previous papers provide code which arguably makes such experiments particularly easy (see [7]). We do not expect the authors to provide state of the art results but they should be at least in the same ball-park.\n\n5.) The authors claim \"(...) this hybrid approach can solve large-scale problems.\" We don't think this is backed up by experiments as most models are rather small (all < 50k parameters except the CNN architecture) and evaluated on toy datasets (for vision standards; while underperforming as already mentioned).\n\n6.) It is not obvious how the noise-based estimation will scale with width and depth. The authors make the observation that \"The number of neurons has an effect\". This can be seen in the appendix in Fig 4E where the relative error increases exponentially with the number of neurons in the first layer (note that the second layer doesn't increase since it remains unchanged). This seems to indicate that the method would not scale well.\n\n7.) Why do the authors decide to train the MNIST networks with such a small learning rate (4e-4) for 2 million steps? Better results can be achieved with just a few thousand steps. Does the method work in the \"general\" setting?\n\n8.) With regards to Figure 2C for layer 2: Given a random initialization of the backward pass for feedback alignment but also the presented method, we would expect that the sign congruence of the first few iterations to be the roughly the same on average. That doesn't seem to be the case according to the figure. Is there are an explanation for this observation?\n\n9.) Why the sudden switch from feedback alignment to direct feedback alignment for the CNN experiment in section 4.3? Does the approach of section 4.2 not work with CNNs?\n\n10) We hope the authors could clear up some confusion with regards to section 3 and the consistency proofs in the appendix.\n\n10.a) Apparently, the notation slightly changes? In section 2, layers are indicated by 1 <= i <= N as the superscript. In section 3, i prevails but a new superscript 1 <= n <= N is used for the layers. Theorem 1 then states that the least-squares estimator for B^(i+1)\\tilde{e}^{i+1} is (\\hat{B}^{N+1})'. There seems to be a notation issue since it is our understanding that every layer should have its own estimated weights.\n\n10.b) We believe A03 should state that E[\\tilde{e}^i (\\tilde{e}^i)'] is of full rank and not simply \\tilde{e}^i (\\tilde{e}^i)' assuming that \\tilde{e} stands for the equivalent X of regular least-squares. From the main text (specifically the eq in section 2.1) one might assume that \\tilde{e}^i are vectors but the proof seems to treat them like the design matrices X in the regular least-squares estimator.\n\n10.c) It is awkward that in the proof of Theorem 1 and 2 the population size is suddenly T, the same symbol for the transpose.\n\n10.d) The regular OLS estimator of the coefficients is b = (X'X)(X'y) where X is a full-rank design matrix and y are the targets. It is our understanding that \\lambda, as defined by eq 12, is equivalent to the y of the regular OLS estimator. But in the following equation (the bottom of page 14), the W^{N+1} of \\lambda is factored out, apparently assuming e^{N+1}e^{N+1}' = I. Maybe we made a mistake, we'd appreciate if the authors could clarify this step in the proof.\n\n11.) Finally, we add some additional minor comments or sources of confusion:\n- we'd appreciate if all equations were numerated\n- In section 2, the definition of the loss is verbose and partially unnecessary. L(y,\\hat{y}(x)) never appears in the text. Furthermore, \\hat{y} is not introduced as a function so writing \\hat{y}(x) is awkward.\n- Similar for the equation in section 2.2.\n- 4.1 first line: the tilde is misplaced\n- Figure 3A, the black line has no label (assumed to be DAE which is missing)\n- 4.2 first inline equation: the tilde is misplaced\n- Table 1: no decimal values, not std. Not clear if the difference between DFA and note perturbation is significant.\n- Figure 4A-E is missing the crucial comparison with feedback alignment\n\n\nReferences:\n\n[1] Grossberg, Stephen. \"Competitive learning: From interactive activation to adaptive resonance.\" Cognitive science 11.1 (1987): 23-63.\n\n[2a] Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis, Univ. Helsinki, 1970. FORTRAN code on pages 58-60. See also BIT 16, 146-160, 1976.\n\n[2b] Werbos, Paul J. \"Applications of advances in nonlinear sensitivity analysis.\" System modeling and optimization. Springer, Berlin, Heidelberg, 1982. 762-770.\n\n[3] Akrout, Mohamed MA, et al. \"Deep learning without weight transport.\" Advances in Neural Information Processing Systems. 2019.\n\n[4a] J.  Schmidhuber. Networks adjusting networks. In J. Kindermann and A. Linden, editors, Proceedings of `Distributed Adaptive Neural Information Processing', St. Augustin, 24.-25.5. 1989, pages 197-208. Oldenbourg, 1990. Extended version: TR FKI-125-90 (revised), Institut für Informatik, TUM. http://people.idsia.ch/~juergen/FKI-125-90ocr.pdf See section \"An Approach to Local Supervised Learning in Recurrent Networks.\"\n\n[4b] Jaderberg, Max, et al. \"Decoupled neural interfaces using synthetic gradients.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n\n[5] Czarnecki, Wojciech Marian, et al. \"Understanding synthetic gradients and decoupled neural interfaces.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n\n[6] Nøkland, Arild. \"Direct feedback alignment provides learning in deep neural networks.\" Advances in neural information processing systems. 2016.\n\n[7] https://paperswithcode.com/sota/image-classification-on-cifar-10?p=maxout-networks\n\nFor now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way. Let us wait for the rebuttal.\n\nEdit: After the rebuttal, we increased our score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1l1YdnaFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper authors study one possible incarnation of more biologically plausible learning scheme, akin to (direct) feedback alignment methods, where the error signal is linearly mapped onto the update direction, without reusing forward weight (so to break the weight sharing issue, that currently is believed to be impossible for brains). \nAuthors approach can be summarised as a mixture of synthetic gradients with a noise-enriched estimation, rather than direct empirical risk minimisation.\nContributions are two fold, first, authors provide some theoretical analysis of the convergence of gradient estimator in a simplified setup, second, the noise-based estimator is empirically evaluated on 2 simple classification tasks.\n\nPaper is well written, and easy to read, with relatively easy to follow notation (which is quite a tricky task for non-gradient based learning methods that require abandoning typical concept of loss minimisation and talking about dynamical systems instead).\n\nI have a few critical comments, that I hope authors can address in the revised manuscript:\n- first, high level thing, that seems to be missing from the manuscript is use of baselines that are actually co-adapted, rather than random (e.g. DFA and FA). To be more specific, in papers like \"Sobolev Training for Neural Networks\" (NeurIPS  2017, https://papers.nips.cc/paper/7015-sobolev-training-for-neural-networks.pdf) one can find 3 basic methods (all requiring one implementation, and differ only in terms of which loss is applied): at each layer h, the gradient predictor g(h, e, B) is composed of (d/dh) CE[softmax(Bh + c), y], which has an analytical form, is biologically plausible (as boils down to a simple affine transformation if h and y. This model can now be supervised in 3 ways: \na) one can put supervision only on the gradient (and bootstrap from higher layers, if needed, as in FA or \"fully decoupled\" synthetic gradient model in Jaderberg et al.) [this is essentially \"pure\" SG from Jaderberg et al. but constrained to the conservative vector fields, so to guarantee convergence]\nb) one can put supervision on loss itself matching the \"topmost loss\" (reminescent of DFA, where error propagation skips entire network) [called Critic Training, Critic Network etc. depending on the source]\nc) twe two above can be used jointly (which leads to the full Sobolev training)\nNeither of these methods have been compared, and in reviewer's opinion it is critical, as approaches are very similar, and while Czarnecki et al. was not analysing biological plausibility, the models proposed do satisfy the same constraints requested by authors of this paper. Note, that with critic learning on MNIST, one can easily get results matching backpropagation (which, in the current manuscript is claimed as an important property of the introduced method). It might also be important to show that well tuned linear model on MNIST can reach 95% test accuracy too.\n- similarly, authors are not disentangling noise-induction, from the overall setup. Which of these two is the actual source of good results? Many previous works would rely on empirical risk minimisation, if one uses noised versions and train \"regular\" fully decoupled affine synthetic gradients, will the results be analogous? Will this improve the Sobolev training (if implemented and verified)? Given, that both methods are known object in the literature, providing understanding of which components are actually important (maybe it is only the combination that works?) would strengthen the claims.\n- Analogously, to further decouple mixed effects, when authors claim that noise-based estimator performs better than adam in the autoencoding case, having an adam with artifically added same amount of noise to gradient estimates would be helpful for the reader to see if the difference lies in the \"second order estimates\" of Adam, or in simply regularising effects of adding noise.  In particular, note, that this is a well established result, that noising inputs to the neural network is equivalent (up to first order terms) to Tikhonov regularisation.\n- the theoretical claim of Thm 1 is quite trivial, and while I really admire authors effort to provide theoretical grounding, it feels a bit overstated in the current form. What authors are showing, is that in the simple setup, where network is not learning, and all the errors are in the sense, independent, the linear predictor is consistent. This property is well known, and both assumptions used - never met in practise, consequently I would strongly suggest downplaying the claim, and stating these results in sentence or two, with proof moved to the appendix, as in the current version of the manuscript section 3 is presented as major contribution, rather than an interesting side note.\n\n\nMinor comments:\n- can authors provide some error intervals for results in Table 1? Results are so close, that claiming that 48 is even \"marginally better\" than 47 seems odd, unless many repeated runs were conducted and some confidence intervals can be provided?\n- it is odd to say \"using backpropagation OR Adam\". Adam is just an optimiser, it still uses backpropagation (aka chain rule). Could the notation be unified so that when talking about comparing optimisers, authors state Adam and SGD (assuming that this is what is currently called backpropagation?) and when talking about chain rule, the name of backpropagation is there?\n\n\nOverall I believe it is an interesting study, however, currently missing important baselines and ablations to be a good ICLR contribution. If authors are willing to add these, I will be happy to revise the score assigned.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1x2cMt5KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It's unclear how multi-layer biological neural networks could implement gradient-based learning, as they don't have the symmetric connections needed for backpropagation. This paper proposes a perturbation-based synthetic gradient estimator that does not rely on symmetric backward connections. Hidden unit perturbation is used to estimate the loss gradient, and backward connections are trained via gradient descent to predict the approximate gradients from the perturbation-based estimator.\n\nThe topic is important and the paper is well-written. I don't follow this area closely, but from what I can tell it's a novel idea. The results are strong. The method beats various alternatives and closely matches backpropagation in terms of performance on the MNIST tasks (less so on CIFAR). It's especially curious that the method performs better than backpropagation on the autoencoder task.\n\nI have an unresolved question: What do the learnable backward connections add beyond the perturbation estimator for the gradients? If the perturbation-based estimator could be used to train the forward model, does the trained backward model have advantages in terms of efficiency or performance? I would appreciate more discussion of this key choice, or a direct comparison with only the perturbation-based estimator (only) to understand the differences.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initial reviews of this paper cited some concerns about a lack of comparison to SOTA and baselines, and also some debate over claims of what is (or is not) "biologically plausible."  However, after extensive back-and-forth between the authors and reviewers these issues have been addressed and the paper has been improved.  There is now consensus among authors that this paper should be accepted.  I would like to thank the reviewers and authors for taking the time to thoroughly discuss this paper.
Decision: Accept (Poster)