Credit Assignment Through Broadcasting a Global Error Vector
Authors: David G Clark
Larry Abbott
SueYeon Chung
Venue: NeurIPS 2021 Poster (2021)
Keywords: biology
biologically plausible
credit assignment
neuroscience
plasticity
learning
nonnegative
brain

Abstract: Backpropagation (BP) uses detailed, unit-specific feedback to train deep neural networks (DNNs) with remarkable success. That biological neural circuits appear to perform credit assignment, but cannot implement BP, implies the existence of other powerful learning algorithms. Here, we explore the extent to which a globally broadcast learning signal, coupled with local weight updates, enables training of DNNs. We present both a learning rule, called global error-vector broadcasting (GEVB), and a class of DNNs, called vectorized nonnegative networks (VNNs), in which this learning rule operates. VNNs have vector-valued units and nonnegative weights past the first layer. The GEVB learning rule generalizes three-factor Hebbian learning, updating each weight by an amount proportional to the inner product of the presynaptic activation and a globally broadcast error vector when the postsynaptic unit is active. We prove that these weight updates are matched in sign to the gradient, enabling accurate credit assignment. Moreover, at initialization, these updates are exactly proportional to the gradient in the limit of infinite network width. GEVB matches the performance of BP in VNNs, and in some cases outperforms direct feedback alignment (DFA) applied in conventional networks. Unlike DFA, GEVB successfully trains convolutional layers. Altogether, our theoretical and empirical results point to a surprisingly powerful role for a global learning signal in training DNNs.

Credit Assignment Through Broadcasting a Global Error Vector
David G. Clark, L.F. Abbott, SueYeon Chung
Center for Theoretical Neuroscience Columbia University
New York, NY
(david. clark, lfabbott, sueyeon. chaneecccnnnnnonoinniii edu
Abstract
Backpropagation (BP) uses detailed, unit-specific feedback train deep neural networks (DNNs) with remarkable success. That biological neural circuits appear perform credit assignment, but cannot implement BP, implies the existence of other powerful learning algorithms. Here, we explore the extent to which a globally broadcast learning signal, coupled with local weight updates, enables training of DNNs. We present both a learning rule, called global error-vector broadcasting (GEVB), and class of DNNs. called vectorized nonnegative networks (VNNs), which this learning rule operates. VNNs have vector-valued units and nonnegative weights past the first layer. The GEVB learning rule generalizes three-factor Hebbian learning, updating each weight by an amount proportional to the inner product of the presynaptic activation and globally broadcast error vector when the postsynaptic unit is active. We prove that these weight updates are matched in sign the gradient, enabling accurate credit assignment. Moreover, at initialization, these updates are exactly proportional to the gradient in the limit of infinite network width. GEVB matches the performance BP VNNs, and in some cases outperforms direct feedback alignment (DFA) applied in conventional networks. Unlike DFA, GEVB successfully trains convolutional layers. Altogether, our theoretical and empirical results point to surprisingly powerful role for global learning signal in training DNNs.
1 Introduction
Peep neural networks (DNNs) trained using backpropagation (BP) have achieved breakthroughs on myriad of tasks 21. The power of BP lies in its ability to discover intermediate feature representations by following the gradient of loss function. In manner analogous to DNN training, synapses in multilayered cortical circuits undergo plasticity that modulates neural activity several synapses downstream to improve performance on behavioral tasks 4 However, neural circuits cannot implement BP, implying that evolution has found another algorithm, or collection thereof [5) This observation has motivated biologically plausible alternatives to BP 7.
Credit assignment algorithms based on broadcasting a global learning signal, such as node perturbation 8), are attractive due their biological plausibility, as the global signal could correspond neuromodulator that influences local synaptic plasticity (9] [10]. However, perturbation methods of little practical relevance as the variance their gradient estimates prohibitively large 7 Instead, recent research has focused on methods that compute or estimate the gradient by transmitting detailed, unit-specific information to hidden layers through top-down feedback. The foremost example of such method is BP. BP's use of the feedforward weights in the feedback
Code accompanying our paper is available htt.::///sstaaa..coo/dddcccc///e
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A backpropagation (BP)
B
feedback alignment (FA)
direct feedback alignment (DFA)
global error-vector broadcasting (GEVB)
Serget 0000000
0
2 OOO0OO
sarget
P
O
000000
O
enor
0000000
O
output
global
W
W
Broadcorting
O p
O
O
O
0 0000000
OOO00O
Tw! O
F
D
p O O O
0 O
O 0
0
p 0 0
inpul
O
000000
Inpal
1 000000
Figure I: Credit assignment methods. (A) Backpropagation (BP) transmits the error vector backwards layer-by-layer using the transposes of the feedforward weights. (B) Feedback alignemnt (FA) transmits the error vector backwards layer-by-layer using fixed, random feedback matrices. (C) Direct feedback alignment (DFA) delivers the enror vector directly to each hidden layer using fixed, random feedback matrices. (D) Global error-vector broadcasting (GEVB) conveys the full error vector to all hidden units without unit-specific feedback. Thus, there are no feedback parameters. GEVB operates in networks in which each hidden unit is vector-valued. Each arrow represents a single vector unit.
pathway, a property called weight symmetryy is not biologically plausible- - this is the weight transport problem [1]. Another method of this class, feedback alignment (FA), operates identically to BP but uses fixed, random matrices in the feedback pathway, thereby circumventing the weight transport problem 13.. FA inspired further method called direct feedback alignment (DFA), which delivers random projections of the output error vector directly to hidden units [5]. DFA approaches the performance BP in certain modern architectures [16), however its performance still lags behind that of BP in many cases [4
16-19 A particularly striking shortcoming of DFA is its inability to train 19 20
convolutional layers [16
To be implemented in neural circuits, both FA and DFA require a
biologically unrealistic "error network" to compute top-down learning signals, though models based on segregated dendrites aim to lift this requirement (21)
Here, we show that DNNs can be trained to perform par with BP by broadcasting a single, global learning signal to all hidden units and applying local, Hebbian-like updates to the weights. Our proposed learning rule, called global error-vector broadcasting (GEVB), is not perturbation-based. but instead distributes information about the output error throughout the network. Unlike DFA, which delivers unique random projection of the output error vector to each hidden unit, GEVB broadcasts the same, unprojected error vector to all hidden units. Thus, GEVB involves no unil-specific feedback. This learning rule operates in new class of DNNs called vectorized nonnegative networks (VNNs), which have vector- valued units and nonnegative weights past the first layer. While vector- valued units neuroscientifically speculative (though see Section for for possible neural implementations)) nonnegative weights are motivated by the fact that cortical projection ncurons are excitatory [4]. The GEVB learning rule updates each weight by amount proportional to the inner product of the presynaptic activation and the global error vector when the postsynaptic unit is active. Three- factor Hebbian plasticity encompasses synaptic update rules that depend on pre- and postsynaptic activity and a global neuromodulatory signal (the third factor) 24 25). The GEVB learning rule is therefore a form of three-factor Hebbian plasticity with vector-valued global third factor (26). Our experimental
results show that this form of global-error learning is surprisingly powerful, performing on par with BP in VNNs and overcoming DFA's inability train convolutional layers.
2 Credit assignment in conventional networks
Consider a fully connected network of L layers with pre-activations of and activations af, where i = 1,....,,, my and l = o...., L. Let wf, denote the weights, bf the biases, and 6 the pointwise nonlinearity. The forward dynamies
h Σ
e Biga + +b,, of = 6 (h!),
(1)
with the understanding that &i is the identity function at the output layer L. The network inputs and outputs are af and at, respectively. We place a loss on the output, and define aL/Onf es where e; is the output error vector. On single training example, negative-gradient weight update can be

---

written
0C da
n1
Dak dag
Awly x 88 (h,) of where of
Ek. A=1
(2)
BP, FA, and DFA are different methods of computing approximating 8f. BP computes of exactly using the recurrence relation
1+1 o' (h
with the initial condition 8. c; (Fig. A). FA approximates of by performing a backward pass identical to Eq. 3 but using fixed, random feedback weights lie in place the transposed feedforward weights wf (Fig. B3. This backward pass given by
Ri+11
fik H Σ
(1++1 1o
with the initial condition 0. 65- Learning occurs under Eq. due to learning-induced partial alignment of wf, with fi 13 Finally, DFA approximates of by setting the derivative 0ak/ in the formula for of Eq. to fixed, random matrix di (Fig. (C). This yields
dyes.
(5)
In a similar manner to FA, learning occurs in DFA due to induced partial alignment of Oat /aa
with de [99 (20]. The learning rule we propose further simplifies Eqns. 3 and 5 by removing
computation of the quantity analogous to 9f entirely (Section 3.2
A fundamental obstacle to accurate credit assignment in the absence of weight symmetry is that the sign of the gradient each weight depends on detailed information about downstream weights to which the feedback pathway does not have access, even in the presence of partial alignment of feedforward and feedback weights. For example, suppose DNN confidently predicts class instead of the target class c. Then, the gradient at wiy depends on of ~ aa²; /0, - òa!: faaf, which can be positive or negative depending on a difference in the strengths of complex polysynaptic interactions between unit in layer l and units and e' in layer T..
Intriguingly, this obstacle dissolves in networks with scalar output, positive weights past the first layer, and o' > 0. In this case, we have Qf = (da /Da with aaf/ /0a > 0. Thus, as per Eq. 2. the gradient sign at wiy is equal to the sign of o' o 01. This expression has the form of a three-factor Hebbian rule. This simplification to credit assignment arising from sign-constrained weights in scalar-output networks was observed by both Balduzzi et al. (77 who proposed the Kickback algorithm, and Lechner [8] who used sign-constrained weights in DFA. However, the restriction to scalar-output (e.g., binary classification) networks is highly limiting. The GEVB learning rule, described in the next section, generalizes this observation to vector-output (e.g., multi-way classification) networks.
2-1
3 Credit assignment in vectorized nonnegative networks
3.1 Vectorized nonnegative networks
Here, introduce VNNs, the class of DNNs in which proposed learning rule operates. Given task with output dimension K, such as K-way classification, each VNN unit is a vector of dimension K. With the exception of the first hidden layer, connections obey "vectorized weight sharing" such that each vector unit computes a weighted linear combination of presynaptic vector units. Our notational convention is to use Latin letters to index different vector units and Greek letters index the components of a single vector unit. Thus, Greek indices always run over 1,..,, Boldface variables represent all K components of vector unit.
For = 1..., T., let iLg denote the pre-activations and af the activations. The inputs, al. are not vectorized and hence lack μ subscript. The vector units in the first hidden layer compute
3

---

representation of the input using the weights wayj. For > 1, connections obey vectorized weight sharing. Thus, these weights lack l subscript and are denoted by wfy- Crucially, these 1 weights are nonnegative, consistent with excitatory cortical projection neurons. For & 1,... L, the biases are vector-valued and denoted by bf.. Finally, the vector-to-vector nonlinearity denoted by @ I2K 7 I This function mixes vector components in general. The forward dynamics are
Vini
b
f=1
hey =
d = 22 (h.)
(6)
ne-1 w
an 1 +
b l>
with the understanding that & is the identity function at the output layer We assume that there is single vector output unit o denoted by af for brevity, which we place loss L. We define 0C/0a% eu. where eN is the output error vector. We will show that our proposed learning rule matches the sign of the gradient when the nonlinearity has the form
p = Ghh
with G(h) 2 0 and piecewise constant. The function induces coupling between different vector components. While there are many options for this function, we choose
G(h) = θ (t h))
(8)
where O is the Heaviside step function and is gating vector that differs across units (for brevity, we suppress its and indices in our notation). We sample the gating vectors uniformly over 1] initialization and hold them constant throughout training and inference. the K case, this choice of & reduces to a ReLU if L > 0 since ((h) = @(hh) = @(h)h Rel.U(h)..
While described the inputs as no scalar units with all-to-all connections the first hidden layer, the inputs can be described equivalently as Kno vector units with vectorized weight-shared connections to the first hidden layer, unifying the l and 1 cases of the learning rule (Appendix B). We use this convention henceforth.
3.2 Global error-vector broadcasting learning rule
The GEVB learning rule works by globally broadcasting the output error vector en all hidden units:
GEVB:
Awf, -G (h.) Σ j eu-
Thus, presynaptic units that are aligned anti-aligned with the output error vector have their weight onto the postsynaptic unit decreased increased, respectively. when the postsynaptic unit is active. Note that this learning rule has no feedback parameters. The GEVB learning rule can be interpreted as three-factor Hebbian rule with vector-valued global third factor e,, 26). Our experimental results in Section show that this extremely simple learning rule is surprisingly powerful.
3.3 Global error-vector broadcasting matches the sign of the gradient
We now show that by choosing d as in Eq. 7 with G(h) 2 0 and piecewise constant, GEVB weight updates are matched in sign to the gradient. The gradient on single training example is
ac Ouly
da Da
G(h
Σ R.
ew
ajv
(10)
The K-by-K K Jacobian /0a,, in Eq. 10 describes how component v of vector unit layer polysynaptically influences component j of the vector output unit. This Jacobian can be computed iteratively a BP-like manner,
dak 0
ne+1
dak
Σ
w
e+1
G(f ++1
(11)
do
k=1

---

the gradient is with the initial condition 0a = luu, where I is the identity matrix. These recursions ensure that da /0a is proportional to j for all l. Thus, write
day Daf
8 pm
where 8f
(12)
Da
Substitution of Eq. 12 into Eq. 11 gives
Dell
8
Σ
Wks
(13)
with the initial condition 8 1. Altogether, substituting Eq.
12 into Eq.
Oc Owf = 5) Eafzeer cpr
(14)
where of is backpropagated according to Eq. 13 The nonnegativity of the >>>>>>>> 1 weights and of G(h) ensure that 8 2 0 for all f. Under mild additional assumption (Appendix C), have strict positivity, 8f 0. Thus, setting the 8 term Eq. 14 to a positive constant yields an expression with the same sign as the gradient. This is precisely the GEVB learning rule of Eq. 9
Starting from a general vectorized network, one can derive weight nonnegativity, the required form of the vector nonlinearity, and the GEVB learning rule itself from the requirement that the gradient sign computable using only pre- and postsynaptic activity and the output error vector (Appendix D).
Recent theoretical work has shown that gradient sign information is sufficient for attaining or improving stochastic gradient descent convergence rates in non-convex optimization [8]. A caveat is that these rates assume that the gradient sign is computed on mini-batches, whereas GEVB matches the gradient sign on individual examples. In Section 6 show that training VNNs using batched GEVB updates yields performance on par with BP. Whether non-batched GEVB updates particularly effective by virtue of matching the gradient sign exactly is question for future study. Finally, we note that prior studies have demonstrated strong performance of a variant of FA, called sign symmetry, in which the feedback matrices have the same sign as the feedforward matrices [293I]]. By contrast, GEVB has no feedback parameters and provides weight updates with the same sign as the gradient. Sign symmetry has no such theoretical guarantee and thus presumably results in a looser approximation of the gradient.
4 Gradient alignment beyond sign agreement
The GEVB learning rule approximates the gradient by setting 8 in Eq. 14 to a positive constant for all units i in every layer f. This approximation is accurate if the empirical distribution of ot in layer is tightly concentrated about a positive value, and poor if this distribution diffuse. The level of concentration can be quantified as the relative standard deviation re azz /Pg. where μ3 and of denote the mean and variance the empirical distribution of 8 layer f. Since 8 O. we have r¹ O. When of 1, this distribution is close to a delta function positive value and 8 is accurately approximated as positive constant. The relative standard deviation is closely related to the angle 8 between GEVB weight updates and the gradient according to lº tan rd (see Appendix E for proof). This alignment-angle metric is commonly used to measure the quality of weight updates for learning [3 19.
Intriguingly, in randomly initialized VNNs, re scales inversely with ng in sufficiently early layers so that, in the limit of infinite network width, rf 0. Concretely, if the elements of W sampled i..d. from distribution with mean pnd > 0 and variance o² with o Pap smaller than order /ne, e obeys the recurrence relation
2 amet1
+ (ret1)2
(15)
ni411 Pott+
5

---

with the initial condition r r = Jwt/Put (see Appendix F for derivation; layer- GEVB up- dates are equal to the gradient by definition). Since / smaller than order yng, r' scales inversely with ny after sufficiently many recursions. Our experiments in VNNs used a non-i.i.d initialization described in Section S whose backward- i..d. weights with T /
behavior is qualitatively similar to using Our measurements of GEVB alignment angles at initialization (Section 6 Fig. [A) were in agreement with Eq.
13 Specifically, in layer L 1, we observed
g 45 corresponding to p ~ in layers to rd 1/ 1/yne+1
L 1, we observed small 0 corresponding
The limit of infinite network width has been shown to dramatically simplify the dynamics of gradient descent in DNNs, an idea embodied in the Neural Tangent Kernel (NTK) 32 33 Our finding that
credit assignment is simplified in the same limit suggests a potentially fruitful connection between the NTK regime and biologically plausible DNN training.
Initializing nonnegative networks with ON/OFF cells
GEVB can be applied in networks initialized with zero weights, method sometimes used with FA and DFA. However. BP incompatible with zero initialization as the gradient vanishes. To enable fair comparisons between GEVB and BP, we used the same nonzero initialization for both training methods. Random nonnegative initializations have the added benefit of placing VNNs in a regime in which GEVB weight updates are highly gradient-aligne, (Section 4 Unfortunately, nonnegative Li.d. initializations are unsuitable for initializing DNNs as they tend to produce exploding forward passes. Specifically, if the weights in layer e are sampled i.i.d. from a distribution with mean ~ and variance aapaance~//nu ~1/ 1, the weight matrix has an outlier eigenvalue size ~ "VTee- 1. and the projection of the pre-activations onto this mode grows across layers. Here, we present nonnegative initialization for VNNs based on ON/OFF cells that yields well-behaved forward propagation.
First, we group the hidden units into pairs with aanalland-opooiiin gating vectors t, defining the structure of ON and OFF cells. We then initialize the weights such that the units each pair have equal-and-opposirr activations. The subnetwork of ON cells has the same activations as a network of half the size with i.i.d. mixed-sign weights, and thus the ON/OFF initialization exhibits well-behaved forward propagation insofar as the underlying mixed-sign initialization does. To initialize W we sample an i...d. mixed-sign weight matrix W of half the size, then construct W according to
W = ( W +
+ + W1
W
E>I
(16)
W
where [-]- denotes positive rectification. These weights mixed-sign for & = 1 and nonnegative for > 1, consistent with the definition of VNNs. During training, the ON/OFF structure of the weights degrades, while the ON/OFF structure of the gating vectors is preserved.
The manner which ON/OFF initialization avoids exploding forward propagation can be under- stood as follows. Let (A, v) denote an eigenvalue/vector pair of W and (X+ x*) such a pair of abs( Wt). Then, (v, -u)/v2 and (u+, /tt//2 are eigenvectors of W with eigenvalues À and x+, respectively 34 Thus, the spectrum of W is the union of the spectra of W and (b.),, which contains a large outlier eigenvalue from abs( ("'), a nonnegative i.i.d. matrix. However, each pre-activation vector has the form h = (h. - T and the projection of ht onto the eigenvector of the outlier eigenvalue is zero since (u ) (h. -h) O.
In BP, the backpropagated signal lacks the ON/OFF structure of the forward prwaard-roppggted signal and therefore possesses a component along the eigenvector of the outlier eigenvalue, resulting in a growing backward pass. Our experiments used an optimizer with a normalizing effect (namely, Adam), preventing large weight updates in early layers when using BP (55 Importantly, when using GEVB, well-behaved forward propagation is sufficient for well-behaved weight updates.
6 Experimental results
Here, we show that GEVB performs well in practice. To disentangle the impact on network perfor- mance of vectorization and nonnegativity, trained vectorized and conventional networks, with

---

Table 1: MNIST test errors (%). Train errors are shown in parentheses if greater than 0.005. Errors corresponding to GEVB in VNNs are shown in color. For each type of architecture, the smallest GEVB or DFA test enror is bold.
Vectorized networks
Nonnegative
Mixed-sign
GEVB
BP
GEVB
BP
Fully connected
1.87 (0.06)
1.93 (0.05)
2.32 (0.33)
1.84 (0.07)
Convolutional
2.33 (1.03)
1.3 (0.19)
1.83 (0.83)
0.8
Locally connected
1.78
1.64
1.84 (0.05)
1.44
Conventional networks
Nonnegative
Mixed-sign
DFA
BP
DFA
BP
Fully connected
2.2 (0.3)
1.36
2.09 (0.19)
1.29
Convolutional
1.56 (0.67)
0.71
1.64 (0.42)
0.65
Locally connected
1.98 (0.32)
1.21
1.48 (0.09)
1.07
Table 2: CIFAR-I0 test emrors (%). Conventions are the same as in Table I
Vectorized networks
Nonnegative
Mixed-sign
GEVB
BP
GEVB
BP
Fully connected
47.62 (1.25)
47.03 (0.72)
48,86 (2.02)
45,98 (0.78)
Convolutional
33.74 (28.83)
30.85 (16.29)
38.43 (20.59)
30.54 (1.71)
Locally connected
41.08 (0.83)
41.01 (0.34)
40.11 (0.83)
38.77 (0.36)
Conventional networks
Nonnegative
Mixed-sign
DFA
BP
DFA
BP
Fully connected
48.69 (3.28)
45.42 (0.96)
49.34 (2.88)
45.69 (0.73)
Convolutional
54.18 (48.43)
32.13 (0.23)
44.07 (17.25)
28.8 (0.15)
Locally connected
41.18 (10.62)
35.51
39.41 (2.94)
32.32
and without nonnegativity constraint. Vectorized networks were trained using GEVB and BP, and conventional networks were trained using DFA and BP (see Appendix for DFA details). The nonlin- earity in conventional networks was the scalar case of the VNN nonlinearity with t = =1. Mixed-sign networks were initialized using He initialization, and nonnegative networks were initialized using ON/OFF initialization with an underlying He initialization (36). In conventional nonnegative net- works, the DFA feedback matrices were nonnegative [8]. We trained fully connected, convolutional, and locally connected architectures. Locally connected networks have the same receptive field structure as convolutional networks, but lack weight sharing [17]. We trained models on MNIST 37 and CIFAR-I0 (8), using wider and deeper networks for CIFAR-10. We used Adam for fixed number of epochs (namely, 190), stopping early at zero training error. For each experiment, we performed five random initializations. Training lasted 10 days using five NVIDIA GTX 1080 Ti GPUs. See Appendices G. H. and for details.
Our experimental results are summarized in Tables [nd 2 (see Appendix A for learning curves). We first examined the impact of vectorization and nonnegativity on model performance by considering errors under BP training. Across tasks and architectures, imposing vectorization or nonnegativity typically increased BP test and train errors, and imposing both increased the errors more than imposing either feature on its own. However, these increases in error were modest. One exception was locally connected networks trained on CIFAR- 10, whose test error increased by 8.69% upon imposition of vectorization and nonnegativity. As the train error increased by only 0.34%, these features primarily diminished the generalization ability, rather than the capacity, of the model.
Next, we compared GEVB to BP in vectorized networks. In fully and locally connected VNNs, GEVB achieved test and train errors similar to those of BP (0.14% and 0.59% maximum discrepancies in test error on MNIST and CIFAR-10. 10, respectively). In convolutional VNNs, we observed a small performance gap between GEVB and BP (1.33% and 2.89% discrepancies in test error on MNIST and CIFAR-10 10, respectively). One possible reason for this gap is that weight sharing in convolutional
7

---

A
Fully Connected
Convolutional
Lacally Connected
90-
90-
eigremei 17)
75-
75
80-
60-
60-
GEVB in VNN
45-
45-
45-
30-
30
30
15
15
15
Dwt/
Corn3
151
B
80
120
100
60
80
120
100
40
80
120
100
90-1
90-
90-
75
75
75
eigremet (")
DFA in conventional nonnegative network
80
60
60
45
45
45
20
30
30
15
1C3
FC2
7C3
Dmts
Cons2
Cond
FCL
LC1
122
40
80
120
160
40
80
120
160
40
80
120
160
spochs
spochs
spochs
Figure 2: Gradient alignment angles over the course of training on CIFAR-10. In the legend, layers progress from early to late from left to right. Weight updates to the readout layer are equivalent to the gradient for GEVB and DFA, so these layers are not shown. (A) GEVB in VNNs. (A) DFA in conventional nonnegative networks. Error bars are standard deviations across five runs. See Appendix A for corresponding plots for mixed-sign networks and MNIST.
networks breaks GEVB's gradient sign match guarantee. Nevertheless, CIFAR-10. GEVB in convolutional VNNs yielded substantially lower test error than all other convolutional experiments using GEVB or DFA, feature we examine in detail below. Finally, in vectorized mixed-sign networks. the performance gap between GEVB and BP was larger than in VNNs across tasks and architectures. Rather than enjoying guaranteed gradient alignment by virtue of nonmegative-constraindd weights, learning in these networks relied on the tendency of GEVB to generate a bias toward positive weights, a special case of the feedback alignment effect 13
As indicated by the bold entries in Tables D and GEVB in vectorized networks in some cases outperformed DFA in conventional networks. GEVB had lower test error than DFA for fully connected architectures on MNIST, and for fully connected and convolutional architectures on CIFAR- 10. Moreover, GEVB tended to produce considerably lower train errors than DFA. This is particularly impressive in light of the fact that vectorized networks have higher errors than conventional networks under BP training. In cases where DFA outperformed GEVB, the gap in train error tended be much smaller than the gap in test error, suggesting that VNNs were overfitting. This is plausible as VNNs have a factor of K more parameters than conventional networks in the first hidden layer (Section 3.1]]
To gain insight into our experimental results, we measured the alignment of both GEVB and DFA weight updates with the gradient over the course of training (Appendix E). Following previous works, we computed the alignment between single-example updates and averaged these values over each mini-batch [13 19. Consistent with the theoretical result of Section 4 GEVB alignment angles at initialization were around 455 for layer I, - 1, and small (typically under 20%) for layers & < T. (Fig. AA). Over the course of training, these small angles increased, plateauing around or below 45%. By contrast, DFA exhibited alignment angles ~000 at initialization, and these angles dropped over the course of training due to the feedback alignment effect (Fig. 2B). The plateaued alignment angles for GEVB were typically smaller than those for DFA. In convolutional networks, DFA updates failed to align with the gradient in convolutional layers, consistent with large errors as well as prior studies By contrast, GEVB had plateaued alignment angles around 45 for convolutional layers.
[16 19 20
Finally, examined the ability of GEVB to train convolutional layers by studying the learned representations on CIFAR-10 at the output of the convolutional layers. t-SNE embeddings of these representations before and after training revealed that GEVB, but not DFA, improved the level of class clustering (Fig. 3A see Appendix K for t-SNE details) (39]. We quantified this improvement using measure of cluster quality defined as 1 - ( Σ r?2/r², where r and r denote the average class-c and average overall squared pairwise distance, respectively, in the t-SNE embedding. For all experiments, the cluster quality was slightly greater than zero at initialization and typically improved during training (Fig. B) DFA largely failed to improve cluster quality. By contrast, GEVB not only matched, but surpassed BP's improvement in cluster quality in vectorized networks.
8

---

A
initialization
trained
0.20
Accluster quality)
0.15
GEVB in VNN
0.10
0.05
0.00
DFA conventional nonnegative network
GEVB BP GEVB BP Nonneg. Mixed
DFA BP DFA BP Nonneg. Mixed
Vectorized
Conventional
Figure 3: Comparison of learned representations at the output of convolutional layers in networks trained on CIFAR-10. (A) t-SNE embeddings of five image classes before and after training for GEVB in VNNs (top) and for DFA in conventional nonnegative networks (bottom). (B) Change in cluster quality (defined main text) due to training for all eight convolutional experiments on CIFAR-10. GEVB in VNNs is shown in color. Error bars are standard deviations across five runs.
The observation that GEVB not only trains convolutional layers, but surpasses BP's improvement in cluster quality in vectorized networks, suggests that GEVB applies more pressure than BP on early layers to extract task-relevant features. Understanding how the representations generated by GEVB differ from those generated by BP is an avenue for future study. Such an understanding could enable disambiguation cortical learning algorithms on the basis of neural recordings 44
7 Discussion
Biological implementation: Our work raises the question of how vector units and the GEVB learning rule could be implemented neural circuits. Here, we suggest two possible implementations. For each solution, we describe the required circuit architecture, implementation of the VNN nonlinearity, and formulation of GEVB in terms of three- factor Hebbian plasticity 24,22)] 25).
We first consider a solution in which each vector unit is implemented by group of K neurons. Consider a presynaptic group j in layer group j projects only to neuron μ in group Let w denote the strength of this connection. As vectorized weight sharing not enforced automatically in neural circuits, this weight has a J index. The nonlinearity silences group of neurons if the activity pattern of the group has positive alignment with the gating vector. This could implemented through an inhibitory circuit mechanism. Given an input example and a target output, the network computes the error vector e. The K components of this error vector are then distributed throughout the network via K distinet neuromodulatory signals, where the p-th signal is proportional to eu- The p-th signal modulates only the w-th weight wfyer Then, each synapse undergoes three-factor Hebbian update, Aut, = - f G(h Here, en is the global third factor, d// is the presynaptic rate, and G(h!) is a function of the postsynaptic rate. We recover the GEVB learning rule by relaxing the strengths of the K updated synapses between groups j and to their average, * Σn why Recent work demonstrates that lateral connections and anti-Hebbaie plasticity can induce such a relaxation to the average during sleep phase 422
1 and a postsynaptic group in layer f. Neuron μ in
Next, we consider a solution in which vectorization unfolds in time. Each vector unit is implemented by single neuron. In each of K time bins, different set of input neurons are active producing single component of the network output in each bin. As the network processes inputs at different times using the same weights, vectorized weight sharing is enforced automatically. As per the nonlinearity, each neuron must be active or inactive at all time bins according to the sign of t h, where h is the pre-activation vector, each element of which corresponds to single time bin. Computing this inner product requires the full temporal input, violating causality. One workaround is to make the gating vector sparse in all but its first component so that each neuron is active or inactive based on input at the first time bin. Alternatively, gating could be implemented by an external inhibitory signal that is independent of the pre-activations [43] In time bin μu, each synapse wi, undergoes a factor
8-1 G(h)). Here, ea is the global third factor at time bin μ,, d is Hebbian update, Awl, ap

---

the presynaptic rate at time bin jr, and G(h is function of the postsynaptic rate. Integrating these updates over all K time bins performs a sum over μ.. implementing the GEVB rule.
These implementations make distinct predictions for the spatiotemporal structure of the neuromodula- tory signal. The "grouped" implementation predicts that this signal is spatially heterogeneous, that each synapse is modulated by the appropriate component of the error vector, and temporally uniform. By contrast, the "temporal" implementation predicts that this signal is temporally heterogeneous, so that the appropriate component of the error vector is broadcast during each time bin, and spatially uniform. Both implementations predict an important role for inhibition silencing groups of neurons or individual neurons in a stimulus deellddeet manner. Raman and O' O'Leary [26] have suggested that vectorized feedback signals could be a general mechanism for increasing the precision of credit assignment in neural circuits, providing evidence for this mechanism in the Drosophila connectome. Our findings are in line with this view.
Prior uses of vector units: Other works have used vector units for computational, rather than credit- assignment, purposes [47]. Capsule networks use vector-valued capsules to represent object parts [48) (see also Hinton [49). Vector neuron networks (not to be confused with our proposed architecture, vectorized nonnegative networks) use three-dimensional vector units achieve equivariance the rotation group, SO(3), for three-dimensional point-cloud input data (50). In gated linear networks, each unit stores vector of parameters describing a probability distribution with the same support as the network output [43 45). Gated linear networks use a form of vectorized weight sharing so that each unit represents a weighted geometric mixture of the distributions represented by units in the previous layer. In some cases, it is natural to constrain these weights to be nonnegative [44]. Vector units in VNNs can be interpreted as performing weighted geometrie mixtures of categorical probability distributions, where each each vector activation encodes a vector of logits.
Prior work on eredit assignment: In addition to works we have already mentioned, a wide variety of solutions to the credit assignment problem have been proposed. One approach involves dynami- cally generating approximate weight symmetry [ [1 [52] Another approach, target propagation, preserves information between layers by training stack of autoencoders 53). In contrast to these methods, GEVB assumes a nonnegative feedforward pathway, obviating the need for a feedback pathway. Rather than minimizing an output loss, as in GEVB, several methods forego optimizing a global objective. For instance, some methods optimize neuron- or layer-local objectives 443
54 56
enable training of DNNs using three- factor Hebbian learning. Several works have proposed local plasticity rules that perform unsupervised learning [59-62]. Still other approaches to credit assignment include equilibrium propagation [63 64 and methods based on meta-learning (65)69). Some works investi- gate the mechanism of credit assignment in specific biological structures 169 70]. Exciting recent work shows that complex tasks can be learned in a segregated-dendrice model which feedback connections trigger bursting events that induce synaptic plasticity (23). In general, we are in need of a theoretical framework linking structure to learning capabilities in neural circuits (26).
Notably, methods based on the information bottleneck (57] and contrastive prediction 58
Scaling to digh-dime outputs: Using a large output dimension K in VNNs is cumbersome as the computational complexity of the forward pass scales in K (by contrast, the complexity of the backward pass does not scale in K). One solution is to use target vectors of size K ~ log C for C-way classification with K bit binary, rather than one-hot, encodings of the target class index. This is especially appropriate if the binary encoding is congruent with hierarchical structure in the data. For example, such structure is present in ImageNet [71]. If vectorization unfolds in time, it is natural to produce more coarse-grand -aarse-grained classifications of the input first. Also, note that the forward pass in VNNs is highly parallelizable: at each layer. the K required matrix- vector products can be performed in parallel. Communication is then required to compute O( h) at each vector unit.
Future directions: Given GEVB's gradient sign match property and ability to train convolutional layers, our method is poised to scale to more complex tasks. GEVB often outperforms DFA, which successfully trains several modern architectures [19]. Biologically, the main work ahead lies in determining whether the core insights of our paper can be adapted to more realistic network models with spiking neurons and biophysically motivated plasticity rules (23] Finally, the GEVB learning rule is designed to be aligned with the gradient, but gradient-based learning may not be optimal in scenarios such as few-shot or continual learning. Thus, while cortical circuits could plausibly use gradient-based based learning, whether they should remains unclear. Future work should consider learning algorithms that are both biologically plausible and overcome shortcomings of BP 67 72
10

---

Acknowledgements
We thank Jack Lindsey for insightful comments on an earlier version of this manuscript. We thank Greg Wayne, Jesse Livezey. and members of the Abbott lab for helpful pointers and discussion. Research was supported by NSF Neuronex 1707398 and the Gatsby Charitable Foundation GAT3708. The authors declare no competing financial interests.
References
[1] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by propagattg errors. Nature, 323(6088):533 -3-6,,66111 1986.
[2] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553):436 2015.
[3] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. deep learning framework for neuroscience. Nature neuroscience, 22 11:17)1-1170000 2019.
[4] Eric R Kandel, James H Schwartz, Thomas Jessell, Steven Siegelbaum, A James Hudspeth, and Sarah Mack. Principles of neural science, volume 4. McGraw New York, 2000.
[5] Francis Crick. The recent excitement about neural networks. Nature, 337(6203):129-1333 1989.
[6] James CR Whittington and Rafal Bogacz, Theories of error back-propagation in the brain. Trends in cognilive sciences, 23(3):235-255 250, 2019.
[7] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
[8] Peter Latham. Node perturbation in vanilla deep networks, Feb 2019. URL http://wwi www gatsby. ucl. ac. wwl..ac.uk/pppl//aa.... http///ootee///ooee http://teeeeoooeeeeeeraaaaeeeeaa
pdf
(9] Pietro Mazzoni, Richard A Andersen, and Michael Jordan. A more biologically plausible learning rule for neural networks. Proceedings of the National Academy of Sciences, 88(10): 1991.
[10] Ronald J Williams. Simple statistical ment learning. Machine learning, 8(3-4):229-255 1992.
algorithms for connectionist reinforce-
[11] Justin Werfel, Xiaohui Xie, and H Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. Neural computation, 17(12):2699-25 0112269997711.. 2718, 2005.
[12] Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23-63 1987.
[13] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. Nature communications, 7(1):1-10, 2016.
[14] Arild Nø kland. Direct feedback alignment provides learning in deep neural networks. In Advances in Neural Information Processing Systems, volume 29, 2016.
(15) Arash Samadi, Timothy P Lillicrap, and Douglas B Tweed. Deep learning with dynamic spiking neurons and fixed feedback weights. Neural computation, 29(3):578-602, 2017.
[16] Julien Launay, lacopo Poli, François Boniface, and Florent Krzakala. Direct feedback alignment scales modern deep learning tasks and architectures. In Advances in Neural Information Processing Systems, volume 33, 2020.
[17] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability biologically-motival deep learning algorithms and architectures. In Advances in Neural Information Processing Systems, volume 31, 2018.

---

[18] Mathias Lechner. Learning representations for binary clasiiyclasssiication without backpropagation. In International Conference on Learning Representations, 2019.
[19] Julien Launay, lacopo Poli, and Florent Krzakala. Principled training of neural networks with direct feedback alignment. arXiv preprint arXiv:/906.044 2019.
[20] Maria Refinetti, Stéphane d' A'cooli Ruben Ohana, and Sebastian Goldt. learning with feedback alignment. arXiv preprint arXiv:2011. 2020.
The dynamies of
[21] Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. ELife, 6xe22901, 2017.
[22] Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current opinion in neurobiology, 54:28-36, 2019.
[23] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, pages 1-10, 2021.
[24] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike-timing-dependeee plasticity, and theory of three-factor learning rules. Frontiers in neural circuits, 9:85, 2016.
[25] Lukasz Kuámierz, Takuya Isomura, and Taro Toyoizumi. Learning with three factors: modulat- ing hebbian plasticity with errors. Current opinion in neurobiology, 46: 46:170-11 1701177, 2017.
[26] Dhruva V Raman and Timothy O'Leary. Frozen algorithms: how the brain's wiring facilitates learning. Current Opinion Neurobiology. 67:207-22 -24,, 2021.
[27] David Balduzzi, Hastagiri Vanchinathan, and Joachim Buhmann. Kickback cuts backprop's s red-tape:
Biologically plausible credit assignment in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[28] Jeremy Bernstein. Yu-Xiang Wang, Kamyar Azizzadenesheli. and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560-569. PMLR, 2018.
[29] Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in back- propagation? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[30] Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausible learning algorithms can scale to large datasets. In International Conference on Learning Representations, 2018.
[31] Theodore H Moskovitz, Ashok Lutai,-Kumar, and LF Abbott. Feedback alignment in deep convolutional networks. arXiv preprint arXiv: 1812.0681,,06488,, 2018.
[32] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemne:: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
[33] Jaehoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Journal of Statistical Mechanics: Theory and Experiment, 2020 (12), dec 2020.
[34] Block matrix, Jul 2021. URL htt.pppp..ee...ii.oocc..oo/////e
[35] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 2014.
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
12

---

[37] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):278-2322 1998.
[38] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tronto, 2009.
[39] David M Chan, Roshan Rao, Forrest Huang, and John Canny, l-sne-cuda: Gpu-accelerated t-sne and its applications to modern data. In 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pages 330-338. IEEE, 2018.
[40] Yinan Cao, Christopher Summerfield, and Andrew Saxe. Characterizing emergent representa- tions a space of candidate learning rules for deep networks. In Advances in Neural Information Processing Systems, volume 33, 2020.
[41] Aran Nayebi, Sanjana Srivastava, Surya Ganguli, and Daniel L Yamins. Identifying learning rules from neural network observables. In Advances in Neural Information Processing Systems, volume 33, 2020.
[42] Roman Pogodin, Yash Mehta, Timothy Lillicrap, and Peter E Latham. Towards biologically plausible convolutional networks. arXiv preprini arXiv:2106.13033 2021.
[43] Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopehand, Christopher Mattern, Ag- nieszka Grabska-Barwinska, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, et al. Gated linear networks. arXiv preprint arXiv:1910.01526. 2019.
[44] David Budden, Adam Marblestone, Eren Sezener, Tor Lattimore, Gregory Wayne, and Joel Veness. Gaussian gated linear networks. In Advances in Neural Information Processing Systems, volume 33, 2020.
[45] Eren Sezener, Agnieszka Grabska-Barwinska. Dimitar Kostadinov, Maxime Beau, Sanjukta Krishnagopal, David Budden, Marcus Hutter, Joel Veness, Matthew Botvinick, Claudia Clopath, al. A rapid and efficient learning rule for biological neural circuits. bioRxiv, 2021.
[46] Chandrashekar Lakshminarayanan and Amit Vikram Singh. Neural path features and neural path kemel: Understanding the role of gates in deep learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
[47] H Eugene Stanley. Dependence of critical properties on dimensionality of spins. Physical Review Letters, 20(12)5589, 1968.
[48] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules, In Advances Neural Information Processing Systems, volume 30, 2017.
[49] Geoffrey Hinton. How to represent part-whole hierarchies in a neural network. arXiv preprint 12X1v:2/02.12627, 2021.
[50] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulcnard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for so (3)-qquivariant networks. arXiv preprint arXiv:2104.12223 12229, 2021.
[51] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances Neural Information Processing Systems, 32, 2019.
[52] Daniel Kunin, Aran Nayebi, Javier Srgastuy-Brenas Surya Ganguli, Jonathan Bloom, and Daniel Yamins. Two routes to scalable credit assignment without weight symmetry. In International Conference on Machine Learning, pages 5511-552) PMLR, 2020.
[53] Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv: /407.7904 2014.
[54] Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs. Deep supervised learning using local errors. Frontiers in neuroscience, 12:608, 2018.
13

---

[55] Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In International Conference on Machine Learning. pages 4839-4850. PMLR, 2019.
[56] Shanshan Qin, Nayantara Mudur, and Cengiz Pehlevan. Contrastive similarity matching for supervised learning. Neural Computation, 33(5):1300-1328, 2021.
[57] R Pogodin and PE Latham. Kernelized information bottleneck leads to biologically plausible
3 3-factor hebbian learning in deep networks. In CoRR, 2020.
(58) Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. arXiv preprint arXiv:2010.05262, 2020.
[59] Yanis Bahroun and Andrea Soltoggio. Online representation learning with single and multi-layer hebbian networks for image classification. In International Conference on Artificial Neural Networks, pages 354 144-363.22 Springer, 2017.
[60] Dmitry Krotov and John J Hopfield. Unsupervised learning by competing hidden units. Pro- ceedings of the National Academy of Sciences, 116(16):7723-7771, som 2019.
[61] David Lipshutz, Yanis Bahroun, Siavash Golkar, Anirvan M Sengupta, and Dmitri Chklovskii.
A biologically plausible neural network for multichannel canonical correlation analysis. Neural Computation, 33(9):2399-2355) 2021.
[62] Siavash Golkar, David Lipshutz. Yanis Bahroun, Anirvan Sengupta, and Dmitri Chklovskii. A simple normative network approximates local non-hebbian learning in the cortex. Advances in Neural Information Processing Systems, 33, 2020.
[63] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
[64] Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation to deep convnets by drastically reducing its gradient estimator bias. Frontiers in neuroscience, 15:129, 2021.
[65] Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. arXiv preprint arXiv: 1804000204 2018.
[66] Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Kording. Learning to solve the credit assignment problem. arXiv preprint arXiv:1906.00SS: 2019.
[67] Jack Lindsey and Ashok Kumai. Learning to learn with feedback and local plasticity. In Advances in Neural Information Processing Systems, volume 33, 2020.
[68] Danil Tyulmankov, Guangyu Robert Yang, and LF Abbott. Meta-learning local synaptic plasticity for continual familiarity detection. bioRxiv, 2021.
[69] Linnie Jiang and Ashok Litwin-Kumar. Models of heterogeneous dopamine signaling in an insect learning and memory center. PLoS Computational Biology, 17(8)ee 2021.
[70] Salomon Muller, Abigail N Zadina, LF Abbott, and Nathaniel B Sawtell. Continual learning in multi-layer network of an electric fish. Cell. 1396)):38221139) 2019.
[71] Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-enn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE international conference computer vision, pages 2015.
[72] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grrabsa-BBrwiinkhe Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13): 3521-3526, 2017.
14


Reviews:
[{"review_id": "ypwXnWkWF_", "reviewer": "Reviewer_Pci4", "summary": "This paper proposes (i) a learning rule called global error-vector broadcasting (GEVB), and (ii) a class of DNNs called\nvectorized nonnegative networks (VNNs) in which this rule operates - to solve the credit assignment problem. While Backpropgation (BP) is remarkably effective at training deep neural networks, it is biologically implausible because of the weight transport problem. In this paper, the weight transport problem is circumvented by globally broadcasting the error signal to all hidden units in the network. However, in order to accommodate the vector error signal a new network model with vector units is proposed. \n\nFurther, by deriving gradients using a BP like approach, the authors demonstrate gradient alignment with the GEVB learning rule. The performance of VNNs trained with GEVB is evaluated on MNIST and CIFAR-10 and compared with BP on VNNs and conventional networks.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Significance__: Understanding credit assignment in biological networks is an important problem. Several approaches have been considered to circumvent the weight transport problem of BP, such as node perturbation and feedback alignment. In this work, an interesting alternative approach based on broadcasting an error signal is considered. However, the impact of this approach hinges on finding a biological plausible implementation of VNNs. \n\n__Originality__:  The main idea in this paper is the broadcasting of the error signal to all hidden units of the network. The broadcast error is combined in a Hebbian-like learning rule for each weight. This is in contrast to sending a unique projection of the error signal to each hidden unit as in direct feedback alignment. However, in order to accommodate a vector error signal, the units in the network are modified to have vector activities instead of scalars. This work also provides analyses that demonstrate gradient matching for the GEVB rule.\n\n__Quality__: The experimental results show the impact of vectorization and nonnegativity of weights on the VNN model performance. In most cases the performance of VNNs trained with GEVB was quite close to BP. Gradient alignment analysis reveals that GEVB results in better alignment than DFA on average. While the model itself is interesting, the biological implementation of vectorization is not quite convincing.  Another important issue is that training large networks with this approach is computationally challenging. \n\n__Clarity__:  The paper is well presented and easy to follow. \nMinor: Caption for figure 2 has label (A) for both panels.\n\n__Questions__:\n* An important consideration is how the error signal is broadcast to all units in the network. Do we expect the influence of a neuromodulatory system to be exactly the same on all units in a large network? How would some stochasticity in the error signal received at each unit affect the performance?\n* The activation function used here is of a very specific form. How could this be biologically implemented? \n* Non-negativity is important for gradient sign matching. And this corresponds well with the fact that a majority of cortical neurons are excitatory. However, inhibitory neurons do constitute a small but crucial neuronal class in the cortex. How well would VNNs with GEVB work without the sign constraint on the weights?\n* The comparison of learned representations was done using networks trained just on CIFAR-10. This is an interesting analysis but is it conclusive enough to say GEVB in VNNs learns better representations? \n \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work have been discussed by the authors - performance discrepancies on more challenging tasks, biological plausibility, and the computational complexity of the forward pass in VNNs. \n\nThere are no other potential negative societal impacts or ethical concerns. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}, {"review_id": "pKUZ5Bq4XZS", "reviewer": "Reviewer_wVMT", "summary": "This paper proposes to overcome an important biological implausibility of neural networks trained by backpropagation using two novelties: 1) to train neural networks using a new globally available error signal, 2) a new type of neural network with vector-valued units and nonnegative weights. Interestingly, the model shows that the proposed learning rules lead to update rules with the same sign as those obtained by gradient-descent. \n\nThis work justifies the biological plausibility of the learning rule by relating it to the growing literature on three-factor Hebbian rules and justifies the choice of nonnegative weights to observed cortical properties, as well as the presence of ON/OFF cells. \n\nThe performance of the proposed network is compared, on MNIST and CIFAR-10 datasets, against a very standard set network trained with BP, as well as more recent models trained using direct feedback alignment. The model outperforms competing DFA-trained models for fully connected and convolutional architectures but not for locally connected architectures. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Below is addressed the clarity. \n\nThis paper is clear and well-written, and the claims are clearly laid out. However, the notations are not easy to follow, in particular the not-standard choice of indexing of the network. It would have been useful to relate the notation with neural architecture presented in Figure 1, where it is very unclear how it relates to the rest of the notations used in the paper. It is also true for the \"biological interpretation\" (section 7) that does not easily relate to the architecture in Fig.1. \nThe results in Table 2 are not easy to compare as the competing models (bold) results are presented in different \"sub\"-tables limiting the readability. The rest of the results, Fig.2 and Fig.3, are clear and well described in the caption and the text. \n\nBelow is addressed the originality. \n\nThis work makes an interesting effort at providing \"local\" learning rules, which is an important limitation of BP; as such, the learning rule is novel. The comparison to three-factor Hebbian rules is mentioned but is not clearly analyzed. The work of [24], mentioned in the paper, offers a generic framework and discusses possible biological structures in which it could occur, unlike in this work. The comparison with this work needs to be clearly stated as the model claims to be biologically sensible due to the presence of the aforementioned third factor. \nThis work claims to be both theoretical and original; however, proposing an ad-hoc learning rule can hardly be qualified to be theoretical when its analysis lacks theorems, or propositions, or even lemmas. It is suggested to the author to either rewrite the paper more mathematically or to provide more experiments and biological realism. The next point relates to the lack of experiments and even mention of competing models that are potentially more biologically plausible than those presented in the paper. Indeed, pieces of work from [i] and [ii] could rely on standard unsupervised Hebbian/Hopfield models followed by an SVM (that can be easily trained with biologically plausible models) and reach competing performance on CIFAR-10. As mentioned in the limitation review, the lack of proper theoretical analysis, and biological implementation, of the algorithm does not allow for a weak experimental section. We invite the reader to strengthen at least two out of three of those contributions. Various other models have also been proposed [iii-v] that have been omitted to be cited, using local biologically plausible learning rules and using 3-factor Hebbian rules. \n\nBelow is addressed the significance. \n\nThis work is definitely a step in the right direction in building biologically plausible algorithms but falls short of making a significant contribution. As mentioned above, the paper needs substantial improvement in at least one of the following contributions, theoretical, experimental, or biological realism. \n\n\n[i] Krotov, Dmitry, and John J. Hopfield. \"Unsupervised learning by competing hidden units.\" Proceedings of the National Academy of Sciences 116.16 (2019): 7723-7731.\n[ii] Bahroun, Yanis, and Andrea Soltoggio. \"Online representation learning with single and multi-layer Hebbian networks for image classification.\" International Conference on Artificial Neural Networks. Springer, Cham, 2017.\n\n[iii] Pogodin, Roman, and Peter E. Latham. \"Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks.\" NeurIPS 2020/\n[iv] Golkar, Siavash, et al. \"A biologically plausible neural network for local supervision in cortical microcircuits.\" NeurIPS 2020.\n[v] Qin, Shanshan, Nayantara Mudur, and Cengiz Pehlevan. \"Contrastive Similarity Matching for Supervised Learning.\" Neural Computation 33.5 (2021): 1300-1328.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Two other important limitations of deep networks have not been addressed. \n\nThe first one is their lack of transparency, and this model remains \"black-box,\" not offering a better understanding of how the learning is performed. The learning rule is compared to that of the gradient methods, which have shown to be hard to analyze and understand and do not address this problem. The claimed biological plausibility could have led to more understanding of the learning in the brain but fails to do so as it only vaguely addresses the topic. However, it appears to be a central point to the paper. \n\nThe second one is the bias that has been observed in many deep neural networks trained with backpropagation. Although it is not expected that the model would solve this problem, it still needs to be mentioned. \n\nOn a side note, the paper claims that a limitation of this work is the lack of experiments reported, i.e., only evaluated on MNIST and CIFAR-10. This paper is mostly experimental rather than theoretical; no theorem, proposition, or lemma is stated in the main text. As a result, more experiments are expected. This is thus not a limitation of the paper but a weakness of the submission. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "CSOX49CGcru", "reviewer": "Reviewer_osc2", "summary": "This paper presents a biologically plausible learning method, called global error-vector broadcasting (GEVB). The authors also propose vectorized nonnegative networks (VNNs) on which the GEVB method operates. The key property of the GEVB method is that the gradient has the same sign as the ground truth gradient returned by back-propagation (BP) on VNNs. Experiments on MNIST and CIFAR are conducted to show that the proposed method can sometimes match the performance of BP and sometimes outperform competitors like direct feedback alignment.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1, The proposed GEVB essentially propagates the error signal from the readout layer to all other layers in a direct manner. It seems to be more efficient and biologically more plausible compared to BP since it avoids the symmetric weights and the successive computation of gradients across layers. I think this is a novel and interesting contribution towards more biologically plausible learning methods.\n\n2, The authors also show that in the limit of infinite network width, the proposed update in GEVB is proportional to the exact gradient at the initialization. This observation provides some potentially interesting connections to the recent study on infinite wide networks like NTK.\n\n3, The empirical results show that the proposed learning method 1) does converge and 2) is comparable or even outperforms other alternatives. In particular, it successfully trains convolutional layers whereas other biologically plausible methods like direct feedback alignment (DFA) have some troubles.\n\nCons and Questions:\n\n1, I agree that the proposed GEVB is more biologically plausible compared to BP. However, the introduction of this new type of architecture, i.e., VNNs, seems to be unsatisfying. In particular, do you have some evidence to support that VNNs are biologically more plausible than other common architectures?\n\n2, Why do you need the random gating vectors per layer? If I understood correctly, your method and theory still work if those random gating vectors are removed. It is unclear to me what the motivation and the impact of this additional design choice are. More importantly, this random gating would inevitably bring some stochasticity to the final predictions. Therefore, it would be important to do a thorough ablation study on this component and show the standard deviations of the final performances corresponding to the aforementioned stochasticity.\n\n3, Could you explain why the training and testing errors of GEVB are inconsistent for fully-connected and convolutional VNNs, i.e., better than or comparable to BP for fully-connected VNNs and significantly worse than BP for convolutional VNNs? Moreover, for convolutional VNNs, why does GEVB outperform DFA on CIFAR but perform worse on MNIST?\n\n4, When you compute the gradient alignment angle, do you compute the angle per sample and then take the average, or do you compute the angle between gradients averaged over mini-batch? I am curious whether these two have significant differences since matching the signs of individual gradients is different from matching the signs of gradients averaged over mini-batch. From the perspective of optimization, we should care more about the latter.\n\n5, In Fig. 2, could you comment on why the angle curve of the last readout (FC) layer seems to be diverging on fully connected VNNs? Also, why do the curves of the last readout (FC) layer first increase and then decrease rather than directly decrease on both fully-connected and convolutional VNNs?\n\n6, The current method seems to be surprisingly slow, e.g., taking 10 days of GPU training even on CIFAR and MNIST. Could you comment on what are the exact factors that slow down the training?\n\n## After Rebuttal\n\nI'd like to thank the authors for the detailed response! It does address most of my concerns. \nI was expecting authors to comment in detail on if there is evidence in neural science to back up the existence of some form of the VNNs. \nHowever, I found the current response on this point less informative.\nIn addition, I agree with *Reviewer wVMT* that thorough experimental comparison on more datasets would significantly improve the paper.\nIn summary, I would like to keep my original rating.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the computational cost as the potential negative societal impact, which I think is fair.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6v6xYs4lkY", "reviewer": "Reviewer_oN9Y", "summary": "The paper proposes to circumvent the need of backpropagation in deep networks by introducing two architectural changes. First, the weights are restricted to be non-negative so the chain rule simplifies significantly. Second, each unit in a network is represented by a vector to match the dimensionality of the error signal in the top layer, further simplifying the chain rule. The resulting weight update approximates backpropagation similar to the sign symmetry method.      ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a very novel and intriguing way to approximate backprop. The discussion on feedback alignment-type (FA) algorithms is missing two important papers, however. First, sign symmetry [1] is another FA-type algorithm that uses signs of the weights for the backward pass, and I think this algorithm works for a similar reason (although GEVB should be a tighter approximation due to non-negative weights; [26] in the paper seems relevant too, but [1] was published a few years earlier). Also, [2] shows how to introduce plasticity in the backward connections of FA to bring it on par with backprop, so it’s worth including into the discussion. Finally, non-FA-type methods [3-7] might be worth discussing (and comparing in experiments) too.\n     \nQuality: The submission is technically sound, and provides sufficient theoretical and experimental justification.\n     \nClarity: The paper is well-written. One comment: line 236: “One possible reason for this gap is that weight sharing in convolutional networks breaks GEVB’s gradient sign match guarantee.” Does it break it though? Convolutions would add a sum over $i$ in Eq.14, resulting in $(\\\\sum_i \\\\delta_i G(h_i)) (\\\\sum_\\\\mu ...)$. GEVB should change $\\\\sum_i \\\\delta_i G(h_i)$  to $\\\\sum_i G(h_i)$, which doesn’t seem to break the approximation due to non-negativity of $G$. \n     \nSignificance: Overall, the paper is a significant result with future potential. The major limitation of the paper is weight sharing within vector-valued units. While the authors do provide a potential mechanism for that (through time unfolding), it implies a rather complicated training curriculum (especially for large K). However, this limitation is unconventional and might be solved with future methods/experimental evidence. \n\n[1] How Important is Weight Symmetry in Backpropagation?, Qianli Liao, Joel Z. Leibo, Tomaso Poggio\n\n[2] Deep Learning without Weight Transport, Mohamed Akrout, Collin Wilson, Peter C. Humphreys, Timothy Lillicrap, Douglas Tweed\n\n[3] Deep supervised learning using local errors, Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs\n\n[4] Training neural networks with local error signals, Arild Nøkland and Lars Hiller Eidnes\n\n[5] Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks, Roman Pogodin and Peter Latham\n\n[6] Local plasticity rules can learn deep representations using self-supervised contrastive predictions, Bernd Illing, Jean Ventura, Guillaume Bellec, Wulfram Gerstner\n\n[7] Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing Its Gradient Estimator Bias, Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier and Damien Querlioz\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method to train neural networks in a way that is more biologically plausible than backpropagation. The key idea is to derive an update that depends on a global error signal. Work in this direction had already been identified for neural network with a scalar output, however the authors generalize the setup with vector outputs, but restricting the architectures to a very particular class of neural networks they introduce called VNN (effectively operating on vectors with vectorized weight sharing). They show the resulting update matches the sign of the gradient. Reviewers were in agreement the idea was interesting and elegant. 
Doubts remained regarding biological plausibility, with reviewers thinking the authors arguments in that direction were somewhat hand-wavy. Nevertheless, we hope this work can inspire further work in that field that could be even better substantiated.
There were some concerns regarding the extensiveness of the literature review that should be addressed for the final version.

Decision: Accept (Poster)