Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
Authors: Beren Millidge
Alexander Tschantz
Anil K Seth
Christopher Buckley
Venue:  (2021)
Keywords: Neural Networks
Biological Plausibility
Backprop

Abstract: The backpropagation of error algorithm (backprop) has been instrumental in the recent success of deep learning. However, a key question remains as to whether backprop can be formulated in a manner suitable for implementation in neural circuitry. The primary challenge is to ensure that any candidate formulation uses only local information, rather than relying on global signals as in standard backprop. Recently several algorithms for approximating backprop using only local signals have been proposed. However, these algorithms typically impose other requirements which challenge biological plausibility: for example, requiring complex and precise connectivity schemes, or multiple sequential backwards phases with information being stored across phases. Here, we propose a novel algorithm, Activation Relaxation (AR), which is motivated by constructing the backpropagation gradient as the equilibrium point of a dynamical system. Our algorithm converges rapidly and robustly to the correct backpropagation gradients, requires only a single type of computational unit, utilises only a single parallel backwards relaxation phase, and can operate on arbitrary computation graphs. We illustrate these properties by training deep neural networks on visual classification tasks, and describe simplifications to the algorithm which remove further obstacles to neurobiological implementation (for example, the weight-transport problem, and the use of nonlinear derivatives), while preserving performance.

Under review as a conference paper at ICLR 2021
ACTIVATION RELAXATION: A LOCAL DYNAMICAL APPROXIMATION TO BACKPROP IN THE BRAIN
Anonymous authors Paper under double-blind review
ABSTRACT
The backpropagation of error algorithm (backprop) has been instrumental in the recent success of deep learning. However, a key question remains to whether backprop can be formulated in a manner suitable for implementation in neural circuitry. The primary challenge is to ensure that any candidate formulation uses only local information, rather than relying on global signals as in standard backprop. Recently several algorithms for approximating backprop using only local signals have been proposed. However, these algorithms typically impose other requirements which challenge biological plausibility: for example, requiring complex and precise connectivity schemes, multiple sequential backwards phases with information being stored across phases. Here, we propose a novel algorithm, Activation Relaxation (AR), which is motivated by constructing the backpropagation gradient as the equilibrium point of a dynamical system. Our algorithm converges rapidly and robustly the correct backpropagation gradients, requires only a single type of computational unit, utilises only single parallel backwards relaxation phase, and can operate on arbitrary computation graphs. We illustrate these properties by training deep neural networks on visual classification tasks, and describe simplifications to the algorithm which remove further obstacles to neurobiological implementation (for example, the eright-trassport problem, and the use of nonlinear derivatives), while preserving performance.
In the last decade, deep artificial neural networks trained through the backpropagation of error algorithm (backprop) (Werbos) 1982, Griewank et al. 1989 Linnainmaa 1970) have achieved substantial success on a wide range of difficult tasks such as computer vision and object recognition (Krizhessky et al., 2012 He et al., 2016). language modelling (Vaswani et al.) [017] Radford et al.
2019 Brown et al. 2020), unsupervised representation learning (Radford et al. 2015| Oord et al.
2018), image and audio generation (Goodfellow et al. 2014 Salimans et al. [0017 Jing et al. 2019
Oord et al. 2016 Dhariwal et al. (020) and reinforcement learning (Silver et al.) 2017 Mnih et al. [015||Schulmg
et al. 2017] Schrittiieser et al. (019). The impressive performance of backprop is due to the fact that it precisely computes the sensitivity of each parameter to the output (Lillierap et al. 2020), thus solving the credit assignment problem which is the task of determining the individual contribution of each parameter (potentially one of billions in deep neural network) to the global outcome. Given the correct credit assignments, network parameters can be straightforwardly, and independently, updated in the direction which maximally reduces the global loss. The brain also faces formidable credit assignment problem - it must adjust trillions of synaptic weights, which may be physically and temporally distant from their global output, in order to improve performance on downstream tasks] [ Given that backprop provides successful solution to this problem (Baldi & Sadowski 2016), a large body of work has investigated whether synaptic plasticity in the brain could be interpreted as implementing
approximating backprop (Whitiigton & Bogacz 2019: Lillicrap ct al. 2020). Recently, this idea has been buttressed by findings that the representations learnt by backprop align closely with representations extracted from cortical neuroimaging data (Cadieu et al.) 2014 Kriegeskorte, 2015).
Due to the nonlocality of its learning rules, a direct term-for-te comm-forttentt for-term implementation of backprop is is an It is unlikely that the brain optimizes a single cost function, assumed here. However, even if functionally segregated areas can he thought of as optimising some combination of cost functions, the core problem of credit assignment remains.
likely biologically implausible (Crick) 1989). It is important to note that

---

Under review as a conference paper at ICLR 2021
amorphous term in the literature, with many possible readings. Here, specifically mean that the algorithm requires only local information to be present at each synapse, information is processed according to straightforward linear, Hebbian-like like update rules. It is important to note that here (as in much of the literature), we use the abstraction of rate-coded leaky-integrate-and- extension to more neurophysiologicall grounded spiking models is an exciting area of future work. In recent years, there has been a substamtial amount of work developing more biologically plausible approximations that rely solely on local information. (Whittington & Bogacz 2017: Bengio & Fischer
neurons. The
2015; Bengio et al., 2017, / Scellier et al. 2018a b. Sacramento et al. 2018
Guerguiev et al.. 2017 7
Lee et al. 2015 Lillicrap et al. 2014 Nokland 2016; Millidee et al.
2020a
Ororbia II et al. 2017
Ororbia & Mali 2019 2019/ Ororbia et al.) 2020). The local learning constraint requires that the plasticity
at each synapse depend only on information which (in-principle) locally available at that synapse, such as pre- and post- synaptic activity in addition to local point derivatives and potentially the weight of the synaptic connection itself. The recently proposed NGRAD hypothesis (Lillicrap et al., 2020) offers unifying view many of these algorithms by arguing that they all approximate backprop in a similar manner. Specifically, it suggests that all of these algorithms approximate backprop by implicitly representing the gradients in terms of neural activity differences, either spatially between
functionally distinct populations of neurons or neuron compartments (Whittington & Bogacz 2017]
Millidge et al. 2020a), temporally between different phases of network operation (Scellier & Bengio 2017 Scellier et al. 2018b).
Another way to understand local approximations to backprop comes from the notion of the learning channel (Baldi & Sadowski] 2016). In short, as the optimal parameters must depend on the global outcomes or targets, any successful learning rule must propagate information backwards from the targets to each individual parameter in the network that contributed to the outcome. We argue that there are two primary ways to achieve " First, a sequential backwards pass could be used, where higher layers propagate information about the targets backwards in layer-wise fashion, such that the gradients of each layer are simply function of the layer above. This is the approach employed by backprop, which propagates error derivatives explicitly. Other algorithms such as target-pron peoeetpropaggtion (Lee et al., 2015) also perform a backwards pass using only local information, but do not asymptotically approximate the backprop updates. Secondly, instead of a sequential backwards pass, information could be propagated through a dynamical relaxation underwritten by recurrent dynamics, such that information about the targets slowly 'leaks" backwards through the network over the course of multiple dynamical iterations. Examples of such dynamical algorithms include predictive coding (Whittington & Bogacz, 2017 Friston| 2003 2005; Millidge 2019) and contrastive Hebbian methods such as equilibrium prop (Xie & Seung 20033] Scellier & Bengio 2017), which both asymptotically approximate the error gradients of backprop using only local learning rules.
To our knowledge, all of the algorithms in the literature, including those mentioned above, have utilised implicit or explicit activity differences to represent the necessary derivatives, in line with the NGRAD hypothesis. Here, we derive an algorithm which converges to the exact backprop gradients without utilising any layer-wise notion of activity differences. Our algorithm, which we call Activation Relaxation (AR), is derived by postulating dynamical relaxation phase in which the neural activities trace out dynamical system which is designed to converge to the backprop extremely simple, and require only local information, meaning
gradients. The resulting dynamics that they could, in principle be implemented in neural circuitry. Unlike algorithms such as predictive coding, AR only utilises a single type of neuron (instead of two separate populations one encoding values and one encoding errors), and unlike contrastive Hebbian methods it does not require multiple distinct backwards phases (only a feedforward sweep and a relaxation phase). This simplification is beneficial from the standpoint of neural realism, as in the case of predictive coding, there is little evidence for the presence of specialised prediction-erse error neurons throughout cortex (Walsh et al., 2020) D. Unlike contrastive Hebbian methods, AR does not require the coordination and storage of information across multiple backwards phases, which would pose a substantial challenge for decentralised neural circuitry.
'A third method is to propagate information about the targets through global neuromodulatory signal which affects all neurons equally. However because this does not provide precise vector feedhack, the implicit gradients computed have extremely high variance, typically leading to slow and unstable learning (Lillicrap et al. 2020).
There is, of course, substantial evidence for dopaminergic reward-prediction error neurons in midhrain areas
Baaer& & Glimcher 20055 Glimcher 2011 Schuliz 1998 Schultz. & Dickinson
2000
2

---

Under review as a conference paper at ICLR 2021
We empirically demonstrate that the AR algorithm accurately approximates the backprop gradients and can be used to successfully train deep neural networks on the MNIST and FashionMNIST tasks, where demonstrate performance directly equivalent to backprop. Finally, show that several of the remaining biologically implausible aspects of the algorithm, such as "weight-transport (Crick) 1989), and the evaluation of nonlinear derivatives, can be removed, resulting in an updated form of AR that requires extremely simple connectivity patterns and plasticity rules. Importantly, we show that this simplified algorithm can still train deep neural networks to high levels of performance, despite no longer approximating exact backprop gradients.
METHODS
We consider the simple case of a fully-connected deep multi-laver perceptron (MLP) composed of layers of rate-coded neurons trained in supervised setting" The firing rates of these neurons represented
single scalar value 24. referred to the neurons activation, and vector of all activations at given layer is denoted as z². The activation's 's of the hierarchically superordinate layer are a function of the hierarchically subordinate layer's ' activations r/+1 fW"''', where W e θ is the set of synaptic weights, and the product of activation and weights is transformed through nonliner activation function f. The final output x² of the network is compared with the desired targets T, according to some loss function L(x²² T). In this work, we take this loss function to be the mean-squared-erroo (MSE) ((x*, other loss function without loss of generality (see Appendix B). We denote the gradient of the loss d2
1 (xg
T.)². although the algorithm applies to any
with respect to the output layer dzt. In the case of the MSE loss, the gradient of the output layer is just the prediction error e² =xx² 1---10 -T) Backprop proceeds by computing the gradient of the loss function with respect to the weights W using the chain rule,
aL ØL Oz
Iz1+1 OWT OW OL 0zl+1 ('(W (W''")a""
where f' denotes the derivative of the activation function. The difficulty lies in computing the OL
BatFT term using only local information. This is achieved by repeatedly applying the chain rule AL ar 0z'+1 Bet в. Hrt allowing the derivatives to be computed recursively from the output loss. However, this procedure is not local since the update at each layer depends on the gradients of all superordinate layers in the hierarchy. Here, we propose a method for computing the activation derivatives, dul using a dynamical systems approach. Specifically, we define dynamical system where the activations of each node at equilibrium correspond to the backpropagated gradients. Note that this is in contrast with more typical approaches where a dynamical system is designed to converge to the minimum of some global loss. The simplest system to achieve this is -ntte-iitterrttor driven by top-down feedback,
OL
dxl - +
0L Ox!
(2)
dt
which, at equilibrium, converges
dx! 0 x² aL
dt
Ox
Furthermore, by the chain rule write Equation 2 as,
dxl - + 0zl+1 Or
aL 0x'-1
dt
where z' the value of x' computed in the forward pass. We can express this in terms of the equilibrium activation of the superordinate layer.
dx! -
0rl+1
(3)
dt
dz
'Extensions to other architectures are relatively straightforward and will be investigated in future work. In appendix A we show that the approach can he extended to arbitrary directed acyelic graphs (DAGs), which encompasses all standard machine learning architectures such as CNNs, LSSMss, ResNets, transformers, etc.
3

---

Under review as a conference paper at ICLR 2021
To achieve these dynamics exactly in multilayered network would require the sequential convergence of layers, as each layer must converge to equilibrium before the dynamics of the layer below can operate. However, enable updates across multiple layers simultaneously, we approximate the equilibrium activation of the layer above with the layer's current activation, yielding.
de'
= x' xx +1+ ax!+1
dt
Ox ri= ~ s¹ + 1+1 0x!-1
(4)
dx
st==x
2 1+ f(WW, , W
(5)
Despite this approximation, we argue that the system nevertheless converges to the same optimum as Equation 3. Specifically, because evaluate dr
at the feedforward pass value s this term remains constant throughout the relaxation phase Keeping this term fixed effectively decouples the each layer from any bottom-up influence. If the top-down input is also constant, because it has already converged so that x/+1 ~ x'+1", then the dynamics become linear, and the system is globally stable due to possessing a Jacobian which is everywhere negative-defiitt. The top-layer is provided with the stipulatively correct gradient, so it must converge. Recursing backwards through each layer, we see that once the top-level has converged, so too must the penultimate layer, and so through to all layers. Although this argument is somewhat heuristic, in section 2 provide empirical results showing that it rapidly and robustly converges to the exact numerical gradients in practice.
2x!
(AR) algorithm. The algorithm proceeds as follows. First, standard forward pass computes the network output, which is compared with the target to calculate the top-layer error derivative é and thus update the activation of the penultimate layer. [6] Then, the network enters into a relaxation phase where Equation 4 is iterated globally for all layers until convergence for each layer. Upon convergence, the activations of each layer are precisely equal the backpropagated derivatives, and are used to update the weights (via Equation 1).
Equation 4 forms the backbone of the
1.1 RELATED WORK
There has been a substantial amount of work addressing the challenge of developing biologically plausible implementations of, or approximations to, backprop, with numerous schemes now available in the literature. Many attempts have been made address or work around the weight transport problem the requirement that backwards information be conveyed by the transpose of the forward weights - by either simply using random backwards weights (Lillicrap et al., 2016), directly transmit- ting gradients backwards to all layers from the output layer (Nokland 2016). simply learning the
backwards weights themselves (Amit 2019: Akrout et al. 2019). In addition, recurrent algorithms have been developed that can converge to representations of the backprop gradients with only local rules in an iterative fashion. These algorithms include predictive coding (Whittington & Bogacz, 2017| Millidge, 2019 Millidge et al. [220aa] Friston 2005). where gradients are implicitly computed by minimizing layerwise prediction errors, equilibrium-prog (Scellier & Bengio 2017] Scellier al. 2018b), which uses constrastive Hebbian learning approach where gradients are computed through the differences between free and a fixed phase, and target- prggetpropaggtion (Lee et al., 20115), in which layers are optimized to minimize a layerwise target, and the recal-representation- wewarrrreeeetttiinniiignneeeeee aaliniimiinnnnnnnntt
(LRA) family of algorithms (Ororbia II et al. 2017 Ororbia et al.) 2018 Ororbia & Mali 2019)
which is similar to target-prop except that the targets computes induce the local layer-wise target minimization to encourage each layer to produce representations which will aid the layer above.
AR differs from equilibrium-pre (EP) in that it only requires one iterative phase (the relaxation phase), and the values of the neurons in the activation phase represent the gradients instead of the difference between the activities in each phase, as in EP. Similarly, AR differs from predictive coding in that it does not require separate population of error units. but instead represents the gradients
'The need to keep this term tixed throughout the relaxation phase does present potential issue of biological plausibility. In theory could be maintained by short-term synaptic traces, and for some activation functions such rectified linear units it trivial. Moreover, later we show that this term be dropped from the equations without apparent ill-effect
Thii lay-layer error is simply prediction error for the MSE loss, but may be more complicated and less biologically- plaussible for arbitrary loss functions
4

---

Under review as a conference paper at ICLR 2021
Algorithm I: Activation Relaxation
Data: Dataset D = {X, T), parameters è = {Wo) W inference learning rate weight
learning rate Tjo-
/* Iterate over dataset
for (x", E D) do
Initial feedforward sweep
for (a', W each layer do
x/11= = xx)
+ Begin backwards relaxation
x1
while not converged do
/ Compute final output error
x/
et =T=
dot - -xx + HcL dr
for x¹, ",,'l''-1pp " for each layer do
x Activation update
x1
dx' =-===o'0 + (+ dx
glttl xx nzde'
/ * Update weights at equi equilibrium
for W E (Wo ...... do
With W + n0xe ax
SWT
directly in the activities of the neurons during the relaxation phase, rather than through prediction errors. The iterative form of LRA proposed in (Ororbia et al.] 2018) is perhaps most similar to the AR algorithm, but significant differences still remain. AR derived straightforwardly from dynamical systems perspective on approximating the backprop gradient, while LRA is based on variant of pngec-propaagttio. More importantly. AR directly optimizes the post-activations using the top-down information in the relaxation phase whereas iterative LRA optimizes the pre-activations before they are passed through the nonlinear activation function against the discrepancy between target and neural output. Due to this nonlinearity, the aurget-iscrepanm does not exactly correspond to the backpropagated gradient in LRA, and hence the overall updates do not converge to backprop.
2
RESULTS
Layerwise Divergences from True Gradient
Learning Rate Comparison
MSE between predicted and troe greatent
2500
layer
3000
Leaming Rate 0.5
Layer 2
Leaming Rate 0.2
2000
Layer 3
Leaming Rate 0.1
Leaming Rate 0.05
Leaming Rate 0.02
2000
Leaming Rate 0.01
1500
Loaming Rute 0.005
1500
1000
1000
0
20
100
100 Iteration
125
150
175
200
Iteration
Figure 1: Mean square enror between AR and the exact backpropagated gradients on a 3 layer MLP. Left: convergence each layer. Right: the effect the learning rate on the rate of convergence.
We first demonstrate that our algorithm can train deep neural network with equal performance to backprop. For training, we utilised the MNIST and Rashion-MNISS (Xiao et al. (017)) datasets. The MNIST dataset consists of 60000 training and 10000 test 28x28 images of handwritten digits, while the Fashion-MNIS: dataset consists of 60000 training and 10000 test 28x28 images of clothing items.
The FaShion-MNIST dataset is designed to be identical in shape and size MNIST while being
5

---

Under review as a conference paper at ICLR 2021
hackprop
Parkppop
Angle
Seagpn
357
Gradien Angle
052
204
045
045
10000
Der
Teration
Steattome
(a) MNIST train accuracy
(b) MNIST test accuracy
(c) MNIST gradient angle
Backprop accuracies
vs Backprap accuracies
Gradient Angle
Gadder Anale
0479
322
08.8
20
13
20
20
40
30
12000
20000
J00C
1000
(d) Fashion train accuracy
(e) Fashion test accuracy
(1) Fashion gradient angle
Figure 2: Train and test accuracy and gradient angle (cosine similarity) for AR vs backprop for MNIST and Mashion-MNIST datasets.
harder solve. We used 4-layer fully-connected multi-layer perceptron (MLP) with rectified-linear activation functions and a linear output layer. The layers consisted of 300, 300, 100, and 10 neurons respectively. In the dynamical relaxation phase, we integrate Equation 5 with a simple first-order
de' Euler integration scheme. -!²+1 x¹ The dr where ya was a learning rate which was set to 0.1. The relaxation phase lasted for 100 iterations, which found sufficient closely approximate the numerical backprop gradients. After the relaxation phase was complete, the weights were updated using the standard stochastic gradient descent opteimizer, with a learning rate of 0.0005. The weights were initialized as draws from a Gaussian distribution with mean of 0 and a variance of 0.05. Hyperparameter values were chosen based off initial intuition and were not found using grid-search. A full table of hyperparameter values is presented in Apppendix C. The AR algorithm was applied to each minibatch of 64 digits sequentially. The network was trained with the mean-squared-error loss.
Figure shows that during the relaxation phase the activations converge precisely to the gradients obtained by backpropagating backwards through the computational graph of the MLP. In Figure is
show that the training and test performance of the network trained with activation-rellxatti nearly identical to that of the network trained with backpropagation, thus demonstrating that our algorithm can correctly perform credit assignment in deep neural networks with only local learning rules. We also empirically investigate the angle between the AR-computed gradient updates and the true backpropagated updates. The gradient angle A was computed using the cosine similarity v3V;; metric A(Vo) = cos | where Vo was the AR-computed gradients and Vi were the backprop gradients. To handle the fact that we had gradient matrices while the cosine similarity metric only applies to vectors, following (Lillicrap et al.) 2016), we simply flattened the gradient matrices into vectors before performing the computation. We see that the updates computed by AR are very close in angle to the backprop updates (within under 10 degrees), although the angle increases slightly over the course of training. The convergence in training and test accuracies between the AR and backprop shows that this slight difference in gradient angle is not enough to impede effective credit assignment and learning in AR. In Appendix C, we take a step towards demonstrating the scalability of this algorithm, by showing preliminary results that indicate that AR, including with the biologically plausible simplifications introduced below, can scale to deeper CNN architectures and more challenging classification tasks.

---

Under review as a conference paper at ICLR 2021
3 REMOVING CONSTRAINTS
3.1 WEIGHT TRANSPORT
Although the AR algorithm only utilises local learning rules to approximate backpropagation. there are still some remaining biological implausibilities in the algorithm. Following (Millidge et al. 2020c), we show how simple modifications to the algorithm can remove these implausible constraints on the algorithm while retaining high performance, The most pressing difficulty is the weight- transport problem (Crick, 1989). which concerns the 0" term in Equation 5. In effect, the update rule for the activations during the relaxation phase consists of the activity of the layer above propagated backwards through the transpose of the forwards weights. However, in a neural circuit this would require either being able to transmit information both forwards and backwards symmetrically across synapses, which has generally been held to be implausible, or else that the brain must maintain an identical copy of 'backwards weights' which kept in sync with the forwards weights during learning. However, in recent years, there has been much work showing that the precise equality of forward and backwards weights that underlies the weight transport problem is simply not required. Surprisingly, Lillicrap et al. (2016) showed that fixed random feedback weights suffice for learning in simple MLP networks, as the forward weights learn align with the fixed backwards weights to convey useful feedback signals. Later work (Amit) 2019; Akrout et al., 2019: Ororbia & Mali 2019) has shown that it is also possible and more effective to learn the backwards weights from a random initialisation, where learning can take place with local and Hebbian learning rule. Feedback alignment replaces the 8² in Equation with fixed random backwards weights th The updated Equation 5 reads,
dæ' =z'- +1 ( (' dt
The (initially random) backwards weights can also be trained with the local and Hebbian learning rule which is simple Hebbian update between the activations of the layer and the layer above.
del
dt = x'1
f'(W"
(6)
The backwards weights were initialized as draws from a O mean, 0.05 variance Gaussian. In Figure 3.a) we show that strong performance is obtained with the learnt backwards weights. We found that using random feedback weights without learning (i.e. feedback alignment), typically converged to a lower accuracy and had a tendency to diverge, which may be due to a simple Gaussian weight initialization used here. Nevertheless, when the backwards weights are learnt, we find that the algorithm is stable and can obtain performance comparable with using the exact weight transposes. This approach of using learnable backwards weights to solve the weight transport problem has been similarly investigated in (Amit 2019] AAkrout et al.] 2019),. however these papers simply implement backprop with the learnt backwards weights. Here we show that learnable backwards weights can be combined with fundamentally local learning rule, while maintaining training performance. In Figure 4.a) we plot the angle between the AR with learnable backwards weights and the true BP gradients. The angle starts out very large (about 70 degrees) since the backwards weights are randomly initialized but then rapidly decreases about degrees as the backwards weights are learnt, which seems empirically to be sufficient to enable strong learning performance. 3.2
NONLINEARITY DERIVATIVES
The second potential biological implausibility in the algorithm is the requirement to multiply the weight and activation updates with the derivative of the activation function. While in theory this only requires derivative to be calculated locally for each neuron, whether biological neurons could compute this derivative is an open question. We experiment with simply removing the nonlinear derivatives in question from the update so that the updated Equation now simply reads,
dr' = x'- -
dt
Although the gradients are no longer match backprop, we show in Figure 3.b) that learning perfor- mance against the standard model is relatively unaffected, showing that the influence of the nonlinear derivative is small. We hypothesise that by removing the nonlinear derivative, we are effectively projecting the backprop update onto the closest linear subspace, which is still sufficiently close in
7

---

Under review as a conference paper at ICLR 2021
boaands Weights
Algeritim
244
Ber atoon
Beratorn
Perattona
(a) MNIST backwards weights
(b) MNIST nonlinear derivatives
(c) MNIST combined
Backwards Weights accuracies
Nonlinear Dertrative
Combined Algor tast accuracios
325
Salymme
ORR
323
080
12
20
20
30
(d) Fashion backwards weights
(e) Fashion nonlinear derivatives
(1) Fashion combined
Figure 3: Train and test accuracy and gradient angle (cosine similarity) for AR vs backprop for MNIST and Reshion-MNIST datasets.
Rackoards Wiightt Angle
Dirivative Angle
Combined Algorithmm Gradiert Angin
Grdden Angle
Gradenn Angle
Barrat bereborn
Ber
Sterattoss
(a) Backwards weight angle
(b) Nonlincar derivatives angle
(c) Combined angls
Figure 4: Angle between the AR and backprop updates in the learnable backwards weights, no nonlinear derivatives, and the combined conditions.
angle to the true gradient that it can support learning. By explicitly plotting the angle (Figure 4.b), we see that it always remains under about 30 degrees, sufficient for learning, although the angle appears to rise over the course of training, potentially due to the gradients becoming smaller and more noisy as the network gets closer to convergence.
combine these two changes of the algorithm such that there is both no nonlinear derivative and also learnable backwards weights. Perhaps surprisingly, when we do this we retain equivalent performance to the full AR algorithm (see Figure 3.c), and therefore valid approximation to backprop in an extremely simple and biologically plausible form. The activation update equation for the fully simplified algorithm is: da' =
Moreover,
(
dt which requires only locally available information and is mathematically very simple. In effect, each layer is only updated using its own activations and the activations of the layer above mapped backwards through the feedback connections, which are themselves learned through local and Hebbian learning rule. This rule maintains high training performance and a close angle between its updates and the true backprop updates (Figure 4.c). and is, at least in theory, relatively straightforward to implement in neural or neuromorphic circuitry.
DISCUSSION
We have shown that by taking a relatively novel perspective on the problem of approximating backprop through recurrent dynamics by explicitly designing dynamical system converge on
8

---

Under review as a conference paper at ICLR 2021
the gradients wish to approximate one can straightforwardly derive from first principles an extremely simple algorithm which asymptotically approximates backprop using only local learning rules. Our algorithm requires only a feedforward pass and then a dynamical relaxation phase which we show empirically converges quickly and robustly to the exact numerical gradients computed by backprop. We demonstrate that our algorithm can be used train deep ML.Ps to obtain identical performance to backprop. We then show that two key remaining biological implausibilities of the algorithm the weight transport problem, and the need for nonlinear derivatives can be removed without apparent harm to performance. The weight transport problem can seemingly be solved by postulating a second set of independent backwards weights which can be trained in parallel with the forward weights with Hebbian update rule. While we have found that the nonlinear derivatives can be simply dropped from the learning rules, and does not appear to significantly harm learning performance. Future work should test whether performance is maintained on more challenging tasks. When these adjustments to the algorithm are combined, we obtain the simple and elegant update rule shown in Equation 7.
The AR algorithm does away with much of the complexity of competing schemes. Unlike contrastive Hebbian approaches, it does not require the storage of information across distinct backwards dynami- cal phases such as the 'free phase' and the "clamped phase'. Unlike predictive coding approaches (Whittington& & Bogacz 2017), error-dendrite methods (Sacramento al., 2018), or "ghost units' (Mesnard et al. 2019). AR does not require multiple distinct neural populations with separate update rules and dynamics. Instead, the connectivity scheme in AR is identical to that of standard MLP where only a single type of neuron is required. Since it can approximate backprop without any implicit or explicit representation of activity differences across time or space, the AR algorithm thus provides counterexample to the NGRAD hypothesis, suggesting a more nuanced definition required. Although the gradients themselves are not represented through an activity or temporal difference, the update rules (Equation 5 and 7) can be interpreted as a prediction error between the current activity and the activity of the layer above mapped through the backwards weights. This rule is strongly reminiscent of the target-prop updates, suggesting some points of commonality which would be interesting to investigate in future work. However, the NGRAD hypothesis as stated in (iilierap al. 2020) requires that gradients be encoded directly in spatial temporal differences. Our method shows that this is not necessary and that instead inter-layer errors can be used to drive dynamics which converge to the correct gradients.
Although the AR algorithm (especially the simplified version) takes a strong step towards biological realism, it still possesses several weaknesses which may render naive implementation in neural circuitry challenging. The primary difficulty is the necessity of two distinct phases a feedforward phase which sets the activations directly by bottom-up connectivity, and dynamical relaxation phase where the activations are then adjusted using local learning rules. In effect, AR uses the 'activation' units for two distinct purposes - to represent both the activations and their gradients at different times. While this is not a problem in the current paradigm where i.i.d items are presented sequentially to the network, for a biological brain enmeshed in continuous-time sensory exchange, the requirement for a dynamical relaxation adjusting the firing rates of all neurons, before the next sensory stimulus can be processed would be a serious challenge. By reusing the activations for two different tasks, the brain must either rigidly stick to two separate phases an 'inference' phase and a 'learning phase", or else must be able to multiplex the different phases together to perform the correct updates. This difficult is compounded the weight updates require the presence of both the gradient and the original activation simultaneously, thus necessitating the storage of the original activation during the relaxation phase. One potential solution the multiplexing problem and the related problem of storing the original activations is through the use of multicompartment models of neurons with segregated dendrites. A number of algorithms have approximated backprop by using apical dendrites to keep separate representations of the activity and the error (Sacramento et al.) 20188] Urbanczik & Senn| (2014). The apical dendrites of pyramidal neurons have many useful properties for this role they receive a substantial amount of top-down feedback from other cortical and thalamic areas (Ohno) et al, 2012). are electronically distant from the soma (Larkum) 2013), and they can operate as a 'third factor' in synaptic plasticity through NMDA spiking (Körding & König 2001). Another potential solution to the multiplexing problem is that different phases could be coordinated by oscillatory rhythms of activity such as the alpha gamma bands (Buzsaki 2006). These oscillations could potentially multiplex the different phases together while handling continuously varying stimuli.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. In Advances Neural Information Processing Systems, pp. 974-982, 2019.
Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in compula- tional neuroscience, 13:18, 2019.
Pierre Baldi and Peter Sadowski. theory of local learning, the learning channel, and the optimality of backpropagation. Neural Networks, 83:51-74, 2016.
Hannah M Bayer and Paul W Glimcher. Midbrain dopamine neurons encode quantitative reward prediction error signal. Neuron, 2791):129-141, 2005.
Yoshua Bengio and Asja Fischer. Early inference in energy-based models approximates back- propagation. arXiv preprint arXiv:1510.02777,, 2015.
Yoshua Bengio, Thomas Mesnard, Asja Fischer, Saizheng Zhang, and Yuhuai Wu. Stdp-compatible approximation of backpropagation in an energy-based model. Neural computation, 29(3):555-577, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 140:: 2020.
Gyorgy Buzsaki. Rhythms of the Brain. Oxford University Press, 2006.
Charles F Cadieu, Ha Hong, Daniel LK Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib Majaj, and James J DiCarlo. Deep neural networks rival the representation of primate cortex for core visual object recognition. PLoS Comput Biol, 10(12)ee003363, 2014.
Francis Crick. The recent excitement about neural networks. Nature, (3766223) 1989.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.
Karl Friston. Learning and inference in the brain. Neural Networks, 16(9):1325-1333 2003.
Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences, 360(1 (60((455))815-836 2005.
Paul W Glimcher.
Understanding dopamine and reinforcement learning: the dopamine reward prediction eтror hypothesis. Proceedings of the National Academy of Sciences, 108(Supplement 3): 15647-15654, 2011.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Andreas Griewank et al. On automatic differentiation. Mathematical Programming: recent develop- ments and applications, 6(6):83-107, 1989.
Jordan Guerguiev, Timothy Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. Elife, 6:e22901, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE transactions on visualization and computer graphics, 2019.
10

---

Under review as a conference paper at ICLR 2021
Konrad P Körding and Peter König. Supervised and unsupervised learning with two sites of synaptic integration. Journal of computational neuroscience, 11(3):207-215, 2001.
Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446 2015.
Alex Krizhevsky, 1lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Matthew Larkum. A cellular mechanism for cortical associations: an organizing principle for the cerebral cortex. Trends neurosciences, 36(3):141-15,, 2013.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference on machine learning and knowledge discovery in databases, pp. 195. Springer, 2015.
Timothy P Lillicrap, Daniel Cownden, Douglas Tweed, and Colin Akerman. Random feedback weights support learning in deep neural networks. arXiv preprint arXiv:/411.00 2014
Timothy P LLlliccrp, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 1-10, 2016.
Timothy P Lillicrap. Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, pp. 1-12, 2020.
Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm as a taylor expansion of the local rounding errors. Master's 's Thesis (in Finmish), Univ. Helsinki, pp. 6-7, 1970.
Charles C Margossian. A review of automatic differentiation and its efficient implementation. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 944)ee3305 2019.
Thomas Mesnard, Gaëtan Vignoud, Joao Sacramento, Walter Senn, and Yoshua Bengio. Ghost units yield biologically plausible backprop in deep neural networks. arXiv preprint 108555191. 2019.
Beren Millidge. Implementing predictive processing and active inference: Preliminary steps and results. 2019.
Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Predictive coding approximates backprop along arbitrary computation graphs. arXiv preprint arXiv:2006. 04166.048112... 2020a.
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher I. Buckley. Investigating the scalability and biological plausibility of the activation relaxation algorithm. arXiv preprint arXiv:2010.06219, 2020b.
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher L Buckley. Relaxing the constraints predictive coding models. arXiv preprint arXiv:2010.01047, 2020c.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533333 2015.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in neural information processing systems, pp. 1037-1045, 2016.
Sachi Ohno, Eriko Kuramoto, Takahiro Furuta, Hiroyuki Hioki, Yasuhiro Tanaka, Fumino Fujiyama, Takahiro Sonomura, Masanori Uemura, Kazuma Sugiyama. and Takeshi Kancko. A morphological analysis thalamocortical axon fibers of rat posterior thalamic nuclei: a single neuron tracing 2012.
study with viral vectors. Cerebral cortex,
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: arXiv:1609.000 1609.03499 2016.

---

Under review as a conference paper at ICLR 2021
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 187.03388 2018.
Alexander Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Continual learning of recurrent neural networks by locally aligning distributed representations. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Alexander G Ororbia and Ankur Mali. Biologically motivated algorithms for propagating local target representations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4651-4658, 2019.
Alexander G Ororbia, Ankur Mali, Daniel Kifer, and Lee Giles. Conducting credit assignment by aligning local representations. arXiv preprint arXiv: 01834, 2018.
Alexander G Ororbia II. Patrick Haffner, David Reitter, and C Lee Giles. Learning to adapt by minimizing discrepancy. arXiv preprint arXiv: 171iv:/711./1522 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0644 2015.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188)9, 2019.
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pp. 8721-8732, 2018.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 2017.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
Benjamin Scellier, Anirudh Goyal, Jonathan Binas, Thomas Mesnard, and Yoshua Bengio. Extending the framework equilibrium propagation to general dynamics. 2018a.
Benjamin Scellier, Anirudh Goyal, Jonathan Binas, Thomas Mesnard, and Yoshua Bengio. Gener- alization of equilibrium propagation to vector field dynamics. arXiv preprint arXiv: arXiv:/808.000 1808.04873, 2018b.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. arXiv preprint arXiv:1911.08266 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: Tttiv:170..000 17070063777 2017.
Wolfram Schultz. Predictive reward signal of dopamine neurons. Journal of neurophysiology, 80(1):
1-27, 1998.
Wolfram Schultz and Anthony Dickinson. Neuronal coding of prediction errors. Annual review of neuroscience, 23(1):473-500 2000.
David Silver, Julian Schritwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of without human knowledge. Nature, 550(7676):354-39) 2017.
Robert Urbanezik and Walter Senn. Learning by the dendritic prediction of somatic spiking. Neuron, 81 81(3):521-522 2014.
Bart Van Merriënboer, Olivier Breuleux, Amaud Bergeron, and Pascal Lamblin. Automatic differen- tiation in ml: Where we are and where we should be going. Advances in neural information processing systems, pp. 8757-8767, 2018.
12

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Kevin S Walsh, David P McGovern Andy Clark, and Redmond G O'Connell. Evaluating the neurophysiological evidence for predictive processing as model of perception. Annals of the New York Academy of Sciences, 1464(1):242, 2020.
Paul Werbos. Applications of advances nonlinear sensitivity analysis. System modeling and optimization. pp. 762-770. Springer, 1982.
James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5):
1229-1262, 2017.
James CR Whittington and Rafal Bogacz. Theories of error proppaaaaantionn in the brain. Trends in cognitive sciences, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. 2017.
Xiaohui Xie and H Sebastian Seung. Equivalence of backpropagation and contrastive hebbian
learning in layered network. Neural computation, 15(2):441-4535 2003.
APPENDIX A: ENERGY FUNCTION, EXTENSION TO ARBITRARY DAGS.
In this appendix, we elaborate on the mathematical background of the AR algorithm. We present a.) candidate energy function that the dynamics appear to optimize, and discuss its limitations and b.) An extension of the AR algorithm to arbitrary computation graphs, which allow, in theory, for the AR algorithm to used to perform automatic differentiation and optimisation along arbitrary programs, including modern machine learning methods.
THE ENERGY FUNCTION
We first define the implicit energy function that the dynamics can be thought of as optimizing. Given hierarchical MLP structure with activations xt, and activation function f and parameters 0; at each layer. This energy function is:
E Σ 1=0
r sWW
(9)
2
Given this energy, function we derive the dynamics gradient descent E
dx!
00 Ozl
Of(W", dr = 111 Of(W! x')
-x' + f(W' 20)
(10)
dl
(11)
dx
where the second step uses the fact that in the forward pass x++1 = = (WW],-' I²). While this energy function derive the required dynamics, it possesses several limitations. At the initial forward pass, the energy is O for all layers except potentially the output layer, and moreover it does not define standard loss function (such as the mean-squared error) at the output. Thus the AR algorithm requires special case to deal with the final output layer, as the 'natural' final layer loss is simply T -[W / which is not standard loss. Secondly, the value of the loss or below. For instance, it cannot easily be interpreted as a (squared) prediction error, which has minimum at Minimizing the energy function thus effectively requires trying to push f(W', ,²)) to be greater than x' much possible, rather than trying to equilibrate the two. Finally, the replacement of f(W ;x)) with [+1 is a subtle issue due to the way the AR algorithm treats the activations in two separate ways. During the forward pass, this substitution is valid, however during the backward relaxation, the activations are changed away from their initial forward- pass values, and thus this substitution is not valid here.
bounded above
13

---

Under review as a conference paper at ICLR 2021
EXTENSION TO DAGS
Here consider the extension of the AR scheme described in the paper to more complex archi- tectures. Specifically, we are interested in any function which can be expressed as a computation graph of elementary differentiable functions. This class includes essentially all modern machine learning architectures. A computation graph represents a complex function (such as the forward pass of a complex NN architecture like transformer (Vaswani al. 2017). or an LSTM (Hochreiter & Schmidhuber 1997) as graph of simpler functions Each function corresponds to a vertex of the graph while there is directed edge between two vertices if the parent vertex is an argument to the function represented by the child vertex. Because we only study finite feedforward architectures (and since it is assumed finite we can 'unroll' any recurrent network into a long feedforward graph), we can represent any machine learning architecture as directed acyclic graph (DAG). Automatic Differentiation (AD) techniques, which are the heart of modern machine learning (Griewank) al., 1989 Van Merriënboer et al., (2018) Margossian 2019), can then be used to compute gradients with respect to the parameters of any almost arbitrarily complex architecture automatically. This allows machine-learning practitioners to derive models which encode complex inductive biases about the structure of the problem domain, without having to be manually derive the expressions for the derivatives required train the models. Here, we show that the AR algorithm can also be used to compute these derivatives along arbitrary DAGs, using only local information in the dynamics,and requiring only the knowledge of the inter-layer derivatives.
Core to AD is the multivariate chain-rule of calculus. Given a node x' on a DAG, the derivative with respect some final output of the graph can be computed recursively with the relation
dL Oxt
ÖL dz
Σ
(12)
X (CCh((z) Ozl Or
Where Chi(a") represents all the nodes which are children of x'. In effect, this recursive rule states that the derivative with respect to the loss of a point is equal to the sum of the derivatives coming from all paths from that node the output. We now derive the AR energy function and dynamical update rule on a DAG. Specifically, we have in the forward pass that x' f(W', x²) E Par(x") (where Par(z) (z) denotes the parents of x) so that each activation some function of its parent nodes. We can thus write out the energy function and dynamics, and compute the equilibrium to obtain:
L
1 S(W), .²)) E Par(x'))²
3
=
2
2
i=0
dx
f(W ( E Par(>")) aS(Wi, at) E Par(w!))
dt
Ox
Or
xi xie c'hhhi()
2 Orv Oz
x²=
Σ
at aiEChi(a
dx
x dr
0
(13)
dt
dzi
ai (iEh(((
Crucially, this recursion satisfies the same relationship as the multivariable chain-rule (Equation 11), and thus if the output nodes are equal to the gradient at the top-level such that xt 81. t at equilibrium the correct gradients will be computed at every node in the graph. Thus AR can be converted into a general-purpose AD algorithm which utilises only local information. Neurally, this only requires the dynamics in the relaxation phase to respond to the sum of top-down input, which is very plausible, and thus perhaps suggests that it possible the brain could be utilizing substantially more complex architectures than feedforward MLPs.
7 Although AR can be applied to any DAG, including the unrolled graphs of recurrent networks like LSTMs. we do not claim that this applying AR in this way is biologically plausible, since such approach would require propagating the AR update rules backwards through time.
14

---

Under review as a conference paper at ICLR 2021
APPENDIX B: CROSS-ENTROPY LOss FUNCTION
is important to note that although we use mean- rean--quure-eror (MSE) in our main experiments, AR does not depend on any specific loss function. All that is necessary to use a different loss function clamp the activations of the final layer to the derivative of the chosen loss function during the relaxation phase. With these units clamped correctly, the AR algorithm will be able to converge to the correct backpropagated gradients for any loss function. To demonstrate this we also present results with the multiclass cross-entropy loss function L Σ; T. a/. For these experiments we used the same neural network architecture the main paper except that the activation function at the output layer was a softmax (instead of linear), to ensure that the outputs correspond to valid probabilities, as required for the cross-entropy loss.
AR vs Backprop accuracies
Backwards Weights accuracles
1,000
1.00
0.935
Accurecy
0.323
Pccuresy
0,900
0.855
0.50
0.850
0.75
0.800
heraaions
porattinss
(a) MNOST standard AR crossentropy accuracy
(b) MNIST Backwards weights crossentropy accuracy
Nonlinear Derivative accuracles
Combined Algorithm
1.000
0:35
0.250
Accuracy
Accuraty
0.93
0.000
0.530
0.75
herations
Derations
(c) MNIST no nonlinear derivatives crossentropy
(d) FMNIST combined AR crossentropy accuracy
Figure 5: Accuracy (averaged over seeds) for AR trained with the crossentropy loss function in the standard AR, learnable backwards weights, no-nonlinearities, and combined conditions.
APPENDIX C: SCALABILITY TO CNNs
We present empirical results showing that AR can scale to larger, more complex network architectures and more challenging tasks than MNIST and Fashion-MNIST. Here we apply AR a small CNN architecture, and test for classification accuracy on the harder CIFAR dataset, composed of 10 classes of 32x32x3 images. The input pixel values were normalized to lie between and The CNN comprised two convolutional layers with maxpooling, following by two fully connected layers. The convolutional layers used 32 and 64 convolutional filters while the fully connected layers had 64, 120, and 10 neurons respectively. All layers used the hyperbolic tangent nonlinearity, except the final layer which was linear. The network was trained using the mean- square-crror loss function.
We showcase performance against the backprop baseline of our CNN under the three simplification conditions of our algorithms with learnable backwards weights, with no nonlinear derivatives, and with
with both simplifications combined. The AR CNN performs comparably with backprop these simplifications applied, thus providing preliminary evidence of the scalability the approach. For more information the scalability of this algorithm, please see (Millidge et al. 2020b). In
15

---

Under review as a conference paper at ICLR 2021
general, we believe that investigating the potential scalability of these algorithms to truly challenging architectures and tasks such as imagenet is an exciting area for future research.
other tann Corw Backwards Weights best accurec
Ciler Each FC barkwards weights accuracte
ctter tarh both test accuracies
6.5
03
10:0
(a) Conv backwards weights
(b) FC backwards weights
Both backwards weights
cifar tarins Conv Bacowards Nontir Nontin eartty test aceus
Clar tanhh FC Backwards Nerliwartty test accurae
cifar tanh use all norts nontmwartty tect accuracio
Sxading
Accuracy
02
2400 ociionn
3.30
1006
300G
2003
2030
10:0
2006
400
(d) Conv no nonlinear derivative
(e) FC nonlinear derivative
(1) Both no nonlinear derivative
Figure 6: Performance (test accuracy), averaged over 10 seeds, on CIFARIO demonstrating the scalability of the Icarnable backwards weights and dropping the nonlinear derivatives a CNN architecture, compared to baseline AR without simplifications. Performance is equivalent throughout.
APPENDIX D: HYPERPARAMETER TABLE FOR MAIN EXPERIMENTS
All hyperparameter values were selected after preliminary experimentation but without large- scale grid search. We typically used standard hyperparameter values taken from the deep learning literature on training ANNs, and found that they worked reasonably well with AR. The most important AR-specific hyperparameter is the learning rate 7/s for the relaxation phase. We undertook a hyperparameter sweep for this parameter, as shown in Figure (right), and setrled on a value of 0.1 which provided good balance between rapid convergence and numerical stability.
Hyperparameter
Value
Batch Size
64
0.0005
Relaxation Iterations
100
T
0.1
Weight Initialization Variance
0.05
16


Reviews:
[{"review_id": "sBDKCvUC9oz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Second Review\n\nThanks for taking all my comments seriously.  After carefully reading other reviews and quick modifications introduced by authors, I believe this work is richer and has shown some potential towards building scalable and robust alternative to BP.  Thanks for including angles as suggested, this further supports  hypothesis proposed in this work. I really appreciate results using CNNs and cross entropy loss, therefore I increase my score to 7. It seems that other reviewers do not appreciate that training a network using hebbian like updates and without  BP requires some nontrivial engineering tricks and theoretical considerations which are now well described in this paper (updated manuscript). It is difficult to match BP gradients, and many popular alternatives including FA, DFA, DTP, EP struggle to match BP performance when tested on complex benchmarks (Cifar, imagenet, etc) with complex architectures (CNNs, RNNs, etc). Beside this given approach is robust even when backward weights are trainable. However I agree storing backward and forward synapses challenges the bio-plausibility of approach, which I think can be handled if local representation are handled differently. Nonetheless, changing few strong words (optimal BP, optimal gradients) and derivation, supports major hypothesis proposed in this work. A better justification on non-local updates as raised by other reviewers is required to further strengthen this work. But i liked the results with activity relaxation and how close gradients are with respect to BP. Combining current approach with other bio-inspired approaches might solve some key aspects of current learning algorithm. \nCurrent approach is still heavily dependent on backprop, and partially gets closer to bioplausible approaches (mainly hebbian like update rule). Testing this on deeper architectures (Resnet, student-teacher etc) might further strengthen your work. \n \n## Minor comments\n\nFigure 4 c) change angls--> angles\nIn appendix, change caption of fig 5a) from MNOST to MNIST, I believe it is a typo\n\nAdd results with FA or DFA with fixed and learnable weights, will further support robustness and closeness of BP claim in this work.\n\n## Summary\nIn this work, authors designed a bio-inspired approach known as activation relaxation (AR), utilizing local information for training deep neural network. Unlike prior work on bio-inspired learning, AR only utilizes single neuron to compute its gradient helpful for neural circuitry. AR is seen to be derived from postulating a dynamical relaxation phase in which the neural activities are tracing out a dynamical system. This modification is hypothesis to be close to backprop gradients getting ride of weight transport problem. Authors show accuracy as a metric to validate their hypothesis by conducting experiments on small scale dataset such as mnist and fmnist.\n\n## Review\nMost of the paper is clear, but experiment section needs more work in approving the hypothesis w.r.t AR.\n\n1] AR converges close to backprop gradients.\nAccuracy is not a valid measure to show robustness and calculate approximately with backprop gradients.  As shown by FA (lillicrap) , DFA (nokland), LRA (ororbia), the updates carried by there model roughly lies within 45 degree compared with backprop. It is better to show model update angles w.r.t to BP, DFA and other bio-inspired approaches.\n\n2] is AR robust? What happens to the model at various initialization and how does it behave when tested with various model hyper-parameter changes?\n\n## Experimental section is missing major chunk of information. \n2.1] Did you perform grid search on model hyper-parameters, if so, what are those? Did you experiment with various activation functions and if so, what are those? Are results consistent whenever MSE is replaced with CE? If so report your findings.\n2.2] What happens to the network when BP gradients are weak resulting into poor information, can AR come out of that low saddle point and converge to better local minima? Few experiments related to better convergence are shown by ororbia, they experimented with various init to show their model can converge to better local minima whenever backprop had issue in converging.\n2.3] Did you perform multiple trails on your experiments? If so, you should report standard error in your paper.\n\n3] AR w.r.t fixed error weights vs AR w.r.t learnable error weights.\nHow does local learning approach, help in improving the model plasticity when you have these two scenarios? Paper mentions few lines on those, but detailed experiments should be conducted to validate the statement stable and robust performance. Also provide information on how error weights are initialized, and ranges experimented w.r.t backward or error weights.\n\n4] Comparison with other bio-inspired approaches.\nCurrent manuscript does not show any comparison with other bio-inspired approaches (DFA, FA, DTP, DTP-sigma, LRA, Weight mirroring). If goal is to show model is robust, close to backprop updates and gradients, then it is better to show comparison with these approaches or at least show angle to understand closeness w.r.t BP updates.\n\n5]  Performance on large scale dataset and CNNs.\nIt is been argued that bio-inspired approaches (DFA, FA, DTP) struggle to match backprop performance when evaluated on large scale dataset with deep CNNs[Bartunov 18] . However recently weight mirroring [Akrout 19] and LRA[Ororbia and Mali 19]  have shown that they can come closer to backprop in terms of performance on large scale datasets. But in this current manuscript, there are no results w.r.t CNNs and updates w.r.t. filters when AR is deployed on such challenging visual recognition tasks.\n\n[Bartunov 18] Bartunov, S., Santoro, A., Richards, B., Marris, L., Hinton, G.E. and Lillicrap, T., 2018. Assessing the scalability of biologically-motivated deep learning algorithms and architectures. In Advances in Neural Information Processing Systems (pp. 9368-9378).\n\n[Ororbia 18] Ororbia, A.G., Mali, A., Kifer, D. and Giles, C.L., 2018. Deep credit assignment by aligning local representations. arXiv preprint arXiv:1803.01834.\n\n[Akrout 19] Akrout, M., Wilson, C., Humphreys, P., Lillicrap, T. and Tweed, D.B., 2019. Deep learning without weight transport. In Advances in neural information processing systems (pp. 976-984).\n\n[Ororbia and Mali 20] Ororbia, A., Mali, A., Kifer, D. and Giles, C.L., 2020. Reducing the Computational Burden of Deep Learning with Recursive Local Representation Alignment. arXiv preprint arXiv:2002.03911.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "l66BkndAsPJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors suggest a \"local\" algorithm to replace backpropagation in feedforward neural networks. Locality is meant here by the compatibility with the NGRAD hypothesis: backpropagation through the layers is replaced by the propagation of the network activity itself. Therefore there is no need for a second implicit error network or error neurons.\n\nThis method introduces dynamics in the neuron to converge towards an equilibrium and remark that, at equilibrium, the activity transmitted backward becomes equal to the back-propagated gradients. The method is tested on MNIST and fashion MNIST.\n\n## Critical review\n\nThe methods seems extremely similar to equilibrium propagation and the account of this resemblance does not seem to be acknowledged in a fair manner. Rather the introduction summarized a large spectrum of plausible learning rules which are more or less related to AR and the authors do not comment in a clear way about the technical resemblance with these algorithms.\n\nThe derivation originate from an interesting observation at equilibrium, but this is certainly not as rigorous as the proofs given for equilibrium propagation. Equilibrium propagation has been demonstrated to work on tasks harder than MNIST and fashion MNIST.\n\nThe simulations are not sufficient to show if the method is competitive or practicable. The plots show the learning curves where the test accuracy is spanning values between 92.5 and 98 on MNIST. Given the variance, this is probably computed only on mini batches of the test set and not over the entire test set. Hence, it is not the regular protocol for reporting the test accuracy. A good back-prop implementation is expected to reach 98.2% accuracy on the test set on MNIST. I do not know whether the BP baseline reaches this level in this paper. It vaguely seem that AR is similar to their BP implementation but I do not know what to judge from this and the simulation results and not commenting in details in the text.\n\nI strongly doubt the statement that the method applies to recurrent networks and LSTMs, it may work for any DAG, but in a recurrent network the temporal dynamics are already meant to model the network computation and the changing inputs, therefore the network cannot simply converge to an equilibrium in this case. Also the activity in NGRAD replace BP by transmitting activity backward through the depth but this does not make sense to replace back-propagation though time in recurrent networks since biological neural cannot transmit activity backward in time.\n\nAs a conclusion, I see a few interesting remark in the paper but neither the simulation nor the theory are bringing substantial insights to the plausible learning rule community or the broader representation learning literature.\n\nI did not undertand what is meant by \"the Jacobian is negative-definite\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "dETC7IPWfC3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a biological model for loss function differentiation which reuses a single pool of neurons to compute predictions and prediction errors (i.e., derivatives of a loss function with respect to the neural acitivity variables) in two sequential phases.\n\nI have a major concern that I need to see addressed by the authors:\n- In my understanding the proposed method is not local in time and this is not at all clear, the way the paper is written. This is best seen on the last line of Algorithm 1: the apparently innocuous term $\\frac{\\partial x^l}{\\partial W^l}$ should be evaluated at the _feedforward activation values_, not using the equilibrium $x^l$. I wonder whether the authors are aware of this, as such a detail could be somewhat easily overlooked when using automatic differentiation.\n\nNote that this is very different from dropping $f^\\prime$ (as per footnote 5) from the gradient, something that has been experimented with in some of the feedback alignment implementations in the past (for example, some of the simulations in Lillicrap et al., 2016, do not backpropagate $f^\\prime$, although the local ${f^\\prime}^l$ is still used when updating $W^l$).\n\nStemming from this issue, the advantages of the proposed scheme against standard predictive coding implementations using two populations of neurons (predictions and errors; e.g., Whittington & Bogacz, 2017) aren't clear. As I understand the algorithm, prediction activity is overridden by error activity (in a way that still requires coordination/phases), but since prediction activity is necessary to compute the aforementioned term, the resulting learning rule becomes nonlocal in time.\n\nA proper evaluation of the significance of the advance brought by the method requires clarifying this issue.\n\nI leave some additional comments for the authors:\n- \"Given that backprop provides an optimal solution to this problem (Baldi & Sadowski, 2016)\"\nThe claim that backprop provides an optimal solution to the credit assignment problem is a very strong one, and I'm not sure that it is supported by the cited reference. Optimal gradient approximation is different from optimal credit assignment.\n- \"Unlike contrastive Hebbian methods, AR does not require the coordination and storage of information across multiple backwards phases, which would pose a substantial challenge for decentralised neural circuitry.\"\nWhile AR does not require storage of information across multiple backwards phases, it requires storage of information across forward and backward phases.This is related to my major comment above, and this issue/potential misunderstanding reappears in multiple points of the paper.\n- In (1), why is $x^L$ and not $x^l$ appearing?\n- In the equation below (2), should $x_i$ be $x^l?\n- Figure 3: these curves are rather hard to see, perhaps consider presenting mean +- error bars over many runs? \n- I could not follow several of the steps in the arguments presented in the appendix titled \"The energy function\". How is (11) related to the dynamics of the algorithm?\n- There are a few typos that should be corrected, e.g., \"is designed to converges to the minimum\".\n\n---\nEdit after rebuttal:\n\nI am sorry for not yet being able to support acceptance of this paper. I appreciate the authors' work during the rebuttal, and I agree that the execution of the paper improved in this version.\n\nBut the issue remains that the proposed algorithm does not fundamentally lift any of the current obstacles in implementing backpropagation in the brain. Most importantly, the algorithm is very similar to exact standard backpropagation, and it still requires the same coordinated phases as backpropagation: the forward phase and the backward phase. The learning rule is non-local in time and while it does not use activity differences, that is not necessarily a good thing. For example, the connection to spike-timing-dependent plasticity becomes harder to establish. The authors verify that learning still works after important approximations are made (notably to avoid weight transport), but these approximations were already previously reported and are more or less obviously applicable here, given how close activation relaxation is to standard backpropagation.\n\nFinally, I really recommend the authors to more explicitly state and discuss the temporal non-locality of the algorithm when introducing the method (and in Algorithm 1), which remains insufficiently clear to me.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9mc6EI20nfO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an algorithm for estimating the correct backprop gradients that is described as biologically plausible. I have very little to contribute as I think this is a clearly-written paper — as such I recommend acceptance.\n\nI would be happier if the authors gave a very very strict definition of the kind of plausibility they wish to capture with falsifiability criteria and deep theoretical underpinnings, as I think it's a bit of a semantic trap without reference to some explicit level of analysis, etc. But this is a field-wise terminological issue, and thus can easily be seen as out-of-scope. \n\nMinor: A paper the authors might appreciate is, given some of the work they mention to motivate their neuro/bio plausible claims: \n\nXu, Y., & Vaziri-Pashkam, M. (2020). Limited correspondence in visual representation between the human brain and convolutional neural networks. bioRxiv.\nhttps://doi.org/10.1101/2020.03.12.989376", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an algorithm to perform backprop in a feed-forward neural network without the need to backpropagate errors. They hence claim that this algorithm is a biologically plausible variant of Backprop.

After a forward-propagation phase, the method introduces a relaxation phase and they remark that at the equilibrium of this phase, the activity is equal to the derivatives. Some related algorithms have been proposed previously (predictive coding, equilibrium-prop, target-propagation). Advantages of the proposed algorithm are that it does not need multiple distinct backwards phases and that it only utilizes a single type of neuron instead of separate populiations (such as in predictive coding).

Their method is tested on MNIST and fashion MNIST using a 4-layer fully connected network. In a revised version after the initial reviews, the authors added preliminary results on CIFAR-10 with a 4 layer CNN.

The authors then study the impact of some unbiological constraints such as symmetric weights.

While the reviewers agreed that the work is interesting, there was some disagreement on the significance of the model. In particular, it was noted that while the learning rules are indeed local in space, they are not local in time (the network has to remember variables from the forward phase until the update at the relaxation phase), which was deemed questionable from the biological perspective.

In addition, it was criticized that the simulations are no sufficient to support the claims of the paper. The datasets are relatively simple, networks are shallow and performances of baseline models are not state-of-the-art.
 In a revised version after the initial reviews, the authors added preliminary results on CIFAR-10 with a 4 layer CNN. However, these results do not seem conclusive, as the baselines are far below SOTA and networks are still quite shallow (a study by Lillicrap found that biological approximations to backprop struggle especially when applied to deep networks).

In summary, the work looks promising but some questions remain about the locality of learning and its applicability to more demanding tasks.

I add that one reviewer gave a very good rating with a poor review and did not respond to any questions about the justification. Therefore I had to neglect this review.
Decision: Reject