Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning
Authors: Beren Millidge
Yuhang Song
Tommaso Salvatori
Thomas Lukasiewicz
Rafal Bogacz
Venue: ICLR 2023 poster (2023)
Keywords: predictive coding
equilibrium propagation
contrastive hebbian learning
backprop
machine learning
computational neuroscience

Abstract: How the brain performs credit assignment is a fundamental unsolved problem in neuroscience. Many `biologically plausible' algorithms have been proposed, which compute gradients that approximate those computed by backpropagation (BP), and which operate in ways that more closely satisfy the constraints imposed by neural circuitry. Many such algorithms utilize the framework of energy-based models (EBMs), in which all free variables in the model are optimized to minimize a global energy function. However, in the literature, these algorithms exist in isolation and no unified theory exists linking them together. Here, we provide a comprehensive theory of the conditions under which EBMs can approximate BP, which lets us unify many of the BP approximation results in the literature (namely,  predictive coding, equilibrium propagation, and contrastive Hebbian learning) and demonstrate that their approximation to BP arises from a simple and general mathematical property of EBMs at free-phase equilibrium. This property can then be exploited in different ways with different energy functions, and these specific choices yield a family of BP-approximating algorithms, which both includes the known results in the literature and can be used to derive new ones.

Published conference paper at ICLR 2023
BACKPROPAGATION AT THE INFINITESIMAL INFER- ENCE LIMIT OF ENERGY-BASED MODELS: UNIFYING PREDICTIVE CODING, EQUILIBRIUM PROPAGATION, AND CONTRASTIVE HEBBIAN LEARNING
Beren Millidge
Yuhang Song
MRC Brain Network Dynamics Unit
MRC Brain Network Dynamics Unit University of Oxford, UK yuhang. soong wrangnnddn...m .onddnn.o www.acc.kk
University of Oxford, UK
peren@millinn 1iidge......... name
Tommaso Salvatori
Thomas Lukasiewicz
Institute of Logic and Computation TU Wien, Austria tommaso. sal vat.aattrrieeo @cs. ax.a..c.........................
TU Wien, Austria
University of Oxford, UK thomas
Rafal Bogacz
MRC Brain Network Dynamics Unit University of Oxford, UK rafal bafal.bbgaczenden. .c.uk
ABSTRACT
How the brain performs credit assignment is fundamental unsolved problem neuroscience. Many 'biologically plausible' algorithms have been proposed. which compute gradients that approximate those computed by backpropagation (BP), and which operate in ways that more closely satisfy the constraints imposed by neural circuitry. Many such algorithms utilize the framework of energy-based models (EBMs), in which all free variables in the model optimized to minim- ize global energy function. However, in the literature, these algorithms exist isolation and no unified theory exists linking them together. Here, we provide a comprehensive theory of the conditions under which EBMs can approximate BP, which lets us unify many of the BP approximation results in the literature (namely, predictive coding, equilibrium propagation, and contrastive Hebbian learning) and demonstrate that their approximation to BP arises from a simple and general math- ematical property of EBMs at free-phase equilibrium. This property can then be exploited in different ways with different energy functions, and these specific choices yield family of aP-approxng algorithms, which both includes the known results in the literature and can used derive new ones.
INTRODUCTION
The backpropagation of eиror algorithm (BP) (Rumelhart et al. 1986) has become the workhorse algorithm underlying the recent successes of deep learning (Krizhevsky et al., 2012 Silver al., 2016 Vaswani et al. 2017). However, from neuroscientific perspective, BP offen been criti- cised as not being biologically plausible (Crick et al.) 1989 Stork 1989). Given that the brain faces credit assignment problem at least as challenging as deep neural networks, there is fundemental perform credit assignment. The answer to
question of whether the brain uses backpropagation this question depends on whether there exist biologically plausible algorithms which approximate BP that could be implemented in neural circuitry (Whittington & Bogacz 2019 Lillicrap et al. 2020). A large number of potential algorithms have been proposed in the literature (Lillirrap et al, 2016 Xie & Seung 20033 Nøkland [016 WWittington & Bogacz 2017] et al. |015] Bengio & Fischer 2015 Millidge et al. 22000c at [Song et al. 2020 Ororbia Mali 2019), however insight into linkages and relationships between them is scarce, and thus far the field largely presents
itself as a set of disparate algorithms and ideas without any unifying or fundamental principles.
In this paper we provide a theoretical framework which unifies four disparate schemes for approx- imating BP - predictive coding with weak feedback (Whitington & Bogacz (017) and on the first step after initialization (Song et al. 2020), the Equilibrium Propagation (EP) framework (Scellier
Bengio 2017). and Contrastive Hebbran Learning (CHL) (Xie & Scung 2003). We show that
these algorithms all emerge as special cases of a general mathematical property of the energy based
"Corresponding author.

---

Published as conference paper at ICLR 2023
model framework that underlies them and, such, can be generalized to novel energy functions and to derive novel algorithms that have not yet been described in the literature.
The key insight is that for energy based models (EBMs) underlying these algorithms. the total energy can be decomposed into a component corresponding to the supervised loss function that depends on the output of the network, and a second component that relates to the 'internal energy" of the net- work. Crucially, at the minimum of the internal energy, the dynamics of the neurons point exactly in the direction of the gradient of the supervised loss. Thus, at this point, the network dynamics implicitly provide an exact gradient signal. This fact can be exploited in two ways. Firstly, this instantaneous direction can be directly extracted and used to perform weight updates. This 'first step' approach is taken in (Song et al.] 2020) and results in exact BP using only the intrinsic dy- namics of the EBM, but typically requires complex set of control signals specify when updates should occur. Alternatively, a second equilibrium can be found close the initial one. If it is close enough - a condition we call the infinitssmal inference limit -, then the vector from the initial to the second equilibrium approximates the direction of the initial dynamics and thus the difference in equilibria approximates the BP loss gradient. This fact is then utilized either implicitly (Whittington & Bogacz 2017) or explicitly (Scellier & Bengio 2017 Xie & Seung, 2003) derive algorithms which approximate BP. Once this equilibrium, weights can be updated in parallel rather than sequentially and no control signals are needed.
The paper is structured as follows. First, we provided concise introductions to predictive coding networks (PCNs), contrastive Hebbian learning (CHL), and equilibrium propagation (EP). Then, derive our fundamental results on EBMs, show how this results in a unified framework for understanding existing algorithms, and showcase how to generalize our framework to derive novel BP-approximating algorithms.
2 BACKGROUND AND NOTATION
We assume we are performing credit assignment on a hierarchical stack of feedforward layers .o.... with layer indices
0... I in supervised learning task. The input layer 2o is fixed to some data element d. The layer states xi are vectors of real values and represent a rate-coded aver- age firing rate for each neuron. We assume a supervised learning paradigm in which a target vector T' is provided to the output layer of the network and the network as a whole minimizes a supervised loss L(xL, T). T is usually assumed to be one-hot vector in classification tasks but this is not necessary. Synaptic weight matrices Wi at each layer affect the dynamics of the layer above. The network states 2y are assumed to be free parameters that can vary during inference. We use x {m²} and W = {Wi] to refer to the sets across all layers when specific layer indices are not important.
The models describe are EBMs, which possess energy function E(xo... ... ZT Wo.. Wr, which is minimized both with respect to neural activities 2j and weights Wi with dynamics.
dxi
OE
Inference:
di
dr
dWi OE
Learning:
(2)
aw,
which are simply a gradient descent on the energy. For notational simplicity we implicitly set the learning rate 7/ throughout. Moreover, we assume differentiability (at least up to second order) of the energy function with respect to both weights and activities. In EBMs, we run 'inference' to optimize the activities first using Equation until convergence. Then, after the inference phase complete we run a single step of weight updates according to Equation 2, which is often called the learning phase. A schematic representation of how EBMs learn with global energy function is presented in Figure TA while visual breakdown of our results is presented in Figure IB.
2.1 PREDICTIVE CODING
Predictive Coding (PC) emerged as a theory of neural processing in the retina Srinivasan et al.
1982) and was extended to general theory of cortical function (Mumford) 1992, Rao & Ballard
1999 Friston| 2005). The fundamental idea is that the brain performs interence and learning by learning to predict its sensory stimulu and minimizing the resulting prediction errors. Such an approach provides a natural unsupervised learning objective for the brain (Rao & Ballard 1999), while also minimizing redundancy and maximizing information transmission by transmitting only unpredicted information (Barlow] 1961] Sterling & Laughlin, 2015). The learning rules used by PCNs require only local and Hebbian updates (Millidge et al. 2020b) and variety of neural micro- circuits have been proposed that can implement the computations required by PC (Bastos et al.) 2012 Keller & Mrsic-Flogell Flogell 2018). Moreover, recent works have begun exploring the use of large-scale PCNs machine learning tasks, to some success (Millidge 2019 Salvatori al. 2021
2

---

Published as conference paper at ICLR 2023
2022 Kinghorn et al., 2021, Millidge et al. 2022 Lotter al. 2016 Ofner & Stober 2021). Un-
terms of vari- ational Bayesian inference (Friston 2005 2003 Bogacz 2017 Buckley et al. 2017 17 Millidge al,
like the other algorithms presented here, PC has mathematical interpretation
2021) and the variables in the model can be mapped to explicit probabiliste elements of gener-
Eiz e where the terms are prediction errors since they denote the difference between 6, = [1) - f(W
ative model. The energy function minimized by a PCN is written as Lie =
the activity z of layer and the 'prediction' of that activity from the layer below f(W(r;-1). The prediction error terms are weighted by their inverse covariance II, and this provides the network with degree of knowledge about its own uncertainty. The ratio of these precisions Ii-; effectively controls the weighting of bottom-up vs top-down information flow through a PCN. Given this en- ergy function, the dynamics of the PCN can be derived from the general EBM dynamics above as follows.
dæ
@Frc I1,es - 111411411 Of(W)))
(3)
di dm:
dW, OFPc dt aw, 111++1
0f(Win)
(4)
aw;
To simplify notation, for most derivations we assume identity precision matrices II, =1 و,ا I. Several limits in which PC approximates BP have been observed in the literature (Whitiintton & Bogacz, 2017 Millidge et al., 2020al Song et al. 2020). We show that the results of (Whittington & Bogacz (001)) and (Song et al. 2020) can be explained through our general framework. since both rely on properties of the infinitesimal inference limit. Specifically, (Whittington & Bogaez, 2017) demonstrate convergence to BP as the precision ratio is heavily weighted towards bottom up information. Secondly (Song et al., 2020) take advantage of the dynamics of the EBM at equilibrium to devise an PCN that implements exact BP.
2.2 CONTRASTIVE HEBBIAN LEARNING
Contrastive Hebbian Learning (CHL) originated a way train continuous Hopfield networks (Hopfield 1984) without the pathologies that arise with pure Hebbian learning (Hebb) 1949). Be- cause pure Hebbian learning simply strengthens synapses that fire together, it induces a positive feedback loop which eventually drives many weights to infinity. CHI. approaches handle this by splitting learning into two phases (Galland & Hinton 1991] Movellan [991]. In the first phase (free phase), the network output is left to vary freely while the input is fixed, and the network state converges to an equilibrium known as the free phase equilibrium E. At the free phase equilibrium, a standard Hebbian weight update is performed AW x ZT" Then in the second phase, the network output is clamped to the desired targets T and the network converges to a new equilibrium: the clamped phase equilibrium T. The weights are then updated with a negative anti-Hebbian update rule A W x - Instead of updating weights individually at each phase, one can instead write a single weight update rule that occurs at the end of the clamped phase,
ACCHL = - IE
(5)
The intuition is that neurons which are co-active at the clamped phase have their synaptic connec- tions strengthened while those that are co-active in the free phase are weakened, thus strengthening connections that are in some sense involved in correctly responding to the target while weakening all others. Mathematically, CHL performs gradient descent on contrastive loss function,
C(W) Ez, W) (I,
(6)
where the individual energies E and E² refer to the standard Hopfield energy (Hopfield 1982) at either the free phase or clamped phase equilibria. The Hopfield energy is defined as,
EcHI(Zr, (a. W) = x Tw w bbb
where b is bias vector. In one of the first results demonstrating an approximation BP, Xie & Seung 2003) showed that in the limit of geometrically decreasing feedback from higher to lower layers, the updates of CHL approximate BP to first order.
2.3 EQUILIBRIUM PROPAGATION Equilibrium propagation (EP) (Scellier & Bengio, 2017) can be considered a contrastive Hebbian method based an infinitesimal perturbation of the loss function. Instead of clamping the output of
3

---

Published as conference paper at ICLR 2023
A
B
E I AC
Infinitesimal Inference
AC
8xz
First Step Methods
C Camped Phase Equilbrum
dovid -llitce phese
BC
Zaro-Cveignacs (Slong
Weak Feedback
Weak Supervision
Stay riose phese by top down infueence
Stay cless plase by cowneeighting supervised abjection
(WW singtto at 2017) Smang
(Sooler I 8317) PC
Free phase equilibrium
Away from equili brium
Clamped phase liuiiibbrium
d.g
dii
01
21
0L
0Xx (xx
1
8x
1.
Figure I: A: An ANN optimizing supervised loss function with BP an EBM. For the ANN, the supervised loss is only affected by the output, and gradicnts are sequentially propagated backwards through the network. For the EBM, the energy is a global function of all variables, and all variables update themselves to minimize the energy according to Equation [ B: The hierarchy of methods approximating BP. Exactly the free phase, the dynamics point the direction of the supervised loss gradient. First-step algorithms exploit this fact to perform exact backprop. If the network is run clamped phase equilibrium, the gradients only converge BP as the clamped phase converges to the free phase equilibrium, which we call the infinitesimal inference limit. This can be assured either by ensuring that top-down information propagation through the network is weak (the weak feedback limit) or else ensuring that the supervised loss has only small effect on the total energy (weak supervision limit). C: A schematie visualization of the infinitesimal inference limit. Away from equilibrium, the dynamics the supervised loss. At the free phase equilibrium, the contribution from the internal energy is 0 so the dynamics follow exactly the gradient of the supervised loss. At the clamped phase equilibrium, the contributions of the internal and supervised losses perfectly cancel each other out.
determined by contributions from both the internal energy and
the network to the target, the outputs are clamped to a 'nudged' version of the input where the targets only possess small influence on the output. This influence is parametrized by scalar parameter A. Like in CHL the network operates in two phases free phase and a 'nudged' phase. In the free phase, the network minimizes an energy function Σ which is usually the Hopfield energy. In the "nudged phase', the network instead optimizes a nudged energy function E = ÉEP + AL where L is the supervised loss function at the output layer and 0 < X I 1 is scalar parameter which is assumed to be small. The activities at the free phase equilibrium, as in CHL, are denoted by I and the clamped phase equilibrium as こ. EP then updates the weights with the following rule, OEA- (a. W)]
1 1 OF
AW
À aw (7. W)
(8)
OW
which is the difference between the weight gradients at the nudged phase equilibrium and the free phase equilibrium. It was proven in (Scellier & Bengio 2017) that in the limit X 0, the EP update rule converges to BP as long as the energy gradients are assumed to be differentiable. Beyond this, it was shown in (Scellier & Bengio 2019) that in a RNN with static input the dynamics of the weight updates in EP are ıdentical to that of recurrent BP (Almeida 1990 Pineda 1987). Recently, there has been significant work scaling up EP to train large scale neural networks with significant success (Laborieux et al. 2021; Ernollt et al. (220)) as well implementations using analog hardware (Kendall et al. 2020 which utilizes the ability implement the inference dynamics directly through physics to achieve significant computational speedups.
THEORETICAL RESULTS
3.1 GENERAL ENERGY BASED MODEL
Here, we demonstrate how these disparate algorithms and their relationship with BP can all be derived from basic mathematical property of EBMs. A key fact about the EBMs utilized in these
4

---

Published as conference paper at ICLR 2023
algorithms is that the total energy E can be decomposed into sum of two components. An energy function the output layer which corresponds to the supervised loss L, and an 'internal energy' function Z which corresponds to the energy of all the layers in the network except the output layer. Typically, the internal energy can be further subdivided into layerwise local energies for each layer I= -E= Et. If we add a scaling coefficient 0 < À < 1 to scale the relative contributions of the internal and supervised losses, we can write the total energy as,
E -
(9)
The EBM then run in two phases. In the free phase, the output is left to vary freely, so that the targets have no impact upon the dynamics. This is equivalent to ignoring the supervised loss and hence setting À 0. In the clamped (or 'nudged") phase, the output layer is either clamped or nudged towards the targets. This means that the targets do have an impact the dynamics and thus 0. We now consider the dynamics of the network the equilibrium at the free phase. Since the dynamics of the activities of the network are simply a gradient descent on the energy, we have that,
dx
OI dr
(10)
dt
free pluse
Since, during the free phase À = O and so E = at the free phase equilibrium of the activities, it is straightforward to see that,
I. If we then begin the clamped phase, but start out
dx
OE = dr
8I
ac
dt clamped phase
dx
8x
0I 0 dt 8x
dr Oc
(1I)
dr
and thus that, when initialized at the free phase equilibrium, the activity dynamics in the clamped phase follow the negative gradient of the BP loss. As mathematical consequence of the decompos- ition of the energy function, at the free phase equilibrium. information about the supervised loss is implicitly backpropagated to all layers of the network to determine their dynamics.
An intuitive way to view this, which is presented in Figure T is that during inference, we can inter- pret the activity dynamics during the clamped phase as balancing between two forces one pushing to minimize the internal energy and one pushing to minimize the supervised loss. At the minimum of the internal energy (which is the equilibrium of the free phase). the 'force' of the internal en- ergy is O and so the only remaining force is that corresponding to the gradient of the supervised loss. Moreover, as long as the activities remain 'close enough to the free phase equilibrium, the dynamics should still be dominated by the supervised loss and thus the activity dynamics closely approximate the BP loss gradients. We call the limit where the clamped phase equilibrium converges to the free phase equilibrium the infinitesimal inference limit, since it is effectively stating that the 'inference phase' of the network is only achieving an infinitesimal change to the initial (free phase) 'beliefs'. This is the general condition which unifies the disparate BP BP-approximating algorithms from the literature which differ primarily in how they attain the infinitesimal inference limit.
Empirically, we investigate this condition in Figure 2. In panel A we plot the evolution of the three energy components during the clamped phase in a randomly initialized 4-layer nonlinear EBM with relu activation functions. We can see that, by starting at the free phase, the internal energy component is initialized at O and hence at the initial step. the slope of the total energy and that of the backprop loss are identical which is the key relationship that underpins the infinitesimal inference limit. In panel B. we quantify this effect and show that the distance between the change in neural activities and the backprop gradients increases rapidly with the number of inference steps taken until the distance saturates at the clamped equilibrium some distance away from the free equilibrium.
It is important to note that in layerwise architectures initialized at the free phase equilibrium, in- formation has to propagate back in layerwise fashion from the clamped output. This means that only the penultimate layer is affected first, and then one layer down is couured at each timestep. What this means is that the "first' motion of the activities in a layer in the clamped phase are guaran- teed to minimize the BP loss - not that all these motions occur simultaneously. This is the property that is exploited in the single step algorithms below. If, instead, it is required to update all weights simultaneously, instead of in layerwise fashion, it is necessary ate through the entire network and ultimately converge to the clamped phase fixed point. It is this requirement that leads to the fixed point algorithms in CHL, EP, and PC which, since more than one step is taken away from the free phase fixed point, results in approximate BP instead.
wait for information to propag-
5

---

Published conference paper ICLR 2023
A
B
Evolution of energy components during inference energy Backprop loss
Distance from backp op gradients during inference Dittance from backprop by distance from equiibrium
14
Internal energy
Drogrut Kuutppe mi Soodee Cl
12
04
18
20
tirestape
Inference trrestop
Distance from free phase
Figure 2: A: The evolution of the energies during an inference phase starting at the free phase. The supervised loss rapidly decreases while the internal energy increases (from 0). The total energy declines as the increase in internal energy is counteracted by a larger decrease of the supervised loss. At the initial condition where the internal energy is O (free phase) the change in energy is identical to the change in the backprop loss. B: Euclidean distance between the backprop gradients and the change in activity values during an inference phase. We again that the beginning of inference, close match to backprop is obtained while if inference continues for long time this match decreases. C: Relationships between distance between the clamped and free phase equilibria and the distance between the estimated gradients and the true BP gradients for multiple initializations of small MIP. As predicted by the Taylor expansion, for most initializations the approximation error grows linearly with distance between equilibria although some curves appear to bend slightly, especially for larger distances, indicating the breakdown of the local linearity assumption. All results obtained in three- layer nonlinear (relu) neural network using the predictive coding quadratic energy function.
3.2 FIRST STEP ALGORITHMS
A simple approach to exploit the dynamics at the free phase equilibrium is to directly use the direc- tion of the activity dynamies the first step update the weights. This approach compute the exact BP gradients since the gradients of the loss with respect to the weights can be expressed as,
dC
OC 0h4+1
1 dE
0
aW,
dm++1
OW.
X
20 1ai1 oW
dx 001+1
(12)
A dt 1844
aW
A specific case of this relation in PCNs was derived by (Song al. 2020). To understand it con- cretely, we specialize the derivation the dynamics of PCNs as in (Song et 2020). The free phase equilibrium of PCNs is equal to the feedforward pass in the equivalent ANN Iz =((W)11(11) where is activation function applied elementwise. A key property of PC is that the internal energy is at the free phase equilibrium. Thus all prediction errors throughout the network are zero: 6, = O. In the clamped phase, the output layer is clamped to the target and a prediction error é, is generated at the output layer. The dynamics of the activities follow Equation 3 but = 0. This implies that after the first step away from the free phase equilibrium the activities are,
dæp
Of(Wzz)
Of(Wia()
el+1
21+1 5o 4
(13)
dt
Oz
And thus that at the first step t + after the free phase equilibrium, the prediction error is,
af(Wa))
Of(Wizf)
4+1
1+1
5t T1 + eit1
Tt =14+1
(14)
Ox
dxf
However, this is simply a scaled version of the BP recursion where E1+1 takes the role of the adjoint. This means that the first step away from the free phase equilibrium, PCNs implement exact BP. Importantly though, this correspondence is not unique PCNs, but emerges as consequence of the mathematical structure of the energy function, and thus can occur in any EBM with such a structure and where the free phase equilibrium is equal to the forward pass of the network.
3.3 THE INFNITESIMAL INFERENCE LIMIT
Instead of utilizing the instantaneous dynamics at the free phase equilibrium, we can also use the property that the dynamics direction is approximately the same the gradient direction as long as

---

Published as conference paper at ICLR 2023
the activities remain 'close' to the free phase equilibrium to derive algorithm that approximates BP and converges to it in the limit of the free and clamped equilibria coinciding. We consider algorithms that converge to a second 'clamped' fixed point based on the dynamics of the total energy function with a contribution from the supervised loss - i.e. with À >> 0. Given both the clamped and free fixed points, we show that we can approximate the supervised loss gradient as the difference between the gradients at the clamped and free fixed points. If we assume that the gradients of each component of the energy with respect to the weights are continuous and differentiable, we can derive this approximation directly by considering a first order Taylor expansion of the clamped fixed point around the free fixed point T.
OE OW
OE
(x* - (e- 0(622
OW 01^
drdW
Oc
X dw
Hgo + 0(6,²²
(15)
aw
x==
Where in the second line we have used the decomposition of the energy (Equation and we have 8²
defined og - (x* =) and Hg Hoew 1z=8- Using Equation 9 and the fact that the internal energy does not depend on X and thus the gradient of the internal energy at the activity values of the free phase equilibrium is the same as the gradient of the total energy at the free phase equilibrium i.e., ar 2gA=0 aw ow
we can rearrange this to obtain,
ac OW
OE
Hg0 - -
(16)
==
OW
==
which is the CHL update rule (Movellan 1991) as an approximation to the gradient of the supervised loss at the free phase equilibrium. This provides a mathematical characterisation of the conditions for when this approximation is accurate and hence the CHL algorithm well approximates the BP gradient. A key metric for the validity of our Taylor expansion approximation is that it predicts that the relationship between the approximation error to the backprop gradient and the distance between the free and clamped equilibria should be linear, at least when the equilibria are close. This is due to the dominance of the first-order linear term in the expansion. We empirically test this in Figure 2C, and show that for a large number of initializations, this tends to be correct, at least for small distances, thus showing that the first order terms appear able to model the approximation error well.
By observing the explicit error term, we see that it depends on two quantities. Firstly, the crossde- rivative of the clamped phase energy at the free phase equilibrium H₃ with respect to both activities and weights, and secondly the distance between activities at the free and clamped phase equilib- rium 6g. Intuitively, the crossderivative term measures how much the weight gradient changes as the activity equilibrium moves. Thus, when this term is large, the energy in a highly sensitive region, so that the gradients computed at a clamped phase equilibrium small distance away from free phase equilibrium might not be as accurate.
This crossderivative term depends heavily on the definition of the energy function, as well as the location of the free phase equilibrium in the parameter space and thus this means of reducing the ap- proximation error has largely been ignored by existing algorithms which focus primarily on reducing on. Our analysis does, however, suggest the possibility of 'correcting" the CHL update by subtract- ing out the first order taylor expansion term 2z For many energy functions, this derivative may be obtained analytically or else by automatic differentiation software (Griewank et al. 1989) and oz waaeeteetataaa straightforward compute. This would then make the corrected CHL update approximate BP to second order instead of the first order approximation that is currently widely used.
Reducing 5, means ensuring that the clamped phase equilibrium is as close as possible to the free phase. This can be accomplished in two ways. The first, which is implemented in CHI, is to still clamp the output units of the network, but to ensure that feedback connections are weak so that even with output units clamped. the equilibrium does not diverge far from the free phase. This approach is taken in (Xie & Scung (003) who require geometrically decaying feedback strength across layers (parametrerized by a scalar parameter 0 < < 1) to obtain a close approximation to BP. This is why we call this condition the 'weak feedback limit'. A similar approach is taken in (Whittington) & Bogacz, 2017) who similarly require geometrically decaying precision ratio across layers to obtain a close approximation
BP in PCNs. Alternatively, we can instead maintain the feedback strength but reduce the importance of the supervised loss on the network dynamics - Le. turn a clamped output to a 'nudged' output which we call the 'weak supervision limit'. This alternate
7

---

Published conference paper at ICLR 2023
approach is taken in the EP algorithm where almost identical algorithm is derived by instead taking an infintesimal perturbation of the À parameter.
3.3.1 PREDICTIVE CODING AS CONTRASTIVE HEBBIAN LEARNING
Using our new framework, it is straightforward to derive PC as CHL algorithm which simply uses the variational free energy instead of the Hopfield energy. PCNs have unique property that the free phase equilibrium simply the feedforward pass the network and that all prediction errors eg in the network are zero which implies the free phase weight gradient is also zero since,
OE
Of(War)) OwL
= EL+1
of =0
(17)
OW
which results in the CHI. update in PC simply equalling the gradient the clamped equilibrium,
AW 1 OF 1 A aw aw
OE
OE
(18)
aw
since the free phase Hebbian weight update in PCNs is O. This means that only a single clamped phase is required to train PCNs instead of two for the Hopficld energy. This results in significant computational savings as well as increased biological plausibility since the gradients of the free phase do not need to be stored in the circuit during convergence to the clamped phase equilibrium.
3.4 EP AS INFINITESIMAL PERTURBATION
While CHL algorithms typically clamp the output of the network to the targets and hence utilize a large output A, necessitating that feedback through the network be weak, EP takes the opposite approach and explicitly utilizes small \ directly. EP can be derived by essentially taking an infin- itesimal perturbation of X and comparing the difference between the two phases the free phase, and an infinitesimally 'nudged' phase where À + O. To derive EP, if we take the derivative of the decomposition of the energy in Equation 9
with respect to X, we can write the supervised loss L as,.
= = L dE dA
(19)
We wish to compute the gradient of the supervised loss with respect to the parameters W,
dC a OF OF dr ow ax
a OE Ô OE Oz Ox ow ax a% aw
(20)
dW
aE = and we have where firstly we have utilized the fact that in the free phase equilibrium dr substituted in Equation 19 In the final step we have utilized the fact that partial derivatives commute. We have thus expressed the gradient of the loss in terms of a derivative with respect to the supervised loss scale term X. One way to compute this derivative is by finite differences, thus for a small X,
OL oW
1 O A X OW
OE
lim
(21)
OW
Where the first gradient is taken the nudged phase equilibrium and the second at the free phase equilibrium where X= 0. By the standard properties of finite difference approximations, this update rule becomes exact as À - 0. This recovers the EP weight update rule. Importantly, we do not make any assumptions about the form of the energy function. This means that this method is general for any energy function and not just the Hebbian energy used in (Scellier & Bengio 2017). For instance, instead the PC squared energy function could be used which would derive additional algorithm that approximates BP in the limit that has not yet been proposed the literature although would be closely related to existing results (Whittington & Bogacz 2017). We implement and investigate this algorithm below and demonstrate that its convergence to BP properties match precisely those predicted by our theory.
4 EXPERIMENTS
Here, we use our theoretical framework to derive a novel BP-approximating variation which we call 'PC-Nudge' that appproximates BP through the use of weak supervision instead of the CHL- weak-feedback limit previously reported in (Whittington & Bogacz] 2017). (o01cz|2017). Instead of adjusting the precision ratio, PC-nudge instead varies the weighting of the supervised loss with like coeffi-
8

---

A
B
Distence true beckprop gredient by lambda
Teeining accuracy "C-Nadge and
Similrity backprop gr gradients during training
neiralae wddden ditrrs
3030
3597
25:80
PC-Hugge
Backprs
Wi wegghnng confiientt
DUGL TUR12ET
batch FunT Eet
Figure 3: A: A linear relationship between the À weighting coefficient and the approximation of the PC-Nudge algorithm to BP was robustly observed at initialization. Error bars represent standard deviations across 20 random initializations. B: The PC-Nudge network trained with almost identical accuracy across batches as the corresponding ANN, C: Cosine similarity during training between BP gradients and those estimated by PC-Nudge. Although similarity appears to decline somewhat, it never goes below 0.999 and has no observable impact on training. Results were obtained with a 4- layer MLP with relu activation functions trained with predictive coding quadratic energy function.
cient A. PC-Nudge can be defined as gradient descent on the following free energy functional,
1-1 es
Epc-Nudge Aeg 1-ر
(22)
Which results in a standard PCN except with downweighted final layer since À is defined to be small. To counteract the downscaling of the supervised loss, all learning rates in the network are rescaled by À with a new learning rate r = X resulting in the same training rate as BP. As a demonstration experiment, in Figure 3 we compared PC-Nudge to BP on training simple 4-layer MLP with relu activation function on the MNIST dataset. The actual scale of the experiment is largely irrelevant since as À - O, the gradients computed by PC-Nudge converge towards the exact BP gradient and hence we expect training performance to be the same for any architecture and scale. In practice, for our network, throughout all of training and correspondingly identical training performance. We plot the training accuracies of the PC-Nudge and backprop algorithms (which are almost identical) in panel B, and the cosine similarity between the PC-Nudge weight updates and gradients in panel C. We found that this similarity decreases marginally throughout training (but always remained high at above 0.999) but not enough affect training in a noticeable way. We conjecture this occurs because the norm of the BP gradients decreases during training, thus necessitating a smaller À to maintain equivalent scale. Finally, in Figure 3A, we also found that the distance between the PC-Nudge and BP gradients at initialization was approximately linear function of À over a wide range of sensible À values. Detailed information regarding our implementation and experiments can be found in Appendix A.2 and an explicit algorithm for PC-Nudge can be found in Appendix A.2.3
found À = 0.01 to achieve extremely close convergence to BP
5 DISCUSSION
In this paper, we have proposed a novel and elegant theoretical framework which unifies almost all the disparate results in the literature concerning the conditions under which various "biologically plausible' algorithms approximate the BP of error algorithm. We have demonstrated that these convergence results arise from shared and fundamental mathematical property of EBMs in the infinitesimal inference limit. This limit is essentialy when the clamped phase equilibrium of the EBM converges towards its initial free phase equilibrium. We have shown that the approximation error scales linearly with the distance between the free and clamped equilibria. This limit can be achieved in one of two ways either by enforcing weak feedback through the network, leading to the CHL algorithm of (Xie & Seung, 2003) and the PC result of (Whitington & Bogaez) (017)) or else by scaling down the contribution of the supervised loss to the energy function, by which one obtains EP (Scellier& & Bengio. 2017) and our novel PC-Nudge algorithm.
While the analysis in this paper covers almost all of the known results concerning approximation or convergence to BP, there exist several other families of biologically plausible learning algorithms which do not converge BP but which nevertheless can successfully train deep networks. These include propaapropag (Le et al. 20155 Meulemans et al.] 2020), the prospective configuration algorithm in PCNs networks Song et al. (2022), and the feedback alignment algorithm and its vari- ants (Lillicrap et al.) 2016 Nøkland 2 et al. (020).. Whether these algorithms fit into an even broader family can be derived exploiting other mathematical properties of shared EBM structure remains to be elucidated. Moreover, our mathematical results rely heavily on the relatively crude machinery of Taylor expansions around the clamped phase equilibrium. An interesting avenue for future work would be to apply more sophisticated methods to obtain better approximations and understanding as to the scaling of the approximation error as the distance between the clamped and free phase equilibria increases.
9

---

Published conference paper at ICLR 2023
ACKNOWLEDEGMENTS
This work has been supported by BBSRC grant BB/S006338/1 and MRC grant MC. I.C.UU00000/1. Thomas Lukasiewicz has been supported by the Alan Turing Institute under the UK EPSRC grant EP/N510129/1 and by the AXA Research Fund.
REFERENCES
Luis B Almeida. A learning rule for asynchronous perceptrons with feedback in combinatorial environment. In Artificial Neural Networks: Concept Learning, pp. 102-111. IEEE Press, 1990.
Horace Barlow. The coding sensory messages. Current Problems Animal Behavior, 1961.
Andre M Bastos, W Martin Usrey, Rick Adams, George Mangun, Pascal Fries, and Karl J Friston. Canonical microcircuits for predictive coding. Neuron, 76(4):695-711, 2012.
Yoshua Bengio and Asja Fischer. Early inference in energy-based models approximates back-
propagation. arXiv preprint arXiv:1510.0277) 2015.
Rafal Bogacz. A tutorial on the free-energy framework for modelling perception and learning. Journal of Mathematical Psychology, 76: 16:119-200
2017. 2017.
Christopher I. Buckley, Chang Sub Kim, Simon McGregor, and Anil K Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55-79, 2017.
Francis Crick et al. The recent excitement about neural networks. Nature, (37 1989.
Maxence Emoult, Julie Grollier, Damien Querlioz, Yoshua Bengio, and Benjamin Scellier. Equilib- rium propagation with continual weight updates. arXiv preprint arXiv:2005.04166, 2020.
Karl Friston. Learning and inference in the brain. Neural Networks, 16(9):1325-1355) 2003.
Karl Friston. A theory of cortical responses. Philosophical Transactions of the Royal Society B:
Biological Sciences, 360(1656)) (8356)8115-835 2005.
Conrad C Galland and Geoffrey E Hinton. Deterministic Boltzmann learning in networks with asymmetric connectivity. In Connectionist Models, pp. 3-9. Elsevier, 1991.
Andreas Griewank et al. On automatic differentiation. Mathematical Programming: Recent devel-
opments and applications, 6(6):83-107, 1989.
Donald O Hebb. The first stage of perception: Growth of the assembly. The Organization of Behavior, 4:60-78, 1949.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-2558, 1982.
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81800)3088-3002 1984.
Georg Keller and Thomas D Mrsic- Mrsic-Flogel. Predictive processing: canonical cortical computa-
tion. Neuron, 100(():424 435, 2018.
Jack Kendall, Ross Pantone, Kalpana Manickavasagam, Yoshua Bengio, and Benjamin Scel- lier. 'rriniin end-to-end analog neural networks with equilibrium propagation. arXiv preprint arXiv:2006.0/99 2020.
Paul F Kinghorn, Beren Millidge, and Christopher L Buckley. Habitual and reflective control in hierarchical predictive coding. arXiv preprint arXiv:2109 200.00000.00066, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.
Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation to deep ConvNets by drastically reducing its gradient estimator bias. Frontiers in neuroscience, 15: 2021.
10

---

Published conference paper at ICLR 2023
Julien Launay, lacopo Poli, François Boniface, and Florent Krzakala. Direct feedback alignment scales to modern deep learning tasks and architectures. Advances in Neural Information Pro- cessing Systems, 33:9346 93600 2020.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp- 498-515, Springer, 2015.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, (1):1-10, 2016.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Back- propagation and the brain. Nature Reviews Neuroscience, (1(6))335- 3466, 2020.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning. arXiv preprint ar0iv: 08104, 2016.
Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin F Grewe. A theoretical framework for target propagation. Advances in Neural Information Pro- cessing Systems, 33:20024-20000 2020.
Beren Millidge.
Implementing predictive processing and active inference: Preliminary steps and results. PsyArXiv, 2019.
Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Predictive coding approximates backprop along arbitrary computation graphs. arXiv preprint arXiv:2006. 04182, 2020a.
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher L Buckley. Relaxing the constraints on predictive coding models. arXiv preprint arXiv:2010.01047, 2020b.
Beren Millidge, Alexander Tschantz, Anil K Seth, and Christopher L Buckley. Activation laxation: A local dynamical approximation backpropagation in the brain. arXiv preprint arXiv:2009.05335 053599 2020c.
Beren Millidge, Anil Seth, and Christopher L Buckley. Predictive coding: A theoretical and experi- mental review. arXiv preprint arXiv:2107.12979 2021.
Beren Millidge, Tommaso Salvatori, Yuhang Song, Rafal Bogacz, and Thomas Lukasiewicz. Pre- dictive coding: Towards a future of deep learning beyond backpropagation? arXiv preprint 2964:22002.094667 2022.
Javier R Movellan. Contrastive Hebbian learning in the continuous Hopfield model. Connectionist Models, pp. 10-17. Elsevier, 1991.
David Mumford. On the computational architecture of the neocortex. Biological Cybernetics, 66 (3):241-251, 1992.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016
André Ofner and Sebastian Stober. PredProp: Bidirectional stochastic optimization with precision weighted predictive coding. arXiv preprint arXiv:2111.0879) 2021.
Alexander G. Ororbia and Ankur Mali. Biologically motivated algorithms for propagating local tar- get representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4651 46588 2019.
Fernando Pineda. Generalization of back-propagation recurrent neural networks. Physical Review Letters, 59(19;22229, 1987.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: A functional interpreta- tion of some extra-classical receptive-field effects. Nature Neuroscience, 2(1))79-87, 1999.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Nature, 323((088):533-555 1986.
Tommaso Salvatori, Yuhang Song, Yujian Hong, Lei Sha, Simon Frieder, Zhenghua Xu, Rafal Bogacz, and Thomas Lukasiewicz. Associative memories via predictive coding. Advances in Neural Information Processing Systems, 34, 2021.
11

---

Published conference paper at ICLR 2023
Tommaso Salvatori, Luca Pinchetti, Beren Millidge, Yuhang Song, Rafal Bogacz, and Thomas Lukasiewiez. Learning arbitrary graph topologies via predictive coding. arXiv preprini arXiv:2201. 13100, 2022.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in Computational Neuroscience, 11 11:24, 2017.
Benjamin Scellier and Yoshua Bengio. Eguivalence of equilibrium propagation and recurrent back- propagation. Neural computation, 3 11)):312-322, 2019.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game Go with deep neural networks and tree search. Nature, 529(7587):
484-489, 2016.
Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Can the brain do backkropaggtioon -xant implementation of backpropagation in predictive coding networks. Ad- vances in Neural Information Processing Systems, 11222566-22777 2020.
Yuhang Song, Beren Millidge, Tommaso Salvatori, Rafal Bogacz, and Thomas Lukasiewicz. Infer- ring neural activity before plasticity: A foundation for learning beyond backpropagation. bioRxiv preprint, 2022.
Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: A fresh view of inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, 216(1205):427 454444979 1982.
Peter Sterling and Simon Laughlin. Principles of Neural Design. MIT press, 2015.
David G Stork. Is backpropagation biologically plausible. In International Joint Conference on Neurul Networks, volume 2, pp. 241-246. IEÉE Washington, DC, 1989.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Informa- tion Processing Systems, 30, 2017.
James CR Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computatioon 29 15:112291262, 2017.
James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in Cognitive Sciences, 23(3):235 250, 2019.
Xiaohui Xie and H Sebastian Seung. Equivalence of backpropagation and contrastive Hebbian learning in a layered network. Neural Computation, 15(2):441-43 2003.
A APPENDIX
A.1
Related Work The main contribution of this paper is in tying together many previous threads in the literature and presenting unified view of algorithms approximating BP. As such many of the ideas and derivations here are present, albeit in isolation, in previous literature. A similar relationship between the gradients of the supervised loss and the dynamics of neural activity at free phase equi- librium has been observed before in (Bengio & Fischer 2015). The single-step update is derived in (Song et al. 2020) although the derivation is specific to PC and was not conceptualized as general property of energy based models. The derivation of EP as finite difference approximation of the gradient the parameter follows that in (Scellier & Bengio 2017).
relationship between the PC approximation result of (Whittington & Bogacz (017) and EP was hinted at in (Whitiington & Bogacz, 2019) but not explicitly worked out, and we argue that CHL is actually a better fit for PC since the output is fully clamped and not 'nudged' gradient descent on a contrastive loss function equal the difference between the free and clamped equilibrium is an explicit part of the CHL literature (Movellan) 1991: Xie & Seung 2003) although to our knowledge our derivation of this loss function as a consequence of a general energy based model is novel. Similarly, our derivation in Section 3.3 3.33expe expressiin the infinitesimal inference limit in its full generality is novel as is explicit working out of the approximation terms.
in EP. The idea of CHL being derived
12

---

Published as conference paper at ICLR 2023
A.2 EXPERIMENTAL DETAILS
All experiments used the MNIST dataset. A standard training-tess test split of 50,000 train images and 10,000 test images was used. All MNIST images were normalized so that their pixel values lay in the range [0.1].
All experiments were undertaken on 3-layer MLP architecture. The MLP consisted three layers of size 784, 128,648, 228,64.0 with a one-hot output vector of the correct class. A relu activation function was used with a softmax activation at the output along with the cross-entropy loss function. The network was initialized with using Pytorch's 's default (Xavier) initialization for linear layers.
The inference phase of the PCN was accomplished by explicitly integrating Equation for 50 itera- tions. An inference learning rate of 0.1 was used for the inference phase. All precision parameters I were implicitly set to the identity matrix.
All code required to reproduce all experiments and figures in this paper will be made publically available at github address.
A.2.1 FIGURE 2 IMPLEMENTATION DETAILS
In Figure 2A, the energy was computed as the PC energy the sum of squared prediction errors at each timestep of inference. The internal energy was defined as the sum of the squared prediction errors for the first two layers and the supervised backprop loss was the prediction error of the final layer. This lets interpret the network effectively redistributing prediction errors from the output layer into the internals of the network, but doing so in a way that allows the total norm of the prediction errors to decrease.
In Figure 2B, we computed the Euclidean distance between the true backprop gradients and that approximated by the EBM during inference. We used an inference step-size of 0.1. We computed our results randomly initialized 3-layer MLP as described above or, equivalently, PC-Nudge with X = Error bars represent the standard deviation across 10 random initializations.
In Figure 2C, we investigated the relationship between the approximation error to the true backprop gradients and that computed by the EBM (in our case a 3-layer MLP PCN), Estimated gradients were computed by our PC-Nudge algorithm where distances between the free and clamped phase were varied in practice by testing different À parameters. The total euclidean distance d(x:, y)
VE,( - )² between both free phase (forward pass) and clamped phase activations as well as between the BP and PC-Nudge estimated gradients were used. The equilibrium distances were summed across all layers while the gradient distances were summed across all parameters (weights and biases) in the network. Since the exact value of the distances is largely irrelevant we simply summed and did not normalize either the gradients or equilibrium distances so in practice the exact values of these distances scale with layer-width and depth of the network. For this figure we chose 10 represenentative examples of the same input image randomly selected mnist image and label) across initializations of the network.
A.2.2 FIGURE 3 IMPLEMENTATION DETAILS
The learning results were obtained with the SGD with momentum optimizer with a learning rate of 0.001 for both BP and PC-nudge with learning rate of 0.001 The momentum parameter was set to 0.9. A minibatch size of 64 was used throughout. À was set to 0.001.
x
Panel A was produced by computing the Euclidean distance between the BP and PC-Nudge gradients after 50 inference steps for various values of A. The shaded area represents the standard deviation over 10 seeds. Panel B shows the batch-by-batch training accuracies of the BP and PC-Nudge network superimposed. The gradients computed by the two approaches were so similar that the weight and gradient updates ended up being almost identical. Panel C shows the cosine similarity (or normalized dot product) between the PC-Nudge estimated gradients and the true BP gradients. We computed the similarity as,
x x
similarity(z. ||=|||||v|ee
(23)
A similarity score of is the highest possible score and - 1 is the lowest.
A.2.3 PC-NUDGE ALGORITHM
13

---

Published conference paper at ICLR 2023
Algorithm 1 PC-Nudge algorithm
Data: Dataset D = 4d,, set of neural activities {ar), set of weights {Wi} inference learning rate M/,, weight learning rate njo, 'nudge' parameter X
begin
/* For each minibatch in the dataset for (d, () E do
/+ Fix input layer to data
To +
/* Forward pass to compute predictions
for कι E network layers (1} do
21 ((Wan))
Compute output error
CL + (T /x Inference phase
while not converged do
for (æ), E1) E network layers {1} do
Compute prediction errors
a t 21 f(W (w-1x))))
/+ Update neural activities
1 1+1 c of
The
0zf
/* Update weights at equilibrium
for Wi E network weights {Wi do
W + WT + x OW?
BE
A.3 BIOLOGICAL PLAUSIBILITY OF EBMS
In the literature, the EBMs described in this paper, are often motivated and described as biologic- ally plausible, unlike backprop. Due to the fact that we have shown that all of these algorithms approximate backprop in the limit, it is important to consider how this affects the claimed biolo- gical plausibility of these algorithms. For the first-step algorithms we describe (section 3.2), these algorithms are exactly equivalent to backprop (up to a scaling factor) and hence have identical bio- logical plausibility properties as backprop. In effect, these first step algorithms simply propose an additional mathematical interpretation of backprop taking single step towards the minimum of some energy function, thus allowing us to situate backprop as special case of more general EBM framework.
EBMs often claimed be biologically plausible in the literature due their properties of highly parallel and local updates in the inference phase. This due to typically layerwise form of the energy functions used to derive the dynamics of neural activities in the EBM. This means that, in theory, the neurons in an EBM can simultaneously update themselves in parallel, responding only to their neighbours in the layers above and below. Information propagates through the network at the same layerwise rate as backprop, however. As such, the general EBMs have some claim to be eliminate some of the biological implausibilities of backprop such as its sequential forward and backward sweeps where each layer must 'wait' for previous layers to complete either the forward or backward pass. EBMs do not address other claimed biological implausibilities such as the weight transport problem (Lillicrap et al. 2016 Crick al., 1989) and also introduce new ones such as how information from the free phase iS stored during the clamped phase, and how the two phases can be coordinated.
Additionally, the infinitesimal inference limit required to closely approximate backprop typically leads to biologically implausible constraints such as extremely weak feedback connections (when feedback connections in the brain are strong) or an extremely small À parameter value leading to tiny differences in values between phases which must be stored and then subtracted accurately while computation in the brain likely takes place in relatively limited precision.
However, regardless of the biological plausibility of these EBM limits, we believe that our work is important since it lets us understand that backprop emerges as general limit of EBMs and how to unity existing EBMs in the literature into a single, extremely simple framework.
14


Reviews:
[{"review_id": "nw6XyuvuYk", "reviewer": "Reviewer_oU1M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Three types of biologically-plausible learning algorithms --- predictive coding, contrastive hebbian learning, and equilibrium propagation --- are reformulated as energy based models. This framework is used to provide a unifying description for when these algorithms closely approximate gradient descent (backprop), and the authors use this new perspective to suggest a new variant of predictive coding called PC-Nudge. Experiments with a multi-layer neural network on MNIST are used to support the proposed ideas. ", "summary_of_the_review": "This is a well-written paper that might be of interest to some in the community, but despite my familiarity with the algorithms discussed, I was unable to see how it is particularly useful to view them from the energy-based model perspective.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written and clear.\n- The new energy-based model perspective is a way to compare and contrast three different non-backprop  learning algorithms, and understand how they deviate from backprop (gradient descent on the supervised loss). \n\nWeaknesses:\n- It is not obvious to me that viewing these algorithms as gradient descent on an energy function is useful.\n- One justification given for proposing this new framework is that it can be used to propose new algorithms.  PC-Nudge is proposed. However, this algorithm seems to be a minor variation on existing algorithms, and the experimental results on MNIST are not particularly interesting. \n- Performing gradient descent on an energy function is not necessarily biologically plausible, and this paper does not have much to say about the biological plausibility of these various algorithms. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d3bhU4rAKI", "reviewer": "Reviewer_Cz4e", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A number of different algorithms have been proposed as more biologically plausible alternatives to backpropagation.  While different works have shown how several of these proposals approximate backprop by converging to it in some limit, the authors here outline an overall framework of energy-based models that encompasses several predictive coding proposals, equilibrium propagation, and contrastive Hebbian learning, showing how they all can be formulated as approximations to backprop.  By breaking the global energy into components corresponding to the internal energy of the network and the network's cost function, the authors show that different algorithms can approximate backprop by finding equilibrium points near the minimum of the internal energy (by, e.g., employing weak feedback).  They finally show that other algortihms can be derived from their approximation, with \"PC-nudge\" described as an example. ", "summary_of_the_review": "Several results are gathered and combined by showing how a number of proposed biologically plausible alternatives to backprop can be shown to approximate backprop when formulated as energy-based models. The work then leads to an example of a novel algorithm that can be derived using this common framework", "strength_and_weaknesses": "Strengths\n\n1. The authors gather previously disparate theoretical results together under the same umbrella of energy-based models.\n\n2. Several versions of predictive coding, along with equilibrium propagation and contrastive Hebbian learning are shown how, under the same framework, they can be derived as approximations to backprop.\n\n3. A novel algorithm, \"PC-nudge,\" is shown to be derivable from the proposed framework\n\nWeaknesses\n\nNo major weaknesses of the manuscript were found\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bro10lNT8K", "reviewer": "Reviewer_wqJn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper starts from a PCN and expands around the limit where the output is not tethered to the target. Perturbing in the parameter $\\lambda$, which may be interpreted as the precision parameter deciding how tightly would the target be tied to the output, builds a useful bridge to many other approaches to training neural networks.", "summary_of_the_review": " I found this paper useful for relating predictive coding networks with other competing approaches. It is a welcome addition to the literature.", "strength_and_weaknesses": "Strengths:\nAnalysis of the small $\\lambda$ limit\nThe PC-nudge model\nDiscussion of contrastive Hebbian learning, starting from PC.\n\nMinor issues:\nIn equations (3) and (4) the indexing conventions of $W_l$ and $x_l$ seems to be inconsistent with the definition of $\\epsilon_l$ in the previous page.\nUnder equation(5), the sentence mentioning “..neurons which are co-active at the clamped phase are strengthened…” is confusing. What does 'neurons being strengthened' mean? What is really strengthened? Synaptic strengths? That seems to have the opposite effect. A clearer explanation here would help.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-b_pYQwBZpn", "reviewer": "Reviewer_mXjX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper unifies several approximation to backpropagation through the energy-based models view. ", "summary_of_the_review": "Potentially interesting connection between algorithms, but I'm not sure what we can do with it. I'm willing to discuss it further, however.\n\n**UPDATE**: updated the score from 5 to 6 post-rebuttal.", "strength_and_weaknesses": "### Strengths\n\nThe paper unifies several backprop approximation under the same energy-based framework.\n\nThis framework leads to different approximations.\n\n### Weaknesses\n\nI don’t understand what the take-home message is. Predictive coding and equilibrium propagation were already formulated in a similar fashion, but what can we do with it now? I'd guess deriving more plausible algorithms would be interesting, but it's not really discussed.\n\nContrastive Hebbian learning narrowly fits the proposed framework. Eq. 18 seems to be just the gradient of the loss (since $I$ is zero in that case), which is not what CHL is doing. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster