Predictive Coding Approximates Backprop along Arbitrary Computation Graphs
Authors: Beren Millidge
Alexander Tschantz
Christopher Buckley
Venue:  (2021)
Keywords: Predictive Coding
Backprop
Biological plausibility
neural networks

Abstract: The backpropagation of error (backprop) is a powerful algorithm for training machine learning architectures through end-to-end differentiation. Recently it has been shown that backprop in multilayer-perceptrons (MLPs) can be approximated using predictive coding, a biologically-plausible process theory of cortical computation which relies solely on local and Hebbian updates. The power of backprop, however, lies not in its instantiation in MLPs, but rather in the concept of automatic differentiation which allows for the optimisation of any differentiable program expressed as a computation graph. Here, we demonstrate that predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules. We apply this result to develop a straightforward strategy to translate core machine learning architectures into their predictive coding equivalents. We construct predictive coding CNNs, RNNs, and the more complex LSTMs, which include a non-layer-like branching internal graph structure and multiplicative interactions. Our models perform equivalently to backprop on challenging machine learning benchmarks, while utilising only local and (mostly) Hebbian plasticity. Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry, and may also contribute to the development of completely distributed neuromorphic architectures.

Under review as a conference paper at ICLR 2021
PREDICTIVE CODING APPROXIMATES BACKPROP ALONG ARBITRARY COMPUTATION GRAPHS
Anonymous authors Paper under double-blind review
ABSTRACT
Backpropagation of error (backprop) is powerful algorithm for training machine learning architectures through end-to-end differentiation. Recently it has been shown that backprop in predictive coding, a mulrepayer-prrreptrons (MLPs) can be approximated using process theory of cortical computation which relies solely on local and Hebbian updates. The power of backprop, however, lies not in its instantiation in MIPs,, but rather in the concept of automatic differen- tiation which allows for the optimisation of any differentiable program expressed computation graph. Here, we demonstrate that predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules. We apply this result to develop a straightforward strategy to translate core machine learning architectures into their predictive coding equivalents. We construct predictive coding CNNs, RNNs, and the more complex LSTMs, which include lane-ayr--like branching internal graph structure and multiplicative interactions. Our models perform equivalently to backprop on challenging machine learning benchmarks, while utilising only local and (mostly) Hebbian plasticity. Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry, and may also contribute to the development of completely distributed neuromorphic architectures.
INTRODUCTION
Deep learning has seen stunning successes in the last decade in computer vision (Krizhevsky et al. 2012, Szegedy et al. 2015), natural language processing and translation (Vaswani et 2017
Radford et al. 2019 Kaplan et al. 2020). and computer game playing (Mnih et al.) [2015] Silver ct al. 2017; Schrittwieser et al. 2019, Vinyals et al., 2019). While there a great variety of architectures and models, they are all trained by gradient descent using gradients computed by automatic differentiation (AD). The key insight of AD is that it suffices to define forward model which maps inputs to predictions according some parameters. Then, using the chain rule of calculus, is possible, as long as every operation of the forward model is differentiable, to differentiate back through the computation graph of the model as compute the sensitivity of every parameter in the model to the error at the output, and thus adjust every single parameter to best minimize the total loss. Early models were typically simple artificial neural networks where the computation graph is simply a composition of matrix multiplications and elementwise nonlincarities, and for which the implementation of automatic differentation has become known as 'backpropagation" (or "backprop'). ). However, automatic differentiation allows for substantially more complicated graphs to be differentiated through. up to, and including, arbitrary programs (Griewank et al.) 1989 Baydin et al. 2017 Paszke et al. 2017 Revels et al. 2016, Innes et al.) [019 Werbos] 1982. Rumelhart and Zipser 1985 Linnainmma 1970). In recent years this has enabled the differentiation through differential equation solvers (Chen et al. 2018 Tzen and Raginsky 2019 Rackauckas et al. 2019).
physics engines (Degrave et al. 2019:] Heiden et al. 2019). raytracers [Pal|2019)] and planning algorithms (Amos and Yarats 2019: Okada al. (017). These advances allow the straightforward training of models which intrinsically embody complex processes and which can encode significantly more prior knowledge and structure about a given problem domain than previously possible.
Modern deep learning has also been closely intertwined with neuroscience (Hassabis et al. 2017] Hawkins and Blakeslee 2007 Richards et al. 2019). The backpropagation algorithm itself arose

---

Under review as a conference paper at ICLR 2021
technique for training multi- perceptrons simple hierarchical models of neurons inspired by the brain (Werbos] 1982). Despite this origin, and its empirical successes, a consensus has emerged that the brain cannot directly implement backprop, since to do would require biologically implausible connection rules (Crick 1989). There are two principal problems. Firstly, backprop in the brain appears to require non-local information (since the activity of any specific neuron affects all subsequent neurons down to the final output neuron). It is difficult to see how this information could be transmitted "backwards' throughout the brain with the required fidelity without precise connectivity constraints. The second problem - the 'weight transport problem' is that backprop through MLP style networks requires identical forward and backwards weights. recent years, however, a succession of models have been introduced which claim to implement backprop in MLPP-stlle models using only biologically plausible connectivity schemes, and Hebbian learning rules (Liao et al. 2016) Guerguiev et al. 2017 Sacrameeoo et al. 2018 Bengio and Fischer 2015: [eengio et al. 201 7 et al.
2020 W Whittington and Bogacz, 2019). Of particular significance is Whittington and Bogacz (2017) who show that predictive coding networks type of biologically plausible network which learn through a hierarchical process of prediction error minimization are mathematically equivalent to backprop in MLP models. In this paper extend this work, showing that predictive coding can not only approximate backprop in MLPs, but can approximate automatic differentiation along arbitrary computation graphs. This means that in theory there exist potentially biologically plausible algorithms for differentiating through arbitrary programs, utilizing only local connectivity. Moreover, in class of models which we call parameter-linear which includes many current machine learning models, the required update rules are Hebbian, raising the possibility that a wide range of current machine learning architectures may be faithfully implemented in the brain, or neuromorphic hardware.
In this paper we provide two main contributions. (i) We show that predictive coding converges to automatic differentiation across arbitrary computation graphs. (ii) We showcase this result by implementing three core machine learning architectures (CNNs, RNNs, and LSTMs) in a predictive coding framework which utilises only local learning rules and mostly Hebbian plasticity.
2 PREDICTIVE CODING ON ARBITRARY COMPUTATION GRAPHS
OL
0L dv aL
OL 0v
aL
OL dv
dv, dvo
dv,
0v22 dv
0v3 0v2
V/
VI
V/2
T
di,
di²
0i
1/-
Eg
el
v
e dv
62
0/2
Ez
63
ōvo
dv2
1/0
V11
1/2
03
12
T
60
El
E3
Figure 1: Top: Backpropagation on a chain. Backprop proceeds backwards sequentially and explicitly computes the gradient at each step the chain. Bottom: Predictive coding on a chain. Predictions, and prediction errors are updated in parallel using only local information.
Predictive coding is an influential theory of cortical function in theoretical and computational neuroscience. Central to the theory is the idea that the core function of the brain is to minimize prediction errors between what is expected to happen and what actually happens. Predictive coding
2

---

Under review as a conference paper at ICLR 2021
views the brain as composed of multiple hierarchical layers which predict the activities of the layers below. Unpredicted activity is registered as prediction error which is then transmitted upwards for a higher layer to process. Over time, synaptic connections are adjusted so that the system improves minimizing prediction error. Predictive coding possesses a wealth of empirical support (Friston 20033
2005; |Boaac 2017: Whittington and Bogacz, 2019) and offers single mechanism that accounts for diverse perceptual phenomena such as repetition-suppressinn (Auksztulewicz and Friston| 2016). end-
stopping (Rao and Ballard 1999), bistable perception (Hohwy et al. 2008 Weilnhammer et al. 2017)
and illusory motions (Lotter et al. 2016: Watanabe et 2018). 8). and even attentional modulation of neural activity (Feldman and Friston
2010 Kanai et al. 2015). Moreover, the central role of top-down predictions is consistent with the ubiquity, and importance of, top-down diffuse connections between cortical areas, Predictive coding consistent with many known aspects of neurophysiology, and has been translated into biologically plausible process theories which define candidate cortical
microcireuits which can implement the algorithm. (Spratling 2008 Bastos et al., 2012 Kanai et al. 2013 Shipp 2016).
In previous work, predictive coding has always been conceptualised as operating on hierarchies of layers (Bogacz, 2017; Whittington and Bogacz 2017). Here we present generalized form of predictive coding applied to arbitrary computation graphs. A computation graph g = {E. V} is a directed acyelic graph (DAG) which can represent the computational flow of essentially any program or computable function as a composition of elementary functions. Fach edge e; E E of the graph corresponds to an intermediate step- - the application of an elementary function while each vertex 2/ E V is an intermediate variable computed by applying the functions of the edges to the values of their originating vertices. In this paper, i; denotes the vector of activations within a layer and we denote the set of all vertices as {v, Effectively, computation flows 'forward' from parent nodes to all their children through the edge functions until the leaf nodes give the final output of the program as whole (see Figure and 2 for an example). Given a target T and loss function L = 9(T, Vout), the graph's output can be evaluated and, and if every edge function is differentiable, automatic differentiation can be performed the computation graph.
Predictive coding can be derived elegantly as a variational inference algorithm under a hierarchical Gaussian generative model (Friston 2005 Buckley et al. | (017)). We extend this approach to arbitrary computation graphs in a supervised setting by defining the inference problem to be solved as that of in- ferring the vertex value U; of each node in the graph given fixed start nodes to (the data), and end nodes UN (the targets). We define a generative model which parametrises the value of each vertex given the feedforward prediction of its parents, p({r;)) = P(un UN) = II N (o,l)P((s))
D and a factorised, variational posterior Q((x: Huo. uN) = Q(v ... - uN) II, N (c:lP(( C(e)). where P(()) denotes the set of parents and C(v,) denotes the set of children of given node Us From this, we can define suitable objective functional, the variational free-energy (VFE). which acts as an upper bound on the divergence between the true and variational posteriors.
F = KL[(((m un.uo, [[x]) 2 KL[(Q(U).... (ww=)))o,wx))lp((
uN)]
N
0
(1) Under Gaussian assumptions for the generative model p((x;)) = II," N(;; ú,, E,), and the vari- ational posterior (((+:)) = II (V,)), where the 'predictions' é, = f(P(c:): 0,) are defined as the feedforward value of the vertex produced by running the graph forward, and all the precisions, or inverse variances, XX_1 are fixed at the identity,
write F simply sum of prediction errors (see Appendix D or (Friston) 2003) Bogacz| [017 Buckley et al. (017)) for full derivations), with the prediction errors defined as e, 2 d; These prediction errors play core role in the framework and, in the biological process theories (Friston) 2005 Bastos et al. 2012). are generally an upper bound on the divergence between true and approximate posteriors, by minimizing F. we reduce this divergence, thus improving the quality of the variational posterior and approximating exact Bayesian inference. Predictive coding minimizes F by employing the Cauchy method of steepest descent to set the
considered be represented by a distinct population of "error units' Since
'This includes the prior p(vo), which simply has no parents.
3

---

Under review as a conference paper at ICLR 2021
dynamics of the vertex variables o gradient descent directly on F Bogacz 2017).
du;
OF
6 2ig
Σ
dc.
(2)
jEC((L)
The dynamics of the parameters of the edge functions 0 such that 6, - ((P(e)); 0), can also be derived as gradient descent on F. Importantly these dynamics require only information (the current vertex value, prediction error, and prediction errors of child vertices) locally available at the vertex.
d0: 0F 00:
ci 00,
dt 00:
To run generalized predictive coding in practice on given computation graph g = {,, VJ. we augment the graph with error units E & to obtain an augumented computation graph 9 {E. V, E). The predictive coding algorithm then operates in two phases a feedforward sweep and backwards iteration phase. In the feedforward sweep, the augmented computation graph run forward obtain the set of predictions {v, J and prediction errors {c,} {or ) for every vertex. Following Whittington and Bogacz (2017). to achieve exact equivalence with the backprop gradients computed on the original computation graph, we initialize v, û; in the initial feedforward sweep that the output error computed by the predictive coding network and the original graph are identical.
In the backwards iteration phase, the vertex activities {t; and prediction errors { are updated with Equation 2 for all vertices in parallel until the vertex values converge to a minimum of F. After convergence the parameters Whittington and Bogacz (2117), that the predictions at each layer are fixed at the values assigned during the feedforward pass throughout the optimisation of the vs. We call this the fixed-prediction assumption. In effect, by removing the coupling between the vertex activities of the parents and the prediction at the child, this assumption separates the global optimisation problem into a local one for each vertex. We implement these dynamics with simple forward Euler integration scheme so that the update rule for the vertices became uf ++ + of 1) aay where 7j is the step-size parameter. Importantly,
updated according Equation 3. Note we also assume, following
dF
if the edge function linearly combines the activities and the parameters followed by an elementwise nonlinearity- condition which we call 'parameter-linest
then both the update rule for the vertices (Equation 2) and the parameters (Equation 3) become Hebbian. Specifically, the update rules for the vertices and weights become du E E, e f (0,0)) and dé. dt (0: (0,")));" respectively.
2.1 APPROXIMATION TO BACKPROP
Here we show that at the equilibrium of the dynamics, the prediction errors e converge to the correct @L and consequently the parameter updates (Equation 3) become precisely
backpropagated gradients der those of a backprop trained network. Standard backprop works by computing the gradient of a vertex as the sum of the gradients of the child vertices. Beginning with the gradient of the output vertex al it recursively computes the gradients of vertices deeper in the graph by the chain rule: dug
ÖL Oyg
ÖL dug Ovg dr,
Σ
(4)
J=C(e:)
In comparison, in our predictive coding framework, at the equilibrium point di = 0) the prediction errors Ef become,
Fo
G = Σ
(5)
Dug
jEC(v.)
Importantly, this means that the equilibrium value of the prediction error at a given vertex (Equation 5) satisfies the same recursive structure as the chain rule of backprop (Equation 4). Since this relationship is recursive, all that is needed for the prediction errors throughout the graph to converge to the backpropagated derivatives is for the prediction errors at the final layer to be equal to the output ØL To see this explicitly, consider a -quree-uurre--rrror loss function 2 at the gradient: et, Bez.
"While the mean- squared-eree loss function fits most nicely with the Gaussian generative model, other loss functions can be used in practice. If the loss function can be represented as a log probability distribution, then the generative model can be amended to simply set the output distribution to that distribution. If not, then there is no fully consistent generative model (although all nodes except the output remain Gaussian). but the algorithm will still work in practice. See (Figure 6) in Appendix for results for CNNs trained with crossentropy loss.
4

---

Under review as a conference paper at ICLR 2021
Algorithm I: Generalized Predictive Coding
Data: Dataset D (X, L), Augmented Computation Graph 9 { V. E), inference learning rate M/., weight learning rate Tjo
begin
For each minibatch in the dataset
!
for
(a, L) do
/+ Fix start of graph to inputs
6 x
/* Forward pass to compute prediction
for 0z VVdo
is + f(P(())):
/* Compute output error
" + --
/+ Begin backwards iteration phase of the descent on the free energy
while not converged do
for (0,, 4) E G do
/* Compute prediction errors
6 + 0 0z
/* Update the vertex values
0 [+1 + Th de
/ Update weights equilibrium for 0, E E do
0+ + 0 +
dF ~0?
output layer = = - )z with T as a vector of targets, and defining e, = T - -z. We then consider the equilibrium value of the prediction error unit penultimate vertex CL-1- By Equation 5, we can see that at equilibrium,
d0L
düg
(T (12)
OvI--1
since, (T'-
-z)
aL B6,
we can then write,
OL 001. 0L
L Dûg :VL-1
OUL-1
Thus the prediction errors of the penultimate nodes converge to the correct backpropagated gradient. Furthermore, recursing through the graph from children to parents allows the correct gradients be computed"] Thus, by induction, we have shown that the fixed points of the prediction errors of the global optimization correspond exactly the backpropagated gradients. Intuitively, we imagine the computation-grap as chain and the error as 'tension' in the chain, backprop loads all the tension at the end (the output) and then systematically propagates it backwards. Predictive coding, however, spreads the tension throughout the entire chain until reaches an equilibrium where the amount of tension at each link is precisely the backpropagated gradient. The full algorithm for training the predictive coding network is explicitly set out in Algorithm 1. Inference is just forward pass through the network, and is identical to the corresponding ANN.
SSoe subtlety is needed here since fL-1 may have many children which each contribute to the loss. However, these different paths together the node Ug-1, thus propagating the correct gradient backwards.

---

Under review as a conference paper at ICLR 2021
By similar argument, it is apparent that the dynamics of the parameters 6, as a gradient descent on F also exactly match the backpropagated parameter gradients.
d0; dF
de.
dt
d0,
d0,
dL
dů: dL
dēg
d0:
d0,
Which follows from the fact that e
de
and that
de: dā;
de,
db
a9.
RELATED WORK
A number of recent works have tried to provide biologically plausible approximations to backprop.
The requirement of symmetry between the forwards and backwards weights has been questioned by
Lillicrap et al. (2016) who show that random fixed feedback weights suffice for effective learning.
Recent additional work has shown that learning the backwards weights also helps (Amit) 2019);
Akrout et al. 2019). Several schemes have also been proposed to approximate backprop using only
local learning rules and/or Hebbian connectivity. These include target-prop (Lee al. 2015) which approximate the backward gradients with trained inverse functions, but which fails to asymptotically compute the exact backprop gradients, and contrastive Hebbian (Seung. 2003 Scellier and Bengio et al. 2018) approaches which do exactly approximate backprop, but which require two separate learning phases and the storing of information across successive phases. There are also dendritic error theories (Guerguiev et al.] 2017: Sacramento et al., 2018) which are computationally similar to predictive coding (Whittington and Bogacz, 2019: Lillicrap et al. 2020). WWhittington and Bogacz (2017) showed that predictive coding can approximate backprop in MLP models, and demonstrated comparable performance on MNIST. We advance upon this work by extending the proof arbitrary computation graphs, enabling the design of predictive coding variants of a range of standard machine learning architectures, which we show perform comparably to backprop on considerably more difficult tasks than MNIST. Our algorithm evinces asymptotic (and in practice rapid) convergence to the exact backprop gradients, does not require separate learning phases, and utilises only local information and largely Hebbian plasticity.
4 RESULTS
4.1 NUMERICAL RESULTS
To demonstrate the correctness of our derivation and empirical convergence to the true gradients, we present a numerical test in the simple scalar case, where we use predictive coding to derive the gradients of arbitrary, highly nonlinear test function ez tan (an())))) + sin(ug) where 0 is arbitrary parameter. For our tests, we set Up to 5 and 0 to 2. The computation graph for this function presented in Figure Although simple, this a good test of predictive coding because the function is highly nonlinear, and its computation graph does not follow a simple layer structure but includes some branching. An arbitrary target of T 3 was
the output and the gradient of the loss L= (vy T)² with respect to the input was computed by predictive coding. We show (Figure 2) that the predictive coding optimisation rapidly converges to the exact numerical gradients computed by automatic differentiation, and that moreover this optimization is very robust and can handle even exceptionally high learning rates (up to 0.5) without divergence.
In summary, we have shown and numerically verified that at the equilibrium point of the global free-energy F on an arbitrary computation graph, the error units exactly equal the backpropagated gradients, and that this descent requires only local connectivity, does not require a separate phases or a sequential backwards sweep, and the case of parameter- linear functions, requires only Hebbian plasticity. Our results provide straightforward recipe for the direct implementation of predictive coding algorithms to approximate certain computation graphs, such as those found in common machine learning algorithms, in potentially biologically plausible manner. Next, we showcase this capability by developing predictive coding variants of core machine learning architectures convolutional neural networks (CNNs) recurrent neural networks (RNNs) and LSTMs (Hochreiter and Schmidhuber 1997), and show performance comparable with backprop on tasks substantially more challenging than MNIST.

---

Under review as a conference paper at ICLR 2021
9/
de
dec
tont -3
H)
Po tankg))
1/
sindia)
M1
(-)
sint
dhg
dec
Leaming Comparison
Log Prediction
30
Mal sup
10
mvg Hole
Mone Dowggene
20
Log Divergonco
10"
10"
10"
0
500
Vanational Naration
Variaiional dwatio
Figure 2: Top: The computation graph of the nonlinear test function uz. tan(VOng sin( Bottom: graphs of the log mean divergence from the true gradient and the divergence for different learning rates. Convergence the exact gradients exponential and robust high learning rates.
4.2 PREDICTIVE CODING CNN, RNN, AND LSTM
CIFAR CNN performance
SVHN CNN performance
CIFAR100 CNN performance
0.
Pracico tura acorsacy
Precidive
Bidiprop hat acouncy
Phocaliv oud hull acoussy
0.90
Becopap acousey
0.25
Accuracy
0.86
Accuracy
0.7
Accuracy
0.23
0.6
0.15
0.5
010
0.70
aacove x: Sedichee
0.4
0.5
0.86
10
40 Epoch
Epoch
Epoch
Figure 3: Training and test accuracy plots for the Predictive Coding and Backprop CNN on WVHN,CIFAR10 and CIFARI0 dataest over seeds. Performance is largely indistinguishable. Due to the need to iterate the us until convergence, the predictive coding network had roughly a 100x greater computational cost than the backprop network.
First, we constructed predictive coding CNN models (see Appendix B for full implementation details). In the predictive coding CNN, each filter kernel was augmented with 'error maps' which measured the difference between the forward convolutional predictions and the backwards messages. Our CNN was composed of a convolutional layer, followed by max-pooling layer, then two further convolutional layers followed by 3 fully-connected layers. We compared our predictive coding CNN to backprop-trained CNN with the exact same architecture and hyperparameters. We tested our models on three image classification datasets significantly more challenging than MNIST SVHN, CIFAR10 10. and CIFAR100. SVHN is digit recognition task like MNIST, but has more naturalistic backgrounds, is colour with continuously varying inputs and contains distractor digits. CIFAR 10 and CIFAR 100 are large image datasets composed of RGB 32x32 images. CIFAR10 10 has
7

---

Under review as a conference paper at ICLR 2021
10 classes of image, while CIFARI00 is substantially more challenging with 100 possible classes. In general (Figure 3). performance was identical between the predictive coding and backprop CNNs and comparable to the standard performance of basic CNN models on these datasets, Moreover, the predictive coding gradient remained close to the true numerical gradient throughout training.
RNN Accuracies
LSTM Accuracies
0.30
0.30
0.25
0.55
Acsurecy
0.20
Anourecy
0.00
0.15
0.10
0.10
0.05
0.5
0.00
0.00
1000
2000
3000
4000
0000
0
1000
2000
3030
4003
Gpochs
Gpochs
Figure 4: Test accuracy plots for the Predictive Coding and Backprop RNN and LSTM on their respective tasks, averaged over 5 seeds. Performance is again indistinguishable from backprop.
We also constructed predictive coding RNN and LSTM models, thus demonstrating the ability of predictive coding to scale to aan--arrmeter-linaal branching, computation graphs. The RNN was trained character-level name classification task, while the LSTM was trained on next-character prediction task on the full works of Shakespeare. Full implementation details can be found in Appen- dices B and C. LSTMs and RNNs are recurrent networks which are trained through backpropagation through time (BPTT). BPTT simply unrolls the network through time and backpropagates through the unrolled graph. Analogously we trained the predictive coding RNN and LSTM by applying predictive coding to the unrolled computation graph. The depth of the unrolled graph depends heavily on the sequence length, and in our tasks using sequence length of 100 we still found that predictive coding evinced rapid convergence to the correct numerical gradient, and that the performance was approximately identical to the equivalent backprop- trriked networks (Figure 3), thus showing that the algorithm is scalable even to very deep computation graphs.
5 DISCUSSION
We have shown that predictive coding provides a local and potentially biologically plausible approxi- mation to backprop on arbitrary, deep, and branching computation graphs. Moreover, convergence to the exact backprop gradients is rapid and robust, even in extremely deep graphs such as the unrolled LSTM. Our algorithm is fully parallelizable, does not require separate phases, and can produce equivalent performance to backprop in core machine-leaming architectures. These results broaden the horizon of local approximations to backprop by demonstrating that they can be implemented on arbitrary computation graphs, not only simple MLP architectures. Our work prescribes straight- forward recipe for backpropagating through any computation graph with predictive coding using only local learning rules. In the future, this process could potentially be made fully automatic and translated onto neuromorphic hardware. Our results also raise the possibility that the brain may implement machine-learning type architectures much more directly than often considered. Many lines of work suggest a close correspondence between the representations and activations of CNNs and activity in higher visual areas Yaamins et al. 20144) Tacchetti et al. 2017 [iicknberg et al., 2017] Chaligh-Razav and Kriegeskorte [2014] Lindsay] (2220), for instance, and this similarity may be found to extend to other machine learning architectures.
It is important to note that predictive coding, as advanced here, still retains some biologically implausible features. Although using only local and Hebbian updates, the predictive coding algorithm still requires identical forward and backwards weights, as well as mandating very precise one- to-one connectivity structure between value neurons 0; and error neurons 6,, However, recent work (Millidge et al., 2020) has begun to show that these implausibilities can be relaxed using learnable backwards weights instead of requiring weight symmetry, and allowing for learnable dense connectivity between value and error neurons, without harm performance in simple MLP settings. An additional limitation to the biological plausibility of our method is the Exxed-prediction assumption, which requires that the feedforward pass values be somehow stored during the hackwards
8

---

Under review as a conference paper at ICLR 2021
iteration phase. In biological neurons this could potentially be implemented by utilizing synaptic mechanisms for maintaining information over short periods, such as cligibility traces, or alternatively through synchronised phase locking (Buzsaki 2006). Alternatively, it is important to note that this exxedprediction assumption is only required for exact convergence to backprop, and predictive coding networks have been shown be able to attain strong discriminative classification performance without it (Whittington and Bogacz (017).
Although we have implemented three core machine learning architectures as predictive coding networks, we have nevertheless focused on relatively small and straightforward networks and thus both our backprop and predictive coding networks perform below the state of the art on the presented tasks. This is primarily because our focus was on demonstrating the theoretical convergence between the two algorithms. Nevertheless, we believe that due to the generality of our theoretical results, "scaling up' the existing architectures to implement performance-matched predictive coding versions of more advanced machine learning architectures such as resnets (He et al.) 2016). GANs (Goodfellow et al. 2014), and transformers (Vaswani et al. 2017) should be relatively straightforward.
In terms of computational cost, one inference iteration in the predictive coding network is about as costly as a backprop backwards pass. Thus, due to using 100-200 iterations for full convergence, our algorithm is substantially more expensive than backprop which limits the scalability of our method. However, this serial cost is misleading when talking about highly parallel neural architectures. In the brain, neurons cannot wait for a sequential forward and backward sweep. By phrasing our algorithm as global descent, our algorithm is fully parallel across layers. There is no waiting and phases to be coordinated. Each neuron need only respond to its local driving inputs and downwards error signals. We believe that this local and parallelizable property of our algorithm may engender the possibility of substantially more efficient implementations on neuromorphic hardware (Furber et al.) 2014; Merolla et al., 2014; Davies et al. 2018), which may ameliorate much of the computational overhead compared to backprop. Future work could also examine whether our method is more capable than backprop of handling the continuously varying inputs the brain presented with in practice, rather than the artificial paradigm of being presented with a series of i..dd datapoints.
Our work also reveals a close connection between backprop and inference. Namely, the recursive computation of gradients is effectively a by-product of a variationallinnfernnce algorithm which infers the values of the vertices of the computation graph under a hierarchical Gaussian generative model. While the deep connections between stochastic gradient descent and inference in terms of Kalman filtering (Ruck et al. 1992, Ollivier 2019) or MCMC sampling methods (Chen et al.
2014. Mandt et al. 2017) is known, the relation between recursive gradient computation itself and variational inference is underexplored except in the case of a single layer (Amari) 1995). Our method can provide principled generalisation of backprop through the inverse-variane Σ parameters of the Gaussian generative model. These parameters weight the relative contribution of different factors to the overall gradient by their uncertainty, thus naturally handling the case of backprop with differentially noisy inputs. Moreover, the Σ- parameters can be learnt as a gradient descent on جك -, - Σ - E This specific generalisation is afforded by the Gaussian form of the generative model, however, and other generative models may yield novel optimisation algorithms able quantify and handle uncertainties throughout the entire computational graph.
F: 0N 4F
REFERENCES
Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. In Advances in Neural Information Processing Systems, pages 974-982, 2019.
Shun-Ichi Amari. Information geometry of the em and em algorithms for neural networks. Neural networks, 8(9):1379-1408 1995.
Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in computa- tional neuroscience, 13:18, 2019.
Brandon Amos and Denis Yarats. The differentiable cross-entropy method. arXiv preprint arXiv: 12830, 2019.
Ryszard Auksztulewicz and Karl Friston. Repetition suppression and its contextual determinants in predictive coding. cortex, 80: 1400 2016.
9

---

Under review as a conference paper at ICLR 2021
Andre M Bastos, W Martin Usrey, Rick A Adams, George R Mangun, Pascal Fries, and Karl J Friston. Canonical microcircuits for predictive coding. Neuron, 76(4):695-711 2012.
Atilm Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey. The Journal of Machine Learning Research, (8())5555--637, 2017.
Matthew James Beal et al. Variational algorithms for approximate Bayesian inference. university of London London, 2003.
Yoshua Bengio and Asja Fischer. Early inference in energy-hased models approximates back- propagation. arXiv preprint arXiv:1510.02777 2015.
Yoshua Bengio, Thomas Mesnard, Asja Fischer, Saizheng Zhang, and Yuhuai Wu. Stdp-compatible approximation of backpropagation in an meerg--bssed model. Neural computation, 29(3):555-577,
2017.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, (2:51888859877, 2017.
Rafal Bogacz. A tutorial on the free-energy framework for modelling perception and learning. Journal of mathematical psychology, 76:198 198-211, 2017.
Christopher L Buckley, Chang Sub Kim, Simon McGregor, and Anil K Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55-79, 2017.
Gyorgy Buzsaki. Rhythms of the Brain. Oxford University Press, 2006.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in neural information processing systems, pages 6571-6583, 2018.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pages 1683-1691, 2014.
Francis Crick. The recent excitement about neural networks. Nature, 337 1989.
Mike Davies, Narayan Srinivasa, Tsung-Har Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1) (881):82-99) -9, 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis Wyffels. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, 13:6, 2019.
Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152: 184-194, 2017.
Harriet Feldman and Karl Friston. Attention, uncertainty, and free-energy. Frontiers in human neuroscience, 4:215, 2010.
Karl Friston. Learning and inference in the brain. Neural Networks, 1325511325-332, 2003.
Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences. 360( (4466)88155 (1556)815-836 836, 2005.
Karl Friston. Hierarchical models in the brain. PLOS computational biology, 4(11), 2008
Steve B Furber, Francesco Galluppi, Steve Temple, and Luis Plana. The spinnaker project. Proceedings of the IEEE, 102(5):652-665, 2014.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pages 2672-2680, 2014.
10

---

Under review as a conference paper at ICLR 2021
Andreas Griewank al. On automatic differentiation. Mathematical Programming: recent develop- ments and applications, ((6):83-177, 1989.
Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites, Elife, 6:e22901, 2017.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245 2017.
Jeff Hawkins and Sandra Blakeslee. On intelligence: How a new understanding of the brain will lead to the creation of truly intelligent machines. Macmillan, 2007.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Eric Heiden, David Millard, and Gaurav Sukhatme. Real2sim transfer using differentiable physics. In Workshop on Closing the Reality Gap in Sim2real Transfer for Robotic Manipulation, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()):
1780- 1997.
Jakob Hohwy, Andreas Roepstorff, and Karl Friston. Predictive coding explains binocular rivalry: An epistemological review. Cognition, 108(3):687-711 2008.
Mike Innes, Alan Edelman, Keno Fischer, Chris Rackauckus, Elliot Saba, Viral B Shah, and Will Tebbutt. Zygote: differentiable programming system to bridge machine learning and scientific computing. arXiv preprint /r 2019.
Ryota Kanai, Yutaka Komura, Stewart Shipp. and Karl Friston. Cerebral hicrarchies: predictive pro- cessing, precision and the pulvinar. Philosophical Transactions of the Royal Society B: Biological Sciences, 370( (1670((66::2001111 40169, 2015.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Seyed-Mahdi Khaligh-Razav and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology, 10(11), 2014.
Alex Krizhevsky, Σlya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference on machine learning and knowledge discovery in databases, pages 498-515. Springer, 2015.
Qianli Liao, Joel Z Leibo, and Tomaso Poggio. How important is weight symmetry in backpropaga- tion? In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82-89, 2019.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 1-10, 2016.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, pages 1-12, 2020.
Grace Lindsay. Convolutional neural networks as a model of the visual system: past, present, and future. Journal of Cognitive Neuroscience, pages 1-15, 2020.
11

---

Under review as a conference paper at ICLR 2021
Seppo Linnainmaa. The representation of the cumulative rounding error of algorithm taylor expansion of the local rounding errors. Master's Thesis (in Finnish). Univ. Helsinki, pages 6-7, 1970.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint
2016.
Stephan Mandt, Matthew D Hoffman, and David Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, ((1))4873-4900) 2017.
Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668-677 2014.
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher L Buckley. Relaxing the constraints on predictive coding models. arXiv preprint arXiv:2010.01047, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-33, 2015.
Masashi Okada, Luca Rigazio, and Takenobu Aoshima. Path integral networks: End-to-end differen- tiable optimal control. arXiv preprint arXiv: 1706,09997 2017.
Yann Ollivier. The extended kalman filter is a natural gradient descent in trajectory space. arXiv preprint arXiv:1901.006600 2019.
Yann Ollivier, Corentin Tallec, and Guillaume Charpiat. Training recurrent networks online without backtracking. arXiv preprint WiXiv:150.007688 2015.
Alexander Ororbia, Ankur Mali, Lee Giles, and Daniel Kifer. Continual learning of recurrent neural networks by locally aligning distributed representations. IEEE Transactions Neural Networks and Learning Systems, 2020.
Avik Pal. Raytracer. jl: A differenniable renderer that supports parameter optimization for scene reconstruction. arXiv preprint arXiv:1907.07/98 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Chris Rackauckas, Mike Innes, Yingbo Ma, Jesse Bettencourt, Lyndon White, and Vaibhav Dixit. Diffeqflux. jl-a julia library for neural differential equations. arXiv preprint 192.:090 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8)9, 2019.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
Jarrett Revels, Miles Lubin, and Theodore Papamarkou. Forward-mode automatic differentiation in julia. arXiv preprint (XXX::667.00892 2016.
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, al. A deep learning framework for neuroscience. Nature neuroscience, 22 177177777111770, 2019.
Dennis W. Ruck, Steven K. Rogers, Matthew Kabrisky, Peter S. Maybeck, and Mark E. Oxley. Comparative analysis of backpropagation and the extended kalman filter for training multilayer perceptrons. JEEE Transactions on Pattern Analysis & Machine Intelligence, (6)686-691, 1992.
David E Rumelhart and David Zipser. Feature discovery by competitive learning. Cognitive science, (1):75-112, 1985.
12

---

Under review as a conference paper at ICLR 2021
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pages 8721-8732, 2018.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Fronfiers in computational neuroscience, 11 11:24, 2017.
Benjamin Scellier, Anirudh Goyal, Jonathan Binas, Thomas Mesnard, and Yoshua Bengio. Gener- alization of equilibrium propagation vector field dynamics. arXiv preprint arXiv: 1808.888777 2018.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08663 2019.
H Sebastian Seung. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron, 1066))1063-10733 2003.
Stewart Shipp. Neural elements for predictive coding. Frontiers in psychology, 7:1792, 2016.
David Silver, Julian Schrittiesser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of without human knowledge. Nature, 550(7676)3354 -399 2017.
Michael W Spratling. Reconciling predictive coding and biased competition models of cortical function. Frontiers in computational neuroscience, 2:4, 2008.
Jochen J Steil. Backpropagatioo-ddcoorre 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541), volume 2, pages 843 848. IEEE, 2004.
online recurrent learning with o (n) complexity, In
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanct, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
Andrea Tacchetti, Leyla Isik, and Tomaso Poggio. Invariant recognition drives neural representations of action sequences. PLoS computational biology, 13(12)xe005885, 2017.
Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. a7Xiv:1702.05043, 2017.
arXiv preprint
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv: 09883, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6688999 6008, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnekki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-3355 2019.
Biji Watanabe, Akiyoshi Kitaoka, Kiwako Sakamoto, Masaki Yasugi, and Kenta Tanaka. Illusory motion reproduced by deep neural networks trained for prediction. Frontiers in psychology, 9:345, 2018.
Veith Weilnhammer, Heiner Stuke, Guido Hesselmann, Philipp Sterzer, and Katharina Schmack. A predictive coding account of bistable perception-a a model-based fmri study. PLoS computational biology, 13(5)ee 1005536, 2017.
Paul Werbos. Applications of advances in nonlinear sensitivity analysis. System modeling and optimization, pages 762-770. Springer, 1982.
13

---

Under review as a conference paper at ICLR 2021
James CR Whittington and Rafal Bogacz. An approximation of the enror backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5): 1229 1229-1262, 2017.
James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in cogmitive sciences, 2019.
Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280) 1989.
Daniel LK Yamins, Ha Hong. Charles F Cadicu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance iifoommance-opmmizeed hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):619-8624, 2014.
APPENDIX A: PREDICTIVE CODING CNN IMPLEMENTATION DETAILS
The key concept in CNN is that of an image convolution, where small weight matrix is 'slid' (or convolved) across an image to produce an output image. Each patch of the output image only depends on relatively small patch of the input image. Moreover, the weights of the filter stay the same during the convolution, so each pixel of the output image is generated using the same weights. The weight sharing implicit in the convolution operation enforces translational invariance, since different image patches are all processed with the same weights.
The forward equations of convolutional layer for a specific output pixel
k=itf 1=j+f
R. = Σ Σ
0g.
k= f ドーjーン
Where Uay is the (i,j)tth element of the output, Fı, is the element of the input image and 0g.1 is an weight element of a feature map. To setup a predictive coding CNN, we augment each intermediate Iy and v; with error units 6; of the same dimension the output of the convolutional layer.
Predictions i are projected forward using the forward equations. Prediction errors also need to be transmitted backwards for the architecture to work. To achieve this must have that prediction emros are transmitted upwards by a bbackwrrds convolution'. We thus define the backwards prediction errors é follows:
i+f
i+f
E = Σ
Σ
O
k=
Where is map zero-padder aero-padded to ensure the correct convolutional output size. Inference in the predictive coding network then proceeds by updating the intermediate values each layer as follows:
du: di
el
Since the CNN is also parameter- linee,,, weights be updated using the simple Hebbian rule of the multiplication of the pre and post synaptic potentials.
dB,
Σ b
el U1-
dt
There an additional biological implausibility here due to the weight sharing the CNN. Since the same weights are copied for each position the image, the weight updates have contributions from all aspects of the image simultaneously which violates the locality condition. A simple fix for this, which makes the network scheme plausible is to simply give each position the image filter with separate weights, thus removing the weight sharing implicit the CNN. effect this gives each patch of pixels local receptive field with its set of weights. The performance and scalability
14

---

Under review as a conference paper at ICLR 2021
9VHNN losses
CIFAN losses
CIF CIFAR100 losses
13
1-
3.0
20
Borations
Bdtotions
Incatore
Figure 5: Training loss plots for the Predictive Coding and Backprop CNN on SIFARI and CIFAR10 dataset over 5 seeds.
of such a locally connected predictive coding architecture would be an interesting avenue for future work, as this architecture has substantial homologies with the structure of the visual cortex.
In our experiments we used relatively simple CNN architecture consisting of one convolutional layer of kernel size 5, and a filter bank of 6 filters. This was followed by max pooling layer with a (2,2) kernel and a further convolutional layer with a (5.5) kernel and filter bank of 16 filters. This was then followed by three fully connected layers of 200, 150, and 10 (or 100 for CIFARION 100) output units. Each convolutional and fully connected layer used the relu activation function, except the output layer which was linear. Although this architecture far smaller than state of the art for convolutional networks, the primary point of our paper was to demonstrate the equivalence of predictive coding and backprop. Further work could investigate scaling up predictive coding to more ttate-of-the-ar architectures.
Our datasets consisted of 32x32 RGB images. We normalised the values of all pixels of each image to lie between O and 1, but otherwise performed other image preprocessing. We did not use data augmentation of any kind. We set the weight learning rate for the predictive coding and backprop networks 0.0001. A minibatch size of 64 was used. These parameters were chosen without any detailed hyperparameter search and so are likely suboptimal. The magnitude of the gradient updates was clamped to lie between -50 and 50 in all of our models. This was done to prevent divergences, as occasionally occurred in the LSTM networks, likely due to exploding gradients.
The predictive coding scheme converged the exact backprop gradients very precisely within 100 inference iterations using an inference learning rate of 0.1. This gives the predictive coding CNN approximately a 100x computational overhead compared to backprop. The divergence between the true and approximate gradients remained approximately constant throughout training, as shown by Figure 5, which shows the mean divergence for each layer of the CNN over the course of an example training run on the CIFARIO 10 dataset. The training loss of the predictive coding and backprop networks for SVHN, CIFAR1O and CIFAR100 are presented in Figure 4.
While the experiments in the main paper all used the mean- suuu---uuueed-erro loss function, it is also possible to use alternative loss functions. In Figure 6, we show performance of the CNN on CIFAR and SVHN datasets is also very close to backprop when trained with multi-class cross-entropy loss L L= Σ, T; In UL;- In this case the output layer used a softmax function as its nonlinearity, to ensure that the logits passed the cross-entropy loss were valid probabilities. The cross-entropy loss is also straightforward to fit into the predictive coding framework since the gradient with respect to the pre-activations of the output is also just the negative prediction error des T Uz. although the softmax function itself may be challenging to implement neurally since normalisation coefficient requires of the exponentiated activities of all neurons in a layer. Nevertheless, this demonstrates that predictive coding can approximate backprop for any given loss function, not simply mean-square -quare--ururrrrr.
non-local as its'
APPENDIX B: PREDICTIVE CODING RNN
The computation graph on RNNs is relatively straightforward. We consider only a single layer RNN here although the architecture can be straightforwardly extended to hierarchically stacked RNNs. An RNN is similar to a feedforward network except that it possesses an additional hidden state h which is maintained and updated over time function of both the current input and the previous hidden
15

---

Under review as a conference paper at ICLR 2021
Layer: : gradient livergence
Layer: gracient divergence
40ah disonie Not oue grsdeer:
0.0000
0.09
0.0000
0.4
0.004
0.2
0.002
0000
Conv Layer 1
(b) Conv Layer 2
Layer: gradient divergence
Layer gradient divergence
0.0000M
0.00000
3.5
0.00307
3.0
0.00006
0.00005
0.00008
0.00003
000002
80
20
Ezoch
Epoch
(c) FC Layer
(d) FC Layer 2
Mean divergence between true numerical and predictive coding backprops over course training. In general, the divergence appeared to follow largely random walk pattern, and was generally neglible. Importantly, the divergence not grow over throughout training, implying that errors from slightly incorrect gradients did not appear to compound.
16

---

Under review as a conference paper at ICLR 2021
crossentropy syhh test accuracies
crossentroyysshhn svhn accuracies
1.00
0.85
0.95
0.90
Accuracy
Accuracy
0.85
0.75
0.80
0.75
0.70
0.85
Scstanno
10
Iterations
Iterations
SVHN training accuracy
(a) SVHN test accuracy
crossentropy cifar accuracies
crossentropy _cffarcifarrre test accuracies
Enckane
0.550
3.8
Accuracy
0.525
3.7
0.450
0.425
3.4
Iberations
Iteracions
(b) CIFAR training accuracy
(c) CIFAR test accuracy
Training and test accuracies of the CNN network on the SVHN and CIFAR datasets using the cross-entropy loss. As be performance remains very close backprop, thus demonstrating that predictive coding al algorithm can be used with different loss functions, not just mcan-squared-crror.
17

---

Under review as a conference paper at ICLR 2021
state. The output of the network is function of h. By considering the RNN at single timestep obtain the following equations.
h f(0p/=1 + 0 - (00h)))
(8)
Where and g are elementwise nonlinear activation functions. And 0n. Or, fy are weight matrices for each specific input. To predict sequence the RNN simply rolls forward the above equations to generate new predictions and hidden states each timestep.
RNNs are typically trained through algorithm called backpropagation through time (BPTT) which essentially just unrolls the RNN into a single feedforward computation graph and then performs backpropagation through this unrolled graph. To train the RNN using predictive coding take the same approach and simply apply predictive coding to the unrolled graph.
It is important note that this is additional aspect of biological implausibility that we do not address in this paper. BPTT requires updates to proceed backwards through time from the end of the sequence to the beginning. Ignoring any biological implausibility with the rules themselves, this updating sequence clearly not biologically plausible as naively it requires maintaining the entire sequence of predictions and prediction emrors perfectly in memory until the end of the sequence, and waiting until the sequence ends before making any updates. There a small literature on trying to produce biologically plausible, or forward- howarddlooking approximations BPTT which does require updates to be propagated back through time (Williams and Zipser 1989 Lillicrap and Santoro 2019)) Steil 2004 Ollivier al., 2015 Tallec and Ollivier| (017). While this is a fascinating area, we do not address it in this paper. We are solely concerned with the fact that predictive coding approximates backpropagation on feedforward computation graphs for which the unrolled RNN graph is a sufficient substrate.
To learn predictive coding RNN, we first augment each of the variables hy and ye of the original graph with additional error units cb, and Ey,, Predictions Dr. hú are generated according to the feedforward rules (16). A sequence of true labels (Tr...rr) } is then presented to the network, and then inference proceeds by recursively applying the following rules backwards through time until convergence.
Eye L D1
Eh, hr his
dhg
dl Eh. Eyi 07
Ehe41 of
Upon convergence the weights are updated according to the following rules.
d0, v
T
(g(0)h
dt
Eye
00,,
d8
0f(0nh + Og T
Chu
dt
00,
t=
d0h
0f 0x. li7-1
dl EN.
00,,
Since the RNN feedforward updates are parameter-linear, these rules are Hebbian, only requiring the multiplication of pre and post-synaptic potentials. This means that the predictive coding updates proposed here are biologically plausible and could in theory be implemented in the brain. The only biological implausibility remains the BPTT learning scheme.
Our RNN was trained https://dowwlood. pytp:// pytorch.org/tutoriiai orggtutorial/daaa.ipp The RNN was presented with sequences characters representing names and had predict the national origin of the name French, Spanish, Russian, etc. The characters were presented to the network as one- hot-edd vectors without any embedding. The output categories were also presented one-hot vector. The RNN has hidden size of 256
simple character-level name-origin dataset which can be found here:
18

---

Under review as a conference paper at ICLR 2021
units. A tanh nonlinearity was used between hidden states and the output layer was linear. The network was trained on randomly selected name-category pairs from the dataset. The training loss for the predictive coding and backprop RNNs, averaged over 5 seeds is presented below (Figure 7).
RNN Losses
Backprop
Predicivee coding
0.4
Loss
0.3
0.0
1000
2000
Epochs
Figure 8: Training losses for the predictive coding and backprop RNN. As expected, they are effectively identical.
APPENDIX C: PREDICTIVE CODING LSTM IMPLEMENTATION DETAILS
dL dos
dL dows dews des
dog diy dog
dig de,
1y 4%
C++1
C,
dL dvg digg dva
dl dvy
dL dis dig dry digo dhin dii,
daay dys
dL des div
Vg = 1yl4
Ny tamblvy) tanh
dL dvg
dys
h
My m (1
Vi - v)) tank
V. ((0
dt. ding Gydvoo dyg
dL dhia dii drn
0,
i Inp
0-
0
dL dy dy dvip
dr. diig dls dry
dr. diia dos
dog dvy dui
1i 4, (
h
dag diry
y= mil
dL de
dT. du, dv, dx,
dvi dli
dL dy
X,
Y;
Figure 9: Computation graph and backprop learning rules for a single LSTM cell.
Unlike the other two models, the LSTM possesses a complex and branching internal computation graph, and thus a good opportunity to make explicit the predictive coding 'recipe' for approximating backprop arbitrary computation graphs. The computation graph for single LSTM cell is shown (with backprop updates) in Figure 8. Prediction for the LSTM occurs by simply rolling forward a copy of the LSTM cell for each timestep. The LSTM cell receives its hidden state h, and cell state c, from the previous timestep. During training we compute derivatives on the unrolled computation graph and receive backwards derivatives (or prediction errors) from the LSTT cell at time l 1.
19

---

Under review as a conference paper at ICLR 2021
The equations that specify the computation graph of the LSTM cell are as follows.
V1 h4 @
o (0,01)
V3 U₁
a (0mpp))) U5 tanh(0.v1)
= U4
V7 v3 + 26
Ug o (0.21) ) Ug tanh(v7)
U10 = Ug
y o (8g v10)
The recipe to convert this computation graph into predictive coding algorithm is straightforward. We first rewire the connectivity so that the predictions are set to the forward functions of their parents. then compute the errors between the vertices and the predictions.
iz h4 @ ae
a (0:01)
i3 C²U2
o U1 05 tanh(0.0h)
14 Ug
f7 v3 + V6
â a (0,01) ig = Lanh(w7)
U Ug Ug
ey = o Og U10)
61 V1 01
22
63 4/3 13
=
65 3/ 05
= 06
E7 07
1/7
= 0g
Eg 6i 0g
610 = U10 010
During inference, the inputs h4,r, and the output yt are fixed. The vertices and then the prediction errors are updated according to Equation 2. This recipe is straightforward and can easily extended to other more complex machine learning architectures. The full augmented computation graph, including the vertex update rules, is presented Figure 9.
Empirically, observed rapid convergence to the exact backprop gradients in the case of very deep computation graphs (as is an unrolled LSTM with sequence length of 100), Although conver- gence was slower than the case for CNNs lesser sequence lengths, it was still straightforward to achieve convergence to the exact numerical gradients with sufficient iterations.
Below plot the mean divergence between the predictive coding and true numerical gradients as a function of sequence length (and hence depth of graph) for a fixed computational budget of 200 iterations with inference learning rate of 0.05. As be seen, the divergence increases roughly
20

---

Under review as a conference paper at ICLR 2021
V4
dig
61
9
=
Hr
dis
Cg
dbg
He
v9t4
66
V6
My lonhtr)) tanh
Vo
dps
diia
Frul
2115
Cin des Mio
Mg 21
dis
Fin dis
V4
W/
tanh
V55
0 mmp
0g
0
10
h
des
duu
dat 19dt
dryp
T
Figure 10:
The LSTM cell computation graph augmented with error units, evincing the connectivity scheme of the predictive coding algorithm.
linearly with sequence length. Importantly, even with long sequences, the divergence is not especially large, and can decreased further by increasing the computational budget. As the increase is linear, we believe that predictive coding approaches should be scalable even for backpropagating through very deep and complex graphs.
We also plot the number of iterations required to reach a given convergence threshold (here taken be 0,005) as function of sequence length (Figure 11). We see that the number of iterations required increases sublinearly with the sequence length, and likely asymptotes at about 300 iterations. Although this a of iterations, the sublinear convergence nevertheless shows that the method can scale to even extremely deep graphs.
Our architecture consisted multiple stacked LSTM layers). The LSTM was trained on a next-character character-level prediction task. The dataset was the full works Shakespeare, downloadable from Tensorflow. The text was shuffled and split into sequences of 50 characters, which were fed to the LSTM one character at time. The LSTM was trained then to predict the next character, so as to ultimately be able to generate text. The characters cell-size of 1056 units. minibatch size of 64 was used and a weight learning rate of 0.0001 was used for both predictive coding and backprop networks. To achieve sufficient numerical convergence to the correct gradient, we used 200 variational iterations with an inference learning rate of 0.1. This rendered the predictive LSTM approximately 200x costly the backprop LSTM of the LSTM training loss for both predictive coding and backprop LSTMs, averaged over 5 random seeds, can be found below (Figure 12).
single LSTM layer (more complex architectures would consist of
presented as mne-hoo-eneoded vectors. The LSTM had hidden size and
A graph
APPENDIX D: DERIVATION OF THE FREE ENERGY FUNCTIONAL
Here derive in detail the form of the free-energy functional used in sections 2 and 4. We also expand upon the assumptions required and the precise form the generative model and variational density. Much of this material is presented with considerably more detail in Buckley et al. (2017). and more approachably in Bogacz (011)).
Given arbitrary computation graph with vertices {ya). which treat random variables. Here treat explicitly an important fact that we glossed over for notational convenience the introduction. The 8,8 which are optimized in the free-energy functional technically the mean parameters of
21

---

Under review as a conference paper at ICLR 2021
Gradient divergence against sequence length
0.00035
0.00030
0.00025
0.00020
0.00015
0.00010
0.00005
0.00000
100
200
300
400
500
Sequence Lengths
Figure 11: Divergence between predictive coding and numerical gradients function of sequence length.
Numerical iterations to convergence against sequence length
150
Ittrations before convergence
140
130
120
110
25
50
75
100
125
150
175
200
Sequence Length
Figure 12: Number of iterations to reach convergence threshold as function of sequence length.
the variational density Q Us o.) i.e. they represent the mean (variational) belief of the value of the vertex. The vertex values in the model, which we here denote as {y: ). are technically separate. However, due to our Gaussian assumptions, and the expectation under the variational density, in effect end up replacing the y with the v and optimizing the e;s, in the interests of space and notational simplicity we began as the v;s were variables in the generative model, but they are not. They parameters of the variational distribution.
22

---

Under review as a conference paper at ICLR 2021
LSTM Losses
3400
coding
3300
Loss
3000
2900
2800
2700
0
1000
2000
3000
4000
Epochs
Figure 13: Training losses for the predictive coding and backprop LSTMs averaged over 5 seeds. The performance of the two training methods is effectively equivalent.
Given input You and a target yvv (the multiple input and/or output case is straightforward general- ization). We wish to infer the posterior P(y2:N-11MM yN). ). We approximate this intractable posterior with variational inference. Variational inference proceeds by defining approximate posterior Q 1; 6) with some arbitrary parameters Φ. We then wish to minimize the KL divergence between the true and approximate posterior.
argmin 1:
vN)]
Although this KI. is itself intractable, since it includes the intractable posterior, we can derive tractable bound on this KL called the variational free-energy.
p KVI.N. Yo. UN);
[L (pp Mo. uN)] )] = KIL[Q(VI..VVIILIII 1)
p(yo- 9N)
(KL : d)I(p() y0)l + In In MN)
+ IKL[Q(INN: o) 0) Bo. UN)] < KLLO@ vx))
We define the negative free-energy -F IKL (Q wiw.ww-w)ll((..N-111 No. yN which is lower bound on the divergence between the true and approximate posteriors. By thus maximizing the negative free-energy (which is identical to the ELBO (Beal al., 20033 Blei al. (017)), equivalently minimizing the free-energy, we decrease this divergence and make the variational distribution a better approximation the true posterior.
To proceed further, it is necessary to define an explicit form of the generative model p(yo, V1:N-1, yN) and the approximate posterior Qi σ). In predictive coding, define hierarchical Gaussian generative model which mirrors the exact structure of the computation graph
N N(w ((P()) Ex);
p(Uo.N) = NNoo: To, Σo) II i=1
Where essentially each vertex yi is Gaussian with mean which is function of the prediction of all the parents of the vertex, and the parameters their edge-functions, Yo is effectively an "input-prion which is set throughout and ignored. The output vertices yn = T are set the target T.
We also define the variational density to be Gaussian with mean U1.N-1 and variance d1:N-1s but under mean field approximation, so that the approximation each node is independent of all others
23

---

Under review as a conference paper at ICLR 2021
(note the variational variance is denoted a while the variance of the generative model is denoted E. The lower-case o is not used to denote a scalar variable both variances can be multivariate but to distinguish between variational and generative variances)
N-1
II f=1
N(g us o,)
We now can express the free-energy functional concretely. First decompose it the sum of an energy and an entropy
-F II
V1.N-1
ln 1I.N-1.
Energy
Entropy
Then, taking the entropy term first, we can express it concretely in terms of normal distributions.
[In QVVI-N -1,N-1, 1,NN-1,0.N-11 = EQuay 11LN-191N
- uPix 1.
EQ - In det deTTo)))))) + BQima. (B) - 22)
20,
In det det(22To))) + 20:
N
In det d2t()
2
The entropy of a multivariate gaussian has simple analytical form depending only the variance. Next we turn to the energy term, which is more complex. To derive a clean analytical result, we must make a further assumption, the Laplace approximation, which requires the variational density be tightly peaked around the mean so the only non-neginn negligiie contribution to the expectation is from regions around the mean. This means that we can successfully approximate the approximate posterior with second-order Taylor expansion around the mean. From the first line onwards we ignore the p(an) and p(yx | (P(y)) which lie outside the expectation.
N
EQ -iliin--tthaa uith - In p(yo.N)] = pn + In pnypy/ +
EQu (((())
In plun P()x))
(qlnn((; Eq!
Oy: (v: v1)
df In p(u, (P(s))
Eel
(vi y1
dy?
N
8a In (P(x)) Oy?
Inp(u,1(())))
Where the second term the Taylor expansion evaluates to 0 since E B3 - v/ = (v) v.) = 0 and the third term contains the expression for the variance Eg v; 0;.
We can then write out the full Laplace-encooeee encoded free-energy as:
In (p(,)(P(u))) 82 In p(u; (P(s))
F
In det(Too))
Oyp
24

---

Under review as a conference paper at ICLR 2021
We wish to minimize F with respect to the variational parameters and o;. There is in fact closed- form expression for the optimal variational variance which can be obtained simply by differentiating and setting the derivative to 0.
OF 0 (IIp(u,[P(uk
do:
Oy;
0
0F 0o:
8 In ( P(t)))
= 0 7 oj
Oy
Because of this analytical result for the variational variance, we not need to consider further in the optimisation problem, and only consider minimizing the variational means ty. This renders all the terms the fnee-energy except the In Inp(u,|P(y)) terms constant with respect to the variational parameters. This allows to write:
F (np(uN(P(NN)) +
In ((u:))
(10)
presented in section 2. The first term In puN (P(y))) ) is effectively the loss at the output (yN T)
so becomes additional prediction error In ((uN(P(uN)) x (T' (x) ) "T (T- fN) which can be absorbed into the sum over other prediction errors. Crucially, although the variational variances have an analytical form, the variances of the generative model (the precisions Σ,) do not and can be optimised directly to improve the log model-evidence. These precisions allow for a kind of "uncertainty-aware" backprop.
DERIVATION VARIATIONAL UPDATE RULES AND FIXED POINTS
Here, starting from Equation 10, show how to obtain the variational update rule for the Uy (Equation 2), and the fixed point equations (Equation 5) (Friston 2008 2005 Bogacz 2017). We first reduce the free-energy to a sum of prediction errors.
F
In pp (P():)
(us - (PP(t)) ( +
Σ 1=1
+ + 2=
Where - )), and we have utilized the assumption made in section 2 that П I. By setting all precisions to the identity, we are implicitly assuming that all datapoints and vertices the computational graph have equal variance. Next assume that the dynamics of each vertex e follow a gradient descent on the free-energy.
dv;
OF
a Ou:
N
Gel
dt
Ov,
j=1
de: +
θej Dv
Σ
Ei Dv.
Ej
SEC(m:)
Σ
4 jEC(eN)
Drs
25

---

Under review as a conference paper at ICLR 2021
Where we have used the fact that deg we simply solve for des 0.
1 and
de1 day
862
To obtain the fixed point of the dynamics,
du;
OF
=0
dt
Ou;
0by Ej Ou:
0 == Ei
jEC(1N)
003
7 c
6j Do
jEC(m:)
Similarly, since ug 07 then 23 = o + 07. So:
0/ E 07
00;
0
6 Oug
jEC(o;)
26


Reviews:
[{"review_id": "lWegAXK_uVJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Update after author responses: \nWhile the author address some of my comments, I would have still liked to see a more detailed discussion of how the algorithm compares in terms of algorithmic scaling, which I think is relevant because it is a fundamental property of the algorithm, even if it is targeted towards understanding biology. So my score remains the same.\n\nSummary:\n\nThe authors extend recent work on MLPs to show that predictive coding converges asymptotically to exact backprop gradients on arbitrary computation graphs. They construct predictive coding networks for common architectures and show that it works well.\n\nOverall, I vote for an accept because I think the generalisation is quite useful and interesting for training deep networks with local learning rules. The authors demonstrate that this method works, but haven't demonstrated its computational advantages clearly enough. There are some issues of clarity that I have also outlined below.\n\nStrengths:\n+ The generalisation of the earlier MLP results to arbitrary computational graphs is quite powerful esp. since it can be applied to most deep learning architectures.\n+ The experimental evaluation includes all popular deep learning architecture, and it's impressive that this works on all of them. The experimental evaluation is also extensive.\n  \nWeaknesses:\n- The increase in computational cost (of 100x) is mentioned quite late and seems to be glossed over a bit.\n- Due to the potential for parallelisation in the predictive coding network, a comparison of wall-clock time for training on highly parallel setups might have been very interesting.\n- For RNNs and LSTMs, the equivalent predictive coding network is generated after unrolling the network, which means that the predictive coding network has no memory advantage over BPTT and a huge performance penalty. This is related to the previous point, where the utility of the predictive coding network is not demonstrated sufficiently.\n\nClarity:\n- Many of the figures are almost unreadable on paper. E.g. Fig. 3.\n- Algorithm 1 does't seem to be referenced anywhere.\n- In fig. 1 bottom, $\\delta$ missing in the denominators\n- Eqn. 3 is a bit sloppy, where derivative w.r.t $\\theta$ is suddenly equated to a derivative w.r.t $\\theta_i$.\n- If $\\epsilon_i = v_i - \\hat{v}_i$ then eqn. 7 is inconsistent with this, since $\\frac{d\\epsilon^*_i}{d\\theta}$ would be $-\\frac{d\\hat{v}_i}{d\\theta_i}$ (missing -ve sign). Unless I misunderstood something.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lOigDgNZdv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nAuthors propose that predictive coding gives similar convergence as backprop algorithms by extending the work of (Whittington &Bogacz 2017,  https://www.mrcbndu.ox.ac.uk/sites/default/files/pdf_files/Whittington%20Bogacz%202017_Neural%20Comput.pdf) to arbitrary graphs. This is an important topic both for the application of classical deep nets to neuromorphic hardware, but also for our understanding of computations in biological tissues.\n\n1. The work presented in this paper follows similar results by Amit (2019) or Lilicrap, and provides with numerical simulations comforting the theoretical predictions. \n2. Authors present an extension of the previous work to arbitrary graphs, and they apply their claims to LSTM and RNN models.\n3. It provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. This makes the point of the paper more convincing and complementary to similar works.  \n\n##########################################################################\n\nConcern:\n \nA major issue of this paper is to not identify its originality. The extension to « arbitrary graphs » or to CNNs is straightforward in theory, While it is not explicitly stated in  (Whittington & Bogacz, 2017), an unwrapped RNN is by definition a feed-forward graph and is therefore a direct application of their work. \n\nConcerning novelty, the actual derivations (eg for the LSTM) are original and the simulations clearly support that original contributions. However this material is at the end of the paper or in appendices. Recentered on this original contributions and how this makes a suitable contribution to the community would make the paper acceptable to be accepted at ICLR.\n\n##########################################################################\nminor:\np.2 « backwawrds »\np7 « dataest » & p 14", "extra_scores": {}, "extra_text": {}}, {"review_id": "b9oWwaQTTlJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper extends prior work on equivalence between predictive coding and backprop in layered neural networks to arbitrary computation graphs. This is empirically tested first on a simple nonlinear scalar function, and then on a few commonly used architectures (CNNs, RNNs, LSTMs), confirming the theoretical results. The importance of this advance is highlighted by noting that the demonstrated equivalence shows how in principle modern architectures could be implemented in biological neural systems, and that the highly parallel nature of predictive coding could lead to efficient implementations in neuromorphic hardware.\n\nThe paper is very well written, easy to read, and includes a nice introduction section with a fairly comprehensive overview of backprop, and the problems related to its potential implementations in biological systems. I also appreciated the \"tension in a chain\" metaphor illustrating the dynamics of backprop and predictive coding. That the exact backprop gradients are computable in a fully local system with Hebbian plasticity for an arbitrary graph is an interesting and promising result.\n\nThroughout the text, predictive coding is quoted as biologically plausible. This isn't strictly true as noted already in (Whittington & Bogacz, 2017), as e.g. dedicated error nodes are not known to exist for every cell in the brain. I'd suggest calling this \"potentially biologically plausible\", and including a short discussion on how these plausibility concerns could be addressed.\n\nAll in all, the results are interesting, open up interesting directions for future work, and I recommend the acceptance of the paper.\n\nAdditional questions/suggestions:\n- Is the fixed-prediction assumption a limitation to biological plausibility?\n- Fig. 2 shows the model converging at high inference learning rates for the case of the scalar function. Is the 0.1 rate used for CNNs the max that was stable, or could higher values be used to reduce the computational overhead?\n- What convergence condition was used?\n- How does convergence speed depend on the size/diameter of the computational graph? This is similar to Fig. 9, but asking a slightly different question -- i.e. how many iterations are needed to reach convergence as a function of graph size.\n\nTypos:\nFig. 1 caption: \"backwawrds\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9evPd1ch91R", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary of the paper\nIn their paper, the authors demonstrate that Predictive Coding (PC) is a local approximation of back-propagation and could then be interpreted with Hebbian learning rule (a neuro-plausible learning rule). This result has been first demonstrated by [1] with MLP network (on the MNIST dataset) and the presented paper extend this finding to CNNs (on CIFAR10, CIFAR100 and SVHN), RNN and LSTM. \n\n#### Pros\n* The authors provided experimental evidences on a wide variety of networks' type and databases.\n* The link between neuro-plausible learning rule and back propagation is interesting.\n* The paper is well situated in the literature.\n* The authors are providing the code for clean reproducibility\n\n#### Cons\n* The mathematical definition and notation of the paper are not rigorous enough. It makes the paper unclear and hard to follow.\n* Some crucial points would have deserved in-depth discussion and are just ignored (see below)\n* The paper is not well enough motivated: what’s the point of such a local approximation beside the neuro-plausibility (faster ? Consume less resources ? …)\n* The demonstration seems to include only one kind of loss function, which does not match the claim of the paper\n\n#### Recommendation\nGiven the limited impact and the lack of clarity of the paper, I would tend to reject the article.\n\n#### Detailed comments:\n* The gaussian parametrization used by the authors constrains the comparison between PC and backprop to networks with L2 loss function. One cannot claim to approximate arbitrary computational graph if one demonstrates the approximation on a specific loss function (which is known to poorly perform on classification problem). So could your framework be generalized to more effective loss function like cross entropy ? If yes, what would be the underlying probabilistic hypothesis ? This should be included in the paper, as it will strongly strengthen your claim.\n\n* The PC framework proposed by the authors propose a solution to the ’non locality’ of the back propagation (to be a bio-plausible mechanism). However the authors also raised the weight transport problem. On my understanding the proposed framework is still suffering from weight transport as the backward connection weights are the transpose of the feedforward one (due to the derivation of the forward operator). The paper would deserve an in-depth discussion concerning this point.\n\n* What is the computational advantage of local approximations ? Is it saving computational resources (computational time, memory…) ? A comparison of the algorithmic complexity between PC and back-propagation would be valuable to support your claim. In the discussion, the authors mention that their framework, being substantially more expensive than back-prop network, could be deeply parallelized across layer. The authors should provide experimental or theoretical evidences that such parallelization is enough to mitigate the higher number of inference steps (i.e. 100-200) needed by their PC framework.\n\n* The concept of ‘fixed-prediction assumption’ introduced by authors in the paper considers that each (v_i) are fixed to their feedforward value. Then what is the point of the Eq. 2, as you already know the value of the activity vector? On my mind this is here a crucial point, as this is dealing with the core principle of PC : an inner-loop (i.e. the expectation step) that find the most likely v_i, and an outer loop (i.e. the maximization step) that update the parameters. I have the intuition that this problem arises because the authors are tackling a discriminative problem (i.e. finding a mapping between inputs and labels) using a generative model (PC is a generative generative model as described by [2, 3]). Can you please clarify this point ?\n\n* What is the testing procedure of your network ? Is it a simple feedforward pass (which I suspect) or is it an Expectation-Maximization scheme. Your algorithm 1 shows only the training procedure (as you need the label information to perform the computation). If this is a simple feedforward pass, what would be the advantages of the inference process (more robustness ? Better prediction ?)\n\n## Typos and suggestions to improve the paper :\n* The authors state (3rd paragraph page 3) that they are considering a generative model. If it is the case, the formula p(v_i) = product(p(v_i | parent(v_i)) is inaccurate as the authors forgot the prior p(v_N).\n* Eq 1 : The derivation between the first and the second line of Eq.1 has to be demonstrated or referenced (at least in annex)…\n* The authors consider the posterior is a marginal probability (see eq1, and subsequent paragraph). In general the posterior is a conditional probability (this specific point makes your equation 1 hard to grasp because readers are not making the link with the classical ELBO, i.e. the negative free energy). In general, the probabilistic notations are not rigorous enough, and it makes the rest of the mathematical derivation complicated to follow. \n* The authors should reorganize the Annex to make sure it follows the reading order (Appendix D is cited first)\n* Caption of Figure 1 : backwawrds —> backwards\n* Page 3, 2 lines below eq 1 : as as —> as \n* Page 4, 4 lines below eq 3 : forwards —> forward\n* Figure 3, which is on my mind the most import one, is shown but not cited in the core text.\n\n[1] Whittington, J. C., & Bogacz, R. (2017). An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5), 1229-1262.\n\n[2] Rao, Rajesh PN, and Dana H. Ballard. \"Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.\" Nature neuroscience 2.1 (1999): 79-87.\n\n[3]Friston, Karl. \"A theory of cortical responses.\" Philosophical transactions of the Royal Society B: Biological sciences360.1456 (2005): 815-836.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends recent work (Whittington & Bogacz, 2017, Neural computation, 29(5), 1229-1262) by showing that predictive coding (Rao & Ballard, 1999, Nature neuroscience 2(1), 79-87) as an implementation of backpropagation can be extended to arbitrary network structures. Specifically, the original paper by Whittington & Bogacz (2017) demonstrated that for MLPs, predictive coding converges to backpropagation using local learning rules. These results were important/interesting as predictive coding has been shown to match a number of experimental results in neuroscience and locality is an important feature of biologically plausible learning algorithms.

The reviews were mixed. Three out of four reviews were above threshold for acceptance, but two of those were just above. Meanwhile, the fourth review gave a score of clear reject. There was general agreement that the paper was interesting and technically valid. But, the central criticisms of the paper were:

1) Lack of biological plausibility
The reviewers pointed to a few biologically implausible components to this work. For example, the algorithm uses local learning rules in the same sense that backpropagation does, i.e., if we assume that there exist feedback pathways with symmetric weights to feedforward pathways then the algorithm is local. Similarly, it is assumed that there paired error neurons, which is biologically questionable.

2) Speed of convergence
The reviewers noted that this model requires many more iterations to converge on the correct errors, and questioned the utility of a model that involves this much additional computational overhead.

The authors included some new text regarding biological plausibility and speed of convergence. They also included some new results to address some of the other concerns. However, there is still a core concern about the importance of this work relative to the original Whittington & Bogacz (2017) paper. It is nice to see those original results extended to arbitrary graphs, but is that enough of a major contribution for acceptance at ICLR? Given that there are still major issues related to (1) in the model, it is not clear that this extension to arbitrary graphs is a major contribution for neuroscience. And, given the issues related to (2) above, it is not clear that this contribution is important for ML. Altogether, given these considerations, and the high bar for acceptance at ICLR, a "reject" decision was recommended. However, the AC notes that this was a borderline case.
Decision: Reject