A Theoretical Framework for Inference and Learning in Predictive Coding Networks
Authors: Beren Millidge
Yuhang Song
Tommaso Salvatori
Thomas Lukasiewicz
Rafal Bogacz
Venue: ICLR 2023 poster (2023)
Keywords: predictive coding
backpropagation
target propagation
machine learning
neuroscience

Abstract: Predictive coding (PC) is an influential theory in computational neuroscience, which argues that the cortex forms unsupervised world models by implementing a hierarchical process of prediction error minimization. PC networks (PCNs) are trained in two phases. First, neural activities are updated to optimize the network's response to external stimuli. Second, synaptic weights are updated to consolidate this change in activity --- an algorithm called \emph{prospective configuration}. While previous work has shown how in various limits, PCNs can be found to approximate backpropagation (BP), recent work has demonstrated that PCNs operating in this standard regime, which does not approximate BP, nevertheless obtain competitive training and generalization performance to BP-trained networks while outperforming them on various tasks. However, little is understood theoretically about the properties and dynamics of PCNs in this regime. In this paper, we provide a comprehensive theoretical analysis of the properties of PCNs trained with prospective configuration. We first derive analytical results concerning the inference equilibrium for PCNs and a previously unknown close connection relationship to target propagation (TP). Secondly, we provide a theoretical analysis of learning in PCNs as a variant of generalized expectation-maximization and use that to prove the convergence of PCNs to critical points of the BP loss function, thus showing that deep PCNs can, in theory, achieve the same generalization performance as BP, while maintaining their unique advantages.

Published as conference paper at ICLR 2023
A THEORETICAL FRAMEWORK FOR INFERENCE AND LEARNING IN PREDICTIVE CODING NETWORKS
Beren Millidge
Yuhang Song
MRC Brain Network Dynamics Unit
MRC Brain Network Dynamics Unit University of Oxford, UK yuhang song eoon@endern eddon ox ac uk
University of Oxford, UK
serensmillicn name
Tommaso Salvatori
Thomas Lukasiewicz
Institute of Logic and Computation
TU Wien, Austria
TU Wien, Austria
University of Oxford, UK thomas wukasiewiczettwieen. acccnaccccc
tommaso. salvatori@tunien
Rafal Bogaez
MRC Brain Network Dynamics Unit University of Oxford, UK
rafal.bogaczedden.. uc
ABSTRACT
Predictive coding (PC) is influential theory in computational neuroscience, which argues that the cortex forms unsupervised world models by implementing hierarchical process of prediction error minimization. PC networks (PCNs) trained in two phases. First, neural activities are updated to optimize the network's s response external stimuli. Second, synaptic weights are updated to consolidate this change in activity an algorithm called prospective configuration. While previous work has shown how in various limits, PCNs can be found to approximate backpropagation (BP), recent work has demonstrated that PCNs operating in this standard regime, which does not approximate BP, nevertheless obtain competitive training and generalization performance to BP-trained networks while outperform- ing them on various tasks. However, little is understood theoretically about the properties and dynamics of PCNs in this regime. In this paper, we provide com- prehensive theoretical analysis the properties PCNs trained with prospective configuration. We first derive analytical results concerning the inference equilib- rium for PCNs and previously unknown close connection relationship to target propagation (TP). Secondly, as variant of generalized expectation- popeetation-maximizatino and use that to prove the convergence of PCNs to critical points of the BP loss function, thus showing that deep PCNs can, in theory, achieve the same generalization performance as BP, while maintaining their unique advantages.
provide theoretical analysis of learning in PCNs
INTRODUCTION
Predictive coding (PC) is influential theory in theoretical neuroscience (Mumford 1992 Rao & Ballard 1999 Friston, 2003 2005), which is often presented as a potential unifying theory of cortical
function (Friston] 2003 2008 2010: Clark 2015b Hohwy al. 2008). PC argues that the brain is fundamentally a hierarchical prediction-ene inndiction-erro---ee error- minimizing system that learns a general world model by predicting sensory inputs. Computationally. one way the theory of PC can be instantiated is with PC networks (PCNs), which are heavily inspired by and can be compared to artificial neural networks (ANNs) on various machine learning tasks (Lotter et al., (016)] Whittington Bogaez| 2017] Millidge et al. 2020a b, [Sng et al.) 2020, Millidge et al. 2022). Like ANNs, PCNs are networks of neural activities and synaptic weights. Unlike ANNs, in PCNs, training proceeds by clamping the input and output of the network to the training data and correct targets, respectively, and first letting the neural activities update towards the configuration that minimizes the sum of prediction errors throughout the network. Once the neural activities have reached equilibrium, the synaptic weights can be
"Corresponding author.

---

Published as conference paper at ICLR 2023
updated with a local and Hebbian update rule, which consolidates this change in neural activities. This learning algorithm is called prospective configuration (Song et al. 2022), since the activity updates appear to be prospective in that they move towards the values that each neuron should have in order to correctly classify the input.
Previous work has shown how, in certain limits such when the influence of feedback information is small or in the first step of inference, PC can approximate backpropagation (BP) and that this approx- imation is close enough to enable the training of large-scale networks with the same performance as BP (Whittington & Bogacz, 2017: Millidge et al., 2020aa [Song et al. 2020); see Appendix [A2 for a full review and comparison. Recent work (Song et al.) (022) has also shown that PCNs trained with prospective configuration under standard conditions can also obtain a training and generalization performance equivalent to BP, with ad advantages in online, few-shot, and continual learning. Intuitively, the difference between learning in BP and PCNs is that in BP, the error is computed at the output layer and sequentially transported backwards through the network. PCNs, there first an inference phase in which the error is redistributed throughout the network until it converges to an equilibrium. Then, the weights are updated to minimize the local errors computed at this equilibrium. Crucially, this equilibrium is in some sense prospective: due to its iterative nature, it can utilize information about the activities of other layers that is not available to BP, and this information can speed up the training by not making redundant updates.
However, while the convergence properties of stochastic gradient descent combined with BP are well understood, there is equivalent understanding of the theoretical properties of prospective configuration yet. In this work, we provide the first comprehensive theoretical study of both the prop- erties of the inference and the learning phase in PCNs. We investigate the nature of the equilibrium computed by the inference phase and analyze its properties in linear PCN, where an expression for the inference equilibrium can be derived analytically. We show that this equilibrium can be interpreted as an average of the feedforward pass values of the network and the local targets computed by target propagation (TP). We show that the same intuition also holds for nonlinear networks, where the inference equilibrium cannot be computed analytically. Furthermore, we investigate the nature of learning in such networks, which diverges from BP. We present novel interpretation of PCNs as implementing generalized expectation maximization (EM) with a constrained expectation step, which complements its usual interpretation as variational inference (Friston 2005 [Bogazz] 2017 Buckley et al. 2017). Moreover, we present unified theoretical framework, which allows to understand previously disparate results linking PC and BP, to make precise the connection between PC, TP, and BP, and crucially to prove that PCNs trained with prospective configuration will converge to the same minima as BP. We thus show that, in theory, PCNs have at least an equal learning capacity and scalability as current machine learning architectures.
2 BACKGROUND AND NOTATION
Although PCNs can be defined for arbitrary computation graphs (Millige et al.) 2020a| Salvatori et al., 2022a b). in this paper we assume an MLP structure. A PCN consists of hierarchical set of layers with neural activations is a real vector of length w,, which is the number of neurons in the layer. In this paper we refer to the input To and output zy layers as the input and output layers respectively. Each layer makes prediction of the activity of the next layer ét = (W where Wi E R x R"i-1 is a weight matrix of trainable parameters, and first layer neural activities are fixed to an input data point 4o d, and predictions are propagated through the network to produce an output âz. During training, both the first and last layers of the network are clamped to the input data point zo = d and a target value œy = T, respectively. Both the neural activity and weight dynamics of the PCN can be derived as a gradient descent on a free-energy functional equal to the sum of the squared prediction errors F = E/= ef, where = 21 - (WWa1-1). Typically, the activity updates are run to convergence x' (called the inference phase), and the weights are updated once at convergence (called the learning phase):
Predictive coding networks (PCNs):
.., where x; E R
an elementwise activation function. During testing, the
4 + W7 W (WWi- for I L. for = L,
Inference: in = OF /0x)
Learning: Wi - (OF/ =
f'(iatt
(2)
In the above equations denotes elementwise multiplication. The network energy function JF presented here can be derived as a variational free energy under hicrarchical Gaussian generative
2

---

Published as conference paper at ICLR 2023
model, thus resulting in PCNs having an interpretation as principled approximate inference. For more detailed description of PCNs, BP, and TP, see Appendix A.I1
Target propagation (TP) is an alternative algorithm for training deep neural networks (Lee et al. 2015; Bengio & Fischer 2015). Its core intuition is that the ideal activity for a layer is the activity that, when passed through the forward pass of the network, outputs the desired label or target exactly, If we can compute this quantity (which we call a local target), then we can update the feedforward weights to minimize the difference between the actual activity and the local target, and ensure that eventually the feedforward pass of the network will compute the correct targets. Computing the local targets involves computing the inverse of network. For instance, given forward pass through layer z = f(Wizt-1) the local target to can be defined in terms of the local target of Given this local target, we can define the local prediction error as T 1 21, which is minimized by weight updates. In practice, since explicit inverses are expensive compute, and most neural networks are not invertible, the pseudoinverse used or approximateds_ce et al., 2015 Bengio 2020). The TP recursion is identical to the BP recursion except that uses inverses instead of derivatives and transposes. Although BP and TP seem very different, key result that we show is that the inference phase of a PCN effectively provides an interpolation between the BP and TP targets.
the layer above as 4 = W
3 RESULTS
3.1 INFERENCE IN PCNs
To build comprehensive mathematical theory of PCNs, we must deeply understand both inference and learning. We first focus on the former, and explore the connection between the inference equilibrium reached in a PCN and both BP and TP. We show that the PCN inference equilibrium interpolates between BP and TP, and that the weight updates approximate BP when the feedforward influence of the data on the network is large, and so the inference equilibrium is close to the feedforward pass values, while approximates TP when the feedback influence of the target is large.
First, we recapitulate the result that PC approximates BP when the feedforward influence of the data predominates, and thus the equilibrium activities are close to their feedforward pass values.
Theorem 3.1 Let M be a PCN with neural activities (za,.... ,..). and prediction errors {e1,-..,eEL) EL.), where the final prediction error equals the gradient of the loss relative to the out- put (cl = -.L/Or)) Then, the prediction error at equilibrium at a layer oj approximates the BP gradient 0C/0xz up to order O(ez) and hence converges to exact BP as xf + Ht, where es =='((W1+12)) -f'(W+111)) with f being the activation function derivative and ãs the feedfor- ward pass value layer I.
Proof. Consider the equilibrium of the activity dynamics of a layer in terms of the predic- tion errors: di Q+1 ((W(+1a))W/,
= G = G- 2 i+1 This recursive relationship is identical to that of the BP gradient recur- sion 0c/ôr: dC ac/001410014 1/0; except that the gradients are taken at the activity equilibrium instead of the feedforward pass values. To make the difference explicit, we can add and subtract the feedforward pass values to obtain
E+ (02t41/0mi)l==z)
á = 44 @f + anI [f((W+107) - J' (WH00(WH)))))00. + +((()))))
BP
This linear dependence of the approximation error to BP on the difference between f' (WW2+127) and ('(W4117) clearly implies that PC converges to BP as 27 - Tt.
In fact, we obtain additional more subtle conditions. The above proof also implies convergence to BP when the feedback error G411 -> 0 and additionally when the activation function derivatives f' are the same even when the equilibrium activities are not the same the feedforward pass values. Overall, the feedforward influence on the PCN has thus the effect of drawing the equilibrium activity close to the feedforward pass, resulting in the PCN approximating BP. This result is closely related to previous results relating the PCN dynamics to BP (Whittington & Bogacz 2017 Millidge et al.
2020a Song al. 2020 Salvatori et al. 20226), reviewed in Appendix
We next consider the second limiting case where there is only feedback influence on the PCN by considering the 'input-unconstraineed case where the input layer of the PCN is not fixed to any data.
3

---

Published as conference paper at ICLR 2023
B
Convergance each layer to largetsprop largels
converging equilbruu
Aserrage distance from lnnar equi lbrium
3.0
Layer
Layer 2
2.5
Layer Layer 4
0.
2.0
1.5
1.0
-0.4
0.5
-0.0
0.0
0.
250
500
750
1000
1250
1500
1750 2000
20 40 60 80
100
Tresstop
Figure 1: A: Evolution of the activitics converging towards the TP targets during an inference phase in a nonlinear uppu-unconstrained network, averaged over 50 seeds. B Example convergence of activities in layer towards the linear equilibrium computed in Theorem 3.3 in 3-layer lincar PCN. C Total Euclidean distance of all activities of a 3-layer PCN from the linear equilibrium during inference, averaged over 50 seeds. Error bars represent standard deviations.
Let M be PCN with neural activities {zo.....; ,LL with an invertible activation function f and square invertible weight matrices (Wi...., W.}. Let the output layer be clamped to a target 21, = ' and the input layer be unclamped. Then, the equilibrium activities zj of each layer are equal to the TP targets of that layer.
Theorem 3.2
Proof. If the input layer is unconstrained, and the network is invertible, then there is unique global minimum where 6 = 0 for all 1. Substitute in the TP target definition for Ty- We have that (++ F1+1 - f(W(4125)
= Ft+1 F1+1 0. This holds for all q. To see why this minimum is unique, consider the prediction error of the final layer, which is clamped to the target ez, = - DWWW(WW
xi+1 f(W (W
By the assumption of invertible f and Wr, this equation has only one solution = WT (- This applies recursively down to each layer of the network substituting in zz-1 for T. If the invertibility assumption is relaxed, as in realistic neural networks, there may be multiple equally valid targets and the input-unconstrrined PCN will converge to one such solution. A more intuitive dynamical proof is given in Appendix A.6.3 In Figure panel A, we demonstrate this empirically by showing that for an appu-unconstrained 5-layer nonlinear network with tanh activation functions, the layer activities reliably towards the local targets. Interestingly, this occurs in a distinctly layerwise fashion with layers closer to the output converging first, highlighting the temporal dynamics of information transmission through the network. Full details for all networks used in the experiments in can be found in Appendix AA. 10
However, during PCN training, the input layer is clamped to the data while the output layer is clamped to the target. While it is not possible to derive an analytical expression for the equilibrium for a general nonlinear network, we can derive analytical formula for the equilibrium in the linear case.
Theorem 3.3 Let M be linear PCN with neural activities {zo...., #L} and prediction function f = Wi then the equilibrium value of a layer oi is computable in terms of the equilibrium values of the layers below z and above 2i+1 as xi = (I + WTT, W1+1)
- [What_1 + W
Proof. By setting कं = 0,we obtain
di -e + + W 1441 = -n W WX [-241 W411m] =0
- + WH,, W 14)1)2 + WT. "141 + W,
(4) We observe that the linear equilibrium consists of a sum of 'feedforward' term W and 'feedback' term WT T1+1, both weighted by prefactor (1-W7T Wi-1), which effectively measures the degree of orthogonality of the forward pass weight matrix. This expression neatly represents the 'prospective equilibrium' for the activities of a PCN in terms of feedforward component, which we can think of as trying to pull the equilibrium state of the network towards the initial feedforward pass values, and feedback component, which we can interpret as trying to pull the equilibrium activity towards the TP targets. Also, note that in the linear case when all prediction errors are W which implics that at the global optimum of the PCN (if it is reachable), the feedforward pass activies exactly equal to the TP targets. Additional mathematical results
= 27 = (I+ W ( WT m-
zero (ei 0), the feedforward and feedback influences are exactly equal Wir (proof in Appendix
4

---

Published as conference paper at ICLR 2023
Simikariy to BP and TP thougg hout inforence
8F
0.8
0.7
0.8
0.5
Similaiity targetprig targets
0.3
Similarity beckgrop gred enits
40 Figure Left: PC interppolates between BP and TP depending on the precision ratio. Right: Intuitions are maintained in the nonlinear case. Right: The cosine similarity between prediction errors and BP gradients as well as neural activities and TP targets during the inference phase for 3-layer nonlinear (tanh) network. As predicted by our linear theory, the similarity to targetprop targets increases during inference while the similarity to the backprop gradient decreases.
107
demonstrating the convexity of the optimization problem in the linear case, and deriving explicit differential equations for the activity dynamics are given in Appendices |A.4 and A.5 In Figure | panel B, we showcase the node values in a 5-node PCN converging to the exact linear equilibrium predicted by the theory. Moreover, in panel C, we plot the mean euclidean distance of all neurons from the predicted equilibrium values as function of inference step for 3-layer linear network which demonstrate rapid and robust convergence. While these results have been primarily shown for the linear case, we also wish to understand the extent to which they hold empirically in the nonlinear nonlinear 3-layer PCN
case. In Figure 2B, we plot the similarities of the equilibrium activities with t tanh activation function to both targetprop targets and backprop gradients and show that the same intuitions hold in the nonlinear case as predicted by our linear theory.
3.2 INFERENCE IN
WEIGHTED PCNS
We have seen that the equilibrium activities are determined by a sum of feedforward and feedback influences. We demonstrate that the analytical precision equilibrium here is correct empirically in Figure (3 and B which shows that linear PCNs rapidly and robustly converge to the analytical solution we derive. However, we have not yet defined a way to change the relative weighting of these influences. The Gaussian generative model that underlies the derivation of the PCN does, however, give us precisely such a mechanism by utilizing the inverse variance or precision parameters of the Gaussian distribution II, Σ, which thus far we have tacitly assumed to be identity matrices. If we relax this assumption, we obtain a new precision-wee wreihten-weighhed dynamics for the activities:
5 = [ī= 11141
(5)
From these dynamics, we can derive an analytical expression for the preision-weighted equilibrium.
Theorem 3.4 Let M be linear PCN with layerwise precisions (III...., {T Then, the precision- weighted equilibrium value of a layer x is computable terms of the equilibrium values of the layers
below 1i_1 and above 2141 as x] (T + 11, -WT (1+1) - + - 1 111+101411
proof is given in Appendix A.6. The result is similar to the previous linear equilibrium and identical to it when 11, = 11 = I. Heuristically, or exactly with diagonal precisions, this expression lets us think of the precision ratios I1 as determining the relative weighting of the feedforward and feedback influences on the network dynamics. These allow us to derive more precise approximate statements about the relative weighting of the feedforward and feedback influences leading to convergence either to the feedforward pass values the TP targets. For instance, suppose the feedback precision ratio is large, so that in the prefactor the term I1, , I1, W4+1 dominates. This will occur when the precision at the next layer is substantially larger than that at the current layer, i.e., the prediction errors at the next layer are 'upweighted'. In this case, for the prefactor, we 11+1
have that (1 + 11, WT' W ( xs , - Wiilu,,- W, since the identity term in the prefactor becomes negligible. As such, can approximate the equilibrium as +T, W 1
at ~ (Wi
~ W " WT:T + Wiitinn = W
(6)
5

---

Published as conference paper at ICLR 2023
B
Activ tics connec giringgggggggg to procsion oqualiaaiunn
Awerage distance from precision equitirium
Simtirityy and by precision rabio
0.6
10.
0.00
0.0
-0.05
-0.10
0.4
0.6
-0.35
0.2
0.4
-0.20
targelprop tarqets Simlleityy backpreo gradients
0.2
-0.35
30
380
3.6
Timastep
Precsion ratio
Figure 3: A Example activities of neurons in layer converging to the analytical precision equilibrium during an inference phase in 3-layer línear PCN. B Total Fuclidean distance from the analytically computed precision equilibrium averaged over 50 random initializations. The similarity between the equilibrium activities and the targetprop targets, as well as the equilibrium prediction errors and the backprop gradients at the end of inference in a nonlinear 3-layer tanh PCN for different precision Ii+ We observe the same qualitative effects of increasing similarity to backprop with high
ratios I1 bottom-up precision and increasing similarity to target-prop with high top-down precision in the nonlinear case as predicted by our linear theory. Error bars are standard deviation over 50 seeds.
Here, we can discount the first term: as 11," *W?, H1++1 W4+1 is assumed to be large, its inverse Wiil I is small. Effectively, if the relative ratio of feedback to feedforward precision is large, then the balance between the feedforward and feedback stream at equilibrium is tilted towards the feedback stream, and the activities equilibrium approximate the TP targets. Conversely, if the precision ratio is low, then the (II,' WT,, 4 W(+1) ) term in the prefactor is very small. In this case, we can approximate the prefactor simply by the identity matrix as (T + IT, " WT 1-1 2 I. This then lets us write the equilibrium
(42 [ W4+1)=+ x
x² -W + " 11+124+1 R Wiai-1;
(7)
where the approximation assumes that the ratio of feedback feedforward precision is small, i.e., I1 Rs 0. That is, the activities at equilibrium approximately equal the feedforward pass values. While this analysis only applies to linear networks, we show in Figure BC that similar qualitative result applies even to nonlinear networks where the similarity to the backprop gradients increases with the ratio of bottom-up to top-down precision and vice-verss versa for the similarity to the targetprop targets, thus showing that the intuitions gained in our analysis generalize beyond the linear setting.
3.3 LEARNING IN PCNS
We now present a theoretical framework for understanding the learning phase of PCNs and its convergence properties. Our key result is that the learning algorithm should ultimately converge to a critical point of the BP loss L. To do so, we need to develop an alternative mathematical interpretation of PCNs as an expectation-maxiiizatie
(EM) algorithm (Dempster et al. 1977), which complements the standard view of PC as variational inference. The ultimate goal of PC is to be able to achieve local learning such that the weights of the network can be updated to minimize only a locally defined loss using local information, but which nevertheless will ultimately result in minimizing the global (output) loss of the network. One intuition as to how to achieve this is to compute local 'targets' for each layer, where the ideal target is the local activity value that would minimize the output loss of the network. If we had such local targets, then local learning would be simple- we would simply update the weights to minimize the difference between the activity of the layer and its local target. Since there would be a local target for each layer, all information would be available locally.
The fundamental problem is that we do not know the correct local targets. However, given the input data value and the label, we can infer the correct local targets at each layer. Thus, the computation of the optimal targets is a Bayesian inference problem. An alternative way think about this is as a missing data problem. Ideally, we would like to minimize the loss with the optimal local targets and the input data. However, unfortunately, the local targets are 'missing" and so to proceed with our optimization of the weights, we must infer them first This is the classic justification for the EM-algorithm (Dempster et al. 1977: Minka 1998), which involves first an 'expectation" (E) step, which infers the posterior distribution over the missing data, and then 'maximization' step, which maximizes the log likelihood of the parameters averaged over the posterior. PCNs do not infer the full posterior distribution but only delta distribution around the most likely value of the targets.

---

Published as conference paper at ICLR 2023
To mathematically formalize this, suppose that we have a discriminative PCN with input data D, output target T. and estimated local targets {zz]. We want to find the most likely values zj for the local targets. This can be expressed as a maximum a-posterioni (MAP) inference problem:
{ai} (z.) pl [T. D, W) = argmar(x)) (xx) In p(T)))) D, w)+ in p(frr]] W). (8)
Given a set of zjj we can then interpret the learning phase maximizing the log likelihood of the targets with respect to the weight parameters averaged around the posterior distribution of the local targets, which we have approximated with its MAP value p( {zi){ D, W) = I 6(x) - 27). That is, we can write out the PC equivalently as an EM algorithm:
E-step: argmax(sx) Pl D, W),
(9)
M-step: argmazw (o(t=))).D..,,(((((())))
(10)
These dual interpretations of PC as both EM and variational inference are not surprising. As shown in Neal & Hinton (1998), any EM algorithm can be expressed as dual descent on variational free energy functional F. By the known convergence properties of the EM algorithm, we can derive that the PCN converges to a minimum of the variational free energy F.
3.4 CONVERGENCE OF PC TO CRITICAL POINTS OF THE LOSS
A-priori, the results of Section 3.3 tell us little about the extent to which the minima of F correspond to useful minima of the BP, or supervised loss C. Next, we address this question by re-interpreting PC from performing EM on the variational free energy F to performing constrained EM on the BP loss C, and hence showing that PCNs are guaranteed to converge to a minimum of C. This means that PC possesses the same convergence properties that BP enjoys, and thus that, theory, PCNs possess the same learning capabilities as BP. This ultimately implies that the PC learning rule, although different from BP, can be scaled to train large-scale deep networks with the same performance as BP.
First, we want to understand how the variational free energy F and the BP loss L are related. Recall that the variational free energy in PCNs is just a sum of the local energies in each layer, thus we can define F = E, Et, where the 'local energy' is defined in PCNs performing classification, the BP loss L is just the loss at the output layer in PCNs, i.e., L C= E,, (while the BP loss also depends on all the other weights on the network, the same is true for the output layer of the PCN where this dependence is made implicit through the inference process). This allows us express the variational free energy sum of the BP loss and remaining 'residual loss' È of the rest of the network:
Ei = of = (z; (W( Secondly,
F= E.
(11)
where the residual energy is defined as E = -1ー11 Ej. Now, we want to understand how to relate general critical points of J to critical points of L. The key will be to show that we can reinterpret the dynamics of PCN as performing a constrained EM algorithm on L and not F and thus by the known convergence of EM NNell & Hinton 1998), the PCN is guaranteed to converge to a critical point of L. We first show that the marginal condition holds.
Lemma 3.1 The marginal condition: At equilibrium, the gradients of the BP loss relative to the activities are exactly equal and opposite to the gradients of the residual loss: ac/Or; -0E/dm
proof is given in Appendix App72 Intuitively, the marginal condition requires that at equilibrium, any decrease of L must be exactly offset by an increase E or vice versa, since if this were not the case, then one could decrease F by moving in direction that decreased L more than É increased or vice versa, which would imply that we were not in an optimum. To gain an intuition for how this works in practice, in Figure we plot the values of these energies in randomly initialized 4-yeerr relu network as they evolve throughout the inference phase. If initialized in the correct way, the total free energy and backprop loss will decrease during inference while the residual loss will increase until an equilibrium, given by the marginal condition, is reached. We can now state our main result.
Theorem 3.5
Let M be PCN that minimizes a free energy F = C É. Given sufficiently small learning rate and that for each minibatch activities are initialized subject to the energy gradient bound: -ac/ dzj"" T aē/00z I (/C//x,)lY (derived in Appendix A.7.3) then, during training. the PCN will converge critical point of the BP loss L.
7

---

Published as conference paper at ICLR 2023
A
energies
0.000
Change backprop lows during inference
C
10"
Gradient norn throuphout fulk batch training
50%1
000
03
10"
010
graeett
BP gradiont norn (508)
PC gradient, porr (500)
Output loss
10
Training (1)
0.0
energy
loss incressing
PC gradient,
2000
4000
6000
10000
Eboch
E
Layer
Layer
0.3
Layer
train accurccy accuracy
2000
6000
8000
2000 4006
8000
Thaniing
Figure 4: A: Evolution of the decomposed energies during an inference phase of a 4-layer relu network, The output loss L continuously decreases throughout the inference phase, while the residual energy E increases. B: To verify the energy gradient bound holds in practice with standard step-size of 0.1, we plot the change in backprop loss between the beginning and end of the inference phase throughout training. We find that the change is always negative (loss decreasing). C: To verify convergence to a minimum of the backprop loss, we plot the training loss and norm of the BP and PC gradients during full-batch training of either 500 or MNIST digits. When the dataset comprises single digit, the PCN minimizes training loss to O with gradient norms of order 10 16 or within numerical precision given finite learning rate, exactly as predicted by the theory. For 500 MNIST digits very low loss and gradient value of order 10 is achieved but then training stalls, perhaps due to the finite learning rate. D: Similarity scores for each layer in a 6 layer network throughout training between the PC and BP weight updates. Even though PC updates are very different from BP, they still empirically converge to the same minima. This difference allows PC to exhibit different and potentially superior optimization performance. E: Accuracy of PC and BP on MNIST dataset showing that both learn at similar rate and reach equivalent performance.
The proof proceeds in several steps. We first express the inference and learning phases as constrained minimizations of L and not F. We use this to write the PCN dynamics as an EM algorithm on L and then apply the known convergence results of EM to prove convergence
critical point of L.
Recall that we assumed an initialization that satisfies the energy gradient bound -OC/On," (a0//,,,TFF) O0E/0n < ||02//m;112. We prove in Appendix A.7.3 that under such condi- tions the supervised loss is guaranteed to decrease during inference. This is because initializing with the energy bound condition implies that the gradient of the backprop loss is greater than the negative gradient of the residual energy. If the dynamics of the gradients of the backprop loss and residual energy are continuous (which technically requires continuous time inference or an infinitesimal learning rate), then by the intermediate value theorem, for the gradients to 'cross" such that the residual energy gradient is greater than the backprop loss gradient, and hence for the loss to increase, they must first attain point at which they are equal. This point is given by the marginal condition and is the inference equilibrium, and hence the inference procedure terminates at this condition, thus the backprop loss cannot increase during inference. Intuitively, the dynamics of inference in these conditions will simply 'roll down the slope' of continuing to decrease L until an equilibrium given by marginal condition reached. With a small enough learning rate, there should never be an "overshoot" where the supervised loss increases. While this argument technically requires an infinitesimal learning rate, in Figure 43, we plot the change of backprop loss across the inference phase throughout training for deep nonlinear network and find it is always negative, even with step-size of 0.1, implying that this condition seems robust to finite step-sizes in practice. Thus, we can interpret the dynamics of inference in another way, namely, constrained minimization of C
8

---

Published as conference paper at ICLR 2023
subject to the constraints of the energy gradient bound:
zi = arrgmnn L such that - -aC/on Tatiit//B//// 00/00, < 110//0x1112
(12)
Under the assumption that the inference phase always converges to some solution which must satisfy the marginal condition, it is clear to see that the solutions of the two optimization processes are the same. Crucially, this has allowed to reinterpret the inference phase as a minimization of L and not F. Intuitively, what we have done is reinterpreted an unconstrained minimization of F, which must find an equilibrium between a trade-off of L and É as instead a constrained minimization of L subject to the condition that the trade-off that it implicitly makes with É is not greater than the equilibrium condition allowed by the unconstrained minimization of F. The next step is to show that we can also express the weight update in terms of minimization of L and not F. Luckily this is straightforward. Recall from Theorem 3.1 that we can express the prediction errors at equilibrium in terms of BP-like recursion through the equilibrium values:
(13)
The reason that the gradients computed by PCNs differ from is that the BP gradients are evaluated at the feedforward pass values Tt, while the oj recursion involves the activities at their equilibrium values a7. This means, however, that we can interpret the weight update step in PCNs as performing BP through the equilibrium values of the network. This lets us express the weight update of the PCN in terms of a gradient descent on C. Since we can write both the inference and learning phases of the PC network in terms of minimization of the loss f., we can express it in terms of EM algorithm:
E-step (inference): 47 argmin,,, L such that - -OC/0at T 05/0m < ||0//Om|11 M-step (learning): argminw, Epcta? (.C.,,,))((((()))))) W)],
(14)
(15)
where we can interpret the posterior over xy as a delta function around the equilibrium value p(aaly, D, W) 8(2,- - );). Crucially, since L is guaranteed either decrease or stay the same each E and M step, the algorithm must converge to a local minimum (or saddle point) of L (Neal & Hinton| 1998). This occurs even though the E step is constrained by the energy gradient bound, since as long as the network is initialized on the correct 'side" of the bound, the minimization is guaranteed to at least decrease
small amount during the convergence and cannot increase unless the marginal condition is violated. To empirically demonstrate convergence to a minimum of the backprop loss, in Figure we took a simple case where convergence is straightforward full gradient descent on a trivial dataset of either a or 500 MNIST digits. We plotted the norms of the backprop gradient and full PCN gradient during training. In both cases we find that both the training loss and backprop gradient norm (which should be 0 at the minimum of the backprop loss) decrease rapidly and quickly reach close to 0 within a numerical tolerance. For a single MNIST input, the training loss goes to 0 and the backprop gradient norm stabilizes at approximately 10 16 indicating full convergence except for numerical errors. For the 500 MNIST digits, the training loss and gradient norms rapidly decrease the order of 10-4 and stabilize there, likely due to a finite learning rate size and potential insufficient expressivity to fully fit the batch. Nevertheless, these results demonstrate that PCNs can robustly and rapidly converge very close minimum of the backprop loss as predicted by our theory. In Figure 4E. we show that a similar accuracy is obtained by training on the full MNIST dataset between PCNs and BP, also validating our theory. Moreover, this equivalent performance and convergence occurs without the PC and BP gradients being similar. In Figure 4D. we plot the cosine similarity during training of each the PCN weight updates of layer of a 5 layer PCN to the BP updates and find that they remain consistently different across layers and during training. This further supports our main theoretical claim that PCNs exhibit different convergence behaviour to BP, but can nevertheless ultimately minimize the backprop loss function.
DISCUSSION
In this paper, we have provided a novel theoretical framework for understanding and mathematically characterising both inference and learning in PCNs. We have provided the first explicit mathematical characterisation of the inference equilibrium in the case of linear networks, explored how they vary with the precision parameter that determines the relative weighing of the feedforward and feedback influences on the network, and discovered a novel connection between PCNs and TP such that the inference equilibrium of PCN lies spectrum between the TP and BP solution. Secondly, have provided a novel mathematical interpretation of PC as a generalized expectation-maxiizzatoi algorithm where the inference phase can be expressed as a constrained optimization.
9

---

Published as conference paper at ICLR 2023
ACKNOWLEDEGMENTS
This work has been supported by BBSRC grant BB/S006338/1 and MRC grant MC_ [C__U___O 0U_00000000311 Thomas Lukasiewicz has been supported by the Alan Turing Institute under the UK EPSRC grant EP/N510129/1 and by the AXA Research Fund.
00003/1.
REFERENCES
Nick Alonso and Emre Neftci. Tightening the biological constraints on gradient- predictive coding. arXiv preprint (tt::204.11137.. 2021.
Andre M Bastos, W Martin Usrey, Rick A Adams, George R Mangun, Pascal Fries, and Karl Friston. Canonical microcircuits for predictive coding. Neuron, (664))695-711, 2012.
Atilim Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey. The Journal of Machine Learning Research, 18(1)55595-5637, 2017.
Matthew Beal. Variational Algorithms for Approximate Bayesian Inference. PhD thesis, UCL (University College London), 2003.
Yoshua Bengio. Deriving differential target propagation from iterating approximate inverses. arXiv preprint arXiv:2007. 15139, 2020.
Yoshua Bengio and Asja Fischer. Early inference in energy-based models approximates back- propagation. arXiv preprint 1rXiv:1510.02777, 2015.
Rafal Bogacz. A tutorial on the free-energy framework modelling perception and learning. Journal of Mathematical Psychology, 76:198-211 2017. URL https: /www. sc encediceiiicc rect
Com/
Christopher L Buckley, Chang Sub Kim, Simon McGregor, and Anil K Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55-79, 2017.
Andy Clark. Radical predictive processing. The Southern Journal of Philosophy, 53:3-27, 2015a.
Andy Clark. Surfing Uncertainty: Prediction, Action, and the Embodied Mind. Oxford Univer- sity Press, 2015b. URL http://bbwws.oooglee...ooooo////
Francis Crick. The recent excitement about neural networks. Nature, 337(6203) 1989.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):
1-22, 1977.
Karl Friston. Learning and inference in the brain. Neural Networks, 16(9):1325-133 2000,
Karl Friston. A theory of cortical responses. Philosophical Transactions of the Royal Society B: Biological sciences, (50(155)::155 (56):815-8366 836, 2005.
Karl Friston. Hierarchical models in the brain. PLDS Computational Biology, 4(11), 2008
Karl Friston. The free-energy principle: unified brain theory? Nature Reviews Neuroscience, 11 (2):127-138, 2010.
Karl Friston, James Kilner, and Lee Harrison. A free energy principle for the brain. Journal of Physiology-Paris, 100(1-3):00-77, 2006.
Karl Friston, Spyridon Samothrakis, and Read Montague. Active inference and agency: Optimal control without cost functions. Biological Cybernetics, (06(8-9):523-541 2012.
10

---

Published as conference paper at ICLR 2023
Zoubin Ghahramani, Matthew J Beal, et al. Graphical models and variational methods. In Advanced Mean Field Methods Thherry and Practice. MIT Press, 2000.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Andreas Griewank et al. On automatic differentiation. Mathematical Programming: Recent develop- ments and applications. 6(6):83-107, 1989.
Jakob Hohwy, Andreas Roepstorff, and Karl Friston. Predictive coding explains binocular rivalry:
An epistemological review. Cognition, 1086))687-701, 2008.
Yanping Huang and Rajesh PN Rao. Predictive coding. URL https://onninelirra
Wiley Interdisciplinary Re-
views: Cognitive Science, 2(5):580 593, 2011.
wi Ley.yy.. com doi wwwp//pp/f// 10020 htt.::aa....e..aaa.aooaaeooootr. http://w..aa.aaa./..////////aa/a
01110A/0GG7770TTTT000//////////0 http://wwww...ooooooooo/oo////// FV w11
@dbIGc1E2WG12.fNNC
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Dong- Hygu Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 498-515. Springer, 2015.
Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm Taylor expansion the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, pp. 6-7, 1970.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv:1605.0811 08004, 2016.
Joseph Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. arXiv preprint arXivv1887.09335. 2018
Alexander Meulemans, Francesco Carzaniga, Johan AK Suykens, João Sacramento, and Benjamin F Grewe. A theoretical framework for target propagation. arXiv preprint arXiv:2006. 14331, 2020.
Alexander Meulemans, Matilde Tristany Farinha, Javier Gareia Ordonez, Pau Vilimelis Aceituno, João Sacramento, and Benjamin F Grewe. Credit assignment in neural networks through deep feedback control. Advances in Neural Information Processing Systems, 34, 2021.
Beren Millidge, Alexander Tschantz, and Christopher I. Buckley. Predictive coding approximates backprop along arbitrary computation graphs. arXiv preprint arXiv:2006.0411 04182, 2020a.
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher L Buckley. Relaxing the constraints on predictive coding models. arXiv preprint arXiv:2010.01047, 2020b.
Beren Millidge, Tommaso Salvatori, Yuhang Song, Rafal Bogacz, and Thomas Lukasiewicz. Pre- dictive coding: Towards a future of deep learning beyond backpropagation? arXiv preprint arXiv:2202.09467. 2022.
Thomas Minka. Expectation-maxiin maxiiiizaiiaiiiiiiiiiiooo as lower bound maximization. Tutorial published on the web at http://wwwwwhit. media. mit. eduftmminka/papersseen html, 7:2, 1998.
David Mumford, On the computational architecture of the neocortex. Biological Cybernetics, 66(3): 241-251, 1992.
Radford M Neal and Geoffrey Hinton. view of the EM algorithm that justifies incremental, sparse, and other variants. In Leurning in Gruphical Models, pp. 355-368. Springer, 1998.
William F Podlaski and Christian K Machens. Biological credit assignment through dynamic inversion of feedforward networks. arXiv preprint arXiv:2007.05112. 2020.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-ficld effects. Nature Neuroscience, 2(1):79-87, 1999.
11

---

Published as conference paper at ICLR 2023
David E Rumelhart and David Zipser. Feature discovery by competitive learning. Cognitive Science, 9()::55-112, 1985.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, (23(688)) 1986.
Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, Rafal Bogacz, and Zhenghua Xu. Predictive coding can do exact backpropagation on convolutional and recurrent neural networks. arXiv preprint ArXiv:2 03725, 2021.
Tommaso Salvatori, Luca Pinchetti, Beren Millidge, Yuhang Song, Rafal Bogacz, and Thomas Lukasiewicz. Learning on arbitrary graph topologies via predictive coding. arXiv preprint arXiv:2201.13180, 2022a.
Tommaso Salvatori, Yuhang Song, Thomas Lukasiowicz, Rafal Bogacz, and Zhenghua Xu. Reverse differentiation via predictive coding. In 35th AAAI Conference Artificial Intelligence, 2022b.
Anil K Seth. The cybernetic Bayesian brain. Open MIND. Frankfurt Main: MIND Group, 2014. an-brain
net http://wwwo.a....//a//ea///eeeea
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484,, 2016.
Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Can the backpropagation in predic-
brain do backpropagation? Exact implementation tive coding networks. Advances in Neural Information Processing Systems, 33. 2020.
URL https://proceecings www.ineu...... //eaur/pps.ccppppee//2////aaaa 111111111a220fff-a111 Abst.rac...t........... html
Fec87a37cdeest
Yuhang Song, Beren Gray Millidge, Tommaso Salvatori, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Inferring neural activity before plasticity: A foundation for learning beyond
backpropagation. bioRxiv, 2022. doi: 10. 11111///2222...1774992233
URL https: /www
biorxiv.orge
05 1.....7...2...... 492325
Alexander Tschantz, Beren Millidge, Anil K Seth, and Christopher L Buckley. Hybrid predictive coding: Inferring, fast and slow. arXiv preprint arXiv:2204.022 169, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Kevin S Walsh, David P McGovern Andy Clark, and Redmond G O'Connel. Evaluating the neurophysiological evidence for predictive processing as model of perception. Annals of the New York Academy of Sciences, 1464(1):242, 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 681-688. Citeseer, 2011.
James CR Whittington and Rafal Bogacz. An approximation of the emror backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computation, 29(5): 1229-12633 2017.
A APPENDIX
A.I INTRODUCTION TO MAIN ALGORITIIMS
A.1.1 PREDICTIVE CODING
While originally motivated by inoomatiion-thorrcee concerns about compressing sensory data and
minimizing redundancy in the brain (Mumford 1992 Rao & Ballard 1999; Huang & Rao 2011).
12

---

Published as conference paper at ICLR 2023
predictive coding can be most elegantly derived as variational Bayesian inference algorithm (Beal 2003; Ghabramani et al., 2000). Specifically, variational methods aim to turn an inference problem into optimization problem by, instead of computing the intractable Bayesian posterior directly, minimizing an upper bound upon the divergence between the approximate posterior and true posterior. Mathematically,
formalize this process follows. We given some observations p(e.x)
o and latent state 2; which we are trying to infer. The true posterior is p(xlo) = p(o) where
p(o, x) p())))p(() is the generative model of the data, and is assumed known. Variational methods approximate the true posterior p(z by beginning with an approximate posterior distribution q and then fitting to the true posterior by minimizing the divergence K [[q()))) [p(2()))) By itself, this objective is intractable, since it contains the intractable posterior. However, the key insight is that we can instead minimize an upper bound on this objective called the variational free energy F.
F = Eyycoo) In (3rlo) p()
KL [p(]()] - pmp(
2 (L( [(x]
(16)
If we parametrize this approximate posterior distribution in some way, for instance with parameters 6, which we denote by q(x) o; 8), then we can write an abstract variational inference algorithm as simply minimizing the free energy,
H" = argmin F(8,0)
0
argmin (x(r)o0)) [Ing( - Inp(o,
(17)
To turn this specification into practical algorithm, we must first specify concretely what the approximate posterior and the generative model are, and secondly we must specify a method of minimizing the free energy. To derive the predictive coding algorithm, we first assume a hierarchical set of latent variables [zg ... z where L is the number of layers in the hierarchy and we set the lowest level to be the input data: Xn = D. We then assume that these layers can be related according to some parametrizable nonlinear function with parameters é such that 24 (1+1) From this, if we assume all 'noise' in the system is Gaussian, we can build a probabilistic Gaussian generative model p(Z0 ... x.) = II p(Z1|///1 = (2r (z)+11 +. at). Similarly, we can define the approximate posterior to be a Gaussian distribution factorized across layers g(01 ... x xx) = n a(2;; #). Then, under a Laplace approximation about the variances of the variational distribution (see Appendix for full derivation), the free energy can be considerably simplified to a sum of squared prediction errors (Friston 2005 Buuckley et al. 2017).
F(o,2,0) 11
(18)
Where e = 24 f(0, x141) which can be thought of the prediction between the estimated mean of the current layer, and the mean predicted by the mean of the layer above, where this prediction done using the nonlinear function f parametrized by 6. Finally, to actually minimize this objective, we perform a gradient descent on the free energy with respect to both æ and ø, for which we can compute the required gradients as follows,
OF
((41)
a Ōr
-1 + C11
dr
OF = E1
af(),))
à =
(19)
OΦ In practice, we typically optimize the æs first convergence, process we call 'inference" and convergence we run a single step of optimization of the Φ parameters, which we call 'learning'. This can be justified from a probabilistic standpoint as the x parameters are parameters of the variational density q<=]o; 2) while the é parameters are part of the generative model p(o, z; Φ). If we specialize predictive coding further, to the case of perceptron like artificial neural network where we have that at (Waat+)) ) where f is an elementwise activation function such as tanh or relu nonlinearity. and we split out the parameters into layerwise weight matrices Φ = {W.). we can write the learning rules as follows.
a6
क = -t + e411 (Whi)
Wi = -ei (W
(20)
13

---

Published as conference paper at ICLR 2023
Where I" denotes the partial derivative of the activation function. Due to the simplicity, and relative biological plausibility of its update rules, predictive coding networks have often considered to be a strong contender
unified theory of cortical function (Friston 2005; Friston et al. 2006 [012 Clark 2015a; Seth 2014). Moreover, as we will explore in more detail shordly, predictive coding the BP of error algorithm used
has been shown converge, under variety of circumstances, to train artificial neural networks, and thus may to provide a close link between modern machine learning techniques and biologically plausible theories of cortical function in the brain (Whittington
& Bogacz 2017
Millidge et al. 2020a Song et al. 2020).
The Gaussian generative model of predictive coding also provides an additional set of precision parameters which correspond to the inverse variance parameters of the Gaussian distribution which we denote II. These precisions modulate the strength of the prediction errors in the dynamics of the predictive coding network (Equations 19) such that we can simply define the "precision-weighted prediction errors' = IIc and otherwise follow the same dynamics.
A.1.2 BACKPROPAGATION (BP)
The fundamental algorithm which has underpinned the recent major advances in machine learning has been the backpropagation of enror algorithm. Arising in the 1970s (Linnainmaa 1970) and popularised the mid 1980s (Rumelhart Zipser 1985] Rumelhart et al. 1986), backpropagation was immediately applied to train connectionist artificial neural networks with some successes. However, only recently, with vastly increased computational power, and the harnessing of the innate parallelism of GPU units to train deep neural networks, has the true power and scalability of this algorithm become apparent (Goodfellow et al. 2016 Silver et al. 2016 Vaswani et al. 2017 Krizhevsky et al. 2012).
Backpropagaatio of error really refers to a more general suite of algorithms known as Automatic Differentiation (AD) (Griewank et al., 1989. Baydin et al. 2017), which take advantage of the chain-rule of calculus to allow for the computation of the gradients of any complex function made up of many differentiable subcomponents with numerical precision and with a cost that is linear that of the original function. The key idea is that given a complex functioo F' which is the composition of many differentiable component functions such that F' = f1 f2 o .... SN. and given some inputs œ, we may compute dr
OF by the chain rule as,
OF OfN 0fN-1 0f dr ON-10SSN- Or
(21)
This can apply not only to straightforward 'chain', but to an arbitrary directed acyclic graph (DAG) of function composition. The generality of this approach comes from the fact that very many complex computations can be broken down in just this manner into a computation graph composed of differentiable functions. Thus, if we know the derivatives 0fn of every component function, then JJ--1
can straightforwardly compute the derivative of the complex function in extremely efficient and accurate way. The backpropagation of error algorithm performs reverse-mode AD in that it evaluates the chain of derivatives in Equation 21 recursively from the left i.e. from the output of the network to the input. This can be implemented by accumulating gradients onto an adjoint vector 8 which is afn and then is updated according to the recursion,
initialized as 8N = 3fN-1
Ofn
8n-1 On 0O7-1
(22)
We can apply backpropagation explicitly to a standard feedforward neural network used for supervised classification to derive the necessary update rules. The network is composed of multiple layers, where the activity at each layer 2; a function of the activity of the layer below 2 = fWW[117-1) At the output layer FL. we are given a target T and we compute how well the output matched the target using a loss function L(x T). We then train the network to minimize this loss by computing the gradient of the loss with respect to the weight matrix for every layer DW. ac
Bc and then updating
each weight matrix according to this gradient W W + -aw, where 77 is scalar learning rate hyperparameter. To compute this gradient, we can apply backpropagation to the computational graph of the neural network. Typically, we first run the network 'forward' to compute the output ø and compute the loss L(x1, T'). Then, we begin the 'backward pass" which is where compute the
14

---

Published as conference paper at ICLR 2023
Predictive Coding
Backprop
Target Prop
ÖL Tpl
f(1i)
c|+1
f(m)
f(m)
OL
F(m(-1)
El
f(1(-1)
pp
Figure 5: Comparison of learning rules for predictive coding, backpropagation, and target propagation. Predictive coding first proceeds with an iterative error minimization step before weight update based upon prediction errors between layers. Backpropagation propagates gradients backwards in its backwards phase and then uses the gradients to directly update weights. Targetpropagation propagates targets backwards in the form of layerwise inverses, and then uses the differences between target and activation to update weights. We will later see that the inference phase of predictive coding interpolates between backpropagation and target propagation.
adjoints using the backprop recursion in Equation 22 Specifically, we initialize the adjoint vector as 5g ōxz.
ac and then recursively compute the adjoints of the layer below using the relationship.
= 61 (W/=()
(23)
Given the adjoints for each layer, we can then compute the weight gradient as aWL While simple and extremely effective, the idea of the brain directly implementing and learning through the backpropagation of error algorithm has been criticized biologically implausible (Crick 1989).
ac = (a)
A.1.3 TARGET PROPAGATION
Target propagation is an alternative algorithm for training deep neural networks which was constructed to be more biologically plausible than backpropagation (Lee al. 2015 Bengio 2020: Bengio & Fischer 2015). The core intuition behind targetpropagation (or targetprop) is that what we really want the activity at a layer to be the activity that, when passed through the forward pass of the network, outputs the desired label or target exactly. If we can compute this quantity which we call local targel- - then we can update the feedforward weights to minimize the difference between the actual activity and the local target, and thus ensure that eventually the feedforward pass of the network will compute the correct targets. turgetpropagation thus relies on a local error signal involving the local targets which is argued to be more biologically plausible.
we consider an output to be the composition a set of functions, for instance x4 = fa(fe(fi (zx))). then we compute the optimal local target Tp from the global target T4 the inverse of the functions To = fi (f, (Tt))). From this, we can obtain a recursive relationship for the local targets, given a feedforward ANN structure (x7 S(W which simply.
T T= (W, 'T441)
(24)
Or, in the linear case Ti = W-TTi4 Crucially, this requires computing the inverse of the weight matrix, which is computationally expensive and, more importantly, biologically implausible to
15

---

Published as conference paper at ICLR 2023
compute directly. Instead, the targetprop algorithms in the literature amortize this computation by utilizing separate learnable set of backwards weights W which are trained to minimize an autoencoder objective and thus ultimately learn to become an approximate inverse If we consider the linear case for simplicity, we can write out the key targetprop equations as,
Ti = WiT++1
Wi -ExY
é 1i WWm
Wi = -exf
I1 Ti
(25)
With the ideal case being where W W-1, In practice, learning the backwards weights often reduces in instabilities due to imperfect inverses and it has been found necessary to add 'correction' term the update rules accounting for the reconstruction loss of the implicit autoencoder formed by the backwards weights a correction known as differential targetprop (Lee et al. (015).
The empirical performance of targetpropagation in training artificial neural networks has been evaluated in number of papers (Lee et al., 2015; Bengio 2020, Meulemans et al., (020), and recently there has been a significant theoretical advance linking the procedure carried out by targetprop to the Gauss-Newton optimization method (Meulemans al. 2020 Bengio 2020) in the case where the target is simply a perturbation of the feedforward pass - Tr. - IL + Bag where is taken to be a small scalar value. Since this perturbation is small, this allows for the network to be lincarised around the initial feedforward pass values, and thus the computation of the targets implicitly computes a stack of inverse jacobians thus approximately implementing Gauss-Newton optimization.
Importantly, in this paper, we demonstrate a close and previously unknown relationship between predictive coding and target propagation. Specifically, predictive coding network, the equilibrium activity of the value neurons precisely equals the targetprop targets, and thus that predictive coding in this case can be thought of as iteratively computing the targetprop targets in way that eschews the necessity to learn a set of backwards weights to compute the approximate inverse. More generally, we can interpret the equilibrium activities of predictive coding networks as being a balance between the feedforward pass values and the targetprop targets, with the ultimate goal of the minimization of the prediction errors being to set the feedforward pass activities equal to the targetprop targets (this being achieved when the prediction error is precisely 0). Moreover, we show that depending on the relative precision between layers, the equilibrium activity values after inference can be seen as an interpolation between their forward pass values and the targetprop targets, and thus that the weight updates in predictive coding networks interpolate between those predicted by backprop and by targetprop.
show that when the input layer of the
A.2 PREDICTIVE CODING AND BACKPROP
While backpropagation has often been dismissed as biologically implausible, a set of recent works has demonstrated how predictive coding can converge to or approximate backprop weight updates in certain conditions. This means that predictive coding networks can be interpreted as performing or approximating the backproapgation of еттоr algorithm
potentially biologically plausible way.
Since predictive coding is generally considered more biologically plausible, and indeed is often thought to be potentially implemented throughout the cortex (Clark) (2055) Walsh et al.) [0200] Bastos al.) 2012), then these equivalences may be extremely important in understanding the brain's 's ability to perform credit assignment.
Firstly, Millidge et al. 2020a) showed that if we keep the feedforward predictions 'frozen' to their feedforward pass values even as we update the activity values x; in accordance with Equation 19 then can show that at the equilibrium of the network activities, the prediction error units satisfy the same recursive relationship as the backprop adjoints (Equation 22) and thus that if the final prediction error is set to the initial gradient of the loss function <,
dc then the prediction errors converge to
der
't is also possible to design algorithms that compute these local targets online. Methods that do this in the literature are known 'deep feedback control' (Podlaski & Machens 2020 Meulemans et al. 202T).
16

---

Published as conference paper at ICLR 2023
PC Approximates Backprop
@f(W++1=1) dr
61-1 =
is small
II
On the first non-zero step Song et 2020
Whittington et al 2017
Millidge et 2020
Figure 6: Summary of the results showing when predictive coding approximates equals backprop. Firstly, when the precision ratio between layers is small (so the lower layers are heavily up-weighted) (Whittington Bogacz 2017). Secondly, under the fixed-prediction assumption, that the opti- mization of each layer is decoupled (Millidge et al. 2020a). Thirdly, on the first update for layer when the prediction eтror at the layer below (Song al. 2020).
the backprop gradients and thus the weight updates in the predictive coding scheme are identical to backprop. This approach, however, requires the relatively restrictive exed-prediction assumption, which essentially prevents the convergence at one layer from influencing the convergence any other layer, since the predictions between them frozen at their initial feedforward pass values.
The key step is to notice that at the equilibrium, a recursive relationship for the prediction errors can be derived,
@f(W(412)
= of EL+1
(26)
Ox
Crucially, if we assume that this partial derivative af(Wi+121) is taken at the feedforward pass values, dez
instead of the current activity values (the frozen feedforward pass assumption), then this is exactly equivalent to the recursive relationship between the adjoints in backprop. Thus, if e = o = Bmr then the prediction errors (and thus the ultimate weight updates) converge to those computed by backprop for any neural network or, indeed, any arbitrary computational graph.
aL
Secondly, Song et al. (2020) recently showed that even we relax the fixed prediction assumption, the first update of the activity values also leads prediction errors precisely equalling the backprop gradients if the activity neurons x are initialized to the feedforward pass values xy f(Waz=i) 1).. This occurs because the prediction errors of the predictive coding network always satisfy the backprop recursion, and so converge to backprop whenever the activities are equal to the initial feedforward pass. This will always occur at the first step of the iteration as long as the activities are initialized to the feedforward pass values.
Finally, Whittington Bogacz 2017 investigated the impact of precision (inverse-variance) param- eters on the performance of predictive coding networks and demonstrated that when the precision of the output layer was very small, so that the activities at equilibrium were very close to their feedforward pass values, the prediction errors converged to the backprop gradients. In this paper provide a general analysis of this phenomenon and show that it emerges when the precision ratio between feedback and feedforward precision is very low -ie. that the feedforward path is weighted much more strongly than the feedback one. In such scenario, the equilibrium activities converge to approximately the feedforward pass values, and so the prediction errors converge to backprop.
17

---

Published as conference paper at ICLR 2023
A.3 BIOLOGICAL PLAUSTBILITY OF PCN LEARNING
A.4 CONVEXITY IN THE LINEAR REGIME
is straightforward to demonstrate that, in the linear case, the objective for a single layer (and thus the global objective which is just a sum of the local objectives) is convex and thus the global optimum can be easily located. This means that in the linear case there are no local minima and convergence is straightforward and rapid, which is indeed what we observe empirically.
The convexity can be straightforwardly shown by taking the second derivative of the free energy as follows,
F dup
= 1 + W Wi+1
(27)
And then noting that I W Wi+1 is positive semi-definite since,
5T + W Wi-1)z = =²=+ = W
W4+1= W44 || + 1|Wi-12112 2
(28)
for any arbitrary vector The fact that the second derivative of the objective function is always positive (indeed constant here) is sufficient to prove convexity.
A.5 PATH TO CONVERGENCE
Since the dynamics linear, also explicitly solve for the activities during convergence as a function of time as a linear differential equation. There is slight complication that not only the current layer is changing during convergence but also the layer above and layer below which affect the dynamics of the current layer. This difficulty prevents an analytical solution to the dynamics for the general case (even when linear), but we can solve for the condition where the layers above and below can be treated as fixed. This can occur in reality either if they are fixed to the data (or label) or else because the other layers are considered to be already at equilibrium.
In this case, we can write out the dynamics
+ 21 W W 1441 + W W4+1
(I + WTT - W - W
(29)
Since, if we assume the endpoints are fixed, then we can take - - W 2441 to be constant with respect to time. In this case, the dynamics for æ simply linear differential equation of the form è = Ar + b where A [ W W(+1) and = - Wiiz-11 W141-++1- Equations of this canonical form can be solved analytically with the solution for time index T.
ZT(T) (W (0) (1(+ I) W [W + 1441
(30)
If we do not assume that the endpoints are fixed (as in general multilayer PCNs), we instead get non-autonomous linear differential equation, with the canonical form Ax(()) + b(T). which has the following solution,
21(7) = e ( I)() + / ds - ( Wo-Nk [War- + 1++1()))7 (31)
Unfortunately in this case, since the feedback and feedforward components are themselves functions of time, this integral cannot be solved analytically and so instead must be approached numerically.
A.6 PROOFS FOR INFERENCE RESULTS
A. PROOF OF THEOREM 3.5
Recall that in the linear case, the activities of a layer have the following dynamics,
i Ile 11411411 WTT
(32)
18

---

Published as conference paper at ICLR 2023
By some somewhat involved algebraic manipulations we can explicitly solve for the fixed point of these dynamics to obtain an analytical expression for the precision-weighter equilibrium activities
iu = eet 11 W?" -
Tb: 11 W
M4. Wip =0 W
PIt W1-14111 -1.p:-----1----3 11+ W/ 1h WIJH =
(7 W ( Wi-1/111 1T, Ww 1
ai (1+ Wi)
(33)
Where we can observe the previous activity equilibrium in Theorem 3.3 special case where
11 = 11 = Precision matrices are always square so this case exact inverses are plausible.
A.6.2 PROOF EQUAL FEEDFORWARD AND FEEDBACK INPUT AT PREDICTION ERROR
By straightforward algebraic manipulation of the dynamics the linear case, we can determine the necessary condition for the prediction error at a layer to be 0,
= - W =0
[WeeL1 + W1+1 - (I+ W4
W7+1 - Wii
= WH+
W
W = -W
(34)
which occurs when the feedforward and feedback influences are exactly equal or, alternatively, when the feedforward prediction layer equals the targetprop target. Thus, the global minimum, where of = for all layers, is attained when throughout the network the feedforward pass value exactly equals the targetprop target or, alternatively, when the feedforward pass value exactly predicts the correct output target. This provides an additional intuitive reason why PCNs converge in practice, which is that their global minimum achieved training error.
A.6.3 DYNAMICAL PROOF OF SIPUT-UNCOSSTRAII
NETWORK
Here provide alternative proof of the fact that input-unconstrained PCN will converge the targetprop targets. The input being unconstrained means that for each layer, the layer below can, in theory, perfectly track the activity of that layer, meaning that consider 0. This means that the dynamics for the activities need only consider the feedback prediction error and can be written as,
W[(W+113))
f = -441
21
(x1+1 1()) (WLH)))WW
From this, we can directly solve for the equilibrium as follows,
t1 = 0 ( [f
31+1 ((W41m)) = 2 = -
(35)
Which is precisely the same recursion as that satisfied by the targetprop targets Equation 24
A.7 PROOFS FOR LEARNING RESULTS
A.7.1 PROOF PROSPOSITION 3.1
Recall that I. F. Thus, F then we have,
0 L+ É
(36)
19

---

Published as conference paper at ICLR 2023
Now, E is defined as sum of layerwise energies which are themselves (in standard PCNs) defined as a squared prediction error,
E
(37)
E
(38)
Since the squared prediction errors cannot be negative, É is non-negative. Thus, if we assume that L also cannot be negative (a standard, but not always fulfilled property of loss function), then we can obtain the desired result.
A.7.2 PROOF OF MARGINAL LEMMA
Recall the decomposition of the free energy into the backprop loss and the residual error F = L + E. If we differentiate this expression with respect to 21, we obtain,
OF
0c
+ 0E
(39)
Dzi
dzi
dzi
Now, at equilibrium we have that,
OF
31 = drl
=0 0
(40)
Thus,
OF
OL
OF
0
(41)
0x,
dri
dzi
(42)
Om
which is simply the marginal condition.
A.7.3 DERIVATION OF ENERGY GRADIENT BOUND
The fundamental condition we wish to preserve is that the supervised loss can never increase during inference, and we want to express this condition in terms of the balance of supervised and internal energies. We can express this condition as,
dC OC T dzs
50
(43)
dt dz, dt
for every sy. Using the fact that the dynamics of the EBM are gradients of the total energy, we can rewrite this
OC T
dx:
OL T OF
<0
(44)
dm
dt
Oxi Ozi
OL T OE
0L T 0L
(45)
0m
dri
0m 0x
Oc T
0E
Oc
||2
(46)
dm dzi
Dzi
which is exactly the condition presented in the main text. Also note that with equality this condition reduces to the margnial condition, since the marginal condition obtains equilibrium when the change of the loss must be
think of the energy gradient bound as requiring that the dot product similarity between the supervised residual loss be at least as large as the negative of the dot product similarity between the backprop loss and itself. In essence, this places bound the degree of negative correla- tion possible between the supervised and residual loss so that the residual loss cannot 'overpower' the supervised loss, although far away from inference both can become very large if they are correlated with each other so that a movement may induce a large decrease in both losses at the same time.
Intuitively,
Also of interest is that when the network is initialized to its forward pass values, supervised loss gradient is guaranteed to satisfy the bound.
0, then any dzr
20

---

Published as conference paper at ICLR 2023
A.8 RELATIONS BETWEEN PREDICTIVE CODING, TARGET PROPAGATION, AND BACKPROPAGATION
Returning to the missing data intuition behind the interpretation of predictive coding as EM algorithm, we can think about the inference phase of the PC network as performing a constrained inference of the activity values or, alternatively, performing Bayesian inference with priors. The inference consist of trying to infer the optimal local targets while also staying close to their feed- forward pass initializations. However, what happens if we perform an unconstrained minimization during the inference phase? If this is the case, then we will infer the optimal local targets, and hence the PC network will converge towards performing target propagation. This intuition verified by our experiments in Figure [ showing that when PC networks are input unconstrained (i.e. there is no prior keeping them close to the feedforward pass values) the equilibrium converges towards the optimal targetprop targets. Unconstrained predictive coding, then, is effectively an iterative form of target propagation where the targets are inferred dynamically for each batch whereas in classical target propagation the targets are inferred using a fixed set of backwards weights, which are learnt across a dataset, thus instead inferring the targets through amortized inference (Marino et al. 2018 Welling & Teh. 2011: Tschantz et al. 2022).
Another way to think about this is to return to the decomposition of the free energy into L and E. We can see that the magnitude of L controls the strength of the likelihood term driving the activities at equilibrium towards the targets while the magnitude of E controls the strength of the prior term keeping the activities constrained to their feedforward pass values. If we postulated parameter 0 S A < 1 that controls this trade-off, such that we could write the free energy F as,
(47) We could see that setting À = 1 results in target propagation since the inference procedure is unconstrained. Alternatively, by setting À = O, we obtain pure backpropagation, since the inference phase becomes trivial as the activities are constrained to exactly equal their feedforwad pass values. We can then understand predictive coding as being an interpolating solution with 0 < À < 1 between target propagation and backpropagation where À is implicitly set by the ratio of precisions between the output layer and the rest of the network layers. This result is shown graphically in Figure Z
F AL + (1 A)
Putting these aspects together also allows us to understand all the previous equivalences between backpropagation and predictive coding presented the literature. [Whittington & Bogacz (2017) found that predictive coding approximated backprop when the ratio of precision was strongest at the input layers of the network, decreasing exponentially towards the output. We can now understand this in terms of effectively setting are very close to the feedforward pass activities, meaning that LPC s LBP where we can define LAP as the backprop loss through the feedforward pass activity values and LPC as the backprop loss on the equilibrium activity values, and so the gradients computed by predictive coding approximate backprop. MMilidge et al. (2020a), on the other hand, show that predictive coding converges to exactly backprop under the 'fixed prediction assumption', This assumption essentially fixes the layerwise as(W==f) and thus directly enforcing convergence gradient a((W)= to the feedforward pass values 8m] dw; to the exact backprop gradients.
such that the E term becomes dominant, the equilibrium activities
Finally Song al. (2020) note that if the activity values are initialized their feedforward pass values, then the initial prediction errors for each layer except the output are O. then for each inference step, errors propagate one layer backwards through the network. They note that after the first inference step for each layer, when the error first becomes nonzero, the error is exactly equal to the backprop gradients. Again this is because at the first step of inference, the activities of the layer are precisely equal to the feedforward pass values. A simple way understand this is to recall the decomposition of the free energy into the residual and the backprop loss (Equation 11 F = E + L, noting that when the network is initialized to its feedforward pass values then the network is at a global minimum az 0. If we differentiate through the free energy
with respect to the activities - i.e., È = 0 and ab
decomposition with respect to the activities x, we obtain,
0F a5 Oc
dr
dr dr
O = 0
0F 0C
Or
dz
dz
21

---

Published as conference paper at ICLR 2023
Thus, the activity gradients at the first update away from the feedforward pass will be exactly in the direction of the gradient. Since we then use the activity gradients to directly compute the weight gradicnts, the weight updates on the first step are the exact backprop gradients. Note that this derivation applies to any energy based model with an additive decomposition of the energy as in Equation 11 and is not specific to predictive coding.
Lastly, our theoretical framework also provides an explanation for the fact that in practice predictive coding networks trained with redistribution work best when they are initialized to the feedforward pass values of the corresponding backprop ANN before the inference phase begins. This "trick' has been widely, albeit mostly implicitly, used in the literature (Song et al.) 2020 Salvatori al. 202
Millidge et al., [2020a] b) and it has been observed that predictive coding networks perform poorly or do not work at all without it. We can now see that the reason for this relates the condition for the constrained minimization interpretation of the inference process needing to be initialized on the 'correct side' of the marginal condition i.e. 0c
>
BE If, through random initialization, the Zur
Oz
< BE then the inference procedure, by convergence to the marginal condition will tend to decrease E by increasing L, thus violating the conditions for the convergence of the E-M algorithm. The reason the network appears to work best at the feedforward pass initialization is that here the network starts out with 0 prediction errors except at the output, meaning that E. 0 at initialization. This therefore the situation in which the network begins 'furthest away" from the marginal condition, and can therefore achieve a large decrease in L before the corresponding increases in E force the network to an equilibrium.
ac network starts out on the 'opposite side"; i.e. 0m dar
A.9 GENERALIZING PC TO OTHER ENERGY FUNCTIONS
As a consequence of these theoretical results, it is also possible to investigate whether the energy function of predictive coding can be generalized to give a family of possible energies and associated network architectures which all possess the favourable inference and learning properties of predictive coding networks and which may even lead to more effective inference learning. Retracing the steps of our argument, several key conditions are apparent. First, the total energy F must be bounded below, ideally with minimum 0. Secondly, the total energy must be expressable as a sum of the backprop loss and some residual energy E such that the marginal condition can be derived. Finally, for the weight updates to be equivalent to backprop through the equilibrium network, the prediction errors must satisfy the adjoint recursion backprop (Equation [2].
Although this condition of satisfying the adjoint recursion appears quite stringent, it actually allows quite degree of generalization. Suppose that our residual energy function E is still composed of layerwise energies which are composed of an 'interaction function' g(2) of the activities of the layer and the layer below, well as an 'activation function' h(g) which is function of the result of the interaction function. This lets us write the residual energy as,
E Σ, h(g(I1, (-)))
(48)
From this generalized formulation we can recover predictive coding by setting g = (z1- - f(W,, and h(x) = x With this definition of the energy, we can derive the dynamics of a layer.
11
Th1, 0g(si,
ahzia
Ōg( 0g(21+1, 0m:)
(49)
0g
dz
0g
Om
Where, for ease of notation, we shorten can recover the adjoint recursion if we set,
ah(y(e))::))) 0g(2).)) 1)
to
0h.: 8g
From these dynamics, it is clear we
()( Ög(af,
of(Waa))
(50)
dxt
Oxt
This condition essentially means that the ratio of the gradients of the interaction functions between the current layer and the layer below and the current layer and the layer above are equal to the
gradients of the feedforward pass of the layer f(Wzz)) This condition holds for predictive coding ((Wxxi) = āx;
because 0g(#ī _i_ = and )u(z(+1=7)) dzg adjoint recursion functions for any 'activation function"' h long the derivative ratio condition is
@f(W= Of note is that the
Ozt
d=;
22

---

Published as conference paper at ICLR 2023
Tbn
met. This is because, matter what they are, the activation function gradients the adjoint terms. This immediately leads many possible generalizations. One potentially useful one, which has already been independently discovered in the literature (Alonso & Neftci 2021) is to set h(z) = log(x). This gives rise to divisive prediction errors since
2g take the form of
thay becomes
Dg
Another, more principled, method of generalization is return the Bayesian interpretation of the inference objective derived in Equation 8 Notice that the inference objective is split into a likelihood term ply| [z, D, W) and a prior term p( D, W). Our decomposition of the free energy into the backprop loss and the residual loss allows us to associate the likelihood term with L and the prior term with E. This means that we can interpret the 'constraints' E in a principled way as Bayesian priors. Specifically, we can make the identication,
E = - (( W) = E, Inp(ar Wi)
(51)
Which derives from the layerwise composition of É or, equivalently, from the factorization of the prior. Given this understanding of the residual loss as Bayesian prior, we can understand that in the case of predictive coding networks this prior is Gaussian around the predictions from the layer below. Effectively, it enforces quadratic penalty to stop the newly inferred activities deviating too far from their initial conditions. The strength of this prior is controlled by the relative precision parameters in each layer. This leads to the suggestion that natural generalization of predictive coding would be to utilize other prior distributions, perhaps ones which are more natural for specific problems, and which depend the statistics of the data in question. For instance, if the data is strongly log-normal, or power-law distributed, it may be that using log-normal priors as well as divisive prediction errors may lead to better inductive biases for the network and ultimately more efficient inference and learning.
A.10 METHODS
A.10.1 INFERENCE RESULTS
For the numerical results inference in predictive coding networks described in this paper we used an extremely simple 3-layer predictive coding scheme designed to showcase our mathematical results in straightforward way.
All layers had five neurons and the input layer and output layer were fixed to the 'data' and the 'target' which in this case were randomly generated from Gaussian distributions with a mean of 1 and -1, respectively, and a standard deviation of 1. The middle-layer was free to vary according to the predictive coding update rules. For the nonlinear networks we used a hyperbolic tangent activation function. We typically ran inference for 100 timesteps using learning rate of 0.05. Weight matrices were randomly generated from a Gaussian distribution with a mean of 0 and a variance of 0.05.
For the results on the nonlinear fixed point iteration algorithm we utilized a randomly initialized four layer perceptron with 20 dimensional random Gaussian inputs and outputs and hidden width of 20 neurons. We initialized all weights from a zero-mean Gaussian with variance of 0.05.
For the precision experiments, we initialized mostly diagonal covariance matrices with a diagonal set to certain value and the value of other entries sampled from random gaussian with mean of zero and a variance of 0.1. To compute the precision matrices we then inverted the covariance matrices.
In the mathematical notation in the paper we freely use the matrix inverse W-1 However, in practice, weight matrices are usually not square. the numerical results we instead replace the true inverse with the pscudoinverse Wi, Although this makes many of our mathematical results only approximate, in practice, our numerical simulations have lead us to believe that this approximation does not dramatically affect the intuitions conveyed in this paper, so long as the layer-wise width ratios do not become too extreme.
A.10.2 LEARNING RESULTS
In the learning experiments, we utilized three layer MLP trained on the MNIST and FaShion-MNIST datasets. The MNIST digits were normalized to lie between 0 and 1. The network had hidden layers of size 128 and 64. The first two layers had ReLU activation functions and the output layer was an identity. The network was trained using the mean squared error loss function using a Nesterov
23

---

Published as conference paper at ICLR 2023
momentum optimizer using learning rate of 0.0001 and momentum value of 0.9. In general, results are averaged over 5 seeds and the shaded area represents the standard deviation.
Code to reproduce all experiments and figures in this paper will be made available after review at authub_address is also included in the supplementary material folder.
24


Reviews:
[{"review_id": "xcqjmM2EHhj", "reviewer": "Reviewer_3qX4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel theoretical framework for understanding the inference and the learning in predictive coding networks (PCN). The math derivations and proof are sound. ", "summary_of_the_review": "I like the deep math analysis of the inference and learning dynamics of PCN and it greatly gain our understanding of this network. ", "strength_and_weaknesses": "This is a paper full of math proof and derivations, and is theoretically sound. It is quite interesting in interpreting the PNC dynamics as performing a constrained EM algorithm, and comparing the PCN with other algorithms including target propagation and back propagation. The theoretical results seem novel based on the author's description but I am not quite sure since I am not an expert on predictive coding networks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IlUxdtR9hU9", "reviewer": "Reviewer_G2QQ", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While backpropagation is the mainstay technique used for artificial neural networks, it is considered biologically implausible, mainly due to the so-called \"weight-transport\" problem, wherein neurons need to know the weights of neurons that lie upstream in order to correctly update their own weights. Two frameworks that have been proposed to be more biologically realizable are predictive coding and target prop.  While both of these methods have been found to be related to backpropagation under certain circumstances, a fuller accounting of their interconnections has remained elusive. Through both mathematical analysis of linear networks and simulations of nonlinear ones, the present work demonstrates how backprop and targetprop approximately represent two endpoints of a continuum of solutions that predictive coding network inferences lie on (i.e., the equilibrium prediction errors lie close to the gradients of backprop on one end, while the equilibrium activities lie close to targetprop target activities on the other). Moreover, the authors show that predictive coding learning results in convergence of the network to the same minima as for backpropagation.", "summary_of_the_review": "The authors have analytically demonstrated the connections among predictive coding, targetprop, and backprop in linear networks. They back up the analysis with numerical experiments on nonlinear networks that demonstrate the same qualitative results.  This work represents an important and novel advance in understanding predictive coding networks, which currently represent one main direction within neuroscience for understanding cortical functionality.", "strength_and_weaknesses": "Strengths\n\n1. The authors provide a full mathematical characterization of linear predictive networks, demonstrating analytically their relationships to backprop and targetprop networks\n\n2. Furthermore, simulated nonlinear networks are analyzed and shown to produce the same qualitative results as found for the linear networks\n\n3. The authors' writing is clear and well-organized\n\nWeaknesses\n\nMajor\n\nNo major weaknesses have been \nobserved.\n\nMinor\n\nThe precision ratio is introduced after Theorem 3.4 as $\\Pi_{l+1} / \\Pi_l$; however, I believe the precision ratio comprising the x-axis values in Fig. 3C is instead $\\Pi_{l} / \\Pi_{l+1}$.  Either changing the figure to have the same ratio that is initially introduced or specifying the ratio that is used in the figure would help to avoid readers' confusion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3XuM7PsL2ls", "reviewer": "Reviewer_3s9Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper attempts to provide a theoretical analysis computation in predictive coding networks (PCNs), relating it to the more commonly used backpropagation-based (BP) training of feedforward artificial neural nets (ANNs).", "summary_of_the_review": "I wish this paper was what it promises to be. I believe far more careful work is necessary to substantiate the claims made here.", "strength_and_weaknesses": "Strengths: \nUnderstanding the relation between PCN and usual ANN with BP is definitely a worthy goal.\nThe observation that PCN ‘inference’ is a combination of a forward and a backward signal and that the balance between the two is decided by precision ratios. This insight may not be completely new, but is brought out well in the discussion.\nWeaknesses:\nThe proofs of some theorems seem to be spoiled by what looks like sloppy or confusing arguments. Take for example theorem 3.1. In the proof, equation 3 ignores the difference between $\\epsilon^*_{l+1}$, the equilibrium value, and $\\epsilon_{l+1}$, the backward propagated error, to make the argument.\nIn some cases, it is not clear whether some conditions are non-trivial or not, if taken literally. For example, for theorem 3.5, if one thinks $\\mathcal{L}$ depends only on $x_{L-1}$ and the target, the partial derivative $\\partial \\mathcal{L}/\\partial x_{l}$ would naively be zero for $l<L-1$, making the inequality trivially true.\nCalling the $x_l$ ‘inference’ via a local mode, not the expectation, the E-step Eq (14), and then invoking theorems about local convergence of expectation maximization seems confusing.\n\nMinor issues:\nIn equation (2), it seems $\\epsilon_l\\cdot f’(..)$ is column vector of the same size as $x_l$. Given that, in equation (1), should the order of $W^T_{l+1}$ and $\\epsilon_{l+1}\\cdot f’(...)$ not be reversed?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "0nTweqB7i_E", "reviewer": "Reviewer_c2Qw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I would like to begin by saying that this paper makes repeated and relentless reference to a given set of papers that induced this reviewer to suspect that it was self reference, and it took barely a minute of google search to verify this fact. I find this very disturbing, since the other papers I have reviewed have by and large played by the rules of double blind.\n\nThat said, this is a very interesting paper. The authors consider a feedforward ANN, and analyze an algorithm called prospective configuration that falls within the perview of predictive coding. Basically, given an data point and its label, the input and output of the network are clamped to their respective input and output values, and each intermediate layer searches thru gradient descent for input values that would minimize the squared difference between its output and the current output. This allows the error at the top layer to propagate downwards. Once equilibiirm has been reached, the input outputs of each layer is clamped and a gradient descent on the weight vectos is initiated, again until convergence. This process is repeated until convergence.\n\nWhat the paper shows is that this this algorithm converges to the same local optima that back propagation converges to. It also show that the algorithm builds a bridge between BP and target propagation (an alternative learning algorithm for deep networks).\nIt builds its results on the intuitive idea that the problem can be rephrased as a missing data problem, that is the activation of the intermediate layers) and can therefore be attacked using EM.", "summary_of_the_review": "This is a well written paper that provides theoretical justification for yet another technique (prospective configuration) for training deep networks.", "strength_and_weaknesses": "Strength: The analysis is quite comprehensive and connections to BP and TP are shown\nWeekness: Several of the results are for linear networks where the problem becomes trivial. Also, a lot of the results are similar to deep gaussian networks that came out many years back.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster