Predictive Coding beyond Gaussian Distributions
Authors: Luca Pinchetti
Tommaso Salvatori
Yordan Yordanov
Beren Millidge
Yuhang Song
Thomas Lukasiewicz
Venue: NeurIPS 2022 Accept (2022)

Abstract: A large amount of recent research has the far-reaching goal of finding training methods for deep neural networks that can serve as alternatives to backpropagation~(BP). A prominent example is predictive coding (PC), which is a neuroscience-inspired method that performs inference on hierarchical Gaussian generative models. These methods, however, fail to keep up with modern neural networks, as they are unable to replicate the dynamics of complex layers and activation functions. In this work, we solve this problem by generalizing PC to arbitrary probability distributions, enabling the training of architectures, such as transformers, that are hard to approximate with only Gaussian assumptions. We perform three experimental analyses. First, we study the gap between our method and the standard formulation of PC on multiple toy examples. Second, we test the reconstruction quality on variational autoencoders, where our method reaches the same reconstruction quality as BP. Third, we show that our method allows us to train transformer networks and achieve performance comparable with BP on conditional language models. More broadly, this method allows neuroscience-inspired  learning to be applied to multiple domains, since the internal distributions can be flexibly adapted to the data, tasks, and architectures used.

Predictive Coding Beyond Gaussian Distributions
Luca Pinchetti' Tommaso Salvatori' Yordan Yordanov
Beren Millidge² Yuhang Song! 1,2,t Thomas Lukasiewicz"
Department of Computer Science, University of Oxford, UK MRC Brain Network Dynamics Unit, University of Oxford, UK 3 Institute of Logic and Computation, TU Wien, Austria luca. pinapptthhetessss ox. ac. uk, tommaso iasaasossaaae .slvvttoriesss........o. ox. ac uk yordan. yordanovêcs ox ac ac.uk, beren. berem.mmllidee...............eee wwl.mmiillimdmmmm ox ac yuhang. song@some ox.ac. ac. uk, thomas lukasiewi ox. uk
Abstract
A large amount of recent research has the far-reaching goal of finding training meth- ods for deep neural networks that can serve as alternatives to backpropagation (BP), A prominent example is predictive coding (PC), which neuroscience-inspired method that performs inference on hicrarchical Gaussian generative models. These methods, however, fail keep up with modern neural networks, they are unable to replicate the dynamics of complex layers and activation functions. In this work, we solve this problem by generalizing PC to arbitrary probability distributions, enabling the training of architectures, such as transformers, that are hard to approx- imate with only Gaussian assumptions. We perform three experimental analyses. First, we study the gap between our method and the standard formulation of PC on multiple toy examples. Second, we test the reconstruction quality on varia- tional autoencoders, where our method reaches the same reconstruction quality as BP. Third, we show that our method allows us train transformer networks and achieve a performance comparable with BP conditional language modcls. More broadly, this method allows neuroscience-inspinn learning to be applied to multiple domains, since the internal distributions can be flexibly adapted to the data, tasks, and architectures used.
1 Introduction
The last decade has seen an explosion machine learning research fueled by the collection of an unprecedented amount of data and the development of models that make of it. Starting with AlexNet [Krizhevsky et al., 2012], deep neural networks trained with backpropagation (BP) [Rumelhart et al., 1986] have been established the best-performing models in many fields [Silver et al., 2016, He et al., 2016, Brown et al., 2020, Ramesh et al., 2021, Devlin et al., 2018J. Despite reaching human-level performance in several tasks [Silver al., 2016, Vinyals al., 2017, 2019], we are still far from artificial general intelligence. The trend has been to constantly increase the number of parameters in such networks, from millions [Devlin et al., 2018] hundreds of billions [Brown et al., 2020]. The limitations and drawbacks given by the large size of modern architectures have motivated research that looks for alternative methods train them. The direction of research that has inspired this work is that of neuroscience-inspiee overcome these drawbacks, due to both interesting properties their credit assignment, such as plasticity [Hebb, 1949], and their biological hardware [Kendall et al., 2020]. These methods have two main advantages relative to standard deep learning models trained with BP. First, it much more feasible to train them analog and neuromorphic chips [endalll et al., 2020). Second, the resulting
alternatives to BP, which promise to
Corresponding author.
36th Conference on Ncural Information Processing Systems (NeurIPS 2022).

---

computational models are extremely flexible in both network layout design and querying techniques (Salvatori et al., 2022a, Millidge al., 2021, Salvatori et al., 2021, Papadimitriou al., 2020J. These two properties could play a crucial role in overcoming the limitations of BP-based learning towards artificial general intelligence.
Predictive coding (PC) is onc of the most influential thcories of information processing in the brain, initially proposed to explain a large number of brain behaviours [Mumford, 1992, Friston and Kiebel, 20091, and now also topic of rescarch in machine learning, thanks to the computational model proposed by Rao and Ballard [1999]. This method has in fact been used in supervised and unsupervised tasks [Ororbia and Kifer, 2020, Whittington and Bogacz, 2017, Han et al., 2018), with an important theoretical result: the original formulation is able to approximate the weight update of BP [Whittington and Bogacz, 2017, Millidge et al., 2020], and to exactly replicate it when introducing small variations [Salvatori et al., 2022b. Song et al., 2020]. These results are important, as they draw strong connection with the aforementioned results obtained by BP in the last decade. PC, however, also presents several interesting propertics that make it different from BP: it has an energy-based formulation that allows the design of powerful associative memory models [Salvatori et al., 2021] and to train graphs of any topology [Salvatori ct al., 2022a].
PC can also be studied from an information theory perspective, as it is hierarchical generative model [Rao and Ballard, 1999]. This a strength of the model, as has allowed such models to achieve a competitive performance to standard models trained with BP on several generative tasks. These results, however, are all obtained on simple models: sequential architectures with element-wise activations and quadratic energy functions. Deep learning, however, has progressed far from those in recent years, and, therefore, it is necessary to evaluate the performance of PC on more up-to-date and complex architectures to obtain complete comparison of PC against BP. In this paper, that the strict Gaussian assumption is limiting when dealing with more complex architectures such as transformers, preventing PC from reaching the performance obtained by BP. Note that this limitation is not unique to PC, but it is shared among all seuroscience--nssiree methods: for example, to our knowledge, none of these frameworks has been successfully used to train language models date.
In this work, we address this problem by generalizing PC to arbitrary distributions. This allows us to both use Gaussian distributions when allowed, and also to deal with intractable ones by approximating them using a sampling scheme. The resulting framework is coherent with the PC theory, as enables the definition of layer-wise energy functions that represent prediction errors [Rao and Ballard, 1999, Whittington and Bogacz, 2017]. In standard PC networks, the error is given by the difference between the expected and actual imput of a layer; here, it is defined as the KL-divergence between the expected and actual distributions. We show that these formulations are equivalent when using Gaussian distributions, meaning that our proposed framework is generalization of standard PC. The results of this paper are briefly summarized as follows:
We generalize PC beyond the assumption of a Gaussian generative model. This lets us define prediction errors "distances" between arbitrary distributions at each hierarchical layer of a PC network, and derive a novel free-energy objective to train such networks.
We empirically show that standard PC is ineffective training models with complex structure and activation functions, as they cannot be approximated by Gaussian generative model. Our proposed method, instead, significantly outperforms it, reaching a competitive performance with BP in training both variational autoencoders [Kingma and Welling, 2014] and transformers [Vaswani et al., 2017J. This further bridges the gap in performance between statc-of-the-art deep learning methods and neuroscience-insprred learning.
The rest of this paper is structured as follows. In Section 2, we introduce the notation used throughout the paper and describe the probabilistic interpretation of PC. In Section 3. we propose how generalize the defimition of PC beyond Gaussian generative models. In Section 4, we evaluate the performance of our proposed method in comparison to standard PC and BP, Finally, in Sections 5 and 6, we discuss related work and provide a conclusion, respectively.
2 Predictive Coding: A Probabilistic Interpretation
In this work, we focus on multi-layer networks organised in sequence of L layers. These networks define a model Ma that learns the relationship between the input and the target o by updating
2

---

aPg
MaP d
va ue impleitl activation
32g
RF1 ang
En
ue held oxp expiiiitlly
MPc
6g
weight
Figure I: Difference between a network trained with BP (up) and PC (down). The nodes ø of cach layer store the extra parameters d1. By following the computational arrows backward, the error flows globally from the last layer to the first when using BP. With PC, instead, each layer computes a local error that gets propagated only to nearby nodes.
its weight parameters (01...., 0r). Networks trained with BP have forward pass, where an output Ó is computed from input d, and a backward pass, where the eπror given by loss function loss(o (o, 6) is propagated back to the network. Every layer of the model has a sct of value nodes It, which contain the value of the prediction μu (i.e., what is normally called activation of a layer), computed accordingly to the value of the previous layer. In detail, we have μ4 = f(( 81), where stored directly inside the value nodes 21,
ft is activation function. In PC, the prediction luz but rather in separate prediction nodes. The nodes xy hold, instead, the neural activation di of layer I. Φi is a parameter of the network that can be optimized by minimizing the so-called prediction error between and itself. Therefore, model trained via BP consists exclusively of the parameters 0, while a PC network requires two sets of parameters, 0 and where & (Øo, ................................ ΦL) represents the neural activities of different layers (Fig. 1). Table summarises the differences between BP and PC.
Method
Value received from I 1-1 Content of value nodes Tι
Value fed to layer + 1
BP
μ4
prediction from / (pt)
PC
μi
neural activation (Φt)
Φ1
Table 1: Clarification of the notation used, highlighting the differences between BP and PC.
2.1 PC as Variational Inference
PC as a learning algorithm can be mathematically interpreted variational inference problem. is in fact possible to consider learning as an intractable Bayesian inference process that can be approximated with tractable optimization problem [Friston, 2003, 2005, 2008]. Under this assumption, the neural activitics Φ, in a PC network represent probability distributions. In particular, Friston based his theory of PC Gaussian generative models. A detailed review is provided in [Millidge et al., 2021]. Assume that we have a generative model = g(z). where o is a data point and set of latent variables, which described by the joint probability p(o, x) = p(o(=))(xx) We need to solve an inverse problem: given a data point o, we need to infer the causes x that generate through g. Similarly to many inverse problems, this compute the true posterior p(z(o). However, computing it by Bayes rule as p(z)o) p(o, <)/p(o) requires the normalizing factor p(o) J p(x, o) dr, which is, for all but trivial problems. intractable. Variational inference aims to approximate the intractable posterior with a family of distributions (z(z))) where the parameters á have be learnt. This is generally done via gradient descent on the KL divergence [Kullback and Leibler, 1951] between the approximated and true posterior. The goal compute
is intractable. In particular, we want
d argmin DKL [p([))]]
(1)
by minimizing an upper bound on the divergence, called the variational free energy F: x)] (=ll)(()))))ppo...))) In p(o) - D
F =
(2)

---

The PC framework assumes a Gaussian form for the generative model p(o, = p(o)x))(())
N(;; [(x, 0), E2)NN(x Jz, H4, Σ1), where E2, Σt, and μ are prior parameters that can optionally be learnt. Using variational posterior the Dirac-delta distribution" 9p(x1l) 6(α Φ), we get that
J = (ttoo (o((oo) (uro)
(olcco) [In æ)] = Boo [In a)] In o), (3)
where the entropy of q is O. This scheme can be applied to deep neural networks, where x does not represent a homogeneous latent space (e.g., a single layer), but is, instead, organised in a hierarchical structure, defined by the multiple layers of PC network with widths w1,..., Wr and nodes To, Ti,..... LL. Under this premise, the generative model as follows:
p( p(zn) II. p( = - Po, En) IT, Nini M4, Ed),
(4)
where μn f((21-1, B1), and Iy corresponds to the observation layer and is set to ry during training. The parameters Σ, are prior diagonal covariance matrices, which can be optionally learnt, and Ao is an arbitrary prior that can be set to some given data d. This is equivalent to the training of a supervised network with data point and label o. The energy becomes:
F = -eg (m = -'' 1== - Inp(())))) = 1 E 1=0
Σ
--12 + In Ex,)) + h; (5)
21, 11 i=
where l is a constant, e = (Ør mi), and Is (zo.z. ld, o) = II δ(zι Φt) (implying that x = Φ1). In this equation, the total energy is given by the sum of the energies & of every layer, where El == In p(p((l)))) By assuming identity covariance matrices (i.c., En = I)², the energy becomes the sum of quadratic errors, introduced by Rao and Ballard [1999]:
F = Σ & E
of-
(6)
In most cases, however, the generative model depends on a set of parameters 0: p( F2; 0) that need to be learned according specific dataset. This can be done via expectation maximization (EM) [Dempster et al., 1977], where we first infer the best possible latent variables é given a data point (E-step) and then use them to update the parameters 0 (M-step). In practice, both these phases are achieved using gradient descent to minimize F. In detail, given labelled point (d, o), the input layer prior is set to μi d, while the output layer nodes are fixed to Φr = o for both the inference phase (E-step) and weight update (M-step). During the inference phasc, the weight parameters are fixed, and the node values @ are continuously updated via gradient descent to minimize the energy F. This process either runs until convergence or for a fixed number of steps T. When the inference phase ends, weight update is performed as follows: the node values & are fixed, and the weights are updated once via gradient descent on the same energy function F.
Generalization to Arbitrary Distributions
In this section, we go beyond the strict Gaussian assumption of PC. According to Eq. 5, we have that & = - In p(pll1 = - In In (Øt fi( 6)), Σ,)). We can highlight the role of each 6,, by introducing the and s superscripts, which, respectively, indicate that a vector value is interpreted as a distribution (i.e., a vector of sufficient statistics that uniquely identifies a probability distribution, such as the mean u of Gaussian distribution), or as a single sample. The S notation does not, in any case, imply any transformation of the value itself. We get that
E = In ( fi( 61), Ex) = In 10 11, Et).
(7)
Thus, neural activation Φ simultaneously interpreted both as of and op This subtle difference has never been highlighted in standard works using hierarchical Gaussian models, since of corresponds to the maximum likelihood estimation of the Gaussian distribution ((ff Σx), and thus of o..
"A Gaussian variational posterior under the Laplace approximation can also be used, resulting in the same learning rules as the PC framework proposed here; see [Buckley et al., 2017].
'Throughout the paper, we assume diagonal covariance matrices for the Gaussian generative model in order simplify the mathematical derivations. However, our approach can be naturally extended to the case of general covariance matrices [Bogacz, 2017].
4

---

E3
PC₃
KDP
8++1
New Interpretation
E3
PC
oi
$i 11+1
Figure 2: Different layer structure between PCy (left) and PCFRı (right). In PCy, the nodes of each layer are simultaneously interpreted as both samples (when evaluating &₁) and as distributions (when computing #1+1). This inconsistency (highlighted by the dashed red arrow) is not present in PCTEL where they always represent probability distributions.
Assuming a Gaussian form for the generative model could, however, become a limiting factor when considering complex architectures. For example, layer employing a softmax activation cannot be easily approximated through multi-variate Gaussian, since the function introduces strong dependency between the nodes (i.e., their values have to sum to 1). As we will show in the next sections, using softmax activation functions can hinder the ability of network to learn using the standard definition of PC, and hence generalized formulation that goes beyond the standard Gaussian assumption is needed.
To generalize and extend the canonical formulation of PC, we do the following: instead of directly maximizing the likelihoods pt == pf lnP), we assume that this process happening repeatedly between each pair of consecutive layers for single optimization step. We consider the nodes 2, as storing the sufficient statistics of of an arbitrary distribution ((P)), we draw N sample (L,....NN) and minimize each individual likelihood pi p(s(") InP)). Furthermore, we remove the Gaussian assumption and consider ni, = Ji 0:) be a parametrization of generic distribution (MP). By doing so, the node values di are interpreted exclusively distribution parameters: both the energies & and the activations ju are functions of pP. It follows that the variational free energy F is also function of the expected values given by the likelihoods p(s) IpP), i (1,...,N, N), for each layer 1. The energy of each layer is then defined as:
points from it, s(") (oP)),
& = - lnppoplpp IPP) Ne H(X((O)), ((()))
(8)
A detailed derivation of the above equation is presented in the supplementary material. Knowing that the cross-entropy between total energy of the network can be optimized by minimizing
distributions H(a, 6) D [a/bal H(a) and that H(a) 2 the
FKL = Ei
Tr & = En_ DKL[Xi(O (1I Σ (.
(9)
This follows, as the entropy of (((P)) can be assumed being constant depending the training data. Figure highlights the difference between the original and new PC formulation. This definition of the variational free energy FuL does rely on the Gaussian generative assumption, and (as long as the KL divergence between X; and & can be efficiently computed) there no limit on the kind of distribution that can be used. Throughout our experiments, we assumed that the distributions X; and X; belong the same family. but different families can be used in different layers. In the supplementary material, we show how FK is equivalent to F when assuming a Gaussian generative model. We also analyze the learning dynamics defined by applying the expectation-maximiaze (EM) algorithm to Eq. (9).
We use notation (1(44) to emphasize the dependency of distribution & exclusively on the parame- ters uu For example, & could be Gaussian distribution, and μu represents mean and variance.
5

---

4 Experiments
We compare the results of different models trained with BP and PC various tasks. The main goal is to make PC competitive with BP for complex deep neural architectures. Only recently, PC has begun to be applied train neural networks for classification tasks [Whittington and Bogacz, 2017], with similar performance but a greater versatility compared to BP [Salvatori et al., 2022a]. We show that our extended probabilistic formulation can be applied more complex architectures. We refer each specific version of PC by specifying its energy function.
MNIST classification
100-
-010
100
3.0
100
95-
96
75-
PCp
Rccuscy
PG
0.05
85
85
25
Accussy
Lows
10
20
10
20
10
30
CIFARIO assification
Accumcy
50-
40
6.05
40
1.5
30-
30*
201
1.5
20
0.00
20
1.0
0-
1.0
20
10
M
M
Figure 3: Classification performance of the three models on the MNIST and CIFARIO datasets. PCFx1, noticeably outperforms PCy, reaching performances comparable with BP. This is true, especially for M2, which reflects the most commonly used architecture among the three. The x-axis represents the number of epochs.
4.1 Classification
As proof-of-coneept experiment, we have trained different models on classification lasks for both the MNIST [Deng. 2012] and CIFARIO (Krizhevsky et al.] datasets. We evaluated the final accuracy of the model when training with BP compared to PC, as well as the training efficiency, measured as improvements over epochs.
Setup: We defined three variations of fully connected network with L 3 hidden layers and
width w = 512:
MM uses tanh as activation function for the hidden and final layers. The mean squared error (MSE) is the loss function used for BP and compute the test loss of both PC and BP.
M2 uses the softmax activation function for the final layer. Consequently, the loss function used is cross-entropy (CE). This architecture corresponds to the one normally used in classification tasks.
M3 is a copy of M2 where the activation function of the second hidden layer replaced with softmax. CE is again the loss function used.
Note that the experiments performed with a softmax activation in a hidden layer are merely presented with the goal of empirically validating the effectiveness of our thcory, as networks of this kind have never been used in practical tasks. Effectively. M2 represents the only widely used architecture. We used a weight learning rate of 3g = 0.0001 for both PC and BP. For PC, we used T 32 é-steps. We assumed identity covariance matrices for the Gaussian distributions of the generative model. Consequently, the energies and Fx differ only for the function used for a soflmux--ctivated layer. For FKL, that is
Elyaarm = DKL ( 21 X
201 (OLS) In(
(10)
PL.
where Xi and & are discrete distributions. Therefore, in the model Mi. PC and PCF are algorithmically equivalent (see the supplementary material). More details about the hyperparameters that we used are given in the supplementary material.
6

---

Tb41
2ib Sp-1
Encoder
O
0000
Bapsoe D
Encoder
O
RG
0
Decodar
199
Figure 4: In PC= (left), Σg is stored as an optionally trainable parameter and does not depend the input d. If we were allow it, and use Σ, to generate 14++1 (red dashed arrows), we would violate the PC locality assumption, the enror coming from the decoder would flow through E, back to the encoder. Using PCFEE (right), instead, possible to have such a dependency by modelling both Eg and Eg.
MNIST reconstruction
8P PC5e
0.06
7
/
4
4
9
Original
7
2
/
4 0
6
4
9 9
9
9
PCrc
7 7
2
9
/ /
4 4
9 9
9 9
BP
7
2
0
4
4
0.02
16
20
30
Epochs
Figure 5: Comparison of BP and PC in training a VAE on the MNIST dataset. The graph on the left shows the test loss over the epochs. The solid and dotted lines represent two different models. Overall PC and BP perform similarly.
The results are plotted in Fig. 3. They show that neural networks trained with BP and PCrx, perform similarly in both M1 and M2. The original formulation of PC, on the other hand, performs much worse in all the considered experiments. The first column of plots shows that PC and BP perform similarly when trained with fully Gaussian assumptions. In the experiment on M2, the standard definition of PC is performing well, even though it is significantly outperformed by BP and PC When training on M3, however, the performances are poor. This again does not happen when training using PCFEL which obtains performances comparable to those of BP. Overall, this clearly indicates that PC= is not suitable to train neural networks that do not rely uniquely on Gaussian generative models. Instead, we experienced solid improvement when using the FKL energy function. It is enough to reach the same accuracy-over-epodee ratio achieved by BP when using M2. Under more uncommon architectural choices, such MM, the ratio is slightly worse favour of BP, but still decisively better than PCr. The difference is particularly noticeable in the first epochs. We believe that may be due to a not ideal initialization of the weights for the PC network, which is currently using the default initialization designed for BP networks [He et al., 2015). Further research in this direction could improve the training performance.
Results:
4.2 Variational Autoencoders
Variational autoencoders (VAEs) [Kingma and Welling, 2014] are models that rely on distributions different from Gaussians with fixed covariance matrix. This follows, as the bottleneck layer b of a VAE required to model the distributions of both the mean and variance of the latent posterior given a sample d, p(u,, Es/d). However, both PC= and PCF are not suitable for that, as they require each layer to represent exclusively the Gaussian mean u. The optionally learnable parameters Σ, do not depend on the particular input sample d. Our proposed method, however, overcomes this limitation by learning the full posterior distribution Na(us, Eb). This is done by considcring the bottleneck layer b as storing the distribution parameters ob = (ug, Ez). In this case, dul = (ū,, Σx) = f 0%). We then employ the reparametrization trick [Kingma and Welling, 2014] by sampling some Gaussian noise ~ (0, 1) to compute Pb+1 = foti(us + w eDiag(E)) 0b+1). which is fed to the next layer.
More details are shown Figure 4.
7

---

15 BP Pora
3 5 3
3
3
3
5
3
3
3
0 0 0 0
0
0
0
0
0
4 9
4
9
9
4
H
4
4
Posterior samping
5
4 3
3 3 3 3
4
7 7
9 9 9 9
0
-5
Posterior traversal
10
15
10
Figure 6: The analysis of the latent space does not highlight any significant differences between PC and BP. UMAP was the algorithm used to obtain the 2D projection [MeInnes ct al., 2018). Sampling from the posterior (of the PC-trained model) does not show any anomalies as well.
Setup: We trained multiple VAES on the MNIST dataset, comparing the PC and BP training algorithms. Our architectures employed fully connected layers, with Hardtanh activation function for the hidden layers and sigmoid for the oulput layer. The bottleneck layer has a total of wa 32 16 + 16) latent units. When training with PC, we assumed Gaussian distributions with identity covariance matrix for all but the bottleneck layer, for which diagonal covariance matrix Σg as explained above. We used T = 32, but for cach data point we sampled single at O. We use the same weight learning rate 3g 0.0001 for both BP and PC. Further details about hyperparameters and implementation details are described in the supplementary material.
model both the mean us and the
Results: We observed similar results on wide range of hyperparameters. In Fig. 5, we report the performance of both training methods on two different architectures. The final test loss is overall similar, with neither method being decisively better than the other. The learning curves are also comparable, despite PC being generally faster than BP in the first training epochs. By reconstructing the maximum likelihood estimation some data points, we can observe how all models produce very similar images. We also performed an analysis of the latent space produced by the encoders and did not detect any significant difference between the two training modes. Figure reports the results. We sampled the latent posterior distribution by encoding data point d and decoding multiple data points d' obtained by sampling from (Mb, ).). To perform a latent traversal, we encoded two different data points, di and d2, and reconstructed the maximum likelihood estimation of the vectors obtained by interpolating the two latent embeddings.
4.3 Transformer Language Models
To show the performance of our method on more complex tasks, we have trained transformer conditional language models based on the BERT architecture [Devlin et al., 2018]. The conditional language model objective is enforced by modifying the self-attention mechanism with triangular mask so that no position can attend to any later position in the sequence.
Setup: We use the 1B Word Benchmark dataset [Chelba et al., 2013], from which we randomly sample 200,000 training and 10,000 dev instances. We choose restrict the model's input length to 32 input tokens from a vocabulary of 8001 tokens generated by the BPE-based SentencePiece tokenizer Kudo and Richardson, 2018]. We use one transformer block with one head and hidden size of 128 throughout the model. For the PC networks, we assume Gaussian distributions with identity covariance matrix for all but the layers that employ a softmax activation function (i.e., the attention layers [Vaswani et al., 2017]). In the latter case, we assume a categorical distribution for the generative model. Consequently, the energy function for those layers is the one defined in Eq. (10). More implementation details and the hyperparameters are given in the supplementary material.
For each model, compare the three training methods B.P, PCy, and PC We found beneficial to run multiple weight updates for a single training iteration when using PC but not for PCr, where led instability. We run a hyperparameter search for each training method, select the best models, compare their training curves, and their test performance, and show qualitative examples of model predictions. Figure 7 shows that PCFÆE significantly outperforms PCF in
Results:

---

PCy
PC
1000
BP
Training Method
Test Perplexity
Perplexity
500
BP
162.64 = 0.76
250
PC
175.90 + 1.74
PCy
590.08 12.60
Epochs
Figure 7: Left: Comparison of language models trained with BP and PC, as shown by dev perplexity. Right: Test perplexity achieved by the various training methods for transformer language models. Average (d:o) of 10 seeds.
terms of test perplexity, while having more stable training curve. We believe that this is because of the softmax in both the attention mechanism and the output layer of the transformer model. The performance of PC is close that of BP and both training curves look stable. In practical terms, the language models trained by BP and PCrK₁ do not differ significantly on the test set. In some cases, the predictions given by PCFÆL bank and its executives are still ready to support specific Democratic [candidates]", PCFxL predicts "leaders" and "candidates" top-2 choices. All models show failure in commonsense reasoning, e.g., for "I've been dreaming about this since was a [child]" they fail to assign "child" with > 1% probability, which shows the limitations of small language models. More examples are given the supplementary material.
closer the ground truth, e.g., for "Yet the
Memory Consumption and Computational Complexity
Memory consumption: PC, in contrast to BP, stores μ and ó as different variables, which results in some memory overhead. On the contrary, the number of weights and training parameters used does not change between BP and PC. Therefore, if MBP is the memory consumption training a model using BP, have that, in general, MPC 2 MBP. Actual values depend the architecture and hyperparameters chosen.
Computational complexity: the complexity operations comparable between PC and BP. The same be said for the backward pass. However, in accordance with the EM algorithm, it is necessary to perform multiple updates on the neurons before updating the network weights 0. This results in multiplicative factor that performance compared to BP. Nonetheless, from our experiments, we noticed that even a value as low T 4 or T 2, where T is the number of updates of the neurons before performing update of the parameters, is sufficient given the right hyperparameters. In fact, the experiments on the transformer reached the best perplexity with exactly 7 = 5. Furthermore, of the features of PC. One of its major strengths is that each layer computation (both in the forward and backward pass) local and, therefore, can be execuled in parallel, removing one of the main bottlenecks of BP when training deep networks (i.c., the computations induced by each layer have to be executed sequentially). Thus, we expect PC scale well large architectures and to bring huge improvements on neuromorphic hardware. Finally, it has already been demonstrated that the speed of energy- based networks can be greatly increased by implementing the relaxation on analog hardware [Foroushani et al., 2020, Hertz et al., 1997], potentially resulting in energy-based networks being faster than BP. Thus, one scientific indication of this work is that the "analog-harl hardwarr-daaaaaeeeere hardwarr-friendly" PC can have a reasonable performance on transformers, which opens the door to designing fast hardware-implemented transformers.
single forward pass terms the number of
impact
can take advantage
6 Related Work
In the last years, an active research direction that lies at the intersection of machine learning and cognitive science focuses on finding training algorithms for deep neural networks that have degree of biological plausibility while obtaining good results on machine learning benchmarks. The most

---

popular ones are PC [Rao and Ballard, 1999, Whittington and Bogacz, 2017], equilibrium propagation [Scellier and Bengio, 2017, Scellier et al., 2018, Scellier and Bengio, 2019), and target propagation [Lee et al., 2015, Meulemans et al., 2020, Ernoult et al., 2022]. These methods share multiple similarities, both theoretically and in terms of performance. The finst two methods, PC and equilibrium propagation, are able to approximate the weight update of BP when provided with label that is close in distance to the neural activities of the last layer [Whittington and Bogacz, 2017, Scellier and Bengio. 2019]. Target propagation fails to have this property. but instead has been shown to approximate Gauss-Newton optimization [Meulemans et al., 2020]. However, PC possesses many unique properties that these methods lack. PC networks can in fact be used to produce efficient associative memory models [Salvatori et al., 2021], have an update mechanism that produces better learning properties than BP under specific conditions [Song et al., 2022], and allow training on graphs of any topology [Salvatori et al., 2022a]. Furthermore, they have achieved good results in classification [Han et al., 2018J, generation [Ororbia and Kifer. 2020], and reinforcement learning Ororbia and Mali [2022a.b]. For recent survey on these aspects, see [Millidge et al., 2022]. To conclude, we are not aware of any neuroscience-inspired learning method before this work that is able to generalize to complex tasks such as language modeling.
Progress in this direction is promising, one of the main limitations of modern architectures is that they are extremely computationally expensive to be trained, with large-scale models sometimes requiring hundreds of GPUs for several weeks [Brown et al., 2020]. On the other hand, significant breakthroughs on neuromorphic and analog hardware have recently been achieved [Strukov et al., 2008, Sebastian et al., 2020J, which can exploit the aforementioned properties of neuroscience- inspired learning methods, as shown in [Kendall et al., 2020], where the authors simulated the training of a multilayer network on an analog chip in an end-to-end fashion.
There has been a lot of research done towards bridging the gap in performance between state-of-the-art deep learning methods and ceuroscience-insprre learning. Both fields can benefit from each other by drawing inspiration from each other's techniques. In neuroscience, understanding how the brain learns to associate different areas (e.g., visual and motor cortices) to successfully drive behaviour is of fundamental importance [Petreanu et al., 2012, Manita et al., 2015, Makino and Komiyama, 2015, Poort al., 2015, Pakan et al., 2016, Zmarz and Keller, 2016, Attinger et al., 2017]. However, how to correctly modify synapses to achieve this has puzzled neuroscientists for decades. This is often referred as the synaptic credit assignment problem [Rumelhart al., 1986, Sutton and Barto, 1998, Roelfsema and van Ooyen, 2005, Bengio, 2014, Lee et al., 2015, Roelfsema and Holtmaat, 2018], for which the BP algorithm provides an elegant solution.
7 Conclusion
The main motivation behind this work was to make PC competitive with BP for complex deep neural architectures. The tasks in this work among the most popular and important in the field: image generation and language modelling. In the first case, we trained a variational autoencoder. This model fully Gaussian, but the bottleneck requires explicitly computable variances. While variations of PC with trainable variances are already defined in the literature [Millidge et al., 2021], they do not allow dependencies between the variance and the input. Rather, they act as a regulariser within the network. Consequently, they have not been used as sampling scheme in a specific layer of a PC network. In the second case, we trained a transformer model, intractable before by PC networks, due to the presence of attention (and hence softmax), and showed results comparable to those of BP. Future work includes applying this method to other complex deep learning architectures, with the far-reaching goal of scaling PC to large-scale machine learning tasks and hence further closing the gap with BP-based learning.
Acknowledgments
This work was supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1, by the AXA Research Fund, the EPSRC grant EP/RO1 3667/1, the MRC grant MC_UU 00003/1, the BBSRC grant BB/S006338/1, and by the EU TAILOR grant. We also acknowledge the use of the EUSRC-funded Tier 2 facility JADE (EP/P020275/1) and GPU computing support by Scan Computers International Ltd. Yuhang Song was supported by the China Scholarship Council under the State Scholarship Fund and by a J.P. Morgan AI Research Fellowship.
10

---

References
A. Attinger, B. Wang, and G. B. Keller. Visuomotor coupling shapes the functional development of mouse visual cortex. Cell, 169(7): 1221-133,2 2017.
Y. Bengio. How auto-encoders could provide credit assignment in deep networks via target propaga- tion. arXiv preprint arXiv:1407.7906 2014.
R. Bogacz. A tutorial on the free-energy framework for modelling perception and learning. Journal of Mathematical Psychology, 76:198 198-11,, 2017.
T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
C. L. Buckley, C. S. Kim, S. McGregor, and A. K. Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55-79, 2017.
C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, Koehn, and T. Robinson. One billion word benchmark for measuring progress statistical language modeling. arXiv:1312.3003 2013.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1-38, 1977.
L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-144 2012.
J. Devlin, M.-W. Chang, K. Lec, and K. Toutanova. BERT: Pre-training of deep bidircctional transformers for language understanding. arXiv:1810.04800 2018.
M. Ernoult, F. Normandin, A. Moudgil, S. Spinncy, E. Belilovsky, I. Rish, B. A. Richards, and Y. Ben- gio. Towards scaling difference target propagation by learning backprop targets. arXiv:2201.13411 2022.
A. N. Foroushani, H. Assaf, F. H. Noshahr, Y. Savaria, and M. Sawan. Analog circuits to accelerate the relaxation process in the equilibrium propagation algorithm. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS), 2020.
K. Friston. Learning and inference in the brain. Neural Networks, 16(9):1325-1352. 2003.
K. Friston. A theory cortical responses. Philosophical Transactions of the Royal Society 2005.
Biological Sciences,
K. Friston. Hierarchical models the brain. PLOS Computational Biology, 4(11):1-24, 11 2008
K. Friston and S. Kiebel. Predictive coding under the free-energy principle. Philosophical Transac- tions of the Royal Society Biological Sciences, 364(1521):1211-22 2009.
K. Han, H. Wen, Y. Zhang, D. Fu, E. Culurciello, and Z. Liu. Deep predictive coding network with local recurrent processing for object recognition. Advances in Neural Information Processing Systems, 31, 2018.
K. He, X. Zhang. S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human- perfor- mance imagenet classification. 1502.0 2015.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 770 778, 2016.
D. O. Hebb. The Organization of Behavior: Neuropsyccological Theory, Wiley, New York, 1949.
11

---

J. Hertz, A. Krogh, B. Lautrup, and T. Lehmann. Nonlinear backpropagation: Doing backpropagation without derivatives of the activation function. IEEE Transactions on Neural Networks, 8(6): 1321-1327, 1997.
Kendall, R. Pantone, K. Manickavasagam, Y. Bengio, and B. Scellier. Training end-to-end analog neural networks with equilibrium propagation. arXiv:2006.01990 009000,6 2020.
D. Kingma and M. Welling. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representations, 2014.
A. Krizhevsky, V. Nair, and G. Hinton. CIFAR-10 (Canadian Institute for Advanced Research).
A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In 26th Annual Conference on Neural Information Processing Systems, 2012.
T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, System Demonstrations, pages 66-71, 2018.
S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
D.-H. Lee, S. Zhang, A. Fischer, and Y. Bengio. Difference target propagation. In Joint Euro- pean Conference on Machine Learning and Knowledge Discovery in Databases, pages 498 515. Springer, 2015.
H. Makino and T. Komiyama. Learning enhances the relative impact of top-down processing in the visual cortex. Nature Neuroscience, 18: 18:1116-1 1122, 2015.
S. Manita, T. Suzuki, C. Homma, T. Matsumoto, M. Odagawa, K. Yamada, K. Ota, C. Matsubara, A. Inutsuka, M. Sato, M. Ohkura, A. Yamanaka, Yanagawa, Nakai, Y. Hayashi, M. Larkum, and M. Murayama. A top-down cortical circuit for accurate sensory perception. Newron, 86, 05 2015.
L. McInnes, J. Healy, and J. Melville. UMAP: Uniform manifold approximation and projection for dimension reduction, 2018
A. Meulemans, F. Carzaniga, J. Suykens, J. Sucramento, and B. F. Grewe. A thcoretical framework for target propagation. Advances in Neural Information Processing Systems, 33:20024-20036, 2020.
B. Millidge, A. Tschantz, and C. L. Buckley. Predictive coding approximates backprop along arbitrary computation graphs. arXiv preprint arXiv:2006.0 04 2020.
B. Millidge, A. K. Seth, and C. L. Buckley. Predictive coding: A thcoretical and experimental review. CoRR, abs/2107.11979, 2021.
B. Millidge, T. Salvatori, Y. Song, R. Bogacz, and T. Lukasiewicz. Predictive coding: Towards a future of deep learning beyond backpropagation? arXiv:2202.09467,, 2022.
D. Mumford. On the computational architecture of the neocortex. Biological Cybernetics, 66(3): 241-251, 1992.
A. Ororbia and D. Kifer. arXiv:2012.03405. 2020.
The neural coding framework for learning generative models.
A. Ororbia and A. Mali. Active predicting coding: Brain-inspired reinforcemeet learning for sparse reward robotic control problems. arXiv preprint 2rXiv:2209.09/74 2022a.
A. G. Ororbia and A. Mali. Backprop-free reinforcement learning with active neural generative coding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 29-37, 2022b.
12

---

J. M. Pakan, S. C. Lowe, E. Dylda, S. W. Keemink, S. P. Currie, C. A. Coutts, and N. L. Rochefort. Behavioral-state modulation of inhibition context-dependent and cell type specific in mouse visual cortex. eLife, 5:e14985, 2016.
C. H. Papadimitriou, S. S. Vempala, D. Mitropolsky, M. Collins, and W. Maass. Brain computation by assemblies of neurons. Proceedings of the National Academy of Sciences, 117(25), 2020.
L. Petreanu, D. A. Gutnisky, D. Huber, N. long Xu, D. H. O'Connor, L. Tian, L. Looger, and K. Svoboda. Activity in motor-sensory projections reveals distributed coding in somatosensation. Nature, 489(74155:299-300 September 2012.
J. Poort, A. G. Khan, M. Pachitariu, A. Nemri, I. Orsolic, J. Krupic, M. Bauza, M. Sahani, G. B. Keller, T. D. Mrsic-Flogel, and S. B. Hofer. Learning enhances sensory and multiple non-sensory representations in primary visual cortex. Neuron, 86(6):1478-143) 2015.
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and Sutskever. Zero- shot tcxt-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 8821-8831. PMLR, 2021.
R. Rao and D. Ballard. Predictive coding in the visual cortex: functional interpretation some extra-classical receptive-ficld effects. Nature Neuroscience, 2:79-87, 1999.
R. RoeIfsema and A. Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 19(3):166, 2018.
P. R. Roelfsema and A. van Ooyen. Attention- gated reinforcement learning of internal representations for classification. Neural Computation, 21110):117--214, 2005.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back- back-propagating errors. Nature, (23(6088).533-555 1986.
Salvatori, Y. Song. Y. Hong, Frieder, L. Sha, Z. Xu, R. Bogacz, and T. Lukasiewicz. Associative memories via predictive coding. In Advances in Neural Information Processing Systems, 2021.
T. Salvatori, L. Pinchetti, B. Millidge, Y. Song, R. Bogacz, and T. Lukasiewicz. Learning arbitrary graph topologies via predictive coding. arXiv:2201. 13180, 2022a
T. Salvatori, Y. Song, Z. Xu, T. Lukasiewicz, and R. Bogacz. Reverse differentiation via predictive coding. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, 2022b.
B. Scellier and Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11:24, 2017.
B. Scellier and Y. Bengio. Equivalence of equilibrium propagation and recurrent backpropagation. Neural Computation, 31(2)3312-32, 2019.
B. Scellier, A. Goyal, J. Binas, T. Mesnard, and Y. Bengio. Generalization of equilibrium propagation to vector field dynamics. arXiv preprint IrXiv:1806 04873, 2018.
A. Sebastian, M. Le Gallo, R. -hjaaeh, and Eleftheriou. Memory devices and applica- tions for in-memory computing. Nature Nanotechnology. 15(7):529-544 2020.
D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Sutskever, Lillicrap. M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489 Jan. 2016.
Y. Song, T. Lukasiewicz, Z. Xu, and R. Bogacz. Can the brain do backpropagation? imple- mentation of backpropagation in predictive coding networks. In Advances in Neural Information Processing Systems, volume 33, 2020.
Y. Song, B. G. Millidge, T. Salvatori, T. Lukasiewicz, Z. Xu, and R. Bogacz. Infer- ring neural activity before plasticity: A foundation for learning beyond backpropagation.
2022.
13

---

D. B. Strukov, G. S. Snider, D. R. Stewart, and R. S. Williams. The missing memristor found. Nature, 88307191):80-83, 2008.
R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning, volume 2. MIT Press Cambridge, 1998.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küt- tler, J. Agapiou, and J. Schrittwieser. StarCraft II: A new challenge for reinforcement learning. arXiv: 170.:0708 04782, 2017.
O. Vinyals, I. Babuschkin, W. M. Czarnecii M. Mathicu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 2019.
J. C. Whittington and R. Bogacz. An approximation the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computation, 29(5): 1229-1262, 2017.
P. Zmarz and G. Keller. Mismatch receptive fields in mouse visual cortex. Neuron, 92, 10 2016.
14


Reviews:
[{"review_id": "c9-IwIVu_Dq", "reviewer": "Reviewer_3JGX", "summary": "This work addresses the problem of generalizing Predictive Coding (PC) to arbitrary distributions. The related literature has presented PC as a biologically plausible alternative of backprop that relies on local (layerwise) updates. However, PC has so far been established as a credible alternative to backprop only when considering Gaussian activations. This is extremely unfortunate, as it limits the applicability of PC to virtually a small fraction of modern deep nets.\n\nThis paper generalizes PC to deal with any distribution, and importantly intractable distributions, approximated via a sampling scheme. The resulting framework is coherent with the PC theory, as it enables the definition of layer-wise energy functions that represent prediction errors.", "questions": "I need some deeper discussions on the weaknesses of the method compared to backprop. For instance, what happens in terms of computational complexity? This must be analyzed in terms of training time, inference time, and required resources (FLOPS). Do you foresee architectures where the proposed method may linger compared to BP? What are the directions for future research to address these limitations?", "limitations": "n/a", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: Strong motivation; a very good mathematical foundation; sound mathematical proofs of on the equivalence with BP; potential for strong impact on the community; convincing simulations.\nCons: I need some deeper discussions on the weaknesses of the method compared to backprop. For instance, what happens in terms of computational complexity?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bYx070H6EOI", "reviewer": "Reviewer_9uRC", "summary": "This paper proposes a modification of the predictive coding (PC) paradigm in the context of variational inference with artificial neural networks. In particular, while the canonical PC framework assumes Gaussian form of the underlying generative model, the proposed PC modification extends PC to arbitrary distributions. The authors provide three sets of experiments which compare the proposed PC modification to two other methods of training, namely, backpropagation and regular PC. The empirical results suggest that the proposed PC modification outperforms regular PC on the task of classification for MNIST and CIFAR-10 and the task of language modeling. It is comparable to backpropagation in the classification of MNSIT and CIFAR-10 datasets and in training a VAE on MNIST data (in terms of reconstruction loss on the test set).\n", "questions": "1. Have the authors explored how the proposed method scales to larger architectures and datasets?\n\n2. What is the relative memory and computational costs of PC vs BP in the three experimental setups?\n\n3. In the experiments with VAEs, what are the two architectures used (line 241)? \n", "limitations": "The authors address limitations of their work. They mention that the KL divergence between $\\chi_l$ and $\\hat{\\chi_l}$ should be computationally efficient to derive (line 171).\nThe authors also discuss the limitation of the biological plausibility of the proposed generalized PC in the Appendix - I believe this could be mentioned in the main text.\nThe authors could mention potential negative societal impacts, for example, in the case when the model might propagate harmful bias that is present in the training data (which is inherent to any machine learning system). \n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The proposed method is well-motivated (addresses a limitation of the existing PC framework).\n- The claims in the paper are supported with empirical evidence.\n- The paper is well-organized and clearly written.\n\nRoom for improvement\n\n- The paper could discuss the computational complexity and scalability of the proposed model.\n- The authors could run multiple seeds in their experiments to provide error bars.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_jrUBx5E7GL", "reviewer": "Reviewer_Bynh", "summary": "A more general variant of the predictive coding (PC) algorithm for learning deep neural networks is introduced. Unlike traditional PC, this generalized PC is also effective in training highly complex neural architectures such as transformer networks. Experiments are conducted on various tasks involving image generation and language modeling and similar performances to backpropagation-trained networks are obtained.", "questions": "- Why should one care about the different interpretation problem? In other words, in what sense is $PC_{\\mathcal{F}}$ \"inconsistent\" and why is it important?\n- Is there a generative model description of $PC_{\\mathcal{F}\\text{KL}}$ like that of $PC_{\\mathcal{F}}$? How should the quantity $\\ln p(\\phi_l^{\\mathcal{D}}|\\mu_l^{\\mathcal{D}})$ be interpreted?\n- Is there a reason for the performance gap between BP and $PC_{\\mathcal{F}\\text{KL}}$ in figure 3 for $\\mathcal{M}_3$?", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The contributions are very clearly stated. \n- The method proposed is novel and it greatly generalizes the PC framework.\n- The paper points out a concrete scenario for which traditional PC fails (i.e. the softmax activation) and clearly demonstrates the improvement of the newly proposed method by comparing different models with and without softmax. I find this to be convincing.\n- The result of successfully training a transformer network with the newly proposed generalized PC is very impressive. \n\nWeakness:\n- The notation and exposition of the predictive coding methods can be quite confusing at times. For example on line 89: \"[t]he nodes $x_l$ of each layer store the activation $\\mu_l$,\" the usage of the word \"store\" can be misleading, making the reader question if $x_l$ is a variable/value or an abstract node object that \"holds\" values. There seems to be three different variables $\\phi$, $\\mu$ and $x$ that are closely related with each other, and the paper does not do a very good job of giving these variables concrete names/definitions. The distinction of different interpretations of the same value ($\\cdot^{\\mathcal{S}}$ vs. $\\cdot^{\\mathcal{D}}$) is also confusing in my opinion.\n- On a similar note, figures 1 and 2 are also very difficult to read, since there are many different visual elements with different colors, line styles, sizes and labels, and some of them even overlap with each other. I would suggest making them simpler and/or adding more description for what each component represents.\n- In the experiment section, no confindence intervals are shown for the training curves.\n\nOverall the paper is well written with impressive results, although some notation and exposition can be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ARdJPatbxT_", "reviewer": "Reviewer_ADdV", "summary": "The authors extend the predictive coding formulation of training neural networks to the setting of non-Gaussian densities for the internal and output units (and non-Dirac approximate posteriors), instead using the more general KL divergence formulation. The authors apply this to supervised classification, VAE training, and training transformers. The results are typically better, in terms of performance and efficiency, than the Gaussian-formulation of predictive coding training and roughly comparable with backpropagation.", "questions": "Why ignore the approximate posterior entropy in Eq. 7? Shouldn’t Eq. 8 contain an expectation over all approximate posterior dimensions? As currently written, this is a mean field assumption over layers, which I would imagine wouldn’t scale particularly well to large model architectures.\n\nWhat is the relative training time for each algorithm?\n\nDoes this more generic formulation retain the biological plausibility benefits of the original formulation? It seems that you now need to obtain more general gradients at each layer, as opposed to the relatively simple form (errors) of the original formulation.\n\nAre the internal activations sampled? This would yield a more expressive model as compared with a Dirac approximate posterior.\n\nHow does this method differ from target propagation?", "limitations": "Limitations are perhaps somewhat addressed. One aspect that the authors have not fully discussed is the increased training time required for these algorithms. As the authors state, they use 32 iterations to infer the activations of the network per training iteration. Naively, I would expect the predictive coding-trained networks to be 32 times slower than their backprop counterparts. Likewise, it’s unclear whether it’s biologically plausible to assume that we can optimize the more generic free energy in Eq. 8. While the authors do not explicitly make claims of biological plausibility, I see this as one potential limitation of moving away from the original predictive coding formulation.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n*Originality*:\nThe paper is somewhat original, in that it generalizes previous approaches to training neural networks with predictive coding, moving to more general distributions and more general functions. I see this as somewhat useful, as it opens up the space of models that can be trained with predictive coding. However, I question whether this formulation isn’t already encapsulated within hierarchical latent variable models (i.e., hierarchical VAEs) or related alternatives to backprop, like target propagation.\n\n*Quality*: \nThe paper is of reasonable quality. The authors take a somewhat established area, predictive coding-based alternatives to backprop, and generalize it to non-Dirac approximate posteriors (and non-Gaussian priors). They attempt to illustrate the key differences of their method using diagrams and mathematical expressions. Finally, they perform comparisons with the original predictive coding formulation and backprop in several settings. While I can see weaknesses in each of these areas, the paper, overall, has many of the structural elements of what could become a high-quality paper.\n\n*Clarity*:\nFor the most part, I was able to follow the writing and mathematical notation (although see below). I appreciate that the authors included diagrams to illustrate the key differences between backprop vs. predictive coding and Gaussian vs. non-Gaussian (or rather, Dirac vs. non-Dirac) predictive coding formulations. With relatively minor improvements, I feel that these aspects of the paper could become quite clear.\n\n*Significance*:\nTo the best of my knowledge, alternatives to backpropagation have been primarily restricted to supervised learning with fully-connected or convolutional model architectures. This paper presents a generalization of the original predictive coding training formulation, which is demonstrated in unsupervised settings and with the transformer model architecture. The fact that this method is able to roughly match backprop in these settings is at least moderately significant.\n\n**Weaknesses**\n\n*Originality*:\nI found the originality of the paper to be somewhat lacking. The main novelty of the paper is in moving beyond the original assumptions of Dirac approximate posteriors. However, in variational autoencoders (VAEs), which are also based around Gaussian latent variable models, these assumptions aren’t even entertained; in practice, almost everyone uses Gaussian priors and approximate posteriors, or even more general distributions like normalizing flows. Thus, the formulation isn’t all that novel, considering that it is standard within other areas of machine learning. Further, given the lack of clarity around the presentation (see below), I found it difficult to determine whether this more general formulation retains the ‘biological plausibility’ / locality benefits of the original predictive coding formulation.\n\n*Quality*: \nMuch of the weakness of the quality of the work stems from the weaknesses outlined in other areas.\n\nHere, I will reiterate that the presentation and empirical evaluation could be improved. In terms of presentation, I would have preferred to see clearer notation and diagrams, as well as the details of training (e.g., with an algorithm box). For the empirical evaluation, I would have liked to see experiments with larger models, e.g., even just convolutional, more datasets, and additional baselines. The only real baseline is the original predictive coding formulation. Given that the presented formulation is strictly more expressive, it’s not all that surprising that the performance is matched or improved.\n\n*Clarity*:\nI found the clarity of the paper to be one of its major weaknesses.\n\nI appreciate the fact that the authors included diagrams in Figures 1 and 2 to attempt to illustrate the differences between 1) predictive coding and backprop and 2) predictive coding with and without Gaussian assumptions. However, I found these diagrams difficult to interpret, so they didn’t add much to my understanding of the key differences. It’s not always clear what each of the arrows and shapes are meant to represent computationally. I also didn’t find the inconsistent vs. consistent dichotomy in Figure 2 to be clear — if I understand correctly, this is just pointing out the fact that typical predictive coding formulations assume a deterministic inference estimate, whereas one could assume other forms of distributions. Inconsistent, to me, implies that there is some type of mistake in the original formulation.\n\nI found the notation to be difficult to follow at times. For instance, the authors use $\\phi$ to denote the approximate posterior estimate, initially a point estimate of the latent variables. Later on, they switch to a notation in which $\\phi^\\mathcal{S}$ denotes this single estimate ($\\mathcal{S}$ for sample) and $\\phi^\\mathcal{D}$ denotes a distribution. While I feel that I understand the main points that they authors are trying to convey, I have to imagine that there are easier ways to present the formulation, particularly for readers that are unfamiliar with predictive coding or alternatives to backpropagation. Instead, I would have kept a consistent notation for the approximate posterior estimate, showing that the standard formulation of predictive coding makes a particular assumption (Dirac), but this is either a) not expressive enough or b) not suitable for categorical/discrete variables, necessitating other distributions.\n\nThe details of the formulation are quite unclear to me. For instance, the authors allude to using the EM algorithm for optimizing both the generative model parameters and inference estimate. In the (typical?) case of Dirac inference estimates, Gaussian priors, and fully-connected networks, this seems fairly reasonable, as the gradient consists of the error at each layer, perhaps including the derivative of the non-linearity. However, in the more general case of non-Dirac inference estimates, non-Gaussian priors, and more general network architectures, it’s unclear what is included in these gradients, let alone whether calculating these gradients is biologically plausible. Further, the authors make several assumptions in their formulation (Eqs. 7 & 8) that seem poorly motivated. For instance, why the ‘energy’ at each layer is assumed to be the cross-entropy between the approximate posterior and prior, rather than the (correct) KL divergence. I suppose this makes sense in the case of Gaussians with fixed variance, but it seems unfounded in more general settings. Likewise, the free energy expression in Eq. 8 is a sum across layers of the network, completely neglecting any dependencies between layers — this is a fairly strong assumption. \n\nThe empirical setup is also unclear. I would have preferred to see a more in-depth description of the network architectures, ideally with clear diagrams. Further, some of the training details were quite unclear, e.g., whether or not the internal Gaussian activations were sampled, the time required for the 32 E-step iterations used for inference, etc. These are very practically important, as they determine the computational efficiency and expressiveness of the network and training algorithm.\n\n*Significance*:\nWhile I admit that training alternative network architectures with predictive coding is mildly significant, I did not find the results to be particularly compelling. And while I understand that the point of the paper is not to obtain state-of-the-art performance on benchmarks, the empirical setups investigated by the authors are almost too simplistic as to be interesting, e.g., classification with 3-layer fully-connected networks, vanilla VAEs. Admittedly, I’m not sure whether transformers have been trained with backprop alternatives, so this could be somewhat significant. However, given my confusion around the specifics of the actual implementation (e.g., how are gradients obtained for inference?), it’s unclear, for instance, whether the model actually retains the biological plausibility of the original predictive coding formulation. And, ultimately, the authors are only able to roughly match the performance of backprop in these fairly standard settings. That is, the authors have not demonstrated that this technique opens up any new training settings. Finally, the authors repeatedly motivate their technique with the idea of being able to apply these algorithms on analog devices, however, it’s entirely unclear to me whether this algorithm is any more suited for these computing architectures than backprop.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept