Predictive Coding with Approximate Laplace Monte Carlo
Authors: Umais Zahid
Qinghai Guo
Karl Friston
Zafeirios Fountas
Venue: Submitted to ICLR 2023 (2023)
Keywords: predictive coding
variational Bayes
Laplace approximation
generative modelling
free energy

Abstract: Predictive coding (PC) accounts of perception now form one of the dominant computational theories of the brain. Despite this, they have enjoyed little export to the broader field of machine learning, where comparative generative models have flourished. In part, this has been due to the poor performance of models trained with standard implementations of PC when evaluated by both sample quality and marginal likelihood. By adopting the perspective of PC as a variational Bayes algorithm under the Laplace approximation, we identify the source of these deficits to lie in the exclusion of an associated Hessian term in the standard PC objective function. To remedy this, we make three primary contributions: we begin by suggesting a simple Monte Carlo estimated evidence lower bound which relies on sampling from the Hessian-parameterised variational posterior. We then derive a novel block diagonal approximation to the full Hessian matrix that has lower memory requirements and favourable mathematical properties. Lastly, we present an algorithm that combines our method with standard PC to reduce memory complexity further. We evaluate models trained with our approach against the standard PC framework on image benchmark datasets. Our approach produces higher log-likelihoods and qualitatively better samples that more closely capture the diversity of the data-generating distribution. 

Under review as a conference paper at ICLR 2023
PREDICTIVE CODING WITH APPROXIMATE LAPLACE MONTE CARLO
Anonymous authors Paper under double-blind review
ABSTRACT
Predictive coding (PC) accounts of perception now form one of the dominant computational theories of the brain, where they prescribe a general algorithm for inference and learning over hicrarchical Gaussian latent general models. Despite this, they have enjoyed little export to the broader field machine learning, where comparative generative modelling techniques have flourished. In part, this has been due to the poor performance models trained with PC when evaluated by both sample quality and marginal likelihood. By adopting the perspective of PC as variational Bayes algorithm under the Laplace approximation, we identify the source of these deficits lie in the exclusion of associated Hessian term in the standard PC objective function. To remedy this, we make three primary contributions: we begin by suggesting simple Monte Carlo estimated evidence lower bound which relies on sampling from the Hessian-paree parameterimed variational posterior. We then derive novel block diagonal approximation to the full Hessian matrix that has lower memory requirements and favourable mathematical properties. Lastly, we present an algorithm that combines our method with standard PC to reduce memory complexity further. We evaluate models trained with our approach against the standard PC framework on image benchmark datasets. Our approach produces higher log-like likelihoods and qualitatively better samples that more closely capture the diversity of the data-generating distribution.
INTRODUCTION
In the last two decades, conceptions of the brain as an organ actively engaged in Bayesian inference have become exceedingly prominent in cognitive neuroscience (Pouget al. 2013 Clark 2013] Kanai et al.) 2015). Under this paradigm, the brain adopts probabilistic generative model of the world, with perception corresponding to inference over latent states, and learning to the inference over its parameters. Predictive coding (PC) (Rao and Ballard 1999 Friston 2018), arguably the most notable instantiation of this perspective, describes a method for parameter learning in hierarchical latent Gaussian generative models with arbitrarily complex and highly non-linear parameterisations governing their conditional distributions. This computational scheme remains one of the foremost and popular computational models for explaining cortical function, (Mumford, 1992 Hosoya et al. 2005; Hohwy et al./ 22088; Bastos et al. 2012 Shippl 2016: Feldman and Friston 2010 Fountas al. 2022). emphasizing the importance of evaluating
successful technique for training deep generative models of the kind presupposed in the brain.
From machine learning perspective, PC bares a close mathematical relationship to Bayesian techniques such as the variational auto-encoder (VAE) (Kingma and Welling 2014), which also relies on optimising an evidence lower bound (ELBO); with a key advantage over VAES ostensibly being in PC's use of non-amortised inference (Cremer et al. 2018). Furthermore, PC also benefits from design principles inherited from its origins as a theory of cognitive function namely asynchronous and local error computation (Whittington and Bogacz, 2019), suggesting far greater amenability to implementation on energy-efficien neuromorphic hardware.
In this work, we show that generative models trained with PC (of the kind described in (Bogacz 2017 Tschantz et al. 2022 Millidge et al.) 2022)), have poor log marginal likelihoods when evaluated on common image datasets, and poor sample quality, despite producing good reconstructions. To diagnose these issues we begin by adopting the perspective of PC as a variational Bayes algorithm
under the Laplace approximation (Friston 2003 2005 2008). Under this approximation, quadratic

---

Under review as a conference paper at ICLR 2023
assumptions over the log joint density of generative model result in a Gaussian variational posterior with precision (inverse variance) equal to the Hessian matrix or curvature - of the negative log joint with respect to its latent states.
We then present simple ELBO-based objective function that accounts for this curvature and thus the uncertainty over latent states using samples from the Laplace-optimal variational posterior. We show that our objective has the additional effect of regularising for the sharpness of the probability landscape. Furthermore, to improve upon the memory complexity of computing the full Hessian matrix required for the Laplace ELBO objective, we present a novel block diagonal approximation to the Hessian that has lower memory complexity and is guaranteed positive semi-definite (PSD) ensuring its associated variational posterior can always be sampled from. Finally, to further remove the dependency of memory complexity on the output image dimensionality, we present a combined model, in which the final layer of our generative model is trained with PC, and all higher layers are trained with approximate Laplace Monte Carlo. The resulting method has memory complexity reduced to O ), from (N2) where n, and N are the dimensionalities of the largest latent layer, and all latent layers combined respectively while retaining improved log likelihoods and sample quality.
PREDICTIVE CODING
Predictive coding is algorithm with origins in computational neuroscience (Rao and Ballard 1999)
Friston 2003 2005; Friston and Kiebel, 2009) that prescribes a method for parameter learning in
hierarchical latent variable probabilistic graphical models. In it's most common form, (Bogacz 2017
Millidge et al. 2022 I'schantz et al. 2022). it can be described succinctly by the following simple recipe:
1. Define a (possibly hierarchical) graphical model over latent (z) and observed (r) states with parameters A (i.e. log ((z,=(0))
2. For x ~ D, where is the data-ge aatta-enerating distribution
Inference: Obtain MAP estimates (SMAP) for the latent states by enacting gradient
descent on log Pazr,
Learning: Update the parameters 0 using stochastic gradient descent with respect to the log joint evaluated at the MAP estimates found at the end of inference: log P(z, 7MAP ())
One common motivation for this algorithm rests upon its interpretation as a variational Bayesian method under Dirac delta (deterministic) approximate posterior distribution (Friston 2003 Bogacz 2017). Under this interpretation, the inference step outlined in the PC algorithm, corresponds to maximisation of ELBO (for particular data point) with respect to the mean of the variational Dirac delta distribution, and learning corresponds to maximising the ELBO (over the entire dataset) with respect to the model parameters 0. Another common interpretation for this algorithm assumes the Laplace approximation (Friston et al. 2007), under which inference corresponds to optimising the mean of a Gaussian variational posterior with covariance equal to the inverse Hessian of the log joint probability. While, this interpretation retains the inference procedure of PC, it has non-trivial implications for the learning procedure, which we detail in the next section.
RELATED WORK AND THE LAPLACE APPROXIMATION
The Laplace approximation has historically been derived in two contexts. The first context adopted Laplace's 's method for the computation of the ordinarily intractable marginalised model evidence after the maximum a posteriori (MAP) value of the latent states had already been identified (Kass) and Raftery| 1995 Tierney and Kadane 1986). The second adopted the Laplace approximation for variational inference, wherein, under quadratic assumptions for the log joint, it can be shown that the Gaussian variational posterior which minimises the ELBO has inverse covariance equal to the Hessian of the negative log joint probability evaluated at the variational mode (Friston et al. 2007). We adopt this second perspective here and thus begin with the definition of the standard ELBO for latent probabilistic model p(z, 2|0). where r and a are sets of observed and latent random variables respectively, and are set of model parameters:
2

---

Under review as a conference paper at ICLR 2023
log P(z|0) F [Q(s)) [log P(z,, 2|0]] - [q(= [log Q())]
(I)
Adopting a quadratic approximation over the log joint, and plugging the optimal posterior under this approximation into the ELBO results in the following analytical expression (See: Appendix A.3 for a recounting of the full derivation):
F = log P(x, 02 + log (2T) N det He(
Here μ4 is the mean of the variational Gaussian posterior over latent states, and He is the Hessian associated with the negative log, joint (-log 2)) with respect to z, evaluated at the variational mode.
In much of the PC literature, it has been common practice to ignore this second log determinant Hessian term as a further simplification. This is either done explicitly (Buckley et al. 2017] Millidge al. 2020; Whittington and Bogacz 2017). or implicitly by adopting a point mass for the posterior density (Friston 2005). Optimising this approximate ELBO with respect to μ4₂ and 0 then results in the PC algorithm, as described in Section 2
It is worthwhile to note that the determinant Hessian of a function evaluated at critical point is equal the Gaussian curvature of that function. When this determinant is well-defined (i.e. the Hessian is positive semi-definite), its eigenvalues are of the same sign, and thus the value of this log determinant monotonically related to the sharpness (the maximum eigenvalue of the Hessian), a metric directly related to the risk of divergence in gradient descent (See Cohenn et al. (2022) for it's in the context of neural network training). is unsurprising then that one of the practical difficulties that arise with PC is managing the risk of divergence during the inference procedure training progresses, which in practice is ameliorated by either finely tuning the inference learning rate, or using an adaptive step size. This increased risk of divergence, can therefore be elegantly explained as a failure to regularise for the curvature of the log joint due to the exclusion of the log determinant Hessian term in the variational Laplace Bayes objective.
In the statistical literature, this Hessian term has been comparatively less neglected. Attempts in this arena have optimised model parameters (0) with respect to equation 2 approximations thereof (Bell) 2001). Alternative approaches have used purpose-built auto-differentiation packages to first compute the Hessian, and subsequently computed gradients of model parameters with respect to Laplace importance sampling estimate of the marginal likelihood, (Skaug) 2002 [Skaug and Fournier [006] Kristensen et al. (0166) Compared to our approach, these have various serious vantdvan such as bias (Breslow and Lin 1995) and computational complexity, in the case of direct optimization of Laplace marginal evidence, or potentially high variance in the case of importance sampling estimates based on the Laplace posterior (Chatterjee and Diaconis 2018).
For completeness we also note that the context in which the Laplace approximation is generally adopted in the existing deep learning literature is distinct from the context it will be used in this paper. In the existing DL literature, (LeCun et al. 1989 MacKay 1992. Daxberger et al. 2022 Immer al., 2022 Ritter al.) 2022), the Laplace approximation is ordinarily used to obtain an approximate posterior over the model parameters (0) for non-latent non-hierarchical model consisting of a log-likelihood parameterised by the output of a (possibly deep) neural network, i.e.
Ex, Togy p (z, yy(0 p (0)). where x and y are observed variables and 0 are neural network parameters. The Laplace approximated posterior over 0 is then used either post-hoc (after training) estimate model uncertainty, or used online for hyperparameter tuning, model selection (Immer et al.) (021). and preventing catastrophic forgetting (Ritter et al., 2018). MAP learning of model parameters in this context is tractable and relatively strright-forward via direct (stochastic) gradient descent on the joint log likelihood due to the absence of latent random variables, which would ordinarily have to be marginalised over. Therefore, while learning is more tractable, weakness of these approaches is in their absence of datapoint specific latent random variables (2) that encode the latent (hidden) probabilistic causes of each datapoint. The presence of data specific probabilistic latent states in most SOTA generative modelling techniques, /Vahdat et al. [222]] Nichol and Dhariwal] 202T Child 2021) suggests they are highly beneficial, if not essential, for modelling complex datasets.
3

---

Under review as a conference paper at ICLR 2023
Conceptually, the methodology presented in this paper is closest that of (Park al. 2019), albeit from different narrative perspectives (non-amortize amortized VAEs rather than PC) and with three key algorithmic differences: adopt block diagonal approximation to the Hessian to reduce the memory complexity of computing a full Hessian, we also present a "combined" model (with both Dirac delta and Laplace approximate posteriors) reduce memory complexity further, and lastly do not propagate gradients through every time step of the inference procedure to the amortization model, which would require memory proportional to the length inference allowing us use significantly greater number of inference steps and thus keep the amortisation gap small.
4
METHOD
(b)
(a)
Actual
LMC
Overlapped
Full
Latent #2
(c)
Approx.
-2
0
2
-2
0
2
-2
Latent #1
Figure 1: (a) Visualisation of the joint probability, P(x, 21; 22) (left), under quadratic (middle) and factorised quadratic approximation for the log joint of model with two latent states (21, E2) connected by leaky ReLU and linear coefficient. We can see the approximate quadratic formulation trades off the accuracy of our variational posterior with tractability. (b) Log absolute values of our approximate block-diagen block-diagonal Hessian for layer, 64 dimension per layer generative model. (c) Log absolute values for the full Hessian for the same model. Layers are demarcated with dashed white lines.
4.1 THE LAPLACE MONTE CARLO ELBO
One can begin by considering the following decomposition of the ELBO into expected energy term and an entropy term:
log P(x) 2 EQ(=s)en)))) [log P(z, x10)] H[Q]
(3)
Since the optimal approximate posterior under the Laplace approximation is known and analytically tractable, and since our model parameters are independent of the entropy of this approximate posterior, we may optimise our model parameters with respect to the free-energy by simply considering the first term the expectation of the log joint probability with respect to our variational posterior. We variational
may then approximate this term by taking Monte Carlo samples from our posterior and optimise model parameters via standard automatic differentiation. Note that this does not require the reparameterisation trick (Kingma and Welling (014)) as we do not require optimising any parameters associated with our variational posterior.
4

---

Under review as a conference paper at ICLR 2023
LMC
[og P(x, []] Σ
(4)
K
Where we are sampling from our Laplace optimal approximate posterior
24 QZZ; (MMP: Eg He !)
(5)
The advantage of this approach is that while we are sampling from the optimal posterior under the Laplace approximation, unlike the analytical expression (equation described in section we do not require this approximation to be true for the resultant objective to still be an ELBO; with the validity of this approximation instead only impacting the tightness of this bound. Furthermore, in section we demonstrate empirically that despite not optimising with respect to the log determinant Hessian directly, our objective function nonetheless successfully regularises for the sharpness of the probability landscape. We denote models trained with the objective in equation 4 as Laplace Monte Carlo (LMC) models.
4.2 APPROXIMATING THE HESSIAN
There remain number of difficulties present however when working with the Hessian under this approach. First, require the Hessian of the log-joint to be positive semi-definite its inverse forms the covariance matrix for the variational Posterior under the Laplace approximation. Second, computing the Hessian has strict bound computational and memory complexity of O where is the total dimensionality of our latent states across the entire network which for deep networks can approach the order of tens of thousands to millions.
To circumvent these issues we present an approximation to the full Hessian that retains only curvature information within layer, resulting in variational posterior that is factorised across layers. The resultant approximate Hessian also has the desirable property of being guaranteed PSD and thus Monte Carlo estimates of our ELBO are always computable.
We consider general probabilistic model consisting of a set of latent (unobserved) random variables Z, and observed random variables X. We define generative model under these random variables factorised such that disjoint subsets of our random variables, (20) Z) and (2(0)12(0 X), have associated with them a multivariate Gaussian conditional distribution with fixed or learnt diagonal covariance matrices ( ()))) زهو diag(a,); and means f( μاlر parameterised by function of subset of the remaining random variables, which we denote with Pa(2(5)) and
())x The negative log joint of this general model can then be defined as follows:
log 2(x,2) 1
Σ (2
(2(0) -(P((((P((())).)))
T Σ(0 (2(0) 1010-f(00 ((P((((()))))))
Σ
- I00 (20) (s(Pa(=()), + +C (6)
(=0122)
We focus on an approximation to the Hessian of this log-joint where we only consider the second order relations of random variables within the same layer, and not between layers. As such the approximate Hessian derived here will be a block diagonal matrix, which we may then guarantee to be positive semi-definite if the constituent blocks its diagonal are also guaranteed to be positive semi-definite.
We adopt the following approximation for single latent block of the Hessian, which is guaranteed to be PSD, and which we note also exact for piece-wise functions f. such as an affine transformation followed by leaky ReLUU. (We place the full derivation in the Appendix for the sake of clarity).
5

---

Under review as a conference paper at ICLR 2023
0fk
T
0fk
-J JV log P(X,2) = ~ف
Σ
(k)
2z(s)
0201
x(MeCh(22)))
0f:
T 20
0f
Σ
(7)
Oz(5)
(Ch((()))
Since have assumed diagonal covariance matrices (E; diag(o)) throughout generative model, the blocks of our block diagonal Hessian thus simplify to a sum of terms that are guaranteed be PSD (see Appendix A.2 for short proof), resulting in Hessian approximation that is also guaranteed to be PSD. A visualisation of the resultant approximation for a layer model, captured during training, can be seen in Figure P alongside the full Hessian (Fig- [Tp. We also visualise the joint probability under quadratic, and approximate quadratic assumptions for model with latent states connected hierarchically alongside the ground truth joint probability in Figure Ta.
J(Vz,) 2 0 -J(VZ)
0
0
22
0
Apppox
(8)
1
((v2))
Because are only computing curvature information with respect to each layer individually, our memory complexity is also reduced to ((max(n?,. ny no)), where nt, 7,,, no are the dimensionalities of the largest latent layer, final latent layer and observation layer respectively, which will generally be significantly lower than O(N²) for full Hessian. Note that our memory complexity might expect because of the Jacobians associated with observed random
not On, variable log likelihood terms, a fact that we will address in the next section.
We also note that the derivation of one block of our approximate Hessian similar in principle the derivation of the generalised Gauss-Newton matrix commonly used to approximate the Hessian of neural network parameters for the purpose of second-order optimisation (Schraudolph) 2002) Martens 2020), as both derivations rely on decomposing the full Hessian into first-order and second-order components.
4.3 COMBINATION MODELS
Naively adopting the aforementioned Hessian approximation may still have impractically high memory requirements as the Jacobians associated with our observed random variables can be exceptionally large if the dimensionality of our observations are high even if the final gram Jacobian matrix is quite small transiently resulting in high memory requirements.
To address this issue and reduce memory complexity further, take approach inspired by the recent class of latent diffusion models (Rombach et al.) 2022 Vahdat et al.) (02T). The key insight here is that one may reduce the untenably high memory requirements of more complex generative models by instead training them on the smaller latent space of an autoencoder trained with the objective of producing good reconstructions. In our experiments, while PC networks failed produce good generative models, they frequently produced excellent reconstructions with very few training samples. Thus we hypothesised that training a combined model, in which lower level/s are trained with the PC objective and higher layers are trained with our Approximate-L.n objective, would allow us to combine the strengths of curvature aware training with the reduced memory complexity of the ordinary PC objective.
Unlike the latent diffusion modelling approach, we train our combination models simultaneously and end-to-end. Mathematically. this approach may still be described by expression 4 with lower layers being "sampled" from Dirac delta variational posterior and higher layers being sampled from our (approximate) Laplace-optimal Gaussian posterior, both centred the MAP estimates found the end of inference. Resulting in the following expression, where we have now segmented our latent states into those trained in accordance with LMC (z²) and those with PC (zP))

---

Under review as a conference paper at ICLR 2023
1 Σ Dog P(r, PMAPP))] K
LMC(O)
(9)
Where we are sampling - from our Laplace optimal approximate posterior:
. Q(2": iMMP Egg He-l)
(10)
5 EXPERIMENTS
To test the proposed objectives, we trained hierarchical models composed of layers of latent states connected via a non-linearity (leaky ReLU or tanh) followed by an affine transformation as skip connections for adjacent layers with equal dimensionality. The decision to have the non- linearity precede the affine transformation was to ensure that the predicted means for each layer were unbounded- - in parity with the unbounded support of our Gaussian variational posterior. We tested three model configurations across four datasets, combined models with learnt variances, combined models with fixed variances, and non-combined models with fixed variances. To initialise our variational modes we adopted the amortisation scheme described by TTchantz et al. (2022), wherein feedforward amortisation network is trained alongside our generative model to initialise close to the non non-amortised MAP estimates found at the end inference. We used K = 20 for our LMC and ALMC objectives. For more extensive details the model architectures refer the reader to appendix A.4
well
As we note in section 4.2 significant difficulty with naively using the Hessian is the inability to guarantee positive semi-deiiniteness- property which, in our experiments, could result in up to 90% of the samples in batch having non-PSD associated Hessians in the early stages of training. To accommodate for this issue we experimented with skipping the relevant samples but found that if done so training quickly deteriorates and the issue of non-PSD Hessians exacerbate training progressed. Thus, for the experiments discussed in this paper non-PSD Hessians were instead replaced with identity matrices, which we found stemmed the continued presence of the problem.
Log Likelihood. We evaluate our models by estimating log marginal likelihoods using Laplace importance sampling of the kind described in (Kuk) 1999). To avoid reporting erroneously inflated log-likelihood values stemming from modelling discrete pixel intensities with continuous models, we follow best practices and dequantize the data by adding uniform noise for both training and evaluation, (Uria et al. 2014); the resultant log-likelihoods are thus guaranteed to be lower bounds on the true log- liglhhodd on the discrete data distribution (Theis et al. 2016). The aforementioned (negative) log marginal likelihoods (reported bits per dimension) can be found in Table
Sharpness. We track the log determinant Hessian for 3200 unseen samples throughout training, Figure 2] We find that Laplace Monte Carlo objectives (approximate or otherwise) regularise for the curvature (log determinant Hessian) despite not explicitly optimising with respect to the term. The log determinant Hessian is also equal to the negative of approximate posterior entropy up an additive constant N (og(2T)) )) and thus the effect of LMC/ALMC can be seen as preventing over confidence in the posterior predictions type of Occam's 's razor.
Interpolations and Samples. By fixing any particular hierarchical layer of our generative model and enacting a feed-forward top-down pass through it we may visualise what the latent states at each hierarchical layer appear to be representing. We do this while interpolating across the latent embeddings for two images and find that LMC models were more likely exhibit semantically meaningful hierarchical structure, with higher layers representing global features such as background, hair colour and gender. An example can be seen in Figure 5 Additionally, examples of samples obtained via ancestral sampling can be seen in Figure 5
6 CONCLUSION
Across (almost) all the configurations and image datasets tested, our LMC and ALMC methods consistently outperformed PC in terms of log marginal likelihood and sample quality. These results serve as a clear demonstration that the PC objective function as commonly presented is insufficient for producing generative models that capture the diversity of the data generating distribution. We
7

---

Under review as a conference paper at ICLR 2023
Method
Dataset
BPD
Method PC
Dataset
BPD
PC
MNIST
6.785=0.00
MNIST
+.000
LMC
MNIST
+.03
LMC
MNIST
9.581
ALMC
MNIST
6.727 0.03
ALMC
MNIST
+.07
PC
CIFARIO
7.007 0.03
PC
CIFARIO 10
9.430
LMC
CIFARI0
6.935 0.006
LMC
CIFARTO
9.405 0.001
ALMC
CIFAR1O
6.900 + 0.009
ALMC
II
9.390 + 0.002
PC
CelebA
6.895 + 0.001
PC
CelebA
9.444 = 0.02
LMC
CelebA
6.896 = 0.002
LMC
CelebA
9.423 = 0.04
ALMC
CelebA
6.895 + 0.001
ALMC
CelebA
9.450 + 0.02
PC
SVHN
5.533 = 0.01
PC
SVHN
9.417 + 0.0007
LMC
SVHN
5.505 + 0.008
LMC
SVHN
9.396 + 0.002
ALMC
SVHN
5.493 0.005
ALMC
SVHN
9.380 0.001
Method
Dataset
BPD
PC
MNIST
9.690 + 0.0002
LMC
MNIST
9.515 + 0.03
ALMC
MNIST
9.374 0.0003
PC
CIFARTO
9.430 + 0.0007
LMC
CI
9.374 + 0.01
ALMC
CIFARIO
9.363 + 0.01
PC
SVHN
9.417 = 0.0007
LMC
SVHN
9.349 + 0.003
ALMC
SVHN
9.365 + 0.03
Table I: Bits per dimension (BPD) for various model configurations:
combination models with variance optimisation (left), combination models with fixed variance (right), and non-combination models with fixed variance (bottom). See configurations 1, and 3 respectively in Appendix [A.4 for more details. Error margins correspond to standard deviation over 5 multiple seed runs. (*) Note configurations and are equal for PC as the combined configuration results in no change.
MNIST
CIFARIO
SVHN
1.0
Log Det Hessian
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
LMC
0.2
0.2
ALMC
0.00
0.25
0.50
0.75
1.00
Learning Iteration
Learning Iteration
Learning Iteration
Figure Log determinant Hessian (curvature) for 3200 unseen samples across the training of our models, for all 3 objective functions. Curves correspond to model configuration (combined with variance optimisation) as described in Appendix A.4 Shaded regions correspond to standard deviation over runs.
have argued in this paper that the source of these deficits lie in failure to accommodate for the uncertainty over our latent states as determined by the curvature our model's log joint probability.
To show that this is the case, we presented simple LMC objective that accommodates for this by sampling latent states from a Gaussian centred upon the MAP estimates found at the end of inference, with covariances equal to the inverse Hessian of the negative log joint an optimal approach under the Laplace approximation. The resultant objective produces consistent improvements in terms of log marginal likelihood and sample diversity, but is marred with difficulties that make it impractical for training larger models such as the inability to guarantee positive comi-definitensss for the Hessian, and poorly scaling memory requirements.
8

---

Under review as a conference paper at ICLR 2023
ALMC
LMC
PC
3
3
3
CelebA
3
6
CIFAR1O
Figure 3: Samples (temp=1) for models trained CclebA (top) and CIFAR1O (bottom) with the ALMC (left), LMC (center), and PC objective (left). While the fidelity of these samples are clearly lacking due the small size of our tested models, LMC and ALMC capture far more of the training set diversity, while PC produces highly uniform samples.
ALMC
PC
Top
Layer
Bottom
Layer
Linear interpolations in the latent states of different hierarchical layers (rows) for ALMC (left) and PC (right). Images refer to a feed-forward top-down pass through our generative model after fixing the latent states at a particular layer.
Figure 4:
We resolved this by then deriving a novel block diagonal approximation to the Hessian that has better memory complexity, and is guaranteed positive semi-definite. Perhaps unexpectedly, the resultant approximate LMC objective consistently performs better than the LMC objective. We attribute this effect to the aforementioned inability to guarantee positive semi-definiteness when utilising the full Hessian, resulting in having to either skip training samples, or as employed here, using fixed alternative Hessian in its stead.
We also demonstrate that accommodating for this curvature can be interpreted as regularising for the sharpness of the loss landscape, an effect that we empirically verify our LMC and approximate LMC objectives exhibit despite not explicitly optimising with respect to the log determinant Hessian. This has important implications for practical implementations of PC which are frequently ailed by gradient divergence,
effect which is the direct consequent of a loss landscape that is too sharp relative to the step size of ones descent. Finally, we noted that the log determinant Hessian is equal to the entropy of the Laplace optimal posterior up an additive constant, and thus results suggest that curvature aware methods induce a type of Occam's 's razor regularisation, where the posterior over latent states is prevented from becoming over-confident.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Detailed information on all the necessary experimental details required to reproduce our results can be found in section S and appendix A.4 Additionally, source code associated with the experiments will be made available after the review process.
REFERENCES
Alexandre Pouget, Jeffrey M. Beck, Wei Ji Ma, and Peter E. Latham. Probabilistic brains: knowns and unknowns. Nature Neuroscience, 16(9): (699:1170-1170 170-1178, 178, September 2013. ISSN 1546-174 1726.
do. 10..0.03///nn4999 URttttps::/www. nature. www.artticiii.. nn. 3495 Number:
Publisher: Nature Publishing Group.
Andy Clark. Whatever next? Predictive brains, situated agents, and the future of cognitive sci- ence. Behavioral and Brain Sciences, 36(3):181-2044 June 2013. ISSN 0140-525X, 1469-1825. 000100..0177/0044555XX0000200000
Publisher: Cambridge University Press.
Ryota Kanai, Yutaka Komura, Stewart Shipp, and Karl Friston. Cerebral hierarchies: predictive
processing, precision and the pulvinar. Philosophical Transactions of the Royal Society B: Biolog-
ical Sciences, 370(1668):201444)
May 2015. doi 1098//shhhhs/ wwtp:///gg..oooooo/oooo 109801.....11111 http:/rttb.....11...... 0169
URL https
//rovalsocietyyubliiss.
Publisher:
Royal Society.
Rajesh P. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpre-
tation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, January
1999. ISSN 1546- 1726. doi TO. 1038/4580] (tRt:ttts::///il. www.:.ntttr..... com/ hamm/arte caamaarticlooo es/ Number: Publisher: Nature Publishing Group.
nnc199 79
Karl Friston. Does predictive coding have future? Nature Neuroscience,
August
2018. ISSN 1546- 1726. doi 10900088/11333-018-020000
URLhttps:/ www.:wwww..........o wawura..ree.... com/ articlest articles/ai 1144555 1593-1930110 01 8 0200-7 Bandiera _atrrsttttttttttaaa Cg_type: Nature Research Journals
Number: Primary_atype: News & Views Publisher: Nature Publishing Group Subject term:
Computational neurosciencc:Neuraal circuits.Neuronal physiology Subject term id: computational-
D. Mumford. computational architecture of the neocortex. II. The role of cortico-cortical loops. Biological Cybernetics, 66(3):241-25) 1992. ISSN 1340-1200. 10t.1000000//F001984///
Toshihiko Hosoya, Stephen A. Baccus, and Markus Meister. Dynamic predictive coding by the retina.
Nature, 436(7047::71-77, July 2005. ISSN 1476-4687. 10030/// 1038/nature03669)
Jakob Hohwy, Andreas Roepstorff, and Karl Friston. Predictive coding explains binocular rivalry: an cpistemological review. Cognition, 108(3):687-701 September 2008. ISSN 0010-0277.
1o010.00166..connitttt 2008.05.010
Andre M. Bastos, W. Martin Usrey, Rick A. Adams, George R. Mangun, Pascal Fries, and Karl J. Friston. Canonical microcircuits for predictive coding. Neuron, 76(4):695-711 November 2012.
ISSN 1097-4199.
Stewart Shipp. Neural Elements for Predictive Coding. Frontiers in Psychology, 7:1792, 2016.
ISSN 1664-1078 doi ww..333//psy..221660777
htw.hhtpss///mmm Ut..ttpss//pw...rtttirrsss.. org/
article/10. 3389/ /tt/8/ffppyae fwsyg 201.. 017.99 01792
Harriet Feldman and Karl Friston.
Attention, Uncertainty, and Free-Energy. Frontiers in Human Neuroscience, 4:215, 2010. ISSN 1662-5161, doi 10.338//nhum 2010,00215 URL https
//ww. www.frontiersinn org httg//rticle//000 3389/ Enhum. 2010. 00215
Zafeirios Fountas, Anastasia Sylaidi, Kyriacos Nikiforou, Anil K. Seth, Murray Shanahan, and Warrick Roseboom. A Predictive Processing Model of Episodic Memory and Time Perception. Neural Computation, 3407)1150-11544, June 2022. ISSN 0899-7667. 1oi:10.1162/nees 01514
URL https:://oi. http::///ollooog1111116/////e/ 0111
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. 1122::1312.6114 [cs, stat], arXiv: 1312.6114,
May 2014. URL http:://aaasiv.orggaaaaaaa/
6114
10

---

Under review as a conference paper at ICLR 2023
Chris Cremer, Xuechen Li, and David Duvenaud. Inference Suboptimality in Variational Autoen- coders. arXiv:1801.0333 .35588 [cs, stat], May 2018. URL ktttp://axxiv.rrr abs/ abs/1801 1801 00355..
03558
arXiv:
James C.R. Whittington and Rafal Bogacz.
Theories of Error Pack-Propagation in the Brain. Trends in Cognitive Sciences, 23(3):235-250, March 2019.
ISSN
13646613. doi 10.1016/j. 201.. 12.005 URLiatta::/Anknngguu http://awwnnnnuuuueeeeeeeeee.e.e elsevier. com/
Rafal Bogacz. A tutorial on the free-energy framework for modelling perception learn-
ing. Journal of Mathematical Psychology, 76:198-211 February 2017. ISSN 0022-2496. doi 0001000000000.mm220011111.110000 ariclle/.../50022299112
URL https: wwww hettss//ww..cceenceed wcw.ncediiiiicccccee rect. cttte.ccoomce coom/science,
Alexander Tschantz,, Beren Millidge, Anil K. Seth, and Christopher Buckley. Hybrid Predictive Coding: Inferring, Fast and Slow. arXiv:2204.022 022222244402999 [cs, q-bio), April 2022. URL http:
//arxiv.org//bs/22288 02169
arXiv: 2204.02169.
Beren Millidge, Anil Seth, and Christopher L.. Buckley. Predictive Coding: a Theoretical and Experimental Review, July 2022. URL cttp://araiv..or//ab//211.1.
arXiv:2107.19799 [cs, q-bio].
Karl Friston. Learning and inference in the brain. Neural Networks, 16(9)1335-11352, November 2003. ISSN 08936080. meoem m00. 2003.06.005
Novemmem URL htttps://linkinghht
Karl Friston. A theory of cortical cortical responses. Philosophical Transactions the Royal Society B: Bio- the
logical Sciences, 360(1) April 2005. ISSN 0962-8436. doi wtw.://9//stb220005..01
www.immmm..........mmmmm.......m httlhttss://wee http://wwwww.poooooooo/ooo///tt/
Karl Friston. Hierarchical Models in the Brain. PLOS Computational Biology, 4(11):
c1000211, November 2008.
ISSN 1553-7358. 101
URL https: plosesnaasppp........mmmmmooosse plos. ppo.ccombbillaarrceeeeell.......
journal. pwbpp.ci....... 1000211
Publisher: Public Library of Science.
Karl Friston and Stefan Kiebel. Predictive coding under the free-energy principle. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, (64(11221):2111-222
May 2009, ISSN 1471-2970. 10. 2008 0300
Karl Friston, Jérémie Mattout, Nelson Trujillo-Barreto, John Ashburner, and Will Penny. Variational free energy and the Laplace approximation. Neurolmage, (4(1):220-234 Jamuary 2007. ISSN 1053-8119. doi http:///cnneeattteoee///////11.. com/
035 URL https: htop:://wwww scienceiirettttt
Robert E. Kass and Adrian E. Raftery, Bayes Factors. Journal of the American Statistical Associaaion... 90(430):773-795, 1995. ISSN 0162-1459, (o..2300//291109) URL (tRthttps:/ pestt:::/wiit /wwww/www....... /t/.://.sssssss////./ jstor
org/stable/229109) Publisher: [American Statistical Association, Taylor & Francis, Ltd.].
Luke Tiemey and Joseph B. Kadane. Accurate Approximations for Posterior Moments and Marginal Densities. Journal of the American Statistical Association, 81339));82-86, March 1986. ISSN 0162- 1459. doi [0. 1000100000122455050 4559 1986. 10478240) URL https::tt //tp:///wwww... wwww http://wwwtaaoo. wow.donnnooonnooo
com/
doi doi/abs/10. http://1...111001111111111000011
1986 10478240 Publisher: Taylor & Francis _eprint: 1477440044..99
Christopher L. Buckley, Chang Sub Kim, Simon McGregor, and Anil K. Seth. The free energy principle for action and perception: A mathematical review. Journul of Mathemutical Psychology. 81:55-79, December 2017. ISSN 0022-2496. 1o. 201.000777700 0.09 URL https //ww.
com www.sp/ccccccccooo/cooc.aoo././a
Beren Millidge, Alexander Tschantz, and Christopher L. Buckley. Predictive Coding Approximates Backprop along Arbitrary Computation Graphs. arXiv:2006.044 04182 [cs], October 2020. URL
http://arxil www.:/oggg/bb//oee 041
arXiv: 2006.04182.

---

Under review as a conference paper at ICLR 2023
James C. R. Whittington and Rafal Bogacz. An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity. Neural Computation, 29(5): May 2017. ISSN 1530-888X. 10.10 00949
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. February 2022. URL https
//oeenrviiew net http:/nee/ttttteeeeeee htr://orummmmmmmm iid jh-hhhhh--- rTt VTtVkGeM
Bradley M. Bell. Approximating the marginal likelihood estimate for models with random pa- rameters. Applied Mathematics and Computation, 119(1)557775, March 2001. ISSN 0096- 3003. doi 00000..0006000000300330000003030
URL http://wwwwccieeeeieeeeee com/
Hans J Skaug. Automatic Differentiation to Facilitate Maximum Likelihood Estimation in Non- linear Random Effects Models. Journal of Computational and Graphical Statistics, 11(2):
458-470 June 2002. ISSN 1061-8600. doi 19811199900666600 160027601806177 URL (hLptt
//doi //doi.org/10 981991110000 0 00010611660000 8602 70000000000006000011000000000011 66188611
Publisher: Taylor & Francis _eprint:
http://doi.org/1011199//00000 0011990/0000000000000008168671
Hans Skaug and David A. Fournier. Automatic approximation of the marginal likelihood in non-Gaussian hierarchical models. Compulational Sutistics & Data Analysis, 51(2):999-709
November 2006. ISSN 0167-9473. 10,100100666)ccca22000...00000
URL, http::://///. //wtt::::ww/ www
sciencedirect. com/ /ttp:///ociiiiieaaaaaa//////////
Kasper Kristensen, Anders Niclsen, Casper W. Berg, Hans Skaug, and Brad Bell. TMB: Auto- matic Differentiation and Laplace Approximation. Journal of Statistical Software, 70(5), 2016. ISSN 1548-7660. doi (0.18637/jss.0070.00) URLhttp:://aan http:////aa.vv..orrab/b///////// 000../06600.00000000 00660
arXiv:1509.00661 [stat].
Norman E. Breslow and Xihong Lin. Bias Correction in Generalised Linear Mixed Models with Single Component of Dispersion. Biometrika, 82(1):81 1995. ISSN 0006-3444.
20022703///3 URL jottl::t/:/// [Oxford University Press, Biometrika Trust].
/
Publisher:
Sourav Chatterjee and Persi Diaconis. THE SAMPLE SIZE REQUIRED IN IMPORTANCE SAMPLING. The Annals of Applied Probability, 1899-1109-1133 2018. ISSN 1050-5164.
URL httpp::/wwww jstor ottg/stable/225423333 Publisher: Institute of Mathemati-
cal Statistics.
Yann LeCun, John Denker, and Sara Solla. Optimal Brain Damage. In Advances in Neural Information 1989. URL http:/pppprrs.pp.....ppppp /c/ Absttaatt--bbttttt......
Processing Systems, volume 2. paper/1989hhash/
David J. C. MacKay. A Practical Bayesian Framework for Backpropagation Networks. Neural Com- putation, 4(3):448 May 1992. ISSN 0899-7667. doiiT0.1162//ecc.19992..4 Conference
Name: Neural Computation.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace Redux Effortless Bayesian Deep Learning, March 2022. URL http:
//arxiv.orrg/aeeeee 22166 4806 arXiv:2106. 14806 [cs, stat].
Alexander Immer, Tycho A. van der Ouderaa, Gunnar Rätsch, Vincent Fortuin, and Mark van der Wilk. Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations, October 2022. URL httthttp:/// /ttpt:::/aaxi....t www.aaawwx.v..oaaa/aaaaaaa
10638 LrXiv:2202.10633 [cs, stat].
Hippolyt Ritter, Aleksandar Botev, and David Barber. A Scalable Laplace Approximation for Neural Networks. February 2022. URL
Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Mohammad Emtiyaz Khan. Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning, June 2021. URL arXiv:2104.04975 [cs, stat].
http://arx: wr..or///aoo wttp:/wwwwr////////////////
04975
12

---

Under review as a conference paper at ICLR 2023
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting, May 2018. URL BtLlttpp//axxxi/ http://ttwwwpiap//p////oooo abs///ba//// orq/abb/1805
07810 arXiv:1805.0788 [cs, stat].
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based Generative Modeling in Latent Space. In Advances in Neural Information Processing Systems, volume 34, pages 11287-11302. Curran As- sociates, Inc., 2021, URL htt.:peee/ppoceeiigg....esssss.s
4a0.4444555442220000055500000003 html
Alex Nichol and Prafulla Dhariwal. Improved Denoiiin Diffussin Probabilistic Models, February
2021.
arXiv:2102.09672 [cs, stat].
Rewon Child. Very Deep VAEs Generalize Auteriaiie  Auttoteeeeesvveeeeeo Models and Can Outperform Them on Images, March 2021. URL httpp///aax...or//aa///11..1...0
arXiv:2011.10650
[cs].
Yookoon Park, Chris Kim, and Gunhee Kim. Variational Laplace Autoencoders. In Proceedings of the 36th International Conference on Machine Learning, pages 5032-5044 PMLR, May 2019. URL httpp::/ppoccceeepptspptt
mlr wwwppppesss.....p /ttpr//eeeaaaaaarpopppprr//////a
9a. html
ISSN: 2640-3498.
Nicol N. Schraudolph. Fast Curvature Matrix-Vector Products for Second-Order Gradi-
ent Descent. Neural Computation, 14(7):1723-1738, July 2002. ISSN 0899-7667.
10011000162/00997660220000
Conference Name: Neural Computation.
James Martens. New Insights and Perspectives the Natural Gradient Method. Journal of Ma-
chine Learning Research, 21(146):1-76, 2020. ISSN 1533-7928. http:///: org/
papers/v21/17-77 .html
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer High- Resolution Image Synthesis with Latent Diffusion Models, April 2022. http:///t///
org/ 2112.11111 10752 arXiv:2112.10752 [cs].
Anthony Y. C. Kuk. Laplace Importance Sampling for Generalized Linear Mixed Computation and Simulation, 63(2):143-158, April doi 10.1110000000000000 0001000000999559900444222222
Models. Journal of Statistical
1999. ISSN 0094-9655.
URL httpp://
doi. .rg. 10. 11080/009496599088808 wtt.:////....000000999999944425.
Publisher: Taylor & Francis _eprint:
Benigno Uria, lain Murray, and Hugo Larochelle.
RNADE: The real-valued neural autore- gressive density-estimator, January 2014. URI. http:://arxivv.rr/abbs... arXiv:1306.011 [cs, stat].
0186
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note on the evaluation of generative models. Technical Report arXiv: 1. .01844, arXiv, April 2016. URL abs/1511 01 844 1rXiv:1511.01844 [es, stat] type: article.
Arak Mathai and Serge Provost. Quadratic Forms Random Variables: Theory and Applications. In Journal of the American Statistical Association, volume 87. Marcel Dekker, December 1992. do::0...307/2206777 Journal Abbreviation: Journal of the American Statistical Association.
A. V. Lazo and P. Rathie, On the entropy of continuous probability distributions (Corresp.). IEEE Transactions on Information Theory, 14(1):120-122 January 1978. ISSN 1557-9654.
10011001199/111199998 1055832 Conference Name: IEBE Transactions on Information Theory.
Richard Zou Horace He. functorch: JAX-like composable function transforms for PyTorch, 2021.
URL https: http://hhhhh...oooo///ttcccc//te
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DERIVATION OF PSD BLOCK DIAGONAL HESSIAN APPROXIMATION
We consider general probabilistic model consisting of a set of latent (unobserved) random variables Z, and observed random variables X. We define generative model under these random variables factorised such that disjoint subsets of our random variables, {200]2 2} and (z(0|z(0 }] have associated with them a multivariate Gaussian conditional distribution with fixed or learnt diagonal covariance matrices ( ( Σ diag(oy); and means ylf), μد parameterised by a function of subset of the remaining random variables, which we denote with Pa(=()) and
P)(x:)). Note that the parent sets Pa(x(()) and Pa(z()) can include both latent and observed variables.
For single set of observed and latent variables, the negative log joint of this general model can then be defined as follows.
llo P(x.2)
Σ
1 (((((((()))))))) 0.))² E(0) (x (:(Pa(x())) 0.))
2
(x(
(2(0) - 15(Pa(=()), (20))2}
Σ
T IGI (20) - s)(Pa(=())) + +C (1))
+ 2
We focus on an approximation to the Hessian of this log-joint where we only consider the second order relations of random variables within the same layer, not between layers. such the approximate Hessian derived here will be block diagonal matrix, which we may then guarantee to be positive semi-definite if the constituent blocks on its diagonal are also guaranteed to be positive semi-definite.
can ask ourselves what one of these blocks looks like, by first applying the gradient operator with respect to the latent variables of one layer unnormalised log joint probability. Note that, solely for the sake of clarity during this derivation, we will exclude any contributions from observed child random variables, i.e. (*) E Ch(2l)), for which the steps below follow identically.
V 200 log P(X, Z) ΣPO (20) (20)-5((Pa(3a ()(P((2())))
dim=N2a1
O Ozúi
Σ
(())
(2(0)_f(PPa (P)(
(12)
ECh((t)))
dim=N, + Ng
No
We can now apply the Jacobian operator the resultant gradient vector field. But first decompose the the matrix vector product inside the summation as a sum of vectors multiplied by scalars.
200 log P(X, y
0fk T) O20)
Σ
Σ
(300 ((0)s((Pa((")) (13)
2 2(h)CC((()) NE
Now, applying the Jacobian operator.
14

---

Under review as a conference paper at ICLR 2023
-J( 20) log P(X,2) =301_
Σ
Σ
(k)
(30)
-s((P(()(()))
NEN
2,0
05k
05k 0z(0)
(k)
(14)
Oz(1)
1.
We may rewrite the second part of the summation as product of the Jacobian tranpose, diagonal covariance, and Jacobian.
0fk T
f
JV 20) log P(X, 2)) = y +
Σ
0z61
0zl0)
204) ECh((C1))
8
Σ
(1)
(2 (((((((()))))
oi
026)2
2 ECh( iEN
(15)
We can then choose to either approximate this by ignoring all terms involving second order derivatives, or alternatively, if the functions fú are piece-wise linear as is the case of for activations functions such as the (leaky) ReLU these terms are guaranteed to be 0, resulting in the following:
JJ(V 20 log P(X,Z)) Σ(0 +
Σ
Ofk Og6i)
05k
(16)
@26i
Including the contributions from any observed random variable log likelihood terms, for which the derivation follows identically, results in the final simplified expression:
05h 0z6)
05k
JJ(V 200 log P(X, = g0) +
Σ
(4)
@261
0fi
T x
0f:
Σ
(17)
Oz(3)
Ozli)
z00 ECh((5))
Since we have assumed diagonal covariance matrices (E; diag(a)) throughout our generative model, the blocks of our block diagonal Hessian thus simplify to sum of terms that guaranteed to be PSD (see Appendix A.2 for a short proof), resulting a Hessian approximation that is also guaranteed be PSD.
A.2 PROOF PSD
Consider the matrix M A diagg(a)) with sigma being a vector of positive real values. We may rewrite this matrix follows:
M AT diag(o A
(18)
A diag(o" )diia(( 1)A
(19)
A diag(o $ + diag(o $)A
(20)
(diag(o 1) AA) (diag(o )A)
(21)
15

---

Under review as a conference paper at ICLR 2023
Call B diag(o -))4, so can write:
M B²B
(22)
Thus M is real gram matrix and guaranteed to be positive semi-definite.
A.3 DERIVING THE VARIATIONAL LAPLACE OBJECTIVE
Consider a generative model, log P(r, =|0), defined over a set of observed (r) and latent states (z), with model parameters given by 0. We are then concerned with optimising model parameters (0) with respect to the log marginal likelihood of observations T.
log P(=00) log f P(z, P(z, 2/0)2 log [ P(e, 210)dz log EQ(s)
[(0ee Q(2)
P P(x,210) Q(2)
(23)
Adopting Jensen's is inequality allows us to define the following lower bound, which we denote with F, often referred to as the negative free energy or the ELBO
E=(=)) log P(z, =10] EQ(=) [log P(z, 2|0) EQ(=) [log Q()) F(4)
(24)
((2)
Note that F is thus far functional of unspecified probability density function q(z).
While the entropy (second) term in equation 24 has an analytically tractable form in the case of our Gaussian assumptions for Q(z), the first expectation term may not as it is dependent on the exact form of the density function specifying our generative model.
The Laplace approximation approximates the log joint density with quadratic approximation using a second-order taylor series centred around the mode of our Gaussian variational distribution ju::
Eq(+) [log 2(x,2(0)) / Q(2)) log 2(x,210)d2
(25)
~ ( (())) log P(z, + (2 Mz) log P(r, 1z
2 (2 )1=) "HH(= p=))
(26)
2 log (1x) EQ(a) ((2 - Mz) V log 11)))11100
Eq(+) [(x us) He(z p=)]
(27)
Using well-known identity on the expectation of quadratic form (Mathai and Provost 1992)
may simplify the third term, such that we obtain significantly simplified expression for our expected energy.
log P(x, p2 0) + Tr [HeEg]
(28)
We can then reintroduce this simplified expected log joint term into our original expression for the ELBO, while plugging in the well-known analytic expression for the differential entropy of the Gaussian distribution for the second term (Lazo and Rathie 1978):
1
F(Jz: Egg 0) = log P(æ, mz + 2
Tr [HeEg] + log ((2Te) det Eg]
(29)
By differentiating this expression with respect to the variational covariance matrix Σg we find that the optimal covariance matrix is equal to the inverse of the negative Hessian our log joint probability. Note that this is an analytical function of the variational modes μı and model parameters 6, which we make explicit here
Σy He(Hu, 01
(30)
16

---

Under review as a conference paper at ICLR 2023
Plugging this into our expression for the free-energy obtain the following significantly simplified expression for the free-energy. which is now a function over μs and parameters (We ignore the dependency for the sake of notational clarity)
F(U2, 0) = log P(x, M2 (2)0) + 2 log (2T) det He 11
(31)
A.4 MODEL. ARCHITECTURE AND EXPERIMENTAL DETAILS
Model Architecture. For the image dataset CelebA, we evaluate models with 5 layers of latent states of dimensionality: [40, 64, 64, 64, 64]. For MNIST, CIFARIO and SVHN we evaluate models with 5 layers of dimensionality: [10, 64, 64, 64, 64].
Each layer of the aforementioned models parameterises the mean of the subsequent layer via non-linearity (either tanh or leaky ReLU), following by an affine transformation. Where adjacent layers have equal dimensionality we also use skip connections.
A summary of the configurations tested for all three datasets and all three methods can be found in Table 2
Configuration Number
Activation Fn
Variance Optimised
Combined
Leaky Relu
Truc
True
2
Tanh
False
True
3
Tanh
False
False
Table 2:
Note, configuration 2 and 3 equal for PC models (for which the combined configuration results in no changes).
Amortisation. Our amortisation models adopt the same basic architecture but in reverse, i.e. for MNIST model, we use an ANN with layer sizes [64,64,6r $4,64,66 64,64,10]. Each layer of the amortisation model is then trained via SGD (with momentum) on an MSE loss between its feed- feedffowwrrd predictions and the MAP latent states identified the end of inference.
Optimisation of O. We use batch sizes of 32 for CelebA and 64 for MNIST, CIFAR10 and SVHN. We use SGD learning rates of 0.01 and 0.0001 for tanh and leaky relu models respectively. We train all generative models using SGD, with momentum set to 0.9. Additionally, for models with learnt variances, we constrain variances to lie between le - and 2.
Optimisation of Maz. We use inference step sizes of 0.05 and 0.001 for tanh and leaky ReLU models respectively and 150 inference steps per batch. Step sizes were reduced by 10% on a per sample basis if an increase in the log joint probability was observed. Furthermore, for experiments with learnt variances we rescale the step size (SS) as max(le-5, SS* Variance)).
Hessian and Jacobian Computation. All Hessians and Jacobians in this text were computed using the general purpose implementations of higher-order automatic differentiation in the Py PoTchh based functorch library (Horace He 2021). Due the 7th digit round-off error associated with single-precision floating point numbers, it was possible for the computed Hessians (or approximate Hessian matrices) to, occasion, be asymmetric; all Hessians (and approximate Hessians) were therefore symmetrised after their computation.
Combination Models. Experiments using combination models used the standard predictive coding approximate posterior for the last layer of the model.
Sampling. Samples were taken with temp=1 until the penultimate layer, which was taken with temp=0, for all models except for those with variance optimisation.
17


Reviews:
[{"review_id": "u9co6--TVCY", "reviewer": "Reviewer_Y7m7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In their manuscript, \"Predictive coding with approximate Laplace Monte Carlo\", the authors propose a modification to the predictive coding method that is designed to improve its performance for multi-layer deep learning style models.", "summary_of_the_review": "There is a kernel of a good idea here but not a sufficient demonstration or comparison against alternatives.\n", "strength_and_weaknesses": "Whereas I usually being with the strengths, I will begin with the weaknesses here. The reason for this is because I think there are some fundamental flaws in the current review and exploration of approaches to this problem, but that in the final implementation the authors start moving into more promising territory. \n\nThe central weakness of this paper is the comparison against strawman alternatives.  First, the Laplace approximation as widely used in the statistical literature is not a Monte Carlo (e.g. importance sampling) algorithm: it is a deterministic approximation.  As described in (e.g.) Rue, Martino & Chopin (2009), the Laplace approximation is not \"Laplace importance sampling\" and it has a variety of error and convergence characterisations, and a relationship to variational approximatons, that are different to the Monte Carlo error studied by Chatterjee & Diaconis.  Recent innovation in the literature include a variation Bayes correction (Niekerk & Rue 2021), a low discrepancy sequence version (Brown et al., 2021), and many more.\nSecond, the proposed algorithm is compared against the old PC implementation which, from the introduction, is acknowledged to have been essentially abandoned as no longer state-of-the-art.\n\nWhere the paper becomes interesting for me is when the authors build their mixed algorithm in which the authors start building combined models to confront the problem-driven complexities of achieving effective approximate solutions.  The use of a blocked Hessian approximation is not novel but the combination with the dirac delta variation posterior starts to get interesting.  Here I would imagine that this investigation of the practical value of this model and perhaps finding additional clever innovations could be pursued in three areas, potentially leading to three separate papers: \n1) comparison (in terms of [spatial] cross-validation-based RMSE and log-likelihood accuracy) against models (the most obvious use case is geostatistics) that use existing Laplace approximation strategies on their home turf, so to speak, i.e., for 'bread and butter' published example datasets; \n2) comparison against the same when they are being pushed to their limits (e.g. in terms of dimensionality: esp space-time models, where further approximations then become necessary: e.g. Cici Bauer et al 2016); and/or \n3) comparison against state-of-the-art deep learning methods based on generative models.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RNW9kABHUV", "reviewer": "Reviewer_ARFQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by ideas in Neuroscience, like predictive coding (PC), the paper proposes a variational inference scheme based on the Laplace approximation with some similarities in behavior to PC. This variational inference method focuses on generative models, where we approximate the latent variable model via a Gaussian auxiliary posterior and the NN parameters are ", "summary_of_the_review": "Not ready for acceptance in its current state. Missing SOTA, baselines on the experiments, thorough results and connection with current advances in Laplace approximation.", "strength_and_weaknesses": "**Strengths:** The paper has the spirit of bringing past ideas from Neuroscience, like predictive coding to build connections with variational inference in generative models.\n\n**Weaknesses:** There are important flaws in the paper that are difficult to ignore. To name a few, the predictive coding is not properly revisited or technically described, which makes difficult to understand the connection made during the work. On a similar direction, the presentation of the Laplace Approximation misses *almost* every key reference on this aspect in the ML community. I would recommend to take a look to the recent paper [Daxberger et al. 2022] and check what is missing and what other contributions around the Laplace approximation for NN-based models and latent variables have been done. Moreover, there are no baselines to compare, and hence the performance of the method is difficult to understand (lower row of Figure 3 shows that the method has not converged for that experiment for example, or that is not working properly). The derivation of the Laplace approximation in the Appendix has errors and mistakes on the notation that seems they were not checked properly (Expectation wrt. $Q(x)$?)\n\n\n\n[Daxberger et al. 2022] -- https://arxiv.org/pdf/2106.14806.pdf\n\nOther recent Laplace approximation papers for generative models, NN methods and autoencoders:\n\n[Immer et al. 2022] -- https://arxiv.org/pdf/2202.10638.pdf\n\n[Park et al. 2019] -- http://proceedings.mlr.press/v97/park19a/park19a.pdf]\n\n[Miani et al. 2022] -- https://arxiv.org/pdf/2206.15078.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PGeSiv_WH8", "reviewer": "Reviewer_58fL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While Predictive Coding (PC) is a prominent theory of perception in the brain, this work addresses why it is not commonly used as a generative modeling technique in the machine learning community. By looking at PC through the lens of variational Bayesian inference, the authors identify that the source of the deficits of PC is the omission of a Hessian term in the objective function. This Hessian captures the uncertainty over the latent states in the model and acts as a regularizer for the sharpness of the underlying joint distribution. \n\nTo overcome these deficits, the authors propose the Laplace Monte Carlo (LMC) approach, in which samples from the Laplace-optimal variational posterior can be used to compute the ELBO. This objective can then be optimized w.r.t the model parameters. However, the covariance of this posterior distribution is the inverse of the negative Hessian of the joint distribution. To ensure that the Hessian is positive semi-definite and to tackle the computational cost of computing it, the authors provide a way of approximating this Hessian. \n\nFinally, the LMC and Approximate LMC are evaluated on MNIST, CIFAR10 and CelebA datasets. As expected, models trained with LMC and ALMC objectives outperform standard PC.", "summary_of_the_review": "This paper identifies the deficits of predictive coding, as viewed through the perspective of variational Bayes, and proposes two new approaches that circumvent these deficits. The experimental results demonstrate that the new methods do indeed outperform standard PC. The paper is well written and the main concepts are clearly conveyed.  ", "strength_and_weaknesses": "**Strengths**:\n- Paper is clearly written and easy to follow\n- Experimental results clearly demonstrate that LMC and ALMC overcome the deficits of standard PC\n\n**Weaknesses**:\n- In spite of the improvements from LMC and ALMC, the CelebA and CIFAR10 samples are very low fidelity. Is this only a consequence of the size of the models? \n- Would using a larger model, perhaps with convolutional layers, result in better samples? \n- In equation 5, it looks like the joint distribution does not have terms coupling $x$ and $z$. Or is it that the subset of variables $\\mathcal{P}a(z^{(j)})$ can include $x$ variables and vice-versa? \n- LMC and ALMC are only compared with standard PC. It might be interesting to also compare it with other generative modeling techniques.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-xpGU30Khrx", "reviewer": "Reviewer_AkoQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a modification of the objective function used for learning predictive coding models by incorporating hessian-parameterized variational posterior. The authors derive the modified objective and perform empirical evaluation of the approach. ", "summary_of_the_review": "It was hard to understand the impact of contributions due to unclear description of problem statement, technical improvements and marginal empirical results. ", "strength_and_weaknesses": "The basic idea makes sense: incorporate the full posterior rather than point estimates of MAP, as the authors claim is currently done in PC literature. \n\nHowever, the paper was hard for me to follow. \n\nIt lacks basic introduction to the Predictive Coding problem and the state of the art in this field. How is PC different than a regular probabilistic graphical model? More introduction to the problem statement (less wordy, more concrete) would be great. \n\nThe math is also hard to follow. Instead of detailed derivations, the main text should rather just highlight the novel contributions. A pseudocode of the proposed algorithm might help with clarity.\n\nEmpirical evaluation is severly limited. The model architecture is unclear. What is the PC model used in Figure 2? A\n\nThe main point that incorporating hessian in objective encourages diversity makes sense from Figure 3 and Figure 4, but the its hard to appreciate this without knowing exactly what PC does (the text can expand on this). Might be better to describe standard PC tasks for someone not familiar.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject