Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a modification of the objective function used for learning predictive coding models by incorporating hessian-parameterized variational posterior. The authors derive the modified objective and perform empirical evaluation of the approach. 
strength_and_weaknesses: The basic idea makes sense: incorporate the full posterior rather than point estimates of MAP, as the authors claim is currently done in PC literature. 

However, the paper was hard for me to follow. 

It lacks basic introduction to the Predictive Coding problem and the state of the art in this field. How is PC different than a regular probabilistic graphical model? More introduction to the problem statement (less wordy, more concrete) would be great. 

The math is also hard to follow. Instead of detailed derivations, the main text should rather just highlight the novel contributions. A pseudocode of the proposed algorithm might help with clarity.

Empirical evaluation is severly limited. The model architecture is unclear. What is the PC model used in Figure 2? A

The main point that incorporating hessian in objective encourages diversity makes sense from Figure 3 and Figure 4, but the its hard to appreciate this without knowing exactly what PC does (the text can expand on this). Might be better to describe standard PC tasks for someone not familiar.

clarity,_quality,_novelty_and_reproducibility: Writing could be improved a lot. See comments above.
summary_of_the_review: It was hard to understand the impact of contributions due to unclear description of problem statement, technical improvements and marginal empirical results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Inspired by ideas in Neuroscience, like predictive coding (PC), the paper proposes a variational inference scheme based on the Laplace approximation with some similarities in behavior to PC. This variational inference method focuses on generative models, where we approximate the latent variable model via a Gaussian auxiliary posterior and the NN parameters are 
strength_and_weaknesses: **Strengths:** The paper has the spirit of bringing past ideas from Neuroscience, like predictive coding to build connections with variational inference in generative models.

**Weaknesses:** There are important flaws in the paper that are difficult to ignore. To name a few, the predictive coding is not properly revisited or technically described, which makes difficult to understand the connection made during the work. On a similar direction, the presentation of the Laplace Approximation misses *almost* every key reference on this aspect in the ML community. I would recommend to take a look to the recent paper [Daxberger et al. 2022] and check what is missing and what other contributions around the Laplace approximation for NN-based models and latent variables have been done. Moreover, there are no baselines to compare, and hence the performance of the method is difficult to understand (lower row of Figure 3 shows that the method has not converged for that experiment for example, or that is not working properly). The derivation of the Laplace approximation in the Appendix has errors and mistakes on the notation that seems they were not checked properly (Expectation wrt. $Q(x)$?)



[Daxberger et al. 2022] -- https://arxiv.org/pdf/2106.14806.pdf

Other recent Laplace approximation papers for generative models, NN methods and autoencoders:

[Immer et al. 2022] -- https://arxiv.org/pdf/2202.10638.pdf

[Park et al. 2019] -- http://proceedings.mlr.press/v97/park19a/park19a.pdf]

[Miani et al. 2022] -- https://arxiv.org/pdf/2206.15078.pdf

clarity,_quality,_novelty_and_reproducibility: The paper is in general under the threshold on quality and novelty. Experiments are limited, without comparison with baselines and some Figures show results that are not working or have not been finished.
summary_of_the_review: Not ready for acceptance in its current state. Missing SOTA, baselines on the experiments, thorough results and connection with current advances in Laplace approximation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: While Predictive Coding (PC) is a prominent theory of perception in the brain, this work addresses why it is not commonly used as a generative modeling technique in the machine learning community. By looking at PC through the lens of variational Bayesian inference, the authors identify that the source of the deficits of PC is the omission of a Hessian term in the objective function. This Hessian captures the uncertainty over the latent states in the model and acts as a regularizer for the sharpness of the underlying joint distribution. 

To overcome these deficits, the authors propose the Laplace Monte Carlo (LMC) approach, in which samples from the Laplace-optimal variational posterior can be used to compute the ELBO. This objective can then be optimized w.r.t the model parameters. However, the covariance of this posterior distribution is the inverse of the negative Hessian of the joint distribution. To ensure that the Hessian is positive semi-definite and to tackle the computational cost of computing it, the authors provide a way of approximating this Hessian. 

Finally, the LMC and Approximate LMC are evaluated on MNIST, CIFAR10 and CelebA datasets. As expected, models trained with LMC and ALMC objectives outperform standard PC.
strength_and_weaknesses: **Strengths**:
- Paper is clearly written and easy to follow
- Experimental results clearly demonstrate that LMC and ALMC overcome the deficits of standard PC

**Weaknesses**:
- In spite of the improvements from LMC and ALMC, the CelebA and CIFAR10 samples are very low fidelity. Is this only a consequence of the size of the models? 
- Would using a larger model, perhaps with convolutional layers, result in better samples? 
- In equation 5, it looks like the joint distribution does not have terms coupling $x$ and $z$. Or is it that the subset of variables $\mathcal{P}a(z^{(j)})$ can include $x$ variables and vice-versa? 
- LMC and ALMC are only compared with standard PC. It might be interesting to also compare it with other generative modeling techniques.

clarity,_quality,_novelty_and_reproducibility: The presentation is very clear in general. One minor comment is that in section 3.2, the idea of model with layers is suddenly introduced. Clearly introducing the hierarchical model at this stage might be beneficial to the reader. Some of the details of the Hessian approximation could instead be moved to the appendix. It might also be useful to briefly introduce the PC objective. 
summary_of_the_review: This paper identifies the deficits of predictive coding, as viewed through the perspective of variational Bayes, and proposes two new approaches that circumvent these deficits. The experimental results demonstrate that the new methods do indeed outperform standard PC. The paper is well written and the main concepts are clearly conveyed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In their manuscript, "Predictive coding with approximate Laplace Monte Carlo", the authors propose a modification to the predictive coding method that is designed to improve its performance for multi-layer deep learning style models.
strength_and_weaknesses: Whereas I usually being with the strengths, I will begin with the weaknesses here. The reason for this is because I think there are some fundamental flaws in the current review and exploration of approaches to this problem, but that in the final implementation the authors start moving into more promising territory. 

The central weakness of this paper is the comparison against strawman alternatives.  First, the Laplace approximation as widely used in the statistical literature is not a Monte Carlo (e.g. importance sampling) algorithm: it is a deterministic approximation.  As described in (e.g.) Rue, Martino & Chopin (2009), the Laplace approximation is not "Laplace importance sampling" and it has a variety of error and convergence characterisations, and a relationship to variational approximatons, that are different to the Monte Carlo error studied by Chatterjee & Diaconis.  Recent innovation in the literature include a variation Bayes correction (Niekerk & Rue 2021), a low discrepancy sequence version (Brown et al., 2021), and many more.
Second, the proposed algorithm is compared against the old PC implementation which, from the introduction, is acknowledged to have been essentially abandoned as no longer state-of-the-art.

Where the paper becomes interesting for me is when the authors build their mixed algorithm in which the authors start building combined models to confront the problem-driven complexities of achieving effective approximate solutions.  The use of a blocked Hessian approximation is not novel but the combination with the dirac delta variation posterior starts to get interesting.  Here I would imagine that this investigation of the practical value of this model and perhaps finding additional clever innovations could be pursued in three areas, potentially leading to three separate papers: 
1) comparison (in terms of [spatial] cross-validation-based RMSE and log-likelihood accuracy) against models (the most obvious use case is geostatistics) that use existing Laplace approximation strategies on their home turf, so to speak, i.e., for 'bread and butter' published example datasets; 
2) comparison against the same when they are being pushed to their limits (e.g. in terms of dimensionality: esp space-time models, where further approximations then become necessary: e.g. Cici Bauer et al 2016); and/or 
3) comparison against state-of-the-art deep learning methods based on generative models.  

clarity,_quality,_novelty_and_reproducibility: The clarity is broadly good throughout; quality and novelty are modest; reproducibility is likely high.

summary_of_the_review: There is a kernel of a good idea here but not a sufficient demonstration or comparison against alternatives.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers recommend rejecting the paper (3-3-3), including Reviewer 58fL who initially had a more positive view of the paper (6) but changed his/her recommendation during the discussion. The reviewers raised major concerns ("important flaws", "fundamental flaws") about the manuscript, including a lack clarity in the presentation of the method, missing references to recent machine learning literature, and a lack of experimental validation against state-of-the-art deep learning approaches. 

Although the authors made some improvements to the manuscript during the discussion with the reviewers, these changes were not enough to convince the reviewers to change their recommendation. The authors should consider submitting a revised version of the paper to a different conference.
justification_for_why_not_higher_score: Fundamental flaws on the clarity, the related work and the experimental validation.
Decision: Reject