Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A more general variant of the predictive coding (PC) algorithm for learning deep neural networks is introduced. Unlike traditional PC, this generalized PC is also effective in training highly complex neural architectures such as transformer networks. Experiments are conducted on various tasks involving image generation and language modeling and similar performances to backpropagation-trained networks are obtained.
strengths_and_weaknesses: Strengths:
- The contributions are very clearly stated. 
- The method proposed is novel and it greatly generalizes the PC framework.
- The paper points out a concrete scenario for which traditional PC fails (i.e. the softmax activation) and clearly demonstrates the improvement of the newly proposed method by comparing different models with and without softmax. I find this to be convincing.
- The result of successfully training a transformer network with the newly proposed generalized PC is very impressive. 

Weakness:
- The notation and exposition of the predictive coding methods can be quite confusing at times. For example on line 89: "[t]he nodes $x_l$ of each layer store the activation $\mu_l$," the usage of the word "store" can be misleading, making the reader question if $x_l$ is a variable/value or an abstract node object that "holds" values. There seems to be three different variables $\phi$, $\mu$ and $x$ that are closely related with each other, and the paper does not do a very good job of giving these variables concrete names/definitions. The distinction of different interpretations of the same value ($\cdot^{\mathcal{S}}$ vs. $\cdot^{\mathcal{D}}$) is also confusing in my opinion.
- On a similar note, figures 1 and 2 are also very difficult to read, since there are many different visual elements with different colors, line styles, sizes and labels, and some of them even overlap with each other. I would suggest making them simpler and/or adding more description for what each component represents.
- In the experiment section, no confindence intervals are shown for the training curves.

Overall the paper is well written with impressive results, although some notation and exposition can be improved.
questions: - Why should one care about the different interpretation problem? In other words, in what sense is $PC_{\mathcal{F}}$ "inconsistent" and why is it important?
- Is there a generative model description of $PC_{\mathcal{F}\text{KL}}$ like that of $PC_{\mathcal{F}}$? How should the quantity $\ln p(\phi_l^{\mathcal{D}}|\mu_l^{\mathcal{D}})$ be interpreted?
- Is there a reason for the performance gap between BP and $PC_{\mathcal{F}\text{KL}}$ in figure 3 for $\mathcal{M}_3$?
limitations: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a modification of the predictive coding (PC) paradigm in the context of variational inference with artificial neural networks. In particular, while the canonical PC framework assumes Gaussian form of the underlying generative model, the proposed PC modification extends PC to arbitrary distributions. The authors provide three sets of experiments which compare the proposed PC modification to two other methods of training, namely, backpropagation and regular PC. The empirical results suggest that the proposed PC modification outperforms regular PC on the task of classification for MNIST and CIFAR-10 and the task of language modeling. It is comparable to backpropagation in the classification of MNSIT and CIFAR-10 datasets and in training a VAE on MNIST data (in terms of reconstruction loss on the test set).

strengths_and_weaknesses: Strengths

- The proposed method is well-motivated (addresses a limitation of the existing PC framework).
- The claims in the paper are supported with empirical evidence.
- The paper is well-organized and clearly written.

Room for improvement

- The paper could discuss the computational complexity and scalability of the proposed model.
- The authors could run multiple seeds in their experiments to provide error bars.

questions: 1. Have the authors explored how the proposed method scales to larger architectures and datasets?

2. What is the relative memory and computational costs of PC vs BP in the three experimental setups?

3. In the experiments with VAEs, what are the two architectures used (line 241)? 

limitations: The authors address limitations of their work. They mention that the KL divergence between $\chi_l$ and $\hat{\chi_l}$ should be computationally efficient to derive (line 171).
The authors also discuss the limitation of the biological plausibility of the proposed generalized PC in the Appendix - I believe this could be mentioned in the main text.
The authors could mention potential negative societal impacts, for example, in the case when the model might propagate harmful bias that is present in the training data (which is inherent to any machine learning system).
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors extend the predictive coding formulation of training neural networks to the setting of non-Gaussian densities for the internal and output units (and non-Dirac approximate posteriors), instead using the more general KL divergence formulation. The authors apply this to supervised classification, VAE training, and training transformers. The results are typically better, in terms of performance and efficiency, than the Gaussian-formulation of predictive coding training and roughly comparable with backpropagation.
strengths_and_weaknesses: **Strengths**

*Originality*:
The paper is somewhat original, in that it generalizes previous approaches to training neural networks with predictive coding, moving to more general distributions and more general functions. I see this as somewhat useful, as it opens up the space of models that can be trained with predictive coding. However, I question whether this formulation isn’t already encapsulated within hierarchical latent variable models (i.e., hierarchical VAEs) or related alternatives to backprop, like target propagation.

*Quality*: 
The paper is of reasonable quality. The authors take a somewhat established area, predictive coding-based alternatives to backprop, and generalize it to non-Dirac approximate posteriors (and non-Gaussian priors). They attempt to illustrate the key differences of their method using diagrams and mathematical expressions. Finally, they perform comparisons with the original predictive coding formulation and backprop in several settings. While I can see weaknesses in each of these areas, the paper, overall, has many of the structural elements of what could become a high-quality paper.

*Clarity*:
For the most part, I was able to follow the writing and mathematical notation (although see below). I appreciate that the authors included diagrams to illustrate the key differences between backprop vs. predictive coding and Gaussian vs. non-Gaussian (or rather, Dirac vs. non-Dirac) predictive coding formulations. With relatively minor improvements, I feel that these aspects of the paper could become quite clear.

*Significance*:
To the best of my knowledge, alternatives to backpropagation have been primarily restricted to supervised learning with fully-connected or convolutional model architectures. This paper presents a generalization of the original predictive coding training formulation, which is demonstrated in unsupervised settings and with the transformer model architecture. The fact that this method is able to roughly match backprop in these settings is at least moderately significant.

**Weaknesses**

*Originality*:
I found the originality of the paper to be somewhat lacking. The main novelty of the paper is in moving beyond the original assumptions of Dirac approximate posteriors. However, in variational autoencoders (VAEs), which are also based around Gaussian latent variable models, these assumptions aren’t even entertained; in practice, almost everyone uses Gaussian priors and approximate posteriors, or even more general distributions like normalizing flows. Thus, the formulation isn’t all that novel, considering that it is standard within other areas of machine learning. Further, given the lack of clarity around the presentation (see below), I found it difficult to determine whether this more general formulation retains the ‘biological plausibility’ / locality benefits of the original predictive coding formulation.

*Quality*: 
Much of the weakness of the quality of the work stems from the weaknesses outlined in other areas.

Here, I will reiterate that the presentation and empirical evaluation could be improved. In terms of presentation, I would have preferred to see clearer notation and diagrams, as well as the details of training (e.g., with an algorithm box). For the empirical evaluation, I would have liked to see experiments with larger models, e.g., even just convolutional, more datasets, and additional baselines. The only real baseline is the original predictive coding formulation. Given that the presented formulation is strictly more expressive, it’s not all that surprising that the performance is matched or improved.

*Clarity*:
I found the clarity of the paper to be one of its major weaknesses.

I appreciate the fact that the authors included diagrams in Figures 1 and 2 to attempt to illustrate the differences between 1) predictive coding and backprop and 2) predictive coding with and without Gaussian assumptions. However, I found these diagrams difficult to interpret, so they didn’t add much to my understanding of the key differences. It’s not always clear what each of the arrows and shapes are meant to represent computationally. I also didn’t find the inconsistent vs. consistent dichotomy in Figure 2 to be clear — if I understand correctly, this is just pointing out the fact that typical predictive coding formulations assume a deterministic inference estimate, whereas one could assume other forms of distributions. Inconsistent, to me, implies that there is some type of mistake in the original formulation.

I found the notation to be difficult to follow at times. For instance, the authors use $\phi$ to denote the approximate posterior estimate, initially a point estimate of the latent variables. Later on, they switch to a notation in which $\phi^\mathcal{S}$ denotes this single estimate ($\mathcal{S}$ for sample) and $\phi^\mathcal{D}$ denotes a distribution. While I feel that I understand the main points that they authors are trying to convey, I have to imagine that there are easier ways to present the formulation, particularly for readers that are unfamiliar with predictive coding or alternatives to backpropagation. Instead, I would have kept a consistent notation for the approximate posterior estimate, showing that the standard formulation of predictive coding makes a particular assumption (Dirac), but this is either a) not expressive enough or b) not suitable for categorical/discrete variables, necessitating other distributions.

The details of the formulation are quite unclear to me. For instance, the authors allude to using the EM algorithm for optimizing both the generative model parameters and inference estimate. In the (typical?) case of Dirac inference estimates, Gaussian priors, and fully-connected networks, this seems fairly reasonable, as the gradient consists of the error at each layer, perhaps including the derivative of the non-linearity. However, in the more general case of non-Dirac inference estimates, non-Gaussian priors, and more general network architectures, it’s unclear what is included in these gradients, let alone whether calculating these gradients is biologically plausible. Further, the authors make several assumptions in their formulation (Eqs. 7 & 8) that seem poorly motivated. For instance, why the ‘energy’ at each layer is assumed to be the cross-entropy between the approximate posterior and prior, rather than the (correct) KL divergence. I suppose this makes sense in the case of Gaussians with fixed variance, but it seems unfounded in more general settings. Likewise, the free energy expression in Eq. 8 is a sum across layers of the network, completely neglecting any dependencies between layers — this is a fairly strong assumption. 

The empirical setup is also unclear. I would have preferred to see a more in-depth description of the network architectures, ideally with clear diagrams. Further, some of the training details were quite unclear, e.g., whether or not the internal Gaussian activations were sampled, the time required for the 32 E-step iterations used for inference, etc. These are very practically important, as they determine the computational efficiency and expressiveness of the network and training algorithm.

*Significance*:
While I admit that training alternative network architectures with predictive coding is mildly significant, I did not find the results to be particularly compelling. And while I understand that the point of the paper is not to obtain state-of-the-art performance on benchmarks, the empirical setups investigated by the authors are almost too simplistic as to be interesting, e.g., classification with 3-layer fully-connected networks, vanilla VAEs. Admittedly, I’m not sure whether transformers have been trained with backprop alternatives, so this could be somewhat significant. However, given my confusion around the specifics of the actual implementation (e.g., how are gradients obtained for inference?), it’s unclear, for instance, whether the model actually retains the biological plausibility of the original predictive coding formulation. And, ultimately, the authors are only able to roughly match the performance of backprop in these fairly standard settings. That is, the authors have not demonstrated that this technique opens up any new training settings. Finally, the authors repeatedly motivate their technique with the idea of being able to apply these algorithms on analog devices, however, it’s entirely unclear to me whether this algorithm is any more suited for these computing architectures than backprop.
questions: Why ignore the approximate posterior entropy in Eq. 7? Shouldn’t Eq. 8 contain an expectation over all approximate posterior dimensions? As currently written, this is a mean field assumption over layers, which I would imagine wouldn’t scale particularly well to large model architectures.

What is the relative training time for each algorithm?

Does this more generic formulation retain the biological plausibility benefits of the original formulation? It seems that you now need to obtain more general gradients at each layer, as opposed to the relatively simple form (errors) of the original formulation.

Are the internal activations sampled? This would yield a more expressive model as compared with a Dirac approximate posterior.

How does this method differ from target propagation?
limitations: Limitations are perhaps somewhat addressed. One aspect that the authors have not fully discussed is the increased training time required for these algorithms. As the authors state, they use 32 iterations to infer the activations of the network per training iteration. Naively, I would expect the predictive coding-trained networks to be 32 times slower than their backprop counterparts. Likewise, it’s unclear whether it’s biologically plausible to assume that we can optimize the more generic free energy in Eq. 8. While the authors do not explicitly make claims of biological plausibility, I see this as one potential limitation of moving away from the original predictive coding formulation.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work addresses the problem of generalizing Predictive Coding (PC) to arbitrary distributions. The related literature has presented PC as a biologically plausible alternative of backprop that relies on local (layerwise) updates. However, PC has so far been established as a credible alternative to backprop only when considering Gaussian activations. This is extremely unfortunate, as it limits the applicability of PC to virtually a small fraction of modern deep nets.

This paper generalizes PC to deal with any distribution, and importantly intractable distributions, approximated via a sampling scheme. The resulting framework is coherent with the PC theory, as it enables the definition of layer-wise energy functions that represent prediction errors.
strengths_and_weaknesses: Pros: Strong motivation; a very good mathematical foundation; sound mathematical proofs of on the equivalence with BP; potential for strong impact on the community; convincing simulations.
Cons: I need some deeper discussions on the weaknesses of the method compared to backprop. For instance, what happens in terms of computational complexity?
questions: I need some deeper discussions on the weaknesses of the method compared to backprop. For instance, what happens in terms of computational complexity? This must be analyzed in terms of training time, inference time, and required resources (FLOPS). Do you foresee architectures where the proposed method may linger compared to BP? What are the directions for future research to address these limitations?
limitations: n/a
review_title: null

Meta-review: metareview: Motivated by advancing the applicability of backpropagation alternatives, the paper extends predictive coding to non-Gaussian distributions, so it can be used to train effectively complex architectures such as transformers.

The reviews are divided: three reviews give a score of 7 (accept) whereas one review gives a score of 4 (borderline reject). The positive reviews cite the following strengths: clearly stated motivation, technical soundness, potential for impact and convincing experiments. The negative review cites lack of clarity as the main weakness (something also mentioned in one of the positive reviews), while the reviewer is not convinced about the originality of the method given similar advances in variational inference and generative modelling.

On balance, given the potential of impact in the field of predictive coding and the technical soundness, I'm happy to recommend acceptance, even though clarity is somewhat lacking.

Some reviewers requested more details on computational complexity, memory consumption and scalability. I encourage the authors to use the extra content page in the camera-ready version to discuss these aspects further.

confidence: Less certain
Decision: Accept