Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a variant of predictive coding (a biologically inspired framework for neural computation and learning) that better aligns with the known physiology and anatomy of cortex. This is achieved by applying a decorrelation constraint, yielding an upper bound on the PC objective. The resulting architecture is proposed to map onto the cortical microcircuitry, with each of the 3 main terms affecting a separate compartment of the pyramidal neurons. Numerical experiments show that this BioCCPC model can approach the performance of vanilla PC on MNIST. 
strengths_and_weaknesses: Strengths:
* Potentially important variant of predictive coding that could be of interest to neuroscientists and ML scientists.
* Clear biological motivation
* Thorough theoretical framework

Weaknesses:
* All results are obtained under an assumption of linearity that could limit the usefulness of the findings in ML applications
* The theoretical derivations are not always easy to follow
* Experiments are restricted to a toy model and dataset (MNIST). 

questions: * Some of the mathematical derivations seem to skip a number of steps, making them difficult to follow. A clear example is Equation (9), whose form is very difficult to relate to any of the previous equations. Similarly, Equation (4) doesn’t naturally follow from the previous ones.
* Should the $v_a$ and $n$ updates be switched in Algorithm 1? $v_a$ is updated using $n$, which isn’t assigned until the next step…
* Line 244, should it be the optimal linear inferred value for $y_{T+1}$ instead of $y_T$ ?

limitations: The main limitation due to the linearity assumption is properly acknowledged.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The main objective of this article is to propose a theoretical extension of the Predictive Coding (PC) algorithm, called the Covariance Constrained Predictive Coding (CCPC), to better ground PC in neurophysiology. The 2 main theoretical novelties compared to PC are: 1) the weights are separated to break symmetry between the feedforward and the feedback weights 2) the eigenvalues of the covariance matrix are upper-bonded using a Lagrange multiplier decomposition. These theoretical changes allow the authors to re-interpret CCPC in light of physiology.
strengths_and_weaknesses: This article is clear and well written. It is easy to follow and solve most of the ’neuro-plausibility’ issue raised by the PC algorithm. This article is mainly grounded in theory (which is 100% fine), but in the same time it do not propose a huge theoretical breakthrough. On my opinion, the 2 main drawbacks of this article are 1) The theoretical part is not rigorous enough, 2) the comparison with neurophysiology is interesting but it more a matter of interpretation than a rigorous quantitative comparison with neurophysiological data (see section "Questions" for more information).
questions: * The authors decided to derive the PC formulation from the probabilistic interpretation (which is good, on my opinion). But the probabilistic description of the model is incomplete as the authors do not define the factorization of the joint distribution. This lack of rigor leads to a misunderstanding : the authors are trying to minimize the negative log likelihood of the joint distribution (as this is done in Rao and Ballard [1]) as in a classical generative model, but here the authors seem to tackle another problem which is inferring the label knowing the image (i.e.  $p(z^n | z^0)$). Given Eq. 2, I imagine the considered factorization is the following one : $p(z^0, z^1, … z^n) = p(z^n | z^{n-1}) * p(z^{n-1} | z^{n-2}) … * p(z^0)$. And $p(z^0)$ is the probability of the input distribution (which is hard to assess). Actually the log probability the authors are minimizing in the right term of Eq 2 (which is also 100% fine) is log $p(z^n | z^{n-1}, z^{n-2}, ..,  z^{0})$, which is equivalent (provided markovian hypothesis, which seems to be the case here) to $p(z^n | z^0)$. I would suggest to update the model section of the article to make it more accurate (just keep in mind the reader does not have to guess the authors hypothesis !).

* In general, I think the section 3.1 is the more important one (this is where the novelty of this paper is). But this part is not rigorous enough for 2 reasons : 1- The authors suggest that splitting $W$ into $W_a$ and $W_b$, doesn’t change the optimization problem. This specific point should be demonstrated. The gradient (w.r.t. $z^l$) depends now on $W_a$ and $W_b$. Meaning that if $W_a \neq W_b$ (I guess this is the case, otherwise I don’t see the point of splitting), the gradient has therefore a different value, which suppose that the optimization problem is not the same anymore. I think this part is crucial, it would deserve a more rigorous demonstration. 2 - I have trouble to understand the inequality presented in Eq. 8. In the right term of Eq 8 there is a minimization w.r.t $W_{b}^{(l)}$, but as expected just after this equation, $L^{(l)}$ does not depend on $W_{b}^{(l)}$ but rather on $W_{b}^{(l-1)}$. Is it a typo in your inequality in Eq.8 ? Even if it is a typo, this inequality would deserve more explanation. 
 
* In Eq.9, the authors call the term $||z^{(l)}||^2$ a ‘prior’.‘Prior’ has a probabilist connotation, and here it doesn’t fit into your probabilist description of the model (nowhere, you have exhibited a normal prior for every $z^{(i)}$ in section 2.1). You can easily circumvent that by calling the term $||z^{(l)}||^2$ a ‘regularization term’ which has no probabilist connotation (I know I am being picky, but as this article is almost fully theoretical, it should be as rigorous as possible)

* Section 6: I do not understand the authors experimental comparison. It seems that the authors are comparing the training loss of PC, PC + cons, BioCCPC (cause on my understanding you have to give both the input and the one-hot to the model). Is it True ? Purely in terms of (training) loss, it seems that the BioCCPC performs not as good as PC (see Fig 3). Then what's the point of BioCCPC ? It is only a more bio-plausible formulation (if yes, it should be fitted to physiological data, see the last bullet point).

* Section 6 again: I do not think the training loss is a good measure to compare algorithms. As the authors already mention (line 141), the whitening regularization added in the CCPC should improve the generalization abilities. But with the training loss only, how is it possible to assess the generalization gap (different between training and testing ? ). Why don’t you show the testing accuracy / testing loss ?

* The authors suggest that their CCPC allows to interpret the variable in terms of neurophysiology. Then I would suggest to conduct quantitative comparison with neurophysiological data to back the authors interpretation.


Minor :

Line 100 : f is not defined at this point. The first time it is defined is line 118. Specifying f is an activation function when you use it, would improve the understanding of the article. 

Line 129 : with respect $z^{(l+1)}$ —> with respect to $z^{(l+1)}$

[1] : Rao, Rajesh PN, and Dana H. Ballard. "Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects." Nature neuroscience 2.1 (1999): 79-87.
limitations: In general, I think have properly discussed the limitations of their work in their article.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper derives an upper bound for the PC objective by imposing a decorrelation-inspired inequality constraint on the latent space, which later comes to a biologically plausible algorithm BioCCPC. The paper shows that the proposed algorithm matches the known physiology of the cortical hierarchy.

strengths_and_weaknesses: Strengths:

(1) The paper is well written, the structure and presentation are good and clear.

(2) The idea and theoretical analysis seem sound.

Weakness:

(1) After reading [4,1,2], the level of novelty contribution is not very clear, especially in the current linear setting.

(2)  I would like to see if the authors could provide additional experimental results on more (complicated) data, and the comparison between different PC baselines.

(3) I would also suggest that the authors can provide a way to interpret the results, to show a better way to understand the model.

questions: Major:
Please refer to the comments above in the Weakness section for the major concerns.

Minor:
The related work section could be added more theoretical or application works that include predictive coding. Simply listing a review paper may not be a good option.

limitations: See comments above.
review_title: null

Meta-review: metareview: A biologically more plausible extension of predictive coding that incorporates physiological detail is presented in this manuscript. I appreciate all discussions and extra experiments and improvements made to the paper. I agree with one of the reviewers who noted that it's a "potentially important variant of predictive coding that could be of interest to neuroscientists and ML scientists." Considering the theoretical contributions and new formulation to a subfield with significant growing interest, I recommend its acceptance to NeurIPS.
confidence: Less certain
Decision: Accept