Stable and Scalable Deep Predictive Coding Networks with Meta-Prediction Errors
Authors: Myoung Hoon Ha
Hyunjun Kim
Yoondo Sung
Youngha Jo
Min S. Kang
Sang Wan Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Neuroscience
Predictive Coding
Meta Predictive Coding
Free Energy
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Meta-PCN fixes deep PCN training instabilities—PE imbalance and exploding/vanishing prediction errors—via a meta-PE loss  and weight variance regularization, yielding statistically significant gains on CIFAR-10/100 and TinyImageNet.

Abstract: Predictive Coding Networks (PCNs) offer a biologically inspired alternative to conventional deep neural networks.
However, their scalability is hindered by severe training instabilities that intensify with network depth. 
Through dynamical mean-field analyses, we identify two fundamental pathologies that impede deep PCN training: 
(1) prediction error (PE) imbalance that leads to uneven learning across layers, characterized by error concentration at network boundaries; and
(2) exploding and vanishing prediction errors (EVPE) sensitive to weight variance.
To address these challenges, we propose Meta-PCN, a unified framework that incorporates two synergistic components: 
(1) a loss based on meta-prediction error, which minimizes PEs of PEs to linearize the nonlinear inference dynamics; and 
(2) weight regularization that employs normalization to regulate weight variance and mitigate EVPE.
Extensive experimental validation on CIFAR-10/100 and TinyImageNet demonstrates that Meta-PCN \rev{achieves} statistically significant improvements over conventional PCNs, outperforming backpropagation in most tested configurations, while preserving the local learning rules of PCNs.

Published as a conference paper at ICLR 2026
STABLE AND SCALABLE DEEP PREDICTIVE CODING NETWORKS WITH META-PREDICTION ERRORS
Myoung Hoon Ha! Hyunjun Kim Yoondo Sung
Youngha Jo Min S. Kang Sang Wan Lee
'Center for Neuroscience-inspired AI, KAIST
Dept. of ECE, Seoul National University
DDept. of Bio and Brain Engineering. KAIST
*LG CNS
School of Electrical Engineering, KAIST "Brain and Cognitive Sciences, KAIST
Kim Jaechul Graduate School of AI, KAIST
(mh. ha. soar, voungha908)@gma (ydsung, minseok21, sangwan aaaisa.....attta.a.a.o
com doctor3390@snu ac. kr
ABSTRACT
Predictive Coding Networks (PCNs) offer biologically inspired alternative conventional deep neural networks. However, their scalability is hindered by se- vere training instabilities that intensify with network depth. Through dynamical mean-field analyses, identify two fundamental pathologies that impede deep PCN training: (1) prediction error (PE) imbalance that leads to uneven learning across layers, characterized by eπrоr concentration network boundaries; and (2) exploding and vanishing prediction errors (EVPE) sensitive to weight vari- To address these challenges, propose Meta-PCN, unified framework that incorporates two synergistic components: (1) a loss based on meta-prediction error, which minimizes PEs of PEs to linearize the nonlinear inference dynam- ics; and (2) weight regularization that employs normalization to regulate weight variance and mitigate EVPE. Extensive experimental validation on CIFAR-10/100 and TinylmageNet demonstrates that Meta-PCN achieves statistically significant improvements over conventional PCNs, outperforming backpropagation in most tested configurations, while preserving the local learning rules of PCNs.
INTRODUCTION
Predictive coding (PC) represents theoretical framework for understanding cortical information processing. It encompasses fundamental functions such learning, prediction, encoding, and mem- orization. As neural architectures, PCNs implement this framework and offer a compelling alterna- tive backpropagation-bban
learning. PCNs are grounded in PC theory (Srinivasan et al., 1982; Mumford, 1992; Rao & Ballard, 1999; Friston, 2005) and formalized through the free-energy frame- work (Friston, 2010: Bogacz, 2017: Bastos et al., 2012). They employ purely local learning rules that respect biological constraints while enabling massive parallelization (Millidge et al., 2022b; Salvatori et al., 2022: Song al., 2020). This positions them as promising candidates for neuro- morphic computing (Schuman et al., 2017; Sacramento et al., 2018). However, PCNs face a critical limitation. As network depth increases, their training becomes progressively unstable. This creates formidable barrier to scalability (Millidge et al., 2022b). The underlying mechanisms driving this instability have remained poorly understood, hindering the practical deployment of deep PCNs in complex applications.
To address these fundamental challenges, we conduct a rigorous analysis of PCN inference dynamics using dynamical mean-field theory (DMFT) (Sompolinsky al., 1988; Poole al., 2016; Schoenholz et al., 2017). Our theoretical investigation (detailed Section 3) reveals two distinct yet interconnected pathologies that impede deep PCN scalability:
(1) PE Imbalance (Pinchetti et al., 2024; Qi et al., 2025; Goemaere et al., 2025: Innocenti et al., 2025): Errors concentrate in boundary layers (input/output) while vanishing in intermediate
"Corresponding author. 'Equal sccond-author contribution. EEqual contribution.

---

Published as a conference paper at ICLR 2026
layers. This creates a characteristic imbalanced distribution. This results in gradient starvation in mid-layers and prevents effective learning.
(2) EVPE: We identify exponential growth and decay patterns in latent states and PEs during inference. These dynamics are controlled by temporal scaling factors that depend critically on weight variance. This leads training instabilities analogous classical exploding and vanishing gradients (Bengio et al., 1994; Hochreiter, 1998; Pascanu et al., 2012; Arjovsky et al., 2016), though arising during inference rather than backpropagation.
To systematically address these pathologies, we propose Meta-PCN (Section 4). This unified frame- work incorporates two complementary solutions operating synergistically. First, we introduce a novel objective hased on reta-prediction error (meta-PE). This objective linearizes the nonlinear equilibrium system by minimizing PEs of PFs. Second, we implement normalization of weight variance. This controls variance and suppresses exponential behaviors. Through these two com- plementary solutions, our framework enables stable training of deep PCNs. We achieve substantial improvements in inference stability, convergence speed, and classification performance. These im- provements are obtained while preserving the local learning rules that underpin PCNs' relevance to neuromorphic applications (see Appendix A for a discussion of biological plausibility).
Extensive experimental validation CIFAR-10, CIFAR-100 and TinylmageNet demonstrates that Meta-PCN achieves substantial performance improvements, with 12-79% gains over conventional PCNs across all tested architectures (Figure 7). Notably, Meta-PCN outperforms backpropagation in 29 out of 30 configurations by an average of 2. 151 (22 statistically significant; see Appendix H), while adhering local learning constraints. These results suggest that Meta-PCN is viable frame- work for scaling PC to deeper architectures while preserving the local learning rules relevant to neuromorphic computing.
2 PREDICTIVE CODING NETWORKS
Z1
#1+1
21
Ri+1
fi(-)
fi(-)
9r(-)
AW; D Abj
v by gi
6
8101
(a) Inference phase
Leamiigg phuse
Figure 1: Inference and learning phases of local PC modules. (a) Inference phase: the PEs (6(41) are calculated and the latent states (z:) are updated. This process is repeated until it reaches the final inference step T'. (b) Learning phase: the weight and bias parameters (W₂ and by) are updated.
PCN Architecture and Objective. PC proposes that the brain continuously generates predictions of the external environment and refines internal representations by minimizing PEs-a computa- tional mechanism that extends beyond perception to reward-guided decision-making (Lee & Sey- mour, 2019; Lee et al., 2019). PCNs implement this principle by connecting local PC modules in a hierarchical chain structure, as illustrated in Figure 1. The forward pass generates predic- tions for subsequent layers, while the backward pass minimizes local PEs (Whittington & Bogaez, 2017; Millidge et al., 2022a). Each layer I with the latent state E RR:N produces predictions via br). where fi(-) represents the forward prediction function, Wi denotes the weight matrix, b is the bias vector, and o(-) is the activation function. The main goal is to minimize PEs for each layer, d; = 21 - 2g. quantified using the notion of free energy:
ut+1 = fi(z)) = a (Wiz
L
F
1
2
1=2
This optimization is subject to the boundary conditions
(input) and z y (target).
Inference and Learning Dynamics. PCNs employ dual optimization process that alternates be- tween inference and learning phases. During the inference phase, latent states evolve according to the fixed- point iteration: =+1 = z' 77 V F(z'),,, where the global latent state vector z is the con- catenation of all layer-wise latent states and denotes the inference rate. The gradient computation
2

---

Published as a conference paper at ICLR 2026
involves forward and backward operations that facilitate bidirectional information flow through- out the network. The forward prediction function fi(zi) = <WWiz + b:) propagates information from lower to higher layers, while the backward operation g( h
W, 1(b++1)6
transmits error signals from higher to lower layers, where h1+1 Wiz + by denotes the pre- activation and 1(h++1)
diag(ø (h1+1)) (1)4( represents the diagonal matrix of activation deriva- tives. The gradient naturally decomposes into bottom-up error and top-down feedback components: F = 8; - gi (8141, h(41). During the learning phase, parameters are updated according to Vw, -DD( 10 1014 and Jb = --D(h,1))0:
Challenges in Achieving Equilibrium. The equilibrium condition V2F 0 yields 6; gi ()))) which resembles standard error backpropagation algorithm (Whitrington & Bo- gacz, 2017; Millidge et al., 2022a). However, achieving equilibrium during inference presents sub- stantial computational challenges. Equivalence to backpropagation holds only under specific con- ditions, such as downweighting the output-layer error term to zero (Whittinton & Bogacz, 2017). The complex interplay between inference and learning dynamies makes understanding deep PCN behavior particularly challenging, motivating systematic analyses of deep PCN pathologies (Sec- tion 3).
DEEP PCN INSTABILITY: UNCOVERING FUNDAMENTAL PATHOLOGIES
This section provides comprehensive investigation into the internal inference dynamics in deep PCNs. We employ DMFT to mathematically characterize the underlying mechanisms driving these instabilities (Section 3.1). Our analysis reveals that deep PCNs suffer from two fundamental patholo- gies that severely impede practical scalability. These pathologies are PE imbalance (Section 3.2) and EVPE (Section 3.3).
Fig. 2
Fig.
15
Lou (A195) (AP)))
3.73
(h)
(9)
2.00
302
0,-0.
10
1020
2013
4913
Layer
(a)
(a)
(b)
(c)
26
(4)
Le(pee(Arr))
120
Lex(o))
Latee)
101+
Lea(el)
185
10
20
Itonation
Itttztion
Layer
(b)
(d)
tanh
(e)
(9)
Figure 2: The layer-wise distributions of lengths in PCNs: (a) len 8)), and (b) len len(vee( AW"))
Setting: randomly initialized linear PCN with Gaussian Gaussian-sampled inputs, outputs, and initial latent states (2 (O, ()); no training is involved, purely for analyzing inference dynamics (see Appen- dices C.I and E for details). We set = 30, the terminal inference step
200, and the latent dimension of each layer is set to 100. The inference rate is set 17 0.05. ow and σb are set to and 0. respecciivv., Figure 3: Dynamics of len(z)), len( and len(vec(AW?)). Dotted lines represent theoretical predictions, while solid lines correspond to empirical observations from linear PCN experiments.
Subfigures: Dynamics of (a) len(z)), (b) len(6;) and (c) len (vve((AW1)) of linear PCNs across
ow E {0.185, 1.0, 5.4]. (d) Dynamics of len 2 across σb E {0.185, 1.0, 5.43. (c) len of nonlinaar (tanh) PCNs across aw {0.185, 1.0,5.4). (f) Layer-wise distributions of len (z)) of
nonlinear (tanh) PCNs at t 20. Further analysis for nonlinear cases can be found in Appendix K.
Setting: In (a), (b), (c), and (d), I = 15, while (e), = 28. The other settings are the same as in Figure 2.
3.1 ANALYTICAL METHODOLOGY: DYNAMICAL MEAN-FIELD THEORY APPROACH
Motivation. PCNs present unique analytical challenges due to their dynamically evolving latent states during the inference phase. This makes difficult to characterize these dynamics using con- ventional analytical methodologies. To address this challenge, develop analytical framework
3

---

Published as a conference paper at ICLR 2026
based on the DMFT approach (Poole et al., 2016; Schoenholz et al., 2017). This framework provides mathematical tractability for the complex inference dynamics of PCNs.
Length-based Statistical Framework. Following the methodology established by Poole et al. (2016), we construct a statistical framework for analyzing PCN dynamics through length-based measures. We define the length of vector N +E= u², where T; denotes the i-th element of the vector x, and N represents the vector di- mension. In this analysis, weights and biases are assumed to be drawn i.i.d as w/y ~ (O N; and b9 ~~ (O0 of). Our analysis systematically tracks three fundamental length variables. These are the lengths of latent states len(z)), the lengths of PEs len(o)), and the lengths of weight up- dates len AW?)) at layer/ and time step t. where vec(-) denotes the vectorization operator that transforms matrix columns into a single vector and the weight update vec(A x V (. To mathematically characterize PCN dynamics,
the mean squared element: len(x) (af)
introduce three interaction matrices that cap- ture the intricate relationships between network components, The Latent Self-Interaction Ma- trix p captures inter-layer latent state interactions through P(+ 7 MM++-11Z2/. Here M1+ = M M ... Mt represents the product of normalized weight matrices and Mi = - Wi denotes the normalized weight matrix. The framework additionally incorporates du
bias-latent interactions via B //4 b MuN-zzi Bias self- inffinteeacctons are characterized through the constant matrix I = I. The interaction matrices enable the systematic derivation of length dynamics for each network component. The temporal evolution is governed by the in- teraction matrices themselves, which evolve according to the dynamics the latent state updates: 2/+1 -M kz) + + 2 1/b: - uM, by. Here, 1)0w governs the information propagation rate, and 1--1--n0 -1 + ow) controls self-interaction dynamies. The latent state length, (z)) = Pi directly corresponds to the diagonal elements of the inter-layer latent state interac- tions matrix P. The PE length emerges from the interaction between these components according to: len(o)) c, Pi=1:1.--:::G + 1-1.1111 where cy [-ow-- T The weight update length follows directly as leenneeee (vec( (sne(ve((AW)))) = len(6) l.n.lellen))))) The complete mathematical deriva- tions are presented in Appendix C, including the evolution equations along with detailed analysis of length dynamics. We emphasize that these results are derived under simplifying assumptions: ii.d. Gaussian weights and biases, linear (or linearized) forward and backward maps, uniform layer widths, and the large-width limit. These assumptions do not perfectly match the deep, nonlinear convolutional architectures used in our experiments (Section 6), leaving inherent gap between theory and practice.
+
3.2 PROBLEM I: IMBALANCED PREDICTION ERRORS
Empirical Observations. The first fundamental pathology of deep PCNs manifests as PE imbalance across network layers. As shown in Figure 2, PEs exhibit a characteristic imbalanced distribution. PEs decrease substantially in intermediate layers during inference. This results in significant error concentration at the boundary layers, while errors vanish in intermediate layers. This phenomenon arises from the fundamental constraints on information propagation in PCNs. Information propa- gates from k layers away at a rate of Or²)), where = 7T as established in our length- framework. With typical values of v & 1.0 (varying with inference rate and weight variance), this exponential decay causes inference to terminate prematurely before information effectively reaches the middle layers.
Theoretical Analysis: Imbalanced Error Distribution and Gradient Starvation. The mathemat- ical foundation of this imbalance can be understood through the effects of boundary conditions and the dynamics of error propagation. With the input layer clamped as Z
during inference, the PE 82 = Z2 22 = Z2 @ (W, + b1) contains a forcing term. This term continuously rein- troduces residual errors near the input boundary. Similarly. clamping the output layer 2,, y yields 5n = d(W,-1ZL- + bt-1). This typically remains nonzero and acts as a persistent error source the output boundary. The equilibrium condition 5; W,T (hh41))14 implies the spectral norm bound [|ō:]| < |Willlllo' (b++1) 1looo | The total bound across layers yields the product bound ||,,|| < (II |[W,llall'(h)z
|| When the terms in this product are less than unity, PEs decay geometrically they propagate downward from the output boundary.
4

---

Published as a conference paper at ICLR 2026
Combined with the term fixed at the input boundary, this creates the characteristic U-shaped error profile.
The PE Dilemma: Error Minimization Impedes Training. As PEs distribute unevenly across layers, certain layers experience 8 x= 0. Mcanwhile, the per-layer weight derivatives follow Vw,FF 1 for 2 - 1 < I 1. Consequently, when ò1-1 O, the gradient Vw,F= = and parameter updates vanish, leading to gradient starvation (Pezeshki et al., 2021). Thus, near-zero PEs at layer / directly eliminate learning signals for weights Wi, compromis- ing the learning capacity of deep PCNs. This presents a fundamental paradox. While PEs con- stitute the primary objective to minimize, reducing them to near-zero in any layer disrupts learn- ing signal propagation. This reveals that learning signals are transmitted through ô values, and their eliminaton blocks this critical information flow. The equílibrium condition's delta relationship (ò) gi (6++1 h(+1)) indicates not PE shrinkage, but rather the necessity of maintaining learning signal transmission through terms. This insight underpins our meta-PE-basee loss design (Sec- tion 4.1), which provides a principled solution to the inherent trade-off between minimizing error and preserving the learning signal.
3.3 PROBLEM 2: EXPLODING AND VANISHIGG PREDICTION ERRORS
Characterization. EVPE represents phenomenon distinct from the classical exploding van- ishing parameter gradients observed in backpropagation (Bengio et al., 1994; Hochreiter, 1998; Pascanu et al., 2012; Arjovsky al., 2016). Our DMFT analysis, corroborated by empirical mea- surements on deep PCNs (Figure 3), reveals multiplicative scaling patterns that manifest across in- ference iteration t. Specifically, we observe a scaling relationship of the form || s Te((..)) [ The multiplicative factors (a ) are governed primarily by the weight variance o and the effec- tive gain | (h)ll When > 1, the corresponding quantities experience geometric growth; conversely, when T. < 1, geometric decay occurs. Critically, stable dynamics (factors approach- ing unity) emerge only within a narrow interval of ow. specifically near one. This stable region contracts as network depth increases, making proper initialization exponentially more difficult for deeper architectures. Although nonlinear activations such as tanh and ReLU impose the constraint || I 1, thereby attenuating these multiplicative effects, they do not eliminate the underlying geometric scaling behavior.
Distinction from Classical Exploding/Vaniin Vanishing Gradients. The implications of EVPE extend beyond classical gradient pathologies due to the unique structure of PC weight updates. Since PC parameter updates follow Vw 1Dh 18 we have the proportionality relationship: vel(A x |18 ||||z|||| ol Il oc. This direct coupling implies that the exploding and van- ishing of latent states and PEs are immediately reflected in the magnitude of the parameter updates. Notably, this instability arises during the inference phase itself, before any parameter updates- fundamental distinction from traditional analyses that localize explosion or vanishing behavior to backpropagated loss gradients alone. In practical terms, large values of t. result in rapidly increas- ing magnitudes ||z]|| || or [|,5|], thereby inducing proportionally larger updates that can destabilize training. Conversely. small multiplicative factors yield near-zero updates, leading to training stag- nation and ineffective learning.
4 META-PCN: A UNIFIED FRAMEWORK FOR DEEP PCN STABILIZATION
Meta-PCN represents a comprehensive framework that addresses the two fundamental pathologies identified in Section 3 through synergistic combination of complementary techniques. Our ap- proach systematically targets each instability mechanism with tailored solutions that work in concert to enable stable deep PCN training. The framework comprises two core components that address the identified pathologies as follows:
Addressing PE Imbalance (Problem 1): We employ a dual approach that combines (i) the meta- PE objective to linearize the nonlinear equilibrium system (Section 4.1) and (ii) systematic weight regularization to control error propagation patterns (Section 4.2).
Mitigating EVPE (Problem 2): We implement comprehensive weight regularization strategies that control the multiplicative scaling factors Ty (a.)), thereby preventing geometric growth and de- cay patterns during inference (Section 4.2).
5

---

Published as a conference paper at ICLR 2026
Each solution component is designed to preserve local learning properties, while systematically addressing the mathematical and computational challenges that have hindered scalability to deep architectures.
4.1 INFERENCE META-PREDICTIO ERROR MINIMIZATION
Motivation. Conventional PC objectives suffer from fundamental structural limitations. First, as demonstrated in Section 3.2, the free-energy formulation creates paradoxical situation where min- imizing PEs leads to gradient starvation, eliminating learning signals and blocking their propagation through the network. Second, a critical train-test mismatch emerges in practice: while model predic- tions at evaluation time rely on direct entirely on latent state updates through the iterative inference process. We propose novel objective that addresses these fundamental issues while preserving core PC principles.
computation without inference, training depends
Loss Based on Meta-Prediction Errors. Our approach redesigns the PCN objective to transform the underlying inference dynamies. While conventional PCNs initialize latent states with feed-
forward predictions, we fix these predictions during inference: 2/4 = f = c₁ where c :=
((h, and h, (0) represents the initial pre-activation. Note that freezing feed-forwart predictions is
inherently tied to the meta-objective; see Appendix 1.4 for clarification and additional ablations. This modification effectively linearizes the nonlinear stationarity system F(z) - V v (() around
the feed-forward initialization point. By introducing error 5 = 2 Ct. we obtain a layer-wise
linear surrogate:
(z) = 8, -
= (z. c,) - w, D(b (2) c(11),
for 2 I establishing a linear fixed-point relationship between consecutive layer errors. Importantly, this represents linearization of the equilibrium map F(z). The Meta-PCN equilibrium approximates the standard PCN equilibrium only when inference trajectories remain near the feed- forward initialization; the do not coincide in general. Building this linearization, define loss function:
|8; 2
J(6)
1 Σ t=
(0) gi 9 1
Since PEs propagate top-down under feed-forwart initialization, treat the top-down transmit- ted signal د stabilized error. Conceptually, سū denotes the top-down error that would be present once inference converges to a fixed point. In practice, we approximate it using the current es-
timate 5(4) analogous to bootstrapping in cemporal-differenct learning. The conceptual innovation lies in treating (-) as a function that predicts the model's ferd-forward PE 8, using the stabilized error signals input. Therefore, I minimizes PEs of PEs meta-level objective that transforms I, the gradient of J with respect to 2, equals the linearized stationarity map: V = 8, - 9 ( Consequently, mini- mizing J drives the linearized equilibrium residual zero, providing tractable linear surrogate for the original nonlinear PC inference problem. This approach directly addresses the motivational problems. First, rather than directly minimizing PEs and causing gradient starvation, this objective encourages PEs to follow the delta relationship, ensuring balanced error propagation across layers in conjunction with weight regularization. Second, by fixing feed-forward predictions, we mitigate the train-test mismatch while transmitting learning signals via PEs. Importantly, iterative inference over latent states 2; preserved: only the predictions e₁ are frozen, while the latent states continue to evolve through 8; z C₁-
the learning dynamics. Since 06z/Oz; I and 06.z1/
Mitigation of PE Imbalance and Enhanced Convergence. Our analysis in Section demon- strates that laser-wise PEs satisfy the spectral bound ||8,]| < ||Wi]la || (h)+ [| When these multiplicative factors less than unity, the resulting product bound yields geometric decay from the output boundary, which-combined with the anchored input forcing- teristic U-shaped error profile. The meta-PE objective addresses this imbalance through its depen- dence on the operator W, (h, ). By regulating the scale of this operator, we directly control the amplification and attenuation factors in the spectral bound. Furthermore, the transformation from
the charac-

---

Published as a conference paper at ICLR 2026
solving nonlinear system linear surrogate provides substantial convergence improvements. The dynamics exhibit more predictable and stable behavior compared to the original nonlinear in- ference, enabling more efficient convergence to surrogate equilibrium states while preserving the essential PC principles.
Parameter Learning. While the meta-PE objective J governs the inference dynamics over latent states, parameter updates follow the conventional PC loss: [()) = _/ ---1 |z
(0) (2 :): This separation allows the meta-objective to stabilize inference while preserving fi-1
the standard PC learning rule for parameters (see Appendix for details).
4.2 WEIGIIT REGULARIZATION
To address the identified pathologies, Meta-PCN employs a variance-based normalization strategy that provides computationally efficient spectral control while maintaining stable weight distributions across network layers. We introduce a practical alternative to direct spectral norm computation that leverages the relationship between weight variance and spectral properties. For weight matrix W with dimensions (m, n) and variance o E Var(W), we apply direct normalization:
W
W (Vm Vn)o
This approach draws upon random matrix theory. For weight matrices with i.i.d. entries having zero mean and variance o" the spectral norm satisfies ||W||2 & (vm vn)ow. Our variance-based scaling ensures [Woomaaizee// RJ 1, serving as a computationally efficient proxy for spectral con- trol. The theoretical foundation rests on the conservative upper bound [[W]|2 S [W]]F R Vmnig and the random matrix theory result ||W]|2 (ym vn)oww enabling accurate spectral norm approximation. For implementation, we determine operator dimensions as follows: linear layers dm: convolutional layers employ m Cou,, n C.n kH kw- This ap- proach offers several advantages: low computational cost with parallel computation, no additional parameters, uniform application across layer types, and robust spectral control by maintaining |[W ||2 near unity. We note that the i.i.d. assumption does not fully capture the structured nature of convolu- tional filters or the evolving weight distributions during training: a complete theoretical treatment for such structured operators remains an important direction for future work. The variance normaliza- tion framework directly addresses the exponential scaling behaviors identified in our analysis while preserving computational efficiency. By regulating o² to maintain spectral norms near unity, this approach simultaneously mitigates both EVPE and PE imbalance pathologies, providing an effec- tive solution for deep architectures. The complete training procedure is summarized in Algorithm (Appendix D).
dowt,
5
META- PCN RESOLVES TWO FUNDAMENTAL PATHOLOGIES
This section demonstrates that Meta-PCN successfully addresses the two fundamental pathologies identified in Section 3. The experiments are conducted under identical conditions to enable direct comparison with the problematic behaviors observed in conventional PCNs (see Appendix E for experimental details).
Balanced PEs. We replicate the PE imbalance analysis from Section 3.2 using the Meta- Meta-PCN frame- work. Figure 4 shows the layer-wise PE distribution under Meta-PCN. Unlike the characteristic U-shaped error concentration observed in Figure 2, Meta-PCN exhibits well-balanced error distri- butions across all network layers. PEs maintain relatively uniform magnitudes across layers without excessive concentration at boundary layers. Note that since the output layer is clamped (zz. = y), the boundary error 8, remains fixed: inference redistributes this error through latent layers. The weight update magnitudes correspondingly show balanced distributions, ensuring that learning sig- nals reach all layers effectively. Consequently, the meta-PE objective and weight regularization regulate the operator W," (h, maintain relatively uniform scaling across layers, preventing the geometric decay patterns that created foundary-layed dominance in conventional PCNs.
For classification tasks, the squared error at the output layer replaced with cross-entropy loss.

---

Published as a conference paper at ICLR 2026
Fig. 4
Fig.
Fig.
t- 15
1- 15
(+), (b)
100
0.000
1.0(66)
Liyer (a)
21
Iteration (a)
Ineration (b)
126 Itensioo (a)
200
100
Lan(we(AAr (AW())
Len (AW?))
(c)
32'4
10 20
Layer
leestton
Layer I
(b)
(c)
(b)
Figure 4: The layer-wise distributions of lengths in Meta-PCNs: (a) len(6)), and (b)
(eneve(( A W))). Setting: same as in Figure 2.
Figure 5: Dynamics of (a) len(z;), (b) len(8)), and (c) (en(vec(AW()) across ow E
{0.185, 1.0, 5.4) in Meta-PCNs. Due to weight variance normalization, the three ow cases over- lap and appear as a single trajectory. Setting: same as in Figure 3.
Figure 6: Convergence dynamics and equilibrium analysis during inference. (a) Objective func- tion convergence showing Meta-PCN's meta-PE objective J over ô (right y-axis) and conventional PCN's standard PE objective F over z (left y-axis) in the inference phase. (b) Layer-wise conver- gence to delta relationships in Meta-PCN, measured by the gradient norm of the objective function.
Stable PEs. Figure 5 demonstrates that Meta-PCN substantially mitigates the exponential growth and decay patterns that characterized EVPE in conventional PCNs. The temporal dynamics re- main stable with controlled magnitudes across all weight variance settings. The latent state lengths (Figure 5a), PE lengths (Figure 5b), and weight update lengths (Figure 5c) all maintain stable trajectories without the dramatic exponential scaling observed in Figure 3. Because we normal- ize weight variance enforce a uniform scale across all conditions, the three different cases (ow E (0.185, 1.0, 5.4]) overlap and are visually indistinguishable, appearing as a single trajectory in the figure. This is expected result since variance normalization maintains the ow of weight matrices at the same scale regardless of the initial ow. Therefore, the multiplicative scaling factors (T that previously caused geometric growth or decay are now effectively controlled through the variance-basse weight regularization strategy. This regulation of multiplicative factors climi- nates the root cause of EVPE while preserving essential PC dynamics. All quantities remain within manageable ranges, preventing both explosion and vanishing dynamics.
Enhanced Convergence Properties. We evaluate convergence improvements by comparing the inference dynamics of conventional PCN and Meta-PCN, measuring the reduction in their respec- tive objectives. Although a direct comparison inherently limited by the differences objectives, Figure 6 reveals substantial convergence improvements achieved by Meta-PCN. The reduction of the objective in Meta-PCN exhibits rapid and definitive convergence to zero, in contrast to the slow convergence dynamics of conventional PCNs (Figure 6a). Layer-wise gradient norm analy- sis (Figure 6b) demonstrates the rapid resolution of high meta-PEs initially observed in some lay- ers, indicating quick convergence to equilibrium. Overall, Meta-PCN achieves better convergence than conventional PCNs while maintaining stability across all layers. The linearized meta-PE-based objective empirically demonstrates improved convergence properties compared to the original non- linear equilibrium system (Figure 6). The transformation from nonlinear to a linearized system addresses fundamental convergence limitation of conventional PC inference.
META-PCN ENABLES SCALABLE DEEP LEARNING
Experimental Setup. We evaluate Meta-PCN on CIFAR-10, 100, and TinylmageNet datasets using VGG and ResNet architectures of varying depths. We directly compare backprop- agation (BP), conventional PCNs with only feed-forward initialization, and the complete Meta-PCN framework. Our three-way comparison is conducted under identical experimental settings except for algorithmic differences: the underlying architectures and all shared hyperparameters (optimizer, learning rate, batch size, etc.) are kept exactly the same across BP, PCN, and Meta-PCN PCN; only the ob- jective function and update rules differ (see Appendix E for details). MeNa-PCN does not introduce
8

---

Published as a conference paper at ICLR 2026
100
$0.58
Top Acturayy (%)
100
Top-1 Accuraty
80
IRC PM
Top- Acturary
R0
60
50.09
53.16
Sec
Meta PUN
40
CN 384
urc PN
60
$4.0
46
42.0
29
35
58.66
20
23.7
19:43
17.56
7.00
1.0
20
17%
1699
72
VC00.7
VGD59
HOC-11
VGG-13
BeaNee-10
VoG.3
voG7
Vo0-
VoG-11
VDG 10
ReaNet:10
VGG-
VGG-7
VGG-9
VGG
V3r-13
ReaSiod
CIFARIO, VGG13
Tha Accuray (ha)
80
28.15
10.35
Top-5 Acc-uay (16)
100
100
DPC
71.02
Top Aecuray
02.79
60
15.29
73.24
36
MENN
90
X01
40
BocScN D0C
22.01
NN
20
885
101
20
30
401
20
VG0-8
VG0-7
VGG--
V30-43
Restles 18
voo.3
volc
Ve0-11
V00-13
Rettiet
Epoch
(d)
Figure 7: Classification accuracy across network architectures (VGG-5, 7,9, 11, 13, & ResNet-18)
and datasets (CIFAR- 10/100 and TinylmageNet.. Performance comparison of Meta-PCN, back- propagation (BP), and PCN on (a) CIFAR-10 (Top-1 acc.). (b) CIFAR-100 (Top-1 acc.), (c) Tiny- ImageNet (Top-1 acc.), (d) Training dynamics on CIFAR-10 with VGG-13 showing test accuracy evolution over 50 epochs. (c) CIFAR-100 (Top-5 acc.), and (f) TinylmageNet (Top-5 acc.). Mean ac- curacy values over 5 runs are annotated for each method, architecture, and dataset (see Appendix H for standard deviations and statistical significance tests). Scattered points indicate literature-reported nudging (PN/NN/CN) (Pinchetti et al., 2024), incremental PC (iPC) (Salvatori et al., 2023b), and Deep PCN (DPC) / DPC with centered nudging (DPC-CN) (Qi et al., 2025).
results for PCN variants: positive/negativeceettee
additional hyperparameters beyond those in conventional PCNs (inference rate, number of inference steps, optimizer settings); its computational overhead is analyzed in Appendix G. Additionally, we compare various PC variants by directly incorporating values from their respective studies. We note that some recent PCN implementations (Pinchetti et al., 2024) employ auxiliary training heuristics such as GELU activations, momentum during inference, and no weight decay-choices orthogonal to the core PC mechanism. Our experiments deliberately use a standard training protocol to isolate the effect of Meta-PCN's ' structural improvements rather than method-specinn tuning.
Performance Across Network Depths and Datasets. Figure 7 presents classification accuracy across different architectures and datasets. The results reveal distinct performance patterns con- sistent with our theoretical predictions about PCN scalability limitations and Meta-PCN's effec- tiveness in addressing them. On CIFAR-10 (Figure 7a). conventional PCNs exhibit severe perfor- mance degradation with increasing depth, achieving only 10-20% accuracy across most architec- tures. In contrast, Meta-PCN demonstrates strong stability, maintaining 80-90% accuracy across all tested depths. Notably, within our controlled experimental setting, Meta-PCN outperforms BP across most architectures, with improvements ranging from 0.04% (ResNet- -8) to 1.73% (VGG-5), four of which are statistically significant, suggesting that local learning rules need not compromise competitive performance (see Table 2 in Appendix H). Similar patterns emerge on CIFAR-100 (Fig- ure 7b) and Tiny ImageNet (Figure 7c). Across all 30 tataset-architectur---eee Meta-PCN outperforms BP in 29 cases (22 statistically significant), with an average improvement 2.1%% The sole exception is CIFAR-100 ResNet-18 Top-1, where BP marginally leads by 0.02% (p=0.84); see Appendix H for full results.
configurations,
Our experimental design employs stringent controls to ensure identical conditions across BP, PCN, and Meta- This rigorous standardization may result subdued BP performance relative lit- erature reports employing method specific optimizations; likewise, literature-reported PCN variants consistently underperformed their corresponding BP implementations in those studies, underscoring the importance of controlled comparative evaluation. Despite this conservative comparison protocol, in deeper architectures V3, ResNet-18), Meta-PCS matches or exceeds BP without requiring architecture-specinn specific tuning, suggesting that the proposed theoretical resolution provides scalability advantages.
Training Dynamics and Stability, Figure 7 illustrates learning trajectories over 50 training epochs, comparing the three approaches on a representative architecture and dataset (VGG-13 on C0F.R-10). The analysis reveals fundamental differences in optimization behavior that complement

---

Published as a conference paper at ICLR 2026
framework. Conventional PCNs exhibit minimal learning progression, plateauing approxi- mately 12% accuracy throughout training. This behavior is consistent with the gradient starvation phenomenon identified in our theoretical analysis. Meta-PCN, by contrast, demonstrates smooth and monotonic improvement, closely paralleling BP's learning trajectory and achieving higher final accuracy in this configuration (Meta-PCN 89.53% vs. BP 87.85%).
Ablation results are reported in Appendix I. The ablation confirms that the meta-PE objective is essential (removing it drops accuracy from 89.5% 10.0%), while weight regularization provides statistically significant but moderate improvement (1.3%).
7 RELATED WORK
With increasing depth, PCN benchmarks report deteriorating accuracy, skewed layer-wise prediction-eree eerrot and energy profiles, and growing imbalances in relaxation speeds (Pinchetti al., 2024; Kinghorn et al., 2023; Qi et al., 2025). Symptomatic remedies tune schedules or normaliz- ers; for instance, interleaving state and weight updates improves robustness and speed but does not give depth-agnostic guarantees for stable inference (Salvatori et al., 2023b). Theoretical reinterpre- tations cast PC updates as implicit stochastic gradients or connect PCNs to backpropagation and equilibrium methods, clarifying step-size stability while offering few constructive prescriptions for removing depth-inde induced pathologies (Alonso et al., 2022; Millidge et al., 2022a).
Some depth-focused methods reduce error-distortion accumulation but rely on temporally non-local updates that do not enforce contraction (Qi et al., 2025). Others reparameterize objectives to opti- mize directly in error space, mitigating exponential signal attenuation (Goemaere et al., 2025), or adopt depth-aware parameterization and learning-rate scaling to train deep residual PCNs (Inno- centi et al., 2025). Collectively, these advances boost performance yet still fall short of general, architecture-anostic guarantees of stable inference.
separate line of work has explored theoretical connections between predictive coding and back- propagation. Whittington & Bogacz (2017) showed that PC updates can approximate backpropaga- tion under certain conditions, and Millidge et al. (2022a) extended this analysis to arbitrary compu- tation graphs. While these studies clarify when PC and BP produce similar learning signals, they do not address the stability pathologies that arise in deep PCNs. Our meta-PE objective is comple- mentary: rather than establishing equivalence to BP, it is designed to stabilize inference dynamics by minimizing prediction errors of prediction errors, directly targeting the EVPE and PE imbalance phenomena identified in analysis.
Our contribution is to treat stability as the primary design objective and to provide a unified, theory- guided remedy for depth-induced pathologies in PCNs. Using a dynamical mean-field, length-based analysis, we show that PE imbalance and EVPE arise from boundary conditions and the spectral statistics of the weights, and we use this account to design Meta-PCN: a meta-PE objective, to- gether with variance-basee weight normalization that controls the effective spectral norms. These components enforce contraction and scale separation during relaxation while preserving fully local learning, yielding stable and scalable training across the architectures and datasets tested. We em- phasize that feed-forward-baeed latent initialization is standard practice in the PCN literature and not part of our claimed contribution; our novelty lies in the DMFT-guides diagnosis of these pathologies and the resulting meta-PE plus variance-normalizza framework.
8 CONCLUSION
This study identifies two fundamental pathologies that impede the scalability of deep PCNs. Through DMFT analysis, we analytically characterize how PE imbalance and EVPE constitute the core obstacles to deep PCN training. We propose the Meta-PCN framework that systematically addresses these problems through two synergistic components: (1) meta-PP--bsee loss function that linearizes the nonlinear equilibrium system to provide stable dynamics, and (2) variance-based weight regularization that suppresses EVPE. Sustained improvements across diverse datasets and architectures, coupled with stable training dynamics, position Meta-PCN as promising step toward a practical alternative to backpropagation-basee optimization while preserving the local learning properties relevant to neuromorphic implementations.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Jung Young Kim for valuable advice and discussions on biological plausibility. This research was partly supported by the MSIT (Ministry of Science, ICT), Korea, under the Global Research Support Program in the Digital Field program (RS-2024-00436680) supervised by the ITP (Institute for Information Communications Technology Planning & Evaluation). This project was partly supported by Microsoft Research Asia. This work was partly supported by Global Research Cluster program of Samsung Advanced Institute of Technology. This work was partly supported by Institute for Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. with high-level brain functions). This work was partly supported by Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) RRS-2019-I119009 Artificial Intelligence Graduate School Program (KAIST)). This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) IRS-2024-00341883) This work partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2022- NR068758).
System3 reinforcement learning

---

Published as a conference paper at ICLR 2026
REFERENCES
Nicholas Alonso, Beren Millidge, Jeffrey Krichmar, and Emre Neftci. A Theoretical Framework for Inference Learning. Advances in Neural Information Processing Systems, 2022.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary Evolution Recurrent Neural Networks. In Proceedings of The 33rd International Conference Machine Learning, pp. 1 1120-1122 PMLR, June 2016.
Andre M Bastos, W Martin Usrey, Rick A Adams, George R Mangun, Pascal Fries, and Karl J Friston. Canonical microcircuits for predictive coding. Neuron, 76(4):695-711, 2012.
Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, March 1994. ISSN 1941-0093. doi: 10. 10.1109/72.279181
Rafal Bogacz. A tutorial on the free-energy framework for modelling perception and learning. Journal of Mathematical Psychology, 76: 16111, 2017.
Matteo Carandini and David Heeger. Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1):51 -2,, 2012. doi: 10.1038/mm3136.
Rishidev Chaudhuri, Kenneth Knoblauch, Marie-Alice Gariel, Henry Kennedy, and Xiao-Jing Wang. A Large-Scale Circuit Mechanism for Hierarchical Dynamical Processing in the Primate Cortex. Neuron, 88(2):49-4311 2015. ISSN 1097-4199. doi: 10. wwwll/o.euroo..05....o.
Sophie Denève and Christian K. Machens. Efficient codes and balanced networks. Nature Neuro- science, 19(3):375-32,, 2016.
Harriet Feldman and Karl Friston. Attention, Uncertainty, and Free-Energy. Frontiers Human Neuroscience, 4, 2010. ISSN 1662-5161. doi: 10.389//hhum.220000025...
Karl Friston. A theory of cortical responses. Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456):815-8333 2005.
Karl Friston. The free-energy principle: unified brain theory? Nature Reviews Neuroscience, 11 (2):127-138, 2010.
Dileep George, Rajeev V. Rikhye, Timothy A. Engel, et al. A detailed theory of thalamic and cortical microcireuits for predictive visual inference. Science Advancex, (()):eadr6698, 2025.
Cédric Goemaere, Gaspard Oliviers, Rafal Bogacz, and Thomas Demeester. Error Optimization: Overcoming Exponential Signal Decay in Deep Predictive Coding Networks, 2025.
Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, Baltimore, MD, 4th edition, 2013. ISBN 978-1-4214-0794444
Neil R. Hardingham, Giles E. Hardingham, Kevin D. Fox, and Julian J. B. Jack. Presynaptic efficacy directs normalization of synaptic strength in layer 2/3 rat neocortex after paired activity. Journal of Neurophysiology. (7(4):2965-22 2975, April 2007. ISSN 0022-3077. doi: 10. 10.1152/jn.01352.2000
Loreen Hertäg and Claudia Clopath. Prediction-error neurons in circuits with multiple neuron types: Formation, refinement, and functional implications. Proceedings of the National Academy of Sciences, 119(13)e21 156991199 19, 2022.
Sepp Hochreiter. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Bant Sys- tems, 6: 6:10-1116, April 1998. doi: 10. 10.1142/5021848885900000999
Roger Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Francesco Innocenti, El Mehdi Achour, and Christopher L. Buckley. SySPC: Scaling Predictive Coding to 100+ Layer Networks, 2025.
12

---

Published as a conference paper at ICLR 2026
Ryota Kanai, Yutaka Komura, Stewart Shipp, and Karl Friston. Cerebral hierarchies: Predictive pro- cessing, precision and the pulvinar. Philosophical Transactions of the Royal Society B: Biological Sciences, 370(1668):2011000)
2015. doi: (0099///st..014...0
Georg B. Keller and Thomas D. Mrsic-Flogel. Predictive Processing: A Canonical Cortical Com- putation. Neuron, 100(2):24 435, October 2018. ISSN 0896-6273, doi: n0.10166..eeuron2201. 10.003.
Stefan J. Kiebel, Jean Daunizeau, and Karl J. Friston. A hierarchy of time-scales and the brain. PLOS Computational Biology, 4(11)xe100000) 2008. ISSN 1553-7358. doi: 10.1371/journal. pcbi.1000200,
Paul F. Kinghorn, Beren Millidge, and Christopher I. Buckley. Preventing Deterioration of Classifi cation Accuracy in Predictive Coding Networks. In Christopher L. Buckley, Daniela Cialfi, Pablo Lanillos, Maxwell Ramstead, Noor Sajid, Hideaki Shimazaki, and Tim Verbelen (eds.), Active Inference, pp. 1-15, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-287 29-7-998. doi: 10.1007/978-3 3-031-28719900
Jee Hang Lee, Ben Scymour, Joel Z. Leibo, Su Jin An, and Sang Wan Lee. Toward high- performance, memory-efficient, and fast reinforcement learning-lessons from decision neuro- science. Science Robotics, 4(26)xeav2975, 2019. doi: 10. 126/scirobotics.av2977.
Sang Wan Lee and Ben Seymour. Decision-making in brains and robots the case for an in- terdisciplinary approach. Current Opinion in Behavioral Sciences, 26:137-145, 2019. doi: 2..0016/..oobhha.20 12.012.
Mathieu Letellier, Florian Levet, Olivier Thoumine, and Yukiko Goda. Differential role of pre- and postsynaptic neurons in the oetivity-dependent control of synaptic strengths across dendrites. PLOS Biology, 1706):e0006223, 2019. ISSN 1545-7885. doi: 10. 1 phi.mmmm.ppp.o20022223.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Jorge F Mejías and Xiao-Jing Wang. Mechanisms of distributed working memory in a large-scale network of macaque neocortex. eLife, 11:e72136, February 2022. ISSN 2050-084X. doi: 10. 7554eeLife.72136
Fabian A. Mikulasch, Lucas Rudelt, Michael Wibral, and Viola Priesemann. Where is the error? Hierarchical predictive coding through dendritic error computation. Trends in Neurosciences, 46 (1):45-59) January 2023. ISSN 0166-2236, 1878-108X. doi: 10.1016/j.tins..oo 200...2..02...00000.....
Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, and Rafal Bogacz. A The- oretical Framework for Inference and Learning in Predictive Coding Networks. In The Eleventh International Conference Learning Representations, September 2022a.
Beren Millidge, Alexander Tschantz, and Christopher L. Buckley. Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs. Neural Computation, 34(6):1329-133) May 2022b. ISSN 0899-7667.
doi: 10. 10.116//ncooa.0014
Rosalyn J. Moran, Pablo Campo, Mkael Symmonds, Klaas E. Stephan, Raymond J. Dolan, and Karl J. Friston. Free Energy, Precision and Learning: The Role of Cholinergic Neuromodulation.
Journal of Neuroscience, 23(19):8227-825) 2013. ISSN 0270-6474, 1529-2401. doi: 10.1523/ INEUROSCI42555 12.2013.
David Mumford. On the computational architecture of the neocortex: the role cortico-cortical loops. Biological cybernetics, 66(3):241-251, 1992.
John D. Murray, Alberto Bernacchia. David J. Freedman, Ranulfo Romo, Jonathan D. Wallis, Xiny- ing Cai, Camillo Padoa- Aadoa-Schioppa. Tatiana Pasternak, Hyojung Seo, Dacyeol Lee, and Xiao-Jing Wang. A hierarchy of intrinsic timescales across primate cortex. Nature Neuroscience, 17(12): 1661-1663, 2014. ISSN 1546-1726. doi: 10.1038/mm.3862.
Erkki Oja. Simplified neuron model as principal component analyzer. Journal of Mathematical Biology, 15(3):6672733, November 1982. ISSN 1432-1416. doi: 10. 10.1000/F002256687.
13

---

Published as a conference paper at ICLR 2026
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. ArXiv, November 2012.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient Starvation: A Learning Proclivity in Neural Networks. In Advances in Neural Information Processing Systems, volume 34, pp. 1256-1272. Curran Associates, Inc.. 2021.
Luca Pinchetti, Chang Qi, Oleh Lokshyn, Cornelius Emde, Amine M M'Charrak, Mufeng Tang, Si- mon Frieder, Bayar Menzat, Gaspard Oliviers, Rafal Bogacz, Thomas Lukasiewicz, and Tom- maso Salvatori. Benchmarking Predictive Coding Networks Made Simple. In The Thirteenth International Conference on Learning Representations, 2024.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Dieksiicst,in, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos, jun 2016.
Chang Qi, Thomas Lukasiewicz, and Tommaso Salvatori. Training Deep Predictive Coding Net- works. In New Frontiers in Associative Memories, 2025.
Rajesh PN Rao and Dana Ballard. Predictive coding in the visual cortex: functional interpreta- tion of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
Youcef Saad. Iterative Methods for Sparse Linear Systems. SIAM, Philadelphia, PA, 2nd edition, 2003. ISBN 978-0-89871-534-7
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcir- cuits approximate the backpropagation algorithm. Advances in neural information processing systems, 31, 2018
Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz. Rafal Bogacz, and Zhenghua Xu. Asso- ciative memories via predictive coding. Advances in Neural Information Processing Systems, 35: 3874-3888, 2022.
Tommaso Salvatori, Ankur Mali, Christopher I.. Buckley, Thomas Lukasiewicz, Rajesh P. N. Rao, Karl Friston, and Alexander Ororbia. Brain-Inspired Computational Intelligence via Predictive Coding, August 2023a.
Tommaso Salvatori, Yuhang Song. Yordan Yordanov, Beren Millidge, Lei Sha, Cornelius Emde, Zhenghua Xu, Rafal Bogacz, and Thomas Lukasiewicz. A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks. In The Twelfth International Conference Learning Representations, October 2023b.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation, apr 2017.
Catherine Schuman, Thomas E Potok, Robert M Patton, J Douglas Birdwell, Mark Dean, Garrett S Rose, and James S Plank. A survey of neuromorphic computing and neural networks in hardware. arXiv preprint
2017.
H. Sompolinsky, A. Crisanti, and H. J. Sommers. Chaos in Random Neural Networks. Physical Review Letters, 61(3):259-262, 262, jul 1988. doi: 10.1103PhhyRReeeett.....55
Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Can the brain do backprop- agation?-exxxe implementation of backpropagation predictive coding networks. Advances in neural information processing systems, 03:22566-22577 2020.
Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: fresh view of inhibition the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, (16(129)):477-59, 1982.
14

---

Published as a conference paper at ICLR 2026
Gina G. Turrigiano, Kenneth R. Leslie, Niraj S. Desai, Lana C. Rutherford, and Sacha B. Nelson. Activity-dependent scaling quantal amplitude in neocortical neurons. Nature, 891 896, February 1998. ISSN 1476-4687. doi: 10.1038/36103.
Richard S. Varga. Matrix Iterative Analysis, volume 27 of Springer Series in Computational Mathe- matics. Springer, Berlin, Heidelberg, 2nd revised and expanded edition, 2009, ISBN 978-3-540- 66321-8.
Xiao-Jing Wang. Synaptic reverberation underlying mnemonic persistent activity. Trends Neuro- sciences, 24(8):455-4633 August 2001. ISSN 0166-2236. doi: (666000006662226000011066666
Xiao-Jing Wang. 50 years of mnemonic persistent activity: quo vadis? Trends in Neurosciences, 44
(11):888-902, November 2021. ISSN 0166-2236, 1878-108X. doi:
James CR Whittington and Rafal Bogacz. An approximation of the enror backpropagation algorithm in predictive coding network with local hebbian synaptic plasticity, Neural computation, 29(5): 1229-1262, 2017.
David M. Young. Iterative Solution of Large Linear Systems. Computer Science and Applied Mathematics. Academic Press, New York, 1971, ISBN 978-0-12-773050-9.
15

---

Published as a conference paper at ICLR 2026
A BIOLOGICAL PLAUSIBILITY OF META-PCNS
A.1 BIOLOGICAL PLAUSIBILITY OF INFERENCE AS META-FREE FREE ENERGY MINIMIZATION
Our inference objective fixes each layer's feed-forward target c₁ and minimizes the norm of meta- PE. [|ō - 9) (81+1 (0) )|2 so that the realized local error 8j matches top-down prediction of that error. This "error of errors" matching is consistent with hierarchical predictive coding accounts in which ascending signals progressively encode residuals of residuals, supported by division of labor between superficial error units and deep representation units, and by laminar asymmetries in feed- forward versus feedback pathways (Rao & Ballard, 1999; Bastos et al., 2012: Keller & Mrsic-Flogel, 2018). Dendritic predictive coding further suggests that basal and apical dendritic compartments can locally compare bottom-up up errors with top-down predictions, implementing error computation with- out requiring distinct error neurons (Mikulasch et al., 2023; Salvatori et al., 2023a). Contemporary cortical PC models operate at two complementary levels: (i) laminar hierarchies where deep layers receive predictions and superficial layers relay PEs, and (ii) local microcircuits where pyramidal cells compute PEs via dendritic comparison (Hertäg & Clopath, 2022: Denève & Machens, 2016). Meta-PCN's s meta-prediction errors can be implemented by interacting ertor neuron populations or distinct dendritic compartments. Recent thalamocortical models with lamina-specific prediction and error pathways further support this circuitry (George et al., 2025). Taken together, these views align with our formulation as an explicit hierarchical comparison between realized errors and their pre- dicted values, while remaining faithful to canonical message-passing in cortical microcireuits (Bas- tos et al., 2012). A complementary line of work interprets meta-error minimization as dynamically adjusting precision weights on errors. Predictive coding theories posit context-dependent gain con- trol of sensory eгror signals, with neuromodulatory systems (notably cholinergic) and thalamocorti- cal circuits (via the pulvinar) regulating the effective gain of error transmission
task-dependent manner (Feldman & Friston, 2010; Moran et al., 2013; Kanai et al., 2015). More broadly, predic- tion errors serve as a core computational currency not only in perception but also in reward-guides decision-making, where interdisciplinary perspectives linking neuroscience and machine learning have proven fruitful (Lee & Seymour, 2019; Lee et al., 2019). Our objective reduces the mismatch between observed errors and expected precision, offering a mechanistic bridge between these the- oretical proposals and graded, circuit-level control of error gain. Furthermore, clamping e during inference is supported by the presence of hierarchical intrinsic timescales in the cortex: higher-order association areas evolve slowly, acting as quasi-fixed boundary conditions over short inference win- dows, while faster superficial error dynamies in lower areas relax toward these stable top-down states (Murray et al., 2014; Chaudhuri et al., 2015; Bastos et al., 2012).
A.2 BIOLOGICAL PLAUSIBILITY OF FIXED FEED-FORWARD PREDICTIONS VIA HIERARCHICAL TIMESCALES AND WORKING MEMORY
The fixed feed-forward predictions our inference scheme are biologically plausible given known temporal hierarchies and active memory mechanisms in the brain. Higher cortical areas exhibit significantly longer intrinsic timescales than lower sensory areas, meaning that higher-level neural activity changes more slowly over time (Murray al., 2014; Kiebel et al., 2008). Consequently, during the brief timescales in which lower-level error neurons undergo rapid update cycles, the top- down signals from higher areas can be treated as essentially constant, providing stable context or boundary condition for lower-level processing. In effect, Meta-PCN's choice to hold each e fixed throughout inference mirrors the brain's 's own use of slowly varying higher-level activity constrain fast iterative dynamics in subordinate regions, in line with hierarchy of intrinsic timescales.
In addition to this passive timescale separation, the brain actively maintains persistent signals through recurrntt circuitry, which underlies working memory. Empirical and computational stud- ies demonstrate that cortical networks- - especially patterns even after an external stimulus is removed, via recurrent excitation and slow NMDA recep- mor-mediated currents that support an attractor state (Wang, 2001; 2021). This mechanism allows information (e.g., sensory prediction) to be held "online" for hundreds of milliseconds to seconds, rather than decaying immediately. Notably, such sustained memory states are not confined to a single high-level region but distributed across multiple interconnected areas (Mejías & Wang, 2022), implying that each layer in a hierarchy can uphold its prediction at its own timescale. Meta-PCN's feeward pass initialization and subsequent clamping of c can thus be interpreted as the network
frontal areas can sustain neural activity
16

---

Published as a conference paper at ICLR 2026
entering regime of active maintenance: the initial top-down predictions are kept stable by internal dynamics (analogous to a global attractor state) while errors are iteratively corrected. This corre- sponds to an active maintenance of predictions within the network, rather than a passive snapshot or cache of activations, In contrast to backpropagation 's practice of storing intermediate activations for later use procedure that violates the real-time, local processing constraint of biological circuits here the information is continuously present in the neural state itself. Such sustained representa- tions respect temporal locality and are energetically efficient, aligning with the brain's tendency to minimize surprise by keeping expected or predictable inputs in stable active state (Friston, 2010).
While these mechanisms provide biologically grounded interpretations, we acknowledge that Meta- PCN does not fully satisfy the strictest definitions of temporal locality. We view it as a step toward more temporally local predictive coding frameworks: demonstrates that deep PCNs can be stabi- lized through principled mechanisms, and future work can build on these stability properties while enforeing stricter temporal constraints.
A.3 BIOLOGICAL PLAUSIBILITY OF WEIGHT REGULARIZATION
Our weight regularizer rescales each weight matrix W multiplicatively according to its layer- wise variance, in order to keep the effective operator near unit spectral norm. This stabilizes the dynam- ies of the feedback term W D(h) across layers. Functionally, this mirrors divisive normalization, canonical neural computation in which neuronal responses are divided by pooled activity term to control gain across populations (Carandini & Heeger, 2012). Interpreting our variance-governed rescaling as a form of parameter- -epae gain control links it to convergent physiological mechanisms for divisive normalization observed across sensory systems and cortical areas (Carandini & Heeger, 2012). The same multiplicative scheme is consistent with global synaptic scaling, which multiplica- tively adjusts all excitatory synapses to maintain target firing rates, and also with theoretical local rules that explicitly normalize weight vectors (Turrigiano et al., 1998; Oja, 1982). finer scales, heterosynaptic plasticity and presynaptic normalization processes can constrain synaptic strength variance branchwise and along individual axonal arbors, yielding population-level effects equivalent to our matrix-level multiplicative regularization (Letellier et al., 2019; Hardingham et al., 2007).
B DISCUSSION
B.1 CONTRIBUTIONS
This work aims not only to document empirical instabilities in deep predictive coding networks (PCNs), but to explain them through a precise structural mechanism and derive a unified, theory- guided remedy. Our main contributions are summarized as follows:
DMFT-based analysis of deep PCN dynamics. We develop a dynamical mean-field, length-based framework for deep PCNs under Gaussian initialization, which analytically tracks the evolution of latent-state, PE, and weight-update lengths across both layer and inference iteration steps. This framework, which does not presuppose reed-forward-bseed initialization, already reveals the characteristic U-shaped PE profile and exponential scaling patterns in a mathematically tractable setting.
Theoretical reproduction and diagnosis of PE imbalance and EVPE. Within this frame- work we reproduce two core pathologies of deep PCNs- PE imbalance and EVPE and show that they arise structurally from the underlying dynamics. PE lengths decay geomet- rically away from boundary layers due to products of spectral factors, while EVPE appears as geometric growth or decay over inference iterations governed by multiplicative factors that depend weight variance and activation gains. This analysis isolates the root causes in terms of boundary conditions, hierarchical layer interactions, and the spectral statistics of the weight operators, going beyond explanations based solely discretization nu- merical heuristics.
Meta-PCN: a unified, theory-guided remedy. Guided by this diagnosis, we introduce Meta-PCN which combines (i) meta peeta-prediction-erroe objective that minimizes "predic- tion errors of prediction errors" by replacing the standard free-energy objective with a loss equal the linearized equilibrium residual, and (ii) variance-based normalization
17

---

Published as a conference paper at ICLR 2026
rule that controls effective spectral norms via random matrix motiivaedd scaling. Together, these components enforce contraction and scale separation during relaxation while preserv- ing fully local learning, thereby mitigating both PE imbalance and EVPE within single, principled framework rather than via separate engineering fixes.
Stable and scalable deep learning with local learning rules. Extensive experiments CIFAR-1O/100 and TinylmageNet with VGG and ResNer architectures demonstrate that Meta-PCN substantially improves inference stability, convergence speed, and classifica- tion performance. Across most configurations, Meta-PCN achieves large gains over con- ventional PCNs and performs competitively with backpropagation, all while maintaining the local learning rules and architectural constraints of PCNs.
B.2 LIMITATIONS
This study has several limitations. First, Meta-PCN's performance has reached complete parity with backpropagation. While this represents substantial progress achieved while maintaining bio- logical constraints, it suggests that performance gaps may still exist in some application domains. Second, the current analysis focuses primarily on computer vision tasks, requiring additional vali- dation of its generalization other domains. Third, comprehensive analysis is lacking regarding whether variance- nareance-based weight regularization is equally effective across all architectures and activa- tion functions. Fourth, for very deep architectures, when the number of inference steps T is smaller than the network depth L 1, prediction errors cannot fully propagate from the output to input boundary within a single inference episode, leading to performance degradation. This limitation may be alleviated in streaming continual settings where inference steps evolve with real-time data.
B.3 FUTURE WORK
This work can be extended in several directions. First, the effectiveness of Meta-PCN should be validated on larger-scale datasets such as ImageNet and in other domains such as natural language processing with text-based architectures and reinforcement learning. Second, more sophisticated weight regularization techniques and adaptive inference schemes could be developed to further im- prove stability across diverse architectures. Third, reinterpreting frozen predictions slowly updat- ing variables and latent states as fast-updating variables may enable Meta-PCN variants naturally adapted streaming settings and continuous-time formulations, which would also strengthen con- nections to biological real-time processing.
B.4 RELATIONSHIP TO PRECISION LEARNING
In the classical predictive coding framework, precision II, ET is defined the inverse covari- ance of latent activity and acts as a reliability weight on prediction errors, yielding the free energy F EE 8, TT,ōt- This scaling can partially compensate for PE imbalance across layers. Our work deliberately fixes I1, = I and focuses on why PE imbalance and EVPE arise in deep PCNs.
There is an important distinction between precision learning and our weight-variance control:
Precision, defined as the inverse covariance of latent activity, rescales 6, directly via II,
Our sirraacce-bssed normalization operates on the variance/spectrum of weights, control- ling of and ||il]2 to prevent amplification/attenuation of errors.
Our DMFT analysis shows that PE variance is primarily determined by weight statistics through multiplicative scaling | || as Te(o..) (( Thus, the most direct control for the pathology is weight variance rather than precision alone. When weight-induced error variance is normalized toward identity, the optimal precision II, Σ, becomes close I, making fixed identity precision appropriate. We view precision learning and weight- variance control as complementary strategies, with their integration future work.
18

---

Published as a conference paper at ICLR 2026
C DERIVATION OF LENGTH DYNAMICS
This section provides the complete mathematical foundation for the length dynamics framework introduced Section 3.1. We present the theoretical aspects how latent states evolve during the inference process, leveraging assumptions of linearity and focus on the statistical distribution Gaussian samples, which serves as the foundation for under- standing the Gaussian ensemble network's behavior under large-scale computations. We present rigorous analysis of interaction matrices of latents and bias and their dynamics, offering insights into the length dynamics of latent states len(z)), PEs len (,)), and weight updates (vec(AW7)) as defined in Section 3.1.
parameters. We
C.I ASSUMPTIONS AND LATENT STATE UPDATE RULE
Our primary goal is to track the changes in the length of the latent state during the inference step To perform this analysis, we adopt the following assumptions:
Gaussian Assumption We assume that the initial latent state at the inference step & = 0 is drawn i..d. = (O, 1). The learnable parameters, weight and bias, are drawn i.i.d. as wijd (O0, No and by. ~ of)
Linearity Assumption The correlation between variables may vary arbitrarily depending the nonlinearity of the activation function d, making it difficult to expand interaction analytically. There- fore, initially analyze the case where is linear function. For cases involving nonlinear tivation functions, empirical verification is performed to confirm the results in Appendix K. In the context of the linearity assumption, the forward and backward transformations defined fol- lows:
fi-1(Z(1) Wi-1Z + b
gi (Z(+1)) W7] Zi+1-
Dimensionality Assumption We assume that all layers share the same dimensionality. If the dimensions differ, the latent spaces must be transformed using matrices like M, resulting in non- generalizable cross-layer interactions.
With these assumptions, can expand the latent state update rule follows:
21+1 z Azi
2ं n ( 8f WT 8)41)
=z + n (z) - 2i) + W,T (zi+1 - 2(+1))
(3)
The update rule can be further simplified as:
2 1+1 (1 n)²t 17. (W1-12/ + bi-1)
+ + W7 2 1 - Wi (Wizi + bi)
= -VM-122f1 + nzj + vM, 2 + 1//111
-vM, b;
(4)
where M I W, v wo and == 1- 17(1 + oZ.)
C.2 THE DISTRIBUTION OF THE PRODUCT OF GAUSSIAN SAMPLES
Before delving into the dynamics length, given that our analysis involves the product of differ- ent forms of Gaussian samples, it is essential to review the distributional properties of products of o2
Gaussian samples. Let u, ~ (O, ) and U; (0, + The square of u, follows a chi-square distribution, while the product u; 02 follows normal product distribution. Our interest lies in
19

---

Published as a conference paper at ICLR 2026
understanding the distribution of the following inner product values
N
N
and
u W2 02
(5)
as N + oo. Applying the Central Limit Theorem (CLT) to these values, we obtain the following:
E[uf]
E
(0,
Var u.)
where Elu = Var(u;) N and Var(u,) E|u u u - Noo 20 N ), and equivalently.
E[u
3
du N
2 0h N7
As a result,
2 u N(1,
N
Similarly, applying the CLT to the cross pross-product yields:
E(uu edl
N
1),
(8)
Var(u; 22))
where Eluy- vs] since u, and o are independent, and Var(u: u.) σā og Hence, we obtain u (0, o Equivalently, for large N,
u ao NO0 )
and if oa o. this converges to of -V(0, 1.
We can conduct similar analysis for the distribution of vector lengths. Let u, NO o and (0, ).) In these cases, we want to understand the asymptotic distribution of the following
length terms:
(u
and (nx- v1) Σ u N
1
1 N
u US:
(10)
as N co. Note that the variance of the Gaussian distribution in the length calculation is not divided by N in contrast to the inner product version. Instead, the length includes division by N. By applying the CLT, similar to the inner product case, we have:
(ut) - of N(1, N
(11)
Using this result, we can apply it to the cases of interest.
Lengths In the case of vector-vector multiplication, consider vectors 21, 2z, and by, where {l.... L 1}. Each of these vectors is assumed to be sampled from Gaussian distribution, i.e., each element is drawn from (0, 1). The length defined by the relationship between these vectors, as N oc, follows:
(u?) - (1,
(12)
while the cross-product between different vectors converges to:
(m (u.-v)) NO0 N
(13)
Consequently, the self-product (length) converges to 1, while the product with a different vector 1v Au, where each element of A,
converges to 0 as N - co. Finally, consider the length Ay, is drawn from (o, }). and each element of u and v follows A/ 1). The transformed vector (Au); ~ (10 1). Therefore, v (A(Au) ~ No 1). Thus, the length follows:
1 ~~ N(0, N2
(14)
20

---

Published as a conference paper at ICLR 2026
Matrix-Matrix Multiplication In the case of matrix- matiixx multiplication, consider C. A A, where each element of A, i.e., Ay, is drawn from (/(,, x). The diagonal entries of C. Cis follow (1, )) The off-diagonal entries Cij. where i * j. follow NN ) Hence, C' approaches the identity matrix N c0. For the product of two matrices D AB, where both Aij and Bij sampled from (0, N ), the resulting matrix Dij shares the same distribution Aig and Bij-
C.3 INTERACTION MATRICES
For the analysis of length dynamics, we define several key variables as follows.
Latent Self-Interaction Let [++, = M for 1 S. 1 < L, where M4 M1+8-M+ ... M, is the product of the series of matrices (as introduced in Section 3.1). By definition, we can observe that P is symmetric, meaning that Pf P/+,,1- The length of the latent state at layer and time step t, pi. can be represented as the diagonal elements of Pt,
of (1))) {
2 2 Pt, With this definition, the matrix P contains
the length information and interactions between latent states at different layers at the inference step Since the input and output are fixed during the inference phase as z z/+1 zf and z/+1 z the interaction terms with the indices 1 and L are constants as P1 = PL.L = 1 and PI PL.1 = 0.
Similarly, at = 0, pa I.
Bias-Latent State Interaction Let bias-latent state interaction - 3tr bi Mi-i- be bilinear term of interaction between the bias and latent states at layers and - h at inference Since the bias, the
step & for S 1,1 - k < L. Likewise, let B = 2 Mi- input (zz) and output (zz) are fixed during the inference phase, the interaction terms between these independent components, B and B.,,, are also fixed at O. At 0, B = 0.
Bias Self-Interaction The term - represents t b, Mit k+1 b 0 for 1 S 1,1 - k < L. Since the bias terms are sampled from A (00, of) and fixed during the inference phase, I = oî. is constant matrix by the properties introduced in Appendix C.2.
C.4 DYNAMICS OF INTERACTION MATRIX
We derive the update rule for P using the definition of the interaction and the latent update rule in Equation 4. For element of Pi, where 1 h > and I < L, the update equation can be described as follows:
p++1 11-k
111T MI
N "
(N: z + v. M M1-12i + U. Mt 2/41
+n b - v. M, bi) x Mi-1atk
x K zi + v. Mi-k-1 1z
MiLzi-144 + b
Mi
We want to expand this equation fully, showing all combinations of terms in the product. First, we identify the components the vectors involved in the equation. The expression consists a sum of transposed vectors, multiplied by matrix M and then multiplied by another sum of vectors. The components of the first sum of vectors are
u1 = V. Mi- 2
U2 = K zú, u1 = 1. -M, 2i+11
b and
un = -v M, b.
21

---

Published as a conference paper at ICLR 2026
The components of the second sum of vectors
V1 = V. M--- 18---11/--11- V2 v2= K =K.2/k
V3 = v M1 V4 = bi-k-1,
and - Mi bi-k-
We can rewrite the original equation using the components defined:
1 N
us + u1 us) MI-is 1
11-k
(u + U2
V2 V3 +v+
We compute all possible products u? M for i 1 to 5.
Terms involving uj:
1n MI v3 (xi-1) MI Mi MI-
xi-x-1 12 Pt. Pi-1.-k-1
u. M- un (z) 1) M,T -
= kv Pl- 1
u, MM-II--Vs = v2 (zf-1) T
2i-141 = v22 1|-111-k41
u, MI-1--AV1 = un (zi_1) M, -
vn B k11-1
T MM -n² (x1_1)
Mi Mi_ Mi
b -v² Bi-K+111-1
Terms involving ug:
m MM-1--VV kV M
= =KV P
u. M = 162 (2iz)
M1-1-1kZi/6
= -
m MM- = =KV (zi) T M
= -KV. P(,-k+1
u2 Mi-1--VV =k]] (z1) Mi--s--bi---1
=KT] Bi-ki
u MM-11-VVs -KY (zi)
M bi-k
= B
22

---

Published as a conference paper at ICLR 2026
Terms involving uy:
Ug MI-1- = =v2 (z/+1) 1.M
zi-b-1 = v22 Pi+1.1--11
ug Mi-1-- = ux (4i+2) T M4
= kv Pi+1--R
11 MM-I--VV = v2 (zi+1) M
2-211 v²2
Pi+1,1-+11
113 MI AV4 = vn (zi+1) T M
=vn] Bi-k,1t1
Ug M---N-VV = -2² (zi+1) 1)
M
b -1? Bi
Terms involving u4:
41 M 1 my (b'-1)² T Mi-Iil- -- M
x 1, = vp7 Bi k-
41 MM mp (b--1) T M 1 --k
))
nn Bi, 1
u1 Mi m1 (b'
1:1 M M
M 2 -
vn- Bi -it
u - 1:- v4 = 1/² (b'-1)² M - 1-k-1
1/² ./t - 1-1
u M -mp (b'-1)² 11-1:lkk - bi-
-mn . -
Terms involving uy:
u5 -11- v1 = -2² (b') M M
z' -h = -1² Bi B
T MM' -1d- = -ux (b') T M' 1-k
u3
- B
us T - -n² (b') M M 1:1- M 1-
- -v² Bj 1,1
ug - = -vn (b') T MM - 1:d bb R-
= -un -
mg - = 1² (bb)² T
- b 1-
2² h
23

---

Published as a conference paper at ICLR 2026
By systematically breaking down the original equation into its constituent components and comput-
ing all possible interactions between them, we have fully expanded the expression:
Pit = 2². Pf-1.1 + Kv P1 v² P/ PI-1--R41
+ +u Bi-k,1-1 -v2
v22 B -1-1-11-1
NV. Pt k + - nv Pi Pi,
+K7- Bi k2 - KV KV BBB.Bikt11
+ 1²2 P -1 v² P 1+1.1-4+1 + KU P(+1A- + un BJ_
22. Bl B1-k+1,141
vn Bii---1 + KT) B Bi k + un
Bi,--411
+1p² ..t-1,1---1 1m 1-1-l
- -v² -BB4411-k-11 KV Bi
-²² BB+111-441 - mn g - + u² p
(15)
On the other hand, when updating Pf it is important to account for the cases where / or / - k are 1 or L. since the values of the latent states are fixed in such cases. For instance, the update equation for the interaction with the input layer, pt+1 can be expressed follows:
P/ = L
1
M 1-1-1:1 1:1
N
N (n.zhe"
1
M1-1il + v. 1 2
+ v. wo.z+ti +1, M 1 21:1)
= P/ +v V- Pl v. P/+111
(16)
Furthermore, by considering the symmetry of p. have Pf = Pf,1- Similarly, the update equation for the interaction with the output layer, pt+1 L.L-k= is as follows:
p++1 1 zh,t 11--1:b-R, - katt1 L.L-k
N
N (k.ztin M 1:1, Lik
1
+ u.ggtt T M 11- 1: k.
+ v-t L M 1:b - k+1 (--tt
k Pt P + +v Pt P V. Pt. Pi.
(17)
Moreover, PL,L-k PL-KLL.
We now aim to express the above update rules, which involve many combination terms, in a more concise matrix and vector calculation form. Let us carefully examine the structure of the update equations for both the latent states and The update equation for z can be divided into two parts. The first part is the sum of the element-wise product the latent states [2]-1, Z1, 2++]] and the coefficients C2 = [p, K, v) The second part is the sum of the element-wise product of the bias terms [b,-1, by] and the coefficients с = [n. -v] The update equation for p. which is derived from the update equation of z, can be seen the outer product of the latent updates of layer and
another layer k. The coefficients are fixed, and the values of and / - correspond to the indices in the P matrix. Utilizing this,
rewrite the update rule from Equations 15 to 17 in matrix
24

---

Published as a conference paper at ICLR 2026
form as follows:
p = Pt G.
+ +c, Bj -1----------1111
+ c. B -
+ +cb [
p T
1. 1,1 C, P 1111141,11 and
Pt-1T
LL C2 1141.
(18)
for 1 < <1,1-k
The update rules for B represent the evolution the interaction between the latent states z and the bias terms b.
B+1 141,1K
1
b' IT
Mii-k 2 1-1,41
N
Nbb b' M k. g/h + v. M1- --
1
+ v. M z -
+1 -b' bi-k-1 M k- bl
1
N (Ix b, MMa-
+v. by M - M1 1
+v.b, bj MiL M M 2 k+1
+7- by MM bi-k-1
V. bj Mis M bi
1
N (1s bj MMa 22
+V- b, M 1
+ b, M -11
+1 bj Mi--bb-k--
- VV bj M bl-k)
= K Bf41.1- + V. B B +v. B411--441
+ Tia-k11-u u. -..
(19)
We can simplify the update rule for B follows:
B
Bi,, -- 1 + 1
(20)
Note that since I' is symmetric matrix, swapping the column and row indices in the update equation for B does not alter the result.
C.5 DYNAMICS LENGTII
Lengths of the Latent States As mentioned earlier, the diagonal elements p represent the lengths of the latent states. That is, r/ = Pl--
25

---

W2-12/-1 + b1-1. By substituting the prediction term, of can be further expanded as Published as a conference paper at ICLR 2026
Lengths of the Prediction Errors
Let the length of the PE at layer / be denoted by qf. We can
express it as follows:
of ((6t,)²)
N 10% 12
N (z) 2 [z] 21)
(21)
where 2/ follows:
1
N z Fw MM-12i_1 - -b--1)
2 Tw M1-12/-1 -((-1)
1 ( IT -w ai-MM]- - bi_1
N %
2 Ow MM- 12/12/ bb-1)
1 (z!1" (z - Ow -Mi-lz/ - -b'-1)
N
Tw z 1-1,t M M1-11 (z - MM 1-1t -b bb-1)
b (z) ow M M1-12/-1 - -bi-1)
1 IT z om ai Mi Mi- 2f bi-1
N 21
2 LIM,I1'
0
+ 2
IT MI M
+ Gw M bi-1
b 2j + ow bT_ MM + +b/
Pf Tm f1,-11 - Bi,
Tw Pt Pf11 + pn pf_ P(-1,1-1 Tu B
Bi, + ow Bí,-1 7-11-1
(22)
The above equation can be simplified as:
qf = c.y P Pi-Lu---s9
+ T1-12-11
(23)
where the coefficient c2 = [-ow,
ow. 1]
Lengths of Weight Updates The length of the weight updates at layer is denoted by rf. and is defined as:
1
N2 IAW!
(24)
where AW, = o4 with 6:z representing the error signal at the next layer and zi being the signal at the current layer.
Before proceeding further, prove simple lemma for the Frobenius norm:
Lemma (Horn & Johnson, 2012): |xy [|: = ||x| 2 ||yl? where x and y are vectors.
Proof:
|xxy 112 Σ (xy T)2, = Σ (xx,))) Σx Σ y
in
2
|x||2
11x112
(25)
26

---

Published as a conference paper at ICLR 2026
Using the above lemma for the Frobenius norm, simplify rt as:
1
r
N2 18 2
1
N2 |
9441
(26)
Since * |Abill = t I| the length of the bias update is equivalent to of and is therefore omit- ted.
D META-PCN TRAINING ALGORITHM
Algorithm presents the complete training procedure for Meta-PCN, The algorithm consists of three main phases per mini-batch: feed-forward initialization, inference, and learning, followed by weight normalization.
Feed-Forward Initialization (Lines 4-9). The input is set the first layer (Line 4), and feed- forward predictions C₁ are computed and frozen for subsequent use during inference (Lines 5-7). Latent states initialized to these predictions. The output prediction ey computed (Line 8), and the output layer set to the target (Line 9).
Inference Phase (Lines 10-18). For iterations, the meta-prediction errors 8:00 z (t) ci
are computed (Lines 11-13). The meta-PE objective I is then evaluated (Line 14), where [+] is
(t)
treated as a constant (stop gradient) to serve as stabilized error signal. Latent states are updated by gradient descent on J (Lines 15-17).
Learning Phase (Lines 19-20). After inference converges, the parameter loss L()) is computed based on the final meta-prediction errors 8(()) (Line 19). Parameters are updated using the optimizer (Line 20).
Weight Normalization (Lines 21-24). After each parameter update, variance-based weight nor- malization is applied to maintain spectral control (Lines 21-24).
E EXPERIMENTAL SETUP
Length Dynamies Analysis. The simulations described in Section 3 analyzed the length dynamics of latent states and PEs during the inference process in a random PCN ensemble. The dataset consisted of samples from random unit Gaussian distribution ((z;, yi) ~ N/(0, I)). The dataset contained 128 samples processed in single batch. The number of inference steps (7) was mainly set to 2000 to track the iterative changes length dynamics. The inference rate was set at 0.05.
The model consisted of 30 layers effectively show the exponential growth in PCNs. The latent dimension was set to 100.
Baseline Comparisons. We compare three approaches under identical experimental conditions: (1) Standard backpropagation (BP). (2) Conventional PCN with only feed-forward initialization, and (3) The complete Meta-PCN framework. All methods use the same network architectures, optimization settings, and training procedures, differing only in their learning algorithms.
Meta-PCN Components. Our framework is built on two core components: (1) Meta-PE objec- tive with feed-forward initialization and (2) Variance-based weight regularization. In addition, all experiments include blocked sweep updates (Gauss-Seidel-Like Seidel-like alternating layer updates), theoretically motivated technique whose empirical contribution is analyzed in the ablation study (Appendix l).
Training Configuration. We train all models for 50 epochs using the AdamW (Loshchilov & Hutter, 2019) optimizer with learning rate of 0.0001 and weight decay of 0.0005. The batch size
27

---

Published as a conference paper at ICLR 2026
Algorithm Meta-PCN Training
Input:
Dataset D {(x, y)). network with layers, parameters (Wi, bi). inference steps T' inference
rate 77, learning rate a, epochs
Output: Trained parameters 0
1. Initialize randomly
2. for epoch = to do
3. for each mini-hatch (x.y) do
4. Z1 + x
Di Input
5.
for 2 L do
6.
Ci + @ @(W1-171-1 + b1-1); (0) t
Frozen prediction
7.
end for
8.
CL + @(WL-
bL-1)
Output prediction
9.
ZL y
Target
10.
for f = to T - do
b Inference phase
11.
for | = 2 do
12.
8 + 2
13.
end for
14.
Compute -= D1
1-11 |(0 9: (6 11
& stop gradient on 8,41
15.
for
2
do
16.
2
(++1) - (8) 1 V
17.
end for
18.
end for
19.
Compute [()) -111 12
b Learning phase
20.
0 - 0 - aV)
for 110 do
Weight normalization
22.
Ow - (ar(Wi)
23.
Wi - Ww/(lm/ vn) o.)
24.
end for
25.
end for
26. for
27. return 0
is set to 128 across all experiments to ensure fair comparison. For PC-specific parameters, we use an inference rate (1)) of 0.2 and perform 20 inference steps (I") per learning step. For classification tasks, we replace the squared error at the output layer with a cross-entropy loss.
Implementation Details. All experiments were conducted using the PyTorch (Paszke et al., 2017) framework on CUDA-enabled GPUs rented from Vast.ai. Each experiment is repeated times with different random seeds to ensure statistical reliability.
Comparison with Prior PCN Configurations. Our PCN bascline differs from the configuration reported in Pinchetti et al. (2024) in three key aspects: (i) network architecture (standard VGG with ReLU vs. custom VGG with GELU), (ii) inference rate (1 0.05 vs. 7 0.01). and (iii) weight decay (5 x 1 vs. 0). controlled study varying these factors on CIFAR-10 shows that accuracy can range from below 20% to above 70% depending on these choices, with high variance across runs (std up to 8.99%). indicating unstable training dynamics inherent to PCNs. Our primary focus is stability and learning dynamics under controlled, standardized setup. rather than on maximizing absolute baseline performance through method-specce speciifc hyperparameter tuning. A detailed comparison provided in Table 1.
28

---

Published as a conference paper at ICLR 2026
Table 1: Controlled comparison of PCN configurations on CIFAR-IO. "pex vge7" refers our implementation of the VGG7-style architecture from Pinchetti et al. (2024); "ours vgg7" is the standard VGG7 used our main experiments. Results are mean std over three runs.
Architecture
Weight Decay
Test Acc. (%)
pex vgg7
0.01
0.0
64.88 2.48
ours 7
0.01
0.0
40.48 + 2.44
pex vgg7
0.01
5 X 10
63.70 - 4.50
ours vgg?
0.01
5 10
4
pex vgg7
0.05
0.0
73.37 8.99
ours vgg7
0.05
0.0
16.33
pex vgg7
0.05
5 10-4
+
ours vgg7
0.05
5 X 10-4
17.85
F PARAMETER LEARNING IN META-PCN
Separation Inference and Parameter Objectives. While the meta-PE objective J governs the inference dynamics over latent states, parameter updates follow the conventional PC loss:
(( =
2 (T) fi- (2.0)
fz-1(2
(27)
We do not directly use the meta-PE objective for parameter updates for the following reasons. The first term ô in the meta-PE loss is subject to prediction freezing and thus does not directly involve parameters. To backpropagate through gi the second term, one would need compute second- order derivatives, which we explicitly avoid due to complexity and computational cost. Instead, the meta-objective reshapes the inference dynamics so that the resulting latent states and error signals are better aligned with the desired theoretical properties.
Empirical Convergence. Figure 8 shows that the training loss L()) decreases smoothly during training, demonstrating stable convergence without pathological behavior.
2.0
80
1.5
Loss
loss
60
1.0
accuracy
48
20
20
30
40
Epoch
Figure 8: Training loss curves for Meta-PCN CIFAR-I0 with VGG-13 over 5 runs.
G COMPUTATIONAL OVERHEAD ANALYSIS
We empirically measured the computational overhead of Meta-PCN compared conventional PCN. Using a representative configuration (batch size 1, = 20 inference steps, averaged over 100 runs), obtained:
Conventional PCN: 127.930 + 0.637 per inference step
Meta-PCN: 129.821 + 0.622 per inference step
This corresponds relative overhead of approximately 1.48% per inference step.
29

---

Published as a conference paper at ICLR 2026
Table CIFAR-IO Top-1 accuracy (%): mean over n=5 runs for each method, with p-values from two-sided Mann- W Whitney U tests. Meta-PCN vs BP; P2: Meta-PCN vs PCN. Bold denotes the best mean accuracy per architecture. Italic p-values indicate statistical significance (p I 0.05).
Architecture
BP
PCN
Meta-PCN
P1
P2
VGG-5
03.91
25.78+
85.64-0.22
0,0079
0.0079
VGG-7
88.0910.47
20.0113.44
88.99_0.40
0.0317
0.0079
VGG-9
84.8410.28
19.43114.8
85.6910.29
0.0079
0.0079
VGG-11
84.5010.50
17.96.13.23
8..04_1..23
0.0556
0.0079
VGG-13
87.85=0.50
12.24+1.50
89.53=0.47
0,0079
0,0079
ResNet-18
02.54=0.35
54.53+0079
82.58=0012
1.0000
0.0079
Table 3: CIFAR-100 Top-1 accuracy (%). Same format as Table 2.
Architecture
BP
PCN
Meta-PCN
P2
VGG-5
51.73+0.24
4.14
55.18-10.23
0.0079
0.0079
VGG-7
51.52.10.22
7.01
55.49.1.499
0,0079
0.0079
VGG-9
45.22.10.34
3.4811122
50.09
0,0079
0.0079
VGG-11
51.13-1.88
1.32.10.76
52.21110.48
0.1508
0.0079
VGG-13
54.19=2.03
1.26+:0.17
59.04
0.0079
0.0079
ResNet-18
53.16:20.43
38.66::0.49
03.14
0.8413
0.0079
Component-wise Analysis.
The overhead attributable to the two core components:
The gradients of the meta-PE objective with respect to la- tent states can efficiently computed a single backward pass using automatic differen- tiation. The overhead is negligible compared to the overall inference cost.
Meta-objective computation:
Weight normalization: Computing the variance of each weight tensor is substantially cheaper than spectral norm computation and benefits from GPU parallel reduction.
Additionally, blocked sweep updates are included in the Meta-PCN configuration but introduce no measurable overhead, our implementation follows existing PC frameworks that do not parallelize updates across modules.
H STATISTICAL SIGNIFICANCE TESTING
We performed Mann-Wai Whitney U tests to verify whether Meta-PCN's performance improvements are statistically significant. This non-parametric test is suitable for comparing performance values it does not assume normality of accuracy distributions. All comparisons use n= runs per method.
H.1 MAIN METHOD COMPARISON: BP VS PCN vs META-PCN
The statistical analysis reveals several key findings across all datasets. Meta-PCN demonstrates statistically significant improvements over conventional PCNs in all architecture and dataset combi- nations. Against backpropagation, Meta-PCN shows statistically significant improvements in most VGG architectures, with the performance gap generally decreasing as architecture complexity in- creases. Notably, ResNet-18 shows minimal or non-significant differences compared to backpropa- gation.
The consistent pattern across Top-1 and Top-5 metrics on CIFAR-100 and TinyImageNet further validates Meta-PCN's robustness. The larger improvements observed in conventional PCN compar- isons highlight the severity of the pathologies addressed by our framework, while the competitive performance against backpropagation supports the practical viability of locally learned algorithms.
The comprehensive statistical analysis provides strong evidence that Meta-PCN's per- formance improvements represent systematic rather than accidental gains. The framework success- fully bridges the gap between local learning and competitive task performance, positioning predic- tive coding promising alternative backpropagationn-hase learning.
Conclusion:
30

---

Published as a conference paper at ICLR 2026
Table CIFAR-100 Top-5 accuracy (%).
Same format as Table 2.
Architecture
BP
PCN
Meta-PCN
P1
VGG-5
79.71-0:226
14.8311.11
81.94110.24
0,0079
0.0079
VGG-7
78.43110.74
22.18.12.05
81.92:10.31
0.0079
0.0079
VGG-9
75.71=0.28
13.1843.84
79.15=0.81
0.0119
0.0079
VGG-11
78. 08.68=40.49
6.57+2.04
79.29=:0.44
0.0952
0.0079
VGG-13
81.06=18.81
5.70
85.12:+1.49
0,0159
0,0079
ResNet-18
09.91
68.29410.44
80.36:10.17
0.0159
0.0079
Table 5: TinylmageNet Top-1 accuracy (%) Same format Table 2.
Architecture
BP
PCN
Meta-PCN
P1
VGG-5
35.11
1.31
38.91110422
0,0079
0.0119
VGG-7
39.35_10.33
0.90110.05
42.28.10.51
0,0079
0.0079
VGG-9
35.99-1.559
0.77.10.30
39.85
0.0079
0.0079
VGG-11
38.06=00.42
0.72+t0.28
39.18=0..35
0.0079
0.0079
VGG-13
45.38=0.81
3,49+2.73
47.43-10.30
0.0079
0.0119
ResNet-18
43.80=0032
17.73#0.79
44.19::0441
0.1508
0,0079
I ABLATION STUDY
1.1 ABLATION STUDY ON META- COMPONENTS
We conducted a systematic ablation study to quantitatively evaluate the contribution of each compo- nent in the Meta-PCN, framework. Table 7 presents results from training the CIFAR-10 dataset using the VGG- architecture for 50 epochs.
1.2 COMPONENT ANALYSIS
The ablation results reveal that each Meta-PCN component contributes differently. Removing the meta-PF: objective causes catastrophic performance degradation, dropping accuracy from 99.53% to 10.01 % (79.52% decrease). This dramatic decline suggests that the standard free-energy objective suffers from severe pathologies, such PE imbalance and EVPE, which prevent effective learning.
Removing weight regularization (normalization) leads to a 1.28% decrease (88.25% vs 89.53%) ), with statistical significance (p 0.0079). This moderate but significant degradation indicates that variance control effectively addresses EVPE and contributes to training stability.
Removing blocked sweep updates shows minimal impact, with only 0.20% decrease (89.33% vs 89.53%) and no statistical significance (p 1.0000). While the convergence improvement from Causs-Seidef-like updates appears modest in this configuration, the benefit may become pro- nounced deeper architectures.
1.3 STATISTICAL SIGNIFICANCE AND COMPONENT RANKING
The Mann-Whitng Whitney test results establish a clear hierarchy component importance. The meta-PE objective is the most critical component, its removal renders the model non-functional. Weight regularization provides statistically significant but moderate improvements, while blocked sweep updates show non-significant effects under current experimental conditions.
Component Criticality Ranking:
Meta-PE Objective (most critical): Essential for basic functionality (79.52% decrease upon removal)
2. Weight Regularization (moderately critical): Statistically significant (1.28% decrease upon moval, p 0.0079)
3. Blocked Sweep GS (least critical): Non-significant (0.20% decrease upon removal, p 1.0000)
31

---

Published as a conference paper at ICLR 2026
Table 6: TinylmageNet Top-5 accuracy (%).
Same format Table 2.
Architecture
BP
PCN
Meta-PCN
P1
VGG-5
60.71-10449
5.22.10.15
65.08.10.43
0,0079
0.0079
VGG-7
64.67-10.40
3.35.10.28
67.79.10.91
0.0079
0.0119
VGG-9
62.98=0.31
3.29:
66.58=10.99
0.0079
0.0079
VGG-11
65.79=0.73
2.95+0.557
66.24-:0.55
0,3095
0.0079
VGG-13
71.24=0.44
8.83+5579
71362::0.36
0.3095
0.0119
ResNet-18
69.55
38.97-1.388
70.02:10.15
0,0159
0.0079
Table 7: Ablation study results showing the contribution of each Meta-PCN component. Perfor-
mance is measured on CIFAR-10 using VGG-13 architecture across 5 independent runs, Statistical significance evaluated using Mann-Whimng Whitney U tests against the full Meta-PCN framework.
Method
Accuracy (%)
p-value
Meta-PCN
89.53 0.47
Blocked Sweep GS
89.33 + 0.56
1.0000
Normalization
88.25 + 0.32
0.0079
Meta-PE Objective
10.01 = 0.02
0.0167
PCN (conventional)
10.01 + 0.02
0.0079
Prediction Freezing
16.31 + 1.00
0.0159
Normalization
10.16 0.33
0.0095
These results demonstrate that the meta-PE objective is fundamental to Meta-PCN's success, while weight regularization provides important stability benefits. The blocked sweep component, while theoretically motivated (Appendix L.5), showed statistically significant effect in our experiments. Accordingly, the main text focuses on the two core components thhesss--hhhhhheeee meta-PE, objective and weight regularization- the primary contributions. Nevertheless, all reported Meta-PCN results include blocked sweep updates to faithfully reflect the complete experimental configuration.
1.4 EFFECT OF ADDING INDIVIDUAL COMPONENTS TO CONVENTIONAL PCN
The lower half of Table 7 examines the effect of adding individual Meta-PCN components to con- ventional PCN. There are conceptually distinct uses of rerd-forward predictions in our work:
1. Latent initialization via feed-forward prediction. This standard practice in the PC literature: feed-forward predictions initialize latent states z. All variants in our Iilation--onventioon PC, Meta-PCN, and ablations-use this initialization.
2. Freezing feed-forward predictions during inference. This our proposal: predictions treated as fixed values throughout inference, which is built into the meta-objective definition. Thus, freezing inherently tied to using the meta-objective.
The Prediction Freezing" row shows the effect of freezing predictions without the meta-objective, yielding the loss J(ō) = 2 Σドー |z - c; cıll.. Since z; initialized cı this setup produces min- imal updates during inference. This variant achieved 16.31 % accuracy, slightly better than conven- tional PCN (10.01%) %) but far below Meta-PCN (89.53%).
The "+ Normalization" row (10.16%) shows that weight regularization alone cannot resolve the structural issues. These results confirm that the meta-objective and normalization act complemen- tarily: the former reshapes the objective target the equilibrium residual, while the latter controls the spectral factors that drive EVPE.
I.5 BLOCKED SWEEP UPDATES FOR PC INFERENCE
Convergence Analysis through Classical Iterative Methods. To understand the convergence limitations of PC inference, frame the free-energy minimization problem in inference within the
32

---

Published as a conference paper at ICLR 2026
context of classical iterative methods. PC inference can be viewed solving the nonlinear stationar- ity system F(z) = Vz = subject to boundary conditions 2 = X and 27 = The standard inference procedure follows the fixed-point iteration z (z,(), where equilibrium point z² satisfies z" = G²2zz)) ) if and only if V xz") O.
This formulation reveals that the PCN inference performs simultaneous updates of all latent states, making it directly analogous to the Jacobi method in classical iterative solvers (Saad, 2003; Golub & Loan, 2013). This analogy provides crucial insights into the inherent limitations of PC inference, as the Jacobi method is well-known for its stringent convergence requirements and inefficient infor- mation propagation across network depth.
For linear systems Au b. Jacobi convergence typically requires both A and 2D A to be sym- metric positive definite (SPD), where D denotes the diagonal of In contrast, the Gauss-Seidel method requires only that A be SPD (see, e.g., Varga, 2009; Saad, 2003). More importantly, for con- sistently ordered SPD matrices, the celebrated convergence rate relationship p(GS) = p(Jacobi)² demonstrates that Gauss-Seidel achieves asymptotically quadratic convergence improvement over Jacobi (Young, 1971; Saad, 2003; Varga, 2009), where p(-) denotes the spectral radius of the itera- tion matrix.
The practical implications for deep PCNs are significant. Because the standard PC update behaves analogously to the Jacobi method, its convergence in deep networks suffers from inherent inefficien- cies: information propagates exclusively through simultaneous, layer-wise exchanges, resulting in slow convergence and potential divergence on ill-conditioned problem instances.
PC-Compatible Blocked Sweep Method. Standard PC inference employs simultaneous (Jacobi- like) updates of the form: z 2=+ = z' VzF(zz), which updates all layers using neighbor infor- mation from iteration t. While this approach offers high parallelizability, it propagates information slowly across network depth, contributing to the convergence inefficiencies.
In contrast, Gauss-Seidel (GS)-type schemes achieve faster convergence by reusing the most cently computed values within the same iteration, though at the cost of reduced parallelism. Our blocked sweep update strategy offers a principled compromise that preserves the locality and mod- ularity inherent to predictive coding, while leveraging newly computed neighbor states to accelerate information propagation.
Theoretical Foundation and Convergence Analysis. The blocked sweep method possesses solid theoretical foundation rooted in classical iterative solver theory. The blocked sweep update is mathematically equivalent to preconditioned gradient step 2 z ---n) r M - VzF(zz), where the preconditioner M = D+ L has a block lower-trianullar structure (with D representing the diagonal blocks and L the strictly lower block components). This corresponds precisely Gauss-Seidel iteration applied to the linearized system.
The convergence advantage is quantified through spectral analysis of the resulting iteration ma- trix TBS = - 7 H.. For consistently ordered SPD problems, the classical result ((T = (TT.ae))² (Young. 1971; Saad, 2003; Varga, 2009) demonstrates that blocked sweeps achieve asymp- totically quadratic convergence acceleration compared to simultaneous Jacobi-like updates. The method thus provides theoretically motivated approach to address the depth-induced convergence slowdown, though its empirical impact may vary across configurations (see Section I).
J WEIGHT NORMALIZATION FOR BACKPROPAGATION
We examined whether our variance- based normalization benefits backpropagation (BP). On CIFAR- 10 with VGG-13, standard BP achieves 87.85 + 0.50% accuracy, while BP with our normalization achieves 89.07 + 0.46%. Although normalization provides modest improvement for BP, the re- sulting accuracy remains below Meta-PCN (89.53 0.47%), indicating that the full Meta-PCN framework offers benefits beyond weight regularization alone.
K LENGTH DYNAMICS WITH NONLINEAR ACTIVATIONS
Figure 9 explores the dynamics of latent state lengths (p/ PE lengths (g'*), and weight update lengths (fr--) in nonlinear PCNs across different activation functions. The analysis focuses on com- mon nonlinearity types such as tanh, ReLU, SELU, and S LU, each applied to layers with varying
33

---

Published as a conference paper at ICLR 2026
tanh
ReLU
SELU
SILU
dt
ton
toi
10a
11.0
1=
10h
4p- -5.4
1oiz
10t
107
10a
10
10-
25
50
75
100
23
50
75
160
20
50
75
100
25
30
73
100
p
Itoration
Iteration
Iteration
lteration
Lanethgla
1011
10a
1013
10-3
Lot
10-3.
02
10n-
Layer
Layer 2
Layer
Lager I
Longthg 2
10
10-4
10
tol4
1003
102
Licration d
Iterationn 5
Itesation
Dicnntion
Langit yt?
1021
10P
16t1
1017
1049,
1013
10:3.
02
10h
10%-
101
10"-
10
20
20
Layer
Layer
Layer
Layer
Langh da
1otz
M
10th
10t
3011.
30
10
10'
104
10 0
25
50
75
100
25
50
70
100
25
50
75
100
25
50
76
100
Ticration
Ticration
Diennikon
Langhyis
1012
10
1015.
10
01 =
102
1n7
10a
16"
10
20
101
20
10
20
10
Layer
Jryor
Taayer
Tayer
Figure 9: The dynamics of ph. g/d and ph,t and their layer-wise results for nonlinear PCNs. In all subfigures, the results are shown for the cases of ow E {0.185, 1.0, 5.4] with different colors, and 0.1. Settings not mentioned or indicated are identical to those in Figure 2. Each column represents the applied nonlinear function. The odd rows are the dynamics of p, q. and r, respectively (1 27). The even
the layer-wist wise distribution of p. q, and r, respectively (T = 20).
weight variances (. The results show that the dynamics for q. and are highly sensitive to Twn even under nonlinear activations. The odd rows depict the temporal evolution of p. q, and r at layer / = 27, while the even rows display the layer-wise distribution of these values at the T = 20 inference step. These subfigures illustrate two key phenomena that occur regardless of the applied nonlinear activation function:
1.
In the odd rows, we observe that even with nonlinearity applied, p, and r exhibit exponential growth near the output layer when a is large (e.g., T. = 5.4). This suggests that while nonlinear activations are typically expected to provide some degree of constraint on the predicted latent state dynamics by squashing the outputs (e.g., tanh), the latent state length growth persists for larger ow. This pattern holds across all activation functions examined, indicating that nonlinearity alone insufficient counteract the destabilizing effects of high weight variance.
2. The even rows reveal that these exponential growth patterns can emerge early in the inference phase, even at = 20, particularly deeper layers. The layer-wise distributions p, q. and r show that the effects of large o,, extend throughout the network, with PEs (q) and weight updates (r) becoming increasingly concentrated toward the output layer. This observation underscores a key challenge in training deep PCNs with nonlinearity. While early inference stages may seem stable, instability can rapidly accumulate in deeper layers due to the interplay between nonlin- earity and large weight variances.
34

---

Published as a conference paper at ICLR 2026
Importantly, this analysis highlights the need for regularization strategies, even in networks with nonlinear activations. The exponential growth seen here mirrors the behavior in linear PCNs, sug- gesting that length regularization and weight variance control critical to preventing runaway dynamics in both linear and nonlinear architectures. Regularization techniques, such as those in- troduced in Meta-PCN, become essential for maintaining stabilit,, partieularly when nonlinearity alone is insufficient to prevent the excessive growth of latent states and PEs.
L ADDITIONAL. RESULTS
7) = 0.45
1) = 0.2
?) = 0.05
?) = 0.01
Lincar
Length giz
1022
Length gitt
100
Lon
1002
1017
104
102
10
107
23
50
75
100
25
30
75
100
25
50
75
130
25
50
75
160
Ieersion
Ttemsionn
Tteration
Iseration
tanh
Longhype g
1034
Lougth g/t
10a
Lengthgt
1012
10*
Lenghpie
107
10*-
100
103
10-
100
100
100
Iteration 1
Inerstion
Iteration
Itcration 2
Figure 10: The dynamics of p: for PCNs (1 = 26 and L 30). Settings: In all subtigures, the results are shown for the cases of ow E (0.5, 1.0, 2.0, 4.0, with different colors, and o, 0.3. Settings not mentioned or indicated are identical to those in Figure 3. Subfigures: (a)-(d) Dynamics of pl of linear PCNs over the 100 inference steps. (e)-(h) Dynamics of pi of nonlinear PCNs (tanh) over the 100 inference steps.
Figure 10 shows the dynamics of the latent state lengths p² for PCNs across varying ow values. The results demonstrate how the network's ' stability depends heavily on the initialization of the weights and inference rate. In both the linear and nonlinear PCN cases, observe that Twe 1) increases, the system becomes more prone to instability, with the exponential growth of the latent state lengths becoming apparent. This is especially visible for higher values of ow (e.g., 8.0), where the growth accelerates drastically. This behavior aligns with the theoretical predictions discussed in the paper, where weight variance ow significantly influences the dynamics of the latent states. For smaller values of ow, such as 1.0, the growth is more contained, allowing the network to maintain more stable latent states across inference steps. However, larger values lead a divergence in p' which necessitates additional regularization techniques, suggested in our proposed framework.
I 15
I 15
100,000
T 100,000
Longhble
10a
Langhh
10P
Lenggh of
2x
Longth det
10
10
10a
10
2.5
5
7.5
10
255
7.5
20
Iteration
Derattnn
layer
layer I
Figure 11: The dynamies of ph and g' for PCNs (T 100, 000). indicated are identical to those in Figure 3.
Settings not mentioned or
Figure 11 explores the effect of extremely large inference steps (T 100, 000) on the dynamics of pi and ghr Despite the large number of steps, the latent states and PEs stabilize after sufficient inference steps when ar 1.0. However, we also observe that PEs tend to concentrate near the output layer, phenomenon consistent with earlier findings that show concentrated PEs major challenge in deep PCNs. This stability over extended inference periods suggests that while PCNs converge in theory, the issue of error concentration near the output layer persists. The results emphasize the need to balance PEs to prevent output-layer dominance, a feature crucial in deep networks for robust training.
Figure 12 provides a heatmap visualization showing the effects of on and 0 on the latent state lengths. For both linear and nonlinear PCNs, we observe that an has much more significant
35

---

Published as a conference paper at ICLR 2026
Lincr
6.
101
100
10'
0.0
30"
6.0
4.5
4.7
4.7
4.5
4.7
63.0
23.0
1010 63.0
10
63.0
30
65.0
1044
13
13
1.8
883
155
30
655
085
45
K0
880 15
60 88
15
30
ORn
15
30
4.5
Figure 12: Heatmap plot of length pi. for linear and nonlinear PCNs. (a {0.6, 1.2, ..., 6.0} and d E {0.6, 1.2,... 6.0)). The total number of inference steps T = 10 and the layer index 1 = 15. Settings not mentioned or indicated are identical to those in Figure 3.
impact the length dynamics than db- This supports the notion that the variance of the weights the primary driver instability, while the bias variance has a more subdued role. The heatmap also reveals that larger ow values result in increasingly longer latent state lengths. These findings underline the necessity of controlling weight variance during initialization, as unchecked variance can lead to runaway growth in latent states.
16,7 20
pengthyph
10
Length par
10th
109
10
8010
10-14
Length
10
10a
20-12
30
20
10
20
Layer
Layer I (b)
Layer (c)
(a)
(d)
Figure 13: (c)-(c) The layer-wise distribution of pht g4d and pf for linear PCNs (o.,. = 5.4 and
T6 0.1). The results are shown for the cases of {10, 20, 50, 100, 200, 500) with different
colors. (d) Direct comparison of p. P. 9, and r for 1 = 15 and f = 20. ý represents the length of the prediction. Settings not mentioned or indicated are identical to those in Figure
Figure 13 presents detailed examination of the layer-wise distribution of ph*,, ght,, and plu in linear PCNs for different inference steps 1. These subfigures aim to capture how the latent state lengths, PEs, and weight update magnitudes evolve across different layers and with varying 1. In Figure 13a- c, for o = 5.4, we observe an exponential growth pattern in the values p. and r across all layers, particularly as increases. This growth expected, given that larger weight variances typi- cally result in larger latent state dynamics, leading to a cascading effect on PEs and weight updates. The increase in p. q, and r with inference steps indicates that the internal representations become increasingly unstable as the inference phase progresses without proper regularization. Figure 13d highlights a direct comparison between p. á (the length of predictions), q, and r for layer = 15 at a 20. Across all values of own we observe that P. and remain within similar range, though their values become more exaggerated for higher ow values. Notably, r, which represents the weight update length, shows explosive growth when ow 5.4, making it impractical to display fully. This behavior confirms that the higher values of aN without regularization lead to unstable weight updates. Interestingly. for lower dw values (e.g., 0ue 0.185), remains small, indicating that proper initialization can contain these dynamics. However, ow = shows a more moderate, controllable behavior in r. This figure emphasizes the need for length regularization and highlights the trade-off between network capacity (as influenced by o..) and the necessity of stability through regularization techniques.
16,7
10-+
10-4
Length
10
Length
10
Length
10-11
10-4
10
10-18
1000
1500
1000
Iteration 6
Iteration (b)
(a)
Figure 14: (a) & (b) The dynamics of g for linear PCNs (a 1 and 06 0.1). The results shown for the cases of different layer index with different colors. (c) Direct comparison of p. ý. q. and for I = 15 and T 2000. á represents the length of the prediction. Settings mentioned or indicated are identical to those in Figure 2.
36

---

Published as a conference paper at ICLR 2026
Figure 14 illustrates the dynamics of q² (PE lengths) in linear PCNs, with ow = 1 and 0 0.1, across different layer indices and inference steps. In Figure 14a and b, we see that the PE length approach the output layer (indicated by red lines). This trend
(q'") increases significantly consistent with the concentration of PEs deeper layers, a challenge observed in deep PCNs that affects the learning capacity of intermediate layers. Conversely, the PE length in earlier layers (indicated by blue lines) starts small and grows gradually with further inference steps, reinforcing the observation that early layers tend stabilize more effectively than deeper layers. Figure 14c compares p. P. 9. and for layer 1 = 15 at T 2000. The comparison shows how the dynamics of prediction lengths (p), latent state lengths (p), and the magnitude of (weight update length) become highly dependent on ow. As noted earlier, the growth in with larger ow values can lead to instability, stressing the need for controlled weight updates via regularization mechanisms.
37


Reviews:
[{"review_id": "7KUf9bl9WN", "reviewer": "Reviewer_gjvy", "summary": "The paper uses a dynamical mean-field theory (DMFT) analysis to diagnose two failure modes in deep predictive coding networks (PCNs): (1) an imbalance of the local energy functions (prediction errors) across depth, (2) exploding/vanishing prediction errors (EVPE). To address these, it introduces Meta-PCN, which (1) uses a meta prediction error objective that linearises the PC equilibrium map around the feed-forward state and trains errors to satisfy the PC delta relation, and (2) applies variance-based weight normalisation for efficient spectral control. In empirical results, Meta-PCN outperforms conventional PC and is competitive with backprop on CIFAR-10/100 and TinyImageNet; ablations indicate the meta-objective is critical. \nThis paper makes a significant contribution to the predictive coding literature, both in its analysis of PCNs in terms of DMFT and the proposed Meta-PCN. I therefore recommend that this paper be accepted, under the constraint that the weaknesses highlighted below are sufficiently addressed.", "questions": "1) What loss do the weights actually optimise?\nYou introduce a new error-based meta-objective J to stabilise inference, but learning appears to proceed as if optimising a different loss. Could you explicitly state the loss function used for weight updates, and, if it is distinct from J, explain why optimising different objectives makes sense conceptually?\n2) How is the “stabilised error” constructed in practice?\nThe text treats the stabilised error as a fixed top-down target, but its computation is not specified.  Is it the last error iterate, an EMA, or something else?\n3) In the Appendix G ablation (“without meta-PC objective”), are feed-forward predictions fixed or iterated?\nThis ablation is central to understanding what Meta-PCN contributes. Does the ablation also freeze the feed-forward predictions? If not, could you include that variant to disentangle the contributions of fixed predictions vs. the meta-objective?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Clear diagnosis of PCN failure modes via DMFT. Identifying depth-wise energy imbalance and EVPE gives concrete targets for stabilisation.\n\nSimple, principled meta-objective. Linearising around the feed-forward state and training errors to satisfy the PC delta relation is elegant and leads to stable inference without heavy tuning. \n\nCompelling empirical evidence. Consistent gains over conventional PC and competitive performance with backprop across CIFAR-10/100 and TinyImageNet, plus ablations that isolate key components.", "weaknesses": "Unclear optimisation of parameters. The paper introduces a new error-based objective to stabilise inference, but does not establish how parameter updates relate to this objective. In effect, inference is performed under one criterion while learning appears to optimise another. Please (i) state explicitly what loss the weights optimise, and (ii) provide diagnostics showing the stated training objective consistently decreases over epochs (a descent argument or convergence guarantee would be even better).\n\nClarify ablation definitions: One of the most convincing results is in Appendix G, where you compare with and without the Meta-PCN objective. However, it is unclear if this includes the fixed prediction (i.e., you fix the initial forward pass and then perform standard free-energy minimisation). If this is the case, please clarify; if not (you do not fix the forward pass), this should be included as an additional ablation to tease apart the Meta-PCN objective from the fixed predictions.\n\nDefinition of the stabilised error: The objective states that the stabilised error is a fixed top-down target, but the paper does not give an explicit construction, presumably from the last iterate, but this is not clear.\n\nAcknowledge trade-offs in freezing feed-forward prediction: In the PC literature, iterative inference over hidden variables is a key benefit. Freezing the feed-forward prediction and optimising errors removes that benefit; this is a reasonable engineering choice, but it should be clearly flagged as a departure from traditional PC.\n\nWeight “variance normalisation” claims overreach given non-iid, structured weights: The normalisation argument assumes random, dense matrices. Convolutional weights are structured and evolve during training. Without measuring actual spectral behaviour before/after normalisation, claims of “robust spectral control” are speculative. This should be clarified in the main text.\n\nLinear DMFT assumptions don’t extend to the nonlinear networks used in experiments: Theory is developed in a linearised regime with random weights; experiments use deep nonlinear models with structure. The paper leans on the theory to justify design choices, but doesn’t bridge the gap. This should be highlighted in the main text rather than dealt with in the appendices.\n\nBackprop baselines aren’t tuned to standard practice for the architectures/datasets you use: Given the modest margins, any claims of “outcompeting backprop” should be softened until stronger baselines are included.\nTraining loop pseudocode: Provide a short, concrete algorithm box covering initialisation, T inference steps, parameter update, and when normalisation is applied.\n\nComputational overhead: The Meta-PCN framework adds multiple components (meta-objective computation, weight normalisation, blocked sweeps) that likely increase computational cost, but this isn't discussed or quantified. \nMinor points Abstract: “demonstrates that Meta-PCN achieves statistically significant improvements …”.\n§3: “depend critically on the variance of the weights”.\n\nEVG/VG phrasing: “are immediately reflected in the magnitude …” (or “are immediately reflected”).\n\nMeta-PCN reintroduces temporal non-locality and activation caching similar to backprop.\nThe method freezes feed-forward activations and runs a distinct inference phase. This requires storing c and h^{0}, analogous to backprop activation caching, and introduces explicit phase separation (forward → inference), which traditional predictive coding avoids. Thus, the biological advantage over backprop is reduced and should be acknowledged in the main text. Although high level justifications are given in the appendix, the authors do not discuss whether the same arguments could be applied to backprop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n2iwBlr25E", "reviewer": "Reviewer_JWY7", "summary": "The paper proposes a theoretical analysis of predictive coding models, as well as a solution to the problem of training very deep networks. This solution is based on two novel proposals: the use of meta-prediction errors, and a new kind of normalization of the weight matrices. A large empirical evaluation shows the effectiveness of the proposed method on a large number of architectures, where the comparison is made agains variations of PC, and backprop.", "questions": "See above. My main concern at the moment is the novelty of the work, so I'd be happy to reconsider my score if this concern is properly addressed. (Also, the score I gave is a little too harsh in my opinion, as the work does have value. However, ICLR this year does not allow for intermediate scores, and 4 is already considered borderline...)", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is well motivated, tackling a timely problem in the field of predictive coding. The empirical evaluation is well done, as it tests on a large number of models and datasets, showing clear improvements in performance. Also, the use of dynamical mean field theory to study  the instability/vanishing error problems of PCNs is novel and interesting.", "weaknesses": "There are a couple of problems in the solutions proposed by the authors, that affect the novelty and bio-plausibility of the meta prediction errors, that is, the fact that they rely on the value of the neurons during the forward pass (so, at t=0). The problems are the following:\n\n1) This is not novel, but it has already been done in three other works, that you also cite: Whittington&Bogacz,2017, B.Millidge 2020 (Predictive coding approximates backprop along arbitrary computation graphs), and C.Qi 2025 (Towards the Training of Deeper Predictive Coding Neural Networks). How is the energy function you describe different from theirs? \n\n2) This is also biologically implausible, as already stated in the three works above, as it breaks the locality in time that define what biologically plausible algorithms are  (operations must rely on information that is locally available at that time step). \n\nA second problem in terms of novelty, is the claim that you have identified the problems of vanishing PE, and PE imbalance across the network. While I agree that the theoretical analysis you propose is novel, I believe you should state that both problems have been already identified in previous work: [1] shows this problem in a small neural network, while [2,3,4] propose their theoretical analysis. So i would rephrase your list of contributions by acknowledging existing work first, the current state of the art, and from there list your contributions. On a side note, I'm not a big fan of works that place the related works in the supplementary material, as it leads exactly to this kind of confusion. Maybe you could move a summary that discusses only these points in the main body?\n\nMinor: \n\nyou propose two methods to improve the results, but you only test them simultaneously. I would add an ablation study so that the reader can understand which method has the largest impact in the final test accuracy.\n\nIn figure 2, what is the exact setup? Is it a randomly initialized model? Is it already trained? On which data?\n\nAt some point you cite:\n\n*Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Predictive coding networks for\ntemporal prediction. Neural Networks*\n\nThis is a wrong citation, maybe you meant this one, with the same title?\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011183\n\n\n\n[1] L .Pinchetti et al. Benchmarking predictive coding networks -- Made Simple (2024)\n\n[2] C.Qi et al. Towards the Training of Deeper Predictive Coding Neural Networks (2025)\n\n[3] Cédric Goemaere et al. Overcoming Exponential Signal Decay in Deep Predictive Coding Networks (2025)\n\n[4] Francesco Innocenti et al. uPC: Scaling Predictive Coding to 100+ Layer Networks (2025)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rqx76wNaXb", "reviewer": "Reviewer_VakM", "summary": "### Summary\n\nThe paper introduces Meta-PCN, a framework designed to stabilize deep Predictive Coding Networks, which traditionally face severe scalability issues. The authors identify two causes for this instability (1) *prediction error (PE) imbalance* and (2) *exploding/vanishing prediction errors (EVPE)*.\n\nThe authors employ dynamical mean-field theory to characterize these instabilities and propose two core innovations:\n\n1. **Meta-prediction error loss**, which minimizes prediction errors of prediction errors, effectively linearizing inference dynamics.\n2. **Variance-based weight regularization**, which controls weight variance to prevent EVPE and ensure balanced signal propagation.\n\nEmpirical results across CIFAR-10/100 and TinyImageNet show substantial stability gains and accuracy improvements of Meta-PCN over PCNs while maintaining biological plausibility.", "questions": "I really enjoyed reading your manuscript, especially section 3. However, there remain a few open questions that would need to be addressed before I could confidently raise my score to an 8. Could you please clarify the following:\n\n**Major**\n\n- **Ablation study:** Based on Figure 2, EVPE appears to be the dominant source of instability in PCNs. However, the ablation results presented by the authors do not include a direct comparison of a standard PCN augmented only with the proposed weight regularization. The experiments instead compare Meta-PCN variants with and without this regularization. Demonstrating that weight regularization alone is insufficient for achieving scalable learning in PCNs would be important to substantiate the necessity of the full Meta-PCN framework.\n- **Precision learning:** It appears that optimizing the precision (1/variance) of PCNs layers would eliminate the PE imbalance by rescaling each error based on its variance (Bogacz 2017). Do you expect this to be the case? While the precisions are often assumed to be identity in PCNs for computational simplicity, they remain an essential part of PCNs that should be considered in your analysis.\n- **backpropagation equivalence:** At equilibrium, the meta-PE formulation appears exactly equivalent to backpropagation ($\\delta_l = g(\\delta_{l+1}, h_{l+1}^0)$) but obtained using a local objective, could you clarify this relationship formally?\n- **Implementation discrepancy:** Why does your predictive coding baseline perform so much worse than previous works (e.g., Pinchetti 2025)?\n\n**Minor**\n\n- **Energy propagation:** Why does error concentration persist at the output boundary (Fig. 4a)? Does this suggest incomplete energy flow across the network?\n- **Baselines:** Did you apply *weight normalization* to the BP baseline for a fair comparison?\n- **Biological interpretation:** Can Meta-PCN be implemented in a local neural circuit model consistent with cortical predictive coding mechanisms?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "### Strengths\n\n- **Convincing Theoretical results:** The paper provides a rigorous dynamical mean-field analysis of PCN inference dynamics, clearly identifying root causes of instability (Sections 3.2–3.3).\n- **Experiments supporting theoretical claims:** Results show experimentally the root causes of instability. Figures 2–3 nicely illustrate the characteristic U-shaped PE imbalance and exponential EVPE dynamics; Figures 4–5 convincingly show that Meta-PCN mitigates these.\n- **Novel conceptual contribution:** The idea of *meta prediction errors* offers an alternative to avoid gradient starvation.\n- **Strong connection to neuroscience:** The work maintains biological plausibility, an property of high relevance for both ML and computational neuroscience.", "weaknesses": "- **Implementation comparison gaps:** The predictive coding baseline underperforms relative to prior works (e.g., Pinchetti 2025), raising questions about whether the reference implementation is optimal.\n- **Connection to backpropagation not fully unpacked:** The equilibrium of the new loss (Section 4.1) appears mathematically equivalent to an iterative local implementation of backpropagation (δₗ = g(δₗ₊₁, hₗ₊₁⁰)), but the paper doesn’t explicitly acknowledge this equivalence or discuss implications for biological plausibility.\n- **Missing mention of the assumptions of the length analysis**: eg. linear activation function and all layers share the same size. These assumptions should be clearly stated in the main text (section 3).\n- **Limited experimental details**: The main text should contain necessary information to understand how the experiment were run broadly. Some additional details should be added, for instance the activity initialisation used for PCNs in figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cm9VLXVfRb", "reviewer": "Reviewer_QiPr", "summary": "The paper empirically illustrates 2 problems, imbalance and unstable PEs, of PCN with high numbers of layers. Based on that analysis, the paper propose a framework to mitigate the problems based on Meta-PC objective and weight regularization. The paper then demonstrates the effectiveness of the framework on VGGs and Resnet-18 in CIFAR10/100 and TinyImagenet.", "questions": "1) Resnet-18 seems to have lower gap between PCN and Meta-PCN. Why is that?\n2) How do you implement PCN and Meta-PCN into VGGs and Resnet-18? Are there any modifications in the architectures or the only changes are in inference and learning procedure?\n3) Ablation results on Meta-PCN objective and Weight regularization, i.e., how each modifications improve PCN?\n4) Are there any hyper/tuning-parameters related to Meta-PCN? It seems this method is parameter-free to me, but I would like to have a clarification on this.\n5) Are there any future research directions can be beneficial from this framework based on PCN?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find the reasoning flow of the paper clear and strong. I am also impressed in how the proposed technicalities can improve PCN significantly, matching traditional back-propagation approach.", "weaknesses": "It will be an excellence paper if the authors can demonstrate the method in larger dataset like Imagenet or demonstrate the scalability in text data on more recent language models. Nevertheless, I think this is a strong, solid paper already.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)