$\mu$PC: Scaling Predictive Coding to 100+ Layer Networks
Authors: Francesco Innocenti
El Mehdi Achour
Christopher Buckley
Venue: NeurIPS 2025 poster ()
Keywords: predictive coding
mup
optimisation theory
hyperparameter transfer
inference learning
local learning
Primary area: neuroscience_and_cognitive_science
TL;DR: We introduce $\mu$PC, a reparameterisation of predictive coding networks that enables stable training of 100+ layer ResNets on simple tasks with hyperparameter transfer.

Abstract: The biological implausibility of backpropagation (BP) has motivated many alternative, brain-inspired algorithms that attempt to rely only on local information, such as predictive coding (PC) and equilibrium propagation. However, these algorithms have notoriously struggled to train very deep networks, preventing them from competing with BP in large-scale settings. Indeed, scaling PC networks (PCNs) has recently been posed as a challenge for the community (Pinchetti et al., 2024). Here, we show that 100+ layer PCNs can be trained reliably using a Depth-$\mu$P parameterisation (Yang et al., 2023; Bordelon et al., 2023) which we call "$\mu$PC". By analysing the scaling behaviour of PCNs, we reveal several pathologies that make standard PCNs difficult to train at large depths. We then show that, despite addressing only some of these instabilities, $\mu$PC allows stable training of very deep (up to 128-layer) residual networks on simple classification tasks with competitive performance and little tuning compared to current benchmarks. Moreover, $\mu$PC enables zero-shot transfer of both weight and activity learning rates across widths and depths. Our results serve as a first step towards scaling PC to more complex architectures and have implications for other local algorithms. Code for $\mu$PC is made available as part of a JAX library for PCNs.

IPCC: Scaling Predictive Coding to 100+ Layer Networks
Francesco Innocenti
El Mehdi Achour UM6P College of Computing Rabat, Morocco e elmehdi awm.urruupccorrrmppppp... ma
School of Engineering and Informatics University of Sussex, UK
F. Innocenniiiiii...... wwwetttissusee..ue ackuk
Christopher L. Buckley
School Engineering and Informatics
University of Sussex, UK VERSES AI Research Lab Los Angeles, CA, USA c. bwwkloyysssseex..ccooooo ac uk
Abstract
The biological implausibility of backpropagation (BP) has motivated many alterna- tive, brain-inspired algorithms that attempt to rely only on local information, such predictive coding (PC) and equilibrium propagation. However, these algorithms have notoriously struggled to train very deep networks, preventing them from competing with BP in large-scale settings. Indeed, scaling PC networks (PCNs) has recently been posed as a challenge for the community [48]. Here, we show that 100+ layer PCNs can be trained reliably using Depth-uu parameterisation [72, 3] which we call "xPCC". By analysing scaling behaviour of PCNs, we reveal several pathologies that make standard PCNs difficult to train large depths. We then show that, despite addressing only some of these instabilities, μPC allows stable training of very deep (up to 128-layer) residual networks simple classifi- cation tasks with competitive performance and little tuning compared to current benchmarks. Moreover, jPC enables zero-shot transfer of both weight and activity learning rates across widths and depths. Our results serve as a first step towards scaling PC more complex architectures and have implications for other local algorithms. Code for μPC is made available as part of a JAX library for PCNs."
Introduction
Backpropagation (BP) is arguably the core algorithm behind the success of modern Al and deep learning [52, 29]. Yet, it is widely believed that the brain cannot implement BP due to its non-local nature [34], in that the update of any weight requires knowledge of all the weights deeper or further downstream in the network. This fundamental biological implausibility of BP has motivated the study of many local algorithms, including predictive coding (PC) [37, 36, 54, 63], equilibrium propagation [59, 74], and forward learning [20], among others [33, 43, 8]. These algorithms offer the potential
for more energy efficient Al and have been argued to outperform BP in more biologically relevant
http:://ggihib..cco//bbbbllllllp [23].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

H = - H =
16
-
H =
- H = 64 - H = 128 PC
=
1
32
H = 64
+ -- 128 μPC
- -H 128 BP
H = 128
Test accuracy (%)
90
Weight
20
10 10
10
10
50
Activity Ir
H 16
Weight ir
1D-1
10
300
600
900
10
Training iteration
Activity le
Figure 1: μPC enables stable training of 100+ layer ResNets with zero-shot learning rate transfer. (Right) Test accuracy of ReLU ResNets with depths H {8, 16, 32, 64, 128} trained to classify MNIST for one epoch with standard PC, μPC and BP with Depth-/uP (see $A.4 4 for details). Solid lines and shaded regions indicate the mean and #1 standard deviation across different random seeds. These results hold across other activation functions (see Fig. A.1 See also Figs. 17-A.19 for asymptotic results with 128-layer ReLU networks trained for multiple epochs MNIST, Fashion- MNIST and CIFARIO 10. (Left) Example of zero-shot transfer of the weight and activity learning rates from 16- to 128-layer Tanh networks. See Figs. 5 & A.31-A.32 for explanation and the complete transfer results across widths as well as depths.
settings such as online and continual learning [61]. However, local learning rules have notoriously struggled train large and especially deep models on the scale of modern AI applications.
For the first time, we show that very deep (100+ layer) networks can be trained reliably using a Depth-uuP-insprred parameterisation [72, 3] of PC which call "μPC" (Fig. 1). To our knowledge, no networks of such depth have been trained before with a local algorithm. Indeed, this has recently been posed challenge for the PC community [48]. We start by showing that the standard parameterisation of PC networks (PCNs) is inherently unscalable in that (i) the inference landscape becomes increasingly ill-conditioned with model size and training time, and (ii) the forward initialisation of the activities vanishes or explodes with the depth. We then show that, despite addressing only the second instability, jPC is capable of training up 128-layer fully connected residual networks (ResNets) on standard classification tasks with competitive performance and little tuning compared current benchmarks (Fig. 1). Moreover, μPC enables zero-shot transfer of both the weight and activity learning rates across widths and depths (Fig. 5). We make code for μPC available as part JAX library for PCNs https: //tps:://gttaii thtbttttu.b.... cww http//theeelllbbbe//////// jpc [23].
The rest of the paper is structured as follows. Following a brief review the maximal update parameterisation (nP) and PCNs ($2), Section exposes distinct pathologies in standard PCNs which make training at large scale practically impossible. Motivated by these findings, we then suggest minimal set of desiderata for more scalable PCN parameterisation ($4). Section presents experiments with jPCC, and Section studies a specific regime where jPC converges to BP. We conclude with the limitations of this work and promising directions for future research ($7). For space reasons, we include related work and additional experiments in Appendix A, along with derivations, experimental details and supplementary figures.
It is possible that these algorithms are more suited to alternative, non-digital hardware, but their scalability can still be investigated on standard GPUs. Indeed, the issues we expose with the standard parameterisation of PCNs can be argued to be hardware-independeer. ($3.1).

---

1.1 Summary of contributions
We show that JPCC, which reparameterises PCNs using Depth-uPP [72, 31, allows stable training of very deep (100+ layer) ResNets on simple classification tasks with competitive performance and little tuning compared to current benchmarks [48] (Figs. & A.17-A.18).
μPC also empirically enables zero-shot transfer of both the weight and activity learning rates across widths and depths (Figs. 5 & A.31 A.32-...2).
We achieve these results by a theoretical and empirical analysis of the scaling behaviour of the inference landscape and dynamics of PCNs ($3), revealing the following two pathologies:
the inference landscape becomes increasingly ill-conditioned with model size (Fig. 2)
and training time (Fig. 3) ($3.1); and the forward pass of standard PCNs vanishes or explodes with the depth ($3.2). To address these instabilities, we propose a minimal set of desiderata that PCNs should aim to satisfy to be trainable at scale ($4), revealing an apparent trade-off between the conditioning of the inference landscape and the stability of the forward pass (Fig. 4). This analysis can be applied to other inference-basee algorithms ($A.2.5).
To better understand jPPC, we study a theoretical regime where the jPCC energy converges to the mean squared error (MSE) loss and so PC effectively implements BP (Theorem 1, Fig. 6). However, we find that μPC can successfully train deep networks far from this regime.
2 Background
2.1 The maximal update parameterisation (nP)
The maximal update parameterisation was first introduced by [70] to ensure that the order of the activation or feature updates at each layer remains stable with the width N. This was motivated by the lack of feature learning in the neural tangent kernel or "lazy" regime [27], where the activations remain practically unchanged during training [6, 31J. More formally, μP can be derived from the following 3 desiderata [70]: (i) the layer preactivations are ON (1) at initialisation, (ii) the network output is ON (1) during training, and (iii) the layer features are also ON (1) during training."
Satisfying these desiderata boils down to solving system of equations for a set of scalars (commonly referred to as "abed"") parameterising the layer transformation, the (Gaussian) initialisation variance, and the learning rate [71, 44]. Different optimisers and types of layer lead to different scalings. One version of μP (and the version we will be using here) initialises all the weights from standard Gaussian and rescales each layer transformation by 1/ /N with the exception of the output which is scaled by 1/Nr.11. Remarkably, jP allows not only for more stable training dynamics but also for zero-shol hyperparameter transfer: tuning a small model parameterised with μP guarantees that optimal hyperparameters such as the learning rate will transfer to a wider model [69, 42].
More recently, μP has been extended to depth for ResNets ("Depth-uP"") [72, 3]. such that transfer is also conserved across depths L. This is done by mainly introducing a I/VL scaling before each residual block. Extensions of standard pP for other algorithms have also been proposed [25, 26, 14, 9].
2.2 Predictive coding networks (PCNs)
We consider the following general parameterisation of the energy function of T-laaerrd PCNs [5]:
2i as Webr(Ze-1) 111
(I)
with weights We E N activities 2, E N: and activation function (.(-). Dense weight matrices could be replaced by convolutions, all assumed be initialised i.i.d. from Gaussian (We)ky (O, be) with variance scaled by be. We omit multiple data samples to simplify the notation, and ignore biases since they do not affect the main analysis, as explained in 1. We also add scalings E R and optional skip or residual connections set by E (0.
Throughout, we will (1) to mean (1) such that the activations neither explode vanish with
3

---

Linear
Tanh
ReLU
24
24-
21-
2
24
21
15
2
10
27-
21
25
26
2 2 2
2
2
H
H
H
Figure 2: Wider and particulary deeper PCNs have more ill-conditioned inference landscape. We plot the condition number of the activity Hessian x(H,) (lower is better) of randomly initialised fully connected networks as a function of the width N and depth (see $A.4 for details). Insets show 2D projections of the landscape of selected networks around the linear solution (Eq. 4) along the maximum and minimum cigenvectors of the Hessian F(z" + a'V + max)). Note that the ill-conditioning is much more extreme for ResNets (see Fig. A.22). Results were similar across different seeds.
The energy of the last layer is defined as FL = 72 - a2 W L for some target Zr = y E Rdua, while the energy of the first layer is Fi _ aj Wizo|l², with some optional Rdw for supervised (vs unsupervised) training.* We will refer to PC or SP as the input parameterisation" with unit premultipliers ag = for all & and standard initialisations "standard
[30, 11, 18] such as by = 1/N2-1, and to μPC as that which uses (some of) the scalings of Depth-juP ($2.1). See Table for a summary.
We fix the width of all the hidden layers N = Ni =... NH where Hl = = 1 is the number of hidden layers. We use 0 == {vec( W } number of parameters and depending on the context, we will use both H and L to refer to the network depth.
IRP represent all the weights with p as the total {zz) ) E RN to denote all the activities free vary. Note that,
PCNs are trained by minimising the energy (Eq. in two separate phases: first with respect to the activities (inference) and then with respect to the weights (learning),
Infer:
min
(2)
Learn:
min F.
Inference acts on single data point and is generally performed by gradient descent (GD), Z FFFF with step size 3. The weights are often updated at numerical convergence of the inference dynamics, when V = 0. Our theoretical results will mainly address the first optimisation problem (Eq. 2), namely the inference landscape and dynamics, but we discuss and numerically investigate the impact on the learning dynamics (Eq. 3) wherever relevant.
Instability of the standard PCN parameterisation
In this section, we reveal through both theory and experiment that the standard parameterisation (SP) of PCNs suffers from two instabilities that make training and convergence of the PC inference dynamics (Eq. 2) at large scale practically impossible. First, the inference landscape of standard PCNs becomes increasingly ill-conditioned with model size and training time ($3.1). I). Second, depending
MMany of theoretical results be extended the unsupervised case (see $A), but for of presentation we will focus on the supervised case.
'ee distinguish between μPC and Depth-uP puPth-pPPon for brevity, to encapsulate both the algorithm and the parameter- isation in single acronym.

---

Linear
Tanh
ReLU
<(E,)
500
900
Training Iteration
Training Iteration
Training iteration
Test accuracy (%)
Test accuracy (%)
Test accuracy (%)
80
80
80
40
20
100
900
500
Training Iteration
Training Iteration
Training Iteration
The inference landscape of PCNs grows increasingly ill-conditioned with training. We plot the condition number of the activity Hessian (Eq. 5) (top) well test accuracies (bottom) for fully connected networks of depths H E {8, 16, 32} during one epoch training. All networks had width N 128 and were trained to classify MNIST (see $A.4 for more details). Similar results are observed for ResNets (Fig. A.9) and FasHion-MNIST (Fig. A.23). Solid lines and shaded regions indicate the mean and standard deviation over 3 random seeds.
Figure 3:
on the model, the feedforward pass either vanishes or explodes with the depth ($3.2). The second problem is shared with BP-trained networks, while the first instability is unique to PC and likely any other algorithm performing inference minimisation ($A.2.5).
3.1 Ill-conditioning of the inference landscape
Here we show that the inference landscape of standard PCNs becomes increasingly ill-conditioned with network width, depth and training time. As reviewed in $2.2, the inference phase of PC (Eq. 2) is commonly performed by GD. For a deep linear network (DLN, Eq. I with Φi = for all f), one can solve for the activities in closed form shown by [26].
V H₂z
z' H
where (Hz)s = FF/OZ ((NH))(NH)) is the Hessian of the energy with respect to the activities, and E N is a sparse vector depending only on the data and associated weights (see $A.2.1 for details). Eq. 4 shows that for DLN, PC inference a well-determined linear problem.
For arbitrary DLNs, one can also prove that the inference landscape is strictly convex as the Hessian positive definite? H2 the energy (Eq. I) is quadratic in z. The result is empirically verified for DLNs in Figs. A.5-A.7 and appears to generally hold for nonlinear networks (see Figs. A.7 & A.22).
(Theorem A.I; see $A.2.2 for proof). This makes intuitive sense since
For such convex problems, the convergence rate of GD known to given by the condition number of the Hessian [4, 41], x(H²) = /mmax / /minn Intuitively, the higher the condition number, the more elliptic the level sets of the energy F(z) become, and the more iterations GD will need to reach the solution (see Fig. A.21), with the step size bounded by the highest curvature direction B < 2/Amax (see Fig. 10 for an example). For non-convex problems, can still be useful to have a notion of local conditioning [e.g. 73].
What determines the condition number H₂? Looking more closely the structure the Hessian
I + a1
WT, W
L = k
E+1
f dz,Ozy
ah+l W WE+t,
k k: -1 else
(5)
WT
0141
E+1
0,
"This contrasts with the weight landscape F(0). which grows nonlinear with depth even for DLNs [22]. We note that this was claimed to be proved by [39]; however, they only showed that the block diagonals of the Hessian positive definite, ignoring the layer, off off-diagonal interactions.
5

---

one realises that it depends on two main factors: (i) the network architecture, including the width N, depth L and connectivity; and (ii) the value of the weights any time during training 9,. We first find that the inference landscape of standard PCNs becomes increasingly ill-conditioned with the width and particularly depth (Fig. 2), and extremely so for ResNets (Fig. A.22). See also $..2.3 for a random matrix theory analysis of the scaling behaviour of the initialised Hessian eigenspectrum with N and L. In addition, we observe that the ill-conditioning grows and spikes during training (Figs. 3, A.9, A.23 & A.25), and using an adaptive optimiser such as Adam [28] does not seem to help (Figs. A.8 & A.24). Together, these findings help to explain why the convergence of the GD inference dynamics (Eq. 2) can dramatically slow down deeper models [23, 48], while also highlighting that small inference gradients- which are commonly used to determine convergence do not necessarily imply closeness to solution.
3.2
Vanishing/explodingg forward pass
In the previous section ($3.1), we saw that the growing ill-conditioning of the inference landscape with the model size and training time one likely reason for the challenging training of PCNs at large scale. Another reason- and the activities can vanish or explode with the depth. This is a classic finding in the neural network literature that has been surprisingly ignored for PCNs. For fully connected networks with standard initialisations [30, 11, 18]. the forward pass vanishes with the depth, leading to vanishing gradients. This issue can be addressed with residual connections [19] and various forms of activity normalisation [24, 11, both of which remain key components of the modern transformer block [64].
will see the key reason-is that the forward initialisation of
However, while there have been attempts to train ResNets with PC [48], they have been without activity normalisation. This is likely because any kind of normalisation of the activities seems odds with convergence of the inference dynamics to solution (Eq. 2). Without normalisation, however, the activations (and gradients) of vanilla ResNets explode with the depth (see Fig. A.30). A potential remedy would be to normalise only the forward pass, but here we will aim to take advantage of more principled approaches with stronger guarantees about the stability of the forward pass ($4).
4 Desiderata for stable PCN parameterisation
In $3, we exposed two main pathologies in the scaling behaviour of standard PCNs: (i) the growing clliconditinning of the inference landscape with model size and training time ($3. 1), and (ii) the instability of the forward pass with depth ($3.2). These instabilities motivate us specify minimal set of desiderata that we would like a PCN to satisfy to be trainable at large scale."
Desideratum 1. Stable forward pass at initialisation. At initialisation, all the layer preacti- vations are stable independent of the network width and depth, ||ze|| ~ ON,H(1) I) for all E, where he (x)) with he(-) as the map relating one layer to the next.
(1)
To our knowledge, there two approaches that provide strong theoretical guarantees about this desideratum: (i) orthogonal weight initialisation for both fully connected [58, 46, 47, 68] and convolutional networks [68], ensuring that W.W, W. = I at every layer E; and (ii) the recent Depth-uP
parameterisation [72, 3] (see $2.1 for a review). For a replication of these results, see Fig. A.30. To apply Depth-uP to PC, we simply reparameterise the PC energy for ResNets (Eq. I with тą 1 for f= ; H and 7; = O otherwise) with the layer scalings of Depth-pp (see Table 1). We call this reparameterisation μPC.
*We do see these desiderata strict (necessary sufficient) conditions, since relatively small PCNs be trained competitively without satisfying them, and other conditions might be needed for successful training. "uP and Depth-uP also include an aptimiser-dependee scaling of the learning rate. However, we found this scaling be suboptimal for PC as discussed in 87.

---

Linear
Tanh
ReLU
21
27
21-
21
10P
21
27-
25-
orthog.
121
27
2' 21
2
2
2
27
2 2
2
2
2"
2
H
H
10P
10
JPC
21
22
F
H
Figure 4: Parameterisations with stable forward passes induce highly ill-conditioned inference landscapes with depth. We plot the conditioning of the activity Hessian of randomly initialised networks over width N and depth !/ for the pPC and orthogonal parameterissatins.. Networks with and without residual connections were used for these respective parameterisations. Note that ReLU networks with orthogonal initialisation cannot achieve stable forward passes (see Fig. A.30). Results were similar across different seeds.
Table 1: Summary of parameterisations. Standard PC has unit layer premultipliers and weights initialised from Gaussian with variance scaled by the input width at every layer Ne-1- μPC uses standard Gaussian initialisation and adds width- and depth-dependent scalings at every layer.
a1 (input weights)
at (hidden weights)
az (output weights)
be (init. variance)
PC
1
NZ1
μPC
1/
(Ne-1
N N
We would like Desideratum 1 to hold throughout training state in the following desideratum.
Stable forward pass during training. The forward pass is stable during training such that Desideratum true for all training steps ,....,
Desideratum 2.
Depth-uP ensures this desideratum for BP, but we do not know whether the same will apply to μPC. We return to this point in $7. For the orthogonal parameterisation, the weights should remain orthogonal during training to satisfy Desideratum 2, which could be encouraged with some kind of regulariser. Next, we address the ill-conditioning of the inference landscape ($3.1), again first at initialisation.
Desideratum 3. Stable conditioning of the inference landscape at initialisation. The condi- tion number of the activity Hessian (Eq. 5) at initialisation stays constant with the network width and depth, 16 (H2) ON.H(1)
Ideally, we would like the PC inference landscape to be perfectly conditioned, i.e. <(H,) = However, this cannot be achieved without zeroing out the weights, H2 0) = I, since the Hessian symmetric and so it can only have all unit eigenvalues if it is the identity. Starting with small weights (Wz) & at the cost of slightly imperfect conditioning is not solution, since the forward pass vanishes, thus violating Desideratum 1. See $A.3.3 for another intervention that appears come at the expense of performance.

---

What about the above parameterisations ensuring stable forward passes? Interestingly, both orthogonal initialisation and μPCC induce highly ill-conditioned inference landscapes with the depth (Fig. 4), similar to standard PC ResNets (Fig. A.22). This highlights potential trade-off between the stability of the forward pass (technically, the conditioning the input-output Jacobian) and the conditioning of the activity Hessian. Because PCNs with ill-conditioned inference landscapes can still be trained (e.g. see Fig. 3), we will choose to satisfy Desideratum at the expense of Desideratum 3, while seeking to prevent the condition number from exploding during training.
Desideratum 4. Stable conditioning of the inference landscupe during training. The condition number the activity Hessian (Eq. 5) stable throughout training such that x(H2(()) x xHH, 1)) for all training steps / T.
Experiments
We performed experiments with parameterisations ensuring stable forward passes at initialisation (Desideratum 1), namely μPPC and orthogonal, despite their inability solve the ill-conditioning of the inference landscape with depth (Desideratum 3; Fig. 4). Due to limited space, we report results only for μPC since orthogonal initialisation was not found be effective (see $A.3.4). We trained fully connected residual PCNs on standard image classification tasks (MNIST, Fashion-MNIST and CIFAR 10). This simple setup was chosen because the main goal was to test whether μPPC capable of training deep PCNs-ap task that has proved challenging with more complex datasets and architectures [48]. We note that the networks used as many inference steps as hidden layers (see Figs. A.14 & A.27 for results with one step).
First, we trained ResNets of varying depth (up to 128 layers) classify MNIST for single epoch. Remarkably, we find that μPC allows stable training of networks of all depths across different activation functions (Figs. & A.1 These networks were tuned only for the weight and activity learning rates, with no other optimisation techniques such as momentum, weight decay, and nudging, as used in previous studies [48]. Competitive performance (= 98%) is achieved in 5 epochs (Fig. A.17), 5x faster than the current benchmark [48]. Similar results are observed on Fashion-MNIST, where competitive accuracy (#= 80%) is reached
fewer than 15 epochs (Fig. A.18). On CIFAR10 10, performance is far from SOTA because of the fully connected (as opposed to convolutional) architectures used, but jPC remains trainable at large depth (Fig. A.19).
Low
Training loss
High
N 64
N 128
N 256
N 512
N 1024
Weight Ir (log)
10
Activity
Activity'
Activity
Activity (log)
(log)
H 8
H 16
H 32
H 64
H 128
Weight Ir (log)
Activity (log)
Activity (log)
Activity (log)
Activity Ir (log)
Activity (log)
Figure 5: μPC enables zero-shot transfer of the weight and activity learning rates across widths N and depths H. Minimum training loss (log) achieved by ResNets of varying width and depth trained with μPC MNIST across different weight and activity learning rates. All networks had Tanh as nonlinearity (see Figs. A.31-A.33 for other activation functions); those with varying width (first row) had hidden layers, and those with varying the depth (second row) had 512 hidden units (see $A.4 for details). Each contour was averaged over random seeds.

---

Strikingly, we also find that μPC enables zero-shot transfer of both the weight and activity learning rates acToss widths and depths (Figs. 5 & A.31-A.32), consistent with recent results with Depth-pP pP [72, 3]. This means that one can tune small PCN and then transfer the optimal learning rates to wider and/or deeper PCNs-a process that particularly costly for PC since it requires two separate learning rates. In fact, this is precisely how we obtained the Fashion-MNIST (Fig. A.18) and CIFARIO (Fig. A.19) results: by performing transfer from to 128-layer networks, avoiding the expensive tuning at large scale.
6
Is μPC BP?
Why does μPC seem to work so well despite failing solve the ill-conditioning of the inference landscape with depth (Fig. 4)? Depth-uP also satisfies other, BP-specific desiderata that PC might not require benefit from. Here we show that while there is practical regime where jPCC approximates BP, it turns out to be brittle, and so BP cannot explain the success of μPC (at least on the tasks considered). In particular, it is possible show that, when the width much larger than the depth N > L, at initialisation the μPC energy at the inference equilibrium converges to the MSE loss. In this regime, PC computes the same gradients as BP and all the Depth-uu theory applies.
Theorem 1 (Limit Convergence of μPC to BP.). Let Fupc((, z) be the PC energy of randomly initialised linear ResNet (Eq. 1 with Te for 8 = 2....,,I and Ti otherwise) parameterised with Depth-uP (Table 1) and (u)(0) its corresponding MSE loss. Then, as the aspect ratio of the network r == L/N vanishes, the equilibrated energy (Eq. 31) converges to the loss (see $A.2.6 proof)
r + 0, Furc(0,z") z²) Lur(
(6)
The result relies on a recent derivation of the equilibrated energy rescaled MSE loss for DLNs [22]. We simply extend this to linear ResNets and show that the rescaling approaches the identity with μPC in the above limit. Fig. 6 shows that the result holds at initialisation (t 0), with the equilibrated energy converging to the loss when the width is around 32x the depth. (Note that the deepest networks (H 128, NN- 512) we tested in the previous section had much smaller aspect ratio, r= 4.) Nevertheless. we observe that the equilibrated energy starts to diverge from the loss with training at large width and depth (Fig. 6). Note also that do not know the inference solution for nonlinear networks. We therefore leave further theoretical study of jPC to future work. See also $A.1 for discussion of how Theorem relates to previous correspondences between PC and BP.
t= 0
t 50
(
200
121
25
10
121
23
12
10?
10P
21
10a
25
12
C/
2
50
54
C
2
14
12
32
21
21
21
10
26
11
11
20
10
120
121
10
26
2
23
2
29
20
2
23
29
22
22
2
23
H
Figure 6: Convergenee/Divergenee of μPC to BP for linear ResNets. To verify Theorem (Eq.
6), we plot the ratio between the MSE loss and the equilibrated μPC energy of linear ResNets (Eq.
31) at different training points L as a function of the width N and depth H (see $A.4 for details). We observe that while at initialisation (t 0) the equilibrated energy converges to the loss as the the width grows relative to the depth (verifying Theorem 1), the correspondence breaks down with training at large depth and width. Results were similar across different runs.

---

7 Discussion
In summary, we showed that it is possible to reliably train very deep (100+ layer) networks with a local learning algorithm. We achieved this via a Depth-juP-like reparameterisation of PCNs which we labelled jPPC. We found that jaPC is capable of training very deep networks with little tuning and competitive performance on simple classification tasks (Fig. 1), while also enabling zero-shot transfer of weight and activity learning rates across widths and depths (Fig. 5).
dPC and inference ill-conditioning. Despite its relative success, μPC failed to solve the growing ill-conditioning of the inference landscape with the network depth (Desideratum 3; Fig. 4). This can be explained by two additional findings. First, the forward pass of jPCC seems to initialise the activities much closer to the analytical solution (Eq. 4) for DLNS than standard PC (Fig. A.35).
Second, training jaPC networks with a single inference step (as opposed to as many as hidden layers) led performance degradation not only during training, but also with depth (Figs, A. & A.27).
Together, these results suggest that a stable forward pass, as ensured by jPPC, is critical not only for performance but also for dealing with landscape ill-conditioning, by initialising the activities closer to solution such that only a few (empirically determined) inference steps are needed. This is also consistent with the finding that while inference convergence necessary for successful training of the SP, it does not appear sufficient for good generalisation (see 6A.3.6). It would be interesting to study μPC in more detail in linear networks given their analytical tractability.
Another recent study investigated the problem of training deep PCNs [12], showing an exponential decay in the activity gradients over depth. This result can be seen as consequence of the ill- conditioning of the inference landscape with depth (Fig. 2), since flat regions where the forward pass seems initialise the activities (see SA.322 have small gradients, and depth drives ill-conditioning. [12] proposed reparameterisation of PCNs leveraging BP for faster inference convergence on GPUs, and it could be interesting to combine this approach with μPC, especially for generation tasks or more complex datasets where more inference steps might be necessary for good performance.
uPC and the other Desiderata. Did μPC satisfy some other Desiderata (544 besides the stability of the forward pass at initialisation (Desideratum 1)? When experimenting with jPC, we tried including the Depth-uP scalings only in the forward pass (i.e. removing them from the energy or even just the inference or weight gradients). However, this always led to non-trainable networks even at small depths, suggesting that the Depth-pP scalings are also beneficial for the PC inference and learning dynamics and that the resulting updates are likely to keep the forward pass stable during training (Desideratum 2). Deriving principled scalings specific to PC could help explain these findings or even lead to better scalings. Finally, jPC did not seem to prevent the ill-conditioning of the inference landscape from growing with training (see Figs. A.28 & A.29), thus violating Desideratum 4.
Is μPC optimal? μPC unlikely to be the optimal parameterisation for PCNs. This because we adapted, rather than derived, principled (Depth-uP) scalings for BP, with only guarantees about the stability of the forward pass. Indeed, we did not rescale the learning rate of Adam (used in all our experiments) by V as prescribed by Depth-uP [72], since this scaling always led to non-trainable networks. We note that depth transfer has also been achieved without this scaling [3, 42] and that the optimal depth scaling is still an active area of research [10]. It would also be useful to better understand the relationship between μPC and the (width-only) μP parameterisation for PC proposed by [26] (see $A.1 for a comparison). More generally, it would therefore be potentially impactful to derive principled scalings specific to PC. While analysis far from inference equilibrium appears challenging, one could start with the order of the weight updates of the equilibrated energy of linear ResNets (Eq. 31).
Other future directions. Given the recent successful application of Depth-uu to convolutional networks and transformers [3, 42], it would be interesting to investigate whether these more complex architectures can be successfully trained on large-scale datasets with juCC. Our analysis of the inference landscape can also be applied to any other algorithm performing some kind of inference minimisation (see $A.2.5 for preliminary investigation of equilibrium propagation), and it could be interesting to see whether these algorithms could also benefit from yPP-like parameterisation.
10

---

Acknowledgements
Fl is funded by the Sussex Neuroscience 4-year PhDD Programme. EMA acknowledges funding by UM6P and the Deutsche Forschungsgemmeinscaaf (DFG, German Research Foundation)- Project number 442047500 through the Collaborative Research Center "Sparsity and Singular Structures" (SFB 1481) as he started this project at RWTH Aachen University. CLB was partially supported by the European Innovation Council (EIC) Pathfinder Challenges, Project METATOOL with Grant Agreement (ID: 101070940). FI would like to thank Alexandru Meterez and Lorenzo Noci for their help in better understanding JaP, and Ivor Simpson for providing access to GPUs used to run some of the experiments.
References
[1] Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint 1rXivv1607.060 2016.
[2] N. Baskerville, P. Keating, F. Mezzadri, Najnudel, and D. Granziol. Universal character- istics of deep neural network loss surfaces from random matrix theory. Journal of Physics A: Mathematical and Theoretical, 55(49):494002, 2022.
[3] B. Bordelon, L. Noci, M. B. Li, B. Hanin, and C. Pehlevan. Depthwise hyperparameter transfer residual networks: Dynamies and scaling limit, arXiv preprint arXiv:2309. I6620, 2023.
[4] S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[5] C. L. Buckley, C. S. Kim, S. McGregor, and A. K. Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55-79, 2017.
[6] I.. Chizat, F. Oyallon, and F. Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
[7] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun. The loss surfaces of multilayer networks. Artificial intelligence and statistics, pages 192-204. PMLR, 2015.
[8] G. Dellaferrera and G. Kreiman. Error-driven imput modulation: solving the credit assignment problem without a backward pass. In International Conference on Machine Learning, pages 4937-4955. PMLR, 2022.
[9] N. Dey, S. Bergsma, and Hestness. Sparse maximal update parameterization: holistic approach to sparse training dynamics. arXiv preprint arXiv:2405.15743, 2024.
[10] N. Dey, B. C. Zhang, L.. Noci, M. Li, B. Bordelon, S. Bergsma, C. Pehlevan, B. Hanin, and J. Hestness. Don't be lazy: Completep enables compute-efficient deep transformers. arXiv preprint arXiv:2505.016/1 01618, 2025.
[11] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[12] C. Goemaere, G. Oliviers, R. Bogacz, and T. Demeester. Error optimization: Overcoming exponential signal decay in deep predictive coding networks. arXiv preprint arXiv:2505.20137. 2025.
[13] D. Granziol. Beyond random matrix theory for deep networks. arXiv preprint arXiv:2006.07721, 2020.
[14] M. Haas, J. Xu, V. Cevher, and L. C. Vankadara. Effective sharpness aware minimization requires layerwise perturbation scaling. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024.
[15] M. Hardt and T. Ma. Identity matters in deep learning. arXiv preprint arXiv:1611.0422 2016.

---

[16] S. Hayou. Commutative scaling of width and depth in deep neural networks. Journal of Machine Learning Research, 25(299):1-41, 2024.
[17] S. Hayou and G. Yang. Width and depth limits commute in residual networks. International Conference Machine Learning, pages 127033 PMLR, 2023.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026 1064,1034,000 2015.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[20] G. Hinton. The forward-forwar algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13344 2022.
[21] R. A. Horn and C. R. Johnson. Matrix analysis. Cambridge university press, 2012.
[22] F. Innocenti, E. M. Achour, R. Singh, and C. L. Buckley. Only strict saddles in the energy landscape of predictive coding networks? Advances in Neural Information Processing Systems, 77:3344-536685 2025.
[23] F. Innocenti, P. Kinghorn, W. Yun- Fumbbrobro,,gh M. D. L. Varona, R. Singh, and C. L. Buckley. Jpc: Flexible inference for predictive coding networks in jax. arXiv preprint arXiv:2412.03676, 2024.
[24] S. loffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
[25] S. Ishikawa and R. Karakida. On the paramcterization of second-order optimization effective towards the infinite width. arXiv preprint UrXiv:2312.12222 2023.
[26] S. Ishikawa, R. Yokota, and R. Karakida. Local loss optimization in the infinite width: Sta- ble parameterization of predictive coding networks and target propagation. arXiv preprint arXiv:2411.02001 2024.
[27] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[28] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1rX.v:1412.6980, 2014.
[29] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521((5533:436 2015.
[30] Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
[31] J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent, Advances in neural information processing systems, 32, 2019.
[32] Z. Liao and M. W. Mahoney, Hessian eigenspectra of more realistic nonlinear models. Advances in Neural Information Processing Systems, 34:20104-20117, 2021.
[33] P. Lillicrap, D. Cownden, B. Tweed, and C. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 731):13276, 2016.
[34] T. P. Lillicrap, A. Santoro, I.. Marris, J. Akerman, and G. Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
[35] V. A. Marchenko and A. Pastur. Distribution of eigenvalues for some sets of random matrices. Matematicheskäi Sbomik, 11 114(4):507-536, 1967.
12

---

[36] B. Millidge, T. Salvatori, Y. Song, R. Bogacz, and T. Lukasiewicz. Predictive coding: towards future of deep learning beyond backpropagation? arXiv preprint arXiv:220..09667, 2022.
[37] B. Millidge, A. Seth, and C. L. Buckley. Predictive coding: a theoretical and experimental review, arXiv preprint arXiv:2107. 12979, 2021.
[38] B. Millidge, Y. Song, T. Salvatori, T. Lukasiewicz. and R. Bogacz. Backpropagation the infinitesimal inference limit energy-based models: Unifying predictive coding, equilibrium propagation, and contrastive hebbian learning. arXiv preprint Sikiv:2206.02623)) 2022.
[39] B. Millidge, Y. Song, T. Salvatori, T. Lukasiewicz, and R. Bogacz. A theoretical framework for inference and learning in predictive coding networks. arXiv preprint 2rXiv:2207.22316, 2022.
[40] B. Millidge, A. Tschantz, and C. L.. Buckley. Predictive coding approximates backprop along arbitrary computation graphs. Neural Computation, 34(6):1329-1363 2022.
[41] Y. Nesterov. Introductory lectures on convex optimization: A basic course, volume 87, Springer Science & Business Media, 2013.
[42] I.. Noci, A. Meterez, T. Hofmann, and A. Orvieto. Super consistency of neural network landscapes and learning rate transfer. Advances in Neural Information Processing Systems, 37: 37:102696-102743, 2025.
[43] A. Payeur, J. Guerguiev, F. Zenke, B. A. Richards, and R. Naud. Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, 24(7): 1010- 1019, 2021.
[44] C. Pehlevan and B. Bordelon. Lecture notes infinite-width limits of neural networks. 2023.
[45] J. Pennington and Y. Bahri. Geometry of neural network loss surfaces via random matrix theory. PMLR, 2017.
In International conference on machine learning, pages
[46] J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
[47] Pennington, S. Schoenholz, and Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pages 1924-11 1932. PMLR, 2018
[48] L. Pinchetti, C. Qi, O. Lokshyn, G. Olivers, C. Emde, M. Tang, A. M'Charrak, S. Frieder, B. Menzat, R. Bogacz, et al. Benchmarking predictive coding networks-made simple. arXiv preprint arXiv:2407.011663 163, 2024.
[49] C. Qi, T. Lukasiewicz, and T. Salvatori. Training deep predictive coding networks. In New Frontiers in Associative Memories, 2025.
[50] D. A. Roberts, S. Yaida, and B. Hanin. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA. USA, 2022.
[51] R. Rosenbaum. On the relationship between predictive coding and backpropagation. Plos one, 170):e0266102, 2022.
[52] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back pack-propagating errors. nature, 323(6088):533-536 1986.
[53] D. K. Salkuyeh. Comments on "a note on a three-term recurrence for a tridiagonal matrix". Applied mathematics and computation, 176(2):442 444, 2006.
[54] T. Salvatori, A. Mali, C. Buckley, T. Lukasiewicz, R. P. Rao, K. Friston, and A. Ororbia. Brain- inspired computational intelligence via predictive coding. arXiv preprint arXiv:2308.07877 2023.
13

---

[55] T. Salvatori, L. Pinchetti, B. Millidge. Y. Song. T. Bao, R. Bogacz, and T. Lukasiewicz. Learning on arbitrary graph topologies via predictive coding, Advances in neural information processing systems, 35:38232-3888
2022.
[56] T. Salvatori, Y. Song, T. Lukasiewicz, R. Bogacz, and Z. Xu. Predictive coding can do exact back- propagation on convolutional and recurrent neural networks. arXiv preprint arXiv:2 103.03225, 2021.
[57] T. Salvatori, Y. Song, B. Millidge, Z. Xu, L. Sha, C. Emde, R. Bogacz, and T. Lukasiewicz. Incremental predictive coding: A parallel and fully automatic learning algorithm. arXiv preprint arXiv:2212.0072 2022.
[58] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint 1rXivv1312.6120 2013.
[59] B. Scellier and Y. Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 1 11:24. 2017.
[60] Y. Song, T. Lukasiewicz, Z. Xu, and R. Bogacz. Can the brain do backpropagation? exact im- plementation of backpropagation in predictive coding networks. Advances in neural information processing systems, 33:22566-22579, 2020.
[61] Y. Song, B. Millidge, T. Salvatori, T. Lukasiewicz, Z. Xu, and R. Bogacz. Inferring neural activity before plasticity: A foundation for learning beyond backpropagation. bioRxiv, pages 2022 05, 2022.
[62] R. Van Handel. Structured random matrices. Convexity and concentration, pages 107-156, 2017.
[63] B. van Zwol, R. Jefferson, and E. L. Brock. Predictive coding networks and inference learning: Tutorial and survey. arXiv preprint arXiv:2407.041 2024.
[64] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[65] H. Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differential- gleichungen (mit einer anwendung auf die theorie der hohlraumstrahlungg) Mathematische Annalen, 71(4):441479, 1912.
[66] J. C. Whittington and R. Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation, 29(5):1229 1262, 2017.
[67] E. P. Wigner. Characteristic vectors of bordered matrices with infinite dimensions i. The Collected Works of Eugene Paul Wigner: Part A: The Scientific Papers, pages 524-540, 1993.
[68] L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference Machine Learning, pages
PMLR, 2018
[69] G. Yang, F. Hu, Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34: 17094-17077 2021.
[70] G. Yang and E. J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pages 11727-11737. PMLR, 2021.
[71] G. Yang and E. Littwin. Tensor programs ivb: Adaptive optimization in the infinite-width limit. arXiv preprint arXiv:2308.0I1 00104, 2023.
[72] G. Yang, D. Yu, C. Zhu, and S. Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244 2023.
14

---

[73] J. Zhao, S. P. Singh, and A. Lucchi. Theoretical characterisation of the gauss- condition- ing neural networks. arXiv preprint arXiv:2411.02139. 2024.
[74] N. Zucchet and Sacramento. Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation. Neurul Computation, 34(12):230-22346, 2022.
15

---

A
Appendix
Contents
A.1
Related work
16
A.2 Proofs and derivations.
17
A.2.1
Activity gradient (Eq. 4) and Hessian (Eq. 5) of DLNs
17
A.2.2
Positive definiteness the activity Hessian
18
A.2.3
Random matrix theory of the activity Hessian
19
A.2.4
Activity Hessian of linear ResNets
21
A.2.5
Extension to other energy-based algorithms
22
A.2.6
Limit convergence of μPC to BP (Thm. 1).
................................
23
A.3
Additional experiments
................................
A.3.1
Ill-ooditioning with training...
23
A.3.2
Activity initialisations
................................
24
A.3.3
Activity decay
25
A.3.4
Orthogonal initialisation
25
A.3.5
μPC with one inference step
26
A.3.6
Is inference convergence sufficient for good generalisation?
26
A.4
Experimental details
27
A.5
Compute resources
28
A.6
Supplementary figures
29
A.1 Related work
uP for PC [26]. The study closest work is [26], who derived μP parameterisation for PC (as well as target propagation), also showing hyperparameter transfer across widths. This work differs from ours in the following three important aspects: (i) derives μP for PC only for the width, (ii) focuses on regimes where PC approximates or is equivalent to other algorithms (including BP) so that all the μP theory can be applied, and (iii) it considers layer-wise scalar precisions Ye for each layer energy term, which are not standard in how PCNs are trained (but are nevertheless interesting to study). By contrast, we propose apply Depth-jPP PC, showing transfer for depth as well as width (Figs. 5 & A.31-A.32). We also study regime where this parameterisation reduces to BP (Fig. 6) while showing that successful training is still possible far from this regime (Fig. 1).
Training deep PCNs [49, 48). Our work also related [49], who following [48] showed that the PC energy (Eq. 1) is disproportionately concentrated at the output layer Fr. (closest to the target) for deep PCNs. They conjecture that this is problematic for reasons: first, it does not allow the model to use (i.e. update) all of its layers; and second, it makes the latents diverge from the forward pass, which they claim leads to suboptimal weight updates. The first point consistent with our theory and experiments. In particular, because the activities of standard PCNs vanish or explode with the depth ($3.2) and stay almost constant during inference due the ill-conditioning of the landscape 1$3.1) (Figs. 10-AA..11 11&1 & A.36), the weight updates are likely to be imbalanced across layers. However, the ill-conditioning contradicts the second point, in that the activities barely move during inference and stay close to the forward pass (see 3..3.2 for relevant experiments). Moreover, divergence from the forward pass does not necessarily lead to suboptimal weight updates and worse performance. For standard PC, deep networks cannot achieve good performance regardless of whether one stays close the forward pass (see $A.3.6). For μPC, on the other hand, as many steps as the number of hidden layers (c.g. Fig. 1) leads to depth-stable and much better accuracy than single step (c.g. Fig. A.14).
16

---

PC and BP. Our theoretical result about the convergence of μPC to BP (Theorem 1) relates to relatively well-established series of correspondences between PC and BP [66, 40, 60, 51, 56, 38]. In brief, if one makes some rather biologically implausible assumptions (such as precisely timed inference updates), it can be shown that PC can approximate or even compute exactly the same gradients as BP. In stark contrast to these results and also the work of [26] (which requires arbitrarily specific precision values at different layers), Theorem applies to standard PC, with arguably interpretable width- and depth-dependent scalings.""
Theory of PC inference (Eq. 2) & learning (Eq. 3). Finally, our work can be seen as a companion paper to [22], who provided the first rigorous, explanatory and predictive theory of the learning landscape and dynamics of practical PCNs (Eq- 3). They first show that for DLNs the energy at the inference equilibrium is rescaled MSE loss with weight-dependent rescaling, a result that build on here for Theorem 1. They then characterise the geometry of the equilibrated energy (the effective landscape on which PC learns), showing that many highly degenerate saddles of the loss including the origin become much easier to escape in the equilibrated energy. Here, by contrast, we focus on the geometry of the inference landscape and dynamics (Eq. 2). As an aside, we note that the origin saddle result of [22] probably breaks down for ResNets, where for the linear case it has been shown that the saddle is effectively shifted and the origin becomes locally convex [15]. We suspect that the results generalise, but it could still be interesting to extend the theory of [22] to ResNets, especially by also looking at the geometry of minima.
..P. For a full treatment of μP and its extensions, we refer the reader to key works of the "Tensor Programs" series [70, 69, 71, 72]. μP effectively puts feature learning back into the width limit of neural networks, lacking from the neural tangent kernel (NKT) or "lazy" regime [27, 6, 31]. In particular, in the NTK the layer preactivations evolve in O(N-1/2) time. In μP, the features instead change in a "maximal" sense (hence "/="), in that they vary as much as possible without diverging with the width, which occurs for the output predictions under SP [70]. More formally, μP can be derived from the 3 desiderata stated in $2.1. juP was extended to depth (Depth-uP) for ResNets by mainly introducing a 1/VT. scaling before each residual block [72, 33. This breakthrough was enabled by the commutativity of the infinite-width and -nfinite-eppth limit of ResNets [17, 16]. Standard μP has also been extended to local algorithms including PC [26] (see jP for PC above), sparse networks [9], second-order methods [25], and sharpness-awe aware minimisation [14].
A.2 Proofs and derivations
All the theoretical results below are derived for linear networks of some form.
A.2.1
Activity gradient (Eq. 4) and Hessian (Eq. 5) of DLNS
The gradient of the energy with respect to all the PC activities of DLN (Eq. 4) can be derived by simple rearrangement of the partials with respect to each layer, which are given by
0F/Oz =
Wix 42 w 22 d? Wzz1
(7)
aF/Ozz Z2 02 2 03 W + w W
(8)
(9)
0F /Ozn = zzz 07-1 - a + W
(10)
Factoring out the activity of each layer
0F/0z: = z 2 + 03 W T W2) - 01 W x 02 u2 22
(11)
0F 08/- = z2 + 0g W Wa) 02 W2 03 W 23
(12)
(13)
/F/Ozn zH + of W W( -1-1 W L- L- 18 01 W.y.
(14)
10e width scaling is inherently local, while the depth scaling is more global but could be perhaps argued to be bio plausible based notion of the brain "knowing its own depth".
17

---

one realises that this can be rearranged in the form of linear system
I + 2 W W2
02 W
0
0
21
@1
Wix
02 W2
I+ 3
W3
W
0
Z2
0
V
43 W
I+ a
W
W4
0
0L-1 WT C OL-1 W It a WT
ZH-1
Z
La
W
y.
H2
(15)
where the matrix of coefficients corresponds the Hessian of the energy with respect the activities (H2) lk == /0 We make the following side remarks about how different training and architecture design choices impact the structure of the activity Hessian:
In the unsupervised case where Zn is left free to vary like any other hidden layer, the Hessian gets the additional terms of WT W, the first diagonal block, -a1 W as the superdiagonal block (and its transpose as the subdiagonal block), and b = 0. 1I This does not fundamentally change the structure of the Hessian; in fact, in the next section we show that convexity holds for both the unsupervised and supervised cases.
Turning on biases each layer such that Fr = / z Og W W bell does not impact the Hessian and simply makes the constant vector of the linear system more dense:
b [a] W1x x + by 02 W2bb, b2- 0g W b3..... ar WTy + b az Wr''''"
Adding an e norm regulariser to the activities _ scales the identity in each diagonal block by 2. This induces unit shift in the Hessian eigenspectrum such that the minimum eigenvalue is lower bounded at one rather than zero (see $A.2.3), as shown in Fig. A.12.
Adding "dummy" latents at either end of the network, such that Fo = zo 2| or F1. = 2 - 2r.12, simply adds one layer to the Hessian with block diagonal given by 21. Compared to fully connected networks, the activity Hessian of convolutional networks is sparser in that (dense) weight matrices are replaced by (sparser) Toeplitz matrices. The activity Hessian of ResNets is derived and discussed in $A.2.44
We also note that Eq. 15 can be used to provide an alternative proof of the known convergence PC inference to the feedforward pass [39] z b b f(x)=
a.... W ix when the output layer is unclamped or free to vary with F az and b = 0.
A.2.2 Positive definiteness of the activity Hessian
Here we prove that the Hessian of the energy with respect to the activities of arbitrary DLNs (Eq. The result is empirically verified for DLNs in $A.2.3 and
5) is positive definite (PD), H also appears to generally hold for nonlinear networks, where we observe small negative Hessian eigenvalues only for very shallow Tanh networks with no skip connections (see Figs. A.7 A.22).
Theorem A.1 (Convexity of the PC inference landscape of DLNS.). For any DIN parame- terised by 0 = (W1...,, W..) with input and outpul (x,y), the activity Hessian of the PC energy (Eq. I) is positive definite
(0) O,
(16)
showing that the inference activity landscape F(z) is strictly convex.
To prove this, we will show that the Hessian satifies Sylvester's criterion, which states that a Hermitian matrix is PD if all of its leading principal minors (LPMs) are positive, i.c. if the determinant of all its square lep-left submatrices positive
[21]. Recall that an n x n square matrix A has n LPMs A, of size h x h for h = 1, ... n. For Hermitian matrix, showing that the determinant of all its LPMs is positive a necessary and sufficient condition determine whether the matrix PD (A > 0), and this result can be generalised to block matrices.
Note that the lack of an identity term in the block diagonal term comes from the fact that the first layer is not directly predicted by any other layer.
18

---

We now show that the activity Hessian of arbitrary DLNs (Eq. 5) satisfies Sylvester's is criterion. We drop the Hessian subscript H for brevity of notation. The proof technique lies in Laplace or cofactor expansion of the LPMs along the last row. This has an intuitive interpretation in that it starts by proving that the inference landscape of mee-hideen-layrr PCNs is (strictly) convex, and then proceeds by induction to show that adding layers does not change the result.
The activity Hessian has N H LPMs of size NE X NE for l 1, ...... H. Let [H] denote the 4th LPM of H, A = [[H]/] its determinant, and D, and Og the 7th diagonal and off-diagonal blocks of H, respectively. Now note that H a block tridiagonal symmetric matrix, as can be clearly seen from Eq. 15. There is a known two-term recurrence relation that can be used to calculate the determinant of such matrices through their LPMs [53]
A = |D/ - 10
t 2
(17)
with 4.
and A Dıl. The first LPM is clearly PD and so its determinant positive, D1 = I I+ 03 W W₂ > 0 = A > 0, showing that the inference landscape of one-hidden-layer linear PCNs is strictly convex. For
2, the first term of the recursion (Eq. 17) is positive, since
D21 = +I+ a3 W Wa > 0 and, A > 0 as we wejust saw. The second term is negative, but it is strictly less than the positive term, a2 W₂ < [I + ag W Wy'Wall
3 Wall + W W2] and so A > O. Hence, the activity landscape of 2-hidden-layer linear PCNs remains convex. The same holds for three hidden layers where (0 < D A₂ = A3 >
We can keep iterating this argument, showing by induction that the inference landscape is (strictly) convex for arbitrary DLNs. More formally. the positive term of the recurrence relation is always strictly greater than the negative term,
De De/A >0 0
(18)
Dz > 1
(19)
and so A, > 0 and H > 0 for all E. Convexity holds for the unsupervised case, where the activity Hessian is now positive semidefinite since the term aj W Wi is introduced (see AA22. The result can also be extended any other linear layer transformation B₃ including ResNets where B, I+WW We-
A.2.3 Random matrix theory of the activity Hessian
Here we analyse the Hessian of the energy with respect to the activities of DLNs (Eq. 5) using random matrix theory (RMT). This analysis follows line of work using RMT to study the Hessian of neural networks, specifically the Hessian of the loss with respect to the parameters [7, 45, 13, 32, 2]. We note that the structure of the activity Hessian is much simpler than the weight or parameter Hessian, in that for linear networks the former is positive definite (Theorem A. $A.2.2), while for the latter this is only true for one hidden layer [22].
In what follows, we recall from $2.2 that the PC energy (Eq. 1) has layer-wise scalings as for all l, and the weights are assumed be drawn from zero-mean Gaussian (We)as ~ (O be) with variance set by be.
Hessian decomposition. The activity Hessian (Eq. 5) is challenging matrix to study theoretically as its entries are not i..dd even at initialization due to the off-diagonal couplings between layers. However, we can decompose the matrix into its diagonal and off-diagonal components:
= D +0
(20)
with D D = offdiag( -a2 W2,.., -05-..,-a--119 -a---W--- tion. Since these matrices are on their own i.i.d. at initialisation, we can use standard RMT results to analyse their respective eigenvalue distributions in the regime of large width N and depth H we are interested in. We will then use these results to gain some qualitative insights into the overall spectrum of Hz.
di + a W.',,. W2,....
+ W.W. and where the off-diagonal part can be seen as perturba-
Analysis of D. As a block diagonal matrix, the eigenvales of D are given by those of its blocks D, -I=I4 + ob WT 1+1 We+1 E R for f= = I,..., H. Note that the size of each block depends only on the network width N. It is easy to see that each block is a positively shifted Wishart matrix. As N + co, the eigenspectrum of such matrices converges to the well-known Maröhenko- Pastur (MP) distribution [35] if properly normalised such that az WT
41 We+1 ~ O1/N).
19

---

As shown in Figs. A.1-A.2. this normalisation can be achieved in two distinct but equivalent ways: (i) by initialising from standard Gaus- sian with by = 1 and setting the layer sealing to ag 1/vN, or (ii) by setting a, = and by = 1/N as done by standard initialisations [30, 11, 18J. either case, in the infinite-width limit the eigenvalues of each diagonal block will converge to a unit-shifted MP density with ex-
1/VV
A(P)
Ap = 1/2
Figure A.I: Empirical eigenspectra of D at ini- tialisation, holding the network width constant (N = 128) and varying the depth H. ag indi- cates the premultiplier at each network layer (Eq. 1), while by is the variance of Gaussian initialisa- tion, with as = and by = 1/N corresponding to the "standard parameterisation (SP).
tremes
lim N
A=(D:) - 1+ (1 2
(21)
= (1.5).
(22)
While the spectrum of D will be a combination of these independent MP densities, its extremes will be the same of D, since all of the blocks are i.i.d. and grow the same rate N -> 00. This is empirically verified in Figs. A.1-A.2, which also confirm that the spectrum of D is only affected by the width and not the depth.
1/VV
A(9)
A(P)
Analysis of O. The off-diagonal component of the Hessian O is a sparse Wigner matrix whose size depends on both the width and the depth and so the correct limit should take both N, H + DG at some constant ratio. Note that the sparsity of O grows much faster with the depth.
Ag m 1/N
Figure A.2: Empirical eigenspectra of D at ini- tialisation, holding the network depth constant (H 128) and varying the width N.
Because sparse Wigner matrices are poorly un- derstood and still an active area of research [62]. we make the simplifying assumption that O is dense.
1/VW
If properly normalised above, know that in the limit the eigenspectrum of dense Wigner ma- trices converges the classical Wigner semicircle distribution [67] with extremes
b 12N
lim H/N-000
( = +2.
(23)
100)
We find that the empirical eigenspectrum of O is slightly broader than the semicircle and, as expected, is affected by both the width and the depth (Figs. A.3-A.4).
Figure A.3: Empirical eigenspectra of at ini- tialisation, holding the network width constant (N = 128) and varying the depth H.
1//33
Analysis of H2. Given the above asymptotic results on D and O we can use Weyl's 's inequali- ties [65] to lower and upper bound the minimum and maximum eigenvalues (and so the condi- tion number) of the overall Hessian at initialisa-
de 1//
tion: Amax D + O) < Amax (D) + Amax (O) and Amin (D ID+00 0) Z Amin (D)+ + + (O). The upper bound (Amax = 7) appears tight, as shown in Figs. A.5-A.7. However, the lower bound pre- dicts negative minimum eigenvalue (Amin -1), which is not possible since the Hessian is positive definite as we proved in $A.2.2.
X60
X000
Figure A.4: Empirical eigenspectra of at ini- tialisation, holding the network depth constant (Hf = 128) and varying the width N.
Nevertheless, we can still gain some insights
into the interaction between D and O by looking at the empirical eigenspectrum of Hg. In particular,
20

---

we observe that the maximum and especially the minimum eigenvalue of the Hessian scale with the network depth (Figs. A.7 & A.22), thus driving the growth of the condition number.
1/V3
1/v
by
by 1/2V
by 1/20
Denbe
ASAL)
AEGG]
ASAL]
2003]
Figure A.5: Empirical eigenspectra of H initialisation, holding the network width con- stant (N 128) and varying the depth H.
Figure A.6: Empirical eigenspectra of H at initialisation, holding the network depth con- stant (H 128) and varying the width N.
Linear
Tanh
ReLU
300
1.75
4.6
125
H
1.6
238
21
328
24
238
27
H
Figure A.7: Maximum and minimum eigenvalues of H, at initialisation as a function of network width N and depth L.
A.2.4 Activity Hessian linear ResNets
Here we derive the activity Hessian for linear ResNets [19], extending the derivation in $A.2.1 1 for DLNs. Following the Depth-uP parameterisation [72, 3], we consider ResNets with identity skip connections at every layer except from the input and to the output. The PC energy for such ResNets given by
H
Fi-skip o ||e2 2
1|
ze at Wiz:-1
Zi-1
(24)
2=
1-skip
21

---

where recall that žς as Wi and Zu == x, = y. We refer to this model as "1-skip" since the residual added every layer. Its activity Hessian given by
21 + 0 +1 2+1
WT Wet1 + aeti + Wet1),
= k + H
I I+ 4 W 4+
Wet1.
{ = k = H
0 Fiskip dz,dzk
H--kipp
ac+1
W A+11 - II,
{ 8- = -
(25)
ae+1 WT
E+ -I,
else
We find that this Hessian is much more ill-conditioned (Fig. A.2 than that of networks without skips (Fig. 2), across different parameterisations (Fig. 4). We note that can extend these results to n-skip linear ResNets with energy
H Σ (=++1
1 11
Σ
F skpp
E2 1|
+
22 a Wezi
Ze Ze-n 11
(26)
2
indeed arbitrary computational graphs [55]. It could be interesting investigate whether there exist architectures with better conditioning of the inference landscape that do not sacrifice the stability of the forward pass (see $4, Fig. 4).
A.2.5 Extension to other energy-based algorithms
Here we include a preliminary investigation of the inference dynamics of other energy-based local learning algorithms. As example, consider equilibrium propagation (EP) [59], whose energy for a DLN is given by
E =
Σ
lly 2
(27)
where 2g x for supervised learning (as for PC), and it is also standard include an 22 regulariser on the activities. Unlike PC, EP has two inference phases: a free phase where the output layer 2 is free vary like any other hidden layer with 3 0; and clamped or nudged phase where the output is fixed to some target y with 3 > 0. The activity gradient and Hessian of the EP energy (Eq. 27) are given by
0E
21
We - 24 W Weth, +
(28)
Oz:
2i
Weze - B(y zi), = L
and
f k #
I+
= k =
E
H2 = O2,0zi
Wi+ls
- k = 1
(29)
WT K+D
l k -1
else
where we abuse notation by denoting the Hessian in the same way as that of the PC energy. We observe that the off-diagonal blocks are equal to those of the PC activity Hessian (Eq. 5). Similar to PC, one can also rewrite the EP activity gradient (Eq. 28) as linear system
WT
Z1
W xx 0
W
I
WT
0
Z2
V,E=
0
Wa
I
:
(30)
WT
ZI.-1
By
W. 1 I+
H
with solution xt H Interestingly, unlike for PC, the EP inference landscape not necessarily convex, which can be easily seen for a shallow 2-layer scalar network where 3X(Hx(w2 > 1)) 0. This is always true without the activity regulariser, in which case the identity in each diagonal block vanishes.
22

---

A.2.6 Limit convergence of PCC to BP (Thm. 1)
Here we provide a simple proof of Theorem 1. Consider a slight generalisation to linear ResNets (Eq. 24) of the PC energy at the inference equilibrium derived by [22] for DLNs:
B
F(z')
T S
r
(31)
2B
H
H as W II at We ar WL II + a. (...l./a.. >
H
T
where
= Id + oz W + Σ
(32)
and the residual error is r; = y, y.-a 0r W. I1 I+ (o ai Wxx, can stand for the batch or dataset size. Note that Eq. 31 is MSE loss with weight-dependent rescaling (Eq. 32). Now, know that, for Depth-juP, the forward pass of this model has ON,H(1) preactivations at initialisation and the residual will also be of order 1. Note that, by contrast, for SP (a. 1 for all l and by = 11) 1) the preactivations explode with the depth (Fig. A.30).
The key question, then, is what happens to the rescaling S in the limit of large depth and width. Recall that for jPC, az = 1/N and weights factor in every term of the rescaling S except for the identity, these terms will all vanish at I/N rate as N + i.c. W W W ~ (L) The depth, the other hand, scales the number terms in S. Therefore, the width will have to grow with the depth at some constant ratio S/N-which can be thought of the aspect ratio of the network [50]- make the contribution of each term as small as possible. In the limit of this ratio r - O, the energy rescaling (Eq. 32) approaches the identity S = I, the equilibrated energy converges to the MSE Furc(z", 0) (6) and so PC computes the same gradients as BP.
1/VNT. for 2,... H (see Table 1). Because the output
A.3
Additional experiments
A.3.1
Ill-conditioning with training
For the setting in Fig. 3, we also ran experiments with Adam as inference algorithm and ResNets with standard GD. All the results were tuned for the weight learning rate (see $A.4 for more details). We found that Adam led to more ill-conditioned inference landscapes associated with significantly lower and more unstable performance than GD (Figs. 3 & A.23).
Linear
Tanh
ReLU
30k
GM
x(H,)
<(H..
400
Training
Training Iteration
Training Iteration
Test accuracy (%)
Test sccuracy (9))
Test acturecy (%6
80
80
60
20
100
500
900
100
500
900
500
900
Training iteration
Training iteration
Training iteration
Figure A.8: Same results as Fig. with Adam as inference algorithm (MNIST).
Interestingly, while skip connections induced much more extreme ill-conditioning (Fig. A.22), performance was equal to, and sometimes significantly better than, networks without skips (Figs. A.9 A.25), suggesting complex relationship between trainability and the geometry of the inference landscape which return to in $....6..
23

---

Linear
Tanh
ReLU
10
O0k
40M
0.58
20M
900
Training Iteration
Training
Training Iteration
Test accuracy (%6
Test accuracy (%)
Test accuracyy (%)
80
80
70
60
100
500
100
500
500
500
900
Training Iteration
Training iteration
Training iteration
Figure A.9: Same results as Fig. 3 with skip connections (MNIST).
A.3.2 Activity initialisations
Here we present some additional results on the initialisation of the activities of PCNs. All experiments used fully connected ResNets, GD as activity optimiser, and as many inference steps as the number of hidden layers. For intuition, we start with linear scalar PCNs chains. First, the ill-conditioning of the inference landscape ($3.1) causes the activities to barely move during inference, and increasing the activity learning rate leads to divergence for both forward and random initialisation (Fig. 10). Similar results are observed for μPC (see Fig. A.35).
verify that
B = 1
B = 10
B = 50
tioory
theory
1xt0a
Shoory
fwd.
1/25
1755
1/5
3/42
Inference iteration
Inference iteration
Inference iteration
theory
theor
random
1/28
1/45
1,25
Inference Itaration
Inference Iteratton
Inference Iteratton
Figure A. 10: -llconditioning the inference landscape prevents convergence to the analytical solution regardless of initialisation. For different initialisations (forward and random) and activity learning rates 33 we plot the activities of 64-layer scalar PCN over inference at the start of training. The theoretical activities were computed using Eq. 4. The task
simple toy regression with = x x + e with x ~ ((111) 1) and ~ ~N 0.5). A standard Gaussian was used for random initialisation, (O, 1). Results were similar across different random seeds.
For wide linear PCNs with forward initialisation, we find similar results except that μPC seems to initialise the activities close the analytical solution (Fig. A.11). The same pattern of results is observed for nonlinear networks (Fig. A.36), although note that in this case we do have an analytical solution. These results might suggest that one does not need to perform many inference steps to achieve good performance with μPC. However, we found that one inference step led to worse performance (including as a function of depth) (Figs. A.14 A.27) compared as many steps as number of hidden layers (Figs. A.16 & A.1
24

---

8 = 1
B = 10
B 50
Diarr
Dhoory
1-1
13H
PC
70
1/32
1/22
5/65
16
16
Inference Iteration
Interence Iteratton
Inference Iteration
1000
μPC
1732
1/22
16
16
Inference Iteration
Inference Iteratton
Inference
Figure A.11: The forward pass of μPC seems to initialise the activities close to the analytical solution (Eq. 4). Similar Fig. A.10, we plot the l² norm of the activities over inference of 16-layer
linear PCNs (N = 128) at the start of training (MNIST), Again, results were similar across different random initialisations.
A.3.3 Activity decay
In $4, discussed how it seems impossible to achieve good conditioning of the inference landscape without making the forward pass unstable (c.g. by zeroing out the weights). We identified one way of inducing relative well-conditionness initialisation without affecting the forward pass, namely adding an f2 norm regulariser on the activities a E. ||ze|? with 1. This effectively induces unit shift in the Hessian spectrum and bounds the minimum eigenvalue at one rather than zero (see $A.2.3). However, we find that PCNs with any degree of activity regularisation a are untrainable (Fig. 12).).
Test accuracy (%)
2*
2.50
60
40
2-
2.75
0.5
0.1
0.05
0.01
H
Activity decay
Figure A.12: Activity decay induces well-conditioned inference at the cost of performance. Left: Same plot as Fig. with an added activity regulariser 2 $||ze| 1 with a = 1. Right: Maximum test accuracy MNIST achieved by linear PCN with N = 128 and H 8 over activity regularisers of varying strength a. Solid lines and (barely visible) shaded regions indicate the mean and standard deviation across 3 random seeds, respectively.
A.3.4 Orthogonal initialisation
As mentioned in $5, in addition to μPC we also tested PCNs with orthogonal initialisation parameterisation ensuring stable forward passes at initialisation for some activation functions ($4; Fig. A.30). We found that this initialisation was not as effective μPC (Figs. A.13 & A.26), likely due to loss orthogonality of weights during training. Adding an orthogonal regulariser could help, but at the cost of an extra hyperparameter to tune. We also find that, except for linear networks, the ill-conditioning of the inference landscape still grows and spikes during training, similar to other parameterisations (e.g. Fig. 3).
25

---

Linear
Tanh
ReLU
30k.
60k
x(Ha)
1.5M
10c
20k
0.5M
400
900
400
900
400
900
Treining iteration
Training iteration
Training iteration
Test accuracy (%6
accuracy
60
Test accuracy
90
75
80
100
500
000
100
500
900
100
500
900
Training iteration
Training iteration
Training Iteration
Figure A.13: Test accuracies in Fig. for orthogonal initialisation. Note that performance is expected drop for ReLU networks which cannot have stable forward passes with orthogonal weights (Fig. A.30), We also plot the condition number of the activity Hessian over training.
A.3.5 JPC with one inference step
All the experiments with μPC (e.g. Fig. 1) used many inference steps hidden layers. Motivated by results of $A.3.2 showing that the forward pass of μPC seems to initialise the activities close to the analytical solution for DLNs (Eq. 4), we also performed experiments with single inference step. We found that this led degradation in performance not only at initialisation but also as a function of depth (Figs. A.14 & A.27), suggesting that some number of steps is still necessary despite μPC appearing to initialise the activities close to the inference solution (Fig. A.11). Similar to other parameterisations, we find that the ill-conditioning of the inference landscape grows and spikes during training.
Linear
Tanh
ReLU
40k
x(H,.
<(H.)
20k
0.5M
20k
900
400
Training Iteration
Training iteration
Training Iteration
accuracy (%)
Test accuracy (%)
Test accuracy (%)
844
852
80
78
80
Training iteration
Training Iteration
Training iteration
Figure A.14: μPC test accuracies in Fig. 1 with one inference step. We also plot the condition number of the activity Hessian during training.
A.3.6 Is inference convergence sufficient for good generalisation?
Our analysis of the conditioning of the inference landscape ($3.1) could be argued to rely on the assumption that converging to a solution the inference dynamics beneficial for learning and ultimately performance. This question has yet to be resolved, with some works showing both theoretical and empirical benefits for learning close to the inference equilibrium [61, 22], while others argue to take only step [57]. As discussed in $7, our results suggest that convergence close to a solution necessary for successful training (or monotonic decrease of the loss), which for brevity we will refer as "trainability". In particular, jPC seems to the activities much closer the analytical solution (Eq. 4) than the SP (SA...2), and training μPC with one inference step leads to worse performance (e.g. Fig. A.14) than with as many hidden layers (e.g. Fig. I).
Here we report another experiment that speaks to this question and in particular suggests that while inference convergence is necessary for trainability, it is insufficient for good generalisation, least
26

---

for standard PC. Training linear ResNets of varying depth on MNIST with "perfect inference" (using Eq. 4), we observe that even the deepest (H 32) networks now become trainable with standard PC in the sense that the training and test losses decrease monotonically (Fig. A.15). However, the starting point of the test losses substantially increases with the depth, and the test accuracies of the deepest networks remain at chance level. These results do not contradict our analysis but highlight the important distinction between trainability and generalisation. Our analysis addresses the former, while the latter is beyond the scope of this work.
Train loss
100p
0.01
Test lass
10k
F
Test accuracy
80
60
1p
100
H
10n
20
100g
0.01
468
936
100
500
900
100
500
900
Training iteration
Training iteration
Training iteration
Figure A.15: Train and test metries of standard PCNs of varying depth trained with analytical
inference (Eq. 4). We plot the training loss, test loss and test accuracy of ResNets (N = 128) trained with standard PC on MNIST by solving for inference analytically (using Eq. 4). All experiments
used Adam optimiser with learning rate 1e_ Solid lines and shaded regions represent the mean and standard deviation across 3 random initialisations.
A.4 Experimental details
Code to reproduce all the experiments is available at https: com/thheuuckkeylabbb experiments http//uuppppppppee We always used no biases, batch size B = 64, Adam as parameter optimiser, and GD as inference optimiser (with the exception of Figs. A.8 & A.24). For the SP, all networks used Kaiming Uniform (We)iz ~ U(-1/No-1, 1/Nr) as the standard (PyTorch) initialisation used to train PCNs.
uPC experiments (e.g. Fig, 1). For the test accuracies in Figs. 1 & A.16, we trained fully connected ResNets (Eq. 24) to classify MNIST with standard PC, μPC and BP with Depth-uP. To ensure fair comparison, BP with Depth-ul employed the same scalings as μPC. All networks had width N = 512 and always used as many GD inference iterations as the number of hidden layers H E (2')7- To save compute, we trained only for one epoch and evaluated the test accuracy every 300 iterations. For pPC, we selected runs based on the best results from the depth transfer (see Hyperparameter transfer below). For standard PC, we conducted the same grid search over the weight and activity learning rates as used for jPC. For BP, we performed a sweep over learning rates 1) E [1e", 5e le-1, 5e- le 56 1e-3 5e-1, 1e-4] at depth H = 8, and transferred the optimal value to the deepest (H = 128) networks presented.
Fig. A.20 shows similar results for jPC based on the width transfer results. Fig. A.17 was obtained by extending the training of the 128 ReLU networks in Fig. I to 5 epochs. Figs. A. & A.27 were obtained with the same setup as Fig. 1 by running μPC for single inference step. As noted in $5, the results on Reshion-MNISS (Fig. A.18) were obtained with depth transfer by tuning 8-layer networks and transferring the optimal learning rates to 128 layers.
Hessian condition number at initialisation (e.g. Fig. 2). For different activation functions (Fig. 2), architectures (Fig. A.22) and parameterisations (Fig. 4), computed the condition number of the activity Hessian (Eq. 5) at initialisation over widths and depths N, H E {2 12) This was the maximum range we could achieve to compute the full Hessian matrix given memory resources. No biases were used since these do not affect the Hessian as explained in $A.2.11. 1. Results did not differ significantly across different seeds input and output data dimensions, predicted from the structure of the activity Hessian (Eq. 5).
For the landscape insets of Fig. 2, the energy landscape was sampled around the linear solution of the activities (Eq. 4) along the maximum and minimum eigenvectors of the Hessian F(x" + a min + (V with domain a, 3 [-2, and 30 x 30 resolution.
27

---

Hessian condition number over training (e.g. Fig. 3). For different activations (e.g. Fig. 3). architectures (e.g. Fig. A.9), algorithms (e.g. Fig. A.8) and parameterisations (e.g. Fig. A.13), we trained networks of width N = 128 and hidden layers H E {8, 16, 32] to perform classification on MNIST and Fashion-MNin MNIST. This set of widths and depths was chosen to allow for tractable computation of the full activity Hessian (Eq. 5). Training was stopped after one epoch to illustrate the phenomenon of ill-conditioning. All experiments used weight learning rate rj = le and performed grid search over activity learning rates B (5e-2 1, 5e 2). A maximum number of I' 500 steps was used, and inference was stopped when the norm of the activity gradients reached some tolerance.
Hyperparameter transfer (e.g. Fig. 5), For the ResNets trained on MNIST with μPC (e.g. Fig. 1), we performed 2D grid search over the following learning rates: 7) E (5e llle 5, -, 3.1e2_ for weights, and 33 E (le², 5e",, le²; 5e", let, 5e." le 5e -1 le -1 5e -2, 1] for the activities. We trained only for one epoch, in part to save compute and in part based on the results of [3. Fig. B.3] showing that the optimal learning rate could be decided after just epochs on CIFAR-10. The number of (GD) inference iterations width transfer results, we trained networks of 8 hidden layers and widths N E 1-=6+ while for the depth transfer fixed the width to N 512 and varied the depth H E (2') Jim3 Note that this means that the plots with title N = 512 and = 8 in Figs. 5 & A.31-A.32 are the same. The landscape contours averaged over 3 different random seeds, and the training loss is plotted log scale to aid interpretation.
always the same as the number of hidden layers. For the
Loss energy ratios (e.g. Fig. 6). We trained ResNets (Eq. 24) to classify MNIST for epoch with widths and depths N, E { To replicate the successful setup Fig. 1, we used the same learning rate for the optimal linear networks trained MNIST, r7 1e To verify Theorem 1, at every training step we computed the ratio between the Depth-uP MSE loss L(O) and the equilibrated jPC energy F(z", O) (Eq. 31), where z computed using Eq. 4. All experiments used the weight learning rate 1e- 4 Fig. A.33 shows the same results for the SP, which used a smaller learning rate 1 = le avoid divergence large depth. All the phase diagrams are plotted easier visualisation. Fig. A.34 shows an example of the ratio dynamics of μPC vs PC for a ResNet with hidden layers and different widths. Results were similar across different random initialisations.
log scale for
A.5 Compute resources
The experiments involving μPC, hyperparameter transfer, and the monitoring of the condition number of the Hessian during training were run on an NVIDIA RTX A6000. The runtime varied by experiment, with the 1 128-layer networks trained for multiple epochs (Figs. A.17-A. 18) taking several days. All other experiments were run on a CPU and took between one hour and half a day, depending on the specific experiment.
28

---

A.6 Supplementary figures
Linear
Tanh
ReLU
Test accuracy (%)
Test accuracy (%)
Test accuracy
300
600
900
300
600
900
300
600
900
Training Iteration
Training iteration
Training Iteration
- H 128
BP
H =
8
H
16
H = 32 H = 64 - H = 128
pPC
16
32 H 64 H 128 PC
Figure A.16: Test accuracies in Fig. for different activation functions. Solid lines and shaded regions indicate the mean and standard deviation across 3 random seeds, respectively. BP represents with Depth-uP.
Test accuracy (%)
100
BP
PC
PPC
55
10
300
2400
4500
Training iteration
Figure A.17: 128-layer residual ReLU network trained with μPC on MNIST for 5 epochs. Solid lines and (barely visible) shaded regions indicate the mean and standard deviation across 5 random seeds, respectively. BP represents BP with Depth-uP.
Test accuracy (%)
90
BP
PC
μPC
50
10
1
15
Epoch
Figure A.18: 128-layer residual ReLU network trained with μPC on Fashion-MNIST.. Solid lines
and (barely visible) shaded regions indicate the mean and standard deviation across random seeds, respectively, BP represents BP with Depth-ud..
29

---

Test accuracy (%)
45
BP
PC
30
PPC
10
10
20
Epoch
Figure 1919: 128-layer fully connected residual ReLU network trained with JaPC on CIFAR1O. Solid lines and (barely visible) shaded regions indicate the mean and standard deviation across random seeds, respectively. BP represents BP with Depth-uP. As for other datasets, we see that μPC remains capable of training such deep networks, although performance slightly lags behind BP. Note that accuracies for all algorithms are far from SOTA because of the fully connected (as opposed to convolutional) architecture used.
Linear
Tanh
ReLU
accuracy
X7 2
X7
95
Test accuracy
99
80
300
600
900
Test
85
300
500
900
85
300
600
900
Training iteration
Training iteration
Training iteration
Figure A.20: Same results as Fig. 1 varying the width N and fixing the depth at H 8, showing that "wider is better" [69, 26].
High
22
F
Low
-22
Figure A.21: Toy illustration of the ill-conditioning of the inference landscape. Plotted is the activity or inference landscape F(21, 22) WI along with the GD dynamics. One weight was artificially a much higher value than the others to induce ill-conditioning.
for a toy linear network with two hidden units f(x) =
30

---

Linear
Tanh
ReLU
101
30
24
25-
21-
107
24
107
201
21-
27-
27-
20
27-
101
H
H
H
Figure A.22: Same results as Fig. 2 for the activity Hessian of ResNets (Eq. 25).
Linear
Tanh
ReLU
1500
20k
<(Ha]
1000
1000
10k
500
500
400
900
400
900
400
900
Training Iteration
Training iteration
Training iteration
(%)
Test accuracy (%)
Test accuracy (%)
80
80
60
20
20
100
500
900
100
500
900
500
900
Training iteration
Training iteretion
Training iteration
Figure A.23: Same results as Fig. 3 for FaShion-MNIST..
Linear
Tanh
ReLU
60M
R(H+)
40M
((H.)
400
900
400
900
400
000
Training iteration
Training Iteration
Training Iteration
Test accuracy (96)
Test accuracy (16)
accuracy
20
0
100
500
900
100
500
900
100
500
900
Training Iteration
Training iteration
Training iteration
Figure A.24: Same results as Fig. A.8 for Fashion-MNIST.
31

---

Linear
Tanh
ReLU
200M
30k
x(Haa)
20k
100M
100M
400
900
400
900
400
900
Training iteration
Training iteration
Training iteration
Test accuracy (9%
Test accuracy (%)
accuracy
80
60
60
100
500
900
100
500
900
100
500
Training Iteration
Training Iteration
Training iteration
Figure A.25: Same results as Fig, A.9 for Fashion-MNIST.
Linear
Tanh
ReLU
200k
x(H.)
20k
400
Training iteration
Training Iteration
Training iteration
Test accuracy (96
Test accuracy (96)
accuracy
75
75
Training iteration
Training
Training
Figure A.26: Same results as Fig. A.13 for Fashion-MNIST..
Linear
Tanh
ReLU
40K
500k
<(H.)
400k
30k.
20k
200h
20k
400
900
400
900
400
900
Training iteration
Training iteration
Training iteration
Test accuracy (%)
Test accuracy (36)
accuracy
17
74
75
75
72
100
500
900
100
500
900
100
500
900
Training iteration
Training Iteration
Training iteration
Figure A.27: Same results as Fig. A.14 for Fashion-MNIST.
32

---

Linear
Tanh
ReLU
15k.
2
63k
<(H))
40k.
400
500
400
900
400
900
Training iteration
Training iteration
Training iteration
Figure A.28: Inference conditioning during training for some μPC networks in Fig. 1.
Linear
Tanh
ReLU
15k
20k
x(H,)
15k
10kk
900
400
Training Iteration
Training Iteration
Training iteration
Figure A.29: Same results as Fig. A.28 for Fashion-MNIST.
SP
orthogonal
Depth uP
10
Linear
10
3/43
0.5
0.05
Depth
Depth
Depth
Tanh
1/4
2246
0.5
20
2
2
Depth
Depth
ReLU
0.6
10ta
1/20
0.4
0.5
Depth
Depth
Depth
Figure A.30: Forward pass (in)stability with network depth for different parameterisations. For different activation functions and parameterisations, plot the mean el norm of the feedforward pass activities at initialisation as a function of the network depth L. Networks (N = 1024) had skip connections for the standard parameterisation (SP) and Depth- but not orthogonal. Results were similar across different seeds.
33

---

Low
Training loss
High
256
512
N 1024
Weight
1913
10
10
10
30
10
10
10
10 10
Activity (leg)
Activity (leg)
Activity (leg)
Activity in (log)
Activity (l0g)
8
32
64
H 128
Weigght
1010
10
12
107
101
(log)
Activity (log)
Activity (log)
(log)
(log)
Figure A.31; Same results as Fig. for Linear.
Training loss
High
N 64
N 128
N 256
N 512
N 1024
Weight Ir (teg)
Activity (log)
Activity (log)
Activity (log)
Activity (log)
Activity (log)
H 8
H 16
H 32
H 64
H 128
Weight Ir (log)
1014
10"
Activity (log)
Activity (log)
Activity (log)
Activity (log)
Activity (log)
Figure A.32: Same results Fig. 5 for ReLU.
t= 0
t= 50
= 200
28
28
28
24
24
21
10f
10a
22.
22
22
10'
10a
10R
20
10a
10a
10a
H
H
H
Figure A.33: Same results as Fig. 6 for the standard parameterisation (SP).
34

---

Linear
Tanh
ReLU
10
C.F.
C
C/F""
PC
10
Training iteration
Training Iteration
Training Iteration
pPC
C F
1.5
465
9036
Training Reration
Training iterattonn
Training iteration
Figure A.34: Example the loss vs energy ratio dynamics of SP and μPC for H 4.
B
1
10
β 50
-0.5
-0.5
fwd.
0.5
1/45
3/40
2/43
32
Inference Iteration
Inference
Inference Iteration
0. 15
random
1/45
005x10m
3/40
2/43
-1x10
Inference iteration
Inference Iteration
Inference iteretion
Figure A.35: Same results as Fig. A.10 for μPC.
3 =
B = 10
B 50
[eell
10
1/20
23k
1/25
PC
16
16
16
Inference Itaration
Inference Iteratton
Inference Itaration
10
1/26
1125
IPC
16
Inference Itaration
Inference Iteratton
Inferance Itaration
Figure A.36: Same results as Fig. A.11 for a ReLU network.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our main claims are clearly stated in the abstract and introduction and are strongly supported by experiments. We include summary of contributions at the end of the introduction, and the main limitation of our work (extension to more complex architectures) is clearly stated in the abstract.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We state the main limitation of our work (referring to the extension to more complex architectures) in the abstract, and address all important limitations in the discussion ($7).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
36

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our two main theoretical results (Thms. & A.1) rely on the key assumption of network linearity, which we state clearly and emphasise where relevant. Complete proofs are provided in the Appendix ($A.2.2 & A.2.6), with intuitions included in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes)
Justification:
We provide details needed reproduce all the experimental results in $A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
37

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We release code that be used reproduce all the experimental results https: https:///ithub.. com/ttheuuckeeyllbbpppp [23]]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm co/
public/ cublic/guides// Seniides/CodSSubiii wommiisssoooiioooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwtppcccc//aaccccccccccccccccclt wwwiic/gg//////eieeieiiiiieiisee guwddes/oodeuuumiiiiiccc Cwweeoomiisssiiiiooooo wwwsssssssiiooooooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify important details of all the experiments in the main text and all other relevant information $A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars where most relevant (e.g. Fig. 1). Where they included, it is always because results did not significantly vary across different seeds runs, stated in the corresponding figure captions. Wherever shown, error bars always represent +1 standard deviation, as noted in the captions.
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on the compute resources used for the different experiments in $A.5.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / //n/ueeursp...// cc
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciii..))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We see no potential positive or negative societal impact of the work since the models tested are too simple for modern AI applications.
39

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of our experiments use JPC [23], an MIT-licensed library for training PCNs, which we cite in the main text.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
40

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We make MIT-licensed modifications to JPC [23] to allow for training of PCNs with pPC.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
41

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpa::///eriipie for what should or should not be described.
42


Reviews:
[{"review_id": "fswPKE4DBI", "reviewer": "Reviewer_aWaW", "summary": "The authors present an analysis of standard PC showing that it is susceptible to vanishing and exploding inference dynamics in networks of increasing size. They reveal a trade-off between conditioning of the inference landscape and the stability of the forward pass, and introduce muPC, showing equivalence to BP, helping to bridge the gap in performance at networks of scale.", "questions": "This would be a stronger accept with more diverse experiments and some reflection on limitations and possible extensions of this approach. Motivation could also be developed as well as further comment on the link between this and other local learning algorithms, the possible benefits of training via local learning , limitations of backprop etc. beyond the bioplausibility issue", "limitations": "No. I imagine experimental results would lend further insight into limitations and advantages of this approach. Can we expect such networks to match bp performance exactly , given theorem 1? Are there other caveats worthy of consideration.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This was a very impressive paper that was well motivated with great theoretical contributions. This work has impact not only for general literature in bioplausible learning algorithms, but by extension other forward-only and local learning algorithms and by extension novel hardware and training paradigms. Overall very impressive work!\n\nI see only one notable weakness, which is limited experimental results. It would be interesting to showcase the direct applicability of this approach to problems of greater complexity, although training such models is surely expensive. Authors might also consider broadening the discussion of the implications of this work for other kinds of local learning algorithms, although it is alluded in the introduction.", "was_revised": true, "final_justification": "I have increased my confidence in this assessment in light of the authors' response.", "rebuttal": "We thank the reviewer for their feedback and are glad that they share our excitement about the work. Below we address the key points made by the reviewer.\n\n**Limited experiments:** First, we completely agree that the main weakness of the paper are the limited experiments testing $\\mu$PC, especially in terms of range of tasks and datasets explored. The main reason for this is that, as correctly noted by the reviewer, training of such deep models is very computationally expensive on GPUs even on simple datasets like MNIST, due to both the size of the networks and the sequential nature of PC inference. This cost further increases for generation tasks where more inference steps tend to be needed. For these reasons, we think that trying to extend our results, in the first instance, to convolutional networks trained on more challenging classification datasets is an important (and exciting) next research step, as mentioned in the conclusion.\n\n**Clarification on $\\mu$PC and BP:** Second, we would like to clarify a potential misunderstanding regarding the relationship between $\\mu$PC and BP explored in Section 6. Theorem 1 shows that under certain conditions $\\mu$PC will compute the same gradients as BP ***at the initialisation of the weights***, with no guarantees about what happens during training. Figure 6 shows that this result indeed holds at initialisation but breaks down after a few steps of training in the regime of large width and depth we are interested in. This means that $\\mu$PC computes (approximately) the same gradients as BP ***only*** at initialisation (at large width relative to depth). Importantly, this does not necessarily mean that $\\mu$PC will underperform BP, and in fact the empirical results show that it is equally performant. As shown by Innocenti et al. (2024), for example, PC can have different gradients than BP that still allow successful (and sometimes even faster) training. Nevertheless, it is still not entirely clear why muPC is so successful, and more theoretical work is needed as noted in the text (lines 234-5). We now make these points clear in Section 6 to avoid any potential confusion.  \n\n**Motivation of local learning:** The reviewer remarks that the study of local learning algorithms as alternatives to standard BP could also be better motivated beyond their biological plausibility. We agree and now mention in the introduction that these algorithms also hold the promise of more energy efficient AI and have been argued to outperform BP in more biologically relevant settings including online and continual learning (Song et al., 2024).\n\n**Expanded limitations and implications:** We agree with the reviewer that, due to limited space, our discussion of the limitations and implications of the work (including connections with other local learning algorithms) was very brief. There are two main implications that we see for other local algorithms which the reviewer specifically asks about. First, our results clearly highlight that the stability of the forward pass is as critical for PC as for standard BP, suggesting that other local algorithms could benefit from a $\\mu$P-inspired parameterisation. Second, as noted in the conclusion (lines 252-4), our analysis of the inference landscape can be applied to any other algorithm performing some kind of inference minimisation. As noted, we include a preliminary investigation of equilibrium propagation in the Appendix (A.2.5).\n\nThe camera-ready version of the paper will also include an expanded Conclusion section addressing the following main points: (i) why $\\mu$PC is successful despite the ill-conditioning, (ii) whether $\\mu$PC seems to satisfy other Desiderata than #1, (iii) the limitations of $\\mu$PC and whether it is optimal, and (iv) additional future research directions. More specifically, we argue that:\n* $\\mu$PC seems to work despite the ill-conditioning of the inference landscape because its forward pass initialises the activities much closer to an inference solution than standard PC (see Fig. A.11), which we show tends to vanish/explode; \n* $\\mu$PC seems to satisfy not only Desideratum 1, but also Desideratum 2 about the stability of the forward pass during training. While we have no theoretical results on the training dynamics of $\\mu$PC, the fact that $\\mu$PC shows good performance throughout training strongly suggest that the forward pass remains stable;\n* $\\mu$PC is unlikely to be the optimal parameterisation in that it was not developed specifically for PC, and more theoretical work is required in this respect.\n\n**References**\n\nInnocenti, Francesco, et al. \"Only strict saddles in the energy landscape of predictive coding networks?\" NeurIPS (2024).\n\nSong, Yuhang, et al. \"Inferring neural activity before plasticity as a foundation for learning beyond backpropagation.\" Nature Neuroscience (2024)."}, {"review_id": "AGoE6nx0aG", "reviewer": "Reviewer_ijZ5", "summary": "The paper *\"µPC: Scaling Predictive Coding to 100+ Layer Networks\"* presents µPC, a novel parameterisation that enables scalable training of deep predictive coding networks (PCNs) with up to 128 residual layers.\nIt identifies that traditional PCNs face challenges with depth in residual architectures due to ill-conditioned inference initialisations. µPC, leveraging Depth-µP scaling, addresses these issues, enabling stable training, competitive performance, and zero-shot transfer of learning rates across varying network widths and depths.\nBeyond these results, the authors provide extensive analysis of four desiderata for stable parameterisations, showing that satisfying the first (desideratum 1) alone suffices for stable training in residual networks.\nThe additional desiderata and corresponding analyses offer future research directions and tools for enhancing training stability in PCNs.\nOverall, this work marks a step toward scaling local, biologically plausible learning algorithms to the level of modern deep learning systems.", "questions": "Major\n\n- Is the proposed method only applicable to residual networks? Figures A.17 and A.18 suggest that deep ReLU networks benefit from µPC. What exactly are these ReLU networks? Do they include residual connections? If not, which initialisation do they use—is it the µP initialisation described in Section 2.1 or the Depth‑µP initialisation proposed specifically for ResNets? Clarifying this will make the scope of the work clearer.\n\nMinor\n\n- In Section 2.2, you state that SP uses a normal initialisation with variance 1/fan_in. However, in your implementation, it appears that you use a uniform initialisation with the same variance, as you rely on equinox.nn.Linear (https://github.com/patrick-kidger/equinox/blob/main/equinox/nn/_linear.py) which uses a uniform initialisation from (https://github.com/patrick-kidger/equinox/blob/main/equinox/nn/_misc.py). Did I misinterpret your codebase? If not, does the choice between uniform or Gaussian initialisation materially affect your results or analysis?\n- You devote a lot of space to the ill-conditioned energy landscape, arguing that this poses a challenge for PC inference. However, from Figures 2 and 4, it appears that µPC results in a significantly more ill-conditioned landscape than standard PC, with κ(Hz) increasing by factors of 1,000 to 10,000. Do you think the improved training stability observed with µPC comes at the cost of slower inference, particularly if you let inference run until convergence instead of fixing the number of inference steps to the depth of the model?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nQuality\n\n- Extensive experimental evaluation: The authors demonstrate stable training of predictive coding networks with up to 128 residual layers on classification benchmarks, supporting their claims with empirical evidence.\n- Clear scaling analysis: They conduct extensive scaling studies, identify two pathologies in standard PCNs, and show how µPC addresses one of them through Depth‑µP parameterisation.\n\nClarity\n\n- Contributions were clearly stated\n- Figures were clear.\n\nSignificance\n\n- Unlocking depth stability for residual PCNs, potentially enabling biologically plausible learning in very deep models.\n- Zero‑shot learning‑rate transfer: Demonstrating that learning rates generalise across network widths and depths is practically valuable for reducing hyperparameter tuning in future PCN research.\n- The proposed techniques likely extend to residual networks with convolutions and other base operations, broadening the paper's impact.\n\nOriginality\n\n- Applying Depth‑µP scaling to predictive coding is a novel approach that bridges recent developments in parameter scaling with biologically plausible learning. While µP has appeared in other contexts, its integration with predictive coding for depth scaling is innovative and opens new research directions for predictive coding.\n\n## Weakness\n\nQuality\n\n- Evaluated on simple classification tasks that do not require deep architectures. Training on MNIST and Fashion MNIST typically does not require more than 2 hidden layers. While the parametrisation improves stability, it remains unclear whether any layers beyond the final few actually learn. Thus, the applicability of µPC to more complex tasks that require depth remains uncertain.\n- Partial stability fix: The proposed initialisation ensures stability only at the start of training. It remains unclear whether the model remains stable throughout learning, especially for more complex architectures.\n\nClarity\n\n- The proposed method µPC is difficult to locate in the main text on first read and requires a deep dive to understand, despite being a relatively simple change. A clearer structure with a dedicated section for the desiderata and another for the proposed parametrisation would improve accessibility.\n- Figure A27 should be moved into the main text to illustrate the problem µPC aims to solve.\n- Figure captions should be improved to make figures stand-alone—for instance, in Figure 2, the activity Hessian κ(Hz) should be explained (e.g., lower values are better).\n- The paper is generally difficult to read and navigate. While it condenses extensive work into 9 pages, it would help to move the ill-conditioned landscape results (Figures 2–4) to the appendix or combine Figures 2 and 4. These could then be replaced with results that directly reflect the properties µPC targets (e.g., Figure A27), helping reader understand what muPC is.\n\nSignificance\n\n- While the work is significant for predictive coding research, backpropagation still outperforms PC on the evaluated tasks. Exploring learning tasks where PC has advantages over backpropagation could yield more insightful comparisons.\n\nOriginality\n\n- The core change to PC parametrisation is directly drawn from the existing µP literature. Adapting this parametrisation specifically to PC inference dynamics and weight updates could lead to more substantial advancements for predictive coding.", "was_revised": true, "final_justification": "I have read the authors' rebuttal, which has addressed my earlier questions. Although I still have reservations regarding whether this method will lead to improved learning performance on more challenging tasks, I am now convinced that this work makes a significant contribution to the predictive coding literature. Consequently, I have increased my confidence score.", "rebuttal": "We thank the reviewer for their detailed feedback and what we think is a very fair assessment of the work. Below we address the points made by the reviewer one by one.\n\n**Quality**\n\n* We completely agree that the tasks we tested $\\mu$PC on were simple and do not require deep architectures. We further agree that, as noted in the conclusion, it will be important (and exciting) to try to extend, in the first instance, $\\mu$PC to convolutional networks so that performance on more complex datasets requiring depth can be properly assessed.\n* The reviewer points out that $\\mu$PC ensures stability only at the start of training. We see two possible interpretations of this statement. The first is that $\\mu$PC does not explicitly satisfy Desideratum 2 about the stability of the forward pass during training. It is true that there is no theoretical guarantee provided; however, the empirical results clearly show that $\\mu$PC maintains good performance throughout training, suggesting (although not proving) that Desideratum 2 is satisfied. This point was not in the paper for space reasons but will be added to the camera-ready version. The second interpretation of the reviewer's point has to do with results of Section 6 (Theorem 1 and Figure 6) where the correspondence between $\\mu$PC and BP breaks down during training. Here we would like to highlight that if an algorithm does not approximate the BP gradients, it does not necessarily mean that the algorithm cannot lead to stable training dynamics or good performance. For PC, this was clearly demonstrated by Innocenti et al. (2024), who showed that PC in fact performs a completely different gradient update at the inference equilibrium and that, under certain conditions, this can even lead to faster convergence. We revised Section 6 to make this point clear and avoid potential confusion. Having said all this, we would like to emphasise that we do not claim that $\\mu$PC is completely stable. $\\mu$PC clearly does not fix the ill-conditioning of the landscape (at initialisation or during training), and in this sense the parameterisation remains unstable.\n\n**Clarity**\n\nTo address the reviewer’s points about the clarity of the paper:\n* We appreciate that the paper condenses a lot of information and that $\\mu$PC can be difficult to locate. The camera-ready version of the paper will aim to clarify the presentation of $\\mu$PC.\n* We did not include Figure A.27 in the main text because it is not a novel result but, as noted in line 172, a ***replication*** of the stability of the forward pass with depth ensured by the Depth-$\\mu$P and orthogonal parameterisations. Despite the importance of these results to our work, we felt that including this figure in the main text could have led to a misunderstanding of our contributions.\n* We clarified the captions of the figures in the main text to make them more stand-alone.\n\n**Significance**\n\nThe reviewer remarks that backpropagation still outperforms PC on the evaluated tasks. Could they please clarify what is meant by this? Our empirical results on $\\mu$PC clearly show that performance is not significantly different from BP, both during training and at convergence, for both MNIST and Fashion-MNIST (see Figs. 1 & A.16-18). We suspect that this could be due to a misunderstanding of Figures A.17-18, which the reviewer also asks about. To answer the specific questions raised:\n* $\\mu$PC does indeed only work for residual networks (ResNets) because ***Depth-$\\mu$P was in turn developed for ResNets***. We mention this in the main text (lines 82 and 172-3), and a brief explanation is included in the related work section on $\\mu$P (from line 486). Essentially, this is because the forward pass of standard MLPs is hard to stabilise with respect to the depth without skip or residual connections.\n* Figures A.17-18 are ***residual*** ReLU networks trained with $\\mu$PC. We thank the reviewer for pointing out this imprecision and now clarify in the figure titles that these networks have skip connections. As discussed in the Experiments section (lines 210-11), Figure A.17 shows the performance of the same ReLU networks of Figure 1 trained on MNIST until convergence (5 epochs). This is also noted in the caption of Figure 1. Figure A.18 shows the same results as Figure A.17 on Fashion-MNIST. We hope that this helps clarify the significance of our results.\n\n**Originality**\n\nWe completely agree with the reviewer that developing a parameterisation that is adapted to the specific inference and learning updates of PC would likely lead to further improvements. Here we would like to note that this is challenging in part because the PC learning dynamics are hard to characterise far from the inference equilibrium. Moreover, even at the equilibrium, the analytical expression for the equilibrated energy is non-trivial (Eqs. 30-31). Nevertheless, we think that theoretical work would be important and impactful here.\n\n**Questions**\n\nTo address the reviewer’s minor questions:\n* We indeed use the Kaiming/He uniform initialisation for all of the experiments except for those on the random matrix theory of the activity Hessian (Section A.2.3), where (for theoretical purposes) we use a Gaussian with variance scaled by 1/fan_in. This initialisation for SP does not materially affect any the theoretical or empirical results. The reason why we use the Gaussian initialisation in the main text to refer to SP is ***purely for easier comparison*** with $\\mu$PC in Table 1, since we can contrast them both on the Gaussian variance. We thank the reviewer for pointing out this imprecision and now clarify in the main text that SP can refer to any of these initialisations.\n* Regarding the ill-conditioning of the inference landscape, Figure 4 should be compared with Figure A.21, which shows the ill-conditioning of ResNets ***without $\\mu$PC***. Together, these figures show that the ill-conditioning of ResNets does not significantly vary with or without $\\mu$PC. Figure 2 shows results for standard MLPs with no skip connections (please note that the caption makes reference to Figure A.21). The reason for this order of presentation is to highlight the trade-off between the stability of the forward pass and the ill-conditioning discussed in Section 5. Namely, we start with MLPs which are not extremely ill-conditioned but have unstable forward passes. This then lead us to ResNets which can have stable forward passes but become much more ill-conditioned. The camera-ready version of the paper will discuss the significance of the ill-conditioning of the inference landscape and $\\mu$PC in more detail, and we are happy to elaborate on this further.\n\n**Summary**\n\nTo summarise, we agree with the reviewer that the main limitation of our work remains the simple datasets on which $\\mu$PC was tested and that this is one of the most important next research directions. We hope that we clarified the significance of the results by making clear the results of Figures A.17-8. We appreciate the reviewer’s suggestions to improve the clarity of the paper and will aim to incorporate these in the camera-ready version.\n\n**References**\n\nInnocenti, Francesco, et al. \"Only strict saddles in the energy landscape of predictive coding networks?\" NeurIPS (2024)."}, {"review_id": "K1Mg8MsB2V", "reviewer": "Reviewer_qKFu", "summary": "The article addresses a major issue in biological network training: it is well-known that almost all of the biologically plausible approaches suffer from depth-scalability problem. This article proposes the adoption of the Depth-$\\mu P$ scheme, which initializes the network weights based on the width and depth of the network. The Depth-$\\mu P$ scheme was proposed mainly for hyperparameter transfer capability for backpropagation training of networks. The current article wisely adopts this approach to overcome the difficulties of training predictive coding (PC) based biologically plausible networks, with the additional gain of enabling hyperparameter transfer for PC training, as originally devised for BP training. The article starts with a summary of the relevant work and the PC loss description. Then it provides an analysis of the unstability of PC based training based on the analysis of the loss function and the corresponding algorithm. Later, the use of Depth-$\\mu P$ parametrization is proposed to enable stable training and hyperparameter transfers. The article provides numerical experiment results for the MNIST dataset, to demonstrate the successful training of deep networks using PC approach, and the hyperparameter transfer capability.", "questions": "-  Could you clarify why you use a different label/acronym ($\\mu PC$) for Depth-$\\mu P$? The footnote in Page-3 references  Section 4 to clarify this distinction, but when you check Section 4 there is no satisfactory explanation. It is clear that PC loss and the MSE loss of BP are not the same but in terms of the initialization/parametrization ($\\mu PC$) is exactly equal to Depth-$\\mu P$. This, in my opinion, creates confusion about whether there is any change specific to the adoption of Depth-$\\mu P$ to the predictive coding.\n\n- Is this approach extendable to more complex classification tasks?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- It is demonstrated that the proposed parametrization approach can be used to successfully train $128$-depth networks based on the predictive coding loss for the MNIST datset. Scaling biologically plausible neural networks is an important research area, therefore, this result raises the hopes by underlining the importance of the initialization schemes in stable training of deep biological networks. \n- The hyperparameter transfer is the main advantage of the existing Depth Depth-$\\mu P$ based parametrization. In this work, it is demonstrated to be applicable to the predictive coding based neural network training.\n\n## Weaknesses\n- One of the main drawbacks of the article is to assess whether the depth scaling property demonstrated for the MNIST dataset generalizes to other data sets. Even CIFAR-10, CIFAR-100 would be informative for this purpose.\n- The article's presentation requires significant improvement. In particular, the  positioning of the article's contributions relative to the existing work and its novelty are  obscured by the terminology and the presentation style of the article. For example, the use of the $\\mu PC$ acronym to replace the existing Depth Depth-$\\mu P$ causes confusion and  understanding the actual contributions. Many of the figures in appendix are lacking captions and discussions/explanations. In this sense, the article requires a major rewrite which would clarify the main contribution points\n\n## Additional Comments\n- it is better fan_in is defined for clarity. \n-  Table 1 could be placed in the second page, where it is first referenced. \n-  Using an extra symbol (H) for $L-1$ is unnecessarily confusing.\n- zero-shot transfer figures in Figure 1 (left-down corner) are not clearly explained in the Figure caption. I was not able to see referrals to this figure part in the article's discussion.", "was_revised": true, "final_justification": "I believe  the article presents a valuable contribution to  the problem of scalability of biologically plausible networks, by using results from theoretical scaling analysis of networks. However, I am a bit concerned about the restriction of numerical examples to MNIST. During the discussion period, I stated this concern, hoping that the authors could share some preliminary results on a different data set such as CIFAR-10. However, the authors responded that they will share results in the camera ready paper. This is a bit concerning, I believe it should not be relatively hard to generate results for CIFAR-10 dataset during the discussion period. I increased my score to 4, but I still feel not comfortable about the lack of further experimental support for the proposed valuable idea.", "rebuttal": "We thank the reviewer for their feedback and fair summary of our work. Below we address their points one by one.\n\n**Limited experiments** \n\nFirst, we completely agree that the main weakness of our work are the limited experiments testing $\\mu$PC, specifically the range of datasets used. The main reason for this is that training of such deep models is highly computationally expensive on GPUs even on simple datasets such as MNIST, due to both the size of the networks and the sequential nature of PC inference. We would also like to point out that, as discussed in the Experiments section, we also tested $\\mu$PC on Fashion-MNIST classification. Nevertheless, we agree that it would be important for future work to test $\\mu$PC on more complex datasets and architectures, as mentioned in the conclusion. \n\nTo directly answer one related question asked by the reviewer, it seems reasonable to expect that $\\mu$PC should extend to more complex classification tasks given that convolutional neural networks (CNNs) parameterised with Depth-$\\mu$P simply involve a different width scaling. Nevertheless, this needs to be empirically shown, and there are other subtleties related to CNNs highlighted by some previous Depth-$\\mu$P work (Bordelon et al., 2023)—such as how to scale hidden layers outside the residual blocks—that might require further changes to the parameterisation for PC. This is why we think that, as noted in the conclusion, trying in the first instance to extend $\\mu$PC to CNNs is one of the most exciting future research directions.\n\n**Clarification of related work:** The reviewer points out that the paper’s summary of contributions could be better positioned relative to previous work. As noted in the text (line 44), all related work and its relationship with our contributions is reviewed in detail in the appendix (from line 445) due to limited space. The reason why our summary of contributions does not directly mention or relate to any previous work is that all the results presented are novel and go significantly beyond the few existing works on training deep PCNs (reviewed in the related work section).\n\n**Clarification of the $\\mu$PC acronym:** The reviewer notes that the $\\mu$PC acronym causes confusion and misunderstanding of the contributions. This was not at all our intention. We used the acronym $\\mu$PC to encapsulate ***both the algorithm and the parameterisation***. As noted in footnote 3 and as the reviewer correctly points out, this was in part to distinguish the different objectives of PC (energy) and BP (MSE loss) even though the parameterisations are the same. However, it was also to avoid repeating ***PC with Depth-$\\mu$P***, which would have been often necessary to distinguish it from ***BP with Depth-$\\mu$P***. We hope this clarifies the acronym’s usage and now make it more clear in the relevant footnote. In terms of the contributions, we would like to stress that $\\mu$PC is essentially “just” (one version of) Depth-$\\mu$P applied to PC and that it was not directly developed from analysing PC. We think that our wording throughout the paper reflects this, where we never claim that we propose a novel parameterisation and often state that $\\mu$PC reparameterises PCNs using Depth-$\\mu$P.\n\n**Clarification of missing captions and explanations:** The reviewer points out that many of the figures in the appendix are missing captions and explanation. These omissions are deliberate and, we believe, appropriate. For example, we discuss all the figures in the Additional experiments section (A.3) and omit captions whenever the experimental details are the same as those of another figure with a detailed caption (and this figure is always referenced in the title). This is true (for example) for many of the figures plotting the condition number of the activity Hessian over training (Figs. A.8-9 and A.22-26), which share the same setup as Figure 3 in the main text of the paper and only change one variable (e.g. dataset or architecture) that is always clearly stated in the figure title. Many of the Supplementary figures in the appendix (A.6) share this feature.\n\n**Additional comments**\n\nTo address the reviewer’s additional comments:\n* We agree that “fan_in” was confusing and now replace it everywhere with $N_\\ell$ for appropriate $\\ell$.\n* In the writing of this paper, we received mixed feedback as to the positioning of Table 1, with some suggesting (like the reviewer) to place it where it is first referenced (Section 2) while others recommending the Desiderata section. We think the latter is the most appropriate because this is the place where we motivate and introduce $\\mu$PC.\n* We agree that the variables $L$ for number of layers or weight matrices and $H = L-1$ for number of hidden layers can be confusing, especially since we use them both to refer to the network depth (as noted in lines 99-100). We found this impossible to avoid without being mathematically imprecise given that (i) $L$ features in the Depth-$\\mu$P scalings, and (ii) many of the quantities related to the PC activities involve $H$ (as there are $H$ hidden layers of states free to vary). \n* We agree that the zero-shot transfer results in Figure 1 are not explained in the caption; however, we immediately after (last two lines of the caption) refer to the full transfer results for explanation. We changed the wording to make this more clear.\n\n**Summary**\n\nTo summarise, we completely agree with the reviewer that the limited experiments testing $\\mu$PC is the main weakness of our work and that this is an important (and exciting) future direction. In this context, we would like to highlight the extensive range of other experiments performed to arrive at the conclusion that $\\mu$PC could help with training at large depth, especially on the conditioning of the inference landscape and other parameterisations (i.e. orthogonal initialisation). We also hope that we clarified the use of the $\\mu$PC acronym and our contributions, as concerns raised by the reviewer. The strength of our work ultimately lies in clearly identifying problems with the standard parameterisation of deep PCNs and exploring what we call $\\mu$PC as a first step towards a more stable and scalable parameterisation. We strongly believe that the local learning community would likely benefit from our findings. \n\n**References**\n\nBordelon, Blake, et al. \"Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit.\" arXiv preprint arXiv:2309.16620 (2023)."}, {"review_id": "9dV5ZOGwvD", "reviewer": "Reviewer_N2W2", "summary": "This paper achieves stable and decent predictive coding (PC) training of deep fully connected ResNets by applying layerwise weight multipliers. These weight multipliers are motivated by Depth-muP, but not equivalent. The conditioning of the inference Hessian is evaluated as a proxy for good optimization properties, but the experiments show examples where muPC performs well despite ill-conditioned inference Hessian.", "questions": "- **Ill-conditioning in ResNets.** Why is the activity Hessian of ResNets so ill-conditioned? Are residual connections not expected to improve conditioning?\n- **Does ReLU behave more favorably than linear or tanh?** For ReLU, the condition number does not appear to monotonically increase with depth. This is arguably the practically most relevant example. How does this behave at larger depth? This may yield important insights into the desirable properties of large ReLU nets.", "limitations": "Some baselines are missing in the experiments. The employed scaling rule is not quite Depth-muP, thus not fully width-independent. Unlike claims in the paper, the results for $1/\\sqrt{L}$ residual scaling are fundamentally limited to ResNets with shallow residual blocks. The experiments are restricted to MNIST and FashionMNIST. It remains unclear what makes the proposed parameterization stable despite diverging condition number of the activation Hessian.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\nTo the best of my knowledge, this is the first paper to scale predictive coding (PC) to deep ResNets. The paper is written clearly and the figures are neat.\n\n**Weaknesses:**\n\nWhile I like the general idea to enable performant, deep PC training by correcting signal propagation both in forward and backward passes via depth-muP-like desiderata, the concrete realization in the paper is not ideal in several important regards. I think the paper should undergo a major revision to really unfold its full potential. I see the following important weaknesses that should be addressed:\n\n- **Unmotivated scaling choices.** The authors claim to study Depth-muP, but then do not use the correct layerwise learning rate scaling rule if I understand lines 245-246 correctly. This choice is not motivated at all and expected to be suboptimal in terms of transfer properties. In addition, Table 1 is incomplete without specifying both the optimizer and the layerwise learning rate scaling, as SGD and Adam require different layerwise choices. As [24] already derive width-scaling muP for PC, the main novelty here would really stem from correcting depth scaling in a principled way. Is there a reason for deviating from the principled Depth-muP scaling rule?\n- **Unfair comparison with SP.** It is clear that a width-independent LR induces training instability at large width. One naive learning rate scaling that could have been employed is LR= width$^{-1}$, but the practical optimal LR exponent may be larger. Arguably, a fair comparison with SP would use the optimal learning rate at each width and depth. Degeneracies at large scale are to be expected even at the optimal LR, so that Depth-muP is still expected to outperform SP under this fair and more informative comparison.\n- **Usefulness of the condition number of the inference Hessian remains unclear.** The conditioning of the inference Hessian does not always seem to be a good proxy for understanding properties of the training dynamics. It remains unclear how useful it is to plot its condition number. The experiments show that muPC provides a counterexample which performs well despite bad condition number. It appears to remain open what is a good metric for predicting training instability. This could be acknowledged more clearly. Noci et al. (2024) suggests that not the Hessian but the Hessian preconditioned by the layerwise learning rate scaling is the correct metric to study, which is more directly related to the training dynamics and becomes width-independent under muP. The two metrics only coincide under the choice of multipliers under which a width-independent global learning rate can be chosen. I recommend that the authors reconsider their analysis with this metric and would expect that the conflict between Desiderata 1 and 3 is not fundamental but rather an artifact of the wrong metric.\n- **Overstated generalizability.** The authors claim that their work could be extended to convolutional and transformer architectures, but this extension is not so clear. The Depth-muP paper (Yang et al., 2024) clearly states that Depth-muP is only guaranteed to work for block depth 1, so that classical convolutional ResNets and Transformers do not directly apply. In fact, recent papers show that Transformers may require the different residual scaling $L^{-1}$ (Bordelon et al., 2024; Dey et al., 2025).\n\n**Further weaknesses in experimental or theoretical execution:**\n\n- Figure 1 is lacking a baseline: Maybe SP also transfers across depth in this setting. If not with which exponents do its optimal (activity, weight) learning rate combination decay?\n- Scaling the number of inference steps with the number of hidden layers is a potential scaling confounder. Why do you not keep this number fixed? Do the SP experiments also employ this scaling for comparability?\n- Figure 6 only scales to width 64. Larger width would be interesting to understand whether larger width retains correspondence to BP up to larger depth in practice.\n- In Figures 1 and 5, the color scale is lacking scalar values.\n- Writing $\\mathcal{O}$ in the desiderata is imprecise. Not only avoiding explosion with $\\mathcal{O}(1)$, but also non-vanishing activations and their updates $\\Omega(1)$ thus $\\Theta(1)$ is required for muP. Otherwise, many parameterizations fulfill these weaker desiderata, which do not induce HP transfer and maximal stable feature learning in the limit.\n- Theorem 1 only holds at the inference equilibrium and quickly follows from the derivation in [20]. Theory that holds throughout the optimization trajectory could hold important practical implications.\n\n**References:**\n\nBordelon, Blake, Hamza Chaudhry, and Cengiz Pehlevan. \"Infinite limits of multi-head transformer dynamics.\" NeurIPS (2024).\n\nDey, Nolan, et al. \"Don't be lazy: CompleteP enables compute-efficient deep transformers.\" arXiv:2505.01618 (2025).\n\nNoci, Lorenzo, et al. \"Super consistency of neural network landscapes and learning rate transfer.\" NeurIPS (2024).\n\nYang, Greg, et al. \"Tensor programs vi: Feature learning in infinite-depth neural networks.\" ICLR (2024).", "was_revised": true, "final_justification": "During the discussion period, it became apparent that the parameterization used is not equivalent to depth-$\\mu$P (as a function of width scaling), as suggested in the paper (e.g. Contribution 1 or Section 2.1). Since the maximal update parameterization is defined through desiderata of maximal stable updates of each trainable weight matrix, but the parameterization used does not fulfill these desiderata, the framing is misleading. I see the practical contribution of trainability of deep PC networks, but due to severe incorrectness concerns I am unable to recommend acceptance of the current form.\n\nI find the bulletpoints in the author final remarks misleading. My correctness concern primarily critizes the wrong width dependence. muP can be realized with any width-dependent multipliers, but only in combination with the correct layerwise initialization variance and layerwise learning rate scaling. The referenced depth-muP papers may differ in their depth-dependence, but all muP-related papers agree in their width dependence. Even in Noci et al. (2024), the used parameterization only differs depth-dependently, but as a function of width a muP-equivalent parameterization is used for both SGD and Adam. The paper under review, however, uses the wrong width-dependent learning rate scaling for the chosen multipliers, and hence does not study muP, nor depth-muP, but merely some parameterization that works better than PyTorch standard. The differences of the used parameterization to muP and depth-muP in this paper are much more severe than those in the cited papers.", "rebuttal": "We thank the reviewer for their detailed feedback. Below we address their points one by one.\n\n**Scaling choice justification:** The reviewer remarks that we do not use the correct layerwise learning rate scaling rule for Depth-$\\mu$P as stated in lines 245-6. The reason for this is that we could not train as deep PCNs as those tested ***without*** the scaling (of the weights and/or activities). We now make this point clear in the discussion. Nevertheless, as alluded to in the subsequent lines (246-7), it is not clear to us whether this is the version of Depth-$\\mu$P that is used in practice. For example, the two original Depth-$\\mu$P papers disagreed on precisely this issue, with Yang et al. (2024) but not Bordelon et al. (2023) proposing to rescale the learning rate when using Adam. We note that this inconsistency was discussed but not resolved in the review process of Yang et al. (2024). Moreover, as we note in lines 246-7, depth transfer was still achieved in two studies (e.g. Noci et al., 2024) using Adam with no learning rate scaling, on both GPT-2 and vision transformers. Regardless of the “correct” Depth-$\\mu$P version, since we did not derive a novel parameterisation for PC, we simply used the Depth-$\\mu$P scalings that worked best empirically. We now make this point clear in the text.\n\nRelated, the reviewer remarks that Table 1 is incomplete without specifying both the optimiser and the learning rate. This is correct, and these details were omitted because we used Adam for all the experiments with no learning rate scaling, as explained above and in lines 245-6. Again, the reason was that this is the only parameterisation that we found to lead to stable training of very deep PCNs. We now make this clear in the main text.\n\n**Comparison with SP:** The reviewer notes that we did not use a width-dependent scaling for the learning rate of SP, making the comparison with $\\mu$PC unfair. We make two points in response. First, we performed an extensive grid search (line 758) that in practice covered many width exponents given that the networks in Figure 5 had width $N = 512$. Second, and more to the point, the transfer results were not the focus of our work given that we do not derive a novel parameterisation for PC, and there was therefore no reason to expect transfer. This is a rather surprising result that needs to be better understood. We agree that any parameterisation specific to PC should perform a more rigorous analysis.\n\n**Usefulness of the condition number of the inference Hessian:** The reviewer questions the usefulness of this metric based on the results that networks can be trained with PC or $\\mu$PC despite high ill-conditioning of the inference landscape. We agree that these results are puzzling but argue that the inference conditioning has still an important impact, even if in a subtle way. In particular, taken together, many results of the paper show that (close) convergence of the inference dynamics to a solution is ***necessary for trainability*** and, therefore, that understanding of the landscape conditioning is crucial. The clearest demonstration is an experiment where we trained standard deep linear PCNs with “perfect inference” (using the solution of Eq. 4). As shown in Figure A.15, we find that, even without $\\mu$PC, very deep PCNs become trainable in the sense that the training loss decreases monotonically. These results suggest that ***if we could reach the solution*** (which is practically impossible at large depth), the networks would be trainable. (To prevent further misunderstandings, please note that trainability is necessary but not sufficient for good generalisation, as shown in Figure A.15.)\n\nWhy does $\\mu$PC work despite the ill-conditioning? It turns out that ***$\\mu$PC initialises the activities much closer to the solution than standard PC*** (see experiment in Figure A.11). At the same time, we find that $\\mu$PC still needs as many inference steps as hidden layers to obtain good performance (see Figure A.14 for results with one step), showing that it does not initialise exactly at the solution. Together, these results show that (i) close inference convergence is necessary for stable training, that (ii) this becomes hard at large depth because of the ill-conditioning, and that (iii) having a stable forward pass is a way of initialising much closer to the solution. Without an understanding of the inference landscape, these results would be much harder to reconcile. For space reasons, this discussion was not included in the paper but will be added to the camera-ready version.\n\nHaving clarified the importance of the conditioning of the inference Hessian, we agree that an analysis of the weight Hessian as performed by Noci et al. (2024) could be useful and complementary to our study. However, we would like to highlight that it is impossible to disentangle the weight dynamics from the inference dynamics, so any analysis of the former would have to make assumptions about the latter. Indeed, this what was done in Innocenti et al. (2024), who studied the weight Hessian of PC ***at the inference equilibrium***, showing that it has some better properties compared to the mean squared error loss of feedforward networks. However, these properties broke down empirically on deep PCNs, showing that an equilibrium was no longer reached in practice. For all these reasons, we do not think that the conflict between Desiderata 1 and 3 is an artifact but could be an inherent limitation of such local models.\n\n**Overstated generalizability:** The reviewer points out that our claim about extending the work to convolutional and transformer architectures is overstated. Here we were being loose with the phrase “could be extended” and simply meant that Depth-$\\mu$P as developed for CNNs and transformers could be applied to PC in the same way we did for fully connected ResNets. We agree with the reviewer that our original wording can be misinterpreted to overstate our results, so we qualify this and similar statements in the paper to mean that our work can be seen as a first step towards a more stable parameterisation of deep PCNs.\n\n**Further weaknesses**\n\nTo address further weaknesses pointed out by the reviewer:\n* We do not have a transfer plot for SP in Figure 1 because as noted above standard PCNs become untrainable at such depths.\n* The reason why we scale the number of inference steps with the number of hidden layers is that one needs at least $H$ steps for the inference (and therefore weight) gradients to propagate (i.e. become non-zero) to all the layers. The easiest way to see this is to note that at the forward pass all the prediction errors except the last one are zero, since the PC energy collapses to the mean squared error loss. The experiments with SP also employ this scaling of the inference steps, and here it is also worth noting that in practice more inference steps are often used, which is consistent with the point made above that the forward pass of standard PC initialises the activities far from a solution.\n* Networks with $H = N = 64$ were the widest and deepest we could compute the theoretical energy for, due to a naive (i.e. inefficient) computation of the activity Hessian. However, we will add plots to the appendix for small depth and larger width similar to Figure A.31.\n* We omitted scalar values from the transfer results of Figure 1 and 5 for a less cluttered visualisation. They do not materially affect the results.\n* We agree that using $\\mathcal{O}(1)$ is imprecise and now correct the text to mean $\\Theta(1)$.\n* We agree that theory that holds throughout the training dynamics of PC would be important and potentially very impactful. As noted above, this is challenging since the PC learning dynamics are hard to characterise far from the inference equilibrium. Moreover, even at the equilibrium, the analytical expression for the equilibrated energy is non-trivial (Eqs. 30-31). Nevertheless, this is clearly an important direction for future work.\n\n**Questions**\n* The reviewer asks why the activity Hessian of ResNets is so ill-conditioned. This is a good question, especially since it is known that ResNets help with the ill-conditioning of the weight landscape as noted by the reviewer. We have no rigorous explanation for why ResNets lead to more ill-conditioned inference, mainly because the activity Hessian is challenging to study even at initialisation from a random matrix theory perspective (Section A.2.3). Intuitively, however, if one examines the structure of the ResNet Hessian (Eq. 24), one realises that the off-diagonal terms dominate the diagonal ones much more so than without skip connections (Eq. 5). A slightly different way of saying this is that ***ResNets increase the network interactions*** (by adding more connections), and as noted in lines 183-5 a naive way to make the Hessian perfectly conditioned is to remove all interactions.\n* It is not entirely clear to us why the reviewer remarks that for ReLU networks the condition number does not appear to monotonically increase with depth. The only plot suggesting this is Figure 3 where at depth 32 the condition number increases more slowly than at half the depth. We note that the conditioning is >10k compared to the Linear and Tanh plots (>1k), and all other relevant plots on ReLU networks show that the conditioning still grows with the depth ***at initialisation*** with or without skip connections (Figs. 2, 4 and A.21).\n\n**References**\n\nYang, Greg, et al. \"Tensor programs vi: Feature learning in infinite-depth neural networks.\" ICLR (2024).\n\nBordelon, Blake, et al. \"Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit.\" arXiv preprint arXiv:2309.16620 (2023).\n\nNoci, Lorenzo, et al. \"Super consistency of neural network landscapes and learning rate transfer.\" NeurIPS (2024).\n\nInnocenti, Francesco, et al. \"Only strict saddles in the energy landscape of predictive coding networks?\" NeurIPS (2024)."}]

Author rebuttal:

Meta-review: This paper introduces µPC, a parameterisation for predictive coding (PC) networks inspired by Depth-µP, enabling the training of deep residual PCNs (up to 128 layers). The authors show that standard PC struggles with vanishing/exploding inference dynamics at scale, and µPC alleviates these issues, achieving stable training, competitive performance, and zero-shot transfer of learning rates across depths and widths. The work provides both theoretical analysis and empirical results, positioning itself as a step toward scaling biologically plausible local learning algorithms to modern deep architectures.

The strengths of the paper are clear. Reviewers highlighted its strong motivation (aWaW), clear theoretical framing of why PC fails at depth (ijZ5), and novel application of µP scaling to PC, which is an important and original contribution (qKFu). The work convincingly demonstrates that stable training is possible with very deep PCNs, a milestone that had not been shown before (N2W2).

The main weakness, consistently noted by reviewers, is the limited experimental evaluation, restricted to simple benchmarks such as MNIST and Fashion-MNIST. This raises questions about generalisability to more complex datasets. Concerns were also raised about the precise equivalence to Depth-µP (N2W2). During rebuttal, the authors clarified scaling choices, added analysis, and acknowledged limitations, committing to adding CIFAR-10 experiments in the final version.

On balance, despite the limited empirical scope, the methodological advance and thorough theoretical treatment make this an impactful and timely contribution. I recommend acceptance.
Decision: Accept (poster)