Title: $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: predictive coding

mup

optimisation theory

hyperparameter transfer

inference learning

local learning
Abstract: The biological implausibility of backpropagation (BP) has motivated many alternative, brain-inspired algorithms that attempt to rely only on local information, such as predictive coding (PC) and equilibrium propagation. However, these algorithms have notoriously struggled to train very deep networks, preventing them from competing with BP in large-scale settings. Indeed, scaling PC networks (PCNs) has recently been posed as a challenge for the community (Pinchetti et al., 2024). Here, we show that 100+ layer PCNs can be trained reliably using a Depth-$\mu$P parameterisation (Yang et al., 2023; Bordelon et al., 2023) which we call "$\mu$PC". By analysing the scaling behaviour of PCNs, we reveal several pathologies that make standard PCNs difficult to train at large depths. We then show that, despite addressing only some of these instabilities, $\mu$PC allows stable training of very deep (up to 128-layer) residual networks on simple classification tasks with competitive performance and little tuning compared to current benchmarks. Moreover, $\mu$PC enables zero-shot transfer of both weight and activity learning rates across widths and depths. Our results serve as a first step towards scaling PC to more complex architectures and have implications for other local algorithms. Code for $\mu$PC is made available as part of a JAX library for PCNs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper achieves stable and decent predictive coding (PC) training of deep fully connected ResNets by applying layerwise weight multipliers. These weight multipliers are motivated by Depth-muP, but not equivalent. The conditioning of the inference Hessian is evaluated as a proxy for good optimization properties, but the experiments show examples where muPC performs well despite ill-conditioned inference Hessian.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nTo the best of my knowledge, this is the first paper to scale predictive coding (PC) to deep ResNets. The paper is written clearly and the figures are neat.\n\n**Weaknesses:**\n\nWhile I like the general idea to enable performant, deep PC training by correcting signal propagation both in forward and backward passes via depth-muP-like desiderata, the concrete realization in the paper is not ideal in several important regards. I think the paper should undergo a major revision to really unfold its full potential. I see the following important weaknesses that should be addressed:\n\n- **Unmotivated scaling choices.** The authors claim to study Depth-muP, but then do not use the correct layerwise learning rate scaling rule if I understand lines 245-246 correctly. This choice is not motivated at all and expected to be suboptimal in terms of transfer properties. In addition, Table 1 is incomplete without specifying both the optimizer and the layerwise learning rate scaling, as SGD and Adam require different layerwise choices. As [24] already derive width-scaling muP for PC, the main novelty here would really stem from correcting depth scaling in a principled way. Is there a reason for deviating from the principled Depth-muP scaling rule?\n- **Unfair comparison with SP.** It is clear that a width-independent LR induces training instability at large width. One naive learning rate scaling that could have been employed is LR= width$^{-1}$, but the practical optimal LR exponent may be larger. Arguably, a fair comparison with SP would use the optimal learning rate at each width and depth. Degeneracies at large scale are to be expected even at the optimal LR, so that Depth-muP is still expected to outperform SP under this fair and more informative comparison.\n- **Usefulness of the condition number of the inference Hessian remains unclear.** The conditioning of the inference Hessian does not always seem to be a good proxy for understanding properties of the training dynamics. It remains unclear how useful it is to plot its condition number. The experiments show that muPC provides a counterexample which performs well despite bad condition number. It appears to remain open what is a good metric for predicting training instability. This could be acknowledged more clearly. Noci et al. (2024) suggests that not the Hessian but the Hessian preconditioned by the layerwise learning rate scaling is the correct metric to study, which is more directly related to the training dynamics and becomes width-independent under muP. The two metrics only coincide under the choice of multipliers under which a width-independent global learning rate can be chosen. I recommend that the authors reconsider their analysis with this metric and would expect that the conflict between Desiderata 1 and 3 is not fundamental but rather an artifact of the wrong metric.\n- **Overstated generalizability.** The authors claim that their work could be extended to convolutional and transformer architectures, but this extension is not so clear. The Depth-muP paper (Yang et al., 2024) clearly states that Depth-muP is only guaranteed to work for block depth 1, so that classical convolutional ResNets and Transformers do not directly apply. In fact, recent papers show that Transformers may require the different residual scaling $L^{-1}$ (Bordelon et al., 2024; Dey et al., 2025).\n\n**Further weaknesses in experimental or theoretical execution:**\n\n- Figure 1 is lacking a baseline: Maybe SP also transfers across depth in this setting. If not with which exponents do its optimal (activity, weight) learning rate combination decay?\n- Scaling the number of inference steps with the number of hidden layers is a potential scaling confounder. Why do you not keep this number fixed? Do the SP experiments also employ this scaling for comparability?\n- Figure 6 only scales to width 64. Larger width would be interesting to understand whether larger width retains correspondence to BP up to larger depth in practice.\n- In Figures 1 and 5, the color scale is lacking scalar values.\n- Writing $\\mathcal{O}$ in the desiderata is imprecise. Not only avoiding explosion with $\\mathcal{O}(1)$, but also non-vanishing activations and their updates $\\Omega(1)$ thus $\\Theta(1)$ is required for muP. Otherwise, many parameterizations fulfill these weaker desiderata, which do not induce HP transfer and maximal stable feature learning in the limit.\n- Theorem 1 only holds at the inference equilibrium and quickly follows from the derivation in [20]. Theory that holds throughout the optimization trajectory could hold important practical implications.\n\n**References:**\n\nBordelon, Blake, Hamza Chaudhry, and Cengiz Pehlevan. \"Infinite limits of multi-head transformer dynamics.\" NeurIPS (2024).\n\nDey, Nolan, et al. \"Don't be lazy: CompleteP enables compute-efficient deep transformers.\" arXiv:2505.01618 (2025).\n\nNoci, Lorenzo, et al. \"Super consistency of neural network landscapes and learning rate transfer.\" NeurIPS (2024).\n\nYang, Greg, et al. \"Tensor programs vi: Feature learning in infinite-depth neural networks.\" ICLR (2024).\n\nQUESTIONS:\n- **Ill-conditioning in ResNets.** Why is the activity Hessian of ResNets so ill-conditioned? Are residual connections not expected to improve conditioning?\n- **Does ReLU behave more favorably than linear or tanh?** For ReLU, the condition number does not appear to monotonically increase with depth. This is arguably the practically most relevant example. How does this behave at larger depth? This may yield important insights into the desirable properties of large ReLU nets.\n\nLIMITATIONS:\nSome baselines are missing in the experiments. The employed scaling rule is not quite Depth-muP, thus not fully width-independent. Unlike claims in the paper, the results for $1/\\sqrt{L}$ residual scaling are fundamentally limited to ResNets with shallow residual blocks. The experiments are restricted to MNIST and FashionMNIST. It remains unclear what makes the proposed parameterization stable despite diverging condition number of the activation Hessian.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article addresses a major issue in biological network training: it is well-known that almost all of the biologically plausible approaches suffer from depth-scalability problem. This article proposes the adoption of the Depth-$\\mu P$ scheme, which initializes the network weights based on the width and depth of the network. The Depth-$\\mu P$ scheme was proposed mainly for hyperparameter transfer capability for backpropagation training of networks. The current article wisely adopts this approach to overcome the difficulties of training predictive coding (PC) based biologically plausible networks, with the additional gain of enabling hyperparameter transfer for PC training, as originally devised for BP training. The article starts with a summary of the relevant work and the PC loss description. Then it provides an analysis of the unstability of PC based training based on the analysis of the loss function and the corresponding algorithm. Later, the use of Depth-$\\mu P$ parametrization is proposed to enable stable training and hyperparameter transfers. The article provides numerical experiment results for the MNIST dataset, to demonstrate the successful training of deep networks using PC approach, and the hyperparameter transfer capability.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- It is demonstrated that the proposed parametrization approach can be used to successfully train $128$-depth networks based on the predictive coding loss for the MNIST datset. Scaling biologically plausible neural networks is an important research area, therefore, this result raises the hopes by underlining the importance of the initialization schemes in stable training of deep biological networks. \n- The hyperparameter transfer is the main advantage of the existing Depth Depth-$\\mu P$ based parametrization. In this work, it is demonstrated to be applicable to the predictive coding based neural network training.\n\n## Weaknesses\n- One of the main drawbacks of the article is to assess whether the depth scaling property demonstrated for the MNIST dataset generalizes to other data sets. Even CIFAR-10, CIFAR-100 would be informative for this purpose.\n- The article's presentation requires significant improvement. In particular, the  positioning of the article's contributions relative to the existing work and its novelty are  obscured by the terminology and the presentation style of the article. For example, the use of the $\\mu PC$ acronym to replace the existing Depth Depth-$\\mu P$ causes confusion and  understanding the actual contributions. Many of the figures in appendix are lacking captions and discussions/explanations. In this sense, the article requires a major rewrite which would clarify the main contribution points\n\n## Additional Comments\n- it is better fan_in is defined for clarity. \n-  Table 1 could be placed in the second page, where it is first referenced. \n-  Using an extra symbol (H) for $L-1$ is unnecessarily confusing.\n- zero-shot transfer figures in Figure 1 (left-down corner) are not clearly explained in the Figure caption. I was not able to see referrals to this figure part in the article's discussion.\n\nQUESTIONS:\n-  Could you clarify why you use a different label/acronym ($\\mu PC$) for Depth-$\\mu P$? The footnote in Page-3 references  Section 4 to clarify this distinction, but when you check Section 4 there is no satisfactory explanation. It is clear that PC loss and the MSE loss of BP are not the same but in terms of the initialization/parametrization ($\\mu PC$) is exactly equal to Depth-$\\mu P$. This, in my opinion, creates confusion about whether there is any change specific to the adoption of Depth-$\\mu P$ to the predictive coding.\n\n- Is this approach extendable to more complex classification tasks?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper *\"µPC: Scaling Predictive Coding to 100+ Layer Networks\"* presents µPC, a novel parameterisation that enables scalable training of deep predictive coding networks (PCNs) with up to 128 residual layers.\nIt identifies that traditional PCNs face challenges with depth in residual architectures due to ill-conditioned inference initialisations. µPC, leveraging Depth-µP scaling, addresses these issues, enabling stable training, competitive performance, and zero-shot transfer of learning rates across varying network widths and depths.\nBeyond these results, the authors provide extensive analysis of four desiderata for stable parameterisations, showing that satisfying the first (desideratum 1) alone suffices for stable training in residual networks.\nThe additional desiderata and corresponding analyses offer future research directions and tools for enhancing training stability in PCNs.\nOverall, this work marks a step toward scaling local, biologically plausible learning algorithms to the level of modern deep learning systems.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nQuality\n\n- Extensive experimental evaluation: The authors demonstrate stable training of predictive coding networks with up to 128 residual layers on classification benchmarks, supporting their claims with empirical evidence.\n- Clear scaling analysis: They conduct extensive scaling studies, identify two pathologies in standard PCNs, and show how µPC addresses one of them through Depth‑µP parameterisation.\n\nClarity\n\n- Contributions were clearly stated\n- Figures were clear.\n\nSignificance\n\n- Unlocking depth stability for residual PCNs, potentially enabling biologically plausible learning in very deep models.\n- Zero‑shot learning‑rate transfer: Demonstrating that learning rates generalise across network widths and depths is practically valuable for reducing hyperparameter tuning in future PCN research.\n- The proposed techniques likely extend to residual networks with convolutions and other base operations, broadening the paper's impact.\n\nOriginality\n\n- Applying Depth‑µP scaling to predictive coding is a novel approach that bridges recent developments in parameter scaling with biologically plausible learning. While µP has appeared in other contexts, its integration with predictive coding for depth scaling is innovative and opens new research directions for predictive coding.\n\n## Weakness\n\nQuality\n\n- Evaluated on simple classification tasks that do not require deep architectures. Training on MNIST and Fashion MNIST typically does not require more than 2 hidden layers. While the parametrisation improves stability, it remains unclear whether any layers beyond the final few actually learn. Thus, the applicability of µPC to more complex tasks that require depth remains uncertain.\n- Partial stability fix: The proposed initialisation ensures stability only at the start of training. It remains unclear whether the model remains stable throughout learning, especially for more complex architectures.\n\nClarity\n\n- The proposed method µPC is difficult to locate in the main text on first read and requires a deep dive to understand, despite being a relatively simple change. A clearer structure with a dedicated section for the desiderata and another for the proposed parametrisation would improve accessibility.\n- Figure A27 should be moved into the main text to illustrate the problem µPC aims to solve.\n- Figure captions should be improved to make figures stand-alone—for instance, in Figure 2, the activity Hessian κ(Hz) should be explained (e.g., lower values are better).\n- The paper is generally difficult to read and navigate. While it condenses extensive work into 9 pages, it would help to move the ill-conditioned landscape results (Figures 2–4) to the appendix or combine Figures 2 and 4. These could then be replaced with results that directly reflect the properties µPC targets (e.g., Figure A27), helping reader understand what muPC is.\n\nSignificance\n\n- While the work is significant for predictive coding research, backpropagation still outperforms PC on the evaluated tasks. Exploring learning tasks where PC has advantages over backpropagation could yield more insightful comparisons.\n\nOriginality\n\n- The core change to PC parametrisation is directly drawn from the existing µP literature. Adapting this parametrisation specifically to PC inference dynamics and weight updates could lead to more substantial advancements for predictive coding.\n\nQUESTIONS:\nMajor\n\n- Is the proposed method only applicable to residual networks? Figures A.17 and A.18 suggest that deep ReLU networks benefit from µPC. What exactly are these ReLU networks? Do they include residual connections? If not, which initialisation do they use—is it the µP initialisation described in Section 2.1 or the Depth‑µP initialisation proposed specifically for ResNets? Clarifying this will make the scope of the work clearer.\n\nMinor\n\n- In Section 2.2, you state that SP uses a normal initialisation with variance 1/fan_in. However, in your implementation, it appears that you use a uniform initialisation with the same variance, as you rely on equinox.nn.Linear (https://github.com/patrick-kidger/equinox/blob/main/equinox/nn/_linear.py) which uses a uniform initialisation from (https://github.com/patrick-kidger/equinox/blob/main/equinox/nn/_misc.py). Did I misinterpret your codebase? If not, does the choice between uniform or Gaussian initialisation materially affect your results or analysis?\n- You devote a lot of space to the ill-conditioned energy landscape, arguing that this poses a challenge for PC inference. However, from Figures 2 and 4, it appears that µPC results in a significantly more ill-conditioned landscape than standard PC, with κ(Hz) increasing by factors of 1,000 to 10,000. Do you think the improved training stability observed with µPC comes at the cost of slower inference, particularly if you let inference run until convergence instead of fixing the number of inference steps to the depth of the model?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an analysis of standard PC showing that it is susceptible to vanishing and exploding inference dynamics in networks of increasing size. They reveal a trade-off between conditioning of the inference landscape and the stability of the forward pass, and introduce muPC, showing equivalence to BP, helping to bridge the gap in performance at networks of scale.\n\nSTRENGTHS AND WEAKNESSES:\nThis was a very impressive paper that was well motivated with great theoretical contributions. This work has impact not only for general literature in bioplausible learning algorithms, but by extension other forward-only and local learning algorithms and by extension novel hardware and training paradigms. Overall very impressive work!\n\nI see only one notable weakness, which is limited experimental results. It would be interesting to showcase the direct applicability of this approach to problems of greater complexity, although training such models is surely expensive. Authors might also consider broadening the discussion of the implications of this work for other kinds of local learning algorithms, although it is alluded in the introduction.\n\nQUESTIONS:\nThis would be a stronger accept with more diverse experiments and some reflection on limitations and possible extensions of this approach. Motivation could also be developed as well as further comment on the link between this and other local learning algorithms, the possible benefits of training via local learning , limitations of backprop etc. beyond the bioplausibility issue\n\nLIMITATIONS:\nNo. I imagine experimental results would lend further insight into limitations and advantages of this approach. Can we expect such networks to match bp performance exactly , given theorem 1? Are there other caveats worthy of consideration.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}