Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation
Authors: Satoki Ishikawa
Rio Yokota
Ryo Karakida
Venue: ICLR 2025 Poster (2025)
Keywords: deep learning
feature learning
local learning
predictive coding
target propagation
infinite width
maximal update parameterization (muP)
Primary area: optimization
TL;DR: We derive the parameterization of major local learning methods that enable feature learning in infinite-width neural networks and demonstrate its benefits.

Abstract: Local learning, which trains a network through layer-wise local targets and losses, has been studied as an alternative to backpropagation (BP) in neural computation. However, its algorithms often become more complex or require additional hyperparameters due to the locality, making it challenging to identify desirable settings where the algorithm progresses in a stable manner.
To provide theoretical and quantitative insights, we introduce  maximal update parameterization ($\mu$P) in the infinite-width limit for two representative designs of local targets: predictive coding (PC) and target propagation (TP). We verify that $\mu$P enables hyperparameter transfer across models of different widths.
Furthermore, our analysis reveals unique and intriguing properties of $\mu$P that are not present in conventional BP. By analyzing deep linear networks, we find that PC's gradients interpolate between first-order and Gauss-Newton-like gradients, depending on the parameterization.  
We demonstrate that, in specific standard settings, PC in the infinite-width limit behaves more similarly to the first-order gradient.
For TP, even with the standard scaling of the last layer differing from classical $\mu$P, its local loss optimization favors the feature learning regime over the kernel regime.

Published as a conference paper at ICLR 2025
LOCAL LOSS OPTIMIZATION IN THE INFINITE WIDTH: STABLE PARAMETERIZATION OF PREDICTIVE CODING NETWORKS AND TARGET PROPAGATION
Satoki Ishikawa Rio Yokota', Ryo Karakida"
Science Tokyo AIST { sishkaww,, rioyokota| @rio..cre. iir. iisiscctaa. ac. jp, ka rakida. ryoorroosssso.. go. jo
ABSTRACT
Local learning, which trains network through layer-wise local targets and losses, has been studied However, its algorithms often become more complex or require additional hyperpa- rameters because of the locality, making it challenging identify desirable settings in which the algorithm progresses in a stable manner. To provide theoretical and quantitative insights, we introduce the maximal update parameterization (uP) in the infinite-width limit for two representative designs of local targets: predictive coding (PC) and target propagation (TP). We verified that juP enables hyperparam- eter transfer across models of different widths. Furthermore, our analysis revealed unique and intriguing properties μP that are not present conventional BP. By analyzing deep linear networks. we found that PC's gradients interpolate between first-order and Newso--ewtt--lieee gradients, depending on the parameterization. We demonstrate that, in specific standard settings, PC in the infinite-width limit behaves more similarly to the first-order gradient. For TP, even with the standard scaling of the last layer, which differs from classical μP, its local loss optimization favors the feature learning regime over the kernel regime.
alternative to backpropagation (BP) in neural computation.
INTRODUCTION
Deep learning has achieved remarkable performance by building upon the backpropagation (BP) algorithm and developing architectures specialized for it (Rumelhart et al., 1986; LeCun al., 1998; 2015). BP, however, is not always suitable method for more general objectives, such as biologically plausible computation (Lillicrap al., 2020; Bredenberg al., 2024) or efficient distributed computation (Amid et al., 2022). A representative alternative is local loss optimization, type of credit assignment problem, in which loss functions are defined layer-wise, and targets are set locally. The basic formulation involves performing regression target signals at each layer to reduce the global error across the entire network: Predictive Coding networks, usually referred to as PC, generate their targets through the internal dynamics of inference (Whittington & Bogacz, 2017; Song et al., 2020; Salvatori et al., 2023), while Target Propagation (TP) generates them using feedback networks (Bengio, 2014; Lee et al., 2015; Ernoult al., 2022).
In many cases, the use of local losses requires additional hyperparameters (HPs) and their careful tuning, making the algorithm configuration significantly more complicated compared that of BP. For example, PC requires not only the usual HPs, such as learning rate and initialization of weight parameters but also those for the inference phase, such as the initialization of the state and the number of inference sequences. These HPs are primary considerations and have been reported as critical for ensuring stable training behavior (Pinchetti et al., 2024; Alonso et al., 2024; Rosenbaum, 2022). A few analyses have succeeded in providing theoretical intuition for such local learning algorithms by introducing specific conditions or additional corrections that bridge them to classical optimization formulations (Song et al., 2020; Alonso al., 2022; Meulemans et al., 2020). However, such conditions are not always met in practice and may not be commonly shared across the entire family of methods. To develop local learning that is more casily manageable across broader range of settings, it is promising to establish a theoretical foundation that enables the analysis of natural learning dynamics under fewer constraints.

---

Published as a conference paper at ICLR 2025
For standard BP, deep learning theory offers insights into the universal properties of learning (Bahri et al., 2020; Bartlett et al., 2021). A key research focus in this area is understanding learning in the infinite-widde limit, including studies on neural tangent kemel (NTK) and feature learning regimes (Jacot et al., 2018; Chizat et al., 2019; Mei et al., 2018; Bordelon & Pehlevan, 2022b). In particular, Yang & Hu (2021) provided unified perspective the parameterizations that realize these learning regimes and proposed maximal update parameterization (μP) as a unique scaling of HPs, such as random initialization and learning rates, that achieves feature learning in the infinite-width limit. Building on this developing theoretical foundation, we expect to gain universal insight into local leamnnn, which has not yet been systematically analyzed.
In this work, we derive the μP for PC and TP and investigate hyperparameter transfer (the so-called j'Transfer) across different widths. Although juP for SGD has been previously derived, the jP depends on the specific training algorithm, making it necessary to derive jPP for each local learning algorithm. Our contributions are summarized as follows:
While it is known that PC inference trivially reduces to gradient computation of BP under the fixed prediction assumption (FPA), a technical and heuristic condition, there is generally no guarantee that PC will reduce to BP, making it highly non-trivial to identify its jPP We first consider PC with a single sequential inference and reveal the jaP even without FPA (Theorem 4.1). We also empirically verify the μ y'Transfer of learning rates, showing that the optimal learning rate does not depend on the order of width.
Second, for a more general context involving multiple inference sequences, we consider the convergence of the inference phase. We find that, for deep linear networks, we can explicitly obtain the local targets and losses at the fixed point of the inference, which depend inference step sizes (Theorem 4.2). Interestingly. it takes similar form to the Gauss- Newton (GN) gradient, but it can be reduced to the conventional first-order gradient descent the parameterization and step sizes. We find that the eventual gradient
(GD) depending is closer to GD for sufficiently wide neural networks under standard experimental settings with μP. We also confirm that a larger inference step size, identified through this analysis, enhances μ Transfer of HPs.
Finally, we derive μP for both TP and its variant difference target propagation (DTP) assuming linear feedback networks (Theorem 5.1). We reveal distinct property that differs from BP and PC; the feedback network of (DyTP changes the preferable scale of the last layer compared to the usual μP and causes the absence of the kernel regime. In this sense, (D)TP favors feature learning more strongly than other learning methods.
Thus, this study provides solid and qualitative foundation for the further development of local learning schemes in large-scale neural networks in the future.
2 RELATED WORK
Local learning: Most research on local learning stems from the exploration of biologically plausible learning (Lillicrap al., 2020), with PC and TP following this line. As deep learning has evolved, local learning has also begun to focus on large-scale networks, and some models have achieved performances close those trained with BP (Emouul al., 2022; Ren al., 2023). Several algorithms are inherently structured to resemble the BP chain (Akrout et al., 2019) or to estimate first-order gradients (Scellier & Bengio, 2017). In contrast, PC relies on an inference phase, which essentially infers the appropriate activation values for hidden layers, and TP uses feedback network, both of which are quite different from BP and seem to be fundamental designs for using local targets. However, their optimization properties are still not well understood. Alonso et al. (2022) proposed a modified PC as proximal point algorithm (implicit SGD), though it requires additional corrections and adaptive rescaling (Alonso al., 2024). Innocenti et al. (2023) proposed an inference phase computed by a GN method, but it requires a quadratic approximation of the local loss around a special initialization. As discussed in the next section, bridging to such classical optimization requires strong conditions that may deviate significantly from the original purpose and algorithm (Rosenbaum, 2022; Meulemans al., 2020).
Infinite width and μP: While the NTK regime guarantees the existence of learning dynamics in the infinite-wiet width limit and its global convergence, it reduces to just kernel method (Jacot et al.,
2

---

Published as a conference paper at ICLR 2025
2018; Lee et al., 2019). To realize feature learning in the infinitewwitth limit, Yang & Hu (2021) proposed JP, which is a non-trivial scaling of HPs with respect to the width. From a theoretical perspective, this serves as those described by the mean-field regime (Mei et al., 2018) or the dynamical mean-field theory (Bordelon & Pehlevan, 2022b). For more applications, ,P or its extension has been validated across various architectures (Yang et al., 2021; Vyas et al., 2023; Everett et al., 2024). It covers not only the naive first-order gradient but also entry-wise adaptive optimizers such as Adam (Yang & Littwin, 2023) and second-order optimization methods like K-FAC (Ishikawa & Karakida, 2024). There has been little previous work on the anfintt-widtth analysis of local learning. Bordelon Pehlevan (2022a) formulated (direct) feedback alignment and (supervised) Hebbian learning using dynamical mean-field theory, which are rather close to BP.
parameterization that enables the dynamics of feature learning, such
3 PRELIMINARIES
In this section, we summarize local learning and μP in an I-layer fully connected neural network f:
h1 = Φ (u), i1 = W (1=1....,
(1)
where Wi E Mr are weight matrices, h4, un E IR MiNN are activations and is the number of data samples, independent of the order of width M. We set the width of the hidden layers to Mi- = M for (I = ,......---- L- 1) for simplicity. To keep the notation concise, for non-linear networks, set Mr. 1; however, we can easily generalize to Mr. (()). The activation function o(-) is usually assumed to be differentiable and polynomially bounded for some theoretical reasons within the μP framework (Yang & Hu, 2021).
3.1 OVERVIEW OF LOCAL LEARNING
3.1.1 PREDICTIVE CODING
Predictive Coding (PC) updates both the states and weights to minimize the following free-energy
function (Whittington& & Bogacz, 2017; Song et al., 2020; Salvatori et al., 2023):
L-1
F(u, W) -cC(y WLd(vL-1)) +
2 |lm Wid(
L denotes a loss function, and both mean squared error loss and cross-entropy loss are allowed in theory and experiments unless explicit assumption is stated. To distinguish the internal state from the forward signal propagation ui we denote this state as vt. Although this algorithm was originally derived from the variational Bayes formulation, it has heen extended beyond the scope of the original framework, aiming instead to develop inference computations that work more effectively in practice. PC is composed of two phases: inference phase, in which the per-layer states it are updated and learning phase, in which weights W; are updated. Its update rule for the inference phase is given by
OF
01 = U1 du
= Uts - net,s + 74100 (vi,s) o WiT1et+1 (I < L),
(3)
where we define et,s = 01,s Wid(ve-1,s) and the Hadamard product. From eq. (3), 7 can be regarded as step size for the inference phase. The update rule for the learning phase is given by
OF = Wi. + m
W = Wt TIl aw,
Note that the inference time index s and the parameter update index are distinct with resetting to at each f. We usually omit the step size in Eq. (S.6) in implementation. Generally, weights are updated after multiple inference steps, while the incremental version of PC (iPC), which updates the weights after just single inference, has also been proposed (Salvatori et al., 2024b). The internal state can be updated simultancously across all layers or computed sequentially in a specified order. In the first part of the next section, we focus on the Sequential Inference (SI) method, where e,,, is computed sequentially by propagating from the output layer to the input layer. For more details on this difference, Algorithm in the Appendix.
3

---

Published as a conference paper at ICLR 2025
Empirically, to improve trainability, PC often relies assumptions that are either rational or, at times, unrealistic. One such reasonable assumption is the initialization method for vi,., which is used to improve the convergence (Song et al., 2020; Alonso al., 2022; Rosenbaum, 2022):
Technique (i): Forward initialization (F-ini). At each training step t, 20, is initialized such that 01,0 W1, which ensures e1,0 = 0.
Generally, the gradient computation of PC does not match that of BP. However, under F-ini and SI, it reduces to BP by adopting the following rather technical assumption (Millidge et al., 2022b; Rosenbaum, 2022):
Technique (ii): Fixed prediction assumption (FPA). Replace (v--1.s) with Φ (0--10) during the inference phase.
Under FPA, the inference is given by Ct (1 7 + n+i d' (b1,0) W4410+11. By substituting F-ini, one can easily verify that this sequential inference computes V C. In Section 4, we reveal that the following scaling of yy, with respect the width M plays fundamental role in characterizing the feature learning of PC and a parameterization that enables stable learning even without such heuristic techniques:
7L >
(5)
with an exponent 7r. and uninteresting constant provided in the extended related work (Appendix.A.1.2))
A more detailed overview of PC is
3.1.2 TARGET PROPAGATION
In target propagation (TP), hy, = hr - Vn is propagated through the feedback network, which generates local targets hu as follows:
h== gr((u+1); (xx)) ((ux)) L=1,,..,-1))
(6)
where Qi RM 1xM are weight matrices, v-)- is activation function of the feedback network. We also analyze the Difference Target Propagation (DTP), a variant of TP, whose definition is provided in the appendix. The feedback network is trained minimize the following reconstruction loss:
Cree 1l (()bb--))-h--1 2
(7)
where fi(x) = <(Wx)) TP updates the weights W; to minimize the following local loss ||ea
|| /211222 The gradient of this local loss provides the update rule for the learning phase as Wi,++1 = W.,- - "W, o eihi For a so-called invertible network, TP computes the Gauss-Newton Target (GNT), i.e., eGGTT ( + p where d; V uL is the BP signal (Meulemans et al., 2020) and er, y h2 is the error vector. Note that the assumption of the invertible network restrictive because the invertible network requires invertible activation functions, regular weight matrices, and the training tp converge to the solution of g (h(41) = fit (he+1) ( = W o (h(+1) For general networks, (DYTP does not necessarily lead the GNT.
Remark on a connection between PC and TP. Some previous studies have argued that PC yields GNT-like solutions, and thus be connected to TP (Alonso al., 2022: Millidge et al., 2022a). These works attempt to gain an intuitive insight from the fixed point equation for each layer:
h = (WII W + 141//11) (WiT hi41 + (iihi))
(8)
where hf means ((r/)) For 7+1/7u & 1, we approximate hj ~ Wiinhigs If we multiply this approximation across layers, the naive expectation is that h7 I W!et. which corresponds to the GNT for linear networks. Thus, we can intuitively see that the PC may be linked to the GNT, although its exact connection requires careful limit operations across layers. Additionally, taking the limits 1/1111 < 1 for all layers means the exponential decay of 7 with depth. raising concerns regarding its practical relevance. For ?1 = 1, Innocenti et al. (2024) has recently derived an explicit formulation of the free energy at the fixed point using an unfolding calculation of a hierarchical Gaussian model. This formulation shows that the obtained gradient differs from that of the exact GNT, supporting the idea that the connection to GNT would be weak.
'The final gradient d.F.dW /FW is equivalent to the special case of K-FAC (Martens & Grosse, 2015) where the preconditioners are applied only the backward signals.
4

---

Published as a conference paper at ICLR 2025
Table 1: Parameterization for weight initialization scale: by and learning rate scale: e₁. Predictive
Coding (PC) with % 0 reduces to SGD's jlP, while one with 71. = reduces to pP for Gauss Newton Target (GNT). (Target Propagation) has the distinctive property of by 1/2.
Layer
SP (Default)
SGD (2021)
GNT (2024)
PC (New)
TP (New)
Input
(0
(0,-1)
(0,0)
(0, -0,-72- 1)
(0, 0)
Hidden
(1/2,
(1
(1/2,1)
(L) (1,1)
(1/2, (1/2,1)
Output
(1/2.0)
(1,
(1,1)
3.2 s&P AND LEARNING REGIMES
The paraaaarttttiinnooonnn {ar, by, chese determines the scaling of weights and learning rates at initialization. It scales the parameters by width as follows ( (Yang & Hu, 2021):
W w//M",
wi ~ N'0, /²/ ) = 1//M
(9)
(uP and its conditions: Consider the temporal change of u by the parameter update:
Amc. == W1.t W1.0 θ (1/M"),
(10)
where (-) denotes the order with respect to the width and x = (M*)) means v
for x E IR M The training dynamics and parameterization are referred to as stable when 41,0 neither vanish nor explode as the network width increases and Ahy do not explode as the network width increases (Definition A.2). Yang & Hu (2021) introduced the following conditions and characterized jPP as a unique stable aibeeearamtttiiottionn under them:
Condition 3.1 (W, updated maximally). AW O(1) where AWL, == Wi. Wi.0- Condition 3.2 (W,, initialized maximally). Wr.oAur.-1, = O(1)
These conditions imply learning regime, the previous work refers ri<r. > and rr. as the kernel regime. The NTK parameterization corresponds to the kernel regime with TıKL = 1/2 Note that the original derivation of the parameterization that satisfics the above conditions is based on the first (infinitesimal) one-step update of the parameters (Yang& & Hu, 2021; Ishikawa & Karakida, 2024) (see Section A.2). Our work also follows the same approach.
0 for all layers and feature learning. In contrast to this feature
uP for Gauss-Newton Target: The following work has recently derived the μP scaling, including both first-order and second-order optimizations.
Proposition 3.3 (Ishikawa & Karakida (2024)). Consider the first one-step update by the GNT: Wí,1 = Wio-mh (W +pl) (ndiag((r))))) where & = V ur and e,, = -h,- In the infinite-widht limit, this update admits the μP for feature learning at
{
A CB b 0, bieKeL 1/2, by 1,
0icieL CB. 01
(11)
where 0, = 2a, + Ct. We obtain uP of SGD for eg 0, and that of GNT for eB =
More precisely, we can also allow by 2 1 for the feature learning regime. However such initialization reduces the case of by 1 in the next parameter update. Thus,
summarize b, 1. The scaling of by = 1 implies that a smaller initialization required compared to the standard parameterization (SP), which is PyTorch's default, for sufficiently wide neural networks. It can also be immediately verified that we can set = 0 due to shift invariance without loss of generality. In Table we summarize the yP from previous work and our results obtained in the following sections.
FEATURE LEARNING OF PREDICTIVE CODING
4.1 μP OF PC WITH SINGLE-SHOT SEQUENTIAL INFERENCE
As noted in section 1.1.1, , PC involves such techniques as F-ini, FPA and SI, which must be clearly distinguished when deriving the yP. It is well-established that when F-ini, SI, and FPA all assumed,
5

---

Published as a conference paper at ICLR 2025
SP
muP for SGD
SP
muP (%
0)
muP (fu -1)
Ted Acc (w PPA)
Test Ane. (we/o FPA)
128
60
256
1024
2048
4096
-2. 90.0-17.5
20
-15
22.5 20.0
-15
-10
15
10
8192
log2 LR
loga LR
log2 LR
Jogg
LR
logg
LR
Figure 1: /aP enables the transfer of learning rates across widths. (Left) PC reduces SGD when F-imi, FPA, and SI are applied. In fact, using the μP of SGD, learning rates are successfully transferred across different widths. (Right) Even without FPA, our μP of PC also allows μTransfer across widths. In this case, inference is performed only once, and the difference in test accuracy between 72 = 0 and 7 is small. Both figures show results with 3-layer MLP on FashionMNIST.
PC reduces to the gradient computation of BP, and the jP matches that of standard BP. Figure 1 shows that when F-ini, FPA, and SI are applied, the yP of BP can be directly transferred PC and leads to learning rate transfer across width. However, this may not hold for general PC and BP as there is no guarantee of their equivalence. To explore this, we first remove FPA. Although initialization (F-ini) and sampling (SI) are inherently arbitrary, the justification for FPA is unclear from both machine learning and biological perspectives. In PC without FPA, find the μP follows:
Theorem 4.1 (μPP for PC (informal)), Let the inference step sizes be " ((1) and 7 > /M with positive constant y/. Consider the first one-step update of the learning parameters after a first single-shot SI with F-ini. Then, PC admits the uPl for feature learning
{ 0 -7t.- -1, 014tet -7r 0, 01-1=1, (0: 2at ct)
(12)
b = 0, bier = 1/2, b1. 1.
Rough sketch of the derivation. Section B.1 of the Appendix presents a detailed and comprehensive derivation. It is based on the perturbation approach, which applies to general networks with nonlinear activation functions. This method inspired by the previous work that derived the jP by evaluating Conditions 3.1 and 3.2 using the perturbations, such dy (AWi, , w/=0 (()) This allows
for systematic and transparent derivation. In PC, we extend the perturbation argument to the inference step size and require
(AW. (WWi,,WWW-LL = 0(1).
(13)
which is an example of Condition 3.1 for the hidden layer. Under the assumption of F-ini (ci,o = 0),
by putting ój = V., " (1 < L.) and 99 WLLL-100 we obtain "4,1 41,0 = - -II 76 and
e1,1 (w,o II 1-141 7dd) $W,, II (r))-
(14)
For the hidden layers, the perturbation term (13) becomes M (0 + &' (((Wnur),))
Wi W ho1o and we obtain 6 + 7z - 1 + (az + by) 0. We can similarly evaluate the other layers. The last condition. by 1, comes from Condition 3.2. We can derive the NTK parameterization of PC in the same way.
As Figure demonstrates, the obtained μP supports ,/Transfer in PC without FPA. Note that μ Transfer defined as satisfying both conditions: the optimal learning rate can be set independently of the order of width, and the empirical rule that "wider is better' holds (Yang et al., 2021). This means that the optimal hyperparameters tuned for smaller-width models can be effectively re-used larger- models. Additionally, consistent with previous work, we observed the empirical rule of "wider is better" in yuP (Yang et al., 2021), where test accuracy improves as the network width increases. The derivation of μP through one-step update can be immediately generalized to cross-entropy loss in the same way as for μP of naive gradient descent. Thus, p/Transfer can similarly be observed for cross-entropy loss, shown in Appendix (Figure $.5

---

Published as a conference paper at ICLR 2025
1.00
1.0
1.00
Cos Sitn with BP Grad
1.00
width
Cos Sim
128
0.98
0.98
256
0.90
0.5
512
0.7
0.96
1021
0.85
2048
(a)
Mr
27
(b)
26
Mi
-1.0
-0.5
0.94
4096
8192
BP Layer1
CN Layer1 -
BP Layer2
CN Layer2
71
Figure 2: (Left) Comparison of gradients with the analytical solution of a linear network. We measured the cosine similarity between the gradients analytically derived in Theorem 4.2 and the BP gradients GN gradients for each layer. (a) As Mr. approaches 1, PC's gradient converges to BP's. (b) As Mj increases, the PC gradient approaches BP's. (c) 7T. = 0 yields gradients closer to BP gradient (which means SGD in this experiment) compared 7L -1. (Right) In nonlinear MLP, PC's gradient also approaches BP's when 7 O.
Thus far, when considering the parameter gradients. it appears that 7r. as free parameter can be absorbed into the learning rate, allowing the feature learning dynamics to remain stable. However, as the following analysis shows, YL modifies the preconditioning of the computed gradients, which may influence wTransferr of both YL itself and the learning rate. The analysis also demonstrates the validity of setting W ((1)
4.2 ANALYSIS WITH LINEAR NETWORK
In the previous section, we derived the μP under the assumption that inference is performed only once using F-ini and SI. However, in practice, the inference phase typically involves multiple update steps. To address this, we found that it is possible explicitly derive the following general solutions (fixed points) of the inference phase for lincar networks. See Section B.2 for the derivation.
Theorem 4.2. Suppose an I-laayered linear network and a mean squared error loss fly, W WI.UL.1), and put oj = 27 - Woit- with * denoting the fixed point of the inference process (3). The following holds:
6 24 WT
L141 O, (W y). C,(W) Σ
7 WL.WT.
(15)
7
7-1
[-1
of Win + ( (T
WT
Σ 9-1
+ -y - ff).
(16)
1a441
and et y WLef.-1 (I C, (WL.1 y) where WL W WiWL-11.h
From this general solution, we can also confirm the following property of the infinite width.
Corollary 4.3. Suppose the setting of Theorem 4.2, MeL O(1) and the random weights given by μP i.e. ag + by = 1. In the infinite-widthh limit, the PC's gradient reduces to the first-order GD for 7r. (1). For 7r. ((M), the preconditioner part C.yy remains of order 1.
The exact solutions eï provide much clearer insight into the gradient computation compared to Eq. (8), which was previously argued but not explicitly solved. First, it becomes evident that PC does not generally coincide with GNT. Consequently, PC is also generally different from TP. In fact, PC coincides with GNT only for the input layer in shallow network (i.e., L 2), where the update vector for PC corresponds to a GNT update with damping term. Although PC does not entirely coincide with GNT, it noteworthy that the scaling of ci in juP for Tr = matches that of GNT. In contrast, for 7t = 0. the PC's gradient aligns with the first-order GD. Because the preconditioner part scales as C.2- = (1)M) in the infinite-width limit, we observe that ef = 8; which reduces to the first-order GD. Naturally, μP matches that of first-order GD in this case. Intuitively, 7r. reflects how effectively the last layer's 's error propagates downward. Like linear regression, the last layer's 's inference solution inherently involves inverse matrix. Thus, when Y is larger than other 7 values, the last layer's 'sss representation is computed first and propagated downward, making the solution resemble GNT.
7

---

Published as a conference paper at ICLR 2025
Loss (Inference)
Loss after inference
T1.
-1.8
Test Acc [CNN)
With F-ini
Without F-ini
-1.6
muP
80
60
(% 0)
-1.2
mul!
-1.0
(5p. -1]
2-3
NTK
-0.4
SP
2.5
5.0
7.5
10.0
12.5
-0.2
2
2
2 222
1ogg Mj
M
M6-1
Figure 3: (Left) YL -1 steadily reduces the local loss width increases. We observed the inference loss in randomly initialized linear network for various Tt.- For "i. =1 the inference loss consistently decreases with increasing width. (Right) The "wider is better" frend holds for uP with OL = 1. With F-ini, this trend holds for μP regardless of the in value. However, without F-ini, the benefits of 7 -1 become particularly prominent.
muP (9z muP (9z 1)
Ah (input) Ah2 (hidden)
width
Test Ace (CNN)
80
1.00
0.75
16
60
32
16
0.50
64
32
0.25
128
64
20
400
20
log, LR
loga
Loga LR.
Epoch
Epoch
Figure 4: (Left) pP can transfer the learning rate across widths (without F-ini). We trained a 3-layer CNN on FashionMNIST with 100 inference iterations. Without F-ini, the stability of the inference becomes more crucial. As a result, unlike the single-shot SI with F-ini shown in Figure 1, the stability provided by 72 -1 becomes critical. Note that additional experiments under different settings, including those with VGG5 (Figure S.3 ) and cross-entropy loss (Figure S.5 ), are presented in Section 1.1.2 of the Appendix. (Right) Ah remains consistent across widths during training. We confirm that the condition Ah- = ((1) required by μP holds throughout the training.
Second, the order of eî in the analytical solution for the linear network matches the order of e, as derived in Theorem 4. Therefore, this theorem implies that in linear networks, the μP of PC would remain unchanged regardless of the presence of F-ini or the number of inference iterations. Moreover, as proved in Section B.2.3, the orders of et and 61,1 align only when Y< 6(1). In practical settings with multiple inferences, is desirable for the pP to be consistent both after single inference and after the inference has fully converged. Therefore, setting 7KL @(1) is reasonable.
Additionally, we found that the dimension of the last layer plays a key role in determining the similarity between PC and BP. According to the solution for linear networks, when M,. 1, the PC's gradient aligns with GD. Figure shows numerical results confirming that for Mr = 1, the gradient direction always corresponds to GD, and for M, >> M2 @(1). the gradient approaches GD as well. We observed that both GN and BP get much closer to each other for sufficiently large widths. In other words, detailed view of the cosine similarity at the large width is shown in Figure 2(c). This result seems reasonable because in the context of second-order optimization, it has also been reported that GNT tends to collapse into an identity matrix owing to damping (Benzing, 2022). In summary, while PC's gradient switches between first-order GD and GNT depending on the parameterization, is important to highlight that GNT behaves similarly to GD in the infinite-wide width limit.
when realize GNT by setting 72 -1, it has a quite close direction.
As minor extension, we can also analyze the nudge-type loss of PC defined by Eq. (S.10) (Alonso et al., 2022; Millidge et al., 2023; Pinchetti et al., 2024). In this case, the damping term 7 in Eq. (15), replaced by (1 T.. Thus, the dependence on the parameterization remains essentially the same as that of the naive PC. Further discussion nudge-type PC can be found in Appendix B.2.4.
4.3 STABILITY OF INFERENCE PHASE
To ensure feature learning in SGD, the μPP framework requires stable activations, ie., Auiet. t. = O(1). It seems natural apply this requirement to the inference phase of PC. That is, let suppose u<L.s
8

---

Published as a conference paper at ICLR 2025
SP
muP (f 0) muP (9 - -1)
SP
muP (9 0) muP (fc -1)
with 7/=
with 7== with 71=1
with y=1ee2
with x= x=1e=2
with x=1e=2
width
Test Ace (MLP)
128
60
256
512
1024
20
2048
10
4096
1og2 7/
log2 7/
logg 7/
log2 7
loga 7/
logg 7/
8192
Figure 5: uP with 7t. = -1 performs consistently well, regardless of m. When is small (n 0.01), uP with 7.. performs poorly, while yP with 7.. -1 shows significantly better performance. This difference is likely due to slower inference convergence in μP with 7 0. For larger values of ? (n 1), both μP configurations exhibit high accuracy. However, for pP with iz = 0, 7 does not transfer effectively across widths, whereas μP with 722 demonstrates the successful transfer of 7/z across widths.
varies by (11 during the inference. Note that in Eq.(3) at L-1, 1, the feedforward signal from the lower layer 71-16--1,s (1), and the error feedback from the last layer 'L (u W eL.e θ (1/MSx+tb). Both terms should be of order (1) for the inference successfully merge both feedforward and feedback signals. When b,. can expect the local loss in the last layer é to decrease most prominently during the inference. Additionally for 7KL (1)) the inference remains stable for layers / I. 1. Empirical results in Figure 3 (left) confirm that when 5n -1, the inference loss decreases consistently as the width increases, verifying that the "wider is better" hypothesis holds even in inference. This facilitates the
this condition requires 7.. ( and
hyperparameter transfer of 7 for the inference dynamics.
We also observe the benefits using ÿ convergence of inference usually deteriorates for SP, making inference stability especially critical in this scenario. As shown in Figure (right), the "wider is better" trend holds with F-ini regardless of 7z. However, without F-ini, this trend holds only when it. = -1. Figure 4 demonstrates that the Transfer the learning rate holds for 5L -1. Additionally, Figure indicates that 72 = - is also preferable from the perspective of Transfer of y2.
for the parameter updates. Without F-ini, the
5
FEATURE LEARNING OF TARGET PROPAGATION
5.1 μP OF TP
As overviewed in Section 3.1.2, TP reduces to GNT in the highly restrictive case invertible networks. However, TP is not equivalent to GNT or BP in general cases (Meulemans al., 2020; Emoult et al., 2022). While TP involves two networks trained using different manners, and one may feel it challenging assumption that the feedback network uses a linear activation function v,, we can systematically derive pP for both TP and DTP.
obtain stable parameterization for learning, we demonstrate that, under the
Theorem 5.1 (pPP for TP and DTP (informal)). Consider a linear feedback network. The forward network is allowed to have nonlinear activation functions. After the first training phase of Qt. take the first one-step update of W. Then, obtain uP follows:
f C1 0,
CieLEL = 1, eL =,
(17)
b1 = 0, bieie== 1/2, bu 1/2.
The derivation presented in Section C. Note that the linear feedback network has trained weights in pseudo-inverse form, that is, Qi = h (h, by + - Stable parameterization also be discussed for the training of the feedback network. For further details, see Section C.2.
As demonstrated in Figure 6, using the μP for TP results in the Transfer appropriately across widths. Furthremore, Figure S.14 in Appendix tracks Ah; during training. μP, Ah; remains consistent across different widths, whereas in SP, Ah; either diverges diminishes the width changes.

---

Published as a conference paper at ICLR 2025
SP
DTP
param
80
0.8
SP
Test Anc
70
0.8
mal
(TF)
muP
(SGD)
60
2048
4096
10g
0.5
(GNT)
-2.5
2.3 6.0
0.0
2.5
8192
21
loga LR
LR
16381
wiith
Figure 6: (Left) μP can transfer the learning rate across widths in TP. (Right) TP does not have kernel regime. We measured Wμ = logaz
across different parameterizations following Everett et al. (2024). In the infinite- width limit, wi converges to a. Therefore, in TP, where wr. remains fixed at 1/2 even as the width increases, the kernel regime disappears.
5.2 DISAPPEARANCE OF THE KERNEL REGIME
is notable that jP in the previous work on the gradient methods requires b, = in TP, juP requires dy = 1/2. For the usual gradient methods, stable parameterization with b,. = 1/2 leads to the kernel regime. This raises the question: does a kernel regime exist in TP? Interestingly, in TP, the kernel regime disappears (see Corollary C.I for the details).
Rough sketch of derivation. Condition 3.2 must hold to achieve stable learning in the hidden layers. Note that this condition is required in both the feature learning and kernel regimes. By expressing AhL-1- = ((1/M") we obtain
0g + by +
(18)
When the inner product Wr follows the Law of Large Numbers (LLN), follows the Central Limit Theorem, a = 1/2 (Everett et al., 2024). Additionally, to prevent the output of the last layer from exploding. it is necessary that 1µ O(), that is, at + 79 2 1/2. Consequently, r S a 1/2. In BP, the dependence between Wy and Ah results in a = 1 by the LLN. We have r S 1/2, allowing for the kernel regime. In contrast, in TP, updating the feedforward network weights does not induce dependence between WL and Åh leading to = 1/2. This because the gradient is computed based on the feedback weight Qī fr.- (h lr. r. + pl) [
and when
rather than W,. Consequently, S 0 and the kernel regime cannot be achieved in TP.
Figure empirically confirms a = 1/2 in TP. TP seems be the first example in the infinite-width limit where ,y 1/2 induces feature learning.
6 CONCLUSION
In this work, revealed μP for local loss optimization that can effectively scale toward the infinite width in a stable manner, supported by our analysis of linear networks. Our study covers two of the most fundamental settings: the local targets computed during the inference phase (i.e., PC) and the feedback network (i.e., TP). Although neither method generally reduces to BP or GNT, making gradient computation non-trivial, identified the μP and highlighted its intriguing properties, such as the gradient switching depending on the parameterization and the disappearance of the kernel regime. Additionally, we empirically confirmed that the derived jaP facilitates hyperparameter transfer across widths.
Limitation and future direction. The derivation of μP assumes a one-step gradient and linear networks, although this prerequisite is not unique to our work (Yang & Hu, 2021; Yang et al., 2024). Ensuring the existence of feature learning dynamics for more general steps in the infinite width limit would require the development of a tensor program. However, handling the dependencies between variables that differ from standard BP, such as those arising from the inference phase and feedback pass, is non-trivial and presents interesting direction for future research. Additionally, would also be valuable to explore the learning dynamics of local learning and its convergence properties by extending the infinite width theory or further analyzing linear networks. We believe that understanding the universal behavior of large-scale limits will provide a foundation for the development of more effective algorithms.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work supported by JST FOREST (Grant No. JPMJFR226Q) and JSPS KAEKENHI (Grant Nos. 22H05 16, 23K16965). S.I. and R.Y. also acknowledge support from JST CREST Grant Number JPMICR2112.
REFERENCES
Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas Tweed. Deep learning without weight transport. In Advances in Neural Information Processing Systems, 2019.
Nicholas Alonso, Jeffrey Krichmar, and Emre Neftci. Understanding and improving optimization in predictive coding networks. In AAAI Conference on Artificial Intelligence, volume 38, pp. 10812-10820, 2024.
Nick Alonso, Beren Millidge, Jeffrey Krichmar, and Emre Neftci. theoretical framework for inference learning. In Advances in Neural Information Processing Systems, 2022.
Ehsan Amid, Rohan Anil, and Manfred Warmuth. Locoprop: Enhancing backprop via local loss optimization. In International Conference on Artificial Intelligence and Statistics, pp. 9626-9642.
PMLR, 2022.
Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam Schoenholz, Jascha Sohl- Sick-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11()):011-528 2020.
Peter I. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta Numerica, 30:87-201, 2021.
Bengio. Deriving differential target propagation from iterating approximate inverses. MIXiv:2007.15139 2020.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. MeXiv:1407.7999 2014.
Frederik Benzing. Gradient descent on neurons and its link to approximate second-order optimization. In International Conference on Machine Learning, pp. 1817-1853, PMLR, 2022.
Blake Bordelon and Cengiz Pehlevan. The influence of learning rule on representation dynamics in wide neural networks. In International Conference Learning Representations, 2022a.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. In Advances in Neural Information Processing Systems, 2022b.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning improve neural scaling laws. arXiv:2409.1788 17858, 2024.
Colin Bredenberg, Ezekiel Williams, Cristina Savin, Blake Richards, and Guillaume Lajoie. For- malizing locality for normative synaptic plasticity models. In Advances in Neural Information Processing Systems, 2024.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, 2019.
Maxence M Ernoult, Fabrice Normandin, Abhinav Moudgil, Sean Spinney, Eugene Belilovsky, Irina Rish, Blake Richards, and Yoshua Bengio. Towards scaling difference target propagation by learning backprop targets. In International Conference on Machine Learning, pp. PMLR. 2022.
Katie E Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter J Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jachoon Lee, and Jeffrey Penning- ton. Scaling exponents across parameterizations and optimizers. In International Conference on Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2025
Karl Friston. Learning and inference in the brain. Neural Networks, 2003.
Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences, 2005.
Mario Geiger, Leonardo Petrini, and Matthicu Wyart, Perspective: A phase diagram for deep learning unifying jamming, feature learning and lazy training. arXiv:2012.15110 2020.
Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 1987.
Francesco Innocenti, Ryan Singh, and Christopher Buckley. Understanding predictive coding as a second-order trust-region method. In ICML Workshop Localized Learning. 2023.
Francesco Innocenti, El Mehdi Achour, Ryan Singh, and Christopher I. Buckley. Only strict saddles in the energy landscape of predictive coding networks? In Advances in Neural Information Processing Systems, 2024.
Satoki Ishikawa and Ryo Karakida. On the parameterization of second-order optimization effective towards the infinite width. In International Conference on Learning Representations, 2024.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances Neural Information Processing Systems, 2018.
Ido Kanter and Haim Sompolinsky. Associative recall of memory without errors. Physical Review A. 35(1):380, 1987.
Ryo Karakida and Kazuki Osawa. Understanding approximate fisher information for fast convergence of natural gradient descent in wide neural networks. Advances in Neural Information Processing Systems, volume 33, pp. 10891-10901, 2020.
Axel Laborieux, Maxence Emoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation to deep convnets by drastically reducing its gradient estimator bias. Frontiers in Neuroscience, 15:633674, 2021.
Yann LeCun, Léon Bottou, Genevieve B OIr, and Klaus-Robert Müller. Efficient backprop. In Neural Networks: Tricks of the Trade, pp. 9-50. Springer, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Machine Learning and Knowledge Discovery in Databases, 498-515, Springer, 2015.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, 2019.
Timothy P Lillicrap. Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 2020.
James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factoree approximate curvature. In International Conference on Machine Learning, pp. 2408 2417. PMLR, 2015.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy Sciences, 115:E7665-E7671, 2018.
Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento. and Benjamin F Grewe. theoretical framework for target propagation. In Advances in Neural Information Processing Systems, 2020.
Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, and Rafal Bogacz. A theoretical framework for inference and learning in predictive coding networks. XXXi:2207.12316 2022a.
12

---

Published as a conference paper at ICLR 2025
Beren Millidge, Alexander Tschantz, and Christopher L Buckley. Predictive coding approximates backprop along arbitrary computation graphs. Neural Computation, 2022b.
Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, and Rafal Bogacz. Back- propagation at the infinitesimal inference limit of energy-based models: Unifying predictive coding, equilibrium propagation, and contrastive hebbian learning. In International Conference on Learning Representations, 2023.
Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, and Antonio Orvieto. Super consistency of neural network landscapes and learning rate transfer. Neural Information Processing Systems. 2024.
Luca Pinchetti, Chang Qi, Oleh Lokshyn, Gaspard Olivers, Cornelius Emde, Mufeng Tang, Amine M'Charrak, Simon Frieder, Bayar Menzat, Rafal Bogacz, et al. Benchmarking predictive coding networks-made simple. arXiv:2407.01166. 2024.
Shikai Qiu, Andres Potapezynski, Mare Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. Compute better spent: Replacing dense layers with structured matrices. In International Conference on Machine Learning, 2024.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 1999.
Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In International Conference on Learning Representations, 2023.
Robert Rosenbaum. On the relationship between predictive coding and backpropagation. Plos One, 17(3)=e0266122, 2022.
2022.
David Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating error.. Nature, 323(6088):533-536 1986.
Tommaso Salvatori, Ankur Mali, Christopher Buckley, Thomas Lukasiewicz, Rajesh PN Rao, Karl Friston, and Alexander Ororbia. Brain-inspired computational intelligence via predictive coding. Ttl.:2308.0787700 2023.
Tommaso Salvatori, Luca Pinchetti, Amine M' C'Carrra,, Beren Millidge, and Thomas Lukasiewicz. Predictive coding beyond correlations. In International Conference Machine Learning, 2024a.
Tommaso Salvatori, Yuhang Song, Yordan Yordanov, Beren Millidge, Lei Sha, Cornelius Emde, Zhenghua Xu, Rafal Bogacz, and Thomas Lukasiewicz. A stable, fast, and fully automatic learning algorithm for predictive coding networks. In International Conference on Learning Representations, 2024b.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in Computational Neuroscience. 11:24, 2017.
Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Can the brain do Packpropagation'--eaae implementation of backpropagation in predictive coding networks. In Advances in Neural Information Processing Systems, 2020.
Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, and Cengiz Pehlevan. Feature-learning networks consistent across widths realistic scales. In Advances in Neural Information Processing Systems, 2023.
James CR Whittington and Rafal Bogacz. An approximation of the envor backpropagation algorithm in predictive coding network with local Hebbian synaptic plasticity. Neural Computation, 29(5): 1262, 2017.
Greg Yang. Tensor programs Neural tangent kernel for any architecture. arXiv:2006.14548 2020.
Greg Yang and Edward J. Hu. Feature learning in infinite- width neural networks. In International Conference Machine Learning, volume 139, pp. 11727-11737. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Greg Yang and Etai Littwin. Tensor programs IVb: Adaptive optimization in the infinite-wite width limit. In International Conference on Learning Representations, 2023.
Greg Yang. Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems, 2021.
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Feature learning in infinite-depth neural networks. In International Conference on Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2025
Appendices
A EXTENDED BACKGROUND
A.1 EXTENDED RELATED WORK
Table S.1 List Abbreviations
Abbreviation
Full Name
Reference
BP
BackPropagation
SGD
Stochastic Gradient Descent
GNT
Gauss Newton Target
Proposition 3.3
μP
Maximal Update Parameterizatton
Definition A.5
NTK
Neural Tangent Kernel
HP
HyperParameler
PC
Predictive Coding
Section 3.1.1 Technique Technique ii
F-ini
Forward initialization
FPA
Fixed prediction assumption
TP
Target Propagation
Scction 3.1.2 Eq S.13
DTP
Difference Target Propagation
A.1.1 pP
While pP was introduced as a parameterization to induce a feature-learning regime the infinite- width limit for theoretical interest (Yang & Hu, 2021), practical advantage highlighted by Yang et al. (2021) is its ability to transfer the learning rate across different widths, phenomenon they experimentally validated. This phenomenon, known y/Transfer, has also been examined from theoretical perspective (Noci et al., 2024). Another notable advantage of jP is the improvement in the scaling law exponent, which has been investigated both experimentally (Qiu et al., 2024) and theoretically (Bordelon et al., 2024).
noteworthy that μP depends on the learning algorithm used and thus should be derived for each specific method. The μP for Adam was introduced in Yang et al. (2021), with its theoretical justification provided in Yang Littwin (2023). For the second-order optimization, including the algorithm, K-FAC, and Shampoo, the jP was derived in Ishikawa & Karakida (2024). These works emphasize the importance of adjusting not only the learning rate also the damping term in second-order optimization using μPP Additionally, μP derived for Adafactor in Everett et al. (2024) and empirically demonstrated that the scaling of the term in Adam also crucial in j.P.
A. A.1.2 PREDICTIVE CODING
Recent progress in deep learning has largely been achieved by the success of backpropagation (Rumel- hart al., 1986; LeCun al., 1998; 2015). This success has increased the interest exploring whether deep networks can also be trained using training algorithms other than backpropagation This includes exploration into biologically plausible training methods and the benchmarking of local learning rules at modern-scale networks deep-learning benchmark datasets; equilibrium propagation (Scellier & Bengio, 2017; Laborieux et al., 2021), target propagation (Bengio, 2014; Lee et al., 2015), predictive coding (Whittington & Bogacz, 2017; Song al., 2020; Salvatori et al., 2023), and forward-forwar algorithms (Ren et al., 2023).
Computation of predictive coding for supervised learning: Predictive coding was originally introduced as an algorithm for solving inverse problems where the goal is to find the parameters W that maximize the marginal likelihood p(v,; W) where Ug denotes variable representing the
15

---

Published as a conference paper at ICLR 2025
network output and 1y = z for the input data (Rao & Ballard, 1999). In this inverse problem, consider latent variables by (also referred to as causes) and a generative function p(m)/u/11: Wi). If assume hierarchical Gaussian generative model, the marginal probability of the causes is as follows:
(op. ...0.....00 Uz; W) p(b))
II 1=1
pleller-1: Wi)
(S.1)
L
= N (vo, To 1)
II
(Wid(vt-i)
(S.2)
1=1
Friston (2003) framed this inverse problem as an EM algorithm aimed at minimizing the following variational free energy:
= (L
PL).. pz)) UL; W)).
(S.3)
where q("...... vI.) is tractable posterior probability distribution for the EM algorithm. In particular, the E-step, referred to inference, minimizes variational free energy by causes ut, while the M-step, referred to as learning, minimizes by parameters W,. We usually apply mean-field approximation or Laplace approximation the tractable probability distribution (Friston, 2005; Salvatori et al., 2024a;b). Under these formulations, we can derive the variational free energy for PC.
1 2
F
lo Wio (m-1))1?3
Wie
(S.4)
While predictive coding networks were originally discussed primarily in the context of generative models for unsupervised learning, Whittington & Bogacz (2017) reformulated PC networks for supervised learning and highlighted their potential for use in the context of deep learning. Specifically, we fix vp = x and v = y for data, this corresponds to supervised learning using the mean squared loss.
Heuristic techniques in PC: After Whittington & Bogacz (2017), there has been an increasing amount of studies evaluating the performance of PC networks as local learning on deep-learning benchmark datasets (Salvatori et al., 2023; Pinchetti et al., 2024). They revealed that the original implementation of PC networks is insufficient for achieving stable training performance, and heuristic modifications have played an important role. For instance, Fixed prediction assumption (FPA) has been introduced achieve higher performance by approximating the gradient computation of PC networks closer to that of BP (Millidge et al., 2022b; Rosenbaum, 2022). Under FPA, ø (ut..) is replaced with ø' (v1,0), resulting in an inference phase given by
U1,++1 = V1,s - 7eL. + 1 (vLo) WII1 LL
(S.5)
and a learning phase is given by
W1.+1 = WT, + (0ree,,6(0-11))
(S.6)
Additionally, FPA is typically used with Forward initialization (F-ini). In Forward initialization, the state 21,0 is initialized with the forward value w,o. While F-ini has been implicitly utilized in most studies on PC (Whittington & Bogacz, 2017; Song et al., 2020; Rosenbaum, 2022), its role was explicitly highlighted in Alonso et al. (2022), where the authors compared the convergence with and without F-ini.
Rosenbaum (2022) pointed out that when both F-ini and FPA assumed, PC networks are entirely reduced to BP and that if the algorithms are fully equivalent to BP, the advantages of biological plausibility and local updates are lost. In this study, we aim to identify a parameterization that enables stable local learning while maintaining distinctions from BP. By leveraging several recently developed heuristics, we clarify the desirable scales for stable and efficient local learning.
Nudged PC: The design of loss functions in PC networks has also been focus of algorithmic improvements. In most of the ML. research, classification tasks generally use cross-entropy loss rather than mean squared loss. Accordingly, PC networks sometimes use cross-entropy loss as well (Pinchetti et al., 2024). The free energy for general loss function is given by
L-1 1 || - Wid Wid(w-1)) (11
F= F W2o(UL-1)) + Σ
(S.7)
16

---

Published as a conference paper at ICLR 2025
Algorithm 1 PC Algorithm (Simultaneous Sequential inference)
1:
for to n do
2:
ET., -- V u... (.............))
3:
for 1 = L - to do
4:
61,s f- U1, - Wi (4
5: C1+1s t 4411,11 W4+1 On un,s) (Sequential Inference)
6: U1,++1 f- u1, Me,, 1 (ut,)) o W 11411s
7: end for
8: end for
and its update rule for the inference phase is given by
01,+1 U1,s 71,,, + n+10' (V1,s) o W441
(<<
(S.8)
UL
01, UL NL duL
(S.9)
Furthermore, there are formulations PC networks that incorporate the nudge term introduced in equilibrium propagation (Scellier & Bengio, 2017). PC networks with nudge term update the state and weights W; to minimize the following free energy function (Alonso et al., 2022; Millidge al., 2023; Pinchetti et al., 2024):
F = 3C(y, +
2
|lue
Wid (1r-1))1
(S.10)
Here, 3 is a nudge coefficient parameter that covers some variants of the PC algorithms in the previous work.
There are several possible orders for computing this inference (Alonso et al., 2024) as illustrated in Algorithm Specifically, et can be calculated sequentially from the output layer to the input layer, or it can be updated synchronously across all layers simultancously from the output to the input. While the main text focuses on Sequential Inference (SI), where computations proceed layer-by-laye from the output to the input, Predictive Coding with synchronous inference is also valid approach worth considering.
A.1.3 TARGET PROPAGATION
Target propagation offers learning rule that is more biologically plausible and easier for the brain to implement compared to BP (Bengio, 2014). Specifically, it addresses the following two issues inherent to BP (Meulemans et al., 2020).
1. Signed error transmission problem: BP propagates the error gradient to the lower layers, whereas the brain propagates target values for the neurons (Lillicrap al., 2020).
2. Weight transport problem: BP requires exact weight symmetry between the forward and backward paths. However, the brain cannot transport weights (Grossberg, 1987; Akrout al., 2019).
Target propagation aims to address the two issues by:
1. Propagating the target value hg hı V,, instead of the error gradient V C for signed error transmission problem.
2. Utilizing a feedback network 9i distinct from the feedforward network fi to propagate the target value h for weight transport problem.
The feedback network gi has the weights Qi distinct from those in the feedforward network.
= 1)) gi
(Quar)
(I 1,..,,, L 1).
(S.11)
Here, 1a denotes the activation function. While it is often the same as the activation function used in the feedforward network, it is also possible consider different activation function. We set s
17

---

Published as a conference paper at ICLR 2025
in experiments and assume v as the identity function only in Theorem 5.1. The feedback network is trained to minimize the following reconstruction loss:
ree ( Qt) 1l (Ji(h111 + +c)) - 11-1 - +ell"
(S.12)
where is small Gaussian noise to improve the robustness of the feedback network. Target propagation attempts to approximate the inverse function of the feedforward network by learning the feedback network through the optimization of this reconstruction loss.
Difference target propagation (DTP) is an improved method of TP, which adjusts the propagation in the feedback network as follows (Lee al., 2015).
h; 9i
diff (ha+1; h.) gi (met1) (g: (h++1) h₂)
(S.13)
In TP, the accumulation of the reconstruction error 9: (:(()(b))) h; during propagation poses an obstacle to optimization. In DTP, subtracting (y, (h(+1) - h1)) mitigates the accumulation of reconstruction error and improves the progress of learning.
As a side note, Meulemans et al. (2020) and Bengio (2020) pointed out that TP can related to the Gauss-Newton method for invertible networks. Additionally, Meulemans et al. (2020) proposed Direct Difference Target Propagation so as to establish this correspondence even in non-invertible networks under some infinitesimal conditions. Ernoult et al. (2022) reported that one stabilize TP by introducing the additional Local Difference Reconstruction Loss which makes the gradient align more closely with Backpropagation rather than Gauss-Newton Targets. our work, aim to clarify the fundamental properties of TPP and DTP from the perspective of parameterization and do consider such additional conditions or loss functions.
A.2 DEFINITIONS FOR STABLE PARAMETERIZATTON
As is common in the μP theory, we also assume that the firing activities are of order at random initialization:
Assumption A.1. ur b4,0 ((1) < L), fo - "I,0 = 0(1.
As shown in Theorem H.6 of Yang & Hu (2021), this assumption immediately leads to
a1 + bj = O, ai<K< bici<L = 1/2, ay + 2y 2 1/2.
(S.14)
In addition, the stability of learning is defined follows (see Definition H.4 in Yang & Hu (2021) for more detail):
Definition A.2 (Stability of learning). We say an abe-parameterizzatonn is stable if, for < and for any fixed 2 1,
Ahi = O(1), Af = O(1),
(S.15)
under Assumption A.I.
Condition S.15 ensures avoiding exploding dynamics with respect to the width, i.e., Abi, O(1/M*) with & < O.
We follow the derivation based on the infinitesimal one-step update from random initialization (Yang & Hu, 2021; Ishikawa & Karakida, 2024), which involves taking the limit of a sufficiently small coefficient the learning rate n/'. This formulation clarifies the proof and enables the systematic derivation of μa across various problems. In the infinitesimal formulation, Conditions 3.1 and 3.2 are expressed as follows:
Condition A.3 (Wr updated maximally).
0, AWWh 1,1 w== = O(1)
(S.16)
where AW; Wi,1 Wi,0-
Condition A.4 (W, initialized maximally).
W.WLLARL-La -(1).
(S.17)
18

---

Published as a conference paper at ICLR 2025
As described in the previous work (Yang & Hu, 2021; Ishikawa & Karakida, 2024), Condition 3.1 (or A.3) naturally appears from the expansion of Eq. (10) by the parameter update, yielding (S.18)
O,y WW = (11/M").
for abc-paramettrivaaton. The stability requires
n
(S.19)
and, in particular, feature learning is characterized by ri 0. Condition 3.2 (or A.4) is required to eliminate an uninteresting case in which the hidden layer provides no contribution to the network output. Both NTK and feature learning regimes are characterized by this condition.
As is shown in Yang & Hu (2021), μP is the unique stable parameterization satisfying Condition A.3 for / S and Condition A.4 for WL. Thus, we can admit this characterization definition of p.P. Definition A.5 (jPP). JPP is the stable abc parameterization satisfying Condition A.3 for / S L and Condition A.4 for WL.
Note that Condition A.3 is required not only for hidden layers but also for the last layer. In the previous work, this eliminates trivial case of learning, i.., Ahi, O(1/M") with k > 0, where the effect of learning vanishes.
B μP OF PREDICTIVE CODING
B. DERIVATION FOR PREDICTIVE CODING WITH SINGLE SHOT SI
Theorem B.1 (Stable parameterization for PC). Set inference step sizes Ne. ((1) and T,. Y' M with a positive constant 7' Suppose F-ini and single-shot sequential inference, and consider one-step update of parameters after the inference. For infinitesimal step sizes 7 and nf, PC admits the yP for feature learning at
C1 c1= 1, Orelet --,, b0
G=L=1,
{
(S,20)
by.
Additionally, admits the NTK parameterization
C1 -L, C1CLL =1 7e: CL =1
(S.21)
b = 0, bieL = 1/2.
Proof. Assuming F-ini, considering the single-shot SI for v, we have
f1 = v1,0 + 1 (ur,)) W 941,1
(S.22)
vn + 0 (vro) WIT (M141, W441h1,)
(S.23)
v1 2411 6 (ur,o) W.T. (1141,1 u141,))
(S.24)
for 1 < L where CL,0 y WLU1-1,0 =: 81. When the CE loss is used instead of MSE loss, 5 = y - becomes ōy = y softmax(f)), and the order analysis remains unchanged. To keep the notation concise, we set Mr. 1 in this proof. A generalization for Mr. O(1) is possible. Next, we define
d auz/Ou.
(S.25)
Note that batch gradient can be used with N training samples where N 0(1). One can regard vn as an M x N matrix in the derivation.
Using
011
00
II 76
(S.26)
1=1+1
we have
11,1 : 01 ((110-111)
(S.27)
II
7: - 6(Wi(u-11
II (
(S.28)
1=1+1
19

---

Published as a conference paper at ICLR 2025
for = 1, L 1. Recall that 01 u1o for F-ini. For I L,
C1,1 y W1h1-1,1
(S.29)
y - W W. -
(S.30)
where diag(z) denotes a diagonal matrix whose diagonal entries are given by x. The above equation comes from
7L V VL 11 Wihh-1l12
11-11 11-1,0
(S.31)
11. - 7 WT. 6r. = u - -
(S.32)
The first one-step update of the weight is expressed
T/
AW11
M2 111141111
(S.33)
In PC, in addition to the usual learning rate 7) there also exists 7. Therefore, in addition to the infinitesimal update the learning rate for the weight update, we also consider the infinitesimal inference step size 7/.- By applying the perturbation of yr. Conditions A.3 and A.4, derive
AU; + 0, AUi 7/ =0 2 @(),,
(S.34)
AVE + 0,1 AVL 7/ -0 O(1)
(S.35)
where we define
AU = 0, AWW,WW,, / / y'=0'
(S.36)
AV, = 0. 11. n==0
(S.37)
is noteworthy that we retain the zero-th order terms, namely, AU; and AVi in the conditions. This is because, even without the inference phase, parameter updates can progress while the internal states remain their initialization. Therefore, even if the maximalization of the order is less than ((1) in the first-order perturbation terms, stable learning can still occur. Since μP aims to maximize the order updates as much as possible, we require the first-order terms of Eqs. (S.34.5.35) to be O(1) whenever possible.
We introduce the following kernel matrix:
K-:= = h1 ht/M.
(S.38)
For the random initialization Wi, from Eq. (S.14), we asymptotically obtain
K²* O(), K = 9 (1/M/Roztbb)
(S.39)
in the infinite-width limit (Yang, 2020).
On Condition A.I.
(i) Case of 1 <
1
0x AU 7/==0 = à
Mr. C1,e=1 -, h 12=0
(S.40)
M O(e -- MMI - 00 Mo. 1 /
2/-=0
M 0 0(e
1/, 1,8= 11
(S.41)
51 + 6 (Wa4-100) o Wi KA -9 (-6
(S.42)
where we used etct,,==1 O(L/Monthi) and AÜ; tactabbb.. we have
| =0= 0 and h he ly, h1 Since òiet
O(Wr.)
AU + 02 AU A =07 ~
(S.43)
20

---

Published as a conference paper at ICLR 2025
The similarity symbol ("~") denotes that the left-hand side is of the same order the right-hand side. Note that if the first-order term becomes negligible, the contribution the inference phase disappears in the parameter update. To maximize the order of the first-order term, require
PL S 0
(S.44)
and obtain
= 0: + 7r. + az + br. 1.
(S.45)
(ii) Case of = 1.
1
dy2 AU
M0 6,K0
(S.46)
71
(S.47)
Here, we used ei< L,=1 7/- =0
and h1,s=1 1,,= h1.=00 Similar the case of 1 < L,
Condition (S.34) leads < 0 and
Ti = 01 + 7 + az + bL.
(S.48)
(iii) Case of I L.
0yg AU
(S.49)
Mor. MeL,o=1)h--1ssse 1/ +
M (..=:1000---s::------1 (S.50)
M ((EL,==)) -11 +
M 0
(S.51)
1
M U1 W WL K
S 1,,0,,11 (S.52)
Mot
2
Mo..tu.-1 8 (W W
-1 00h4-,,011 (S.53)
M
Note that from Eq. (S.30), we have
LL =0=0 = - = - 0 0
(S.54)
Since CL,s=1 1/2 =0 = 5z + 0,
Wr. Wr(0'r-11 o 1) 0r.
(S.55)
For the second term in Eq. (S.53), we have
-- ( = -0 16=0
(S.56)
-hI-10' W 1 1
(S.57)
76-1 1T (8 o W 1i_11d1-2)
(S.58)
M
where we used FL-211 UL-2,0 - from Eq. (S.26). Let us recall that a variable without an index indicates the initial state at 0. The variable ô includes Wt whereas h is independent of it. Therefore, by applying the Central Limit Theorem with respect to Wr, we have
hT 0lh 1y0 1/M
(S.59)
Then,
0y AU ly +i
(S.60)
In contrast, we have
AUG 1/10L-1
(S.61)
21

---

Published as a conference paper at ICLR 2025
Comparing the zero-th and first order terms (S.60,8.61), obtain
min(0r. - 1, 1, + Yz. + 2(az + by)) 2, 0z. + 7t. + az - 2y 1/2] =
(S.62)
Because ag + +b 1/2 2 from Eq. (S. 14), we obtain
Or. - 1 = 0.
(S.63)
On Condition A.2.
0z AVE 2/=0 2V WL,u(0' (uL(0(( O -f
= eM (6L-10
M By (AW / 2
(S.64)
G(1/MeL 1-11
(S.65)
where ear denotes an M-dimensional vector with all entries equal to 1. Note that the product with EM means the summation over M.
Finally, from Conditions and 2, the jPP is given by
+ "L + az + 2y = (1=1). 1),
(S.66)
0 + îL + ar + -1=1 (1<l<L),
(S.67)
0
(1=1) L)
(S.68)
aL + 1y - 1 = 0,
(S.69)
and 5z. S 0. That is,
-tc 1, Crelet. -5, 20, G=L=11 0, heer = 1/2, by. =1=1. 1.
(S.70)
The above μP case assumes a₁ 0. It is important to note that there is no issue in replacing with 0, = 20z + <t, which introduces an indeterminacy of a; = ar + a and ai 2a.
We also derive the NTK parameterization, which is commonly used term for the kernel regime for TıL = 1/2 (Yang & Hu, 2021):
01 3 + ag 1い = 1/2 (1=1). 0 + + az + -b,-11 1- 1/2 <<<<)),
(S.71)
(S.72)
9 0, (L=L)
(S.73)
ar bb 1/2 0.
(S.74)
It is noteworthy that the gradient computed by Eq. (S.28) differs from ò; in standard SGD, implying that the NTK matrix also deviates from Vof Vef. Even in this case, the NTK regime can emerge with certain modified kernel composed of and hr. A similar situation arises in the NTK regime of second-order optimization (Karakida Osawa, 2020). Although the preconditioner modifies the NTK matrix, the linearization of the model still holds, allowing the emergence of the kernel regime.
B.2 FIXED POINTS OF PC IN LINEAR NETWORKS
B.2. PROOF FOR THEOREM 4.2
In this section, we analyze the fixed point of the inference phase using linear network:
f(x) W.Wr.....W1r.
(S.75)
Even for linear networks, the properties of the fixed points have rarely been analyzed. An exception a recent study by Innocenti et al. (2024). They explicitly derived the free energy at a fixed point to analyze the parameter loss landscape of a naive PC. However, their analysis uses unfolding calculation of a hierarchical Gaussian model to directly derive the free energy. Although this is an elegant derivation, it is not method for explicitly obtaining the fixed points themself. Additionally,
22

---

Published as a conference paper at ICLR 2025
since their proof is based on 1, need another method to determine the dependence the inference size. Here, we provide a derivation of the states at the fixed point that can be used more generally for various inference sizes and add nudge term (in Section B.2.4).
Proof. We consider the inference of naive PC:
F(, vr.) = 7t lyy WLUL- +
R 2
110-1112
(S.76)
Taking
Oyg aF = 0, we obtain the following fixed-point equations:
7u. WI (y WLPL-1) + 7 Wi-191-2) 0, (1= L)
(S.77)
w, (m W1m-1) + n-1 W1-19122 = 0, (1<l< < L)
(S.78)
W2 (12 W221) + 1 - W1x) = 0 (1=1). 1).
(S.79)
These equations are summarized the following matrix form:
0
O
16-16I-
Yz W.
W L
0
YL 2L-20
0
WT
I
0
(S.80)
:
220
0
W2
I
nei
where ef := of W and ez := W
Here, we use the following lemma:
Lemma B.2. Define
0
WT
1
0
Ag ==
0
W.T1
(S.81)
WW
Its inverse matrix is given by
I
0
WI
0
WT
WT
O
AZ
(S.82)
W.T.
W
W.
Proof. One can easily derive this inverse matrix. A simple derivation is achieved by induction. We can express
1
0
A
(S.83)
K
A
where KJ WW, O, OJ. Suppose that the inverse of Ar is given by Eq. (S.82). Then,
0
A7
KA
A7-L.
(S.84)
Since KA-11 = [Wr,, W,-i.,,... W2.1] the inversion of Ar. is also given by Eq. (S.82).
By using Lemma B.2, we can transform Eq. (S.80) as follows:
WTe
et
T
1
76-2
(S.85)
C
WT
2.1
23

---

Published as a conference paper at ICLR 2025
Although this equation can not be solved explicitly for of-1- can, nonetheless, solve it by summing over e7 as follows:
vi-11 WL-1111
(S.86)
-6i.-1
+
Wi.-107 + ... + 1:26
1
1
71
I +
WI. L-1 ....
+ W W. Wi.1:2Wi-12 W
(S.87)
1/-1
7L-2
7
This leads to
vi-1- 1+ 7
WT.,
7
WT,
W L-1 WI Wr. Wr,
74
WT, 1- 1:2 W2 L-1:2WLW
71.11
7L-2
71
(WL-111 +
1
+
1
W
W
1
W WL-1:2WT-112
W
Y1-1
71.22
71
(S.88)
Set an Mr Mr matrix
C,(W) = Σ
IL
WL W.:WT..
L:3
(S.89)
1=2 7--1
Then,
e WL (1 (W)) (WLaw (I CW))) v)
(S.90)
= +T+ (()) (u f)-
(S.91)
Thus, at the fixed point, the local loss is explicitly obtained as
I W La+ CWW)) f).
ej
(S.92)
7
We also obtain of- From ei, have
of = Wix + Yr N + f).
(S.93)
By induction, we have 27 = =] + Wioi-1
(S.94)
WI.1 +
Th WT
Σ 7 WinW ( f).
(S.95)
Li++1
n
ύi-1
1=2
B.2.2 PROOF FOR COROLLARY 4.3
Proof. We consider the order of W..WT. in (W)= T WL.WT. L:3- Note that computing the vector WT.. La is equivalent to signal propagation in deep linear network with random weights WT W:L-1 and an input vector W,T. Therefore, in Eq. (S.39), kernel computations in existing studies of random neural networks. Specifically, this is equivalent to computing hn by in the following random neural network:
evaluate ||e/] using
ug Wi h = Ō(ux) (k: I......). J; ~ N(0 100001/Meeae 1/Mretbb) (1 (d=1, 1..,,..,MM M).
(S.96)
(S.97)
Wk,0 (O 1/VM) (i.j 1,,,,, M).
(S.98)
From Theorem 7.2 in Yang (2020), for any x. 9b = 11 ha/M converges almost surely to
V2T / due 1 (
1
(S.99)
24

---

Published as a conference paper at ICLR 2025
in the infinite-width width limit. Since we considering a linear activation function, have qi 9t-1 and
M x.
(S.100)
Since is an i.i.d. Gaussian vector, the law large numbers leads to qi of (L1/M2(ar+bb)) For random weights given by μP i.e., az by I, we find WL.: WT La (1/MM). Consequently, for IL = (M), C,()) remains of order 1. contrast, for 7 = (1), C, is O(1/M), causing I+ (W()) approach I in the infinite-widhe width limit. As result, ef òn and of We, and the gradient in the infinite-width limit matches that of first-order gradient descent.
IT
B.2.3 BALANCE CONDITION DETERMINING "
Here, we consider the order of 61, with respect to M<z. For linear network with one-shot SI, we obtain
L
61, = II
76 - 11 /ttl:///////aaaaaaaa 1/Ammin(0.9)422 90
(S.101)
im++1
In contrast, recall that the order of eî at the fixed point is
of 1/M 1/M-3++min(0.n1)
(S.102)
Therefore, to satisfy 61,1 ~ ef, the following is necessary:
Σ 7: -7
(S.103)
for all I < L. This is equivalent to 7 = 0 for all I < L. Thus, 0t, o7 holds if and only if 7142 0.
B.2.4 NUDGED PREDICTIVE CODING
We can extend Theorem 4.2 to the nudged PC.
Theorem B.3. Suppose an L-layered linear network and put oî = = Wiu where denotes the fixed point of the inference given by Eq. (S.10). The following holds:
ej =z -Wioi=1
(
7 +++
(WL,10-)
(S.104)
7
ej = - -WLUi-1=
2 2 c,(w)) 1 1+
11+2214
(WL.10
(S.105)
where WLA WWWL-
Proof. Put
L
F F(v U1) = Blyy 22
+
|1 Wio
(S.106)
1=1
Taking aF By = 0, we have
8(UL - 3) + (2 WL =
(S.107)
+ -
= 0 (1<13L)
(S.108)
W2 (02 - W2r1) + ( - Wix) = (1= 1).
(S.109)
Putting the system of equations can be written in matrix form as follows:
0
T
8x
W
T
0
7
7-16
0
WT
O
(S.110)
:
T26
0
0
W.
T_
71e
25

---

Published as a conference paper at ICLR 2025
From Lemma B.2, the above equation is transformed into
1 W
6L-1-
74-1
= =33
(S.111)
1
2
71
Take the following summation:
3 WreZ-1 +
WT.WW1-2 = WWWWWa. W
3
+1+ C,))))))
(S.112)
NI
Thus, we can explicitly obtain uz.
ot. (6=(2r10
- 1+
3
3
-1 c,
WW 8 (I (I+C)(W))))
(S.113)
7L
7L
7
Thus, X can be written follows:
x L
(S.114)
3
3
-1 c,(W)
WL11+ + (1 (W))))
= y
I +
(S.115)
I
1
8
1
B
C,
-1
(y s).
(S.116)
7
From the above, we conclude that
ei
WT Li+11
(++ E
B c,()))
-1
(y -)
(S.117)
YL WT 1:141
[ IL I C,(W))
(1 (
(S.118)
at the fixed point.
C μPP OF TARGET PROPAGATION
C.1 DERIVATION OF THEOREM 5.1
Assume that the feedback network linear: gi (x) = Qiæ. Here, we consider a reconstruction loss with L2 regularization:
((Q),)) (W h4 11
(S.119)
with 2 0. Note that while some work adds noise to 11-1- it does not affect the order; therefore, we will ignore it in this derivation. As described below, by taking the ridge-less limit of μ,, we can evaluate the parameterization of the original TP and Difference Target Propagation (DTP) in clear and unified manner. Considering the fixed point for QL, since ai(Qi) 0 holds, we have aQ:
Qi = - (he h + 1q1)
(2.12
where h oWW The feedback network is given by the network with Eq. (S.120). As side note, this weight is essentially the same as the pseudo-inverse weight, which is known as an extension of the Hebbian weight (Kanter & Sompolinsky, 1987).
Local targets of DTP. DTP is improved method of TP, where h, is propagated follows:
h, = -i-1 (he4t, h141, 11) = gt+1 (het.) (gr41 (lu41) - hu).
dill
(S.121)
26

---

Published as a conference paper at ICLR 2025
For the last layer, the error is given by For linear feedback network, we have
h hr 3(y hL)
(S.122)
h = gi+1 (hit1) - (gL+1 (h(+2) h1)
(S.123)
hu Q++1 Q4+1 h(+1)
(S.124)
hu - Q1+1 + Q1+2(11+2 1l+2))) h(11)
(S.125)
= hu Q++1 11r1Q42 h142)
(S.126)
h4 II Q
(S.127)
(=1+1
Therefore, at the equilibrium point for Qi, for I S have
h h4 - II Q7o
(S.128)
i=/+1
L-1
= -B1u II
(h. ha + pzT) (o + 111)-111.65
(S.129)
=++1
= -h
II (K + 1-++
(KΛ) + 1--11-111 '1755
(S.130)
where ó1 = Oc/Of and for l 1. we have h hL-1 = - -BhL-1hK3 + - To avoid uninteresting change of order, introduce 14 p4/M/ and require that it have the same order as K;A. This is essentially equivalent to the valid condition argued in Ishikawa & Karakida (2024), which requires the damping term to have the same order as the preconditioner the second-order optimization. We note that typically set to be regular at random initialization the neural tangent kernel literature (Jacot et al., 2018; Yang, 2020). For instance, this holds true for normalized input samples with || = 1.
take the ridge-less limit μd₁ - 0+ because KÅA (S.39)
Local targets of original TP. The signal propagation in the feedback network is
hi = Qi+1 ... Qi,h
(S.131)
L-1
hi II
(h h4 + 2h h.(hTTh2 + (LT) "h 35L)
(S.132)
im1+1
L-1
lu
Bhg
II t=++1
<<>)
118 (4 + 0+).
(S.133)
Thus, the target is reduced to essentially the same as that in DTP (3.130) and we can treat both in the same manner.
On Condition A.1. The update for the last layer is identical to that of SGD with BP, thus
1 36 36,
oy AW
(S.134)
Next, consider the L 1 layer.
o, I. = M (h1-1-
11 1
(S.135)
M 1-1 Bhr (1 h1 11.1) -1 K
(S.136)
M 3hr.-1 (KΛ n1 1) (Kg -M
(S.137)
27

---

Published as a conference paper at ICLR 2025
Similarly, when hy h4 - -BII QioL. we have
1
Dy WWiWWhh--ll Mo:
(hr hi)li-zhu11
(S.138)
L-1
M Bhg
II i=++
(h) ha + (= heth 1
(S.139)
1 Bhy
II i=/+1
(K₃ + (4) + M k
M
(S.140)
for .... L 2. On the right- right-hand side, hy 1/Mottbe-1/ and from Eq. (S.39), we have
K - M h1-h, h1 y ~ max max 1/Mastbe+1/2y
(S.141)
=1/Moutbrt1/2
(S.142)
In the last line, we used az +b2 2 1/2. Here, from Assumption A.1, which states that a₂ ty 2 1/2, we obtain
01 az - by + 1/2 (1=1).
01 1 as b + 1/2 <1<
(S.143)
0 - L).
On Condition A.2.
(W M W,, oida - h1
(S.144)
M 3hg h1 16 1
(S.145)
11P-117311(K33 +12 - M M (S.146)
Thus, its order is
Oy (W ~ -0
(S.147)
(S.148)
Finally, from Conditions A. and A.2, the yP given by
az b, + 1/2 0 (I=1),
(S.149)
0 bt - 1 = 0 <))
(S.150)
0,
L),
(S.151)
ar by 1/2 0.
(S.152)
C.I.1 DISAPPEARANCE OF KERNEL REGIME
Corollary C.1. Stable learning satisfying Condition A.2 leads to Fr.-1 = 0 for TP and DTP.
Proof. From Eq. (S.148), we have
TL-1 + (aL + by) 1/2 =
(S.153)
From Eq. (S.14), 0, + b,. 2 1/2 and we have FT 2 0. In contrast, from the stability of learning, have TL-1 0. Thus, FT-1 0.
28

---

Published as a conference paper at ICLR 2025
Note that, precisely speaking, TL-1 does not necessarily imply rı<L-1 = 0. However, it is often considered unnatural (or uninteresting) to examine cases in which the progress of learning depends on individual layers. Therefore, the μP typically assumes uniform parameterization, meaning Tic (see Theorem G.4 Yang & Hu (2021)). In this sense, FL-1 indicates the disappearance of the kernel regime.
One might be surprised by the fact that d, = 1/2 is allowed in the feature learning and that the kernel regime disappears. Note that the feedback weight in the last layer (S.120) essentially differs from Wr. The feedback weight receíves ,r as input whereas Wi receives - This makes
QL 1/M (az
(S.154)
W, ~ 1/MoLtbe, The gradient is proportional to Qr. in TP and Wr in BP. The feedhack weight contributes more significantly to TP's gradient when ay + 1y 2 1/2. This eventually makes the index of the hidden layer (S.143) get quite large even for careful about the order of condition A.2 (S. 148). Because the feedback weight (2.) has lower alignment exponent (Everett et al., 2024), this causes the condition 2 of TP to be smaller than that of SGD (or K-FAC), i.e., (u Therefore, stable feature learning is possible even for ag + 7y 1/2.
+ 1₁ 1/2. We also need to be
Remark on zero head initialization. Related to the size of by. the parameter initialization with by > 1/2 (9, > 1 for SGD) reduces to the yP of b, = 1/2 (b,, = for SGD) because Wí,o becomes negligible compared to AWi.o. To illustrate the intuition, let us introduce the case where the weight of the last layer in a feedforward network is initialized as W = O, that is, by = DC.
In this case, only the last layer is updated during the first step because Q7, O does not propagate the local error to the downstream layers. After the first-step update, the weight is given by
WL,1
M 1-1
(5.155)
and WKLL11 WILLO Thus,
1
Ar 11
8ih L- h
(S.156)
M
M 8x Kr.
(S.157)
and
h1,1 0(1/M'-1), KY = O(L
(S.158)
Substituting these into Eqs. (S.137,S.140), we obtain
0 0₂ +1 (1=1), 1),
ri
0: 0r. (1<l L),
(S.159)
1 - (1= L).
From Eq. (S.146),
0(1/M/(---1)
(S.160)
Finally, Conditions A.I and A.2 lead to
0, (0,, - 1) 0 (1=1),
(S.161)
0 1 (0, 1) = 0 (L)
(S.162)
0, =
(S.163)
Thus, the /PP is the same as in the case of random head initialization.
29

---

Published as a conference paper at ICLR 2025
itcration vs similarity
width vs similarity
Targes
Figure S.1 During the training of the linear net- work, converges to the an- alytical solution. We trained 3-layer linear network using synthetic data.
(layer1)
cosl (ct, ot)
(layert)
Analytical
0.8
(layer
2
BP
0.7
(layer3)
ONT
(layer2)
100
200
300
6.0
7.3
10.0
12.5
Analetical
Jog2 width
Itcration
C.2 STABLE PARAMETERIZATION FOR FEEDBACK NETWORK
The feedback network minimizes the following loss function:
I(Qi) 2M 11 - 11-1
(S.164)
where dividing by M ensure that L(Qi) (1). which is the default setting in Py Torch We consider the parameterization in the feedback network:
Qi O0~N(0,02/
7
(S.165)
M
where 7) denotes the learning rate for the feedback network.
To ensure that the update AQth4 (1) holds, we have
T'
AQth M (ő(Qthi) -h111) d' (Qrhi)hi hn-
(S.166)
Here, because M (MM) and hiee +0(1 (((/MRbb-11), assuming AQih: ((1/M'). we obtain:
π (1<1 < L),
(S.167)
FL + 2b2 (1 = L)
Therefore,
pKL 0, iL = 22b-
(S.168)
If we optimize the feedback network for one step, we have
7'
QKLL = QieLo M 1
(S.169)
Mmiu(1.g)
QL = QL.
M M h1- ~ M
(S.170)
And,
Qicl 1 hy + pl) -1p] 1
(S.171)
M
Q2 h (1 h2 + ul) 'II 1
(S.172)
111/2-b1."
Therefore, in this case, Qt,1 = ii holds when
qe 2 1, Tr. 2 1/2 bi.
(S.173)
D ADDITIONAL EXPERIMENTS
D.1 PREDICTIVE CODING
D.1.1 LINEAR NETWORK
In Figure S.I. we measure the similarity of the inference vector with BP, GNT, and the analytical solution. With fewer inference iterations, the model behaves more like BP: however, the number of iterations increases, the model converges toward the analytical solution. Furthermore, as the middle layer width M, increases, the gap between GNT and BP decreases. Figure S.2 further demonstrates that reducing the output dimension M,. brings the model closer to BP. However, increasing Mr, moves the model away from BP, though this divergence is more gradual as 7t. approaches zero.
30

---

Published as a conference paper at ICLR 2025
Sim with BP gradient
1.000
Figure S.2 As Mr. approaches 1, the update vector in PC converges to that of BP. We conducted infer- ence training on 3-layer linear network and measured the similarity between PC and BP. The results demon- strate that PC approaches BP as Mr decreases and " increases.
Cos (Inference)
0.976
0.2
0.4
0.950
0.926
1.2
1.4
10
logg Mz
SP
NTK
muP (9z.
muP (fr. -1)
(VGGS 71 = (0.01)
Test Ace
80
75
70
-20
-10
-20
-15
-0
20
10 -20
15
loga LR
logg LR
logg LR
logg LR
SP
NTK
muP (9z. 0)
mmP (fr. 1)
(VGG3 nu = 0..1
width
Test Acc
80
70
-20
-15
-10 20
-15
-10 20
15
10
-20
15
10
loga LR
log, LR
loga LR
loga LR
Figure S.3 In VGG5, the learning rate also transfers across widths. In SP, the optimal learning rate shifts based model width, whereas in pP, remains fixed. Additionally, we trained with two different " values, and under yP (5z = -1), 1), the learning rate consistently transfers across widths, regardless of m. The model was trained for epochs on 1024 samples from FashionMNIST.
D.1.2 ADDITIONAL EXPERIMENTS ON μ TRANSFER FOR PC
Architecture In the main text, primarily focused MLP and CNN. However, our μP is architecture- urchitcturr-indeeendent The results for VGG5 are presented in Figure S.3. Furthermore, the μTransfer observed in Figure 4 also holds for MLP, demonstrated in Figure S.4
Loss Type In the main text, we mainly used mean squared error (MSE) loss. However, this can be replaced with cross-entropy (CE) loss. As demonstrated in Figure S.5 juP for PC also transfers the learning rate across widths when using cross-entropy loss.
Optimizer In this paper, primarily focus on weight updates using SGD. However, it is also possible to update the weights using Adam instead of SGD. In this case, the corresponding jP is as follows:
=, bictel 1/2, by { = 9>1 1.
(S.174)
For Adam. the scaling bj and c does not depend on F.. Additionally, in Adam, the gradients are normalized, which means that jP remains unchanged regardless of whether the gradients are generated by BP or PC. When considering the stability of the inference, scaling with respect to 7 can be treated in the same manner in the case of SGD.
NoL O, 7L
(S.175)
Train sample We reduced the number of training samples in most of the graphs for y/rannfer. By reducing the number of training samples, finite-width models are known behave more similarly to
31

---

Published as a conference paper at ICLR 2025
NTK
muP
mmP (9z -1)
80
80
Test
512
40
1024
2048
1096
-20
-10
-20
-15
-10
20
-15
-10 -20
-15
10
8192
1ogg I.R
logg L.R.
logg L.R.
logg LR
NTK
muP
muP (fz
80
width
0.01,7t
Test
40
1024
VA
2048
4096
20
16
10
-00
20
15
100
8192
1ogg
LR
logg
LR
Toga
logg
Figure S.4: Without F-ini, μP with 7z -1 transfers the learning rates across widths also in MLP. We trained 3-layer MLP on FashionMNIST without F-ini.
SP
NTK
mnP (5z - 03
muP (f 1)
85
width
80
128
Ted Arc
[mip o0)
75
256
512
70
1024
2048
4096
8192
1osg
LR
Toga LR
logg
LR
1ong
LR
Figure S.5 : μPP for PC transfers learning rates across widths even with Cross Entropy. We train 3-layer MLP for 40 epochs on 1024 samples from FaShionMNIST.
32

---

Published as a conference paper at ICLR 2025
NTK
muP (iL 0)
muP (ix -1)
0.1
width
mip F-ini)
128
Test. Ace
<p = -0001,1 72
256
512
1024
2048
4006
-20
20
8192
loga
loga
loga
loga
SP
NTK
muP (Fc. 0)
muP (fs -1)
3) 00.01. T1 0.1
width
mip (wefo F-in)
128
Test Acc
256
512
1024
2048
4006
-15
100
-
15
-15
-0
15
10
-5
8192
ToKa L.R.
Toya T.R.
Toga LR
Toga LR
NTK
muP (fe
mmP (%a
mip (w/fo F-in))
0..1.7y 1.. 0.1
80
width
128
Ted Acc
60
40
1024
=
20
2048
4036
10
Towa L.R
long I.R.
lowg T.R
Togg LR
muP (fz 0)
muP (9c 1)
= (1.1, 7T. = 0.1
80
widch
emn F-ini)
Test Ace
16
32
64
128
-20
10
-20
20
10
20
10
256
loga LR
loga LR
log,
log LR
SPP
NTK
muP (fr. 0)
miP (9p -1)
con (w/f F-in))
0.011.72 77. = 0.1
80
widch
Tes Ace
60
40
20
64
128
loga 1.R
loga L.I.
logu LR
loga LR
SP
NTK
muP (fz. 0)
muP
(Rz. = -1
onn (w/o F-ini)
0.1.7 71 0.11
wirlth
Test Acc
16
32
64
128
-20
10
20
10
200
-20
10
256
logg LR
loga Lit
log; LR
loga LR
Figure S.6 JP with 74. = -1 can constantly transfer the learning rates across width We confirmed Transfer when training PC with Adam to update parameters in both MLP and CNN. In the training of MLP without F-ini, we observe that pP with 7.. = -1 consistently stabilizes training and performs well. All experiments were conducted MNSSI with 1024 samples.
33

---

Published as a conference paper at ICLR 2025
SP
NTK
muf (92
muP (9c -1
80
60
[mipp]
Test
512
1021
2048
4096
-10
10-10
0
10-10 10
10-10 10
10
8192
loga 1
loga 7
log2
logg
Figure S.7 When training with Adam, muP with iz transfer 7t. width. We trained a 3-layer MLP on FasHionMNIST with Adam.
SP
NTK
muP (5c
muP (FL -1
wkdth
128
Tes Ace (mip)
256
1024
1096
8192
loga LR
log, LR
log LR
loga LR
SP
NIK
muP (5z =
muP (9z 1))
width
80
128
Test Acc (mipp
40
1021
2045
1096
-10
10-10
10-10
0
10-10
8192
loga 7/
loga 7
loga %
loga %
SP
NTK
muP (5t.
muP (fx. 1)
width
128
Test Arc
(xuip)
256
512
1024
20
2018
8192
loga 1
loga T(
loga 7f
loga 7/
Figure S.8 The results of yP for PC are independent of the number of training samples. We train a 3-layer MLP on FaSH with full training samples. The stability of μP holds even with all training samples.
34

---

Published as a conference paper at ICLR 2025
SP
NTK
mnP 11 0)
muF (%
width
Infermnc Loss
(MLPP F-ini)
10P
128
266
512
1021
10-1
2048
4096
20
-10
-00
15
-10
-20
15
-10
20
15
8192
logg LR
To52
logg
lowg LR
SP
NTK
muP (iz 0)
muP (Tz
10a
width
Inference Looe
w/o F-ini
128
10"
256
1024
10'
2048
-30
L.
loga L.R.
loga L.R.
L.R.
Figure S.9 When evaluating the loss after inference, only jP with 52 satisfies the empir- ical rule of "wider is better" Regardless whether F-ini is applied, J&P with &n = -1 consistently reduces the loss during inference with stability. We trained 3-layer MLP on FashionMNIST.
Infinite-widtl models, as has often been seen in papers examining the theoretical aspects of feature learning (Geiger al., 2020; Ishikawa & Karakida, 2024). However, even when training on the full dataset, μP remains stable across widths, as shown in Figure S.8
Inference Loss When considering the stability of inference, updating the parameters after inference. As shown in Figure S.9 when training a 3-layer MLP on FashionMNIST, only μP with 71 -1 consistently reduces the inference loss as the model width increases."
observe the loss before
Base width and inference iterations In ju-transfer, some research set the width of the smaller model used for tuning the learning rate, as the base width, denoted by M', and adjusts the learning rate using 1/t 1//(M/M') sometimes make jP with 7L more sensitive.
As shown in Figure S.10. the choice of M' (a smaller M') can
As shown in Figure S.11 the shift in the optimal learning rate at 7ı 0 with M' = 128 becomes more evident as the number of inference iterations increases. This is likely because, with more iterations, the dynamics of inference play more critical role in weight updates. In summary, to achieve stable 4 v/Transfer independent the base width and the number of inference iterations, we should use μP with YL- -1
Sequential Inference and Synchronous Inference In the main text, focused on Sequential Inference, where u, is updated layer by layer, starting from the output layer. However, Synchronous Inference, where all layers are updated simultaneously, can also be considered. For the differences between Sequential Inference and Synchronous Inference, see Algorithm. 1. As shown in Figure S.12 since μP for PC validated at fixed points, is also applicable Synchronous Inference.
Additional Experiments with Figure 3 Figure S.13 presents the results the same experiment shown in Figure 3 (Right), but with the CIFAR- CIFAR-10/CIFARR 1000IFAR=100 dataset and the VGG5 model. It is evident that even with CIFAR- 10, CIFAR- 100 and VGG5, jPP achieves higher accuracy compared to SP and NTK.
35

---

Published as a conference paper at ICLR 2025
SP
NTK
muP (%c.
muP (9
80
width
128
Tel. Are
(bact=]
60
40
1021
20
-20
20
8192
loga
loga Lit
loga LR
logg
SP
NTK
muP (8r. = 0)
muP (TL. -1)
80
wkdth
128
Ted Ace
(base=
60
256
512
1024
2048
20
1096
-20
-20
I.RL
LR
LR
logg LR
SP
NTK
mup (5z. = 0)
mul'
(iL 11
width
128
Tea Acc
(base=64)
60
256
512
1024
2048
20
4096
-20
15
10
-20
15
10
-10
-20
8192
Tosta LR
Toga L.R
loga LR
logt LR
NTK
(Sz.
(9z
width
Tes Acc
[base: 1288
128
60
256
512
40
1024
2048
20
4096
-20
15
10
-20 15
1.0
20
15
110
20
15
10
8192
logg
LR
logg LR
log, LR
loga
LR
Figure S.10 When the base width M' is large, uP with 7t. = 0 tends to fail with ptransfer. We train a 3-layer MLP on FashionMNIST. This suggests that μP with 74 -1 should be used when setting the base width, even with F-ini. The inference is performed for 100 iterations.
36

---

Published as a conference paper at ICLR 2025
(5x = 0)
(f4 11
80
width
(Inf 300
128
Test Are
60
256
512
40
1024
2048
20
4096
20
-10
-20
-15
-10 20
-1
-10
20
16
100
8192
Jogg LR
1og2 LR
logg LR
logg LR
NTK
(5t
muP (Fr. 1)
80
wkith
Test
(Inf Itee
512
40
1024
2048
1096
-20
15
10
-00
15
10
20
15
-10
20
-15
100
8192
Joga LR
log, LH
log2 LR
loga LR
SP
NTK
muP
muF (TL.
80
Iter 10)
Test Acc
40
2048
4096
-2
-15
10 -20
-55
-10 -20
-55
-10 20
-15
8192
108g
logg LR
logg
logg
maintains high inference stability and successfully performs ju-transfer even with large number of inference iterations. We conducted the experiment shown in Figure S.10 with varying numbers of inference iterations. Even with a larger number inference iterations, ,P with 72 -1 consistently transfers the learning rate across different widths.
Figure S.11 : μP with 77. -1
SP
NTK
muP
muP (7
01.01.1 TM 0.1
80
width
128
Tedt Ace
60
256
512
40
1024
20
2048
4096
-15
25
20
20
8192
logg T.R.
Ingg T.R.
1ngg L.R.
logg L.
NTK
(92 0)
muP
(To
1)
01
80
width
60
Tei
40
20
2048
1096
-25
20
15
25
15
15
8192
log. LR
logg LR
loga
LR
logg LR
Figure S.12 yP for PC also transfers learning rates across widths in synchronous inference. μP for PC can also be applied in synchronous inference. Note that when the base width is set to 128, in SI, the learning rate does not transfer in jP with 7g 0.
37

---

Published as a conference paper at ICLR 2025
CNN om CTPARIO
CNN on CIFAR100
VGG on MNIST
unP
Test Aee
(fu O)
60
(fu -1] NTK
maP
68
62
SP
66
M4-1
Ma-1
Mu-1
Figure S.13 pP scales better than SP and NTK. We trained CNN by PC with F-ini on CIFAR10/CIFAR 100 and a VGG5 on the full FashionMNIST dataset. The "wider is better" principle holds for paP.
SP
muP
SP
mul
width
0.8
0.8
128
A/1 (imput)
0.6
Ahr (ontput)
256
512
0.4
0.4
1021
2048
0.0
8192
20
40
0
20
20
16384
Epoch
Epoch
Epoch
Epoch
Figure S.14 : In Target Propagation, using μP ensures that Ah; remains consistent across widths. This figure shows the RMS norm of Ah; during training. For SP, Ah, in the input layer diminishes as the width increases, while Ah in the output layer diverges with increasing width. Consequently, the training dynamics become unstable. In contrast, with μPP, Ah; remains consistent across different widths in both the input and output layers.
Feedbacck Network (TanH)
Feedback Network (ReLU)
widsh
1.0
width
256
256
0.6
512
512
0.8
1024
1024
0.4
2018
2018
1096
1096
8192
$192
16384
16384
1os2 LR
loKg LRR
Figure S.15 Learning rate transfer in Feedback networks. We demonstrate that the learning rate in feedback networks transfers effectively across widths using toy data. Both the feedforward and feedback networks include Tanh/ReLU activation function following the linear layer.
SP
muP
SP
1.2
wislth
(bIL Tanll)
(MLP ReLU)
256
Loss
0.75
Loss
1.0
1024
0.25
0.8
2048
0.6
4096
1.0
10
5
10
10
8192
Logg,
logg
loga LR
loga LR
Figure S.16 : Learning rate transfer in Feedback networks (output layer). We show that the
learning rate in feedback networks transfers across widths using toy data. Unlike the hidden layers, the learning rate in the output layer does not transfer under the default setting, which requires jP scaling. Both the feedforward and feedback networks include Tanh/ReLU activation function after the linear layer.
38

---

Published as a conference paper at ICLR 2025
Figure S.17 Even when training the feedback network with DRL, ja demonstrates greater stability com- pared to SP. We trained 3-layer MLP on the FashionMNIST using DRL.
SP (Fash)onMNIST) NNIS))
width
80
128
Test Acc
256
512
1024
While SP exhibits shift in the maxi-
2048
mum learning rate as the model width increases, jP consistently transfers the
4096
8192
optimal learning rates across different
logg LR
logg LR.
16384
widths.
SI (FiIbioMMNISTT) mul (FasbionMNISST)
SP (CIFARIO)
moMP (CIPARIO)
willhh
50
width
128
128
Toot Aoc
256
1024
40
1024
2048
2048
0096
4006
8192
30
-2.5
2.5
5.0-2.5
8192
L.R.
Ioga LR
Iodg
logg
LR
Figure S.18 : /PP for TP remains stable regardless of the dataset or the number of training samples. We trained a 3-layer MLP on both FashionMNIST and CIFAR-10 using the full training samples. With μP, the learning rate successfully transfers across widths, ensuring that the maximum learning rate remains consistent regardless of the model width.
D.2 TARGET PROPAGATION
D.2.1 ADDITIONAL EXPERIMENTS ON TRAANSFER FOR TP
Temporal change of activation In Figure S.14, we observed Ah during the training of an MLP on FashionMNIST. SP exhibits a dependency of Ah on width, whereas jP demonstrates consistent behavior, independent of width.
Feedback Network As discussed in Section C.2, stable parameterization is crucial not only for feedforward but also for feedback networks. We verified this with μP, shown in Figures S.15 and S.16.
DRL Meulemans al. (2020) proposes the difference reconstruction loss (DRL) for constructing feedback networks. In Figure $.17, we empirically confirm that our jPP works effectively with DRL when training an MLP on FashionMNIST.
Training samples As with PC, in the case of TP, Figure 6 uses 1024 training samples. Similar results were observed when using the full training dataset, as shown in Figure S.18
39

---

Published as a conference paper at ICLR 2025
E
EXPERIMENTAL SETTINGS
Architecture and dataset
We trained the following three models:
MLP: We trained a 3-layer multilayer perceptron (MLP) with Tanh activation. The MLP
models do not include bias.
CNN: We trained a 3-layer CNN with Tanh activation. The models consist of two-layer convolutional layers and linear layer. We trained with different hidden widths where the width is proportional to the input dimension of the output layer. (For example, when the width is set 4, the input dimension of the final layer is 512.) Max pooling applied after the activation function.
VGG5:
We trained VGG-like model consisting of 4 convolutional blocks and 3 linear blocks, based on the structure described in (Pinchetti al., 2024). When the width to 8, it matches the VGG5 model in Pinchetti et al. (2024), with the channel sizes being [128, 256, 512, 512].
Dataset and batch size We used FashionMNIST and CIFAR- 10 datasets without applying any data augmentation. The settings for batch size and training samples were as follows:
PC In the experiments on y'rransfer, FashionMNIST was generally trained with 1024 training samples and batch size of 1024, except for Figure S.8 However, when training VGG5, the batch size was reduced verifying the scaling of μP with respect to width (Figure 3), all training samples were used, with a batch size of 1024.
due to memory constraints. In the experiment
TP In Figures S.14 and 6, we trained a 3-layer MIP using 1024 training samples. Note that in Figure S.18 in the Appendix, FashionMNIST,, and CIFAR-10 10 were trained using the full datasets. For the activation function of feedback networks, the same activation function as one used in the forward pass is utilized (i.e., v = Φ).
Training recipe Weight decay was not applied during the parameter updates for feedforward networks. For SGD, the momentum was set to 0.9, and for AdamW, the parameters (11, B2) were set to (0.9, 0.99).
PC The reduction mode for the loss function was set to "sum" to align the order of all terms in the free energy function.
TP For feedback networks, weight decay was set to 10 and the learning rate for the target was set to v 0.01. Before starting the main training, only the feedback network was trained for 5 epochs with the feedforward network fixed.
40


Reviews:
[{"review_id": "GTPFAJqe7y", "reviewer": "Reviewer_akLS", "summary": "This paper investigates local learning methods for neural networks as alternatives to backpropagation. It specifically focuses on predictive coding (PC) and target propagation (TP). The authors introduce maximal update parameterization (µP) for PC and TP in the infinite-width limit to enable hyperparameter transfer across different model widths, aiming to stabilize local loss optimization. They analyze deep linear networks to show that PC’s gradient transitions between first-order and Gauss-Newton-like gradients depending on parameterization. For TP, they find that µP favors feature learning over the kernel regime, a distinction from traditional BP.", "questions": "1. In the preliminaries I can't find any definition for $\\mathcal{L}$, I assume it is the loss function, are there any particular assumptions about it? \n2. A precise definition of the Maximal Update Parametrization in the text would be useful, especially for readers unfamiliar with the related work. \n3. In equation 5, where is $\\psi$ defined? \n4. Could the authors provide a more precise definition of stable training dynamics? \n5. Is the transfer of learning rates also observed for Cross Entropy loss? \n6. How sensitive are $\\mu P$  and $\\mu$Transfer to different initialization schemes?", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written. \n- The problem is conceptually well motivated.\n-  The manuscript includes both rigorous proofs and empirical experiments that validate the findings. \n- The observation that the optimal learning rate does not depend on the order of width is interesting. \n- The disappearance of the kernel regime in TP is interesting.", "weaknesses": "1. The derivation of $\\mu P$ assumes one-step gradient \n2. The derivation also assumes linear networks \n3. It is unclear how the findings extend to more general settings", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S8ay6Bdq3Q", "reviewer": "Reviewer_SFwu", "summary": "The submission presents derivation of the initialization and step-size scaling required to achieve hyperparameter transfer across width using the $\\mu P$ parameterization in the infinite width limit for two local learning algorithms, predictive coding and target propagation.", "questions": "The discussion of the implications of the $\\mu P$ parameterization for PC after corollary 4.3 and in §4.3, on the impact of the free parameter $\\gamma$, is very technical. It is not clear what the broader implications of $\\gamma = 0$ or $\\gamma = 1$ or the fact that the the PC gradients are more similar to those of standard backprop GD or GNT. A higher-level description of what the parameter $\\gamma$ represents, and what it means to be closer to 0 or 1, would help the reader grasp the importance of the results.\n\n- It would be worth to explicitly state that, by definition, $\\mu P$ depends on the algorithm used, and therefore needs to be derived to figure out what $\\mu P$ means in the context of PC and TP, which is the contribution of this paper. The sentence before the list of contributions leaves the \"$\\mu P$ depends on the algorithm\" part implicit.\n\n- The word \"inference\" is heavily overloaded in ML and a clarifying note as to what it means for PC in §2 would help. This is adressed later in §3.1 but a short parenthetical might be worth including.\n\n- \"The parameterization that induces the feature learning regime is called $\\mu P$\"\n  This phrasing implies that there is a unique parameterization that induces feature learning. It seems that the authors instead mean that \"$\\mu P$ induces the feature learning regime\"?\n\n- \"The training dynamics are referred to as stable when they neither vanish nor explore as the network width increases\"\n  What quantitiy should neither vanish nor explode?\n\n- \"the parameterization that achieves this [non-vanishing and non-exploding] is called a stable parameterization\"\n  The phrasing as \"the parameterization that achieves this\" implies that this parameterization is unique. Is this what is meant or should it be \"a\"?\n\n- In table 1; reminding the reader of what the parameters $b_l$, $c_l$ and $\\gamma_l$ would help, e.g. $b_l: initialization of the weights, $c_l$: scaling of the step-size. Adding a legend would help remind the reader of what SP, SGD, GNT, PC and TP mean.\n\n- Line 271-272 \"the $\\mu P$ of BP can be directly transferred to PC\" should also mention \"and leads to parameter transfer across width\", as this is what the figure shows.\n\n- Please add that $\\theta_l = 2a_l + c_l$ to the statement of Thm 4.1, assuming that those parameters indeed have the same meaning as in Prop 3.3.\n\n- An additional experiment showing that the $\\mu P$ for SGD does not lead to hyperparameter transformer on PC without FPA would strenghten the claim that the $\\mu P$ for PC is necessary \n\n- It is not clear whether the difference between GD, (S)GD and SGD is intended to meaningful or whether they are used interchangeably. If it is not, would it make sense to standardize to GD or (S)GD?\n\n- Figure 2 should explicitly state what is being compared; The cosine similarity measures the similarity between what and what?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "To my knowledge, the results are new as the infinite width limit has not received as much attention for local learning as for the typical backpropagation-based methods. The objectives of the paper are clear and the paper executes on its premises. The experimental results support the claim that the derived parameterization does achieve hyperparameter transfer.", "weaknesses": "The paper deals with two very specialized subjects, local learning and infinite-width limits. As a result, it has a difficult job in making its argument clear to a general audience. The current writing makes a reasonable attempt, but there are still points that can be clarified to help guide the reader that might be only be vaguely familiar with one of the topic. I give a list of specifics in the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nXliXcbAp3", "reviewer": "Reviewer_Nzx8", "summary": "This work studies parameterisation for local learning algorithms: for predictive coding and target propagation. The main goal is to understand maximal update parameterization ($\\mu P$), which allows transfer of hyperparameters across different widths. Authors prove optimal $\\mu P$ from one step of descent or for linear networks. Experimental results are provided based on Fashion MNIST.", "questions": "How is $\\eta'_l$ chosen in Equation (8)? Is it tuned in the experiments?\n\nFigure 3: what is the cause of sharp loss increase in the loss of wide networks when $\\bar \\gamma_L < -1$? Is there some theoretical explanation? Can be it that for $\\bar \\gamma_L = -1$, training would also diverge for wide enough network?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Authors give a thorough analysis of parametrization in local learning, motivated by previous progress of $\\mu P$ in backpropagation. The question is well-motivated by the fact that model sizes for local learning are increasing, thus it is important to understand which parameterizations are robust to scaling up the models. \n\nExperimental results on FashionMNIST confirm theoretical findings.", "weaknesses": "Presentation of the paper can be improved, in particular the notation section. Choice of $\\mathcal{L}$ is not introduced, notation for $\\gamma_l$ should be defined before Section 3.1. Also, $\\bar \\gamma_L$ is an important quantity, but is only defined implicitly in Theorem 4.1.\n\nBoth sections on PC and TP were a bit hard to follow (for someone outside of the field): In PC, for example, Equation (3) should also be complemented by the case $\\ell = L - 1$.\n\nIn Table 1, adding references to prior work and highlighting new contributions will be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJOtlmWR4e", "reviewer": "Reviewer_ubuK", "summary": "This paper considers maximal update parameterization for local learning, an alternative to backpropagation for optimizing neural networks. In particular, the authors consider predictive coding and target propagation. \n\nThe authors develop theory to show that hyperparameters can be transferred between networks of differing widths, matching existing guarantees for backpropagation, which helps reduce the amount of computation required for training with local learning. Further, they prove that under certain assumptions, target learning reduces to gradient descent, which implies that backpropagation and local learning will converge to the same model.", "questions": "Line 200: what is abc-parametrization?\n\nLine 278: what is $M^{\\overline{\\gamma}_L}$?\n\nLine 309: what is $\\mu$Transfer? A reference here would be helpful.\n\nWhat would the challenges be in extending the analysis to non-linear networks? \n\n[HRSS19 Mean-field Langevin dynamics and energy landscape of neural networks, arXiv preprint arXiv:1905.07769] studies infinite-width neural networks via the lens of mean-field Langevin dynamics. Would it be possible to study local learning through this lens, and could this potentially mitigate the requirement to develop a tensor program (line 535)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Originality: I’m not so familiar with this subfield, but in so far as I can tell, this work seems very novel as it seems to be the first work to introduce $\\mu$P for local learning. The authors provide theoretical guarantees showing that it is possible to reduce the required amount of hyperparameters in local learning. \n\nQuality: The theoretical results are well-justified by detailed proofs in the appendix and references to prior works. The authors provide empirical justification on real-world datasets as well.\n\nSignificance: The paper is significant as it provides justification on why local learning can match performance of models trained by back-propagation.", "weaknesses": "The content is rather inaccessible as the assumed knowledge is too specific for a broad conference such as ICLR. I have background in mean-field networks, but not local learning or $\\mu$P. It would be useful to add a brief appendix on these as a background (beyond what is already described in Section 3) to help the reader understand the paper.\n\nThe paper is also hard to follow due to the amount of abbreviations. It would also be useful to add backreferences for the techniques F-ini and FPA in lines 158, 162.\n\nSome of the theorems are missing assumptions, notation, or intuition. For example, in Prop. 3.3 (line 250), what are $\\delta_l$, $e_L$? In Cor. 4.3 (line 350), why do we need $a_L + b_L = 1$? \n\nThe experimental results are on relatively small datasets, e.g. FashionMNIST and CIFAR10: it would be nice to have validation on CIFAR100 as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)