The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks
Authors: Blake Bordelon
Cengiz Pehlevan
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Deep Learning Theory
Learning Rules
Representations

Abstract: It is unclear how changing the learning rule of a deep neural network alters its learning dynamics and representations. To gain insight into the relationship between learned features, function approximation, and the learning rule, we analyze infinite-width deep networks trained with gradient descent (GD) and biologically-plausible alternatives including feedback alignment (FA), direct feedback alignment (DFA), and error modulated Hebbian learning (Hebb), as well as gated linear networks (GLN). We show that, for each of these learning rules, the evolution of the output function at infinite width is governed by a time varying effective neural tangent kernel (eNTK). In the lazy training limit, this eNTK is static and does not evolve, while in the rich mean-field regime this kernel's evolution can be determined self-consistently with dynamical mean field theory (DMFT). This DMFT enables comparisons of the feature and prediction dynamics induced by each of these learning rules. In the lazy limit, we find that DFA and Hebb can only learn using the last layer features, while full FA can utilize earlier layers with a scale determined by the initial correlation between feedforward and feedback weight matrices. In the rich regime, DFA and FA utilize a temporally evolving and depth-dependent NTK. Counterintuitively, we find that FA networks trained in the rich regime exhibit more feature learning if initialized with smaller correlation between the forward and backward pass weights. GLNs admit a very simple formula for their lazy limit kernel and preserve conditional Gaussianity of their preactivations under gating functions. Error modulated Hebb rules show very small task-relevant alignment of their kernels and perform most task relevant learning in the last layer.

Published as a conference paper at ICLR 2023
THE INFLUENCE OF LEARNING RULE ON REPRESEN- TATION DYNAMICS IN WIDE NEURAL NETWORKS
Blake Bordelon & Cengiz Pehlevan
School Engineering and Applied Science Harvard University Cambridge, MA 02138, USA
(b (blake_bordelon, legnn]e harvard.
ABSTRACT
It is unclear how changing the learning rule of a deep neural network alters its learning dynamics and representations. To gain insight into the relationship be- tween learned features, function approximation, and the learning rule, we analyze infinite-width deep networks trained with gradient descent (GD) and biologically- plausible alternatives including feedback alignment (FA), direct feedback align- ment (DFA), and error modulated Hebbian learning (Hebb), well as gated linear networks (GLN). We show that, for each of these learning rules, the evolution the output function infinite width is governed by time varying effective neural tangent kernel (eNTK). In the lazy training limit, this eNTK is static and does not evolve, while in the rich mean-field regime this kernel's evolution can be deter- mined self-consistently with dynamical mean field theory (DMFT). This DMFT enables comparisons of the feature and prediction dynamics induced by each of these learning rules. In the lazy limit, we find that DFA and Hebb can only learn using the last layer features, while full FA can utilize earlier layers with scale determined by the initial correlation between feedforward and feedback weight matrices. In the rich regime, DFA and FA utilize a temporally evolving and depth- dependent NTK. Counterintuitively, regime exhibit more feature learning if initialized with smaller correlation between the forward and backward pass weights. GL.s admit a very simple formula for their lazy limit kernel and preserve conditional Gaussianity of their preactivations under gating functions. Error modulated Hebb rules show very small task-relevant alignment of their kernels and perform most task relevant learning in the last layer.
find that FA networks trained in the rich
INTRODUCTION
Deep neural networks have now attained state of the art performance across a variety of domains including computer vision and natural language processing (Goodfellow al. 2016 LeCun al.) 2015). Central to the power and transfcrability of neural networks is their ability to flexibly adapt their layer-wise internal representations the structure of the data distribution during learning.
In this paper, we explore how the learning rule that is used to train a deep network affects its learn- ing dynamics and representations. Our primary motivation for studying different rules is that exact gradient descent (GD) training with the back-propagation algorithm is thought to be biologically im- plausible (Crick 1989). While many alternatives standard GD training were proposed (Whit Wittiing- ton & Bogacz 2019). |019. it is unclear how modifying the learning rule changes the functional inductive bias and the learned representations of the network. Further, understanding the learned represen- tations could potentially offer more insight into which learning rules account for representational changes observed in the brain (Poort et al.) 2015 Kriegeskorte & Wei. 2021 Schumacher et al. 2022 Our current study is step towards these directions.
The alternative learning rules we study are eтror modulated Hebbian learning (Hebb), Feedback alignment (FA) (Lillicrap et al. 2016) and direct feedback alignment (DFA) (Nokland) 2016). These rules circumvent one of the biologically implausible features of GD: the weights used in the back- ward pass computation of emror signals must be dynamically identical to the weights used the

---

Published as a conference paper at ICLR 2023
forward pass, known the weight transport problem. Instead, FA and DFA algorithms compute an approximate backward pass with independent weights that are frozen through training. Hebb rule only uses global error signal. While these learning rules do not perform exact GD, they are still able to evolve their internal representations and eventually fit the training data. Further, experiments have shown that FA and DFA can scale to certain problems such as view-synthesis, recommendation systems, and small scale image problems (Launay et al. (2020). but they do not perform as well in convolutional architectures with more complex image datasets (Bartunov et al., 2018 However, significant improvements to FA can be achieved if the feedback-weighi have partial correlation
with the feedforward weights (Xiao et al. 2018 Moskovitz et al.] 20188 [Boopathy & Fiete [2022).
We also study gated linear networks (GLNs), which use frozen gating functions for nonlinearity (Fiat al. 2 Variants of these networks have bio-plausible interpretations in terms of dendritic
gates (Sezener et al. 202T). Fixed gating can mitigate catastrophic forgetting (Veness et al., 2021
Budden et al., (020) and enable efficient transfer and multi-task learning Saxe et al. (2022).
Here, we explore how the choice of learning rule modifies the representations, functional biases and dynamics of deep networks at the infinite width limit, which allows a precise analytical description of the network dynamics in terms of collection of evolving kernels. At infinite width, the network can operate in the lazy regime, where the feature embeddings at each layer are constant through time, or the cnlffeattur-learniig regime (Chizat et al. 2019 Yang & Hu 2021 Bordelon & Pehlevan (0022) The richness is controlled by a scalar parameter related to the initial scale of the output function.
In summary, novel contributions are the following:
1. We identify class of learning rules for which function evolution is described by dynamical effective Neural Tangent Kernel (eNTK). We provide a dynamical mean field theory (DMFT) for these learning rules which can be used to compute this eNTK. We show both theoretically and empirically that convergence to this DMFT occurs at large width N with error 1/2-1
We characterize precisely the inductive biases of infinite width networks in the lazy limit by computing their eNTKs at initialization. We generalize FA to allow partial correlation between the feedback weights and initial feedforward weights and show how this alters the eNTK.
3. We then study the rich regime so that the features are allowed to adapt during training. In this regime, the eNTK is dynamical and we give a DMFT to compute it. For deep linear networks, the DMFT equations close algebraically, while for nonlinear networks we provide numerical procedure to solve them.
4. We compare the learned features and dynamics among these rules, analyzing the effect of rich- ness, initial feedback correlation, and depth. We find that rich training enhances gradient- pseudogradient alignment for both FA and DFA. Counterintuitively, smaller initial feedback cor- relation generates more dramatic feature evolution for FA. The GLN networks have dynamics comparable GD, while Hebb networks, as expected, do not exhibit task relevant adaptation of feature kernels, but rather evolve according to the input statistics.
1.1 RELATED WORKS
GLNs were introduced by Fiat et al. (2019) as simplified model of ReLU networks, allowing the analysis of convergence and generalization in the lazy kernel limit. Veness et al.) (202T) provided a learning rule for deep GLNs which was extended by Budden
simplified and
et al. (2020) and provided an interpretation in terms of dendritic gating Sezener et al. (202 l.. These
works demonstrated benefits to continual learning due to the fixed gating. [Saxe et al. (2022) derived
exact dynamical equations for a GLN with gates operating at each node and each edge of the network
graph. Krishnamurthy et al. (2022) provided a theory of gating in recurrent networks.
Lillicrp et al. (2016) showed that, in a two layer linear network the forward weights will evolve to align to the frozen feedback weights under the FA dynamics, allowing convergence of the network to loss minimizer. This result was extended to deep networks by Frenkel et al.] (2019), who also introduced a variant of FA where only the direction of the target is used. RRffinetti et al. (202I) studied DFA in two-layer student-teacher online learning setup, showing that the network first undergoes an alignment phase before converging to one the degenerate global minima of the loss. They argued that FA's worse performance in CNNs is due to the inability of the forward pass gradients to align under the block- Toeplitz connectivity strucuture that arises from enforced weight sharing (d' Ascoli et al. 2019). Garg & Vempala (2022) analyzed matrix factorization with FA,
2

---

Published as a conference paper at ICLR 2023
proving that, when overparamettrrzzed it converges to a minimizer under standard conditions, albeit more slowly than GD. Cao et al. (2020) analyzed the kernel and loss dynamics of linear networks trained with learning rules from space that includes GD, contrastive Hebbian, and predictive coding rules, showing strong dependence of hierarchical representations on learning rule.
Recent works have utilized DMFT techniques to analyze typical performance of algorithms trained on high-dimensional random data (Agoritsas et al. 2018) [Mgnaacoo et al., 2020; [eeentaao et al.] 2021 Gerbelot al., 2022). In the present work, we do not average over random datasets, but rather over initial random weights and treat data as an input to the theory. Wide NNs have been analyzed infinite width both lazy regimes with the NTK (Jacot al.. 2018 Lee et al., 2019) and rich feature learning regimes (Mei et al., (2018). In the feature learning limit, the evolution of kernel order parameters have been obtained with both Tensor Programs framework (Yang & Hu 2021) and with DMFT (Bordelon & Pehlevan 2022). [Song et al. (202T) recently analyzed the lazy infinite width limit of two layer networks trained with FA and weight decay, finding that only one layer effectively contributes to the two-layer NTK. (Boopathy & Fiete (002) proposed alignment based learning rules for networks at large width in the lazy regime, which performs comparably to GD and outperform standard FA. Their Align-Ada rule corresponds our p-FA with p 1 in lazy large width networks.
2 EFFECTIVE NEURAL TANGENT KERNEL FOR LEARNING RULE
We denote the output of neural network for input Tx E RD as fy- For concreteness, in the main text we will focus on scalar targets fa E R and MLP architectures. Other architectures such as multi- class outputs and CNN architectures with infinite channel count also be analyzed the Appendix C For the moment, we let the function be computed recursively from a collection of weight matrices 0 (WW W ,.... W!..., w²) in terms of preactivation vectors h R.N where,
show in
fM
1 wh o(h)), htt No
w W' (( h2
W Z VD
1
where nonlinearity Φ is applied element-wiss The scalar parameter to controls how rich the network training is: small Tu corresponds to lazy learning while large % generates large changes to the features (Chizat et al. 2019). For gated linear networks, we follow Diit et al. (2019) and modify the forward pass equations by replacing o(h(,) with a multiplicative gating function @ (m,, hi, where gating variables m vD ay are fixed through training with Mij (O 1). To minimize loss L = -n (sSur (()))u), we consider learning rules to the parameters of the form
d w = -0 dt
2 Σ (1)(())) u. dt
d W'
E ((h VN
70
d w = 70 Σ A.
dt
VD
ac (0))))) The last layer weights w- are always updated with their true gradient. This corresponds to the biologically-plauibbe and local delta-rule, which merely correlates the error signals A, and the last layer features ((h()) (Widrow & Hoff 1960). In interme- diate layers, the pseudo- gradient vectors g% are determined by the choice of the learning rule. For concreteness, we provide below the recursive definitions of g for our five learning rules of interest.
where the error signal is (1) = -
o(h() o ((h))
(t) gft1| -thh = =b(ht) o ow- T w'( gf g = o(m f (t)
GD
VN
(pW + v1- pl- 9 get1
W. ~ 1(0,
p-FA
VN
5/h o(h()
(3)
~~~10011)
DFA
)(m)) (1),
GLN
Hebb
While GD uses the instantaneous feedforward weights on the backward pass, P-FA uses the weight matrices which do not evolve throughout training. These weights have correlation p with the initial forward pass weights (0). This choice is motivated by the observation that partial correlation between forward and backward pass weights initialization improve training (Liao al. 2016
3

---

Published as a conference paper at ICLR 2023
Xiao et al., 2018 Moskovitz et al. 2018), though the cost is partial weight transport initialization. However, we consider partial correlation at initialization more biologically plausible than the demanding weight transport each step of training, like in GD. For DFA, the weight vectors & are sampled randomly at initialization and do not evolve in time. For GLN, the gating variables m frozen through time but the exact feedforward weights are used in the backward pass. Lastly, which
we modify the classic Hebb rule (Hebb, 1949) to get A x E, (t) "h weighs each example by its current error. Unlike standard Hebbian updates, this learning rule gives stable dynamies without regularization (App. G). For all rules, the evolution of the function is determined by time-dependent eNTK Ky which is defined as
fl 0f. d0 ōt
L = Σ E 61) NN
(eoK(olh)),,, k
00
dt
1=0
1 (1, = N (1 (b), (/o011,0) - 1 (h) (00(((()(((())))
1
(4)
where the base cases G (t, = and (1, a 2u are time-invariant. The kernel G computes an inner product between the true gradient signals g V
of and the pseudo-gradient ah
e which is set by the chosen learning rule. We see that because & is not necessarily symmetric, K is also not necessarily symmetric. The matrix & quantifies pseudo-gradient / gradient alignment.
3 DYNAMICAL MEAN FIELD THEORY FOR VARIOUS LEARNING RULES
For each of these learning rules considered, the infinite width N - co limit of network learning can be described by dynamical mean field theory (DMFT) (Bordelon & Pehlevan 2022). At
infinite width, the dynamics of the kernels &' and G become deterministic over random Gaussian initialization of parameters O. The activity of neurons in each layer become i.i.d. random variables drawn from a distribution defined by these kernels, which themselves are averages over these single- site distributions. Below, we provide DMFT formulas which are valid for all of our learning rules
P M(,(1) (1) 70 S ds Σ (1,)(1)))
+ Ch (1,)))((()) Di (6,))),))))(
200 = (1) + +mo J ds Σ B (4, (1t (1,5)()))))))) ((h(())), 02(1) = 001,010)=
(u((0)) ~ CP(O, x 11-1),, (t,,s) = (00001(()))))))0)))) (1, = = 5
[r((s) ((12((0)
(r)()) ~ GP )) & (,,(t, 8) = (o)(())((())) (t, = (oull(s) 9p L+l (t)
(5)
The definitions of of. (l) depend on the learning rule and are described in Table | The (1) is the pre- greddetd field defined so that oh (t) (1h,,(()) The dependence of these DMFT equations on data comes from the base case $0 (t, 8) = F a, and error signal A ac
ar.
Rule 5,(() (t)e&
GD
FA (t)EE))
DFA (())
GLN )(m{)&f((t)
Hebb
(x(t)
Table 1: The field definitions for each learning rule. For p-FA, the field has definition 20) + +7o fo ds E. (t) where (t),(()), (t)) are Gaussian with (1)(())(())) = (t.s.. The Ç" field is an independent Gaussian with correlation (Z))))())) =
(t))(t) + vi- -i- -()((()))
= (t,).. For DFA, the zl field is static z ~ ~N(0
For GLN, we use (mf) ~ NO0,, K*) gating variable. Cº 0 except for p-FA with 0.
4

---

Published as a conference paper at ICLR 2023
We that, for {GD, p-FA, DFA, Hebb} the distribution of (t), (t) Gaussian throughout training only in the lazy To 0 limit for general nonlinear activation functions o(h). However, conditional on {my). the {h² 2 fields are all Gaussian for GLNs, For all algorithms except p-FA,
C O. For p-FA we have Cp (t, 8) -00 (اड(
( (00, (1))
((K,))")
0.5
0.4
0.0
150
250
352
450
500
1.00
220
302
420
500
202
200
350
(a) Loss Dynamics
(h) NTKK-Trget Alignment
(c) G Dynamics
GIN
Hebb
NN Exgt.
10
G
o DMFFT
2 DRIFT
(d) Final h Distributions
Final & Kernels
Final G kernels
Figure 1: The DMFT predicts feature dynamics of wide networks trained with gradient descent
(GD), feedback alignment (FA) with p = 0, gated linear network (GLN), and a error-modulated 8 1 Hebb rule (Hebb) in the feature learning regime. (a) The loss dynamics in a two layer
(L = I, = 2000) network trained with these learning rules at richness %o 2. The network is trained collection of P 10 random vectors in D 50 dimensions.
(b) The cosine similarity of the eNTK with the targets A(K, yy y Ky reveals increasing alignment for all algorithms. Flv Though FA starts with the lowest alignment, its final NTK task alignment exceeds that of GD. (c) The dynamics of the
kernel G also reveals increasing correlation of g with ÿ. FA starts with G 0 but G increases to non-zero value. (d) The distribution of hidden layer preactivations after training reveals non- on-Gaussian statistics for both GD and FA, but approximately Gaussian statistics for GLN. (e)-(f) The final & and G kernels from theory and experiment.
As described in prior results on the GD case (Bordelon & Pehlevan 2022), the above equations can be solved self-consistently in polynomial (in train-set size P and training steps T) time. With an estimate of the dynamical kernels { {Tf.. (t, s), (t, s), Gf, s)}, one computes the eNTK (ww(t) and eтror dynamics (t). From these objects, we can sample the stochastic processes { 2 2 which can then be used to derive new refined estimates of the kernels. This procedure is repeated until convergence. This algorithm can found App. A An example of such solution is provided in Figure for two layer ReLU networks trained with GD, FA, GLN, and Hebb. We show that our self-consistent DMFT accurately predicts training and kernel dynamics, as well as the density of preactivations (t)} and final kernels per, Gpm) for each learning rule. We observe substantial differences in the learned representations (Figure Te), all predicted by our DMFT.
3.1 LAZY OR EARLY TIME STATIC- KERNEL LIMITS
When Yo + O, we see that the fields h (t) and (t) are equal to the Gaussian variables uf, (0) and of (0). In this limit, the eNTK Ky, remains static and has the form summarized in Table 2 in terms of the initial feature kernels te and gradient kernels G We derive these kernels in Appendix D
The feature P matrices Th G in Table 2 are computed recursively as
t- = (o(u)o(u) ) Gº Gl+1 o (o(u))(u)))
(6)
u-/10,00 1)
5

---

Published as a conference paper at ICLR 2023
L
FA 0.5
GLN
1.0
1.00
100
0.6
0.6
055
0.50
0.4
021
0:000
p.0055
0.00
0.25
0.00
02:
p=100
(a) ReLU FA varying P
(b) ReLU FA varying L
(c) ReLU GLN varying
10-
L=2,0002
0 P.10.
L=2omi0
10a
12-
13-3
13-
10
10'
16
10a
101
10-
10t
N
N
(d) o convergence
(c) G convergence
(1) eNTK convergence
Figure 2: The lazy infinite width limits of the various learning rules can be fully summarized with their imitial eNTK. (a) The kemels of p-aligned ReLU FA and ReLU GLN for inputs separated by
angle 0. (a) The kernels for varying P in p-aligned FA. Larger p has sharper peak in the kernel around 0 = 0. The p + 0 limit recovers the NNGP kemne & while the - limit gives the back- prop NTK. (b) Deeper networks with partial alignment
0.5. (c) ReLU-GLN kernel sharpens with depth. (d)-(e) The relative error of the infinite width &º, G kernels in width N ReLU neural network. The late layer & and early layer G kernels have highest errors since finite size effects accumulate on forward and backward passes respectively.
(f) Finite width corrections to eNTK are larger for small p and large depth L. All square errors go as KN Kxl ~ (1/N))
Rule
GD
P-FA
DFA
GLN
Hebb
KE
Ei=o
G pv
El pz pli-e
(ő(m)))((m)) )
K
Lh
Juv
The initial eNTK Ke for each learning rule. The GD kernel is the usual initial NTK of Jacot et al. 2018). For p-aligned FA, each layer l's contribution to the eNTK is suppressed by factor p For DFA and Hebb, only the last layer feature kernel &- contributes the NTK, For GLN, each layer has an identical contribution.
Table 2:
with base cases T K and GL+1 11 We provide interpretations of this result below.
Backpropagation (GD) and = FA recover the usual depth L NTK, with contributions from every layer Kar = Dn G L at initialization. This kernel governs both training dynamics and test predictions in the lazy limit %
pu
0 (Jacot et al. 2018 Lou et al. 2022).
p = 0 FA, DFA and Hebb are equivalent to using the NNGP kernel K Kyn LL giving the Bayes posterior mean (Matthews et al. 2018 Lee et al. 2018 Hron et al. (020). In the 7o. P 0 limit, only the dynamics of the readout weights w contribute to the evolution of fu since error signals cannot successfully propagate backward and gradients cannot align with pseudo- gradddngss (App D). The standard = 0 FA will be indistinguishable from merely training w- with the delta-rule unless the network is trained in the rich feature learning regime 7o > 0, where & can evolve. This effect was also noted in two layer networks by [Song et al. (2021).
p-FA weighs each layer with seale p--(, since each layer's pseudo-gradient is only partially correlated with the true gradient, giving recursion & = p with base case ÕL+1 = GL+11
GLN's kernel in lazy limit is determined by the Gaussian gating variables {mf} NN(0 K*).
We visualize these kernels for deep ReLU networks and ReLU GLNs for normalized inputs |x|
| D. by plotting the kernel function of the angle 0 separating two inputs cos(()

---

Published as a conference paper at ICLR 2023
D "x='', x'. We find that the kernels develop sharp discontinuity at the origin 0 = O, which becomes more exaggerated as and L increase. We further show that the square difference width N kenels and infinite width kernels go O(N=1). We derive this scaling with perturbative argument in App. H. which enables analytical prediction of leading order finite size effects (Figure 7 In the lazy 70 limit, these kernels define the eNTK and the network prediction dynamics.
3.2 FEATURE LEARNING ENABLES GRADIENT/PSEUDO-GRAR GRADDIEN ALIGNMENT AND KERNEL/TASK ALIGNMENT
In the last section, we saw that, in the "Yo - limit, all algorithms have frozen preactivations and pregradient features { (1), (t)). A consequence of this fact is that FA and DFA cannot increase their gradient-pseudogradient alignment throughout training in the lazy limit To 0. However, we increase then the gradient features gf. (t) and
0 (t) evolve in time and can increase their alignment. Figure 3 we show the effect of increasing "To on alignment dynamics in depth 4 tanh network trained with DFA. In (b), we see that larger is associated with high task- alignment of the last layer feature kernel Φ_ which becomes essentially rank one and aligned to yy The asympotic cosine similarity between gradients and pscudogradients also increase with To- The eNTK also becomes aligned with the task relevant directions (shown in Figure c), like has been observed in GD training (Baratin al. 2021 [Shan & Bordelon 2021 Geiger et al. 202 Atanasov al. 2022). We see that width N networks have a dynamical eNTK K which deviates from the DMFT eNTK (t) by O(1/N) in square loss. DMFT is more predictive for larger % networks, suggesting a reduction in finite size variability due to task-relevant feature evolution.
-02
1.0-
04
0.5
0.:
0.6
Corlg 5
0.6
0.-
02
01
(a) DFA Train Loss
(b) $4 Alignment
(c) 9 g Correlation
4071 Ditt
402 CWNT
(d) Final NTK Aligns to Task
(e) Dynamical NTK Convergence
Figure 3: Feature Learning enables alignment for depth 4 (L 3 hidden layers) tanh network trained with direct feedback alignment (DFA) with varying To- (a) Training loss for DFA networks with width N 4000 with varying richness to shows that feature learning accelerates training, as predicted by DMFT (black). (b) The alignment (cosine similarity) of the last layer kernel $ with the target function reveals successful task depedent feature learning at large "oo (c) The dy-
gf (())((())) namics of pseudo-grad./rrad. correlation corr(g. .) Er, (x) (oT averaged over lay- ers l and datapoints μ.. Larger generates more significant alignment between pseudogradients and gradients. (d) The final NTKs as function To reveals increasing clustering of the data points by class. (c) The error of the DMFT approximation for K's dynamics as function of N:
( (ox (0)x000)) 2/,
O(N-1), where the averages are computed over the time interval of training. This error is smaller for larger feature learning strength 70-

---

Published as a conference paper at ICLR 2023
3.3 DEEP LINEAR NETWORK KERNEL DYNAMICS
When To > 0 the kernels and features in the network evolve according to the DMFT equations. For deep linear networks we can analyze the equations for the kernels in closed form without sampling since the correlation functions close algebraically (App. [E. In Figure 4 we utilize our algebraic DMFT equations to explore p-FA dynamics in a depth linear network. Networks with larger p train faster, which can be intuited by noting that the initial function time derivative de Ic=0 ~ E/= p'- I-p 1-p
"+ is an increasing function of p. We observe higher final gradient
pseudogradient alignment in each layer with larger P. which is also intuitive from the initial condi- tion (0) = p However, surprisingly, for large initial correlation p. the NTK achieves lower task alignment, despite having larger (+ (t). We show that this is caused by smaller overlap of each layer's 's feature kernel H²(I) with yy" __ Though this phenomenon is counterintuitive, we gain more insight in the next section by studying an even simpler two layer model.
1-1
1-2
1-3
0.6
666
106
426
384
360
366
420
269
360
480
100
200
330
430
(a) A-iigneed Loss Dynamics
(b) Prradeet-Pseedogreen Kernel Dynamics
AUK-29
35
125
0.0
126
200
d2
303
260
364
400
100
203
340
460
100
200
300
408
(c) NTK-Task Alignment
(d) Feature Kernel Task Overlap
Figure 4: The initial feedback correlation p alters alignment dynamics in on the FA dynamics in depth 4 (L- = 3 hidden layer) linear network. (a) Larger leads to faster initial training since the scale of the eNTK is larger. (b) Further, larger leads to larger scales of G(t) = g (t). (c) However, smaller o leads to more alignment of the NTK K(t) with the task-relevant subspace, measured with cosine similarity A (, yy ). (d) The feature kernel H (t) overlaps with y reveal that H (t) aligns more significantly in the small p networks.
3.3.1 EXACTLY SOLVEABLE DYNAMICS IN TWO LAYER LINEAR NETWORK
We can provide exact solutions to the infinite width GD and p-FA dynamics in the setting of Saxe et al. (2013), specifically a two layer linear network trained with whitened data Kÿ = 5pر- Unlike Saxe et al. 2013)'ss 's result, however, we do not demand small initialization scale (or equivalently large 7o). but rather provide the exact solution for all positive To. We will establish that large initial correlation P results in higher gradient/sseudogradde feature kernel (t) with the task relevant subspace yy" T.
alignment but lower alignment of the hidden
We first note that when K = I, the GD or FA hidden feature kernel (t) only evolves in the rank-one yy" T subspace. thus suffices to track the projection of H() on this rank one subspace, which we call (t). In the App- F we derive dynamics for Hy for GD and p-FA
((() 1 A())= 2G(t) 20:1-2p11+-2
dA
II+ (())
GD P-FA
(t)
#t
(7)
=
70y
-1++
We illustrate these dynamics in Figure The fixed points are H, = V1 + n002 for GD and for p-FA, Hy = 1 + d² where a is the smallest positive root of _aaa + (1 + p)a = Moy. For both GD and FA, that increasing To results in larger asymptotic values for H, and G. For p-FA the fixed point of a's dynamics is strictly decreasing function of p since
da < 0, showing that the final dp
8

---

Published as a conference paper at ICLR 2023
value of Hy is smaller for larger p. On the contrary, have that the final -= 2²a² is a strictly increasing function of since d = fa²+(1+0) > O. Thus, this simple model replicates
the phenomenon of increasing G and decreasing Hy as p increases. For the Hebb rule with K I, the story is different. Instead of aligning H along the rank-one task relevant subpace, the dynamics instead decouple over samples, giving the following P separate equations
d To (3) - (t)) de Hpp = pt)
dt A = -
(8)
From this perspective, we see that the hidden feature kernel does not align to the task, but rather increases entries in overall scale as is illustrated in Figure(5((b)
1.0
0.25
0.8
pw3
20
DNTT
201
0.25
0.4
02
0.15
820
10"
13
16F
15"
(a) Loss Dynamics
(b) Kernel-Task Alignment
(c) Feature Learning vs To
Figure 5: The feature kernel dynamics and scaling with go for GD, p-FA, and Hebbian rules in an exactly solveable two layer linear network. (a) The loss dynamics for all algorithms reveals that = 0 FA and Hebb rules have same early time dynamics and that p 1 FA and GD have same carly-time dynamics. However all loss curves become distinct at late times due to different eNTK dynamics. (b) The alignment of the kemnel to the target function (t) = - H Hy (t)TH
increases significantly for GD, and FA, but not for Hebb, reflecting the task independence of the learned representation. (c) The movement of the feature kernel AH, = lim,, Hy)() Hy (0) as function of 7o for GI), and p 0, FA. At small feature learning strength, all algorithms have updates on the order AH, ~ -0- At large "o. GD has AHl ~ To while FA has AH, ~ 70 The p 1 FA (green) has lower A, than the p 0 FA across all yo-
2/3
DISCUSSION
We provided an analysis of the training dynamics of a wide range of learning rules at infinite width. This set of rules includes (but is not limited to) GD, p-FA, DFA, GLN and Hebb as well as many others. We showed that each of these learning rules has an dynamical effective NTK which concentrates over initializations infinite width. In the lazy To - 0 regime, it suffices to compute the the initial NTK, while in the rich regime, we provide a dynamical mean field theory to compute the NTK's dynamies. We showed that, in the rich regime, FA learning rules do indeed align the network's 's gradient vectors to their -reudingradients and that this alignment improves with "to We show that initial correlation p between forward and backward pass weights alters the inductive bias of FA in both lazy and rich regimes. In the rich regime, larger networks have smaller eNTK evolution. Overall, our study step towards understanding learned representations in neural networks, and the quest to reverse-engineer learning rules from observations of evolving neural representations during learning the brain.
Many open problems remain unresolved with the present work. We currently have only implemented our theory in MLPs. An implementation in CNNs could explain some of the observed advantages of partial initial alignment in p-FA (Xiao et al., 2018 Moskovitz et al. 2018 Bartunov et al.] 2018]] Refinetti et al. 2021). In addition, our framework is sufficiently flexible to propose and test new learning rules by providing new of.(() formulas. Our DMFT gives recipe to compute their initial kernels, function dynamics and analyze their learned representations. The generalization perfor- mance of these learning rules varying Yo is yet to be explored. Lastly, our DMFT is numerically expensive for large datasets and training intervals, making it difficult to scale up to realistic datsets. Future work could provide theoretical convergence guarantees for our DMFT solver.

---

Published as a conference paper at ICLR 2023
REFERENCES
Elisabeth Agoritsas, Giulio Biroli, Pierfrancesco Urhani, and Francesco Zamponi. Out-of- cquilibrium dynamical mean-field equations for the perceptron model. Journal of Physics A: Mathematical and Theoretical, (10e: 51(8):085002, 2018.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners:
The silent alignment effect. In International Conference on Learning Representations, 2022.
URL bttps:://ppereeee
net www.:/eeet/trttte/oeee
forum? id- INvffqAdoom
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vin- cent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Arti- ficial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 226992277 PMLR, 13-15 Apr 2021. URL https: /ttpoc////rcceeeeee ngs mlr ptwpss/v1s.1/////
baratin2la. html
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of lologgicall--moivateee deep learning algorithms and archi- tectures. Advances in neural information processing systems, 31, 2018.
Carl M Bender, Steven Orszag, and Steven Orszag. Advanced mathematical methods for scientists and engineers I: Asymptotic methods and perturbation theory, volume 1. Springer Science & Business Media, 1999.
Akhilan Boopathy and Ila Fiete. How to train your wide neural network without backprop: An input-weight alignment perspective. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Confer- ence Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2178-2205. PMLR, 17-23 Jul 2022. URL htttps://proceedi: wttpsss....rrrs//iii. wwwpppressvll. boopathyy2a. html
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evo- lution in wide neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
fett:/orrmmiddsiiinrreee pwrPPTT
David Budden, Adam Marblestone, Eren Sezener, Tor Lattimore, Gregory Wayne, and Joel Veness. Gaussian gated linear networks. Advances in Neural Information Processing Systems, 33:16508- 16519, 2020.
Yinan Cao, Christopher Summerfield, and Andrew Saxe. Characterizing emergent representations in a space of candidate learning rules for deep networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8660-866777 8670. Curran Associates, Inc., 2020. URL btttps://proceedinggg .ww.iinnnnnnneu....e..emmm.....r hcc/paper/202//EL110 (tw.el//27577777 http:///22...1112022222222266661
Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics of first
order methods with random data. arXiv preprint 2rXiv:21/2.05522 2021.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
Francis Crick. The recent excitement about neural networks. Nature, 337 1989.
A Crisanti and H Sompolinsky. Path integral approach to random neural networks. Physical Review E, 986))00), 2018.
Stéphane d' Ascoli, Levent Sagun, Giulio Biroli, and Joan Bruna. Finding the needle in the haystack with convolutions: on the benefits of architectural bias. Advances in Neural Information Process- ing Systems, 32, 2019.
Jonathan Fiat, Eran Malach, and Shai Shalev-Shwarzz. Decoupling gating from linearity. arXiv preprint arxiv: 100e: 0000200032,2..0.00 2019.
10

---

Published as a conference paper at ICLR 2023
Charlotte Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: direct random target projection as a beedback-alignmeet algorithm with layerwise feedforward training. arXiv preprint 1XXx::1909.01111, 10, 2019.
Shivam Garg and Santosh Vempala. How and when random feedback works: A study of low-rank matrix factorization. In Gustau Camps- Valls, Francisco R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 4070-4108. PMLR, 28-30 Mar 2022. URL TRL.https:: httpt: ftttps://proceebee ngs . ml ppress/vl5) ga httpg/gggg22aaa...
Mario Geiger, Leonardo Petrini, and Matthieu Wyart. Landscape and training regimes in deep learning. Physics Reports, 924:1-18, 2021.
Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborova. Rigorous dynamical mean field theory for stochastic gradient descent methods, 2022. URL
htttps://arxxz wwtp:////ppaa/....oo//// 06591
Gabriel Goh. Why momentum really works. Distill, 2(4):e6, 2017.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Donald O. Hebb. The organization of behavior: A neuropsychological theory. Wiley, New York, June 1949. ISBN 0-8058-4300-0.
Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, and Jascha Sohl-Dickstein. Exact posterior distributions of wide bayesian neural networks. arXiv preprint 200::2000.005444 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Mehran Kardar. Statistical physics fields. Cambridge University Press, 2007.
Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Nature Reviews Neuroscience, 22(11):703-718, 2021.
Kamesh Krishnamurthy, Tankut Can, and David J Schwab. Theory of gating in recurrent neural networks. Physical Review X, 12(1):01101, 2022.
Julien Launay, lacopo Poli, François Boniface, and Florent Krzakala. Direct feedhack alignment scales to modern deep learning tasks and architectures. Advances in neural information processing systems, 33:9346-9360, 2020.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):3364444 2015.
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. International Confer- Learning Representations, 2018. URL hattps://openreeeee wwwp:e/w.ee/ffoorreeee
1LEA-M-OO
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Jachoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: empirical study. Advances in Neural Information Processing Systems, 33:15156-15177 2020.
Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in backpropaga- tion? In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 1-10, 2016.
11

---

Published as a conference paper at ICLR 2023
Yizhang Lou, Chris E Mingard, and Soufiane Hayou. Feature learning and signal propagation in deep neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine learning, volume 162 Proceedings of Machine Learning Research, pp. 14248-14222 PMLR, 17-23 Jul 2022. URL htt.hhts////ppooee
ngs. ml wtt.:sm.w........... w.w..p.essv... ou22a. .html
Alessandro Manacorda, Grégory Schehr, and Francesco Zamponi. Numerical solution of the dy- namical mean field theory of infnite-dimensional equilibrium liquids. The Journal of chemical physics, 152(16):164500) 2020.
Paul Cecil Martin, ED Siggia, and HA Rose. Statistical dynamics of classical systems. Physical Review A, 8(1):423, 1973.
Alexander G.D.G. Matthews, Jiri Hron, Mark Rowland, Richard E. Tumer, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. International Conference on Learn- ing Representations, 2018. URL htLLttttss//ppereerrr/
net/ forum -ttp:uumm-----------e---
Song Mei, Andrea Montanari, and Phan-Minhe Nguyen. A mean field view of the landscape two- layer neural networks. Proceedings of the National Academy of Sciences, (1163))E7665-E7777 2018.
Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33:9540-9955 9550, 2020.
Theodore H Moskovitz, Ashok Litwin- and LF Abbott. Feedback alignment in deep convo- lutional networks. arXiv preprint
2018.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Jasper Poort, Adil Khan, Marius Pachitariu, Abdellatif Nemri, Ivana Orsolic, Julija Krupic, Mar- ius Bauza, Maneesh Sahani, Georg B Keller, Thomas D Mrsic-Flogel, et al. Learning enhances sensory and multiple non-sensory representations primary visual cortex. Neuron, 86(6):1478- 1490, 2015.
Maria Refinetti, Stéphane d' Ascooi,, Ruben Ohana, and Sebastian Goldt. Align, then memorise: the dynamics of learning with feedback alignment. In International Conference on Machine Learning. pp.
PMLR, 2021.
Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. In International Conference Machine Learning, pp. 19287- 19309. PMLR, 2022.
Andrew M Saxe, James I. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6122 2013.
Joseph Schumacher, Matthew K McCann, Katherine Maximov, and David Fitzpatrick. Selective enhancement of neural coding in vl underlies fine-discrimination learning in tree shrew. Current Biology, 32(15))324,33260, 2022.
Eren Sezener, Agnieszka Grabska- Barkkkkkaaiiikaa,, Dimitar Kostadinov, Maxime Beau, Sanjukta Krish- nagopal, David Budden, Marcus Hutter, Joel Veness, Mathew Botvinick, Claudia Clopath, et al. A rapid and efficient learning rule for biological neural circuits. BioRxiv, 2021.
Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv e-prints, pp. arXiv-2105, 2021.
Ganlin Song. Ruitu Xu, and John Lafferty. Convergence and alignment of gradient descent with ran- dom backpropagation weights. Advances in Neural Information Processing Systems, 34:1 19888- 19898, 2021.
12

---

Published as a conference paper at ICLR 2023
Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Agnieszka Grabska- Barkinkka, Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, et al. Gated linear networks. In Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pp- 10015-10023, 2021.
James CR Whittington and Rafal Bogacz. Theories of error back propppropngttion in the brain. Trends in cognitive sciences, 23(3):235-255) 2019.
Bernard Widrow and Marcian E Hoff. Adaptive switching circuits. Technical report, Stanford Univ Ca Stanford Electronics Labs, 1960.
Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio. Biologically-plausibe learning algo- rithms can scale to large datasets. arXiv preprint arXiv:1811.03355 2018.
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural net- works. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 18-24 Jul 2021. URL https::/ https::/proceeddnngss mmr. mtt.:ssssm...p..///omm P139/yang2lc..te html
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A ALGORITHM TO SOLVE NONLINEAR DMFT EQUATIONS
Algorithm 1: Alternating Monte-Carlo Solution to Saddle Point Equations
Data: K y, Initial Guesses ( G' C", D' Sample count S,
Update Speed 3
Result: Network predictions through training (4),, correlation functions
{ Gh G. GELL 1==1 response functions (A', B , D.
T0 K & 11", GL++ =11 11"
while Kernels Not Converged do
From { ,G) compute KNTK((1 (1, and solve $L((1)= E. (N())KTTK(1,,)
{= 1; while 8 < L++1d do
Draw S samples {u (u (t)]
(r (4) uf (1)) GP (0.
G++1
Ge+1]
Ge+IT
Ge+1
Solve equation 5 for each sample to get (he (t), of (,)(t), (t), of (t) Use learning rule (Table to compute (t)]s
Compute new correlation function { G G., G9) estimates:
hh.new (t, 5) = Ene(s) (n (t)))(h (s)),
10
G.aen (t, S Ene(s] Ian (n)))..)s))
11
Ghyew (1,8) 3 Ene(s) Gf.n (1)T (8),
Ghinew (t,s)= (t, Ene(s) Ee (t)gl. (s)
13
Solve for Jacobians on each sample Od(h()) do(h")
0g
0g'
14
Br
Gog
Aut
But
Compute response functions (A B
C D
estimates:
16
A &,ne CLaow >
E @o(k") Ort
Be-1,ncu
Ene(s) (S)
0g
B8(A Ov
D De-1acm 3/ Ene(s]
88
Dut
18
1t E +++ 1;
19
end
{= 1;
21
while < + 1 do
Update correlation functions
&h (1- -² + 3$ neeu G (1- G) f.ae
23
G - (1- m.w. E -1- + B BB neew
G)
24
if <<<11 T. then
26
Update response functions f A' + (1 A + 3A Enee
B t (1 (1-3) B² + B neew
27
(1- + L.w end
Dd + - D' + 3D knew
28
29
{ E +1
end
32 end
return (f.(0), (0' G G.G.G.GGGCC.. G.G (' C C,D
The sample-and-solyy procedure we developed and describe below for nonlinear networks is based numerical recipes used in the dynamical mean field simulations in computational physics Man- acorda et al.] (2020) and is similar to recent work in the GD case [ordelon & Pehlevan (2022). The basic principle is leverage the fact that, conditional order parameters, easily draw
14

---

Published as a conference paper at ICLR 2023
samples (t)
from their appropriate GPs. From these sampled fields, identify the kernel order parameters by simple estimation of the appropriate moments. The algorithm is provided in Algorithm !
The parameter 3 controls recency weighting of the samples obtained at each iteration. If 3 1, then the rank of the kernel estimates is limited the number of samples S used in a single iteration, but with 3 < 1 smaller sample sizes S can be used to still obtain accurate results. We used / = 0.6 in our deep network experiments.
B DERIVATION OF DMFT EQUATIONS
In this section, we derive the DMFT description of infinite network dynamics. The path integral theory we develop is based on the Vartin-Sigga-RRos--DD Dominicis-lanssen (MSRDJ) framework Martin et al. 1973). A useful review of this technique applied to random recurent networks can be found here Crisanti & Sompolinsky (2018). This framework was recently extended for deep learning with GD in (Bordelon & Pehlevan 2022).
B.1 WRITING EVOLUTION EQUATIONS IN FEATURE SPACE
First, we will express all of the learning dynamics in terms of preactivation features (t, VN (() pre-gradient features (1) VN T and pseudogradient features (t). Since we would like to understand typical behavior over random initializations of weights
(()) = (0), (O)...... (0)). we want to isolate the dependence of our evolution equations by (0). We achieve this separation by using learning dynamics for (t)
70 J ds
- - (0)
The inclusion of the prefactor VN in the weight dynamics ensures that f O no (1) and 4r
Tx (71) at initialization (Chizat al. 2019 Bordelon & Pehlevan 2022). Using the forward and backward pass equations, we find the following evolution equations for our feature vectors
h0,00 = xx,10 + 70 / ds Σ
(1) xx(1)
1 w (())(hE)))))
VN
u=1
21,()) (t) + 7o / ds
(s tt
w (0) 9, (t), (10)
VN
where we introduced the following feature and gradient/pseudo srragpgdesudddggggeeeteee kernels
1 Th. (1, N ohh (1)) (s)) (1, N- 9 (s).
(11)
The particular learning rule defines the definition of the pseudo-gradient (f.)). We note that, for all learning rules considered, the pseudogradient (t is function of the fields (t) 2f, (t), (t), (t), (E)(()))))))) conditional on the value of the kernels { Gr.. The additional fields have definitions
1 1 W
(t)
VN w'(() (t) 2 (1+)(t)
g g(+)(t)
(12)
VN
and are specifically required for p=FA with > O since ().(1) = po(h(((1) @ ()(()) +
pi-pP (t)) e (t). The fields m 1 M'rx,
x are required for GLNs.
VD
All of the necessary fields (t), (t). are thus causal functions of the stochastic fields (x/ (t), E (t), m (t), (t)} and the kernels ( G.). It thus suffices to characterize the dis- tribution of these latter objects over random initialization of (0)) in the - oo limit, which we study in the next section.
15

---

Published as a conference paper at ICLR 2023
B.2 MOMENT GENERATING FUNCTIONAL
We will now attempt to characterize the probability density of the random fields
X₁ -1) (O)o() (1)
W (0)
gr (t),
mi M'
VN
VD
(1) 1 (0) ( (1)
W 9F (1).
(13)
VN
It is readily apparent that the fields mf. are independent of the others and have Gaussian distribu- tion random Gaussian M These fields, therefore do not can be handled independently from the others, which are statistically coupled through the initial conditions. We will thus characterize the moment generating functional of the remaining fields (x,10) (,,5),))))))))))) (0,0,,),)))) (1) over random initial condition and random backward pass weights
(1, (1,, ((((0.P())))
Foo exp EL di (1:00) xx.0) k(0) (0 m0,00 () p0()) -,
(14)
where E. C. are regarded as functions of 0(0), [W*). Arbitrary moments of these random variables can be computed by differentiation of Z near zero source. For example, a two-point correlation function can be obtained as
(X
lim J. p.
8 on A0E00 K) (P.P))
(15)
More generally, we let (i., t) be tuple containing the neuron, time, and sample index for an entry of one these fields so that X - xx,,,() (4). Further, we let Ner. be index sets which contain sample and time indices as well as neuron indices A = {Mr .-. ( for all of the indices we wish to compute an average over. Then arbitrary moments can be computed with the formula
II
G
MEN
MEN
a
BEN
8
lim j.h.p.
(11 (1),0, kf,,), (() (H)))
8j4 veN
SHE NEN
dn BEN
5p
MEN
(16)
We now to study this moment generating functional Z in the large width N + 0o limit.
B.3 PATH INTEGRAL FORMULATION AND INTEGRATION OVER WEIGHTS
To enable the average over the weights, we multiply Z by an integral representation of unity that enforces the relationship between X 2++ (t), (0),,, (h
1= / dx (t)
X
(t)
1 (())((((,())))
VN
RN
- f (1)
JNN JRN (2T))
exp
s (t)
X, (t)
(t)
VN
w (0)0((())))))))
(17)
In the second line, we used the Fourier representation of the Dirac-Delta function for each of exp (ifr). We repeat this procedure for the other fields 27
the N neuron indices 5(r) I dF
E (1), (1), (1))) each time and each sample μ. After inserting these delta functions, find
16

---

Published as a conference paper at ICLR 2023
the following form of the moment generating functional
dx (t)d (t) de (t)d (t)(() (t)
III Epe
(2T)
(2T)
(2T)
(2x)
x exp
dt
(x (,(0) + (x) (K/ + [t)
E
X exp
di
EM
(o) exp 1T [ +x
x
[
E (
x exp
VN
f f
II E exp
T-WeT
(18)
VN
We see that often have simultaneous integrals over time and sums over samples μr so we will again adopt a shorthand notation for indices = (p, t) and define a summmation convention En b = -oo dt 1 (t) To perform the averages over weights, note that for standard normal variable Way, that Eww,, exp (iWaya,by) = exp (-)) Using this fact for each of the weight matrix averages, we have
)
Eo exp
Tr (0)
Σ e e+1 (hh
+ ++1 E + gfhh c++ GeT
1 2N
exp
Xx
E+1 ((h())
+
e-1 EeT + 1+1 GiT
F
X s+1 xet m.u Ey Eo G + Et. E. u. + Et. -2G
setl
G
exp
X exp
gi A + Rx gftl Cuel)
(19)
In the above, we introduced a collection of order parameters ( G. G, å, A, C), spond correlation and response functions of our DMFT. These are defined as
which will corre-
Th m. = N (h()) d(h(). G. 1 N gú Gj = g gt
1
1
getl 1
No gt. ihf N (hh))) E, ic (((h)) Et.
1
1
(20)
N
We perform a similar average over we can be obtained directly
E W exp (-> TT
Σ f 1) exp
1
Σ
Et Et
E nu
(21)
Now that we have defined our collection of order parameters, we enforce their definitions with Dirac-Delta functions by multiplying by one. For example,
1 N f
d ' (N uN ((h()) ((h())
exp (N T Tf. The (h ((ht))
(22)
2T
17

---

Published as a conference paper at ICLR 2023
We enforce these definitions for all order parameters { yv Guu G E We let the corresponding Fourier duals for each of these order parameters be
uui GnrrAhuu Af C.u:.:
{ uu: Geue Gr Gr pv: -B, - }- the next section we show the resulting formula for the moment generating function and take the N oc limit derive our DMFT equations.
B.4 DMFT ACTION
After inserting the Dirac-Delta functions to enforce the definitions of the order pa- rameters, we derive the following moment generating
functional in terms of
{2, d, G. G, G,
d di II E.p.v
dG dG dG dG dõ, yv dG d.A. dB ju wu
2
wa
N-N-1 2=N-1 2T exp (NS(g))
2TN-1
N=N-1
2TN-1
where Siq] is the (1( DMFT action which takes the form
Σ
D
Di Be mee + G
Guu + G G uu + &r Et.
pa uee - Al Be
C De
De
pee
Z
(23)
The single-site moment generating functionals (MGF) z involve only the integrals with sources (if, kf, nf.p() for neuron i E [N] in layer t. For given set of order parameters q zero source, these functionals become identical across all neuron sites i. Concretely, for any E [L], e [N]. the single site MGF takes the form
dx, dE dét
act dé
2
(24)
2T
2x
2T
2x
Xys Xu 1
Te
3t Entt) Ghttl + Gu
exp
Sett SpeSu G + 2 1.0)
cxp
De o(h ((hh))())))))) + 6 Gpan Grusiigh Ee Senthoe
Σ
exp
Xtat gu
e+1 Alwe + Xth De + (lh Bin + (bh Dine)
sttl
exp
m
Σ Ku + +xx)) + (Ke + iitt) + (i + +(29) + 7/t ( + 1221
exp
As promised, the only terms in Z, which vary over site index i are the sources {j. k. n, p}- To simplify later saddle point equations, will abstract the notation for the single site MGF, letting
dxudxn dEndEn dCndbn deudee exp w (-H((x.&&&&&
2
II
(25)
2x
2x
2x
where H. is the single site effective Hamiltonian for neuron and layer E. Note that zero source, Hi are identical for all i [N].
18

---

Published as a conference paper at ICLR 2023
B.5 SADDLE POINT EQUATIONS
Letting the full collection of concatenated order parameters be indexed by b. We now take the
N + 0o limit, using the method steepest descent
dd 2n
2 =
exp (NS(a)) exp (NV(4")) VS[g]1a = =0, N - cc.
(26)
We see that the integral over q is exponentially dominated by the saddle point where S 0. We thus need solve these saddle point equations for the q". To do this, we need introduce some notation. Let O(x, & &.C. be arbitrary function of the single site stochastic processes. We define the f-th layer i-th single site average, denoted by (
dxndRu dEndEu dGudGu deuds
1
II
z
2x
2T
exp
(27)
which can be interpreted as an average over the Gibbs measure defined by energy H. With this notation, we now about computing the saddle point equations which define the primal order parameters (Φ) G,
as
1
(o(h)))(hh)))
Jh
mv
N
as
Gd
1
N
(ofLoi) =
O0 pv
N
N
0s
1
Gt
(o,
=0
a
N
pu
i=1
0s
1
N
8
N
aG
19

---

Published as a conference paper at ICLR 2023
We further compute the saddle point equations for the dual order parameters
as
N
AM
2N
Xxx
E+1
Xv
++1)
=0
1=1
as
Gh mm
1
28 28-1
=0
aG pMv
2N
1-1,8
1=1
as
1
26-1
10 m
arl
N
1-1,1
80 W
as
ai
2N
Σ
N
(IGH
((t--1
5c
+
Sps
Re-1
52
1-1,3
=0 0
1=1
0s
N Σ
0A
B
xet1
IN
E+1)
e41, =0
pv
as
A 10
Σ
=0
OB
1,2
pvv
as
i
De
(xtt
xft1)
/041,
uv
N
1=1
0S
C²
(o(h))5)) =0.
(28)
aD juv
The correlation functions involving real variables (h, 9. have straightforward interpetation. However, is not immediately clear what do with terms involving the dual fields (x, E. () As starting example, let's consider one of the terms for B namely -i (f We make progress by inserting another fictitious source term un and differentiating near zero source
(Restt)
lim θ (wn) Oug
o exp
(
EiEun w uo
(29)
Introducing a vectorization notation u = Vectufuuu X² Mat($6-1) Mat u.v J.e. we can perform the internal integrals over X
VectXubu and Φ--1
dx
1 X
xT
-f +ix (x -w' A -
exp
2
exp
2 (x"
u² A 1--1 96 - C -
(x' - u-- AA--1g' 1--1--)--1gg))
1
exp
2 In det o. i
(30)
We thus need to compute a derivative of the above function with respect to u' at uee 0, which gives
- ((x A -
-1--1)-
(31)
20

---

Published as a conference paper at ICLR 2023
From the above reasoning, can also easily obtain d using
0²
(xxx1 ET =
u= (exp (-iu' x))
du
/ dx
0²
Ju tu
u4=0
x exp
7 (x u1-
-A--1 - - (0/ u' - A -C"-lg))
1--1j-1
1 1 ((x - A g" - 1") - A--1g² - -C--1g)) -
(32)
Tpe v
Performing similar analysis, we insert source fields r+
for E
and define G
G
G++1 and Bº [B DJ] and then we can compute the necessary averages using
Ge+IT
G++1
the same technique
-i (8(()))
-0 ( (6(h"))x)
exp (-irf, - (0(h)(E) BTT T(T)()())
gr
0²
1r9=00 eex -ir )
Ôr Dr
|
[G ((E) B o(h))(E B BTT((h)))) |G
(33)
We now have formulas for all the necessary averages entirely in terms of the primal fields Ex.&,S.33
B.6 LINEARIZING WITH THE HUBBARD TRICK
Now, using the fact that in the N oo limit q concentrates around q², we will simplify our single
site stochastic processes can obtain final formula for we utilize the Seabbard-SSratanovve identity
To so,
o² (aa'R²
/ du
V2T02 exp (-
1 u² iku 2Ø
exp
12
(34)
which is merely a consequence of the Fourier transform the Gaussian distribution. This is often referred to "linearizing" the action since the term quadratic in k was replaced with an average of an action which is linear in k. In our setting, we perform this trick on a collection of variables which appear in the quadratic forms of our single site MGFs 2 For example, for the gf+1 fields, we have
exp
Σ
s+1 4
exp
-i
-
xet
6+1 #
(35)
2
Similarly, we perform a joint decomposition for the {Et, (t) fields which gives
exp
1
Et こ8 g++1 + -2 Geth + 26 ge+1
&
(36)
2
G++l
G++1
G
exp
usp + (
c++l
P
(1
We thus see that the Gaussian sources fri) and (uh are mean zero with correlation given by Now that we have linearized the quadratic components involving each of the dual fields
0++
21

---

Published as a conference paper at ICLR 2023
(8,5,5), we now perform integration over these variables, giving
dxf
i Σ &1
-
Σ Alsel gi
exp
Xx -
2T
II
Xj1
Σ
Afr Iu
Σ che Iu
14
Σ
28
ct
Σ Ben ((1))(())
II 8
Ep - 11 Σ
Be (eun((h))
2T
exp
Sp
1
1
dC
exp
Σ
Su
Si
1/1
E Deu (iiusblit))
II à
ct 1
Σ Deu Dieus(li))
2m
(37)
This reveals the following set of identities
X = u/ + AAw oe + EC Cp 9v
5/ = r + Σ Biy Bly Sq = uiy +
Des DDen up
Der o(h)).
(38)
Since we know by construction that u' = x - A Cè-1g" covariance Φ--1 1-1 we can simplify our expressions for B and l- using Stein's 's Lemma
g" is a zero mean Gaussian with
B -11 1 N
0g du
(u'g
N
1=1
JgghT
D
c
1p--11-1 (u'g²T)
Σ
(39)
N
N
Oue
N
1
[ 10-111-1
c
8 - 11
(u (T
10'-1-11 = 0.
2
2N
Similarly, using the Gaussianity of r S.. Ee we have
A
oR(h!)
c
do(h))
Get1
ge+1 Get1
dreT
OoeT /
B.7 FINAL DMFT EQUATIONS
We now take the limit of zero source jr, k nn, p + 0. In this limit, all single site averages () become identical so we can simplify the expressions for the order parameters. To "symmetrize" the equations we will also make the substitution B + B D + D Next, we also rescale all of the response functions {A Bd, C", D'} by % so that they are Onp (1) small %o. This gives us the following set of equations for the order parameters
(t, (O(H,
(t, (s)(())9(())) (t,(t,s) = (1)
(80(h((0))
Chi(l,)) = -0= ( 8o((8)
/80(h) (1)
(1, (1) (1, = =01
(r((s)
By (t, s) = 7%01 / (gg(t(1))
50gh
Suutt(s) (t, = -/=0
(s))t(s)
22

---

Published as a conference paper at ICLR 2023
Tr d (t.s)
Tr (i))s) Tr (t.ss)
Expi..
Expr.
DMFT.
DMFT.
p
Final Kemel
(b) p Kernel Dynamics
@
Tr o (t,s)
(t.,
Tr 20, (t.s)
Expe
Expt.
DAIFT.
DMFT.
(c) p= Final Kernel
(d) = 1.0 Kerel Dynamics
Figure 6: Feature kernels &e and their dynamics predicted by solving full set of saddle point equa- tions equation 40 for p-IAA in depth 3 tanh network with 7o 1.0. Solving deep nonlinear p-IA requires sampling the full triplet of Gaussian sources {u,, r of) for each layer and computing all four response functions {A C" D'}. DMFT thcoretical predictions are compared to width N 3000 neural network.
For the fields (1,,(1), (1), we have the following equations
A5,(1) (t,(() + [ ds (1, (c (t,)))) ss))
=000 200) + +70 [ ds
(4,
((){(())))((
GD
2(1,(00) -1- + po, + ds EP DE (t, (1,)
P-FA
((h((())))* E/~ ~(011
DFA
(t)2,
GLN
(n())(((()))))
Hebb
(2)())) GP & ) (00) (2(0))
GC+1 Ge+1
G4t1=
(40)
Ge+I,T ge+1
C EXTENSION TO OTHER ARCHITECTURES AND OPTIMIZERS
In this section, we consider the effect of changing architectural details (multiple output channels and convolutional structure) and also optimization choices (momentum, regularization).
23

---

Published as a conference paper at ICLR 2023
C.I MULTIPLE OUTPUT CLASSES
Similar to pre-existing work on the GD case (Bordelon & Pehlevan 2022). our new generalized DMFT can be easily extended to C' output channels, provided the number of channels C' is not simultaneously taken to infinity with network width N. We note that the outputs of the network are now vectors fa E RC and that each eNTK entry is now C C matrix (t, E RCx. The relevant true gradient fields are vectors gé afay We construct pseudo-gradients Ke,p as before using each of our learning rules. The pseudorr-ps kernel G E RCxC is Ge Geed.pu g ge The eNTK K = >e 1 nu can be used to derive the function dynamics
05, Øt
E A Oc
(41)
0fμ
At infinite width N + 20, the field dynamics for 11 R. gf R satisfy
= (t) 70 L ds
(s)
As(s) 1- (t,s) + A st, gc(s) + C [s)s
2t r(((t) + 7 J ((((())) G (t, (t,s) Btt(t,
(42)
(t))(((t) 5r((e)
(oo(k((e)) ōut(e)
)
R.C and Bpa while the gradient-sseudogradient kernel is Gh = (o (OgE (s)) E RCx. The pseudogradient fields ğ' are defined analogously for each learning rule as in the single class setting.
where (t,)) 7o
E R.C (t, = 7a
0g (1) (a) E RC, The feature kernels are the same as before & (t, = (t)h
((1,,(0)),
GD
((1,(()) (00(() + v1- + pro mr (t) = ((),())) L ~(0,
ds E. (((()))0()(6)))
p-FA
DFA
((m{, (1)((h))))))
GLN
Hebb
(43)
C.2 CNN
The DMFT described for each of these learning rules can also be extended to CNNs with infinitely many channels. Following the work of Bordelon & Pehlevan (2022) Appendix on the GD DMFT limit for CNNs, we let W represent the value of the filter at spatial displacement from the center of the filter, which maps relates activity at channel j of layer to channel of layer E + The fields he a satisfy the recursion
N
A
E+1 p,i,a
Σ
w o(he (b)) iE [N],
(44)
j=1 be.t
where Se is the spatial receptive field layer l. For example, a (2k + 1) (2k 1) convolution will have St = 5i E 2² -k k < kj. The output function is obtained from the last layer is defined as fy To Di=i E. ( The true gradient fields have the same N which as before enjoy the following recursion on
1 ΣN
definition as before of N 81,
L/ No.N Σ
OS
ah
p.b
(h (
I VN
Σ Wt
e+1 gua-b
(45)
0h E+1
oh!
n.b
i1a
D
24

---

Published as a conference paper at ICLR 2023
We consider the following learning dynamics for the filters
70 W
Σ Au (hp,,ct0)
(46)
dt
VN
where before g' is determined by the learning rule. The relevant kernel order parameters have spatial indices. For instance the feature kernel at each layer has form &e peccaab N ( (!)) ph (s)). At the infinite width N > co, the order parameters and field dynamics have the form
(t) (t) 7o ' ds Σ (s) wo u,b,c
(t. (s)
(47)
+ 70 S ds DlA (1 ())))())) + hana 3)
2,b
(t) (t) a (t) + ( ds (
w.h.c
/ ds Σ By
+ 70
(48)
2,b
where correlation and response functions have the usual definitions
df & (t, (0 (t)) Gia,,e (t, ) (gaa(t)g
G (t, (t)
60 (t))
(t)
A (1,
Ba ao 70
(49)
8r (s)
(s)
du e+1
(s)
C.3 L2 REGULARIZATION (WEIGHT DECAY)
1.2 regularization on the weights W (weight decay) can also be modeled within DMFT. We start by looking at the weight dynamics
P
d w di
70
Σ 4
T -
p=1
('( M
w'
+
To
[ ds -((ts) A(t-s) "
(50)
VN
In the second line we used an integrating factor et, We can thus arrive at the following feature dynamics in the DMFT limit
A4(()) = - Xx((t) + [ ds M(t-s)
Σ (z)
=1(1) = - - +0 L' ds dse- Σ (t (s)0())()))
The ÿ dynamics are also modified appropriately with factors of - and e x for each of our learning rules. We see that the contribution from the initial conditions x. are suppressed late times while the feature learning update which is O ()) in the first layer dominates scale of the final features.
C.4 MOMENTUM
Momentum uses low-pass filtered version the gradients to update the weights (Goh 2017). A continuous time limit of momentum dynamics on the trainable parameters (W*) would give the
25

---

Published as a conference paper at ICLR 2023
following differential equations
a (((() = (t)
01
dt (1)) = QQ + 70
(51)
VN-
We write the expression this way that the small time constant - limit corresponds to classic gradient descent. Integration the (*(1) dynamics gives the following integral expression for W
VNT I dt' [ dt"
(' (0)
70
(("
(52)
These weight dynamics give rise to the following field evolution
(t)= Xp 2+1 (t)+ + To [" dt' ' dt" c (t
T
7a J dt' S dt" et"c-W---h Σ ( (t, (t,
(53)
We that in the limit, the 1" integral is dominated by the contribution t" i' recov- ering usual gradient descent dynamics. For 7 >> O, we see that the integral accumulates additional contributions from the past values of fields and kernels.
D LAZY LIMITS
In this section discuss the lazy 7o - limit. In this limit we see that h²(t) (f( and ((t)=r(() for all time f. Since the input data gram matrix 40 pur = f xy is constant in time the sources in the first hidden layer u/, are constant in time. Consequently, the first layer feature kernel is constant in time since
T yu (t, 8)
(o(u)))u)) w 400
(54)
Now, we see that this argument can proceed inductively. Since T time-independent, the second layer fields h u² ~ (O, D are also constant in time, implying & is constant in time. This argument repeated for all layer E [L]. Similarly, we can analyze the backward pass fields z'. Since I4 (0, GL+1) constant, then z are time-independent for all l. It thus suffices to compute the static kernels {Ø², Ge, G* at initialization
Φ = (o(u'))(u)) uu
G ((((l') o r To((")) o r9]T)
= G क. क. = (((()) u'~A((000-1""
(55)
where in the last line we utilized the independence of u", rf. These above equations give forward pass recursion for the & kernels and the backward pass recursion for G Lastly, depending on the learning rule, we arrive at the following definitions for G for l (1,..., L}
G o oe
GD
p $
p-FA
Gº a-= ( (
(56)
DFA.Hebb
G @ (0lm²)
(6(m)))(m)))
GLN
Using these results for G. we can compute the initial eNTK prediction dynamics.
E G++l @ 4' which governs
26

---

Published as a conference paper at ICLR 2023
D.1 LAZY LIMIT PERFORMANCES ON REALISTIC TASKS
We note that, while the DMFT equations on P datapoints and T timesteps require O(P)TS) time complexity solve in the rich regime, the lazy limit gives neural network predictions in O time, since the predictor can be obtained by solving a linear system of P equations. The performance of these lazy limit kernels on realistic tasks would match the performances reported by Lee et al. (2020). Specifically, GD and p = 1 FA would match the test accuracy reported for "infinite width GD while p 0 FA, DFA, and Hebbian rules would match "infinite width Bayesian" networks in Figure Lee et al. (020).
E DEEP LINEAR NETWORKS
In deep linear networks, the DMFT equations close without needing any numerical sampling proce- dure, as was shown prior work on the GD case (Yang & Huj 202T; Bordelon & Pehlevan, 2022). The key observation is that for all of the following learning rules, the fields {h, 9.9) g) are linear combinations of the Gaussian sources {u. v),, and are thus Gaussian themselves. Concretely, we introduce a vector notation h (ec and g² Veclg (s (1)) etc. We have in each layer
h' R + R Ru + R
g c Ry Rg + u + R
g' Rg + R + Ra + Rg
where the matrices R depend on the learning rule and the data. The necessary kernels H (h' (T can thus be closed algebraically since all of the correlation statistics of the sources {u, P; b} have known two-point correlation statistics.
E.1 LINEAR NETWORK TRAINED WITH GD
The R matrices for GD were provided in (Bordelon & Pehlevan 2022). following DMFT equations for h h'gg
We start by noting the
he u + (0 ')'', g =r (B' + G )
(57)
where [H livtes (1, Isolating the dependence of these equations on u and r, have
lww,ts
T- AA 1 (B* + G(tl)] h = 7 H G.+1) )re
[ -- 1 ( (A (A- - )] g" = -
f-1 + H 1)" (A
(58)
These equations can easily be closed for H' and G¹.
E.2 PLALIGNDD FEEDBACK ALIGNMENT
In p-FA we define the following pseudo-gradient fields
= v'=vI-ppf 5²5 + pof + D'
(59)
Next, we note that, at initialization, the & can be computed recursively
G P
(60)
We note that a h which implies A 0. Similarly we have a h2 0. Thus A O. Proceeding inductively, we find A 0. Similarly, we note that 0g Inductively, we have D for all Using these facts, we thus find the following equations
0 D4-1
0.
h r + (C H g9
(61)
g = r' r + G '
(62)
g¹ = -1= --11-p -5- p²5² po
(63)
27

---

Published as a conference paper at ICLR 2023
We can close these equations for H. and &
H' H 10 (C + H G (C ( 1
G p T [ + G + G
C (C H
=B= + G
(64)
The matrices G 11 are all rank one. Thus suffices to compute the vectors cf (C -H Further, suffices to consider d' G/1//1/3 With this formalism have
= de TT d' pd BB' G
(65)
The analysis for DFA and Hebb rules is very similar.
F EXACTLY SOLVEABLE 2 LAYER LINEAR MODEL
F.1 GRADIENT FLOW
Based the prior results from (Bordelon & Pehlevan
2022), the Hy Hyyyy/lll dynamics for
GD are coupled to the dynamics for the error A(t)
3 y (+) have the form
d
dt = A)A
A 2Hy
(66)
(y A)2. Integrating this conservation law from time 0 to time find (t)² 1 1 70 A(t)) We can therefore solve single ODE for Att), giving the following simplified dynamics
These dynamics have the conservation law d H T0 de
2
d:
d
dt -
76 A)2A H, = V -
(67)
These dynamics interpolate between exponential convergence (at small 70) and logistic conver- gence (at large 7o) of (t) zero. Since A + late time, the final value of the kernel alignment Hy = /1 + ?..?
F.2 PLALINEED FA
For the two layer linear network, the p-FA field dynamics are
h. = 90 Σ (s) at 9n()) = 7o Σ (t((h)a(tt)..
(68)
FA we have (t) = -- ~ (O, 1) which is a constant standard normal. We let (t = (āh (t)) The dynamics for Hw and an coupled
H mar Σ AU)KE + (1) Σ (l)
d
di (((() = Σ A.K
G(L) Yo Σ ())())an() _ AA(0) = -IM (00KK))) dl.
(69)
Whitening the dataset K² I and projecting all dynamics subspace gives the reduced dy- namics
d - H = 2y0A, d =a= vo
de Aa d _ - GAA.
(70)
di
di
d1
28

---

d Published as a conference paper at ICLR 2023
From these dynamics identify the following set of conservation laws
2 d G
d a²
d
H
dt
dt.
dt
2G 2p = d² = 1.
(71)
Writing everything in terms of A, we have
2 + (1 p) A - d
3
1
di 70 A
2aa (1 p)a
di
Integrating both sides of this equation from 0 to gives A y %0 Ia (1 p)a]. Thus, the a dynamics now one dimensional, giving
1 a + p)a.
di a = Y0y
(72)
2
When run from initial condition 0, this will converge the smallest positive root of the cubic
equation _a²² + (1+ p)a 70y. This implies that, for small To we have ~ Y0 I+p so that All
~nv² pI+p)² and that larger initial alignment p leads smaller changes in the feature kernel
and seeudo-grrdient alignment kernel. At large Yuy, have that (2 that AH 2AG (2709)2/3.
F.3 НEBB
For the Hebb rule, G₁. = (ghh)) A - Tof2A, r0 - - A,, Under the whitening assumption K Sue. the dynamics decouples over samples
d di Hn 200 ب. AA - + +70011 An)
d
(73)
We see that Hmd strictly increases. The possible fixed points for A are A or A, Nu + // 2 One of these roots shares sign with yn and has larger absolute value. The other root has the opposite sign from Vμ- From the initial condition Aμ Y/ and Hμ 1, Ap is initially approaching decreasing in absolute value so that ا E (0, (8ul) and will have the same sign as yn- In this regime ay ا < O. Thus, the system will eventually reach the fixed point A 0, rather than increasing in magnitude the root which shares sign with y, or continuing to the root with the opposite sign as Vy.
d
G DISCUSSION OF MODIFIED HEBB RULE
We chose to modify the traditional Hebb rule to include weighing of each example by its instanta- neous error. In this section discuss this choice and provide brief discussion of alternatives
d W & Er (( ((hE)) In the absence of regularization or Traditional Hebb Learning: oe normalization, this learning rule will continue to update the weights even once the task is fully learned, leading to divergences at infinite time t. - oc.
Single Power of the Error: σi W A, (o6t()+))))((( While this rule may naively appear plausible, it can only learn training points with positive target values yn in linear network if Yo Further this rule only gives Hebbian updates when Aμ >
This was our error modified Hebb rule. We note that the update always has the correct sign for Hebbian update and the updates stop when the network converges to zero error, preventing divergence of the features at late time.
Two Powers of the Error: di W x Σ, (o(hh))((()))))
H FINITE SIZE EFFECTS
We can reason about the fluctuations of q around the saddle point q" at large but finite N using Taylor expansion of the DMFT action S around the saddle point. This argument will show that
29

---

Published as a conference paper at ICLR 2023
at large but finite N. we treat q fluctuating over initializations with mean q" and variance O(N ') We will first illustrate the mechanics of this computation of an arbitrary observable with scalar example before applying this to the DMFT.
H.1 SCALAR EXAMPLE
Suppose have scalar variable y with distribution defined by Gibbs measure J ddge-Ns.g N. for
action S. We consider averaging some arbitrary observable (g)) over this distribution
(0 [g exp (-N5 (())
(74)
exp (-NS(g)
We Taylor expand S around its saddle point 9" giving S/d] S(0") + "00"("'(0 9²)² + Ex Sth) [g" )* This gives
dg exp (-NT) (9 9²)² Σ kk= 5 19' (1) O(g)
(0( =
(75)
I dq exp (-M125 --)² -- Sth
(0 (4 )*)
The exp(NS(q")) terms canceled in both numerator and denominator. We let the variable q - q- Vz6. 8. After this change of variable, we have
(O()) = fd exp 10 E N= (4"]00) O4p + N f dx (- " E N1-k/25((k (ok)
(76)
We note that all the higher order derivatives (k: 2 3) are suppressed by at least N 1/2 compared to the quadratic term. Letting U = E N1- [a" [5 represent the perturbed potential, we can Taylor expand the exponential around the Gaussian unperturbed potential exp (-50² S"""""""""") [g")). We let (O(8)) = -== represent an average over this unperturbed potential
(O(g)) = [dō exp (- 1--5s""1 " [1- U JU2
(77)
J dō exp (-]
U +
(0())) (O())U) (O()))))) +... (U), (U2)
(78)
Truncating each series in numerator and denominator at certain order in 1/N gives a Pade- Approximant to the full observable average (Bender et al. 1999). Alternatively, this can be ex- pressed in terms cumulant expansion (ardar 2007)
(0)
(OUM))
(79)
R!
where (OU are the connected correlations, or alternatively the cumulants. The first two con- nected correlations have the form
(OU%) (OU), - (U)(())) ( (OU2) (OU), (U), - (0)) (U3) (0) (
(80)
Using Stein's lemma, can now attempt extract the leading O(N-1) behavior from each of these terms. First, we will note the following useful identity which follows from Stein's Lemma
(O()))*) 2/2 (O **)*)
(81)
= -22211990000-100000000--------0- (O'(wa +-
g-)
-A (1=1))1"~~000)) -37)0'0'0))00-11
(82)
30

---

Published as a conference paper at ICLR 2023
Using these this fact, can find the first few correlation functions of interest (O(g)U) (00))))))))) + +3-- +
(O())U?) 15, (00" +(N-))
(83)
Thus, the leading order Pade-Approximant has the form
(O(4)) (O),
(5 (O(g)) + 15 [--" (O(4))
2N
-13115 -544)
2N [5
15
(84)
H.1.1 DMFT ACTION EXPANSION
The logic of the previous section can be extended our DMFT. We first redefine the action as its negation - S to simplify the argument. Concretely, this action S[g] defines a Gibbs measure over the order parameters q which we can use compute observable averages
(O())) = f exp (- (1)
(85)
exp
As before, one can Taylor expand the action around the saddle point q
Sig S|q') + + "'a==""""'(q""ll -q)-(((+++... +
(86)
As before, the linear vanishes since VgS(q"] 0 at the saddle point q². We again change variables to 8 VN((q q²) and express the average as
(0) f dō exp 6 T
+ 0())
f dd exp 8 4258 (())
(-1)*
(OU%)
(87)
k!
k=0
where On denotes Gaussian average over N(g" N (
H.2 HESSIAN COMPONENTS OF DMFT ACTION
To gain insight into the Hessian, we will first restrict our attention to the subset Hessian entries related to & $² We again adopt a multi-index notation = (H, v, t, s) so that & = Φ (t, 8)
025
a.e. =0 1
OTe 04.00
0P5
80,441 aΦe
of+1 (t Moth - Te Φe
05 δ2
04 a. n The first equation follows from the fact that X has vanishing moments due to the normalization of the probability distribution induced by z Similarly, for the G, G kernels we have
8PS
aG '''
05
8 oe de. 8 aG P
Gt-1 de, (9 (t) ( (s')) G.
aG 82s
04 04,00%
Proceeding in similar manner, compute all off off-diagonal components such 90aG and
Once all entries computed, seek inverse of the Hessian obtain the covariance of the order parameters.
0's a6ad
31

---

Published as a conference paper at ICLR 2023
H.3 SINGLE SAMPLE NEXT-TO-LEADING ORDER PERTURBATION THEORY
In order to obtain exact analytical expressions, we will consider L-hidden layer ReLU and linear neural networks in the lazy regime trained on single sample with K iel" To ensure
preservation norm, we will use ((h) = (2) for ReLU and ((h) = h for linear networks. First, note that in either case, the infinite width saddle point equations give
Φ (o(h)²) h~N10,0e-1)
Φ 10 =1
G = ())))22)
-G++1 G =1
Φ6 1, G =1, VLE L].
(88)
At large but finite width, the kernels therefore fluctuate around this typical mean value of æ and G 1. We now compute the necessary ingredients to invert the Hessian
ReLU
V = <o(h))) - - 2
Linear
Vi = (a(h))2)) - G2 =
-
5 ReLU
(89)
Linear
Next, we compute the sensitivity of each layer's kernel to the previous layer
a
G =1
(90)
a
aG
First, let's analyze the marginal covariance statistics for + Vec 45 We note that the DMFT action has Hessian components
1/21
and $
1e
1-1
0
0
0
0
1
-1
0
0
V2
S
7
S
0
U 1
Ho =
U
(91)
72
UT
$
1
We seek a (physical) inverse C which has vanishing lower diagonal entry, indicating zero variance in the dual order parameters & This gives us the following linear equations
0
U
Cn
C12
1 07
H4C
UT
V&T
C22
0 I
UCT I, U C" + V.C =0
Cn+VoC
(92)
The relevant entry is Cn -V-
This matrix has the form
0
07
1
1
0
0
0
1
1
1 2
Cn VV
1
0
0
0
1
1 = V
1
2
3
3
(93)
2
L.
Using the fact that the covariance is the negative of the Hessian inverse multiplied by 1/N, we have the following covariance structure for {$'}
1 N
Cov( 6 Φ = V. mi.
(94)
This result be interpreted as the covariance of Brownian motion. Following an identical argu- ment, we find
Cov(G' ,. Vy min (,,+ 1'}.
(95)
N
We verify these scalings against experiments below in Figure 7
32

---

Published as a conference paper at ICLR 2023
Covariance
Empirical
0
(a) Cross-Layer &6 Covariance
(b) Cross-Layer G Covariance
DMFT ==
10a
DMFT t
DMFT
DMFT 2
10a
Var of
DMFT =
Var G
DMFT t 4
DMFT 1- 5
DMFT t-5 5
10
DMFTI=
10-1
DMFT 2=
10-4
10-1
10a
10a
10a
10a
10a
102
N
(c) NLO-DMET of Variance Scaling
(d) NLO-DMFT G Variance Scaling
Figure 7: Verification of kernel fluctuations through next-to-leading-ordee (NLO) perturbation the- ory within DMFT formalism. (a) The cross layer covariance structure of {Φ') in L 10 hidden layer ReLU MLP. The empirical covariance was estimated by initializing a large number (500) of random networks and computing their &f kernels. We that variance for & increases as é in- creases. (b) The cross-layer covariance structure {G*). ]. The variance of G larger for smaller (c) The predicted variance of &e for different layer é and widths N. All layers have variance scaling as 1 /NN, consistent with NLO perturbation theory. (d) The scaling of G variance.
33


Reviews:
[{"review_id": "tzTJxpOBIre", "reviewer": "Reviewer_k9iv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies several biologically (more) plausible alternatives to backprop in the lazy/rich NTK regime. This is done using the recent developments in dynamic mean field theory for backprop, and produces non-trivial predictions for activity of those learning rules in wide networks.", "summary_of_the_review": "Good paper with a mostly technical contribution. Some parts could be improved/better explained, but overall this could be an important result for biologically plausible learning studies.", "strength_and_weaknesses": "### Strengths\n\nThe paper clearly theoretical differences between solutions found by different learning rules.\n\nThe theoretical analysis of learning rules' dynamics is important for determining how the brain learns.\n\nThe calculations are written in detail and can be repeated for other learning rules.\n\n### Weaknesses\n\nThe paper seems to be a straightforward application of [Bordelon & Pehlevan, 2022] to other learning rules.\n\nIt's not clear to me if the rich NTK regime contributes to any of the final conclusions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lKPuq1-0Ako", "reviewer": "Reviewer_BwSf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the learning dynamics of neural networks in the mean field limit. The authors focus on gradient descent, Hebbian learning, feedback alignment and direct feedback alignment on networks of arbitrary depth, as well as gated linear networks. For these cases we are presented with a set of self-consistent stochastic differential equations to be solved numerically.\nThe self consistent stochastic equations simplify in the static limit and the linear network case and reduce to other results in the literature.\nSpecifically, the authors obtain an explicit characterisation of the dynamics of two layer linear networks.", "summary_of_the_review": "The paper presents a set of self-consistent stochastic differential equations that describe the learning dynamics of deep networks for a number of learning rules. The paper is well-written and clearly details the analytical derivation as well as the numerics involved. The results are easily reproducible and a code is provided in the supplementary materials. \n\nWhile the treatment of the dynamics and the phenomenology extracted from it are not completely novel it's a significant contribution that could be expanded in many directions", "strength_and_weaknesses": "Strengths:\n1) The paper is reasonably well-written and easy to follow.\n2) The analytical procedure to obtain a theoretical characterisation of the dynamics is easily generalisable to different learning rules and architectures beyond what is shown in the paper.\n3) The predictions of the theory are compared with experiments and the plots display for the most part quantitative agreement between theory and experiments.\n4) The authors check numerically that the large width limit is approached polynomially fast in the width of the network.\n\nWeaknesses:\n1) The role of the data on the theory is not explicitly stated in the main test.\n2) The authors propose a fixed point scheme to solve the self-consistent equations. Do you have a theoretical argument for its convergence and some bounds on the number of iterations required?\n3) In many plots (for example figure 3a, blue line) there is a small disagreement between theory and experiments. What could be the reason for the discrepancy?\n4) While the numerical checks of the finite width effects are very convincing, the theoretical argument for the finite width expansion seems not so transparent. Can you plot the finite width corrections to the theory? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5qOss1NzeKf", "reviewer": "Reviewer_6oec", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the learning dynamics of infinite-width networks trained using various biologically plausible variations to gradient descent, including feedback alignment, direct feedback alignment, error modulated Hebbian learning, and gated linear networks. The paper derives an effective Neural Tangent Kernel (NTK) that governs the learning dynamics. Through theory and simulation, the paper identifies a number of interesting phenomena about the learning dynamics of these biologically plausible learning rules.", "summary_of_the_review": "Overall, I think this paper is an important contribution in our understanding of the learning dynamics of networks trained with biologically plausible (or at least biologically inspired) learning rules. Feedback Alignment and subsequent variants are an intriguing learning rule because they seem so far from gradient descent; this paper helps us understand when and why they work.", "strength_and_weaknesses": "Strengths:\n- Nice use of the theory of infinite width neural networks to study learning dynamics in networks trained with variants of gradient descent.\n- Identifies a number of interesting phenomena regarding similarities and differences between the representations learned using the different learning rules.\n\nWeaknesses:\n-  Analysis is limited to MLP networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4WgssX47JX", "reviewer": "Reviewer_ZSh1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors investigate an important and open question: how does the the learning rule (gradient descent, Feedback alignment, Direct Feedback Alignment, error-modulated Hebbian learning, Gated linear networks) influence the learning dynamics during neural network training. The authors focus on wide neural networks, and study these using dynamical mean field theory. Their approach allows them to make statements about the learning dynamics with different learning rules. Some of these statements are expected (but still good to formally analyze), e.g. that error-modulated Hebbian learning does not enable effective learning. Other statements are more surprising (e.g. that smaller initial feedback alignment later leads to better feature learning in the Feedback Alignment method). ", "summary_of_the_review": "The paper is relevant to a big portion of the ICLR community, and provides theory and insights that will likely result in follow-up research, and possible better learning rules. \n\nWhile the paper is generally well-written, the authors should make small fixes/improvements during the rebuttal period. \n\nIn my opinion, the biggest weakness of the paper is that many of the findings that the paper works out are unsurprising. The authors could strengthen the paper by highlighting which of the findings are \"common knowledge\", but have not yet been investigated formally. ", "strength_and_weaknesses": "# Strengths:\n\n- The paper is relevant to a big portion of the ICLR community, and provides theory and insights that may result in follow-up research, and possible better learning rules. \n- The papers release their code in the supplementary material. This includes code to generate the graphics included in the paper. \n- The paper is well-written. \n\n# Weaknesses:\n\n- The empirical work done in the paper is on fairly small tasks, considering that many of the learning rules considered in the paper are supposed to work on networks as complex as the human brain. \n\n- Some of the paper’s insights are not very surprising. For example, the fact that Hebbian learning converges slowly, or that it does not closely follow the gradient, are generally accepted as true in the community. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%