Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systems
Authors: Ori Shem-Ur
Khen Cohen
Yaron Oz
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Tangent Kernel
Linearization
Wide Neural Networks
Correlations
NTK
Weak Correlations
Primary area: learning theory

Abstract: Deep learning models, such as wide neural networks, can be viewed as nonlinear dynamical systems composed of numerous interacting degrees of freedom. When such systems approach the limit of infinite number of degrees of freedom, their dynamics tend to simplify. This paper investigates gradient descent-based learning algorithms that exhibit linearization in their parameters. We establish that this apparent linearity, arises from weak correlations between the first, and higher-order derivatives of the hypothesis function with respect to the parameters, at initialization. Our findings indicate that these weak correlations fundamentally underpin the observed linearization phenomenon of wide neural networks. Leveraging this connection, we derive bounds on the deviation from linearity during stochastic gradient descent training. To support our analysis, we introduce a novel technique for characterizing the asymptotic behavior of random tensors. We validate our theoretical insights through empirical studies, comparing the linearized dynamics to the observed correlations.

Published as a conference paper at ICLR 2026
NEURAL TANGENT KERNEL PARAMETER-SPACE S SYMMETRIES
PERSPECTIVE ON
Ori Shem-Ur, Khen Cohen, Yaron Oz School Physics and Astronomy Tel Aviv University Tel Aviv-Yafo, 6997801, Israel (ori oorihheor,, khencohen) mmail.tau ac. yaronoz@tauex.tau.
ABSTRACT
Deep learning models, such as wide neural networks, can be viewed as nonlinear dynamical systems composed of numerous interacting degrees of freedom. When such systems approach the limit infinite number of degrees of freedom, their dynamics tend simplify. This paper investigates gradient descent-based learning algorithms that exhibit linearization in their parameters. We establish that this apparent linearity, arises from weak correlations between the first, and higher-order derivatives of the hypothesis function with respect to the parameters, at initialization. Our findings indicate that these weak correlations fundamentally underpin the observed linearization phenomenon of wide neural networks. Leveraging this connection, we derive bounds on the deviation from linearity during stochastic gradient descent training. To support our analysis, we introduce a novel technique for characterizing the asymptotic behavior of random tensors. We validate our theoretical insights through empirical studies, comparing the linearized dynamics to the observed correlations.
INTRODUCTION
Deep learning in general, and particularly over-parameterized neural networks, revolutionized various fields Graves et al. 2013); He et al. (2016); Krizhevsky et al. (2012); Silver et al. (2016), and they are likely to do much more. Yet, underlying reason for their unprecedented success remains elusive. These systems can be interpreted as non-linear dynamical physical systems, characterized by a multitude interacting degrees of freedom, which makes an exact description of their behavior exceedingly hard. However, is well established that dynamical physical systems, when expanded to an infinite number of degrees of freedom, tend to exhibit a simplified form of dynamics AAnderson (1972); therefore, it seems plausible to consider such limit in the context of deep learning systems.
seminal study in 2018 Jacot et al. (2018) demonstrated that wide, fully connected neural networks undergoing deterministic gradient descent behave as though they were linear with respect their parameters (while maintaining highly non-linear structure in their inputs). This structure has been denoted as the neural tangent kernel (NTK). The result sparked a plethora of subsequent research, generalizing it to other architectures, investigating the rate of convergence towards this linear limit, exploring the deviation of the parameters themselves from their initial configuration, decoding the structure of the kernels, and leveraging this knowledge to enhance our understanding of wide neural networks in general Lee al. (2019;; Li et al. Cao Gu (201)); Karniadakis et al. (202T):
Huang et al. (2021): Bartlett et al. (2021) Woodworth et al. (2020)
Subsequent discussions arose regarding the role of this limit in the exemplary performance of wide neural networks. Several studies have demonstrated that in certain contexts, infinitely wide neural networks converge to their global minimum at an exponential rate Jacot et al. (2018): Lee et al.
(2019); Du et al. (2019); Allen-Zhu et al. 200 9a b); Daniely (2017): Li & Liang 2018 Du et al.
(2018;; Xu al. 2020). Moreover, wide neural networks have been posited as effective tools for generalization, with connections drawn to the double descent phenomenon Belkin al. (201));
Nakkiran et al. (202T); Mei & Montanari (2022). Although simplified, first-order approximations shown capture many of the critical properties of tnith-width neural networks, making it

---

Published as a conference paper at ICLR 2026
valuable framework for understanding neural network behavior in general Li al. (2019) Littwin et al. (2021); Yang Hu (2020).
These conclusions however encounter
contention when juxtaposed with empirical evidence. Notably, several experiments indicate that for real-world data, NTK- based learning is less effective than its wide (albeit finite) neural network counterparts Lee al. (020);; Fort al. (2020). This apparent "N7K inferiority paradox" suggests that the relationship between the NTK limit and the success of finite neural networks may be more intricate than initially presumed.
relatively understudied aspect within the framework of the neural tangent kernel pertains to the fundamental mechanisms underpinning the phenomenon of linearization. Previous research, such as Chizat et al. (2019). suggests that any gradient-basee learning algorithm inherently possesses an intrinsic scale dictating its lincarization behavior. Furthermore, incorporating an external parameter can modify this intrinsic scale, thereby directly influencing the extent to which linearization manifests.
In a related context, Liu et al. (2020) demonstrated that the ratio between the spectral norm of the Hessian and the Euclidean norm of the gradient governs the rate of linearization. Their analysis also established that, wide neural networks, this ratio typically remains small, thus facilitating linearization.
Another relevant result in this field, presented by Liu et al. (2022). who proposed that the linear behavior observed in wide neural networks emerges fundamentally due to their structural composition as ensembles numerous weak sub-models.
The closest study to this work is that of Dyer & Gur-Ari (019), which introduced methodology grounded in Feynman diagrams to systematically analyze wide neural networks. Their technique enables precise computation of the asymptotic behavior of correlation functions, notably the NTK, in the limit of infinite network width. By leveraging methods from theoretical physics, their work derives finite-width corrections training dynamics, thus providing deeper insights into the evolutionary behavior of wide neural networks beyond the infinite-width approximation. However, their results limited in scope, their setup is restricted, and only considers the average values of these correlations.
1.1 OUR CONTRIBUTIONS
1. We establish that for gradient descent-based learning, linearity is equivalent weak correla- tions between the first and subsequent derivatives of the hypothesis function with respect its parameters at their initial values (Section 3.3). This equivalence is suggested as the fundamental cause for the linearization observed in wide neural networks.
2. We prove directly that wide neural networks display this weak derivative correlations structure. By relying on, and extending the tensor programs formalism Yang & Littwin (2021), our approach uniformly addresses a broader spectrum of architectures at once than any other proof are aware of (Section
3. Drawing from the same concepts, we demonstrate how modifications in the architecture of linearizing learning systems, and more specifically, wide neural networks, affect the linearization rate. This finding is juxtaposed with Chizat et al. (2019)'s 's result regarding the implications of the introduction of an external scale (Sections 3.3.2 and 4.2
4. Harnessing weak correlations formalism, derive bound the deviation from lineariza- tion over time during learning when utilizing stochastic gradient descent (Section 4.1). This is generalization of the traditional result for deterministic gradient descent Lee et (2019). This is crucial, as in most practical scenarios, stochastic gradient descent generalizes better than deterministic gradient descent Lee al. (2200 Fort et al. 2020
5. We introduce the notion of random tensor asymptotic behavior an effective analytical tool to describe the asymptotic behavior of random tensors (Section 2 Such tensors are not only integral to machine learning, but also serve pivotal role in diverse mathematical and physical frameworks. Understanding the typical asymptotic behavior of these tensors is relevant for addressing many questions across these fields.
2

---

Published as a conference paper at ICLR 2026
The overarching simplicity and broad applicability of our findings suggest that weak derivative correlations could very well be the foundational cause for the prevalent linearization attributes observed in wide neural networks, and possibly for other linearizing systems.
RANDOM TENSOR ASYMPTOTIC BEHAVIOR
Random tensors play fundamental role in machine learning in general, and in this work in particular. In this section, we demonstrate the effectiveness of employing the stochastic big-O notation of the subordinate norm to characterize the asymptotic behavior of a general random tensor sequence (hereinafter referred to as a random tensor). Addressing the asymptotic behavior of such tensors involves two inherent challenges: the complexity arising from their multitude of components, and the stochastic nature of these components, In this part, we define an effective way to characterize their asymptotic behavior.
Our primary norm in this work is the Subordinate Tensor Norm, defined as in Kreyszig 1991):
|M = sup { M (o* x x... x o²) o'....,,'' SS....., SN.}
We provide detailed explanation of this definition and discuss its advantages in Appendix B.
We combine this concept with the Stochastic Big- O Notation, introduced in Appendix B.2 which is defined for a sequence of random tensors, denoted by M = {M,)a= Henceforth, we regard M as random tensor depending on a limiting parameter e N This leads us to the definition of new asymptotic upper bound for random tensors.
Denoting V {J N - RR+) as the set of all functions from N to the non-negative real numbers, we introduce the following definition.
Definition 2.1 (Asymptotic Upper Bound of Random Tensors). random tensor M, as defined above, is said to be asymptotically upper bounded by E if:
M =
(2)
and only if:
Vg s.t. 0 (g) lim P (IIMM S (n)) 11
(3)
n-+00
The lower asymptotic bound, f = (M)), is defined analogously, but with the inequality reversed and g o(f).
As with an infinite collection of deterministic sequences, where pointwise convergence often falls short and uniform convergence is required, require definition of a uniform asymptotic bound when discussing an infinite number of random tensors. This concept is rigorously defined in Appendix C.I
Remark 2.1. For a finite number of tensors, it can be shown directly that the uniform bound coincides with the pointwise asymptotic bound, analogously to sequence convergence.
We demonstrate Lemma C.6 that this notation inherits many of the properties of the norm on which it is defined, including all properties of the subordinate norm, delineated in Lemma C.I Furthermore, satisfies several additional useful properties, outlined in Appendix C.3
2.1 PROPERTIES
Remark 2.2. We denote f 5 g or O()) < Ogg) if and only if Ogg). We also denote
or O()) < Ogg) if and only if O(g) and f o g, where f ~ if and only if O (J) 0 (g).
equivalently O (g) and g 0 (f). .. It f= o(g).
important note that f y may hold without requiring
can be readily shown that for any random tensor M, there exist upper and lower bounds such that O (h_) < O (M) O (f+), with h < h Furthermore, if h4 and h satisfy h4 ~ h_, then their asymptotic behavior is unique. That is, for any other pair h4, h", the relation h4 ~ M4 ~ h ~ h
'The results are applicable not only N, but also any other endowed with total order.
3

---

Published as a conference paper at ICLR 2026
still holds (Lemma C.5 In such cases, say that M possesses exact asymptotic behavior, denoted by O (h+) (h-).
The existence of such a pair. however, is not guaranteed, illustrated by a random variable that, for every n E N. has equal probability of one-half to take the value or n. For this variable, the optimal upper bound is (n). and the optimal lower bound is O(1), yet these bounds do not share the same limiting behavior. Analogously, deterministic sequences may exhibit similar behavior, featuring multiple distinct partial limits. In the deterministic case, however, the limsup and liminf serve as the appropriate upper and lower limits, respectively. This observation leads to the question of whether an appropriate asymptotic bound exists in the random case. It turns out that it does.
Theorem 2.1 (Definite Asymptotic Bounds for Tensors). Consider a random tensor M with limiting parameter n as described above. There exists I E NN serving as a tight asymptotic upper bound for M, satisfying:
M = (J) A s.t. f gg: M + (g)
Furthermore, the asymptotic behavior of f is unique.
Explanation. Although the theorem's is statement is intuitive, the challenge arises from the fact that the order defined on N above is not total, even when considering only asymptotic behavior. For example, none of the following relations hold:
sin (m)) cos (mn), cos (Tn) sin (mm), sin (mr) cos (mT)
(5)
We address this issue by employing Zorn's 's lemma, as demonstrated in Appendix C.2
Since every such random tensor M admits a unique definite asymptotic bound f, we refer to this bound the random tensor's asymptotic behavior, and write:
O(M) = O())
(6)
3
WEAK CORRELATIONS AND LINEARIZATION
3.1 NOTATIONS FOR SUPERVISED LEARNING
3.1.1 GENERAL NOTATIONS
Supervised learning involves learning a classifier: a function 9 X + Y that maps an input set (here X C Rdx) an output set (here Y C Rdd), given dataset of its values X' C X, denoted as the "target function". This is achieved by using a hypothesis function, in our case of the form F EN + {f: X + Y} which depends on certain parameters E EN (in the case of fully connected neural networks, for example, the weights and biases). The objective of supervised learning is to find the optimal values for these parameters such that F captures ý best with respect to a cost function C, assumed here convex. We use E X to denote elements in the input set, and i,j = 1, .......... dy to denote the output vector indices. The parameters 0 are enumerated as 0a, a - 1..., [0 = N, and their initial values are denoted by 0g- - 0(0).
We work within the optimization framework of single-input- batch gradient descent-based training, which is defined such that for every learning step N:
A F. 0 (s) 0 (s + 0 (s)= Vn (F (0) (I.), (x.)) (-))))) -IVF (0 (s)) (z.) (F (s)) (I.). .0000 (z.))
Here, V. = 2 represents the gradient operator, x, denotes the s E N-th input data, and C' (w) v,C (y) refers to the derivative of the cost function. The derivative matrix/the Jacobian VF is defined such that for every index pair a, (VF).. = F. We denote as the learning rate and (Ia, 9 (Ix)) as the inputs and labels, respectively. The training path is defined as the sequence of inputs upon which we trained our system, represented by {x, E x We assume that each input along this path is drawn from the same random distribution P. neglecting the possibility of drawing the same input multiple times. The same distribution is used for both training and testing. Moreover, we assume that the hypothesis function and the cost function F, C are analytic in their parameters. We
20,
study learning in the limit where the number of parameters N = |0] 0o, with N = N(n)) being
4

---

Published as a conference paper at ICLR 2026
function of some other parameter N, denoted the "limiting parameter". For neural networks, n is typically chosen as the width of the smallest layer, but we can choose any parameter that governs the system's 's linearization.
Remark 3.1. This framework can be greatly generalized, as we discussed in Appendix !
3.1.2 NEURAL TANGENT KERNEL. NOTATIONS
Numerous gradient descent learning systems (GDML) with different neural network architectures display linear-like structure in their parameters in the large-width limit. In this linear limit, the hypothesis function takes the following form:
Fiin (0) = F (00), Vs E No Fun (s +
Fin (o) - Oo (= (Fimn (+) (T.), (x.)) ))
(8)
with the kernel O defined as:
Vx, x' E X @ (0) (z, (10 O = (0o)
where F is the transpose of VF, the Jacobian.
3.2
THE DERIVATIVES CORRELATIONS
3.2. THE DERIVATIVES CORRELATIONS DEFINITION
In the following, we prove that linearization equivalent to having weak correlations between the first and higher derivatives of the hypothesis function with respect to the initial parameters. We define the derivative correlations as follows:
Definition 3.1 (Derivatives Correlations). We define the derivatives correlations of the hypothesis function for any positive integer E N and non-negative integer No as:
D.d (0) 11 +d
VXD+1F (0)² (VF
(10)
Did!
where the higher order derivatives are defined such that for every d E N and indices i, on
(Vxd F) o Van
F.
More explicitly, presenting the inputs and indices of these tensors:
eD. (0) a14d (Io, I1 (1... =
4 Dld
EN Ea
V Dx+d D1.
Fl, (0) (zo) (Va, Fi, (x1) V (0) (rx))
(11)
The objects in Equation 10 are the correlation of the derivatives in the sense that (1) ... Oa can be viewed as random variables drawn from a uniform distribution of (1...N), while and all other indices are fixed instances and hence deterministic. In this context, V and V
F in Equation 10 can be viewed as random vectors of the variables a Qa, and the summation in Equation 10 represents the (unnormalized) form of the "Pearson correlation" between the two random vectors. The overall coefficient of the learning rate 1y 2 serves as the appropriate normalization, as we will demonstrate in Appendix E and Appendix We will also denote: e (0) = C0 (0).
D.d eDd (0o). cd = ed (fo).
Essentially, D + d represents the degree of the derivative under consideration when interacting with first derivative, whereas specifies the number of copies of the first derivative involved in the interaction.
An example for these correlations is the D 0, d = 1 correlation, the correlation of the first derivative with itself, the kernel (Equation 9):
e (0) V (0) VF = (
(12)
The definition for the asymptotic behavior for these derivative correlations is slightly nuanced due to the many different potential combinations distinct inputs. We rigorously define in Appendix [D.]]
In the remainder of the paper will show how these correlations serve as effective tool for the theoretical analysis of the lincarization of wide neural networks. While this is their main purpose, they can also be used to evaluate linearization rate numerically.
5

---

Published as a conference paper at ICLR 2026
4th (MNNISTT)
g- (CIFAR10)
gh (FastoonMNIST)
agens, reU
syets,
0.035
layers relu
ligers
0.05
agers, reki
0.04
ayers,
0.030
layers
liges mku
0.255
layers, relu
Corndaatoon
4.04
ligers, reli
6.000
0.02
0.003
0.01
0.00 2"
0.000
0.00
27
21
Witth
Figure 1: A comparison of the second-order correlation approximation versus the width of the network, for different datasets (MNIST, CIFARIO and FMNIST) and for different activation functions (ReLU, Erf, Sigmoid) and numbers of layers (1,2,3).
At first glance, this may seem computationally impractical, as computing the D, d-th derivative requires summing (N*) elements. However, we do not actually need to compute the full d-th gradient of F to obtain its correlation. Instead, we can use the chain rule:
111-11 (Ira, 31 ......... .1.). = CD.d (0)) io.ia
2 Dla! On ... Oadia Fio
0u + a1 V., (00, T1) +... + V., (00, xD) + a1+dCar+a + +... aD+dCan+as To
)
(13)
computed for = .... = 0, where ea is the a-th standard basis vector. This approach reduces the computation to summing only O (N) elements.
3.3 EQUIVALENCE OF LINEARITY AND WEAK DERIVATIVES CORRELATIONS
Our main theorems concern the equivalence of linearity and weak derivative correlations. In other words, weak correlations can be regarded as the fundamental reason for the linear structure of wide neural networks. These theorems condition, meaning that when taking n - oo the different components of the system remain finite. We define rigorously exactly what this means in Appendix D.2. We denote such systems properly normalized GDMLs or PGDMLs.
applicable for systems that properly scaled at the initial
3.3.1
OUR MAIN THEOREMS
The relationship between linearization and weak derivative correlations is formalized through the equivalence theorems, which are characterized by a monotonically increasing sequence m())). where lim,-+00 m(n) = Do. This sequence captures the rate of lincarization or correlation decay and constitutes an intrinsic parameter of the system. For instance, in the case of wide neural networks, one typically has mn))= = /n. Nevertheless, m(n) may take any form that satisfies the stated conditions, with its mathematical role lying in defining the equivalence relation.
Theorem 3.1 (Fixed Weak Correlations and Lincarization Equivalence). Given the setup described in this section, for sufficiently small learning rate 1) < 1Jthes the two properties are equivalent, where the asymptotic bounds are uniform for every d. D E N:
1. m(n) fixed weak derivatives correlation:
ed = 1 m (m) ) cDd- -0
(14)
m (n)
2. Simple linearity: for every fixed training step s N:
1
(0) - Flin (8) 0
(15)
m (n)
1 (VxDF(0(3) - VDDF -0000()()>=<((00000 = - (vvn(())
(m(()
(16)
6

---

Published as a conference paper at ICLR 2026
The is defined such that all the correlations uniformly bounded by O(1) to ensure the converges, as shown in Appendix [22] Any system that does not satisfy this condition will diverge within only a few training steps, show in Appendix E.2 For fully connected networks, for example, The
The next theorem delineates
stronger equivalence, which is also relevant for wide neural networks. It also encompasses the scaling of the learning rate.
Theorem 3.2 (Exponential Weak Correlations and Linearization Equivalence). Given the setup described in this section, the two properties are equivalent, where the asymptotic bounds are uniform for every D No, d N:
1. m(n) power weak derivatives correlation: for (D. + (0,1):
CD. = V (m)
(17)
2. Strong lincarity: for every reparamctrisation of the learning rate 1j - r(n)), r(n) > 0 and for every fixed training step E N:
r (()))
(0 Fisn (s)= =
(18)
m (n)
and for every D E N:
- =0
r(()) (v((n) (v(inn))
(19)
F(n)
Explanation: We prove the theorems by considering, for a general learning step s E N, the hypothesis function and its derivatives' Taylor series expansion around step Utilizing Equations 7 and we find that the evolution of the derivatives of F' during learning is governed by a linear combination of correlations of the form:
17 D!
(0) = Σ P.d cD) (- (F (0) 0)))
(20)
for every D E No. where AV*DFI is the change of VxDF. For deterministic functions, it is straightforward to prove the equivalences by employing the arithmetic properties of the big-O notation, and that (i) one can choose any F' (as long as its asymptotic behavior is appropriate), and (ii) different components our sum cannot cancel each other, since we can change 7) continuously; thus, for the sum to be small, all components must be small. The adjustments needed for our case of stochastic functions are minor, as, as we show in Appendix C.3) our tensor asymptotic behavior notation satisfies many of the same properties as the deterministic big-O notation. The complete proofs are Appendix F These theorems describe how our weak correlations govern the transition from the "feature learning" regime to the NTK limit, exemplified in Figure I and Appendix A
3.3.2 RELATION TO RELATED RESULTS
As shown in Theorem 3.2 a rescaling of 7), such as + r(n)n, can either promote or impede the process of linearization. T his observation remains valid for Theorem 3.2 long as 1) < Thhe- This insight offers a deeper understanding of the findings presented by Chizat et al. (019). specifically elucidating how an alteration of an external scale influences lineariattonn by affecting the scales of higher-order correlations differently from those of lower-order correlations.
notable connection to another principal work, [iu et al. (2020). concerns the definition of derivative correlations themselves. In Liu et al.) (2020), the authors established that linearization results from small ratio between the spectral norm of the Hessian and the norm of the gradient. The derivative correlations can be interpreted as a vector. This interpretation refines the results presented in Liu et al. (020). Unlike their approach, which required this ratio be small within neighborhood (ball), our framework demands its minimization specifically at the initialization point. Consequently, it necessitates the decay of higher-order correlations well.
spectral norm, but concerning solely the gradient when considered
7

---

Published as a conference paper at ICLR 2026
Another related work is the work of Huang & Yau (2020). In their work, they characterize the dynamics of wide neural networks using a hierarchy of kemels, where higher-order kernels evolve on slower time scales. Similarly to our paper, these time scales are proportional to -n effectively capturing the deviation from the NTK limit. The relation of their work to ours is most evident in the gradient flow case, where their kemelss can be expressed as linear combinations of our correlations. However, our result more general, as does not rely on the structural assumptions of wide neural networks, and also generalizes to finite learning rate GD. The most immediate benefit of that is that our framework applies to learning systems which are not captured by Huang & Yau (2020)² s framework. More fundamentally, by avoiding restricting ourselves only to neural networks, and instead introducing these new correlations, we obtain not only a sufficient condition for linearization, but an equivalence, providing a universal criterion applicable to generic learning systems.
The connection to Liu et al. (2022) is more abstract. Their argument, that the linear behavior observed in wide neural networks fundamentally emerges from their structural composition as ensembles of numerous weak sub-models is related to our work via the concept that neurons become independent in the infinite-width limit, precisely manifesting the absence of correlation that emphasize.
The most closely related paper we are aware of is Dyer & Gur-Ari (2019). In this work, the authors introduced method using Feynman diagrams to analyze wide neural networks. This approach systematically computes the asymptotic behavior of correlation functions, such as the Neural Tangent Kemel, in the wrge-widdh limit. The main difference between our and their approach is that they measure the asymptotic behavior of correlations directly, rather than averaging their values. This distinction significantly restricts their setup, rendering many of their conclusions more conjectural and less practical compared to our findings.
3.3.3 THE CHICKEN AND THE EGG OF LINEARIZATION AND WEAK CORRELATIONS
The relationship between linearization and weak correlations in over- parameterized systems can be comprehended from two different viewpoints. The first perspective suggests that effective learning in such systems necessitates form of implicit regularization, which inherently favors simplicity Belkin et al. (2019). This preference can be directly incorporated by imposing a linear (or at least approximately linear) structure in highly weer-parameterized regimes. Notably, in certain scenarios, lincarization can facilitate exponential convergence rates, especially with respect to the training datasets, but in some instances, even with respect to the testing datasets Jacot al. (2018) Lee et al. )019); Du et al. (2019): Allen-Zhu et al. (2019b); Daniely (2017); 7); Li & Liang (2018) 8): Du al. 2018); Xu et al. 2020); Allen-Zhu al. (2019a). Hence, weak derivative correlations can be interpreted as pragmatic approach for achieving linearization.
An alternative interpretation, aligning more closely with the spirit of this paper, suggests that weak derivative correlations do not primarily serve as a dynamic mechanism for linearization, but rather as its underlying cause. In this context, persisting derivative correlations may indicate an inherent bias within the system, typically an undesirable one. Therefore, linearization can be viewed as consequence of our attempt to avoid counterproductive biases by demanding weak correlations.
This interpretation suggests that the prevalent perception of kernel learning as biased, and neural networks as unbiased, a result-based fallacy. Had kernel learning empirically outperformed neural networks, it would seem natural to interpret linear learning in the function space (assigning large cigenvalues to simpler functions and smaller ones to complex functions) as unbiased. In other words, we interpret linear learning overly unbiased, while finite neural networks (through mechanisms not fully understood) prioritize the inherent bias of realistic data.
Moreover, if we possess some prior knowledge about an inherent bias in problem. it might be advantagcous to allow some non-decaying correlations, counteracting the process of lincarization. Furthermore, certain biases can enhance general learning algorithms (in the form of implicit and explicit regularization), this perspective might provide valuable insights into the "NTK inferiority paradox" in the Introduction] The reason why linear learning underperforms in comparison to finite neural networks might be that it lacks some beneficial biases in the form of non non-vanishing correlations.
We elaborate this point in Appendix
H
8

---

Published as a conference paper at ICLR 2026
4 PROPERTIES OF WEAKLY CORRELATED PGDMLS
4.1 APPLICATION: DEVIATION FROM LINEARITY DURING LEARNING
Multiple studies have examined the deviation of the hypothesis function F from its linear approxima- tion Fiez (Equation 8) as a function of n for a fixed learning step (especially in the context of wide neural networks). Yet, it seems that no research has explored the deviation between these functions with respect to the learning step for stochastic GD (Equation 7. This aspect crucial since even F - Frin vanishes for the initial learning steps, if it deviates too fast during learning, the linearization may not be evident for realistically large n.
To study how learning systems deviate from their linearization during the training process, we examine the case of an exponentially mnnn-weekly correlated PGDML, with learning rate satisfying ?) Tleor. Here, Yooor is the standard critical learning rate, ensuring that the system is stable in the NTK limit, as explained in Appendix F We consider the problem over the span of
learning
steps, and assume that within this phase the linear solution approaches the true solution exponentially fast for some typical time 0 < T, such that for every = S:
C' (Fisn (s). O ( ()=$). cc" (Fimn (s),97 0
(21)
As show in Appendix F this not a restrictive assumption, especially at the beginning of training, where the deviation from linearization matters the most.
Corollary 4.1 (Weakly Correlated PGDML Deviation Over Time). Given the conditions described above, we obtain that for every s I......:
F (0 Fis (s)=0
(22)
(n)
where the asymptotic bounds are uniform in s, and 8 denotes s the power of zero.
While this result addresses single-input-bater stochastic GD, as we explained in Appendix I can be greatly generalized. Notably, the analysis for stochastic GD may be more relevant even for deterministic GD than conventional approaches that presuppose a fixed training dataset. This is because, while the batch might be fixed, its initial selection is from a stochastic distribution.
We prove the corollary by using a similar induction process as in Theorems 3.J and 3.2 However, here we also consider the dependency on the learning step, as detailed in Appendix F
Explanation.
We are able to bound the deviation over time by leveraging the fact that in the NTK limit, during the
initial phases of the learning process, the system converges towards the target function exponentially
fas Jacot al. (2018); Lee et al. 22019) Du et al. 2019); Allen-Zhu et al. (2019b); Daniely (2017);
Li & Liang (2188)] Du et al. (2018) Xu et al. (2020): Allen-Zhu et al. (2019a). We believe that
subsequent research will be able to produce more refined bounds.
4.2 EXAMPLE: WIDE NEURAL NETWORKS
Numerous studies have demonstrated that wide range of neural network architectures exhibit linearization as they approach the infinite-width limit. However, existing proofs tend to be specific to particular architectures and are often intricate nature. The most comprehensive proof we are aware of that uniformly encompasses diverse set of architectures is presented in Yang & Littwin (202I): Yang (2020). These works employed the tensor product formalism Yang (2019). which can describe most relevant variants of wide neural network architectures the composition of global linear operations and point-wise non-lineat functions.
1. Relying on the semi-linear structure of FCNNs, we show explicitly by induction that for appropriate activation functions, wide neural networks are /n-weakly correlated, and weakly correlated, as shown in Appendix G
'The known bounds for C' ( (Fin f are typically bounds over the variance. In Appendix C.4 discuss how average exponential bound be translated into uniform probabilistic bound.

---

Published as a conference paper at ICLR 2026
2. The framework of low correlations proves effective in discerning how modifications to din)
network influence its linearization. For instance, is evident that suP [n+I]] governs the rate of linearization in FCNNs (Appendix GG. This observation is why we demand for FCNNs that, over the relevant domain, the activation function satisfies:
L O ((n + 1)!)
(23)
where (in) the n-th derivative of the network's ' activation function Φ
3. Our proof for FCNNs be generalized for any wide network described by the tensor programs formalism (Appendix (.5.T). This is because, similarly to FCNNs, all such systems exhibit wide cemi-linear form by definition. Demonstrating that the linearization these systems arises from weak correlations allows us to utilize all of the insights we're 've found for weakly correlated systems in general. We also conceive linearizing network-based systems that fall outside the scope of the tensor programs formalism (Appendix G.5.2).
Leveraging the notation of the asymptotic tensor behavior, our proof accommodates broad spectrum of initialization schemes, extending beyond the Gaussian initialization predominantly employed in other studies.
5 DISCUSSION AND OUTLOOK
The linearization of large and complex learning systems is widespread phenomenon, but our comprehension of it remains limited. We propose that the weak derivatives correlations (3.T) are the underlying structure behind this phenomenon. We demonstrate that this formalism is natural for analyzing linearization: (i) allows us to determine whether, and how fast, a general system undergoes linearization 4.22; and (ii) it aids us in analyzing the deviations from linearization during learning 14.1).
The strength of approach is that it does not rely the structure of wide neural networks. This allows us to describe not only a sufficient condition for linearization, but a true equivalence. Furthermore, enables us identify precisely the structural properties that generate linearization. With this, we were able to provide new insights into the linearization of wide neural networks, accounting for factors such as training duration and activation function properties. Many of these findings were previously unknown or difficult to derive using existing methods.
These insights carry practical implications. Effective systems should neither remain too close their linear limits nor deviate excessively. A cohesive framework that relates convergence behavior to network width, training dynamics, and activation function characteristics can guide the design of more robust and efficient future models.
Our approach raises a pivotal question (3.333) Is the emergence of the weak correlations structure simply tool to ensure linear limit for overparameterized systems? Or do weak correlations indicate an absence of inherent biases, leading to lincarization? If the latter is true, it suggests that in systems with pre-existing knowledge, specific non- non-linear learning methodologies reflecting those biases might be beneficial. That could partially explain why the NTK limit falls short in comparison to finite neural networks.
At the core of our weak derivatives correlation framework the random tensor asymptotic behavior formalism, outlined in Section 2 We have showcased its efficacy in characterizing the asymptotic behavior of random tensors, and we anticipate its utility will extend across disciplines that involve such tensors.
We demonstrate our results empirically in Appendix A. and further discuss generalizations and limitations in Appendix[]
ACKNOWLEDGMENTS
We would like to thank Aviv Orly for conducting the final simulations that concluded this research.
This work is supported in part by the Israeli Science Foundation excellence center, the US-Israel Binational Science Foundation, and the Israel Ministry of Science.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019a.
Zeyuan Allen-Zhn Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International conference on machine learning, pp. 242-252. PMLR, 2019b.
Philip Anderson. More is different: broken symmetry and the nature the hierarchical structure of science. Science, 177(4047):393-396 1972.
Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin, Deep learning: statistical viewpoint. Acta numerica, 30:87-201, 2021.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- -eaaning practice and the classical variance trade-off. Proceedings of the National Academy of Sciences, 116632):5849-115854 2019.
Yvonne M Bishop. Stephen E Fienberg, and Paul W Holland. Discrete multivariate analysis: Theory and practice. Springer Science & Business Media, 2007.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems, 32, 2019.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Amit Daniely. Sgd learns the conjugate kernel class of the network. Advances in Neural Information Processing Systems, 30, 2017.
Yadolah Dodge. The Oxford dictionary of statistical terms. OUP Oxford, 2003.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient descent finds global minima of deep neural networks. International conference on machine learning, pp. 1675-1685. PMLR, 2019.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.0205: 2018
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. arXiv preprint arXiv:1909.1 11304, 2019.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861, 2020.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks, 2013 IEEE international conference on acoustics, speech and signal processing. Icee, 2013.
pp.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hsin-Yuan Huang, Michael Broughton, Masoud Mohseni, Ryan Babbush, Sergio Boixo, Hartmut Neven, and Jarrod R McClean. Power of data in quantum machine learning. Nature communica- tions, 12(1):2631 2021.
Jiaoyang Huang and Homg-Tzee Tzer Yau. Dynamics of deep neural networks and neural tangent hierarchy. In International conference on machine learning, pp. 4p..4542-444 4542-4551. PMLR, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.

---

Published as a conference paper at ICLR 2026
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-infoman informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Erwin Kreyszig. Introductory functional analysis with applications, volume 17. John Wiley & Sons, 1991.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study, Advances in Neural Information Processing Systems, 33: 151155166-112.. 2020.
Sanjeev Li, Arora, Simon Du, Zhiyuan Hu, Wei, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient
descent on structured data. Advances in neural information processing systems, 31, 2018.
Etai Littwin, Tomer Galanti, and Lior Wolf. On random kernels of residual architectures. In Uncertainty in Artificial Intelligence, pp. 897-907. PMLR, 2021.
Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel constant. Advances Neural Information Processing Systems, 33: 15954-15964, 2020.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Transition to linearity of wide neural networks is an emerging property of assembling weak models. arXiv preprint arXiv:2203. 104,0, 2022.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 (4):667-766, 2022.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 202 (12): 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwesser, Ioannis Antonoglou, Veda Panncershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484448)) 2016.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrrize models. In Conference on Learning Theory, pp. 3635-3673. PMLR, 2020.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.1 1848, 2020.
Greg Yang. Wide feedforward recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint
MrXiv:2006114544 14548, 2020.
Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.145 14522, 2020.
Greg Yang and Etai Littwin. Tensor programs iib: Architectural universality of neural tangent kemel training dynamics. In International Conference on Machine Learning, pp. 11762-11772. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2026
CFF FFF)
0.006
MNIST
CIFARIO
0.005
Loss
0.003
0.002
0.000
225
2 Width
2
Figure 2: Relative loss between the neural network and its linear approximation versus the width, for three datasets. We used a learning rate of 1, with 160 samples, ReLU activation, and 1000 epochs.
A EXPERIMENTAL RESULTS
A.1 THEOREM 4.1 EXPERIMENTS
To support our arguments, we present numerical experiments. We show the training and testing dy- namics a neural network and its lincarized approximation for varying network widths. We consider fully connected architecture with mini-batch gradient descent, using learning rates according the NTK normalization Lee et al. (2019), where we chose Y/u = 1. From a computational perspective, focus 10-class classification task, with total of 160 training samples and 32 test samples. We use the MSE loss for training, where each class is represented by different one-hot vector (a 10-dimensional vector).
We perform the analysis for three datasets: CIFARIO, MNIST and FMNIST, and for different activation functions: ReLU, Sigmoid, Erf, and for different numbers of layers L: 1, 2, and 3 (in addition to the output layer). For instance, network with layer and width 128 on MNIST means: 784 128 10.
The simulations were done in the JAX package and were based on Lee et al. (011). We share our code on GitHub. All the results were obtained on an Apple M1 Pro CPU with 32GB of memory; the total rumning time was about hour.
The difference function between f and fin was taken to be:
C (f. 10 xx
ΣΣ Σ (f(x))
fx)m)))))
(24)
x=
where the sub-index represents the output vector index (it depends on the class), and X the set of the data samples, which consists of 32 samples.
Calculating high-order derivatives is very costly in terms of computational resources. Therefore, we estimated the high-order partial derivatives by randomly sampling set D of weights at each layer, and averaging over batch of samples X. Practically, we set [D] = 60 and |X| 160, d, =
2 R
Σ alEaED
(x) (x) on Ongdi(s)
(25)
D
1=1
ZEX
and the same goes for 0.
13

---

Published as a conference paper at ICLR 2026
( layers, relu)
MNIST
0.012
CIFARIO
FashionMNIST
0.010
Correlation
0.008
0.006
0.004
0.000
25
2
2
Width
Figure 3: Third-order correlation approximation function versus different widths, for three datasets. We used learning rate of I, 1160 samples, ReLU activation, and 1000 epochs.
A.2 THEOREM 3.2 EXPERIMENTS
To support arguments, we present numerical experiments that compare the training dynamics of nonlinear neural network with its first-order linearized (NTK/Taylor) approximation, while varying the network width and the learning-rate scaling. The goal is to quantify how accurately the linearized model tracks the true network during (finite-time) SGD training, and how the mismatch scales with width.
Architecture and parametorization. We consider fully connected MLP with two hidden layers,
784
n 10,
(26)
using GELU activations in the hidden layers and no activation on the output logits. Importantly, we the standard PyTorch parumeterization (default nn Linear with Kaiming/He initialization), i... we do not apply the explicit 1/vn NTK output scaling. This choice intentionally places the model outside the strict "NTK parameterization" setting and allows us to probe how learning-rate scaling alone controls the degree of "lazy" inaaiinntiallzztion)) training.
Linearized model (Jacobian- first-order approximation). tialization and 0 the trainable parameters. The linearized model is the first-order Taylor expansion around 0g.
Let 0g denote the random ini-
fun(r; f(x;00) + (x) (0-0u),
(27)
where Je, (x) is the Jacobian of f with respect parameters, evaluated at initialization. In practice, we do not form Ja, (x) explicitly. Instead, we evaluate the product Jo (x) A0 product (JVP), with AØ
via a facobian-vecte (torch. func. sncciionallcall jvp). The initialization 6g the mini-batch order, and all randomness are shared between the nonlinear and linearized models for each seed.
6u, using PyTorch's functional API
Dataset and task. We MNIST with the standard split (60,000 training images and 10,000 test images). Images are flattened to 784 and normalized to [o, 1] by division by 255, without additional standardization. Labels are converted one-hot vectors in R and train using a regression-style mean squared error (MSE) loss between logits and one-hot targets (not cross-entropy).
Training protocol. We train both models using SGD (no momentum, no weight decay), batch size 128, for 250 gradient steps (about 250 128 32,000 samples, i.e., 0.53 epochs). We chose 250 steps because, in this regime, the network behavior is stable but does not yet converge to the true solution, allowing
isolate the effect of -tK-st linearization cleanly possible.
The data loader uses shuffling with a fixed seed per run and arop_last=Teee to ensure identical batch structure across models.
14

---

Published as a conference paper at ICLR 2026
NTK Linearization Error vs Width by Learning Rate Scaling (a)
Norallao Relativ Loss
Normaloon Telestoee Losts
10
10
Nornallze
Width (n)
0.75
Normaltsed staatve LO05
Normolized steettvve
004
10h
10
10'
Width 10"
Wodth
Figure 4: NTK linearization error on MNIST versus network width n for five learning-rate scalings ") 1)on (shown: E 0.255 0.5, -0.75, -1}, with тo 0.1). The y-axis reports the (nor- malized) test-set discrepancy between the trained nonlinear network and its first-order lincarization around initialization. Each marker corresponds to an independent random seed, and the dashed line is a log-log power-law fit, yielding slopes -1.14 (a: 0), ~ 1.14 (a = -0.25), - 1.3
(a = 0.55), 1.73 (a -0.5), and -272 (a 1).
Width sweep and randomness.
We sweep 14 widths,
n {2048, 2435, 2896, 3444, 4096, 4871, 5793, 6889, 8192, 9741, 11585, 13777, 16384, 19484).
(28)
chosen to be approximately multiplicatively spaced by 2/ (four points per doubling). For statistical stability, we average across 100 random seeds; each seed controls both initialization and data ordering, and the nonlinear/linearize pair shares the same seed.
Learning-rate scaling.
We use a width-dependent learning rate
nn))
1o n
170
0.1,
E (0. -025, -0.5, -0.75, -1.03-
(29)
Thus a corresponds to a constant learning rate, while promotes "lazy" dynamics (smaller parameter displacement) as width grows.
enforces 77(n) 70/n, which
Logged metrics and key diserepancy measure. During training (every 10 steps), we log the nonlinear and linearized training losses and the instantancous mini-batch discrepancy between their outputs. After training, we evaluate on the full MNIST test set and define the key metric
MSEnat (f. fin))
Σ X
IIf(;; 0m) fin (z; 0 113
(30)
[Xtesl
15

---

Published as a conference paper at ICLR 2026
where [Xea 10,000 and the norm is over the 10-dimensional logit vector. Lower values indicate more faithful linearization.
Scaling analysis.
For each a, we a power law in width by linear regression in log-log coordinates,
log MSE (f, ftm)) = s(a) log(n) + b(a),
(31)
and report the slope s(o), which quantifies how rapidly the linearization error decreases with width under the learning-rate schedule 17(2) = '/n"".
Compute and reproducibility. The experiment comprises 5 learning-rate exponents x 14 widths 100 seeds 7000 training runs. All runs executed an NVIDIA B200 GPU using CUDA and PyTorel (t (torch. func) for JVP-based linearization. Results are stored per-seed and per- configuration in JSON files containing the full hyperparameters, learning rate, step-wise losses, and the final test metrics in equation 30 enabling full reproducibility.
As shown in Figure 4 the empirical trends do perfectly match the theoretical expectation: the observed slopes are falter than predicted. Several mechanisms may explain this gap. First, the networks may still be too small for the asymptotic regime fully emerge. Second, our theoretical results provide upper bounds, and the two models may stay closer in practice because they learn the same data, which can mask the scaling in finite time.
B ADDITIONAL MATHEMATICAL BACKGROUND
In this section, we elaborate several mathematical concepts that form the foundation for the ideas introduced Section 2 We begin by defining the subordinate tensor norm and its key properties, then introduce a stochastic variant of the "Big O" notation to characterize the asymptotic behavior of random tensors.
B.1 THE SUBORDINATE TENSOR NORM
Let M be tensor of rank r E No- Denote all its indices using the vector such that each ie for 1...r assume values ie Ne. Consequently, the tensor comprises total of N = N1 ... N, elements.
We will the subordinate norm, defined as Kreyszig (1991):
||M|| sup { [M (i' x... x v*) wlnnr E SN..... = sup Σ No.. (Mi .., yt.. SN
(32)
where SN, = f E R: :v. represents the unit vectors of the appropriate dimensions. This norm satisfies certain algehraic properties outlined in Lemma C.T] including: [i] the triangle inequality; [ii] for tensor M and vectors vi ...., .e with appropriately defined product, the condition
| (w) x... x xl S [|M]] ||o'|| -|. ....[--"]] holds; [iii] given two tensors M!!) ?M defining
M3 = M 22) then || = ||| ||
Also, one has M <
M (with equality for vectors) (Lemma where the Frobenius norm is:
|| = Σ M
(33)
B.2 EFFECTIVENESS OF THE STOCHASTIC "BIG O" NOTATION
Consider general random tensor sequence, denoted by M = {M which henceforth we will
consider as a random tensor that depends on a limiting parameter n E
'The results are applicable not only for N, but also for any other set possessing total order
16

---

Published as a conference paper at ICLR 2026
Our objective in this section is to identify method to describe and bound the asymptotic behavior of such a tensor that adheres to elementary algebraic properties. Specifically, we aim for the product of multiple bounded random tensors to be constrained by the product of their respective bounds.
Employing our defined norm (32) we can simplify our problem from general random tensors to positive random variables (rank-zero tensors),
norm satisfies the elementary algebraic properties established in Lemma C.T This reduction substantial; however, the challenge of addressing the non-deterministic nature of variable remains.
One might initially consider the expectation value of the tensor's norm as a solution. This approach unfortunately falls short, because for two positive random variables M1, M2 their product variance is not bounded by the product of their variances. In fact, for M1 = M2, the converse is true:
Var (M,M), 2 Var (M2) Var (M1)
(34)
This issue becomes more pronounced when considering the product of multiple such variables, frequent occurrence in this work. For instance, even with basic zero-mean normal distribution with standard deviation o, the higher moments of this distribution factor as p!! p(p 2p 4)..
Vp E N: (M") p!!o".
(35)
When multiplying multiple such variables. these factors can accumulate in the lower moments, rendering this definition impractical for our purposes. Similarly, any attempt to define asymptotic behavior using the variable's 's moments encounters comparable difficulties.
To circumvent these challenges, we adopt the stochastic big-O notation Dodge 2003 Bishop et 2007
C RANDOM TENSORS ASYMPTOTIC BEHAVIOR
In the following sections, we repeatedly use the results of this section. Due to their intuitive nature, we may not consistently specify when we so, which lemma/theorem we are employing.
C.I PROPERTIES OF OUR NORM
In this subsection, we explore the properties satisfied by the subordinate norm. We omit the proofs as these properties are either well known or straightforward to prove (and also enjoyable to derive).
Lemma C.1 (Algebraic properties of the subordinate norm). The subordinate tensor norm 32 satisfies the following algebraic properties:
1. Given a tensor sequence (M D d== where D E NUfoo), it satisfies the triangle inequality:
Σ
(a)
S Σ
M (4)
(36)
where equality holds when the tensors are positively linearly dependent.
2. Given tensor Mi 1 S in < N₂ for < k S r, and S r E N vectors f/ ...ug (with the same range of indices), then:
| wt x D S || 1- 11
(37)
3. Given two tensors m! and 2 their direct product M7,7, (M M (222)
7,52
! (2) satisfies:
| = |
M
(38)
The generalization to an arbitrary finite number of tensors is trivial.
'Our definition slightly differs from the standard definition of big-O in probability notation, but it is straightforward to show their equivalence.
17

---

Published as a conference paper at ICLR 2026
Remark C.1. Parts and B are also satisfied by the Frobenius norm.
Lemma C.2 (Relation to the Frobenius Norm). Given a tensor M of rank r E N, the following holds:
1. For any tensor .::
|M S ||
(39)
and = =1 (i.e., the tensor a vector), then:
|M
|
1 Σ M².
MM
(40)
2. For every r' = ....r:
|M = sup [|Ne (
o² x ( wr't1 x x''tlx...x x v
(ot
0r -01--x 1x
w² E SN, ...... - SN rttt1 E SN.44 o' E SN.
(41)
The first part of the lemma demonstrates that our norm is always bounded by the Frobenius norm, and the two norms coincide for vectors. The second part generalizes the first, indicating that when reducing any tensor to vector, the two norms once again agree.
Lemma C.3 (Properties of the Maximizing Vectors). Given a tensor M of rank r E N, there exist vectors o² ...." of norm 1 such that:
M = M u x ... x u".
(42)
This result indicates that the supremum is indeed maximum. The vectors v also the ones that maximize the cases demonstrated the previous lemma.
Moreover, the tensor is symmetric with respect to the permutation of the indices i i2, ig and non-zero, then:
n = 2 pf
(43)
Remark C.2. For = O, any set of vectors maximizes our result, irrespective whether the vectors are identical distinct.
C.2 EXISTENCE AND UNIQUENESS OF THE TENSOR ASYMPTOTIC BEHAVIOR
In this section, we discuss some of the more general properties that the tensor asymptotic behavior notation satisfies, regardless of the norm is defined with respect to. The first lemma present is useful equivalent definition for bounding tensor asymptotic behavior. This equivalent definition will be beneficial for our later discussion.
Lemma C.4 (Equivalent Definitions for Tensor's Asymptotic Bound). For any random tensor M and f E N,, the two definitions for bounding the tensor's 's asymptotic behavior O(M) S O(f) are equivalent (the first is the original definition, (2.1)
1.
Vg E N.t. o lim ((] S (n) 1.
(44)
n-000
2.
lim lim P S of (n)) = 1
(45)
c-000 100
(The same applies for 0()) S O(M)).
The order in which we take the limits in Equation 45 is crucial, as any random tensor satisfies the equation for any f if we take the limit in e first
is straightforward to show that any random tensor M has lower and upper bounds: Lemma C.5 (Bounding Tensor Asymptotic Behavior), Given random tensor M, there exist h_ h E N such that:
( ( (0()))
(46)
18

---

Published as a conference paper at ICLR 2026
To prove that the asymptotic tensor behavior has meaning, need to show that bounds not only always exist, but that there is always one well-defined "best" upper bound (Theorem 2.1.. We prove this theorem after Lemma C.3 by using Zorn's lemma.
Remark C.3. It is simple to show that if there exist lower and upper bounds such that h4 = h__ and
the exact asymptotic behavior is well defined, then they are the "definite bound" of Theorem 2.1
Proof- - Lemma
C.4
We will prove the two directions of the lemma separately.
Assuming the second condition in Equation 45 is satisfied:
Given some 0 < p < 1, we know using Equation 45 that there some < such that for sufficiently
large N:
p S (I) S cf (f))
(47)
Given some g E A such that f o(g), we know that for sufficiently large E N:
cf (n) g
(48)
which means that for sufficiently large E N: p P MI S of (n)) - P (IlMMll g As we proved that for any < p < we get that:
(49)
lim (P ([]lall S (n))) = 1.
(50)
n
And as we proved that for any arbitrary g E N such that f = o(g). we proved the first part of the lemma.
Assuming the first condition in Equation 44 is satisfied:
If we assume in contradiction that Equation 45 is not satisfied, we get that there is some 0 < p < 1 such that:
Vno E N, O c, 3mg S n N::::: P ([IM, I cf (n)) <p.
(51)
In particular, this means that if we choose the sequence {c- = i} no such that:
there are îu < nn < ñ..... N N
e P (IIMa, ll < if (ñ.)) <pp
(52)
The reason that we can require that { is increasing is that we know that we can find such n-values for any sufficiently large no and for any c. So by induction we can require every time that every π4 is bigger than all previous values..
Assuming the second condition of Equation 45 is satisfied:
Suppose, by contradiction, that Equation is not satisfied. Then, there exists some 0 < p < 1 such that:
Vno E N, 0 < c, 3ng I n E N: P( s cf (n)) <p. In particular, if we choose the sequence Vi E N: c i, there exist ân < nl ny N such that: V : M|, || S if (ñx) <p.
(53)
(54)
Since we can find such n-values for any sufficiently large no and any c, we can require by induction that each ñ, greater than all previous ñ-values.
We can now define the function:
Vn N: g (n) (max e i S {n)
(55)
Since { is increasing, we know by the Archimedean property that max {i E N n, S <n) is also increasing and unbounded, which implies:
lim g (n)
lim max (iEN) n, S 5n) = 00. no
(56)
n 000 f (n)
However, by using Equations 52 and 55
we also have:
Vng N, 3ng S N: P (II S (n))) <p,
(57)
19

---

Published as a conference paper at ICLR 2026
which means that:
lim P ((I S (n)) 1.
(58)
This contradicts our assumption Equation 44 Therefore, by reductio ad impossibile, Equation must be satisfied, completing the proof for the second direction.
Proof- Lemma C.5
For trivial lower bound, we choose h such that Vn N. (n) O.
We define h as follows:
Vn N inf m E 1- 1 - PP
S m)
(59)
The infimum and the function are well defined because:
1. The is well defined.
2. The set non-empty: it were empty, it would imply that there some probability that ||M]], which is positive number, is larger than any real number, which is impossible.
3. The is defined with total order and has lower bound, 0.
Since for any 0 < p < 1, there exists some no E N such that:
Vno S n E - P (| E m),
(60)
we know that for any h4 < E N, this also true, which implies:
O(M) S 0h+)),
(61)
completing the proof.
Proof- Theorem 2.1
General Idea of the Proof:
The proof proceeds follows:
We consider the set of all upper bounds for M, denoted by 2, and use Zorn's lemma to show that every chain this set has a lower bound within Z.
Applying Zorn's lemma again, we demonstrate that 2 has a minimum.
We then show that the limiting behavior of this minimum unique.
Existence of an Infimum for the Upper Bound Set: We begin by defining the set:
2 = (0(
(62)
This set is:
1. Well defined.
2. Non-empty (as proven in Lemma C.5
3. Defined with partial order h < h2 + O (h1) 0 (h2).
'A chain, defined in set theory, is subset for which the given partial order becomes total order.
20

---

Published as a conference paper at ICLR 2026
According to Zor''s lemma, if all chains in this set have lower bound in 2, then 2 has at least one minimum.
Given some chain in the set, C Z, know it is lower bounded by the function h_ which means (by using Zorn's 's lemma) it has at least one infimum (a lower bound without any larger lower bounds). We choose such infimum and denote it by I E.N.
Proving that the Infimum is in Z:
We assume, by contradiction, that this infimum is not in 2, which means there exists W such that = o(g) and for every 0 < p < 1 and no E N, there exists no S n E N such that:
P M|l S g (n)) <p
(63)
Since = o(g), we know that for any E IR and sufficiently large n E N:
el (n) S (n))
(64)
Combining these equations, we obtain:
vO C, no N 3no
N: P M(]l S cl (n)) <p.
(65)
In particular, if we choose the sequence Vi E N 4 = 22, there exist ūn < ū2 < it. E such that:
Vi N: P (II S (ñ:)) <p.
(66)
We require that { is increasing for the same reason before, we know that find such arbitrarily large n-values for any sufficiently large no and for any c, so we can, by induction, demand that each n; is greater than all previous á1 -1
Now, we define the function:
{ iT (m) E J (n) (
(67)
((m) : else
This function is well defined because there is only one for any n such that n = ñg as it is an increasing sequence.
Using Equations 66 67
we find that the subsequence (n+ satisfies:
(68) Applying Lemma C.4 for the equivalency of the asymptotic bound definition, conclude that above this subsequence E 2, which implies that above this subsequence J is a lower bound of and consequently also of C. Moreover, for all other n, have J = and since I is lower bound of C, so is J. Since every E N belongs to one of these subsequences, we conclude that J is lower bound of C in general.
VIEN: P (l) (2),, i (f:)) <p.
Furthermore, for every i E N, as 1 S c; we have:
Vn [n) < J(n)+ S OJJ).
(69)
However, since { is increasing and unbounded, we know that there exists at least one subse- quence such that:
J (
lim R
lim G -
(70)
7 T(m))
This implies:
( <
+ ~
(71)
We have discovered that J is greater than 7, but smaller than all functions in C, which implies that a larger lower bound than the infimum, which is impossible, and implies by reductio ad impossibile that every chain in Z has a lower bound in Z.
Existence and Uniqueness the Minimum:
Using Zorn's 's lemma, now know that Z has at least minimum, denoted by f N. Our remaining task is to show that all other minima in Z exhibit the same limiting behavior as f. which implies the uniqueness of the minimal limiting behavior.
21

---

Published as a conference paper at ICLR 2026
Let g E N be another minimum. We define:
Vn E h (n) min {f (n), ,9 (n))
(72)
We know that h S f.g. and we also know that h E since f.g E and for every 0 < p < 1 we can choose the maximal from f and Thus, h 2, but h S f.9 well, where f,g are minima themselves. This implies:
0 - 0 = (g) - (f) = (
(73)
Therefore, there exists a unique minimal limiting behavior, which implies that the tensor's 's asymptotic behavior always well defined.
Remark C.4. In our proof, we employed Zor''s lemma twice. First, we used it to demonstrate the existence of an infimum for every chain, and then, after showing that these infima belong to Z, we employed it again to establish that Z has minimum. At first glance, it may seem perplexing that needed to rely on Zorn's lemma, an incredibly abstract and powerful tool equivalent to the somewhat controversial axiom of choice, prove that the tensor's asymptotic behavior, which has much more grounded and intuitive meaning, is well defined.
One possible explanation for this discrepancy that we may have actually required the full power of the axiom of choice, and our structures could be simple enough that an alternative approach could have been taken to prove our theorem without using Zorn's 's lemma. We believe. however, that in the most general case, Zorn's 's lemma was indeed necessary, but it was only relevant for extreme distributions lacking any tangible "physical meaning." " For any well-defined set of distributions with a clear underlying meaning, one could potentially find an alternative method for demonstrating the existence of tight bound without invoking Zorn's s lemma.
In any case, as we demonstrated in Lemma C.5 there is no need for any of these high-level tools to prove the existence an upper bound.
C.3 PROPERTIES OF THE ASYMPTOTIC BEHAVIOR NOTATION
Having established that notation is meaningful, now aim demonstrate its usefulness. First, we need to address our earlier issue and define a "uniform asymptotic bound." Once again, we omit the proofs in this and the following sections.
Definition C.1 (Uniform Tensor Asymptotic Bound). Given a sequence of random tensors
1> {M d=1' where a limiting parameter n, we say that it is uniformly asymptotically upper bounded by f E W under some rising monotonic function K!
N U {oo} (or, more precisely, a sequence of random tensor sequences) with
R::
=d.1... D: (M) s SOCK
Uniformly,
(74)
if and only if:
Vy E N s.t og) lim P P(vd (vd==1. -dd=1...
M K g =1.
(75)
n=000
The definition for uniform lower asymptotic bound is analogous with reversed directions.
Remark C.5. As discussed in Definition 2.1 it is clear that if D is finite, then uniform bound is equivalent to pointwise bound.
Lemma C.6 (Asymptotic Notation Inherits its Norm Properties). Given a random tensor M and
sequence of jointly distributed random tensors {M(}] d=1 (with M as well), where D E NU {co],
such that they are all uniformly bounded:
AA = 1... id.....D:O. 1::0MM(0) (M < (KC f)
Uniformly
(76)
then:
If some positive linear combination of (d) norms satisfies an inequality of the form:
D
D2
|| <
Σ A
II
|Mall
(77)
d=1
d=
D
D
22

---

Published as a conference paper at ICLR 2026
where all of the coefficients positive: =d= À and divided 1... D into a sequence of finite intervals: 0 == D2 <
D2 <... < Dp = D. Then the asymptotic behavior of all the tensors satisfies the same inequality well for every h f:
D Σ
Dā
(M) oo
A
II
10t oh
(78)
d=1
d-
and if the inequality is an equality for the norm, it is also an equality for the "large O" notation.
2. Our asymptotic notation inherits all of the properties presented in Lemma
Remark C.6. The lemma still holds even if the tensor has additional indices, as will see in Section G.4 provided the number of additional index possibilities remains finite in n.
C.4 EXPLORING THE RELATIONSHIP BETWEEN ASYMPTOTIC BEHAVIOR NOTATION AND THE TENSORS' MOMENTS
The final aspect of the asymptotic behavior notation wish to explore is the relationship between this notation and the moments of our tensors' norm or variables. This relationship is relatively intuitive and straightforward, and will be useful in Section G We first introduce a simple notation for tensor M7 that will assist in examining tensor moments, the norm expectation value, defined as:
V N (IM1))).
M
(79)
Lemma C.7 (Asymptotic Behavior and Tensor Moments Equivalency). Given a random tensor M and function
N,, then:
O(M) S O(f),
(80)
if and only if, with probability arbitrarily close 1:
[M] (f)
(81)
The lemma is also applicable for the uniform bound in the case of an infinite collection of random tensors.
In Section 4.1 we highlighted that most assertions concerning the convergence of CF 9) relate to its expected value. However, we can now also associate it with its asymptotic behavior throughout the entire training trajectory. This association stems from the understanding that, if our system exhibits a known average decay, the likelihood of significant deviations from this typical variance range must also decrease, and exponentially (at any decaying rate that is slower than our original rate). Given that decaying geometric sums are convergent, we can infer that the overall probability of the system defying our predicted asymptotic behavior is likewise convergent. Given that we can choose the scaling of this probability arbitrarily, we can set conditions such that the cumulative probability of any deviation is arbitrarily small. We introduce this notion for the reader's consideration and propose a detailed formulation as a future exercise.
D ADDITIONAL DEFINITIONS
D.1 DERIVATIVES CORRELATIONS ASYMPTOTIC BEHAVTOR
In our main text 3.2.1 we discussed that the definition for the asymptotic behavior of the derivatives correlations is slightly nuanced due to the many different potential combinations of distinct inputs. Here we define it rigorously.
Definition D.1 (Derivatives Correlations Asymptotic Behavior). For every D E No, d E N, and d < d2 < ... d such that d1 +... d d:
Oz (cD.d) = O 0rg. (cDid to, xd1 ... x Xdj
(82)
23

---

The general idea has been outlined at Published as a conference paper at ICLR 2026
The input order does not matter, the correlations symmetric with respect their first derivatives. The factor d=l--dd!! accounts for the possible combinations. If f E N,, we say:
(83)
if and only if all combinations uniformly bounded by f. In the continuous limit (extended training time), only d1 = = dd = 1 remains relevant.
D.2 PROPERLY NORMALIZED GDML
Our main Theorems 3. 3.2 and Corollary (4. are applicable for systems that are properly scaled at the initial condition as oC, defined follows.
Definition D.2 (PGDML). Given GDML described in Section 3.1 we say that it is properly normalized (and denote it as PGDML) if and only if:
F (0o) (n")
(84)
AF (0)) = F (0 (1)) - F (00) = O (n²)
(85)
e¹ (Ni) OVF (0n))²
(86)
AA e N: O s OVF (0o))* Uniformly.
(87)
Where n symbolizes n in the power of zero.
The first two conditions 84 85 ensure that our system scale remains finite at the initial condition. Condition 86 stipulates that the asymptotic behavior of the kernel is maximal, given the asymptotic behavior of the first derivative. This condition ensures that system is genuinely learning and not only memorizing. This is because the kernel for different inputs is responsible for extrapolation, while the kernel with the same input twice is responsible for memorization Condition asserts none of the higher derivatives dominate the first for n - oc, property that most realistic scalable GDMLs satisfy, because if it satisfied, gradient descent becomes irrelevant. We show that wide neural networks in general satisfy this property in Appendix G.5
asserts that
E PROOF OF THEOREMS 3.1 3.2
We can now proceed with the proofs of Theorems 3.] and 3.2 the end of Section 33.1
F. FIRST DIRECTION OF THROREMS 3.1
Now that understand how to work with the asymptotic behavior of random tensors, we can proceed to prove our main theorems and corollary. We begin with the first direction of the theorems.
Lemma E.1 (Linearization Requires Weak Correlations).
1. In Theorem 3.1
if condition I is satisfied, then condition 2i satisfied well.
2. In Theorem
3.2 conditionn | is satisfied, then condition iis satisfied as well.
Proof. We only demonstrate that the O₂ (C') are bounded; the proof that the rest are bounded the same, by considering more learning steps after the initial condition.
"This is direct consequence of the NTK equation of motion g For example, in the case of single input point, the system hehaves like memorization algorithm for that one input. However, the term O(x, x') governs how the value of the function is influenced by its values other points x'.
24

---

Published as a conference paper at ICLR 2026
For the initial condition, know that any reparameterization 0 satisfies 8 (82))
(0 - Flin (1)
Ed=1
(mg)
4 xxdp (0o) (F (11)²) xd
(21) (())))
d!
(rT) VF (0n) VF (0o) (21) 1 (-C(( (F)) =
(88)
vxdp (0n) VF (211) xd
2d=2 rd
(-C' (x1) (x))))
Ed= pd (e :a (-C' (F (B)) (2)))((2))))
and in the same way, for every D N:
(rr) A ( (rn) x Fin (0 (1)) 2d 1 +d (cb.d) (-C' (F (0o) (z1) . (z1))) xd
(89)
Utilizing Lemma C.6 becomes evident that for properly normalized gradient descent- systems:
o ( (F' (1) x)) S So D.)) O (e' (F (0o) =)) (C
(90)
However, since our theorem should work for any ú. we can choose á = F (f6) + and obtain:
o ( (F (0o). x)) o (c" (CD.00' (c) (c) 0 (c
(9)
we can choose such that ('(c) the vector that maximizes the correlation, since C' is convex and the correlations are symmetrical.
Given that we can choose an open set of different scalings of r, we know the different elements the series cannot cancel each other out. Consequently, for F - Fin to decay, all the distinct elements must decay.
Assuming condition in Theorem 3.1
Given that (FF( Fiin (1))
0 m(n)
and for every D N
we have
O 1 VD F (0o)
o Vmnts it follows that each correlation must
decay at least like:
1 m (n)
v2 < de< O (C)) 0
Uniformly,
(92)
and
VD,dEN:0(c2.) 50
Uniformly.
(93)
mmm (n)
This completes the first part of the proof.
Assuming condition] T in Theorem 3.2
By taking r(n) arbitrarily close to m(n), we find that for F (0) - Fisn (1) to decay, ddd must decay as well, which implies that:
m (n) )
:EN::((E)) 50
(94)
and
00 VD IDEN deN: (pD.))
(95)
m (n)
This concludes proof.
25

---

Published as a conference paper at ICLR 2026
E.2 SECOND DIRECTION OF THEOREMS 3.1 3.2
We now prove the other direction of the theorems, focusing on Theorem 3.1 since the proofs for the other theorems are essentially the same. should also be noted that Corollary [..] which will be proven next, is almost generalization of this direction, except that it is only applicable for sufficiently small learning rates.
Lemma E.2 (Asymptotic Behavior Normalization for Weakly Correlated PGDML). Consider
weakly correlated PGDML described in Theorems 3.2 Then we have:
AD E P
(1) Uniformly.
(96)
With Lemma E.2 at hand, we can now demonstrate the second direction of the theorem by proving slightly stronger version of it.
Lemma E.3 (Weak Correlations Create Linearization (First Theorem)). Assuming the conditions of Theorem part then for every 1...S:
1.
0 (FF(O())) - Fin (s)) 0 m (n))
(97)
2.
0 VF - VF (Bo) ST
(98)
3. For every 2 < DEN
so
o
uniformly.
(99)
v (n)
Here, is an asymptotic notation such that 7 = o mm(n)
(v) and when multiplied with first
derivative of the hypothesis function in its initial condition, it exhibits an asymptotic behavior of O (Tom)) F (Bo)
O m(n)
From proving Lemmas E. and E.3
conclude that Theorems 3.1 and 3.2 have been proven.
Proof of Lemma E.2
Assume that the lemma not satisfied, i.e.,
O (VF (06)) N (1).
(100)
then for some probability 0 <<p < 1, we have:
(1) < OVF
(101)
Utilizing the third property of PGDML systems (86). we conclude that for some relevant probability:
( < 0
(102)
However, for the reasons discussed earlier, the different elements in the equation of motion cannot cancel each other out, 7) can be chosen from an open set. This implies that the second property of PGDML systems
85 cannot be satisfied, leading to the conclusion that:
17 ( (0o)) < (1),
(103)
must hold.
By employing the fourth property (87 of PGDML systems, obtain the desired result.
26

---

Published as a conference paper at ICLR 2026
Proof of Lemma E.3
We will prove the lemma using induction over the learning steps. The induction base for the "zero" step, where 0 Co. is trivial. Assuming the lemma holds for E No, we observe that for every (D E No, E N) * (0, 1). the (D, corrrellatnoon satisfies the following for sufficiently small learning rate ry:
p (0(())
= n - + A (8)))) (8
11 1x F (00) +2)²
-
( (06) +7) xd
7x (F(FF(0)) xd-1
eDd I4 (AEVF (80)
(104)
Pid P (0o) (xx
(ク (06)
xd-1
+ comb + +0
mmnm)
CD. cD.+0
+ 0
1
+ de
x x
mfn)
m(n)
m(nn
@Dd +0 m(n))
Here, used the derivatives correlation definition, the relevant lemmas, the induction hypothesis, the bound of the correlations from condition] [ and the definition of 7-
By employing the derivatives correlation definition and condition 1 observe that:
<e N: O (C*) 0 m(n) edNN 0 (Cl.a) =7,
(105)
V2 < DEN,dEN,
=0 (Tie(t))
Furthermore:
xd =
ClA (O)) = n7a + x F (86)² 172 V (0o) (+1gx4+1F (0o) V (f.)) xd+1 = cd+1
(106)
Hence, using this equation, well. By incorporating this equation into our equation of motion and employing the lemmas, we find that for sufficiently small learning rate, F (0 (s + 1)) also satisfies the lemma. Consequently, by induction, the lemma holds for all E N.
deduce that P.d (s 1)) satisfies the given conditions as
F PROOF OF COROLLARY
In this section, we prove Corollary direction of Theorems 3.1 and 3.2 albeit with an additional focus on the evolution of the deviation
The general approach for this proof is akin that of the first
throughout the induction process.
Given the complexity of tracking all the derivatives simultaneously, our strategy involves monitoring the difference between the parameters and their linearization, as expressed in Equation [13 A significant challenge arises in solving the equation of motion that these parameters must satisfy.
To circumvent this issue, we establish link between this deviation and the deviation of the general- ization function from its lincarization (113) up the highest order, outlined in Equation 118] By considering only the lowest-order terms, we obtain an equation of motion (25).. In cases where the cost function decays exponentially, are able to bound the deviation of this equation.
F.1 RELATIONS BETWEEN DIFFERENT LINEARIZATIONS
In the main text, we linearized as Finn X by first considering only the linear part of F, and then examining how it changes time for given training path. However, there alternative ways
27

---

Published as a conference paper at ICLR 2026
linearize F that can be useful to consider. One such method involves taking only the linear part of F, without considering the training path:
F(O) F (0o) VF (0o) (0- (0- 0n)
(107)
Another useful definition is to examine how would develop over time under the linear approximation for our training path:
8jan (0) Gu E iin (s + 1) 8tin (s) V (6n) (x.) C' (Fisn (s)
(108)
y (x.))
can be observed that Fiin: F, Oion satisfy the following relation:
vs E No: Fiin (s) (fisn (s))
(109)
more refined relation is the one between F (ftin) and Fisn (0), defined for every 8 = 0...S follows:
S as
e22
O (F (Otin(s)) Finn (s)) S 0 (mm))) (n))
(110)
where @ is defined as:
Definition F.1 (Typical Linear Cumulative Deviation). We define the typical linear cumulative deviation the bound of the cumulative deviation of Fiin from ÿ:
8-1 Oc' (Fin (5) 1))
(e(s))
(111)
and in case:
(e(ss) so (1- 00
(112)
This implies that e(s) = (n)), which is essential for proving 110p. We will not provide this proof here, will not use it directly in the remainder of this paper, and will soon prove many similar identities.
F.2 SMALL PERTURBATION FROM THE LINEAR SOLUTION
The initial approach of the proof aimed demonstrate that F only deviates slightly from Fiins and that its derivatives also deviate slightly from their initial values. The intention was to use induction to show that this holds at each time step. This method effective if the goal is merely to prove that F for fixed time step. However, it poses challenges when
converges to Fin rate of O (m))s
attempting to understand how the two functions deviate from each other over time. This is due to the necessity of simultaneously tracking the evolution of all derivatives and the changes in correlations over time, which is nearly impossible.
To cireumvent this issue, rather than tracking all derivatives, we will calculate how F (0 (s)) deviates from Fiam (s) by utilizing similar relationship to the one we discovered between Ozn and Fin- This will allow to establish bounds on /' Fin. Although the two approaches are equivalent, and the first one is more intuitively clear, the second approach simplifies the analysis by focusing on single object, F Fimn-
In the following lemma, we demonstrate how a small perturbation at a given step (s = 0...S -) results in small perturbation the subsequent step (s + 1). Then, we will use these results to inductively show the deviation over time between the hypothesis function and its linear approximation.
We denote:
8(s) - F F Fiin (s) ((s) 0(s) Otin (s),
(113)
and assume that the deviation from linearity is small, hence:
f(s)
(8())) s 0
(0(())))))))))
(114)
m(n)
28

---

Published as a conference paper at ICLR 2026
where
fs) e(s) = o
= (n))
(115)
For some parts of our lemma, it will also be relevant to separate the deviation of the parameters into two components:
((s) &r (8) + Cm (s)
(116)
such that:
9m (s)
O (G7 (s)) - 0 (0, (s)) 7, O (Sm (s)) = O
(117)
m (n)
Remark F.1. Here, we consider the case of general rate of convergence for C' (Fin:: 9), rather than exclusively focusing on an exponential one. This is done to simplify the generalization of our results for the reader.
Remark F.2. In the following lemma and its proof, we use the symbol ">-" to denote higher-order terms of the expressions. This is justified by our assumption that we are working within the framework of analytic functions, where the sum of all higher-order terms still converges.
Lemma F.1 (Deviation of the Parameters and of the Hypothesis Function Relationships). Given the conditions described above, then up the leading order:
1.
VF (s) = F (0())) - Fin (s) (5)~ 17a VF (0o)² 5m (s) (
So (s)+
E. EgT C²C' (Fin (s1) 9) x C' (Fin (82) 9) +
(118)
2 > Cl. S4 (s) C' (Fis (8') 9) + n (0o)² T 5y (s) x2
which means:
O (5 (F Fiin (s)) S
[m)
0
[g- (g)
< 0
(g(a)+
(119)
(
m2(n)
2.
0 n -FF(()))""~-V(((0))))) (X0 +(())))
(120)
3.
C' (F C' (Fin (s) ~ C" (Fim (8), (),
(121)
where C" (Fii (s). 9) denotes positive random matrix such that, if the asymptotic behavior of C' (Fi,, 9) is bounded, then C" (Fiins), D) is bounded as well (as in our setting).
4.
Oun (s+1)-n7t C' (Fisn (8).0)n²7)
(122)
FI (0o) C" (Frin
which means:
o ( S [((]
+ + (Fin (s), {s))) (123)
m (n)
5.
10 0OC" (Fim (s), <
o ((s())+(())) OC' (Fi. s))
(124)
m(n)
Remark F.3. An important note for our proofs is that all these components can be generalized the case where {(s), ((s) the "original" deviations, long they satisfy Equation 114
to
29

---

Published as a conference paper at ICLR 2026
We can now this result prove Corollary 4.1 by induction. In fact, for the conditions of the corollary at s = O, the induction hypothesis is trivially satisfied as (())(O) = Fiin (0) and 8(0) Otan (0). It is straightforward to show that the contributions of the part multiplied by O (C' (Fiin (+), ú)) are irrelevant for the possible deviation, as C' (Fii (s), 9) + 0 and O (e (s)) S are left with equations of motion for the asymptotic behavior of the form:
O (1). Consequently,
{() m (n)
((s)) 0 -(()))))(((3) 0
=s))
8 + 1) (s) + OOC" (Fitn (s).0) 00 (125)
However, O and C" are positive definite and bounded matrices, so for a learning rate that is sufficiently small (which would be of the same order of magnitude as the learning rate needed for our system to consistently learn, and for the case where C(x) Fx² exactly the same), we find that on average this term can only contribute to the shrinkage of (s)). This means that neglecting this term for large provides an upper bound for the rate of deviation. Thus, we find that the asymptotic behavior of 8 (and consequently, () with respect time for large is bounded by:
8 (8 + 1) 5 (8) = 0.
(126)
This proves our corollary.
Proof.
Part
F() = F (imn (8) 1
F (0 17 - VF (86) C' C'mm (s)) + n (s) =2
F (0o) Ex- (Fim (5'),0) +) (0o)² (()) +
E C2C (Fii 0)) x C² (Fin (82), ,0) +
(127)
2E (()) (Fiin (s'), 9) + k (0o)² T Ç (s) +... 23
Fiin (3) + V (0o)² (s) + Ex CCC (Fin (s1) 1) x C' (Fin (32)
2 > ed ' (s) C' (Fim (5).0) + x 2
(0))² (ss)x²
where in (1) we used Equation 108 and the definition of 0jan. in (2) we expanded our generalization function as a Taylor series and used the definition of the derivatives correlations (3.T). In (3) we used the fact that under our assumptions our system is exponentially weakly correlated. Using this result, we obtain our desired identity.
Subtracting Flins we get, up to the leading order:
(0 Fin (8) ~
( (s) + E... C2C² (Fimn (51).0 C' (Fiin (52)
2 2Σ 1 (s) (Fim (5'),) + x2 (0o)² (s))x²
V (06)* Sm (s) + 172 F (0o)² G, (s)+
E Ee C (Fisn (81),) x C¹ (Fimn (82), +
(128)
15-1 2
Cli Sy (8) C' (Fiin (s'),0) + x2 F (0o)² So xs)x2=
O
+ 0
o(s)²
+ 20 ple)a-(a)
+
m
mln]
m(n)
8m (8) O
+
(o)+
((()))
mx)
m(n)
m(n)
which finishes our proof.
Part 2
30

---

Published as a conference paper at ICLR 2026
Using the same ideas, we get:
n P (0))* = v/ (0rm (s) + ()
r
17 VEVF 0 - 13 " -
VF (0)) C² (Fiin (5') 7p
(129)
Ex C (Pion + ((8) +..
77 V (0o) + 0 (s) +z)
Taking the transpose on both sides completes the proof.
Part 3
Using the definition of 8 and the fact that C is analytical, we know that up to the highest order:
C' (0 (8)) 0) C' (Fran (5) + 6 (s) 0) = C' (Fisn (+) () + C" (Fin (8) 0) 6 (+)
(130)
Since is convex (3). its second derivative is always a positive matrix, and if the first derivative is bounded, then so is the second.
Part
Using the equation of motion for 0 108 and parts and 3 of this lemma, we get, up to leading order:
(ss - 1) = 0ss) - VF (F (F)
0 (s) 1 ( e n² (
(F) (
C' (F (s)
C" (Fiin ())
(131)
(s) - VF (Fim (s), - V/FF C" (Fiin (s), ())0)((s) (s)+
+ (s)
C' (Finn (8) ())
and since:
(8) (F)
Own (8) (N 0) -iin (s + 11 + (
(132)
we obtain the desired result.
Part 5
Using the equation of motion for 0, one can see that:
- - (())))))) (Fiin (s) (8).0)n m )
Onh (s + -VV (Fim (s)
(0 = F (
Fiin + - -
VF (0)) C" (Fin (8),0)(88) +
n )FVF T (g))+ (Fiin ) V (0.)² 7+
Em C²C² (Fin (51) x C' (Fiin (82),
2 3
(8) C¹ FFin ())),)+
(133)
+ (s)
C' (Fim (8) ,1) Ex C212 (Fin (8)) .'),9) .0)+ +
1/ (00)² ((s)*² x2 (s) + (s)) C¹ (Fim (s).0) 1/ F (0o)² x2+
20 (s) + e (s)) C² (Fim (s) 1 v/ (0o)* (7 x (s))
Fin (s + 1) - 0nC" (Fimn (
20 o(s))+ (x)
O (C' (Fim (s)
(g())+ 0 (C (Frin (s)
m(n)
20
+(()())))))())) m(n)
O (C' (Fim (8),9) 0)
31

---

Published as a conference paper at ICLR 2026
m(a) so it can be considered as Sm- In (2) we use the definition of Fiin. Oo and part 1 once again, where gathered all of the components that have only ((s) to get 0()). We then used the asymptotic behavior of all components and took the worst-case scenario obtain Equation 124
where in (1) we use part of the lemma, noting that 0()) I
G WIDE NEURAL NETWORKS ARE WEAKLY CORRELATED PGDML SYSTEMS
G.1 GENERAL IDEA
We start with fully connected neural networks. Although the proof is technically intricate, its underlying concept is straightforward: for the first layer, we observe that all higher correlations exhibit the appropriate asymptotic behavior. We then proceed by induction to show that all layers exhibit the same asymptotic behavior. Consider the second correlation, for instance, which we analyze as follows:
For any layer =1::....... l...., L, defining --1 as the derivatives with respect to parameters from layers 1 to 1 - G.2 employ the equation for fully connected neural networks (399)
1 0 F = - (F()-1) + 0(0) Ve E : (I) = F(L) (z), F(1) (x) =
(134)
to demonstrate that:
6-2 F()) = V/-.) (0(--11) (F(L-1)) + =00)) = (-D) (-D) (00. D (F /(0) Vi-n x (0011 (-00)))))))))))) (1-(-11)) (-00(00))))
(135)
V--) x (0(1-11) (FQ-1) V ((-)
gu (FU-1)) ((-)) (-n) 1(11-1) (FUH-1) Vi-0
Consequently, the contribution to the (-th correlation
10 from this part proportional to:
g(..-1)0" ( C2-11
Ct--1 + gid-11) (FU-1) C(-1)
(136)
Here we have two terms. We can show the right-hand term is small by induction. Showing that the left-hand term is also small is more involved: it requires establishing that, for all hidden layers, the relevant contribution from the first correlation originates from its diagonal terms, i.e. (C)-))
We can now show that, in this term, the left index is identical for both correlations. It follows that, for most indices, the relevant terms are offset by the irrelevant ones, keeping the overall expression small.
For the case where of the derivatives does not belong layers 1 to I 1, explicitly show the corresponding term is negligible, since for most indices it simply resets:
V qaa-i F 6,4
(137)
In the general case of the D-th correlation, while tracing the combinatorial terms arising from different derivative combinations is somewhat intricate, the underlying principle remains the same.
The generalization of this approach to other architectures is discussed in Section G.5
G.2 ASYMPTOTIC BEHAVIOR OF WIDE FCN AT INITIALISATION
Remark G.I. Throughout this paper have considered M| O(M) our way evaluate the size of our random tensors. But here we mainly consider the normalised terms instead:
MII
and
O (M)
(138)
This is because, in practice, what we are interested the average asymptotic behavior of a tensor, and not the accumulative one.
32

---

Published as a conference paper at ICLR 2026
Fully connected neural networks of depth 2 S L E N, characterized by L parameter vectors (the biases ((1), ,..., ((4)) and I. parameter matrices (the weights O(L,L-1), ...., O(1.0)), such as:
1 o........ LL FO0 = 1) (E(1-1)) + +0(0, Vo E X: F(O) (x) p(L) (æ), p(u) (x) =a
(139)
In this representation, F(),, F()) ..... F(I-1), and F(L) constitute the input, inner, and output layers, respectively. The activation function @ is analytical, and all of its derivatives are bounded as described (23).
Remark G.2. Generally, when working with FCNNs we do not apply the activation function to the zeroth layer (the input). However, to make the induction slightly easier, we will simplify the expression by assuming that operates all layers. This makes material difference.
We focus on "wide" neural networks where the depth is fixed. As long as L = (log(n)), we can expect NTK-like behavior for large n, but for simplicity constant in n. We introduce limiting parameter n E N such that the width of all hidden layers satisfies n R1...., -1--1 To simplify our work, replace this assumption by postulating that all hidden layers exhibit the same asymptotic behavior in n, i.., Ri...... RL-1 n. This modification does not affect our theorems and lemmas, it merely establishes a lower bound implied by the
focus on the scenario where L remains
original assumption. Since the sizes of the zeroth and last layers are constant (the input and output dimensions remain fixed in n), we obtain:
71 RL-1 ~ n and no. nL 1.
(140)
Back in the 1960s, it was demonstrated that with Gaussian initialization, we can keep our layers normalised by selecting initial parameters as follows:
T4 - 000000000 (0,11)
Vl = 1, 1,..,, T. 00 11L--11 ~ N
(141)
Despite the specificity of this initialization algorithm, it contradicts the broader spirit of this paper. It not only overly restrictive, but also complicates our work by colliding with our framework for tensor asymptotic behavior. Rather than focusing a particular initialization scheme such as the normal distribution, we identify and use the relevant properties of the distribution.
Definition G.1 (Appropriate Initialization Scheme for Wide Neural Networks). Given wide neural network as defined above, we characterize the distribution for the initial condition @ as appropriate and only if, for every probability arbitrarily close 1, the following properties hold:
1. Different elements of 0 are independent. Moreover, for each layer / = 1,..., L,, the elements g(t.-1) and of g(1) share the same distribution within their respective tensors.
2. 0 symmetric around (implying that all odd moments vanish):
MD N 2N (0 0.
(142)
3. For every layer / ,,, L, all moments of 0 are uniformly normalized:
O11) < + ((00)) 2) < DIO (
N:
Uniformly
VNi (1ot1-111)) s DIO
(143)
where N; m:M4-1 the total number of parameters in the 1-th layer.
where the elemental tensor power is defined by:
VD E N: (M; M!
(144)
The first two conditions ensure that our system is unbiased, while the third condition guarantees that it is not dominated by a disproportionate probabilistic "tail."
We delegate to the reader the verification that Gaussian initialization qualifies as an appropriate initialization.
33

---

Published as a conference paper at ICLR 2026
Remark G.3. Conditions I and can be generalized to hold in the limit of large n, provided this convergence occurs rapidly enough. Nevertheless, any complexities arising from this generalization technical and do not affect our analysis.
For the remainder of this section, will omit the biases from our discussion, they do not add any substantial insights or implications for the points under consideration and do not change any of our results.
Lemma G.1 (Normalization of Layers in Proper Wide Neural Networks). Given a wide neural network, if the initial condition is appropriately set, then all the moments across every layer / 1. 1.... L are well normalized:
Vmu o =
=
(145)
The final parameter that need to normalize in system is the dynamic one-the learning rate, denoted by 7). In an attempt to generalize Gaussian initialization, we adopt the standard normalization for ry:
7
(146)
This condition, coupled with the demand for an appropriate initialization strategy, is sufficient to demonstrate that wide neural networks are exponentially weakly correlated PGDML systems.
In the remainder of this section, we will proceed under the assumption that our parameters are initialized appropriately and that 1 ~ +.
We now use this result find the asymptotic behavior of the layer derivatives:
Lemma G.2 (Asymptotic Behavior of Layer Derivatives). Given established conditions and initialisation, all derivatives are uniformly bounded for each natural number D and each layer I ,....,,.. L. Specifically, we have:
VND o (VXDDF)) S (1) Uniformly
1
(147)
Here, ND = mmm represents the asymptotic behavior of the number of elements in the derivatives.
Proof of Lemma G.1
We approach the proof by induction. Throughout the proof we use Lemma C.7 to establish equivalence
between the asymptotic behavior of the system and its tensorial average (79) The base case, the zeroth layer, satisfies the lemma trivially. Assume inductively that layer satisfies the lemma. We then prove the lemma for the (-th layer for
all (
L:
[
2
L E, (EE,
0 (1) (1,1-1)
Ex ((1--1) 1
11)--11)
To
1 Σ. ((E,,
pll pill
11
11
=2
1 Σ, E, 1
011. (1111-1)
(1111)))
(1-1)
pil (1-11)
ik
((11)-1) (0(1--1) 2) \
FF) (1(--1) 21
1 EE EE E/tt jpt (0(11-1) (t,-1))
(Ff
(11) FR
(-11)
+ 1 Σ,E,
E (00) (0(1-1))) (1 (1)(-1) 21
=1 Enj क
(00
Σ mi
[ (1-11))
m
[0(1-113
2
[1]-1))
= (1)0
(148)
Throughout these equalities, we rely on the premise of a proper initialization. Specifically:
In "I" and "5", we employ the structure of neural networks and the definition of the moment norm.
34

---

Published as a conference paper at ICLR 2026
In "2" and "4", note that F(!-1) depends only on the inner parameters of which are independent of 00.111) This is enabled by the proper initialization ensuring ((1,-11 is uniformly distributed.
In "3", we invoke the fact that different clements of (1,1-1) are independent and symmetric. Hence, for every and j # k:
1 11l 1
- (0 ( O.
(149)
ik
In "6", we apply the induction hypothesis and observe that for proper initialization (G.
VI = I....,L: L: [ou.t-11] = -0
(150)
VTI
By mathematical induction, the lemma holds for all / - I,...,
Using Lemma C.7 again, we obtain O (F") s (1). Since the proof should still hold even after excluding event of sufficiently small probability, conclude that:
0 =
(151)
exactly.
Proof of Lemma G.2
Given w, drawn from another proper initialisation, observe that + is also properly initialised (or at least sub-properly initialised). Hence, assuming we initialise F()) accordingly, we find:
O (Fr)) < (1)
(152)
Vml
Since F() is analytical, we can apply its Taylor expansion around 0 to get:
o
Σ
V x p (0) wD <0())
(153)
Vnu
By continuously rescaling without violating the proper property, we that all components of the expression must be uniformly bounded:
VDEN: 1 Lo (VXD p() (0) (1))) Uniformly.
O VTI
(154)
This is because all terms scale differently with w, meaning the only way to ensure the expression remains bounded under any finite rescaling of w is to bound each term separately and uniformly.
Considering the symmetry of the derivative in its components, and invoking Lemma C.3 we can identify vector of size 1 that maximises it, yielding vector with size equal to its norm. By setting w be this vector and rescaling be proper, we obtain, using Lemma
C.7 that:
YDEN O (VxD (8) nH
1
o (VXD p(0 (0) (0) < 50
Uniformly.
T
n
(155)
Given that:
11
11
(156)
T1
nP_
/ /nmm1
/mmm nD
VND
arrive the desired result.
35

---

Published as a conference paper at ICLR 2026
G.3 REPRESENTATION OF THE NETWORK'S ' LAYERS AS COMPOSITION OF PREVIOUS LAYER COMPONENTS
In this part, between the correlations of the 1-th layer and those of the / - 1-st layer. We then use this relation in the next part to show, by induction, that the correlations are weak. To that end, we define the following useful notation:
the semilinear structure of wide neural networks to establish linear relation
Definition G.2 (Inner and Outer Derivatives). Given layer 1, I, we denote the I-th layer's outer parameters, which include its weights (and biases), follows:
(157)
Meanwhile, the inner parameters are defined as any of the weights (and biases) from layers 1, I and are denoted by:
E 0(-).
(158)
Following the same notation, denote the gradient with respect the outer parameters by V(J). and the gradient with respect to the inner parameters by (--)-- The same applies to the correlations, denoted by (g) and C(-1).
Remark G.4. is important to note that, as F(I-1) depends only on the inner parameters of the I-th layer, the following relationship holds:
= VF(11))
(159)
This notation can be employed express the derivative of the 1-th layer as a combination of derivatives from the (2 1)stt layer.
Lemma G.3 (Representation of the I-th Layer Derivative as Combination the Previous Layer Derivatives). Given fully connected wide neural network specified above, for each layer 1, 1...., L. the D E N-th derivative can be presented as follows:
1. When all derivatives are inner, we have:
(v(-)) XD 1O0 -(x-1))x p(t-))
(160)
2. When one derivative outer and the rest are inner, we have:
x (V(-1) x(D-1) r!l) = dis, 4 x (DD (1-1))
(161)
3. When D 2 2 and 2 < d E N < D derivatives outer, have:
xd
(v(-)) x(D-d)
FO 0.
(162)
Here, ((--) denotes the compound derivative, defined follows for D N:
di +d,=D
FxD Σ
Σ
bld (F()) (vxch F(!) .... x xdda p(i) + comb
(163)
d=1
and for D = 0:
4. :j F = ف,² (F,)
(164)
The "comb" term refers all possible combinations of the derivatives' indices. For instance, if we consider one term of the third derivative:
1(,1-1) (6 (F(-1)) (VFll-1) x vxzp(--))
(165)
then, for any three distinct derivative indices a1, 022 03, there are three unique ways to assign the indices (disregarding irrelevant parts):
Va, (11-1)
X p(l-1) Va, F(1-1)
xx Rapos p(1-1), Vag F(l-1) x xx2 Piaa
((1-1) (166)
36

---

Published as a conference paper at ICLR 2026
While the first combination naturally arises from expression, the "comb" term accounts for the other two.
It should be mentioned that only unique terms are counted, if they arise from differ- ent orderings of the derivatives. Therefore, for another component of the third derivative, VF(t-1)), and distinct 01, a2,
0(--11) (e (FUID)) VF((1) (((-)))F(-1) x VF
F(1-)) Va, F(1-'llaa F(1-1))) Vag F(I-))..
(167)
are identical, and thus should only be counted once.
We can use this result construct the 1-th layer correlations using the correlations from the 1)-st layer:
Lemma G.4 (Representation of the /-th Layer Correlations as Combination of the Previous Layer
Correlations). Under the same conditions in Lemma G.3 have:
c1) = 011-1111 x (0U1--1)
xd
D.a (1-1)
nd I x nao ( () (BULL-1) x(d-1)
D. (1-1)
+ comb
(168)
xd (((--1)
DD-1d (1-1)
+ comb.
or, when showing the indices explicitly (Einstein summation convention):
D.d
((11111) g(l--1)
1(l.-11)
(20-
l0j0
saja
(1-1)
josi.nJa
n Segi ne (1) 0(1--1) ...
(..1-))..000111 g(11-1)
iasa
(Qu-1)
+ comb +
(169)
JoJamnda
g((--1)
gul.-1)
(aD-1.))
iifi
saja
(t-1)
io.i....aa
where the "comb" term includes all index pairings with the zero index, i.e., (io, (2o,;2))..... ..... (fo. ia), and i is defined by:
-00- (1
(170)
The first compound correlation is defined as follows for D E No and d E N:
Σ
D+d
Cabole) (F())
(1) di x ... x CDordds
di + ... + dæ d D1 D DDx =
Comb
d'=1
(171)
where:
(Di!... ... Da (d!!...d))
(172)
Did!
Also, for D No and d = 0:
3D.0 . (
(173)
(1)
The second compound correlation is defined as follows for D E and d e N:
(DD11d)) 1,d
(
= 80al1 (Hade)
(CD)
+ comb,
(174)
io.ii..jaa
jo.h....aa
where the "comb" term is defined as before. For D = O, this compound correlation vanishes.
Remark G.5. For the following lemma and the subsequent section, we assume that D < n. This assumption is permissible even though, in taking limits, the limit in D should technically be taken prior to that in n. This is because higher-order derivatives typically exert a decreasing influence on system behavior, leading to treat them as negligible beyond a certain order.
This assumption is not strictly necessary: we could instead address the combinatorial factors directly. Nevertheless, we adopt it to avoid introducing unnecessary complications into our analysis.
Lemma G.5 (Counting Combinations of Derivatives and Correlations).
37

---

Published as a conference paper at ICLR 2026
1. For the conditions of Lemma G.3 for every di...., da. the number of combinations of the derivative indices is:
D!
(175)
dj!... dz!
and the total number of combinations over all d = 1...., D is the D-th Bell number (which is very close to D!).
2. For the conditions of Lemma G.4 for every di...., de and Di...., Da. the number of combinations of the compound correlations is:
1 d!
D!
(176)
d'! ....... dol Dj!.. Do!
We assume for this lemma that the indices different, as D < n.
Proof- Lemmas
G.3 aadd G.4
We prove the lemma by induction for a general layer 1...., 1, starting with | 1.
The induction base is simple: this is a direct consequence of differentiating the network equation (139). This calculation hinges on the fact that, by definition, the inner derivatives are independent of the outer parameters:
F
V vaguut-1) (F(l-1)) p(l.t- Va (FU-1)) gill-1) (oll) (F(--1) (())((-1))
(FF(F))))
(177)
which yields the base case.
Assume by induction that the lemma holds for some D 1 E N: the inner D-th derivative satisfies:
VXD = (-n x V xx(D=))F(1 =
(-1)
(-- x 1(1-11) DD1 Rd - ald (FUI-1) (Vxd) F(1-1) x .... x E(l-1(
ddg..ddEN
+comb
1111-1)
D-1 -d V x Gld (F(I-1)) (Vid) (( x Xxds F(l
dd=1 EEE
111-1) Ed=
t. Li (F(--)) (V x (xd1 x F(l-1))
comb
(178)
We thus obtain a sum of two different contributions; we analyze each separately. Starting with the first:
+d V x wid (E(-1) ( ( (px 1il-1) x x x ((l-11)) 
Ee 2dd
DD11 1 Adit...de=--1 gid+1) ( ( اللاا 2dii.ddEN
x xx F x(11))x x x 1()-
Ed=1
End dd (l++1) (F((-)) (pxd f(11) x x ((-1)
x V xds++ F((-1))
Ede Edj ddg
td dld (FU-II) (vxd) x (idep((-1) xxdap((-1)
(179)
The second contribution can be rewritten as:
E
di 4d gld (F(L-1) (V Vxd. x xda p(--)() ) deNN
Σ EdinddEN
+dd 0l (F(U-1)) (
xd F(deF(l-1) F(l-1))
(180)
Ed Sdit... dld (FU-1)) (ixdi x F(l-)()
38

---

Published as a conference paper at ICLR 2026
Combining the two sums yields exactly the desired form, which completes the proof of the first case. Lemma follows directly.
Proof- Lemma G.5
Proof of the first part:
The number of ways to partition into distinct sets with di...., da objects is:
(d1 + +.... da)!
D!
(181)
dj!..-da!
dj!-..d.
but our sets are not distinct, so we divide by the appropriate coefficient. If the sets are not identical, then they repeat under different permutations, yielding the factor 1 gives the D-th Bell number.
dt Summing over all possibilities
The second part is analogous.
G.4 WIDE FCNNS ARE WEAKLY CORRELATED PGDML SYSTEMS
Here we provide detailed heuristic argument for why wide neural networks are weakly correlated PGDML systems, described in Lemma G.2
Remark G.6. For this section we assume that the width the last layer, i.e. the L-th layer, exactly nz 1. This does not affect any of our results the system's asymptotic behavior, since L is fixed in n, as discussed in Remark [C.6]
Remark G.7. Throughout this section use Einstein's 's summation convention liberally.
We initiate our exploration of wide neural network correlations (and derivatives) by focusing on the most critical one-the the kernel-e'.
For the final layer L, the kernel norm simply: ||{(,)]| = [C(z)
(182)
Given that n,, = 1, the kernel a scalar.
Leveraging Lemma G.4 layer:
construct the L-th layer kernel from the components of the preceding
C(z) 0.T.L -0 (c(r-1),,, (C)rr))) + no (1)4-1)²
(183)
Applying Lemma G.1 and the Lipschitz property of d. find that the right- hand term satisfies
no (1)6-1)) (1). For the left-hand term, Lemma G.4 again yields:
(C(L-1) = 0
-0/6-110--1-000 2
(CL-2)
+ &yyn (F (1)
(184)
pq
Thus, we obtain an O(1) term and another term depending on the previous layer. Continuing inductively and using the fact that all contributions are symmetric (and hence nonnegative), conclude that the kernel has asymptotic behavior O(1). In combination with (G.2) this shows that our system satisfies the criteria of a PGDML system (D.2)
Let us now consider general final correlation cDdd with D E No and d N. By Lemma C.3 there (L)
exists a vector E SN achieving the norm:
| cD2 v xD
(185)
Applying Lemma G.4 we can express this quantity in terms of correlations from earlier layers. Considering only the first term among the three contributions (the others are treated analogously). and focusing on first correlations, we obtain (up to factor 1 distinguish different index arrangements):
z! while omitting fi since we do not
(ola+D) (F(L-1) (B(L.L-1)) x (B(L--1))* xd ((ca-))) xd ())VE(L-1)) XD xx
(186)
39

---

Published as a conference paper at ICLR 2026
Using 23 and the fact that the layers L - 1 and L are independent at initialization, can discard the ø-terms without changing the asymptotic behavior (we discuss the d! factor below):
(OLL.--I) x(e+1) ((CL--))* xd x (nAVyul-u) XD
(187)
When constructing kernels from the preceding layer, each kernel contains two terms 183) resulting in 2d total terms. This factor does affect the asymptotic behavior, consider only the maximal contributions. We focus on the terms built solely from the diagonal contributions and treat the remaining terms by induction:
8 g (..1-1)(([t11)) ... ( (Bns no (F%
(F(
(Broinn ( (F (14
t0 11
(188)
( (nAVEE XD
..D
Since no (1(l-2)) behavior of at most:
O(1,, after contracting the Kronecker deltas we obtain an asymptotic
(1(:اي-1) a+1 ((1AV)) xD oxx)
(189)
Now, since O (17² (NtPF(L < 0 (1), for D E N the worst-case asymptotic behavior is:
(D d+2
(190)
By proper initialization, this uniformly bounded for all by:
dlO
(191)
Reintroducing the factor I yields:
1
(192)
If D 0, the factor (1t (ntVE!! XD xx disappears, and are left with:
(o(lib-1)) dt1
d+1
(193)
For odd d, symmetry of still yields O
1 vn)
For even d, we obtain:
o
(194)
This explains why our system is /n-weakly and power correlated. Nonetheless, one can verify that this term remains negligible in the time deviation + s0.
Of course, there are many other terms besides the first-derivative ones, but they can be treated similarly.
Assume that for layer l 1,
alld"] (pll-1))
((-1) x ... x Dardy
(195)
(I-
contributes at most:
d d-1
1
0
(196)
Vn
40

---

Published as a conference paper at ICLR 2026
Using Lemma G.5 and replacing ".4' (F(I-1) by d'! (as warranted by (23)) we find that the total contribution is bounded by:
E 2n dg! Di....D
dl (idditude d-1
Drt..Pael or d-1
DI
vn
(197)
d
22++0
similar argument shows that products of correlations exhibit the same behavior at the I-th layer. Consequently, can proceed by induction, as in the kernel case, to show that all correlations scale in the same way, thereby concluding our heuristic proof.
G.5 GENERALIZATION BEYOND FCNNS
G.5.1 TENSOR PROGRAMS
While FCNNS are the prototypical network architecture, numerous other architectures are used in practice, as discussed in Section 4.2 The tensor programs formalism, as detailed in Yang & Littwin (2021), offers a unified language to encapsulate most relevant neural network architectures by viewing them as composites of global linear operations and pointwise nonlinear functions. This formalism encompasses a wide array of architectures, including recurrent neural networks and attention-bane based networks. In their work, they demonstrated that any wide network described within this formalism exhibits linearization.
Our weak-correlation approach naturally aligns with the tensor programs framework, simplifying the proof that such networks not only exhibit linearization, but also are weakly correlated PGDML systems. This comes with additional implications, such as deviations over learning and the influence of network augmentation on the linearization rate.
Our proof for FCNNs can be generalized to any wide network described by this formalism, because, similarly to FCNNs, all such systems exhibit wide semilinear form by definition.
G.5.2 BEYOND TENSOR PROGRAMS
Given the broad generality of the tensor programs formalism, it is challenging to devise linearizing networks that fall outside its scope. However, suggest two network-based architectures that demonstrate linearization and, to our belief, fall outside this formalism.
The first is the FCNN
139 but where each neuron possesses a distinct activation function:
(11-1) 1)) Ø5 (1/4-1) + 0.5h
(198)
The proof of linearization for this system, assuming di satisfies the condition parallels proof for FCNNs.
Not all such systems lie outside the random tensor formalism's purview: if we can represent Φ; function of two distinct inputs-F; and an external input indexed by j E N-then we can write:
Vj= 11.......11 1 oj (F) ())"")) (F)""1))
(199)
However, since ó and its derivatives must remain bounded by some polynomial to fit within the theorems of Yang & Littwin 22021; Yang 2020) for wide neural networks, if the collection {o:} is sufficiently diverse, identifying such may be very challenging even impossible.
more definite (albeit synthetic) example of a linearizing network- based system outside the tensor programs realm is:
n
(x) = Σ (x)+
Σ
A-
(200)
1
initialized by 0 = 0, where a 90" rotation matrix acting on the relevant axis as n - 0o, and f. chosen eigenfunctions of an external kernel.
41

---

Published as a conference paper at ICLR 2026
This system can be viewed as NTK approximation, but with nontrivial second derivative that is orthogonal to the first. Hence, the system still behaves linearly as n - 0o. is also not evident how this system can be derived from the tensor programs framework.
While one might contend that this example artificially contrived, it underscores the existence of weakly correlated, retwork-based systems that not captured by the tensor programs formalism.
Furthermore, in line with our discussion in Section 3.3.3 if we manage to identify effective correla- tions that are beneficial in practice, such systems might find useful applications.
H THE CHICKEN AND THE EGG EXTENDED
In this section, we elaborate on the points made in Section 3.3.3 We begin by explaining why we view derivative correlations as a form of bias in the system.
The simplest way to see the equivalence between weak derivative correlations and an inherent bias in the system to consider wide neural networks. In our demonstration that wide neural networks exhibit weak derivative correlations (Appendix G we assumed that the initial distribution of 0 is uncorrelated in the infinite-width limit. If we introduce correlations into 0, then these correlations contribute to the derivative correlations, so that they no longer vanish. The converse also holds: in the
persistent derivative correlations are equivalent to correlations in the initial distribution large-wide width limit. Such correlations in the initial distribution of 9 indicate inherent bias in the initial hypothesis function, since they imply a predisposition toward specific regions of parameter space. Therefore, weak derivative correlations inherent bias in the initial hypothesis function.
be understood as manifestation of weak
Finite neural networks, by virtue of having finite number of parameters, are restricted to small subset of parameter space. Indeed, they can be viewed as infinite networks in which many parameters are set zero and are not allowed to change during learning. This explains why, even when drawing the initial parameters from an i.i.d. distribution, finite neural networks still exhibit -on-vanishing derivative correlations, which are reduced as the width increases.
The equivalence between weak derivative correlations and inherent bias is also reflected in gradient descent (Equation 7 Inspecting this update, one observes two objects that the optimization process attempts to reduce: the derivative of the cost function, (O)F(0), 5). and the gradient of the hypothesis function, F(O). Minimizing the norm of the first term corresponds to learning the data, since this term is reduced when the hypothesis function fits the target function. In contrast, minimizing [[V)()) corresponds to the system learning its own structure independently of the data, i.e., expressing bias. To enable gradient descent to substantially reduce the second quantity, the higher derivative correlations must share the same asymptotic behavior as the gradient, as seen in Equation for < D. Thus, weak derivative correlations impede the system's 's ability to learn its own structure rather than the data, thereby reducing bias.
Furthermore, we argue that this interpretation helps explain both why linear learning is common and why linear systems are typically outperformed by their nonlinear counterparts. We interpret derivative correlations an inherent bias, and view linear learning as a consequence of our attempt to minimize this bias. However, in some contexts, certain biases can facilitate learning, as exemplified by explicit and implicit regularization. Thus, weak but nonzero derivative correlations can be beneficial, which helps explain why near near-linear learning is often preferable to strictly linear learning. In other words, strictly línear learning takes the weak-correlation principle to an unproductive extreme.
I LIMITATIONS, FURTHER DISSECTION, AND GENERALIZATION
In this section, enumerate the key assumptions that underpin analysis and propose potential extensions beyond these preconditions. Additionally, we identify potential avenues for further related research.
42

---

Published as a conference paper at ICLR 2026
1.1 SECTION 2
Our analysis here did not rely on any hidden or nontrivial assumptions, except for those explicitly stated in the tensor definitions. Our findings are generalizable and applicable any random tensor variable that depends on some limiting parameter E N. Extending our results to any set with a total order is straightforward.
We anticipate that this analytical tool will be beneficial not only for investigating wide neural networks, but also for studying random tensors and variables more broadly, particularly when focusing on their limiting behavior for the reasons delineated in this paper. It upholds several useful algebraic properties C.3 provides well-defined optimal asymptotic bound for any tensor and harmonizes naturally with the notion of "convergence in distribution." Further, owing to its generality, it offers broad applicability. We recommend further exploration of this tool in other settings.
1.2 SECTIONS
1.2.1
ASSUMPTIONS
1. We assume that F, C, and & are analytical in their parameters, i.e., they are smooth and their Taylor series converge.
2. All derivatives of è are bounded as in Equation 23
3. Our analysis is restricted to single- batch stochastic gradient descent, and we assume that the training and testing distributions coincide.
4. We assume that C is convex, i.e., C" is positive definite.
5. Theorems and 3.2 and Corollary 4.1 apply only to PGDML systems, defined in D.2
6. Theorem 3.1 and Corollary 4.1 are valid only for sufficiently small of the same order as the required for effective linearization analyses.
7. Corollary assumes that the first derivative of C decays exponentially, and that the second derivative remains bounded over time for the linear solution.
8. The equivalence shown Theorems 3.1 and 3.2 requires that all derivatives remain fixed. One can, however, state more nuanced equivalence in which the derivatives change substantially, but the network still behaves linearly, provided this change is orthogonal to V F (0)). Nevertheless, since neural networks satisfy simpler conditions, we adhere the version stated above.
1.2.2 GENERALIZATION OF THE ASSUMPTIONS
For Condition [ while we typically deal with smooth analytical functions, non-continuous hypothesis functions common, for example with the ReLU activation in neural networks. If the system can be represented as a linear approximation plus function that is analytical on pieces, and if the non-smooth points form a of measure zero, then the techniques presented herein can be applied.
Regarding the derivative bound Condition 2 this requirement is relatively non-restrictive, especially given that Φ should be analytic and that the bound need only hold on an arbitrarily high-probability set, not necessarily on the entire probability space.
Extending the single-input batch gradient descent case in Condition 3 to other batch schemes, such as multiple-input batches or deterministic single-bate bath GD. is straightforward. This extension amounts repeating our analysis while adjusting the specifics of the optimization algorithm of interest. The generalization to more complex gradient-based algorithms follows similar lines, albeit with additional nuances.
43


Reviews:
[{"review_id": "X3aSeJlemA", "reviewer": "Reviewer_C8ci", "summary": "This paper offers a nice unified explanation for NTK‑style linearization (weak derivative correlations) and a careful asymptotic calculus to make that explanation precise. It provides some helpful intuition for understanding the driver of the lazy regime vs. feature‑learning regimes.", "questions": "1. Main question (feature learning regime). Can you evaluate on feature‑learning regimes (even on small datasets as described in weaknesses), and show that your correlation diagnostics measured at initialisation predict the gap between finite‑width training and NTK?\n2. Learning‑rate scaling predictions. Theorem 3.2 claims reparametrising $\\eta \\mapsto r(n)\\eta$ modulates linearity. Can you add experiments that sweep (r(n)) with width to confirm the predicted $O(r(n)/m(n))$ deviation and the $O((1/\\sqrt{m(n)})^d)$ decay of $C_{D,d}$?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear unifying idea. The paper puts forward weak derivative correlations, small correlations at initialisation between the first and higher‑order parameter derivatives, as the underlying mechanism for NTK‑style linearization, with precise definitions $(C_{D,d}$ and two equivalence theorems (Theorems 3.1 and 3.2) tying correlation decay to linearised dynamics and learning‑rate scaling. This gives a compact, testable lens on “lazy training.”\n- Technical framework. Section 2 builds a random‑tensor asymptotics calculus (subordinate tensor norm + stochastic big‑O, uniform bounds), and proves existence/uniqueness of a tight upper bound.\n- Deviation‑over‑time statement. Corollary 4.1 bounds the SGD deviation $F- F_{\\text{lin}}$ by $O(1/m(n))$ over (finite) time under an exponential NTK‑phase contraction assumption, making the linearisation guarantee feel more operational.\n- Attempt at architectural breadth. By leaning on tensor‑programs, the authors argue many wide architectures satisfy weak correlations (with rates tied to activation derivatives, Equation 22), offering a route to reason about how architectural choices and learning‑rate reparametrization $\\eta \\mapsto r(n)\\eta$ push systems toward or away from linearisation.", "weaknesses": "- All experiments use tiny subsets of MNIST, CIFAR‑10, and Fashion‑MNIST; fully connected nets; MSE loss; and NTK‑normalised learning rate with long training (1,000 epochs). These datasets/architectures in this setup typically do not require rich feature learning and are well known to be close to the lazy/NTK regime already. Thus, showing that the relative discrepancy $|f - f_{\\text{lin}}|$ shrinks with width (Figure 1) and that estimated 2nd/3rd‑order correlation proxies decrease (Figure 2-3) does not validate the central predictive claim in regimes where feature learning matters. That is - nowhere do we empirically validate that networks that are known to feature learn do not also have weak correlations between the first and higher order derivatives. I appreciate that it is hard to compute the derivatives with large amounts of data (where feature learning typically happens) but there are toy models that show feature learning with relatively small datasets (e.g. sparse parity, multi-index, staircase functions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ySsr2FsWS", "reviewer": "Reviewer_ERGz", "summary": "This paper tries to connect the linearity of training wide neural networks with the weak correlations between the first and higher-order derivatives. The authors propose a novel formalism for this purpose by investigating the asymptotic behavior of random tensors, which might be generalized to other cases of machine learning.", "questions": "Please see the first point of the Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The framework of this paper is generally novel. The goal of this paper is clear and straightforward. To this end, the authors are able to develop a sophisticated asymptotic theory for random tensors, which might be of independent interest. This formalism is roughly applied consistently across the proofs, providing a uniform treatment of various architectures.", "weaknesses": "1. I think the presentation is obviously under the bar of ICLR, which sometimes makes the paper challenging to read. Below I list some examples: \n\n   - line 278 \"under the conditions described above\". I'm confused about the exact conditions mentioned here: it seems that there is not any condition above this line. Similar issues exist in Theorem 3.2 line 298.\n\n    - line 279 \"sufficiently small learning rate $\\eta < \\eta_{the}$\". How is this $\\eta_{the}$ obtained? And how small it should be? \n\n    - line 294, I think a \"for example\" should not be stated in a formal Theorem.\n\n    - line 405-407 \"then exists some 0< T, such as for every s =1 ... S, if:\". I'm confused about the statement: \n\n        - $0 < T$ is obvious, why exists? \n\n        - \"such as for every $s$\": Why \"such as\" here? It seems that \"such as\" should not be in a Corollary. \n\n         - Again, how is $\\eta_{cor}$ obtained here? Is it a same one as $\\eta_{the}$ in Theorem 3.2?\n\n2. The authors do not sufficiently justify that existing tools are inadequate for deriving the main results, which makes the motivation of designing a new formalism a bit unclear. In addition, the overwhelming technical machinery in fact raises a barrier for readers to appreciate the methodology, and I'm only able to roughly read the proofs.  \n\n3. The authors make a new and even bold claim, yet the empirical validation is far from sufficient to support this new claim.\n\n4. Indeed, the framework is general. However, the core insight seems to be a deep reformulation of existing knowledge (lazy training, NTK, infinite-width limits). From this perspective, I think the contribution is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZqMXNwqn06", "reviewer": "Reviewer_fCjc", "summary": "This paper proposes a criterion, that of having weak correlations between the first derivative of the model. and the higher derivatives, and shows that it is equivalent to the model being in a linear/NTK regime where one can take a Taylor approximation around initialization. Thus any model with weak correlation can be proven to convergence exponentially fast, and wide DNNs can be thought as just a special case of this result.", "questions": "- It seems that you have missed a paper that is very closely related (it is very similar to Dyer & Gur-Ari 2019): https://arxiv.org/abs/1909.08156 . I wonder how close your condition is to assuming that the higher order terms vanish in the Neural Tangent Hierarchy defined in this paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This is to my knowledge the first equivalent condition to NTK-type linearization, and offers a pretty novel point of vue. Other conditions such as the ratio of the Hessian norm to the gradient norm were more sufficient conditions. This aligns well with an intuition I (and other) had: proving NTK dynamics by looking at a ball around initialization often yields loose rates, instead one has to leverage the fact that the parameters typically move along directions where the NTK moves little.", "weaknesses": "Some parts of the paper are very technical and a bit hard to follow, as well as some parts of the discussion. \n\nAlso the criterion is probably very hard to compute in practice because it involves high-dimensional objects (higher order derivatives). It seems that it would be easier in practice to simply compute how much the NTK moves rather than computing these correlation values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)