A Closer Look at NTK Alignment: Linking Phase Transitions in Deep Image Regression
Authors: Giuseppe Castiglione
Christopher Buckley
Ivor J A Simpson
Venue: NeurIPS 2025 poster ()
Keywords: neural tangent kernel
phase transitions
learning dynamics
Primary area: theory
TL;DR: We mathematically model the evolution of the NTK in a fully deep neural network trained to represent natural images.

Abstract: Deep neural networks trained with gradient descent exhibit varying rates of learning for different patterns. However, the complexity of fitting models to data makes direct elucidation of the dynamics of learned patterns challenging. To circumvent this, many works have opted to characterize phases of learning through summary statistics known as order parameters.   In this work, we propose a unifying framework for constructing order parameters based on the Neural Tangent Kernel (NTK), in which the relationship with the data set is more transparent.  In particular, we derive a local approximation of the NTK for a class of deep regression models (SIRENs) trained to reconstruct natural images.  In so doing, we analytically connect three seemingly distinct phase transitions: the emergence of wave patterns in residuals (a novel observation), loss rate collapse, and NTK alignment.  Our results provide a dynamical perspective on the observed biases of SIRENs, and deep image regression models more generally.

A Closer Look at NTK Alignment: Linking Phase Transitions in Deep Image Regression
Giuseppe Castiglione
Christopher Buckley
School of Engineering and Informatics University Sussex
School of Engineering and Informatics University of Sussex
g castigglioneesssex...... ac.uk
c.1. www.kkeyyssssee..er ac. ac.uk
Ivor Simpson
School Engineering and Informatics
University of Sussex
i. s simpson@suseex ac. uk
Abstract
Deep neural networks trained with gradient descent exhibit varying rates learning for different patterns. However, the complexity of fitting models to data makes direct elucidation of the dynamics of learned patterns challenging. To circumvent this, many works have opted to characterize phases of learning through summary statistics known as order parameters. this work, we propose a unifying frame- work for constructing order parameters based the Neural Tangent Kernel (NTK), in which the relationship with the data set is more transparent. derive a local approximation of the NTK for a class of deep regression models (SIRENs) trained to reconstruct natural images. In so doing, we analytically con- nect three seemingly distinct phase transitions: the emergence of wave patterns in residuals (a novel observation), loss rate collapse. and NTK alignment. Our results provide dynamical perspective the observed biases of SIRENs, and deep image regression models more generally.
particular, we
Introduction
Classical learning theory suggests that models with sufficient capacity specifically, one whose parameters outnumber the training samples tend to "memorise" individual examples rather than learn underlying patterns, leading to poor generalisation [T]. However, while Deep Neural Networks (DNNs) typically ver-parameterised. growing body of research highlights the role of Gradient Descent (GD) in constraining their effective capacity 3 A recurring observation is that GD biases neural networks to prioritise learning simple patterns before more complex ones, resulting in distinct phases of learning 4 5). These phases are characterised by changes in the collective evolution of the network's 's weights, which can be quantified by statisties known as order parameters 66 7 8, (.. Although numerous authors have independently proposed statistics to account for changes in convergence [10 [T] and correspondingly, the memorisation 22 and over-fitting [13] of complex/noisy patterns- their interrelationships remain under-explored. More significantly, these existing approaches provide limited insight into the actual content being learned during each phase and consequently, which patterns models systematically struggle to learn. Addressing these gaps is essential developing unified understanding of learning dynamics in DNNs.
major obstacle in understanding this inductive bias lies in the inherent complexity of GD itself. While conceptually GD can be viewed as function mapping from the dataset, hyperparameters, and initial weights to the final learned weights, in practice, the thousands iterations through high-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

dimensional parameter space obscure the relationship between order parameters and the underlying dataset characteristics. recent years, the Neural Tangent Kernel (NTK) [4] has emerged as an alternative perspective on the dynamics of learning, recasting them in terms of the evolution of pointwise errors. Critically, in phenomenon known as Neural Tangent Kernel Alignment (NTKA), the eigenspectra of the NTK undergo a sudden transition of their own, spontaneously aligning with the class structure of the dataset without direct supervision. NTKA has been widely documented and is suggested counterparts [5 16 17 18 19 20). However, despite repeated empirical demonstrations of NTKA, theoretical exploration of the phenomenon has been largely restricted to classification problems with toy models, such as two-layer neural networks 21 22 and deep linear networks [22].
reason why real-world DNNs often outperform their infinite-wide limit
In this work, we move beyond these simple classification models and study NTKA in a considerably
more complex setting: deep image regression using multi-layer SIRENS 23 These Implicit Neural Representations (INRs) learn mappings from IR² + R. representing images as continuous functions, and find increasing application in tasks such as super-resolution. Despite the low input dimensionality, the depth and non-linear (sinusoidal) activations of these networks pose significant analytical challenges, exceeding the complexity of previously studied models. However, in addition to facilitating visualisation, this low-dimensionality permits us to leverage insights from computer vision to introspect the learning process. Our study is structured around three primary contributions:
1. We derive novel approximations for the local structure of the SIREN NTK, allowing us to approximate: the principal eigenvector (3.3); order parameters such as the minimum value of the Cosine NTK (3.4); and the correlation lengthscale (3.2). In so doing, we theoretically establish connections between the onset of NTKA and other dynamical phase transitions.
2. We identify a novel learning phase in deep image regression, characterized by the appearance of diffusion-like like wavecrests in the residuals, and relate this behaviour to the evolution of the NTK.
3. We experimentally verify that the critical points for these different phase transitions cluster in time. We also empirically investigate the impact of image complexity and SIREN hyperparameters on the occurrence and timing of phase transitions, and provide evidence that NTK alignment in image regression tasks occurs in response difficulties modelling edges.
2 Preliminaries
In this work, we consider 2D grayscale images, where pixel coordinates and their intensity form dataset D of N samples indexed with (z,, (x;)), where x; I²2 and 1 R² + R. On
this dataset, we fit SIREN models f(x; 0) of depth Nr, defined recursively by: h(()) x;; h()
sin wo (W 6(9); fz; W + b(M)) Here (1)) denotes the output of the learnable parameters, and wo is a bandwidth
I-th layer, 0 (W10, 6(0) I. is the hyperparameter. we is generally chosen to ensure the sin function spans multiple periods (and thus frequencies) over the inputs. In the continuum limit, we assume the data distributed uniformly Patta (z) = DD -1. We identify two fields: the local residual field r(r:; #(t) = [(x) f(;; 0(t)) and gradient field V (e.((;;0()) @(t)) Dynamics are induced by gradient flow 0 V the mean square error: (()) 2vol(D) J dx r(x; 0)2. Through the chain rule, the residuals evolve as follows:
f(z; 0(t)) 0(t)) 0
= -oll(D) / da' r(a')Ver(z;((()) Vor(z';0(t))
(2)
= -- da' r(x')
1 Vol(D) V Vos(x; Vof(z'
KxTx(
In the last line, we defined the NTK. Equation 3 is a linear dynamical system with a time-varying kernel. The eigenvectors vn 1) represent distinct normal modes of the dataset, each learning at rate governed by its associated eigenvalue (t). This framework formalizes the intuitive notion that
neural networks learn different patterns at different speeds.

---

0.313
(a)
(c)
55
92
116
55
92
116
10
0.00
131
155
230
155
230
(b)
(d)
AUCI[Hal
-0.15
Leval
peak
10
10
10
10
Raw image
Edges
Epoch
Figure 1: A Single Phase Transition Through Three Lenses: (a) The magnitude of the residuals over the training process. Near the critical point see the formation of wavecrests. (b) Evolution of evaluation loss rate during training, which reaches a peak at the critical point. (c) Evolution of the principle eigenvector of the NTK, which reveals sudden shift from disorder to structure. (d) Quantification of NTKA in terms of alignment between edges and the principal eigenvector.
Finally, for notational brevity, we will drop the explicit dependence on 0, and write x' = + u. We also define a kernel closely related to the NTK, the Cos NTK:
Vol(x) Vof(a +
(2TK(E,I+u)) Vol(D) [[ ([a u)ll
(4)
Deriving Order Parameters from the NTK
We illustrate the different phases of learning in Figure [ we train a (five-layer, 256-unit wide, wh = 60) SIREN model on a 128 x 128 grayscale image using full-batch GD (learning rate=10-$), We evaluate the model super-resolution at 256 x 256. We examine the learning dynamics through three different lenses, each revealing a sudden shift These shifts are quantitatively identified using statistics, known order parameters. We demonstrate below how order paramters for each transition may be related to a common set of features, which control the local NTK structure. The three lenses are follows:
Spatial Distribution of Residuals: Early in training, the loss decreases uniformly over the dataset (Drift Phase). However, at critical point, observe the formation of "wave-crests" corresponding regions low-loss, which propagate across the dataset (Diffusion Phase). To best of our knowledge,
the first to report this behaviour in SIREN models. We attribute this behaviour (in sec. 3.1 to changes the equal-time correlation functions of the gradient field o whose parameters we derive in Section 3.2
Principal Eigenvectors of the NTK: The principal eigenvector vp is initially static and appears highly-disordered (Disordered Phase). However, at critical point, vo experiences a brief, sudden shift, in which it aligns with the edges of the image (Aligned Phase). Although NTKA has previously been studied in the context of classification problems (24 25] 26 27 there are additional subtleties to consider for regression task such as INR training. To this end, introduce metric, AUl([vol, VI) in Section 3.3 to identify when alignment occurs. We also derive an approximation of vg based on the local structure of the NTK, as outlined in Sections and
Training Curve Analysis: There a rapid shift in the slope of the training curve, which we call the loss rate Í.. Learning is initially fast (high î.), but after critical point, slows abruptly (low i.). Several works have studied this transition using order parameters, but in this work, we focus

---

epoch
Ajre()
Bqeaa))
298
0.8
9478
0.00
Nondi
Blend)
Grend)
0.4
08.0
0.4 0.8
Figure 2: Spatiotemporal Evolution of the Cosine NTK (CNTR) Left: Global correlation function of the CNTK different epochs. Dashed lines show fitted Gaussian approximation from equation equation 6 and error bars show variance across datasets. Right: Visualization of the CNTK around three points [A, C} for small separations, at the beginning and end of training.
In Section 3.4 derive an approximation of this parameter based on the local structure of the NTK outlined in Section 3.2
the concept of gradient confusion, described in 13
12].
3.1 Correlation Functions and the Onset Diffusion
The form of equation is reminiscent of the linear response functions in statistical field theory 28 29 to find the rate of change of the residual field point x. the kernel aggregates information about the residual at points x u. To quantify the range of these interactions, we examine the local, equal-time correlation functions for the gradients Vaf(r) separated by distance e:
A(r, e) = Ex [of(x) Vof (x+ + + = E.g [KTK + cε,)]
(5)
Here, è denotes unit vector in direction ø. Similarly, the global, equal-time correlation function is given by k(c) E, [k(z, c)]. We may define similar quantities for the CNTE, which we denote by c(z, E) and c(e). We expect the range of these interactions to be short, as INRs are often carefully designed ensure diagonally dominant NTK 30 31 32). To verify this, we group pairs of datapoints based on their distance, and then compute the mean CNTK value. An example of this empirical correlation function is shown in Figure 2 The observed structure motivates the following proposition, which we examine qualitatively in Figure [nd and validate numerically in Appendix F.1
Proposition 3.1. For SIREN models, the equal-time correlation functions of the NTK are well- approximated by Gaussians of the form:
c(E) (1 Coole" -2/22 + Coos
(6)
k(x,e) [[
Coo (oo(e)) -2 (x)]
Our approximation introduces two important order parameters: the first, the correlation length- Ecorr. controls the rate at which correlations decay with distance, defining the range of interactions. The second, the asymptotic value Coo, describes the interactions between points separations e much greater than Suspending consideration of anisotropic effects, the above correlation functions are consistent with the following form for the full NTK:
Proposition 3.2. Gaussian Approximation of the SIREN NTK. The NTK may be be approximated as a Gaussian kernel with spatially- amplitude o[ (x)),,), bandwidth {(x)). and asymptotic value (x) 12 C (z):
KGanss (I, + w. V (z)()))) (x)( exp((llu1) (VVo))++////////(////\<>>>>>)
(8)
Dynamically, we see from the left of Figure 2 that both & and Coo evolve during training, and we shall demonstrate that changes in these values account for the onset of diffusion.
Theorem 3.1. Diffusive Evolution of the Residuals: Let the mean residual be denoted by p. Ex [r]. and let Ko denote the mean contribution to the K KNK at large distances ||u]| >> {(x). Then, as
μ, K5 + assuming KNTK KGauss the residuals approximately evolve under the following diffusion equation:
d ra,,t)
(9)

---

Proof Sketch. (Full details in Appendix A.2). background in determining the evolution of r(z) in equation
As n, 30
0, local interactions dominate the
Thus we may approximate:
dt - dr
I do r(x + (x, 2s / r(o + +.) +) [a]())) exp(pelpppp((0000 (10)
The exponeniiallydeecyii NTK will suppress all contributions r(x) from residuals r(z + u) for which |u|| > f(r). When {(x) is small, we may perform Taylor expansion around the point x::
r(r + 0; ~ r(:; 0) + u Var(z; 0) + (2r(2;)) O)u
(11)
Inserting this into equation 10 the full integral may be solved using Gaussian integration. We obtain:
d ग
(12)
which resembles standard diffusion equation
3.2 Beyond the Isotropic Gaussian Approximation
Though the Gaussian approximation of the NTK can account for the appearance of the diffusion wavecrests, fails to capture other empirical properties. Namely, whereas the true NTK anisotropic (see Figure 2 and can contain negative entries, the Gaussian kernel is isotropic, and satisfies K Kanus (r, x u. > O for all u. To overcome these limitations, we present the following refinement:
Theorem 3.2. Cauchy Approzimation of the SIREN NTK: For small separations u, the Cosine NIK locally takes the form of a Cauchy Distribution with structure parameters Og, D2, Hz:
2a + u Dr
ONTK uu) Rd
(13)
20 + D2 + u Hgu
These parameters are obtained from the model gradients as follows:
Q₂ = [[Vef())ll: D. = Vel\v)())) H. (VoVof(
(14)
Proof Sketch. (Full details Appendix A.3). Via the Law of Cosines, the CNTK satisfies:
[af(z>]| + Vof(zz + - -of(=))
CNTK(E, u)
(15)
2 (o](>) Vol(= u)]l
The result then follows by Taylor expanding the numerator and denominator to second order in
benefit of this new approximation is that it can be used to predict the correlation length-scale: Corollary 3.2.1. The correlation length-scale for the NTK about a point x may be constructed from its local structure parameters agg Day Hy and asymptotic value Coc (x)) follows:
(x) 1 coo())
az
(1(1-c00(1)
1 Coo
2D D.HH'
(2)) x
+
1 coo
(16)
dee H
det H
Proof Sketch. (Full details in Appendix A.4p. Note that the levels sets of equation 13 correspoon to ellipses. For a given value c, the area of the level set can be shown to be:
Aellipee (x; c) = 2n 1 11-c)
(²=²
det H + (1--0)
DJ H
(17)
det H
The correlation lengthscale is then approximated as {(x)= v Mattipse(1r, (xr R,/ where we choose 1/2 + / to account for the asymptotic value of the CNTK.
5

---

3.3 Order Parameters for the Onset of NTK Alignment
In the classification problems typically studied in the NTKA literature, the principle eigenvector vg (a(2) is seen to learn class- separatin boundaries [24, [25]. Similarly, for our 2D image reconstruction task, we see the NTK learns information about the distribution of edges in the image (Figure 3. To quantify this alignment, we use a Canny Edge Detector [3 to estimate connected image edges. We then quantify the utility of (t)(a) in predicting edges in terms of average recall, as measured by the area under the Receiver Operating Characteristic Curve (ROC AUC). We denote this measure AUC(| VI), and it has the advantage of being insensitive to monotonic transformations of |vol- This invariance is beneficial in two respects: (I) as reliability measure for binary predictor (edges), obviates the need to specify a threshold, facilitating comparisons across datasets (see Section 4.1 and (2) empirically, it saturates during training, facilitating the the identification of critical point.
Another hallmark of NTKA is early anisotropic growth of the NTK spectrum [5] as the NTK becomes stretched along small number of directions correlated with the task. This is especially the case for the principal eigenvalue Åo. which grows orders of magnitude larger than the next leading eigenvalue. In Section 4.1
we will demonstrate empirically that this is also true for INRs.
The divergence of Ao enables particularly simple approximation of the princpal eigenvector Up: Corollary 3.2.2. The principal eigenvector vo(x) of the NTK admits the following approximation in terms of the local asymptotic value Coo (x) and the local correlation lengthscale E(x)):
vo(x) x a Goc (o))o() + 2Tt
coo((())
cac (x))
(18)
Proof. Because the principal eigenvalue is so dominant, KNTK becomes effectively low-rank, and so power iterations converge quickly. Thus, choosing a vector of ones e = 1 as our initial vector, we expect K1/1J1 1 1 to have strong cosine alignment with the principal eigenvalue. In the continuum limit, this is simply given by: K1/N + E. EE[KK(, + u)] E. x + u)l ||-|| c]]
(19)
de k(z, P(),))
(20)
Here, P(x,e) e) denotes the density of points that are located distance e from the point æ, and Emaz is an upper bound on the distance that we assume is much greater than Eoorr. Close to this a P(r,
t)
grows like 2 Thus, leveraging equations] 7 and [4
we have:
vo(x)= = 2
de (ee(co coo(() + (1- coo (z)) - (x)
(21)
a Coo ( + (1) coo - -moo (
(22)
a² coo (z) + (x)(1 coo (z))
We evaluate the fidelity of this approximation in Appendix F. As we approach the phase transition, the asymptotic values tend towards 0, and the second term dominates. Considering the approximation for the correlation length-scale & in Corollary 3.2.1 we note that Un(z) grows as V[ (x) This implies particular sensitivity to pixels in regions with substantial high-frequency information, such as edges and corners. As natural images tend to be piecewise smooth, pixels on boundaries have the strongest spatial gradients, and are therefore the greatest source of information, being poorly compressible due to the lack of smoothness, and accordingly disagreement in parameter gradients. Given the inability of models to accurately describe sharp discontinuities these edge pixels are considered influential datapoints, which accounts for their prominence within the principal eigenvector. We discuss other parallels between the moments of the NTK and traditional corners detection algorithms in Appendix E In particular,
introduce another order parameter, termed MAG-Ma (Magnitude of the A Average Gradient of the Log Gradient-Fiede Magnitudes). to monitor the breakdown of stationarity (ie local translation invariance) of the NTK. It is obtained [Dz/ ?
'The true form of P(x, <) is complicated and varies from point to point, due to edge effects. However, these effects suppressed as P(r, e) only appears when multiplied the Gaussian kg-

---

3.4 Order Parameters for the Loss Rate Collapse
and related works, the authors examine the role of gradient alignment statistics in determining the speed of learning under stochastic gradient descent. They note the emergence of negative alignments between batches correlates with a reduction learning speed. Intuitively, when sample gradients become negatively aligned, the sum of the gradients approaches zero, resulting in diminished learning signal. The minimum alignment is simply the minimum value of the CNTK. which we may obtain explicitly from Theorem 3.2 as follows:
In
Corollary 3.2.3. The minimum value of the ONTK admits the following approximation in terms of the local structure parameters oz. Dz. Hz:
DJ D D2 min CNTK(z,
(23)
HT D2 Sa?
Proof Sketch. (Full details in Appendix A.5 Setting 0,, (xr, + u) 0 yields two solutions:
u = 0, corresponding to the maximum (T), and another corresponding to the minimum. C
The min CNTK is then simply the minimum of 23 across the whole dataset.
4 Experimental Results
Setup: We fit SIREN models to a set of thirty 64 x 64 downsampled images and evaluate the MSE
Leval on a super-resolution task (at 256 x 256). We used five random seeds and also varied the width, depth and bandwidth wo (ranges are given in Appendix [B.2). We compute the eigenspectra of the NTK using Randomized SVD 34. In addition to the order parameters described in Section [3] we examine three NTK-based order parameters from the literature: (1) The principal eigenvalue Ào of the NTK, which diverges at the critical point; (2) The variance of the gradients of which peak during the Fast-Slow learning phases [10], and which may be connected (see Appendix A.6) to the trace of the NTK: (3) The Centred Kernel Alignment (CKA) between the NTK and a task kernel Ky. For INR regression, we use Ky (z, z + u) exp(-50) - [(z + =))l²). The similarity between kernels is measured using the normalized Willbert-Schmin Information Criterion (HSIC), as in 22
Full experimental details may be found in Appendix B.2
4.1 Examining the Distribution of Critical Points
The left-hand side of Figure illustrates our procedure for identifying critical points in given trial. We simple peak detector to identify the region of interest for the loss rate Levat and the gradient variance de, using the FWHM to define a confidence region. For the min CNTK, we look for zero-crossings, with a confidence region constructed from the cumulative variance. For every other order parameter, we fit sigmoid, where the inflection point marks the critical point, and the slope defines the confidence region (full details in Appendix B.2). The right side of Figure [ demonstrates how frequently these confidence regions overlap across our experimental sweep"] Remarkably, the phase transitions described by the order parameters despite being derived to measure different phenomenon in the literature- consistently occur at the same time during training.
Critical points cluster around run-specific times:
Hyperparameters alter the timing of run-specific transitions: in Table I we observe that both depth and bandwidth wp have critical roles in controlling the shift in the loss rate Levat- Generally, increasing depth and decreasing wi result earlier transition times lartt However, these changes have opposite effects on the model performance: for fixed wp. deeper models to converge to better (lower T.evat) solutions faster. However, it seems that lower values of wi cause models to converge prematurely. This may be deduced by studying the correlation between the final residuals (details in Appendix B.2). For equivalent depth (and therefore, equivalent traditional capacity), models with lower wo exhibit more correlations in the residuals. This is indicative of remaining structure in the residuals, and can be interpreted as evidence of under-fitting
II computing the coincidence matrix, we exclude trials where the detection of the critical point was unreliable. In Appendix C we comment how image properties impact the detection rate.
7

---

0.00
Low
-0.15
0.1
CKA(Ky, KaTT)
minCwix
-0.4
0.98
1.00
0.88
0.98
0.97
0.78
0.91
0.83
Ao
0.2
CKA(K,, Kimc)
0.88
0.86
1.00
0.85
0.95
0.58
0.72
0.76
AUC VI)
0.99
0.98
0.85
1.00
0.99
0.57
0.89
0.99
AUCI[val, V1)
0.5
1200
Aa
0.60
0.78
0.59
0.57
0.79
1.00
0.10
0.47
minCNnk
0.89
0.91
0.72
0.98
0.97
0.10
1.00
3.71
urk-
0.025
MAG-Ma
0.000
Ao
AUC([vall 91
Lowa
06
minCai
u.K.
MAG-Ma
0.002
M.K.
C.A(Ky,
10
10'
Epoch
Figure 3: Alignment of Order Parameters. Left: Order parameter evolution and critical points during training of a SIREN model on the astro image. The red vertical lines denote the location of the critical points, and the green vertical lines denote confidence regions. Right: Heatmap showing the frequency of intersections between the confidence regions. Additional figures in Appendix G.1
Table 1: Variation in model performance with hyperparamters: Comparing the dependence of transition times (ferit), super-resolution performance (logıo Ieva)), and resdual correlation on depth and bandwidth. Also featuring expected correlation lengthscale (B(Soorr (t))) and correlation between log V [a; 0) and (x)]( Values averaged over the same sweep defined in Section 4.1
depth
E[Ecorr (1))
AUC([ao):1) /)
Grad. Corr.
log10 lerit
1o810 Leval
Res. Corr.
3/90
0.04 + 0.00
0.59 + 0.05
0.06 + 0.10
2.83 + 0.19
-2.01
0.31
0.35 + 0.06
3/60
0.06 0.00
0.60 0.05
0.16 + 0.11
2.84 + 0.12
-2.00
0.32
0.40 + 0.07
3/30
0.11 + 0.00
0.60 0.05
0.27 + 0.08
2.44 + 0.25
-1.92
0.44 + 0.07
3/15
0.18 0.01
0.60 0.05
0.26 0.07
2.03 = 0.34
-1.83 0.29
0.48 + 0.07
4/90
0.04 0.00
0.67 + 0.07
0.23+00.11
2.66 - 0.17
-2.02
0.32
0.35 + 0.06
4/60
0.06 0.00
0.69 + 0.07
0.12
2.61 0.16
-2.04 0.33
0.39 # 0.07
4/30
0.10
0.71 + 0.08
0.41
2.22 0.27
-1.99 0.31
0.41 + 0.07
4/15
0.16 0.01
0.68 + 0.09
0.55
+ 0.35
-194 + 0.30
0.43 + 0.07
5/90
0.03 0.00
0.68 0.07
0.24+0.14 0.14
2.54 + 0.18
0.32 -2.01
0.34 4: 0.06
5/60
0.05 + 0.00
0.71 + 0.07
0.30+0.11
2.42 + 0.17
-2.05
0.39 + 0.07
5/30
0.09 0.00
0.73 0.08
0.39+0.10 0.12
2.15 + 0.12
-2.02 0.32
0.40 + 0.07
5/15
0.15 0.01
0.72 0.08
0.52
1.85 - 0.27
-1.98 0.31
0.41 + 0.07
4.2
Influence of Hyperparameters on Edge Alignment
In the previous section, for fixed depth, we demonstrated that lower wi correlates with (1) earlier phase transitions, (2) higher validation loss, and (3) correlated residuals. Together, these observations suggest that models with lower w/ converge prematurely, underutilizing their capacity. A natural question arises: which patterns do these models struggle to capture? Given the observed concurrence of loss rate collapse and NTK alignment, we analyse the NTK eigenspectrum to gain some insight.
In Figure we train four SIRENs on the sax dataset with (wn, depth) E [15, 60) {3.5). We visualise both the log magnitudes of the parameter gradients, log [>) and the principal eigenvector, to (z),, at the end of training. Additional Figures may be found in Appendix C.2

---

Alllgl/Vafll. -|0.1]]==0.88
Wafll. ||0.1]0=0.422
AUC[|val. 90)=0.801
AUC([Ve] Vel. V1)=0.722
0.050
0.030
depth=4, 60
depth-4, 6A-000
depth=4, 4J-330
deptn=4, w0-990
pllogilVefll. 11, |10.80)0.5882
pllog] Vofll. [[9,1]])=04480
AUC[[val, VI)=0.807
AUC[[Va]: VIJ=0.740
20
0.050
0.030
10
0.015
depth=5, M6
depth-5, 6c-90
depth=5, 46-30
deptn=5, 69y-90
Edge Alignment: Left (magma colormap): The norm of the parameter gradients Vo [x))] for (wo, depth) E (15, 60} x {3,5), 5J. labelled with the Pearson correlation p between the log of the norm and the spatial gradient V. of the target image. Right (viridis colormap): visualizing the principal eigenvector uo of the NTK for the same models, labelled with the edge alignment score AUC([va): VI). More images in Appendix C.2
Figure 4: Effect of Hyperparameters
Generally, we observe that for low wp, log [o] 0)]l swells and concentrates near image edges, becoming sparser, and more correlated with the spatial gradient magnitudes [ (aggregated statistics may be seen Table D This edge prominence in vo matches expectations, as per Corollary 3.2.2 (z) V ( (x))l"). Overall, while edge alignment is seen across most settings, it is especially prominent deeper models with lower values wn. This indicates prioritization of these patterns by the NTK, and correspondingly, the patterns the model is most invested in.
Discussion
To explain the impact wp. in Table we track the expected correlation lengthscale E( (t)] the course of training. In Figure 17 of the Supplementary Materials, also examine the variance in (oorr(0) with w/ in all models. Consistently, we observe that lower wo is associated with larger values of the correlation lengthscale. This implies that the NTK integrates information across larger neighborhoods. implicitly averaging over high-frequency features. Consequently, we expect these SIRENS to struggle when modeling tigh-frequency patterns, which is consistent with other observations in the literature 32 Following the discussion in Section 3.3 we expect that the difficulty of modeling edges responsible for the swelling of the parameter gradients [x))f
Intriguingly, we observe that the principal eigenvector becomes sparser as we increase depth, leading to stronger edge alignment, seen in Table I Yet. this sparsification is associated with completely different generalization behavior: models achieve lower validation loss, with less correlated residuals, increase depth. We hypothesise different mechanism underlies this sparsification in compari- son with wy. Figure digurs/ademosstratts increasing Wp increases the sensitivity of the gradient magnitudes (and hence the principal eigenvector) noise in the images. For DNNs, gradient magnitudes decom- pose into sum across layers, namely || ||Vof fl2 deeph (/a(() fll2. In effect, preference is given to points which are consistently confusing across layers, thus mitigating the effects of noise.
Related Work
Fast and Slow Phases Neural Network Training: The literature highlights a dynamical phase transition in DNN training between fast to slow learning regimes [10]. The initial fast phase is characterized by large gradient norms and low fluctuations, yielding rapid loss reduction via broad agreement across examples. It is followed by slow phase in which gradient fluctuations dominate

---

and progress decelerates. This transition can be tracked by order parameters including gradient signal-to-noise [TO], gradient confusion, [T3 [2] and correlation lengthscales [33. Furthermore, it reflects move from learning simple, shared patterns to fitting complex, idiosyncratic ones []] and thus offers a window into feature learning. Related work studies the representation dynamics in ReLU networks [5]; here, we study them via the NTK in SIREN models.
Neural Tangent Kernels for Implicit Neural Representations: Previous research has investigated the inductive biases of INRs using the Neural Tangent Kernel (NTK), focusing on aspects such as spectral properties (36] and dependencies on uniformly sampled data [3]. Furthermore, studies by [37] and [38] have analyzed the eigenfunctions of the empirical NTK to elucidate the approximation capabilities of INRs. These investigations, however, primarily examine static properties of the NTK at initialization, which do not account for feature learning dynamics. This is known to be a poor approximation [39] In contrast, our work concentrates on the evolution of the NTK, aiming to deepen our understanding of how INRs learn to model images.
Neural Tangent Kernel Alignment In practical settings, recent studies have shown that during training, the NTK dynamically aligns with limited number of task-relevant directions [40 41
26 27 Concurrently, the eigenfunction level, the modes increasingly reflect salient features of the dataset, such as class-separating boundaries [24] 25), and Fourier frequencies [25] The widespread occurrence and influence of kernel alignment suggest its critical role DNN feature learning, contributing to the superior performance of DNNs over models based on infinite- width NTKs [26). Direct optimisation of aligmment measures has even been suggested as one way to enhance the convergence of GD and generalization of models [2 43 That said, theoretical investigation into spontaneous NTKA often focus on shallow networks 22], toy models 226 25). and deep linear networks [22]. In contrast, the INRs we study are deep (3-6 layers), nonlinear models that see frequent use in Computer Vision problems.
24
21
25
7 Conclusion
We have developed new formulations that leverage the NTK to characterise the dynamics of feature learning in deep image regression models (SIRENs). By analytically deriving approximations for the local structure of SIREN NTKs using Gaussian and Cauchy distributions we were able to obtain approximate expressions for the correlation lengthscale, the minimum value of the CNTK. and the principal eigenvector. We related these expressions order parameters for three phase transitions identified in different dynamical perspectives on learning: the appearance of diffusion wave-crests in residual evolution (first identified in this paper): the collapse of the loss rate; the onset of NTK alignment. We argued, based on these derivations and empirical demonstrations that critical points cluster in time, that these distinct phase transitions share common, underlying mechanism.
The following picture emerges from our analysis: as long range correlations between gradients decay, residuals only interact with their immediate neighbours (onset of diffusion), leading to increased gradient variance (loss rate collapse) and translational symmetry breaking. In parallel, the growth of the principal eigenvalue the NTK leads the principal eigenvector memorize the distribution of influential points, as measured by accumulating gradients. In images, one influential class of points edges, leading to their prominence in the principal eigenvector (NTK alignment).
In this study, we focused on SIREN models trained on a 2D super-resolution task using full-batch gradient descent. However, SIRENS are used in a variety of inverse problems, and it remains to be seen whether our observations extend to these settings. Future work may also explore the impact of different optimizers, such as ADAM [44), which adaptively adjusts learning rates and may influence the stability and divergence of the principal eigenvalue- a key factor in our study of NTK alignment.
This work has demonstrated that the NTK provides a rich theoretical tool for deriving and relating order parameters to understand training dynamics. We provide new methodology to rigorously study the influence of inductive biases, such as model architectures and hyper- parameters, on the underlying learning process and may have practical utility in diagnosing causes of poor learning outcomes.
Acknowledgements
Supported by the University of Sussex Ae. doctoral scholarship, funded by the Leverhulme Trust.
10

---

References
[1] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, edition, 2007.
[2] Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. In Neural Information Processing Systems, 2019.
[3] Maxwell Nye and Andrew M. Saxe. Are efficient deep representations learaale?? ArXiv, abs/1 807 107.06699, 2018.
[4] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. (arXiv:1006.05394) 2017.
[5] Xiao Zhang and Dongrui Wu, Rethink the connections among generalization, memorization, and the spectral bias of dnns. In International Joint Conference on Artificial Intelligence, 2020.
[6] Yu Feng and Yuhai Tu. Phases learning dynamics in artificial neural networks the absence or presence of mislabeled data. Machine Learning: Science and Technology, 2(4))043001, jul
2021.
[7] Cory Stephenson and Tyler Lee. When and how epochwise double descent happens. ArXiv, abs/2108. 12006, 2021.
[8] Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning, ArXiv, Abs/2205.10343, 2022.
[9] Liu Ziyin and Masakuni Ueda. Exact phase transitions in deep learning, ArXiv, abs/2205.12510, 2022.
[10] Ravid Shwartz-ZZ Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. (arXiv:1703.000) 2017.
[11] Karthik A. Sankararaman, Soham De. Zheng Xu, W. Ronny Huang, and Tom Goldstein. The impact of neural network overparameterization on gradient confusion and stochastic gradient descent. (arXiv:1904.0m 904606663), 2020.
[12] Yu Feng and Yuhai Tu. Phases learning dynamics in artificial neural networks in the absence presence of mislabeled data. 2(4):043001, 2021. Publisher: IOP Publishing.
[13] Stanislav Fort, Pawet Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness:
A new perspective generalization in neural networks. (arXiv: 1,.44911),.0 2020.
[14] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. CoRR, abs/1 806.07572, 2018.
[15] Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. 1iXXi::909.00989,, 2019.
[16] Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hierarchy. (arXiv:1909 1ete:099990000 156), 2019.
[17] Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks. (arXiv:1910.08013) 2020.
[18] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. (1rX1v:1812.07956),, 2020.
[19] Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. (arXiv:2007.15801,, 2020.
11

---

[20] Mariia Seleznova and Gitta Kutyniok. Neural tangent kernel beyond the -nfintte-width limit: Effects of depth and initialization. (arXiv:2202.00553)) 2022.
[21] Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural nets. 202((4)04
[22] Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. DIcXi:221110003320 2021.
[23] Vincent Sizzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
[24] Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. (arXiv:1910.08720) 2020.
[25] Aristide Baratin, Thomas George, César Laurent. R. Devon Hjelm, Guillaume Lajoie, Pas- cal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. (arXiv:2008.00938) 2021.
[26] Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. (arXiv:2 105. 100.14311134 2022.
[27] Abdulkadir Canatar and Cengiz Pehlevan. A kernel analysis of feature learning in deep neural networks. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1-8. IEEE.
[28] James P. Sethna. Statistical mechanics: Entropy, order parameters, and complexity. Oxford University Press, 2021.
[29] Joseph W Goodman. Statistical optics. New York, Wiley-Interscience, 1985, 567 p., 1, 1985.
[30] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. (arXiv:0006.10739), 2020.
[31] Filipe de Avila Belbute-Peres and J. Zico Kolter. Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth, 2022.
[32] Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions, 2023.
[33] John Canny. A computational approach to edge detection. IEEE Transactions on Puttern Analysis and Machine Intelligence, PAM1-866:669--988, 1986.
[34] Nathan Halko, Per- Ger-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev., 53:217-288, 2009.
[35] John Lazzari and Xiuwen Liu. Understanding the spectral bias of coordinate based MLPs via training dynamics. (arXiv:2301.05816 2023.
[36] Zhemin Li, Hongxia Wang. and Deyu Meng. Regularize implicit neural representation by itself. In 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10280-10288, IEEE.
[37] Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard. A structured dictionary perspective on implicit neural representations. (arXiv:2112.01911) 2022.
[38] Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G. Baraniuk. WIRE: Wavelet implicit neural representations. (arXiv:2001.05117
2023.
12

---

[39] Jeremy Vonderfecht and Feng Liu. Predicting the encoding error of sirens, 2024.
[40] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural langent kernel. (arXiv:2010.15110) 2020.
[41]
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. 2020(1 1): 13301.
[42] Shervin Khalafi, Saurabh Sihag, and Alejandro Ribeiro. Neural tangent kernels motivate cross- covariance graphs neural networks. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 13577-23622 PMLR, 21-27 Jul 2024.
[43] Johannes Schwab, Bryan T. Kelly, Semyon Malamud, and Teng Andrea Xu. Training ntk to generalize with kare. ArXiv, abs/2505.1 1347, 2025.
[44] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.698D) 2014.
[45] Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: image processing in python. PeerJ, 2:e453, 2014.
[46] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. Ieee, 2009.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035, Curran Associates, Inc., 2019.
[48] Richard Zou Horace He. funetorch: Jax-like composable function transforms for pytorch. 2021.
[49] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. Journal of machine learning research, 2(0cc))2225-2883)) 2011.
[50] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero. Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul Mulbregt, and SeiPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
[51] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
[52] C. Harris and M. Stephens. combined corner and edge detector. Proceedings of the Alvey Vision Conference, pages 23.1 23.1-23.6. Alvety Vision Club, 1988. 23:.10.5244/C..223..
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims in the abastract are that (1) we derive a local approximation of the NTK for SIRENs (all of Section 3); (2) construct order parameters from this approximation, namely, in Section 3.1 for diffusion wavecrests, loss rate collapse (Section 3,4), and NTK alignment (3.3). We discuss SIREN biases in section 4.1 and 4.2 and the Discussion (Section 5). Considering more general deep image regression, we compare against ReLU+PE in Appendix D.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: addition to the conclusion (which clarifies we only consider full batch GD), make numerous assumptions outlined in Section 3 that only know to hold for
14

---

SIRENs (which we validate in Appendix F). What's more, as outlined in the introduction, our analysis is only tractable because we are working in low dimensional domains (ie image regression).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The main paper, this is all in Section 3. Extra proofs found in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 4 begins with summary of our setup, and Appendix B gives the full details. Our actual experiments are very simple: training a variety of SIREN models and tracking summary statistics of the NTK. The bulk of the paper is derivations.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will releasing code as part of the supplementary materials, namely, our training code, monitoring code, and analysis code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptaips.cc/puulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We search over a range of hyperparameters summarized at the beginning of Section and in detail in Appendix B. This for exploration, rather than optimization, as this paper is about considering the inductive biases induced by different hyperparameters. Re optimizer only use full batch GD (described in Section 2) with small, constant learning rate described in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table 1 summarizes results from across several runs (aggregated across architectures, datasets, and random seeds). The heatmap in Figure obtain through analysis of the confidence bounds we used to detect phase transitions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
17

---

Answer: [Yes]
Justification:
This is in Appendix B.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://heurips..c http://rppp...ppooo/occcccccaoaa
Question:
Answer: [Yes]
Our work doesn't involve any human subjects, and fairly concentrated in its scope to theoretical aspects of learning dynamics in 2D image regression problems. Our research of critical nature, though it can potentially be used help interpret image regression models, and thus could be the foundation of tools to improve safety.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is a theoretical nature specialized to very specific models, and we do not believe it has wider societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
18

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We do not believe our research warrants this. is a theoretical investigation on the dynamics of learning in specific class of image regression models. We do anticipate that our code or our derivations can be misused.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix B outlines the images used from public datasets, and the means through which we obtained them. make use no other assets, beyond pytorch and the scipy ecosystem, which we cite.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the sesrrssithhcdee package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
19

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't do any crowdsourcing work with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: There were no study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs not used in this paper.
Guidelines:
20

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https://neurip:: /ttp://pp.ccc//offfeeeee//////// for what should or should not be described.
21

---

Supplementary Materials
Deferred Proofs
22
Decomposition the NTK over layers
22
A.2
Proof of Theorem 3.1
Diffusive Evolution of the Residuals
23
A.3
Proof of Theorem 3.2 Local Cauchy Approximation of the CNTK
24
A.4
Proof of Corollary 3.2.1
Obtaining the Correlation Lengthscale from the Cauchy
Approximation
26
A.5
Proof of Corollary 3.2.3 Minimum Value of CNTK
27
A.6 Relating Loss Gradient Variance the NTK
28
Experimental Details
28
B. Model Training
28
B.2 Order Parameter Estimation
29
Occurrence Rates of Phase Transitions
30
Impact of Image Features
30
Additional Figures: Impact of Hyperparameters
31
Comparison with ReLU Activations
32
Implications of Local Image Structure Feature Learning
34
the Relationship Between Structure Tensors and Tangent Kernels
34
E.2 MAG-MA: Order Parameters From Translational Symmetry Breaking
36
Evaluating Fidelity of Approximation
36
Local Structure of the NTK.
36
Cauchy Approximation
37
G Additional Experimental Results
38
Order Parameter Trajectories for Single Runs
38
G.2 Influence of Hyperparameters on Order Parameter Trajectories
38
A Deferred Proofs
A.1 Decomposition of the NTK over layers
Consider a feedforward neural network, denoted by f(x) h(D) .... (x). We furthermore define:
200 ((--) +00)
(24)
A00 (2
(25)
22

---

In this way, we may calculate the parametric gradients as follows:
0f
V wwaf= = 0z(1)
!h
(26)
0f
vec Ww = 18
11-11)
(27)
dz(1)
(x:) ( (18
0f(zi)
) (
of(zz)
vee(V [(x))) vec(Vin (ww = 11 (1-1) (xxi)
180 0z11)
(
11
(1-1)
(xx))
0z(1)
(28)
= (xa)) (10
10
Of(z))
of(z,)
11 (x)))
(29)
02l1)
0zl1]
(
of(zz)) (x;)) 1 (1))))
(
(30)
02l1]
The first term in this product defines functional similarity between points, while the second defines representational similarity. Thinking of each term as separate kernel, the overall layer kernel ic the product is defined via an AND operation. A similar formula holds for the other layers. For the biases, we have, more simply:
Of
VoO f 0z())
(31)
0f(x;)
0f(zz) 0z(1
Vo fa:) (ans
(32)
0z(1)
The full NTK is then given simply by:
Of(x,) T *a((zz)
(1 1 1++h0)))xx)))) ( (x))
K (z;1
(33)
0z(1)
0z())
N
k (Z,,
(34)
In particular, we have:
0f(z) 0z(0)
Kix, 2;
Σ
(1+
(x)
(35)
Following the same logic, the full NTK is defined OR over all the layers. For INRs, these layers tend to be frequency separated, so that lower layers correspond to lower frequencies.
A.2 Proof of Theorem 3.1 Diffusive Evolution of the Residuals
The motivation for our ansatz in equation 10 is the empirical form of the correlation function in Written fully, we have:
equation 7
K(x, x u) x (tt.
(36)
Thus the residuals evolve according to:
f(x) - /
du rlx
(37)
<IVoo((=)) / du exp(pplliili (xtpcpp//////////))///// co)()) / du r(x + +wx
(38)
[ooof( / du cxp(-Ilwll/ + u) (x)
(39)
When μ4, = E[(]] and Coo decay to zero, the second, background term the above equation becomes dominated by local interactions. Thus, in Section we will track the following order parameter:
Kx
Ex Vo (x)]
(40)
23

---

The order parameter is large in the Drift phase, and small in the Diffusion phase. In Section B we overview the specifics of how detect changes in the phase, For the remainder of this section, analytically study kernels of the form:
K (x, x + = A(x)e (Mx))--(/25'() (x)
(41)
0,
(42)
That is, kernels without background term. Here, NN pe, 23) denotes the d-dimensional; multivariate Gaussian Distribution:
p(1), [(x)) V(2T (2 det E(x) exp (-2(u-n(a))
m(x))
(43)
For our case, d = 2, and E(x) = (1) 1. The determinant of the covariance is as follows:
det E(x) = (52)² det! = (z)
(44)
We now consider the integral of the following quadratic form:
f du (u T Hu)e x² (x) (x) / du (u Nu)
(45)
m (2) Hu]
(46)
(x)((((()))) (x)
(47)
(48)
Now, let's look at the following Taylor expansion:
r(x + u) = r(x) + u T V or + z" (V:r)u
(49)
When integrating the above equation symmetric and anti-symmetric functions. Thus, we have
3
the second term vanishes, because it involves product of
f du r(o + u) (xx x + u) A(x) f du r(x + uuee -u/-=2/22" (x)
(50)
-A(x) fdu [r(a)e (20 + 2
Tu.T(Ve (V2r)ue-
(51)
Leveraging our result for the quadratic term, we have, finally:
I du r(x + u) K() 2mm) +
(52)
=
(53)
Thus, the diffusion equation becomes:
A.3 Proof of Theorem 3.2 Local Cauchy Approximation of the CNTK
A.3.1 Notation and Derivation
We consider an arbitrary vector valued function f()), and consider the cosine of the angle between f()) and f(r + u) for small displacements u. To ease notation, let make use of the following shorthands:
a f(x) b = f(z + u)
(54)
(55)
c a
(56)
= Vaa
(57)
D = V V
(58)
24

---

To first order in we have:
o a u
(59)
c cxuu! JJJ
(60)
|||| ~ laa + u J12
(61)
|||a|1 + u D + ||0 J|
(62)
Our goal is to discern the local behaviour of the cosine of the angle 0 between and (as illustrated
Figure 5). To that end, our starting point the law of cosines:
B
b
c
C
A
Figure 5: Triangle with vectors a, b, and b a, inscribed in a circumcircle.
||a||2 ||b|12 ||c||2
cos Φ
(63)
2||a|| ||a||
2|| + TTTD
T
J|1? F
1+
||a|| +
(64)
2//a||2
||a||
To proceed, note that, for small scalar e, we have the following identity:
2
(1 (1+c)+
1+ 8
(65)
Thus:
2 || + u D
COS Φ
(66)
2 2|
D + [uu J|
(u' D)²
dlaleal
(67)
For the NTK, where we will have Vol. ||a| is so large that we may ignore the term of order
||||| -2 We illustrate our approximation in Figure 6
A.3.2
Specialization for Feed Forward Neural Networks
want to consider the case where, per our previous derivation, a Vof. This procedure
straightforward for the biases. For the weights wr(() we have:
() aw!!
0f(z; 0) (1 (1-1) (x; 0)
(68)
0z
25

---

Figure 6: Cauchy Approximation of the Cosine NTK. Left: Sample image, and test point x = A. Middle: visualization of CNTk(E, x) in the vicinity of the point for small separations Right: the Cauchy approximation, capturing both the range, orientation, and the local minima of the true CNTK.
Therefore:
0f 8)
0f((;; 0) (1-1) (x:0)+ + 0f(z; 8) 0h (x;
(1-1)
0)
Ox ow dz 12
(69)
2z
) 1) + (J - am ff
(70)
Before proceeding, us note that the following holds:
Σ )im (0 (0.0)) - || 200 S112
1
(71)
2
EnS Jimh 2zm
11
(72)
2
The covariance matrix in our Gaussian approximation thus given by:
0f2
0f2
H
(73)
Oz ow!! Ox ow!!
i
ij
Σ (ng (1- 1) (2) Jm (J Km + 10 (0 () "i (5m(( (1(
=
(74)
i.j
+(J
1-11 (J )in (0 (0,00 (() jmm
(1-1)
= -1
(75)
0 /|I + V. /1l 1=
The contribution from the bias comparatively simple:
H (1) =
(76)
A.4 Proof of Corollary 3.2.1: Obtaining the Correlation Lengthscale from the Cauchy
Approximation
To determine the level sets of the Cauchy Approximation. must solve:
20 + u D2 c
CNTK(I, + = 2a3 + u D + u H
(77)
26

---

Rearranging, and collecting terms, we have:
2 + u D2 c(2ngz + u D uu Hzu) 0
T
(78)
1 T D3
= 2(1 c)a² c
u H = 0
(79)
u T Hgu
1 -
T D=
2(1 - c)
(80)
c
c
1 i
T
1 -
(1 (1
2(1 - c)
HD' D
H
H
H H
H D D
(81)
2c
2c
402
1-c H D
H 11 - LHEH'D D
u
2c
=1
(82)
2(1-c)
+
(1-c)²
D H-
de
This is the equation of an ellipse centred at v 1-c
D, and with shape matrix:
2c
H
Eshape
(83)
2(1-c)
a2
(1-4)
D
40
The area of this ellipse is (noting that H is 2x2 matrix):
Acllipee
π
(84)
det Ishape
1
2(1
c) a
(1 (1-
-DTH'DD
+
(85)
det HI
c
4ea
The correlation lengthscale is then obtained from:
E =
Aettipso/ π
(86)
A.5
Proof of Corollary 3.2.3: Minimum Value of ONTK
We consider minimizing the following function:
f(u) = Quu)
(87)
P(u)
Q(u) = 2a² + +u² D
(88)
P(u)= Q(u) + uT Hl
(89)
Here, H is non-degenerate and positive definite. Thus:
0f P.QP- - Q0oP 0
(90)
du
p2
0,QP Q0., P
(91)
Thus:
(u Hu)D = (4a² + 2u. 2u D)Hu
(92)
Clearly u = 0 is a solution, and knowing that our expression locally approximates the cosine, we expect this to be maximum. To find the other solution, which will be a minima, we take the dot product of both sides of the above equaiton with u. After simplifying, we obtain:
" T D 4a2
(93)
If we insert this into equation 92
we get:
(u. T Hu) D -4a² Hu
(94)
+ (m Hu)H
4a2 Fu
(95)
+ (u H H = 16a 4
(96)
16a 4
u Hu DH
(97)
27

---

Armed with an expression for u D and u Hu, we derive the following formula for the min:
2a² u D
Smm
(98)
2a² +u + u
Huu
u-agminn
2a² 4a²
(99)
2a² 4a + DH DD
16a4
DH-1D
(100)
DH-1D= - 8a
A.6 Relating Loss Gradient Variance to the NTK
Our goal is to quantify the amount of noise in the gradients of the local loss C(x,) = fx(,;; 0) We have, in terms of the Jacobian Jip Ōo f (xz), the following sample matrix for the gradients:
G = R.
(101)
Here have defined:
R = diag(r) For dataset with N samples, the sample mean and covariance given by:
(102)
IN
(103)
N
JTr
(104)
N
C J R² un
(105)
From the cycle property of the trace, we have:
tr(J R² = (
(106)
-tr(12
(107)
We also have:
Tr(uu() =11u12
(108)
N22 T JJJ r
(109)
1
N2"
r KNIKT
(110)
Thus the variance of the loss gradients given by:
1
1
oe N
Tr(R"KNTH
N2 KNTKT
(111)
B Experimental Details
B.1 Model Training
All SIREN models trained on the images shown in Figure 7 which obtain through the python package seikit-image [45], and the ImageNet dataset [46]. These images are down-sampled to resolution of 64 x 64 for training, but validation task, track the reconstruction error on the images downsampled to 256 x 256 resolution. Our SIREN models are implemented using Pytorch [7]. and trained using NVIDIA RTX A6000 48GB GPUs for 10000 epochs, using full batch gradient descent with a learning rate of 1e-3. In our experimental sweeps, we consider the following ranges:
Random seeds from interval [0, 5).
Width from set (64, 128).
Depth from set (3, 4, 5).
Wo from set (15, 30, 60, 90).
28

---

camera
astro
coins
coffee
microphone
viper
baseball
S8X
calamaran
violin
bonnet
groom
syringe
African Atrican_grey
Figure 7: Fifteen of the thirty images used for training INRs
B.2 Order Parameter Estimation
Analytical Order Parameters: To compute the NTK, we use manual implementation of back- propagation compute the gradients V fx) for each layer, along with the hidden activations (x). The NTK is then constructed efficiently using the decomposition across layers outlined in Section A. To evaluate the local structure components and /) defined in Theorem 3.2 we obtain the spatial gradients using functorch [48]. We also assemble the defined Theorem 3.2 in this way, except we leverage the decomposition outlined in Section A.3.2 2 to streamline this process, and occupy less memory.
Empirical Order Parameters: Below we describe the estimation procedure for each of the empirical order parameters.
To estimate the correlation funetions empirically, we group pairs of datapoints into 50 bins based on a uniform division of the range of distances. Based on the coordinate range, the minimum distance is 0, and the maximum distance is 2 Within each bin, we evaluate the mean of the CNTK, defining c(c). Based on these groups, we estimate our order parameters as follows:
To estimate the asymptotic value Coo. compute the mean value of c(e) over the last ten bins (corresponding to points with the furthest separation).
<()) and then use linear Given the asymptotic value, we rescale all c(s) + ē(e) I-ex-
interpolation to find the value of e for which ě(c) = 0.5, the FWHM. We then have FWHM 2
Scorr
As an additional measure the correlation length-scale (which we will use in Appendix D). we may calculate the number of points Nc for which CNTk is greater than some cutoff' (we use (1 + coo)). )) The effective correlation lenght-scale is then given by / /NedA/a n, where d.A the area of the coordinate grid cells. We denote this estimate &FW HM-
estimate AUC([vo). V/), the ground truth edges are identified using the Canny Edge Detector distributed through scikit-image [45]. We then evaluate the Area Under the Receiver Operating Characteristic Curve (ROC AUC) using the implementation in scikit-learn [49].
The principal eigenvector Vg. and the principal eigenvalue λo are both computed using
29

---

own implementation of the Randomized Singular Value Decomposition built with pytorch using iterations and 10 oversamples.
To evaluate the Centred Kernel Alignment, in order prevent modes from obscuring alignment, the following centred-variant of the normalized Hilbert-Schmidt Information Criterion (HSIC) is employed:
Kr(
CK)(K,K')> V (r(K)K)
(112)
Here, Ke denotes that centrering operation has been applied, and is defined as:
Kg = (I 11 - 11T)
(113)
7
n
For both Kx and Ky,
bandwidths K 0.1.
To determine the residual correlations in Table 1 we randomly sample (and flatten) 15000 x 15 patches from the validation residuals, and compute the pearson correlation matrix. We then record the mean correlation between all pixels in the patch and the patch centre.
Identifying Critical Points:
For the gradient variance og. the loss rate Treaat and the background contribution J4, the location, and confidence region, for the critical points are identified using the peak detection algorithm distributed through sigpal [50]. For the gradient variance, filter for peaks with a prominence case where multiple peaks are found, select the peaks which appear closest in time. Finally, for H, K, the phase transition occurs not at the peak itself, but after the signal decays to zero. Thus and the right-most boundary.
0.2, loss rate we use 0.4, and for the background we use 0.2. In the
as confidence region the interval between the identified peak
For the min CNTR. we linearly interpolate find the time where min CNTK crosses 0. To compute the confidence interval, we also track the cumulative std of min CNTK. denoted <tt). We then use the same linear interpolation strategy to find the times where min CNTK = (t) and min CNTK = -(()..
For all other parameters, we fit a sigmoid using the curve fitting function from scipy.optimize, with the default settings. The curve we fit has the form:
f(x; A, B, w) A (B - A) a) (1-BB--)(--(---- (
(114)
We identify the time / - with the critical point, with confidence region defined by μ + 2w. For MAG-MA, where the goal is to detect deviation from zero, we fit this sigmoid to the cumulative STD.
Occurrence Rates of Phase Transitions
C.1 Impact of Image Features
There are three main cases in which a critical point cannot be reliably identified in an order parameter trajectory:
1. Peaks in the gradient variance dy may be absent, or not prominent enough, be detected using standard peak detector.
2. A zero-crossing cannot be found for the min CNTK because, at initialization, it is already less than 0.
3. The order parameters do not saturate, and thus, are poorly represented as sigmoids. This is really only problem for edge alignment AUC([ V and the task alignment CKA(Ky, K WNTx). ). In the latter case, in some trials we see CKA A steadily decrease after the inflection point of the loss. Numerically, we omit runs where the mean squared error the fitted sigmoid is greater than 0.01.
30

---

Table Proportion of runs with errors:
Frequency which runs were ommitted in constructing
Figure
3
as a function of depth and bandwidth wg.
depth
Wp
AUC( VI)
CKA(Ky, KNTK)
de
min CNTK
90
0.570
0.293
0.007
1.000
60
0.447
0.177
0.023
1.000
3
30
0.503
0.157
0.093
0.700
15
0.643
0.773
0.583
0.600
90
0.117
0.067
0.000
0,500
60
0.067
0.013
0.000
0.500
30
0.043
0.057
0.037
0,500
0.257 15
0.693
0.170
0.343
90
0.017
0.017
0.000
0.470
5
60
0.037
0.003
0.000
0.203
0.033
0.060
0.000
0.083
5 15
0.070
0.643
0.070
0.093
The occurence rates, as function of the hyperparameters used, are shown in Table is important to note, phase transitions may still occur parameter may be simply too weak") to be identified by the change detection algorithm outlined in Section B.2 It is for this reason that we employ multiple order parameters to identify the same transition (ex the min CNTK and the gradien variance o). Nevertheless, what properties of image datasets may be used predict the aforementioned failure modes. To this end, for each experimental run, we determine if any of the previously mentioned failure modes has occurred, and then record the frequency of success for each image studied. In Figure 8 we that these frequencies correlate with the complexity of the image, as measured the variance of the that
during these failure modes the shift in the order
instructive to identify
spatial gradient magnitudes /./lI. Namely, peaks of the parameter gradient variance ag. but collapse of the kernel alignment as measured by C (NTK). This is reflected in their strong negative/positve spearman correlations. These same properties correlate strongly with the best model performance achieved across all hyperparams (left of Figure 8 These correlations give additional support to the mechanism described in Section 3.3 whereby SIREN models struggle to fit edges as they have sharp gradients. Finally, we note that the image complexity seems have little impact on the error rates for the edge alignment AUC([vo) VI) (spearman correlation -0.017) and the minimum value of the CNTK (spearman correlation -0.1320). By contrast, these parameters are more sensitive the model architecture.
complex images result in sharper
C.2 Additional Figures: Impact of Hyperparameters
The broad effects of varying depth and we on AUC([v)), Vi) are summarized in Table To gain deeper insight into how these parameters influence the principal eigenvector, we examine the two case studies illustrated in Figures T0 11 A lower wa, by broadening the correlation lengthscale, inducing smoothing effect, retaining only the sharpest edges. Increasing depth also removes noise.
1 Also note, even when AUC([vol, V1) is weak, edges are still visible in the principal eigenvector, seen in Figure
31

---

corned
04
Gor
CKA(Ky Kwrrd errr rate
china cakine
0.50
log Loval
1.8
Conezeo Raker
sou trd moyayy
chonk caline
afrert
besaga
cortiion X
binder
0.2
tiis cabinel
0.25 bedro
Artcar
chew
coP
Mad
caul Boer
lujo not Alicc Main-rarig Pollee monsptrre
Suncbull poesi
CaETTNNe ber 20
0.02
0.04 vtd([
0.05
0.32
0.06
0.06
0.02
0.34
0.06
std VIIl
std[|/VIll>
Figure 8: Image Complexity Affects Detection of Phase Transitions. We measure the image complexity according the standard deviation of the magnitude of the spatial gradients () a/D.. Dashed red line indicates line of best fit. Legend records spearman correlation. Left: higher complexity images are positively correlated with higher losses (and therefore, worse performance). Middle: higher complexity images do not saturate the target kernel alignment, causing errors in our sigmoidal fits. Right: higher complexity images lead to sharper peaks in the paramter gradient variance, making their identification easier.
-- 15, logi.. 46557
--- ml. logh 4.6674
-. 60. mi log
4.8660
L- 90, min log L... 4.7927
dush AuC.v.. Vn 0.5775
dosth 3, AuCwv, Vit -0 5014
doat 3, AUCC 90 0.5156
0.5304
loglave
logLest
Leg!
log Lo
degah 4, AUCOs, 2i1 6.5762
dugth 4. AUC/V. 201 0.9960
dosen AUCiM 100 0.3089
AUCIv.
m=. logloss -4.2200
min logtes -4.9363
60, min log
5.0061
90, min - 4.7502
Ogpth AuCove on 0.9913
cuppn 5, AUCCr, 911 0.6224
AUCore D0 0,5078
copth
Figure 9: Variation in NTK Alinment with Hyperparameters (coffee). Principle eigenvectors of the NTK at the end of training. Best performing architecture highlighted in blue.
D Comparison with ReLU Activations
To justify our focus on sinusoidal neural networks, in this section we examine the learning dynamics of ReLU-MLPs, based on the positional encoding scheme used in [51]. The positional encoding layer is kept static, and we pre-compute the nyquist frequencies corresponding to our image size (64 x 64), as is done in [23). We denote this architecture ReL.U-PE. All other architectural choices are identical to those described Appendix B
We observe a number of differences between SIRENs
and ReLU-PEs (visualized in Figure 1
32

---

[Voll [[ 6.62
R -1 01815
AUC 00)00849
AUCIVs. 0.700
16
0.06
12
0.03
3.02
depti-4, -00
deptn-4, 60-900 90
depti-4, G/o
depth-4, 40--0
ollog r1v.//--... 0.672
AUC(Ve. 91)-0.711
AUCIVs 0..0.822
40
20
0.04
0.02
deptheb,
MJ-330
deptn=5, 6S790
depth=5, 6ip-30
dopthoo, 6g-90
Figure 10: Effect of Hyperparametters Edge Alignment: Reproduction of Figure microphone dataset
for the
Vell 1/9//1)00.488
Vell. [/WI|
AUC(ve, W)00.790
AUC(v, W/=0.731
0.050
5.0
0.030
0.025
2.5
0.015
depthe4, WEP
dept 09
depth#4, WA-30
dopth=4, 4y-30
ellogl| 119
V.ll. l/v.|||=0.245
AUC(ve, W)=00828
AUC|v, W)50.754
12
0.04
0.02
0,015
deptn=5,
WC-30
depthe5, 0,-90
depth=s, WA-30
depth=5, 6A-90
Figure 11: Effect of Hyperparameters Edge Alignment: Reproduction of Figure 4 for the coins dataset
Firstly, SIREN models exhibit strong locality: over the course of training, the asymptotic value of the ONTK k decays to O, whereas it grows in ReLU-PE models. What's more, the range of interaction as measured by ÊFWHM is larger in ReLU-PE models. An example comparing the correlation functions for both architectures is shown in Figure 12
Secondly, learning is much slower ReLU-PE models than is in SIRENs. One explanation for this is that there is more gradient confusion (II), that is. the minimum value of the CNTK is lower. In particular, min CNTR is less than zero across all ReLU-PE runs, so that these models are always operating in the "slow" phase of learning.
The principal eigenvalue Åo of the NTK grows to be orders of magnitude larger for SIREN models than for ReLU-PE models. That said, tangent kernel aligmment still occurs in ReLU-PE models, it is just much slower process. In Figurq14 we train a 7-layer deep, 128-unit wide MLP full-batch with learning rate le for 250k epochs, varying only the activation function. To reach the edge-alignment achieved by SIREN model after 453 epochs, the ReL.UPE model must train for 239986 epochs. We also see that more of the edges are present in the principal eigenvector of the SIREN model's NTK.
33

---

RELU-PE
SIREN on
BIREN A
SIREN k
cact
03
35 24
15
18 35
32
Figure 12: Effect of Hyperparameters on Correlation Functions At Initialization: In ReLU-PE models, the Gaussian approximation of the CNTK correlation function is poor for all depths, due to high- sigh-varancee, long range interactions. By contrast, for SIREN models, there much less variance, and the range of the interactions shrinks for increasing w/.
SIREN
ReluPE
SIREN
ReluPE
0.50
0.30
3.30
0.30
0.15
0:15
0.15
0.50
0.00
000
0.00
0.30
030
minCex
0.15
15
0000
c03
-1.0
-1.0
10
10a
10'
10
109
NNB Ma
30
NNG Ma
30
10
10
40
1.0
30 Eaaach
10
10
30
1.0
30
30
10'
3.0
10'
10'
304
30'
10'
30'
10'
10-
Spocc
Epoch
Esedd
Figure 13: Learning Trajectories for SIREN and ReLU-PE models: Histograms visualizing the distribution of various order parameters throughout training. See Section for full details on models and datasets used.
At initialization, MAG-Ma is orders of magnitude lower for SIREN models than for Relu-PE models, indicating the latter are already operating in phase where translational symmetry is broken.
In summary, while ReLU-PE models exhibit Neural Tangent Kernel alignment, it is a much slower, non-local process, that does not coincide with loss-rate collapse translational symmetry breaking.
E
Implications of Local Image Structure on Feature Learning
E.1
On the Relationship Between Structure Tensors and Tangent Kernels
We are now positioned to elucidate the features learned during NTK alignment. As proposed in Section 3.3 the local structure of the NTK adapts to the spatial variations in parameter gradients. In this section, we delve into the spectral consequences of this adaptation. We contend that the principal eigenvectors evolve into edge detectors, resembling the auto-correlation structure tensors
34

---

Raw image
SIREN: Vo
SIREN: V1
SIREN: V2
0.05
0.03
Image edges
ReLU-PE: Vo
ReLU-PE: V1
RoLU-PE: V2
0.02
0.00
Figure 14: NTK alignment in SIREN and ReLU-PE models: The principal eigenvectors of the NTK at the end of training. Final AUC([v)), V/) for the ReLU-PE is 0.754, whereas AUC([vo), V/) for the SIREN model is 0.804. The training time required achieve edge-alignment score greater than 0.75 for the SIREN model was 453 epochs, whereas for the ReLU-PE model it was 239986 epochs.
commonly employed in traditional computer vision. This observation reinforces the concept of translation symmetry breaking: computer vision, the utility of auto-correlation structure tensors stems from the premise that the most informative features are those that minimize redundancy. The auto-correlation function quantifies this through metrics of translational symmetry breaking.
Per the discussion in Section 3.3 the principal eigenvector is closely related to the auto-correlation function. By leveraging the decomposition of the NTK in equation 30 we may relate the features considered in computer vision. Let us define:
Th (w; = 1+ (x)
(115)
that the largest contribution comes from the immediate neighbourhood of æ. This motivates to perform a Taylor expansion of the remaining terms as follows:
K11 Σ
Ki(z, x + u)
(116)
0f(z) 0f +
Σ wi (u; x.
(117)
Ozid
Ozid
0f(x) 0f(x)
Σ wi wi(u; x)) Σ
+ h.o.l
(118)
Ozid
Ozia
(r()(();)) + h.o.t
(119)
Here, A, denotes the structure tensor used in the Harris-Coree detector [52] Accordingly, that K1 and thus, the principle eigenvector assess the extent of local translational symmetry disruption point This principle underlies feature selection in computer vision, concept mirrored in NTK feature learning, as evidenced by the principal eigenvectors that are predominantly maximized around dataset edges and corners.
crucial to highlight that A, pertains the structure tensor specific layer Collectively, the entire DNN's NTK facilitates feature selection across scale pyramid.
35

---

[1961199f11
lie
[|9I| PIi at point
119-1196111 "1l
400
850
0.20
250
200
100
D:25
Figure 15: Evolution of spatial variation of the parameter gradients. At initialization, there very small amount of variance (note the scale of the variations). As the variance grows, translational symmetry is broken, and dynamical phase transition occurs.
Bön't)
01 Luvil
Alend]
Biend)
Clend)
MAG-Ma
033
(e)
Figure 16: Evolution of the Cosine NTK: We visualize CNTK (It, x + u) around three points E B, for small separations u. At initialization, ONTK locally resembles an isotropic, connslationinvaaiin RBF. However, as training progresses, these symmetries are broken. MAG-Ma (described Section E.2 is an ander-parameter that monitors the original symmetry, and changes at the critical point.
E.2 MAG-MA: Order Parameters From Translational Symmetry Breaking
While previous sections have focused bottom-up construction of order parameters, this section adopts a top-down approach rooted in symmetry principles. In Sections 3. 33.4. we expressed several order parameters in terms of the parameters a, D, H, characterizing the local structure of the CNTK- Notably, each of these parameters is now function of the spatial variation of the parameter gradients, whose evolution is showcased in Figure 15] This suggests is translation symmetry which is broken at the phase transition. Indeed, from Figure 16 we observe the ONTK is an approximately stationary, isotropic kernel desirable property for INRs 30) As such, the Kernel exhibits bias for location or direction. Over the course of training, we may monitor the emergence of such bias with the following metric
ER V log || || [
(120)
We refer to this statistic as MAG-Ma: the Magnitude the Average Gradient of the Log Gradient- Field Magnitudes. Intuitively, this order parameter captures the statistical preference for spatial direction in the dataset. The evolution of this quantity is plotted in Figure [6 and its alignment with the other order parameters is shown in Figure 3 We that throughout the Fast Phase of training (before the peak in the loss rate Levat), the local structure of the CNTK statistically translation invariant, and MAG-Ma is close to zero. However, just after the critical point, it grows rapidly coinciding with the edge memorization described in Section 3.3
F Evaluating Fidelity of Approximation
F.1 Local Structure of the NTK
As described in Section 3.1 INRs are often carefully designed to ensure diagonally dominant
NTK [30 31 32). In higher dimensions, diagonal dominance is equivalent to bias towards local
36

---

0.30
(com(
0.15
30
90
3
WJb
depth
0.30
(=)
0.15
30
Wo
depth
Figure 17: Hyperparameters Affect Local NTK Structure. Boxplots visualizing the distribution of structural parameters for the CNTK- Top row: variation in the initial correlation lengthscale Scorr (0). Bottom row: variation in the initial asymptotic value of the CNTK (Coc (0)).
1.0
apech
opoch
2.0
0.8
296
296
9478
1.5
0.6
k(x,0 c)
c
1.0
0.4
0.5
0.2
0.0
0.0
0
2
2
Figure 18: Visualization showing the empirical correlation function for the normalized parameter gradients. On the left-hand side is the global corelation-functiin for the ONTK- On the right is the local-correlatio function for the KNTK around a test point æ. Dashed lines show fitted Gaussian approximation, and error bars show variance across dataset. Over the course of training, both the global correlation lengthscale Eoorr. and the terminal value Coo, evolve.
interactions. We see in Figure the hyperparameters that most affect this local structure: we observe that, while depth has small impact the initial correlation lengthscale Ecory(() higher values of wi cause the CNTK to become dramatically more localized. The converse true for the asymptotic value C.c (0): wa has minor effect, but increasing depth leads to stronger interactions across large distances.
Beyond initialization, in Figure 18 layer deep, 128-unit wide SIREN model on 128 x grayscale image of a cat, with bandwidth we 30. Emprirically, we see that the Gaussian approximation described in Section 3.1 remains valid across training, with the asymptotic value Coo of the CNTK decaying to zero.
we examine the evolution of the correlation function for five-
Cauchy Approximation
To ascertain the fidelity of the Cauchy Approximation, we estimate the Pearson correlation between the true values of the correlation lengthscale and min CNTK, and the prediction based only on the local model. We choose this metric because the identification of critical points is insensitive to linear
37

---

Table Fidelity of Cauchy Approximation: Pearson correlation between the true order parameter and predictions using the local Cauchy Approximation, Mean and standard deviation are calculated over the spread of models and datasets described in Section B
depth
Wp
min CNTK
2
3
15
0.980 + 0.017
0.889 = 0.068
0.980 = 0.011
30
0.924 + 0.110
0.909 = 0.063
0.985 = 0.006
60
0.830 + 0.208
0.946 0.043
0.988 + 0.004
90
0.856 + 0.220
0.967 + 0.020
0.986 + 0.006
4
15
0.983 + 0.015
0.925 + 0.059
0.982 = 0.009
30
0.964 + 0.036
0.956 = 0.039
0.985 = 0.007
60
0.920 + 0.152
0.955 + 0.036
0.986 = 0.008
90
0.974 + 0.026
0.961 + 0.031
0.984 + 0.009
5
15
0.985 0.011
0.921 = 0.049
0.978 = 0.010
30
0.969 + 0.023
0.953 = 0.038
0.983 = 0.008
60
0.947 + 0.066
0.966 = 0.033
0.982 = 0.028
90
0.959 + 0.037
0.974 = 0.026
0.982 = 0.010
The results are shown in Table B] Similarly, we evaluate our approximation of the principle eigenvector vp, by looking at the absolute cosine distance between our approximation and the ground-truth.
transformations.
Finally, in Section 3.3 we approximated the principal eigenvector to of the NTK with the row mean K1/1TL 1. The median cosine aligmment between the row mean and the true Uo was found to be 0.99995 across all epochs surveyed, across all models and datasets. The IQR 0.00446. The strength of this approximation is a testament to the extreme spectral gap of the NTK, which itself is consequence of NTK alignment.
G
Additional Experimental Results
G.1 Order Parameter Trajectories for Single Runs
This section contains additional illustrations of the order parameter trajectories, and the corresponding confidence region estimates, similar to the left side of Figure The results are shown in Figure [9] Each model is 5 layer deep, 128-unit wide SIREN network, trained with full-batch gradient-descent with learning rate of le-3.
G.2 Influence of Hy Hyperparameters Order Parameter Trajectories
In this section, perform ablation study understand the impact of different hyperparameters on the order parameter trajectories. The baseline model a 5-layer 128-unit wide SIREN with wh = 60. Figures 20 22 showcase the effect of depth. Figures 23 25 showcase the effect of the bandwidth parameter w.
When depth (and therefore model capacity) is decreased, we observe corresponding increase in the validation error. In shallower models, the initial gradient confusion (min CNTK) lower, delaying learning, and thus, the peak in the loss rate Levat- While the location of the phase transition changes, the trajectory of the order parameters shapes remain consistent, and exhibit less variance with increased depth. By contrast, there is dramatic change in the shape of the trajectories as we vary wg. When wi is high, &cory starts very low, favouring interactions with immediate neighbours, leading to low overlap with the RBF. During training, the range broadens rapidly, causing CKA (xx K KNTK) to grow sigmoidally at the critical point. Conversely, with low Wo. the range starts large but shrinks during training.
We additionally track the CKA between the NTK and static RBF kernel with fixed bandwidth Kx, as described in [. The evolution of this hyperparameter reflects the evolution of the correlation lengthscale Ecorr. When this value is large (as it is when wo is small), the NTK has broad diagonal,
38

---

0.0
Loo
-0.2
0.2
0.02
0.1
minCurx
0.0
minCais
0.0
-0.3
-0.1
0.30
CKA(Ks
CKA(Ky, Kind
0.1
0.15
0.75
AUC/ve. W)
0.75
AUCIVe
0.80
0.80
400
200
0.006
MAG-Ma
MAG-Ma
0.000
0.00
0.0003
1.K
0.002
0.0000
0.000
10"
10"
Epoch
Epoch
(a) aircraft carrier
(b) espresso maker
$ Laval
-0.025
-0.08
minGGm
0.0
minCam
0.0
-0.4
0.2
CKAIKs Kond
0.2
CKAIK; Kwnd)
0.1
0.7
0.54
AUC(ve, W
AUC(ve.
0.8
0.51
0.5
1000
600
Ap
500
300
MAG-Ma
MAG-Me
0.000
0.00
44K
10'
10'
10'
10" Epoch
10
Epoch
(c) chain
(d) violin
Figure 19: Alignment of Order Parameters. Order parameter evolution and critical points during training of SIREN model. The red vertical lines denote the location the critical points, and the green vertical lines denote confidence regions.
and thus overlaps well with the RBF. Over the course of training, Ecory shrinks, and thus, so does CKA(Kx. KNTK).
39

---

minC
Ecomr
CKA(Kx, KNTK)
cepth
dkpp
daper
0.0
0.06
-0.2
0.3
0.05
0.2
-0.6
10
10
10
10
10
10
10
10"
10"
10"
10"
10
10"
10
10"
Epoch
Epoch
Epoch
Figure 20: Effect of depth on Critical Behaviour (Microphone): Average MSEs, in order of ascending depth: 2.561e-2 + 9.355e- 2.555e + 8.970e 2.572. + 7.209e Dashed vertical lines denote the location of the peak of the loss rate Leval, marking the phase transition.
minCwTk
corr
CKA(Kx, KNTK)
0.065
deptr
0.060
0.055
dusfi
0.050
-0.6
0.045
10
10
10
10
10
10
10
10
10*
Epoch
Epoch
Epoch
Figure 21: Effect of depth Critical Behaviour (Sax): Average MSEs, in order of ascending
depth: 1.628e + 1.312 1.513e 2 + 3.384e -5 1..94e - 6.605e Dashed vertical lines denote the location of the peak of the loss rate Loval. marking the phase transition.
minGNTK
Ecorr
CKA(Kx, KNTK)
0.070
0.5
0.0
0.085
4
0.4
0.060
0.055
0.050
0.045
10
10
10
10
10"
10
10
10
10
10
10
10
10
Epoch
Epoch
Epoch
Figure 22: Effect of depth on Critical Behaviour (Violin): Average MSEs, in order of ascending
depth: 6.885e + 1.677e 5.930e 5.016e 5.665e 3 3.640e Dashed vertical lines denote the location of the peak of the loss rate Levat. marking the phase transition.
40

---

minCwTk
ccorr
CKA(Kx, KNTK)
15
15
30
30
60
60
90
90
-0.6
-0.8
10'
10
10
10" Epoch
10
10
10
10
Epoch
Epoch
Figure 23: Effect of We Critical Behaviour (Microphone): Average MSEs, in order of ascending
Wo: 2.601e + 1.804e 2.566e = 1.327e 2.572e = 7.209e -5 2.807e - + 7.688e
Dashed vertical lines denote the location of the peak of the loss rate Leval, marking the phase transition.
minCwIk
Ecorr
CKA(Kx, KNTK)
0.8
0.0
15
15
30
0.6
30
-0.2
60
60
90
0.4
90
-0.6
-0.8
0.025
10
10"
10"
10"
10"
10
10"
10"
Epoch
Epoch
Epoch
Figure 24: Effect of wp Critical Behaviour (Sax): Average MSEs, in order of ascending we:
1. 1.680e -2 2 + 1.666e -4 1.561e -2 +.6666.8 6.552e 1.494e 6.605e- 1.639e -2 + 3.938e -4, Dashed
vertical lines denote the location of the peak of the loss rate Levat, marking the phase transition.
minCwvk
Ecorr
CKA(Kx, KNTK)
0.8
0.150
0.125
Mu
0.6
15
-0.2
0.100
30
15
'60
30
0.075
0.4
60
0.050
0.2
90
0.025
10
10
10
10
10
10
10
10
10"
10"
10
10
10
10
10
Epoch
Epoch
Epoch
Figure 25: Effect of we on Critical Behaviour (Violin): Average MSEs, in order of ascending wp:
7. 7.223e 3 + 1.503e 6.305e + 3.199e 5.665e + 3.640e -5, 6.698e + 3.359 Dashed vertical lines denote the location of the peak the loss rate Leval marking the phase transition.
41


Reviews:
[{"review_id": "wKETsUL37r", "reviewer": "Reviewer_qFsx", "summary": "This paper theoretically establishes connections between the onset of NTKA and other dynamical phase transitions by deriving novel approximations for the local structure of the SIREN NTK and identify a novel learning phase in deep image regression, characterized by the appearance of diffusion-like wavecrests in the residuals. Experiments demonstrate the critical points for these different phase transitions cluster in time.", "questions": "I am not familiar with this area and topic, the only question I can raise can be seen in above weakness. Authors need to answer them.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.This paper is the first to observe the emergence of diffusion-like wavecrests in residuals during SIREN training, theoretically linked to NTK, filling a gap in visualizing learning dynamics.\n\n2.It empirically investigate the impact of image complexity and SIREN hyperparameters on the occurrence and timing of phase transitions,, offering practical guidance for SIREN design.\n\n3.Sufficient experiments prove the validity of this theory.\n\nWeakness:\n\n1.The paper discusses SIREN model in the regression task of image reconstruction. I wonder if it can be used in other models or tasks?\n\n2.The author doesn't seem to give the result of computational efficiency. It is suggested giving the result of computational efficiency of this method since computing eigenvectors requires high-dimensional operations.", "was_revised": true, "final_justification": "I am not familiar with this area and topic，consider other reviewers' opnions, I would like to keep the score", "rebuttal": "We'd like to thank the reviewer for taking the time to read and review our work.  To answer your questions:\n\n> The paper discusses SIREN model in the regression task of image reconstruction. I wonder if it can be used in other models or tasks?\n\nWe believe that it can, with some modifications.  Rather than considering the residuals $r(x;\\theta) = I(x) - f(x;\\theta)$, we would consider the generalised residual $r(x;\\theta) = \\frac{\\partial L}{\\partial f(x;\\theta)}$ Then, per the chain rule, we have:\n\n$\\dot{r}(x;\\theta) = \\nabla_{\\theta} r \\cdot \\dot{\\theta} = \\bigg(\\frac{\\partial^2 L}{\\partial f(x)^2}\\bigg)\\nabla_{\\theta}f(x)\\cdot \\dot{\\theta}$\n\nThe gradient flow dynamics are:\n\n$\\dot{\\theta} = -\\nabla_{\\theta}L = -\\int dx^{\\prime} r(x^{\\prime};\\theta) \\nabla_{\\theta}f(x^{\\prime};\\theta)$\n\nInserting the above into the residual evolution equation yields:\n\n$\\dot{r}(x;\\theta) = -\\bigg(\\frac{\\partial^2 L}{\\partial f(x)^2}\\bigg)\\int dx^{\\prime} r(x^{\\prime};\\theta) \\nabla_{\\theta}f(x^{\\prime};\\theta)\\cdot \\nabla_{\\theta}f(x;\\theta) = \\bigg(\\frac{\\partial^2 L}{\\partial f(x)^2}\\bigg)\\int dx^{\\prime} r(x^{\\prime};\\theta) K_{NTK}(x,x^{\\prime};\\theta)$\n\n\n\nHence, the NTK still emerges, although the residual equation is a little more complicated:\n- $r$ is now potentially a vector-valued function.\n- There is an additional, scalar coefficient that multiplies the NTK.\n\nNeither of these necessarily pose a problem for our approach.  The biggest question is whether the NTK $K_{NTK}$ experiences the same rapid decay with spatial separation.  At initialization, in the large width limit, the SIREN NTK decays like a sinc function [31].  If the decay remains sharp throughout training, then our derivation of the diffusion equation is largely unchanged: we make the Gaussian approximation, and perform a second-order Taylor expansion to obtain $r(x^{\\prime})$ from $r(x)$ and its spatial gradients.  The Gaussian integration is unaltered, with the net result that the RHS of equation (10) for the diffusion equation is simply multiplied by $\\bigg(\\frac{\\partial^2 L}{\\partial f(x)^2}\\bigg)$.\n\nIn summary, there are reasons to believe that our results would generalize beyond the INR problem considered.  However, we would like to emphasise that, while this is an interesting future direction, such an investigation is not necessary to support the core results presented in this paper.\n\n> The author doesn't seem to give the result of computational efficiency. It is suggested giving the result of computational efficiency of this method since computing eigenvectors requires high-dimensional operations.\n\nTo compute the eigenvectors of the NTK, we use a pytorch implementation of the Randomized SVD.  We described this in Section B.2 of the supplementary materials.  We can be more explicit by mentioning the method in the main body of the experimental section (Section 4) in the camera ready version.  That said, since the focus of our work lies in the analysis of the NTK rather than the computational method itself, we did not explicitly report detailed runtime performance.\n\nIf the concern is whether or not our technique can be adapted to study larger datasets, the answer is yes.  Randomized SVD approximates the singular value decomposition by projecting the matrix onto a lower-dimensional subspace using random sampling, significantly reducing the computational complexity compared to exact SVD methods.  It is commonly used for dimensionality reduction of high-dimensional datasets with millions or even billions of rows, such as those that emerge in genomics studies or recommender systems."}, {"review_id": "caSq70JwWF", "reviewer": "Reviewer_Myuw", "summary": "In this article, the authors aim to develop a unified framework that uses order parameters to characterize different learning phases by leveraging the NTK framework. Specifically, the authors focus on deep image regression with deep sinusoidal representation networks (SIRENs) to address a more complex use case than existing works, which typically employ two-layer or deep linear models. By using the NTK formalism, the authors identify 3 order parameters that delineate three learning phases: \n - Diffusion-like behavior of the network residuals, \n - Alignment of the network NTK, \n - Loss-rate collapse.\n\nSubsequently, through empirical analysis, they demonstrate that these order parameters undergo change almost simultaneously over training, and thus argue for a shared underlying mechanism controlling the learning phases described by them.", "questions": "- Is the fact that the NT-kernel measure in this work is defined across different spatial points of the same image (rather than different examples/images, as in a typical setting) the cause for it to have negative eigenvalues?\n- Which of the order parameters from Figure 3 would correspond to depicting the onset of diffusion?\n- How does AUC best capture alignment and “edge affinity,” in particular? Furthermore, why is this metric chosen over other alternatives?\n- While the authors note that the Gaussian NTK approximation holds “anytime away from a phase transition,” its applicability to generative regression tasks is claimed but not supported by empirical or theoretical evidence.\n- The finding that the NTK “is conveying information about image edges” raises the question: why edges specifically, and how would the framework adapt to other salient structures or modalities?", "limitations": "See Weaknesses & Questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- $\\textbf{Strengths}$:\n  - The authors consider the non-trivial case of SIREN models, moving beyond simpler 2-layer or DLNs considered in prior work, in their analysis\n  - Under an isotropy assumption (on the data), the authors demonstrate that a diffusion equation describes the evolution of the residual\n  - Under practical conditions, where the above assumption does not apply, the authors derive a closed-form expression for the correlation length, which acts as an order parameter to describe the onset of diffusion.\n  - Similarly, the authors provide closed-form expressions for the other two order parameters, namely the top-eigenvector of the NTK and the minimum value of the related cosine-NTK\n  - The authors observe that the critical epoch occurrence can be preponed by:\n    - By increasing network depth\n    - By decreasing $\\omega_0$\n  - The authors observe that:\n    - Lowering $\\omega_0$ causes gradient-energy hotspots to concentrate sharply on true image edges\n    - $\\log\\|\\nabla_θf(x)\\|^2$ and $v_0(x)$ show high spatial correlation with $\\|\\nabla_x I(x)\\|$, confirming that edge pixels drive NTK alignment\n    - Deeper models maintain richer, more distributed gradient features before focusing, enabling better utilization of model capacity.\n\n- $\\textbf{Weaknesses}$:\n  - Generally speaking, the clarity, as well as notational inconsistencies, of the paper can be improved.\n  - Key approximations are stated, but intermediate algebra is too often relegated to appendices. Hence, the reader can struggle to bridge the gap from Eq. (6) / (7) ⟶ (15).  \n  - Reliance on a second-order Taylor expansion in neighborhood distance presumes small perturbations; the boundary of this approximation’s validity vis-à-vis real image data is not discussed. \n  - MAG-Ma is introduced without sufficient theoretical motivation or linkage to prior stationarity-breakdown measures, leaving its relevance and novelty in question.\n  - It is unclear what additional information Eq. 12 provides about the loss gradient beyond what is already encapsulated by the order parameter $\\sigma(\\theta)$. \n  - Basing full-batch gradient-flow experiments on only 30 samples raises questions about the statistical significance and the generalizability of the conclusions drawn from the results.\n  - The caption for Figure 3 suggests that order-parameter trajectories were studied on a single image. If true, it precludes assessment of variance or significance across a representative dataset.\n  - A fixed hidden-layer width of 256 may not suffice to enter the NTK regime; without width-scaling experiments, it’s unclear whether the empirical setup aligns with the asymptotic NTK assumptions.\n  - It is unclear what the alignment between the theoretical predictions in this article and that of a practical training regime (away from gradient flow dynamics, e.g., SGD) would be.\n\n  - $\\textbf{Notation}$:\n    - C_{NTK}: Previous literature (e.g., Arora et al., 2019) uses CNTK to refer to Convolutional NTKs. The notation used here, while slightly different in appearance, can be confused with that. \n      - On line 131, you again use Cos NTK\n    - Line 116:  ξ should be ξcorr\n    - Line 179:  minimum of 21 should be minimum of Eq. 12\n    - Line 187: The reference for Centred Kernel Alignment is not provided", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and the specificity of their questions.\n### Response to Weaknesses\n\nBefore we answer the reviewer's questions, we would like to address some of the concerns raised re weaknesses.\n\n> The caption for Figure 3 suggests that order-parameter trajectories were studied on a single image. If true, it precludes assessment of variance or significance across a representative dataset.\n\nEach order parameter trajectory is indeed obtained from a single training run ie a pairing of an image (the dataset) and a model architecture.  To clarify, it is not possible to obtain a single order parameter trajectory from multiple images because our models are trained on only one image at a time (this is the setup commonly used in constructing implicit neural representations of images).  \n\nThat said, there are 3600 runs total, corresponding to the experimental sweep described in Section 4 and more detail in Appendix B.2.  However, direct comparison between trajectories is difficult: the scale of the order parameters and the timing of the phase transitions varies quite significantly as a function of model and dataset.  An example of this variation can be seen in Figure 13, where multiple order parameter trajectories are shown together in the same plot.  \n\nHowever, **this does not preclude** examination across datasets/architectures:\n1. Within a given run, we can look for the intersection rate of the confidence regions for the different phase transitions.  The result of this is the heatmap shown in Figure 3, whereby we see the remarkable coincidence between the different phase transitions. \n2. When we groupby hyperparameters such as depth and $\\omega_0$, this variation becomes more tame, and it becomes possible to compare statistics between runs.  An example of this analysis is found in Section 4.1, with the expectation and variance across runs shown in Table 1.  \n\n> It is unclear what additional information Eq. 12 provides about the loss gradient beyond what is already encapsulated by the order parameter σ(θ).\n\nWe politely ask the reviewer to clarify this question.  $\\sigma(\\theta)$ describes the loss gradient variance and Eq 12 is the point-wise local structure of the NTK.  The latter is a significantly richer object - is the reviewer perhaps referring to another equation?\n\n> Reliance on a second-order Taylor expansion in neighborhood distance presumes small perturbations; the boundary of this approximation’s validity vis-à-vis real image data is not discussed.\n\nWe thank the reviewer for highlighting this point.  It was discussed in the paper, but we agree it could me made more explicit.\n\nAs shown in Equation 8 and discussed throughout Section 3.1, the relevant lengthscale to consider when performing the Taylor expansion is the correlation lengthscale.  Typical values for the correlation lengthscale can be seen in Figure 2, Table 1, and Figure 17 of the Appendices.\n\nWe believe the best place to make this clearer is in the snippet before Eq 9.  In particular, the camera-ready version will say something to the effect of:\n\n*An exponentially-decaying NTK will suppress all contributions to $\\dot{r}(x)$ from residuals $r(x+u)$ for which $||u|| \\gg \\xi(x)$.  When $\\xi(x)$ is small, only the local neighbourhood of $r(x)$ influences its dynamics, justifying a Taylor expansion around the point $x$...*\n\n> MAG-Ma is introduced without sufficient theoretical motivation or linkage to prior stationarity-breakdown measures, leaving its relevance and novelty in question.\n\nWe are unaware of any studies that track the dynamical breakdown of stationarity in the NTK, and would be grateful if the reviewer could point us towards a reference.  \n\nOur theoretical justification for MAG-Ma is alluded to at the end of Section 3.3, and described in much greater detail in Appendix E.2.  In stat physics, order parameters are often derived as a measure of symmetry violation, with the phase transition heralding a symmetry breaking.  In this work, we showed how various order parameters for SIREN training dynamics could approximated in terms of the local structure of the NTK.  This local structure is in turn determined by the spatial gradients of the parameter magnitudes, ie. $\\nabla_x||\\nabla_{\\theta}f||^2$.  Hence, it makes sense to consider translation symmetry (ie stationarity) of the NTK to be the symmetry that is broken at the phase transition.  MAG-Ma is derived to measure this violation in symmetry explicitly.  We include it for the sake of completeness, and indeed find that there is a sudden shift in its value near other phase transitions.  \n\nWe do not consider this conclusive evidence of a shared underlying mechanism, but it is interesting, especially in connection with edge/corner detectors in computer vision.  Here, translation symmetry plays a crucial role in the identification of image features by studying auto-correlation functions.  This connection is explored more in Section E.1.\n\n> A fixed hidden-layer width of 256 may not suffice to enter the NTK regime; without width-scaling experiments, it’s unclear whether the empirical setup aligns with the asymptotic NTK assumptions.\n\nThis paper does not focus on the asymptotic NTK regime (i.e., the lazy-learning regime of infinitely wide networks), but rather on the practical (ie finite-width) settings where NTK alignment is commonly observed.  As discussed in the Introduction and Section 6 (Related Work), the asymptotic NTK regime is a poor approximation for studying SIREN models and, more broadly, for understanding real-world neural networks, where NTK alignment evolves dynamically instead of remaining fixed.  Furthermore, prior work (e.g., [15,16,17,18,19,20]) demonstrates that this dynamic NTK evolution is a key factor in the superior performance of real-world DNNs compared to their infinite-width counterparts, as it reflects feature learning rather than mere kernel behavior. Our investigation specifically aims to (1) characterize the timing of this dynamic alignment by framing it as a phase transition and (2) explore the patterns learned in implicit neural representation tasks.\n\n>  It is unclear what the alignment between the theoretical predictions in this article and that of a practical training regime (away from gradient flow dynamics, e.g., SGD) would be.\n\nThis is a limitation common to other works on NTK alignment (ex [21,22,24,25,26]).  We agree that it is worth investigating, but it is out of scope for the present work.  We retained the same setup from previous works but generalized the analysis to more complex models.\n\n### Response to Questions\n\n> Is the fact that the NT-kernel measure in this work is defined across different spatial points of the same image (rather than different examples/images, as in a typical setting) the cause for it to have negative eigenvalues?\n\nThe NTK is PSD by construction, hence there are never negative eigenvalues.  \n\nThe source of this misunderstanding is likely the beginning of section 3.2, where we described the Gaussian approx as positive definite ie the $K(x_i, x_i) > 0$ regardless of the points $(x_i, x_j)$ studied.  Empirically, however, gradients become unaligned, so that we may have points where $K(x_i, x_j) < 0$.  This is easy to correct for the camera ready revision.\n\n> Which of the order parameters from Figure 3 would correspond to depicting the onset of diffusion?\n\nThe relevant order parameter is $K_{\\infty}$, as defined on line 118. When this decays to zero, it signifies the decay of long-range interactions, and thus the dominance of local interactions.  We show that such interactions can be modelled via a diffusion equation.  Trajectories can be seen on the bottom of Figure 3.\n\n> How does AUC best capture alignment and “edge affinity,” in particular? Furthermore, why is this metric chosen over other alternatives?\n\nOur reasoning is summarized briefly in Section 3.3, insofar as it is insensitive to monotonic transformations of the principal eigenvector $|v_0|$.  \n\nIn more detail, this insensitivity is important because we want to study the alignment across different models and datasets (which all have very different distributions for the gradient magnitudes $||\\nabla_x I||$).  We settled on using a canny edge detector as providing a \"ground truth\" definition of 'edginess' of the point $x$ as a binary label.  Treating $|v_0(x)|$ as a score for $x$, the AUC enables us to measure the utility of $|v_0(x)|$ in predicting edge membership (without needing to choose a threshold).  \n\n> While the authors note that the Gaussian NTK approximation holds “anytime away from a phase transition,” its applicability to generative regression tasks is claimed but not supported by empirical or theoretical evidence.\n\nWe politely ask the reviewer for clarification on this question.  We never claimed to study generative regression tasks, only supervised learning of INRs.  \n\n> The finding that the NTK “is conveying information about image edges” raises the question: why edges specifically, and how would the framework adapt to other salient structures or modalities?\n\nWe investigate this question at the end of Section 3.3, and in Appendix E.1.  In short, there are similarities between the NTK and the auto-correlation structure tensors used in computer vision for edge/corner detection.  Such detectors are based on the idea that edges/corners are more \"complex\" than flat regions, which is formalized through violations of translation symmetry/local variance.  \n\nIn DNN learning, higher local variance may cause DNNs to treat edges as \"noisy datapoints.\"  Hence gradient energy accumulates at the points, which (via Eq 20), increases the prominence of such points in the principal eigenvector $v_0$.  We expect that other salient structures or modalities for other tasks should show similar behaviour: if a point has higher local-variance, the gradient magnitudes will diverge from its neighbours, and thus become more prominent in $v_0$."}, {"review_id": "M0UVQIDuma", "reviewer": "Reviewer_RheY", "summary": "The paper proposes a unifying, Neural-Tangent-Kernel (NTK)-based framework for constructing order parameters that make the dynamics of deep image-regression models (SIRENs) more interpretable. Analytically, it derives a local Gaussian/Cauchy approximation of the NTK, then shows how three apparently distinct phase transitions during training — (i) emergence of diffusion-like wave-crests in residuals, (ii) collapse of the loss-rate slope, and (iii) sudden NTK alignment with image edges. Empirically, the study demonstrates that the critical points for these transitions cluster in time and show that their relation over the network depth and the sinusoidal bandwidth hyper-parameter $\\omega_0$.", "questions": "- What happens If each layer adopts its own bandwidth $\\omega_0$ ?  The corresponding results will be dramatically changed ? \n- Can the Gaussian + Cauchy NTK approach generalize to 3-D radiance-field tasks ?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "##  Strengths\n- Rigorous, closed-form approximation of the local SIREN NTK, leading to explicit expressions for correlation length, minimal cosine NTK, and principal eigenvector.\n- First report of diffusion-like wave-crests in INR residuals, linked analytically to short-range NTK structure.\n- Clear practical insight: deeper networks or larger $\\omega_0$ postpone critical points and improve super-resolution accuracy, whereas small $\\omega_0$ can cause premature convergence.\n\n## Weakness\n- Scope is restricted to full-batch gradient descent on 2-D grayscale super-resolution.\n- Computing NTK spectra for 256×256 evaluation grids is expensive; runtime and memory requirements are not quantified.", "was_revised": true, "final_justification": "The rebuttal clearly explains why per-layer bandwidth mainly matters in the first layer and gives a reasonable path for extending the NTK analysis to 3-D radiance fields; this resolves my main conceptual concerns. The theory is solid and the clarifications are credible, but empirical support remains thin. On balance, I recommend borderline accept.", "rebuttal": "We thank the reviewer for taking the time to read out paper and leaving thoughtful suggestions.  To address your questions:\n\n> What happens If each layer adopts its own bandwidth ω0 ? The corresponding results will be dramatically changed ?\n\nThis is a subtle question.  As [30] (the Fourier Features paper) shows, MLPs have specific difficulty learning high-frequency functions from low-dimensional inputs. The authors specifically highlight how the NTK is poorly conditioned in low-dimensional spaces, and how fourier feature mappings can help alleviate this.  The intermediate layers, by contrast, are mapping between high dimensional latent spaces, for which the spectral bias isn't so damning.  The SIREN architecture (used in this paper) can be seen as a learnable fourier mapping, whose conditioning is controlled by the bandwidth $\\omega_0$.  As such, we expect that tuning the $\\omega$ of the intermediate layers would have significantly less impact than tuning the first layer bandwidth.\n\n> Can the Gaussian + Cauchy NTK approach generalize to 3-D radiance-field tasks ?\n\nWe expect that our results should generalise, all else being equal.\n\n3D radiance field tasks take in low-dimensional inputs (here, 3D position + view direction) and predict RGB$\\sigma$ values through mean-squared error (MSE) minimization.  This setup is very similar to the grayscale INR case considered in this paper, with two notable differences:\n- The outputs for 3D radiance tasks are higher-dimensional, but the MSE criterion is still used.  Thus, via the chain rule, little changes in Eq 3 save that the matrix equation becomes a matrix equation between flattened matrices.\n- The inputs for the 3D-radiance task are higher-dimensional.  In this case, the most crucial question has to do with the correlation lengthscale.  If we still see rapid decay in $K_{NTK}(x,x+u)$ for separation distance $||u||$, then all of our derivations should hold, since they are principally based on Taylor expansions justified by the locality of the NTK.  To that end, we expect that our results should generalize.  Refrence [31] shows that, at initialization in the large width limit, the SIREN NTK decays like a sinc function.  However, experiments will need to confirm whether this result holds across training as it does for our case."}, {"review_id": "IB489t2rhj", "reviewer": "Reviewer_TPRi", "summary": "The paper studies the NTK alignment for image regression models a more complicated non-linear training dynamics problem. They find that the new observation of wave patterns simultaneous occurs with two other phase transitions that of the loss rate collapse and NTK alignment. Moreover, the presence of these phase transitions is both derived theoretically and illustrated in experiments. The evaluation is done by tracking the corresponding order parameters and visualizations of the NTK at several time steps.", "questions": "Does the onset of wavecrests not appear when the residuals are small? Therefore, they would directly coincide with the decaying loss.\n\nEq.(10) has a Laplacian operator and and a time derivative in it with respect to which variables are they? Given that previously theta seems to evolve with time and that the Laplacian is with respect to x it seems that it is not a real diffusion equation. Could the authors please clarify this?\n\nWhat does the wavecrest effect imply in general? Is the effect desirable, neutral or bad?\n\nIn Figure 1a, what is the scaling of each plot?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\nThe motivation is clear and the study of NTK alignment for image regression is well founded.\nThe new observed phase transition in the residual is interesting and to the best of my knowledge novel.\nThe experimental validations supports the new derived observation of the wavecrests. \n\nWeakness:\nThe main theoretical results are not in a theorem-proof format. This makes it harder to distinguish the contributions from previous known results. Moreover, it strongly decreases the readability and rigorousness of the manuscript. \n\nThe occurrence of the transitions at the same time-scale indicates that they are the same phase transition but different effects. Moreover, the significance of the wavecrest effect is unclear. \n\nFurthermore, in the relate work there are only two recent papers cited from 2024 and 2023. For example relevant NTK literature might be:\n\nYu, Zixiong et al. “Divergence of Empirical Neural Tangent Kernel in Classification Problems.” ArXiv abs/2504.11130 (2025): n. pag. \n\nKhalafi, S., Sihag, S. &amp; Ribeiro, A.. (2024). Neural Tangent Kernels Motivate Cross-Covariance Graphs in Neural Networks. <i>Proceedings of the 41st International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 235:23577-23621 Available from https://proceedings.mlr.press/v235/khalafi24a.html.\n\nThe work holds promise but in its current form falls short of the conference' standards. Improving the structure of the theoretical contributions is key.\n\nMiscellaneous: typos and style\n\nTable1: the word hyperparamters\n\nSame line 82\n\nline 179: 21 should be (21)\n\nThe hyperparameters mentioned in the text for Figure 4 mismatch the ones in the plot.\n\nAlmost all formulas are numbered this is unnecessary.", "was_revised": true, "final_justification": "The clarifications of the authors where helpful to understand the nature of their contributions better. This eases my initial concern with the structure of the paper. However, I believe it still can be improved and the authors have made concrete plans for doing so. To summarize their contribution is interesting and it is possible to follow their main storyline. Therefore, I have updated my score to borderline accept.", "rebuttal": "We thank the reviewer for taking the time to read out paper and leaving thoughtful suggestions.\n\n### Concerning Paper Organization\n\nWe acknowledge your feedback regarding the lack of a ‘theorem-proof’ structure for the theoretical results, and recognize that it can help us distinguish our contributions.  Our approach is primarily phenomenological, focusing on deriving approximate closed-form expressions based on empirical observations and validating them experimentally. While we did not conceptualize these results as formal proofs, we did include derivations and detailed justifications in the appendix. \n\nWe understand that presenting these results in a theorem-proof format may enhance clarity and rigor, and we would be happy to adopt this structure in a revision.  This seems like a minor structural change that could be accomplished without major re-structuring.  Unfortunately we cannot upload revisions during this rebuttal, but as some examples:\n- In Section 3.1, we would merge equations (6,7,8) into a Proposition block, as these are import ansatz/approximations.  Eq 10 would then be wrapped inside a theorem block.  The subsequent Proof, which would occupt the remainder of Section 3.1, would still be a high-level sketch due to constraints.  However the more rigorous derivation contained in Appendix A.2 could certainly be formatted using this structure.\n- The other important theorem block would wrap around our Cauchy Approximation in Eq 12, and a subsequent proof would be offered.  Equations 15, 20, and 21 would be labelled as Corollaries, since we derive them as consequences of the main theorem.\n\nIn summary, we believe, for the camera-ready version, the theorem-proof structure could be adopted via (1) minor formatting adjustments and (2) minor shuffling within the sections.  The overall structure of the paper, the results, figures, and contributions, would be otherwise unchanged.\n### Concerning the significance of wavecrests\n\nWe understand that many of your concerns primarily focus on the wavecrest phenomenon.  Notably:\n\n> The occurrence of the transitions at the same time-scale indicates that they are the same phase transition but different effects. Moreover, the significance of the wavecrest effect is unclear...Does the onset of wavecrests not appear when the residuals are small? Therefore, they would directly coincide with the decaying loss.\n\nThe reviewer is correct in noting that the co-occurence of the phase transitions is indeed suggestive of a shared underlying mechanism.  However, it is not quite so simple as saying that the wavecrests appear when the residuals are \"small.\" While we cannot include figures in this rebuttal, if one were to plot $\\mu_r$ (the mean residual) as a function of time for the experimental run in Figure 3 (left-hand side), you would see the residuals become \"small\" around $t=32$ epochs, though the phase transition for the loss rate (after which we see wavecrests) occurs near $t=324$ epochs.  \n\nTo understand why this happens, we must look at Eq 35 for the evolution of the residuals.  We obtain this equation when we insert our Gaussian Ansatz for the NTK (Eq 8) into the residual evolution equation (Eq 3).  Eq 35 shows a competition of two terms:\n1. The \"local term,\" corresponding to the interactions between grid points and their close neighbours (defined by a correlation lengthscale $\\xi_{corr}$).\n2. The \"long range term,\" corresponding to the product of the mean residual, the local gradient magnitudes, and the asymptotic value of the Cosine NTK.  \n\nThe appearance of wavecrests is assosciated with the vanishing of this long range term: it swells close to the critical point as gradient magnitudes swell and then decays once more (right subfigure in the provided plot).  Stated another way, **the appearance of wavecrests is a visual hallmark of the dominance of local interactions in the evolution of the residuals.**  \n\nThis is significant because it justifies our focus on the local structure of the NTK when constructing (1) our approximation of the principal eigenvector (Eq 20), and (2) our approximations for the minimum value of the Cos NTK (Eq 21).  \n\nTo answer your other question:\n\n> Is the effect desirable, neutral or bad?\n\nStrictly speaking the effect is neither bad nor good.  In this paper we focus on the timing of the transition, and how it aligns with the timing of other phase transitions.  Phase transitions in GD dynamics shed light onto the effective capacity of a model: given two architectures with the same capacity on the same dataset, if one transitions earlier, it likely converges earlier, and that might be indicative of under-fitting (which we explore in Section 4.1).\n### Concerning the Laplacian in Eq 10\n\nIn response to your question:\n\n> Eq.(10) has a Laplacian operator and and a time derivative in it with respect to which variables are they? Given that previously theta seems to evolve with time and that the Laplacian is with respect to x it seems that it is not a real diffusion equation. Could the authors please clarify this?\n\nThe Laplacian is with respect to the spatial coordinate $x$.  This is stated explicitly in Appendix A.2 but this is an easy notational matter to correct in the camera-ready version of Eq 10.\n\nWhile the reviewer is correct in stating that parameters $\\theta$ are the dynamical variable, these dynamics are very high-dimensional and difficult to analyze.  However, the NTK provides an alternative perspective on gradient descent (GD) dynamics.  In particular, if we take a continuum limit in both the time (gradient flow) and the data, we end up with field equations for the evolution of the residuals.\n\nMore explicitly, the residual 'field' is $r(x,t) = I(x) - f(x;\\theta(t))$, where $I(x)$ is the image intensity at the coordinate $x$, and $f(x;\\theta(t))$ denotes the output of the model at the point $x$ given the parameters $\\theta$ at time $t$.  Under the NTK, the evolution of the residual field looks like (per Eq 3):\n$$\n\\dot{r}(x,t) = -\\int du\\; r(x+u,t) K_{NTK}(x,x+u;\\theta(t))\n$$\nIn our work, we propose phenomenological approximations for the structure of $K_{NTK}$ based on experimental observation.  One such model is a Gaussian with spatially-varying correlation length (Eq 8), motivated by the rapid decay of correlation-functions with separation distance $||u||$.  \n\nBecause the NTK is now approximated as a Gaussian, the evolution equation above can be explicitly integrated.  To do this, we approximate $r(x+u,t)$ via a second-order Taylor expansion in the separation $u$ (Eq 9), which is an expansion in the spatial derivatives of $r(x+u)$.  The full details are in appendix A.2.  The Laplacian term emerges because it is the trace of the Hessian $\\nabla_x^2 r$ in the Taylor expansion (Eq 19).\n\nIn summary, because the NTK strongly favours local interactions, the first and second order spatial gradients of $r$ tell us a lot about how it evolves in time, because these describe the local structure of the residuals.  We think this can be easily clarified using the suggested revision we shared with Reviewer Myuw:\n\n*An exponentially-decaying NTK will suppress all contributions to $\\dot{r}(x)$ from residuals $r(x+u)$ for which $||u|| \\gg \\xi(x)$.  When $\\xi(x)$ is small, only the local neighbourhood of $r(x)$ influences its dynamics, justifying a Taylor expansion around the point $x$...*\n\n### Concerning Related Works\n\nWe thank the reviewer for directing us to the two recent NTK works.  While the work on the divergence of the NTK is rather distinct from our own, the work on Cross-Covariance Graphs further reinforces the role that NTK alignment has in understanding convergence in GD (indeed, the paper's theoretical contributions shows that the convergence of GD is positively correlated with their alignment metric).  However, in the referenced paper, the goal is to directly maximize alignment, whereas in our investigation, we study how, when, and why NTK alignment happens *spontaneously* in GD without direction supervision.  We will add a sentence to our Related Works section for the camera-ready version that highlights this distinction.  Something to the effect of (at the end of Section 6):\n\n*...contributing to the superior performance of DNNs over models based on infinite-width NTKs [26].  **Direct optimisation of alignment measures has even been suggested as one way to enhance the convergence of GD [cite the provided work and others].**  That said, theoretical investigation into spontaneous NTKA often focus on shallow networks...*"}]

Author rebuttal:

Meta-review: The paper presents  theoretical and empirical analysis of training dynamics in SIRENs. Its novel observation of wave patterns in residuals and the connection to NTK alignment and loss rate collapse offers interesting insight. The authors' rigorous rebuttals effectively addressed reviewer concerns regarding scope, methodology, and theoretical structure, solidifying the paper's contributions and making it a candidate for acceptance.
Decision: Accept (poster)