Empirical Limitations of the NTK for Understanding Scaling Laws in Deep Learning
Authors: Nikhil Vyas
Yamini Bansal
Preetum Nakkiran
Venue: Accepted by TMLR ()

Abstract: The ``Neural Tangent Kernel'' (NTK) (Jacot et al 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks. In this work, we study NTKs through the lens of scaling laws, and demonstrate that they fall short of explaining important aspects of neural network generalization. In particular, we demonstrate realistic settings where finite-width neural networks have significantly better data scaling exponents as compared to their corresponding empirical and infinite NTKs at initialization. This reveals a more fundamental difference between the real networks and NTKs, beyond just a few percentage points of test accuracy. Further, we show that even if the empirical NTK is allowed to be pre-trained on a constant number of samples, the kernel scaling does not catch up to the neural network scaling. Finally, we show that the empirical NTK continues to evolve throughout most of the training, in contrast with prior work which suggests that it stabilizes after a few epochs of training. Altogether, our work establishes concrete limitations of the NTK approach in understanding scaling laws of real networks on natural datasets.

Published in Transactions on Machine Learning Research (MM/YYY)
Empirical Limitations of the NTK for Understanding Scaling Laws in Deep Learning
Nikhil Vyas Harvord University
nikhil@ghaaruan edu
Yamini Bansal
ybansal@kgoogle.com
Google DeepMind
Preetum Nakkiran
preetum@nakkirnnnco
University of California, San Diego'
Reviewed on OpenReview: https: / openreview. /ttp::///penreevee
net/ forum? id= XXXX
Abstract
The "Neural Tangent Kernel" (NTK) (Jacol et al, 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks. Tn this work, we study NTKs through the lens of scaling laws, and demonstrate that they fall short of explaining important aspects of neural network generalization. In particular, demonstrate realistic settings where finite-width neural networks have significantly better data scaling exponents as compared to their corresponding empirical and infinite NTKs at initialization. This reveals a more fundamental difference between the real networks and NTKs, beyond just a few percentage points of test accuracy. Further, we show that even the empirical NTK is allowed to be pre-trained on a constant number of samples, the kernel scaling does not catch up to the neural network scaling. Finally, we show that the empirical NTK continues to evolve throughout most of the training, in contrast with prior work which suggests that it stabilizes after few epochs of training. Altogether, our work establishes concrete limitations of the NTK approach in understanding scaling laws of real networks on natural datasets.
1
Introduction
The seminal work of Jacot et Jacot et al. (2018) introduced the "Neural Tangent Kernel" (NTK) as the limit of neural networks with widths approaching infinity. Since this limit holds provably under certain initializations, and kernels
more amenable to analysis than neural networks, the NTK promises to be useful reduction to understand deep learning. Thus, it has initiated rich research program to use the NTK to explain various bchaviors of neural networks, such as convergence to global minima (Du et al. 2018 2019), good generalization performance (Allen-Zhu el al. 2018 Arora el al. 2019a), implicit bias of networks (Tancik et al. 2020) as well as neural scaling laws (Bahri et al. 2021).
In addition the infinite NTK, the empirical NTK the kernel with features that are gradients of a finite-width neural network- can be a useful object to study, since it is an approximation both the true neural network and the infinite NTK. This has also been studied extensively
tool to understand deep
learning (Fott et al.) 2020 Long
2021
Paccolat et al. 2021 Ortiz-Jiménez et al. 2021
In this work, we probe the upper limits of this research program: we want to understand the extent to which understanding NTKs (empirical and infinite) can teach us about the success of neural networks. We
"ook doue while at Harvard University. Work done while at UCSD; currently Apple.

---

Published in Transactions on Machine Learning Research (MM/YYYY)
After-kernel trained with varying samples
Empirical NTK trained with varying time
NTK at initialization
Nearal Network Emppiicall NTK
(2000
0.335
Intrate NTK
Test Errr
0.09
Test Erroe
0.09
Tost Error
0.05
Neural network has a higher scaling exponent
0.04
A
1000
4000
Train size
10000
84000
B
Dataset size used to train the kernel
200k
C
0.04
10
Number train batches
Iel
10
Figure 1: Summary of results: (A) Neural network scales better than NTK at imitialization: We compare the scaling exponent of neural network, its corresponding infinite and empirical NTK at initialization. Details in Section 3 (B) After-kernel continues to improve with more training samples: We train a neural network with m = [1K 2K., (OK...024K] samples, extract the empirical NTK at completion, and use this kernel to fit 500 samples. Details in Section 4 (C) Empirical NTK improves with constant rate with respect to
training time: We extract the empirical NTK various times in training and use to fit the full train dataset. Details in Section 5
study this question under the lens of scaling (Kaplan et al. 2020 Rosenfeld et al., 2019 how performance improves as a function of samples and as a function of time since the scaling is an important "signature" of the mechanisms underlying any learning algorithm. We thus compare the scaling of real networks to the scaling of NTKs in the following ways.
1. Dala scaling of inilial kernel (Seclion 3/: We show that both the infinite and empirical NTK (at initialization) can have worse data scaling exponents than neural networks, in realistic settings (Figure D. We find that this is robust to varions important hyperparameter changes such as learning rate (in the range used in practice), batchsize and optimization method.
2. Width scaling of initial kernel (Section 3): Since neural networks provably converge to the NTK at infinite width, we investigate why the scaling behavior differs at finite width. We show (Fig- ure [(b] [[c]] realistic settings where as the width of the neural network increases to very large values, the test performance of the network gets worse and approaches the performance of the infinite NTK, unlike existing results in literature which suggest that increasing the width in the pwerpparameterized regime is always good. This also raises new questions about scaling of neural networks with width, in particular the "variance-lmmitad" neural scaling regimes (Bahri et al. 2021
3. Data scaling of after-kernel (Section ::We consider the after-kernel (Long) 2021 ) i.c. empirical NTK extracted after training to completion on fixed number of samples. We show (Figure B)) [(b) that the after-kernel continues to improve as we increase the training dataset size. On the other hand, we find (Figure 4(c) that the scaling exponent of the after-kernel, extracted after training a fixed number of samples remains worse than that of the corresponding neural network.
4. Time scaling (Section 5): We show (Figure [C),
realistic settings where the empirical NTK continues to improve uniformly throughout most of the training. This is in contrast with prior
5(a)
work (Fott et al. 2020 Ortiz-Jiménez et al. 2021 Atanasov et al. 2021 Long 2021) which suggests that the empirical NTK changes rapidly in the beginning of the training followed by a slowing of this change.
We demonstrate these phenomena occur in certain settings which are based on real, non-synthetic data, and modern architectures (for e.g.: for datasets CIFAR-10 and SVHN and convolutional networks). While we
2

---

Published in Transactions on Machine Learning Research (MM/YYYY)
do not claim that these phenomena manifest for all possible datasets and architectures, we believe that our examples highlight important limitations to the use of NTK to understand the test performance of neural networks. Formalizing the set of distributions or architectures for which these phenomenon occur is an important direction for future theoretical research.
1.1 Comparison to Prior Work on NTK Generalization
Our main focus to understand feature learning occurring due to finite width. To do this, we make the following deliberate choicos in all of our experiments: a) We use the NTK paramcterization, this makes sure that infinite width networks will be equivalent to kernels b) We use the same optimization setup for the neural network, empirical NTK and Infinite NTK, this makes sure that as width tends to infinity all 3 models will have the same limit. We make sure that our comparisons are robust by c) using scaling laws to compare these models and d) doing various hyperparameter ablations (Figure
Below we describe several lines of related works and how our work differs from them.
Small initialization and representation learning at infinite width. Infinite widths neural networks
in the NTK and standard initialization are equivalent to kernels (Jacot et al., 2018
Yang & Hu 2021). On
the other hand it has been shown YYang& & Hu 2021 Sirignano & Spiliopoulos
2019 Nguyen & Pham 2020:
Araújo et al. 2019 Fang et al.) 2020) that with small initialization feature learning is possible at infinite width. The feature learning displayed in our experiments is not due to small initialization as we initialize our networks in the NTK parameterization. This was a deliberate choice as we are interested in feature learning occurring due to finite width as this the kind of feature learning displayed by empirical neural networks (which usually do not have a small initialization).
Data Scaling for NTKs and neural networks. Scaling laws have been empirically shown Kaplan 20200; Rosenfeld et al. (2019) for neural networks and have been theoretically proven [Boddclon et al. ct
al.
: Canatar et. al. 2021g; Bordelon & Pehlevan (2022) for NTKs under natural assumptions. Comparison between the scaling laws for neural network and empirical NTKs has been previous looked at by Paccolat et al. 2021h and [Orti-JJiénnee et al. (2021) and both find that neural networks have better scaling than empirical NTK at initialization. Both of these papers do not compare to infinite NTKs which leaves open the possibility that neural networks and infinite width NTKs behave the same wrt their scaling constants.
2020
Pointwise comparisons of neural networks and corresponding infinite NTKs has also been studied extensively in the literature (Arora et al., 2019b Lee et al., 2020 Simon et al. 2021 but the results have been divided. As discussed earlier we focus on comparing scaling laws. We argue that scaling laws, instead of point-wise comparisons, are the appropriate tool to compare neural networks and NTKs. Practically, pointwise comparisons between any two models can be fraught with issues as the ordering can flip depending on dataset size, as well as the specific choice of hyperparameters. On the other hand, scaling exponents have been found to be more robust to the choice of hyperparamcters (Bansal et al. 2022 Kaplan et al.) 2020). More importantly, the claim that NTK can capture 'most" of the performance of the neural network can be subjective, specially when we are comparing small error or loss values. We show that when we look closely at the scaling exponents of these objects instead, we find major differences.
Theoretical studied effects of finite width with respect to the NTK regime. Finite width cor- rections to the NTK theory have been studied by Andreassen & Dyer (2020); Roberts et al. (2021); Bahri et al. p202I.. While these results do not need infinite widths they still require much higher than practically used widths particularly for the training sizes used in practice. These papers either consider a) the finite width corrections of empirical NTK or b) they consider the change in NTK but predict that the higher order analogues of empirical NTK remain constant. For a) we show that the empirical NTK is very far from the performance of finite width neural networks. Regarding b), in Appendix D we show that the higher order analogues of empirical NTK change significantly.
After Kernel and Time Dynamics We discuss these in detail in Section and
We describe other related works in Appendix G
3

---

Published in Transactions on Machine Learning Research (MM/YYY)
2 Experimental Methodology
Here we describe the common methodology used in our experiments.
The core object we want to understand is the data-scaling law of real neural networks that is, what is its asymptotic performance as a function of the number of train samples? Concretely, in this work we restrict to classification problems, where we mcasure performance
terms of test classification error. For a given classification algorithm, let L(n) be its learning curve: its expected test error as a function of number of samples n. In practice, many neural networks exhibit power-law decay in their learning curves [Kaplan ett al. 2020). In such settings, we have L(n) an and we are interested primarily in the scaling exponent 5, which determines the asymptotic rate of convergence.
Empirical and Infinite NTK Let [w, x) be a neural network with w representing the weights and x an input. By Taylor expansion around wy we have:
1 f(w, x) = (wo, 2) + V Vwf(w, (w wo) + =2(
2 w -
(ww)))" w V fw. wo (w wo) +...
Empirical NTK of the neural network around weights wo refers to the model 9i (w, x) fw x) (w) Note that this is not the same as lincarizing the network as we omit the f(wo, 2) term. Empirical NTK is linear model with respect to the weights w. Infinite NTK refers to the limit of the empirical NTK of the network around initial weights as width tends to infinity.
For given learning problem and given neural network architecture NN, we want to understand its data- scaling law (n). We consider the infinite NTK of the NN and the empirical NTK of the NN at initializa- tion and their corresponding learning curves, LNTK(1) (1) and LENTK(N) (n). Now we ask: is the scaling-expones of LNN always close to the scaling-exponee of either LENTK or LNTK, in realistic settings? That is, how well does the NTK approximation capture the generalization of real networks, on natural distributions?
Recall that this question is especially interesting because the three objects involved (Neural Network, NTK, and ENTK) all become provably equivalent in the appropriate width - oo limit. Thus, at infinite- width we know their scaling laws must be the equivalent. The question is then, how far are we from this limit in practice? Are the widths used in practice large enough for their scaling-behavior to be captured by the infinitewiddt limit? To probe these questions, we empirically study scaling laws of these methods on imsge-llasiifictionn problems.
Remark on comparisons. We intentionally only compare a neural network to its corresponding NTK, and not to other kernels. Our motivation not address the question of "can (some) kernel perform well as as given neural network?" indeed, there may be some better kernel to consider than the NTK. However, our goal is to study the specific kernels given by the NTK approximation, in correspondence with real networks.
Datascts. We use the following datasets:
1. A 2 class subset (dog, horse) of CIFAR-5m Nakkiran et al. 2021) dataset, as binary classification problem, which we denote CIFAR-5m--rr This is a dataset of synthetic but realistic 32 x 32 RGB images similar to CIFAR-10, generated using a generative model.
2. A binary classificaaion lask the SVHN datasel (Ntzer et al. 2011 with the labels being the parity of the digit, denoted by SVHN- parrity For the training data we use balanced random subset of 'train' and 'extra' partitions, for test data we use the 'test' partition.
We focus on the CiF experiments in the main body. Corresponding SVHN-parity experiments can be found in Appendix F
We use these particular datasets because we need datasets with a large number of samples in order to mcasure data-scaling, and CIFAR-ōm-bin and the SVHN datasct both have 2 600k samples. We chose consider binary tasks as this makes the kernel experiments computationally feasible. Although there are other datasets with similar sample sizes (e.g. ImageNet), the datasets we have the advantage that they
4

---

Published in Transactions on Machine Learning Research (MM/YYYY)
are low-resolution and an easier task thus, scaling-law experiments are far more computationally feasible. We also do some experiments on synthetic dataset in Appendix F
Architectures. We use the following base architectures: Myrtle CNN (Page 2018 Shankar et al. 2020
for the CIFAR-6m-bin task and a layer CNN with 64 channels for the SVHN-parity task. We consider various width scaling for these networks: For the Myrtle CNN we vary the width from 16 to 1024 and from 16 to 4096 for the 5 layer CNN. See Appendix B for more details.
Experimental Details. We describe some subtleties in the experimental setup. We use NTK parameteri- zation (Jacot et al. 2018) for both the neural network and the kernels as this is the parameterization used in proving the equivalence of neural network and NTK at infinite width. We train with MSE loss and +1 labels. We use lest error as the metric for all the plots except in Appendix | where we recreate some of the most important plots for test loss, All of our networks are in the overparameterized regime i.e. are able to reach train error. To preserve the correspondence between the neural networks, empirical NTKs and infinite NTKs we train all of them with SGD with momentum with the same hyperparameters (Appendix [.4] This also ensures that in all experiments neural networks will be trained below the critical learning rate, i.e the learning rate at which training of the empirical and infinite NTK can converge (Appendix B.3 Training for the empirical NTK is done by linearizing the initialized neural neural network using Novak et al. (2020) library while for infinite NTK we directly use SGD with momentum on the linear system given by the infinite NTK and the labels. We describe further experimental details for each individual experiment in Appendix
3 Data Scaling Laws of Neural Networks and NTKs in the Overparameterized Regime
In this section we compare the data-scaling laws of neural networks to their corresponding emperical NTKs and infinite NTKs. Our main claim is the following.
Claim 3.1. There erists a natural setting of task and network architecture such that the neural network trained with SGD has a better scaling constant than its corresponding infinite and empirical NTK at initial- isation. Further, this gap in scaling continues to hold over wide range of widths and learning rates used in practice.
The above claim can be interpreted as stating that there exists natural settings where the regime in which real neural networks are trained is meaningfully separated from the NTK regime, and real neural networks have a better scaling law.
In Figure 1 (A), we train a Myrtle CNN Page 2018 Shankar et al. 2020). its empirical NTK at initialization, and its infinite NTK on the CIFAR-5in-bl lask. In each case, we train to fit the train set with SGD and optimal early stopping. We then numerically fit scaling laws, and find, scaling-exponents ,3 of: .185 (empirical NTK), .213 (infinite NTK), .291 (neural network). Thus, in this image-classffication setting, the real neural network significantly outperforms its corresponding NTKs with respect to data-scaling. We show the istical significance of this result in Appendix A See the Appendix B for full experimental details.
We now investigate how robust this result is to changes in the width of the architecture and optimizer, within realistic bounds.
Effect of Width. We explore the effect of width. In Figure 2(a) we train neural networks with widths much smaller (16) and much larger (1024) than the width (64) used in Figure [[]. We find that these networks behaved similarly with respect to their scaling constants (.276 and .279 respectively), and performed better than the infinite width NTK (scaling constant: .213), confirming that real neural networks are far from the NTK regime. However, we know that in the infinite width limit, all these methods will perform identically. Moreover, as ment in Section 2 we are careful to ensure this limit is preserved by our optimization and initialization setup. This implies that at some point, increasing width of the real network will start to hurt performance- although it may be computationally infeasible to observe such large widths. To explore the width- dependecpn,,,,, in Figure 2(b] we plot the expected performance of empirical NTK and Neural network we increase the width, using fixed training size of 4000. Here see that (a) the empirical NTK
5

---

Published in Transactions on Machine Learning Research SMM/YYY)
CIF-A--S--äi with 4k samples
SVHIN SVHIl-parity parity with 1000 training samples
Zhe with
0.301
Text Eror
2.09
Expected Text Erer
0
5.006
Infilte NTK
Enorical NTN
Ne Network
Expected Text Erer
9.315
Infens NTK
Enpriell
2.0
0007
0.280
0.50
0.009
1000
4000
19000
64002
16
256
1024
16
1004
40%
Train size
Width
Width
(a)
(b)
(c)
On the Effect of width. In Figure 2(a) we plot data scaling laws of the Myrtle-CNN at small (16) and large (1024) widths and the infinite NTK. We observe that both finite widths have similar scaling constant which is better than that of the infinite NTK. In Figure 2(b) we plot the performance of Myrtle- CNN and its empirical NTK for a fixed training size while varying width. In Figure 2(c) we do the same for a 5-layer CNN and the SVHN-parity task. Both Figure 2(b) and 2(c) we observe that (a) the empirical NTK performance continues to improve with width, moving lowards the infinite NTK performance while (b) neural network performance improves initially and then starts to deteriorate towards the infinite NTK performance. Error bars represent estimated standard deviation. See Appendix B for more details.
Figure 2:
initialization continues improve with larger width and approaches the infinite NTK's error from above, while (b) the neural network improves initially and then starts to deteriorate and approach the infinite NTK's error from below. In Figure [(c) we repeat the experiment for the SVHN-parity task. In this setup it computationally feasible to try out much larger width (upto 4096) with smaller training size of 1000. Hence in this experiment as the width increases, we can observe stronger deterioration of the performance of neural network, lowards the infinile NTK performance.
Together these results suggest that "intermediate" widths (not too large, not too small) are important for the performance of overparameterizee neural networks, and any explanatory theory must be consistent with this.
Effect of Learning Rate. We now study how robust results are to changes in the learning rate, within practically used bounds. Note that changing the learning rate only affects the neural network training, and does not affect any of their corresponding NTKs, In Figure 3(a) we train networks in the same setting as Figure - but with varying learning rates. We find that after moderate modifications of the lcarning rate the neural network still has a better scaling law than infinite and empirical NTK at initialization, suggesting that practically used learning rates (for practically used widths) are far from the NTK regime. The scaling constants are .333, 22,,2222.22 .213 for the 3x higher learning rate, 10x lower learning rate and the infinite neural network. We discuss the effects of more drastic changes (1000x) in the learning rate in Appendix C
Other Changes in Optimization. We now study whether our results hold under other changes to optimization parameters. In Figure 3(b) 3(c) 3(d) we see the effect of doing GD instead of SGD, effect of training without momentum and of using the final test error instead of doing optimal early stopping respectively. We see that in all of these cases, while there is some change in the scaling laws, the neural network scaling constant is still always better than the one for infinite NTK. The scaling constants for neural networks in Figure 3(b] 3(c) 3(d are .294, .310 and .292 respectively. The scaling constant for the infinite NTK is .213 in Figure 3(b) 3(c) and .219 for Figure 3(d) This suggests that these optimization factors (within commonly used values) are not the fundamental reason behind the improved scaling laws of neural networks.
Various extensions to the NTK regime have been proposed (Roberts et al. 2021 Andreassen & Dyer 2020
in the literature which allow for the change in empirical NTK but posit that higher order analogues of the NTK remain constant. This would predict that higher order analogues of the empirical NTK at initialization would be sufficient. to match the performance of neural networks. Tn Appendix D we show that this is not
6

---

Published in Transactions on Machine Learning Research MYYY
1.33
HeC suppleiie lariing
MiD NTV.
Vili HiX
144
Train 104
Tran 1D1
Trals A54
Traln 104
(a)
(b)
(c)
(d)
Figure 3: Neural networks continue to have a better scaling constant under various hyperpa- rameter choices. We compare the data-scaling for a Myrtle-CNN, its empirical NTK and its infinite NTK on CIFAR-5m-bii bin task under various hyperparameter changes: (a) Higher and lower learning rate compared
to Figure [ (b) GD instead of SGD (c) SGD without momentum (d) Training until convergence (no early
stopping)
the case suggesting that these theories may also not be sufficient to explain the performance of practical neural networks.
Discussion and Future Questions. The equivalence between neural networks and corresponding NTKs applies when width >> train-size. On the other hand nearly all overparameterized networks and natural tasks fall in the regime of width < train-size (though width is still large enough to fit the dataset). The results of this section - showing scparations between neural networks in the latter regime and NTKs lead to following concrete question on the gap between theory and practice which could guide future work.
Question 3.1. How con we undersland the the behavior of overparamelerized nelworks in the width << train-size regime?
4 Exploration of After- wrt Dataset size
In the previous section, we studied the empirical NTK when linearized around weights at initialization. In this section we will study the behaviour of empirical NTK when lincarized around the weights obtained at the end of braining. This is known as the afler-kernel for empirical NTK, in the terminology of Long (202I). We will show, in the more precise sense defined below, that (1) the after-kernel continues to improves with dataset size, and thus (2) no fixed-time after-kernel is sufficient to capture the data scaling law of its corresponding neural network.
Formally, denote the after-kernel from the neural network trained m samples Km- We will denote the accuracy of Km when fit on n samples as Km(n). Here, the n samples are a subset of the original m samples. When we use fresh n samples to fit we use the notation (n). We study the as improved performance of neural networks over NTKs has been attributed (Ortiz-Jiménez et al. 2021 Atanasov et al. 2021) to the adaptation of the empirical NTK of the neural network to the task. Concretely, prior works (Long) 2021 Paccolat el al., 2021) have shown that this explanation is complete in the following sense: The behaviour of (n) is similar to that of the neural network fit on n samples. In other words, when we fit after-kernel obtained from training
samples to the same samples we get an accuracy very closep to that of the neural network fit on the samne samples. We verify this for our setup in Figure ((a) This tells us that the following two factors are sufficient to explain the behaviour of neural networks fit on n samples: (1) Change in empirical NTK from empirical NTK around initial weights to the after-kernel due to training on n samples. (2) Fitting the after-kernel on n samples.
What this does not tell us how these two improvements scale with training size n. In particular, we know that Ko(n) i.e. the empirical NTK at initialization fit on samples does not match the neural network trained on samples the other hand K, (n) does. This raises the following natural question: How data
We again note that empirical NTK does not refer the lincarization of the network (Sce Section for exact definition). If had linearized the network this statement would be trivially true as the linearized network around final weights would start out with an accuracy matching that of the trained neural network.
7

---

Published in Transactions on Machine Learning Research (MM/YYYY)
dependent does the kernel need to be to recover the performance of the neural network? For example, it is possible that for some sample size mg and all m 2 mo, the after-kernel Km is roughly constant, and has same scaling law as the neural network itself. We find that this is not the case the after kernel continuously improves with dataset size m.
4.1 Experimental Results
After-Kernel continues to improve with dataset size. In Figure [(b) we plot (500) versus m for our We observe that (500) improves as m goes from 500 to 1024k showing that the after-kernel keeps improving with larger dataset sizes. Corresponding SVHN-parity experiments can be found in Appendix
base Myrtle CNN from Figure] 1
Fixed after-kernel is not sufficient to capture neural network data scaling. In Figure 4(b) plot the data scaling curves for the base Myrtle-CNN, its empirical NTK at initialization, K16s, K64k with scaling constants 199,,,,8,,1003, .097 respectively. We find that the neural network has the best scaling constant. This shows that the scaling of the after-kernel with training size is an important component of neural network scaling laws as even the after-kernel learnt with 64k samples (on the simple CIFAR-ōm-bin task) not sufficient to explain the data scaling of neural networks. We also see that K6tk has better performance than K Kist; another evidence towards the fact that after-kernel improves with dataset size.
Kaini
K.,000
0:335
0.155
0.13
Espired NTK
Kip
2.99
Test Erer
Text Errer
Text Ereer
0.00
2.06
2.34
0.04
0.37
10
2504
1C24c
106
2004
1024c
1000
4000
10000
60000
Train Size
Datavet sine used train the knrnel
Train Size
(a)
(b)
(c)
After-Kernel continues to improve with dataset size. In Figure 4(a) we plot data scaling curves of (n), K. (m) and the neural network and observe that they behave very similarly. In Figure 4(b) we plot (500) versus m and observe that the performance improved with increasing m. In Figure [(c) we plot data scaling curves of empirical NTK at initialization, Kink, Ke4k and the neural network. We observe that the neural network has the best scaling law amongst these.
Figure 4:
5 Time Dynamics
In the previous section, we saw that the change in the empirical NTK from initialization to the end of training (the after-kernel) is sufficient to explain the improved performance of neural networks. Thus the empirical
NTK must have evolved throughout training, and in this section take a closer look at this evolution. Our main focus in this section is to investigate the following informal proposal in the literature (Fort et al. 2020:
Long 2021 about how the empirical NTK evolves:
Hypothesis 5.1 ((Informal, from Fort et al. (2020); Long (2021) ). The empiricnl NTK evolues rapidly in the beginning of training (< 5 epochs), but then undergoes "phase transition" into a slower regime.
One way to interpret the above hypothesis is that there is both qualitative and quantitative difference in the empirical NTK between the "early phase" of training (the first few epochs) and the later stage of training. This is called a "phase transition" in the literature, in analogy to physics, where systems undergo discontiniitess between two regimes with quantitatively different dynamics.
In this section we will give evidence that suggests, contrary prior work, that there is no such "phase transition". We show that if empirical NTK performance is measured at the appropriate scale, performance
8

---

Figure I Published in Transactions on Machine Learning Research (MM/YYY)
Neural Network
0.0653
K
0.135
Empirical
Test Error
R*. t 3308
0.053
K" t- 11861
Test Error
0.09
0.4
0.06
network Empirical
Test Error
0.2
0.04
0.1
10
10a
10
10
1000
4000
16000
64000
256000
Number of train batches seen
Train size
(b)
Figure 5: Empirical NTK keeps improving uniformly throughout most of the training. In Fig- ure ((a) we plot the test error of Myrtle-CNN, its empirical NTK at initialization and Ky at time t. We observe that the slope Kia docs not decrease with time suggesting that the change in kernel does not slow down after an initial part of training. Using this same setup, we plot the data scaling curves of Kt for various and the data scaling of Myrtle-CNN in Figure 5(b) We observe that the Myrtle-CNN has the best scaling law.
appears to continnously improve throughout training (from early to late stages), at approximately the same "rate." Our experiments are fact compatible with the experiments in prior work (e.g. Fort et al. (2020) ): we simply observe that if performance and time are measured on a log-log scale (as is appropriate for measuring multi-scale dynamics), then the NTK is seen improve continuously throughout most of the training.
5.1 Experiments
Setup. We now describe the setup more formally. Let Kt refer to the empirical NTK (as described in Scction 2 extracted at time in the training, where measure time in terms of number of SGD batches seen in optimization. Kj denoles the model K fit to the whole training data. Prior works (Fort et al.
2020 Long 2021) have used the slope of the curve of test error of Kf versus to decide if the kernel is changing rapidly not. We will do the same with one crucial difference: We will measure this slope log-log plot instead of directly plotting test error and time. We do this empirically scaling laws with respect to time (or tokens processed) have been observed (Kaplan et al. 2020 for natural language tasks in neural networks and formally proved for kernels (Velikanov & Yarotsky/ 2021) for natural tasks. These results suggest the need for log-log plots to observe qualitative phase transitions in training dynamics.
Results. Our main claim that the test error of Kfac as function of time t is approximately linear on a log-log scale, throughout the course of training. Recall that Ki is the model obtained by extracting the empirical NTK after batches of training the real neural network.
In Figure ((a) we compare the test error of the base Myrtle-CNN at time i, test error of empirical NTK initialization at time and Kt fie when trained 64k samples with the same hyperparameters Since we want to probe Hypothosis 5.1 which is about the beginning of training, we plot these quantities until train error reaches < 5% (which requires 32 epochs in our experiments). This should be sufficient to cover any reasonable definition of "beginning of training".
Observe that Figure 5(a) we do not observe a "phase transition" after which the improvement in kernel test error (in red) slows down. In fact, we observe that the kernel starts out being essentially constant and then starts and continues to improve uniformly.
9

---

Published in Transactions on Machine Learning Research (MM/YYYY)
We instead observe the following two regimes: (1.) In the first regime (before the dashed vertical line) the empirical NTK at initialization and the neural network have very similar behaviour, and jia is nearly constant. This only lasts for around 140 batches = 0.5 epocha] (2.) In the next regime (after the dashed line) the empirical NTK at initialization and the neural network diverge. As they diverge, the extracted kernel K2 fu also starts to improve with constant slope, and this improvement continues uniformly until the terminal stage of training.
Importantly, the kernel Kis does not transition into a "slower phase" of learning at any point in our experiments. Corresponding SVHN experiments can be found in Appendix
Next, we measure the performance of K* in termn of its data scaling law. Due to computational limitations (since measuring data-scaling is expensive), we can only measure the scaling law for several selected values of time t, instcad of every batch (as in Figure 5(a)p. In Figure 5(b) we plot data-scaling of K for t 0 (empirical NTK at initialization), 3308 (10 epochs), 11861 (32 epochs), in the same setup as Figure ((a) We also plot the data scaling of the base Myrtle-CNY CNN with the same hyperparameters. As in Figure ((c) of Section 4 we again observe that the neural network has the best scaling law, outperforming any of the extracted kernels. This shows that representations learnt after any constant time of training are not sulficient to explain the data scaling of neural networks. Rather, these representations improve throughout training, and the entire course of training must be considered to recover the correct scaling law.
References
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 74. PMLR, 2020. URL http::/ppooceeding mlr, wwest.pees/vvee v 11t.v/1111ldaammmmmm .ttll
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. arXiv preprint orXiv:1811.04911 2018.
Anders Andreassen and Ethan Dyer. Asymptotics of wide convolutional neural networks. CoRR, abs/2008.08675, 2020. URL https: http://aaxvvvv. wttp://rrrgaabb/0000000005 08675
Dyego Araújo, Roberto I. Oliveira, and Daniel Yukimura. A mean-field limit for certain deep neural networks, 2019. URL
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fino-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322 332. PMLR, 2019a.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bur Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Pro- cessing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8139-8148, 2019b. URL htttps://proceedingss ceurips.cc/papee http//aaerr/211110 12209//ahhh////
weeellleffbbbaae3cc Abstract abstract.hhtml
Alexander Atanasov, Blake Bordclon, and Congiz Pehlevan. Neural networks as kernel learners: The silent
alignment effect, 2021.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. CoRR, abs/ 2bs/2112006701, 2021. URL https:://arii httpe://arrvv.or///b/////U 06701
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in NMT: the effect of noise and architecture. CoRR, abs/ abs/2222.001994 2022. URL https://arxiv: /tttp://arriv.or//bb////e http://b/22222.9999/
Note that this means that if we plot per epoch this phase would not be visible at all. 3 As we keep training at some point train loss will tend to 0 and Kt. will converge to fixed value. This does not affect our results as are only interested in the initial part of training as described in Hypothesis 5.1
10

---

Published in Transactions on Machine Learning Research (MM/YYYY)
David G. T. Barrett and Benoit Dherin. Implicit gradient regularization. In 9th International Conference on Learning Representations, TCLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http:///ppeeneeeeeeeeee/frorrooo
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off, 2019.
Mathieu Blondel. kernel 11-15.
http:////w....ga.bo..oo////aaa/a 2012. Accessed: 2022-
Blake Bordelon and Cengiz Pehlevan. Learning curves for SGD on structured features. In International I2VVkA111
Conference on Learning Representations, 2022. URL
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning, JCML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1024-1034- PMLR, 2020. URL http::ttproceeeiinnggee mlr wttp:p/s/////////eeeeooo222///// http://poooollgll...p.epepe////p
html
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12((): 1-12, 2021.
Amit Daniely and Eran Malach. Learning parities with neural networks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annaal Conference on Newral Information Processing Systems 2020, NewrIPS 2020, December 6-1 2020, virtual, 2020. URL
hash/
eaae5e04a2259000aaa (at.://00000.a01100eeeeee7/ed-e. Absttraatddbbsrtrtt...... html
Stéphane d' d'Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfit- ting: where & why do they appear? In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL
Abstract. html
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pp. 1675-1688 PMLR, 2019.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- parameterized neural networks. arXiv preprint 1xx::111100 2..002054, 2018
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. In 8th Interna- lional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nen net., 2020. URL
Cong Fang, Jason D. Lee, Pengkun Yang, and Tong Zhang. Modeling from features: a mean-field framework for wver-araaeeteriee deep neural networks, 2020. URL https://arxiv httpaaawwivrra/aaeee...ee www.oogg//.//.ooooooooooo..ooeo http://bb0000001000 01452
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geom- etry and the time evolution of the neural tangent kernel. In Hugo Larochelle, zato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Newrul Information Processing Systems 33: Annual Conference on Newral Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URI. htttps://proceeddnggg wwwpeeedddnnnnnuuisssso.m cttp:ppp.cccpppp/220000/// has////h//0hhh/hh////// 41507669991664443317777777888888 Absb:e84tt--Atttatt..... html
Ran-
Mario Geiger, Stefano Spigler, Stéphane d' Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthicu Wyart. Jamming transition as paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100(1), Jul 2019. ISSN 2470-0053. doi: 10. 11100//pyyrree..0..0222222 URL
http://d:: doi.
http::/ot..aa..orr/0011///////11 http/10.1103/PhysReeee..ee 100.001111
11

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Maria-Florins Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http::///prceeiinnseeeriiipp heeurips.ccpppper/20000000
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in Newral Information Processing Systems 31: An- nual Conference on Neural Information. Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580 8589, 2018. URL Abstract. html
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXie preprint ar (xXv:2000.083300 2020.
Stefani Karp, Ezra Winston, Yuanzhi Li, and Aarti Singh. Local signal adaptivity: Provable feature learning in neural networks beyond kernels. In Marc' Aurelio Ranzato, Alina Beygelziner, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-
14, 2021, virtual, pp. 24883 24897, 2021. URL http::/pproeedig.g...uuuo....ppp a064bf 1666 200-:2226667700-0strtra.. Atbt-act.tttttttccocmeeeoe hhml
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (cds.), Advances in Neural Information Processing Systems 32: Annual Con- ference on Neurul Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 8570 8581, 2019. URL 11la6651444238881111111111888881
couripp..c/paperree
hash/
Abstract. html
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dicksteen. Finite versus infinite neural networks: an empirical study, In Hugo Larochelle, Aurelio Ranzato, Raia Hadsell, Maria-Florrna Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Processing Systems 33: Annual Conference Newral Information Processing Systems 2020, NewrIPS 2020, December 6- 2020, virlual, 2020. URL httpps:/pproceeddngg/ wttp:eceeddinnneennnn......e courrpp..c/papeee 1222:h/he//005550fffff1132222222
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rale phase of deep learning: the calapult mechanism. CoRR, abs/2003.02216, 2020. URL https: /tapps://arxxii /aaxivv.
org/ ab////33.00..... 02218
Zhu Li, Zhi-Hua Zhou, and Arthur Gretton. Towards an understanding of benign overfitting in neural networks. CoRR, abs/2106.03212, 2021. URL
Xin Liu and Zhisong Pan. Provable convergence of nesterov accelerated method for over-parameterized neural networks. CoRR, abs/2107.01882, 2021. URL httpps//aaaxxv..rrg////////.
Philip M. Long. Properties of the after kernel. CoRR, abs/2105. 2021. URL abs/2105.1055
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Llya Sutskever. Deep
double descent: Where bigger models and more data hurt. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
12

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In 9th International Conference on Learning Representations, TCL.R 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://operreiie..ee
forum?id guetrILLFI
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Phan-Minl Nguyen and Huy Tuan Pham. A rigorous framework for the mean field limit of multilayer neural networks. CoRR, abs/2001.11443, 2020. URL httpss//arxiv... org/ abs//bs/////0 abs///0ss000.... 11443
Roman Novak, Lechao Xiao, Jiri Hron, Jachoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference Learning Representations, 2020. URL
Guillermo Ortiz-Jiméncz, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What linearized neu- ral networks actually say about generalivation? In Annual Conference on Newral Information Process- ing Systems 2021, NewrlPS 2021, 2021. CRL
hhahh//
Abstract. html
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2021 (4):044001, 2021.
David C. Page. How to train your resnet 4: Architocture. Accessod: 2022-01-25.
https://ueb.archiin org/webb/
2018.
Danicl A. Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. abs/2106.10165, 2021. URL https:///rrai http://///www..o/aabaa////////// wttp//.rrr/aaa//1...1..
CoRR,
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXit:1999.12673 2019.
Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Keil, Jonathan Ragan-Kelley, Ludwig Schmidt, and Benjamin Recht. Neural kernels without langents. In Proceedings of the 37th International Confer- ence on Machine Learning, ICML 2020, 18-18 July 2020, Virtual Event, volume 119 Proceedings of Machine Learning Research, pp. 8614-8623. PMLR, 2020. URL http:://pproceednngs.. mlr wtt.prees/vvi9/ shankar20a.. html
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. CoRR, 10002004.00802, 2020. URL. ftttps://arxiv.oo/ Saarrg/ba///0ee aawpa//a.....00000
James B. Simon, Madeline Dickens, and Michael Robert DeWeese. Neural tangent kernel eigenvalues accu- rately predict generalization. CoRR, abs/2110.03922; 2021. URL httpp::/aarxi..orrrbbb//// 03922
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of deep neural networks, 2019. URL btttp:///axxv..rg//ae 0440
Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee net, 2021. URL https://operrrrie...eeeee nwt.oorrruwtt/errr
Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arxio preprint ar 10739, 2020.
13

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Maksim Velikanov and Dmitry Yarotsky. Explicit loss asymptotics in the gradient descent training of neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advunces in Neural Information Processing Systems, 2021. URL https:// http:p//pppaneeeeeeee/erouuaadad wtt.:///ooruuuuddddolll111111
Greg Yang and Edward J. Hu. Tensor programs IV: feature learning in infinitewwddth neural networks. In Marina. Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virlual Evend, volume 139 of Proceedings of Machine Learning Research, pp. 11727- 11737. PMLR, 2021. URL http://proceeeding: mlr, pttss//113/yaagg1c....oo 1ttm
14

---

Published in Transactions on Machine Learning Research (MM/YYYY
A Statistical Significance of Figure [(AA)
In Figure A),, for cach of the dataset sizes 500, 1000, 2000,.... 64000 we have >= 4 neural network runs. Taking our distribution to be the cross product over there runs and dataset sizes we calculate the distribution of scaling constants of the neural network. We get that the mean is .292 and the standard deviation is .0106, The infinite NTK scaling constant (.213) is 7.5 standard deviations away. The mean and the standard deviations were calculated with a sample size of 10000. Even the minimum scaling constant across these 10000 samples
.265 which is closer to the neural network mean scaling constant, (.292) than it is to the infinite NTK scaling constant (.213). The empirical NTK scaling constant: .185 is even smaller and hence the result is even more statistically significant (10.5 standard deviations away).
B Experimental Details
B.1
Architecture
The exact architecture of the Myrtle-CNN we use is the following: 3 Conv Layer with e channels, Relu, 3 Conv Layer with 2c channels, Relu, 2x2 Avg-pooling, Conv Layer with 4e channels, Relu, 2x2 Avg-pooling, Conv Layer with 8c channels, Relu, x 2 Avg-pooling, 4 x 4 Avg-pooling, Dense Layer with output. Stride is always Here is our code for the network in the staz module of the Neural Tangents (Novak al.
2020) library:
init In , apply In kernel In stax. serial
stax Conv(c, (3, 3), strides=(1, 1), padding='SAMB. 'SAME') stax Relu ()
stax Conv (2*c. (3, 3), strides=(1, 1), padding= (aadingg'SAMEE)) stax Relu ()
stax AvgPool ((2 (2,2)),
stax Conv (4*c. (3, 3), strides= (1, 1), Bodding-'SAME") stax. Relu ()
stax AvgPool ( (2 22) (2,2)),
stax Conv (3, 3), strides=(1, 1), padding='SAME') stax. Relu ()
stax AvgPool ( (2,2), (2,2)),
stax AvgPool (4,4), (4,4)),
stax Flatten (),
stax Dense (1)
We refer to as the "width" of the network. Our base network in Figure has 64.
We the following 5 layer CNN for the SVHN-parity task:
init fn apply fn kernel fn stax. serial
stax Conv (c, (3, 3), strides= (1, 1), padding- 'SAME'
stax Relu ()
stax Conv(c; (3, 3). strides= (1, 1), padding= 'SAME'
stax Relu
stax. Conv(c, (3, 3). strides= (1, 1), padding= 'SAME"
stax. Relu ()
stax Conv(c, (3, 3). strides= (1, 1), padding= 'SAME'
stax. stax.Rolu ()
stax Flatten (),
stax. Dense (1)
The base network has 64.
The MLP that we use in our synthetic dataset experiments is depth-4 MLP with the following code:
init
In , apply In kernel In stax. serial
stax. Flatten ()
stax. Dense c),, stax. Relu ()
stax. Dense c), stax. Relu ()
stax Dense (), stax Relu ()
15

---

Published in Transactions on Machine Learning Research
stax Dense (1)
B.2
Scaling Laws
In our plots we use the scaling laws for the form L(n) = A(1/n a where 3 is referred to as the scaling constant. This is take into account the fact that any given neural network has a maximum possible accuracy. Note that as our task is a deterministic there is no label-noise to be accounted for. We calculate A, B by solving the least squares problem between lo)(L(n)) and the log of empirically found test errors.
B.3 SGD with momentum, equivalence between infinite NTKs and neural networks
The standard equivalence between neural networks and infinite NTKs (Lee al. 2019 used GD with momentum. Recent works (Dyer & Gur-Ari 2020 Liu & Pan 2021 have extended this to SGD and SGD with momentum.
B.4 Kernel-SGD
We optimize our neural network with SGD, the empirical NTK (as the the linearized neural network) with SGD and the infinite NTK with kernel-SGD (deseribed later), all with the same optimization hyperparam- eters. For all of the experiments the limiting bchaviour of all 3 models width tends to infinity will exactly be the same in contrast with prior works which have usually optimized these 3 models in different ways. This is important for having a fair comparison between all three models.
early stopping except in Figure 3[d] This allows to ensure that
We now describe kernel-SGD. Intuitively, kernel-SGD refers to doing SGD in the kernel space. We now define more formally, beginning with the notation. Let.
denote the number of samples.
K = FFT where K is the NTK matrix and F is x d feature matrix. The of F corresponding to training sample x is given by f(x)² where f is the function that maps a training sample to their features. Define k(z) as Ff(x) and k(æ, x') as f(z)* f(z')
Yx. be the label corresponding to sample T.
|| - 2 be the loss function. As we will be doing SGD start with 0 initialization we can equiva- lently state the loss function as || y| where w = p'22.
Kernel-SGD refers to doing SGD on the loss || v optimizing for Specifically if our batch contains single sample then we do one step of GD on the loss [[K(x)*= 12 See Blondel (012)) for an implementation.
In the following claim we show that this is equivalent to doing SGD on the loss F yll long our selection of the the batches and SGD hyperparameters is the same. The following claim only works for one step and SGD with batch size but can be casily generalized. The result is casy to prove and probably follows from prior works. But we were not able to find reference so we present it here for completeness.
Claim B.1. The prediction on a input sample x' is the same in the following two cases
We start from w FT za and do one step of SGD on imput sample x the feature space i.e. one step of GD with loss (f(x)² j'' y2)2. The learning rate is n,
We start from z 2y and do one step of SGD on input sample x the kernel space i.e. one step of GD with loss (k(x)T== 2 The learning vale is n-
16

---

Published in Transactions on Machine Learning Research (MM/YYYY
Proof. The update in the first case leads to w' wi 2n(f(x) wo vo) 1T [n(f())(p" -Vo)(()) FT 2n(((xx)22 20 vo)s(x)
The prediction on sample x' changes to w'')= I('')
2n(k(2)** vx)[(x)) = k(x')=== 21 (k(())))*= -y=))
The update in the second case leads to
=' 22(20-2((((x) vx)k(())
The prediction on sample x' changes to
h(d')=²' = (=0) - 2n)(((x)² kx)) =0=')=== 2n/h(r, (()(x)))) 20 yz))
which is exactly the same as the update in the first case.
B.5
Additional Experimental Details for Figures
64 bit versus 32 bit precision Our neural network experiments are done with 32 bit precision while kernel experiments are done with 64 bit precision. To verify that this difference is not the cause of better performance of neural networks we ran the experiment in Figure with 64 bit precision for train size of 64k. The test error actually improved, test error with optimal early stopping was .03993 while it was .04118 32 bit precision.
Starting with O output neural networks The convergence between neural networks, empirical NTK and the infinite NTK as width tends to infinity needs that the neural networks have 0 output at initialization. This can be done by subtracting the initial outputs from the neural network output. While we do not do this for the experiments in the paper, we verify Table | that this makes almost not difference for Figure We table instead of plot as most of the differences too small to be visible plot.
Dataset Size
500
1k
2k
4k
8k
16k
32k
64k
Usual Neural Networks
.1504
.1177
.09655
.08083
.0658
.0545
.04958
.04118
Neural Netwoks with 0 output
.1501
.1204
.09672
.08302
.0668
.05462
.04585
.04152
Table 1: Performance of usual neural networks and their 0 output counterparts.
We now describe additional experimental details for each figure boyond those described in Section 2
In Figure [ all 3 models were trained with batch size 200 and SGD with learning rate of 4.0 and momen- patpprrmmentall.ptmmiiieee mommemdumm For all models we used optimal
tum of .9 (as implemented in early stopping where the test error is logged after increase of 1.1 multiplicative factor in the number of gradient steps. Unless mentioned otherwise this will the optimization setup we will use. For neural network and empirical NTK each model is averaged over 4 random initializations and error bars denote standard de- viations (except the last two points of empirical NTK which we were not able to rerun due to computational constraints). The infinite NTK is a deterministic model hence we only have only have a single run for it. We also trained the kernels by directly solving the linear system with optimal T.uulagulaiiitioon but that yielded worse test performance.
In Figure 2(b] for empirical NTK with large widths (128, 256) the values across different initializations were very similar hence we did not do multiple runs of even higher widths (512, 1024). For the neural network standard error of the mean (SEM) was calculated using 6 runs for widths up to 128 and 4 runs for higher widths.
In Figure [(c) the models are trained with SGD: batch size of 200, learning rate of 10, .9 momentum. All models are averaged across random initializations. In this figure we report the final test error at convergence (train loss < 2 x 10-6 for neural network) for all models.
17

---

Published in Transactions on Machine Learning Research (MM/YYYY)
In Figure c train with learning rate of 10.0 with no momentum.
In Figure d we train the width 512 version of the Myrtle-CNN CNN till it reached train loss - .001 by which point their test етror converges to a nearly fixed value.
In Figure all after-kernels are extracted at the optimal early stopping point of training. In Figure 4(a)
and 4(b) the train sizes used are 500, 1k, 2k, 4k, 8k; 16k, 32k, 64k; 256k, 1024k.
In Figure with fit K to the data with same optimization hyperparameters used in Figure batch size 200, SGD with learning rate 4.0 and momentum .9.
C Effect of Very Low Learning rate
0114
WTK
28 10
26
24. 00
Text Errer
Text Erre
Text Erer
0.08
2.5x
0.00
2x
Nh. 103b lrannig
Empacca
Iriute NTK
MIP
D/34
1000
4003
L00D
M400
1033
3374
10
10"
11P Learnigg Rate
Train size
Leaming Rate
(a)
(b)
(c)
Figure 6: In Figure 6(a) we redo the experiment from Figure except that training the neural network with 1000x smaller learning ratc. This leads to significant drop in the performance and the scaling constant explore the effect of learning rate on fixed training size.
of the neural network. In Figures 6(b) and 6(c)
In this section we explore the effects of very low learning rate. The motivation is to understand the gradient flow limit i.e. the limiting behaviour of trained neural networks as the learning rate tends to O. In Figure 6(a) we repeat Figure [ except that we train the neural network with learning rate ,004. The measured scaling constants are reasonably close by: .2044 204,.185,.. .233 for the MyrtleCCNN, its empirical NTK and infinite NTK respectively. This suggests a natural question:
Question C.1. Do the benefits (with respect to scaling constant) of fivile width networks over corresponding empirical NTK vanish in the gradient flow limit?
To answer this question affirmatively we would would need repeat the plot of Figure 6(a) for various widths which was computationally infeasible for us. We leave this for future work.
We now move to considering the effect of learning rate a fixed training size.
In Figure b(b) we plot the performance with respect to lcarning rate for training sizes 4000 (From the setup in Figure and observe that at low learning rates performance is worsd than infinite NTK but still better than empirical NTK at initialization. From this plots it is not clear if at the lowest learning rates which we could train the performance has converged to the gradient flow performance. In this setup it was computationally infeasible for us explore smaller learning rates. To do this we move to the synthetic setting (with the same setup as in Figure in Figure 6[c] Here the performance (final test error) converges we go towards smaller learning rates which indicates that we have converged to the gradient flow limit. In this limit the neural network performs better than the infinite and empirical NTK at initialization. Note that higher learning rates still leads to even better performance.
We interpreted all of these experiments as suggesting that while high learning rate plays an important role in the performance of empirical networks it may not be necessary in having a improved
Note that this is just for single width. We do know how width affects these results. We do know that infinite width the learning rate does have any effect, other than due to optimal early stopping.
18

---

Published in Transactions on Machine Learning Research (MM/YYYY)
performance over corresponding NTKs.. But more experimental evidence is needed to understand the role of learning rate and in understanding the gradient flow limit, particularly for natural tasks.
Related Works: Barrett & Dherin (2021) and Smith et al. 2021 describe regularizers which potentially explain the benefits of high learning rate in GD and SGD, The effect of higher than critical learning rate on very wide networks has been studied by Lewkowycz et al. (2020). They claim that perhaps the improved
generalization of neural networks can be explained by the use of higher than critical learning rate. This
does not apply to our experiments all of our experiments use learning rates which are below the critical learning rate.
Higher order analogues of the NTK
Let f(w,x) x) with w representing the weights and x a sample. By Taylor expansion around wp we have:
1 fw, = f(wo 2) V (w (w wo) + 2((w (w- u f x) mo (w wo) ....
The empirical NTK of the neural network around weights wi refers to the model 91 V (w, wa).
We consider the following two second order analogue of the NTK: 92 only (w, .)) = (w "at x 1
wo) and 92 full (w, x) = (Wo,, x)) + Vwf(w, x) wo (w wo) + [w wa) (oo)T V (w, x²) mm (w wo)
0.135
Empriial 2//4
Emprical NIK
Text Erorr
Error
0.09
0.00
0.04
0.027
0.04
64000
Train size
Train Size
(a)
(b)
Figure 7: Higher order analogues of Empirical NTK.
In Figure 7(a) we use the setup of Figure to plot the performance of the 2nd order analogue of the empirical NTK (g) referred by full in the plot). This shows that even this higher order analogue is not sufficient to recover the scaling law of neural networks.
In Figure 4(a) we saw that the afrer-kernel was sufficient explain the improved performance of neural network over the empirical NTK. We now show an analogous result for the 2nd order analogue of NTK. As want to understand the effect of change in the higher order terms need to remove the influence the after-kernel. We do so by defining the higher order analogue of after-kernel as the model go which does not contain the after-kernel. We will denote this by Honly when we use the weights after training on m samples. In Figure 7(b) we show that the performance of Honly(n) is very close to that of the neural network.
Both of these experiments suggest that theories which assume that higher order analogues of the NTK remain fixed throughout the training may not be sufficient. to explain the performance of neural networks.
19

---

Published in Transactions on Machine Learning Research (MM/YYY)
E
Experiments on Synthetic Data
Some of our experiments were not feasible on the C1FnR-5m-bin and SVHN-parity tasks. We did these experiments on the following synthetic task: Sample 2 ~ {-1, 1,,, ~ (0, 120); the input sample is x' = 2 + e and the label = 21
Experiments on very low learning rates for this synthetic task can be found in Appendix
In Figure we do an analogous experiment to Figure 2(b) and 2(c) for this synthetic task. We again observe that neural networks at small width improve with increase in width but at high width they start to worsen off with increase in width. The models are trained with SGD: batch size of 100, learning rate of 10, no momentum. All models are averaged across random initializations. In this figure we report the final test error at convergence (train loss < 2 x 10- for neural network) for all models.
0.396
NN
Empiricll NTK
Expected Test Error
0.288
0.2
10
10
Width
Figure 8: Analogous experiment to Figure 2 and
for the synthetic task ((c)
F SVHN-parity Experiments
0.39
Imepicaa NIK
Test Error
0.18
ca Erdr
0.39
Tex Ertrr
0.12
6.
Emarce
011
0.00
Train
Datanct uned
(a)
(b)
(c)
Figure 9: Experiments for the SVHN-parity task, analogous to Figure Error bars in Figure ((a)
denote
standard deviation.
In Figure the analogue of Figure 1 The scaling constants Figure [(a are .8,, .22 and .19 for the neural network, infinite NTK and the empirical NTK at initialization respectively. We also observe that unlike the CIFAR-5m-bin task here the neural network outperforms the infinite NTK even at small dataset sizes. This may be because the inductive bias of the NTKs is not suited for the parity task. In Figure 9(c) we plot the more extensive figure corresponding to Figure (a)) We again observe that the kernel continues to improve for most of the training.
All of the above models are trained with SGD: batch size of 200, learning rate of 10, momentum. In Figure [a] the neural network and empirical NTK experiments are averaged over 4 runs with error bars denoting standard deviation. The infinite NTK is deterministic model and hence only do single run.
20

---

Published in Transactions on Machine Learning Research (MM/YYY)
Figure 2 is another SVHN-parity experiment.
G Other related work
Beyond Double Descent: Double descent (Belkin al. 2019 Geiger et al. 2019 Nakkiran et al, 2020) predicts that in the regime of overparameterized models increasing the width improves the test error. We observe that the performance of overparameterizzed models is better than that of infinite width models showing that there is a natural setting where there is at least one more ascent after the double descent phenomena. Behaviours beyond the double descent phenomena have been predicted (Adlam & Pennington, 2020 d' A'Accoli et al., 2020 Li et al. 2021 and also obscrved (Lee et al.) 2020p in empirical neural networks. Our works different from these works as we show that in our setup simultaneously a) empirical NTK displays a monotonic improvement in the overparameterized regime towards the infinite NTK performance while b) the neural network performs better than the infinite NTK. This directly points towards another ascent after the double descent and also pinpoints its cause the divergence between finite width neural networks and the empirical NTK at initialization.
After Kernel The empirical NTK after the training of neural network has been termed as after-kernel (Long 2021j. It has been shown (ongg 2021 Paccolat et al., 2021 We extend these works by studying how the after-kernel changes with dataset size and show that it continues improve with dataset size.
Time dynamics of training from the NTK perspective. has been studied by Fort et al. 2020 5 Ortiz- Jiménez et al. (2021): Atanasov al. (2021) Long [022]) These papers suggest that the empirical NTK changes rapidly in the beginning of the training followed by slowing of this change. We argue against this
interpretation in Scction 5
Explanations for Scaling Laws Current explanations of scaling laws (Sharma & Kaplan 2020 Bahri et al. 2021) rely on a fixed representation space. Operationalizing representation as the after-kernel, our results suggest that in practical neural networks the representation itself improves as data-size increases. Hence we may need more refined theories for explaining scaling laws of neural networks which take this into account.
Provable differences between neural networks and NTKs have been shown Ghhoraani et al. 2020:
Daniely & Malach 2020 Karp et al. 2021 though they have been restricted to synthetic datasets.
Effect of learning rate: See the last paragraph of Appendix C
H Scaling curves of various after-kernels
In Figure [(c) we saw that the neural network has a better scaling constant than any fixed after-kernel. Another curious observation from the plot is the comparison between the empirical NTK and K16t scaling curve. We observe that
K Kies outperforms the empirical NTK of all training sizes we tested.
The empirical NTK has a better scaling constant than KiGk-
We start by noting that this issue of having worse scaling but always generalizing better does not arise for other comparisons in the paper which involve the neural network. This is because the neural network has better a scaling constant and it also starts to perform better for larger dataset sizes.
Going back to empirical NTK and Kisa we can ask the following question: Does empirical NTK having a better scaling constant imply that for large enough sample sizes empirical NTK would outperform K KI Not necessarily, see Figure 10(a) for an example of two scaling curve where where the curve with the worse scaling constant (.25) always outperforms the curve with the better scaling constant (.5). We think this will also
TTee irreducible component of the error necessary this situation arise. Irreducible error refers to error that the model will have no.
21

---

Published in Transactions on Machine Learning Research (MM/YYYY)
hold for empirical NTK and K16k- The intuition behind this is that both empirical NTK and K16k have the same number of parameters and both are linear models but Kiss has a better implicit bias (as it has better performance 6
On the other hand, our current scaling curves do not predict this i.. extrapolating them predicts that empirical NTK would outperform Ki credible prediction for multiple reasons:
million, see Figure 10(b) But we think this is not a
At n million we will have more samples than parameters (~ 1.5 million). This means that we might no longer be overparaneeerrieed and hence extrapolating data scaling curves is not valid.
This could occur due to small fluctuations in the fitting of the scaling curves (particularly we are extrapolating a lot, factor of 100). Specifically we think that our estimation of irreducible loss for these two curves is noisy.
( ozy
Empirical NTK
Curve 2 (.01) [5] 4 .01
0.135
Test Error
4x 10-1
0.06
3x 10
x 10-1
x 102
x 10a
5 x 10
S 10P
x 10
10a
10
10a
10a
Train Size
(a)
(b)
Figure 10:
Plots with Test Loss
Throughout the paper we have focused on the test error. Here we recreate some of the important plots for
test loss instead of test error. These correspond to Figure (aaueeaa)... [a), 2bb and 5(a) respectively. All the phenomena that we observed for test error in these plots continue to hold for test loss.
The scaling constants in Figure l1 .22 .171 and .135 for the neural network, infinite NTK and the empirical NTK respectively.
6This intuition does apply NTK model and neural network they models of different types, linear and nonlinear respectively.
22

---

Published in Transactions on Machine Learning Research
1i-AR--mm-iin with samples
0.355
Neari Nerwert
Emplrcal MIK
Text Loss
0.12
Expected Text Lots
Tox Losa
Erg ricl NTK
2.10
0.18
kdiis NTK
Erpiial
0.1
0.16
(a)
(b)
(c)
Figure 11: Plotting some of our important figures with test loss as the metric.
23


Reviews:
[{"review_id": "jeb2BE89BM", "reviewer": "Reviewer_Lg8z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths.__\n- The experiments are well-designed and conducted with a significant computational effort.\n- The empirical results demonstrate a very clear pattern, making the claim very convincing (although this is usually true for many log-log style plots).\n- The paper is well-organized and clearly written.\n\n__Weaknesses.__\n- The paper has a relatively weak followthrough; having observed the clear discrepancy between the scaling behaviors of NTK and NN, the paper does not attempt to provide an original explanation of why such a phenomenon takes place (or come up with an idea to take advantage of it). Without such followthrough, it is quite difficult to see the immediate merits of the observations over the prior works on the limitations of the NTK. Given the present manuscript, I can only think of one intellectual merit: a sanity check for the future theoretical framework that they should be able to explain this discrepancy in the scaling behavior. Other than that, I am not sure. In this regard, the paper should have given more detail on (1) in which regard the classical theory fails, and (2) in which sense extending the current theory to explain the observation is challenging and important.\n\n\n- If I understood correctly, the experiments took place only on Myrtle CNN and 5-layer CNN, which are small and simplified versions of ConvNets. In this regard, I wonder whether the same pattern will also take place on other architectures without being confined to ConvNets.\n\n- Some of the claims should be rephrased/corrected to avoid misunderstanding. On page 9, the paper argues that \"we will give evidence that suggests, contrary to prior work, that there is no such \"phase transition.\"\" I believe that the correct way to put it should be \"we did not observe any phase transitions under the experimental settings tested. In other words, the empirical results can only rigorously disprove the statement that \"the phase transition __always__ takes place,\" and not proving that \"the phase transition __never__ happens.\" Because this paper is almost 100% about experiments & observations, one must describe them very carefully. (In fact, I am not really sure if this claim is even true; the log-log scale does not seem to be an ideal way to capture such a phenomenon.)\n\n- One could make this paper clearer to unexperienced readers (like me) by adding some detailed explanations and experimental setups. For instance, authors could clarify which initialization schemes the models use, and give more formal details on what \"NTK parameterization\" is.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reports several pieces of empirical evidence that illustrate the limitations of the neural tangent kernels---both the infinite width limit and the empirical NTK---in capturing and representing the behavior of neural networks. The focus is on the scaling behavior of neural networks and kernels with respect to the size of the dataset. The main finding is that NTKs have a noticeably worse scaling property than NNs, consistently over varying widths and hyperparameters for the NNs. In addition, the paper reports that the after-kernel continues to improve with the size of the dataset, and no noticeable \"drift\" during the early phase of training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the \"weaknesses\" above.\n- The experiments on other models will be too much for the given timeline, but I expect at least more justifications on why looking at small CNNs is enough, in authors' opinion.\n- Also fix some typos: (1) Figure 2 caption, \"widths and its the infinite NTK\" -> \"widths and the infinite NTK\" (2) page 7, \"kenel\" -> \"kernel.\" (3) Appendix B, what is \"V Relu\"?", "broader_impact_concerns": "I do not think this paper needs to address any ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kmt4Fze8mt", "reviewer": "Reviewer_MrBR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n\n* Scaling exponents are a solid and thorough way of comparing two classifiers in the light of recent studies.\n* The findings on the stable rate of evolution of the ENTK are interesting.\n* It is interesting that in the small-dataset regime in Figure 2-(b-c) the neural networks of small width perform the best and then deteriorate gradually as the width increases (following the classical U-shape curve and overfitting in the malign sense).  \n\n*Weaknesses:* \n\n* It is expected that as feature extractors, neural networks will perform better than a random features linear predictor (NTK at init). It is fair to compare the after-kernel with finite-width NN as done in the previous work. The thing that I find unsatisfactory is why should one use the after-kernel as a linear predictor with learned features, say instead of re-training the last layer of the neural network. I think the more interesting part of the paper is the evolution of the ENTK. \n* Overall I would appreciate more interpretations in terms of feature learning and also in general. \n* I might be missing something but in Figure 5-a, the caption writes empirical NTK **at initialization**. How is it possible that it evolves during training if it is fixed at initialization? I am guessing that it improves because its training dataset also has grown. Is that correct?  \nThe text needs update accordingly, it is confusing as it is. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the generalization errors between predictors (i) neural network (ii) after-kernel and (iii) empirical kernel (linear-classifier) in terms of the-called scaling law that is the gen. error as a function of training dataset size. These three predictors converge to the same predictor in the infinite width limit. In the finite width case, the parameters move from initialization and across datasets and architectures neural networks perform better than the after kernel which is better than the empirical kernel. Moreover, unlike the recent results on the stability of the NTK after a few epochs of training, the paper shows that the empirical kernel keeps constant improvement at a linear rate in the log-log scale (beyond the initial phase of training; over 32 epochs). ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is a valuable contribution as it contributes to the study of neural networks through the lens of NTKs. However, I think the paper is too focused on the standard comparison between NTKs and neural networks. It is expected that there is separation for finite-width networks and that after-kernel does better than the empirical kernel. The more interesting insight would be what the scaling laws of NTKs tell us about the performance of neural networks. Are they helpful to predict or understand the generalization of finite-width neural networks?\n\nFollow-up question: Is there any link between the performance of the after-kernel and the evolution speed of ENTK? \n\nTypos:\n* emperical NTK (page 1, italic) -> empirical\n* Question 3.1: the the (double the)\n* truly infinite width -> infinite width (there is only one limit in this scaling)\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BDUSvmt1S1", "reviewer": "Reviewer_Yjp1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper is the systematic empirical exploration of the scaling laws\n\nThe weakness is the lack of testing of the statistical significance of the results, and the over-claims associated with it. See the requested changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work empirically shows that the scaling of the generalization error of a finite width network is different from NTK", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. There are a lot of typos in the references. For example, the first sentence of the paper has \"Jacot et al. Jacot et al (2018),\"  and similar mistakes appear throughout the paper. The authors should be careful and patient, checking obvious errors thoroughly before submission\n\n2. A lot of figures are not quite visible. For example, Figure 3. The authors should make the font sizes of the figures match the font sizes of the main text. In this regard, all figure (not just Figure 3) can be improved\n\n3. The paper is all about scaling exponents. Why not show the exponents in the figure or in the captions?\n\n4. This is the first major problem: none of the reported scaling exponents have an error bar. While the paper is about saying that the exponents are different, there is no scientific evidence that there is a difference. The authors need to state the error bar and p-value to convince the readers that these measured quantities are significantly different\n\n5. Also, I think the authors need to significantly tone down their claims. Even if the authors show that the exponents are statistically different, the difference is still quite minor in my opinion. For example, for Figure 1, the NTK has 0.213 while the actual NN has 0.291. Supposing both models start from the same performance, how many times more data points are required for the NN to have half of the test error of the NTK? Solving the condition $0.5 * n^{-0.213} = n^{-0.291}$ shows that $n\\approx 7000$. We need to have a 7000 times larger dataset to see a significant difference between NTK and actual NN. To be honest, I think this does not convince me (or any ordinary reader) that NN is very different from the NTK. To me, it feels like the empirical evidence only shows that NTK is quite good -- I am not saying that NN is the same as NTK, but that the authors should be much more humble in many claims. For example, in the abstract, the authors call this a \"fundamental difference,\" but I am not convinced. A difference between 0.2 and 2 feels fundamental, but a difference between 0.21 and 0.29 does not.\n\n6. A lot of claims do not feel right. The paper mentioned quite a few times \"neural networks provably converge to the NTK at infinite width.\"\nIs there a reference to this? I think a lot of qualifiers are required here, for example, \"under gradient flow and special learning rate scaling\" and \"width certain type of initializations.\"\n\n7. The term \"SGD\" in claim 3.1 bepuzzles me. Are the experiments not using GD? Which part of the experiment is using SGD? If SGD is used, a crucial question arises. Is the difference due to finite step size or minibatch training? There is one more parameter to control, and I think the authors should investigate the causes of the difference in greater depth. ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Summmary
The paper examines how the neural tangent kernel (NTK) scales with dataset size. The main finding is an empirical observation that the NTK scales much worse than neural networks (NN), highlighting a fundamental difference between NN and NTK. This observation holds true for both infinite-width and empirical NTKs. Additionally, the paper demonstrates that empirical NTK evolves continuously during training, contrary to the previous belief that it stabilizes after a few epochs. These results emphasize the limitations of the current NTK approach for studying the scaling law of NN.

### Reviewer Recommendations
The reviewers generally agree that the empirical observations are interesting, and it's a good fit for TMLR. However, their main concerns are on the overclaim in the contribution and title. Specifically, the reviewers pointed out that the paper should clarify that its results are "empirical" and limited to "scaling laws," especially in the title.

### AE Recommendation
The AE agrees with the reviewers' concern. Therefore, my recommendation is to accept the paper with minor revisions. We expect the authors to clarify their contribution in the revision. This includes:
- Change the title to "Empirical Limitations of the NTK for Understanding Scaling Laws in Deep Learning"
- Clarify in their abstract and main text that their contributions focus on "empirical" observations and "scaling laws" rather than generalization.
Decision: Accept (poster)