Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond
Authors: Alan Jeffares
Alicia Curth
Mihaela van der Schaar
Venue: NeurIPS 2024 poster (2024)
Keywords: Empirical theory
double descent
grokking
linear mode connectivity
gradient boosting
Primary area: other

Abstract: Deep learning sometimes appears to work in unexpected ways. In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network consisting of a sequence of first-order approximations telescoping out into a single empirically operational tool for practical analysis. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena in the literature -- including double descent, grokking, linear mode connectivity, and the challenges of applying deep learning on tabular data -- highlighting that this model allows us to construct and extract metrics that help predict and understand the a priori unexpected performance of neural networks. We also demonstrate that this model presents a pedagogical formalism allowing us to isolate components of the training process even in complex contemporary settings, providing a lens to reason about the effects of design choices such as architecture & optimization strategy, and reveals surprising parallels between neural network learning and gradient boosting.

Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond
Alan Jeff'ares University of Cambridge aj6590cam. ac uk
Alicia Curth University of Cambridge amc2530cam ac uk
Mihaela van der Schaar University of Cambridge mv4720cam. ac uk
Abstract
Deep learning sometimes appears work in unexpected ways. In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network consisting of a sequence of first-order approximations telescoping out into a single empirically operational tool for practical analysis. Across three case studies, we illustrate how it can be applied to derive empirical insights on a diverse range of prominent phenomena in the literature- including double descent, grokking, linear mode connectivity, and the challenges of applying deep learning on tabular data - highlighting that this model allows us to construct and extract metrics that help predict and understand the a priori unexpected performance of neural networks. We also demonstrate that this model presents a pedagogical formalism allowing us to isolate components of the training process even in complex contemporary settings, providing a lens reason about the effects of design choices such as architecture & optimization strategy, and reveals surprising parallels between neural network learning and gradient boosting.
Introduction
Deep learning works, but it sometimes works in mysterious ways. Despite the remarkable recent success of deep learning applications ranging from image recognition [KSH12) text generation 2M there remain many contexts in which it performs in apparently unpredictable ways: neural networks sometimes exhibit surprisingly non-monotonic generalization performance (BHMMI9
2BE+22 continue to be outperformed by gradient boosted trees tabular tasks despite successes elsewhere [GOV22] and sometimes behave surprisingly similarly to linear models [FDRC20]. The pursuit of a deeper understanding of deep learning and its phenomena has since motivated many subfields, and progress on fundamental questions has been distributed across many distinct yet complementary perspectives that range from purely theoretical to predominantly empirical research.
Outlook. In this work, take a hybrid approach and investigate how we can apply ideas primarily used in theoretical research to investigate the behavior of simple yet accurate model of neural network empirically. Building upon previous work that studies linear approximations learning in neural networks through tangent kernels (c.g. [JGHT8 COB19], see Sec. 2). we consider model that uses first-order approximations for the functional updates made during training. However, unlike most previous work, we define this model incrementally by simply telescoping out approximations to individual updates made during training (Sec. 3 such that it more closely approximates the true behavior of fully trained neural network in practical settings. This provides us with pedagogical lens through which we can view modern optimization strategies and other design choices (Sec. 5). and a mechanism with which we can conduct empirical investigations into several prominent deep learning phenomena that showcase how neural networks sometimes generalize seemingly unpredictably.
* Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Across three case studies in Sec. 4 we then show that this model allows us to construct and extract metrics that help predict and understand the priori unexpected performance of neural networks. First, in Sec. 4. we demonstrate that it allows us to extend [CJvdS23] 's recent model complexity metric to neural networks, and use this to investigate surprising generalization eurves - discovering that the non-monotonic behaviors observed in both deep double descent [BHMM]9] and grokking [2 are associated with quantifiable divergence of train- and test-time model complexity. Second, in Sec. 4.2 we show that it reveals perhaps surprising parallels between gradient boosting [FrO]] and neural network learning, which we then use to investigate the known performance differences between neural networks and gradient boosted trees tabular data in the presence of dataset irregularities M3KV+33.. Third, in Sec. 3.3
we use it to investigate the connections between gradient stabilization and the success of weight averaging (i.e. linear mode connectivity (FDRC20).
2 Background
Notation and preliminaries. Let so X' C + TR denote a neural network parameterized by (stacked) model weights 0 E IRP Assume we observe a training sample of input-output pairs (x, 31 i.i.d. realizations of the tuple (X, Y sampled from some distribution P, and wish to learn good model parameters 0 for predicting outputs from this data by minimizing an empirical prediction loss :, Em= {(fo(x))))))) Vi), where f: I x R + I denotes some differentiable loss function. Throughout, we let lo 1 for ease of exposition, but unless otherwise indicated our discussion generally extends to k > 1. We focus on the case where O is optimized by initializing the model with some 0g and then iteratively updating the parameters through stochastic gradient descent (SGD) with learning rates " for 7 steps, where at each / E (T] = {1..... 7) we subsample batches
B, S [n] = [1....,n] _n of the training indices, leading to parameter updates ~0, == 0, 0,1 as:
O2 = 02-1 + A0, = 0
7a B.
EiER, Vele., = 0,- - Tgf
(I)
where g
0d De(fo,_) (x:).v)))
- (x.) is the gradient of the loss w.r.t. the model prediction for the 7th training
IT, and the p x n matrix 1(ne B,) (x,,,_((x,,) has as columns the gradients of the model prediction with respect to its parameters for examples in the training batch (and 0 otherwise). Beyond vanilla SGD, modern deep learning practice usually relies on a number of modifications to the update described above, such as momentum and weight decay; we discuss these in Sec. 5
example, which we will sometimes collect in the vector gi I...... I
T, = T [1(1BB))
[B,| (x)))_,xx),...
[B:]
Related work: Linearized neural networks and tangent kernels. A growing body of recent work has explored the use of linearized neural networks (linear in their parameters) as tool for theoretical [JGHI8 19 LXS+ 19] and empirical [FDP+21 20. LZB20 OJMDF21 study. In this paper, we similarly make extensive use of the following observation (as in e.g. 2DP+ we can linearize the difference A (x) = So. (x) So. (x) between two parameter updates
( = Vofo. (x) Δ., + ((1 13) R (ojo,,, T A0: == Af((x)
where the quality of the approximation A fe (x) is good whenever the parameter updates A0, from a single batch are sufficiently small (or when the Hessian product || |,,,, Vale. (x) vanishes). If Eq. (2) holds exactly (e.g. for infinitesimal 7x), then running SGD in the network's parameter space to obtain AO, corresponds to executing steepest descent on the function output fe (x) itself using the neural tangent kernel K. (x, xx) at time-step [IGH8), i.e. results in functional updates
fx) R2 7 Exepw K. (x, x:)9 where KJ (x, x:) = 1 B.) [B
VoJo.
(x) (x.). (3)
Lazy learning COB19 occurs as the model gradients remain approximately constant during training, L.e VoJo, (x) VoJa, (x). Vi E [T]. For learned parameters 0r, this implies that the approxi-
mation for (x) for (x) Volo. (x) (Or- -o) holds which linear function of the model parameters, and thus corresponds to a linear regression in which features are given by the model gradients Vofe, (x) instead of the inputs x directly whose training dynamics can be more eas- ily understood theoretically. For sufficiently wide neural networks the Velee (x), and thus the tangent kernel, have been theoretically shown to be constant throughout training in some settings JGH18 19). but in practice they generally vary during training, as shown theoretically in L.ZB20 and empirically in (FDP+ 20). A growing theoretical literature [GPK22 investigates con- stant tangent kernel assumptions to study convergence and generalization of neural networks (c.g.

---

Linear approximations Af((x) = Vofo.
B2
Dirain
B1
Random prediction
A0
00
Telescoping prediction
Soo (x)
(x)
+
A (xx
+ +...+
Afr(x)
for (x)
Vefe.(x)
Vofo, (x)
Figure Illustration of the telescoping model of a trained neural network. Unlike the more standard framing of neural network in terms of an iteratively Icarned set of parameters, the telescoping model takes functional perspective training a neural network in which arbitrary test example's initially random prediction, Se, (x), is additively updated by linearized adjustment A ft (x) at each step & as in Eq.
[GHI8 LXS 19 DLL 19 BM19 GMMMI9 GSJW20). This present work relates more closely
to empirical studies making use of tangent kernels and linear approximations, such as [LSP+20 OIMD221 who highlight differences between lazy learning and real networks, and (FDP+20 who empirically investigate the relationship between loss landscapes and the evolution of K (x, x;).
3 A Telescoping Model of Deep Learning
In this work, we explore whether we can exploit the approximation in Eq. (2) beyond the laziness assumption to gain new insight into neural network learning. Instead of applying the approximation across the entire training trajectory at once as in flim (x), we consider using incrementally each batch update during training to approximate what has been learned at this step. This still provides us with greatly simplified and transparent model of neural network, and results in a much more reasonable approximation of the true network. Specifically, we explore whether instead of studying the final model fo, (x) as whole we can gain insight by telescoping out the functional updates made throughout training, i.e. exploiting that we can always equivalently express Jo, (x) as:
fe, (x) fe. + IT (x) (x)] = (x) + ΣT= Af((x)
(4)
This representation of trained neural network in terms of its learning trajectory rather than its final parameters is interesting because we are able to better reason about the impact of the training procedure the intermediate updates A (x) than the final function fe (x) itself. In particular, we investigate whether empirically monitoring behaviors the sum Eq. (4) while making use of the approximation in Eq. (2) will enable to gain practical insights into learning in neural networks,
while incorporating a variety of modern design choices into the training process. That is, we explore the use the following telescoping model for (x) as an approximation a trained neural network:
Telescoping model of trained neural network
(x) (x) = fo, (x) +
Vefo. T0: (x)
Eie(n K? (x, (x,xx, Σ t=1
x;)
(5)
(i) The weight--vennging representation
(i) The kernel representation
where K.' (x,x;) is determined by the neural tangent kernel as 7 KK x;) in the case of standard SGD (in which case (ii) can also be interpreted as a discrete-time approximation of [Dom20]* 's path kernel), but can take other forms for different choices of learning algorithm
explore in Sec. 5
Practical considerations. Before proceeding, is important to emphasize that the telescoping approximation described in Eq. (5) is intended as tool for (empirical) analysis of learning in neural networks and is not being proposed as an alternative approach to training neural networks. Obtaining for (x) requires computing Vefe... (x) for each training and testing example at each training step e [T], Icading to increased computation over standard training. Additionally, these computational costs are likely prohibitive for extremely large networks and datasets without further adjustments; for this purpose, further approximations such [MBS23] could be explored. Nonetheless, computing for (x) - or relevant parts of it is still feasible in many pertinent settings later illustrated in Sec.
3

---

How good is this approximation? In Fig. we examine
the quality of fo, (x) for 3-layer fully-connected ReLU net- work of width 200, trained to discriminate 3-vs-5 from 1000 MNIST examples using the squared loss with SGD or Adam W [H17]. In red, we plot its mean average approximation error ExeX IJo, (x) - So, (x)I) and observe that for small learning rates y the difference remains negligible. In gray we plot the same quantity for fli" (x) (i.c. the first-order expansion around 0o) for reference and find that iteratively telescoping out the updates instead improves the approximation by orders of magnitude- which is also reflected in their prediction perfor- mance (see Appendix D.I. Unsurprisingly, y controls approxi- mation quality as it determines || Further, interacts with the optimizer choice- - e.g. AW) [KB14, LH17 naturally makes larger updates due to rescaling (see Sec. 5) and therefore requires smaller 7 to ensure approximation quality than SGD,
SCD
10
10-3
lanning
10
1000
approdinstion podiinttion crnver
10
AdomW
10-3
Lescning 1660
110-1
50
Optimization step.
Figure 2: Approximation error of the telescoping (fo, (x), red) and the linear model flam (x). gray).
4 A Closer Look at Deep Learning Phenomena Through a Telescoping Lens
Next, turn to applying the telescoping model. Below, we present three case studies revisiting existing experiments that provided evidence for a range of unexpected behaviors of neural networks. These case studies have in common that they highlight cases in which neural networks appear to generalize somewhat umpredictably, which is also why each phenomenon has received considerable attention in recent years. For each, then show that the telescoping model allows us construct and extract metrics that can help predict and understand the unexpected performance of the networks. In particular, we investigate (i) surprising generalization curves (Sec. 4.1). (ii) performance differences between gradient boosting and neural networks on some tabular tasks (Sec. (4.2), and (iii) the success
of weight averaging (Sec. 4.3 We include an extended literature review in Appendix A detailed discussion of all experimental setups in Appendix C and additional results in Appendix D
4.1 Case study 1: Exploring surprising generalization curves and benign overfitting
Classical statistical wisdom provides clear intuitions about overfitting: models that can fit the training data too well because they have too many parameters and/or because they were trained for too long are expected to generalize poorly (e.g. (HTF099 Ch. 7]). Modern phenomena like double descent [HMMM19, however, highlighted that pure capacity measures (capturing what could be learned in- stead of what is actually learned) would not be sufficient to understand the complexity complexity-generalzzaii generalizzain relationship in deep learning [Bel2T]. Raw parameter counts, for example, cannot be enough to under- stand the complexity of what has been learned by neural network during training because, even when using the same architecture, what is learned could be wildly different across various implementation choices within the optimization process and even at different points during the training process of the same model, as prominently exemplified by the grokking phenomenon [PBE+22] Here, with the goal of finding clues that may help predict phenomena like double descent and grokking, explore whether the telescoping model allows us to gain insight into the relative complexity of what is learned.
A complexity measure that avoids the shortcomings listed above because it allows to consider specific truined model - was recently used by [ClvdS23] in their study of non-deep double descent. As their measure 1g builds on the literature on smoothers (HT90)) it requires to express learned predictions as a linear combination of the training labels, i.e. as f(x) = $(x)y = Σiejn) 8 (x))x: Then, [JvdS23] define the effective parameters p used by the model when issuing predictions for some set of inputs {x])ieI, with indices collected in To (here, Io either Zerain = {L, .1,.... n) or Ieest = (n + 1, I,..., n + m) as 1 = P(Io, $()) = EZl
x2 EjEzo [1? Intuitively, the larger yg.
the less smoothing across the training labels is performed, which implies higher model complexity.
Due to the black-box nature of trained neural networks, however, it is not obvious how to link learned predictions to the labels observed during training. Here, we demonstrate how the telescoping model allows us to do precisely that enabling to make use of pe as a proxy for complexity. We consider the special case of a single output (k 1) and training with squared loss {f y) = = - f(x))²

---

and note that we can now exploit that the SGD weight update simplifies to
A = 7 (y fo. where = [y1,,... ynl and fo. = [fo, (xi)..... So. (x,))* Assuming the telescoping approximation holds exactly, this implies functional updates
(6)
Af((x) >/2 Ve.e._ (x) TT(y fo...)
(7)
which use a linear combination of the training labels. Note further that after the first SGD update
Sor (x) - (x) + + VoJa. (x) Tiy fo. (x) 71 Volo. (x) T Tfo
(8)
so,
dg (x)
which means that the first telescoping predictions So, (x) are indeed simply linear combinations of the training labels (and the predictions at initialization)! As detailed Appendix [.. this also implies that recursively substituting Eq. into Eq. ( further allows us to write any prediction fe, (x) as a linear combination of the training labels and (.), i.c. fo, (x) = se, (x)y + cb, (x) where the 1 X n vector Se, (x) is a function of the kernels of the { (-, Je and fo (-). We derive precise expressions for so, (x) and co, (x) for different
and the scalar cg, (x) is a function
optimizers in Appendix B.1 enabling us to use so, (x) to compute y proxy for complexity below.
Double descent: Model complexity vs model size. While training error always monotonically decreases model size (measured by pa- rameter count) increases, [BHMM19] made a surprising observation regarding test error in their seminal paper on double descent: they found that test error initially improves with additional parameters and then worsens when the model is increasingly able to overfit to the training data (as is expected) but can improve again as model size is increased further past the so-called interpolation threshold where perfect training performance is achieved. This would appear to contradict the classical U-shaped relationship between model complexity and test error (HTF09) Ch. 7]. Here, we investigate whether tracking 1 on train and test data separately will allow us to gain new insight into the phenomenon in neural networks.
0.4
MSE
10
10
13
102
Eilertioe Paramezzes
500
Train
Test
10
10
10a
Num hidden nonrons
Figure 3: Double descent in
MSE (top) and effective param- eters P (bottom) on CIFAR-10.
In Fig. 3 replicate the binary classification example of double descent in neural networks of [BHMM19] training width to distinguish cats and dogs CIFAR-10 (we present additional results using MNIST in Appendix D.2)). First, we indeed observe the characteristic behavior of error curves as described in [8HMM19] (top panel). Measuring learned complexity using pg. then find that while pgrein monotonically increases as model size is increasing, the effective parameters used on the test data plest implied by the trained neural network decrease model size is increased past the interpolation threshold (bottom panel). Thus, paralleling the findings made in [CIvd233] for linear regression and tree-based methods, we find that distinguishing between train- and test-time complexity of neural network using z provides new quantitative evidence that bigger networks are nol necessarily learning more complex prediction functions for unseen test examples, which resolves the ostensible tension between deep double descent and the classical U-curve. Importantly, note that ptest can be computed without access to test-time labels, which means that the observed difference between pfrain and preat allows to quantify whether there is benign overfitting [BLLT20 21]+ in a neural network.
ReLU networks of increasing
Grokking: Model complexity throughout training. The grokking phenomenon 2B then showcased that improvements in test performance during a single training run can occur long after perfect training performance has been achieved (contradicting carly stopping practice!). While [LM222 attribute this to weight decay causing ||0,|| || to shrink late in training which they demon- strate on an MNIST example using unusually large 0 [KGP224] highlight that grokking can also polynomial regression task. In Fig. we replicate both experiments while tracking pg to investigate whether
occur as the weight norm ||0,]| grows later in training which they demonstrate
"As detailed in Appendix into binary classification task with lower learning rate 7 to enable the use of fe, (x). The reduction of y is needed here as the 20, are otherwise too large to obtain an accurate approximation and has a side effect that the grokking phenomenon appears visually less extreme as perfect training performance is achieved later in training.
replicate [KHGP24 's experiment exactly hut adapt MLM122 's experiment
5

---

(1) Polynomial Regression
(2) MNIIST Large Initialization
(3) MNIST Standard Initielization
1.0
0.5
Brrex
0.3
0.3-
0.00
0.0
0.00
10
10'
10
10"
10
10
10
Elliotive Pararattess
500
Train
1000
Trai
1000
Test with atL
Test
Test
Ted w/o
250
300
500
10
10 10
10
10'
10
10"
10a
10
Optimization stop
Optimization step
Optinizzation
stop f
Figure 4: Grokking in mean squared error on polynomial regression task (1, replicated from [BGP24] ) and
in misclassification error MNIST using network with large initialization (2, replicated from [LM222 (top),
against effective parameters (bottom). Column (3) shows test results MNIST with standard initialization
(with and without sigmoid activation) where time to generalization quick and grokking does not occur.
this provides new insight into this apparent disagreement. Then, we observe that the continued improvement in test error, past the point of perfect training performance, is associated with diver- gence of parain and pgaa in both experiments (analogous to the double descent experiment in Fig. 3) suggesting that grokking may reflect transition into measurably benign overfitting regime during training. In Appendix [D2] we additionally investigate mechanisms known to induce grokking, and show that later onset of generalization indeed coincides with later divergence of pg
parasn and yeett
Inductive biases & learned complexity. We observed that the large Og in [LMT222's MNIST exam- ple of grokking result in very large initial predictions [fe, (x) 1. Because no sigmoid is applied, the model needs to learn that all y E [0, 1] by reducing the magnitude of predictions substantially large O, thus constitute very poor inductive bias for this task. One may expect that the better inductive bias is, the less complex the component of the final prediction that is learned from data. To test whether this intuition is quantifiable, and without sigmoid activation (.). in column (3) of Fig. (training results shown in Appendix D.2 for readability). We indeed find that both not only speed up learning significantly (a generalizing solution is found in 102 instead of 10 steps), but also substantially reduce effective parameters used, where the stronger inductive bias using (-)) indeed leads the least learned complexity.
repeat the MNIST experiment with standard initialization scale, with
Takeaway Case Study 1. The telescoping model enables us to use p as proxy for learned complexity, whose relative behavior on train and test data can quantify benign overfitting in neural networks.
4.2 Case study 2: Understanding differences between gradient boosting and neural networks
Despite their overwhelming successes on image and language data, neural networks are perhaps surprisingly - still widely considered to be outperformed by gradient boosted trees (GBTs) on tabular data, an important modality in many data science applications. Exploring this apparent Achilles heel of neural networks has therefore been the goal of multiple extensive benchmarking studies [GOV22] MKV+23] Here, we concentrate on specific empirical finding of [MKV+23] their results suggest that GBTss may particularly outperform deep learning on heterogeneous data with greater irregularity in input features, a characteristic often present in tabular data. Below, we first show that the telescoping model offers useful lens to compare and contrast the two methods, and then use this insight to provide and test a new explanation of why GBTs can perform better in the presence of dataset irregularities.
Identifying (dis)similarities between learning in GBTs and neural networks. We begin by introducing gradient boosting |TOI closely following [HTF09 Ch. 10.. Gradient boosting (GB) also aims to learn predictor jGB X - RR* minimizing expected prediction loss f. While deep learning solves this problem by iteratively updating a randomly initialized set of parameters that transform inputs to predictions, the GB formulation iteratively updates predictions directly without requiring any iterative learning of parameters thus operating in function space rather than parameter space. Specifically, GB, with learning rate and initialized at predictor ho (x), consists of sequence upon BGB (x) hx(x) + Σi= hx(x)) where each h, (x) improves upon the existing predictions fCB (x).
The solution to the loss minimization problem can be achieved by executing steepest descent in function space directly, where each update h₂ simply outputs the negative training gradients of the loss function with respect to the previous model, i.c. hr -ge where gf, = a80fFB /a (xx).

---

However, this process is only defined at the training points fx,, Vi hejn]. To obtain an estimate of the loss gradient for an arbitrary test point x, each iterative update instead fits a weak learner (-) to the current input-gradient pairs (x,, - Jeli which can then also be evaluated new, unseen inputs. While this process could in principle be implemented using any base learner, the term gradient boost- ing today appears to exclusively refer to the approach outlined above implemented using shallow trees as h..()) [riOT]. Focusing on trees which issue predictions by averaging the training outputs in each leaf, we can make use of the fact that these are sometimes interpreted as adaptive nearest neighbor es- timators or kernel smoothers [[J06] BDI0 CJvdS24), allowing us to express the learned predictor as:
T
1 1le (x) = (x.))
T
jGB (x) = hn(()) -
Dit = -ho(x)x
Σ
Σ
Ki, (x, xi)g (9)
(((x)
t= RE(M)
1=1
where 1/, (x) denotes the leaf example x falls into, (1(x)) = Eiglni 10,, (x) = I, (x:) is the number of training examples in said leaf and i,, (x,x;) = 1/mccussss 1 (7, (x) (x,)) is thus the kernel learned by the 4th tree f.).)). Comparing Eq. (9 to the kernel representation of the telescoping model of neural network learning in Eq. ( we make a perhaps surprising observation: the telescoping model of neural network and GBT's have identical structure and differ only in their used kernel! Below, we explore whether this new insight allows to understand some of their performance differences.
Why can GBTss outperform deep learning in the presence of dataset irregularities? Comparing Eq. ( and Eq. ( thus suggests that at least some of the performance differences between neural networks and GBTs are likely to be rooted in the differences between the behavior of the neural network tangent kernels K. (x, xx) and GBT's tree kernels K,, (x, x.). One difference is obvious and purely architectural: it is possible that either kernel encodes better inductive bias fit the outcome-generating underlying outcome- generating process of dataset at hand. Another difference is more subtle
and relates to the behavior of the learned model on new inputs x: the tree kernels are likely to behave much more predictable at test-time than the neural network tangent kernels. To see this, note that for the tree kernels we have that Vx E X and Vi E [n], 0 I Ki,, xx) S and Eiein) Ki (x, x,) = 1; importantly, this is true regardless of whether x = x, for some or not. For the tangent kernels on the other hand, K2 (x, x:) is in general unbounded and could behave very differently for x not observed during training. This leads us to hypothesize that this difference may be able explain [MKV NKK'+23J's s observation that GBTs perform better whenever features heavy-tailed: if test point x is very different from training points, the kernels implied by the neural network K (x) == [K] (x, x1), -..., K2 (x, x," may behave very differently than at train-time
while the tree kernels kg, (x) == j R, (x,x1), xI,....... Kj, (x, xn)] will be less affected. For instance, vn |k, (x)]l2 I for all x while ||k] (x)]l2 is generally unbounded.
We empirically test this hypothesis on standard tabular benchmark datasets proposed in [GV22]]. We wish to examine the performance of the models and the behav- ior of the kernels inputs become increasingly irregular, evaluating if GBT's kernels indeed display more consis- tent behavior compared to the network's tangent kernels. As a simple notion for input irregularity, we apply prin- cipal component analysis to the inputs to obtain a lower dimensional representation of the data and sort the ob- servations according to their distance from the centroid.
300
Relative increase in
MSB difference (96)
Tist MSE
200
NN
00.08 (= 0.00
GBT
0.861 (4 (U000)
100
0.125
0.25
0.375
0.5
Normmlized mnximauu
harnel weight norm
2.0
NN
GBT
For a fixed trained model, we then evaluate on test sets consisting of increasing proportions p of the most irreg-
1.5
ular inputs (those in the top 10% furthest from the cen-
troid). We compare the GBTs to neural networks by ex- amining (i) the most extreme values their kernel weights
take at test-time relative to the training data (measured ET. max max ||ke (az)) l2
0.125
0.25
0.375
0.5
as E. maxieFerac (x.12 and (ii) how their rela-
Proportion of irregular examples
Figure 5: Neural Networks vs GBTs: Rel- alive performance (top) and behavior ker- nels (bottom) with increasing test data irreg- ularity using the houses dataset.
tive mean squared error (measured MSE" MSF GeT changes as the proportion p of irregular examples in-
creases. In Fig. 5 using houses and in Appendix D.3
7

---

using additional datasets, we first observe that GBT's outperform the neural network already in the absence of irregular examples; this highlights that there may indeed be differences in the suitability of the kernels in fitting the outcome-generating processes. Consistent with our expectations, we then find that, as the test data becomes more irregular, the performance of the neural network decays faster than that of the GBTss Importantly, this is well tracked by their kernels, where the unbounded nature of the network's 's tangent kernel indeed results in it changing its behavior on new, challenging examples.
Takeaway Case Study 2. Eq. (5) provides a new lens for comparing neural networks to GBTs, and high- lights that unboundedness in k. (x) can predict performance differences due to dataset irregularities.
4.3 Case study 3: Towards understanding the success of weight averaging
The final interesting phenomenon we investigate is that it is sometimes possible to simply average the weights O2 and 0g obtained from two stochastic training runs of the same model, resulting in a weight- averaged model that performs no worse than the individual models (FDRC20 AHS22 which has important applications in areas such as federated learning. This phenomenon is known as linear mode connectivity (LMC) and is surprising as, a priori, it is not obvious that simply averaging the weights of independent neural networks (instead of their predictions, as in a deep ensemble [LPB17). which are highly nonlinear functions of their parameters, would not greatly worsen performance. While recent work has demonstrated empirically that it sometimes possible to weight-average an even broader class of models after permuting weights [SJ20, ESSNNT AHS222, we focus here on understanding when LMC can be achieved for two models trained from the same initialization to-
In particular, we are interested in 2FDRC20)s 's observation that LMC can emerge during training: the weights of two models 0% E {1,2}, which are initialized identically and follow identical optimiza- tion routine up until checkpoint t' but receive different batch orderings and data augmentations after t', can be averaged to give an equally performant model as long as t' exceeds a so-called stability point t", which was empirically discovered to occur early in training in [FDRC20] Interestingly, 2D Sec. 5] implicitly hint at an explanation for this phenomenon in their empirical study of tangent kernels and loss landscapes, where they found an association between the disappearance of loss barriers between solutions during training and the rate of change in K. (:.-.. We further explore potential implications of this observation through the lens of the telescoping model below.
Why a transition into a constant-gradient regime would imply LMC. Using the beight-averaggng representation of the telescoping model, it becomes easy to see that not only would stabilization of the tangent kernel be associated with lower linear loss barriers, but the transition into lazy regime during training- - i.e. reaching point t* after which the model gradients no longer change can be sufficient to imply LMC during training as observed in [DRC20 under a mild assumption on the performance of the two networks' ensemble. To see this, let L(J) == (xxx=p((([[X)))) Y)] denote the expected loss of ʃ and recall that if [ -00 -1--)))) - [aL(Ser 0
)] S 0 then LMC is said to hold. If we assume that ensembles (x(x = ale; (x) +1-1aa)) (x) perform no
)+(+ (1 La) 0i
worse than the individual models (i.e. L(fa) S af + (1( ) Va [0, 1]. as is usually the case in practice (ABCC23)) ) then one case in which LMC guaranteed is if the predictions of weight-averaged model and ensemble are identical. In Appendix B.2 we show that if there exists some t" E [0, T) after which the model gradients Voloç: (.) no longer change (i.e. for all t' 2 t" the
learned updates 0j lie in a convex set @xtable in which Voleg (-) ~ Voso.- (-)), then indeed
T (x( R Saor, +(-000,, (x) Ro fo, (x) + Volo.. (x) Tx) Ee
+ +(1 - a))0))) (10)
That is, transitioning into a regime with constant model gradients during training can imply LMC because the ensemble and weight-averaged model become near-identical. This also has as an immediate corollary that models with the same O, which train fully within this regime (e.g. those discussed in [IGH88 LXS+19) will have t = O. Note that, when using nonlinear (final) output activation (-) the post-activation model gradients will generally not become constant during training (as we discuss in Sec. for the sigmoid and as was shown theoretically in [LZB20] for general nonlinearities). If, however, the pre-activation model grudients become constant during training and the pre-uctivation ensemble which averages the two model's 's pre-activation outputs before applying (.)) performs no worse than the individual models (as is also usually the case in practice ULCvdS24]), ), then the above also immediately implies LMC for such models.
8

---

Accuracy 8ap
Geadiont change (rand
(3) Gradiont change (protrained)
Latar
0.5
Rand init Pretrained
0.8
HEdden layens
0.3
Hidden Jayers
layers
Lineor outpot lager
Linear output layee
0.1
0.1
0.0
0.0
BatehNorm params
BatehNorm params
02
0.2
0.1
0.1
0.0
R011
2001
1973
10001
100
100
12
3007
4473
0.0
235
1997
N75
Barar
layers
Figure 6: Linear mode connectivity and gradient changes by t'. (1) Decrease in accuracy when using
averaged weights a0'r + (1 - a 0z7 for randomly initialized (orange) and pre-trained ResNet-20 (green).
(2) & (3) Changes in model gradients by layer for randomly initialized (2) and pretrained (3) model.
This suggests a candidate explanation for why LMC emerged at specific points in [DRC20) To test this, we replicate their CIFAR-10 10 experiment using a ResNet-20 in Fig. 6 In addition to plotting the maximal decrease in accuracy when comparing f faor +00 (x) to the weighted average of
the accuracies of the original models as [FDRC20) to measure LMC in (1), we also plot the squared change in (pre-softmax) gradients (Vofo,-+300 (x) VoJo, (x))² over the next epoch (390 batches) after checkpoint L', averaged over the test and the parameters in each layer in (2). We find that the
disappearance of the loss barrier indeed coincides with the time in training when the model gradients become more stable across all layers. Most saliently, the appearance of LMC appears to correlate with the stabilization of the gradients of the linear output layer. However, we also continue to observe some changes in other model gradients, which indicates that these models do not train fully linearly.
Pre-training and weight averaging. Because weight averaging methods have become increasingly popular when using pre-trained instead of randomly initialized models NSZ20 WIG 22 CVSK22) we are interested in testing whether pre-training may improve mode connectability through stabilizing the model gradients. To test this, we replicate the above experiment with the same architecture pre-trained on the SVHN dataset (in green in Fig. I)). Mimicking findings of [NS720], we first find the loss barrier to be substantially lower after pre-training. In Fig. 63,, we then observe that the gradients in the hidden and final layers indeed change less and stabilize earlier in training than in the randomly initialized model- yet the gradients of the BatchNorm parameters change more. Overall, the findings in this section thus highlight that while there may be a connection between gradient stabilization and LMC, it cannot fully explain it suggesting that further investigation into the phenomenon using this lens, particularly into the role of BatchNorm layers, may be fruitful.
Takeaway Case Study 3. Reasoning through the learning process by telescoping out functional updates suggests that averaging model parameters trained from the same checkpoint can be effective if their models' gradients remain stable, however, this cannot fully explain LMC in the setting we consider.
5 The Effect of Design Choices on Linearized Functional Updates
The literature on the neural tangent kernel primarily considers plain SGD, while modern deep learning practice typically relies range of important modifications to the training process (see e.g. [Pr223] Ch. 6]) - this includes many of the experiments demonstrating surprising deep learning phenomena we examined in Sec. 4 To enable to use modern optimizers above, we derived their implied linearized functional updates through the weight-averaging representation A Vofo.,, (x) (x),, which in turn allows to define K2' (---) in Eq. (5) for these modifications using straightforward algebra. As a by- hy-product, we found that this provides us with an interesting and pedagogical formalism to reason about the relative effect of different design choices in neural network training, and elaborate on selected learnings below.
Momentum with scalar hyperparameter 31 smoothes weight updates by employing exponentially weighted average over the previous parameter gradients AQ, - 1-3
1-3 Ei (
instead of using the current gradients alone. This implies linearized functional updates
fx) - EiE(n) (K? (x, )9f + - -
1-3
(xx
(x, x; xx)9
(11)
where K. x,) = 1(iCB.) V Peffo_.00 (x) Vofo._, (x,) denotes the cross-temporal tangent kemel. Thus, the functional updates also utilize previous loss gradients, where their weight is determined

---

using an inner product of the model gradient features from different time steps. If V Vefe, (x) is constant throughout training and we use full-batch GD. then the contribution of each training example i to fx( reduces to - X (x, xx) 1 E 88 I an exponentially weighted moving average over its past loss gradients making the effect of momentum on functional updates analogous to its effect on updates in parameter space. However, if Vefe, (x) changes over time, it is e.g.
possible that K2, (x, x;) has opposite sign from K2 (x, x,) in which case momentum reduces instead of amplifies the effect of previous g This is more obvious when re-writing Eq. to collect all 1-3, s
terms containing a specific Ih. leading to KT x;) Σk= 7k
K x;) for Eq. (5)
1-
Weight decay with scalar hyperparameter À uses 20, - (T.g. + x0, 1). For constant learning
rate y this gives 0, = 0u - Σx= > (T + x0x=1) = (1 - -x)) 00 7 Σ k= (1 "TT)
This then implies linearized functional updates
(f((x) = -7 ExEma (K((x, x;)g - Xy E1-1 (1 -17) 1 Ke, (x, x; (196))
(12)
->1(1 - Xx) - ( - (x) 00
Vo....
For full-batch GD and constant tangent kernels, ->K x; )0se - xo (1- 17)"
the contribution of each training example to the functional updates, which effectively decays the previous contributions of this example. Further, comparing the signs in Eq. (12) to Eq. (II highlights that momentum can offset the effect of weight decay on the learned updates in function space (in which case weight decay mainly acts through the term decaying the initial weights On).
learning rates are another important modification in practice which enable the use of different step-sizes across parameters by dividing A0, elementwise by p x 1 scaling vector Φi. Most prominently, this is used to adaptively normalize the magniude of updates
Adaptive & parameter-dependent
/1-8 Σi= 82 (Tag e). When combined with plain SGD, (e.g. Adam KB uses Φ; V 1-33
this results in kernel (x,,)) = 1(EEB.) Vojo._, (x) ( de (x)o This expression highlights that Φb admits an elegant interpretation as re-scaling the relative influence of features on the tangent kernel, similar to structured kernels in non- con-paraettric regression [HTF09, Ch. 6.4.1].
[8,
Architecture design choices also impact the form of the kernel. One important practical example whether (x) applies a non- activation function to the output (x) E of its final layer. Consider the choice of using the sigmoid (22) = Ite-- for binary classification problem and recall 02 a (z) = a (=) - (z)) E (0, 1/4), which is largest where (=) = 1/2 and smallest when (=)) + V 1. If k. (x, x.) = [B.1 Va 9o, .x))
T Vo ga: , (x:) denotes the tangent kernel of the model without activation, it is easy to see that the tangent kernel of the model (go, (x)) is
B.
(x,x:) - (g)))())
(go, (x)))o(ga, (xx)) ogo, (x))) x;)
(13)
indicating that K.'"(x, x;) will give relatively higher weight in functional updates to training examples i for which the model is uncertain (o((xx)) 1/2)) and lower weight to examples where the model certain (g (x:) Rs OV 1) regardless of whether (go, (x:) is the correct label. Conversely, Eq. (13) also implies that when comparing the functional updates of (x)(x)) to those of (x) across inputs x X. updates with (-)) will be relatively larger for x where the model is
uncertain (a (ge, (x)) se 1/2)). Finally, Eq. (13 also highlights that the (post-activation) tangent kernel of a model with sigmoid activation will generally not be constant in f unless the model predictions a (ge, (x)) do not change.
6 Conclusion
This work investigated the utility of a telescoping model for neural network learning, consisting of a sequence of linear approximations, as a tool for understanding several recent deep learning phenomena. By revisiting existing empirical observations, we demonstrated how this perspective provides a lens through which certain surprising behaviors of deep learning can become more intelligible. In each case study, we intentionally restricted ourselves to specific, noteworthy empirical examples which we proceeded to re-examine in greater depth. We believe that there are therefore many interesting opportunities for future research to expand on these initial findings by building upon the ideas we present to investigate such phenomena in more generality, both empirically and theoretically.
10

---

Acknowledgements
We would like thank James Bayliss, who first suggested to us to look into explicitly unravelling SGD updates to write trained neural networks as approximate smoothers to study deep double descent after a seminar on our paper [CJvdS23] on non-deep double descent. This suggestion ultimately inspired many investigations far beyond the original double descent context. We are also grateful to anonymous reviewers for helpful comments and suggestions. AC and AJ gratefully acknowledge funding from AstraZeneca and the Cystic Fybrosis Trust, respectively. This work supported by G-Research grant, and Azure sponsorship credits granted by Microsoft's AI for Good Research Lab.
References
[ABNH23] Gül Sena Altintas, Gregor Bachmann, Lorenzo Noci, and Thomas Hofmann. Dis- entangling linear mode connectivity. In UniReps: the First Workshop on Unifying Representations Neural Models, 2023.
[ABPC23] Taiga Abe, Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. Pathologies of predictive diversity in deep ensembles. Transactions on Machine Learning Research, 2023.
[AHS22] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint S0tiv:2220.0488200 2022.
[AP20]
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine- grained bias-variance decomposition. Advances neural information processing systems, 33:1 33:11022-11032 2020.
[ASS20]
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-446, 2020.
[BBL03]
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning theory. In Summer school on machine learning, pages 169-207. Springer, 2003.
[BD10]
Gérard Biau and Lue Devroye. On the layered nearest neighbour estimate, the bagged nearest neighbour estimate and the random forest method in regression and classification. Journal of Multivariate Analysis, 101(10):2499-511 2010.
[Bel21]
Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203-248, 2021.
[BHMM19]
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical biricvari trade-off. Proceedings of the National Academy of Sciences, 116(3)):
2019.
[BHX20] Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features.
SIAM Journal on Mathematics of Data Science, 2(4): 2020.
[BLLT20] Peter L. Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117643)30006-30000 2020.
[BM19]
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
[BMM18] Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pages 541-549. PMLR, 2018
[BMR+20] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
[BO%6]
Siegfried Bös and Manfred Opper. Dynamics of training. Advances in Neural Informa- tion Processing Systems, 9, 1996.
[Bre01]
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
11

---

[BSM 22] Frederik Benzing, Simon Schug, Robert Meier, Johannes Von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger. Random initialisations performing above chance and how to find them. arXiv preprint arXiv:2209.07500 2022.
[CIvdS23] Alicia Curth, Alan Jeffares, and Mihaela van der Schaar. A u-turn on double descent:
Rethinking parameter counting in statistical learning, Advances in Neural Information Processing Systems, 36, 2023.
[CJvdS24] Alicia Curth, Alan Jeffares, and Mihacla van der Schaar. Why do random forests work? understanding tree ensembles as self- self-regul adaptive smoothers. arXiv preprint arXiv:2402.01502, 2024.
[CL21] Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. The Journal of Machine Learning Research, 22(1):5721-5750, 2021.
[CMBK21] Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve. Advances in Neural Information Processing Systems, 34:8898-8912 12, 2021.
[COB19] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
[Cur24]
Alicia Curth. Classical statistical (in-sample) intuitions don't t generalize well: A note on bias-variance tradeoffs, overfitting and moving from fixed to random designs. arXiv preprint arXiv:2409. 18842, 2024.
[CVSK22] Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, arXiv preprint arXiv:220400004 2022.
[Die02]
Thomas G Dietterich. Ensemble learning. The handbook of brain theory and neural networks, 3(1):110-133 2002.
[DLL*19] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
[DLM20]
Michal Derezinski, Feynman Liang, and Michael W Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. Advances in neural information processing systems, 33:5152-5164, 2020.
[Dom20]
Pedro Domingos. Every model learned by gradient descent is approximately kernel machine. arXiv preprint arXiv:2012.0011: 2020.
[dRBK20]
Stéphane d' Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning, pages 2280-2290. PMLR, 2020.
[DVSHI8] Felix Draxler, Kambis Veschgini. Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318. PMLR, 2018.
[ESSN21] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXivv21100062966 2021.
[FB16]
C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint (tt:::6611.05444 2016.
[FDP+20] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an em- pirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861. 2020.
[FDRC20]
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[FHL119] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv: 2019.
12

---

[Fri01]
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[GBD92] Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
[GIP+18) Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dons. Advances in newral information processing systems, 31, 2018.
[GK24] Samuel James Greydanus and Dmitry Kobak. Scaling down deep learning with mnist- 1d. In Forty-first International Conference on Machine Learning, 2024.
[GMMM19] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limita- tions of lazy training of two-layers neural network. Advances in Neural Information Processing Systems, 32, 2019.
[GOV22] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? Advances in neural information processing systems, 35:507-520, 2022.
[GPK22] Eugene Golikov, Eduard Pokonechnyy, and Vladimir Korviakov. Neural tangent kernel: A survey. arXiv preprint
2022.
[GSJW20]
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11): 13301, 2020.
[HHLS24] Moritz Haas, David Holzmüller, Ulrike Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. Advances in Neural Information Processing Systems, 36, 2024.
(HMRT22) Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani.. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statisties, 50(2):949 986, 2022.
[HT90]
Trevor Hastie and Robert Tibshirani. Generalized additive models. Monographs statistics and applied probability. Chapman & Hall, 43:335, 1990.
[HTF09]
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[HXZQ22] Zheng He, Zeke Xie. Quanzhi Zhu, and Zengchang Qin. Sparse double descent: Where network pruning aggravates overfitting. In International Conference on Machine Leurning, pages 8635-8659. PMLR, 2022.
[HZRS16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[Ide]
Yerlan Idelbayev. Proper ResNet implementation for CIFARIO/CIFARI00 in PyTorch. comp http://www.....aaoooooooooooeer.
Accessed: 2024-
05-15.
[IPG 18]
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gor- don Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint a803 800.0054770 2018.
[IWG+22] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Ha- jishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open- vocabblcyyy models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262 2729276-22277, 2022.
[JGH88] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Conver- gence and generalization in neural networks. Advances neural information processing systems, 31, 2018.
[JLCvdS24] Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela van der Schaar. Joint train- ing of deep ensembles fails due to learner collusion. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[KAF*24] Devin Kwok, Nikhil Anand, Jonathan Frankle, Gintare Karolina Dziugaite, and David Rolnick. Dataset difficulty and the role of inductive bias. arXiv preprint Sttv:2401.01866) 2024.
[KB14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:/412 /4l..:9902 2014.
[KBGP24]
Tanishq Kumar, Blake Bordelon, Samuel Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. In The Twelfth International Conference on Learning Representations, 2024.
[KH+09] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[KSH12]
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[LBBH98]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bart basee learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-233) 1998.
(LBBS24)
Noam Levi, Alon Beck, and Yohai Bar-Sinai. Grokking in linear estimators-an solv- able model that groks without understanding. International Conference on Learning Representations, 2024.
[LD21]
Licong Lin and Edgar Dobriban. What causes the test error? going beyond bias-variance via anova. The Journal of Machine Learning Research, 22(1))6955-7006 2021.
[LH17]
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
[LJ06]
Yi Lin and Yongho Jeon. Random forests and adaptive nearest neighbors. Journal of the American Statistical Association, (0144))):578-55 590, 2006.
[IJL+224] Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon Shaolei Du, Jason D Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking. In The Twelfth International Conference on Learning Representations, 2024.
[2 Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34663, 2022.
[LMT22] Ziming Liu, Eric Michaud, and Max Tegmark. Omnigrok: Grokking beyond algo- rithmic data. In The Eleventh International Conference on Learning Representations, 2022.
[LPB17] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
ILSP+20] 20] Jachoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 33:15156-1517,, 2020.
[LVM+200]
Marco Loog, Tom Viering, Alexander Mey, Jesse H Krijthe, and David MJ Tax. brief prehistory of double descent. Proceedings of the National Academy of Sciences, 1 17720)):0225-1022 2020.
[LXS+ 19]
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances neural information processing systems, 32, 2019.
[LZB20]
Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non- non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 13:15554-1596 2020.
[Mac91] David MacKay. Bayesian model comparison and backprop nets. Advances in neural information processing systems, 4, 1991.
14

---

[MBB18] Siyuan Ma. Raef Bassily, and Mikhail Belkin. The power of interpolation: Understand- ing the effectiveness of sgd in modern over-parametrizer learning. In International Conference on Machine Learning, pages 3325-3334. PMLR, 2018.
[MBS23] Mohamad Amin Mohamadi, Wonho Bae, and Danica J Sutherland. A fast, well-founded approximation to the empirical neural tangent kernel. In International Conference on Machine Learning, pages 25061-25081. PMLR, 2023.
[MBW20] Wesley J Maddox, Gregory Benton, and Andrew Gordon Wilson. Rethinking pa- rameter counting in deep models: Effective dimensionality revisited. arXiv preprins ar arXiv:2003.021.39 2020.
[MKV*23] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C. Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2023.
[MOB24] Jack Miller, Charles O' O'Neill, and Thang Bui. Grokking beyond neural networks: An empirical exploration with model complexity. Transactions on Machine Learning Research (TMLRR, 2024.
[Moo91]
John Moody. The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems. Advances in neural information processing systems, 4, 1991.
[MSA (MSA+22] 22] Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Processing Systems, 35:1 182-1195, 2022.
[NCL+23] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
[9ea19]
Brady Neal. On the bias-variance tradeoff:: Textbooks need an update. arXiv preprint arXiv: 1rXiv:/9/2.000 191200886,, 2019.
[NKB+21] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of 2021.
Statistical Mechanics: Theory and Experiment, 2021
[NMB+188 18] Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julient and loannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint arXiv:/810.000
2018.
[NSZ20]
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 13:512--23, 2020.
[NVKM20] Preetum Nakkiran, Prayaag Venkat, Sham Kakade, and Tengyu Ma. Optimal regular- ization can mitigate double descent. arXiv preprint arXiv: 01897, 2020.
[NWC+11] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 7. Granada, Spain, 2011.
[OJFF24]
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
[OJMDF21] Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? Advances Neural Information Processing Systems, 34:8998-900 90100, 2021.
[PBE* 22]
Alethea Power, Yuri Burda, Hari Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
[Pri23]
Simon JD Prince. Understanding Deep Learning. MIT press, 2023.
15

---

[PVG*11]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, Dubourg, J. Vanderplas. A. Passos, D. Coumapeeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[2 Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamoniy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution gen- eralization. Advances in Neural Information Processing Systems, 35:10821-10836, 2022.
[SGd+ 18] Stefano Spigler, Mario Geiger, Stéphane d' Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to per--araamtrization affects loss landscape and generalization. arXiv preprint arXiv: 11100096000.0 2018.
[SIvdS23]
Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar. Dissecting sample hard- ness: Fine-grained analysis of hardness characterization methods. In The Twelfth International Conference on Learning Representations, 2023.
[SJ20]
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22085 2020.
[SKR 23] Rylan Schaeffer, Mikail Khona, Zachary Robertson, Akhilan Boopathy, Kateryna Pistunova, Jason W Rocks, Ila Rani Ficte, and Oluwasanmi Koyejo. Double descent demystified: Identifying, interpreting & ablating the sources of a deep learning puzzle. arXiv preprint 1rXiv:2303.1415/ 2023.
[TLZ*22] Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. arXiv preprint
2022.
[Vap95]
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1995.
[VCR89] F Vallet, J-G Cailton, and Ph Refregier. Linear and nonlinear extension of the pseudo- inverse solution for learning boolean functions. Europhysics Letters, 9())3115, 1989.
[VSK*23] Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXXv:2309.0233: 2023.
[VvRBTI3] Joaquin Vanschoren, Jan N. van Rijn. Bernd Bischl, and Luis Torgo. Openml: networked science machine learning. SIGKDD Explorutions, 15(2):99-60, 2013.
[WIG+22] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 11965-23998, PMLR, 2022.
[WOBM17] Abraham Wyner, Matthew Olson, Justin Bleich, and David Mease. Explaining the success of adaboost and random forests as interpolating classifiers. Journal of Machine Learning Research, 18(48):1-33, 2017.
[YHT+21] +21]
Yaoqing Yang, Liam Hodgkinson. Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Advances in Neural Information Processing Systems, 14:18722-18777 2021.
16

---

Appendix
This appendix is structured as follows: Appendix A presents extended literature review, Ap- presents an extended discussion of experimental setups and Appendix D presents additional results. The NeurlPS paper checklist is included after the appendices.
pendix [B] presents additional theoretical derivations, Appendix C
A Additional literature review
In this section, we present an extended literature review related to the phenomena we consider in Sec. and Sec.
A.1 The model
relationship (Sec. 4.1
Classical statistical textbooks convey well-understood relationship between model complexity historically captured by model's parameter count and prediction error: increasing model complexity is expected modulate transition between under- and overfitting regimes, usually represented by a U-shaped error-curve with model complexity on the x-axis in which test error first improves before worsens the training data can be fit too well [HT90) Vap95 HTF09]. While this relationship was originally believed to hold for neural networks as well (GBD92 later work provided
evidence that when using parameter counts to measure complexity this U-shaped relationship no longer holds [NMB* 18 Nea19].
Double descent. Instead, the double descent [BHMM19] shape has claimed its place, which postulates that the well-known U-shape holds only in the underparameterized regime where the number of model parameters p smaller than the number of training examples nı; once we reach the interpolation threshold increasing p further into the overparametrized (or: interpolation) regime leads to error improving again. While the double descent shape itself had been previously observed in linear regression and neural networks in [VCR89 BO%
at which models have sufficient capacity to fit the training data perfectly,
ASS20 NMB 18 SGd 18] (see also the historical note in 20)) the seminal paper by [HMMM1 both popularized it as a phenomenon and highlighted that the double descent shape can also occur tree-based methods. addition double descent as function of the number of model parameters, the phenomenon has since been shown to emerge also e.g. the number of training epochs NNBB 22] and sparsity (HXZQ22). Optimal regularization has been shown to mitigate double descent INVKM20]
Due to its surprising and counterintuitive nature, the emergence of the double descent phenomenon sparked a rich theoretical literature attempting to understand it. One strand of this literature has focused on modeling double descent in the number of features in linear regression and has produced precise theoretical meslyses for particular data- matta-enerating models A BLLT20 DL2
HMRT22 SKR 23 CCMK221] Another strand of work has focused on deriving exact expressions of bias and variance terms as the total number of model parameters is increased in a neural network by
taking into account all sources of randomness in model training [NMB + 18 A dRBK20 LD2I
A different perspective was presented in [CIvdS23, who highlighted that in the non-deep double descent experiments of [BHMMI9], a subtle change in the parameter- introduced exactly at the interpolation threshold, which is what causes the second descent. [ClvSS23 also demonstrated that when using a measure of the test-time effective parameters used by the model to measure complexity on the x-axes, the double descent shapes observed for linear regression, trees, and boosting fold back into more traditional U-shaped curves. In Sec. 4.1 we show that the telescoping model enables to discover the same effect also in deep learning.
parameter-increasing increasing mechanism is
Benign overfitting, Closely related to the double descent phenomenon is benign overfitting (c.g. [MM18 MBB18 BLLT20 CL21 MSA+ WOBMI7 HHLS244), i.e. the observation that, incompatible with conventional statistical wisdom about overfitting [HTF09), models with perfect training performance can nonetheless generalize well to unseen test examples. In this literature, it is often argued in theoretical studies that overparameterized neural networks generalize well because they are much more
around unseen test examples than examples seen during training [MSA 22 H24. Sec. 4.1 we provide new empirical evidence for this by highlighting that there is difference between ps and pg
17

---

Understanding modern model complexity. Many measures for model complexity capture some form of capacity of hypothesis class, which gives insight into the most complex function that could be learned- e.g. raw parameter counts and VC dimensions [BBL03] The double descent and benign overfitting phenomena prominently highlighted that complexity measures that consider only what could be learned and not what is actually learned for test examples, would be unlikely to help understand generalization in deep learning [Bel21]. Further, [CJvdS23] highlighted that many other measures for model complexity so-called measures of effective parameters (or: degrees of freedom) including measures from the literature of smoothers [HT90, Ch. 3.5] as well as measures relying on the model's Hessian [Mo09I [ac91] (which have been considered for use in deep learning in [MBW20] were derived in the context of in in-sample prediction (where train- and test inputs would be the same) and do thus not allow to distinguish differences in the behavior of learned functions on training examples from new examples. [Cur24] highlight that this difference in setting - the move from in-sample prediction measuring performance in terms of out-of-sample generalization is crucial for the emergence of apparently counterintuitive modern machine learning phenomena such as double descent and benign overfitting. For this reason, (CJvdS23) proposed an adapted effective parameter measure for smoothers that can distinguish the two, and highlighted that differentiating between the amount of smoothing performed on train- vs test examples is crucial to understanding double descent in linear regression, trees and gradient boosting. In Sec. 4.1 we show that the telescoping model makes it possible use [CydS233's 's effective parameter measure for neural networks, allowing interesting insight into implied differences in train- and test-time complexity of neural networks.
Grokking. Similar to double descent in the number of training epochs as observed in [NKB (where the test error first improves then gets worse and then improves again during training), the grokking phenomenon (P demonstrated the emergence of another type of unexpected be- havior during the training run of single model. Originally demonstrated on arithmetic tasks, the phenomenon highlights that improvements in test performance can sometimes occur long after perfect training performance has already been achieved. [LMT22] later demonstrated that this can also occur on more standard tasks such as image classification. This phenomenon has attracted much recent attention both because it appears to challenge the common practice of early stopping during training and because it showcases further gaps in our current understanding of learning dynamics. A number of explanations for this phenomenon have been put forward recently: [ attribute grokking to delayed learning of representations, [NCL- use mechanistic explanations to examine case studies of grokking, [VSK*2] 223 attribute grokking to more efficient circuits being learned later in training. [LM122 attribute grokking to the effects of weight decay setting in later in training and 2T attribute grokking to the use of adaptive optimizers. [KBGP24] highlight that the latter two explanations cannot be the sole reason for grokking by constructing an experiment where grokking occurs as the weight norm grows without the use of adaptive optimizers. Instead, [KBGP24 LJI. 24 conjecture that grokking occurs model transitions from the lazy regime to a feature learning regime later in training. Finally, [LBBS24] show analytically and experimentally that grokking can also occur in simple linear estimators, and [MOB24] similarly study grokking outside neural net- works, including Bayesian models. Our perspective presented in Sec. 1.1 is complementary to these lines of work: we highlight that grokking coincides with the widening of a gap in effective parameters used for training and testing examples and that there is thus a quantifiable benign overfitting effect at play.
A.2 Weight averaging in deep learning (Sec.
Ensembling Die02). i.e. averaging the predictions of multiple independent models, has long estab- lished itself as a popular strategy to improve prediction performance over using single individual models. While ensembles have historically been predominantly implemented using weak base learn- ers like trees to form random forests [BreOl], deep ensembles [LPB17] 1 i.c. ensembles of neural networks- have more recently emerged as a popular strategy for improving upon the performance of a single network [PB17] FHL19). Interestingly, deep ensembles have been shown to perform well both when averaging the predictions of the underlying models and when averaging the aetiviiiiasioss of the final network layers [LCvdS24).
A much more surprising empirical observation made in recent years is that, instead of averaging model predictions as in an ensemble, is sometimes also possible to average the learned weights 0 and Og of two trained neural networks and obtain model that performs well [IPG+ 18 FDRC20).
18

---

This is unexpected because neural networks are highly nonlinear functions of their weights, so it unclear a priori when and why averaging two sets of weights would lead to a sensible model at all. When weight averaging works, it is a much more attractive solution relative to ensembling: an ensemble consisting of models requires k X p model parameters, while a weight--veraged model requires only p parameters making weight-averaged models both more efficient in terms of storage and at inference time. Additionally,, weight averaging has interesting applications in federated learning because it could enable the merging of models trained on disjoint datasets. [IP 18] were the first to demonstrate that weight averaging can work in the context of neural networks by showing that model weights obtained by simple averaging of multiple points along the trajectory of SGD during training- - a weight-space version of the method of fast geometric ensembling [GIP* 18 could improve upon using the final solution directly.
Mode connectivity. The literature on mode connectivity first empirically demonstrated that there are simple (but nonlinear) paths of nonincreasing loss connecting different final network weights obtained from different random initializations (FB16 DVSH18 18] As discussed in the main text, [DDC220] then demonstrated empirically that two learned sets of weights can sometimes be linearly connected by simply interpolating between the learned weights, as long as two models were trained together until some stability point t" [ perform an empirical study investigating which networks and optimization protocols lead to mode connectivity from initialization (i.e. t" = 0) and which modifications ensure t' > O. As highlighted in Sec. 4.3 our theoretical reasoning indicates that one sufficient condition for linear mode connectivity from initialization is that models stay in a regime in which the model gradients do not change during training. In the context of task arithmetic, where parameters from models finetuned on separate tasks are added or subtracted (not averaged) to add remove a skill, [QIFF24 find that pretrained CLIP models that are finetuned separate tasks and allow to perform task arithmetic do not operate in a regime in which gradients are constant.
Methods that average weights Beyond [IGG 18) 's stochastic weight averaging method, which averages weights from checkpoints within a single training run, weight averaging has also recently gained increased popularity in the context of averaging multiple models finetuned from the same pre- trained model [NSZ20] WIG + 22 CVSK22]: while [NSZ20] showed that multiple models finetuned from the same pretrained model lie in the same loss basin and are linearly mode connectible, the model soups method of [WIG** 22) highlighted that simply averaging the weights of multiple models fine-tuned from the same pre- praitr parameters with different hyperparameters leads performance improvements over choosing the best individual fine-tuned model. A number of methods have since been proposed that use weight-averaging of models fine-tuned from the same pretrained model for diverse purposes (e.g. [RKR (2)). Our results in Sec. 4.3 S.3. complement the findings of [NSZ20] by investigating whether fine-tuning from pre-trained model leads to better mode connectivity because the gradients of a pre-trained model remain more stable than those trained from a random initialization.
Weight averaging after permutation matching, Most recently, a growing number of papers have investigated whether attempts to merge models through weight-averaging can be improved by first performing some kind of permutation matching that corrects for potential permutation symmetries in neural networks. [ESSNT]) conjecture that all solutions learned by SGD are linearly mode connectible once permutation symmetries are corrected for. [SI20] AHS22 BSM use different methods for permutation matching and find that this improves the quality of weight-averaged models.
B Additional theoretical results
B.1 Derivation of smoother expressions using the telescoping model
Below, we explore how we can use the telescoping model to express a function learned by a neural network as fo, (x) = Se, (x)y + cg, (x). where the 1 xxn vector Se, (x) is a function of the kernels (K) Jess. and the scalar <9, (x) is a function of the (K$ (:. le and the networks' initialization fe. (-). Note that, as discussed further in the remark at the end of this section, the kernels (,-),) for i are data-adaptive as they can change throughout training.
Vanilla SGD. Recall that letting y [M....., (ii....,y))) vn]] and fo [o, (xI),...,, So, (x.)). the SGD weight update with squared loss e(f(x). y) = [ - [(x))2 in the special case of sin- gle outputs k 1. simplifies to 20, Trly ff._,), where T, is the p x matrix
19

---

T, = [ V
1.nB.. V Vofo.,, (x,)). If we assume that the telescoping model [B
holds exactly, this implies functional updates (f.(x) 7a V Vofo. T, T, f)...) If we could write fo... 1 1 + co. then we would have
Af, = 7/ VoJo (x) Tely (S + co,-1)
(14)
To Vele. ( TI, Sam-ly Veje.1 (x) T T,c0.-1
where I, the n x n identity matrix. Noting that we must have Cd. = fao and Sa. 0 at initialization, we can recursively substitute Eq. (14) into Eq. (5) which then allows to write the vector of training predictions as
for
Σ
(I, 7 Tit Totk)
n.., T y TitxTetk)
S
(15)
TT A T T- k TT kk)
foo
A==0
coT
where the matrix T, VoJo, (xi)..... Vefe, (x,)] differs from Tr only in that it includes all training examples and not normalized by batch size. Then note that Eq. (5) is indeed a function of the training labels y, the predictions at initialization fe, and the model gradients (T.) traversed during training (captured in the n x n matrix Sø, and the n 1 vector co,) alone. Similarly, we can also write the weight updates (and, by extension, the weights 0r) using the same quantities, i.c. A., = y (T,, So,,))y 7 T By Eq. 5 this also implies that we can write predictions arbitrary test input points function of the same quantities:
(x) =
N Voso. (x) T.(I. So...) y+ (xo
Σ t=1
Yo Vofor. TTco...
S0z (x)y
coz (x)
where the matrix So, is as defined in Eq. 15 which indeed has so, (xx) as its i-th row (and analogously for co.-))
General optimization strategies. Adapting the previous expressions to enable the use of adaptive learning rates is straightforward and requires only inserting diag( 2, ,T, into the expression for
(x) instead of T, alone; then defining the matrices similarly proceeds by recursively unraveling ( TT, fo..1). 1). Both momentum and weight decay lead to somewhat more tedious updates and necessitate the introduction of additional notation. Let (x, = se, (x) se._, (x), with sa, (x) = 0 xn and Ac, = ce, (x) ce._, (x), with ce (x) fo, (x), that so, (x) Σt (x) and co, (x) fo, (x) ΣT (x). Further,
updates using (x) >h Velo. (x) diag( te
we can write
fx((x (x)y + (x) yi Volo, UU + 7o Volo. (x) U
(16)
which means that to derive Sg, (x) for each f, we can use the weight update formulas to define the p x update matrix U and the p x 1 update vector Uf that can then be used to compute (s, as 7 Vofe... (x) U and (x;(x) as Vo Vofo. (x) TU. For vanilla SGD,
U = (I, S and U -T,cO.-.
(17)
while SGD with only adaptive learning rates uses
1 TT, So. .) and U -diag( 1 - TTCo.
U diag(
(18)
Φ
Momentum, without other modifications, uses U 1
1-0 U and C, = I-3 UC, where
U (1 (--B1),(1,, S..-1) Se.-1)+ + and Uf - ((1 T. + +B,U (19)
20

---

with U = 0pxn and U = 0PX1
Weight decay, without other modifications, uses
U (In Se.- 1 + AD and U Tc(... + +DD
(20)
where D = Nt - + (1 A and D. = 1t-1U--1 + (1 - X with
DØ = 0px" and Dg -6 6g.
Putting all together leads to Adam W LH17 (which decouples weight decay and momentum, that weight decay does not enter the momentum term), which uses
U
diag(
U + XT,D; and C
1 1 3 diag
1 -g + AT.D.
(21)
Φt
31
where all terms are as in Eq. 19 and Eq.
Remark: Writing før = S + ca, is reminiscent of a smoother as used in the statistics literature [HT90]. Prototypical smoothers issue predictions
Sy which include k-Nearest Neighbor regressors, kernel smoother, and (local) linear regression as prominent members -, and are usually linear smoothers because S does not depend on y. The smoother implied by the telescoping model is not necessarily a linear smoother because Se,, can depend on through changes in gradients during training, making fe,, an adaptive smoother. This adaptivity in the implied smoother is similar to trees as recently studied in [CIvdS23 CJvdS24]. In this context, effectivee parameters as measured by pe can be interpreted as measuring how non-uniform and extreme the learned smoother weights are when issuing predictions for specific inputs [CvddS23.
B.2 Comparing predictions of ensemble and weight-averagee model after train-time transition into a constant-gus teattatt-gradiet regime
Here, we compare the predictions of the weight-averager model faof, +(-aa)( (x) to the ensemble
fo (xx = afao (x) + (1 - a)oe (x) if the models transition into lazy regime at time t* S t'.
We begin by noting that the assumption that the gradients no longer change after t" (i.e. Vofo: (-) R Vole.- (-) for all L 2 (*) implies that the rate of change of Vale.. (x) in the direction of the weight updates must be approximately O. That is, Vo.o.- (x) 0.-) R 0 for all 0 E @gtable, or equivalently all weight changes in each @stable are in directions that are in the null-space of the Hessian (or in directions corresponding to diminishingly small eigenvalues). To avoid clutter in notation, we use splitting point t' below, but note that the same arguments hold for t' t
First, we now consider rewriting the predictions of the ensemble, and note that we can now write the second-order Taylor approximation of each model føy: (x) around 0,.
Sozt (x) = fo. (x) + Vefo..
Σ
AO je +
Σ
AOt jt
Volo.. (x)
Σ
A
j0
[=+++1
200
Ra( Σ
A0%)
t
T
T
2 Sa (x) + Vole.- (x)
Σ I=t++
A + Ro
40%)
t=t+1
where Re BT A0))) contains remainders of order 3 and above. Then the prediction of the ensemble can be written as
T
(x( ~ fe.- (x) + se.. (x)T
(as0 - -11- a)-)
t=
(22)
+aRa( Σ
A0ti) + (1 a)) R2(
Σ
1=
t=
21

---

Now consider the weight- averaged model I faog, +1- (x). Note that we can always write
OTT 0. + Ei= A0ji O.. Ei A0%, and thus affr + (1 0) 0.. 21=r+1 (a + (1 a)) Further, because Volo, (x) Ei=r s0ij ss 0 for each
j E (0, we also have that
Vife.- (x)
Σ
aAon (1- (1--a)44 a)2012 2 a0 (1-a)0 =0
(23)
(=!+1
Then, the second-order Taylor approximation of Saoj +1- (x) around 0.- gives
fao +0-0000--000 (x) = (x) + VoJo.- (x)
(aS011 + (1-a))0m2))
(=
(24)
+ R2
A0 + (1 -0--011 2)2))
1=
Thus, Saoj +11 (1-a)) (x) v (x) up remainder terms of third order and above.
C Additional Experimental details
In this section, we provide a complete description of the experimental details throughout this work. Code is provided at https: //githu com/ alan jenferefses teeesstelleee /tt//ellescccoc telescoping- lens Each section
also reports their respective required compute which was performed on either Azure VMs powered by 4 X NVIDIA A100 GPUs or an NVIDIA RTX A4000 GPU.
C.1 Case study 1 (Sec. 4.1 and approximation quality experiment (Sec.
3
Fig.
Double descent experiments. In Fig. 3 we replicate [BHMMT9 Sec. S.3.3]'s only binary classifi- cation experiment which used fully connected ReLU networks with a single hidden layer trained using the squared loss, without sigmoid activation, cat and dog images from CIFAR- 10 KH+09) Like [HMMM9), we grayscale and downsize images to d = 8 8 format and use n = 1000 training exam- ples and use SGD with momentum $1 0.95. We use batch size 100 (resulting in B = 10 batches), learning rate y = 0.0025, and test on Netest = 1000 held out examples. We train for up to e = 30000 epochs, but stop when training accuracy reaches 100% when the training squared loss does not improve by more than 10-4 for 500 consecutive epochs (the former strategy was also employed in [BHMM19,, we additionally employ the latter detect converged networks). We report results us- ing (1, 5, 7, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 70, 85, 100, 200, 500, 1000, 2000, 5000} hidden units. We repeat the experiment for random seeds and report mean and standard errors in all figures.
In Appendix D.2 we additionally repeat this experiment with the same hyperparameters using MNIST To create a binary classification task, we similarly train the model to distinguish 3-vs-5 from 1000 images downsampled to d = 8 x 8 format and test on 1000 examples. Likely because the task is very simple, we observe no deterioration in test error in this setting for any hidden size (see Fig. 9). Because INKB 21) found that double descent can be more apparent in the presence of label noise, we repeat this experiment while adding 20% label noise to the training data, in which case the double descent shape in test error indeed emerges. As above, we repeat both experiments for 4 random seeds and report mean and standard errors in all figures.
images [L BBH98
Further, in Appendix D.2 we additionally utilize the MNIST-ID dataset [GK24] which was pro- posed recently as a sandbox for investigating empirical deep learning phenomena. We repli- cate a binary classification version of their MLP double descent experiment with added 15% label noise from [GK24] (which was itself adapted from the textbook (P723)) We select only examples with label 0 and 1, and train fully connected neural networks with single hidden layer with batch size 100, learning rate y = 0.01 for 500 epochs, considering models with (1,2,3, 5, 10, 20, 30, 40, 50, 70, 100, 200, 300, 400) hidden units.
Compute: We train num_ num_setttetttetee tim_settings x num num_hiddes wwwddhdddiiiiee sizes x num_ num__eeee seeds (> x 22 x 352) models for up To = B x 300000 gradient steps. Training times, which included all gradient
22

---

computations to create the telescoping approximation. depended on the dataset and hidden sizes, but completing single seed for all hidden sizes for one setting took an average of 36 hours.
Grokking experiments. panel (1) Fig. from KBGP24| Sec. 5] exactly. [KBGP24 use a neural network with single hidden layer, using custom nonlinearities, of width ny = 500 which the weights of the final layer are fixed, that is they use
we replicate the polynomial regression experiment
n
60, x) where ((h)= h h++ 1/2 2
(x
(25)
Thp
Inputs x. E Rd are sampled from an isotropic Gaussian with variance 1 and targets y are generated as v(x 1 [3Tx²² x)². In this setup, used in the activation function of the network controls how easy it is to fit the outcome function (the larger c, the better aligned is for the task at hand), which in turn controls whether grokking appears. In the main text, we present results using Appendix D.2 we additionally present results using = .05 and e = 0.5. Like KBGP24), we use d d= 100, Pltrain 550, Nitest 500, initialize all weights using standard normals, and train using full-batch gradient descent with = B = 500 on the squared loss. We repeat the experiment for random seeds and report mean and standard errors in all figures.
.2; in
In panel (2) of Fig. 4 we report an adapted version of [LMT22]'s experiment reporting grokking on MNIST data. To enable the use of our model, we once more consider the binary classification x 8 features and test on 1000 held-out examples. Like (LMT22), we a 3-layer fully connected ReLU network trained with squared loss (without sigmoid activation) and larger than usual initialization by using aQ, instead of the default initialization 9,. We report = 6 in the main text and include results with
task 3-vs-5 from n = 1000 images downsampled to
5 and a = 7 Appendix D.2 Like [LM222] we use the A optimizer [LH77] with batches of size 200, 81 -9 and 32 = .99, and use weight decay À = .1. While [LMT22 use learning rate 10-3, we need to reduce this by factor 10 to y 10-4 and additionally use linear learning rate warmup over the first 100 batches to ensure that weight updates are small enough to ensure the quality of the telescoping approximation; this is particularly critical because of the large initialization which otherwise results in instability in the approximation early in training. Panel (C) of Fig. 4 uses an identical setup but lets a = (i.e. standard initialization) and additionally applies a sigmoid to the output of the network. We repeat these experiments for random seeds and report mean and standard errors in all figures.
Compute: Replicating KBGP24)'s 's experiments required training num_ sen_stttinggssssse x num_ seeds (3 x 5 15) models for T = 100, 000 gradient steps. Each training run including all gradient computations took less than hour to complete. Replicating [MM22]'s 's experiments required training num_ tamm_settingg x num_ num_seeds (3 x 4 = 12) for = 100, 000 gradient steps. Each training run including all gradient computations took around 5 hours to complete. The MNIST experiments with standard initialization required training num_ aum_settings x num_ _en__seds (2 x 4 = 8) for T = 1000 gradient steps, these took more than 2 hours to complete in total.
The approximation quality experiment uses the identical MNIST setup, training process and architecture as in the grokking experiments (differing only in that we use standard initialization a and no learning rate warmup). In addition to the vanilla SGD and A experiments presented in the main text, we present additional settings using momentum alone, weight decay alone and using sigmoid activation - Appendix D.] In particular, we use the following hyperparameter settings for the different panels:
Approximation quality experiment (Fig. 2
"SGD": X 0, 31 = 0, no sigmoid.
"AdamW": X 0.1, 31 = 0.9, 82 .99, no sigmoid.
"SGD + Momentum", ": X = 0, 31 = 0.9, no sigmoid.
"SGD + Weight decay": X = 0.1, 31 = 0, no sigmoid.
"SGI) + a.)" X 0, 31 0, with sigmoid activation.
We repeat the experiment for 4 random seeds and report mean and standard errors in all figures.
23

---

Compute: Creating Fig. F rrquirrd training num_settings num num__seed (5x4= = 20) for T = 5. 000
gradient steps. Each traming run including all gradient computations took approximately 15 minutes to complete.
C.2 Case study 2 (Sec. 4.2)
In Figs. andd 14 we provide results on tabular benchmark datasets from GOV22] We select four datasets with 20,000 examples (houses, superconduct, california, house_ house_sales) to ensure there is sufficient hold-out data for evaluation across irregularity proportions. We apply standard preprocessing including log transformations of skewed features and target rescaling. As discussed in the main text, irregular examples are defined by first projecting each (normalized) dataset's 's input features onto its first principal component and then calculating each example's is absolute distance to the empirical median in this space. We note that several recent works have discussed metrics of an examples irregularity or "hardness" (e.g. KAFF 24 [IvdS233 finding the choice of metric to be highly context-depender on its simplicity and transparency. The top K irregular examples are removed from the data (these form the "irregular examples at test-time") and the remainder (the "regular examples") is split into training and testing. We then construct test datasets containing 4000 examples, constructed from a mixture of standard test examples and irregular examples according to each proportion p.
Therefore we select a principal component prototypicality approach based
We train both standard neural network (while computing its telescoping approximation as de- scribed in Eq. ) and a gradient boosted tree model (using [PVG* on the training data, We select hyperparameters by further splitting the training data to obtain a validation set of size 2000 and applying random search consisting of 25 runs. We use the search spaces suggested in [GVV22] Specifically, for GBTs we consider learning_rate E LogNormal (lg(0000)) log((1)]], num_estimators E tive probabilities LogUniformlnt( (10.5 1000.5). and max_ _eathh E None,, 2,3, 4, 5] with respec- 0.6, 0.1, 0.1). For the neural network, we consider learning_ rate 128, num_ aum_layers = 3. and hidden dim 64 with ReL.U activations throughout. Each model is then trained on the full training set with its optimal parameters and is evaluated on each of test sets corresponding to the various proportions of irregular examples. All models are trained and evaluated for random seeds and we report the mean and standard error in our results.
E LogUniform le - 5. le 2] and set batch_siae
As discussed in the main text, we report how the relative relative mean squared error of neural MSEN MSE
network and GBT (measured as MSE MSE
changes as the proportion p of irregular examples
+ -T= max
(x;)]l which measures how the kernels
increases and relate this to changes in Eizi maxie [x]]]
behave at their extreme during testing relative to the maximum of the equivalent values measured for the training examples such that the test values can be interpreted relative to the kernel at train time (i.e. values > 1 can be interpreted as being larger than the largest value observed across the entire training set).
Compute: The hyperparameter search results in num_ searches num_datasets num_modell (25 x 4 x 2 = 200) training runs and evaluations. Then the main experiment requires num num_seeds x num_de Lum_datasets num_ num_moode models (4 4 2 = 32) training runs and num_ seeds x num, nam_nnuaaaaaaaaaaaaa datasets X num_models x num_ www__proportiieo tions (4 x x 2 5 = 160) evaluations. This results in total of 232 training runs and 360 evaluations. Individual training and evaluation times depend on the model and dataset but generally require < hour.
C.3 Case study 3 (Sec. 4.3
In Fig. @wee follow the experimental setup described in [FDRC20). Specifically, for each model we train for total of 63,000 iterations over batches of size 128 with stochastic gradient descent. At predetermined set of checkpoints (t' E [0, 4, 25, 50, 100, 224, 500, 1000. 2000, 4472, 10000, 25100)) we create two copies of the current state of the network and train until completion with different batch orderings, where linear mode connectivity measurements are calculated. This process sometimes also referred to as spawning 20] and is repeated for 3 seeds at each 1'. The entire process is repeated for 3 seeds resulting in total of 3 x 3 = 9 total values over which we report the mean and standard error. Momentum is set to 0.9 and a stepwise learning rate is applied beginning at 0. and decreasing by a factor of 10 at iterations 32,000 and 48,000. For the ResNet-20 architecture (HZRS16), use
24

---

an implementation from [ Experiments are conducted on CIFAR-10 [KH* 09) where the inputs are normalized with random crops and random horizontal flips used as data augmentations.
Pretraining of the finetuned model model is performed on the SVHN dataset [NCC+1] which is also an image classification task with identically shaped input and output dimensions CIFAR-10. We use a training setup similar to that of the CIFAR-I0 model but set the number of training iterations 30,000 and perform the stepwise decrease learning rate at iterations 15,000 and 25,000 decaying by a factor of 5. Three models are trained following this protocol which achieve validation accuracy of 95.5%, 95.5%, and 95.4% on SVHN. We then repeat the CIFAR-I0 10 training protocol for finetuning but parameterize the three initialization with the respective pretrained weights rather than random initialization. We also find that shorter finetuning period sufficient and therefore finetune for 12,800 steps with the learning rate decaying by factor of 5 at steps 6,400 and 9,600.
Also following the protocol of (DDCC20) for each pair of trained spawned networks (So, &fo,) consider interpolating their losses (i.e. ta == ar &ooo (x), y) +1 -a)- (x), y)) and parameters (i.e. clne {fe (x), y) where @lme a0 (1-a))1) 02) for 30 equally spaced values of E [0, 1]. In the upper panel of Fig. 6 we plot the accuracy gap at each checkpoint t' (i.e. the point from which two identical copies of the model are made and independently trained to completion) which is simply defined as the average final validation accuracy of the two individual child models minus the final validation accuracy of the weight averaged version of these two child models. Beyond the original experiment, we also wish to evaluate how the gradients V fee evolve throughout training. Therefore, in panels (2) and (3) Fig. 6 each checkpoint t' we also measure the mean squared change in (pre-softmax) gradients (VoJo,+ (x) V (x))² between the current iteration t' and those at the next epoch 1' 390, averaged over a set of n 256 test examples and the parameters in each layer.
Compute: We train num_ outer _eeds x num_innee inneinnneedsssssee X num wwwphhhhihhheeeee x num_ checkpoints (3 x 3 x 2 x 12 = 216) networks for the randomly initialized model. For the finetuned model this results in 3 x 3 x 2 x 10 = 180 training runs. Additionally, we require the pertaining of the base models on SVHN. Combined this results in a total of 216 + 180 + 399 training runs. Training each ResNet-20 on CIFAR-16 10 required <1 hour including additional gradient computations.
C.4 Data licenses
All image experiments are performed on CIFAR-10 [KH MNIST [LBBH98] MNISTID
[GK24], or SVHN [1] Tabular experiments are run on houses, superconduct. california, and house_sales from OpenML. [VvRBTI3] as described in [GOV22]. CIFAR-
10 released with an MIT license. MNIST is released with Creative Commons AttributionShan
Alike 3.0 license. MNISTID is released with an Apache-2.0 license. SVHN is released with CCO:Public Domain license. OpenML datasets are released with a 3-Clause BSD License. All the datasets used in this work are publicly available.
D Additional results
D.1 Additional results on approximation quality (supplementing Fig.
SGD
Achan
SGD + Momeream
SGD + Weicht Deaar
SGD of-)
10
10
tike
90
500
$000
30
$00
5000
530 5030
300
5000
500
$30
Oyciniatiioeee strpp
Opeimiortien don stop
Opeinizntionn step.
Oprimioati scap
Op stap
Figure 7: Approximation error of the telescoping (fo. (x), red) and the model linearized around the initialization (fi (x), gray) by optimization step for different optimization strategies and other design choices. Iteratively telescoping out the updates using Fo, (x) improves upon the lazy approximation around the initialization by orders of magnitude.
25

---

Lazy meenaey Telesoping accuricy
SGD
AdaaW
SGD Momentran
SGD
Weight Dengy
SGD o
10
10m11
10 mam Hi??
0.0
30
900
2000
3000
50
3XC
2000
Oytimiaztion stop
Optimiazion
Oytimizztion step
Optimizzion
Optimization
Figure 8: Test accuracy of the telescoping (fo, (x), red, top row) and the model linearized around the initialization (f0." (f,,"()), (x), blue, bottom row) against accuracy of the actual neural network (gray) by optimization step for different optimization strategies and other design choices. While the telescoping model visibly matches the accuracy of the actual neural network, the linear approximation around the initialization leads to substantial differences in accuracy later in training.
In Fig- 7 we present results investigating the evolution of approximation errors of the telescoping and linear approximation around the initialization during training using additional configurations compared the results presented in Fig. in the main text (replicated in the first two columns of Fig. 7. We observe the same trends as in the main text, where the telescoping approximation matches the predictions by the neural network by orders of magnitudes better than the linear approximation around the initialization. Importantly, we highlight in Fig. [ that this is also reflected in how well each approximation matches the accuracy of the predictions of the real neural network: while the small errors of the telescoping model lead to no visible differences in accuracy compared to the real neural network, using the Toylor expansion around the initialization leads to significantly different accuracy later in training.
D.2 Additional results for case study 1: Exploring surprising generalization curves and benign overfitting
MNIST (- noise)
MNIST (w/o noise)
0.2
0.2
MSE
0.1
0.1
0.0
0.0
Elfoctive Paramoters
1000
1000
Train
Test.
500
500
10
10
10a
10a
10"
10
10a
10
Num hidden neurons
Num hidden neurons
Figure 9: Double descent experiments using MNIST, distinguishing 3-vs-5, with 20% added label noise during training (left) and no added label noise (right). Without label noise, there no double descent in error on this task; when label noise is added we observe the prototypical double descent shape in test error.
Double descent on MNIST. Fig. 9 we replicate the CIFAR-10 experiment from the main text while training models to distinguish 3-vs-5 on MNIST. We find that in the absence of label noise, no problematic overfitting occurs for any hidden size; both train and test error monotonically improve with increased width. Only when we add label noise to the training data, do we observe the
26

---

0.3
MSE
0.2
0.1
Elfectiive Paraneters
0.0
10
10
102
800
Train
400
Test
Num hidden neurons
Figure 10: Double descent experiment using MNIST-1D, distinguishing class O and 1, with 15% added label noise during training. Mean squared error (top) and effective parameters (bottom) for train and test examples by number of hidden neurons.
characteristic double descent behavior in error - this is in line with NNB++11 observation that double descent can be more pronounced when there is noise in the data. Importantly, observe that as in the main text, the improvement of test error past the interpolation threshold associated with the divergence of effective parameters used on train and test data. In Fig. 10 we additionally repeat the experiment using the MNIST-ID dataset with 15% labelnoise as in [K24], and find that the decrease in test error after the interpolation threshold again accompanied by decrease in effective parameters as the number of raw model parameters is further increased in the interpolation regime.
Additional grokking results. In Fig. I1 replicate the polynomial grokking results of [KBGP24 with additional values of e. Like (BGPP24), we observe that larger values of 0.5 lead to less delayed generalization. This is reflected in gap between effective parameters emerging earlier. With very small = .05, conversely, we even observe a double descent-like phenomenon where test error first worsens before it improves later in training. This is reflected also in the effective parameters, where plest first exceeds pS parinn before dropping below it as benign overfitting sets in later in training. In Fig. [2] we replicate the MNIST results with additional values of a; like [LMT22] we observe that grokking behavior is more extreme for larger a. This is indeed also reflected the gap between pl test and ps prrii emerging later in training.
test and train
Additional training results on MNIST with standard initialization. In Fig. 13 we present train and test results on MNIST with standard initialization supplement the test results presented in the main text. Both with and without sigmoid, train and test behavior is almost identical, and learning is orders of magnitude faster than with the larger initialization. The stronger inductive biases of small initialization, and additionally using sigmoid activation, lead to much lower learned complexity on both train and test data as measured by effective parameters.
D.3 Additional results for Case study Understanding differences between gradient boosting and neural networks
In Fig. 14 we replicate the experiment from Sec. 4.2 on three further datasets from GOV22 tabular benchmark. We find that the results match the trends present in Fig. in the main text: the neural network is outperformed by the GBTs already at baseline, and the performance gap grows as the test dataset becomes increasingly more irregular. The growth in the gap is tracked by the behavior of the normalized maximum kernel weight norm of the neural network's 's kernel. Only on the california dataset do we observe slightly different behavior of the neural network's 's kernel: unlike the other ET max,ez? [1k((1
three datasets, E maxic [x](2 stays substantially below at all p; this indicates that there may have been examples in the training set that are irregular in ways not captured by our
$ Σl=; max Jerf (1,)(
experimental protocol. Nonetheless, we observe the same trend that increases in relative terms as p increases.
4 ΣT maxie [x,
27

---

(1) Tisk aligunent
0.5
(2) Task alignment
0.2
(3) Task alignnent 0.05
1.0
1.0
1.0
MSE
0.5
0.5
0.5
0.00
0.00
0.0
10"
10"
10
Iifeetive Parametons
500
Train
500
Test
250
250
250
107
Optimisation step
Optimization stepp
Optimication step
Figure 11: Grokking in mean squared eπrrr (top) on a polynomial regression task (replicated from
KBGP24
against effective parameters (bottom) with different task alignment parameters c.
(1) Initiolization =
(2) Initialization =
(3 Initialization a
Ellective perameters Mielaossfiatton mite
0.6
0.6
0.6
0.3
0.3
0.0
1000
Test
Train
10
10
10
10
10
10'
10
10'
10
10'
Optimization
stop
Optimization step
Optimization step
Figure 12: Grokking in misclassification error on MNIST using a network with large initialization ( replicated from (LM222) (top), against effective parameters (bottom) with different initialization
scales a.
MNIST with o(.)
MNIST w/o a(-)
Misclasifiiation
0.6
rate
0.3
0.3
0.0
0.0
10
10
10a
10a
10"
10
10
10
150
Effective
parameters
Test
Train
75
10P
10
10a
10a
10"
10a
10a
10a
Optimization step
Optimization step t
Figure 13: No grokking in misclassification enror MNIST (top), against effective parameters (bottom) using a network with standard initialization (a = 1) with and without sigmoid activation.
28

---

houno saless
cuporconduct
colifornia
Rodubee introsse in
MSE fifferetoee (55)
Tee HER
Tex ME
Toa VSE
NN BMB ta DURS
-1377 LE BEDD
100
NN -0.8 490)
CBT--20 1230
1000
GBT RL2S 14 (URG)
GBT DIGL Is (UC3)
0-
1.4
Normallzed mextmmn
XN
1.2
CST
1.2
1.0
bernet
0.6
Proportion Imegrlar commples
Propoction of irregiiar examples
Proportion inoogulrr coumplos
Figure 14: Neural Networks vs GBTs: Relative performance (top) and behavior of kernels (bottom) with increasing test data irregularity for three additional datasets.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As indicated in the abstract, present model for learning in neural networks in Sec. ] We discuss how this allows to reason about the effects design choices in Sec. and use derive new empirical insights the listed phenomena in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Sec. 3 we highlight important practical limitations regarding the use of the telescoping approximation
requires significant additional computation. Further, in Sec. and in Sec. 6 we highlight that in each case study we intentionally limited ourselves to specific noteworthy empirical observations, which implies that future work is needed to further establish their generality beyond the case studies we consider.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
30

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Beyond the arguments presented in the main text, all additional derivations included in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems. The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All details needed to replicate the experiments are described in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
31

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code
is
provided
com
teleacoping-lemt
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
publite/guidee//deeettbmiss seiiibmmsssoooooccri..... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips /nips.cc/puut
guides http:///iw///......oooo/////aaaa
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: detailed account of all experimental details is presented in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All figures report mean and one standard error across multiple replications of the same experiment, as outlined in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Estimates of used compute resources are presented in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////eaaappsscccpppppt////a
Answer: [Yes]
Justification: research conducted the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning by advancing our understanding of surprising deep learning phenomena. While there are many potential societal consequences of the use of machine learning in general, there are none that we feel must be specifically highlighted here. If anything, we believe that better understanding of unexpected deep learning behaviors will help limit unexpected behaviors of systems deployed in practice.
33

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The used datasets and their licenses are discussed in Appendix C.4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
34

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "8PPz8xKjpp", "reviewer": "Reviewer_YCvv", "summary": "This work proposes a new analytical model of neural networks (NNs) extending popular 'linearized' approximations over the initial parameters built from the gradient vectors. In particular, they consider approximating the full learning trajectory of NNs as a sequence of first-order approximations rather than a single one. The authors connect their model to several prominent empirical observations of the properties of neural networks, showing how it can be used to compare different optimizers and learning frameworks, and also justify phenomena such as grokking, double-descent, and linear-mode connectivity.", "questions": "I would appreciate if the authors could address the criticism highlighted above in their rebuttal.", "limitations": "The authors explicitly discuss the main limitation of the proposed methodology (increased complexity) at the end of Section 3, although mostly from a practical (and not theoretical) viewpoint. Overall, I found the work well-written, quite comprehensive, and insightful - building directly on top prior work, yet, providing some interesting new perspectives. Hence, I am leaning toward acceptance.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The authors provide a comprehensive first look into the properties and implications of the proposed framework. For instance, they consider extensions to vanilla SGD updates, including momentum and weight decay - showing how they can connect quite naturally through a weight averaging perspective (start of Sec 3.1).\n2) The Appendix is functionally used to relegate more comprehensive secondary details, allowing the authors to keep the text free of long proofs and extended literature reviews. This makes the main text concise and pleasant to read, while still allowing the interested readers to delve deeper into specific topics.\n3) The paper does a good job at highlighting the relevance of the proposed framework, providing grounded examples that connect it with several popular understandings of neural network dynamics and behavior in Section 4. Out of the three examples, I found the discussion regarding comparing neural networks and gradient boosting through their relative approximated dynamics and kernels (4.2) of particular interest.", "weaknesses": "1) The proposed telescoping approximation partially defeats the purpose of the simplified linear models, by introducing back much complexity that linearization precisely seeks to abstract away. Perhaps, some additional discussion about the limitations in terms of theoretical analysis would have been useful.\n2) Also due to this increased complexity, most experiments are carried out in relatively toy-ish settings (e.g., MNIST).\n3) There is some overuse of inline math notation (e.g., Section 4.1) hindering readability. While I understand the limitations due to the imposed page constraints, I would encourage the authors to try and adjust their text to lay out their equations more clearly in later revisions. \n\nTypos:\n\n\"due rescaling\" -> due to rescaling (117)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their very constructive review and the positive assessment of our work! We were especially delighted to read that the reviewer found our uncovered connection between the telescoping model and gradient boosting to be particularly instructive. We respond to the comments raised in the review below. \n\n**(1) Increased complexity over linear models.** Indeed, we sacrifice some of the analytic simplicity of the fully linearized lazy model in order to obtain a more accurate model of a neural network. Unfortunately, we believe this to be unavoidable as a linear model will typically be unable to accurately model a full neural network in standard applied settings (indeed including even the most of toy-ish settings we consider, e.g. MNIST in Fig 1). We would argue that any conclusions drawn from a proxy model can only be valid if that proxy is a good approximation. Therefore, we believe the telescoping model sacrifices only the minimal degree of simplicity necessary in order to maintain an accurate approximation of the full network. We appreciate the reviewer's suggestion to extend the limitations section beyond practical considerations by highlighting that this additional complexity can come at the cost of complicating potential future theoretical analyses, and will do so in the updated manuscript!\n\n**(2) Scale of experimental settings.** While we agree that our experiments were generally conducted in relatively moderate-scale settings compared to e.g. much of the recent LLM focused empirical work, we would like to emphasize that we were able to operate at a sufficiently large scale to satisfy the three provided case studies, all of which have received substantial attention in the recent literature (despite their relatively moderate scale). For case study 2 we operate on typical size tabular datasets from a standard benchmark from the literature.  For case studies 1 & 3, we primarily revisit the exact well-known, canonical experimental settings in which the studied phenomena have been observed in previous work (e.g. grokking, double descent, LMC from [BHMM19, LMT22, KBGP24, FDP+20]). While larger scale experiments were thus not necessary in order to address our selected research questions here, we certainly agree that considering how to apply the proposed telescoping model to larger scale networks would enable many interesting research directions in future work (as we allude to in l.123).\n\n**(3) Use of in-line maths.** We were delighted to read that this reviewer appreciated our efforts to make the reading experience as natural as possible by relegating extended details to the appendix for interested readers. We also appreciate that the quantity of content and breadth of phenomena covered in this work required certain parts of the paper to be written quite densely. This was a fair point raised by several reviewers. We will therefore put considerable effort into using the extra page of a camera-ready version to decompress the text (e.g. less in-line maths particularly in Sec 4.1) and expand on details where there is an opportunity to improve clarity. \n\n_Typo:_ Thank you, fixed!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T2BIJTpvoM", "reviewer": "Reviewer_b22S", "summary": "The paper replaces the traditional \"lazy learning\" regime approximation with an approximation using a telescoping sum. This splits up the usual interval of approximation $\\theta_T - \\theta_0$ into many smaller pieces. The authors then show why this approximation is useful for explaining various interesting aspects of modern deep learning:\n1. Generalisation behaviour (double descent, grokking and the effect of inductive bias on model complexity)\n2. The connection between gradient boosting and modern deep neural networks (DNNs)\n3. The success of weight averaging\n4. Modern design choices regarding architecture and optimiser", "questions": "Some questions:\n1. As someone who has read relatively few papers using tangent kernel arguments, I am curious how other groups have extended the traditional lazy -> rich approximation?\n2. Could the authors provide a better explanation of $p_{\\hat{s}}^0$ and associated nomenclature? What is $\\mathcal{I}_0$ for example?\n3. Are there meaningful experiments to run with the insights in section 3.1? For example, could the authors look at verifying the reported trade-off between momentum and weight decay?\n\nSome other suggestions:\n1. A few grokking papers which are not included in the literature review are:\n   - [Paper 1](https://openreview.net/forum?id=GH2LYb9XV0). Grokking in linear estimators without regularisation, which may go against intuitions with the measure proposed\n   - [Paper 2](https://openreview.net/forum?id=ux9BrxPCl8). Discusses complexity as a theory of grokking.\n   - [Paper 3](https://openreview.net/forum?id=UHjE5v5MB7). Talks about corrupted labels, might be interesting in discussion of future work where the proposed measure could be used to analyse cases with corrupted labels\n2. Paper on double descent not mentioned: [Double Descent Paper](https://arxiv.org/pdf/2111.08234).", "limitations": "I think limitations are appropriately addressed given that the paper is not focused on performance. It might be interesting to see cases where the telescoping sum does not approximate some set of phenomena well. Perhaps some discussion of this instead of the current limitations paragraph would be more meaningful.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "In general, I think the authors did a reasonably good job. Particular praise goes to the following aspects of the work:\n1. The extension of the lazy regime approximation to a telescoping sum seems like a very natural extension of the current set of ideas. The use of the sum seems logical and may become a useful theoretical insight.\n2. I liked the choice of case studies. Phenomena like grokking, double descent, and weight averaging are clearly of interest to the community.\n3. The explanations provided seem relatively intuitive under the telescoping approximation.\n\nI will now touch on how well I think the paper does on the areas of originality, quality, clarity, and significance:\n1. *Originality.* The use of a telescoping sum is clearly somewhat original. In addition, I thought the analysis of GBTs seemed fairly novel and interesting.\n2. *Quality.* There are some elements of high-quality work in the paper. I think the analysis and empirical evidence for the generalisation section were fairly good. I also thought the analysis of momentum and weight decay seemed interesting. (However, I have issues/suggestions for both of these sections in weaknesses).\n3. *Clarity.* I thought the authors did a fairly good job of justifying their use of a telescoping sum. However, I find it hard to come up with other strengths, as I think clarity is what this paper is most lacking.\n4. *Significance.* I think the insights are of decent significance. For example, if it was demonstrated that the effective parameter count metric had further predictive power, it could be a useful tool in the analysis of grokking. (I provide suggestions as to how it could be expanded upon).", "weaknesses": "I will structure this section into three subsections. The first two subsections concern general where I believe the paper may be lacking. In the third section, I list additional minor issues. I then end with a general assessment.\n\n### (1) Lack of clarity and writing style\n\nI think the paper lacks clarity in the writing. Here are some major areas where this causes issues:\n1. **Abstract.** The abstract does not describe the idea of the paper well nor provides a clear explanation of the impact of the work. I found it very hard to read at first.\n2. **Explanation of insights.** Due to the density of the writing, I found it hard to understand the theoretical insights.\n3. **Explanation of setup in section 4.1.** I found the explanation of the $p_{\\hat{s}}^0$ measure to be quite confusing. I could not see where $\\hat{s}(x)$ was clearly defined.\n\n### (2) Importance of theoretical insights and empirical backing\n\nIn section 3.1, there are several interesting observations about modern optimisation choices and how they can be more easily reconciled under the telescoping approximation. However, I am unsure of several things:\n1. **Utility of telescoping sum.** Are the observations provided specific to the telescoping sum approximation or would they hold with the traditional lazy approximation?\n2. **Claims in real networks.** Do the claims made hold in real networks? It would have been nice to see experiments demonstrating the claims made given that we are still working with an approximation.\n3. **Utility of insights.** What should I do with the insights provided (supposing they hold in real cases)? For example, now that I know that momentum offsets the effect of weight decay on learned function updates, should I try and avoid using weight decay and momentum? In my experience, it seems that weight decay and momentum do work pretty well together in practice.\n\nIn section 4, there are also many interesting observations. However, I have a few concerns:\n1. I would have liked to have seen more experiments demonstrating that cases of double descent and effective parameter counts are related in the way described. I think it is likely to be true, I just don't think that one experiment is sufficient.\n2. In the grokking cases, I would like to see whether the metric proposed (namely mismatch in the effective parameter count) has more predictive power. For example, in the literature, there have been ways of inducing grokking proposed. For example, in [this paper](https://openreview.net/pdf?id=ux9BrxPCl8) it is shown that adding spurious dimensions to the input reliably increases grokking. I would like to see whether cases with a greater grokking gap correlate with a greater divergence in effective parameter use.\n3. In the second case study, the theory of why GBTs perform better than neural networks seems reasonable. However, I think the empirical evidence is lacking. A single plot with correlation does not seem sufficient.\n4. In the third case study, I am unsure as to whether the trend in the pre-activation gradients aligning with the accuracy gap is sufficient to demonstrate the hypothesis.\n\n### (3) Minor gripes with the paper\n\n1. If I have understood correctly, the paragraph discussing approximation error seems unnecessary. I would place this in an appendix and spend two sentences on it in the main discussion. It seems quite obvious that the telescoping sum is going to approximate things better.\n2. The practical limitations section also seems unnecessary. We know that it is a tool for analysis rather than an alternative approach for training. This does not need to be explained.\n3. I'm not sure how I feel about the use of the word \"pedagogical\" throughout the paper. I don't know if the impact of a NIPS paper is meant to be a tool for teaching. Rather, the desired impact should be a novel result or an improvement to the understanding of the field.\n\n### A general assessment\n\nGenerally, I think the paper provides some interesting insights but is attempting to do too much. I think if it were to be rejected, I would encourage the authors to rewrite it in a much simpler style focused on far fewer topics with many more experiments.\n\nUpdate: I have moved my overall rating from a 5 to a 6 as I believe some of my concerns were addressed in the rebuttal. I also moved my rating of soundness from a 2 to a 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very detailed and constructive review! Limited by space constraints, we respond to major comments below. \n\n**(1) Clarity.** We appreciate that our writing was somewhat densely packed in the mentioned sections, and will utilize the camera-ready extra page to decompress for improved clarity. We will particularly expand the paragraph on $p^0_s$ and will more explicitly write out $\\hat{s}(x)$ in the main text (currently derived for different optimizers in Apx B.1), which can recursively be obtained from eq 5 after substituting in the squared loss gradient. We will also rework the abstract to include a more explicit summary of the finding of each case-study.\n\n**(2) Comments on Sec 3.1.**\n1. The insights on functional updates that can be derived from the telescoping sum subsume those of the lazy regime as a special case. As we highlight in the paragraphs on momentum and weight decay, assuming constant gradients will simplify some of these expressions. Some observations could thus also be made when assuming lazy learning – e.g. adaptive learning rates acting by rescaling the kernel – however, we have not seen these observations discussed explicitly anywhere. We will make even more clear which observations also hold in the lazy regime!\n2. _Claims in real networks._ This is point is precisely why we included experiments on approximation quality – Fig 1 and Fig 6 in Apx D show that the approximation matches real networks very closely for the different implementation choices and that the insights in Sec 3.1 thus apply to real networks. We appreciate the suggestion to additionally numerically verify the tradeoff between momentum and weight-decay empirically, and have included an experiment that does so in the pdf in the top-level response!\n3. _Utility of insights._ We believe that Sec 3.1 will be of interest to anyone wishing to gain more intuition and understanding of the effect different optimization strategies will have on model predictions. Usually, these are only discussed in terms of their effect in parameter space, and not in function space as we do here – even though function space is what ML is ultimately interested in! We comment further on the specific momentum weight decay example in the experiment in the top-level response. \n\n**(3) Comments on Sec 4.**\n1. _Double descent._ Note that Apx. D.2 already includes additional experiments, indeed demonstrating the same behavior for double descent on MNIST. Additionally, we provided a new experiment in the pdf of the top-level response.\n2. _Grokking._ In Apx. D.2, we actually already conducted additional experiments varying two such mechanisms –  the initialization scale [LMT22] and task alignment [KBG24]. As discussed in Apx D.2  we found in both cases that later generalisation (i.e. more extreme grokking) is predicted by an effective parameter gap emerging later in training. As suggested in the review, we repeated this experiment for another mechanism – the number of spurious dimensions – on the parity prediction task of Miller et al (TMLR2024). The results, included in the top-level comment, indicate that also here grokking is predicted by the effective parameter gap growing. We will better link to this in the updated main text, thank you for the suggestion!\n3. _GBTs._ Note that results using additional datasets are included in Apx D.3 where a similar behavior is observed in the kernels.  \n4. _LMC._ We agree that unlike the preceding sections, the case study in Sec 4.3 is more exploratory in nature. The key point of interest is using the telescoping lens to come to the observation in eq 10 that constant gradients are sufficient for LMC. However, in practice gradients will not reach perfect stabilization and therefore this cannot provide a complete explanation for this phenomenon. We were therefore deliberate in not claiming any overly strong conclusions, and only include preliminary evidence of a specific factor contributing to LMC emerging by empirically showing that the model gradients indeed become closer to stable throughout training, and that – like the loss barrier – the change in model gradients is lower in pretrained than in randomly initialized models. We think there is significant opportunity to further investigate this perspective, and will update the section to be more explicit on scope and potential future avenues.\n\n**(4) Minor points.** \n\n1. _Approximation quality section (Fig 1)._ Building on the answer to point 2 on Sec 3.1 above, we found this section essential as it demonstrates that (and when) the telescoping sum is indeed a reasonable model of a neural network in a practical setting [note that the neural networks used here are the same as in Sec 4.1] and that therefore the derived insights hold, while those of the lazy model may not. Further, the main purpose of including the lazy model here was to provide a _reference point_ for the scale of the approximation error, allowing us to better evaluate how good an approximation the telescoping model is. We will make the purpose of this section more clear in the updated manuscript.\n2. _Limitations._ We found it important to be upfront about potential computational limitations. The comment on training is in reference to the lazy model, which _is_ sometimes used as an alternative way of training (as a linear regression in the tangent features). We will make this more clear! \n3. _The term pedagogical._ We are in full agreement that the goal of a NeurIPS paper should be an ``improvement to the understanding of the field''. Our intention was to imply the telescoping model to be a pedagogical device in the sense that it is a tool through which researchers can better understand aspects of deep learning. However, we are happy to substitute this term as this was obviously not clear. \n4. _Additional relevant references._ Thank you for these, we will include them in our literature review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q4UWURoosv", "reviewer": "Reviewer_gQrQ", "summary": "This paper tries to understand the design of optimizer, model architectures and some deep learning phenomenon empirically. Additionally, the authors find a proxy model of neural networks other than the simply linearized model.", "questions": "N/A", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "1. The topic is interesting, especially the proxy model of neural networks. I would like to see more theoretical insights derived from the proxy model, but the current paper didn't put efforts on it.", "weaknesses": "1. First of all, limited insights gained from the proxy model of neural networks when dealing with the modern optimizer and architectures. In section 3.1, the authors try to connect their proposed proxy model with some modern optimization techniques. However, no rigorous proof or insightful experiments are provided. Complex formulas including Eq. 6-8 only provides limited insights.\n2. As for case study part, the writing confuses me a lot. For example, in Section 4.1, in line 200-201, \"A candidate complexity measure that avoids the shortcomings listed above because it only considers the behavior of the final fitted model was recently used by [CJvdS23] in their study of non-deep double\ndescent.\" I cannot understand what this mean.\n3. Still the case study part, I strongly disagree that \"LMC and train-time transition into lazy regime.\" (As I am not familiar with GBT and grokking, I skimmed those parts). [cite 1] and [cite 2] demonstrates that lazy regime is not sufficient to explain LMC and model merging.\n\n[cite 1] Fort, Stanislav, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. \"Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel.\" Advances in Neural Information Processing Systems 33 (2020): 5850-5861.\n[cite 2] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. \"Task arithmetic in the tangent space: Improved editing of pre-trained models.\" Advances in Neural Information Processing Systems 36 (2024).\n\n4. There are seeming no many experiments in both case study part and Sec 3. I would expect more experiments relating their proposed proxy model to the case studies and optimization techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We are delighted that the reviewer finds our proxy model for a neural network interesting! Before we move to address the specific points raised in the review below, we would like to make two high-level comments.  First, while we similarly believe that interesting theoretical insights can be derived from this model in follow-up work, we would like to note that the _explicitly stated goal_ of the current paper was to use it as a tool for generating new _empirical_ insight, unifying the study of a range of modern deep learning phenomena. Second, we also find it unfortunate that Secs 4.1 and 4.2 were only \"skimmed\" and therefore barely feature in the review, and would like to highlight that these were the parts that received particular praise in all other reviews. \n\n**1. Insights from Sec 3.1.** First, we note that as stated in L131, eqs 6-8 were not only included to provide insight, but were also _necessary_ in order to replicate later experiments from existing work that required Adam, weight decay, and momentum to be applied in the telescoping model. Further, we believe that Sec 3.1 will be of interest to anyone wishing to gain deeper _intuition_ into the effect different optimization strategies will have _on model predictions_. Usually, these strategies are only discussed in terms of their effect in parameter space, and not in function space as we do here – even though function space is what ML is ultimately interested in. While we appreciate that this reviewer was not interested in the insights derived from Sec 3.1, we note that both YCvv and b22S explicitly mentioned this section as a strength of the paper. We hope that in examining the new empirical results we have now provided (see the pdf in the top-level response) that demonstrates how these insights are reflected in training a real neural network this reviewer will also find value in Sec 3.1. \n\n**2. Clarifying cited paragraph on complexity.** This sentence highlights that the cited complexity metric of [CJvdS23] accounts for what the model has learned during the fitting process, and is thus able to capture the interaction of model, optimizer and a particular dataset. Many other metrics only consider what is _learnable_ by the entire hypothesis class for arbitrary data. We note in the opening paragraph of this section that the former type of metric is preferable for understanding double descent and grokking, where the phenomena only arise due to interaction of the model, its learning process and the data. We hope that this alleviates any confusion. We would also be happy to clarify any further unclarities in the writing if more precise issues were pointed out!\n\n**3. LMC.** We thank this reviewer for raising this question and are happy to clarify why there is no contradiction between these works and ours. \n- Fort et al., which we cite numerous times as [FDP+20], find that “deep network training exhibits a highly chaotic rapid initial transient” for the first 2-3 epochs after which the relative change in the NTK decreases throughout training and plateaus at a low value. This is in complete agreement with our work which studies this through the lens of the telescoping model. Note that our Fig 5 and their Fig 7 - which are similar in spirit but study slightly different objects and use different metrics - observe a very similar trend. We further supplement their results by examining differences in tangent feature changes between pretrained and randomly initialized models.\n- Ortiz-Jimenez et al. focus on a very different setting: they consider the context of task arithmetic, where a generalist model is being finetuned several times, each time on a new dataset, where the input space $\\mathcal{X}$ of each task is disjoint from the others, with resulting task vectors then _added_ to the base models weights and, therefore, is subtly different to _weight averaging_ (they consider addition and subtraction). Please note that we strictly focus on finetuning routines that only differ in “batch orderings and data augmentations” (L356) followed by weight averaging between the resulting models, and do not claim that our empirical findings will transfer to their setting. Nonetheless, we agree that this reference is relevant to include and will discuss it in the updated version! Thank you for pointing this out.\n\n**4. Experiments.** The goal of this work was to provide a tool that can help us to better understand _specific previously observed phenomena in deep learning_. Therefore, in our experimental evaluation we primarily revisit the exact well-known, canonical experimental settings in which the studied phenomena have been observed in previous work (e.g. grokking, double descent, LMC). We have rigorously pursued this approach by reproducing key experiments from several works (e.g. [BHMM19, LMT22, KBGP24, FDP+20]) and integrating aspects of the telescoping model to better understand their findings. We also verified that the telescoping approximation holds for the different implementation choices discussed in Sec 3.1 in Apx D.1. \n\nWe also appreciate that our concise presentation of results in the main text (which are reinforced by 4 further pages of experiments in the Appendix) may have disguised the empirical contribution of this work, making it appear that our evaluations are less extensive on the surface. For example, the double descent experiments alone - which only constituted a part (Fig 2) of case study 2 - required training 88 models per setting resulting in 432 hours of compute time on an A100 GPU. \n\nFinally, in the pdf in the general comment we provide several additional results which were specifically requested in other reviews and further reinforce the findings of our paper. In particular, we also provide some empirical verification of our expressions from Sec 3.1 which we hope will further reassure this reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u2yQnWRC5Z", "reviewer": "Reviewer_EFqT", "summary": "The paper proposes a new approach to understanding neural networks by examining a model consisting of a sequence of first-order approximations telescoping into an empirically operational tool. This model aims to isolate components of the training process and offers a lens to analyze the effects of design choices in neural networks. The authors illustrate the applicability of this model through case studies on double descent, linear mode connectivity etc.", "questions": "In what ways does this model bridge the gap between purely theoretical analysis and empirical observations in neural network research?\n\nHow does the model handle non-linearities and complex interactions within neural networks? Can the authors provide a quantification of how these features affect their approximation?\n\nCan you elaborate on how the telescoping model reveals linear mode connectivity within the loss landscape?\nWhat are the implications of this connectivity for network training and generalization?", "limitations": "None", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a novel telescoping model that bridges the gap between theoretical and empirical research in neural networks.\n\nThe approach is used to derive new insights into well-known phenomena such as double descent.\n\nThe paper covers a wide range of phenomena and provides a detailed analysis of each, contributing significantly to the understanding of neural network behavior.\n\nThe model offers a simplified yet effective way to understand the impact of various design choices on neural network training.", "weaknesses": "The telescoping model involves increased computational costs, especially for large networks and datasets, which may limit its practical applicability.\n\nThe quality of the approximation depends heavily on the learning rate and the optimizer choice, which might not generalize well across different settings.\n\nWhile the model is intended to simplify understanding, the incremental approximation can become complex to implement and analyze for certain networks and tasks.\n\nAlthough the paper provides case studies, more extensive empirical validation across diverse datasets and architectures could strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their in-depth review and their positive assessment of our work! We were delighted by their appreciation of the utility of the telescoping model itself, the wide range of phenomena we are able to consider using our model and the insights we provide. We respond to major points raised in the review below.\n\n**(1) Computational burden.** We agree that, as highlighted upfront in Sec 3, the telescoping model is more expensive to obtain than a standard neural network and therefore may not be possible to implement for large scale architectures such as modern LLMs. However, we note that the model is sufficiently tractable that it could be applied across all three case studies on the standard benchmarks from existing literature (all of which have received substantial attention in the recent literature despite their relatively moderate scale), and we were able to utilize it to provide significant insight. We believe that exploring computationally efficient methods of extending this approach to larger scale settings provides an exciting direction for future work (as alluded to in l.123)\n\n**(2) Approximation quality.** We also appreciate that, with the exception of models for which the first-order Taylor expansion holds exactly, the approximation quality will _always_ deteriorate for a sufficiently high learning rate (note that this is effectively by definition as eqn (2) holds exactly as $\\gamma \\to 0$). Indeed, making this potential limitation explicit was the purpose of including the experiment with varying learning rates in Fig 1.  However, we generally found the appropriate learning rates for provided experiments to achieve good approximation quality (across different optimization choices, as further investigated in Apx D.1).\n\n**(3) Increased complexity over linear models.** Indeed, we sacrifice some of the analytic simplicity of the fully linearized lazy model in order to obtain a more accurate model of a neural network. Unfortunately, we believe this to be unavoidable as a linear model will generally be unable to accurately model a full neural network in standard applied settings (indeed including even the most of toy-ish settings we consider, e.g. MNIST in Fig 1). We would argue that any conclusions drawn from a proxy model can only be valid if that proxy is a good approximation. Therefore, we believe the telescoping model sacrifices only the minimal degree of simplicity necessary in order to maintain an accurate approximation of the full network. \n\n**(4) Experiments.** Allow us to re-emphasize that the goal of this work was to provide a tool that can help us to better understand _specific previously observed_ phenomena in deep learning. Therefore, in our experimental evaluation we primarily revisit the specific, well-known, canonical experimental settings in which the studied phenomena have been observed in previous work (e.g. grokking, double descent, LMC). We have rigorously pursued this approach by reproducing key experiments from several works (e.g. [BHMM19, LMT22, KBGP24, FDP+20]) and integrating aspects of the telescoping model to better understand their findings.  Further, Apx D includes additional results for different settings of some of the case studies. We also appreciate that our concise presentation of results in the main text (which are reinforced by 4 further pages of experiments in Apx D) may have disguised the empirical contribution of this work, making it appear that our evaluations are less extensive on the surface. In the general comment, we do provide several additional results on different datasets and using different experimental settings, which were specifically requested in other reviews and further reinforce the findings of our paper. We hope that these additional results may also further reassure this reviewer.\n\n**Questions.**\n- Q1. Related to (3) above, important theoretical progress in understanding neural networks has emerged from the NTK literature. Many of these results require some assumptions e.g. infinite width limit, lazy learning, etc. The telescoping model provides a setting in which an arbitrary neural network and optimization routine can be iteratively cast using NTKs without large deviations from the exact network. We envision this can provide a bridge between theoretical and empirical research upon which future theoretical research may be able to build by applying tools from this literature to this iterative approximation instead.\n- Q2. Indeed, a powerful component of the telescoping model is that it does not require us to discard complex aspects of the optimization process or architecture. As shown in Sec 3.1, important practical extensions to vanilla SGD (i.e. momentum, weight decay, Adam) can be explicitly derived such that they can be exactly integrated into the telescoping model. Complexities of the model architecture, such as nonlinearities, on the other hand are entirely captured by the tangent features $\\nabla_\\theta f_{\\theta_t}(x)$. That is, these gradients of the model outputs with respect to the model parameters capture how the networks architecture propagates between input and output.\n- Q3. We agree that it is an interesting next step to seek insights on the nature of loss landscapes through insights derived from LMC. In this case, we are hesitant to make strong general claims about loss landscapes on the basis of our findings. While it is clear from eqn (10) that constant gradients are a _sufficient_ condition for LMC, in practice we cannot broadly claim that gradients will become _exactly_ constant and a complete theory of LMC may need to take into account further intricacies of neural networks. We hope that a more complete theory of LMC can be developed through future research taking into account the telescoping insights we provide here, which would provide valuable insights into the nature of loss landscapes in deep learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for the time and effort put into the review process! We are grateful for the constructive nature of the reviews and were delighted by the largely positive assessment of our work. We were especially excited by the recurring appreciation of the new insights on (i) the effects of optimization strategies, (ii) the phenomena relating to model complexity (double descent and grokking), and (iii) the relationship between neural networks and gradient boosting, all of which the construction of the telescoping model allowed us to provide in this work. 

We respond to all reviewers individually below, but re-address some recurring comments globally here. We also include additional experimental evidence in the pdf attached to this comment. Note that due to the strict 6000 character limit in rebuttals our responses were forced to be brief. However, we would be happy to expand our discussion on any point if further details are helpful!

**1. Density of writing.** We appreciate that the quantity of content and breadth of phenomena covered in this work required certain parts of the paper to be written quite densely. We will therefore put considerable effort into using the extra page of a camera-ready version to decompress the text (e.g. using less in-line maths particularly in Sec 4.1) and expand on details where reviewers pointed out that there is an opportunity to improve clarity!

**2. Further experiments.** Allow us to re-emphasize that the goal of this work was to provide a tool that can help us to better understand _specific, previously observed_ phenomena in deep learning. Therefore, in our experimental evaluation we primarily revisit the exact well-known, canonical experimental settings in which the studied phenomena have been observed in previous work (e.g. grokking, double descent, LMC). We have rigorously pursued this approach by reproducing key experiments from several works (e.g. [BHMM19, LMT22, KBGP24, FDP+20]) and integrating aspects of the telescoping model to better understand their findings. 

Where specific additional experiments were requested we have provided them in the attached pdf. In particular, the pdf includes:
- Empirical verification of some of the theoretical predictions made in Sec 3.1 on the opposing effects of weight decay and momentum on model predictions using real networks obtained through standard training.
- Additional experiments on grokking in a parity prediction task with varying number of spurious features, further demonstrating that the behaviour of effective parameters indeed predicts grokking behaviour.
- A double descent experiment on a third dataset, the MNIST-1D dataset (Greydanus  \& Kobak, (ICML2024). ``Scaling Down Deep Learning with MNIST-1D.'') which was proposed recently as a sandbox for investigating empirical deep learning phenomena and was also used to demonstrate deep double descent in the textbook [Pri23].

Meta-review: This paper introduces an analytics tool that consist of a sequence of first-order approximations with a telescoping sum to aid the understanding of various deep learning phenomena, as well as the explanation of several empirical observations. 

Most reviewers think the method is technically sound, the presentation is fair (while improvement on the readability can further enhance its quality), and the insights presented are of decent significance. 

While there is some concern on the limited scope on insights, in that more insights than what’s already included in this paper will certainly boost the paper’s strength linearly, most think that the breadth of the insight coverage is significant enough. There is also one remaining concern on the LMC front, which questions the lazy regime hypothesis. While a proxy model would certainly exhibit some performance gap, the insights gleaned from it (and explanations derived) can still contribute positively to one’s understanding of the original model. The only reviewer that recommended against accepting seems to fixate on 1 case study only and shows low confidence on the other 2. 

Therefore I choose to side with the majority of reviewers and recommend accepting.
Decision: Accept (poster)