Deep Double Descent via Smooth Interpolation
Authors: Matteo Gamba
Erik Englesson
Mårten Björkman
Hossein Azizpour
Venue: Accepted by TMLR ()

Abstract: The ability of overparameterized deep networks to interpolate noisy data, while at the same time showing good generalization performance, has been recently characterized in terms of the double descent curve for the test error. Common intuition from polynomial regression suggests that overparameterized networks are able to sharply interpolate noisy data, without considerably deviating from the ground-truth signal, thus preserving generalization ability. At present, a precise characterization of the relationship between interpolation and generalization for deep networks is missing. In this work, we quantify sharpness of fit of the training data interpolated by neural network functions, by studying the loss landscape w.r.t. to the input variable locally to each training point, over volumes around cleanly- and noisily-labelled training samples, as we systematically increase the number of model parameters and training epochs. Our findings show that loss sharpness in the input space follows both model- and epoch-wise double descent, with worse peaks observed around noisy labels. While small interpolating models sharply fit both clean and noisy data, large interpolating models express a smooth loss landscape, where noisy targets are predicted over large volumes around training data points, in contrast to existing intuition.


Published in Transactions on Machine Learning Research (04/2023)
Deep Double Descent via Smooth Interpolation
Mattco Gamba
mgamba@kih.ss
KTH Royal Institute of Technology
Erik Englesson
engless@kth.ss
KTH Rogal Institute of Technology
Mårten Björkman
celle@kth.st se
KTH Royal Institute of Technology
Hossein Azizpour
azizpour@kth.c se
KTH Royal Institute of Technology
Reviewed on OpenReview: https: // openreview. net/ forum? id= fempQstMb?
Abstract
The ability of overparameterized deep networks to interpolate noisy data, while the same time showing good generalization performance, has been recently characterized in terms of the double descent curve for the test error. Common intuition from polynomial regression suggests that overparameterized networks are able to sharply interpolate noisy data, without considerably deviating from the ground-truth signal, thus preserving generalization ability.
At present, procise characterization of the relationship between interpolation and general- ization for deep networks is missing. In this work, we quantily sharpness of fit of the training data interpolated by neural network functions, by studying the loss landscape w.r.t. to the in- put variable locally to each training point, over volumes around clcanly- and noisily-labelled training samples, as we systematically increase the number of model parameters and train- ing epochs. Our findings show that loss sharpness the input space follows both model- and epoch-wise double descent, with worse peaks observed around noisy labels. While small interpolating models sharply fit both clean and noisy data, large interpolating models ex- press smooth loss landscape, where noisy targets are predicted over large volumes around training data points, in contrast to existing intuition
1 Introduction
The ability of overparameterized deep networks to interpolate noisy data, while at the same time showing good generalization performance (Belkin et al. 2018 Zhang et al.] 2018 has been recently characterized in terms of the double descent curve of the test framework, as model size increases, the test error follows the classical bias-variance trade-off curve (Geman et al. 1992), peaking as models become large enough to perfectly interpolate the training data, and decreasing as model size grows further (Belkin et al.) 20119). This phenomenon, largely studied in the context of regression (Bartlett et al.) 2020 Muthukumar et al.] 2020 and random features (Belkin et al. 2020), at present lacks a precise characterization relating interpolation lo generalization for deep networks.
[Blliin et al. 2019 Geiger et al. 2019). Within this
Current intuition from linear and polynomial regression suggests that, under some hypothesis on the training sample, large overparameterized models are able to perfectly interpolate both cleanly- and noisily-labeled samples, without considerably deviating from the ground- truth signal, thus showing good performance de-
spite overfitting the training data (Muthukumar et al. 2020 Bartlett et al. 2020 Nakkiran et al. 2019a
Source code reproduce our results available http:://ggtaae....///ggaaaaaaaaa

---

Published in Transactions on Machine Learning Research (04/2023)
poth puth m2
K=1
pulh
K-3
Ground Truth Ob vertoons
Ground Truth Oeservattons
58000 -1
zn
(a)
(b)
(c)
(d)
Figure 1: Intuition from overparameterized regression. a) Polynomial of large degree, trained with gradient descent to fit noisy scalar data, reproducing the polynomial regression experiment of [akkiran ctt al. (2019a), and reflecting common intuition on double descent, suggesting that the generalization ability of large interpolating models is tied to sharply fitting of noisy data, thus resulting in models that do not deviate considerably from the ground truth signal. b) In this work we show that, contrary to intuition, deep networks smoothly interpolate both clean and noisy data, and that improved generalization in the interpolating regime is tied to smoothness of the loss w.r.t. the input variable. Gcodesic MC integration. For each base training point, we generate geodesic paths by connecting sequence of augmentations of increasing strength, which
to cover volumes of increasing size in the loss landscape around each training point. We compare points that are c) sharply interpolated from those that are d) smoothly interpolated.
Figure la illustrates this phenomenon, showing a polynomial of large degrec that perfectly fits the training data, with predictive function sharply interpolating noisy samples (intuitively corresponding a spike at each training point), while overall remaining close to the data-generating function.
In this work, study the emergence of double descent for the test error of deep networks (Nakkiran et al., 2019b) through the lens of smoothness of interpolation of the training data, model size well as the number of training epochs vary, for models trained in practice. To quantify smoothness of interpolation, we conduct an empirical exploration of the loss landscape w.r.t. the input variable, by providing explicit measures of sharpness of the loss, focusing on image classification.
Due to the inherently noisy nature of Euclidean estimators in pixel space, and following the manifold hy- PPoee
pothesis Pope et al. (2020) ; Bengio (0133;; Narayanan & Mitter (2010), postulating that natural data lies on combination of manifolds of lower dimension than the input data's 's ambient dimension, constrain measures to the support of the data distribution, locally to each training point.
Our empirical study shows that the polynomial intuition in Figure la does not hold in practice for deep networks, which instead smoothly interpolate both clean and noisy data (Figure Tb Specifically, smooth interpolation emerging both for large overparameterized networks and prolonged training results in large models confidently predicting the (noisy) training targets over large volumes around cach training point.
Contributions
We present the first systematic empirical study of smoothness of the loss landscape of deep networks in relation overparameterivation and interpolation for natural image dalasets.
Starting from infinitesimal smoothness measures from prior work, we introduce volumetric measures lhat capture loss smoothness when moving away from training points.
We develop a geodesic Monte Carlo integration method for constraining our measures local approximation of the data manifold, in proximity of each training point..
2

---

Published in Transactions on Machine Learning Research (04/2023)
We present empirical study of model-wise and epoch-wise double descent for neural networks trained without confounders (explicit regularivation, data augmentation, batch normalization); as well as for commonly-fout found training settings. By decoupling smoothness from generalization, we empirically show that overparameterization promotes input-space smoothness of the loss landscape. Particularly, we produce practical examples in which smoothness of the learned function of deep networks does not result improved generalization, highlighting that the implicit regularization effect of overparaaeetrriztion should be studied in terms of reduced variation of the learned function.
2
Related work
Recent years have seen increased interest in the study of smoothness of deep networks in relationship to generalization. For studies of learned representations, interpreting networks as functions of their parameters,
loss landscape smoothness has been related to improved generalization (Ma & Yingl 2021; Foret et al.) 2020:
Rosca et al. 2020). increased stability to perturbations (Keskar et al. 2017), reduced minimum description length (Hochreiter & Schmidhuber 1997), as well as better model compression (Chang et al. 2021
Additionally, for networks interpreted as functions of their imput, for a fixed porumelerization, sensilivily of the networks' learned function has been connected to generalization performance (LeJeune et al.,
2019
Novak ct al. 2018). Indoed mounting evidence, both empirical (Gamba et al., 2022 2020 Novak et al.
2018
as well as theoretical (Bubeck & Sellke, 2021 Neyshabur et al. 2018). suggests that large overparameterized
models achieve robust generalization (Ma & Ying 2021) ) via smoothness of the learned function. While
overparameterizztion alone is not enough to guarantee strong robustness (Chen et al. 2021 Rice et al, 2020), the large number of parameters of modern networks is thought to promote implicit regularization of the network's ' function (Gamba et al.; 2022 Bubeck & Sellke 2021; Neyshabur et. al., 2018
2015). In this context, a direct study of interpolation via the parameter-space interpretation is limited by confounders, such as symmetries of linear layers (Singh & Jaggi 2020 Li et al./ 2015), for which different parameterization may yield the same equivalent interpolating function (Simsek et al. 2021). Thus, our work adopts the input-space view of the loss landscape, to directly study sharpness of interpolation around each training point.
Our methodology builds upon input-space sensitivity analyses for neural networks, presenting a first system- atic study of the role of overparaneterization in promoting smoothness of the network's learned function. The smoothness measures presented in section 3 are inspired by the vast body of work on the loss landscape of neural networks in parameter space. Due to the extensive theoretical literature on double descent in simplisied controlled settings such as linear regression (Muthukumar et al., 2020 Bartlett el al. 2020 Belkin et al. 2018), in the following we mainly draw connections to prior work targeting deep networks.
Deep Double Descent Double descent Belkin et al. 2019 Geiger el al. 2019) was first observed for several machine learning algorithms for increasing model size (model-wise). Later, Nakkiran et al. (2019b) showed a similar trend during training of deep networks (epoch-wise), as well as w.r.t. dataset size (sample-
wise). The phenomenon has been studied from various perspectives: bias-variance decomposition (Yang
et al.
2020
Neal et al. 2018 , samples to parameters ratio (Belkin et. al. 2020 parameter norms (Belkin
let al.
2019 and decision boundaries (Somepalli et al. 2022).
In this work, we study model-wise and epoch-wise double descent in terms of smoothness of the loss landscape with respect to the input, and separate the analysis in terms of clean and noisily-labeled data points. Importantly, in contrast to existing studics of double descent (Belkin et al., 2020),, we focus on input space and on the training loss quantity that does not follow double descent and study sharpness metrics based on training data, showing that they strongly correlate with the test error.
The most related work to ours is the concurrent one of (Somopall et al. (202)) studying decision boundaries in terms of reproducibility and double descent. Our works differ in that we study double descent in the loss landscape. Furthermore, our study takes a closer look at the impact of clean and noisily-labeled samples, and presents scttings in which the emergence of regularity of the loss landscape does not result in improved generalization. Finally, we focus on implicit regularization (by disabling batch norm, data augmentation) and a simpler optimization procedure (constant learning rate SGD) to reduce confounding factors.
3

---

Published in Transactions on Machine Learning Research (04/2023)
Loss Landscape of Neural Networks To understand the remarkable generalization ability of deep 2020 Geiger et al.) 2019; Keskar et al., 2017), well as to design better training 2020 , several works study the loss landscape of deep networks in parameter space,
networks (Xie et al.
criteria (Foret et al.
focusing on solutions obtained by SGD (Kuditipudi et al.) 2019), as well as the optimization process (Arora 2021). Inspired by such literalure, we quantily smoothness of the loss landscape by estimating the sharpness of the loss, as proposed by Forct et al. (020)) and Keskar et al. (2017) for the parameter-space, but we perform our analysis in inpul-space. Importantly, in this work we focus on image classification tasks, and study smoothness of interpolation of training data points.
et al. 2022 Li et al.
Input Space Sensitivity and Smoothness Novak et al. 2018 present an empirical sensitivity study of fully-connected networks with piece-wise linear activation functions through the input-out put. Jacobian norm, which is shown to strongly correlate to the generalization ability of the networks considered. Their study proposes an infinitesimal analysis of the Jacobian norm at training and validation points, as well as the use of input-space trajectories in proximity of the data manifold to probe trained networks. LeJeune et al. (2019) analyse second-order information (the tangent Hessian of a neural network) by using weak data augmentation to constrain their measure to the proximity of the data manifold. Lastly, Gamba et. al. (2022) introduce a nonlinarity measure for piece-wist wise linear networks, that strongly correlates with the test error in the second descent for large overparamcterized models. Similar to the first two works, we study smoothness of neural networks, using the Jacobian and Hessian norm of neural networks trained in practice, and similar to the latter work, we provide a systematic study of double descent, which we further exlend to epoch-wise trends.
Finally, Rosca et al. (2020) postulate a connection between model-wise double descent and smoothness: during the first ascent models fit the training data at the expense of smoothness, while the second descent happens as the model size becomes large enough for smoothness to increase.
Later, Bubeck & Sellke (2021) theoretically prove universal law of robustness highlighting trade-off between the model size and the Lipschitz constant of a learning algorithm w.r.t. its input variable. Our work provides empirical evidence supporting the postulate of Rosca et al. (220)) and the law of robustness
of Bubeck & Sellke (2021)
3 Methodology
Our leading research question is to quantify smoothness of interpolation of training data for deep networks trained on classification tasks, as the number of model parameters is increased. We interpret network as a function with input variable x E Rdd and learnable parameter 0, incorporating all weights and biases. Our study focuses on the landscape of the loss Lo y) := [( x, v) treated as a function of the input x, with larget y. Inspired by the literature on the loss landscape of neural networks in parameter space (Forel et al. 2020 Dinh et al.) 2017 Keskar et al., (017)), we quantify (the lack of) smoothness by devising explicit measures of loss sharpness in neighbourhood of training points (Xn; for I....,N. Crucially, for any given network, we focus on sharpness w.r.t. the input variable x, keeping the parameter 0 fixed.
We begin by describing infinitesimal sharpness in section 3.1 which we compute in proximity of the data manifold local to each training point in section 3.2 Finally, we introduce a method for estimating sharp- ness over data-driven volumes by exploiting data augmentation in section 3.3, and in section 3.4 we detail the chosen data augmentation strategics. The proposed methodology enables us to measure sharpness of interpolation of the training data, by restricting our study near the support of the data distribution.
3.1 Sharpness at Data Points
To estimate how sharply the loss changes w.r.t. infinitesimal perturbations of the input variable x, we study the Jacobian of the loss
J y) = Ø Ox Lo v)
(1)
4

---

Published in Transactions on Machine Learning Research (04/2023)
To measure sharpness at point (Xn: Un), follow Novak al. (2018) and compute the l norm of [xn; which we take in expectation over the training set DD {(x,; n=11
N
J = ED (xx
(2)
assuming that the loss is differentiable one time at the points considered. Intuitively, sharpness is mcasured by how fast the loss Lo(x, y) changes in infinitesimal neighbourhoods of the training data, and network is said to smoothly interpolate a data point x, if the loss is approximately flat locally around the point and the point is classified correctly according to the corresponding target ya- Throughout our experiments, the Jacobian J is computed using a backward pass w.r.t. the input variable x.
Equation provides first-order information about the loss landscape. To gain knowledge about curvature, we also study the Hessian of the loss w.r.t. the imput variable,
H(x, 0x0x7 Co(x,y)
(3)
whose Frobenius norm again we take in expectation over the training
Ep H(x, y)ll2
(4)
The Hessian tensor Equation 3 depends quadratically on the input space dimensionality d, providing a noisy Euclidean estimator of loss curvature in proximity of the input data. Following the manifold hypothe- sis (Bengio 2013 Narayanan & Mitter 2010 stating that natural data lies subspaces of dimensionality lower than the ambient dimension d, we restrict Hessian computation to the tangent subspace of each train- ing point X,. Starting from Equation 1 throughout our experiments, Equation 3 is estimated by computing the langent Hessian, as outlined in the next section.
3.2 Tangent Hessian Estimation
To constrain Equation 3 to the support of the data distribution, we adapt the method by LeJeune et al. 2019) and estimate the loss Hessian norm projected onto the data manifold local to each training point.
For any input data point (Xa, Un) and corresponding Jacobian J(x,, yv). we generate M augmented data points X, + Um by randomly sampling displacement vector un using weak data augmentation. For each sampled um we then estimate the Hessian Hx,,, Vn) projected along the direction x, Um by computing the finite difference [X(n yn) J(x, 50m,Ba)) Then, following Donoho & Grimes (2003) we estimate the Hessian norm directly by computing
J (x,, Vn) J(x, + Sun vn)113) +
=
ED M28
which equivalent to rescaled version of the rugosity measure of LeJeune al. (0199) Importantly, different from rugosity, we generate augmentations x,, um by using weak colour transformations in place of affine transformations (1-pixel shifts), since weak photometric transformations are guaranteed to be fully on-manifold Details about the specific colour transformations are presented in appendix C
3.3 Sharpness over DriveDriven Volumes
The measures introduced in Equations 2 and 5 capture local sharpness over infinitesimal neighbourhoods of input data points. To study how different networks fit the training data, we devise a method for estimating loss sharpness over volumes centered at each training point Xa, as one moves away from the point. Essentially, we exploit a variant of Monte Carlo (MC) integration to capture sharpness over data-driven volumes, by applying two steps. First, we integrate the Jacobian and Hessian norms along geodesic paths म C Rd basod at Xx, on the data manifold local to each training point, for p ......A P. Second, we estimate sharpness over the volume covered by the loss along the P paths via MC integration. The following details each step.
5

---

Published in Transactions on Machine Learning Research (04/2023)
Sharpness along geodesic paths For each training point (Xn, Un) D, aim to estimate loss sharpness we move away from Xn, while traveling the support of the data distribution. To do so, we exploit a sequence of weak data augmentations of increasing strength to generate / paths Tp C g in the input space, each formed by connecting augmentations of x, in order of increasing strength.
Formally, let Ts : RRd + Rdd, represent a family of smooth transformations (data augmentation) acting on the input space and governed by parameter 8, controlling the strength S = ||s|| as well as the direction of the augmentation in R In general, the parameter s, interpreted models the randomness of the transformation. Randomly sampling s, yields value g corresponding to fixed transformation Tap. of strength S For instance, for affine translations, sp.k models a random radial direction sampled from a hypersphere centered at Xn, with strength S denoting the magnitude of the translation (e.g. 4-pixel shift). For photometric transformations, sp, may model the change in brightness, contrast, huc, and saturation, with total strength S*
suitably distributed random variable,
To generate on-manifold paths up starting from Xn: we proceed as follows. First, fix sequence of K + 1 strengths S < 8' < ... S with 8 = 0 denoting the identity transformation V p. Then, for each strength Sk, with k 1, p random directions gp. are sampled, each with respective fixed magnitude ||s** [|2 = S*. This yields P sequences of transformations (Tom. 1 k==o cach producing augmented versions x of Xn, ordered by strength, x <... < xp_K, and forming a path Fp C Rdd Specifically, each path xp approximates an on on-manifold trajectory by a sequence of Euclidean segments x xp.++l xp. for k = ......, The maximum augmentation strength S controls the distance traveled from Xn, while the number of strengths used controls how fine-gre grained the Euclidean approximation is. Pseudocode for generating geodesic paths is presented in section D
Volume integration Once sequence of paths {Tp}p== is generated for Xa, volume-based sharpness is computed by integrating over each path Ty, and normalizing the measure by the length len(TTp) of each path:
/ (nx,xx)dx
P Σ len(TTp) Jp
(6)
where o represents an infinitesimal sharpness mcasure, namely the Jacobian and tangent Hcssian norms at (xn, vn). The same method can also be applied to accuracy and crossentropy loss to evaluate consistency and confidence of the models predictions over volumes. Figures [c and [d] illustrate geodesic MC integra- tion. For each training point, P geodesie paths are generated, each anchored to the data manifold by K augmentations. Integrating infinitesimal measures over each path returns MC sample of sharpness along Tp- Then, volumetric sharpness is estimated by MC integration over P samples. Importantly, the number P of paths is fixed throughout all experiments, representing the number of MC samples for volumo-basod integration. Finally, we take mean-filed view by averaging over the training set D:
((x, / n, ED Σ len (TT))))) P
1
N Σ pm1
axx, Bn) / len (TTp) J
NP
p=l
Importantly, extending LeJeune et al. (2019). we replace Fuclidean integration by geodesic integration over local approximation of the data manifold, by generating augmentations of increasing strength.
Crucially, the proposed MC integration captures average-case sharpness in proximity of the training data and is directly related to the generalization ability of the studied networks, as opposed to worst-case sensitivity, as typically considered in adversarial settings [Moosavi-Dezfood et al. 2019). In fact, the random sampling performed in Equation 7 is unlikely to hit adversarial directions, which are commonly identified by searching the input space through an optimization process (Goodfellow et al. 2014 Szegedy et al. 2013).
To conclude our methodology, in section 3.4 we present the family of transformations To used for generating trajectories np throughout our experiments.

---

Published in Transactions on Machine Learning Research (04/2023)
10-
Than -1.
1.0
accuracy
crossentropy
30-
jacobian
tangent hesslan
Volume
0.3m
Emar
Train loss
02-
1100
22-
-0.7
0.0
6,2-
1-
20-
Width
Bose Wicth
Width
Width
(a)
(b)
Figure 2: a) Double descent curve for the test error for ConvNets trained on CIFAR-10 with 20% noisy labels. b) Average metrics integrated over volumes of increasing size. Volumes are denoted by the number K of weak angmentations used to generate each geodesic path. From left to right: average training accuracy, training loss, Jacobian norm and Hessian norm, each plotted against. model size. The dotted vertical line marks the model width that achieves zero train error (i.c. the interpolation threshold). All models are trained for 4k epochs. We observe accuracy over volumes increases monotonically with model size, while crossentropy follows double descent. Combined, the two observations suggest that large networks confidently predict the training targets over increasingly large volumes around the training data (for increasing model size). Importantly, interpolation is sharp at the interpolation threshold, even infinitesimally at each training point (blue curves), while increasing overparameterization produces smooth interpolation, contrary to existing intuition. Shaded areas mark standard deviations over seeds.
3.4 Weak Data Augmentation Strategies
Computing sharpness of interpolation via Equation each data point x,, requires generating P trajectories Tp composed of augmentations of Xn of controlled increasing strength. Furthermore, the augmented data 1==0 should lie in proximity of the base point x,, in order for the Euclidaan approximation to be meaningful. Finally, to correctly estimate correlation between smoothness and the generalization ability of the networks considered, volume- based sharpness should not rely on validation data points, i.e. the augmentations xfi should be strongly correlated to X,,, for each p, k.
points (xx {
To satisfy the above, we modify weak data augmentation algorithm introduced by Yu et al. 2018 which allows to efficiently generate augmentations that lie in close proximity to the base training point x,, for image data. Specifically, each base image Xa, consisting of C' input channels (e.g. C = 3 for RGB images) and h x w spatial dimensions, is interpreted as C independent matrices x,|c,
R g each factorized using Singular Value Decomposition (SVD), yiclding a docomposition x, :,; = UEV. where E' is diagonal matrix whose entries are the singular values of xn(c;a;; sorted by decreasing magnitude. In the original method, Yu et al. (2018) produce weak augmentations by randomly erasing one singular value from the smallest ones, thereby obtaining modified matrix Σ', and then reconstructing each channel of the base sample via V In this work, in order to generate P random augmentations of strength k, Σe is Essentially,
P- obtained by erasing & singular values Di for i = w - - w w-p. and p = O...., the augmentation strength is given by the number of singular values erased, and P augmentations similar strength are generated by erasing subsets of size from the smallest singular values, for each channel
We note that this method produces augmented images that are highly correlated with the corresponding base training sample, and as such they do not directly amount to producing validation data, points. We refer the reader lo appendix E for further details. In the next section, we present our empirical study of sharpness of interpolation for neural networks in relationship to double descent.
4 Experiments
In this section, we present our empirical exploration of input-space smoothness of the loss landscape of deep networks as model size and number of training epochs vary. Focusing on implicil regularization (Neyshabur
2 "Assuming square spatial dimensions

---

Published in Transactions on Machine Learning Research (04/2023)
et al. 2015 promoted by optimization and model architecture, evaluate our sharpness measures net- works with increasing number of parameters, trained without any form of explicit regularization (e.g. weight decay, batch normalization, dropout). We extend our analysis to common training settings in section 4.4
Experimental setup We reproduce deep double descent by following the experimental setup of Nakkiran et al. (011b)). Specifically, we train family of ConvNets formed by 4 convolutional stages of controlled base width [w, 2w, 4w, 8w], for w = 1,...., on the CIFAR-10 datasct with 20% noisy training labels and on CIFAR-100. All models are trained for 4k epochs using SGD with momentum 0.9 and fixed learning rate. Following Arpit et al. (2019). to stabilize prolonged training, we use a learning rate warmup schedule. Furthermore, we extend our empirical results to training settings more commonly found in practice, and validate our main findings on a series of ResNet18s (He et al., 2015) of increasing base width w = 1....,64, 64, with batch normalization, trained with the Adam optimizer for 4k epochs using data augmentation. We refer the reader to section B for a full description of our experimental setting. In section G.1. we extend our main results to Transformer networks trained on machine translation tasks.
We begin our experiments by reproducing double descent for the test error for the ConyNets (Figure 2a Starting with small models and by increasing model size, a U-shaped curve is observed whereupon small models underfit. the training data, as indicated by high train and test error. As model size increases, the optimal bias/variance trade-off is reached (Geman et al.) 1992]. Mid-sized models increasingly overfit training data as shown by incrcasing test error for decrcasing train error and loss until zero training error is achieved, and the training data is interpolated. The smallest interpolating model size is typically referred to as interpolation threshold (Belkin et al.) 2019). Near said threshold, the test error peaks. Finally, large overparameterized models achieve improved generalization, as marked by decreasing test error, while still interpolating the training set.
4.1 Loss Landscape Smoothness Follows Double Descent
In this section, we establish a strong correlation between double descent of the test error and smooth interpolation of noisy training data. Figure 2b studies fitting of training data for models at convergence (training for 4k epochs) as model size increases. Starting with (infinitcsimal) sharpness at training points (blue curve), we observe that training accuracy at convergence monotonically increases with model size, with 100% accuracy reached at the interpolation threshold and maintained therefrom. At the same time, crossentropy loss over volumes follows double descent, with peak near the interpolation threshold, and then decreasing model size grows. Similarly, the Jacobian and Hessian norms peak at the interpolation threshold and then rapidly decrease, showing that all training points become stationary for the loss, and that the landscape becomes flatter as model size grows past the interpolation threshold. When all measures are integrated over volumes of increasing size (number K of augmentations per path), we observe how large overparameterized models are able to smoothly fit. the training data over large volumes. This finding suggests that in contrast to the polynomial intuition of Figure lab
overparamcterized networks interpolate training
data smoothly (as intuitively depicted in Figure 1b
Our finding extends the observations of Novak et al. 2018) and LeJeune et al. (019 from fixed-size networks to spectrum of model sizes, and establishes a clear correlation with the test error peak in double descent. Finally, the results substantiate the universal law of robustness (Bubeck & Sellke 2021), showing that at the interpolation threshold highest sensitivity to input perturbations is observed, while overparaneterization beyond the threshold promotes smoothness. Intriguingly, our findings represent mean sharpness as opposed to the worst case studied by Bubeck & Sellke (2021), showing that the observed regularity is much stronger in practice. In the following section, we study this behaviour in proximity of cleanly- and noisily-labeled
training samples. We refer the reader to section 4.4 for analogous results on ResNets trained with Adam.
4.2 Smooth Interpolation of Noisy Labels
In this section, we break down the noisily labeled training set into two subsets: cleanly-labeled points, and training points with corrupted labels, and explore how fitting is affected by the training labels.
8

---

Published in Transactions on Machine Learning Research (04/2023)
accuracy
crossentropy
jacobian
tangent hesslan
1.0
Volume
120-
6m
100
clean
0.6
30
20-
60
40
10-
0.2
0.8
40-
120
100
corrupted
20
0.4
40
10-
0.2
20-
0
20
Width
Width
Width
Figure 3: Average accuracy, crossentropy, Jacobian and Hessian norms integrated over volumes of increasing size (augmentations per path) around clean (lop) and noisy (bottom) subsets of the CIFAR-10 training set with 20% noisy labels. For models near the interpolation threshold, we observe a large increase in the loss for increasing neighborhood size. At the interpolation threshold, sharp interpolation is observed for both clean and noisy samples, with crossentropy, sensitivity (Jacobian norm) and curvature peaking over all volumes considered. Larger models present smoother loss landscape around training points, with the largest models expressing a locally flat landscape around each point. This finding shows that large networks confidently and smoothly predicting the noisy labels around data points whose label was corrupted, suggesting that snoothness emerging from overparameterization in fact hinders generalization locally to those points.
accuracy
crossentropy
jacoblan
tangent hessian
T:tn Brir
-35
40-
Volumi
Enor
-10
0.8-
120
10-
100
0.5-
20
0.6-
80
15
20-
60
-00
0.0m
10-
40
-05
20-
0.2
25
20
Base Whath
Base Wiath
Bose width
Bose Whath
Base Width
(a)
(b)
Figure a) Double descent for the test error for ConvNets trained on CIFAR-100. b) Average metrics integrated over volumes of increasing size (number K of augmentations per path). From left right: average training accuracy, crossentropy, Jacobian, and Hessian norm, cach plotted against model size. All models are trained for 4k epochs. For relatively complex datasets (i.e. with few samples per class), findings hold even without artificially corrupted labels, suggesting that the trends reported in this work are not caused by synthetic noise. Shaded areas depict standard deviations over 3 seeds.
Figure 3 reports accuracy, crossentropy, as well as sharpness measures computed on the clean subset of CIFAR-10 (top), as well as the corrupted subset (boltom), for volumes of increasing size. We begin by noting that small models fit mostly the cleanly labeled data points, and show close to zero accuracy on the noisily labeled data points, showing bias towards learning simple patterns. We hypothesize that most cleanly labeled samples act as "simple examples", while noisily labeled ones provide "hard examples", akin to support vectors, for small size models. This behaviour is aligned with prior observations, reporting that
9

---

Published in Transactions on Machine Learning Research (04/2023)
Tost Error
crossentrasy
Icoollon
-20
1001
-1.
100
300
-1.
100-
-03
-0.
10-
-04 10-
10-
d
d
1
80
Buse
Figure (Left) Test error (Middle) Train crossentropy (Right) Jacobian norm for ConvNets trained on CIFAR-10 with 20% noisy labels. The healmaps show each metric for increasing training epochs (y-axis) and base width (x-axis). Models past the interpolation threshold (base width 15) undergo epoch-wise double descent for each metric. Similar trends are observed for curvature, as measured by the Hessian norm. All metrics are computed on the training set only, without geodesic Monte Carlo integration.
deep net admit support vectors Toneva et al. 2018 and that deep networks share the order in which samples are fitted (Hacohen et al. 2020). We refer the reader to Hacohen et al. (2020) for details.
As model size grows toward the interpolation threshold, networks fit both clean and noisy samples (as marked by increasing accuracy on both subsets), with large models consistently predicting the clean and noisy labels over large volumes. At the same time, crossentropy local to each training point (blue curve) approaches zero past the interpolation threshold, while volume- based crossentropy undergoes double descent. Interestingly, this trend is observed both around cleanly- and noisily-labeled training samples, with peaks at the interpolation threshold which are considerably more marked for noisy labels.
Our sharpness measures follow double descent for all volumes considered, even when no Monte Carlo integra- tion performed (blue curve). Importantly, curvature as measured by the Hessian norm rapidly decreases model size grows, showing that large networks smoothly interpolate both clean and noisy samples. Im- portantly, observe how the second descent in test error corresponds to improved fitting of cleanly-labeled samples, while the network lose their generalization ability locally to noisy labeled points.
In Figure 4a we extend the observations to CIFAR-100, where modcl-wise double descent is obsorved even on the standard dataset without artificially corrupted labels. Similarly to what observed on CIFAR-10, Figure 4b shows the loss landscape peaking in sharpness at the interpolation threshold, and then rapidly decreasing as model size grows, with large networks smoothly fitting the training set over increasingly large volumes. This finding suggests that double descent is tied to dataset complexity, and that the trends reported in this work are not caused by artificially corrupted labels.
4.3 Epochwise Bouble-Descees
We now turn our attention epoch-wise double descent, first reported for the test error of deep networks by (Nakkiran et al.) 2019b). Figure shows the test error (left), train crossentropy (middle), well Jacobian norm (right) for ConvNets trained on CIFAR-10 with 20% noisy labels. We consolidate obscrvations for each metric with hcatmaps, in which the y-axis represents training epochs, and the x-axis denotes the models' base width. We observe that models past the interpolation threshold (base width w = 15) undergo epoch-wise double descent for each metric. At the same time, models with base width w < 15 are unable reduce their test error within 4k training epochs, and this is associated to non-decreasing training loss as well as Jacobian norm. We hypothesize that the model size affects model's ability to interpolate the training data, and therefore affects the training dynamics and the occurrence of epoch-wise double descent..
10

---

Published in Transactions on Machine Learning Research (04/2023)
accuracy
crossentropy
jacobian
tangent hessian
1.0
Trai Emor
110
Vilume
Nost Enar
-1.35
0.9-
2.2
25-
0.8-
102
0.4-
0./-
3.75
0.2-
-035
4.50
0.
3.52
10-
35
An
db
40
db
de Base Widhh
80
do
Base Wicth
Base Whdth
Width
(a)
(b)
Figure 6: (a) Double descent for the test error the CIFAR-10 with 20% noisy labels for family of ResNet18s of increasing base width w, trained with data augmentation. (b) Accuracy, crossentropy, Jacobian and Hessian norms over volumes. All models are trained for 4k epochs. Analogous trends as observed for the ConvNets holds in this case. However, the largest integration volume considered (K = 7 augmentations per path), now shows considerably increased sharpness and loss curvature, while still undergoing double descent.
4.4 Practical Training Settings
So far, the training setting included the least amount of confounders (e.g. adaptive learning rates, explicit regularization, skip connections, normalization layers) and focused on implicit regularization. In Figure C we extend our findings to ResNets trained on CIFAR-10 with 20% noisy labels, with the Adam optimizer, data augmentation (4-pixel shifts and random horizontal flips), as well as batch normalization layer (see appendix B for details). Both model-wise and epoch-wise trends reported for the ConvNets also hold for this setup, with the interpolation threshold occurring at base width w = 18. We consolidate our model-wist -ise and epoch-wise findings with heatmaps in Figure 12 and 14 Interestingly, data augmentation causes the peak in test error to occur earlier than the interpolation threshold. We hypothesize that the mismatch which can also be observed in related works (Nakkiran et al. 2019b is due to lack of fine grained control over model size as base width w varies. Importantly, for large volumes around training points (K = 7 augmentations per path), training accuracy degrades and loss sharpness increases. However, all sharpness metrics undergo double descent as model size grows, confirming the trends reported in simpler training settings.
4.5 Towards Decoupling Smoothness from Generalization
Our experimental results suggest that double descent in the lest error is closely related to input-space smoothness. One possible interpretation is that models at the interpolation threshold learn small and irregular decision regions, marked by high loss sharpness, while large models learn more regular decision regions with wider margins; supporting the observations of Jiang et al. (2019).
In fact, as consistently observed in our experiments, on the one hand, models near the interpolation threshold fail to smoothly interpolate all clean samples, while on the other hand large models smoothly interpolate the entire training set. This effectively enforces a trade-off for which large models lose generalization ability around noisy samples, but can correctly classify all clean samples. Assuming the train and test distributions are similar (i.e. excluding covariate shifts), this would in turn result in improved average test error past the interpolation threshold, as indeed observed in practice. To assess the validity of our interpretation, we decouple smoothness from generalization by studying training settings in which smooth training set interpolation hurts gencralization. In this sctting, we expect smooth interpolation to consistently emerge with overparameterization, but this time without produeing double descent in the test. error. To corroborate our interpretation, in principle, one would need to construct nearest neighbour classifier (either in input in feature space), and test whether predictions for each test sample are affected by proximity to corrupted samples. In the following, we propose simple experiment to decouple smoothness from generalization, without requiring knowledge of proximity of test samples to train samples.
First, we corrupt 20% of the CIFAR-100 training set with asymmetric label noise, such that 80% samples of 20 randomly selected classes are perturbed. At test time, this enables us to split the test set into (1) samples whose classes have been corrupted, and (2) samples belonging to unperturbed classes. Figure 7a
11

---

Published in Transactions on Machine Learning Research (04/2023)
accuracy
crossentropy
iacobian
tangent hessian
Volume
0.2m
02-
0--
TrE OTD!
deen
0.2
54
10
0.2m
2540
Volume
communtee
0.--
10-
0.4-
40
0.2-
0
20
20
Base Width
Base Witth
Base Width
Bose Witth
20
Base Width
(b)
Figure 7: Decoupling smoothness from generalization. We present an experiment in which 20 randomly selected classes of the CIFAR-100 training split are corrupted with asymmetric label noise, perturbing 80% training labels within each class, for total of 20% corrupted training samples. At lest time, this enables splitting the test set into classes that have been corrupted at train time, and unperturbed classes. (a) Overparameterization promotes a smooth and flat loss landscape around both cleanly-labeled as well as noisy training samples under asymmetric label noise. (b, top) Confirming our hypothesis, double descent for the test error can still be observed for the unperturbed classes, while the trend disappears for the corrupted classes. This finding shows that overparameterization promotes smoothness in the input variable, which is aligned with generalization only around cleanly labeled points. (b, bottom) For networks trained 100% noisy labels, smooth interpolation still follows double descent over volumes of increasing size around training points, but such property in this case is not aligned with generalization.
shows that, even under strong asymmetric noise, overparametterizatin promotes input-space smoothness over increasingly large volumes around both clean and noisy training samples. Perhaps surprisingly, in Figure 7b (top) at test time double descent is still observed for Lest. samples belonging to unperturbed classes, while the trend disappears for the corrupted classes. This confirms our interpretation and shows that double descent should be understood in terms of input-space smoothness, and its relation to generalization.
Second, we train ResNets CIFAR-10 with all training labels corrupted (Figure 7b bottom). Also in this setting, loss sharpness over volumes follows double descent, peaking near the interpolation threshold, and decreasing with increasing model size. Trivially, all networks in this setting lose their generalization ability, with performance close to random chance. This finding shows that overparameterization promotes smooth interpolation of the training data, and that such property is not necessarily aligned with generalization.
5 Conclusions
In this work, we present geodesic Monte Carlo integration tools to study the input space of neural networks, providing intuition built extensive experiments how neural networks fit training data. We present strong correlation between epoch-wise as well as model-wise double descent for the test error and smoothness of the loss landscape in input space. Our experiments show that overparameterization promotes input space regularity via smooth interpolation of clean and noisy training data, which is aligned with improved gener- alization for datasets with relatively low ratio of label noise. Crucially, contrary to intuitions in polynomial regression, deep networks uniformly prodict noisy training targets over volumes around noisily-labcled train- ing samples behaviour which may have severe negative impact practical applications with imbalanced training sets with covariate shifts of the population distribution.
12

---

Published in Transactions on Machine Learning Research (04/2023)
Consistently in experiments, we observe peak in test error and loss sharpness near the interpolation threshold, which decreases for better generalizing models. Finally, for increasing volumes around each training point, we observe that overparametrization promotes flatter minima of the loss in input space, providing initial clucs as to why large overparameterized models generalize better, and corroborating the findings of [Somepalli et al. (2022) on regularity of decision boundaries of overparameterized classifiers, as well as Gamba et al. (2022) on input-space regularity.
Our analysis substantiates the law of robustness of Bubeck & Sellke 2021 and extends the findings of Novak et al. (0188) to experimental settings with controlled model size. We hypothesize that overparameterization affects the dynamics of optimization and interpolation, promoting a smooth loss landscape. An intcresting open problem is characterizing the impact of individual layers on interpolation, as model size grows.
Finally, our analysis opens the question of whether increased interpolation smoothness is to be attributed the model architecture, the optimizer, or a combination of both. First, increased network width, as controlled in our experiments, has been recently connected to the existence of paths connecting critical points for the optimizer (Simsek et al. (021), suggesting that model width plays important role in affecting the dynamics of the optimizer. Particularly, one mode connected manifold of minima may allow wider networks to retain interpolation while at the same time optimizing for input-space smoothness (Li et al. 2021). Second, understanding the existence of implicit regularization promoted by the optimizer is at present an active area of research. On the one hand, several studies argue that stochastic optimization, and the potential implicit regularization effect of muiii-bttel noise, are not required for generalization (Chiang et al. 2023 Paquette al. 2022 Geiping et al., 2020). On the other hand, current models of double descent hypothesize that stochastic noise an important component in explaining implicit regularization and double descent in deep learning
Li et al. 2021
Blanc et al. 2020
Acknowledgments
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Scientific computation was enabled by the supercomputing resource Berzelius provided by National Supercomputer Centre at Linköping University and the Knut and Alice Wallenberg foundation. The work was partially funded by Swodish Research Council project 2017-04609.
References
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on edge of stability in deep learning. arXiv preprint arXi::2205.0914 09745, 2022.
Devansh Arpit, Victor Campos, and Yoshua Bengio. How to initialize your network? robust. initialization for weightnorm & resnets. Advances in Neural Information Processing Systems, 32, 2019.
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in lin- car regression. Proceedings of the National Academy of Sciences, 117(48):3006333000 2020. doi: 0tt.7//pnaa..907788..
Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance 15854, 2019.
variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathemalics of Data Science, 2(4):1167-1188) 2020.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Yoshua Bengio. Deep learning of representations: Looking forward. In International conference statistical language and speech processing, pp. 1-37. Springer, 2013.
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an orustein-ublenbeck like process. In Conference on learning theory, pp. 483-513. PMLR, 2020.
Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Advances in Neural Information Processing Systems, 34, 2021.
Mauro Cettolo, Christian Girardi, and Marcello Federico. Wit3: Web inventory of transcribed and translated talks. In Proceedings of the Conference of Europeon Association for Machine Translation (EAMT), pp. 261-268, 2012.
Xiangyu Chang, Yingcong Li, Samet Oymak, and Christos Thrampoulidis.. Provable benefits of overparam- cterization in model compression: From double descent to pruning neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6974 6983, 2021.
Tianlong Chen, Zhenyn Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Represenlations, 2021.
P. Chiang, R. Ni, D.Y. Miller, A. Bansal, J. Geiping, M. Goldblum, and T. Goldstein. Gradient-basee opti- mization is not necessary for generalization neural networks. In International Conference on Learning Representations, 2023.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. Tn International Conference on Machine Learning, pp- 1019-1028. PMLR, 2017.
David L. Donoho and Carrie Grimes. Hessian eigenmaps: Locally linear embedding techniques for high- dimensional data. Proceedings of the National Academy of Sciences, 100(10):5591-5396 2003. doi: 10. 1073/pnase http/paaa.1111610000 1031596100.
Pierre Foret, Ariel Kloiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awarr aware minimization for efficiently improving generalization. In International Conference on Learning Represenlations, 2020.
Matteo Gamba, Stefan Carlsson, Hossein Azizpour, and Mårten Björkman. Hyperplane arrangements of trained convnets are biased. arXiv preprint ar 077997 2020.
Matteo Gamba, Adrian Shmielewski-Anderr Josephine Sullivan, Hossein Azizpour, and Märten Björkman. Are all linear regions created equal? In International Conference on Artificial Intelligence and Statistics, pp. 6573 6590. PMLR, 2022.
Mario Geiger, Stefano Spigler, Stéphane d' d'Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart. Jamming transition as a paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100(1):012115, 2019.
Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. In International Conference on Learning Representations, 2020.
Stuart Geman, Elic Bienenstock, and René Doursat. Neural Networks and the Bias Bias/Variance Variiace Dilemma. Neural Compulation, 4(1):1 58, 01 1992. ISSN 0899-7667. doi: 10. 10.1162/necoo http://nncc.......... 1992.4.1.1.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint ar Xro: 14..6572... 2014.
Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datascts. In Hal Daumé III and Aarti Singh (ods.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3950 3960. PMIRR 13 18 Jul 2020.
14

---

Published in Transactions on Machine Learning Research (04/2023)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE International Conference Computer Vision, pp. 1026 1034, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Newral compulation, 9()):1-42, 1997.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In International Conference on Learning Representations, 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Con- ference on Learning Representations, 2017.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems, 32, 2019.
Daniel LeJeune, Randall Balestriero, Hamid Javadi, and Richard G Baraniuk. Implicit rugosity regularization via data augmentation. ar preprint arXiv:1905.11639, 2019.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXie preprint orXi::1511.0754,, 2015.
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?-a mathematical framework. In International Conference on Learning Represenlations, 2021.
Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 16805-16817. Curran Associates, Inc., 2021.
Matouš Macháček and Ondřej Bojar. Results of the wmn114 metries shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 293 100-0,,,400 2014.
Seyed-Mohsen Moosavi-Dezfoolii Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1):67 83, 2020.
Prectum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Πya Sutskever. Deep double descent, 2019a. URL
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and 1lya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2019b.
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. Advances in neural information processing systems, 23, 2010.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julien, and Toannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arxio preprinl arXiv:1810.0859/ 2018.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In International Conference on Learning Representations Workshop Track, 2015.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrization in generalization of neural networks. In International Conference on Learning Rep- resentations, 2018.
15

---

Published in Transactions on Machine Learning Research (04/2023)
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations, 2018.
Courtney Paquette, Elliot. Paquette, Ben Adlam, and Jeffrey Pennington. Implicit regularization or implicit conditioning? exact risk trajectories of sgd in high dimensions. In Advances in Neural Information Processing Systems, 2022.
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact learning. In International Conference on Learning Represenlations, 2020.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8093-8104 PMLR, 13-18 Jul 2020.
Mihaela Rosca, Theophane Weber, Arthur Gretton, and Shakir Mohamed. A case for new neural network smoothness constraints. NewrIPS Workshops, 2020.
Berfin Simsek, François God, Arthur Jacot, Francosco Spadaro, Clement Hongler, Wulfram Gerstner, and Johanni Brea. Geomelry of the loss landscape in overparameetrriee neural networks: Symmetries and invariances. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9722 9732. PMLR, 18-24 Jul 2021.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045- 23:22004-22055,, 2020.
Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Ych-Chiang. Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13699-13708, 2022.
Christian Szegody, Wojeiech Zaremba, Ilya Sutskover, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprinl Xiv::13312 6199, 2013.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Ge- offrey Gordon. An empirical study of example lorgetting during deep neural network learning. In International Conference on Learning Representations, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you noed. Advances in neural information processing systems, 30, 2017.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gra- dient descent exponentially favors flat minima. In International Conference on Learning Representations, 2020.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pp. 10767- 10777. PMLR, 2020.
Tao Yu, Huan Long, and John E Hopcroft. Curvature-based comparison of two neural networks. In 2018 24th International Conference on Pallern Recognilion (ICPR), pp. 441 447. IEEE, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalivation. International Conference on Learning Representations, 2018.
16

---

Published in Transactions on Machine Learning Research (04/2023)
A
Appendix
D and D respectively detail the tangent Hessian computation method, the geodesic path generation algorithm, and the weak data augmentation strategy used for geodesic Monte Carlo integration. Finally, in section we extend our discussion of related works. Additional experiments are reported in section
Section B summaarizes our experimental setup, while Section C
B Network Architectures and Training Setup
Network Architectures The ConvNets and ResNets used follow the experimental settings of Nakkiran et al. (019bb), with the only difference that we disable batch normalization in order to focus our study on implicit regularization. In summary, the ConvNets are composed of 4 convolutional stages (cach with a single conv + ReLU block) with kernel size 3 x 3, stride 1, padding 1, each followed by maxpooling of stride and kernel size 2 x 2. Finally, a max pooling layer of stride 2 and kernel size 2 x 2 is applied, followed by a linear layer. The Residual networks used in this study are ResNet18s (He al. 2015) without batch normalization.
Both ConvNets and ResNets are formed by 4 convolutional stages at which the number of learned feature maps doubles, i.e. the base width of each stage follows the progression [w, 2w, 4w, Swi, with w = 64 denoting a standard ResNet18. To control the number of parameters in each network, the base width w varies from to 64.
Throughout our experiments, augmentations &, of sample (X,: Un) are labelled with their respective (potentially noisy) training larget yn-
Dataset Splits To tune the training hyperparameters of all networks, validation split of 1000 samples drawn uniformly at random from the training split of CIFAR-10 and CIFAR-100.
ConvNet Training Setup The training settings the same for CIFAR-10 and CIFAR-100. All Con- vNets are trained for 4k epochs with SGD with momentum 0.9, fixed learning rate le - 3, batch size 128, and no weight decay, All learned layers are initialized with Pytorch's default weight initialization (version 1.11.0). To stabilize prolonged training in the absence of batch normalization, we use learning rate warmup: starting from a base value of le - the learning rate is linearly increased to le 1e-3 3 during the first 5 epochs of training, after which remains constant at 1
ResNet Training Setup All Res are trained for 4k epochs using Adam with base learning rate le 4, batch size 128, and no weight decay. All learned layers are initialized with Pytorch's default initialization (version 1.11.0). All residual networks are trained with data augmentation, consisting of 4 pixeel random shifts, and random horizontal flips.
Computational Resources Our experiments are conducted on local cluster equipped with NVIDIA Tesla A100s with 40GB onboard memory. For each dataset and architecture, we train 64 different networks for 4000 epochs with 3 different seeds. The total time for computing our experiments, excluding training networks and hyperparameter finetuning, amounts to approximately 6 GPU years. Furthermore, computing our statistics requires evaluating per-sample Jacobians for each training point and corresponding augmenta- tions, for increasing volumes around each point. For cach training setting, this was performed for 72 model checkpoints collected during training, to produce the healmaps in Figures 5
13 and 14
C Tangent Hessian Computation
To estimate the tangent Hessian norm at a point x,, through Equation 5 we approximate the tangent space the data manifold local to x, by using a set of random weak augmentations of x,. To guarantce that all augmentations x, + ums as well the displacements x, + oun lie on the data manifold, we use weak colour augmentations follows.
17

---

Published in Transactions on Machine Learning Research (04/2023)
20
25
30
Color augmentations
Erased SVD values
Figure 8: (Teft) Visualization of random colour augmentations used to estimate the tangent Hessian norm. Each row represents a set of random augmentation, with the first image per-row showing the corresponding base sample. (Right) Each row represents SVD augmentations of increasing strength. Also in this case, the first column represents the base sample used to generate the corresponding augmentations each row.
For each sample Xn, we apply in random order the following photometrie transformations:
random brightness transformation in the range [0.9, 1.1), with 1. denoting the identity transforma- tion.
random contrast transformation in [0.9, 1.1], with 1. denoting the identity transformation.
random saturation transformation in [0.9, 1.1], with 1. denoting the identity transformation.
random hue transformation in 0.05, 0.05), with 0. denoting the identity transformation.
Furthermore, a step size
is used for computing the finite differences Equation 5
augmentations
X, Um are sampled for each point. All randomness is controlled to ensure reprodueibility. Figure S (left)
shows a visualization of the colour augmentations used.
Geodesic Paths Generation
In this section, we provide pseudocode for the algorithm used for generating geodesic paths, used for Monte Carlo integration. Lot Xo E Rd denote a training point, which we use as the starting point of goodesic paths up emanating from Xg. Let. To Rd Rd denote family of smooth transformations (data augmentation), dependent
parameter controlling the magnitude and direction of the transformation (e.g. radial direction and displacement for pixel shifts). Let S (s!....,st gK} denote a sequence of parameters for the family Tg each with strength S = ||s* for k = 1,..., K, such that 51 .... < SK Then, Algorithm ] returns a geodesic path म [0. [ + Ræd based at Xo. iee. (0) = Xo, which is anchored to the data manifold local to Xy by a sequence of augmentations of increasing strength, for I...,,..
Particularly, Algorithm D can be applied times to generate paths яº emanating from X Finally, by integrating metrics of interest (e.g. Jacobian and tangent Hessian norms) along each path ny. obtain
18

---

Published in Transactions on Machine Learning Research (04/2023)
Algorithm Generate geodesic path emanating from training point
function GEODESIC PXTH(Xo, Tax S {ss ....,.k))
P {xo} for s E S do
Set of on-manifold points.
sample ~ g
> Sample augmentatiin of strength S = |||||2.
5:
= (x(o)) P PU 7: end for return P
> Generate weak data augmentation.
6:
Set data augmentations forming a path n, with points sorted by distance from Xo.
end function
estimates of sharpness of the loss along each path, which we use as Monte Carlo samples in Equation 7 for estimating volume-sharpness. We recall that the size of the volume considered controlled by the maximum augmentation strength 5* used for generating weak augmentations, which is proportional to the distance travelled away from X6 in input space.
SVD Augmentation E
The SVD augmentation method presented in section 3.4 allows for generating images that lie in close prox- imity to the base sample X,- Figure shows an illustration of the original image (first column) and several augmented images, as the augmentation strength (number of erased singular values) increases. Figure g shows the average (over the CIFAR-10 training set) Euclidean distance of augmented samples from their respective base sample, as well as the length of the polygonal path formed by connecting augmentations of increasing strength. We note that for k 30, in expectation, augmentations lie in close proximity to the original base sample in Euclidean space.
35
sample
Path length
30
25
L22 distance
20
15
10
5-
0-
20
Augmentation strength
Figure Average L2 distance from the base samples, for augmentations of increasing strength.
F Extended Related Works
In this section, we extend the related work discussion of section 2 to contextualize our findings in relationship to linear models.
In linear regression, the model-wist wise double descent phenomenon has been studied terms of harmless inter- polation (Muthukumar al., 2020) or berign overfitting of noisy data (Bartlett et al.) 2020). by controlling the number of input features 0 considered. Particularly, for the least squares solution to noisy linear
19

---

Published in Transactions on Machine Learning Research (04/2023)
regression problem with random imput and features, the impact of noise generalization is mitigated by the abundance of weak features (Belkin et al. 2020 In this context, interpolation is studied for data whose population is described by noisy observations from a linear ground truth model. In the following, we delineate the main differences between lincar regression and the experimental setting considered in our study.
We begin by noting that, since the model function of linear models has zero curvature (both w.r.t. model input and parameters 0), the only source of nonlinearity and curvature in linear regression is the error function (MSE). To see this, let f(x 0) 0"x denote linear regression model, estimated by minimizing
the mean squared error C x, v) 2N Σ (x n 8) -va)² where Yn is noisy target Vn. Then, the error
function & has constant curvature H = 0x0x r ll2 ||00 independent of x.
In contrast, we study the case of nonlinear classification problems and nonlinear models, which have notable differences from the linear case. First, there is no a priori closed form solution of the learning problem, thus providing relevance empirical studies. Second, curvature of the model function is non-constant, and the lunction may oscillate arbitrarily outside of the training data (this is known the Runge phenomenon). Third, studies that rely exclusively the test error suggest that interpolation is harmless also in overpa- rameterized nonlinear models. Finally, the model function of convolutional architectures is independent of input-data dimensionality, and the relationship between complexity of the model function and its underlying parameterization is therefore implicit.
this setting, we experimentally show that, in the interpolating regime, (1) curvature at training points depends monoooniccalll on model size; (2) oscillations occur especially for small interpolating models, which are worst affected by noise; (3) large models achieve low-curvature interpolation of both clean and noisy samples (in contrast with the polynomial intuition), and such property is observed over large volumes (non- zero measure) around each training point (in contrast with the Runge phenomenon, thus providing evidence of implicit regularization); (4) Interpolation of noise impacts generalization even for large models (contrary to the overparameterized linear regression case): (5) Double descent observed for input space curvature occurs even when fitting 100% noisy data, more clearly pinpointing properties that are consistently promoted by overparameteriiation in deep nonlinear networks.
Our methodology enables the study of sharpness of fit of training data for nonlinear models, providing comparative study of the regularity with which different parameterizations achieve interpolation and (in some cases) generalization.
G Additional Experiments
G.1 Transformers
500
Iwit14 de.en
iesk14, wmt14_en_r
400
Loss|acobias Jacobian norm
0.025
0.600
Enor
200
8,015
Loss
0.005
1.0
200
300
600
300
Embedding Dimension
Embedding Dimension
(b)
Figure 10: a) Double descent of the test error for transformers trained translation tasks, as the embed- ding dimension and model width vary, b) Average Jacobian norm.
20

---

Published in Transactions on Machine Learning Research (04/2023)
We consider multi-head attention- Transformers (Vawwani et al. 2017] for neural machine translation tasks. We vary model size by controlling the embedding dimension de, well as the width h of all fully connected layers, which we set to h = 4d, following the architecture described in Vaswani et al. (017)). We train the transformer networks on the 1MT'14 En-Fr task (Macháčok & Bojar 2014), as well as ISWLT'14 De-En (Cettolo al. 2012). The training set of WMT' reduced by randomly sampling 200k sentences, fixed for all models. The networks are trained for 80k gradient steps, to optimize per-token perplexity, with 10% label smoothing, and no dropout, gradient clipping or weight decay.
For both datasets, Figure 10a shows the double descent curve for the test error for both datasets considered. Figure 10b extends our main result beyond vision models, showing that loss sharpness at each training point, as measured by the Jacobian norm, follows double descent for the test error.
G.2 ConvNets
Figures 11 and 13 summarize our main findings with healmaps showing modelwise and epochwise trends for the test error, train loss, as well as our sharpness metrics, individually computed over the clean and noisy subsets of CIFAR-10.
crossentropy
cressentrosy
Test Eror
-29
10
25
2.
15
-0.6
110
10-
10
-5
-04
30 Width
30
Width
Width
Figure 11: Test error (left), crossentropy loss over cleanly-labelled training samples (middle) and corrupted training samples (right) over epochs (y-axis) for different model sizes (x-axis), for ConvNets on CIFAR-10.
G.3 ResNets
Figures 12 and 14 present heatmaps showing modelwise and epochwise trends for the test error, train loss, as well as our sharpness metrics, individually computed over the clean and noisy subsets of CIFAR-10.
crassentropy Clean
cressentrosy corrupted
Enar
335
1000
2.5
Epoctts
100
-05
100
15
-0.4
-03
10-
10-
05
02
:0
30
do
Figure 12: Test error (left), crossentropy loss over cleanly-labelled training samples (middle) and corrupted training samples (right) over epochs (y-axis) for different model sizes (x-axis), for ResNets on CIFAR-10 10.
21

---

Published in Transactions on Machine Learning Research (04/2023)
crossentropy
crossentropy
-.00
1000
1000
335
3.0
-1.5
Epochs
2.5
100
-1.0
100
2.0
1.5
005
110
10
10
0.5
jacobian
jacobian
-10
25
20
-6
Epocits
100
100
-4
-2
-5
10
10-
tangent hessian
tangent hessian
1000
-12
1000
-10
Epochs
100
100
-6
-10
10
-2
10-
5
1-
1
15
30
45
60
15
30
60
Base Width
Base Width
Figure 13: (Left column) Metrics evaluated on the training set without Monte Carlo integration on ConvNets. (Right column) Monte Carlo integration over neighborhood with paths consisting of 7 augmentations.
22

---

Published in Transactions on Machine Learning Research (04/2023)
crossentropy
crossentropy
-.
-2.
1000
2.0
1.8
1.5
Epochs
166
Epochs
100-
-1.0
10-
10-
0088
1-
1-
jacobian
jacobian
1000
1000
8
Epochs
Epochts
6
100
10-
-0.5
10-
tangent hessian
tangent hesslan
-1
1000
-7
1000
10
-6
Epachs
5
Epochs
100
6
4
10-
-2
1
30
80
Figure 14: (Left column) Metrics evaluated on the training set without Monte Carlo integration on Nets. (Right column) Monte Carlo integration over a neighborhood with paths consisting of 7 augmentations.
23


Reviews:
[{"review_id": "rVvCph9nQg", "reviewer": "Reviewer_egGL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly motivated and relatively easy to follow. Empirical results are rich and are performed under relevant and interesting settings. The following questions might be relevant.\n\n1). Is 20% label corruption needed to study the smoothness of interpolation? Or changing different levels of label corruption, can we find different conclusions? As a sanity check, if the labels are totally corrupted, does overparameterized networks still produce smooth interpolations?\n\n2). How accurate is the approximation to the tangent hessian estimation? As we know, data are high dimensional and in order to approximate a high-dimensional integral, exponentially many (exponent depending on data dimension, also known as the curse of dimensionality) samples are needed.\n\n3). The finding of smooth interpolation contradicts previous intuition. Can authors comment on the cause of smooth interpolation? Is it a consequence of algorithmic regularization in SGD or is it an unique feature of large neural networks? The paper is expository yet a bit weak in explaining the thinking behind the smooth interpolation phenomenon. The paper should be stronger with more discussions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides empirical study of the landscape smoothness in the overparameterized regime of neural networks. Through computing the Jacobian and tangent hessian at training data points, the paper finds that after reaching the interpolation threshold, neural networks tend to smoothly interpolate data, rather than forming spikes at training data points. This discovery enhances the understanding of neural networks' behavior in the practical setting (overparameterized). The reviewer believes this is a timely study and interesting discovery regarding overparameterized neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest the authors to read through the manuscript once more to clear a few typos and grammatic issues.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RYWfR6oc8f", "reviewer": "Reviewer_5jmG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n1. The novelty is good, studying the effect of overparameterization and its relationship to the generalization via characterizing the input-space smoothness is a nice direction.\n2. The finding is also interesting: the overparameterization helps reduce the sharpness of the input space.\n3. The idea of characterizing the smoothness via estimating the geodesic paths is also interesting and potentially useful.\n\nWeakness:\n\n1. Lack of comparison. It seems that the authors try to argue that the input-space smoothness could be better than the parameter-space smoothness. To better explain this, it would be better to draw similar plots as Figures 2&3 in terms of Jacobian/Tangent Hessian metrics for model parameters. Then we are able to see whether overparameterization can also encourage the parameter-space smoothness.\n\n2. The experiment setup is not clearly presented. For instance, the authors may need to explain what's the number of augmentations in Figure 2. I can understand that this will affect the estimations of jacobian and Tangent Hessian, but why the crossentropy will also be different, do you use augmented data to calculate the loss? Or do you use the augmented data to train the model?\n\n3. In order to show the double descent, it would be good to also include the test error/loss.\n\n4. It seems that the sharpness will be largely different if using different numbers of augmentations. However, this is confusing to me as it seems that using different numbers of augmentations will also affect the estimation of sharpness, which implies they should be at least in the same order since the true sharpness of the loss does not depend on the estimating approaches.\n\n5. Regarding Figure 7, the authors claim that the double descent for the corrupted cannot be observed. However, I actually do not see the difference between the trends for the clean and corrupted classes. Could you clarify this with more details?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper performs an empirical study of the smoothness of the loss landscape of DNNs in the input space. The key findings include:\n\n(1) overparameterization improves the input-space smoothness\n(2) the double descent phenomenon can be observed with respect to both training epochs and model sizes.\n\nOverall, this paper highlights the importance of studying the smoothness of the input-space to gain a better understanding of the implicit regularization effect of overparameterization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness section.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hZllFEulfd", "reviewer": "Reviewer_MGoA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n[1] This paper clearly states the setups and observations of the experiments.\n\n[2] This paper reveals a difference between clean data and noisy data (Section 4.5).\n\nWeakness:\n\n[1] The writing of this paper needs improvement.\n\n  (1.1) The title \"Deep Double Descent via Smooth Interpolation\" from my understanding means that the paper developed a method of smooth interpolation and it caused the double descent phenomenon. This is different from the aim of this paper.\n\n  (1.2) From the sentences in the abstract, \"Common intuition...\" and \"While small interpolating ... in contrast to existing intuition\", the authors study the smoothness in clean and corrupted samples and have some observations different from the existing literature. However, the authors put a lot of effort in introducing double descent phenomenon, e.g., \"Our work presents an empirical study of deep double descent...\" in Section 2. It is not clear why double descent is helpful to achieve the goal in the abstract.\n\nI would suggest the authors to clarify what is the aim of this paper and how does double decent helps the understanding in this paper. I started to get confused when reading the introduction section.\n\n[2] From the paper\n\nBelkin, Mikhail, Daniel Hsu, and Ji Xu. \"Two models of double descent for weak features.\" SIAM Journal on Mathematics of Data Science 2.4 (2020): 1167-1180.\n\nmy understanding is that, the model they considered have already implies that over-parameterized models some times smoothly interpolates all training data (including both noisy and clean). Although Belkin's work is on simple models rather than neural networks, it is not surprising that neural networks also share similar properties. I would suggest the authors to compare with this work.\n\n[3] The numerical experiments use only CIFAR-10 and CIFAR-100. Since this paper only conducts empirical studies, using only these two datasets is not sufficient. In addition, the neural networks used in this paper are only ConvNet and ResNet, which is not sufficient.\n\n[4] Some descriptions in the paper are not rigorous. For example, on Page 9, it is mentioned that \"deep networks are biased towards learning training samples in approximately the same order\". Please provide more concrete definitions of \"bias\" and \"order\".\n\n[5] Some descriptions are not accurate. For example, in the abstract, it mentions that \"Our findings show that loss sharpness in the input space follows both model- and epoch-wise double descent, with worse peaks observed around noisy labels\". However, from Figure 7, noisy data does not have double descent.\n\n\nOther issues:\n[1] Is J(x,y) a vector or a matrix? If $L_\\theta$ outputs a scalar loss, then how can we have the $||\\cdot||_F$ norm of a vector? Please also double check if the words \"Jacobian\" and \"Hessian\" are used properly.\n\n[2] Please adjust the layout of the paper. For example, Figure 2 appears on Page 6, but only until the end of Page 7 Figure 2 is mentioned.\n\n[3] Please report the computation time for the proposed method in the experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a way to calculate smoothness of neural networks, and  conducts empirical experiments to study the relationship among over-parameterization, smoothness, and testing performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please polish the paper to make it more clear, comparing with Belkin's work mentioned in the weakness part, and add some more experiments using other data sets.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, a method for computing the smoothness of neural networks is proposed and empirical experiments are conducted to investigate the relationship between overparameterization, smoothness, and test performance. The paper finds that the investigated neural networks tend to interpolate data smoothly after reaching the interpolation threshold instead of forming spikes at the training data points. This discovery improves the understanding of overparametrized neural networks and can be seen as a way to extend the existing theoretical analysis on the double descent phenomenon to neural networks.

The reviewers cautiously note that the numerical study may not be comprehensive enough to make generalizable statements for other networks/datasets. In addition, the metrics measurement methods chosen may have an impact on the transferability of the results to other applications. Despite these reservations, the reviewers believe that this is a timely study that meets the acceptance criteria of the TMLR.
Decision: Accept (poster)