Learning sparse features can lead to overfitting in neural networks
Authors: Leonardo Petrini
Francesco Cagnetta
Eric Vanden-Eijnden
Matthieu Wyart
Venue: NeurIPS 2022 Accept (2022)
Keywords: feature learning
kernel methods
neural networks
overfitting

Abstract: It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data. Yet, understanding when and how this feature learning improves performance remains a challenge: for example, it is beneficial for modern architectures trained to classify images, whereas it is detrimental for fully-connected networks trained for the same task on the same data. Here we propose an explanation for this puzzle, by showing that feature learning can perform worse than lazy training (via random feature kernel or the NTK) as the former can lead to a sparser neural representation. Although sparsity is known to be essential for learning anisotropic data, it is detrimental when the target function is constant or smooth along certain directions of input space. We illustrate this phenomenon in two settings: (i) regression of Gaussian random functions on the $d$-dimensional unit sphere and  (ii) classification of benchmark datasets of images. For (i), we compute the scaling of the generalization error with number of training points, and show that methods that do not learn features generalize better, even when the dimension of the input space is large. For (ii), we show empirically that learning features can indeed lead to sparse and thereby less smooth representations of the image predictors. This fact is plausibly responsible for deteriorating the performance, which is known to be correlated with smoothness along diffeomorphisms.

Learning sparse features can lead to overfitting in neural networks
Leonardo Petrini
Francesco Cagnetta" Institute of Physics
Institute of Physics
École Polytechnique Fédérale de Lausanne leonardo petrini @epf1.ch ch
École Polytechnique Fédérale de Lausanne
francesco wwt.ssco..caaeeteepp....c.ccc ch
Eric Vanden- Eijeen Courant Institute of Mathematical Sciences New York University eve20cims. nyu. edu
Matthieu Wyart
Institute of Physics
École Polytechnique Fédérale de Lausanne matthieu. wttpchiuuuyaar@epfl....
Abstract
It is widely believed that the success of deep networks lies in their ability to learn meaningful representation of the features of the data. Yet, understanding when and how this feature learning improves performance remains challenge: for example, it is beneficial for modern architectures trained to classify images, whereas it is detrimental for fully-connected networks trained on the same data. Here propose an explanation for this puzzle, by showing that feature learning can perform worse than lazy training (via random feature kernel the NTK) the former can lead to a sparser neural representation. Although sparsity is known to be essential for learning anisotropic data, detrimental when the target function constant smooth along certain directions of input space. We illustrate this phenomenon in two settings: (i) regression of Gaussian random functions on the d-dimensional unit sphere and (ii) classification of benchmark datasets of images. For (i), we compute the scaling of the generalization error with the number of training points and show that methods that do not learn features generalize better, even when the dimension the input space is large. For (ii), we show empirically that learning features can indeed lead to sparse and thereby less smooth representations of the image predictors. This fact is plausibly responsible for deteriorating the performance, which is known to be correlated with smoothness along diffcomorphisms.
Introduction
Neural networks are responsible for technological revolution in variety of machine learning tasks. Many such tasks require learning functions of high-dimensional inputs from finite set of examples, thus should be generically hard due to the curse of dimensionality [1, 2]: the exponent that controls the scaling of the generalization error with the number of training examples inversely proportional to the input dimension d. For instance, for standard image classification tasks with d ranging in 103 10", such exponent should be practically vanishing, contrary what observed in practice [3]. In this respect, understanding the success of neural networks is still an open question. A popular explanation is that, during training, neurons adapt to features in the data that are relevant for the task [4], effectively reducing the input dimension and making the problem tractable [5, 6, 7]. However, understanding quantitatively this intuition is true and how depends on the structure of the task remains challenge.
"Equal contribution (a coin was flipped).
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

Overlitting in Feature Learning
MXIST
10-1-
F-MNIST
CIFARIO
test ezzot,
10 10-1-
6 10-
Lazy
8 10
Lazy
Fodi/me
10-1
Lazy
Haatize
Staincet stoe,
traizeet eioce
Izaizzel aloe,
Figure 1: Feature vs. Lazy in image classification. Generalization error function of the training- set size n for infinite-width fully-connected networks (FCNs) trained in the feature (blue) and lazy regime (orange). In the latter case the limit is taken exactly by training an SVC algorithm with the analytical NTK [23]] In the former case, the infinite-width limit can be accurately approximated for these datasets by considering very wide nets (H 109), and performing ensemble averaging on different initial conditions of the parameters as shown in [24, 25]. Panels correspond to different benchmark image datasets [26, 27, 28]. Results are averaged over 10 different initializations of the networks and datasets.
Recently much progress was made in characterizing the conditions which lead to features learning, in the overparameterized setting where networks generally perform best. When the initialization scale of the network parameters is large [8] one encounters the lazy training regime, where neural networks behave as kernel methods [9, 10] (coined Neural Tangent Kernel or NTK) and features are not learned. By contrast, when the initialization scale is small, a feature learning regime found [11, 12, 13] where the network parameters evolve significantly during training. This limit is much less understood apart from very simple architectures, where can be shown to lead to sparse representations where a limited number of neurons are active after training [14]. Such sparse representations can also be obtained by regularizing the weights during training [2. 15].
In terms of performance, most theoretical works have focused on fully-connected networks. For these architectures, feature learning was shown significantly outperform lazy training [16, 17, 18, 19, 11] for certain tasks, including approximating a function which depends only on a subset or a linear combination of the input variables. However, when such primitive networks are trained on image datasets, learning features is detrimental [20, 21], as illustrated in Fig. I (see [19, Fig. 3] for the analogous plot in the case of target function depending on just one of the input variables, where learning features is beneficial). A similar result was observed in simple models of data [22]. These facts are unexplained, yet central to understanding the implicit bias of the feature learning regime.
1.1 Our contribution
Our main contribution is to provide an account of the drawbacks of learning sparse representations based on the following set of ideas. Consider, for concreteness, image classification problem: (i) images class varies little along smooth deformations of the image; (ii) due to that, tasks like image classification require continuous distribution of neurons be represented; (ii)) thus, requiring sparsity can be detrimental for performance. We build our argument as follows.
In order to find a quantitative description of the phenomenon, we start from the problem of regression of random target function of controlled smoothness on the d-dimensional unit sphere, and study the property of the minimizers of the empirical loss with n observations, both in the lazy and the feature learning regimes. More specifically, we consider two extreme limits-the NTK limit and mean-field field limit-as representatives of lazy and feature regimes, respectively (section 2). Both these limits admit simple formulation that allows to predict generalization performances. In particular, our results on feature learning rely on solutions having an atomic support. This property can be justified for one-hidden-layer neural networks with ReLU activations and weight decay. Yet, we also find such a sparsity empirically using gradient descent in the absence of regularization, if weights are initialized to be small enough.
We find that lazy training leads to smoother predictors than feature learning. As a result, lazy training outperforms feature learning when the larget function is also sufficiently smooth. Otherwise, the performances of the two methods are comparable, in the sense that they display the same asymptotic decay of generalization error with the number of training
2

---

Overlitting Feature Learning
examples. Our predictions are obtained from asymptotic arguments that we systematically back up with numerical studies.
For image datasets, it is believed that diffeomorphisms images are key transformations along which the predictor function should only mildly vary obtain good performance [29]. From the results above, a natural explanation as to why lazy beats feature for fully connected networks is that leads to predictors with smaller variations along diffeomorphisms, We confirm that this is indeed the case empirically on benchmark datasets.
Numerical experiments are performed in PyTorch [30], and the code for reproducing experiments is available online at github. hettau/ccoom/ssssspplleeeessssso
1.2 Related Work
The property that training ReLU networks the feature regime leads a sparse representation was observed empirically [31]. This property can be justified for ome-hideen-laye networks by casting training as LI minimization problem [32, 2]. then using representer theorem [33, 15, 34]. This analogous to what is commonly done in predictive sparse coding [35, 36, 38].
Many works have investigated the benefits of learning sparse representations in neural networks. [2, 16, 17, 18, 19, 39, 40J study cases in which the true function only depends on a linear subspace of input space, and show that feature learning profitably capture such property. Even for more general problems, sparse representations of the data might emerge naturally during deep network training-4 phenomenon coined neural collapse [41]. Similar sparsification phenomena, for instance, have been found to allow for learning convolutional layers from scratch [42, 43]. Our work builds on this body of literature by pointing out that learning sparse features can be detrimental, if the task does not allow for it.
There is currently general framework to predict rigorously the learning curve exponent / defined as (n2) = Onn-9) for kernels. Some of our asymptotic arguments can be obtained by other approximations, such as assuming that data points lie on lattice in R2 [44], or by using the non- rigorous replica method of statistical physics [45, 46, 47J. In the case d 2, provide a more explicit mathematical formulation of our results, which leads to analytical results for certain kernels. We systematically back up our predictions with numerical tests d varies.
Finally, in the context of image classification, the connection between performance and 'stability' or smoothness toward small diffeomorphisms of the inputs has been conjectured by [29, 48]. Empirically, a strong correlation between these two quantities was shown to hold across various architectures for real datasets [49]. In that reference, it was found that fully connected networks lose their stability over training: here we show that this effect much less pronounced in the lazy regime.
2 Problem and notation
Task We consider supervised learning scenario with n training points (xx) ) uniformly drawn on the d-dimensional unit sphere $d-1, We assume that the target function f" an isotropic Gaussian random process on gd-1 and control its statistics via the spectrum: by introducing the decomposition of f* into spherical harmonics (see App. A for definitions),
Nu.a
(x)
ΣΣ K20
fh (L) with E [fā O, E [fkeffiee dr
(2.1)
We assume that all the Cę with is odd vanish apart from oj: this is required guarantee that f can be approximated as well as desired with a one-hiddes see-hidden--layee layer ReLU network with no biases, as discussed in App. A. We also assume that the non-zero decay as a power of for A > 1, C₁ ~ -2 - (d-1), The exponent v > 0 controls the (weak) differentiability of f on the sphere (see App. A) and also the statistics of f* in real space:
E (x) ('(1) 0 0 y o ((1 x )
as x
(2.2)
Examples of such target function for d 3 and different values of 14 are reported in Fig. 2.

---

Overlitting Feature Learning
(a): 3, 1
(b): d= 3,4 1/2
(x)
Figure 2: Gaussian random process on the sphere. We show here two samples of the task introduced in section when the target function f"xx) is defined the 3 -immennsional unit sphere. (a) and (b) show samples large and small smoothness coefficient Vt, respectively.
Neural network representation in the feature regime In this regime we aim to approximate the target function f (I) via a one-hidden hidden-layer neural network of width H,
1
(x)
1n x).
(2.3)
h=1
where {0,]h=1 (the features) and {wx ) (the weights) are the network parameters to be optimized, and (z)) denotes the ReLU function, o(x) = max x0, If we assume that {0,, wh wh are independently drawn from a probability measure μ on gd-1 R such that the Radon measure Sg wp(, dw) exists, then as H + 0o,
lim fn = [
fga-1 o(0 (l.x))) (0)
a.e. on gd-1
(2.4)
00000
This is the so-called mean-field limit [11, 12], and it is then natural to determine the optimal y via
7 arg min [
(d)
subject to:
( x. (d> Vi= 1........r n. (2.5)
ga-
In practice, we can approximate this minimization problem by using network with large but finite width, constraining the feature to be on the sphere 00 1, and minimizing the following empirical loss with L1 regularization on the weights,
1 Σ
min (w.,0% 3 1
(x:
wno
wh
(2.6)
H
|0,
This minimization problem leads to (2.5) when H and X + 0. Note that, by homogeneity of ReLU, (2.6) can be shown to be equivalent to imposing a regularization on the L2 norm of all parameters [32, Thm. 10], i.c. the usual weight decay.
To proceed we will make the following assumption about the minimizer y": Assumption 1. The minimizer 7" of (2.5) is unique and atomic, with nAA S n. atoms, i.e. there exists {wf, 0 A such that
W d0r
(2.7)
1=1
The main component of the assumption is the uniqueness of y"; if holds the sparsity of follows from the representer theorem, see e.g. [33]. Both the uniqueness and sparsity of the minimizer can be justified as holding generically using asymptotic arguments involving recasting the L1 minimization problem 2.5 as a linear programming one: these arguments are standard (see e.g. [50]) and are presented in App. B for the reader convenience. In arguments below to deduce the scaling of the generalization error we will mainly use that nı = O(n) we shall confirm this fact numerically even in the absence of regularization, if the weights initialized to be small enough. Notice that from Assumption it follows that the predictor in the feature regime corresponding to the minimizer y takes the following form
FRATURE (x)
o o (0 x).
(2.8)
1=1

---

Overlitting Feature Learning
Neural network representation in the lazy regime. In this regime we approximate the target function (a) via
(x) (x) Σ K NTK (xx
(2.9)
where the weights [9:b- solve
(aa) Σ 9: KNKK (z, xz))
j 1,...,
(2.10)
and KNIK (x y) the Neural Tangent Kernel (NTK) [9]
/ xg
KNIK y)
(o(0-2))) y) + w² y d x) y)) d0 w).
(2.11)
Here Ho is fixed probability distribution which, the NTK training regime [9], is the distribution of the features and weights at initialization. It is well-known [51] that the solution to kernel ridge regression problem can also be expressed via the kernel trick as
NTK
go (gw (0, w a:) wa go w æ)) (0, w.
(2.12)
where go and Iw are the solutions of
min [ 9o
(w, 0) (go(w, 0)1²) (0,00,00
Jzd 3d-1x8
1d-1XR (ww () xx)) + wa, [o 0)0'(0 xx)) duo(O, w) = f'(x)) (2.13)
subject to:
=i= 1,.....,,m
Another lazy limit can be obtained equivalently by training only the weights while keeping the features to their initialization value. This is equivalent to forcing go w) vanish in Eq. 2.13, resulting again in a kernel method. The kernel, this case, is called Random Feature Kernel (K RFFF,, and can be obtained from Eq. 2.11 by setting dug w) (w=odio(()) The minimizer can then be written as in Eq. 2.9 with K NTK replaced by KRFR
3 Asymptotic analysis of generalization
In this section, we characterize the asymptotic decay of the generalization error ε(n) averaged over several realizations of the target function f". Denoting with d the uniform measure on gd-1
/ dr (x) (z)x)-
{(n) E,.
Am +o(n
(3.1)
for some constant Aa which might depend on d but not on n. Both for the lazy (see Eq. 2.9) and feature regimes (see Eq. 2.8) the predictor can be written as sum of O(n) terms:
(nx)
(x)x)= = Σ g)p(2 4,) =-= f (")))(( y)dr(y) j-1
(3.2)
In the feature regime, the 9,'s 's (y₁) coincide with the optimal weights w (features 67), &o with the activation function o. the lazy regime, the are the training points Vy, is the neural tangent or random feature kernel the 9j 's are the weights solving Eq. 2. We have defined the density g" (z) E, gd (ggō(x y₁) as to cast the predictor as convolution the sphere. Therefore. the projections of fr onto spherical harmonics YE, read fī, rkk where Le is the projection of g" (x) and 4ox that of pfx: y). For ReLU neurons one has (as shown in App. A)
LAZY k- (d-
with 1/2 (NTK), 3/2 (RFK), &F
FEATURE
k- 4-1 -3/2
(3.3)

---

Overlitting Feature Learning
lazy
feature
prodictor
prodictor
predistor
training points
trwining points
x/2
1
m 2
angle
angle
Feature vs. Lazy Predictor. Predictor of the lazy (left) and feature (right) regime when learning the constant function on the ring with 8 uniformly-sampled training points.
Figure 3:
Main Result Consider target function f with smoothness exponent 1t defined above, with data lying on gd-1, II' f is learnt with ane-hidden-layee network with ReLU neurons in the regimes specified above, then the generalization error follows ε(n) with:
gLAZY min (2(d - 1) + 4v, 2v1}
with V = [1/2 for NTK,
(3.4a)
d-1 BFEATURE min {(d - 1) + 3, 20
3/22 for RFK,
(3.4b)
d-1
This is our central result. implies that if the target function smooth isotropic Gaussian field (realized for large v1), then lazy beats feature, in the sense that training the network in the lazy regime leads to a better scaling of the generalization performance with the number of training points.
Strategy There is no general framework for rigorous derivation of the generalization error in the ridgeless limit À + predictions such as that of Eq. 3.4 can be obtained by either assuming that training points (for Eq. 3.4a) and neurons (for Eq. 3.4b) lie on a periodic lattice [44], or (for Eq. 3,4a) using the replica method from physics [45] shown in App. first characterizing the form of the predictor for d= (proof in App. C). This property alone allows us to determine the asymptotic scaling of the generalization error. We use to analytically obtain the generalization error in the NTK case with a slightly simplified function y (details in App. D). This calculation motivates a simple ansatz for the form (xz) entering Eq. 3.2 and its projections onto spherical harmonics, which extends naturally to arbitrary dimension. We confirm the predictions resulting from this ansatz systematically in numerical experiments.
Here we follow a different route, by
Properties of the predictor in d = 2 On the unit circle S' all points are identified by polar angle [0, 2T). Hence both target function and estimated predictor are functions of the angle, and all functions of a scalar product are in fact functions of the difference in angle. In particular, introducing (xx)) ((cos(z)).
2x dy gj - - = frx
(z(x)
()) -y),
(3.5)
2T
where we defined
n
9" (x) Σ (2 zz). 5=1
(3.6)
Both for feature regime and NTK limit, the first derivative of ((x)) continuous except for two values of (0 and æ for lazy, / and 7/2 for feature), so that (2x)" " has singular part consisting of two Dirac delta functions.
As result, the sccond derivative of the predictor (f*)" has a singular part consisting of many Dirac deltas. If we denote with (fm)" the regular part, obtained by subtracting all the delta functions, we can show that (see App. C):
Proposition (informal) As n + oo, (f")"" converges to a function having finite second moment, i.e.
lim E,- (z)) const. oc.
(3.7)
=-000

---

Overfitting in Feature Learning
In the large n limit, the predictor displays singular second derivative at (n) points. Proposition 1 implies that outside of these singular points the second derivative is well defined. Thus, as n gets large and the singular points approach each other, the predictor can be approximated by chain of parabolas, as highlighted in Fig. 3 and noticed in [47] for a Laplace kernel. This property alone allows to determine the asymplotic scaling of the error in d = 2. In simple terms, Prop. I follows from du (v)) (x - y). which is guaranteed the convergence of g" to the function satisfying (z) f 2x
under our assumptions on the larget function a detailed proof is given in App. C.
Decay of the error in 2 (sketch) The full calculation is in App. D. Consider a slightly simplified problem where o has a single discontinuity in its derivative, located at x = 0. In this case, (a(( is singular if and only if x: is data point. Consider then the interval x C [2g 2i+-1 1] and set ó, = Fi+1 Fig 1/+1/2 = (#++1 + z2)/2. If the target function is smooth enough (vz > 2), then a Taylor expansion implies |f (1 - f" (x(x+1122) ~ of. Since the distances 6, between adjacent singular points are random variables with mean of order 1/n and finite moments, is straightforward to obtain that {(n) ~ Ef(
(=((=)1/2)) ~ Σi 8, n-1, By contrast if f is not 0,2r0, leading to E(n) n 2", Note that for this asymptotic argument to apply to the feature learning regime, one must ensure that the distribution of the rescaled distance between adjacent singularities nō; has finite fourth moment. This is obvious in the lazy regime, where the 6; 's are controlled by the position of the training points, but not in the feature regime, where the distribution of singular points determined by that of the neuron's s features. Nevertheless, we show that it must be the case in our setup in App. D.
sufficiently smooth (1t S then (f(++1))2) ( ff
Interpretation in terms of spectral bias From the discussion above it is evident that there is a length scale ó of order I/n such that f"(xx is good approximation of f (ax) over scales larger than 5. In terms of Fourier modes?, one has: i) (*)) matches (ki(k) at long wavelengths, i.e. for k & ka ~ 1/n. ii) In addition, since the phases exp(ikz,) become effectively random phases for k >>>h kc, ge E, Bj exp(ikx..) becomes Gaussian random variable with zero mean and fixed variance and thus iii) (k) - 9" (k) (1k) decorrelates from f" for k > ke- Therefore
((n)
Σ Esr I//
( (A))))) 2
Σ Ey [G* ((k)² + Es- [()(()) 1k/2kc
(3.8)
For v4 2. one has E, 03 n lim ʃ (x) n- It follows (see App. E for details) that the sum is dominated by the first term, hence entirely controlled by the Fourier coefficients of (k)) large k. A smoother predictor corresponds to faster decay of f (k) with k. thus a faster decay of the eπrоr with n. Plugging the relevant decays yields ē ~ n for feature regime and lazy regime with the NTK, and n for lazy regime with the RFK (which smoother than the NTK). For V/ S 2, the two terms have comparable magnitude (sce App. E), thus غ ~ n 2v:
Generalization to higher dimensions The argument above can be generalized for any d by replacing Fourier modes with projections onto spherical harmonics. The characteristic distance between training points scales as n 1/((d11), thus k ~ n 11((-1),, Our ansatz is that, as in d= 2: i) for k < kc, the predictor modes coincide with those of the target function, f z fū, (this corresponds to the spectral bias result of kernel methods, stating that the predictor reproduces the first (n) projections of the target in the kernel eigenbasis [45]); ii) For k >> kes Lh, is a sum of uncorrelated terms, thus a Gaussian variable with zero mean and fixed variance; iii) fk. = P decorrelates from fū.e for k k i), ii) and tii) imply that:
Nk. Es- [ 1m.)
N ΣΣ Ey- [GAA,)]] 12 + k- 2n 20--(d-1 (d-1)
(n)
(3.9)
kzke
k2ke
As shown in App. E, from this expression it is straightforward to obtain Eq. 3.4. Notice again that when the target is sufficiently smooth so that the determined by the smoothness of the predictor. In particular, as d > 2, the predictor of feature learning is less smooth than both the NTK and RFK ones, due to the slower decay of the corresponding (Pa.
term dominates, the error is
TThe Fourier transform of a function fxx is indicated by the hat, f(k)).

---

Overlitting in Feature Learning
4 Numerical tests of the theory
We test successfully our predictions by computing the learning curves of both lazy and feature regimes when (i) the target function is constant on the sphere for varying d, see Fig. 4, and (ii) the target Gaussian random field with varying smoothness Vt, as shown in Fig. G.I of App. G. For the lazy regime, we perform kernel regression using the analytical expression of the NTK [52] (see also Eq. A.19). For the feature regime, we find that our predictions hold when having a small regularization, although it takes unreachable times for gradient descent to exactly recover the minimal- norm solution-a a more in-depth discussion can be found in App. G. An example of the atomic distribution of neurons found after training, which contrasts with the initial distribution, is displayed in Fig. 5a, left panel.
Another way to obtain sparse features is to initialize the network with very small weights [14], as proposed in [8]. As in the presence of an infinitesimal weights decay, this scheme also leads to sparse solutions with A (nn) an asymptotic dependence confirmed in Fig. G.3 of App. G. This observation implies that our predictions must apply that casc too, as we confirm in Fig. G.3.
2
3
70
Pat, c 10²
Iod 01707, cox h-
Rest, 3-107
Lazy Eaperiments
10
Laze
10-4
Post, C- Lasy
10-2
Experimanc
Thnoey
Theser
10-5
3p
4.0
3p
2.5
3y 1.75
4.0
10-7
3.0
10 -5
3r.
2.5
10-10-
16'
10a
10 101
10'
10
107
trainset sizo,
trainset sise, n
trainset size, n
Figure 4: Generalization error for constant function (x) 1. Generalization error function of the training set size n for a network trained in the feature regime with LI regularization (bluc) and kernel regression corresponding to the infinite-width lazy regime (orange). Numerical results (full lines) and the exponents predicted by the thcory (dashed) are plotted. Pancls correspond to different input-space dimensions (d 2, 3, 5). Results are averaged over 10 different initializations of the networks and datasets. For = 2 and large n, the gap between experiments and prediction for the feature regime is due to the finite training time Indeed our predictions become more accurate as t increases, as illustrated in the left.
Evidence for overfitting along diffeomorphisms in image datasets
For fully-connected networks, the feature regime is well-adapted to learn anisotropic tasks [16]: if the target function does not depend on a certain linear subspace of input space, e.g. the pixels at the corner of an image, then neurons align perpendicularly to these directions [19]. By contrast, our results highlight a drawback of this regime when the target function is constant or smooth along directions in input space that require continuous distribution of neurons be represented. In such a case, the adaptation of the weights to the training points leads to a predictor with a sparse representation. Such a predictor would be less smooth than in the lazy regime and thus underperform.
Does this view hold for images, and explain why learning their features is detrimental for fully- connected networks? The first positive empirical evidence is that the neurons' distribution of networks trained on image data becomes indeed sparse in the feature regime, as illustrated in Fig. 5a, right, for CIFAR10 [28]. This observation raises the question of which are the directions in input space i) along which the target should vary smoothly, and ii) that are not casily represented by a discrete set of neurons. An example of such directions the input and do not change the image class: the lazy regime predictor is indeed smoother than the feature with respect to translations of the input (see App. H). Yet, these transformations live in space of dimension 2, which is small in comparison with the full dimensionality d of the data and thus may play a negligible role.
global translations, which conserve the norm of
much larger class of transformations believed to have little effect on the target are small diffeomor- phisms [29]. A diffcomorphism 7 acting an image is illustrated in Fig. 5b, which highlights that
8

---

Overlitting Feature Learning
Spherical Task
CIFARIO
4
2
PC2
02
7 4
-0.3
poinis Peature
-2
Feature Lazy
-1.0
Lazy
-1.0
-0.5
0.0
0.5
1.0
2
01
0 0pc1
(a) Features sparsification. Panee: Distribution of neuron's 's feature for the task of learning constant function the sphere in 2D. Arrows represent subset of the network features {0, ) after training the lazy and feature regimes. Training performed on n = data data-points (black dots). 2 Panel: FCN trained on CIFARIO On the axes the first two principal components of the features {0,, ) after training on n = 32 points the feature (blue) and lazy (orange) regimes, Similarly to what observed when learning a constant function, the 0 angular distribution becomes sparse with training in the feature regime.
(b) Example of diffeo- morphism. Sample of a max max- entropy deforma- tion t [49] when applied to a natural image, illus- trating that it does not change the image class for the human brain.
Figure 5: Features sparsification and example of a diffeomorphism.
our brain still perceives the content of the transformed image as in the original onc. Near-invariance of the task these transformations is believed to play a key role in the success of deep learning, and in explaining how neural networks beat the curse of dimensionality [48]. Indeed, if modern architectures can become insensitive to these transformations, then the dimensionality of the problem is considerably reduced. In fact, it was found that the architectures displaying the best performance are precisely those which learn to vary smoothly along such transformations [49].
Small diffcomorphisms are likely the directions we are looking for. To test this hypothesis, follow- ing [49], we characterize the smoothness of function along such diffeomorphisms, relative that of random directions in input space. Specifically, we use the relative sensitivity:
- Eg. ( f
R₁ = (f + 1)) (amn))((a)) (x))]
(5.1)
In the numerator, the average is made over the test set and over an ensemble of diffeomorphisms, reviewed in App. I. The magnitude of the diffeomorphisms is chosen so that each pixel is shifted by one on average. In the denominator, the average runs over the test set and the vectors 1) sampled uniformly on the sphere of radius ||>]| Er,-llrz x||. and this fixes the transformations magnitude.
We measure R2 as function of n for three benchmark datasets of images, shown in Fig. 6. We indeed find that Ry is consistently smaller in the lazy training regime, where features are not learned. Overall, this observation supports the view that lcarning sparse features is detrimental when data present (ncar) invariance to transformations that cannot be represented sparscly by the architecture considered, Fig. supports the idea that for benchmark image datasets -this negative effect overcomes well-known positive effects of learning features, e.g. becoming insensitive to pixels on the edge of images (see App. H for evidence of this effect).
Conclusion
Our central result is that learning sparse features can detrimental if the task presents invariance or smooth variations along transformations that are not adequately captured by the neural network architecture. For fully-connected networks, these transformations can be rotations of the input, but also continuous translations and diffcomorphisms.
9

---

Overlitting Feature Learning
defermatioe sabibo, Ky
10a
MNIST
F-MNIST
CIFARIO
301
x 10P
Feature
2 10
Peature
Pesture
Lany
Lacy
Lozy
10P
107
10
307
10
104
107
107
301
Trainset sibe,
trainent
trnimsrt
Figure 6: Sensitivity to diffeomorphisms vs number of training points. Relative sensitivity of the predictor small diffeomorphisms of the imput images, in the regimes, for varying number of training points n and different image datasets. Smaller values correspond to a smoother predictor, on average. Results computed using the same predictors in Fig. 1.
Our analysis relies on the sparsity of the features learned by shallow fully-connected architecture: even in the infinite width limit, when trained in the feature learning regime such networks behave as (n) neurons. The asymplotic analysis we perform for random Gaussian fields on the sphere leads to predictions for the learning curve exponent / in different training regimes, which we verify, Such kind of results is scarce in the literature.
Note that our analysis focuses on ReLU neurons because (i) these are very often used in practice and (ii) in that case, B will depend on the training regime, allowing for stringent numerical tests. If smooth activations (e.g. softplus) are considered, we expect that learning features will still be detrimental for generalization. Yet, the difference will not appear in the exponent B3, but in other aspects of the learning curves (including numerical coefficients and pre-asymptotic effects) that are harder to predict.
Most fundamentally, our results underline that the success of feature learning for modern architectures still lacks a sufficient explanation. Indeed, most of the theoretical studies that previously emphasized the benefits of learning features have been considering fully-connected networks, for which learning features can be in practice drawback. It is tempting argue that in modern architectures, learning features is not at a disadvantage because smoothness along diffcomorphisms can be enforced from the start-due due the locally connected, convolutional, and pooling layers [53, 29]. Yet the best architectures often do not perform pooling and are not stable toward diffeomorphisms at initialization. During training, learning features leads to more stable and smoother solutions along diffeomorphisms [54, 49). Understanding why building sparse features enhances stability in these architectures may ultimately explain the magical feat of deep CNNs: learning tasks in high dimensions.
Acknowledgements
We thank Lénaïc Chizat, Antonio Sclocchi, and Umberto M. Tomasini for helpful discussions. The work of MW is supported by grant from the Simons Foundation (#454953) and from the NSF under Grant No. 200021-165509. The work of EVE is supported by the National Science Foundation under awards DMR- 142007,, DMS-201255 2510510, and DMS-213422 by the Simons Collaboration Wave Turbulence, Grant No. 617006, and by a Vannevar Bush Faculty Fellowship.
References
[1] Ulrike von Luxburg and Olivier Bousquet. Distance-based classification with lipschitz functions. Journal of Machine Learning Research, 5(Jun):669-695, 2004.
[2] Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-66) 2017.
[3] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 1tXiv1111..004000 2017.
[4] Quoc V Le. Building high-level features using large scale unsupervised learning. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 8595-8598, IEEE, 2013.
10

---

Overlitting Feature Learning
[5] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information, arXiv preprint urXivv:/03..00mm I0, 2017.
[6] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems, pages 611 1-6122, 2019.
[7] Stefano Recanatesi, Matthew Farrell, Madhu Advani, Timothy Moore, Guillaume Lajoie, and Eric Shea-Brown Dimensionality compression and expansion in deep neural networks. arXiv preprint arXiv: 1906: 00443, 2019.
[8] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. In Advances in Neural Information Processing Systems, pages 2937-2947, 2019.
[9] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS' 18, pages 8580-8589, USA, 2018. Curran Associates Inc.
[10] Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2019.
[11] Grant M Rotskoff and Eric Vanden-Eijnden. Neural networks as interacting particle systems: Asymptotic convexity the loss landscape and universal scaling of the approximation error. arXiv preprini arXiv: 1805.80055115 2018.
[12] Song Mci, Andrea Montanari. and Phan Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E765-
E7671, August 2018. Publisher: National Academy of Sciences Section: PNAS Plus.
[13] Justin Sirignano and Konstantinos Spiliopoulos. Mean Field Analysis of Neural Networks: A
Law of Large Numbers. SIAM Journal on Applied Mahhmatics, 80(2):725-752, January 2020. Publisher: Society for Industrial and Applied Mathematics.
[14] Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and Rich Regimes in Overparametrized Models. In Conference on Learning Theory, pages 3635-3673. PMLR, July 2020. ISSN: 2640-3498.
[15] Jaume de Dios and Joan Bruna. On sparsity in overparametrised shallow relu networks. arXiv preprint arXiv:2006. 10225, 2020.
[16] Lénaïc Chizat and Francis Bach. Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss. In Conference on Learning Theory, pages 1305-1338.
PMLR, July 2020. ISSN: 2640-3498.
[17] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When Do Neural Networks Outperform Kernel Methods? Advances in Neural Information Processing Systems,
33, 2020.
[18] Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdcborová. Classifying high- dimensional gaussian mixtures: Where kernel methods fail and ncural networks succeed. arXiv preprint arXiv:2102.11742 2021.
[19] Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in ncural networks. Journal of Statistical Mechanics: Theory and Experiment, 2021 (4):044004 April 2021. Publisher: IOP Publishing.
[20] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11): 133001, November 2020. Publisher: IOP Publishing.
[21] Jachoon Lee, Samuel S Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein Finite versus infinite neural networks: an empirical study. arXiv preprint arXiv:2007. /5801, 2020.
[22] Guillermo Orüzz-imééez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. What can linearized neural networks actually say about generalization? Advances in Neural Information Processing Systems, 34, 2021.

---

Overlitting in Feature Learning
[23] Yilan Chen, Wei Huang, Lam M. Nguyen, and Tsui-Wei Weng. On the Equivalence between Neural Network and Support Vector Machine, arXiv:21 1021111.06663 [cs, math, stat], November 2021. arXiv: 2111.06063.
[24]
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d' Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2):03301,, February 2020. Publisher: IOP Publishing.
[25] Mario Geiger, Leonardo Petrini, and Matthieu Wyart. Landscape and training regimes in deep learning. Physics Reports, 924:1-18, August 2021.
[26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2277--2324, 1998.
[27] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST; a Novel Image Dataset for Benchmarking Machine Learning Algorithms. 7xXiv:1708.07747 [cs, stat], September 2017. arXiv: 777.07747.
[28] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[29] Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions
on pattem analysis and machine intelligence, 3588):1872-1886. 2013.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Porformance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[31] Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. arXiv preprint arXivv:8033 083667, 2018
[32] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. Conference on Learning Theory, pages 1376-1401. PMLR, 2015.
[33] Claire Boyer, Antonin Chambolle, Yohann De Castro, Vincent Duval, Frédéric De Gournay, and Pierre Weiss. On Representer Theorems and Convex Regularization. SIAM Journal on Optimization, 29(2):120011288 May 2019.
[34] Lenaic Chizat. Sparse optimization measures with over-parameterized gradient descent. Mathematical Programming, pages 1-46, 2021.
[35] Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, (6683))))60066669 June 1996. Number:
6583 Publisher: Nature Publishing Group.
[36] Julien Mairal, Jean Ponce, Guillermo Sapiro, Andrew Zisserman, and Francis Bach. Supervised Dictionary Learning. In Advances in Neural Information Processing Systems, volume 21, Curran Associates, Inc., 2008
[37] Nishant A Mehta and Alexander G Gray. Sparsity-Based Generalization Bounds for Predictive Sparse Coding. page 9.
[38] Jeremias Sulam, Ramchandran Muthukumar, and Raman Arora. Adversarial Robustness of Supervised Sparse Coding, January 2021. arXiv:2010. 12088 [cs, stat].
[39] Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for derstanding neural networks. In Advances in Neural Information Processing Systems, pages 6598-6605, 2019.
[40] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of lazy training of two-layers neural network. In Advances in Neural Information Processing Systems, pages 9111-9121. 2019.
[41] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training, Proceedings of the National Academy of Sciences, 1 117(40):4452--4666 2020.
[42] Behnam Neyshabur. Towards Learning Convolutions from Scratch, arXiv:2007.1363 [cs, stat]. July 2020. arXiv: 2307.13657.
12

---

Overlitting Feature Learning
[43]
Alessandro Ingrosso and Sebastian Goldt. Data-driven emergence of convolutional structure in neural networks. Proceedings of the National Academy of Sciences, 119(40)::221185411) 19,
2022.
[44] Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus reacher-student paradigm. Journal of Statistical Mechanics:
Theory and Experiment, 2020(12): December 2020. Publisher: IOP Publishing.
[45] Blake Bordelon, Abdulkadir Canatar, and Cengiz. Pehlevan. Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks. In International Conference Machine Learning, pages 1024-1034. PMLR, November 2020. ISSN: 2640-3498.
[46] Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34, 2021.
[47] Umberto M Tomasini, Antonio Sclocchi, and Matthieu Wyart. Failure and success of the spectral bias prediction for kernel ridge regression: the case of low-dimensional data. arXiv preprint arXiv: 033488 2022.
[48] Stéphane Mallat. Understanding deep convolutional networks. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065):2015020,3 2016.
[49] Leonardo Petrini, Alessandro Favero, Mario Geiger, and Matthieu Wyart. Relative stability
toward diffeomorphisms indicates performance in deep nets, In Advances in Neural Information Processing Systems, volume 34, pages 8727-8723 8739. Curran Associates, Inc., 2021.
[50] Scott Shaobing Chen, David L. Donoho, and Michael A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20(1):33-61, 1998.
[51] Bernhard Scholkopf and Alexander Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2001.
[52] Youngmin Cho and Lawrence K. Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems 22, pages 342-350. Curran Associates, Inc., 2009.
[53] Alberto Bietti and Julien Mairal. Group invariance, stability to deformations, and complexity of deep convolutional representations. The Journal of Machine Learning Reseurch, 20(1):876-224, 2019.
[54] Avraham Ruderman, Neil C. Rabinowitz, Ari S. Morcos, and Daniel Zoran. Pooling is neither necessary nor sufficient for appropriate deformation stability in CNNs. arXiv:1804, 1804.04488488 [cs, stat], May 2018. arXiv: 1804.04438.
[55] Alex Smola, Zoltán Ovári, and Robert C. Williamson. Regularization with dot-product kernels. Advances in neural information processing systems, 13, 2000.
[56] Kendall Atkinson and Weimin Han. Spherical harmonics and approximations the unit sphere: an introduction, volume 2044. Springer Science & Business Media, 2012.
[57] Costas Efthimiou and Christopher Frye. Spherical harmonics in p dimensions. World Scientific, 2014.
[58] Alberto Bietti and Francis Bach. Deep Equals Shallow for ReLU Networks in Kernel Regimes. arX9v:2009. 14397 [cs, stat], August 2021. arXiv: 1439.14397.
[59] Francis Bach. Learning Theory from First Principles. In preparation, 2022.
13

---

Overlitting Feature Learning
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Yes, in the conclusion we state that this paper provides a better understanding of the feature regime for FCNs while better understanding of its advantages in other architectures like CNNs is still lacking,
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work theoretical nature and we don't 't foresee possible negative societal consequences of
work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including thcoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Yes, proofs are either provided in the main text or in appendix, depending space constraints.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material links to the (anonymized) github repository, code and instructions on how to run it are found there.
URL)? [Yss] We provide
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Ys]] Training details are either in the text or in the experiments.md file, as indicated App. G.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Statistical errors are reported shaded area around the curves.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details are provided in App. G.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the Py PyTorch framework and cite the authors in the lext.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] The code written by the authors for running experiments is available URL.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "r-weE8KnHf-", "reviewer": "Reviewer_Rp2y", "summary": "The papers provide insights into why learning features in a certain setting are not robust (may overfit) compared to the lazy training regime. They show that learning features results in sparse features, and they argue that when the target function is smooth along certain input dimensions, this is detrimental. Hence, the non-learning features method generalizes better. ", "questions": "1. See my comment above on wording for generalization/robustness. The statements in this regard are confusing and should be rephrased. The paper mainly talks about robustness, not the generalization to train distribution.\n\n2. Provide supporting argument/applications where the target function is smooth (e.g., for line 48).\n\n3. What is the role of overparameterization?\n\nMinor. On line 39, the author claims that feature learning outperforms lazy training in tasks such as approximating a function. Please demonstrate this similar to Figure 1.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I enjoyed reading the paper. The presentation is clear and well organized. It includes proper experiments with good visualization and comparisons. See my comments below for improvement.\n\n- Confusion of several terminologies. The paper uses generalization terminology. However, it covers the robustness of their methods against specific deformations without providing literature on robustness. Moreover, the discussed deformations may be seen as robustness or out-of-distribution generalization. Hence, it is not completely proper to phrase that learning-feature networks with sparse representations \"overfit\".\n\n- The paper argues that sparse features are not good in fully connected, but they are good in CNN (there is sparsity in CNN and such networks contain robust sparse features). Why this is the case? The results are not convincing of the fact that sparsity is the leading factor in the lack of robustness.\n\n- The sparsity of the features is related to the neural collapse. Please include a discussion related to this and generalization. Please specify at which training size n, the networks become overparameterized. \n\n- This paper talks about sparsity. However, it is missing literature on the generalization of sparse coding [1,2].\n\n\n[1] Mehta, Nishant, and Alexander Gray. \"Sparsity-based generalization bounds for predictive sparse coding.\" International Conference on Machine Learning. PMLR, 2013.\n\n[2] Sulam, Jeremias, Ramchandran Muthukumar, and Raman Arora. \"Adversarial robustness of supervised sparse coding.\" Advances in Neural Information Processing Systems 33 (2020): 2110-2121.\n\nMinor: In Figure 5, the label should be \"train set\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U9bVQMKGk1A", "reviewer": "Reviewer_vPhN", "summary": "The authors consider the asymptotic behavior of the generalization error in two\ndifferent regimes of idealized (infinite-width) neural network training: the\n\"feature regime\" where the network has a small initialization and\nregularization on the parameters, and the \"lazy regime\" where the\ninitialization scale is larger and the network behaves like a kernel ridge\nregression method in the limit. The setting they study is regression from iid\nsamples of an isotropic gaussian random field on the sphere, with data and\nhidden layer weights uniformly distributed on the sphere. The results show that\nwhen the smoothness of the target function is sufficiently high, the asymptotic\ngeneralization error of the learned predictor in the kernel regime has a\nsuperior scaling exponent to that of the feature regime predictor: the\nintuition given is that the feature regime induces a sparse learned predictor,\nwhich is inferior for fitting very smooth functions. They connect the notion of\nfitting highly smooth predictors to invariance to small diffeomorphisms in\nimage classification, and present numerical experiments that suggest that a\nsimilar performance gap is observable there.\n\n", "questions": "In connection with the discussion of small diffeomorphism stability, it would\nbe interesting if the authors can point out connections or implications to\nother works that study deep networks in kernel regimes for computing with\nlow-dimensional data models, such as manifold classification [1] or RF kernel\nregression on subspheres [2], relative to learning such data in the feature\nregime.\n\nI do not quite understand why Assumption 1 (the uniqueness part) is necessary\nhere. There seems to be a significant amount of generic structure on the\nproblem (2.5), given that the covariates are uniformly random and the target\nfunction is also random; in light of the heuristic argument in appendix B, why\nis it not possible to exploit this structure to argue that (2.5) has a unique\nsolution?  This is a standard process in similar finite-dimensional problems\narising in e.g. sparse approximation with $\\ell^1$ regularization and random\nindependent measurements -- I am just curious why similar ideas can't be\ninstantiated here to remove this assumption.\n\nWhat can be said (qualitatively) about the behavior of predictors that are\nsmooth enough for (3.4a, b) to show an advantage for lazy regime training?\nEvidently we need $\\nu_t$ to grow with $d$ -- does this become equivalent to\nthe predictor being in the kernel's RKHS? (e.g. Sobolev embedding theorems and\nRKHS equivalence results?)\n\n[1] https://openreview.net/forum?id=O-6Pm_d_Q-\n\n[2] http://arxiv.org/abs/2006.13409\n", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper presents an interesting finding that offers an alternate viewpoint\n  to one of the predominant ones in the DL theory literature -- that the\n  two-layer \"feature learning\" training regime is inherently superior to the\n  kernel regime -- and supports the settings where this does not hold by\n  arguing that invariance to nuisance transformations in image classification\n  might be such a setting. This will be an interesting addition to this\n  conversation in general.\n- The paper is clearly written. Although technical, it provides ample\n  well-chosen references to background/related works that allow technical\n  aspects of the problem to be understood. Intuitions are explained so clearly\n  that the conclusions of the paper end up feeling natural after some thought.\n- The experiments presented are thought-provoking and make a connection to\n  practical structured data in a way that many in the deep learning theory\n  literature do not.\n\n## Weaknesses\n\n- Proofs are somewhat terse. I could not follow the proof of Proposition 2\n  around eqn C.8 very well -- is there any need to justify the interchanging of\n  limits/integrals here? The calculation in section D appears to apply to\n  $\\varphi$ with a single discontinuity, rather than to the functions\n  associated to the NTK or the ReLU. Some clarification on whether the\n  calculations in Appendices D and E are being claimed as rigorous might be\n  helpful -- this word is used in the main paper at points, and the intuitive\n  discussions make sense, but some of the arguments appear heuristic (perhaps\n  this is just because the details are often left to the reader).\n- It might make sense to examine some of the hypotheses around diffeomorphism\n  stability in a more 'toy dataset' setting relative to just testing on\n  benchmark datasets (since the issue of varying the train set size is somewhat\n  strange here, given non-iid-ness, and the performance separations in Figure 5\n  do not always support the hypothesis). For example, what about creating a\n  synthetic dataset of small diffeomorphisms (or even moderate-dimensional\n  parametric deformations, which will be easier to sample and compute) of two a\n  fixed templates embedded into some background, and trying to classify?\n  Additionally, here a metric that truly captures diffeomorphism stability\n  could be used rather than the relative one of eqn (5.1).\n\n## Minor Issues\n\n- Hyperref links are broken in the pdf. I also cannot access the github\n  repository link in line 69. Although there may be other fixes, I always enjoy\n  using the `xr` package in Latex to avoid this.\n- I did not follow how the Lazy regime described in lines 123-135 relates to the\n  training of an actual neural network for fitting the data (it is not clear to\n  me how the \"Assume that...\" at the start of the paragraph is linked to an\n  actual neural network training; the definition (2.3) also has a\n  nonstandard-seeming initial \"lazy regime\" network, which is identically\n  zero).  I only mention this because of the contrast with the description of\n  the feature regime above, where eqn (2.6) makes the link fairly\n  well-motivated. Since the experiments for \"lazy regime\" are eventually just\n  simulating the associated KRR method, perhaps the description could be\n  amended accordingly, and/or a reference could be added that describes the\n  connection to neural network fitting with this initialization scheme.\n- I was not familiar with the precise result alluded to in lines 110-112; would\n  be slightly helpful to add a more precise pointer e.g.  [31, Section 5]\n  perhaps.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J6LD_F7PRV_", "reviewer": "Reviewer_mho4", "summary": "The paper tackles an important question of the impact of feature learning on generalization. While it is a common wisdom that feature learning is beneficial, experimentally it is not the case for fully-connected networks on image datasets.\n\nThe authors show that when learning a random isotropic Gaussian process on a sphere with a wide, single-hidden-layer ReLU network, models trained in the feature and lazy regimes will exhibit different scaling (with the dataset size) of the generalization MSE error. The scaling depends on the smoothness of the target function, and for sufficiently smooth functions the lazy regime will actually scale better.\n\nThe authors verify their theory with experiments, and provide some discussion and further experiments connecting these results to the observed phenomena in practice (that lazy fully-connected networks perform better on image datasets than feature learning fully-connected networks).\n", "questions": "- Could you provide a companion plot to Figure 1 plotting classification error instead of the MSE?\n\n- On line 111 you say that the feature regime is equivalent to L2 regularization. Does your result then imply that on image datasets as considered in Figure 1, fully-connected networks shouldn't use weight decay? If so, how does this reconcile with the common wisdom that weight decay tends to improve generalization (see e.g. top row of Figure 1 in https://arxiv.org/pdf/2007.15801.pdf)? \n\n- Could you elaborate on your reasoning on lines 245-246, i.e. why the low dimension of the transformation makes it negligible? Even though translations are 2D, aren't they still not learnable by a few neurons, hence could be a good space of transformations to test your theory? Same for rotations (which are 1D).\n\n\t* Relatedly, it would also be nice to contrast Figure 5 with equivalent plots for deformations that are learnable with a few neurons (perhaps brightness, corruption to a few fixed pixels, etc).\n\n- Is there an explanation for why in Figure 5 $R_f$ is usually $> 1$, i.e. networks are more sensitive to image diffeomorphisms than to random noise?\n\n- Why is $R_f$ chosen to be the ratio of sensitivities, as opposed to just the numerator? Intuitively, in connection to generalization, I would expect the absolute (vs relative) stability to matter. Relatedly, in Figure 1 error curves for CIFAR-10 diverge for large $n$, but stability curves, in contrast, converge for large $n$ (with lazy even becoming more sensitive than feature all the way to the right), which casts doubt on stability explaining the generalization gap here.\n\n- Line 292: which best architectures don't use pooling?\n\n- Do you have thoughts on connecting more explicitly $\\nu_t$ to some metric of the dataset like CIFAR-10? Perhaps, fitting a Gaussian process with smoothness $\\nu_t$, picking/optimizing $\\nu_t$ for max likelihood, then using it to predict whether lazy or feature will do better?", "limitations": "All good.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a specific and novel result on the importance of feature learning in deep learning, which is also well-backed by experiments, hence I recommend acceptance. \n\nThe evidence and discussion on transferring these results to real-world datasets appear promising but preliminary, which is why my recommendation is less strong.\n\n## Pros:\n\n- Interesting and important topic.\n- Clear and novel result, well-backed by experiments.\n- Well-written.\n\n## Cons:\n\n- Considered construction is rather specific, and it will require more future work to understand how relevant these results are to explaining the actual generalization gaps on real datasets observed in practice (e.g. in Figure 1). The authors propose an interesting hypothesis in section 5, but I find it inconclusive (see also some questions below). While more connections to practical settings would make this a much stronger paper, I find the presented results already significant enough for publication.\n\n## Minor\n\n- Definition of stability (Eq. 5.1) is somewhat counter-intutivie, since high stability actually implies high sensitivity (ergo low stability) to $\\tau$. Also, superscript $n$ can be omitted to declutter.\n\n- All links, including the code repository, are not clickable.\n\n- I would use more visually-distinct line styles in Figure G.1 (e.g. `:` vs `-.`).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IJPpuTA5m3L", "reviewer": "Reviewer_ZBQb", "summary": "This paper studies the dynamics of how a neural network learns from data. Authors focus on the claim that neural networks learn sparse representation of data, and demonstrate that when the underlying phenomenon is not sparse, encouraging the model to learn sparse features will lead to bad generalization.\n", "questions": "Please see the comments above.", "limitations": "Overall, the arguments put forward in the paper do not naturally generalize to real world applications of neural networks on images and such. This is a limitation that can be acknowledged in a clearer way.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nPaper is well written and its arguments are clear. Its message seems intuitive, even trivial, but it seems to put into perspective some of the ideas that are present in the literature. Overall, it seems to me that the research community is better off having this paper in the literature, but I would like to see it significantly improved before publication.\n\n\nWeaknesses:\nThe main argument seems trivial to me: if the underlying phenomenon is not sparse, one should not try to learn sparse features, or try to impose sparsity into the learning process.\n\nThe notion of feature learning vs lazy learning seems fluid and ambiguous. It is not clear why we should categorize learning into these two types, and if we do this categorization, where the exact line would be that separates these two categories of feature learning vs lazy learning.\n\nPaper does not consider the argument that models learn a \"low-dimensional feature space\" from the original data, rather it focuses on the argument that models learn \"sparse features\". These two arguments are not equivalent, but they are related, and authors can at least acknowledge those studies on the literature. For example, see:\n\nCohen, U., Chung, S., Lee, D.D. and Sompolinsky, H., 2020. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1), pp.1-13.\n\nZarka, J., Guth, F. and Mallat, S., 2021, May. Separation and Concentration in Deep Networks. In ICLR 2021-9th International Conference on Learning Representations.\n\nBengio, Y., Courville, A. and Vincent, P., 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8), pp.1798-1828.\n\nMany of the arguments in the literature that suggest models learn a sparse representation, imply that models learn a low dimensional feature space. So, sparsity may not be the property of that low-dimensional feature space, rather, it could be the property of original data.\n\nFor example, consider a dataset that has linearly dependent features and therby, its data matrix is rank deficient. By dropping the zero singular values, one can project that data into a lower dimensional space without losing any information. The data and its projection may not be sparse at all.\n\nPaper does not consider the studies that impose sparsity on the parameters of the model. For example, this paper shows that convolutional layers can be learned using feed-forward networks and imposing sparsity on the parameters of the model:\n\nNeyshabur, B., 2020. Towards learning convolutions from scratch. Advances in Neural Information Processing Systems, 33, pp.8078-8088.\n\nWhen authors talk about high dimensions, especially in introduction and in the closing, it is not clear what they actually mean. How many dimensions are considered high dimensions, and how many dimensions would be low? Some of these arguments seem hand wavy and ambiguous.\n\nFCN is not defined in the paper the first time it is introduced.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept