Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles an important question of the impact of feature learning on generalization. While it is a common wisdom that feature learning is beneficial, experimentally it is not the case for fully-connected networks on image datasets.

The authors show that when learning a random isotropic Gaussian process on a sphere with a wide, single-hidden-layer ReLU network, models trained in the feature and lazy regimes will exhibit different scaling (with the dataset size) of the generalization MSE error. The scaling depends on the smoothness of the target function, and for sufficiently smooth functions the lazy regime will actually scale better.

The authors verify their theory with experiments, and provide some discussion and further experiments connecting these results to the observed phenomena in practice (that lazy fully-connected networks perform better on image datasets than feature learning fully-connected networks).

strengths_and_weaknesses: The paper presents a specific and novel result on the importance of feature learning in deep learning, which is also well-backed by experiments, hence I recommend acceptance. 

The evidence and discussion on transferring these results to real-world datasets appear promising but preliminary, which is why my recommendation is less strong.

## Pros:

- Interesting and important topic.
- Clear and novel result, well-backed by experiments.
- Well-written.

## Cons:

- Considered construction is rather specific, and it will require more future work to understand how relevant these results are to explaining the actual generalization gaps on real datasets observed in practice (e.g. in Figure 1). The authors propose an interesting hypothesis in section 5, but I find it inconclusive (see also some questions below). While more connections to practical settings would make this a much stronger paper, I find the presented results already significant enough for publication.

## Minor

- Definition of stability (Eq. 5.1) is somewhat counter-intutivie, since high stability actually implies high sensitivity (ergo low stability) to $\tau$. Also, superscript $n$ can be omitted to declutter.

- All links, including the code repository, are not clickable.

- I would use more visually-distinct line styles in Figure G.1 (e.g. `:` vs `-.`).

questions: - Could you provide a companion plot to Figure 1 plotting classification error instead of the MSE?

- On line 111 you say that the feature regime is equivalent to L2 regularization. Does your result then imply that on image datasets as considered in Figure 1, fully-connected networks shouldn't use weight decay? If so, how does this reconcile with the common wisdom that weight decay tends to improve generalization (see e.g. top row of Figure 1 in https://arxiv.org/pdf/2007.15801.pdf)? 

- Could you elaborate on your reasoning on lines 245-246, i.e. why the low dimension of the transformation makes it negligible? Even though translations are 2D, aren't they still not learnable by a few neurons, hence could be a good space of transformations to test your theory? Same for rotations (which are 1D).

	* Relatedly, it would also be nice to contrast Figure 5 with equivalent plots for deformations that are learnable with a few neurons (perhaps brightness, corruption to a few fixed pixels, etc).

- Is there an explanation for why in Figure 5 $R_f$ is usually $> 1$, i.e. networks are more sensitive to image diffeomorphisms than to random noise?

- Why is $R_f$ chosen to be the ratio of sensitivities, as opposed to just the numerator? Intuitively, in connection to generalization, I would expect the absolute (vs relative) stability to matter. Relatedly, in Figure 1 error curves for CIFAR-10 diverge for large $n$, but stability curves, in contrast, converge for large $n$ (with lazy even becoming more sensitive than feature all the way to the right), which casts doubt on stability explaining the generalization gap here.

- Line 292: which best architectures don't use pooling?

- Do you have thoughts on connecting more explicitly $\nu_t$ to some metric of the dataset like CIFAR-10? Perhaps, fitting a Gaussian process with smoothness $\nu_t$, picking/optimizing $\nu_t$ for max likelihood, then using it to predict whether lazy or feature will do better?
limitations: All good.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors consider the asymptotic behavior of the generalization error in two
different regimes of idealized (infinite-width) neural network training: the
"feature regime" where the network has a small initialization and
regularization on the parameters, and the "lazy regime" where the
initialization scale is larger and the network behaves like a kernel ridge
regression method in the limit. The setting they study is regression from iid
samples of an isotropic gaussian random field on the sphere, with data and
hidden layer weights uniformly distributed on the sphere. The results show that
when the smoothness of the target function is sufficiently high, the asymptotic
generalization error of the learned predictor in the kernel regime has a
superior scaling exponent to that of the feature regime predictor: the
intuition given is that the feature regime induces a sparse learned predictor,
which is inferior for fitting very smooth functions. They connect the notion of
fitting highly smooth predictors to invariance to small diffeomorphisms in
image classification, and present numerical experiments that suggest that a
similar performance gap is observable there.


strengths_and_weaknesses: ## Strengths

- The paper presents an interesting finding that offers an alternate viewpoint
  to one of the predominant ones in the DL theory literature -- that the
  two-layer "feature learning" training regime is inherently superior to the
  kernel regime -- and supports the settings where this does not hold by
  arguing that invariance to nuisance transformations in image classification
  might be such a setting. This will be an interesting addition to this
  conversation in general.
- The paper is clearly written. Although technical, it provides ample
  well-chosen references to background/related works that allow technical
  aspects of the problem to be understood. Intuitions are explained so clearly
  that the conclusions of the paper end up feeling natural after some thought.
- The experiments presented are thought-provoking and make a connection to
  practical structured data in a way that many in the deep learning theory
  literature do not.

## Weaknesses

- Proofs are somewhat terse. I could not follow the proof of Proposition 2
  around eqn C.8 very well -- is there any need to justify the interchanging of
  limits/integrals here? The calculation in section D appears to apply to
  $\varphi$ with a single discontinuity, rather than to the functions
  associated to the NTK or the ReLU. Some clarification on whether the
  calculations in Appendices D and E are being claimed as rigorous might be
  helpful -- this word is used in the main paper at points, and the intuitive
  discussions make sense, but some of the arguments appear heuristic (perhaps
  this is just because the details are often left to the reader).
- It might make sense to examine some of the hypotheses around diffeomorphism
  stability in a more 'toy dataset' setting relative to just testing on
  benchmark datasets (since the issue of varying the train set size is somewhat
  strange here, given non-iid-ness, and the performance separations in Figure 5
  do not always support the hypothesis). For example, what about creating a
  synthetic dataset of small diffeomorphisms (or even moderate-dimensional
  parametric deformations, which will be easier to sample and compute) of two a
  fixed templates embedded into some background, and trying to classify?
  Additionally, here a metric that truly captures diffeomorphism stability
  could be used rather than the relative one of eqn (5.1).

## Minor Issues

- Hyperref links are broken in the pdf. I also cannot access the github
  repository link in line 69. Although there may be other fixes, I always enjoy
  using the `xr` package in Latex to avoid this.
- I did not follow how the Lazy regime described in lines 123-135 relates to the
  training of an actual neural network for fitting the data (it is not clear to
  me how the "Assume that..." at the start of the paragraph is linked to an
  actual neural network training; the definition (2.3) also has a
  nonstandard-seeming initial "lazy regime" network, which is identically
  zero).  I only mention this because of the contrast with the description of
  the feature regime above, where eqn (2.6) makes the link fairly
  well-motivated. Since the experiments for "lazy regime" are eventually just
  simulating the associated KRR method, perhaps the description could be
  amended accordingly, and/or a reference could be added that describes the
  connection to neural network fitting with this initialization scheme.
- I was not familiar with the precise result alluded to in lines 110-112; would
  be slightly helpful to add a more precise pointer e.g.  [31, Section 5]
  perhaps.

questions: In connection with the discussion of small diffeomorphism stability, it would
be interesting if the authors can point out connections or implications to
other works that study deep networks in kernel regimes for computing with
low-dimensional data models, such as manifold classification [1] or RF kernel
regression on subspheres [2], relative to learning such data in the feature
regime.

I do not quite understand why Assumption 1 (the uniqueness part) is necessary
here. There seems to be a significant amount of generic structure on the
problem (2.5), given that the covariates are uniformly random and the target
function is also random; in light of the heuristic argument in appendix B, why
is it not possible to exploit this structure to argue that (2.5) has a unique
solution?  This is a standard process in similar finite-dimensional problems
arising in e.g. sparse approximation with $\ell^1$ regularization and random
independent measurements -- I am just curious why similar ideas can't be
instantiated here to remove this assumption.

What can be said (qualitatively) about the behavior of predictors that are
smooth enough for (3.4a, b) to show an advantage for lazy regime training?
Evidently we need $\nu_t$ to grow with $d$ -- does this become equivalent to
the predictor being in the kernel's RKHS? (e.g. Sobolev embedding theorems and
RKHS equivalence results?)

[1] https://openreview.net/forum?id=O-6Pm_d_Q-

[2] http://arxiv.org/abs/2006.13409

limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the dynamics of how a neural network learns from data. Authors focus on the claim that neural networks learn sparse representation of data, and demonstrate that when the underlying phenomenon is not sparse, encouraging the model to learn sparse features will lead to bad generalization.

strengths_and_weaknesses: Strengths:
Paper is well written and its arguments are clear. Its message seems intuitive, even trivial, but it seems to put into perspective some of the ideas that are present in the literature. Overall, it seems to me that the research community is better off having this paper in the literature, but I would like to see it significantly improved before publication.


Weaknesses:
The main argument seems trivial to me: if the underlying phenomenon is not sparse, one should not try to learn sparse features, or try to impose sparsity into the learning process.

The notion of feature learning vs lazy learning seems fluid and ambiguous. It is not clear why we should categorize learning into these two types, and if we do this categorization, where the exact line would be that separates these two categories of feature learning vs lazy learning.

Paper does not consider the argument that models learn a "low-dimensional feature space" from the original data, rather it focuses on the argument that models learn "sparse features". These two arguments are not equivalent, but they are related, and authors can at least acknowledge those studies on the literature. For example, see:

Cohen, U., Chung, S., Lee, D.D. and Sompolinsky, H., 2020. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1), pp.1-13.

Zarka, J., Guth, F. and Mallat, S., 2021, May. Separation and Concentration in Deep Networks. In ICLR 2021-9th International Conference on Learning Representations.

Bengio, Y., Courville, A. and Vincent, P., 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8), pp.1798-1828.

Many of the arguments in the literature that suggest models learn a sparse representation, imply that models learn a low dimensional feature space. So, sparsity may not be the property of that low-dimensional feature space, rather, it could be the property of original data.

For example, consider a dataset that has linearly dependent features and therby, its data matrix is rank deficient. By dropping the zero singular values, one can project that data into a lower dimensional space without losing any information. The data and its projection may not be sparse at all.

Paper does not consider the studies that impose sparsity on the parameters of the model. For example, this paper shows that convolutional layers can be learned using feed-forward networks and imposing sparsity on the parameters of the model:

Neyshabur, B., 2020. Towards learning convolutions from scratch. Advances in Neural Information Processing Systems, 33, pp.8078-8088.

When authors talk about high dimensions, especially in introduction and in the closing, it is not clear what they actually mean. How many dimensions are considered high dimensions, and how many dimensions would be low? Some of these arguments seem hand wavy and ambiguous.

FCN is not defined in the paper the first time it is introduced.
questions: Please see the comments above.
limitations: Overall, the arguments put forward in the paper do not naturally generalize to real world applications of neural networks on images and such. This is a limitation that can be acknowledged in a clearer way.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The papers provide insights into why learning features in a certain setting are not robust (may overfit) compared to the lazy training regime. They show that learning features results in sparse features, and they argue that when the target function is smooth along certain input dimensions, this is detrimental. Hence, the non-learning features method generalizes better. 
strengths_and_weaknesses: I enjoyed reading the paper. The presentation is clear and well organized. It includes proper experiments with good visualization and comparisons. See my comments below for improvement.

- Confusion of several terminologies. The paper uses generalization terminology. However, it covers the robustness of their methods against specific deformations without providing literature on robustness. Moreover, the discussed deformations may be seen as robustness or out-of-distribution generalization. Hence, it is not completely proper to phrase that learning-feature networks with sparse representations "overfit".

- The paper argues that sparse features are not good in fully connected, but they are good in CNN (there is sparsity in CNN and such networks contain robust sparse features). Why this is the case? The results are not convincing of the fact that sparsity is the leading factor in the lack of robustness.

- The sparsity of the features is related to the neural collapse. Please include a discussion related to this and generalization. Please specify at which training size n, the networks become overparameterized. 

- This paper talks about sparsity. However, it is missing literature on the generalization of sparse coding [1,2].


[1] Mehta, Nishant, and Alexander Gray. "Sparsity-based generalization bounds for predictive sparse coding." International Conference on Machine Learning. PMLR, 2013.

[2] Sulam, Jeremias, Ramchandran Muthukumar, and Raman Arora. "Adversarial robustness of supervised sparse coding." Advances in Neural Information Processing Systems 33 (2020): 2110-2121.

Minor: In Figure 5, the label should be "train set".
questions: 1. See my comment above on wording for generalization/robustness. The statements in this regard are confusing and should be rephrased. The paper mainly talks about robustness, not the generalization to train distribution.

2. Provide supporting argument/applications where the target function is smooth (e.g., for line 48).

3. What is the role of overparameterization?

Minor. On line 39, the author claims that feature learning outperforms lazy training in tasks such as approximating a function. Please demonstrate this similar to Figure 1.
limitations: N/A
review_title: null

Meta-review: metareview: This paper studies different behaviors of feature learning and lazy training in the regression of Gaussian random functions and image classification. The main contribution includes insights into why feature learning in a particular setting is not robust (may overfit) compared to the lazy training regime for fully connected networks. 

The reviewers pointed out several concerns about the presentation of the proofs, different behaviors of FCN and CNN, etc., which have been appropriately addressed by the authors. Overall, all reviewers appreciate the contribution of this paper, so I recommend accept. 
confidence: Certain
Decision: Accept