Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a theoretical analysis 
- bounding the difference between a kernel dynamics of finite width network trained for time T with a finite set of samples and the ideal infinite width NTK kernel.
- that concludes a finite width network is biased toward learning the top eigenfunction of the infinite width-based kernel 
strengths_and_weaknesses: - The paper positions itself well amongst related works and is presented such that even an outsider to the topic is able to understand the problem, analysis, and results.
- The paper is constrained to bounding the difference in the kernels and it is not clear how the network architecture and activation function fits into the analysis. A discussion on how the model design affects the results could make clear the limitation as well as applicability to future model designs.
- The analysis considered requires twice differentiable activation function which does not include ReLU. I think this is an admissible assumption for theoretical analysis though it makes analysis disconnected from practice.
Minor comment
- the notation for non-linearity and eigenvalue seems to be overloaded and might create confusion.

questions: - The theorem associates the minimum width of the network and the number of samples to the number of iterations trained. I'm confused by this connection - scaling the size of the network is possible but the size of the data is, in practice, fixed. Does this mean that for fixed-size data, the results are applicable only up to a certain training step? 

limitations: - The paper targets a simple question of spectral bias of networks using NTK kernels and provides a characterization that is applicable to deep networks.
- The dependency on stopping time limits the consequences of the analysis to when the network is initialized (with few training steps)
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a new theoretical analysis of the bounds of function space difference between a finite-width network in a finite sample size regime and the corresponding kernel dynamics in finite width and finite data regime.
The results are presented based on fully connected, convolutional and residual neural network architectures.
By investigating the spectrum of the NTK and its connections to the Fisher Information Matrix, this work provided some insights into the spectral bias of neural network learning dynamics.
strengths_and_weaknesses: 
#### Strengths
- This work continues to explore the spectral bias of the neural networks from the perspective of the neural tangent kernel theory. This work made some novel theoretical analyses on measuring the distance in function space and the scaling properties of the learning with respect to the stopping time.
- This paper is generally clearly written. Alongside the main theoretical results, it provides some description and interpretation of the main theorems.

#### Weaknesses
- My main concerns are about the lack of more discussions about the connections with existing literature ([1,2,3,4,5]) on analyzing the spectral structure of neural networks and corresponding kernel formulations, for example, existing works [1] provided theoretical results on the spectral alignment of neural network and kernel methods in learning dynamics, [2] addressed the low-rank spectral structure of both neural tangent kernel and the Fisher information matrix. Not limited to the missing references below, a more detailed discussion on the theoretical and empirical connections to the relevant literature is needed.

[1] Atanasov, Alexander, Blake Bordelon, and Cengiz Pehlevan. "Neural networks as kernel learners: The silent alignment effect." arXiv preprint arXiv:2111.00034 (2021).

[2] Zhang, R., Zhai, S., Littwin, E., & Susskind, J. (2022). Learning Representation from Neural Fisher Kernel with Low-rank Approximation. arXiv preprint arXiv:2202.01944.

[3] Karakida, Ryo, Shotaro Akaho, and Shun-ichi Amari. "Pathological spectra of the fisher information metric and its variants in deep neural networks." arXiv preprint arXiv:1910.05992 (2019).

[4] Pennington, Jeffrey, and Pratik Worah. "The spectrum of the fisher information matrix of a single-hidden-layer neural network." Advances in neural information processing systems 31 (2018).

[5] Hazan, Tamir, and Tommi Jaakkola. "Steps toward deep kernel methods from infinite neural networks." arXiv preprint arXiv:1508.05133 (2015).
questions: - Besides the presented theoretical results, have the authors tried on any synthetic or MNIST-scale datasets to see if the results are consistent with the empirical experimental results?
limitations: N/A.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper bounds the distance between the dynamics of wide DNNs and the dynamics of kernel gradient descent on the population with the limiting NTK. This paper improves over previous bounds, by considering the projected error along the top k eigenfunctions of the integral operator $T_{K^\infty}$ for any $k$.

They show that the approximation is better along the top eigenvalues. This allows them to show that with early stopping, one can guarantee learning along the top eigenvalues of DNNs. In comparison to previous bounds, the results in this paper only require a width that grows linearly with the number of datapoints, at least with a specific early stopping.
strengths_and_weaknesses: The paper is well written and gives a good overview of the previous bounds and how their bound improves on previous ones. The paper nicely connects two lines of research: the NTK regime approximations for finite widths and the spectral bias of DNNs.

The results are new to my knowledge and describe an interesting phenomenon. I did not check the proofs in detail but I did not see any technical issues.

The only weakness that concerns me is the fact that the results only apply to gradient flow. The discussion of early stopping is much more interesting in the context of gradient descent since it then relates to computational complexity. There exist many proofs of convergence to NTK-dynamics for gradient descent, so I think the proofs should extend to this setting easily. I do not understand why the authors did not try to extend their proof to gradient descent.
questions: I do not have any questions or suggestions regarding the aforementioned weakness (except to try and do the proofs for gradient descent but I realize that it would require a lot of work).
limitations: As already mentioned I think the paper would be greatly improved with a discussion of gradient descent.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This article theoretically offers quantitative bounds between the trajectory of a finite-width network from the idealized kernel dynamics of infinite one, which implies that networks tend to learn the top eigenfunctions of the NTK at rates corresponding to their eigenvalues. Furthermore, the authors observe that if the width of network grows linearly with the number of samples, it will remain the bias of the kernel at beginning. 
strengths_and_weaknesses: Strengths:
1.	The paper is rich in theory and offers quantitative bounds between the trajectory of a finite-width network from the idealized kernel dynamics of infinite one, which measure the L2 difference in function space and imply that networks tend to learn the top eigenfunctions over the entire input space. I'm sure the audience of NeurIPS will be interested in it.
2.	The theoretical proof is detailed, the order is reasonable and the structure is clear.

Weaknesses:
1.	Although the theory of this article is rich, there are very few experiments that could be found in it, which could verify the rationality of the hypothesis and whether the theoretical derivation is true or not. One suggestion is that: It will make the conclusions more convincing if corresponding graphs of experimental results could be attached after important conclusions. Especially, authors can use a 1d experiments to show the estimation in theorem 1 is meaningful.

questions: 1.	It seems that this article does not consider the bias term in the neural network. If the bias term is considered, do the corresponding results still hold?
2.	Many Relu-like activation functions do not satisfy Assumption 3.1. Can authors discuss more about such activation functions?
3.	If I understand correctly, as t goes to infinity, the RHS of the first inequality converges to a non-zero constant. Does mean this estimation is too loose. And, how can this inequality relates the behavior of the convergence outside the training set?
4.	In “NTK Eigenvector and Eigenfunction Convergence Rates”, a related reference is missed: https://arxiv.org/abs/2010.08153

limitations: 1.	The main limitation of this paper is that although the theory is rich, there are too few experiments to tell the reader at a glance what it does. The interpretation of theoretical result is not clear enough, especially, for “outside the training set”.
review_title: null

Meta-review: metareview: This paper focuses on theoretically bounding the difference between a finite-width network in a finite sample size regime and the corresponding kernel dynamics in finite width and finite data regime for various neural network architectures. Using the spectrum of the NTK they provide some insights into the spectral bias of neural nets. All reviewers were positive and recommend acceptance. I concur with this decision.

confidence: Certain
Decision: Accept