Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper analyzes feature learning in infinite-width neural networks trained with gradient descent. In this limit, the distribution over hidden unit activations and gradients in each layer becomes i.i.d. and can be characterized with some order parameters using self-consistent dynamical field theory. These order parameters are inner-product kernels that are determined self-consistently from the aforementioned distributions. These self-consistent equations can be solved exactly for linear networks, as they reduce to matrix equations in this case, or numerically for nonlinear networks. The authors demonstrate very good agreement between their theory and simulations in finite-width neural networks. Finally, limitations of the theory are discussed, showing where the theory breaks down, and a comparison to other analyses of feature learning via perturbations of the NTK is made.

strengths_and_weaknesses: Strengths
- Understanding feature learning is arguably the most important open theoretical problem for neural networks and is certainly of interest and significance to the NeurIPS community.
- Obviously, the work builds on previous techniques and analyses but the paper makes a strong original contribution with interesting results.
- The discussion of connections to previous work are helpful for the reader to build intuition.

Weaknesses
- Most of the calculations are relegated to the supplement. This is probably expected given their complexity.
- It is not completely clear what is proved rigorously and what results are formal calculations.
- The experiments are on extremely small datasets due to the computational limitations discussed in Section 7.
- There is little interpretation of the features that are learned in this limit.

Minor typos
- Citation on line 198.
- "numerically method" on line 249.
- "Which" and "that" are mixed up several times.
- Some compound adjectives are missing hyphens, e.g. "infinite width neural networks" -> "infinite-width neural networks"
- "Figures 5 and Figure 6" on line 165
questions: - Do the authors have any sense how much of the performance gap between finite-width neural networks and infinite-width neural networks is closed by the feature learning in this limit?
- For linear networks, is the main obstacle to scaling the experiments to larger datasets in solving the linear system of Equation (12) or are there other challenges?
limitations: No concerns about negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes to simulate the dynamics of infinitely-wide neural networks under gradient descent through a self-consistent dynamical field theory. The key idea is to adopt techniques from dynamical kernels, which can govern the evolution of a neural network. The authors provide an analytic result for the linear network while providing a sampling method to approximate the dynamics. Compared to various approximation sketches, the proposed method is shown to obtain consistent solutions across different regime. Lastly, the authors provide experiments in more realistic settings which demonstrate that the method is still valid.
strengths_and_weaknesses: Overall, the work is of high quality. First of all, in terms of writing, this article has a clear structure and is relatively easy to read. From a method perspective, it is a novel idea to use dynamicsl field theory to simulate the dynamics of infinitely wide neural networks, especially the feature learning area. From the simulation results, compared with other basic methods, the method proposed by the author can effectively capture the dynamics of the neural network.

On the other hand, the method proposed in this work has a relatively large limitation in computational efficiency, and there is still a certain gap from the actual neural network application scenario. However, I also agree that this limitation can be addressed in the next step.

questions: We know that NTK methods cannot fully explain the full capabilities of neural networks. What makes me curious is, what is the difference between the method in this work and the finite width neural network in practical performance, especially in the feature learning area?
limitations: I appreciate that the authors acknowledge the limitation of expensive computation. I encourage the authors to try to further improve the computational efficiency of the algorithm in future research.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors continues the line of research that uses dynamical mean field theory (DMFT) to study neural networks. 
Using statistical physics heuristics, they derive self-consistent equations that governs the evolutions of the kernels, the (internal) representations and the gradients of the networks in the infinite-width feature learning regime for deep neural networks. Although these equations are too expensive to solve in the general setting, the authors propose a sampling-based approach to solve the equations numerically. They show good agreement between theory prediction and simulation. Finally, the authors have a very nice discussion explaining that their framework capture several recently developed perturbation-based approaches to understand neural networks. 

Overall, I find the paper interesting, insightful and have valuable contribution to the NeurIPS community. 
strengths_and_weaknesses: ## Strengths 

- Generating several previous DFMT related works to multiple layer networks setting
- Technical contribution in deriving self-consistent equations 
- Proposed framework captures several related existing works, e.g. finite-size correction [26, 27] in the paper.   
- Good agreement between theory and (small scale) simulation. 


## Weaknesses 
- The calculations are very far from rigorous. It is totally unclear that under what assumptions the results of the paper are correct. 
- The current presentation is not very friendly to readers without DMFT background. E.g. Section 3 (main theoretical contribution) is very hard to follow and hard to extract key ideas and insights behind the equations and the techniques for deriving them. Walking the readers through the deviation in the simplest possible setting (e.g. 2-layer linear networks) will be much appreciated. 
- The self-consistent equations are not very interpretable (at least at the current form). It is not clear, at least to me, what extra insights regarding feature learning (beyond linear models) can we get from those equations.
-  Computationally, these equations are much much more expensive to solve than just training the original networks and unscalable (cubic dependence on both training steps and training samples)
questions: - Is it possible to formulate the assumptions under which the results are theoretically sound?   

- Yang&Hu (Feature Learning in Infinite-Width Neural Networks) also derives similar (recursive) equations that depend on the distribution of of the (hidden) representations and their derivatives. Followup work (EFFICIENT COMPUTATION OF DEEP NONLINEAR ∞-
WIDTH NEURAL NETWORKS THAT LEARN FEATURES) also derives efficient approximation scheme to solve the recursive equations, which is scalable to full CIFAR10. Can you add more detailed discussion about these two papers? What are the advantages and disadvantages? Is it also possible to scale up your algorithm to full CIFAR10?


limitations: The paper is purely theoretical. No potential negative societal impact as I can tell. 

The authors discussed the limitation regarding computation. The non-rigorousness nature of the approach should also be discussed.
review_title: null

Meta-review: metareview: This paper analyzes a dynamical mean field theory that describes feature learning via gradient flow for certain infinite-width neural networks. Self-consistent equations for the order parameters characterizing the dynamics are presented and methods for approximate numerical evaluation are discussed. Overall, this is a solid paper that advances the theory and understanding of feature learning for neural networks of large width and the reviewers and I unanimously support acceptance.


confidence: Certain
Decision: Accept