Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This submission studies the optimization of the neural network with infinite-dimensional input in the mean-field regime and provides a convergence guarantee based on weak convergence. This result utilizes the proximal gradient method which solves the optimization task by linearization and Langevin sampling approximation. The authors study the convergence rate in both finite-sum and online settings. Compared to previous work, this submission considers the infinite-dimensional case and makes use of the analysis in infinite-dimensional Langevin. Besides, the authors provide the experiment on functional regression. The primary contribution is the convergence guarantee of the algorithm with infinite-dimensional input.
strengths_and_weaknesses: Strengths:

This paper provides a complete and detailed analysis on infinite-dimensional-input P-SDCA and PDA and provides the weak convergence result. First the paper is well-written and the analysis is rigorous and well-orginized. The assumptions are stated explicitly. The authors also llustrate the effectiveness by numerical experiments.

Weakness:

Lack disscussion on the theoretical result. It would be better to discuss the implication of the convergence rate e.g. on $\lambda_2$. I am not familiar with this field but it seems that this line of research cannot overcome the exponential dependency on sample size M which makes the bound less informative. 

Minor:

In proof of thm 4, drop the weight t.
questions: Refer to strengths and weaknesses.
limitations: Refer to strengths and weaknesses
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors extend the mean-field limit in the two-layer neural networks with finite-dimensional input to the infinite-dimensional input setting. The generalization bound was derived to show the regularized empirical risk minimizer is provable generalized at rate of $O(1/\sqrt{n})$. Two optimization algorithms (PDA, P-SDCA) based on gradient Langevin dynamics are proposed. Authors proved that two algorithms could converge to optimal solution under certain assumptions. Experiment results also support the theoretical analysis.
strengths_and_weaknesses: Strengths 
1. In general the paper is well organized and clearly written.
2. Extending mean-field limit to the infinite-dimensional input setting to allow feature learning seems to be new. The two proposed optimization algorithms as well as the generalization bound also seem to be new in the literature.
3. The theoretical results are well supported with proof ideas in the main text and complete proof in the appendix.
4. This paper gives generalization bound $O(1/\sqrt{n})$ that shows regularized empirical risk minimizer generalizes provided enough samples in the infinite-dimensional data setting. 

Weaknesses
1. While the two proposed optimization algorithms could converge to optimal solution, their running time are exponential in regularization parameter $\lambda_2$. As suggested by Thm 1, $\lambda_2$ should be set to $1/\sqrt{n}$, which implies an exponential runtime. This seems unavoidable in the general mean-field setting with gradient Langevin dynamics. However, it would be better if authors could clearly state this, since the current abstract and contribution part seems to suggest the runtime is polynomial.
2. The generalization bound depends on $KL(\pi^o||\nu)$ where $\pi^o$ is the target distribution and $\nu$ is Gaussian distribution. It would be better if some discussion could be added to clarify in some common settings when this KL divergence is bounded (e.g., $O(1)$) and when it is not.

questions: 1. In Theorem 7, at the last line what is the expectation taken over? Is it taken over the random noise during training?
2. It would be better if authors could write a main theorem for PDA to combine Theorem 3,4,5 (just like Theorem 7 for P-SDCA).

limitations: Authors discuss the limitations of current work. As far as I could see, there are no potential negative social impacts.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the  mean field limit of two-layer neural network in an infinite-dimensional parameter space. The main contributions are:
       
- A generalization error bound of the regularized empirical risk minimizer.
       
- Two optimization algorithms for our infinite-dimensional neural network by incorporating the  infinite-dimensional Langevin dynamics in the inner loop of PDA and P-SDCA.

- Theoretical guarantees for the convergence of the two algorithms. 
strengths_and_weaknesses: - Strengths. This is a very nice (and perhaps the first) paper on the training analysis of  two-layer neural network on an infinite-dimensional space in the mean field regime. The extension from finite dimensional setting to infinite dimensions is highly nontrivial.  It is among the best of all papers I have reviewed in NeurIPS 2022. 
    
 - Weaknesses. In my opinion, the paper is strong enough to be accepted. I do not have comments on the weakness, except minor comments which requires some clarifications; see below. 
questions: I only have minor comments and questions. 


- Are the two distributions $P_{\mathcal{X}}$ and $P_{X}$ in line 222 the same?

- Can the author comment on the choice of the Gaussian measure as the reference measure in the regularization? In the finite dimensional case, obviously the flat Lebesgue measure is used as the reference. In the infinite dimensional case, I understand that Gaussians are the canonical choice. However, does the choice of Gaussian measure affect the generalization error? I would image the covariance of the Gaussian reference is essential since it determines the smoothness of functions that the Gaussian measure is supported.  

- Is the Nesterov’s dual averaging essential for the main results (Theorems 3-5) to hold? I am not an expert on stochastic optimization, but is it possible to establish these results if one only use the  gradient information at the most recent step?

limitations: I am not aware of  potential negative societal impact of this work. The author did not discuss the limitations explicitly in the conclusion, but comment on some limitations of the main results after the statements.
review_title: null

Meta-review: metareview: The paper considers the two-layer neural network in the mean-field regime and proposes an algorithm with complexity independent of the input dimension. Overall, I think the paper is very interesting. I recommend an acceptance.
confidence: Certain
Decision: Accept