Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents theoretical results on the fluctuations in the mean-field limit. Building on the framework proposed by Nguyen and Pham, the paper goes beyond the strict mean-field limit and derives next-order results about the fluctuations of weights and network output. Specifically, the paper shows that weight fluctuations scale as `1/\sqrt{N}` (for width `N`), and derives a differential equation that describes the fluctuation. It further shows that the network output obeys a CLT-like result, placing an upper bound on the error in terms of an inverse power of the width. Finally, the paper shows that gradient descent close to a global optimum tends to send the variance in the network output to zero.

UPDATE FOLLOWING AUTHOR RESPONSE:

I would like to thank the authors for sharing the results of numerical experiments verifying key results from the paper, and for addressing my and other reviewers' concerns. In light of these results I am updating my score and recommend accepting the paper.
main_review: The paper makes an interesting and valuable contribution to the study of large width networks in the mean-field regime. Making the connection between infinite-width and finite-width networks is worthwhile, and the paper makes a step in this direction by studying the fluctuations around the large width limit in detail. The results are highly technical: They rely on explicit expressions for the weight fluctuations in the mean-field limit, which are expressed in terms of a complicated ODE. This level of complexity is to be expected from second-order analysis. 

The exposition is as clear as can be given the significant technical complexity. The notation is consistent, there is a section devoted to reviewing the existing framework on which these results build, and heuristic explanations are provided in several instances which aid understanding. I do suggest numbering all equations and referring to these numbers in the text to help the reader navigate the text more easily. I also suggest referring to the main theorems in the introduction by number, to help guide the reader toward the main results when reading the later sections.

My main concern is that it is not possible to verify the results of the paper from the main text alone. There are no experimental results to back up the theoretical claims, and the proofs are not in the main text. The authors include useful heuristic arguments for several results, but given the technical complexity these are not enough to be convincing arguments. (See for example equation 3, a 4-line equation that describes the gradient descent evolution of the mean-field fluctuations.) I do believe these results are valuable. I will vote to accept the paper if the authors provide empirical evidence supporting Theorem 5 (including equation 3), and the results of Section 4.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This manuscript proposes a framework for analyzing multilayer neural nets based on the mean field theory, and prooven that gradient descent (GD) in a large-width multilayer network has a variance reduction effect.
This research is important and highly novel because mean-field analysis of "multilayer" neural networks is considered to be very difficult.

main_review: I had a strong interest in the purpose of the manuscript, but unfortunately I did not understand it.
This is because the manuscript was confusing to me. 
From my view point, I think there are the lack of explanation and numerous careless mistakes.
The following is a list of the relevant points.

Lack of explanation:
1.(Line 29) I cannot understand the word "deterministic limit" means what kind of limit, because there are no explanation.

2. (Line 94) What is the meaning of $ j_i $?

3. (Line 97) Is $ J_i \sim Uinf([N_i]) $ means $ J_i \sim N_i $ ?

4. (Line101) What is the meaning of index 2 in $ \partial_2 $?

5. (Line106) There might be no definition and explanation of $ \mathcal{F} $?

6. (Line 142) There seem to be some sentences which have no information, such as the following one.
The initialization assumption is also mild.

Careless mistakes:
1. In Reference, there are no author name for reference [12], [21], and [22].
I don't think this is a response for double blind either, since other papers written by the same author have the names listed.
Is it suggested that the paper is incomplete?

2. There is "Theorem 2" in Line 179. But there is not "Theorem 1" in the manuscript.
Is it also suggested that the paper is incomplete?

3. The words "GD" and "ODEs" are suddenly omitted without formal names.

4. There are some elementary errors in English grammar, such as the following
(Line 20) "helps reducing and" -> "helps to reduce, and"


In addition, since the theoretical results are not verified numerically, the validity of the derivation process cannot be verified by me.
In case other reviewers have not verified the validity of the derivation process, the authors should add numerical experiments.


For your information, here are my attributes.
I have no experience in theoretical analysis of DNNs using mean field theory.
Also, since I was educated in the Department of Physics, I am not familiar with mathematical paper structure.
Since NeuRIPS has participants with various attributes, I think that the influence of the manuscript will be greater if the authors take care to make it readable by people from a wide range of fields.
I am happy if I can understand your nice work.
limitations_and_societal_impact: It was noted around Line 74 in the manuscript.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the fluctuations of a wide but finite multilayer network about the mean field theory limit. The first main result of the paper establishes the form of these fluctuations as a sum of two components, one given by a Gaussian process and another by what the authors call the second-order mean field limit. Additionally, the paper shows that in the large-width limit and near the global optimum, the gradient descent reduces the variance of the network output.   
main_review: The paper addresses the important topic of network fluctuation about the infinite-width limit. The paper proposes a novel solution in the multilayer case and also shows that the (previously observed) output variance reduction effect extends to this case. The paper is well written.  

The two (mutually connected) main issues with the paper are a rather unwieldy form of the main result and the absence of its independent validation. The obtained structure of the fluctuations is quite complex and is only fully described in the appendix. Because of this complexity, it is not obvious how useful is this theorem. The paper is purely theoretical and does not include any numerical experiments or applications. Moreover, the main text contains only a couple of very brief sketches of the arguments, with 50 pages of detailed proofs deferred to the appendix. I have not checked these detailed proofs, so it's hard for me to judge if the results are correct. Accordingly, I think that a journal publication with a comprehensive review process would be more appropriate for this paper. Also, the paper would benefit substantially from some experimental confirmation of its findings - at least the variance reduction effect that should be easy to demonstrate. 
limitations_and_societal_impact: -
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work focuses on the random fluctuation of the multilayer network in the mean-field limit. Leveraging on the neuronal embedding framework introduced in previous work, the authors derived a second-order mean-field limit to captures the limiting fluctuation distribution. In the large-width regime, it is shown that the gradient descent helps to reduce and eventually eliminating the variance when the learning dynamics converge sufficiently fast to a global optimum.
main_review: Originality: It seems that the technique introduced in this work is new and makes a novel combination of previous techniques. The related work is adequately cited and discussed. For example, the authors make a detailed comparison to the related work in terms of networks and optimization settings [1,2].   

Quality: This submission is technically sound. It seems a complete piece of work in progress. Besides, the authors careful and honest about evaluating both the strengths and weaknesses of their work.

Clarity: This submission is clearly written with good organization.

Significance: My main concern is the significance of the result. It extends a similar phenomenon (gradient descent helps to reduce and eventually eliminating the variance) previously shown for shallow networks [3]. Compared with the previous work [3], the loss function setting of this work is more general. However, how the result could be used in the practice or understanding deep learning is unclear to me.

Even though this work focuses on the theoretical perspective, it would be better if the authors could provide some simple (perhaps on toy model/dataset) but insightful numerical verifications. For example, [3] complemented their results with numerical experiments.

[1] Phan-Minh Nguyen and Huy Tuan Pham, A rigorous framework for the mean field limit of multilayer neural networks, arXiv preprint arXiv:2001.11443 (2020). 

[2] Justin Sirignano and Konstantinos Spiliopoulos, Mean field analysis of neural networks: A central limit theorem, Stochastic Processes and their Applications 130 (2020), no. 3, 1820–1852. 

[3] Zhengdao Chen, GrantMRotskoff, Joan Bruna, and Eric Vanden-Eijnden, A dynamical central limit theorem for shallow neural networks, arXiv preprint arXiv:2008.09623 (2020).
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The paper presents theoretical results on neural network training for finite width networks, which extends prior work which assumes an infinite width limit. The theory proves (among other results) bounds on the output variance of standard feed forward networks as a function of network width when trained with GD. Reviewers agreed that the technical contribution is significant enough to warrant publication, however no reviewer had time to go over the 50 pages of proofs provided in the Appendix. Unfortunately, the ML conference system is ill equipped to handle these kinds of submissions (a reviewer for a mathematics journal may have over a year to review a highly technical submission). 
During the rebuttal period, authors provided experimental confirmation of the theoretical predictions for a simple network trained on MNIST which was enough to satisfy reviewers. Although I am still hesitant to accept an almost purely theoretical work where no reviewer had time to go over the proofs, I will side with the reviewers in light of the experimental confirmation provided during the rebuttal period and recommend acceptance. 

Decision: Accept (Poster)