Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content:  In this paper, the authors propose an analytical framework for DNN training with noisy gradient descent and weight decay. For infinite width, the authors have established a correspondence between DNNs trained with
noisy gradients and the NNGP, and for finite width, the authors introduce a finite width correction (FWC). Overall, the paper is quite interesting and well organized. However, I still have several concerns.

While I agree with the authors that this is an analytical framework different from the Neural Tangent Kernel, my major concern is that this paper doesn't show the advantage of the new framework because this paper hasn't provided any theoretical result for convergence or generalization. 

The authors have commented in the introduction that for NTK "deterministic training is qualitatively different from the stochastic one used in practice, which may lead to poorer performance when combined with a small learning rate." However, as far as I know, there are plenty of good works that have considered NTK trained with SGD([1], [2], [3]). The authors may want to comment on these papers. In the infinite width limit, the authors establish a correspondence between DNNs trained with noisy gradients and the NNGP. Actually, there is a line of work that directly studies the infinite wide NN trained with noisy gradients and weight decay regularization in the so-called mean-field regime([4], [5], [6]).  It would be much better if the authors can characterize the difference between NNGP and those papers I have mentioned.

My other minor concerns are as follows:

1. In the eq 2 of this paper, it seems that the probability measure of the parameter is very close to the gaussian with \sigma_{w}^2 variance. Would it be better for the NN to initialize at \sigma_{w}^2 too? If so, whether the parameters will still only make a small change in the distribution space?

2. The analysis in this paper highly depends on the ergodicity of the dynamics. The authors mention that a small ACT of the dynamics implies this key property, especially for the output. It would be better if the authors can explain more about why the ACT can imply ergodicity.
3. There is no explicit theorem, lemma, or proof in this paper, which is quite hard to read or follow. I suppose the finite width correction would be the key theorem of this paper?

References:

[1] Allen-Zhu, Z., Li, Y., & Song, Z. (2019, May). A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning (pp. 242-252). PMLR.

[2] Zou, D., Cao, Y., Zhou, D., & Gu, Q. (2020). Gradient descent optimizes over-parameterized deep ReLU networks. Machine Learning, 109(3), 467-492.

[3] Daniely, A. (2017). SGD learns the conjugate kernel class of the network. In Advances in Neural Information Processing Systems (pp. 2422-2430).

[4] Mei, S., Misiakiewicz, T., & Montanari, A. (2019). Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. arXiv preprint arXiv:1902.06015.

[5] Chen, Z., Cao, Y., Gu, Q., & Zhang, T. (2020). Mean-Field Analysis of Two-Layer Neural Networks: Non-Asymptotic Rates and Generalization Bounds. arXiv preprint arXiv:2002.04026.

[6] Tzen, B., & Raginsky, M. (2020). A mean-field theory of lazy training in two-layer neural nets: entropic regularization and controlled McKean-Vlasov dynamics. arXiv preprint arXiv:2002.01987.






review_title: An interesting paper but has some flaws

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper shows a correspondence between deep neural networks (DNN) trained with noisy gradients and NNGP. It provides a general analytical form for the finite width correction (FWC) for NNSP expanding around NNGP. Finally, it argues that this FWC can be used to explain why finite width CNNs can improve the performance relative to their GP counterparts on image classification tasks.

This paper is written in a physics style which makes it hard to read. Many notations are defined in words instead of mathematical formulas. This is quite annoying since it is hard to follow what are scalers, vectors, matrices, functions, random variables, and what are the arguments of the functions.
For the first claimed contribution, I feel that this is a simple observation rather than a significant contribution: the authors just pointed out that the stationery distribution of Langevin dynamics is a Gibbs measure.
For the third claimed contribution, I feel that the authors over-claimed what their results can suggest. The authors showed that the FWC contains the information of the weight sharing structure in CNN, and then cited a 'common practice' to argue that this weight sharing structure will lead to better test error. This doesn't give a convincing argument that finite width CNN will outperform its corresponding GPs.

The derivation of the finite width correction of the NNGP could indeed be a concrete contribution, but the result is presented in a very ambiguous way. For example, the $U_{x1, x2, x3, x4}$ quantity seems to be a very important quantity in defining this finite width correction. However, its definition in Eq. (9) is quite ambiguous. Especially, U_{x1, x2, x3, x4} depend on $\phi_\alpha = \phi(z_i^{l-1}(x_\alpha))$, and I didn't see where does the subscript $i$ come from. The authors used the Einstein convention in subscripts but I don't know what are ranges of \alpha, \beta, \gamma, and \delta to be summed over (is it 1 to n or 1 to 4?). There are many other places where the quantities are not concretely defined and I can only guess what do these notations mean in order to follow the results.

Although the finite width correction of NNGP (if properly written) could be an interesting result, due to the ambiguous writing style of the paper, I don't think it achieves the goal of clearly presenting the results. I will suggest a rejection.

After reading the rebuttal: 
I increased my score to 5. I still feel that the writing style is hard to follow. Besides the examples that I wrote in my original review, there are many other places where the notations and definitions are not clearly written. 
review_title: The presentation is very ambiguous. Overclaimed the implications. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies equivalence between neural networks and Gaussian process. The authors make three main contributions: (1) they prove that infinite-width DNNs trained with noisy gradients are corresponded to neural network Gaussian process (NNGP), not neural tangent kernel (NTK); (2) they prove that the finite width correction (FWC) for DNNs with arbitrary activation functions and depth has a general analytical form which can help predict the outputs of empirical finite networks with high accuracy; and (3) they show how these FWCs can improve the performance of finite convolutional neural networks (CNNs) on image classification tasks.

Overall, I vote for accepting. The paper establishes solid correspondence between neural networks and Gaussian process which may have a real impact to applications. The authors also give empirical results to support their arguments.

Pros:

+ The paper takes one of the most important problem in deep learning theory – the correspondence between neural networks and Gaussian kernels.

+ The paper makes real contributions compared with existing works on NNGP (in NNGP, weights were determined by the distribution of the DNN weights at initialization; and in this paper, the weights are sampled across the stochastic training dynamics).

+ Give results for finite-width networks, while many works only give asymptotical analysis.

+ Explain why finite CNNs outperforms infinite CNNs and quantify the difference.

Cons:

- The presentation can be improved. For example, Sections 2 and 3 would better if the contents can be further organized into smaller parts.

Questions: 

* How different is the techniques used in the proves comparing to existing results? I have gone through the proves and the techniques wherein seem similar to existing works for me. Could you please give detailed proof skeletons in the manuscript and discussion the differences with related works?
review_title: Interesting and new correspondence between neural networks and Gaussian kernels

final_score: rating: 4: Ok but not good enough - rejection
review_content: It has been show that practical performance of finite width DNNs deviates from the infinite limiting cases of both the NTK and GP. The reason behind this is still not well understood. This papers develops theory trying to explain this by showing the finite width correction (FWC) for DNNs. The results look very interesting. However, as an educational guest on this topic, I find the paper written in a way that causes a lot of confusions, making me not fully understand the results.

First of all, the paper is talking about the FWC. Although it has been used in previous work and I can get some sense of the meaning, I think a formal definition is needed in order to make the paper more readable.

Regarding the technical details:

1. What is Df in eq.4? And why can it be eliminated? And it is not clear to me about the sentence below eq.4: why the rhs of eq.4 equals the kernel of the NNGP only in highly over-parameterized DNNs?
2. I am not sure how eq.5 is obtained?
3. I think the description below eq.5 can be made more formal because this gives the definition of NNSP. Based on the current description, I still cannot get why NNSP is special? Is it still a GP and why?
4. Second line below section 3.1, it says e^{-L[f]/2\sigma^2} is independent of the DNNs, but why? I think f corresponds to the DNN?
5. In Section 3, the paper derives complicated formulations for the posterior mean and variance. I am not sure how these results are useful? It seems to say that these results describe how the finite width DNNs are deviated from the NTK and NNGP, but by looking at these formulas, it is hard for me to figure out why NNSP is better than NTK and NNGP?
6. Paragraph above Figure 1: it says "the above cubic term would lose its explicit dependence on n", but I think there is still a quadratic term, which still depends on n. How can it say that the FWC is negligible in the large n regime?

Overall, I found this paper addressing an important and interesting problem, but the current presentation cannot convince me a pass.
review_title: Interesting theory on finite width DNNs but not clearly explained

Meta-review: This paper develops an interesting new angle on the behavior of large-width neural networks by elucidating the connection between the NNGP and noisy gradient descent and by examining finite-width corrections through an Edgeworth expansion. While these contributions are important, the paper would better serve the community if its presentation were significantly improved before publication. The main issue is not one of presentation style -- papers with physics-style prose are welcomed and appreciated at ICLR -- but rather one of presentation substance. In addition to the various specific points raised by the reviewers, I would add that the figures and captions are difficult to interpret, the experiments need a more in-depth discussion, and the notations should all be defined at the time of their introduction, among other things. For these reasons, I cannot recommend accepting the paper in its current form, but I hope to see a more polished version of the manuscript at a subsequent conference.
Decision: Reject