Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to explore what structural properties account for transition to linearity when a neural network becomes wider. The authors propose a new perspective where the neural network is considered as a multi-level assembly model. Each hidden layer can be viewed as an assembly model built up from a set of independent sub-models. Furthermore, they assume that the proposed assembly model is not dominated by a single or minor group of sub-models. This leads to neurons within the same hidden layer being independent when the network width grows to infinity, and to show that the network is linear in an O(1)-neighbourhood around the initialization.
main_review: The paper is very well written, with proper style and easy to follow. The findings look interesting and expand the work of other manuscripts. My main concerns are as follow:

1. The introduction mainly focuses on the proposed new perspective (assembly model) without much discussion of the benefits of it. This could also be solved at the end of the manuscript with a discussion and limitations section that explains better which future work does this work provide.

2. The related work at the end of the introduction is a bit short. I think further discussion of the similarities and limitations of the proposed perspective with works like Liu et al. 2020a and Lee et al. 2019 is needed. I also miss some references to works that analyse the mean field gradient descent dynamics of training neural networks in the large-width limit and the relation to the submitted manuscript.

3. The key finding of this manuscript is "[...], when the network width is large, the neurons within the same layer become essentially independent to each other in the sense that their gradient directions are orthogonal in the parameter space.". This is linked to assembling independent sub-models to obtain the O(1)-neighbourhood linearity. It seems like the proposed perspective relies directly onto Assumption 1. However, my impression is that if Assumption 1 is not there, other properties could emerge that explain the transition to linearity. The authors start by fixing the property of the neural network being an assembly model and work from there, but does that imply that there are no other structural properties that could explain the transition to linearity on wide neural networks? Is it guaranteed that the constructed assembly model is not dominated by one or a minor group of sub-models?

Minor typos:
page 3: Further more --> Furthermore
pages 4, 5, 7 and 8: the the --> the

summary_of_the_review: The paper looks solid and well written, although the contribution and usefulness of the findings is not made totally clear. I am not an expert on this field, thus some more relevant related work might be missing. Most of the mathematical formulation seems correct, although I might have missed some details. My main concern is the way in which the findings are presented seem to point as to the proposed perspective (assembly model) being one of the explanations under Assumption 1, but not necessarily the only one, or valid when said assumption is not true.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors present a theoretical understanding for the "transition to linearity" phenomenon of wide neural networks with linear output layer under gradient descent. Presented line of thought develops from the perspective where one considers a neural network as a recursively-built assembly of sub-models which correspond to individual neurons from the preceding layers. Authors prove that the transition to linearity phenomenon when the number of sub-models converge to infinity (i.e., the network becomes wider) is a result of assembling weak sub-models that do not dominate the assembly from network initialization.

main_review: Authors present a very interesting and novel theoretical contribution to the understanding of contemporary deep learning architectures with increasing network width. Contributions of this study neatly follows the line of investigations presented by [Lee et al. 2019], and [Liu et al., 2020a]. The paper is very clearly written, and I find this work to be of interest for presentation at ICLR.

The paper provides a rigorous theoretical understanding of the phenomenon, while it naturally lacks empirical contributions and demonstrations overall. Some comments that the authors can discuss/include:
- What does the results tell us about the impact of common practices in network regularization (e.g., with dropout)?
- Any discussions on how can the performed assembly analysis extend to residual networks (e.g., WideResNets)?
- Is it possible to perform any synthetic data based empirical/numerical validation (e.g., for wide neural networks with linear output and a bottleneck) of the finding on the orthogonality of the gradient directions for neurons within the same layer of a network with large width?
summary_of_the_review: The paper is well written and presents a novel theoretical contribution to the understanding of the recently highlighted phenomenon on how wide neural network architectures with linear output layer inherently evolve as linear models. I find this work to be of good quality in terms of theoretical contribution, whereas on the other side empirically weak. My rating would be to accept the paper for presentation at ICLR, as long as the authors can present some numerical verifications or discussions on the how to make an understanding of these results for a broader audience.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to explain the phenomenon that neural networks with infinite width tend to be linear in the neighborhood of initial optimization points, from an assemble model perspective. The key point is to prove that the quadratic term from Taylor expansion disappears when the width approaches infinity. The authors first proved in the simple two-layer case and then extend the results to deep neural networks with L layers.
main_review: Strength:

The authors managed to providing a new perspective on why the wider neural networks tend to be linear, which looks interesting and also consistent with previous results. They also extend the basic results to more complex multi-layer cases. 

As this paper is out of my expertise and purely theoretical, I hope other reviewers with more background can comment more on its novelty and significance.



Weakness:

It's not clear to me how the main theoretical results compare against previous methods on linearity of neural networks. Taking Theorem 1 for example, is the convergence term s(m) faster or slower when compared with previous results? Another general comment is that a specific section on "related work" may make the connection with previous work more clear. 
summary_of_the_review: This paper provides a new perspective towards an interesting observation in deep neural networks, while the connection with previous work could be stated more explicitly.
review_title: null

Meta-review: The authors provide in this manuscript a theoretical analysis to explain why deep neural networks become linear in the neighbourhood of the initial optimisation point as their width tends to infinity. They approach this question by viewing the network as a multi-level assembly model.

All reviewers agree that this is an interesting, novel, and relevant study. The paper is very well-written.

Initially, a weak point raised by a reviewer was that an empirical evaluation of the theory was missing. The authors addressed this issue in a satisfactory manner in their response.

In conclusion, this is a strong contribution worth publication.
Decision: Accept (Spotlight)