Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors studied the empirical phenomenon of condensation during NN training, where the weight vectors of individual hidden neurons in a two-hidden-layer network concentrate in a few directions. The authors demonstrated empirically that the number of concentrated directions is related to the neuronal nonlinearity and specifically its multiplicity. The author then supplied a theoretical analysis under some heuristic assumptions of how this relationship arises.
strengths_and_weaknesses: Strength: the writing is very clear. Concepts, assumptions and definitions are all well-defined, and the empirical and theoretical claims are clearly stated. 
The empirical results connecting the number of condensed directions and the multiplicity of the nonlinearity seem very convincing, at least in the settings that the authors explored. 

Weakness: My main critique is that I'm not sure that the studied regime is an interesting one. If I am interpreting Fig.2 correctly, the authors are suggesting that when using the tanh nonlinearity, the network condenses to an effective network with just two hidden neurons in this initial stage. The trained networks in practical settings will need much bigger effective sizes to fit training sets, and it is not clear to me how behaviors in this early, condensed regime are related to generalization and other phenomena of interest later on in training. The authors appear to suggest that condensation is important because condensation = smaller effective networks = fewer parameters = less overfitting. But first of all fewer parameters does not equate less overfitting, given what we now understand about various double descent phenomena. Second, while the fully trained networks may have effective sizes smaller than their real sizes, the effective size is almost certainly not as small as 2, as suggested by the tanh analysis.
questions: I would appreciate if the authors can better elucidate how the very condensed stage that the authors study is related to expressive, fully trained NNs.
limitations: See above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Understanding how the weights of a neural network converge during training is an important task in deep learning. In this paper, the authors empirically investigate this problem for deep neural networks at the initial stage of training. Through a set of experiments on both synthetic and realistic data sets, they show that for small enough initialization, the orientation of the weight vectors to hidden neurons converge in a maximal number of directions as given by twice the multiplicity of the activation function. Here, the multiplicity is defined as the integer $p$ for which the derivative of the activation function at $x=0$ is zero up to the order $(p-1)$, and is non-zero for the $p$-th order. 

Then, they provide some theoretical support for their experiments by presenting a preliminary analysis, in which they consider a Taylor approximation of the activation function at zero and show that, in the limit of vanishing initialization, convergence of orientations at early stage of training basically implies the alignment along a fixed number of directions as specified by the multiplicity. This analysis is done for activation functions with multiplicity 1 and high dimensional data, and for general multiplicity with 1 dimensional data.
strengths_and_weaknesses: **Originality**
Previous work [Luo et al 2022] empirically observe that for a two-layer ReLU network with small initialization, the weights of hidden neurons condense on certain directions. However, it was not studied why it happens, and how it behaves for other activation functions. This paper fills this gap by studying the phenomenon for deep nets with smooth activation, and providing some intuitive explanation for why it happens.

**Quality**
The paper is easy to understand. However, there is not much discussion on the motivation and significance of their finding. For example, why it is important to study vanishing initialization if it is not even used in practice, how the behavior of the network at the initial stage of training could be informative for generalization where we only care about the final solution (the later stage of training).

**Clarity**
The writing is mostly clear.

**Significance**
I find the connection between the maximal number of condensed orientations and the multiplicity of activation interesting. I hope that this observation could be helpful for future researches in understanding implicit bias of gradient descent in the early stage of training. 
questions: Comments and questions:
- There is some gap between the theoretical and empirical evidence: the theoretical analysis of the paper is done for gradient flow, but all the experiments seem to be done with Adam. How do the results look like if you use gradient descent (full batch size) with a small learning rate?
- It is mentioned in the introduction that condensation of orientation implies networks with small complexity. I agree that if the hidden neurons have identical directions then one can use the homogeneity of ReLU to rewrite the output and obtain a narrower network. However, this seems not possible for smooth activations. Thus, what are exactly the implications of condensation studied in your paper? What kind of generalization bounds can be proved for condensed weights?
- If my understanding is correct, the proof of Theorem 5.1 requires an implicit assumption, that is, the weight orientation must converge, and the convergence must occur near the initialization so that all the weights are still very small and the Taylor approximation is valid. However, I feel that this assumption is quite strong. It is not even clear a priori why the direction of the weight vectors should converge for this setting.
limitations: Limitations are discussed above. I do not see any societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the condensation phenomenon qualitatively in neural networks with various activation functions. The number of condensed directions can be at most twice the multiplicity of the activation function, which is shown empirically and some theoretical calculations are given to back up. 


strengths_and_weaknesses: Originality: As indicated by the authors, condensation is analyzed before for example by Maennel et al. The paper generalizes the analysis for multiple activation functions. 

Quality: Various experiments are presented to study the phenomenon in models from ResNets to two-layer models, to a six-layer network with skip connections. The problem is with Theorem 5.1, although the calculations are informative, they should not be called theorems as they are not rigorous. 

Clarity: Writing is coherent, but some individual sentences have slight English issues. The figures are nice and illustrative. 

Significance: I think it is interesting for the NeurIps community to understand the initial training dynamics in neural networks. This can yield new inductive biases and training algorithms; also relevant for pruning, compressed networks etc. 


questions: Can the authors comment on the difference between Figures 2 and 4 for the sigmoidal and softmax regarding the number of condensed directions? tanh is an odd function, so it makes sense to get $u$ and $-u$, but it is less clear for the other activation functions. 

line 64: missing citations! The so-called embedding principle is identified first by Fukumizu and Amari [1] for the addition of one neuron, then generalized in Fukumizu et al. [2] and even more the construction and scaling of such manifolds of critical points is studied in Simsek et al. [3] from the perspective of permutation-symmetry. 

line 101: dose -> does

line 103: "can across" unclear what is meant in this sentence

line 184: "It is reasonable" -> "it is reasonable"

line 220: "suppose we only consider the leading term" -> this is in no way a theorem statement, please remove the theorem format from the calculations. 

I'm willing to increase my score if my concerns are addressed. 

[1] Fukumizu, Kenji, and Shun-ichi Amari. "Local minima and plateaus in hierarchical structures of multilayer perceptrons." Neural networks 13.3 (2000): 317-327.

[2] Fukumizu, Kenji, et al. "Semi-flat minima and saddle points by embedding neural networks to overparameterization." Advances in neural information processing systems 32 (2019).

[3] Şimşek, Berfin, et al. "Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances." arXiv preprint arXiv:2105.12221 (2021).
limitations: Yes.
review_title: null

Meta-review: metareview: Three reviewers recommended borderline accept, borderline accept, and weak accept. Reviewers found the work clearly written and the claims clearly stated. Of particular interest were the empirical results connecting the number of condensed directions and multiplicity of the nonlinearity, where the article was found to fill gaps and provide new intuitive explanations for important phenomena. A main critique in the initial reviews was the interest in the considered setting and the insufficient discussion of the significance of the findings, which was partly clarified during the discussion leading to updated more favorable reviewer ratings. Overall I find the strengths outweigh the weaknesses and hence I am recommending accept. However, as evidenced in the discussion, the article still can improve in some ways and I strongly encourage the authors to carefully consider the detailed feedback of the reviewers for the preparation of the final manuscript. Specific recommendations include that the revised manuscript better elucidates how the very condensed stage that the authors study is related to expressive, fully trained NNs, and includes a clearer discussion of the motivation and significance, and the promised discussion of activation functions and limitations of the theoretical analysis. Following up on the discussion in response to reviewer Mpot, I would like to suggest the authors do not use their proposed title 'informal proposition’ (or 'Theorem’ or 'Proposition’ etc.) unless the statement has a formal proof and instead use titles such as 'Conjecture’ or 'Empirical observation’ if the statement does not have a formal proof. 

confidence: Less certain
Decision: Accept