Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the role of activation functions (via their multiplicity at the origin) in the condensation of neural networks at the initial training stage. Condensation can be viewed as a feature learning process, where the wide network can be described effectively as a narrower network and the input weights condense on isolated orientations during training. This mechanism may provide a plausible explanation for the performance of the wide network. In particular, the paper shows empirically that the maximal number of condensed orientations is twice the multiplicity of the activation function used. Moreover, using polynomial approximations, the paper provides theoretical support in two cases: when the activation function is of multiplicity one and when the input is one-dimensional. 
main_review: Strengths: 
- Provide an interesting explanation of the condensation mechanism through the lens of multiplicity of activation functions

Weaknesses: 
- Theoretical analysis is limited to the case of one-dimensional inputs and activation functions of multiplicity one 
- Experimental demonstrations are somehow limited -- could be further improved by considering how similar phenomenon occurs for a range of other datasets and for other loss functions (cross-entropy, etc.)

Minor comments/questions:
- the focus seems to be on how condensation appears during the initial training stage, but there is no discussion of how initial this stage is (in particular, the selection of the number of epochs in the plots shown in the paper seems unjustified)
- how does the learning rate play a role in the appearance of condensation? Would be nice to at least mention this
summary_of_the_review: This paper provides a study of the role of multiplicity of activation functions in the condensation mechanism of neural network training, claiming that the maximal number of condensed orientations in the initial training stage is twice the multiplicity. Some experimental demonstrations are provided to support the claim but the theory is somehow limited. I believe that the paper could be further improved and will go with a weak reject for now.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the relation between condensation of neurons activations and the multiplicity of the used non-linearity. Essentially, it is found an empirical link between the multiplicity of the activation function and the number of condensation directions.
main_review: The paper aims at giving an intriguing analysis of the properties of NN activations in early stages of learning, which would motivate the intrinsic generalization abilities of deep neural architectures (e.g., when over parametrized).
The paper includes both an empirical and a theoretical analysis.
My major concern with this paper is that I found the experimental session not fully convincing. First, the analysis is limited to a set of specific condition, for which it is shown that a sort of empirical relation is in place. These assumptions should be made very clear in order to avoid overstating and confusion in readers. Second, the paper focuses on claiming to analyze the behavior of the neurons activations at initialization and early stages of learning, with small weights. In this respect, I could find that the experiments use a number of 100 (or 1000) epochs, which seems something different from initial stages of training. An evolution analysis (e.g., showing the behavior over all the epochs) would have been much more informative in this regard. Moreover, while the theoretical analysis assumes small weight values, in some plots (e.g., Fig 4) the results are shown for weight values larger than a threshold (0.4), and this is a bit confusing.

In order to better understand the novelty of this contribution I also suggest considering clarifying the differences between this current work and the work in Luo et al., 2021.

The paper presents several typos and unclear sentences in the current form.

[edit after the revision]
I would like to sincerely thank the authors for the effort in improving the manuscript. I am going to increase slightly my evaluation of the paper. Thanks.

summary_of_the_review: The quality of the paper is hampered by
- not fully convincing and unclear experimental analysis
- low readability due to several typos in the manuscript
- unclear improvement wrt the work in Luo et al, 2021
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This well-written paper takes a step forward in understanding the implicit regularization in neural net optimization. The authors offer empirical evidence that the complexity of the function initially learned by nets is related to the multiplicity of the activation function at zero (i.e., the number of derivatives with nonzero values when evaluated at zero). Then, analytically, they show that all input weights converge towards the same or opposite direction for a multiplicity of 1 (which is the case for common activation functions like tanh, sigmoid, softplus), and that this holds for any multiplicity in the special case of a 1-dimensional hidden layer. Broadly, this work is intriguing, but could stand to benefit from a few improvements, suggested below.
main_review: - The biggest thing that I would like is a better characterization of the experiments they run.  Currently they show multiplicity of 1 with little training and multiplicity >1 at much more training. I want the 2x2: what is the correlation matrix for the p>1 at short time scales and for p==1 at longer time scales? E.g., in Fig 1, does relu net at epoch 61 exhibit the bifurcation seen in the xtanh(x)? And is the xtanh(x) condensing on one before?
  - More broadly, it would be great to see these matrices for a few example networks at a dozen timepoints over the course of training (spanning the periods before they condense to after they condense) -- this could be an informative appendix figure
  - Also, please state what proportion of units are discarded because the l2 norm was below threshold (both in Fig 1 and in all cases)

- Is there any rigorous or quantitative sense of what the authors mean by "initial"? 

- A key component of the intuitive argument and analytic results is that the weights have a very low magnitude upon initialization. It is important that the authors did indeed show that at least for CIFAR10, a network with small initialization achieves comparable performance to a more standardly initialized net and does so in a similar amount of training time. But I was left wondering: What are the implications of these results for the more standard case of larger initializations? Presumably this condensation on one or a few axes is not seen in these cases, but I'd be curious about the authors speculations about this. Do the authors have any suggestion as to whether there is some sense in which the net is effectively lower capacity in those cases?

- Why is it natural to vary the nonlinearity across layers of a single network (e.g., second paragraph on page 5)? This feels relatively unmotivated to me, as that's not a standard approach in the field.

- A minor point that is not necessary here, but would helpful in future cases: It is often clearer to use diverging colormaps for positive and negative values rather than a sequential colormap (e.g., https://matplotlib.org/stable/tutorials/colors/colormaps.html). The fact that beige was an anti-correlation, not orthogonality, took me longer to realize than it should have.

- A few small typos:
- pg 2: "quantitatively theory explanation" --> ?? "quantitative theoretical explanation"?
- pg 4: "full batch expect for..." --> "full batch *except* for..."
summary_of_the_review: This intriguing paper demonstrates, through experiment and analysis, a connection between the implicit regularization of certain networks and the multiplicity of their activation function.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the condensation of weights of neural networks during the initial training stage. It showed theoretically and empirically that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function under the small initialization of weights. This condensation restricts the capacity of NNs at the beginning, working as implicit regularization.
main_review: Strengths:
- The paper is well written, organized, and easy to follow. The experimental part is comprehensive and a detailed visualization of condensation orientations is provided.
- The conclusion that the number of condensation orientations is twice the multiplicity of the activation function gives an possible explanation why small initialization works as implicit regularization at initial training and is insightful for diving into the nonlinear dynamics of NN.

Weaknesses:
- For experiments on real datasets such as MNIST and CIFAR10, the output dimension $d_\text{out}$ is still 1. In that case, what is the target value? Is it just the class index, e.g., 0-9? 
- It is hard to analyse the case of the general  multiplicity with high-dimensional input data. However, as Figure 2 displays condensation orientations for activation functions with the multiplicity>1 such as $x\tanh(x)$ and $x^2\tanh(x)$, I think the analysis of $p=2,3$ can make the conclusion more stronger.
- An extension to more practical cases where the output dimension is greater than one is interesting and needs some discussions. Also, when combined with the softmax function for a classification task, does the conclusion still hold?

summary_of_the_review: This paper presents a possible perspective to explain why small initialization can serve as implicit regularization during the initial training stage and provides insights to investigate the nonlinear dynamics of neural networks. While it can be improved by conducting more experiments as well as analysis of more complicated cases, it is above the acceptance threshold.
review_title: null

Meta-review: *Summary:* Study isolated orientations of weights for networks with small initialization depending on multiplicity of activation functions. 

*Strengths:* 
- Interesting analysis of properties in early stages depending on activations. 

*Weaknesses:* 
- Reviewers found the settings limited. 
- Reviewers found experiments limited.  

*Discussion:*

In response to ejGJ authors reiterate scope of covered cases and submit to consideration that their experiments should be adequate for basic research. Reviewer acknowledges the response, but maintains their assessment (limited scope of theory, limited experiments). KucV found the experimental part limited in scope, the settings unclear (notion of early stage, compatibility with theory), and review of previous works lacking. KucV’s sincerely acknowledged authors for their efforts to address their comments and improving the manuscript, and raised their score, but maintained the experimental analysis is not fully convincing and unclear, and the comparison with prior work insufficient. zuZq also expressed concerns with the experiments and the notions and settings under consideration. They also raised questions about the comparison with standard initialization. Authors made efforts to address zuZq concerns. zuZq acknowledged this but maintained initial position that the article is just marginally above threshold. jDJ5 found the paper well written and the conclusion insightful. However, also raised concerns about the experiments the settings under consideration. Authors made efforts to address jDJ5’s concerns, who appreciated this but was not convinced to raise their score. 

*Conclusion:*  
Two reviewers consider this article marginally above and two more marginally below the acceptance threshold. I find the article draws an interesting connection pertaining an interesting topic. However, the reviews and discussion conclude that the article lacks in several regards that in my view still could and should be improved. Therefore I am recommending reject at this time. I encourage the authors to revise and resubmit.
Decision: Reject