Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies temperature-dependent differentiable approximations to the ReLU activation function. The paper derives the critical initialization for this approximation at all temperatures and shows empirically that these smooth activations have similar NTK to ReLU under their proposed initialization.

strength_and_weaknesses: Strengths:  
1. The paper derives critical initializations for many commonly used relaxations of ReLUs like SWISH and GeLU. The derivation seems correct.  
2. The proposed initializations train better than He et. al. initialization at all temperatures in the experiments with MLPs.   

Weaknesses:  
1. Experiments: It would have been useful to see how these initializations perform for realistic tasks compared to standard initializations used in practice. Activations like Swish are being used for transformers widely and it would be great if a theoretically-motivated initialization can provide improved training dynamics. I understand that residual connections and normalizations add further complexities to the derivation, but it could be useful to see if the initialization matters in the practical settings anyway.  
2. While it is interesting that the NTK updates look smoother for these neural networks, could the authors comment on why we should care about this quantity for practical settings? Is there an analogous quantity outside the NTK regime that could be useful to look at?   
3. I find the impact of this paper to be incremental. Theoretical impact: While it is nice that these activation functions are smooth and have nicer NTK properties, I am not sure if these highlighted properties are the theoretical reason these activation functions are preferable to ReLUs in practice. Practical impact: the proposed initializations could potentially lead to faster training dynamics to He et. al., but that has not been demonstrated in this paper i.e. they show that He et. al. behaves differently for different temperatures, but the overal dynamics are not faster than He et. al. The other remaining reason this paper could be impactful is if it develops new techniques in their derivations, but I am not well versed with the literature enough to know if so.

Minor comments. 
1. Would help readability if you can relabel axes in figures with what the quantity indicates. For eg: In Figure 3, replace n with ‘width’

clarity,_quality,_novelty_and_reproducibility: Clarity: The figures in the paper could have been made clearer to convey the main takeaways
Novelty: The problem of critical initializations for differentiable ReLUs has not been addressed in prior work to my knowledge
summary_of_the_review: While it is useful to know the critical initializations of relaxations to ReLUs being used in practice, the paper does not show that these initializations are practically useful compared to standard initializations. The nicer properties of these activations under their proposed initialization are interesting, but again do not give us much theoretical or practical insight.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper compares the ReLU activation function to its many smooth variants that converge to it in the limit of low temperatures. The paper proposes to investigate the role of these two sub-classes of activation functions by looking at their gradient propagation at initialization time. In particular, prior work has shown that there are some initializations that produce the known phenomenon of exploding/vanishing gradients. In addition, there are specific initializations where the magnitude of gradients stays constant on average throughout the network. When the latter is possible the paper defines it as a criticality. In this work this property is studied for a number of variants of smoothed ReLUs and how it changes with temperature. The paper proves the sufficient and necessary conditions for a smoothed ReLU kind of function to have a critical initialization scheme, and then examines how it manifests for the various activations, including whether this property is stable or not (stable for small pertubations). The paper continues with examining the various activation function through the lens of Neural Tanget Kernel, and use that to argue when smooth ReLU follows similar properties as the non-smooth ReLU. Finally, the paper examines how these activation functions affect the training dynamics, showing a very different behavior of the smooth vs. the non-smooth case.
strength_and_weaknesses: Strengths:
* With an evergrowing zoo of activation functions, often very similar to each other, there is a growing need for a crisp theory to understand their inherent properties, and when one should prefer a specific one. The family of ReLU-like activation functions is vast, and this paper proposes a method to study a large set of them if they can be represented as $f(z, T) = z \cdot a(\frac{z}{T})$ for some sigmoid function $a(z)$ that converges to the Heaviside function for $T \to 0$. Such a general result could have a major impact on how activation functions are used and how new ones could be designed.
* The paper also examines 3 novel kinds of smooth ReLUs based on this generic definition, and examines their properties. This demonstrates the breadth of the theoretical framework.
* As a side note, I suggest the authors to consider adding to their table the Softplus function (the "original" smooth alternative to ReLU). It too can be put into the studied form by rewriting it as $Softplus(z) = z \cdot a(z/T)$ for $a(x) = \frac{\ln(1+\exp(x))}{x}$, and it can be shown that $a(x)$ converges to the Heaviside function as the temperature goes to zero, but $a(x)$ is not a sigmoid function. It is unclear in your analysis if $a(x)$ being a sigmoid is critical to your results or not, and having a function that does not follow this pattern could shed more light.

Weaknesses:
* The paper has serious clarity issues, starting with not giving a clear definition of the core expressions in its analysis. For example, a very loose definition of $< \cdot >$ is given in the text, and the definition for $< \cdot >_K$ is completely absent. One has to read one of the cited works [1] to understand what it means, and the paper doesn't even bother directing the reader to do so. Moreover, after examining the other paper that defines these terms, it raised even more questions. This is because only at the end of section 2.1 is it apparent that the prior definitions were for the limit of infinite width/depth. A much more in-depth background is needed to explain how this limit is taken. Similarly, NTK is poorly introduced, giving no insight into its definitions and why the reader should care. This paper essentially requires you to be well versed in the literature on NTK and the works on the limit of infinite NN as Gaussian processes.
* Given this point, I had a hard time understanding and verifying the correctness of this work. Therefore, I cannot judge whether the analysis is accurate or evaluate its limitations.
* Furthermore, due to the various clarity issues, many of the results are essentially left unexplained and unmotivated. This is too bad, as beyond the point on criticality, which I understood, I cannot really say I understood the point of the later sections that are based on NTK, which is a shame as I'd like to believe there might be interesting insights there.
* Of a minor note, it appears that the style file was modified. I'm not taking this into account in my rating, but simply pointing this out for the AC for their consideration.

[1] - B. Hanin. Correlation functions in random fully connected neural networks at finite width. arXiv preprint arXiv:2204.01058, 2022.
clarity,_quality,_novelty_and_reproducibility: Clarity: A lot to be desired. See weakneeses above for details.

Quality: Difficult to judge given many clarity issues. Of the little I could verify, I didn't find any mistakes.

Novelty: The results and analysis build on prior works (NTK and NN as Gaussian processes), but the specific application on smooth ReLUs is novel and could have significant impact on practical usage.

Reproducibility: Again, clarity issues make this not very reproducible.
summary_of_the_review: Due to the severe clarity issues, I cannot recommend acceptance. It could be that a reviewer that is more versed in the prior works this work builds upon could evaluate the merits of this submission independently of its presentation, but I am (sadly) unable to do so. I hope the authors will take this criticism and revise their manuscript to be accessible to a broader readership, as what I did understand I liked.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers several smooth relaxations of ReLU. The smoothness of these networks is controlled by a parameter called “temperature”. The authors theoretically analyze fully-connected networks with these activations and derive temperature-dependent critical initialization schemes. Namely, initializations such that the values in different layers in a deep network do not explode or vanish. The authors also demonstrate empirically that representation variance and NTK of smooth ReLUs resemble those of ReLU under the stable initialization scheme we provide, for any temperature, and show that smooth-ReLUs typically have stable and continuous NTK updates during training, while ReLU shows stochasticity instead. 
strength_and_weaknesses: The subject of understanding which initialization schemes work well for smooth relaxations of ReLU is well-motivated and interesting.

I read the paper, but unfortunately, I didn’t understand it (starting from Section 3). There are two possible reasons:

(1) It is my fault, namely, I am not sufficiently familiar with this subject and/or did not devote enough time to understand the paper.

(2) The paper is not well-written.
clarity,_quality,_novelty_and_reproducibility: I suspect that the paper is not sufficiently well-written.
summary_of_the_review: I recommend weak-reject with low confidence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies several smooth relaxation to ReLU activation function which is controlled by a temperature parameter. For fully connected networks, authors theoretically analyze these activations and derive temperature dependent critical initialization schemes. Criticality is based on forward and backward pass of deep networks not to explode or vanish. Authors examine these activation functions using Neural Tangent Kernel and study training dynamics between smooth and non-smooth cases. 

Strength
- Theory of activation is well motivated.
- Theoretical breadth by covering 3 novel kinds of smooth ReLU and derive critical initialization for commonly used relaxation of ReLUs. 
- Well motivated and interesting problem of understanding initialization of networks with smooth relaxation of ReLU
- Empirical benefit over He initialization at all temperatures in the experiments with MLPs


Weakness
- Severe clarity issues: One of the reviewers had a hard time understanding the paper and other reviewers also raised serious clarity issues. This is exacerbated by the fact that many concepts are not introduced and rely on familiarity with NNGP and NTK. 
- Many results are left unexplained or unmotivated
- Issues with practicality and impact: Experiments are not done in realistic tasks used in practice and only proven useful in toy-ish settings. Theoretical justification doesn't seem to explain why smooth versions of ReLU are used in practice. Also practical impact hasn't been fully demonstrated. 

justification_for_why_not_higher_score: There is a clear clarity issue the authors need to work on, which was raised by two of the reviewers. Although, one reviewer with expertise in the topic could understand the main points, the paper as is required significant knowledge of related works and not applicable for a broad ICLR audience. Beyond clarity, reviewers raised issues that many results are unexplained or unmotivated. Finally, there is weakness on practicality and impact.

In the end, all reviewers rated the paper as below acceptance threshold thus not ready for publication at ICLR at this moment.
Decision: Reject