Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary 

The paper proposes two modifications to SELU activation function to improve it with regards to preserving forward-backward signal propagation in neural networks. The work builds on top of the mean-field theory literature and provides a modified self-normalization property (additional constraints compared to SELU). Further, it discusses some heuristics (mixup, weight centralization) to improve performance in practice.

## Strengths

1. The problem is interesting and important to improve trainability in neural networks. It is shown recently that SELU suffers from gradient explosion and this work attempts to circumvent it by redefining the self-normalization condition. This condition leads to one more scalar parameter in SELU function and discusses a method to obtain this scalar.

2. Experiments are conducted on a 56-layer CNN for cifar and tinyimagenet datasets and modified mobilenetv1 for imagenet. These proposed modifications to SELU seem to result in improved accuracy even though the improvement is minimal on imagenet.

3. Overall the paper is clearly written.

## Weaknesses

The main weaknesses of the manuscript in my opinion are as follows:

1. There seem to be multiple gaps in the theory and the modifications do not mitigate gradient explosion:
	- The theoretical analysis lacks rigor and the main purpose of mitigating the gradient explosion issue of SELU is not sufficiently addressed. In fact, if I understand correctly, even with the redefined self-normalization property gradient vanishing/explosion can occur. Note, according to Eq. 15, when $q_l <1$, the numerator can grow arbitrarily. Please clarify.
	- Related to the above point, the 3rd condition in Eq. 12 and the subsequent argument of gradient norm converges to the fixed point is unsubstantiated. To my understanding, Eq. 15 only gives a lower/upper bound depending on the value of $q$ and it does not convey anything regarding convergence to such a fixed point. If it does a rigorous proof would be required.
	- The assumption in Eq. 4 is unjustified. In Fig. 3 there is a plot provided but it has several underlying assumptions (eg, small $\epsilon$ and $E[x^2]$ etc.) and it is not clear what functions would satisfy this assumption. I recommend the authors to look at Gaussian-Poincare inequality for connecting a function and its derivative under the Gaussian distribution. In fact, GP inequality is recently used to define a class of activation functions to solve gradient vanishing/explosion in neural networks [a].
	- The condition on $\lambda$ (Eq. 19) is derived without any theoretical basis and it seems like it is found empirically. This also weakens the theoretical contribution of the paper.

2. Contradicting arguments compared to dynamical isometry and mean-field theory literature:
	- In dynamical isometry and mean-field theory literature it is shown that dynamical isometry is sufficient to ensure stable gradients and for RELU the scalar is computed to be $\sqrt{2}$. Refer [b]. To this end, it is confusing to me when it is mentioned that this condition "will lose self-normalization" in the second last para of page 4. Please precisely define what is meant by self-normalization (including its purpose) in this paper and also clarify this confusion.
	- In the literature it is known that when the width is large, the preactivations will be close to a 0 mean Gaussian distribution due to the central limit theorem (Assumption 2 in this paper). However, in 3rd last para of page 7, it is mentioned that "networks with large fan-in are more likely to lose the self-normalization effect". This seems to be contradicting as well.

3. Discussion about the mean of activations:
	- The discussion about the mean of activations exploding in Sec. 6 seems irrelevant to me. In the theoretical derivation of this paper, I could not find any discussion on the mean of activations affecting any of the desired quantities such as $q$ or $\phi(q)$. Please improve the clarity on how this is connected to the theory of this paper.

4. No experiments on gradient vanishing or exploding behaviour:
	- As far as I understood, the main motivation is to fix the gradient exploding issue of SELU. However, there is no experiment showing this effect and this seems to reinforce my concern that the proposed modification does not solve the gradient vanishing/exploding issue (also mentioned briefly in the para before conclusion). This questions the significance of the contributions.
	- I understand that generalization error is the quantity that we mostly care about. However, self-normalization or the work in mean-field theory literature concerns on trainability. Therefore, I believe, it is important to show the trainability behaviour and the propagation of gradients. Note, the theory does not convey anything about the generalization error directly but the signal propagation in the forward and backward directions.

In retrospect, I just want to say that in the current form theory and heuristics are mixed together making it difficult to see where the benefit is coming from and I think if theory is tightened this could be a good paper. 

## Minor Comments

1. It seems the modifications could be done to other activation functions as well such as tanh etc. Please consider.
2. Please explain what is the meaning of $\epsilon$ in Eq. 10. This is not clear from this paper.
3. First para in Sec. 7.1 : "considerably higher"

## References

- [a] Lu, Y., Gould, S. and Ajanthan, T., 2020. Bidirectional Self-Normalizing Neural Networks. arXiv.
- [b] Pennington, J., Schoenholz, S. and Ganguli, S., 2017. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. NeurIPS.
review_title: There seem to be multiple gaps in the theory and the modifications do not mitigate gradient explosion

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose a new definition for the self-normalization property of a network. Using this definition, the authors propose two new activation functions: sSELU and lSELU. Performance of the new activation functions is tested on benchmarks like CNNs on CIFAR10/CIFAR100/Tiny-ImageNet and MobileNet on ImageNet.

The new definition of self-normalization is interesting and could potentially be useful. The paper is also written reasonably clearly (see below for minor comments on how I think the writing could be improved). However, I think the experimental section of the paper using the two proposed activation functions have a number of limitations (which I describe below) that explain my rating. I am happy to raise my score if the authors address some of my concerns.

Main comments/questions:
- In the end, it seems like solving the problem of the shift in the mean is the main thing that makes SELU, sSELU and lSELU work on MobileNets. Is there a problem of gradient explosion in SELU apart from growing means that sSELU or lSELU solves, and if so, what is an experiment that shows this? Without this, it is not clear to me whether the slightly better results of sSELU and lSELU is simply a result of sub-optimal hyperparameter tuning of the baselines (see comment below).
- The BN+ReLU baseline run on MobileNet seems a bit suboptimal to me. The original MobileNet paper reports 71.7% accuracy, and other work reports even higher numbers, e.g., 72% in https://arxiv.org/pdf/1710.05941.pdf. This could be a result of the learning rate not being tuned for any of the activation functions. It is not obvious to me that the optimal learning rate should be the same across all activation functions, so ideally optimal performance should be shown over a learning rate sweep for each method.
- The authors mention that neural networks with larger fan-in are more likely to lose the self-normalization effect on the mean based on the observation that if the mean is less than 1/N, then multiplication with the weight decreases the mean of the pre-activations. I do not however follow this argument (and don't think it is true), since the weights are typically initialized from N(0, 1/N), and therefore the probability that the mean is less than 1/N does not change with larger N?
- How sensitive is performance to epsilon? The fact that epsilon needs to be tuned seems to be a major disadvantage of the proposed activation functions.
- It would be interesting to see the following ablations as well for the MobileNet problem:
    - SELU + Weight Centralization
    - dSELU + Mixup
    - dSELU + Weight Centralization

Other minor comments:
- It would have been good to report the values of the parameters alpha, beta and lambda of sSELU and lSELU for at least one problem (such as the MobileNet experiment), and compare them with SELU.
- Would be good to add SELU in figures 1b, 1d and 3.
- The authors mention in definition 2 that the self-normalization property is stronger when phi(q) is closer to 1/q. Can the authors expand on why this is? It is not obvious to me from the definition.

Additional comments on the writing:
- It is not clear to me how the new definition is "easier to use both analytically and numerically".
- Explanation of related prior work should be improved, as it is not quite clear right now. In particular, the explanation of the prior work in mean field theory, as well as the explanation of "gradient norm equality".
- The first paragraph on the second page needs to be checked for a number of typos (Forbenius -> Frobenius, etc) and grammatical errors.
- Typo at the end of page 2: boarder -> border
- Equation 13 and 15 need to be made clearer by adding "if q_l > 1", etc.

----------

Update after rebuttal: I thank the authors for the detailed response. Some of my comments have been addressed. However, some of my major concerns remain such as the insufficiency of the hyperparameter tuning for the baselines to do a fair comparison. I am keeping my score.
review_title: Interesting results but needs better experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Update

I thank the authors for extensive replies and updates to the paper. Most of my questions are answered, and the paper quality is substantially improved. I would not be opposed if other reviewers recommends to accept it. Unfortunately I still can't raise the score and advocate for it myself, since:

1) Table 2 doesn't really show superiority of new nonlinearities, since the bolded (bottom) entry has both trainable $\lambda$, centralization and Mixup, while the baseline of dSELU only has mixup and trainable $\lambda$, no centralization. Without centralization (Mixup + trainable $\lambda$), lSELU/sSELU/dSELU perform comparatively. Without trainable $\lambda$ (only Mixup), they perform a bit worse than dSELU. With only centralization, BN is still better. Further, even after rebuttal, I still believe that making $\lambda$ trainable effectively cancels the preceding theoretical discussion, and makes this subset of results somewhat unrelated to the main idea of the paper. Finally, Table 1 shows a more robust benefit over dSELU on CIFAR-100, but not on CIFAR-10/TinyImageNet (where s/lSELU can be both better and worse than dSELU), which is in my opinion underwhelming given the added implementation complexity.

2) Figure 4 is very promising, but I find that SELU doesn't look that bad on it, which again makes me wonder whether the new improved self-normalization actually matters in Tables 1/2, especially given gradient clipping and other heuristics in Table 2.

So at the moment I find that the proposed nonlinearities are promising in terms of both self-normalization (Figure 4), and generalization (Table 1/2/6), but these results appear to be largely unrelated to each other, and neither of them in separation is strong enough to convince me to try s/lSELU over dSELU (given additional implementation complexity + the boolean hyper-parameter of whether to use lSELU or sSELU) or over BN (given additional hyper-parameter $\epsilon$). I wish the paper either showed clear use-cases where one can't train BN/SELU/dSELU networks at all in reasonable time (but new nonlinearities allowed it due to superior normalization), or more robust generalization results.

Original review below:

# Outline 
The paper proposes two new nonlinearities designed to normalize the second moment of activations and the Frobenius norm of gradients, and therefore avoid exploding/vanishing activations/gradients. The nonlinearities are evaluated on multiple image datasets. 

# Review

On one hand, I find the theoretical motivation and the idea of adding the new minimization constraint compelling, and empirical performance of nonlinearities promising (notably on CIFAR-100). I appreciate evaluation on multiple datasets, and providing native CUDA code. Further, several decisions along the design process appear well-motivated and backed by experiments (e.g Figure 2, 3, 4).

On the other hand, I find the theoretical contribution to be relatively incremental from Chen et al (2020b), and unrelated to the uptick in generalization performance in Tables 1 and 2 (motivation for deriving the nonlinearities concern improving trainability, which is not studied experimentally in the paper; better generalization is a nice bonus, but is not predicted by theory). 

On the empirical side, I believe there are important gaps in the experimental results that leave me uncertain of how robust the improved generalization is, and hence whether new nonlinearities justify the added complexity (I believe the claim that no new hyper-paramaters are introduced is false) over SELU or dSELU from Chen et al (2020b). 

As such, at the time of submission neither theory nor experiments appear sufficiently convincing for me to accept the paper, but I am open for discussion.


Below are my specific questions/concerns.


## Motivation/Theory:
1. Do I understand correctly that Definition 2, “new self-normalization definition”, and what follows within section 4 is effectively the same as “partial normalization” in Chen et al (2020b), section 5.3? In either case, I suggest drawing a more explicit connection with Chen et al (2020b) in this section, highlighting precisely what is novel relative to Chen et al (2020b), and potentially softening/clarifying the first bullet point of main theoretical results in the Introduction respectively.
2. One important weakness of this work is that the paper does not evaluate whether the designed nonlinearity ends up doing what it was designed to do. Namely, we know that in finite networks Assumptions 1-3 don’t hold, Eq. (4) is not exact, and, for $\lambda \geq 1$ $\frac{d \phi(q)}{d q}|_{q=1}$ is clearly far from $-1$ in Figure 1, notably even deviating further from it than dSELU in Figure 1.c. As such I am not persuaded that s/lSELU has better gradient behavior than dSELU in practice. One could demonstrate such benefits in different ways, perhaps by showing that s/lSELU allows higher learning rate in deep networks on some toy task, or comparing how the gradient norms evolve with depth in deep random networks compared to dSELU etc. Further, even in generalization experiments, as mentioned in appendix B, both this paper and Chen et al. (2020b) clip the gradients by $[-2; 2]$, which, from my understanding, further conceals whether these nonlinearities help normalize gradients or not.
3. Making $\lambda$ trainable in s/lSELU is a somewhat disappointing decision/necessity since there is no theoretical reason to do so, and from a quick look at the referenced Zhang et al. (2019) I could not find the justification there - could you please clarify which part of the paper you were referencing? Does $\lambda$ remain constrained to $\geq 1$ when trained? 
4. I would like the discussion about “large fan-in networks are likely to lose self-normalization on the mean” (section 6, 3rd paragraph) to be either elaborated, referenced, or confirmed empirically (e.g. on a toy task with networks of increasing fan-ins). Precisely, the conclusion is reached under the assumption that the mean of weights $\Delta \mu$ (during training) is independent of fan-in $N_{l-1}$, but isn’t this clearly not true? I.e. weights with larger fan-in are sampled with smaller variance, and I expect them to move less and less during training as the network gets wider, hence increasing fan-in could proportionally decrease $\Delta \mu$. In this case these changes cancel out, and the conclusion does not appear justified.


## Experiments:
1. As presented, experiments do not support the introductory claim that “no additional hyper-parameter is introduced". Firstly, I would prefer if the claim was more explicit, i.e. “no additional hyper-paramater relative to dSELU”. Secondly, even apart from $\epsilon$, there are still hyper-parameters of a) which among s/lSELU to use; b) whether to have $\lambda$ trainable or not; c) whether to use Weight Centering, or mixup, or nothing. From reading the abstract, I was expecting a drop-in, no-hyper-parameter activation, but unfortunately there are quite a few knobs to tune.
2. In both Tables, I believe it’s important to have entries with hyper-parameter-free d/s/lSELU as a baseline, i.e. with $\epsilon = 1 / L$, to get an understanding of what order of improvement is gained from the additional $\epsilon$ hyper-parameter. I’m concerned that If $\epsilon$ is necessary, the relative performance of d/l/sSELU against BN/SELU may be less compelling, especially if the best-performing models were selected on the test set (see next question). 
3. In both tables, was a validation set used to select best performing numbers, or were the numbers selected on the test set (please expand appendix B with this detail)? 
4. Table 2: what $\epsilon$ was used?
5. In both Tables, it would be useful to see how sensitive different nonlinearities are to $\epsilon$, so I suggest adding respective tables for all values in the appendix (also noting the specific values of $\lambda, \alpha, \beta$).
6. In Table 2: since Chen et al. (2020b) is the primary point of reference for this work, I would argue that for this Table to be convincing, it should also include “dSELU + Mixup” and “dSELU + Mixup + trainable $\lambda$” (and, ideally,  “dSELU + Weight Centralization”).
7. In Table 2, could you please include results with non-trainable $\lambda$ as well?
8. Was trainable $\lambda$ a single shared parameter for the whole network, or one per layer?

## Mentioned references from the paper:
* Chen et al. (2020b): [A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks](https://arxiv.org/abs/2001.00254)
* Zhang et al. (2019): [Fixup Initialization: Residual Learning Without Normalization]( https://arxiv.org/abs/1901.09321)
review_title: Solid effort, but theoretical motivation and experimental evidence aren't convincing enough

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The authors propose variants of the SELU activation 
function that yield a stronger self-normalization property. The 
analysis in done using mean field theory with several
assumptions on the randomness of quantities in a neural
network. 

Pros:
a) The paper is clearly written and the line of thought can 
easily be followed. 
b) The notation is clean. 
c) The mathematical formulations are sound.
d) The connections between SNNs and the paper series
by Poole et al. and Schoenholz et al. has been made
quite clear. 

Cons:
a) The work has limited relevance due to the 
fact that it is unclear whether the proposed property is 
indeed crucial for learning and that the empirical results
are biased and far from state-of-the art. 
As stated by Goodfellow et al. in their "Deep Learning Book"
(Section 6.3.3), "New hidden unit types that perform roughly comparably to known types are so common
as to be uninteresting." They provide an example using cos() as activation function reaches a test error
below 1% on MNIST. This implies that the machine learning community should be rigorous in the
assessment whether a new activation function is worth publishing in order to avoid drowning literature
about activation functions. Furthermore, the number of potential activation functions for neural networks is
infinite. Therefore, activation function research should be focused around those activation functions with
interesting theoretical properties or -- if no theoretical properties are given -- the new activation
functions should increase the state-of-the-art of predictive performance.
i) The mathematical derivations use several assumptions on the quantities, such as weights and
activations. These assumptions would require commenting on how strong they constrain
the applicability of this theory. The author should comment on how strong those assumptions are
and how they can be leveraged.
ii) It is unclear why a stronger normalization property should improve learning. The authors
should back this more.
iii) Overall, the derivations are rather related to the backward pass. The authors should 
include a view on their activation functions in terms of vanishing/exploding gradients (see also [1]).
iv) The main concern of the reviewer are the experiments. Firstly, SNNs were introduced with
relatively large-scale experiments on fully-connected networks. In order to demonstrate 
improvements, the suggested activation functions shoudl be compared in that set of experiments.
The  presented experiments are done with architectures that are relatively far 
from the current SOTA on CIFAR and Imagenet [2], such that their relevance cannot be judged well. 
The authors should introduce their suggested activation functions into SOTA architectures.
v) The presented performance metrics suffer from a hyperparameter selection bias, since 
the best epsilon-parameters are selected (section 7.1.). The authors should select 
hyperparameters of their method on a validation set. 

b) The novely of this could be stated clearer. Large parts of the derivations are 
similar to those by Poole et al, 2016. The authors should make their theoretical contributions
clearer. 


Questions: 
Questions are implicitly contained in the suggestions above.
- Can you elaborate more on the connection to mixup?
- Skip/shortcut connections typically pose a problem both theoretically and practically [3]. How are they handled in this work?



Minor: 
1) English editing might be required. Should it say "Redefining THE self-normlization property"?


References:
[1] Hoedt, P.J., Hochreiter, S. and Klambauer, G. Characterising activation functions by their backward dynamics around forward fixed points. Critiquing and Correcting Trends in Machine Learning workshop at NeurIPS 2018.
[2] Xie, Q., Luong, M. T., Hovy, E., & Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10687-10698).
[3] Huang, Z., Ng, T., Liu, L., Mason, H., Zhuang, X., & Liu, D. (2020, May). SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6854-6858). IEEE.


review_title: Interesting paper with lacking experiments

Meta-review: This paper proposed two variants of the SELU activation function, termed the leaky SELU (lSELU) and scaled SELU (sSELU), respectively, in order to yield a stronger self-normalization property. The review process and the discussion find the following issues:

- The hyperparameter tuning for the baselines is insufficient for the baselines so that the comparison may be unfair. 
- The experiment results (Table 2) do not show superiority of the proposed activation functions. In addition, the results appear to be unrelated to each other. (see Reviewer 3's detailed update)
- Reviewer 2 pointed out that the architecture that the authors used was far from the SOTA. I read the authors' response. This paper may benefit from adding some even naive workaround and making fair comparisons under the SOTA architecture. 

I do not think (6) is a good way to present this equation. The authors may want to perform change of variable and replace $\sqrt{q}z$ by $z$ in the integral and add this form to the right-hand side of (6). In addition, $\epsilon$ appears in Definition 2. However, when the authors mention the self-normalization property in Definition 2, they omit $\epsilon$. It might be better to call it $\epsilon$-self-normalization property to stress that this definition depends on $\epsilon$.  

Other minor issues:
- The line right below (15) on page 11, the authors did not need to capitalize "orthogonal".
- In eq (16), $W_l^{,T}$, the comma is unnecessary. 
Decision: Reject