Redefining The Self-Normalization Property
Authors: Zhaodong Chen
Zhao WeiQin
Lei Deng
Guoqi Li
Yuan Xie
Venue:  (2021)
Keywords: Self-normalizing Neural Network
Activation Function

Abstract: The approaches that prevent gradient explosion and vanishing have boosted the performance of deep neural networks in recent years. A unique one among them is the self-normalizing neural network (SNN), which is generally more stable than initialization techniques without explicit normalization. The self-normalization property of SNN in previous studies comes from the Scaled Exponential Linear Unit (SELU) activation function. %, which has achieved competitive accuracy on moderate-scale benchmarks.  However, it has been shown that in deeper neural networks, SELU either leads to gradient explosion or loses its self-normalization property. Besides, its accuracy on large-scale benchmarks like ImageNet is less satisfying.  In this paper, we analyze the forward and backward passes of SNN with mean-field theory and block dynamical isometry. A new definition for self-normalization property is proposed that is easier to use both analytically and numerically. A proposition is also proposed which enables us compare the strength of the self-normalization property between different activation functions. We further develop two new activation functions, leaky SELU (lSELU) and scaled SELU (sSELU), that have stronger self-normalization property. The optimal parameters in them can be easily solved with a constrained optimization program. Besides, analysis on the activation's mean in the forward pass reveals that the self-normalization property on mean gets weaker with larger fan-in, which explains the performance degradation on ImageNet. This can be solved with weight centralization, mixup data augmentation, and centralized activation function. On moderate-scale datasets CIFAR-10, CIFAR-100, and Tiny ImageNet, the direct application of lSELU and sSELU achieves up to 2.13% higher accuracy. On Conv MobileNet V1 - ImageNet, sSELU with Mixup, trainable $\lambda$, and centralized activation function reaches 71.95% accuracy that is even better than Batch Normalization.(code in Supplementary Material)

Under review as a conference paper at ICLR 2021
REDEFINING THE SELF-NORMALIZATION PROPERTY
Anonymous authors
Paper under double-blind review
ABSTRACT
The approaches that prevent gradient explosion and vanishing have boosted the performance of deep neural networks in recent years. A unique one among them is the self-normalizing neural network (SNN), which is generally more stable than initialization techniques without explicit normalization. The self noll-oormalizaioon property of SNN in previous studies comes from the Scaled Exponential Linear Unit (SELU) activation function. However, it has been shown that in deeper neural networks, SELU either leads to gradient explosion or loses its self-normalization property. Besides, its accuracy on large-scale benchmarks like ImageNet is less satisfying. In this paper, we analyze the forward and backward passes of SNN with mean-field theory and block dynamical isometry. A new definition for self- normalization property is proposed that is casier to use both analytically and numerically. proposition is also proposed which enables us to compare the strength of the beff-normalization property hetween different activation functions. We further develop two new activation functions, leaky SELU (ISELU) and scaled SELU (sSELU), that have stronger self-normalization property. The optimal pa- rameters in them can be easily solved with a constrained optimization program. Besides, analysis on the activation's 's mean in the forward pass reveals that the self- normalization property on mean gets weaker with larger fan-in, which explains the performance degradation on ImageNet. This can be solved with weight centraliza- tion, mixup data augmentation, and centralized activation function. On moderate- scale datasets CIFAR-I0 10, CIFAR- 100, and Tiny ImageNet, the direct application of ISELU and sSELU achieve up to 2.13% higher accuracy. On Conv MobileNet V1 ImageNet, sSELU with Mixup, trainable X, and centralized activation fune- tion reaches 71.95% accuracy that is even better than Batch Normalization.
INTRODUCTION
In recent years, deep neural networks (DNNs) have achieved state-of-the-art performance on dif- ferent tasks like image classification (He et al. 2013; Zheng et al.) 2019). This rapid development can be largely attributed to the initialization and normalization techniques that prevent the gradient explosion and vanishing. The initialization techniques (He et al.) 2015 Xiao et al.) (2018) initialize the parameters in networks to have good statistical property beginning, and assume that this prop- erty can be more or less maintained throughout the training process. However, this assumption is likely to be violated when the network gets deeper or trained with higher learning rate. Hence, normalization techniques are proposed to explicitly normalize the network parameters (Salimans & Kingmai 2016 /Arit al., 2016) or the activations (Toffe & Szegedy, 2015b Ulyanov et 2016) during training. In particular, Batch Normalization (BN) (loffe & Szegedy 2015a) has become a standard component DNNs, as not only effectively improves convergence rate and training stability, but also regularizes the model to improve generalization ability.
However, BN still has several drawbacks. First, when calculating the mean and variance, the ac- cumulation must be done under FP32 to avoid underflow (Micikevicius et al. 2018). This brings challenges when training neural networks in low bit width. Second, the performance degradation under miero batch size also makes it more difficult design training accelerators, the large batch size increases the memory size to store the intermediate results for backward pass (Deng et al. 2020). Besides, Chen et al. (2020a): Wu et al. (2018) show that BN introduces considerable overhead.
The self-normalizing neural network (SNN) provides promising way to address this challenge. SNN initializes the neural network to have good statistical property at the beginning just like

---

Under review as a conference paper at ICLR 2021
initialization techniques. However, the statistics deviation in forward and backward passes can be gradually fixed during propagation, thus it is more robust to the deviation from initial properties (Chen et al., 2020b). For instance, the mean and variance of output activations with SELU in Klam- bauer et al. (017)) automatically converge the fixed point (0. 1). Chen et al. (2020b) analyze the Frohenius norm of backward gradient in SNN activated with SELU. They reveal trade-off between the self-normalization property and the speed of gradient explosion in the backward pass, and the hyper-parameters need to be configured according to the depth of the network. The resulting acti- vation function, depth-aware SELU (dSELU), has achieved even higher accuracy than the original configuration on moderate-scale datasets like CIFAR-10. 10, and makes the SNN trainable on ImageNet.
However, in deeper neural networks, the dSELU gradually degenerates to ReLU and loses its self- normalization property. Moreover, even with dSEL.U, the test accuracy on ImageNet with Conv MobileNet V1 (Howard et al. 2017) is still 1.79% lower than BN (Chen et al. we aim to answer the following three questions in this paper: 1). Is SELU the only activation function that has self-mormalization property? 2). If it is not, is there better choice? And how do we compare the strength of self-normalization property between different activation functions? 3). Why the performance of SNN on ImageNet is less satisfying? Is there any way to alleviate that?
2020b). Therefore,
In this paper, we analyze the signal propagation in both forward and backward passes in serial deep neural networks with mean-field theory (Poole et al., 2016) and block dynamical isometry (Chen et al. 2020b). Our main theoretical results are summarized as follows:
We illustrate that activation function would demonstrate self normaazaaaiinoo property if the second moment of its Jacobian matrix's 's singular values Φ0(g) is inversely proportional to the second moment of its input pre-activations q. and the property gets stronger when ó(q) gets closer to 1/g. A new definition of the self-normalization property is proposed that can be easily used both analytically and numerically.
We propose leaky SELU (ISELU) and scaled SELU (sSELU). Both of them have addi- tional parameter, B, that can be configured to achieve stronger self-normalization property. The hyper-parameter can be solved with a constrained optimization program, thus no ad- ditional hyper- parameter relative to dSELU is introduced.
We reveal that models with larger fan-in have weaker normalization effectiveness on the mean of the forward pass signal. This can be solved with explicit weight centralization, mixup data augmentation (Zhang et al. 2018). and centralized activation function.
On CIFAR- 10, CIFAR-100, and Tiny ImageNet, ISELU and sSELU achieves up to 2.13% higher test accuracy than previous studies. On ImageNet- Conv MobileNet VI, sSELU with Mixup. trainable X. and centralized activation function achieves comparable test accuracy (71.95%) with BN. Besides, we provide CUDA kernel design for ISELU and sSELU that has only 2% overhead than SELU.
RELATED WORK
In this section, present an overview of existing studies on the self-normalizing neural networks (SNN) as well as statistical studies on forward and backward signals in deep neural networks.
Self-normalizing Neural Network. Scaled Exponential Linear Unit (SELU) (Klambauer et al. 2017) scales the Exponent Linear Unit (ELU) by a constant scalar A. The À and original parameter in ELU are configured such that the mean and variance of output activation have fixed point (0, 1). The authors further prove that this fixed point is still stable and attractive even when the input activations and the weights are unnormalized. [Chnn et al. (2020b) investigate the fixed point in backward gradient. They reveal that the gradient of SNN is exploding with the rate (1 + e) per layer, where is small positive value. The norf-noona property gets stronger when e larger, whereas the gradient will explode at a higher rate. Therefore, they propose the depth-aware SELU in which the e se 1/ L is used to derive the optimal a and À in SELU for network with depth L.
Statistical Analysis of Deep Neural Networks. Schoenholz et al. (2016); Poole et al. (2016); Burkholz& & Dubatovka 2018) investigate the forward activations under the limit of large layer width with mean-field theory. They have identified an Order-to-Char Chaos phase transition characterized by the second moment singular values of the network's 's input-output Jacobian matrix. The neural network has good performance when it is on the border of the order and chaos phases. On the other
2

---

Under review as a conference paper at ICLR 2021
hand. CChnn et al. (2020b) develop very handy framework for analyzing the Frobenius norm of gradient. They illustrate that the gradient norm equality is a universal philosophy behind various different initialization, normalization techniques, and even some neural network structures. The gradient norm equality means the Frobenius Norm of the gradient more or less equal in different layers that the information flow in the backward pass be well preserved. (Arpit & Bengio, (2020)
3 SELF-NORMALIZATION PROPERTY
In this section, we formally define the self-normalization property under the problem formulation, notations, and assumptions follows.
Problem Formulation. Let's consider DNN with L layers. Each layer performs linear transform followed by non-linear element-wise activation function f, i.e.
x f(i),
h4 W bi =
where æ E -RM is the output feature vector of layer h; is the pre-activation vector, Wi is the weight of fully-connected layer or the expanded doubly block circulant matrix (Sedghi et al.) 2019) of 2D convolution, by is the vector of biases, and we denote the loss as L. Besides, without loss of generality, for and (0, g). we have
6,+ [P()]] (((((=))/dz) E[a"),
where 6g is function of q. Following previous studies (Poole et al. 2016 Chen et al. 2020b), for V l, we make the assumptions as follows:
Assumption The mean of entries in Wi and by are zero.
Assumption With central limit theory, the entries hu follow i...d. N qi). ai = N, h h..
Assumption 3 The eigenvalues of W. Wi are independent with entries in h1-1-
Klambauer et al. (017) first define the self- elff-normalizaioon property of neural network as follows.
Definition (Selfnnormalizing Neural Network) neural network self-norn if possesses mapping Ω - Ω for each activation that maps mean and variance from one layer to the next and has stable and attracting fixed point depending on (w, the variance remain in the domain Ωt that is g(2)) C Q, where Ω ((H,,)]u E [Mmmiag pmaz), E [pmin, {maa]]] When iterutively applying the g, each point within 52 converges to this fixed point.
Q. Furthermore, the mean and
This definition imitates the explicit normalization techniques like BN, which ensures that the feed- forward signal is normalized. Based on Definition T Klambauer et al. (2017) propose the SELU:
if f(x) = of or - if x 20
(3)
Besides, Klambauer al. (017) initialize the entries in Wi with N 1/N that the output pre-activation will have the same second moment of input activation. With the stable fixed points of mean and variance around (0, 1). the optimal choice for À and a can be derived from
f(z) e d= O,
dz 1
(4)
V2T
2
Furthermore, the authors prove that the fixed points for mean and variance are still attractive even when the statistical properties of the parameters in the neural network deviate from the initial setup.
However, the statistical fixed point in the forward pass doesn't 't necessarily lead to good dynamics of gradient. [Chen et al. (2020b) analyze the Frobenius norm the gradient in neural networks activated by SELU. With the same activation function shown in equation their analysis shows that the optimal À and a can be configured by preserving the Frobenius norm of backward gradient and second moment of forward activations with equations follows:
(f)
2
dz= +G,
p(s)
P(x)
dz =
(5)
dz
2
/2T
00
3

---

Under review as a conference paper at ICLR 2021
where is small positive constant, without which the only solution for equation would would be X = v22 and a = 0, and the activation function degenerates back to ReLU with the initialization technique proposed in He et al. (2015). Thus it will lose the self-normalization property. Conversely, rel- atively large will bring stronger self-normalization property, but meanwhile make the Frobenius norm of gradient explode with rate + per laver. Notably. the original configuration of SELU can be obtained by setting = 0.0716. Therefore, Chen et al. (2020b) assert that having R I could bring good trade-off between gradient norm stability and nerllaaomai property. Experiments on CIFAR-10 10 and ImageNet show that the new configuration results in higher accuracy.
Inspired by Chen al. 2020b). we formally redefine the self normalization property as follows:
Definition 2 (Self-normalization Property) Given an activation function define operand & as
(
00
(df(vz)) dv/z
(()) =
dz.
(6)
2m
If satisfies:
- o(1) < max ) -11)=1+6, [ P(x) e d= 1, min(1, y
00
f())=
2
then say f has the self-normalization property.
While the first two equations in equation 7 are identical to equation 5 that constructs fixed-points for both the second moment of activations and the Frobenius norm of the gradient, the third one makes these fixed points attractive,
have the proposition follows.
Proposition 3.1 (Strength of Self-normalization Property) Under all the three Assumptions and Definition 2 we represent ((g) linear interpolation between and 1/q as follows.
s(4) = 1 1+ (1-
1/9 1//
q a<1
(8)
q>1
where 74 E (0, is function of q. Then the following conclusions hold (Proof: Appendix A.2]:
The self- normalizationn property gets stronger when 7441 and 74>1 get closer to O. In particular, we have 1Tg<1| R 1
dò(g)
dq 1==1 + 1| when q is around 1.
For layer the gradient explodes under the rate (1 + pnd, i.e. 11 +
6m_1)
BC Sh
E
8C
Proposition 3.1 is derived based on Assumption I whereas the mean weight matrices may shift during training. Fortunately, Proposition (3.2 shows that the deviation of the mean of forward activa- tions can also be normalized by simply multiplying with the weight matrix.
Proposition 3.2 (Normalization of Mean) Under the assumption that the entries in the weight ma- trix Waj are independent with the input activations, and their expectation has upper bound μs, Le. i,j. E[wiz] S p.. Then we say multiplication with the weight matrix normalizes the mean if < Ni-1 holds, where Ni-1 is the fan-in of the current layer 1. Moreover, the mean is scaled down by ratio smaller than 1N (Proof: Appendix A.3
NOVEL SELF-NORMALIIZNG ACTIVATION FUNCTIONS
reveals that ff with ((g) closer to 1/q may have stronger self-normalization property. Therefore, the basis of SELU, propose to add an additional hyper-parameter B that can be configured to bring Φ(g) closer to 1/g and encode other interesting properties. As demos, we find the following two activation functions are quite promising.
Proposition 3.1
4

---

Under review as a conference paper at ICLR 2021
Scaled Scaled Exponential Linear Unit (sSELU). The sSELU is defined as follows
if 0 f(x) f(x)= X a0 - if 2 S
(9)
The negative pre-activations are scaled by 8 before fed into the activation function. This design is motivated by the observation that without the curvature provided by the exponential term ae ((g) of SELU will be constant value without self-normalization property.
Leaky Scaled Exponential Linear Unit (ISELU). The ISELU is defined as follows
f(x) ac 3x {
if x x>>0 if <0
(10)
which has an additional negative slope Br. This inspired by the observation that leaky ReLU helps to avoid the saturation of negative activations. Besides, Chen al. (2020b) show that leaky ReLU alone also improves the stability of the training process.
(a)
(b)
(c)
(d)
1.75
1.75
ReLL
ReiCI
1.50
258LU 2.53
452211 1.02
4-
OSELO
1.50
692211 - 1:50
ASELU
ISPII 0.
45221/ 1120
1.07
(7229 3:440
1ST1O A-1.00 1.00
1.25
1.25
ISEU 1.40
1.00
1.00
0.75
0.75
0.50
0.50
2
Figure I: The ((q) ~ and f(x ~ x of sSELU (a) & (b) and ISELU (c) & (d) under different X.
Determine the optimal X, ar and B. Figure | shows that given proper parameters X, a, and 3. our sSELU and dSELU can be configured to get closer to 1/g, which indicates stronger self- normalization property. With the first conclusion in Proposition 3.1 and equation 7 the X, a, and B can be obtained by solving the optimization problem below when c provided:
db(()) dq
s.t. =1+, / (())
min A.
+
de 1, A21
(11)
2T
9=1
In particular, the constraint A 2 1 is inspired by the argument in Klambauer et al. (017) that "a slope larger than one can increase the variance if is too small in the lower layer". In this paper, we find that constraining À 2 1 provides two other benefits. First, having À = helps to maintain the mean of the output activations around 0. Second, having larger slows down the gradient explosion in the backward pass. The detailed discussion can be found in Appendix A.4
Determine the e. While Chen et al. 2020b) propose to have 1/T. to avoid gradient explosion, where 7. is the depth of the network, their derivation is based the assumption that 6, ss in equation 2 How- ever, after taking the nonzero 6, into consideration, our Proposition 3.1 shows that the is actually (1+6,) rather than ee.. We plot the relationship between (1 5g) and under different e in Figure 2
When
1.10
dSeLU
8 0:07
SSelU
0.05
1.05
= 0.03
0.01
2
First of all, because of the first term in equation 7 we have 6, =
when = 1, this illustrates the intuition behind using (1 + E) to characterize the rate of gradient explosion. Therefore, R3 1/T. is still a good choice to arbitrarily determine c, especially for relatively shallow networks. As ISELU and sSELU has relatively higher 8 in Figure 2 relatively smaller than that of dSELU may yield better performance. Last but not least, in deeper neural networks, q has more chance to deviate from the fixed point q = 1, and 6g gets larger when g gets larger. Therefore, the trade-off between the strength of self-norralizzaiio property and the speed of gradient explosion may become too complex to be captured by e ss 1/L, and it might be more promising to determine the proper the validation set.
Figure 2: (1 + 5g) q of SSELU & ISELU under dif- ferent c.
5

---

Under review as a conference paper at ICLR 2021
5 LARGE-SCALE SELF-NORMALIZING NEURAL NETWORK
Chen et al. (020b) evaluate SELU and dSELU on Conv MobileNet V1 (Howard et al. 2017 7) on Im- ageNet. While SELU suffers from gradient explosion, the accuracy of dSELU 1.79% lower than the BN baseline. We observe that there are two major reasons behind this performance degradation and propose several tricks that improve the performance on large-scale SNNs.
Nonzero Mean in the Forward Pass. Proposition 3.2 reveals that the nonzero mean can be dimin- ished by multiplying with the weight matrices when l < Ni-1" On small-scale SNNs, as Ni-1 relatively small, this condition is easy to satisfy, and we don't have to worry about the deviation of the mean from However, in large-scale SNNs for large datasets like ImageNet, larger fan-in is required to ensure the network has enough parameters to model the more complex problem. In appendix[A.5]] we empirically show that models with larger fan-in tend have larger JNNi-1. which implies weaker seff-normalization property on the mean. As our Proposition 3.1 is based on As- sumption [ greatly biased mean may violate the assumption. As a result, for large-scale SNNs, we have to consider the influence of nonzero-mean.
While the influence of the weight matrix on the mean is well captured by Proposition 3.2 the in- fluence of the activation function is more complex. In particular, for layer I, we assume the pre- activations follow i.i.d. [(E(b)) )²), and the output mean can be computed with
1 20
E E
f(x)
dx
(12)
V2T02
We plot the relationship in Figure 3 in which the solid line represents the theoretical value and the dash line is the value measured via numerical experiments. When the variance o² large, there will be a positive bias on the mean of output. The explanation is quite intuitive: the saturated region in the negative axis has an asymmetric growth rate compared with the positive axis. Hence, when the variance is large, the positive part contributes more than the negative part, which increases the mean.
SELU
dSELU
SSELU
SELU
15
0.11
-011
0.31
- 0.31
0 D:25
0.25
0.25
= 0.25
1.00
1.00
1.00
-
1.0
9.00
e- 9.00
0.5
0.0
E[h)]
E(hi]
Elh]
E(h]
Figure The absolute value of the output mean under different input mean and variance.
Lack of Regularization during Training. Luo et al. (0088) show that batch normalization also regularizes the training process. In particular, using the statistics of minibatch j4 and oB for explicit normalization introduces additional Gaussian noise that regularizes the training process, and also discourages the reliance a single neuron and penalizes correlations among neurons. However, activation functions with the nommaaommaiinnoonn property don't 't have these features as they do not rely on the statistics from minibatch.
Based on the analysis above, we find that three techniques can be used to improve the performance of large-scale SNNs: mixup data augmentation (Zhang et al. 2018). weight centralization, and centralized activation functions.
Mixup Data Augmentation. The mixup is simple data augmentation routine that constructs virtual training examples via linear interpolation: T = -yz, (1 -x) V/ y?; + (1 11--yy -y)j,- where (æ,, v.) and (zz,. are two training samples randomly drawn from the training set, 7 (0,1). 1). In particular, we find that using Mixup with SNN brings two benefits.
First, Mixup reduces the variance second moment of the inputs. Under the assumption that the corresponding entries and x, in the two samples are independent and E|0;] Elae] = E[x?), E|æ,} = Eir,] 0. we have E [(y; + (1 ) (72 (11++(11130000 (1= For instance, when = 0.7, the second moment of the sample entries is 0.58 of the original training samples, hence the variance of the input samples is implicitly decreased. With smaller q in the first few layers, on

---

Under review as a conference paper at ICLR 2021
one hand, shown in Figure 2 smaller second moment leads to smaller 6q. which reduces the gradient explosion rate the backward pass. On the other hand, as shown in Figure a smaller variance also reduces the shift of output mean caused by the activation function.
Second, Mixup creates additional training samples from the dataset, which provides additional reg- ularization that could further boost the accuracy. The same property is also used in Zhang et al. (219). Besides, we empirically find that making À trainable is also helpful when applying ISELU and SSELU large datasets like ImageNet. The trainable can be viewed as the scalar multiplier initialized at I used in Zhang et al. (2019). Together with the bias of each layer, they serve as the affine transform applied batch normalization (loffe & Szegedy, 2015a). which increases the representational power of the network (Michalski et al. 20199).
Weight Centralization. When μ 1/NN-1, multiplication with the weight can effectively nor- malize the mean of activations. Therefore, we can explicitly centralize the weights, i. W - W mean(W) As the weights usually much smaller than the feature maps, the overhead of Weight Centralization is usually quite small. Moreover, as it doesn't 't rely on the batch, Weight Centralization can still be utilized under micro-batch scenarios.
Centralized Activation Function. When the network with large fan-in is relatively shallow, we trade the strength of self- normalizaaiinio property with the deviation of the mean caused by the activation function. While 6(1) = 1 + c, E[f(=) = 0, and (x) = can not simultaneously hold in SELU and dSELU they only have two parameters, the X, a, and in our SSELU and ISELU can be solved with
(1) 1+c, Elf(=)]= = _
f(z)
de =
E[f(()) L f(
( de 1, (13)
2
22T
which ensures that the output activations still have zero-mean when the input is at the fixed point.
6 EXPERIMENTS
this section, we validate our activation functions on multiple image classification benchmarks. In Appendix B we present an efficient CUDA kernel design, under which the overhead of ISELU and sSELU are only 2% higher than SELU. The experiment setup in Appendix Cland the value of the parameters X. a, B, and the resulting 7g= summarized in Appendix
6.1 NORMALIZATION EFFECTIVENESS
We empirically show that our new activation functions have better normalization effectiveness than existing studies, which is demonstrated by the second moment of the output pre-activation of each convolutional layer (F)) and the Frobenius norm of the gradient of the weight (l) aw I|
0C
SELU
dSELU
SSEW
ISELU
1.5
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
-05
0.5
0.5
005
20
0
20
40
20
40
20
40
0.0
-0.5
0.5
05
-05
-10
1.0
1.0
1.0
-15
1.5
155
1.5
-2.0
2.00
2.0
2.0
Layer
Laywr
Layer
Layer
Figure 4: The distribution of the second-moment of forward pre-activation and the Frobenius norm of backward gradient on weights.
As shown in Figure in the forward pass, sSELU and ISELU normalize the second moment in the forward pass better than dSELU. In the backward pass, compared with SELU and dSELU, both sSELU and ISELU have much flatter and more concentrated distribution of the Frobenius norm in the backward pass. Notably, SELU has e ze 0.0716, and higher lead to stronger self-normalization property. This explains why it also has good dynamies in the forward pass. However, e 0.0716

---

Under review as a conference paper at ICLR 2021
also increases the speed of gradient explosion, which explains why SELU has worse backward dynamics. Last but not least, the further the E[h²] deviates from in the forward pass, the faster the 8C
Byy || increases in the backward pass. As larger = ETAh]] will lead to larger bg. this observation
justifies the second conclusion in Proposition 3.1
6.2 MODERATE-SCAA DATASETS
We summarize the results on CIFAR-10 10, CIFAR-100, and Tiny ImageNet in Table !
Table 1: Test accuracy on CIFAR-10. 10, CIFAR-100 100, and Tiny ImageNet (CI=95%).
Method
CIFAR-I0
CIFAR-100
Tiny ImageNet
BN+
85.64%
1.15%
52.60% + 1.98%
43.43%
0.58%
Slabbaa et al. 2017
85.69% t 1.10%
55.08% + 0.37%
45.50%
0.47%
0.01
86.07%
0.40%
54.09% + 0.79%
45.63%
-0.99%
0.017
86.39%
0.37%
53.92% + 0.88%
46.92%
-0.9%%
dSELU et al. 2020b
0.03
86.93%
Σ 0.61%
54.49% + 0.56%
46.45%
0.33%
0.05
85.99%
1.30%
54.33%
1 0.33%
47.01%
+0.36%
0.07
86.88%
0.48%
54.83%
+ 0.29%
46.55%
0.50%
0.01
87.18%
+ 0.38%
57.21%
+ 0.50%
46.26%
+ 0.49%
0.017
87.29%
+ 0.20%
56.30%
1 0.79%
46.67%
0.98%
sSELU (Ours)
0.03
87.25%
0.266
56.15% 1 0.70%
46.73%
0.46%
0.05
86.09%
+: 1.14%
55.43% + 0.85%
46.82%
1 0..6%
0.07
86.99%
0.33%
55.06% + 0.39%
46.69%
0..4%
0.01
83.19% +: 0.23%
54.61% 1 0.82%
44.77%
= 0.38%
0,017
85.80% + 0.28%
55.51% + 0.999%
46.46% = 0.66%
ISELU (Ours)
0.03
86.69% + 0.84%
56.78% + 0.44%
47.80%
+ 0.58%
0.05
86.15% + 1. 1.29%
56.35%
+ 0.71%
47.71%
0.34%
0.07
86.61%
+ 0.47%
54.47% + T.59%
47.27%
+ 0.45%
First of all, under most of c, our ISELU and sSELU are comparable or even better than dSELU. In particular, SSELU achieves consistent accuracy improvement on CIFAR- 10 and CIFAR- 100, while ISELU has better performance is not always the best choice for dSELU, ISELU, and sSELU, but the best accuracy achieved in our sSELU and ISELU are under relatively smaller than dSELU (Chen et al., 2020b These two observations accord with our arguments in Section 4 on the selection of proper c.
Tiny ImageNet. Second, the results show that ~ 1/L ~ 0.017
Table 2: Test accuracy under different configura- tions on ImageNet (CI=95%).
6.3 LARGE SCALE SNN
Method
Test Acc.
In this part, we evaluate our conclusions in Sec-
BN + RcLU
11.12% 0.07%
tion 5 First, adding Weight Centralization or
BN + ReLU Mixup
71.54% 0.38%
Mixup successfully solve the gradient explo-
KlambuKammsurrr al. 2017
Explode the lirst epuch
sion problem in SELU, ISELU, and sSELU. Second, in dSELU, ISELU, and sSELU, mak- ing À trainable brings additional performance improvement than using Mixup alone. More- over, after relaxing the constraint X Z to A > 0.5, the test accuracy "ISELU (A > M.x)+ drops by 8.25% this demon- strates the importance of constraining to be no less than 1. Last but not least, by combining centralized ISELU and sSELU with Mixup and trainable X. we achieve 71.82% and 71.95% 1 accuracy.
SEL.U Weight C entraliztion
60%40% 0.00%
SELU + Mixup
71.37% 0.14%
WSELL
(20206
69.63% 0.10%
dSF:1. + Weight Centraization
69.11% 0.00%
dSELU + Mixup
71.03% 0.06%
dSELU + Mixup (trainable X)
71.74% =: 0.09%
ISELU
Explode in the tirst epoch
ISELU + Weight Centralization
10.33% 0.05%
ISELU Mixup
70% :: 0.06%
ISELU + Mixup (inamable
71.55% 0.00%
ISHLLU (centmal) + Mixup (trainalle A)
71.82 + 0.00%
ISELU LA 0.6)+Mixup
63.30% 0.25%
SSELU
Explode the first epoch
SSELU +
Weight Centralization
69.07% 0.11%
SSELU + Mixup
11.12% 0.00%
$SELU +
Mixup (trainsbll X)
71.77% 0.09%
SSELU (central) + Mixup (trainable A)
71.95% 1 0.07%
7 CONCLUSION
In this paper, we analyze the forward and backward pass signals in SNNs and redefine the self- normalization property. Two novel activation functions, ISELU and sSELU, are developed under this definition. A constrained optimization program is proposed to solve the optimal configurations. Moreover, reveal the reason behind the performance degradation of SNN under large fan-in, and several solutions are proposed. With our novel methods, advanced results are achieved on multiple benchmarks. Our study demonstrates
research direction for the design of activation functions.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Devansh Arpit and Yoshua Bengio. The benefits of over-paaramettrriationn at initialization in deep re(lu] networks, 2020. https:// http:///ppeeevieeeeeeee www.rreeeee.eeeoorunnn hit.:mmmdddddr.ggX00ee
Devansh Arpit, Yingbo Zhou, Bhargava Kota, and Venu Govindaraju. Normalization propagation: A parametric technique for removing internal covariate shift in deep networks. In Maria Florina
Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1168-1176, New York, New York, USA, 20-22 Jun 2016. PMLR. URL Tress/v4//arpiie atll1.11.... ht.m
ngs. .ll
Rebekka Burkholz and Alina Dubatovka. Exact information propagation through fully-connected feed forward neural networks. arXiv preprint arXiv:/806.066: 863666660066.... 2018.
Zhaodong Chen, Lei Deng, Guoqi Li, Jiawei Sun, Xing Hu, Ling Liang, Yufei Ding, and Yuan Xie. Effective and efficient batch normalization using a few uncorrelated data for statistics estimation. IEEE Transactions Neural Networks and Learning Systems, 2020a.
Zhaodong Chen, Lei Deng, Bangyan Wang, Guoqi Li, and Yuan Xie. A comprehensive and modu- larized statistical framework for gradient norm equality in deep neural networks. arXiv preprint arXiv:2001.00254, 2020b.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Nicholas Higham. The accuracy of floating point summation. SIAM Journal on Scientific Com- puting, 14(4):783-799, 1993.
Andrew G Howard, Mengiong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04661 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015a.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Ma- chine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015b. PMLR. URL http: /pprocediings.o www.mmrr..ss.....mmmmmmmommmmmse http://////offff5.htmmmmm
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing
neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. arXiv preprini arXiv: I 1809. 00846. 2018.
Vincent Michalski, Vikram Voleti, Samira Ebrahimi Kahou, Anthony Ortiz, Pascal Vincent, Chris Pal, and Doina Precup. An empirical study of batch normalization and group normalization in conditional computation. arXiv preprint arXiv: 000600001, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL
http:::///ppeeeviii www.errrriee..tt//ooou.......... www.:riddddllssggpaer
.ne//
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl- Dickstiis,,,,,, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. In Advances in neural informa- fion processing systems, pp. 3360-3363 2016.

---

Under review as a conference paper at ICLR 2021
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterizzttoon accelerate training of deep neural networks. In Advances neural information processing systems, pp. 901- 909, 2016.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint (oXiiv6611.00 2016.
Hanie Sedghi, Vineet Gupta, and Philip M. Long. The singular values of convolutional layers. In International Conference on Learning Representations, 2019. URL UtL/hhttps:///ppeeeviee
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 107.v0160..08022, 2016.
Shuang Wu, Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie. and Luping Shi. /1-norm batch nor- malization for efficient training of deep neural networks. IEEE transactions on neural networks and learning systems, 30(7):2043-2551, 2018.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of cnns: How to train 10, 000-layer vanilla con- volutional neural networks. In JCML, pp. htmml
2018. URL http://proceeainng
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In International Conference on Learning Representations, 2018. URL
https://openreviee net/ forum?iddii ?idm 1Ddp1-DdRRR
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv: /901. 09321, 2019.
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou. Hardness-aware deep metric learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 72 81, 2019.
Denny Zhou, Mao Ye, Chen Chen, Tianjian Meng, Mingxing Tan, Xiaodan Song, Quoc Le, Qiang Liu, and Dale Schuurmans. Go wide, then narrow: Efficient training of deep thin networks. arXiv preprint 1,Xiv12007.00811, 2020.
10

---

Under review as a conference paper at ICLR 2021
A PROOFS
A.1 SIGNAL PROPAGATION IN DEEP NEURAL NETWORKS
For convenience, we denote the Jacobian matrix 0f(h:-1) as X where is the normalized trace.
Di-1, and tr (rWWWWTtrrDL--DDD
h--1
Proposition A.1 (Forward Signal under Mean Field Theory) Under the formulation, notations, and assumptions above, the evolution of the second moment of pre-activations QEE[LLL] in the forward pass can be described with
Xi
=901
1= L=,,..,, L.
(14)
8 9
Proof. Under Assumption [ & 2 the pre-activation vector input in layer can be characterized with Gaussian random variable / where z is random variable following N 1). With these definitions, we can investigate how evolves between layer and l:
9I Ni (Wif + bi)" ( + bi) =o + Ni f(hn-)) AU ((u (15)
where of N. bf. U is Orthogonal matrix, and A is diagonal matrix of eigenvalues in W. Wi. We characterize the diagonal entries in A with random variable À whose probability den- 3 we have
sity function is p(A). With Assumption
② =
NH Ni
/ (P(VL12)
dz of PWWa)1))
dz, (16)
22T
22
Then, we substitute equation 2 into equation
16 which yields
= 9111 tr (w.W?? / [[(V)==))]
9-1 tr (D e= de === of
145
22T
1+8
(17)
As the bias vector is usually initialized with zero and shared among multiple feature entries, db has lower impact than the second term. Therefore, if we neglect the of with the notation XI tr tr(,.WF))rr (D.) have
X²
Qi = go
1= 1,1,,,, L.
(18)
Sq...
Proposition A.2 (Backward Gradient under Block Dynamical Isometry) Under the formulation, notations, and assumptions above, the evolution of the Frobenius norm of the gradient in the back- ward pass can be described with
OC Dh
dc
E
/E
(19)
The
Xi
Proof. Given the gradient Th,
ac with the chain rule, we have Sh:
ac
DJ WT Th:
dC
and
dL Tho
dC
11' T
(20)
Oh In particular, we are interested in the Frobenius norm of oh; represented as || ah; Chen et al. (2020b), its expectation can be computed with
According to
dC
dC
R (ITL (TW))
(21)
Tho
0h,
Chen al. 2020b) proves the theorem follows.

---

Under review as a conference paper at ICLR 2021
Definition Chen et al. 2020b)) (kth Moment Unitarity Invariant) Let [Ai] Az.., AL}
be a series independent random matrices. Let {UI} := {U1, U3.... U be series independent haar unitary matrices independent of moment unitarily invariant v0 < & k, we have
AL). We say that (II,A) is the kth
tr ((ULA))A)))) = ((LLUA)))U)))))))
(22)
Theorem A.1 Chen et al. 2020b) (Multiplication). Given J : := 11 i= 1 Jr where {Jı E R=
is series of independent random matrices. If (W) (i is at least the 1st moment unitarily invariant (Definition 3) we have
tr I. Ji)") I (J,J,7)
(23)
Therefore, equation can be further simplified with Theorem A.1 as follows.
dc
ac
1
12
/E
I
(24)
Oh
Oho
(n(W.W7) (D: D
Xi
A.2 PROOF OF PROPOSITION 3.1
Proposition 3.1 (Strength of Self-normalization Property) Under Definition 2 as linear interpolation between 1 and 1/q as follows.
we represent ((g)
<(1) {
+ (1-7) 1) if q<l 1/q+%/-111111
<< if 9>>
(25)
where 7/ E (0, 1) is a function of Then the following conclusions hold:
The self-normalizatioon property gets stronger when Te<i and 7/51 get closer to 0. In
particular, hy R hpp>1
dō(g)
g=1 + 1| when q is around 1.
dq
For layer 1, the gradient explodes under rate (1 + 6g), i.e. II II. (1 + bq,_)) [| /1 11
||
8C The
Proof. When 7a<1 and 7-21 are approaching 0, p(g) gets closer to 1/g. With equation 6 have ((q) Ir( tr( Df) where D, is the Jacobian matrix of the activation function in layer 7, As the weights are initialized with N(0, ). we have tr tr(W.W7)
In the forward pass, with equation 2 we have Q++1 148
p(a)).
We can substitute equation
and get
q1+1
I+bg
"9p501 (1 q1) + (1 Itō
if q <
(26)
9i+1
I+dn
Tgp
(q) (1 Ito
if q
In the backward pass, with equation 14 and equation 19 we have
E
8C Jii
1/9
(27)
go
F
11
II 1= (1
+ 5p-.)
Because of
0c
E
BC
E
Thy
BC
E
112
(28)
0h411
tr
(D.
tr W W
(b())
12

---

Under review as a conference paper at ICLR 2021
have
77
if qi <
40
J.C.
11 (1+60-1) a m+((--9g=1)(1--
(1+b
(29)
Dh141
0.C.
1
1/0 y x1
if q1 >
:
E
05
m (1+6,1))
17
(148
where
Vy <1
E (0,
11 91)
(30)
7/211
E (0,
1
the monotonically increasing functions of 7ip and >p respectively.
Similarly, we can derive how the deviation from the fixed point evolves during the back propagation.
1
E
AC
7/
+ (1 -axx1) IT (1+8
1), if q1 <
11
DC
3
F2
||
BC 57hg
113
AC
11
88
12
7
+ (1
11
T )) if qi
20
BC
1
#C Bhg
(31)
First of all, when dy are neglectable, equation 26
and equation
31
can be simplified as
E
=1 - (1) 1 -
Q1-1 1-q-11
2/
is q1 <1
113
113
[1
11
E
11+1 - -g (41 1),
if |>
4o
J.C. 9P5g
11
90
2C Bhg
(32)
As %g and N are the monotonically increasing functions of Tar<1 and Ya>1, 1, it is obvious that with smaller Ta and > the deviation from the fixed point both forward and backward passes shrinks faster.
In particular, when y is around the fixed point q = as ensured by the second term in equation 7 approximate ()g) and 1/g with their first-order Taylor expansion around q with the definition of <1 and 7 in equation we have
1/<1 T9s1
1
do(g) dq
is
Aq <
1/(1+Aq)
Aq
q=1
(33)
1/21 A Tps1 2(1+Ag)-11((+4))) 1/(1-1+A4)
Adig) dg
if A >
q=1
As result, we can reduce the number of layers required to diminish the deviation by minimizing dd(a) dq
1-=1 + 1/.
Then, we discuss the influence of 6g. The fixed point of the two recursive functions in equation 26 and equation can be computed
2C Dhy
8
ni (1+8 - 1, if <
1+5 <1
Tox3
go
E
||
113
(34)
MK;
118
q
1+6 77001
9..6
[|:
m (1+bmu))))))
if 9>1
While the fixed point of deviation slightly deviates from 0, in the backward pass, we have 11 (1+ + ac
5q-,) )EEE 8h,
ac
113 F.
[i] which suggests that the gradient explodes with rate (1 + 8g.)
aho
at layer l.
A.3 PROOF OF PROPOSITION 3.2
Proposition 3.2 (Normalization Mean) Under the assumption that the entries in the weight matrix why are independent with the input activations, and their expectation has an upper bound ns,
13

---

Under review as a conference paper at ICLR 2021
Le. 4 i.j, E[wyy] S μ. Then say multiplication with the weight matrix normalizes the mean if N:- holds, where N-11 is the fan-in of the current layer 1. Moreover, the mean is scaled down by ratio smaller than 1-N-1-
Proof. With equation
the j entry in the output pre-activation hy can be computed with
h
Wj. 11,311-11
(35)
Therefore, with the assumption on independence between weight and input activations, have
Ni-1
E[bry]
Σ
F(w S NI-IPE
T 1'==
(36)
Ni-1
As the term E N; 1 - can be viewed as the mean the input activations, when j < Ni equation 36
reveals that the mean is reduced after multiplying with the weight matrix.
A.4 BENEFITS OF HAVING A 2
First of all, show that having À Rs 1 helps to maintain the mean of the output activations around As we normalize the mean by multiplying with the weights, we don't require the F [(]( when æ ~ (00, 1) like Klambauer et al. (2077). However, according to Proposition 3.2 the speed of the mean converging to 0 gets slower when the expectation of entries in the weight matrix deviates from O and when the fan-in gets larger. Therefore, it's still ideal to avoid shifting the mean too much when the activations flowing through the activation functions. As shown in Figure 5 we simulate the forward pass in 64-layer cunnecco neural network, and plot the distribution of output activations in layer 1, 4, 16, and 64.
Depth=1
Depth=4
Depth=16
Depth=64
0.14
SSELU
0.14
#SELU
0.14
#SELU
0.14
#SELU
#SELU 1-0.2
SSELU 1-02
#SELU 1-1.02
0.12
#SELU 1.0.02
0.08
0.08
0.08
0.08
0.06
0.06
0.06
0.06
0.00
0.00
0.00
0.00
Depth-1
Depth-4
Depth-16
Depth-64
ISELU A= 0.117
0.14
ISELU = 0.17
ISELU A= 0.17
ISELU 0.17
ISELU 1.00
ISELU
ISELU
ISELU
0.10
0.10
0.10
0.10
0.04
0.04
0.04
0.04
0.02
0.02
0.02
0.02
-5
-5
-5
-5
Figure 5: The distribution of output activations of sSELU and ISELU in different layers.
obvious that when < 1, a spike around 0 is observed for both sSELU and ISELU, and this leads to large negative mean of the output activations. On the other hand, for instance, by solving
(11) = 1+
f (2)
dz
I
f(z)= de 0
f(2)
(37)
2
2m
under 0.03, we have À ~ 1.0360 and 1.0362 for sSELU and ISELU, respectively. Therefore, is a good starting point for the optimization.
14

---

Under review as a conference paper at ICLR 2021
Second, show that having larger À slows down the gradient plosion in the backward pass. According to the second conclusion in Proposition 3.1 the backward gradient explodes under rate (1 + 6,); thus keeping 0g low is critical for avoiding gradient explosion. Ac- cording to the definition in equation 3 (6) be computed with
maXge (g)
1.050
SSELU ISELU
1.045
x(())
1 1+ 6g
(38)
1.040
a (2) V2* dz
0.8
1.0
A
We plot the relationship between the maximum (1 6n) under 9 (0, 2] and in Figure Obviously, the maximum (1+ + 6g) decreases when À gets larger. This observation is quite intuitive. The + de characterizes the relative deviation between [[]] and El(f(()) (x)/dr)) 3 E[a"]. For the positive pre-activations, we have
Figure 6: Influence of on
1 8g
1-- AV72
v22 e
dz -
dz ([(d)(z+)/dz)) 1|
(39)
2
2
Hence, the deviation is contributed only by the negative part. With larger X, the positive activations are scaled up, thus the negative activations have to be scaled down to preserve the overall second moment. Therefore, the negative part contributes less the overall second moment, and the relative deviation between (x)) and [[[[f [ [xx") gets smaller. All in all, a larger leads to smaller ōg and smaller 6, reduces the gradient explosion rate (1 + 6g).
A.S THE pN-1 WITH INCREASING FAN-IN
Here we empirically illustrate that 1NN-1 increases when the fan-in Ni-1 gets larger. The ex- periment is performed on a 32-layer CNN activated with dSELU. We collect the JN of each convolutional layer and the final fully-connected layer after each epoch among total 10 epochs. The learning rate is set to 0.005, Let the number of input channels of layer be Ct, the k in each title dSELUXA x indicates that the number of input channel is scaled to k x
dSELUx 1
dSELUx
dSELUx
dSELUx
30
20
20
20
20
15
5-
5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Figure The 1uNN-1 with Increasing Fan-in.
As shown in Figure 7 with larger fan-in, the layers tend to have larger J&N According to Propo- sition 3.2 larger p&Vi-1 will lead weaker self-normalization property on the mean. Notably, if we further scaling the number of input channels with & greater than 4, gradient explosion happens. These two observations justify our conclusion that the shift of mean is more influential networks with larger fan-in.
B EFFICIENT IMPLEMENTATION OF LSELU AND SSELU ON GPU
In this section, we present an efficient implementation for ISELU and sSELU on GPU. In particular, take ISELU example, as the same strategy be directly applied sSELU.
15

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Forward Kernel of ISELU.
Data: Input Feature: X IN,, Output Feature: Y E RV,, a, 3 E R; Threadidx: t; BlockIdx: b; Thread Block Size: Number of Thread Blocks: B;
begin
for i = x T to N step x T do
if X i then
1 [
else
Y
x
(a x + 3x X(i aa)
Forward Pass. The forward pass kernel for ISELU is shown in Algorithm In the forward pass, have
x ac + Br
x> x
(40)
The implementation quite straightforward, all the threads stride across the feature map and element- dlement-wiscly compute the output activations. The input and output feature maps are treated as ID array. therefore the kernel achieves good coalescing in both read and write. While we take T' = 1024, the number of thread blocks B is computed by B L(N +T 1)/TJ, the number of thread blocks is large enough to achieve a high utilization rate.
Algorithm 2: Backward Kernel of ISELU.
Data: Gradient of Input Feature: Gx E IR Gradient of Output Feature: Gy E R Input Feature: X; X, a, R; Gradient of x: Gx E R; ThreadIdx: t; Blockldx: b; Thread
Block Size: T; Number of Thread Blocks: B;
begin
float = 0, float = 0
for
x T + 1. to N step T
du float æ = [ float de
float y, float gy = Gyli]
if [ [
then
dy dz
By x x -
else
dy X x (a cº 3), y = 9v (axe +3xx a)
dr
float p + y, c (t p) V. ==
G [1] = gv x syncthreads()
dy
10
dr
11
12
summ BlockReduce(()
13
if then atomicAdd(&&G), sum)
The backward pass kernel shown in Algorithm backward pass ISELU is shown follows:
Backward Pass.
When the trainable, the
ac ØT. Oy ØT.
x
1 +3
if 2 ax = Σ
OL
OL
(41)
dr Oy dr ay
Oy
and UC BC is used to compute both de DX can be cached registers data reuse (line 4). As the By When the threads stride through the whole feature map, each thread holds a partial sum in a private register p (line 2). In order to avoid the underflow of floating point accumulation, Khan summation algorithm (Higham] 1993) is applied (line 9). At last, we use the block reduction in the CUB library to get the partial sum of the whole thread block, and the final result is atomically added to the 8C Øx
Different from the forward pass, we choose to be a few thousands (usually much smaller than B = [(N + T 1)/TJ). The motivation behind this is that while it is large enough keep all the
16

---

Under review as a conference paper at ICLR 2021
streaming multiprocessors busy, it is also small enough to keep most of the reduction chip and reduce the atomic transactions.
We evaluate new CUDA kernels NVIDIA V100 GPU. We randomly generate input fea- ture map with size of [512, 64, 56, 56] as the input of the activation function, and we compare the kernel latency of forward and backward passes with the native SELU in PyTorch. The results summarized in the table below:
Table 3: Forward and Backward pass Latency of SELU and ISELU.
Forward Pass
Backward Pass
SELU (Klambauer et al. 2017]
10.487 ms
14.833 ms
ISELU
10,571 ms
15,284 ms
SSELU
10.570 ms
15.260 ms
Compared with the original SELU, our new implementation with the trainable only increases the latency by around 2%, which is neglectable. The reason behind this is that the latency of activation functions are bounded by the DRAM bandwidth of GPU (Chen al.) 2020b), and the computation units are underutilized. As our CUDA kernels don't 't introduce additional DRAM access, it has low impact on the latency.
C EXPERIMENT SETUP FOR MODERATE-SCALE BENCHMARKS
The experiments in Section 6.1 and 6.2 are based 56-layer Convolutional Neural Network shown in Table The H and W are 32 for CIFAR-10, CIFAR-100, and 64 for Tiny ImageNet. Following Klambauer et al. (2017): Chen al. (2020b), the weights in the convolving kernels are initialized with i.i.d. NO, Ku. where kn, and kw are the height and width of the filters and cmn is the input channels. The models are optimized with SGD with momentum=0.9, weight decay=0.0005.
Table 4: 56-layer Convolutional Neural Network.
Out Size
Serial Network
conv
W x H
3 x 3, 16, sl
3 16,
block1
W x H
x 16,
x 88
ds1
W I x 4
3 x 3, 32, s2
3 3.3, s1
x
3 x 3, 32, s1
block2
2 x
3 x 3.
32, s1
x
1,3, 64, x2
ds2
4 x
x
x 3,
64, s1
block3
of x 4
3
x 64,
64,
x
1
average pool, fc
For Section 6.1 we train the model from scratch for 3190 iterations (10 epochs) on CIFAR-10 under the learning rate 0.015. The choice of learning rate is based on the observation that it is large enough to simulate the fierce update of parameters but also small enough to avoid gradient explosion. Following Chen et al.] (2020b), we set 0.017 for dSELU, ISELU, and sSELU. We collect the second- of the output pre-activations of each convolutional layer as well as the Frobenius norm of backward gradient on the convolving kernels in each iteration.
For Section 6.2 as the model has relatively small fan-in, we directly apply sSELU and ISELU without techniques mentioned in Section 5 Besides, We clip the gradient by 2,- 2)" for all the experiments to increases stability. All the results are averaged among 4 independent runs to reduce fluctuation. For CIFAR-10 and CIFAR- 100, the models are trained with batch size 128 for 130 epochs with the initial learning rate set to 0.01, and decayed to 0.001 epoch 80. For Tiny Ima- geNet, the models are trained with batch size 64 for 200 epochs. The initial learning rate is set to 0.001, and decayed by 10 epoch 130, 180.
17

---

Under review as a conference paper at ICLR 2021
For Section 6.3] following Chen al. (2020b), choose the Conv MobileNet V1 (Howard al. 2017). The "Conv" indicates that traditional convolutions rather than depthwise separable convo- lution is used, the latter requires more epochs to converge (Zhou et al.) 2020). The model trained for 90 epochs with batch size 512 under leaning rate 0.02 (decayed by 10x at epoch 60 and 75). Following [Zhang al.] (2018), the y for interpolation is drawn from Beta distribu- tion Beta(0.7, 0.7). For all the experiments of dSELU, ISELU, and sSELU, we follow Chen et al. (2020b) and set e 0.06.
D COMPARISON OF PARAMETERS BETWEEN DIFFERENT ACTIVATION FUNCTIONS
We summarize the value of the parameters X, a, 3. and the corresponding 79 under different configurations in Table [S] According to Proposition 3.I smaller 7p=1 will lead to stronger self- normalization property. As shown in Table 5 under the same our SSELU and ISELU have lower 7y=1 compared with dSELU, this justifies our intuition that ISELU and sSELU can be configured to have stronger cefflnormalization property. Second, the result shows that for each activation func- tion, 7/=1 increases when gets larger. However, as shown in Figure 2 larger also leads to larger 5q. which increases the speed of gradient explosion in the backward pass. Last but not least, for experiments on MobileNet VI, our ISELU and sSELU under e = 0.06 achieve approximately the same 7/=1 with SELU with 0.0716, whereas the latter one has higher gradient explosion rate.
Table 5: A, a, 8, and the corresponding 7g under different configurations.
Method
X
7==1
5a-er CNN
al.
2017
1.05
1.67
0.806
0.01
1.37
0.48
0.973
0.017
1.34
0.64
0,954
dSELU et al. 2020b
0.03
1.27
0.89
0.919
0.05
1.17
1.25
0.865
0.07
1.06
1.64
0,810
0.01
1.00
4.09
0.31
0.905
0.017
1.00
3.28
0.41
0.881
sSELU (Ours)
0.03
1.02
2.46
0.58
0.855
0.05
1.00
2.17
0.75
0.819
0.07
1.06
1.94
0.92
0.794
0.01
1.00
0.66
0.65
0.914
0.017
1.00
0.86
0.55
0.890
ISELU (Ours)
0.03
1.00
1.14
0.39
0.857
0.05
1.00
1.47
0.21
0.821
0.07
1.00
1.74
0.07
0.794
MobileNet VI
(lamuaalre al. 2017
1.05
1.67
0.806
dSELUU(hen et al. 2020b)
0.06
1.12
1.44
0.837
SSELU
0.06
1.00
2.04
0.84
0.805
ISELU
0.06
1.00
1.61
0.14
0.807
sSELU (central)
0.06
1.05
1.79
0.89
0.817
ISELU (central)
0.06
1.05
1.54
0.08
0.818
E EXPERIMENTS ON FULLY-CONNECTED NEURAL NETWORKS
(0017) is first demonstrated fully- The performance of SELU proposed in Klambauer al.
connected neural networks. In this section, we compare our sSELU and ISELU against the orig- inal SELU in laylr fully-connected neural network three typical datasets: UCLminiboone, UCI adult, and HTRU. The results are summarized in Table 6 As the neural network has 64 layers, only evaluate SSELU and ISELU (0.01, 0.017, 0.03). The results show that with all these
18

---

Under review as a conference paper at ICLR 2021
three e, our sSELU and ISELU achieve consistent improvement over SELU, which further justifies the effectiveness of our activation functions.
Table 6: Test accuracy on UCI mCiiiibbone, UCI addll,, and HTRU2 (CI=95%). Method
UCLminiboone
UCI adult
HTRU2
laaubal al. 2017
92.80%
0.13%
84.55% + 0.35%
97.64%
0.26%
0.01
93.01% + 0.30%
84.68% + 0.23%
98.03% E 0. 16%
sSELU (Ours)
0.017
92.90%
0.29%
84.66% + 0.24%
97.73%
0.29%
0.03
92.85%
0.17%
84.56% = 0.27%
97.92% 0.34%
0.01
93.29% + 0. 0.11%
84.86% + 0.18%
97.79%
0.20%
ISELU (Ours)
0.017
93.
4 0.19%
84.63% + 0.24%
98.03% - 0..1%
0.03
92.81%
1 0.23%
84.67% 1 0.23%
98.01% = 0.20%
19


Reviews:
[{"review_id": "mNs0SmVoC8_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Update\n\nI thank the authors for extensive replies and updates to the paper. Most of my questions are answered, and the paper quality is substantially improved. I would not be opposed if other reviewers recommends to accept it. Unfortunately I still can't raise the score and advocate for it myself, since:\n\n1) Table 2 doesn't really show superiority of new nonlinearities, since the bolded (bottom) entry has both trainable $\\lambda$, centralization and Mixup, while the baseline of dSELU only has mixup and trainable $\\lambda$, no centralization. Without centralization (Mixup + trainable $\\lambda$), lSELU/sSELU/dSELU perform comparatively. Without trainable $\\lambda$ (only Mixup), they perform a bit worse than dSELU. With only centralization, BN is still better. Further, even after rebuttal, I still believe that making $\\lambda$ trainable effectively cancels the preceding theoretical discussion, and makes this subset of results somewhat unrelated to the main idea of the paper. Finally, Table 1 shows a more robust benefit over dSELU on CIFAR-100, but not on CIFAR-10/TinyImageNet (where s/lSELU can be both better and worse than dSELU), which is in my opinion underwhelming given the added implementation complexity.\n\n2) Figure 4 is very promising, but I find that SELU doesn't look that bad on it, which again makes me wonder whether the new improved self-normalization actually matters in Tables 1/2, especially given gradient clipping and other heuristics in Table 2.\n\nSo at the moment I find that the proposed nonlinearities are promising in terms of both self-normalization (Figure 4), and generalization (Table 1/2/6), but these results appear to be largely unrelated to each other, and neither of them in separation is strong enough to convince me to try s/lSELU over dSELU (given additional implementation complexity + the boolean hyper-parameter of whether to use lSELU or sSELU) or over BN (given additional hyper-parameter $\\epsilon$). I wish the paper either showed clear use-cases where one can't train BN/SELU/dSELU networks at all in reasonable time (but new nonlinearities allowed it due to superior normalization), or more robust generalization results.\n\nOriginal review below:\n\n# Outline \nThe paper proposes two new nonlinearities designed to normalize the second moment of activations and the Frobenius norm of gradients, and therefore avoid exploding/vanishing activations/gradients. The nonlinearities are evaluated on multiple image datasets. \n\n# Review\n\nOn one hand, I find the theoretical motivation and the idea of adding the new minimization constraint compelling, and empirical performance of nonlinearities promising (notably on CIFAR-100). I appreciate evaluation on multiple datasets, and providing native CUDA code. Further, several decisions along the design process appear well-motivated and backed by experiments (e.g Figure 2, 3, 4).\n\nOn the other hand, I find the theoretical contribution to be relatively incremental from Chen et al (2020b), and unrelated to the uptick in generalization performance in Tables 1 and 2 (motivation for deriving the nonlinearities concern improving trainability, which is not studied experimentally in the paper; better generalization is a nice bonus, but is not predicted by theory). \n\nOn the empirical side, I believe there are important gaps in the experimental results that leave me uncertain of how robust the improved generalization is, and hence whether new nonlinearities justify the added complexity (I believe the claim that no new hyper-paramaters are introduced is false) over SELU or dSELU from Chen et al (2020b). \n\nAs such, at the time of submission neither theory nor experiments appear sufficiently convincing for me to accept the paper, but I am open for discussion.\n\n\nBelow are my specific questions/concerns.\n\n\n## Motivation/Theory:\n1. Do I understand correctly that Definition 2, “new self-normalization definition”, and what follows within section 4 is effectively the same as “partial normalization” in Chen et al (2020b), section 5.3? In either case, I suggest drawing a more explicit connection with Chen et al (2020b) in this section, highlighting precisely what is novel relative to Chen et al (2020b), and potentially softening/clarifying the first bullet point of main theoretical results in the Introduction respectively.\n2. One important weakness of this work is that the paper does not evaluate whether the designed nonlinearity ends up doing what it was designed to do. Namely, we know that in finite networks Assumptions 1-3 don’t hold, Eq. (4) is not exact, and, for $\\lambda \\geq 1$ $\\frac{d \\phi(q)}{d q}|_{q=1}$ is clearly far from $-1$ in Figure 1, notably even deviating further from it than dSELU in Figure 1.c. As such I am not persuaded that s/lSELU has better gradient behavior than dSELU in practice. One could demonstrate such benefits in different ways, perhaps by showing that s/lSELU allows higher learning rate in deep networks on some toy task, or comparing how the gradient norms evolve with depth in deep random networks compared to dSELU etc. Further, even in generalization experiments, as mentioned in appendix B, both this paper and Chen et al. (2020b) clip the gradients by $[-2; 2]$, which, from my understanding, further conceals whether these nonlinearities help normalize gradients or not.\n3. Making $\\lambda$ trainable in s/lSELU is a somewhat disappointing decision/necessity since there is no theoretical reason to do so, and from a quick look at the referenced Zhang et al. (2019) I could not find the justification there - could you please clarify which part of the paper you were referencing? Does $\\lambda$ remain constrained to $\\geq 1$ when trained? \n4. I would like the discussion about “large fan-in networks are likely to lose self-normalization on the mean” (section 6, 3rd paragraph) to be either elaborated, referenced, or confirmed empirically (e.g. on a toy task with networks of increasing fan-ins). Precisely, the conclusion is reached under the assumption that the mean of weights $\\Delta \\mu$ (during training) is independent of fan-in $N_{l-1}$, but isn’t this clearly not true? I.e. weights with larger fan-in are sampled with smaller variance, and I expect them to move less and less during training as the network gets wider, hence increasing fan-in could proportionally decrease $\\Delta \\mu$. In this case these changes cancel out, and the conclusion does not appear justified.\n\n\n## Experiments:\n1. As presented, experiments do not support the introductory claim that “no additional hyper-parameter is introduced\". Firstly, I would prefer if the claim was more explicit, i.e. “no additional hyper-paramater relative to dSELU”. Secondly, even apart from $\\epsilon$, there are still hyper-parameters of a) which among s/lSELU to use; b) whether to have $\\lambda$ trainable or not; c) whether to use Weight Centering, or mixup, or nothing. From reading the abstract, I was expecting a drop-in, no-hyper-parameter activation, but unfortunately there are quite a few knobs to tune.\n2. In both Tables, I believe it’s important to have entries with hyper-parameter-free d/s/lSELU as a baseline, i.e. with $\\epsilon = 1 / L$, to get an understanding of what order of improvement is gained from the additional $\\epsilon$ hyper-parameter. I’m concerned that If $\\epsilon$ is necessary, the relative performance of d/l/sSELU against BN/SELU may be less compelling, especially if the best-performing models were selected on the test set (see next question). \n3. In both tables, was a validation set used to select best performing numbers, or were the numbers selected on the test set (please expand appendix B with this detail)? \n4. Table 2: what $\\epsilon$ was used?\n5. In both Tables, it would be useful to see how sensitive different nonlinearities are to $\\epsilon$, so I suggest adding respective tables for all values in the appendix (also noting the specific values of $\\lambda, \\alpha, \\beta$).\n6. In Table 2: since Chen et al. (2020b) is the primary point of reference for this work, I would argue that for this Table to be convincing, it should also include “dSELU + Mixup” and “dSELU + Mixup + trainable $\\lambda$” (and, ideally,  “dSELU + Weight Centralization”).\n7. In Table 2, could you please include results with non-trainable $\\lambda$ as well?\n8. Was trainable $\\lambda$ a single shared parameter for the whole network, or one per layer?\n\n## Mentioned references from the paper:\n* Chen et al. (2020b): [A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks](https://arxiv.org/abs/2001.00254)\n* Zhang et al. (2019): [Fixup Initialization: Residual Learning Without Normalization]( https://arxiv.org/abs/1901.09321)", "extra_scores": {}, "extra_text": {}}, {"review_id": "l-_XsfvMHco", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new definition for the self-normalization property of a network. Using this definition, the authors propose two new activation functions: sSELU and lSELU. Performance of the new activation functions is tested on benchmarks like CNNs on CIFAR10/CIFAR100/Tiny-ImageNet and MobileNet on ImageNet.\n\nThe new definition of self-normalization is interesting and could potentially be useful. The paper is also written reasonably clearly (see below for minor comments on how I think the writing could be improved). However, I think the experimental section of the paper using the two proposed activation functions have a number of limitations (which I describe below) that explain my rating. I am happy to raise my score if the authors address some of my concerns.\n\nMain comments/questions:\n- In the end, it seems like solving the problem of the shift in the mean is the main thing that makes SELU, sSELU and lSELU work on MobileNets. Is there a problem of gradient explosion in SELU apart from growing means that sSELU or lSELU solves, and if so, what is an experiment that shows this? Without this, it is not clear to me whether the slightly better results of sSELU and lSELU is simply a result of sub-optimal hyperparameter tuning of the baselines (see comment below).\n- The BN+ReLU baseline run on MobileNet seems a bit suboptimal to me. The original MobileNet paper reports 71.7% accuracy, and other work reports even higher numbers, e.g., 72% in https://arxiv.org/pdf/1710.05941.pdf. This could be a result of the learning rate not being tuned for any of the activation functions. It is not obvious to me that the optimal learning rate should be the same across all activation functions, so ideally optimal performance should be shown over a learning rate sweep for each method.\n- The authors mention that neural networks with larger fan-in are more likely to lose the self-normalization effect on the mean based on the observation that if the mean is less than 1/N, then multiplication with the weight decreases the mean of the pre-activations. I do not however follow this argument (and don't think it is true), since the weights are typically initialized from N(0, 1/N), and therefore the probability that the mean is less than 1/N does not change with larger N?\n- How sensitive is performance to epsilon? The fact that epsilon needs to be tuned seems to be a major disadvantage of the proposed activation functions.\n- It would be interesting to see the following ablations as well for the MobileNet problem:\n    - SELU + Weight Centralization\n    - dSELU + Mixup\n    - dSELU + Weight Centralization\n\nOther minor comments:\n- It would have been good to report the values of the parameters alpha, beta and lambda of sSELU and lSELU for at least one problem (such as the MobileNet experiment), and compare them with SELU.\n- Would be good to add SELU in figures 1b, 1d and 3.\n- The authors mention in definition 2 that the self-normalization property is stronger when phi(q) is closer to 1/q. Can the authors expand on why this is? It is not obvious to me from the definition.\n\nAdditional comments on the writing:\n- It is not clear to me how the new definition is \"easier to use both analytically and numerically\".\n- Explanation of related prior work should be improved, as it is not quite clear right now. In particular, the explanation of the prior work in mean field theory, as well as the explanation of \"gradient norm equality\".\n- The first paragraph on the second page needs to be checked for a number of typos (Forbenius -> Frobenius, etc) and grammatical errors.\n- Typo at the end of page 2: boarder -> border\n- Equation 13 and 15 need to be made clearer by adding \"if q_l > 1\", etc.\n\n----------\n\nUpdate after rebuttal: I thank the authors for the detailed response. Some of my comments have been addressed. However, some of my major concerns remain such as the insufficiency of the hyperparameter tuning for the baselines to do a fair comparison. I am keeping my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UeGZ29bAmPa", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose variants of the SELU activation \nfunction that yield a stronger self-normalization property. The \nanalysis in done using mean field theory with several\nassumptions on the randomness of quantities in a neural\nnetwork. \n\nPros:\na) The paper is clearly written and the line of thought can \neasily be followed. \nb) The notation is clean. \nc) The mathematical formulations are sound.\nd) The connections between SNNs and the paper series\nby Poole et al. and Schoenholz et al. has been made\nquite clear. \n\nCons:\na) The work has limited relevance due to the \nfact that it is unclear whether the proposed property is \nindeed crucial for learning and that the empirical results\nare biased and far from state-of-the art. \nAs stated by Goodfellow et al. in their \"Deep Learning Book\"\n(Section 6.3.3), \"New hidden unit types that perform roughly comparably to known types are so common\nas to be uninteresting.\" They provide an example using cos() as activation function reaches a test error\nbelow 1% on MNIST. This implies that the machine learning community should be rigorous in the\nassessment whether a new activation function is worth publishing in order to avoid drowning literature\nabout activation functions. Furthermore, the number of potential activation functions for neural networks is\ninfinite. Therefore, activation function research should be focused around those activation functions with\ninteresting theoretical properties or -- if no theoretical properties are given -- the new activation\nfunctions should increase the state-of-the-art of predictive performance.\ni) The mathematical derivations use several assumptions on the quantities, such as weights and\nactivations. These assumptions would require commenting on how strong they constrain\nthe applicability of this theory. The author should comment on how strong those assumptions are\nand how they can be leveraged.\nii) It is unclear why a stronger normalization property should improve learning. The authors\nshould back this more.\niii) Overall, the derivations are rather related to the backward pass. The authors should \ninclude a view on their activation functions in terms of vanishing/exploding gradients (see also [1]).\niv) The main concern of the reviewer are the experiments. Firstly, SNNs were introduced with\nrelatively large-scale experiments on fully-connected networks. In order to demonstrate \nimprovements, the suggested activation functions shoudl be compared in that set of experiments.\nThe  presented experiments are done with architectures that are relatively far \nfrom the current SOTA on CIFAR and Imagenet [2], such that their relevance cannot be judged well. \nThe authors should introduce their suggested activation functions into SOTA architectures.\nv) The presented performance metrics suffer from a hyperparameter selection bias, since \nthe best epsilon-parameters are selected (section 7.1.). The authors should select \nhyperparameters of their method on a validation set. \n\nb) The novely of this could be stated clearer. Large parts of the derivations are \nsimilar to those by Poole et al, 2016. The authors should make their theoretical contributions\nclearer. \n\n\nQuestions: \nQuestions are implicitly contained in the suggestions above.\n- Can you elaborate more on the connection to mixup?\n- Skip/shortcut connections typically pose a problem both theoretically and practically [3]. How are they handled in this work?\n\n\n\nMinor: \n1) English editing might be required. Should it say \"Redefining THE self-normlization property\"?\n\n\nReferences:\n[1] Hoedt, P.J., Hochreiter, S. and Klambauer, G. Characterising activation functions by their backward dynamics around forward fixed points. Critiquing and Correcting Trends in Machine Learning workshop at NeurIPS 2018.\n[2] Xie, Q., Luong, M. T., Hovy, E., & Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10687-10698).\n[3] Huang, Z., Ng, T., Liu, L., Mason, H., Zhuang, X., & Liu, D. (2020, May). SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6854-6858). IEEE.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2jyUA7LDqFQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary \n\nThe paper proposes two modifications to SELU activation function to improve it with regards to preserving forward-backward signal propagation in neural networks. The work builds on top of the mean-field theory literature and provides a modified self-normalization property (additional constraints compared to SELU). Further, it discusses some heuristics (mixup, weight centralization) to improve performance in practice.\n\n## Strengths\n\n1. The problem is interesting and important to improve trainability in neural networks. It is shown recently that SELU suffers from gradient explosion and this work attempts to circumvent it by redefining the self-normalization condition. This condition leads to one more scalar parameter in SELU function and discusses a method to obtain this scalar.\n\n2. Experiments are conducted on a 56-layer CNN for cifar and tinyimagenet datasets and modified mobilenetv1 for imagenet. These proposed modifications to SELU seem to result in improved accuracy even though the improvement is minimal on imagenet.\n\n3. Overall the paper is clearly written.\n\n## Weaknesses\n\nThe main weaknesses of the manuscript in my opinion are as follows:\n\n1. There seem to be multiple gaps in the theory and the modifications do not mitigate gradient explosion:\n\t- The theoretical analysis lacks rigor and the main purpose of mitigating the gradient explosion issue of SELU is not sufficiently addressed. In fact, if I understand correctly, even with the redefined self-normalization property gradient vanishing/explosion can occur. Note, according to Eq. 15, when $q_l <1$, the numerator can grow arbitrarily. Please clarify.\n\t- Related to the above point, the 3rd condition in Eq. 12 and the subsequent argument of gradient norm converges to the fixed point is unsubstantiated. To my understanding, Eq. 15 only gives a lower/upper bound depending on the value of $q$ and it does not convey anything regarding convergence to such a fixed point. If it does a rigorous proof would be required.\n\t- The assumption in Eq. 4 is unjustified. In Fig. 3 there is a plot provided but it has several underlying assumptions (eg, small $\\epsilon$ and $E[x^2]$ etc.) and it is not clear what functions would satisfy this assumption. I recommend the authors to look at Gaussian-Poincare inequality for connecting a function and its derivative under the Gaussian distribution. In fact, GP inequality is recently used to define a class of activation functions to solve gradient vanishing/explosion in neural networks [a].\n\t- The condition on $\\lambda$ (Eq. 19) is derived without any theoretical basis and it seems like it is found empirically. This also weakens the theoretical contribution of the paper.\n\n2. Contradicting arguments compared to dynamical isometry and mean-field theory literature:\n\t- In dynamical isometry and mean-field theory literature it is shown that dynamical isometry is sufficient to ensure stable gradients and for RELU the scalar is computed to be $\\sqrt{2}$. Refer [b]. To this end, it is confusing to me when it is mentioned that this condition \"will lose self-normalization\" in the second last para of page 4. Please precisely define what is meant by self-normalization (including its purpose) in this paper and also clarify this confusion.\n\t- In the literature it is known that when the width is large, the preactivations will be close to a 0 mean Gaussian distribution due to the central limit theorem (Assumption 2 in this paper). However, in 3rd last para of page 7, it is mentioned that \"networks with large fan-in are more likely to lose the self-normalization effect\". This seems to be contradicting as well.\n\n3. Discussion about the mean of activations:\n\t- The discussion about the mean of activations exploding in Sec. 6 seems irrelevant to me. In the theoretical derivation of this paper, I could not find any discussion on the mean of activations affecting any of the desired quantities such as $q$ or $\\phi(q)$. Please improve the clarity on how this is connected to the theory of this paper.\n\n4. No experiments on gradient vanishing or exploding behaviour:\n\t- As far as I understood, the main motivation is to fix the gradient exploding issue of SELU. However, there is no experiment showing this effect and this seems to reinforce my concern that the proposed modification does not solve the gradient vanishing/exploding issue (also mentioned briefly in the para before conclusion). This questions the significance of the contributions.\n\t- I understand that generalization error is the quantity that we mostly care about. However, self-normalization or the work in mean-field theory literature concerns on trainability. Therefore, I believe, it is important to show the trainability behaviour and the propagation of gradients. Note, the theory does not convey anything about the generalization error directly but the signal propagation in the forward and backward directions.\n\nIn retrospect, I just want to say that in the current form theory and heuristics are mixed together making it difficult to see where the benefit is coming from and I think if theory is tightened this could be a good paper. \n\n## Minor Comments\n\n1. It seems the modifications could be done to other activation functions as well such as tanh etc. Please consider.\n2. Please explain what is the meaning of $\\epsilon$ in Eq. 10. This is not clear from this paper.\n3. First para in Sec. 7.1 : \"considerably higher\"\n\n## References\n\n- [a] Lu, Y., Gould, S. and Ajanthan, T., 2020. Bidirectional Self-Normalizing Neural Networks. arXiv.\n- [b] Pennington, J., Schoenholz, S. and Ganguli, S., 2017. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. NeurIPS.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed two variants of the SELU activation function, termed the leaky SELU (lSELU) and scaled SELU (sSELU), respectively, in order to yield a stronger self-normalization property. The review process and the discussion find the following issues:

- The hyperparameter tuning for the baselines is insufficient for the baselines so that the comparison may be unfair. 
- The experiment results (Table 2) do not show superiority of the proposed activation functions. In addition, the results appear to be unrelated to each other. (see Reviewer 3's detailed update)
- Reviewer 2 pointed out that the architecture that the authors used was far from the SOTA. I read the authors' response. This paper may benefit from adding some even naive workaround and making fair comparisons under the SOTA architecture. 

I do not think (6) is a good way to present this equation. The authors may want to perform change of variable and replace $\sqrt{q}z$ by $z$ in the integral and add this form to the right-hand side of (6). In addition, $\epsilon$ appears in Definition 2. However, when the authors mention the self-normalization property in Definition 2, they omit $\epsilon$. It might be better to call it $\epsilon$-self-normalization property to stress that this definition depends on $\epsilon$.  

Other minor issues:
- The line right below (15) on page 11, the authors did not need to capitalize "orthogonal".
- In eq (16), $W_l^{,T}$, the comma is unnecessary. 
Decision: Reject