Bidirectionally Self-Normalizing Neural Networks
Authors: Yao Lu
Stephen Gould
Thalaiyasingam Ajanthan
Venue:  (2021)

Abstract: The problem of exploding and vanishing gradients has been a long-standing obstacle that hinders the effective training of neural networks. Despite various tricks and techniques that have been employed to alleviate the problem in practice, there still lacks satisfactory theories or provable solutions. In this paper, we address the problem from the perspective of high-dimensional probability theory. We provide a rigorous result that shows, under mild conditions, how the exploding/vanishing gradient problem disappears with high probability if the neural networks have sufficient width. Our main idea is to constrain both forward and backward signal propagation in a nonlinear neural network through a new class of activation functions, namely Gaussian-Poincaré normalized functions, and orthogonal weight matrices. Experiments on both synthetic and real-world data validate our theory and confirm its effectiveness on very deep neural networks when applied in practice.

Under review as a conference paper at ICLR 2021
BIDIRECTIONALLY SELF-NORMALIZING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of exploding and vanishing gradients has been a long-standing ob- stacle that hinders the effective training neural networks. Despite various tricks and techniques that have been employed to alleviate the problem in practice, there still lacks satisfactory theories or provable solutions. In this paper, we address the problem from the perspective of high-dimensional probability theory. We provide a rigorous result that shows, under mild conditions, how the exploding/vanishing gradient problem disappears with high probability if the neural networks have sufficient width. Our main idea to constrain both forward and backward signal propagation in nonlinear neural network through new class of activation fune- tions, namely Gaussian- Poincaréé normalized functions, and orthogonal weight matrices. Experiments on both synthetic and real-world data validate our the- ory and confirm its effectiveness on very deep neural networks when applied in practice.
INTRODUCTION
Neural networks have brought unprecedented performance in various artificial intelligence tasks (Graves et al. 2013 Krizhevsky al., 2012, Silver et al. 2017). However, despite decades of research, training neural networks is still mostly guided by empirical observations and successful training often requires various heuristics and extensive hyperparameter tuning. It is therefore desir- able to understand the cause of the difficulty in neural network training and to propose theoretically sound solutions.
A major difficulty is the gradient exploding/vamishing problem (Glorot & Bengio 2010 Hochreiter 1991 Pascanu et al., 2013] Philipp et al., 2018). That is, the norm of the gradient in each layer either growing or shrinking at exponential rate as the gradient signal is propagated from the top layer to bottom layer. For deep neural networks, this problem might cause numerical overflow and make the optimization problem intrinsically difficult, the gradient in each layer has vastly different magnitude and therefore the optimization landscape becomes pathological. One might attempt to solve the problem by simply normalizing the gradient each layer. Indeed, the adaptive gradient optimization methods (Duchi et al., 2011] Kingma & Ba 2015) Ticleman & Hinton 2012) implement this idea and have been widely used practice. However, one might also wonder there is a solution more intrinsic to deep neural networks, whose internal structure if well-exploited would lead to further advances.
To enable the trainability of deep neural networks, batch normalization (loffe & Szegedy 2015) was proposed recent years and achieved widespread empirical success. Batch normalization is differentiable operation which normalizes its inputs based on mini-batch statistics and inserted between the linear and nonlinear layers. is reported that batch normalization can accelerate neural network training significantly (Ioffe & Szegedy) 2015). However, batch normalization does not solve the gradient exploding/vanishing problem (Philipp et al. 2018). Indeed is proved that batch normalization actually worsen the problem (Yang al. 2019). Besides, batch normalization requires separate training and testing phases and might be ineffective when the mini-batch size is small (loffe) (2017). The shortcomings of batch normalization motivate us to search for a more principled and generic approach to solve the gradient exploding/vanishing problem.

---

Under review as a conference paper at ICLR 2021
Alternatively, self- nofmmnorma neural networks (Klambauer et al. 2017) and dynamical isometry theory (Pennington et al., 2017) were proposed to combat gradient exploding/vamishing problem. In self-normalizing neural networks, the output of each network unit is constrained to have zero mean and unit variance. Based on this motivation, a new activation function, scaled exponential linear unit (SELU), was devised. In dynamical isometry theory, all singular values of the input-output Jacobian matrix are constrained to be close to one at initialization. This amounts to initializing the functionality of network be close to an orthogonal matrix. While the two theories dispense batch normalization, it is shown that SELU still suffers from sxploding/vanislee gradient problem and dynamical isometry restricts the functionality of the network to be close to linear (pseudo- linearity) (Philipp et al. 2018).
In this paper, we follow the above line of research to investigate neural network trainability. Our contributions are three-fold: First, we introduce bidirectionally seff-normalizing neural network (BSNN) that consist of orthogonal weight matrices and new class of activation functions which we call Gaussian-Poineee Poincaré normalized (GPN) functions. We show many common activation functions can be easily transformed into their respective GPN versions. Second, we rigorously prove that the gradient exploding/ vanisshin problem disappears with high probability in BSNNs if the width of each layer is sufficiently large. Third, with experiments on synthetic and real-world data, we con- firm that BSNNs solve the gradient vanishing/exploding problem to large extent while maintaining nonlinear functionality.
2 THEORY
In this section, we introduce bidirectionally self normalizng neural network (BSNN) formally and analyze its properties. All the proofs of our results are left to Appendix.
To simplify the analysis, we define a neural network in restricted sense as follows:
Definition 1 (Neural Network), A neural network is a fuunattinnfaan Rn" to R composed of layer- wise operations for I,.... I. h
(0)
x(+1) = o(h()),
(1)
where ( E Raxn 6: R + R is differentiable applied element-wise to a vector, x (1) is the input and x (I+1l) is the output.
Under this definition, @ is called the activation function, [W are called the parameters, v is
called the width and L. is called the depth. Superscript (?) denotes the I-th layer of neural network. The above formulation is similar to (Pennington et al. 2017) but we omit the bias term in (T) for simplicity plays role in our amalysis.
Let E be the objective function of (W and D (!) be diagonal matrix with diagonal elements D = 6((h.)) ("), where ø' denotes the derivative of d. Now, the error signal is back propagated via
OE
(L)
(L)
d = D (w
and the gradient of the weight matrix for layer can be computed as
OF
a
(3)
OW
To solve the gradient exploding/vanishing problem, we constrain the forward signal x( and the backward signal ( in order to constrain the norm of the gradient. This leads the following definition and proposition.
Definition 2 (Bidirectional Seff-Normalization) A neural network is bidirectionally self- normalizing if
[xx ||2 = ||x(2| - (1x(2) 2
(4)
|d = (2 _.. (1
(5)
Proposition 1. If a neural network is bidirectionally self-normalizing, then
OE
0E =..- =..-= 0E
(6)
awa)
2w
2

---

Under review as a conference paper at ICLR 2021
In the rest of this section, derive the conditions under which bidirectional self- ceff-normalization is achievable for a neural network.
2.1 CONSTRAINTS ON WEIGITT MATRICES
We constrain the weight matrices be orthogonal since multiplication by an orthogonal ma- trix preserves the norm of vector. For linear networks, this guarantees bidirectionally self- noramliinggoomalizttoo and its further benefits discussed in (Saxe al. 2014). Even for nonlinear neural networks, orthogonal constraints are shown to improve the trainability with proper scaling (iihhkin & Matas 2016 Pennington al. 2017
2.2 CONSTRAINTS ON ACTIVATION FUNCTIONS
To achieve bidirectionally eeffnoormiizignnraaizzziiii for a nonlinear network, it is not enough only to constrain the weight matrices. We also need to constrain the activation function in such a way that both forward and backward signals are normalized. To this end, propose the following constraint that captures the relationship between a function and its derivative.
Definition 3 (Gaussian-Poincar Normalization). Function
E - R is Gaussian-Poincard
normalized if it is difserentiable and
(0.))00)))(()))) = (/t (=))= 11
(7)
The definition is inspired by the following theorem which shows the fundamental relationship be- tween function and its derivative under Gaussian measure.
Theorem Slauusian-Poincaaa Inequality (Bogachev 1998)). Iffunction IR + R is differen-
tiable with bounded ())))) and (e 0 (z)=). then
(o)x)))))(=))= S Eg.-No.
Note that there is an implicit assumption that the input is approximately Gaussian for a Gaussian- Poincaré normalized (GPN) function. Even though this is standard in the literature (Klambauer et al. 2017 Pennington et al. 2017 Schoenholz et al. (017). we will rigorously prove that this assumption valid when orthogonal weight matrices used in equation I Next, we state a property of GPN functions.
Proposition 2. Function R - TR. is Gaussian-Poinnaar normalized and E (. = 0 and only if dδ())
(xz) -r.
This result indicates that any nonlinear function with zero mean under Gaussian distribution (e.g.. Tanh and SELU) is not GPN. Now show that large class of activation functions can be converted into their respective GPN versions using an affine transformation.
Proposition 3. For any differentiable function è : R + R with non-zero and bounded E (==NN(o,)(((a))) [=xx]²] and Ez-N (o' (2) there exist two constants a and b such that aδ(x) + b
Gaussian-Poinenn Poincaré normalized.
To obtain and b, one can use numerical procedure to compute the values of Eo (z)²1 ((=()()))(((()²)) and (#(()) and then solve the quadratic equations
E o' (z) 1,
(9)
Berss((.))((a(t)) + =
(10)
We computed and (not unique) for several common activation functions with their default hy- perparametere and the results are listed in Table 1. Note that ReLU, ReakyReLU and SELU are not differentiable ignore such point and evaluate the integrals for x E (-oo, 0) U (0, oo).
but they can be regarded approximations of their smooth counterparts. We
With the orthogonal constraint the weight matrices and the Caussian-Poincart normalization the activation function, we prove that bidirectionally aalinorrrmiizznnommaiiaattiooooo is achievable with high probability under mild conditions in the next subsection.
We 0.01 for LeakyRel.U, 1 for ELU and ((x) 2/(11 exp(-1.7022)) for GELU.
3

---

Under review as a conference paper at ICLR 2021
Tanh
ReLUU
LeakyRel.U
ELU
SELU
GELU
a
1.4674
1.4122
1.4141
1.2234
0.9660
1.4915
6
0.3885
0.0000
0.0000
0.0742
0.2585
-0.9097
Table 1: Constants for Caussian-Poincart normalization of activation functions.
2.3 NORM-PRESERVVAIIN THEOREMS
The bidirectionally noramiizinngnoomalizttii may not be achievable precisely in general unless the neural network is a linear one. Therefore, we investigate the properties of neural networks in a probabilistic framework. The random matrix theory and the high-dimensional probability theory allow us to characterize the behaviors of large class of neural networks by its mean behavior, which is significantly simpler to analyze. Therefore, we study neural networks of random weights whose properties may shed light on the trainability of neural networks practice.
First, we need probabilistic version of the vector norm constraint.
Definition (Thin-Shell Concentration). Random vector x E IR" is thin-shell concentrated if for any
P
| -0-111 1 7
-
(11)
as 72 + DC.
The definition is modified from the one in Bobkoo 2003). Examples thin-shell concentrated dis- tributions include standard multivariate Gaussian and any distribution on the n-diimensionll sphere of radius vn.
Assumptions. To prove the main results, ie., the norm-preservation theorems, we require the following assumptions:
1. Random vector E RD is thin-shell concentrated.
2. Random orthogonal matrix W = (W1, W2, w,) is uniformly distributed.
Coausiaan-Poiccat normalized. 4. Function & and its derivative are Lipschitz continuous.
3. Function R +
The above assumptions are not restrictive. For Assumption 1, one can always normalize the input vectors of a neural network. For Assumption 2, orthogonal constraint its relaxation has already been employed neural network training (Brock et al.) 2017). Note, in Assumption 2, uniformly distributed means that W is distributed under Haar measure, which is the unique rotation invariant probability measure on orthogonal matrix group. We refer the reader to (Meckes) 2019) for de- tails. Furthermore, all the activation functions their smooth counterparts listed in Table satisfy Assumptions 3 and 4.
With the above assumptions, can prove the following norm-preservation theorems.
Theorem (Forward Norm-Preservation). Random vector
(ø(w (((wTx)),
(12)
thin-shell concentrated.
This result shows the transformation (orthogonal matrix followed by the GPN activation function) can preserve the norm of its input with high probability. Since the output is thin-shell concentrated, it serves as the input for the next layer and so on. Hence, the forward pass can preserve the norm of its input in each layer along the forward path when n sufficiently large.
Theorem 3 (Backward Pres-rreerrvttion). Let D be the diagonal matrix whose diagonal el- (w!x) x) and y E 12n be a fixed vector with bounded ||y|loo- Then for any
ements are Du = p'(w?'x
<>
p
Dy h
1 2 0
(13)
CC.
4

---

Under review as a conference paper at ICLR 2021
This result shows that the multiplication by D preserves the norm of its input with high probability. Since orthogonal matrix W also preserves the norm of its input, when the gradient error signal is propagated backwards in (2. the norm is preserved in each layer along the backward path when n sufficient large.
Hence, combining Theorems and proved that bidirectionally self nnnnnaamiizinnnomaizztttonn achievable with high probability the neural network is wide enough and the conditions in the Assumptions are satisfied. Then by Proposition the gradient exploding/vanishiin problem disap- pears with high probability.
Sketch of the proofs. The proofs of Theorems 2 and B are mainly based on a phenomenon in high-dimensional probability theory, concentration of measure. We refer the reader to (Vershynin 2018) for an introduction to the subject. Briefly, can be shown that for some high-dimensional probability distributions, most mass is concentrated around certain range. For example, while most mass of low-dimensional standard multivariate Gaussian distribution is concentrated around the center, most mass of high-dimensional standard multivariate Gaussian distribution is concentrated around a thin-shell. Furthermore, the random variables transformed by Lipschitz functiios are also concentrated around certain values. Using this phenomenon, can shown that rows { {w,} of random orthogonal matrix W in high dimension are approximately independent random unit vectors and the inner product w. x for thin-shell concentrated vector x can be shown be approximately Gaussian. Then from the assumptions that ø is GPN and ø and ø' are Lipschitz continuous, the proofs follow. Each of these steps is rigorously proved Appendix.
EXPERIMENTS
We verify our theory on both synthetic and real-world data. More experimental results can be found in Appendix. In short, while very deep neural networks with non-GPN activations show vanishing/cxploding gradients, GPN versions show stable gradients and improved trainability in both synthetic and real data. Furthermore, compared to dynamical isometry theory, BSNNs do not exhibit pseudo- lineariiy and maintain nonlinear functionality.
3.1 SYNTHETIC DATA
We create synthetic data to test the preeperesevvtiion properties of the neural networks. The input x! is 500 data points of random standard Gaussian vectors of 500 dimension. The gradient error 0E/0xL+1 is also random standard Gaussian vector of 500 dimension. All the neural networks have depth 200. All the weight matrices random orthogonal matrices uniformly generated. No training is performed.
In Figure D we show the norm of inputs and gradient of the neural networks of width 500. From the results, we can see that with GPN, the gradient exploding/vanishing problem is eliminated to large extent. The neural network with Tanh activation function does not show gradient explod- ing/vanishing problem either. However, ||x(]]| is close zero for large and each layer is close to a linear one since Tanh(x) Rs x when x 0 (pseudo-linearity). for which dynamical isometry is achieved.
One might wonder if bidirectionally sef-noraaliiingoomaiizziooo has the same effect as dynamical isometry in solving the gradient exploding/vanishing problem, that is, to make the neural network
close to an orthogonal matrix. To answer this question, we show the histogram of ø' (((h()) in Fig- ure If the functionality of neural network is close to orthogonal matrix, since the weight matrices are orthogonal, then the values o' d'(h,") (?) would concentrate around one (Figure 2 (a)), which is not the case for BSNNs (Figure (b)). This shows that BSNNs do not suffer from the gradient vanishing/explosion problem while exhibiting nonlinear functionality.
In Figure in Appendix, show the gradient norm of BSNNs with varying width. There note that as the width increases, the norm of gradient in each layer of the neural network becomes more equalized, predicted by our theory.
5

---

Under review as a conference paper at ICLR 2021
0.4
0.2
0
100
150
200
50
100
150
200
Layer
Layer
(a) ||x (9) |1 /. Tanh.
(b) ow ||e. Tanh.
30
y WMM M
Mnt
25
0.8
150
100
200
Layer
Layer
(c) ||x (x) 11/n,, Tanh-GPN.
(d) || 8E
aW Il. Tanh-GPN.
x10
1.2
0.8
100
150
200
150
Layer
Layer
(e) |x 18/n, SELU.
BE (f)
UW Ir., SELU.
1.2
My M M M M.
MW
25
20
0.8
50
100
150
200
50
100
150
200
Layer
Layer
||x(*)
|1//n, SELU-GPN.
(h) ||
8E
I1.. SELU-GPN.
Figure 1: Results on synthetic data with different activation functions. "-GPN" denotes the function Gaussian- Gaussian-Poincaré normalized. ||x(() ||2 denotes the l2 norm of the outputs of the 1-th layer. denotes the width. W || p the Frobenius norm of the gradient of the weight matrix in the 1-th layer.
x10
x10
2
0
0.2
0.4
0.6
0.8
0.5
1.5
(a) Tanh.
(b) Tanh-GPN.
Figure 2: Histogram of d((h;)) ()).. The values of g' (1())) are accumulated for all units, all layers and all samples in the histogram.

---

Under review as a conference paper at ICLR 2021
20
20
GR LUGPN
GB LHRR
20
40
60
80
100
20
40
60
80
100
Epoch
Epoch
(a) Tanh.
(b) SELUU
Figure 3: Test accuracy (percentage) during training on CIFAR-10. "-BN" denotes that batch nor-
malization is applied before the activation function.
MNIST
CIFAR-IO 10
Train
Test
Train
Test
Tanh
99.05 (87.39)
96.57 (89.32)
80.84 (27.90)
42.71 (29.32)
Tanh-GPN
99.81 (84.93)
95.54 (87.11)
96.39 (25.13)
40.95 (26.58)
RcLU
11.24 (11.24)
11.35 (11.42)
10.00 (T0.00)
10.00 (T0.00)
ReLU-GPN
33.28 (11.42)
28.13 (11.34)
46.60 (10.09)
34.96 (9.96)
LeakyReLU
11.24 (11.24)
11.35 (11.63)
10.00 (10.21)
10.00 (10.06)
43.17 (11.19)
49.28 (11.66)
51.85 (9.89)
39.38 (10,00)
ELU
99.06 (98.24)
95.41 (97.48)
80.73 (42.39)
45.76 (44.16)
ELU-GPN
100.00 (97.86)
96.56 (96.69)
99.37 (43.35)
43.12 (44.36)
SELU
99.86 (97.82)
97.33 (97.38)
29.23 (46.47)
29.55 (45.88)
SELU-GPN
99.92 (97.91)
96.97 (97.39)
98.24 (47.74)
45.90 (45.52)
GELU
11.24 (12.70)
11.35 (10.28)
10.00 (10.43)
10.00 (10.00)
GELU-GPN
97.67 (11.22)
95.82 (9.74)
90.51 (10.00)
36.94 (10.00)
Table 2: Accuracy (percentage) neural networks of depth 200 with different activation functions on real-world data. The numbers in parenthesis denote the results when batch normalization is applied before the activation function.
3.2 FEAL-WORLD DATA
We run experiments on real-world image datasets MNIST and CIFAR-10. The neural networks have width 500 and depth 200 (plus one unconstrained layer at bottom and one at top to fit the dimensionality of the input and output). We use stochastic gradient descent of momentum 0.5 with mini-batch size 64 and learning rate 0.0001. The training run for 50 epochs for MNIST and 100 epochs for CIFAR-10, 10. We do not use data augmentation. Since it is computationally expensive to enforce the orthogonality constraint, we simply constrain each row of the weight matrix have la norm one relaxation of orthogonality by the following parametrization W = [vi/[[villa. v2/l|v2lla,.... v./l/v,,l2)) and optimize V V2., Vn.) Vn) as an uncon- strained problem.
We summarize the results in Table 2 We can see that, for activation functions ReLU, LeakyReLU and GELU, the neural networks not trainable. But once these functions are GPN, the neu- ral network can be trained. GPN activation functions consistently outperform their unnormalized counterparts terms of the trainability, as the training accuracy increased, but not necessarily generalization ability. We show the test accuracy during training in Figure 3 from which we can see the training is accelerated when SELU is GPN, ReL.U, LeakyRel.U and GELUU if not GPN, are completely untrainable due to the vanished gradient (see Appendix).
We observe that batch normalization leads to gradient explosion when combining with any of the activation functions. This confirms the claim of (Philipp et al., 2018) and (Yang et al. 2019) that batch normalization does not solve the gradient exploding/vaniee vanishing problem. On the other hand, without batch normalization the neural network with any GPN activation function has stable gradient magnitude throughout training (see Appendix). This indicates that BSNNs can dispense with batch normalization and therefore avoid its shortcomings.

---

Under review as a conference paper at ICLR 2021
4 RELATED WORK
We compare our theory to several most relevant theories in literature. A key distinguishing feature of our theory that we provide rigorous proof's of the conditions under which the exploding/vanishing problem disappears. To the best of our knowledge, this is the first time that the problem is provably solved for nonlinear neural networks.
4.1 SELF-NORMALIZING NEURAL NETWORKS
Self-normalizing neural networks enforce zero mean and unit variance for the output of each unit with the SELU activation function (Klambauer et al. 2017). However, as pointed out in (Philipp et al., 2018) and confirmed in experiments, only constraining forward signal propagation does not solve the gradient exploding/vanishing problem since the norm of the backward signal can grow or shrink. The signal propagation in both directions needs to be constrained as in theory.
4.2 DEEP SIGNAL PROPAGATION
Our theory is developed from the deep signal propagation theory (Poole et al. 2016 Schoenholz ct al. 2017). Both theories require (0 (x)) 1. However, ours also requires the quantity
Bg~AA(o.1) (()))) to be one while in Poole et al. (Poole et al. 2016 Schoenholz et al. 2017 it can be an arbitrary positive number. We emphasize that is desirable to enforce Ez-,((0.1)
((x)
to avoid trivial solutions. For example, (xx) = Terh(ex)) with e as 0, then ((er) ex and the neural network becomes essentially a linear one for which depth is unnecessary (pseudo-linearity (Philipp et al. 2018). This is observed in Figure (a). Moreover, in (Poole et al. 2016 Schoenholz
et al. 2017) the signal propagation analysis is done based random weights under i.i.d. Gaussian distribution whereas we proved how one can solve gradient vanishing/exploding problem assuming the weight matrices are orthogonal and uniformly distributed under Haar measure.
4.3 DYNAMICAL ISOMETRY
Dynamical isometry theory (Pennington et al., 2017) enforces the Jacobian matrix of the input- output function neural network to have all singular values close to one. Since the weight matrices are constrained to be orthogonal, the identity matrix, which implies the functionality of neural network at initialization is close to an orthogonal matrix (pseudo-linearity.. This indeed enables trainability since linear neural networks with orthogonal weight matrices do not suffer from the gradient exploding/vanishie neural networks need to learn a nonlinear input-output functionality to solve certain tasks, during training the weights of neural network are unconstrained so that the neural network would move to a nonlinear region where the gradient xxploding/vanishing problem might return. In our theory, although the orthogonality of weight matrices is also required, approach the problem from different perspective. We do not encourage the linearity at initialization. The neural network can be initialized to be nonlinear and stay nonlinear during the training even when the weights constrained.
equivalent to enforce each D in (2) to be close to
problem. As
5 CONCLUSION
In this paper, we have introduced bidirectionally self-normalizing neural network (BSNN) which constrains both forward and backward signal propagation using new class of Caussian-Poincart normalized activation functions and orthogonal weight matrices. BSNNs are not restrictive in the sense that many commonly used activation functions can be Caussian-PPoneaan have rigorously proved that gradient vanishing/exploding problem disappears in BSNNs with high probability under mild conditions. Experiments on synthetic and real-world data confirm the validity of our theory and demonstrate that BSNNs have excellent trainability without batch normalization. Currently, the theoretical analysis limited to same width, fully-connected neural networks. Future work includes extending theory more sophisticated networks such convolutional architec- tures as well as investigating the generalization capabilities of BSNNs.
normalized. We
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tom Alberts and Davar Khoshnevisan. Calculus on Gauss Space: An Introduction to Gaussian Analysis. 2018.
Sergey G Bobkov. On concentration of distributions of random weighted sums. Annals of Probabil- ity, 2003.
Vladimir Igorevich Bogachev. Gaussian Measures. American Mathematical Society, 1998.
Andrew Brock, Theodore Lim, James Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. International Conference on Learning Representations, 2017.
John Dawkins. Normalized vector of gaussian variables is uniformly distributed on the sphere. 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011.
Nate Eldredge. When do ø² and 6² have the same expectation under gaussian random variable? MathOverflow 2020. https:/maathovrffloo.nee///////
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. International Conference on Artificial Intelligence and Statistics, 2010.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. IEEE International Conference Acoustics, Speech and Signal Processing, 2013.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Uni- versitär München, 1991.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch mozealnormmllzed models. Advances in Neural Information Processing Systems, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference Machine Learning. 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International
Conference on Learning Representations, 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing
neural networks. Advances in Neural Information Processing Systems, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in Neural Information Processing Systems, 2012.
Elizabeth Meckes. The Random Matrix Theory of the Classical Compact Groups. Cambridge University Press, 2019.
Dmytro Mishkin and Jiri Matas. All you need is good init. International Conference Learning Representations, 2016.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. International Conference Machine Learning, 2013.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep leamn ing through dynamical isometry: theory and practice. Advances in Neural Information Processing Systems, 2017.
George Philipp. Dawn Song, and Jaime G Carbonell. The exploding gradient problem demystified- definition, prevalence, impact, origin, tradcoffs, and solutions. arXiv preprint arXiv:1712.055: 2018.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Dickksicsst,n, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in Neural Information Processing Systems, 2016.

---

Under review as a conference paper at ICLR 2021
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics learning in deep linear neural networks. International Conference on Learning Represen- tations, 2014.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. International Conference on Learning Representations, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 2017.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Roman Vershynin. High- Digh--Dimesiooall Probability: An Introduction with Applications in Data Sci- ence. Cambridge University Press, 2018.
Greg Yang. Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel Schoenholz. mean field theory of batch normalization. International Conference on Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX A PROOFS
Proposition If a neural network is bidirectionally self-normalizing, then
0E
=...= = 0E
Taw
2W
(w(
(14)
F
Proof. For each I, we have
OE
DE OWO OW()) )")
OE
@ww
trace
(15)
trace(d (x(1) (d")))
(16)
V (race((x()) (d))
(17)
V (x(0))xX x()) (d
(18)
1aax"1 1|2
(19)
By the definition of bidirectional self-normalization, we have
||
AE
BE
F.
SWiLT
Proposition 2. Function @ : R + R is Gaussian-Poincaar normalized and [(--((01)00(=) = 0 if and only if dbx) or ((x) x.
Proof. Since (0, ( < oo and (an_ co, d(x) and (xxx) can be expanded in terms of Hermite polynomials. Let the Hermite polynomial of degree & be
(-1)
d
2
Hx(x)=
exp
exp(
(20)
Vh!
2 du
2
and due to Hx)x) VkHH-1 we have
<(x) =
Σ
a Hx)))
(21)
(x) =
Vhay (x))
(22)
Since ((..))(((())) we have
ag (on
(23)
Em-ro.nl(==)
(24)
0.
(25)
Since
(o- = (o_ =1
(26)
and Hermite polynomials are orthonormal, we have
E (o, =
a = Ea Σ
ko 1.
(27)
1=1
k=
Therefore, we have
kaj
03 = 0
(28)
1=1
k=1
that is
2 (k 1)a 0.
(29)
Since each term in - 1)a2 nonnegative, the only solution is os 0 for k 2 2. And since [o=x))) of = have a = +1. Hence, x(x) + Hi (2) tr.
n
11

---

Under review as a conference paper at ICLR 2021
Proposition 3. For any differentiable function é R R with non-zero and bounded
Eg-N [ and E=~N10.n (x)²}, there exist two constants a and such that aδ(x) + normalized.
is Gaussian- Poincaré Caussian-Poincaré
Proof. Let ((x) ((x) + c. Then let
(c) = E -aw.:uulllaall----ovvolllllll
(30)
= Varr=-voon)(((=)]] +
- -En-woon)((d Var=-wro.n]((==]] (E=) + +c)² ((nN)a))l((==))
(31)
(32)
Therefore, (b() is quadratic function of c. We also have v(() vb( ((-BB--N(0.1)0000 I due to Pooncaa inequality. Hence, there exists e for which
as c oo and
(c)) such that Bz=-wionn(((())+)) Vio.)oo)) (x))) Let (LEz) (1)))))())))11111 and ac, we have E.-N + b)3] E (a
The proof is largely due to (Eldredge 2020 with minor modification in here.
Assumptions.
1. Random vector E Rd is thin-shell concentrated.
2. Random orthogonal matrix W = (W1, W2,... w,,) is uniformly distributed. Cotussian-Poincan normalized.
3. Function R >
4. Function & and its derivative are Lipschitz continuous.
Theorem (Forward Sorm-Preservation), Random vector (6(wix)
(33)
is thin-shell concentrated.
Theorem 3 (Backward Norm-Preservation Let D be the diagonal matrix whose diagonal el- ements Di o6 (w(w(x) and y E Rn be a fixed vector with bounded ||y||>. Then for any
6 <> 0
Dy
1y
2 2 -0
(34)
oC.
||x|| = 1]. O(n) is the orthogonal matrix group of size n. (() denotes the indicator function. O, denotes the vector of dimension and elements equal to zero. I, denotes the identity matrix of size
Notations. (x
Lemma 1. If random variable NN 1) and function f IR - R is Lipschitz continuous, then random variable fxx is sub -aussaassian.
Proof. Due to the Gaussian concentration theorem (Theorem 5.2.2 (Vershynin 2018) we have
[[(x) - EE](())]l < CK
(35)
where || 11,,, denotes sub-gaussian norm, C' is constant and is the Lipschitz constant of f. This
implies f(x) [xf is sub-gaussian (Proposition 2.5.2 in /Vershynin 2018 Therefore f(x) is
sub-gaus gaussian (Lemma 2.6.8 in (Vershynin) 2018)).
Lemma 2. Letx (I1, I2, I2,..., .... Zn.) E R be a random vector that each coordinate z; is independent
and sub- sub-gaussiee gaussian and B|n;]| = 1. Let y (21, y2 ....
yn.) E Rnn be a fixed vector with bounded
loy. Then
PP
x 2
Σ 3
20 +
(36)
DC.
12

---

Under review as a conference paper at ICLR 2021
Proof. Since y; is sub-gaussian, then 2 is sub-exponential (Lemma 2.7.6 in Vershynin 2018 ). Since Eigf2i] = y = yr. yj x yf is sub-exponential with zero mean (Exer-
cise 2.7.10 in (Vershynin 2018)). Applying Bernsteins inequality (Corollary 2.8.3 in /erssyyii 2018 )
we proved the lemma.
Lemma 3. Let (0., In). Then for any 0 << < <
P(z R2 -0)m |z]|a S (1 + (1+6) 8) 2 1 2exp(-nod).
(37)
See (Alberts & Khoshnevisan 2018) (Theorem 1.2) for proof.
Lemma 4. Let z (O,,, In). Then z/ //|z2|22 is uniformly distributed S"-1
See (Dawkins) 2016) for proof.
Lemma 5. Let z (21, 22, 22, kn) NO In), = (a1, .2..... ..., ar) ) be a fixed vector with bounded ||a||oc and R + IR be a continuous function. Then for any e 0
P
!
Σ
Lu uf(()s)))) >c}
(38)
as 72 OC.
Proof. Since
Σ @af(vn//zzll23) - Σ a=)((z) S Σ Jail 1f(/n///zlll2) - f(2))),
(39)
if, - 00.
Σ lasl. |F(Vn/112|222) - f(z))) >{ + 0,
(40)
then
P
Σ yn v - - Eul(zz) >{} + 0.
(41)
For < 8 < let
A R" 1 Σ |ail (((v7/1z//22)) - -f f(2)) >
(42)
Us { 2 R", (1 8) v s |2/12 (1+ (1+
(43)
Then
aul |f(vn/|z|222) - f(x))) >> (xa))>>]=/// =/e d(meA)
P
(44)
n
{ jRm ((EEa)d2+ /
Ju. (nEaa)
(45)
Let 5 1144 From Lemma 3, have, 0o,
[ J ((eAA) < { 22n2e dz = 1a=1-Pa P(z E U6} < 2eepp-nn8 - 0.
(46)
1/4 1144, we have ||z||2 + vn, vn/l|zllazi + z; and therefore f( f(vn/|lz|/2z) f(z:) + co. Hence, Sun 1(zEA) + 0, as n 20.
For z E Us and 8 = n
13

---

Under review as a conference paper at ICLR 2021
Lemma 6. Let random matrix W be uniformly distributed ((n) random vector 0 be uniformly distributed on 8"-1 and random vector x E R" be thin-shell concentrated. Then Wx + vn0 as + 00.
Proof. Let y E In be any vector with |ylla vn and e = OV.,0....000 0) E I" Since W is uniformly distributed, Wy has the same distribution as We. We is the first row of vnW, which is equivalent to random vector vnO. Since x is thin-shell concentrated, x Wx + vno as n 20.
and therefore
Tx|
Proof of Theorem 2. Let = (21, 22, 22, ~ (2) (0,
Due Lemma random variable ((2+) is normalized, E, ((00))((()))) = 1. Applying Lemma
sub-gan -a-sgaassiann. Since d is Gaussian- with each y/1 = 1, we have for
Pf Σ ozx) - 12126 2 - -0
(47)
as 12 + 00.
Due Lemma 4 and (with each a; 1), for random vector A (01, 02, uniformly distributed on gm-1, we have
p Σ Φ nn.)
1 )(()) 2 c) +
(48)
and therefore
P
-
Σ (VVT0.)) 1 2 2 +0
(49)
OC.
Then from Lemma 6, we have Wx + vno and therefore
P
Σ (w x)2 - 1 2 +0
(50)
as 72 + DC.
Proof of Theorem 3. Let z = (21, 22, .... 2n) be the standard multivariate Gaussian random vectors. Due to Lemma 1, random variable d' (2,,) ) is sub-uuu sabb-gaussian. Since ø is Couussian-Poincant normalized, Eg (0 (z,) = Applying Lemma we have
P
Σ v2)
5i 2 +0
(51)
OC.
Due to Lemma and 5 (with each a; = y!), for random vector 0 = =(01, 0n) uniformly distributed 1n-1, we have
P | Σ
n E Ezfd(())
1
2 2 +
(52)
and therefore
P
Σ
v of (Vr0.)²
v2 2 2
(53)
as 72 + oc.
Then from Lemma 6, we have Wx Vno and therefore
IP Σ vfo'(w?x)) - 2o { 2 +0
(54)
as nl + D0.
14

---

Under review as a conference paper at ICLR 2021
APPENDIX B ADDITIONAL EXPERIMENTS
Due to the space limitation, we only showed the experiments with Tanh and SELU activation func- tions in the main text. In this section, we show the experiments with ReLU, LeakyReLU, ELU and SELU, Additionally, we also measure the gradient exploding/vanishing during training on the real-world data.
B.1 SYNTHETIC DATA
We show the figures of the experimental results in addition to the ones in the main text.
x10 29
0.6
0.4
2
0.2
1.5
50
100
150
200
200
Layer
Layer
(a) ||x 11) /n. ReLU.
(b)
aw |r.. ReLU.
0g
30
1.5
20
0.5
10
100
150
200
100
150
200
Layer
Layer
(c) ||xx [12/ n, ReI.U-GPN.
(d)
||
DE
||.. ReLU-GPN.
(W (3)
x10
0.6
12 M
0.4
10
8
0.2
6
100
150
200
100
150
200
Layer
Layer
(e) |xx [| n, LeakyRel.U.
(f) ||
P. LeakyReLU.
SWOT
2.5
2
1.5
25
20
0.5
15
100
150
200
100
150
200
Layer
Layer
(g) [|x0) n,, LeakyReLU-GGNN.
(h) || (w
ag
|r. LeakyReLU-GGP
Figure 4: Results on synthetic data with different activation functions. "-GPN" denotes the function Gaunsie normalized. ||x("||2 || denotes the le norm of the outputs of the 1-th layer. denotes the width. || || is the Frobenius norm of the gradient of the weight matrix in the I-th layer.
15

---

Under review as a conference paper at ICLR 2021
0.32
0.6
0.4
0.3
0.2
0
50
100
150
200
50
100
150
200
Layer
Layer
(a) ||x (1) |1
ELU.
(b)
BE SW0
ELU.
1.2
20
wwm
W
18
0.8
50
100
150
200
50
100
150
200
Layer
Layer
(c) (1) 11/111/m ELU-GPN.
(d)
25
||F. ELU-GPN.
0W
x10
-99
0.4
2.5
0.2
2
50
100
150
200
50
150
200
Layer
Layer
(c) [x |13 /, GELU.
(f)
||
ow el.. GELU.
8g
x10
4
1.2
M wy My Iv
Mm
1.2
Mwh
0.8
100
150
200
50
150
200
Layer
Layer
(g) ||x 12/n,, GELU-GPN.
(h)
[[., GELU-GPN.
JWO
Figure 5: Results synthetic data with different activation functions. "-GPN" denotes the function Coussian-PPincart normalized. ||x(0) || denotes the le norm of the outputs of the 1-th layer. denotes the width. || (/away || is the Frobenius nomn of the gradient of the weight matrix in the I-th layer.
16

---

Under review as a conference paper at ICLR 2021
x10
X10
2-
0.2
0.4
0.6
0.8
0.5
1.5
(a) Tanh.
(h) Tanh-GPN.
x10
x107
3
0
0.2
0.4
0.6
0.8
0
0.5 (d) ReLU-GPN.
1.5
(c) ReLUU.
x10
x107
1
0
0.2
0.4
0.6
0.8
0.5
1.5
(e) LeakyReLU.
(1) LeakyReLU-GPN.
x107
x10
2
0
(g) ELU.
(h) HLU-GPN.
x10
x100
3
2
0.5
0.5
1.5
(i) SELU.
SELU-GPN.
10
0
0.2
0.4
0.6
0.8
0.5
1.5
(k) GELU.
(1) GELU-GPN.
Figure 6: Histogram of g' (l(h.)), The values of ø' o'(h,) are accumulated for all units, all layers and all samples in the histogram. Except for ELU, none of them has values concentrated around one.
17

---

Under review as a conference paper at ICLR 2021
15
15
10
5
500
1000
1500
500
1000
1500
Width
Width
(a) Tanh-GPN.
(b) ReL.U-GP.
6
500
1000
1500
500
1000
1500
Width
Width
(c) LCPNyRLLUUGN.
(d) ELU-GPN.
10
10
5
0
500
Width (e) SeLU-GPN.
Width
(i) GELU-GPA GPN..
Figure 7: Gradient norm ratio for different layer width synthetic data.
The ratio is defined
as max; sWoy //// miny 8E
awi ||.. The width ranges from 100 to 1500. The error bars show
standard deviation.
B.2 REAL-WORKS DATA
In Figure we show the test accuracy during training on MNIST. In Figure 9 we show the test accuracy during training on CIFAR-10.
we show a measure of gradient exploding/vanishiin during training for different activation functions. The measure is defined as the ratio of the maximum gradient norm and the minimum gradient norm across layers. Since we use the parametrization
In Figure I0
12 and 13
Vi
V2
V
(55)
|vill
with V = (V1, V2,... VT, the gradient norm ratio is defined on the unconstrained weights V, that is,
BE
maxi
av
F
(56)
minz
BE
Note that for ReLU, LeakyReLU and GELU, the gradient vanishes during training in some exper- iments and therefore the plots are empty. From the figures, we can see that batch normalization leads to gradient explosion especially at the early stage training. On the other hand, without batch normalization, the gradient is stable throughout training for GPN activation functions.
18

---

Under review as a conference paper at ICLR 2021
100
100
50
50
www...00mmo 201
Epoch
Epoch
Tanh.
(b) ReLU.
100
100
50
M
ROLU J PUILU 3N
-C (1
20
30
50
10
20
30
40
50
Epoch
Epoch
(c) LeakyReLU.
(d) ELU.
100
100
50
50
LL CPN
10
20
30
40
50
10
20
30
40
50
Epoch
Epoch
(c) SELU.
(f) GELU.
Figure 8: Test accuracy (percentage) during training on MNIST.
40
40
20
20
PODLU
DELL--R
20
40
60
80
100
20
40
60
80
100
Epoch
Epoch
(a) Tanh.
(b) SELU.
40
40
20
R2:0
Lankenaaiicco GPN
Rallk-GP
0
100
80
100
Epoch
Epoch
(c) ReLU.
(d) LeakyReLU.
40
40
20
-ORLD OPN
-10-57N-30 2h
0
-OBLU-GPP-0 9N
20
80
80
100
20
40
80
80
100
Epoch
Epoch
(e) ELU.
(I) GELU.
Figure 9: Test accuracy (percentage) during training on CIFAR-10.
19

---

Under review as a conference paper at ICLR 2021
100
20
50
2
3
x10
x10
(a) Tanh.
(b) Tanh-GPM
x10
x10
2
1
2
x10
x x10*
(c) Tanh-BN.
(d) Tanh-GPN-BN.
300
200
0.5
100
x100
(e) ReLU.
(f) ReLU-GPN.
x10
x10
5
2
0
x10
x10
(g) ReLU-BN.
(h) ReI.U-GNN-BN.
800
600
0.5
400
200
x10
x x10
(i) LeakyReLU.
LeakyReLU-GPN.
x10
x10
2
2
x10
x10
(k) LeakyRel.U-BN.
()) LeakyRdl.UGGPN-BN.
Figure 10: Gradient norm ratio during training on MNIST. Horizontal axis denotes the mini-batch updates. Vertical axis denotes the gradient norm ratio maxi | F// min: || Vavi || The gradient
vanishes (I pvo | Rs 0) for ReLU and LeakyReLU during training and hence the plots are empty.
20

---

Under review as a conference paper at ICLR 2021
40
100
50
0
0
2
3
2
4
x10
x10
(a) ELU.
(b) ELU-GPN.
1000
1000
500
3
4
2
3
x10
x10
(c) ELU-BN.
(d) ELU-GPN-BD BNN.
2000
1000
20
0
3
4
3
4
x10
(e) SELU.
(l) SELU-GPN.
1500
1000
1000
500
0
0
1
3
4
2
3
x10
x10
(g) SELU-BN.
(h) SELU-GPN-BR
400
0.5
200
x10
x10
GELU.
GELU-GPN.
x:10
20
8
6-
0.5
4.
2
x10
x10
(k) GELU-BN.
()) GELU-GPN-BN,
Figure 11: Gradient norm ratio during training on MNIST, Horizontal axis denotes the mini-batch updates. Vertical axis denotes the gradient norm ratio max; V P miny Vav ll.. The gradient aE
vanishes (I) gyur || F R 0) for GELU and GELU-BN during training and hence the plots are empty.
21

---

Under review as a conference paper at ICLR 2021
40
100
20
2
x100
x10
(a) Tanh.
(b) Tanh-GPP
x10
x10
3
2
2
6
x10
x10
(c) Tanh-BN.
(d) Tanh-GPN-BN.
400
0.5
200
x10
x10
(e) ReLU.
(f) ReLU-GPN.
x10
8
2
x10
x10
(g) ReLU-BN.
(h) ReLU-GPN-BN,
400
0.5
200
x10
x10
(i) LeakyReL.U.
LeakyReLUUGGN.
x10
x10
2
x10
x10
(k) LeakyReLU-BN.
(1) LeakyReLU-GPN-BI
Figure 12: Gradient norm ratio during training on CIFAR-10. Horizontal axis denotes the mini- aE
batch updates. Vertical axis denotes the gradient norm ratio maxi || (l |//ming
min; V ll.. The
gradient vanishes (I) T || Re 0) for ReLU and LeakyReLU during training and hence the plots empty.
8E
22

---

Under review as a conference paper at ICLR 2021
100
50
40
20
0
0
2
6
x10
x10
(a) ELU.
(b) ELU-GPN.
1000
1000
500
500
2
6
4
x10
x10
(c) ELU-BN.
(d) ELU-GPN BBN.
100
1500
1000
2
x10
x10
(e) SELU.
(I) SELU-GPN.
1500
1500
1000
1000
500
500
2
6
4
(g) SELU-BN,
(h) SELU-GPN-BN.
300
0.5
200
100
x10
x10
(i) GELU-BN.
(j) GELU-GPN.
0.5
2
x10
x10
(k) GELU-BN.
(I) GELU-GPN-BN,
Figure 13: Gradient norm ratio during training on CIFAR-10. Horizontal axis denotes the mini- batch updates. Vertical axis denotes the gradient norm ratio max, || ( / gradient vanishes (I De || 0) for GELU, GELU-BN and GELU-GPN-BI BN during training and hence the plots empty. For GELU-GPN-BN, both gradient exploding and gradient vanishing observed.
min; | 2 ||r. The
23


Reviews:
[{"review_id": "pTSLSBagIoe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nIn this paper, the authors introduce the bidirectional self-normalizing neural networks (BSNN) that preserve the norms in both forward and backward passes. To serve such purpose, a new class of activation functions, GPN, is proposed, which can be obtained via the affine transform of existing activation functions like tanh and SELU. The authors prove that under orthogonal weights and GPN, the norm for both forward and backward passes can be well preserved. Besides, the conclusions are also supported by experiments on synthetic data and real-world data like MNIST and CIFAR 10.\n\n\nStrength:\n\n--The paper is well organized and easy to follow.\n\n--The proofs seem rigorous.\n\n--The experiments on synthetic data and real-world data are consistent with the theoretical prediction.\n\n\nWeakness:\n\n--One major strength of the self-normalizing neural network powered by SELU as well as batch normalization is that, even if the statistical properties deviate from the ideal point at some layers, the succeeding layers can gradually fix the deviation. This property greatly stabilizes the training process even under a high learning rate. On the other hand, methods like dedicated initialization and weight normalization/standardization do not have this property, and previous studies have shown that they are less effective compared with SELU and BN. While GPN can be achieved via adding a simple affine transform to the existing activation functions like tanh, does it mean that it is more like an initialization technique? Or it still has the property of fixing deviations in preceding layers?\n\n--For the experiments on real-world data, the depth 200 may be too deep for these small datasets, and the accuracy reported is less satisfying compared with networks with 30-50 layers. Although I understand that the authors want to demonstrate the performance of self-normalization in very deep models, it would be more convincing if the proposed activation functions are also compared with existing ones like SELU under a more practical depth (e.g., 30-50 layers).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "b7GaFdZXG81", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of vanishing/exploding gradient in neural networks by imposing constraints on the weights and activation functions that ensure consistent Frobenius norm across updates to the weight matrices in all layers of the network.  \n\nThe paper is very well written, straight to follow, theory is well laid out and it is interesting.  The only (minor) issue I have on the presentation side is the name “bidirectional self-normalizing neural networks”.  This phrasing initially made me think the the network is bidirectional…but the network itself is not, it’s the self-normalizing that is bidirectional.   Wouldn’t it be more precise to say “bidirectionally self-normalized neural networks”?\n\nHowever, despite a well outlined theory, I think that in the end the experimental evaluation demonstrates a major weakness of the proposed approach.  Credit to the authors for including Table 2 with evaluation on real datasets, but the generalisation performance there seems to show quite clearly that -GPN variants of the networks are equivalent to linear classifiers.  A quick check of the performance of linear classifier on MNIST and CIFAR10 without augmentation reveals 93% and 41% testing accuracy - pretty close to the performance shown in Table 2.  Sure, -GPNising allows for training a 200-layer model, but it seem to come at the cost of reducing the model to barely more than a linear classifier.  The sacrifice of all non-trivial representational power in order to gain stable learning dynamics is just not worth it.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CiZpzKfXQZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting approach to normalizing the signals in a deep network both in the forward pass and in the backward pass with the goal of preventing vanishing and exploding gradients for better training convergence (and perhaps general elegance). \n\nPreviously, this has been partly solved by the use of orthogonal weight matrices (and this completely solves the problem for linear networks). The authors show that additionally a specific modification of the non-linearity (essentially a scale and shift) is enough to guarantee good bi-directional normalization with high probability.  \n\nI did not check the math. It would be good if the authors could provide some intuition for the main contribution: that an appropriate scaling and shifting of the nonlinearity is what is needed. \n\nEmpirical results are demonstrated in essentially toy settings (synthetic, and mnist/cifar with fully connected networks) but these experiments are thorough (but see question below). \n\nMy positive review of the paper is mostly based on viewing this as a natural completion of an existing line of work (around the use of orthogonal weight matrices for normalization), but I worry that this line of work may not lead to practically useful results since the orthogonality constraint is hard to enforce exactly, and it is not clear if this would work for convolution given the requirement for sufficient width. This is why I am hesitant to give it a higher score. \n\nQuestions, mostly around the practicality of the proposed techniques.\n\n1. How would this work on convolutional networks where one does not typically expect large “width” i.e. is this work only of theoretical interest?\n\n2. How does this impact generalization? There is reason to believe that mucking with gradients can hurt generalizations. This has been discussed for  adaptive methods (Duchi et al., etc. alluded to the in the introduction) in https://arxiv.org/abs/1705.08292.\n\n3. How would you compare with residual networks? Particularly in terms of run-time and generalization.\n\nMinor:\n\na. What is the significance of Theorem 1?\n\nb. Theorem 2 statement should say $\\phi$ is Gaussian-Poincare normalized?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6YbssJTKZiT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors introduce a variant\nof self-normalizing networks (SNNs) that also maintain\nthe norm of the errors during backpropagation. The presented\nBidirectional self-normalizing networks (BSNN)\nhave interesting theoretical properties, but are hardly \nassessed empirically. There are severe flaws in \nthe method evaluation. \n\n\nPros: \na) The mathematical formulations are very clear \nand sound.\nb) The connection between SNNs and works in\nmean-field theory (Poole, Schoenholz) are very\nwell put in place. \nc) Making both the forward- and the backward pass \nself-normalizing appears desirable for NNs.\nd) The work is well embedded into related works. \n\n\nCons: \na) Unclear relevance due to lacking and small-scale experiments, furthermore,\ndue to missing error bars and significance tests. \n\ni) Firstly, SNNs were introduced with relatively large-scale experiments on fully-connected networks. \nIn order to demonstrate improvements due to bidirectional normalization, \nthe suggested activation functions should be compared in that set of experiments. \nThe presented experiments are done with architectures that are extremely far from the current SOTA on CIFAR10, \nwith 46% versus 90% accuracy already reported in 2013 (see ref[2]). The authors\nshould perform experiments on large scale, e.g. CIFAR100 and ImageNet or voice recognition [3], \nwith architectures at or close to the state-of-the-art to demonstrate \nthe broad applicability and relevance of their method. \n\nii) It is unclear whether BSNNs represent an advance at all because \nall presented performance metrics come without repetitions, error bars\nand significance tests. Purported improvements could be just by chance. \nThe authors should use repetitions, present all metrics with confidence\nintervals and perform statistical tests. \n\niii) Conclusions of the experiments are not justified due to limited\nhyperparameter search. The setting of the experiments that activation\nfunctions are compared with the same architecture, same learning rate, \nmomentum term, and batch size strongly limits the conclusions since\nactivation functions could work with slightly different settings. \nThe authors should allow each method (here: activation function) \nto optimize hyperparameters. \n\niv) It is not necessary that almost exact preservation \nof the norm in the backward pass must improve learning, especially,\nbecause it is traded against a bias shift. \nFor standard SNNs, with the SELU activation, there is \na small exploding gradient, which, however leads\nto lower layers learning faster [1]. This could even assist learning.\nThis point should make clear that stringent empirical\nassessment is necessary. Therefore, the authors should provide\na much wider empirical assessment of BSNNs.\n\nb) Strong uncommented assumtions. The presented work requires\northogonal matrices, and several other assumptions. Those assumptions\nare left mostly uncommented and leave unclear how applicable this \ntheory is and how strong it restricts the results and conclusion.\nThe authors should comment on all assumptions and relate it to learned \nneural networks. \ni) The assumption of orthogonal weight matrices is a strong one. \nThis is computationally extremely costly, even in the relaxed way\nof the authors (Section 3.2). The authors should elaborate on this.  \n\nc) Neglection of learning dynamics. Whereas, the derivation of SNNs\ninvolves learning dynamics by showing that even with changes in the\nweights, a fixed points remains, BSNNs neglect learning dynamics. \nEven definition of self-normalization requires exact preservations\nof the norm. The authors should aim at requiring Eq (4) and (5) to \nbe kept on expectation and not exactly. Otherwise, no learned \nnetwork is self-normalizing. \n\n\nMinor:\n\n\nQuestions:\na) The expectation over E[\\phi(x)] is not more equal zero, which introduces\na bias shift. How does this bias shift interact with learning? \nCan you state this difference to SNNs clearer in the manuscript?\nb) Lemma 3: Can you comment on how this bound arises from (Ball, 1997)?\n\n\nJustification of score:\nDespite my many points of concerns, the theory in this paper and the main \nidea are good. I would be willing \nto vote for acceptance if the experimental part is strongly improved. \n\n\nReferences:\n[1] Hoedt, P.J., Hochreiter, S. and Klambauer, G. Characterising activation functions by their backward dynamics around forward fixed points. Critiquing and Correcting Trends in Machine Learning workshop at NeurIPS 2018. \n[2] Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y. (2013, May). Maxout networks. In International conference on machine learning (pp. 1319-1327). PMLR.\n[3] Huang, Z., Ng, T., Liu, L., Mason, H., Zhuang, X., & Liu, D. (2020, May). SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6854-6858). IEEE.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three knowledgeable referees rate this paper ok but not good enough or borderline positive (4,4,6), and one fairly confident referee rates it borderline positive 6. The referees discussed the authors' responses and, while they considered the idea and some of the theory good, they remain concerned, in particular about the experimental part and generalization discussion. The scores remained unchanged after the discussion. Hence I must reject the article. 
Decision: Reject