Redefining The Self-Normalization Property

1 INTRODUCTION . In recent years , deep neural networks ( DNNs ) have achieved state-of-the-art performance on different tasks like image classification ( He et al. , 2015 ; Zheng et al. , 2019 ) . This rapid development can be largely attributed to the initialization and normalization techniques that prevent the gradient explosion and vanishing . The initialization techniques ( He et al. , 2015 ; Xiao et al. , 2018 ) initialize the parameters in networks to have good statistical property at beginning , and assume that this property can be more or less maintained throughout the training process . However , this assumption is likely to be violated when the network gets deeper or is trained with higher learning rate . Hence , normalization techniques are proposed to explicitly normalize the network parameters ( Salimans & Kingma , 2016 ; Arpit et al. , 2016 ) or the activations ( Ioffe & Szegedy , 2015b ; Ulyanov et al. , 2016 ) during training . In particular , Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015a ) has become a standard component in DNNs , as it not only effectively improves convergence rate and training stability , but also regularizes the model to improve generalization ability . However , BN still has several drawbacks . First , when calculating the mean and variance , the accumulation must be done under FP32 to avoid underflow ( Micikevicius et al. , 2018 ) . This brings challenges when training neural networks in low bit width . Second , the performance degradation under micro batch size also makes it more difficult to design training accelerators , as the large batch size increases the memory size to store the intermediate results for backward pass ( Deng et al. , 2020 ) . Besides , Chen et al . ( 2020a ) ; Wu et al . ( 2018 ) show that BN introduces considerable overhead . The self-normalizing neural network ( SNN ) provides a promising way to address this challenge . SNN initializes the neural network to have a good statistical property at the beginning just like initialization techniques . However , the statistics deviation in forward and backward passes can be gradually fixed during propagation , thus it is more robust to the deviation from initial properties ( Chen et al. , 2020b ) . For instance , the mean and variance of output activations with SELU in Klambauer et al . ( 2017 ) automatically converge the fixed point ( 0 , 1 ) . Chen et al . ( 2020b ) analyze the Frobenius norm of backward gradient in SNN activated with SELU . They reveal a trade-off between the self-normalization property and the speed of gradient explosion in the backward pass , and the hyper-parameters need to be configured according to the depth of the network . The resulting activation function , depth-aware SELU ( dSELU ) , has achieved even higher accuracy than the original configuration on moderate-scale datasets like CIFAR-10 , and makes the SNN trainable on ImageNet . However , in deeper neural networks , the dSELU gradually degenerates to ReLU and loses its selfnormalization property . Moreover , even with dSELU , the test accuracy on ImageNet with Conv MobileNet V1 ( Howard et al. , 2017 ) is still 1.79 % lower than BN ( Chen et al. , 2020b ) . Therefore , we aim to answer the following three questions in this paper : 1 ) . Is SELU the only activation function that has self-normalization property ? 2 ) . If it is not , is there a better choice ? And how do we compare the strength of self-normalization property between different activation functions ? 3 ) . Why the performance of SNN on ImageNet is less satisfying ? Is there any way to alleviate that ? In this paper , we analyze the signal propagation in both forward and backward passes in serial deep neural networks with mean-field theory ( Poole et al. , 2016 ) and block dynamical isometry ( Chen et al. , 2020b ) . Our main theoretical results are summarized as follows : • We illustrate that an activation function would demonstrate self-normalization property if the second moment of its Jacobian matrix ’ s singular values φ ( q ) is inversely proportional to the second moment of its input pre-activations q , and the property gets stronger when φ ( q ) gets closer to 1/q . A new definition of the self-normalization property is proposed that can be easily used both analytically and numerically . • We propose leaky SELU ( lSELU ) and scaled SELU ( sSELU ) . Both of them have an additional parameter , β , that can be configured to achieve stronger self-normalization property . The hyper-parameters can be solved with a constrained optimization program , thus no additional hyper-parameter relative to dSELU is introduced . • We reveal that models with larger fan-in have weaker normalization effectiveness on the mean of the forward pass signal . This can be solved with explicit weight centralization , mixup data augmentation ( Zhang et al. , 2018 ) , and centralized activation function . On CIFAR-10 , CIFAR-100 , and Tiny ImageNet , lSELU and sSELU achieves up to 2.13 % higher test accuracy than previous studies . On ImageNet - Conv MobileNet V1 , sSELU with Mixup , trainable λ , and centralized activation function achieves comparable test accuracy ( 71.95 % ) with BN . Besides , we provide a CUDA kernel design for lSELU and sSELU that has only 2 % overhead than SELU . 2 RELATED WORK . In this section , we present an overview of existing studies on the self-normalizing neural networks ( SNN ) as well as statistical studies on forward and backward signals in deep neural networks . Self-normalizing Neural Network . Scaled Exponential Linear Unit ( SELU ) ( Klambauer et al. , 2017 ) scales the Exponent Linear Unit ( ELU ) by a constant scalar λ . The λ and original parameter α in ELU are configured such that the mean and variance of output activation have a fixed point ( 0 , 1 ) . The authors further prove that this fixed point is still stable and attractive even when the input activations and the weights are unnormalized . Chen et al . ( 2020b ) investigate the fixed point in backward gradient . They reveal that the gradient of SNN is exploding with the rate ( 1 + ) per layer , where is a small positive value . The self-normalizing property gets stronger when is larger , whereas the gradient will explode at a higher rate . Therefore , they propose the depth-aware SELU in which the ≈ 1/L is used to derive the optimal α and λ in SELU for a network with depth L. Statistical Analysis of Deep Neural Networks . Schoenholz et al . ( 2016 ) ; Poole et al . ( 2016 ) ; Burkholz & Dubatovka ( 2018 ) investigate the forward activations under the limit of large layer width with mean-field theory . They have identified an Order-to-Chaos phase transition characterized by the second moment of singular values of the network ’ s input-output Jacobian matrix . The neural network has good performance when it is on the border of the order and chaos phases . On the other hand , Chen et al . ( 2020b ) develop a very handy framework for analyzing the Frobenius norm of gradient . They illustrate that the gradient norm equality is a universal philosophy behind various different initialization , normalization techniques , and even some neural network structures . The gradient norm equality means the Frobenius Norm of the gradient is more or less equal in different layers so that the information flow in the backward pass can be well preserved . ( Arpit & Bengio , 2020 ) 3 SELF-NORMALIZATION PROPERTY . In this section , we formally define the self-normalization property under the problem formulation , notations , and assumptions as follows . Problem Formulation . Let ’ s consider a DNN with L layers . Each layer performs a linear transform followed by a non-linear element-wise activation function f , i.e . xl = f ( hl ) , hl = Wlxl−1 + bl , l = 1 , ... , L , ( 1 ) where xl ∈ RNl is the output feature vector of layer l , hl is the pre-activation vector , Wl is the weight of fully-connected layer or the expanded doubly block circulant matrix ( Sedghi et al. , 2019 ) of 2D convolution , bl is the vector of biases , and we denote the loss as L. Besides , without loss of generality , for f and x ∼ N ( 0 , q ) , we have ( 1 + δq ) E [ f2 ( x ) ] = E [ ( df ( x ) /dx ) 2 ] E [ x2 ] , ( 2 ) where δq is a function of q . Following previous studies ( Poole et al. , 2016 ; Chen et al. , 2020b ) , for ∀ l , we make the assumptions as follows : Assumption 1 The mean of entries inWl and bl are zero . Assumption 2 With central limit theory , the entries in hl follow i.i.d . N ( 0 , ql ) , ql = 1Nlh T l hl . Assumption 3 The eigenvalues ofW Tl Wl are independent with entries in hl−1 . Klambauer et al . ( 2017 ) first define the self-normalization property of a neural network as follows . Definition 1 ( Self-normalizing Neural Network ) A neural network is self-normalizing if it possesses a mapping g : Ω→ Ω for each activation y that maps mean and variance from one layer to the next and has a stable and attracting fixed point depending on ( ω , τ ) in Ω . Furthermore , the mean and the variance remain in the domain Ω , that is g ( Ω ) ⊆ Ω , where Ω = { ( µ , ν ) |µ ∈ [ µmin , µmax ] , ν ∈ [ νmin , νmax ] } . When iteratively applying the g , each point within Ω converges to this fixed point . This definition imitates the explicit normalization techniques like BN , which ensures that the feedforward signal is normalized . Based on Definition 1 , Klambauer et al . ( 2017 ) propose the SELU : f ( x ) = λ { x if x > 0 αex − α if x ≤ 0 . ( 3 ) Besides , Klambauer et al . ( 2017 ) initialize the entries in Wl with N ( 0 , 1/Nl−1 ) , so that the output pre-activation will have the same second moment of input activation . With the stable fixed points of mean and variance around ( 0 , 1 ) , the optimal choice for λ and α can be derived from∫ ∞ −∞ f ( z ) e− z2 2 √ 2π dz = 0 , ∫ ∞ −∞ f2 ( z ) e− z2 2 √ 2π dz = 1 . ( 4 ) Furthermore , the authors prove that the fixed points for mean and variance are still attractive even when the statistical properties of the parameters in the neural network deviate from the initial setup . However , the statistical fixed point in the forward pass doesn ’ t necessarily lead to good dynamics of gradient . Chen et al . ( 2020b ) analyze the Frobenius norm of the gradient in neural networks activated by SELU . With the same activation function shown in equation 3 , their analysis shows that the optimal λ and α can be configured by preserving the Frobenius norm of backward gradient and second moment of forward activations with equations as follows : ∫ ∞ −∞ ( df ( z ) dz ) 2 e− z2 2 √ 2π dz=1 + , ∫ ∞ −∞ f2 ( z ) e− z2 2 √ 2π dz = 1 . ( 5 ) where is a small positive constant , without which the only solution for equation 5 would be λ = √ 2 and α = 0 , and the activation function degenerates back to ReLU with the initialization technique proposed in He et al . ( 2015 ) . Thus it will lose the self-normalization property . Conversely , a relatively large will bring stronger self-normalization property , but meanwhile make the Frobenius norm of gradient explode with rate ( 1+ ) per layer . Notably , the original configuration of SELU can be obtained by setting = 0.0716 . Therefore , Chen et al . ( 2020b ) assert that having ≈ 1L could bring a good trade-off between gradient norm stability and self-normalization property . Experiments on CIFAR-10 and ImageNet show that the new configuration results in higher accuracy . Inspired by Chen et al . ( 2020b ) , we formally redefine the self-normalization property as follows : Definition 2 ( Self-normalization Property ) Given an activation function f , we define operand φ as φ ( q ) = ∫ ∞ −∞ ( df ( √ qz ) d √ qz ) 2 e− z2 2 √ 2π dz . ( 6 )

Summary: The paper proposes two modifications to SELU activation function to improve it with regards to preserving forward-backward signal propagation in neural networks. The work builds on top of the mean-field theory literature and provides a modified self-normalization property (additional constraints compared to SELU). Further, it discusses some heuristics (mixup, weight centralization) to improve performance in practice.