Bidirectionally Self-Normalizing Neural Networks

1 INTRODUCTION . Neural networks have brought unprecedented performance in various artificial intelligence tasks ( Graves et al. , 2013 ; Krizhevsky et al. , 2012 ; Silver et al. , 2017 ) . However , despite decades of research , training neural networks is still mostly guided by empirical observations and successful training often requires various heuristics and extensive hyperparameter tuning . It is therefore desirable to understand the cause of the difficulty in neural network training and to propose theoretically sound solutions . A major difficulty is the gradient exploding/vanishing problem ( Glorot & Bengio , 2010 ; Hochreiter , 1991 ; Pascanu et al. , 2013 ; Philipp et al. , 2018 ) . That is , the norm of the gradient in each layer is either growing or shrinking at an exponential rate as the gradient signal is propagated from the top layer to bottom layer . For deep neural networks , this problem might cause numerical overflow and make the optimization problem intrinsically difficult , as the gradient in each layer has vastly different magnitude and therefore the optimization landscape becomes pathological . One might attempt to solve the problem by simply normalizing the gradient in each layer . Indeed , the adaptive gradient optimization methods ( Duchi et al. , 2011 ; Kingma & Ba , 2015 ; Tieleman & Hinton , 2012 ) implement this idea and have been widely used in practice . However , one might also wonder if there is a solution more intrinsic to deep neural networks , whose internal structure if well-exploited would lead to further advances . To enable the trainability of deep neural networks , batch normalization ( Ioffe & Szegedy , 2015 ) was proposed in recent years and achieved widespread empirical success . Batch normalization is a differentiable operation which normalizes its inputs based on mini-batch statistics and inserted between the linear and nonlinear layers . It is reported that batch normalization can accelerate neural network training significantly ( Ioffe & Szegedy , 2015 ) . However , batch normalization does not solve the gradient exploding/vanishing problem ( Philipp et al. , 2018 ) . Indeed it is proved that batch normalization can actually worsen the problem ( Yang et al. , 2019 ) . Besides , batch normalization requires separate training and testing phases and might be ineffective when the mini-batch size is small ( Ioffe , 2017 ) . The shortcomings of batch normalization motivate us to search for a more principled and generic approach to solve the gradient exploding/vanishing problem . Alternatively , self-normalizing neural networks ( Klambauer et al. , 2017 ) and dynamical isometry theory ( Pennington et al. , 2017 ) were proposed to combat gradient exploding/vanishing problem . In self-normalizing neural networks , the output of each network unit is constrained to have zero mean and unit variance . Based on this motivation , a new activation function , scaled exponential linear unit ( SELU ) , was devised . In dynamical isometry theory , all singular values of the input-output Jacobian matrix are constrained to be close to one at initialization . This amounts to initializing the functionality of a network to be close to an orthogonal matrix . While the two theories dispense batch normalization , it is shown that SELU still suffers from exploding/vanishing gradient problem and dynamical isometry restricts the functionality of the network to be close to linear ( pseudolinearity ) ( Philipp et al. , 2018 ) . In this paper , we follow the above line of research to investigate neural network trainability . Our contributions are three-fold : First , we introduce bidirectionally self-normalizing neural network ( BSNN ) that consist of orthogonal weight matrices and a new class of activation functions which we call Gaussian-Poincaré normalized ( GPN ) functions . We show many common activation functions can be easily transformed into their respective GPN versions . Second , we rigorously prove that the gradient exploding/vanishing problem disappears with high probability in BSNNs if the width of each layer is sufficiently large . Third , with experiments on synthetic and real-world data , we confirm that BSNNs solve the gradient vanishing/exploding problem to large extent while maintaining nonlinear functionality . 2 THEORY . In this section , we introduce bidirectionally self-normalizing neural network ( BSNN ) formally and analyze its properties . All the proofs of our results are left to Appendix . To simplify the analysis , we define a neural network in a restricted sense as follows : Definition 1 ( Neural Network ) . A neural network is a function from Rn to Rn composed of layerwise operations for l = 1 , . . . , L as h ( l ) = W ( l ) x ( l ) , x ( l+1 ) = φ ( h ( l ) ) , ( 1 ) where W ( l ) ∈ Rn×n , φ : R → R is a differentiable function applied element-wise to a vector , x ( 1 ) is the input and x ( L+1 ) is the output . Under this definition , φ is called the activation function , { W ( l ) } Ll=1 are called the parameters , n is called the width and L is called the depth . Superscript ( l ) denotes the l-th layer of a neural network . The above formulation is similar to ( Pennington et al. , 2017 ) but we omit the bias term in ( 1 ) for simplicity as it plays no role in our analysis . Let E be the objective function of { W ( l ) } Ll=1 and D ( l ) be a diagonal matrix with diagonal elements D ( l ) ii = φ ′ ( h ( l ) i ) , where φ ′ denotes the derivative of φ . Now , the error signal is back propagated via d ( L ) = D ( L ) ∂E ∂x ( L+1 ) , d ( l ) = D ( l ) ( W ( l+1 ) ) Td ( l+1 ) , ( 2 ) and the gradient of the weight matrix for layer l can be computed as ∂E ∂W ( l ) = d ( l ) ( x ( l ) ) T . ( 3 ) To solve the gradient exploding/vanishing problem , we constrain the forward signal x ( l ) and the backward signal d ( l ) in order to constrain the norm of the gradient . This leads to the following definition and proposition . Definition 2 ( Bidirectional Self-Normalization ) . A neural network is bidirectionally selfnormalizing if ‖x ( 1 ) ‖2 = ‖x ( 2 ) ‖2 = ... = ‖x ( L ) ‖2 , ( 4 ) ‖d ( 1 ) ‖2 = ‖d ( 2 ) ‖2 = ... = ‖d ( L ) ‖2 . ( 5 ) Proposition 1 . If a neural network is bidirectionally self-normalizing , then∥∥∥ ∂E ∂W ( 1 ) ∥∥∥ F = ∥∥∥ ∂E ∂W ( 2 ) ∥∥∥ F = ... = ∥∥∥ ∂E ∂W ( L ) ∥∥∥ F . ( 6 ) In the rest of this section , we derive the conditions under which bidirectional self-normalization is achievable for a neural network . 2.1 CONSTRAINTS ON WEIGHT MATRICES . We constrain the weight matrices to be orthogonal since multiplication by an orthogonal matrix preserves the norm of a vector . For linear networks , this guarantees bidirectionally selfnoramlizingnormalization and its further benefits are discussed in ( Saxe et al. , 2014 ) . Even for nonlinear neural networks , orthogonal constraints are shown to improve the trainability with proper scaling ( Mishkin & Matas , 2016 ; Pennington et al. , 2017 ) . 2.2 CONSTRAINTS ON ACTIVATION FUNCTIONS . To achieve bidirectionally self-noramlizingnormalization for a nonlinear network , it is not enough only to constrain the weight matrices . We also need to constrain the activation function in such a way that both forward and backward signals are normalized . To this end , we propose the following constraint that captures the relationship between a function and its derivative . Definition 3 ( Gaussian-Poincaré Normalization ) . Function φ : R → R is Gaussian-Poincaré normalized if it is differentiable and Ex∼N ( 0,1 ) [ φ ( x ) 2 ] = Ex∼N ( 0,1 ) [ φ′ ( x ) 2 ] = 1 . ( 7 ) The definition is inspired by the following theorem which shows the fundamental relationship between a function and its derivative under Gaussian measure . Theorem 1 ( Gaussian-Poincaré Inequality ( Bogachev , 1998 ) ) . If function φ : R→ R is differentiable with bounded Ex∼N ( 0,1 ) [ φ ( x ) 2 ] and Ex∼N ( 0,1 ) [ φ′ ( x ) 2 ] , then Varx∼N ( 0,1 ) [ φ ( x ) ] ≤ Ex∼N ( 0,1 ) [ φ′ ( x ) 2 ] . ( 8 ) Note that there is an implicit assumption that the input is approximately Gaussian for a GaussianPoincaré normalized ( GPN ) function . Even though this is standard in the literature ( Klambauer et al. , 2017 ; Pennington et al. , 2017 ; Schoenholz et al. , 2017 ) , we will rigorously prove that this assumption is valid when orthogonal weight matrices are used in equation 1 . Next , we state a property of GPN functions . Proposition 2 . Function φ : R→ R is Gaussian-Poincaré normalized and Ex∼N ( 0,1 ) [ φ ( x ) ] = 0 if and only if φ ( x ) = x or φ ( x ) = −x . This result indicates that any nonlinear function with zero mean under Gaussian distribution ( e.g. , Tanh and SELU ) is not GPN . Now we show that a large class of activation functions can be converted into their respective GPN versions using an affine transformation . Proposition 3 . For any differentiable function φ : R → R with non-zero and bounded Ex∼N ( 0,1 ) [ φ ( x ) 2 ] and Ex∼N ( 0,1 ) [ φ′ ( x ) 2 ] , there exist two constants a and b such that aφ ( x ) + b is Gaussian-Poincaré normalized . To obtain a and b , one can use numerical procedure to compute the values of Ex∼N ( 0,1 ) [ φ′ ( x ) 2 ] , Ex∼N ( 0,1 ) [ φ ( x ) 2 ] and Ex∼N ( 0,1 ) [ φ ( x ) ] and then solve the quadratic equations Ex∼N ( 0,1 ) [ a2φ′ ( x ) 2 ] = 1 , ( 9 ) Ex∼N ( 0,1 ) [ ( aφ ( x ) + b ) 2 ] = 1 . ( 10 ) We computed a and b ( not unique ) for several common activation functions with their default hyperparameters1 and the results are listed in Table 1 . Note that ReLU , LeakyReLU and SELU are not differentiable at x = 0 but they can be regarded as approximations of their smooth counterparts . We ignore such point and evaluate the integrals for x ∈ ( −∞ , 0 ) ∪ ( 0 , ∞ ) . With the orthogonal constraint on the weight matrices and the Gaussian-Poincaré normalization on the activation function , we prove that bidirectionally self-noramlizingnormalization is achievable with high probability under mild conditions in the next subsection . 1We use α = 0.01 for LeakyReLU , α = 1 for ELU and φ ( x ) = x/ ( 1 + exp ( −1.702x ) ) for GELU .

Summary: In this paper, the authors introduce the bidirectional self-normalizing neural networks (BSNN) that preserve the norms in both forward and backward passes. To serve such purpose, a new class of activation functions, GPN, is proposed, which can be obtained via the affine transform of existing activation functions like tanh and SELU. The authors prove that under orthogonal weights and GPN, the norm for both forward and backward passes can be well preserved. Besides, the conclusions are also supported by experiments on synthetic data and real-world data like MNIST and CIFAR 10.