On the Selection of Initialization and Activation Function for Deep Neural Networks

1 INTRODUCTION . Deep neural networks have become extremely popular as they achieve state-of-the-art performance on a variety of important applications including language processing and computer vision ; see , e.g. , LeCun et al . ( 1998 ) . The success of these models has motivated the use of increasingly deep networks and stimulated a large body of work to understand their theoretical properties . It is impossible to provide here a comprehensive summary of the large number of contributions within this field . To cite a few results relevant to our contributions , Montufar et al . ( 2014 ) have shown that neural networks have exponential expressive power with respect to the depth while Poole et al . ( 2016 ) obtained similar results using a topological measure of expressiveness . We follow here the approach of Poole et al . ( 2016 ) and Schoenholz et al . ( 2017 ) by investigating the behaviour of random networks in the infinite-width and finite-variance i.i.d . weights context where they can be approximated by a Gaussian process as established by Matthews et al . ( 2018 ) and Lee et al . ( 2018 ) . In this paper , our contribution is two-fold . Firstly , we provide an analysis complementing the results of Poole et al . ( 2016 ) and Schoenholz et al . ( 2017 ) and show that initializing a network with a specific choice of hyperparameters known as the ‘ edge of chaos ’ is linked to a deeper propagation of the information through the network . In particular , we establish that for a class of ReLU-like activation functions , the exponential depth scale introduced in Schoenholz et al . ( 2017 ) is replaced by a polynomial depth scale . This implies that the information can propagate deeper when the network is initialized on the edge of chaos . Secondly , we outline the limitations of ReLU-like activation functions by showing that , even on the edge of chaos , the limiting Gaussian Process admits a degenerate kernel as the number of layers goes to infinity . Our main result ( 4 ) gives sufficient conditions for activation functions to allow a good ‘ information flow ’ through the network ( Proposition 4 ) ( in addition to being non-polynomial and not suffering from the exploding/vanishing gradient problem ) . These conditions are satisfied by the Swish activation φswish ( x ) = x ·sigmoid ( x ) used in Hendrycks & Gimpel ( 2016 ) , Elfwing et al . ( 2017 ) and Ramachandran et al . ( 2017 ) . In recent work , Ramachandran et al . ( 2017 ) used automated search techniques to identify new activation functions and found experimentally that functions of the form φ ( x ) = x · sigmoid ( βx ) appear to perform indeed better than many alternative functions , including ReLU . Our paper provides a theoretical grounding for these results . We also complement previous empirical results by illustrating the benefits of an initialization on the edge of chaos in this context . All proofs are given in the Supplementary Material . 2 ON GAUSSIAN PROCESS APPROXIMATIONS OF NEURAL NETWORKS AND THEIR STABILITY . 2.1 SETUP AND NOTATIONS . We use similar notations to those of Poole et al . ( 2016 ) and Lee et al . ( 2018 ) . Consider a fully connected random neural network of depth L , widths ( Nl ) 1≤l≤L , weights W lij iid∼ N ( 0 , σ 2 w Nl−1 ) and bias Bli iid∼ N ( 0 , σ2b ) , where N ( µ , σ2 ) denotes the normal distribution of mean µ and variance σ2 . For some input a ∈ Rd , the propagation of this input through the network is given for an activation function φ : R→ R by y1i ( a ) = d∑ j=1 W 1ijaj +B 1 i , y l i ( a ) = Nl−1∑ j=1 W lijφ ( y l−1 j ( a ) ) +B l i , for l ≥ 2 . ( 1 ) Throughout the paper we assume that for all l the processes yli ( . ) are independent ( across i ) centred Gaussian processes with covariance kernels κl and write accordingly yli ind∼ GP ( 0 , κl ) . This is an idealized version of the true processes corresponding to choosing Nl−1 = +∞ ( which implies , using Central Limit Theorem , that yli ( a ) is a Gaussian variable for any input a ) . The approximation of yli ( . ) by a Gaussian process was first proposed by Neal ( 1995 ) in the single layer case and has been recently extended to the multiple layer case by Lee et al . ( 2018 ) and Matthews et al . ( 2018 ) . We recall here the expressions of the limiting Gaussian process kernels . For any input a ∈ Rd , E [ yli ( a ) ] = 0 so that for any inputs a , b ∈ Rd κl ( a , b ) = E [ yli ( a ) yli ( b ) ] = σ2b + σ2wEyl−1i ∼GP ( 0 , κl−1 ) [ φ ( y l−1 i ( a ) ) φ ( y l−1 i ( b ) ) ] = σ2b + σ 2 wFφ ( κ l−1 ( a , a ) , κl−1 ( a , b ) , κl−1 ( b , b ) ) , where Fφ is a function that depends only on φ . This gives a recursion to calculate the kernel κl ; see , e.g. , Lee et al . ( 2018 ) for more details . We can also express the kernel κl in terms of the correlation clab in the l th layer used in the rest of this paper qlab : = κ l ( a , b ) = E [ yli ( a ) yli ( b ) ] = σ2b + σ2wE [ φ ( √ ql−1a Z1 ) φ ( √ ql−1b ( c l−1 ab Z1 + √ 1− ( cl−1ab ) 2Z2 ) ) ] where ql−1a : = q l−1 aa , resp . c l−1 ab : = q l−1 ab / √ ql−1a q l−1 b , is the variance , resp . correlation , in the ( l − 1 ) th layer and Z1 , Z2 are independent standard Gaussian random variables . when it propagates through the network . qla is updated through the layers by the recursive formula q l a = F ( q l−1 a ) , where F is the ‘ variance function ’ given by F ( x ) = σ2b + σ 2 wE [ φ ( √ xZ ) 2 ] , where Z ∼ N ( 0 , 1 ) . ( 2 ) Throughout the paper , Z , Z1 , Z2 will always denote independent standard Gaussian variables . 2.2 LIMITING BEHAVIOUR OF THE VARIANCE AND COVARIANCE OPERATORS . We analyze here the limiting behaviour of qLa and c L a , b as the network depth L goes to infinity under the assumption that φ has a second derivative at least in the distribution sense1 . From now onwards , we will also assume without loss of generality that c1ab ≥ 0 ( similar results can be obtained straightforwardly when c1ab ≤ 0 ) . We first need to define the Domains of Convergence associated with an activation function φ . 1ReLU admits a Dirac mass in 0 as second derivative and so is covered by our developments . Definition 1 . Let φ be an activation function , ( σb , σw ) ∈ ( R+ ) 2 . ( i ) ( σb , σw ) is in Dφ , var ( domain of convergence for the variance ) if there exists K > 0 , q ≥ 0 such that for any input a with q1a ≤ K , liml→∞ qla = q . We denote by Kφ , var ( σb , σw ) the maximal K satisfying this condition . ( ii ) ( σb , σw ) is in Dφ , corr ( domain of convergence for the correlation ) if there exists K > 0 such that for any two inputs a , b with q1a , q 1 b ≤ K , liml→∞ clab = 1 . We denote by Kφ , corr ( σb , σw ) the maximal K satisfying this condition . Remark : Typically , q in Definition 1 is a fixed point of the variance function defined in equation 2 . Therefore , it is easy to see that for any ( σb , σw ) such that F is increasing and admits at least one fixed point , we haveKφ , corr ( σb , σw ) ≥ q where q is the minimal fixed point ; i.e . q : = min { x : F ( x ) = x } . Thus , if we re-scale the input data to have q1a ≤ q , the variance qla converges to q . We can also re-scale the variance σw of the first layer ( only ) to assume that q1a ≤ q for all inputs a . The next result gives sufficient conditions on ( σb , σw ) to be in the domains of convergence of φ . Proposition 1 . Let Mφ : = supx≥0E [ |φ′2 ( xZ ) + φ′′ ( xZ ) φ ( xZ ) | ] . Assume Mφ < ∞ , then for σ2w < 1 Mφ and any σb , we have ( σb , σw ) ∈ Dφ , var and Kφ , var ( σb , σw ) =∞ Let Cφ , δ : = supx , y≥0 , |x−y|≤δ , c∈ [ 0,1 ] E [ |φ′ ( xZ1 ) φ′ ( y ( cZ1 + √ 1− c2Z2 ) | ] . Assume Cφ , δ < ∞ for some δ > 0 , then for σ2w < min ( 1 Mφ , 1Cφ ) and any σb , we have ( σb , σw ) ∈ Dφ , var ∩Dφ , corr and Kφ , var ( σb , σw ) = Kφ , corr ( σb , σw ) =∞ . The proof of Proposition 1 is straightforward . We prove that supF ′ ( x ) = σ2wMφ and then apply the Banach fixed point theorem ; similar ideas are used for Cφ , δ . Example : For ReLU activation function , we have MReLU = 2 and CReLU , δ ≤ 1 for any δ > 0 . In the domain of convergence Dφ , var ∩Dφ , corr , for all a , b ∈ Rd , y∞i ( a ) = y∞i ( b ) almost surely and the outputs of the network are constant functions . Figure 1 illustrates this behaviour for d = 2 for ReLU and Tanh using a network of depth L = 10 with Nl = 100 neurons per layer . The draws of outputs of these networks are indeed almost constant . To refine this convergence analysis , Schoenholz et al . ( 2017 ) established the existence of q and c such that |qla−q| ∼ e−l/ q and |clab−1| ∼ e−l/ c when fixed points exist . The quantities q and c are called ‘ depth scales ’ since they represent the depth to which the variance and correlation can propagate without being exponentially close to their limits . More precisely , if we write χ1 = σ2wE [ φ′ ( √ qZ ) 2 ] and α = χ1 + σ2wE [ φ′′ ( √ qZ ) φ ( √ qZ ) ] then the depth scales are given by r = − log ( α ) −1 and c = − log ( χ1 ) −1 . The equation χ1 = 1 corresponds to an infinite depth scale of the correlation . It is called the edge of chaos as it separates two phases : an ordered phase where the correlation converges to 1 if χ1 < 1 and a chaotic phase where χ1 > 1 and the correlations do not converge to 1 . In this chaotic regime , it has been observed in Schoenholz et al . ( 2017 ) that the correlations converge to some random value c < 1 when φ ( x ) = Tanh ( x ) and that c is independent of the correlation between the inputs . This means that very close inputs ( in terms of correlation ) lead to very different outputs . Therefore , in the chaotic phase , the output function of the neural network is non-continuous everywhere . Definition 2 . For ( σb , σw ) ∈ Dφ , var , let q be the limiting variance2 . The Edge of Chaos , hereafter EOC , is the set of values of ( σb , σw ) satisfying χ1 = σ2wE [ φ′ ( √ qZ ) 2 ] = 1 . To further study the EOC regime , the next lemma introduces a function f called the ‘ correlation function ’ simplifying the analysis of the correlations . It states that the correlations have the same asymptotic behaviour as the time-homogeneous dynamical system cl+1ab = f ( c l ab ) . Lemma 1 . Let ( σb , σw ) ∈ Dφ , var ∩Dφ , corr such that q > 0 , a , b ∈ Rd and φ an activation function such that supx∈S E [ φ ( xZ ) 2 ] < ∞ for all compact sets S. Define fl by cl+1ab = fl ( clab ) and f by f ( x ) = σ2b+σ 2 wE [ φ ( √ qZ1 ) φ ( √ q ( xZ1+ √ 1−x2Z2 ) ) q . Then liml→∞ supx∈ [ 0,1 ] |fl ( x ) − f ( x ) | = 0 . The condition on φ in Lemma 1 is violated only by activation functions with exponential growth ( which are not used in practice ) , so from now onwards , we use this approximation in our analysis . Note that being on the EOC is equivalent to ( σb , σw ) satisfying f ′ ( 1 ) = 1 . In the next section , we analyze this phase transition carefully for a large class of activation functions . 3 EDGE OF CHAOS To illustrate the effect of the initialization on the EOC , we plot in Figure 2 the output of a ReLU neural network with 20 layers and 100 neurons per layer with parameters ( σ2b , σ 2 w ) = ( 0 , 2 ) ( as we will see later EOC = { ( 0 , √ 2 ) } for ReLU ) . Unlike the output in Figure 1 , this output displays much more variability . However , we will prove here that the correlations still converges to 1 even in the EOC regime , albeit at a slower rate .

Summary: The authors prove some theoretical results under the mean field regime and support their conclusions with a small number of experiments. Their central argument is that a correlation curve that leads to sub-exponential correlation convergence (edge of chaos) can still lead to rapid convergence if the rate is e.g. quadratic. They show that this is the case for ReLU and argue that we must ensure not only sub-exponential convergence, but also have a correlation curve that is close to the identity everywhere. They suggest activation functions that attain conditions as laid out in propositions 4/5 as an alternative.