Effect of Activation Functions on the Training of Overparametrized Neural Nets

• For non-smooth activations such as ReLU , SELU , ELU , which are not smooth because there is a point where either the first order or second order derivative is discontinuous , all eigenvalues of the associated Gram matrix are large under minimal assumptions on the data . • For smooth activations such as tanh , swish , polynomial , which have derivatives of all orders at all points , the situation is more complex : if the subspace spanned by the data has small dimension then the minimum eigenvalue of the Gram matrix can be small leading to slow training . But if the dimension is large and the data satisfies another mild condition , then the eigenvalues are large . If we allow deep networks , then the small data dimension is not a limitation provided that the depth is sufficient . We discuss a number of extensions and applications of these results . 1 INTRODUCTION . It is now well-known that overparametrized feedforward neural networks trained using gradient-based algorithms with appropriate hyperparameter choices reliably achieve near-zero training error , e.g. , Neyshabur et al . ( 2015 ) . Importantly , overparametrization also often helps with generalization ; but our central concern here is the training error which is an important component in understanding generalization . We study the effect of the choice of activation function ( we often just say activation ) on the training of overparametrized neural networks . By overparametrized setting we roughly mean that the number of parameters or weights in the networks is much larger than the number of data samples . The well-known universal approximation theorem for feedforward neural networks states that any continuous function on a bounded domain can be approximated arbitrarily well by a finite neural network with one hidden layer . This theorem is generally stated for specific activation functions such as sigmoid or ReLU . A more general form of the theorem shows this for essentially all nonpolynomial activations ( Leshno et al. , 1993 ; Pinkus , 1999 ; Sonoda & Murata , 2017 ) . This theorem ∗Work done when the author was at Microsoft Research India . concerns only the expressive power and does not address how the training and generalization of neural networks are affected by the choice of activation , nor does it provide quantitative information about the size of the network needed for the task . Traditionally , sigmoid and tanh had been the popular activations but a number of other activations have also been considered including linear and polynomial activations ( Arora et al. , 2019a ; Du & Lee , 2018 ; Kileel et al. , 2019 ) . One of the many innovations in the resurgence of neural networks in the last decade or so has been the realization that ReLU activation generally performs better than the traditional choices in terms of training and generalization . ReLU is now the de facto standard for activation functions for neural networks but many other activations are also used which may be advantageous depending on the situation ( e.g . ( Goodfellow et al. , 2016 , Chapter 6 ) ) . In practice , most activation functions often achieve reasonable performance . To quote Goodfellow et al . ( 2016 ) : In general , a wide variety of differentiable functions perform perfectly well . Many unpublished activation functions perform just as well as the popular ones . Concretely , Ramachandran et al . ( 2018 ) provides a list of ten non-standard functions which all perform close to the state of the art at some tasks . This hints at the possibility of a universality phenomenon for training neural networks similar to the one for expressive power mentioned above . Search for activation functions . A number of recent papers have proposed new activations—such as ELU , SELU , penalized tanh , SiLU/swish—based on either theoretical considerations or automated search using reinforcement learning and other methods ; e.g . Clevert et al . ( 2016 ) ; Klambauer et al . ( 2017 ) ; Xu et al . ( 2016 ) ; Elfwing et al . ( 2017 ) ; Ramachandran et al . ( 2018 ) . For definitions , see Section 2 and Appendix B . These activation functions have been found to be superior to ReLU in many settings . See e.g . Eger et al . ( 2018 ) ; Nwankpa et al . ( 2018 ) for overview and evaluation . We quote once more from Goodfellow et al . ( 2016 ) : The design of hidden units is an extremely active area of research and does not yet have many definitive guiding theoretical principles . Theoretical analysis of training of highly overparametrized neural networks . Theoretical analysis of neural network training has seen vigorous activity of late and significant progress was made for the case of highly overparametrized networks . At a high level , the main insight in these works is that when the network is large , small changes in weights can already allow the network to fit the data . And yet , since the weights change very little , the training dynamics approximately behaves as in kernel methods and hence can be analyzed ( e.g . Jacot et al . ( 2018 ) ; Li & Liang ( 2018 ) ; Du et al . ( 2019a ) ; Allen-Zhu et al . ( 2019 ) ; Du et al . ( 2019b ) ; Allen-Zhu et al . ( 2018 ) ; Arora et al . ( 2019c ) ; Oymak & Soltanolkotabi ( 2019 ) ) . There are also many other approaches for theoretical analysis , e.g . Brutzkus et al . ( 2018 ) ; Mei et al . ( 2018 ) ; Chizat & Bach ( 2018 ) . Because of the large number of papers on this topic , we have chosen to list only the most closely related ones . Analyses in many of these papers involve a matrix G , either explicitly ( Jacot et al. , 2018 ; Du et al. , 2019a ; b ) or implicitly ( Allen-Zhu et al. , 2019 ) . ( This matrix also occurs in earlier works ( Xie et al. , 2017 ; Tsuchida et al. , 2018 ) . ) λmin ( G ) , the minimum eigenvalue of G , is an important parameter that directly controls the rate of convergence of gradient descent training : the higher the minimum eigenvalue the faster the convergence . Jacot et al . ( 2018 ) ; Du et al . ( 2019b ) show that λmin ( G ) > 0 for certain activations assuming that no two data points are parallel . Unfortunately , these results do not provide any quantitative information . The result of Allen-Zhu et al . ( 2019 ) , where the matrix G does not occur explicitly , can be interpreted as showing that the minimum eigenvalue is large under the reasonable assumption that the data is δ-separated , meaning roughly that no two data points are very close , and the activation used is ReLU . This quantitative lower bound on the minimum eigenvalue implies fast convergence of training . So far , ReLU was the only activation for which such a proof was known . Our results in brief . A general result one could hope for is that based on general characteristics of the activations , such as smoothness , convexity etc. , one can determine whether the smallest eigenvalue of G is small or large . We prove results of this type . A crucial distinction turns out to be whether the activation is ( a ) not smooth ( informally , has a “ kink ” ) or ( b ) is smooth ( i.e . derivatives of all orders exist over R ) . The two classes of functions above seem to cover all “ reasonable ” activations ; in particular , to our knowledge , all functions used in practice seem to fall in one of the two classes above . • Activations with a kink , i.e. , those with a a jump discontinuity in the derivative of some constant order , have all eigenvalues large under minimal assumptions on the data . E.g. , the first derivatives of ReLU and SELU , the second derivative of ELU have jump discontinuities at 0 . These results imply that for such activations , training will be rapid . We also provide a new proof for ReLU with the best known lower bound on the minimum eigenvalue . • For smooth activations such as polynomials , tanh and swish the situation is more complex : the minimum eigenvalue can be small depending on the dimension of the span of the dataset . We give a few examples : Let n be size of the dataset which is a collection of points in Rd , and let d′ be the dimension of the span of the dataset . For quadratic activation , if d′ = O ( √ n ) , then the minimum singular value is 0 . For tanh and swish , if d′ = O ( log0.75 n ) , then the minimum eigenvalue is inverse superpolynomially small ( exp ( −Ω ( n1/2d′ ) ) ) . In fact , a significant fraction of eigenvalues can be small . This implies that for such datasets , training using smooth activations will be slow if the dimension of the dataset is small . A trade off is possible : assuming stronger bounds on the dimension of the span gives stronger bounds on the eigenvalues . We also show that these results are tight in a precise sense . We further show that the above limitation of smooth activations disappears if one allows sufficiently deep networks . Unless otherwise stated , we work with one hidden layer neural nets where only the input layer is trained . This choice is made to simplify exposition ; extensions of various types including the ones that drop the above restriction are possible and discussed in Section 5 . 2 PRELIMINARIES . Denote the unit sphere in Rn by Sn−1 : = { u ∈ Rn : ‖u‖2 = 1 } where‖u‖2 : =‖u‖22 : = ∑n i=1 u 2 i . For u , v ∈ Rn , define the standard inner product by 〈u , v〉 : = ∑ i uivi . Given a set S , denote by U ( S ) the uniform distribution over S. N ( µ , σ2 ) denotes the univariate normal distribution with mean µ and variance σ2 . Let IS denote the indicator of the set S. For a matrix A containing elements aij , ai denotes its i-th column . We define some of the popular non-traditional activation functions here : swish ( x ) : = x1+e−x ( Ramachandran et al. , 2018 ) ( called SiLU in Elfwing et al . ( 2017 ) ) ; ELU ( x ) : = max ( x , 0 ) + min ( x , 0 ) ( ex − 1 ) ( Clevert et al. , 2016 ) ; SELU ( x ) : = α1 max ( x , 0 ) + α2 min ( x , 0 ) ( e x − 1 ) , where α1 and α2 are two different constants ( Klambauer et al. , 2017 ) . See Appendix B for more definitions . We consider 2-layer neural networks F ( x ; a , W ) : = cφ√ m m∑ k=1 akφ ( wTk x ) , ( 1 ) where x ∈ Rd is the input and W = [ w1 , . . . , wm ] ∈ Rm×d is the hidden layer weight matrix and a ∈ Rm is the output layer weight vector . Activation function φ : R → R acts entrywise on vectors and c2φ : = Ez∈N ( 0,1 ) φ ( z ) 2 . In the case of one hidden layer nets , we set cφ = 1 to simplify expressions ; this is without loss of generality . For deeper networks we do not do this as this assumption would result in loss of generality . Elements of W and a have been initialized i.i.d . from the standard Gaussian distribution . This initialization and the parametrization in Equation 1 are from Du et al . ( 2019b ) . Together , these will be referred to as the DZPS setting . Parametrization in practice does not have cφ√ m in Equation 1 ; standard initializations in practice include He ( He et al. , 2015 ) and Glorot initializations ( Glorot & Bengio , 2010 ) and variants . In the DZPS setting , the argument of the activation can be larger compared to the standard initializations , making the analysis harder . Our theorems apply to the DZPS setting as well as to standard initializations and for the latter easily follow as corollaries to the analysis in the DZPS setting . We defer this discussion to the appendix . Unless otherwise stated , we work in the DZPS setting with one hidden layer neural nets with initialization as above with only the input layer trained . Given labeled input data { ( xi , yi ) } ni=1 , where xi ∈ Rd and yi ∈ R , we want to find the best fit weights W so that the quadratic loss L ( { ( xi , yi ) } ni=1 ; a , W ) : = 12 ∑n i=1 ( yi − F ( xi ; a , W ) ) 2 is minimized . We train the neural network by fixing the output weight vector a at random initialization and training the hidden layer weight matrix W ( output layer being trained can be easily handled as in Du et al . ( 2019a ; b ) ; See Appendix M for details ) . In this paper , we focus on the gradient descent algorithm for this purpose . The gradient descent update rule for W is given by W ( t+1 ) : = W ( t ) − η∇WL ( W ( t ) ) , where W ( t ) denotes the weight matrix after t steps of gradient descent and η > 0 is the learning rate . The output vector u ( t ) ∈ Rn is defined by u ( t ) i : = F ( xi ; a , W ( t ) ) . Next , we define the matrix alluded to earlier , the Gradient Gram matrix G ∈ Rn×n associated with the neural network defined in ( 1 ) , referred to as the G-matrix in the sequel : gi , j : = 1 m ∑ k∈ [ m ] a2k φ ′ ( wTk xi ) φ ′ ( wTk xj ) 〈xi , xj〉 . ( 2 ) We will often work with the related matrix M ∈ Rmd×n , whose i-th column is obtained by vectorizing ∇WF ( xi , yi ) , i.e. , Md ( k−1 ) +1 : dk , i : = akφ′ ( wTk xi ) xi for k ∈ [ m ] . G is a Gram matrix : G = 1 mM TM . Denote by λi ( G ) the i-th eigenvalue of G with λ1 ≥ λ2 ≥ . . . , and similarly by σi ( M ) the i-th singular value of M. These quantities are related by λi ( G ) = 1mσ 2 i ( M ) . Following Allen-Zhu et al . ( 2019 ) , we make the following mild assumptions on data . Assumption 1 . ‖xi‖2 = 1 ∀i ∈ [ n ] . Assumption 2 . ‖ ( In − xixTi ) xj‖2 ≥ δ ∀i , j ∈ [ n ] , i 6= j i.e. , the distance between the subspaces spanned by xi and xj is at least δ > 0 . Assumption 1 can be easily satisfied by the following preprocessing : renormalize each xi so that ‖xi‖ ≤ 1/ √ 2 , add another coordinate to each xi so that ‖xi‖ = 1/ √ 2 and then add another coordinate with value 1/ √ 2 to each xi . This ensures that ‖xi − xj‖2 ≥ δ implies Assumption 2 for xi , xj , which we later verify empirically for CIFAR10 .

Summary: The paper aims to characterize for different activation functions the minimum eigenvalue of a certain gram matrix that is crucial to the convergence rate for training over-parameterized neural networks in the lazy regime (small learning rate). On this front, the paper shows that for non-smooth activations the minimum eigenvalue is large under separation assumptions on the data improving on prior work. However, for smooth activations the authors show that the minimum eigenvalue can be exponentially small (or even 0 in case of polynomials) if the data has low-dimensional structure or the network is sufficiently deep. The authors experimentally validate the observations on synthetic data.