Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems

1 INTRODUCTION . In recent years , many studies have been devoted to explaining the great success of over-parameterized neural networks , where the number of parameters is much larger than that needed to fit a given training dataset . This study also treats over-parameterized two-layer neural networks using smooth activation functions and analyzes the convergence and generalization abilities of the gradient descent method for optimizing this type of network . For over-parameterized two-layer neural networks , Du et al . ( 2019 ) ; Arora et al . ( 2019 ) ; Chizat & Bach ( 2018a ) ; Mei et al . ( 2018 ) showed the global convergence of the gradient descent . These studies are mainly divided into two groups depending on the scaling factor of the output of the networks to which the global convergence property has been demonstrated using different types of proofs . For the scaling factor 1/m , ( m : the number of hidden units ) , Chizat & Bach ( 2018a ) ; Mei et al . ( 2018 ) showed the convergence to the global minimum over probability measures when m → ∞ by utilizing the Wasserstein gradient flow perspective ( Nitanda & Suzuki , 2017 ) on the gradient descent . For the scaling factor 1/mβ ( β < 1 ) , Du et al . ( 2019 ) essentially demonstrated that the kernel smoothing of functional gradients by the neural tangent kernel ( Jacot et al. , 2018 ; Chizat & Bach , 2018b ) has comparable performance with the functional gradient as m → ∞ by making a positivity assumption on the Gram-matrix of this kernel , resulting in the global convergence property . In addition , Arora et al . ( 2019 ) provided a generalization bound via a fine-grained analysis of the gradient descent . These studies provide the first steps to understand the role of over-parameterization of neural networks and the gradient descent on regression problems using the squared loss function . For the classification problems with logistic loss , a few studies ( Allen-Zhu et al. , 2018a ; Cao & Gu , 2019a ; b ) investigated the convergence and generalization abilities of gradient descent under a separability assumption with a suitable model instead of the positivity of the neural tangent kernel . In this study , we further develop this line of research on binary classification problems . Our contributions . We provide fine-grained global convergence and generalization analyses of the gradient descent for two-layer neural networks with smooth activations under a separability assumption with a sufficient margin using a neural tangent model , which is a non-linear model with feature extraction through a neural tangent . We demonstrate that a separability assumption is more suitable than the positivity condition of the neural tangent kernel because ( i ) the positive neural tangent kernel leads to weak separability and conversely , ( ii ) separability leads to the positivity of the neural tangent kernel only on a cone spanned by labels , which is very restrictive compared to the whole space . Therefore , the separability condition is rather weak in this sense but it is enough to ensure global convergence for the classification problems . Thus , a significantly better convergence and generalization analyses with respect to network width can be obtained because the positivity of the neural tangent kernel is not required . Consequently , our theory provides a generalization guarantee for less over-parameterized two-layer networks trained by gradient descent , while most existing results relying on the positive neural tangent kernel essentially require quite high overparameterization . To the best of our knowledge , there are no successful studies for our problem setting ( i.e. , less over-parameterized two-layer neural networks with smooth activation functions for the classification problems with logistic loss ) in the literature . Most studies have focused on highly over-parameterized neural networks with ReLU activation , and less over-parameterized settings have been considered difficult for showing the global convergence property of gradient descent even in the few studies using a separability condition ( Allen-Zhu et al. , 2018a ; Cao & Gu , 2019a ; b ) . However , we note that these studies provided global convergence and generalization analyses of the ( stochastic ) gradient descent for challenging settings ( i.e. , deep ReLU networks ) by making a similar but different assumption than ours . Thus , our and these studies do not include each other because of the difference of the network structure ( i.e. , network depth and activation type ) and assumptions . We here describe the main result informally . A neural tangent model is an infinite-dimensional non-linear model using transformed features ( ∂θσ ( θ ( 0 ) > x ) ) θ ( 0 ) ∼µ0 , where σ is a smooth activation and µ0 is a distribution used to initialize the parameters of the input layer in two-layer neural networks . Theorem 1 states that gradient descent can find an -accurate solution in terms of the expected classification error for a wide class of over-parameterized two-layer neural networks under a separability assumption using a neural tangent model . Theorem 1 ( Informal ) . Suppose that a given data distribution is separable by a neural tangent model with a sufficient margin under L∞-constraint . If for any > 0 , the hyperparameters satisfy one of the following ( i ) β ∈ [ 0 , 1 ) , m = Ω ( −1 1−β ) , T = Θ ( −2 ) , η = Θ ( m2β−1 ) , n = Ω̃ ( −4 ) , ( ii ) β = 0 , m = Θ̃ ( −3/2 ) , T = Θ̃ ( −1 ) , η = Θ ( m−1 ) , n = Ω̃ ( −2 ) . then with high probability over the random initialization and choice of samples of size n , the gradient descent with a learning rate η achieves an expected -classification error within T -iterations . Related work . A few recent studies ( Allen-Zhu et al. , 2018a ; Cao & Gu , 2019a ; b ) are closely related to our work because they also treated the logistic loss function . As stated above , problem settings in our and these studies are somewhat different , but we compare our result with those specialized to two-layer network to show a better property of our problem setting and analyses . Separability assumptions were made on an infinite-width two-layer ReLU network in Cao & Gu ( 2019a ; b ) and on a smooth target function in Allen-Zhu et al . ( 2018a ) . For generalization analyses , our result exhibits much better dependency on the network width owing to a better problem setting with a fine-grained analysis . Table 1 provides a comparison of the hyperparameter settings of networks and gradient descent in related studies to achieve an expected -classification error . As evident in Table 1 , for more comprehensive sizes of two-layer networks with respect to the network width , our theory ensures the same generalization ability as those of Allen-Zhu et al . ( 2018a ) ; Cao & Gu ( 2019a ; b ) . In fact , the network width Ω ( −1 ) and Ω ( −3/2 ) are sufficient in our setting . For the stochastic gradient descent for two-layer networks , Brutzkus et al . ( 2018 ) ; Li & Liang ( 2018 ) provided generalization analyses . Brutzkus et al . ( 2018 ) assumed that datasets are linear separable and this restrictive assumption was relaxed to mixtures of well separated data distributions in Li & Liang ( 2018 ) . However , the analysis in Li & Liang ( 2018 ) is also tailored to only highly overparameterized settings . Concretely , a very large width m = Ω̃ ( −24 ) and the number of samples ( iterations ) n = Θ ( T ) = Õ ( −12 ) are required to achieve an expected -classification error in Li & Liang ( 2018 ) . In addition , it should be noted that global convergence analyses ( Allen-Zhu et al. , 2018b ; Zou et al. , 2018 ) in terms of optimization without the specification of network size will yield loose generalization bounds because the complexities of neural networks can not be specified . Apart from the abovementioned studies , there are many other studies ( Brutzkus & Globerson , 2017 ; Zhong et al. , 2017 ; Tian , 2017 ; Soltanolkotabi , 2017 ; Du et al. , 2019 ; Zhang et al. , 2018 ; Arora et al. , 2019 ; Oymak & Soltanolkotabi , 2019 ; Zhang et al. , 2019 ; Wu et al. , 2019 ) that focus on regression problems but our study focuses on classification problems and demonstrates a better property of gradient descent for over-parameterized networks by utilizing the problem structure of a binary classification . Especially , we show that a separability assumption is more preferable than the positivity condition of the neural tangent kernel ( Du et al. , 2019 ; Arora et al. , 2019 ; Zhang et al. , 2019 ; Wu et al. , 2019 ) and show that the required network width can be significantly reduced . Concretely , the required network widths are at least Ω ( n6 ) ( Du et al. , 2019 ; Wu et al. , 2019 ) , Ω ( n7 −2 ) ( Arora et al. , 2019 ) , and Ω ( n4 ) ( Zhang et al. , 2019 ) . Thus , these widths are very large , as compared to our results because sample complexities are generally slower than or equal to n = Ω ( −2 ) . In addition , proof techniques are also different for the squared loss and the logistic loss functions because the latter function lacks the strong convexity . Thus , we can not utilize the linear convergence property for the logistic loss and parameters will diverge , which also causes the difficulty of showing better generalization ability without a fine-grained analysis . 2 PRELIMINARY . Here , we describe the problem setting for the binary logistic regression and discuss the functional gradients to provide a clear theoretical view of the gradient methods for two-layer neural networks . 2.1 PROBLEM SETTING . Let X = Rd and Y be a feature space and the set of binary labels { −1 , 1 } , respectively . We denote by ν a true probability measure on X ×Y and by νn an empirical probability measure , deduced from observations ( xi , yi ) ni=1 independently drawn from ν , i.e. , dνn ( X , Y ) = ∑n i=1 δ ( xi , yi ) ( X , Y ) dXdY/n , where δ is the Dirac delta function . The marginal distributions of ν and νn on X are denoted by νX and νXn , respectively . For ζ ∈ R and y ∈ Y , let l ( ζ , y ) be the logistic loss : log ( 1 + exp ( −yζ ) ) . Then , the objective function to be minimized is formalized as follows : L ( Θ ) def= E ( X , Y ) ∼νn [ l ( fΘ ( X ) , Y ) ] = 1 n n∑ i=1 l ( fΘ ( xi ) , yi ) , where fΘ : X → R is a two-layer neural network equipped with parameters Θ = ( θr ) mr=1 . When we consider a function fΘ as a variable of the objective function , we denote L ( fΘ ) def = L ( Θ ) . The two-layer neural network treated in this study is formalized as follows . For parameters Θ = ( θr ) m r=1 ( θr ∈ Rd ) and fixed constants ( ar ) mr=1 ∈ { −1 , 1 } m : fΘ ( x ) = 1 mβ m∑ r=1 arσ ( θ > r x ) , ( 1 ) where m is the number of hidden units , β is an order of the scaling factor , and σ : R → R is a smooth activation function such as sigmoid , tanh , swish ( Ramachandran et al . ) ) , and other smooth approximations of ReLU . In the training procedure , the parameters Θ = ( θr ) mr=1 of the input layer are optimized . This setting is the same as those in Du et al . ( 2019 ) ; Arora et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Wu et al . ( 2019 ) , except for the types of activation functions , scaling factor , and loss function .

Summary: The authors study the problem of binary logistic regression in a two-layer network with a smooth activation function.  They introduce a separability assumption on the dataset using the neural tangent model.  This separability assumption is weaker than the more Neural Tangent Kernel assumption that has been extensively studied in the regression literature.  In that case, a certain Gram-matrix must be nonnegative.  In the current work, the authors observe that the structure of the logistic loss in the binary classification problem restricts the functional gradients to lie in a particular space, meaning that nonnegative of the Gram-matrix is only needed on a subspace.  This is the underlying theoretical reason for why they can get improvement over those methods in the setting they study.  Under the separability assumption, the authors prove convergent gradient descent and generalization of the ensuring net, while assuming the two-layer networks are less overparameterized than what would have been possible under the Gram-matrix perspective.