Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropy

1 INTRODUCTION . Neural networks have attracted a significant amount of research interest in recent years due to the success of deep neural networks ( LeCun et al. , 2015 ) in practical domains such as computer vision and artificial intelligence ( Russakovsky et al. , 2015 ; He et al. , 2016 ; Silver et al. , 2016 ) . However , the theoretical underpinnings behind such success remains mysterious to a large extent . Efforts have been taken to understand which classes of functions can be represented by deep neural networks ( Cybenko , 1989 ; Hornik et al. , 1989 ; Barron , 1993 ; Telgarsky , 2016 ) , when ( stochastic ) gradient descent is effective for optimizing a non-convex loss function ( Dauphin et al. , 2014 ) , and why these networks generalize well ( Zhang et al. , 2016 ; Bartlett et al. , 2017 ; Brutzkus et al. , 2017 ) . One important line of research that has attracted extensive attention is a model-recovery setup , i.e. , given that the training samples ( xi , yi ) ∼ ( x , y ) are generated i.i.d . from a distribution D based on a neural network model with the ground truth parameterW ? , the goal is to recover the underlying model parameterW ? , which is important for the network to generalize well ( Mondelli & Montanari , 2018 ) . Previous studies along this topic can be mainly divided into two types of data generations . First , a regression problem , for example , assumes that each sample y is generated as y = 1K ∑K k=1 φ ( w ? > k x ) , wherewk ∈ Rd is the weight vector of the kth neuron , 1 ≤ k ≤ K , and the input x ∈ Rd is Gaussian . This type of regression problem has been studied in various settings . In particular , ( Soltanolkotabi , 2017 ) studied the single-neuron model under ReLU activation , ( Zhong et al. , 2017b ) studied the onehidden-layer multi-neuron network model , and ( Li & Yuan , 2017 ) studied a two-layer feedforward networks with ReLU activations and identity mapping . Second , for a classification problem , suppose each label y ∈ { 0 , 1 } is drawn under the conditional distribution P ( y = 1|x ) = 1K ∑K k=1 φ ( w ? > k x ) , wherew ? k ∈ Rd is the weight vector of the kth neuron , 1 ≤ k ≤ K , and the input x ∈ Rd is Gaussian . Such a problem has been studied in ( Mei et al. , 2016 ) in the case with a single neuron . For both the regression and the classification settings , in order to recover the neural network parameters , all previous studies considered ( stochastic ) gradient descent over the squared loss , i.e. , ` qu ( W ; x , y ) = 1 2 ( y − 1 K K∑ i=1 φ ( w > i x ) ) 2 , ( 1 ) which yields gradient and Hessian in relatively simple forms to assist the landscape characterization of the function as well as model recovery analysis . However , for the classification problem , the cross entropy objective used in practice takes the following form ` ( W ; x , y ) = −y · log ( 1 K K∑ i=1 φ ( w > i x ) ) − ( 1− y ) · log ( 1− 1 K K∑ i=1 φ ( w > i x ) ) . ( 2 ) The geometry as well as the model recovery problem based on the entropy loss function have not yet been understood . It is expected that such a loss function is very challenging to analyze , not just because it is nonconvex with multiple neurons , but also because the gradient and Hessian take much more complicated forms compared with the squared loss . The main focus of this paper is to develop technical analysis for guaranteed model recovery under the challenging cross entropy loss function in eq . ( 2 ) for the classification problem in the multi-neuron case . Furthermore , previous studies provided two types of statistical guarantees for such model recovery problems using the squared loss . More specifically , ( Zhong et al. , 2017b ) showed that in the local neighborhood of the ground truth , the Hessian of the empirical loss function is positive definite for each given point under independent high probability event . Hence , their guarantee for gradient descent to converge to the ground truth requires a fresh set of samples at every iteration , thus the total sample complexity will depend on the number of iterations . On the other hand , studies such as ( Mei et al. , 2016 ; Soltanolkotabi , 2017 ) establish certain types of uniform geometry such as strong convexity so that resampling per iteration is not needed for gradient descent to have guaranteed linear convergence as long as it enters such a local neighborhood . However , such a stronger statistical guarantee without per-iteration resampling have only been shown for the squared loss function . In this paper , we aim at developing such a strong statistical guarantee for the loss function in eq . ( 2 ) , which is much more challenging but more practical than the squared loss for the classification problem . 1.1 OUR CONTRIBUTIONS . This study provides the first performance guarantee for the recovery of one-hidden-layer neural networks using the cross entropy loss function , to the best of our knowledge . More specifically , our contributions are summarized as follows . • For multi-neuron classification problem with sigmoid activations , we show that , if the input is Gaussian , the empirical risk function fn ( W ) = 1n ∑n i=1 ` ( W ; xi ) based on the cross entropy loss in eq . ( 2 ) is uniformly strongly convex in a local neighborhood of the ground truthW ? of size O ( 1/K3/2 ) as soon as the sample size is O ( dK5 log2 d ) , where d is the input dimension and K is the number of neurons . • We further show that , if initialized in this neighborhood , gradient descent converges linearly to a critical point Ŵn ( which we show to exist ) , with a sample complexity of O ( dK5 log2 d ) , which is near-optimal up to a polynomial factor in K and log d. Due to the nature of quantized labels here , the recover of W ? is only up to certain statistical accuracy , and Ŵn converges to W ? at a rate of O ( √ dK9/2 log n/n ) in the Frobenius norm . Furthermore , such a convergence guarantee does not require a fresh set of samples at each iteration due to the uniform strong convexity in the local neighborhood . To obtain -accuracy , it requires a computational complexity of O ( ndK2 log ( 1/ ) ) . • We adopt the tensor method proposed in ( Zhong et al. , 2017b ) , and show it provably provides an initialization in the neighborhood of the ground truth . In particular , our proof replaces the homogeneous assumption on activation functions in ( Zhong et al. , 2017b ) by a mild condition on the curvature of activation functions aroundW ? , which holds for a larger class of activation functions including sigmoid and tanh . In order to analyze the challenging cross-entropy loss function , our proof develops various new machineries in order to exploit the statistical information of the geometric curvatures , including the gradient and Hessian of the empirical risk , and to develop covering arguments to guarantee uniform concentrations . Our technique also yields similar performance guarantees for the classification problem using the squared loss in eq . ( 1 ) , which we omit due to space limitations , as it is easier to analyze than cross entropy . 1.2 RELATED WORK . Due to page limitations we focus on the most relevant literature on theoretical and algorithmic aspects of learning shallow neural networks via nonconvex optimization . The parameter recovery viewpoint is relevant to the success of non-convex learning in signal processing problems such as matrix completion , phase retrieval , blind deconvolution , dictionary learning and tensor decomposition ( Sun & Luo , 2016 ; Candès et al. , 2015 ; Ge & Ma , 2017 ; Ge et al. , 2016 ; Sun et al. , 2015 ; Bhojanapalli et al. , 2016 ; Ma et al. , 2017 ) , to name a few . The statistical model for data generation effectively removes worst-case instances and allows us to focus on average-case performance , which often possess much benign geometric properties that enable global convergence of simple local search algorithms . The studies of one-hidden-layer network model can be further categorized into two classes , landscape analysis and model recovery . In the landscape analysis , it is known that if the network size is large enough compared to the data input , then there are no spurious local minima in the optimization landscape , and all local minima are global ( Soltanolkotabi et al. , 2017 ; Boob & Lan , 2017 ; Safran & Shamir , 2016 ; Nguyen & Hein , 2017 ) . For the case with multiple neurons ( 2 ≤ K ≤ d ) in the under-parameterized setting , the work of Tian ( Tian , 2017 ) studied the landscape of the population squared loss surface with ReLU activations . In particular , there exist spurious bad local minima in the optimization landscape ( Ge et al. , 2017 ; Safran & Shamir , 2017 ) even at the population level . Zhong et . al . ( Zhong et al. , 2017b ) provided several important characterizations for the local Hessian for the regression problem for a variety of activation functions for the squared loss . In the model recovery problem , the number of neurons is smaller than the dimension of inputs . In the case with a single neuron ( K = 1 ) , under Gaussian input , ( Soltanolkotabi , 2017 ) showed that gradient descent converges linearly when the activation function is ReLU , i.e . φ ( z ) = max { z , 0 } , with a zero initialization , as long as the sample complexity is O ( d ) for the regression problem . On the other end , ( Mei et al. , 2016 ) showed that when φ ( · ) has bounded first , second and third derivatives , there is no other critical points than the unique global minimum ( within a constrained region of interest ) , and ( projected ) gradient descent converges linearly with an arbitrary initialization , as long as the sample complexity is O ( d log2 d ) with sub-Gaussian inputs for the classification problem using the squared loss . Moreover , ( Zhong et al. , 2017b ) shows that the ground truth From a technical perspective , our study differs from all the aforementioned work in that the cross entropy loss function we analyze has a very different form . Furthermore , we study the model recovery classification problem under the multi-neuron case , which has not been studied before . Finally , we note that several papers study one-hidden-layer or two-layer neural networks with different structures under Gaussian input . For example , ( Brutzkus & Globerson , 2017 ; Du et al. , 2017a ; b ; Zhong et al. , 2017a ) studied the non-overlapping convolutional neural network , ( Li & Yuan , 2017 ) studied a two-layer feedforward networks with ReLU activations and identity mapping , and ( Feizi et al. , 2017 ) introduced the Porcupine Neural Network . These results are not directly comparable to ours since both the networks and the loss functions are different .

Summary: This paper studies the problem of recovering a true underlying neural network (assuming there exists one) with cross-entropy loss. This paper shows if the input is standard Gaussian, within a small ball around the ground truth, the objective function is strongly convex and smooth if there is a sufficiently large number of samples. Furthermore, the global minimizer is actually the true neural network. This geometric analysis implies applying gradient descent within this neighborhood, one can recover the underlying neural network. This paper also proposed a provable method based on spectral learning to find a good initialization point. Lastly, this paper also provides some simulation studies.