Learning Two-layer Neural Networks with Symmetric Inputs

y = Aσ ( Wx ) + ξ , whereA , W are weight matrices , ξ represents noise , and the number of neurons in the hidden layer is no larger than the input or output , our algorithm is guaranteed to recover the parameters A , W of the ground-truth network . The only requirement on the input x is that it is symmetric , which still allows highly complicated and structured input . Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions . We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks . Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions . 1 INTRODUCTION . Deep neural networks have been extremely successful in many tasks related to images , videos and reinforcement learning . However , the success of deep learning is still far from being understood in theory . In particular , learning a neural network is a complicated non-convex optimization problem , which is hard in the worst-case . The question of whether we can efficiently learn a neural network still remains generally open , even when the data is drawn from a neural network . Despite a lot of recent effort , the class of neural networks that we know how to provably learn in polynomial time is still very limited , and many results require strong assumptions on the input distribution . In this paper we design a new algorithm that is capable of learning a two-layer1 neural network for a general class of input distributions . Following standard models for learning neural networks , we assume there is a ground truth neural network . The input data ( x , y ) is generated by first sampling the input x from an input distributionD , then computing y according to the ground truth network that is unknown to the learner . The learning algorithm will try to find a neural network f such that f ( x ) is as close to y as possible over the input distribution D. Learning a neural network is known to be a hard problem even in some simple settings ( Goel et al. , 2016 ; Brutzkus & Globerson , 2017 ) , so we need to make assumptions on the network structure or the input distributionD , or both . Many works have worked with a simple input distribution ( such as Gaussians ) and try to learn more and more 1There are different ways to count the number of layers . Here by two-layer network we refer to a fullyconnected network with two layers of edges ( two weight matrices ) . This is considered to be a three-layer network if one counts the number of layers for nodes ( e.g . in Goel & Klivans ( 2017 ) ) or a one-hidden layer network if one just counts the number of hidden layers . complex networks ( Tian , 2017 ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi , 2017 ; Zhong et al. , 2017 ) . However , the input distributions in real life are distributions of very complicated objects such as texts , images or videos . These inputs are highly structured , clearly not Gaussian and do not even have a simple generative model . We consider a type of two-layer neural network , where the output y is generated as y = Aσ ( Wx ) + ξ . ( 1 ) Here x ∈ Rd is the input , W ∈ Rk×d and A ∈ Rk×k are two weight matrices2 . The function σ is the standard ReLU activation function σ ( x ) = max { x , 0 } applied entry-wise to the vector Wx , and ξ is a noise vector that has E [ ξ ] = 0 and is independent of x . Although the network only has two layers , learning similar networks is far from trivial : even when the input distribution is Gaussian , Ge et al . ( 2017b ) and Safran & Shamir ( 2018 ) showed that standard optimization objective can have bad local optimal solutions . Ge et al . ( 2017b ) gave a new and more complicated objective function that does not have bad local minima . For the input distributionD , our only requirement is thatD is symmetric . That is , for any x ∈ Rd , the probability of observing x ∼ D is the same as the probability of observing −x ∼ D. A symmetric distribution can still be very complicated and can not be represented by a finite number of parameters . In practice , one can often think of the symmetry requirement as a “ factor-2 ” approximation to an arbitrary input distribution : if we have arbitrary training samples , it is possible to augment the input data with their negations to make the input distribution symmetric , and it should take at most twice the effort in labeling both the original and augmented data . In many cases ( such as images ) the augmented data can be interpreted ( for images it will just be negated colors ) so reasonable labels can be obtained . 1.1 OUR RESULTS . When the input distribution is symmetric , we give the first algorithm that can learn a two-layer neural network . Our algorithm is based on the method-of-moments approach : first estimate some correlations between x and y , then use these information to recover the model parameters . More precisely we have Theorem 1 ( informal ) . If the data is generated according to Equation ( 1 ) , and the input distribution x ∼ D is symmetric . Given exact correlations between x , y of order at most 4 , as long as A , W and input distribution are not degenerate , there is an algorithm that runs in poly ( d ) time and outputs a network Â , Ŵ of the same size that is effectively the same as the ground-truth network : for any input x , Âσ ( Ŵx ) = Aσ ( Wx ) . Of course , in practice we only have samples of ( x , y ) and can not get the exact correlations . However , our algorithm is robust to perturbations , and in particular can work with polynomially many samples . Theorem 2 ( informal ) . If the data is generated according to Equation ( 1 ) , and the input distribution x ∼ D is symmetric . As long as the weight matricesA , W and input distributions are not degenerate , there is an algorithm that uses poly ( d , 1/ ) time and number of samples and outputs a network Â , Ŵ of the same size that computes an -approximation function to the ground-truth network : for any input x , ‖Âσ ( Ŵx ) −Aσ ( Wx ) ‖2 ≤ . In fact , the algorithm recovers the original parameters A , W up to scaling and permutations . Here when we say weight matrices are not degenerate , we mean that the matrices A , W should be full rank , and in addition a certain distinguishing matrix that we define later in Section 2 is also full rank . We justify these assumptions using the smoothed analysis framework ( Spielman & Teng , 2004 ) . In smoothed analysis , the input is not purely controlled by an adversary . Instead , the adversary can first generate an arbitrary instance ( in our case , arbitrary weight matrices W , A and symmetric input distribution D ) , and the parameters for this instance will be randomly perturbed to yield a perturbed instance . The algorithm only needs to work with high probability on the perturbed instance . This limits the power of the adversary and prevents it from creating highly degenerate cases ( e.g . choosing the weight matrices to be much lower rank than k ) . Roughly speaking , we show 2Here we assume A ∈ Rk×k for simplicity , our results can easily be generalized as long as the dimension of output is no smaller than the number of hidden units . Theorem 3 ( informal ) . There is a simple way to perturb the input distribution , W and A such that with high probability , the distance between the perturbed instance and original instance is at most λ , and our algorithm outputs an -approximation to the perturbed network with poly ( d , 1/λ , 1/ ) time and number of samples . In the rest of the paper , we will first review related works . Then in Section 2 we formally define the network and introduce some notations . Our algorithm is given in Section 3 . Finally in Section 4 we run experiments to show that the algorithm can indeed learn the two-layer network efficiently and robustly . The experiments show that our algorithm works robustly with reasonable number of samples for different ( symmetric ) input distributions and weight matrices . Due to space constraints , the proof for polynomial number of samples ( Theorem 2 ) and smoothed analysis ( Theorem 3 ) are deferred to the appendix . 1.2 RELATED WORK . There are many works in learning neural networks , and they come in many different styles . Non-standard Networks Some works focus on networks that do not use standard activation functions . Arora et al . ( 2014 ) gave an algorithm that learns a network with discrete variables . Livni et al . ( 2014 ) and follow-up works learn neural networks with polynomial activation functions . Oymak & Soltanolkotabi ( 2018 ) used the rank-1 tensor decomposition for learning a non-overlapping convolutional neural network with differentiable and smooth activation and Gaussian input . ReLU network , Gaussian input When the input is Gaussian , Ge et al . ( 2017b ) showed that for a two-layer neural network , although the standard objective does have bad local optimal solutions , one can construct a new objective whose local optima are all globally optimal . Several other works ( Tian , 2017 ; Du et al. , 2017b ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi , 2017 ; Zhong et al. , 2017 ; Zhang et al. , 2018 ) extend this to different settings . General input with score functions A closely related work ( Janzamin et al. , 2015 ) does not require the input distribution to be Gaussian , but still relies on knowing the score function of the input distribution ( which in general can not be estimated efficiently from samples ) . Recently , Gao et al . ( 2018 ) gave a way to design loss functions with desired properties for one-hidden-layer neural networks with general input distributions based on a new proposed local likelihood score function estimator . For general distributions ( including symmetric ones ) their estimator can still require number of samples that is exponential in dimension d ( as in Assumption 1 ( d ) ) . General input distributions There are several lines of work that try to extend the learning results to more general distributions . Du et al . ( 2017a ) showed how to learn a single neuron or a single convolutional filter under some conditions for the input distribution . Daniely et al . ( 2016 ) ; Zhang et al . ( 2016 ; 2017 ) ; Goel & Klivans ( 2017 ) ; Du & Goel ( 2018 ) used kernel methods to learn neural networks when the norm of the weights and input distributions are both bounded ( and in general the running time and sample complexity in this line of work depend exponentially on the norms of weights/input ) . Recently , Du et al . ( 2018 ) showed that gradient descent minimizes the training error in an over-parameterized two-layer neural network . They only consider training error while our results also apply to testing error . The work that is most similar to our setting is Goel et al . ( 2018 ) , where they showed how to learn a single neuron ( or a single convolutional filter ) for any symmetric input distribution . Our two-layer neural network model is much more complicated . Method-of-Moments and Tensor Decomposition Our work uses method-of-moments , which has already been applied to learn many latent variable models ( see Anandkumar et al . ( 2014 ) and references there ) . The particular algorithm that we use is inspired by an over-complete tensor decomposition algorithm FOOBI ( De Lathauwer et al. , 2007 ) . Our smoothed analysis results are inspired by Bhaskara et al . ( 2014 ) and Ma et al . ( 2016 ) , although our setting is more complicated and we need several new ideas .

Summary: This paper pushes forward our understanding of learning neural networks. The authors show that they can learn a two-layer (one hidden layer) NN, under the assumption that the input distribution is symmetric. The authors convincingly argue that this is not an excessive limitation, particularly in view of the fact that this is intended to be a theoretical contribution. Specifically, the main result of the paper relies on the concept of smoothed analysis. It states that give data generated from a network, the input distribution can be perturbed so that their algorithm then returns an epsilon solution.