Learning Two-layer Neural Networks with Symmetric Inputs

y = Aσ ( Wx ) + ξ , whereA , W are weight matrices , ξ represents noise , and the number of neurons in the hidden layer is no larger than the input or output , our algorithm is guaranteed to recover the parameters A , W of the ground-truth network . The only requirement on the input x is that it is symmetric , which still allows highly complicated and structured input . Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions . We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks . Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions . 1 INTRODUCTION . Deep neural networks have been extremely successful in many tasks related to images , videos and reinforcement learning . However , the success of deep learning is still far from being understood in theory . In particular , learning a neural network is a complicated non-convex optimization problem , which is hard in the worst-case . The question of whether we can efficiently learn a neural network still remains generally open , even when the data is drawn from a neural network . Despite a lot of recent effort , the class of neural networks that we know how to provably learn in polynomial time is still very limited , and many results require strong assumptions on the input distribution . In this paper we design a new algorithm that is capable of learning a two-layer1 neural network for a general class of input distributions . Following standard models for learning neural networks , we assume there is a ground truth neural network . The input data ( x , y ) is generated by first sampling the input x from an input distributionD , then computing y according to the ground truth network that is unknown to the learner . The learning algorithm will try to find a neural network f such that f ( x ) is as close to y as possible over the input distribution D. Learning a neural network is known to be a hard problem even in some simple settings ( Goel et al. , 2016 ; Brutzkus & Globerson , 2017 ) , so we need to make assumptions on the network structure or the input distributionD , or both . Many works have worked with a simple input distribution ( such as Gaussians ) and try to learn more and more 1There are different ways to count the number of layers . Here by two-layer network we refer to a fullyconnected network with two layers of edges ( two weight matrices ) . This is considered to be a three-layer network if one counts the number of layers for nodes ( e.g . in Goel & Klivans ( 2017 ) ) or a one-hidden layer network if one just counts the number of hidden layers . complex networks ( Tian , 2017 ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi , 2017 ; Zhong et al. , 2017 ) . However , the input distributions in real life are distributions of very complicated objects such as texts , images or videos . These inputs are highly structured , clearly not Gaussian and do not even have a simple generative model . We consider a type of two-layer neural network , where the output y is generated as y = Aσ ( Wx ) + ξ . ( 1 ) Here x ∈ Rd is the input , W ∈ Rk×d and A ∈ Rk×k are two weight matrices2 . The function σ is the standard ReLU activation function σ ( x ) = max { x , 0 } applied entry-wise to the vector Wx , and ξ is a noise vector that has E [ ξ ] = 0 and is independent of x . Although the network only has two layers , learning similar networks is far from trivial : even when the input distribution is Gaussian , Ge et al . ( 2017b ) and Safran & Shamir ( 2018 ) showed that standard optimization objective can have bad local optimal solutions . Ge et al . ( 2017b ) gave a new and more complicated objective function that does not have bad local minima . For the input distributionD , our only requirement is thatD is symmetric . That is , for any x ∈ Rd , the probability of observing x ∼ D is the same as the probability of observing −x ∼ D. A symmetric distribution can still be very complicated and can not be represented by a finite number of parameters . In practice , one can often think of the symmetry requirement as a “ factor-2 ” approximation to an arbitrary input distribution : if we have arbitrary training samples , it is possible to augment the input data with their negations to make the input distribution symmetric , and it should take at most twice the effort in labeling both the original and augmented data . In many cases ( such as images ) the augmented data can be interpreted ( for images it will just be negated colors ) so reasonable labels can be obtained . 1.1 OUR RESULTS . When the input distribution is symmetric , we give the first algorithm that can learn a two-layer neural network . Our algorithm is based on the method-of-moments approach : first estimate some correlations between x and y , then use these information to recover the model parameters . More precisely we have Theorem 1 ( informal ) . If the data is generated according to Equation ( 1 ) , and the input distribution x ∼ D is symmetric . Given exact correlations between x , y of order at most 4 , as long as A , W and input distribution are not degenerate , there is an algorithm that runs in poly ( d ) time and outputs a network Â , Ŵ of the same size that is effectively the same as the ground-truth network : for any input x , Âσ ( Ŵx ) = Aσ ( Wx ) . Of course , in practice we only have samples of ( x , y ) and can not get the exact correlations . However , our algorithm is robust to perturbations , and in particular can work with polynomially many samples . Theorem 2 ( informal ) . If the data is generated according to Equation ( 1 ) , and the input distribution x ∼ D is symmetric . As long as the weight matricesA , W and input distributions are not degenerate , there is an algorithm that uses poly ( d , 1/ ) time and number of samples and outputs a network Â , Ŵ of the same size that computes an -approximation function to the ground-truth network : for any input x , ‖Âσ ( Ŵx ) −Aσ ( Wx ) ‖2 ≤ . In fact , the algorithm recovers the original parameters A , W up to scaling and permutations . Here when we say weight matrices are not degenerate , we mean that the matrices A , W should be full rank , and in addition a certain distinguishing matrix that we define later in Section 2 is also full rank . We justify these assumptions using the smoothed analysis framework ( Spielman & Teng , 2004 ) . In smoothed analysis , the input is not purely controlled by an adversary . Instead , the adversary can first generate an arbitrary instance ( in our case , arbitrary weight matrices W , A and symmetric input distribution D ) , and the parameters for this instance will be randomly perturbed to yield a perturbed instance . The algorithm only needs to work with high probability on the perturbed instance . This limits the power of the adversary and prevents it from creating highly degenerate cases ( e.g . choosing the weight matrices to be much lower rank than k ) . Roughly speaking , we show 2Here we assume A ∈ Rk×k for simplicity , our results can easily be generalized as long as the dimension of output is no smaller than the number of hidden units . Theorem 3 ( informal ) . There is a simple way to perturb the input distribution , W and A such that with high probability , the distance between the perturbed instance and original instance is at most λ , and our algorithm outputs an -approximation to the perturbed network with poly ( d , 1/λ , 1/ ) time and number of samples . In the rest of the paper , we will first review related works . Then in Section 2 we formally define the network and introduce some notations . Our algorithm is given in Section 3 . Finally in Section 4 we run experiments to show that the algorithm can indeed learn the two-layer network efficiently and robustly . The experiments show that our algorithm works robustly with reasonable number of samples for different ( symmetric ) input distributions and weight matrices . Due to space constraints , the proof for polynomial number of samples ( Theorem 2 ) and smoothed analysis ( Theorem 3 ) are deferred to the appendix . 1.2 RELATED WORK . There are many works in learning neural networks , and they come in many different styles . Non-standard Networks Some works focus on networks that do not use standard activation functions . Arora et al . ( 2014 ) gave an algorithm that learns a network with discrete variables . Livni et al . ( 2014 ) and follow-up works learn neural networks with polynomial activation functions . Oymak & Soltanolkotabi ( 2018 ) used the rank-1 tensor decomposition for learning a non-overlapping convolutional neural network with differentiable and smooth activation and Gaussian input . ReLU network , Gaussian input When the input is Gaussian , Ge et al . ( 2017b ) showed that for a two-layer neural network , although the standard objective does have bad local optimal solutions , one can construct a new objective whose local optima are all globally optimal . Several other works ( Tian , 2017 ; Du et al. , 2017b ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi , 2017 ; Zhong et al. , 2017 ; Zhang et al. , 2018 ) extend this to different settings . General input with score functions A closely related work ( Janzamin et al. , 2015 ) does not require the input distribution to be Gaussian , but still relies on knowing the score function of the input distribution ( which in general can not be estimated efficiently from samples ) . Recently , Gao et al . ( 2018 ) gave a way to design loss functions with desired properties for one-hidden-layer neural networks with general input distributions based on a new proposed local likelihood score function estimator . For general distributions ( including symmetric ones ) their estimator can still require number of samples that is exponential in dimension d ( as in Assumption 1 ( d ) ) . General input distributions There are several lines of work that try to extend the learning results to more general distributions . Du et al . ( 2017a ) showed how to learn a single neuron or a single convolutional filter under some conditions for the input distribution . Daniely et al . ( 2016 ) ; Zhang et al . ( 2016 ; 2017 ) ; Goel & Klivans ( 2017 ) ; Du & Goel ( 2018 ) used kernel methods to learn neural networks when the norm of the weights and input distributions are both bounded ( and in general the running time and sample complexity in this line of work depend exponentially on the norms of weights/input ) . Recently , Du et al . ( 2018 ) showed that gradient descent minimizes the training error in an over-parameterized two-layer neural network . They only consider training error while our results also apply to testing error . The work that is most similar to our setting is Goel et al . ( 2018 ) , where they showed how to learn a single neuron ( or a single convolutional filter ) for any symmetric input distribution . Our two-layer neural network model is much more complicated . Method-of-Moments and Tensor Decomposition Our work uses method-of-moments , which has already been applied to learn many latent variable models ( see Anandkumar et al . ( 2014 ) and references there ) . The particular algorithm that we use is inspired by an over-complete tensor decomposition algorithm FOOBI ( De Lathauwer et al. , 2007 ) . Our smoothed analysis results are inspired by Bhaskara et al . ( 2014 ) and Ma et al . ( 2016 ) , although our setting is more complicated and we need several new ideas .

Summary: This paper studies the problem of learning the parameters of a two-layer (or one-hidden layer) ReLU network $y=A\sigma(Wx)$, under the assumption that the distribution of $x$ is symmetric. The main technique here is the "pure neuron detector", which is a high-order moment function of a vector. It can be proved that the pure neuron detector is zero if and only if the vector is equal to the row vector of A^{-1}. Hence, we can "purify" the two layer neural network into independent one layer neural networks, and solve the problem easily.