Improved Learning of One-hidden-layer Convolutional Neural Networks with Overlaps

1 INTRODUCTION . Giving provably efficient algorithms for learning neural networks is a core challenge in machine learning theory . The case of convolutional architectures has recently attracted much interest due to their many practical applications . Recently Brutzkus & Globerson ( 2017 ) showed that distributionfree learning of one simple non-overlapping convolutional filter is NP-hard . A natural open question is whether we can design provably efficient algorithms to learn convolutional neural networks under mild assumptions . We consider a convolutional neural network of the form f px , w , aq “ k ÿ j “ 1 ajσ ` wJPjx ˘ ( 1 ) where w P Rr is a shared convolutional filter , a P Rk is the second linear layer and Pj “ r 0lomon pj´1qs I lomon r 0 lomon d´pj´1qs ` r s P Rrˆd selects the ppj ´ 1qs ` 1q-th to ppj ´ 1qs ` rq-th coordinates of x with stride s and σ p¨q is the activation function . Note here that both w and a are unknown vectors to be learned and there may be overlapping patches because the stride size s may be smaller than the filter size r. Our Contributions We give the first efficient algorithm that can provably learn a convolutional neural network with two unknown layers with commonly used overlapping patches . Our main result is the following theorem . Theorem 1.1 ( Main Theorem ( Informal ) ) . Suppose s ě t r2 u ` 1 and the marginal distribution is symmetric and isotropic . Then the convolutional neural network defined in equation 1 with piecewise linear activation functions is learnable in polynomial time . We refer readers to Theorem 3.1 for the precise statement . Technical Insights Our algorithm is a novel combination of the algorithm for isotonic regression and the landscape analysis of non-convex problems . First , inspired by recent work on isotonic regression , we extend the idea in Goel et al . ( 2018 ) to reduce learning a CNN with piecewise linear activation to learning a convolutional neural network with linear activation ( c.f . Section 4 ) . Second , we show learning a linear convolutional filter can be reduced to a non-convex matrix factorization problem which admits a provably efficient algorithm based on non-convex geometry ( Ge et al. , 2017a ) . Third , in analyzing our algorithm , we present a robust analysis of Convotron algorithm proposed by Goel et al . ( 2018 ) , in which we draw connections to the spectral properties of Toeplitz matrices . We believe these ideas may inspire further development in designing provable learning algorithms for neural networks and other complex models . Related Work From the point of view of learning theory , it is well known that training is computational infeasible in the worst case ( Goel et al. , 2016 ; Brutzkus & Globerson , 2017 ) . Thus distributional assumptions are needed for efficient learning . A line of research has focused on analyzing the dynamics of gradient descent conditioned on the input distribution being standard Gaussian ( Tian , 2017 ; Soltanolkotabi , 2017 ; Li & Yuan , 2017 ; Zhong et al. , 2017b ; Brutzkus & Globerson , 2017 ; Zhong et al. , 2017a ; Du et al. , 2017 ) . Specifically for convolutional nets , existing analyses heavily relied on the analytical formulas which can only be derived if the input is Gaussian and patches are non-overlapping . Recent work has tried to relax the Gaussian input assumption and the non-overlapping structure for learning convolutional filters . Du et al . ( 2017 ) showed if the patches are sufficiently close to each other then stochastic gradient descent can recover the true filter . Goel et al . ( 2018 ) proposed a modified iterative algorithm inspired from isotonic regression that gives the first recovery guarantees for learning a filter for commonly used overlapping patches under much weaker assumptions on the distribution . However , these two analyses only work for learning one unknown convoutional filter . Moving away from gradient descent , various works have shown positive results for learning general simple fully connected neural networks in polynomial time and sample complexity under certain assumptions using techniques such as kernel methods ( Goel et al. , 2016 ; Zhang et al. , 2015 ; Goel & Klivans , 2017a ; b ) and tensor decomposition ( Sedghi & Anandkumar , 2014 ; Janzamin et al. , 2015 ) . The main drawbacks include the shift to improper learning for kernel methods and the knowledge of the probability density function for tensor methods . In contrast to this , our algorithm is proper and does not assume that the input distribution is known . Learning a neural network is often formulated as a non-convex problem . If the objective function satisfies ( 1 ) all saddle points and local maxima are strict ( i.e. , there exists a direction with negative curvature ) , and ( 2 ) all local minima are global ( no spurious local minmum ) , then noise-injected ( stochastic ) gradient descent ( Ge et al. , 2015 ; Jin et al. , 2017 ) finds a global minimum in polynomial time . Recent work has studied these properties for the landscape of neural networks ( Kawaguchi , 2016 ; Choromanska et al. , 2015 ; Hardt & Ma , 2016 ; Haeffele & Vidal , 2015 ; Mei et al. , 2016 ; Freeman & Bruna , 2016 ; Safran & Shamir , 2016 ; Zhou & Feng , 2017 ; Nguyen & Hein , 2017a ; b ; Ge et al. , 2017b ; Zhou & Feng , 2017 ; Safran & Shamir , 2017 ; Du & Lee , 2018 ) . A crucial step in our algorithm is reducing the convolutional neural network learning problem to matrix factorization and using the geometric properties of matrix factorization . 2 PRELIMINARIES . We use bold-faced letters for vectors and matrices . We use } ¨ } 2 to denote the Euclidean norm of a finite-dimensional vector . For a matrix A , we use λmax pAq to denote its eigenvalue and λmin pAq its smallest singular value . Let Op¨q and Ω p¨q denote standard Big-O and Big-Omega notations , only hiding absolute constants . In our setting , we have n data points txi , yiuni “ 1 where xi P Rd and y P R. We assume the label is generated by a two-layer convolutional neural network with filter size r , stride s and k hidden neurons . Compactly we can write the formula in the following form : yi “ fpxi , w˚ , a˚q , xi „ Z where the prediction function f is defined in equation 1 . To obtain a proper scaling , we let } w˚ } 2 } a˚ } 2 “ σ1 . We also define the induced patch matrix as P pxq “ rP1x . . . Pkxs P Rrˆk which will play an important role in our algorithm design . Our goal is to properly learn this convolutional neural network , i.e. , design a polynomial time algorithm which outputs a pair pw , aq that satisfies Ex „ Z ” pfpw , a , xq ´ fpw˚ , a˚ , xqq2 ı ď . Algorithm 1 Learning One-hidden-Layer Convolutional Network Input : Input distribution Z . Number of iterations : T1 , T2 . Number of samples : T3 . Step sizes : η1 ą 0 , η2 ą 0 . Output : Parameters of the one-hidden-layer CNN : w and a . 1 : Stage 1 : Run Double Convotron ( Algorithm 2 ) for T1 iterations with step size η1 to obtain apT1q . 2 : Stage 2 : Run Convotron ( Algorithm 3 ) using apT1q and ´apT1q for T2 iterations and step size η2 to obtain wp ` q and wp´q . 3 : Stage 3 : Choose parameters with lower empirical loss on T3 samples drawn from Z from ` wp ` q , apT q ˘ and ` wp´q , ´apT q ˘ . 3 MAIN RESULT . In this section we describe our main result . We first list our main assumptions , followed by the detailed description of our algorithm . Lastly we state the main theorem which gives the convergence guarantees of our algorithm . 3.1 ASSUMPTIONS . Our first assumption is on the input distribution Z . We assume the input distribution is symmetric , bounded and has identity covariance . The symmetry assumption is used in Goel et al . ( 2018 ) and many learning theory papers Baum ( 1990 ) . The identity covariance assumption is true if the data whitened . Further , in many architectures , the input of certain layers is assumed having these properties because of the use of batch normalization ( Ioffe & Szegedy , 2015 ) or other techniques . Lastly , the boundedness is a standard regularity assumption to exclude pathological input distributions . We remark that this assumption considerably weaker than the standard Gaussian input distribution assumption used in Tian ( 2017 ) ; Zhong et al . ( 2017a ) ; Du et al . ( 2017 ) , which has the rotational invariant property . Assumption 3.1 ( Input Distribution Assumptions ) . We assume the input distribution satisfies the following conditions . • Symmetry : P pxq “ P p´xq . • Identity covariance : Ex „ Z “ xxT ‰ “ I . • Boundedness : @ x „ Z , } x } 2 ď B almost surely for some B ą 0 . Our second assumption is on the patch structure . In this paper we assume the stride is larger than half of the filter size . This is indeed true for a wide range of convolutional neural network used in computer vision . For example some architecture has convolutional filter of size 3 and stride 2 and some use non-overlapping architectures ( He et al. , 2016 ) . Assumption 3.2 ( Large Stride ) . s ě t r2 u ` 1 . Next we assume the activation function is piecewise linear . Commonly used activation functions like rectified linear unit ( ReLU ) , Leaky ReLU and linear activation all belong to this class . Assumption 3.3 ( Piece-wise Linear Activation ) . σpxq “ `` x if x ě 0 αx if x ă 0 . 3.2 ALGORITHM . Now we are ready to describe our algorithm ( see Algorithm 1 ) . The algorithm has three stages , first we learn the outer layer weights upto sign , second we use these fixed outer weights to recover the filter weight and last we choose the best weight combination thus recovered . Stage 1 : Learning the Non-overlapping Part of the Convolutional Filter and Linear Weights Our first observation is even if there may be overlapping patches , as long as there exists some nonoverlapping part , we can learn this part and the second layer jointly . To be specific , with filter size being r and stride being s , if s ě t r2 u ` 1 , for j “ 1. . . . , k we define the selection matrix for the non-overlapping part of each patch Pnonj “ r 0lomon pj´1qs ` r I lomon 2s´r 0 lomon d´pj ` 1qs s P Rp2s´rqˆd . Note that for any j1 ‰ j2 , there is no overlapping between the selected coordinates by Pnonj1 and Pnonj2 . Therefore , for a filter w , there is a segment rwr´s ` 1 , . . . , wss with length p2s´rq which acts on the non-overlapping part of each patches . We denote wnon “ rwr´s ` 1 , . . . , wss and our goal in this stage is to learn w˚non and a ˚ jointly . In this stage , our algorithm proceeds as follows . Given wnon , a and a sample px , yq , we define g pwnon , a , x , yq “ 2 1 ` γ ´ f̂pwnon , a , xq ´ y ¯ k ÿ j “ 1 aiP non j x ` 1 4 ´ } wnon } 22 ´ } a } 2 2 ¯ wnon ( 2 ) h pwnon , a , x , yq “ 2 1 ` γ ´ f̂pwnon , a , xq ´ y ¯ ¨ ˝ wJnonP non 1 x . . . wJnonP non k x ˛ ‚ ` 1 4 ´ } a } 22 ´ } wnon } 2 2 ¯ a ( 3 ) where f̂ pwnon , a , xq “ řk j “ 1 ajσ ` wJnonP non j x ˘ is the prediction function only using wnon . As will be apparent in Section 4 , g and h are unbiased estimates of the gradient for the loss function of learning a linear CNN . The term 14 ´ } wnon } 22 ´ } a } 2 2 ¯ wnon and 14 ´ } a } 22 ´ } wnon } 2 2 ¯ a are is the gradient induced by the regularization 14 ´ } wnon } 22 ´ } a } 2 2 ¯2 , which is used to balance the magnitude between wnon and a and make the algorithm more stable . With some initialization wp0qnon and ap0q , we use the following iterative updates inspired by isotonic regression ( Goel et al. , 2018 ) , for t “ 0 , . . . , T1 ´ 1 wpt ` 1qnon Ðwptqnon ´ η1g ´ wptqnon , a ptq , xptq , yptq ¯ ` η1ξptqwnon , ( 4 ) apt ` 1q Ðaptq ´ η1h ´ wptqnon , a ptq , xptq , yptq ¯ ` η1ξptqa ( 5 ) where η1 ą 0 is the step size parameter , ξptqwnon and ξ ptq a are uniformly sampled a unit sphere and at iteration we use a fresh sample ` xptq , yptq ˘ . Here we add isotropic noise ξptqwnon and ξ ptq a because the objective function for learning a linear CNN is non-convex and there may exist saddle points . Adding noise can help escape from these saddle points . We refer readers to Ge et al . ( 2015 ) for more technical details regarding this . As will be apparent in Section 4 , after sufficient iterations , we obtain a pair ` wpT1q , apT1q ˘ such that either it is close to the truth pw˚non , a˚q or close to the negative of the truth p´w˚non , ´a˚q . Remark 3.1 ( Non-overlapping Patches ) . If there is no overlap between patches , we can skip Stage 2 because after Stage 1 we have already learned a and wnon “ w. Stage 2 : Convotron with fixed Linear Layer In Stage 1 we have learned a good approximation to the second layer ( either apT1q or´apT1q ) . Therefore , the problem reduces to learning a convolutional filter . We run Convotron ( Algorithm 3 ) proposed in Goel et al . ( 2018 ) using apT1q and ´apT1q to obtain corresponding weight vectors wp ` q and wp´q . We show that the Convotron analysis can be extended to handle approximately known outer layer weight vectors . Stage 3 : Validation In stage 2 we have obtained two possible solutions ` wp ` q , apT q ˘ and ` wp´q , ´apT q ˘ . We know at least one of them is close to the ground truth . Closeness in ground truth implies small squared loss ( c.f . Lemma A.1 ) . In the last stage we use a validation set to choose Algorithm 2 Double Convotron Initialize wp0qnon P R2s´r and ap0q P Rk randomly for t “ 1 to T do Draw pxptq , yptqq „ Z Compute g ` wnon , a , x ptq , yptq ˘ and h ` wnon , a , x ptq , yptq ˘ according to equation 2 and equation 3 . Set wpt ` 1qnon “ wptqnon ´ η1g ´ w ptq non , aptq , xptq , yptq ¯ ` η1ξptqwnon Set apt ` 1q “ aptq ´ η1h ´ w ptq non , aptq , xptq , yptq ¯ ` η1ξptqa Return apT ` 1q Algorithm 3 Convotron ( Goel et al. , 2018 ) Initialize w1 : “ 0 P Rr . for t “ 1 to T do Draw pxptq , yptqq „ Z Let gptq “ pyptq ´ fpwptq , a , xptqqq ´ řk i “ 1 aiPix ptq ¯ Set wpt ` 1q “ wptq ` ηgptq Return wT ` 1 the right one . To do this , we simply use T3 “ poly ` k , B , 1 ˘ fresh samples and output the solution which gives lower squared error . pw , aq “ arg min pw , aqPtpwp ` q , apT qq , pwp´q , ´apT qqu 1 T3 T3 ÿ i “ 1 ´ ypiq ´ f ´ w , a , xpiq ¯¯2 . ( 6 ) Since we draw many samples , the empirical estimates will be close to the true loss using standard concentration bounds and choosing the minimum will give us the correct solution .

Summary: This paper gives a new algorithm for learning a two layer neural network which involves a single convolutional filter and a weight vector for different locations. The algorithm works on any symmetric input data. The techniques in this paper combines two previous approaches: 1. the algorithm Convotron for learning a single convolutional filter (while the second layer has fixed weight) on any symmetric input distributions; 2. non-convex optimization for low rank matrix factorization.