On the Landscape of Sparse Linear Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved remarkable empirical successes in the domains of computer vision , speech recognition , and natural language processing , sparking great interests in the theory behind their architectures and training . However , DNNs are often found to be highly overparameterized , making them computationally expensive with large amounts of memory and computational power . For example , it may take up to weeks on a modern multi-GPU server for large datasets such as ImageNet ( Deng et al. , 2009 ) . Hence , DNNs are often unsuitable for smaller devices like embedded electronics , and there is a pressing demand for techniques to optimize models with reduced model size , faster inference and lower power consumption . Sparse networks , that is , neural networks in which a large subset of the model parameters are zero , have emerged as one of the leading approaches for reducing model parameter count . It has been shown empirically that deep neural networks can achieve state-of-the-art results under high levels of sparsity ( Han et al. , 2015b ; Gale et al. , 2019 ; Louizos et al. , 2017a ) . Modern sparse networks are mainly obtained from network pruning ( Zhu & Gupta , 2017 ; Lee et al. , 2018 ; Liu et al. , 2018 ; Frankle & Carbin , 2018 ) , which has been the subject of a great deal of work in recent years . However , training a sparse network with fixed sparsity patterns is difficult ( Evci et al. , 2019 ) and few theoretical understanding of general sparse networks has been provided . Previous work has already analyze deep neural networks , showing that the non-convexity of the associated loss functions may cause complicated and strange optimization landscapes . However , the property of general sparse networks is poorly understood . Saxe et al . ( 2013 ) empirically showed that the optimization of deep linear models exhibits similar properties as deep nonlinear models , and for theoretical development , it is natural to begin with linear models before studying nonlinear models ( Baldi & Lu , 2012 ) . In addition , several works ( Sun et al. , 2020 ) have show bad minimum exists with nonlinear activation . Hence , it is natural to begin with linear activation to understand the impact of sparsity . In this article , we go further to consider the global landscape of general sparse linear neural networks . We need to emphasize that dense deep linear networks already satisfy that every local minimum is a global minimum under mild conditions ( Kawaguchi , 2016 ; Lu & Kawaguchi , 2017 ) , but findings are different and complicated for sparse linear network . The goal of this paper is to study the relation between sparsity and local minima with the following contributions : • First , we point out that every local minimum is a global minimum in scalar target case with any depths , any widths and any sparse structure . Besides , we also briefly show that similar results hold for non-overlapping filters and orthogonal data feature when sparsity only occurs in the first layer . • Second , we find out that sparse connections would already give sub-optimal local minima in general non-scalar case through analytic and numerical examples built on convergence analyze . The local-min may be produced from two situations : a sub-sparse linear network which owes its minimum as a local-min of the original sparse network ; a rank-deficient solution between different data features due to sparse connections , while both cases verify the fact that sparsity cuts out the decreasing path in original fully-connected networks . Overall , we hope our work contributes to a better understanding of the landscape of sparsity network on simple neural networks , and provide insights for future research . The remainder of our paper is organized as follows . In Section 2 , we derive the positive findings of shallow sparse linear networks , providing similar landscape as dense linear networks . In Section 3 , we give several examples to show the existence of bad local-min for non-scalar case . In section 4 , we briefly generalize the results from shallow to deep sparse linear networks . Some proofs are in Appendix . 1.1 RELATED WORK . There is a rapidly increasing literature on analyzing the loss surface of neural network objectives , surveying all of which is well outside our scope . Thus , we only briefly survey the works most related to ours . Local minima is Global . The landscape of a linear network date back to Baldi & Hornik ( 1989 ) , proving that shallow linear neural networks do not suffer from bad local minima . Kawaguchi ( 2016 ) generalized same results to deep linear neural networks , and subsequent several works ( Arora et al. , 2018 ; Du & Hu , 2019 ; Eftekhari , 2020 ) give direct algorithm-type convergence based on this benign property , though algorithm analysis is beyond the scope of this paper . However , situations are quite complicated with nonlinear activations . Multiple works ( Ge et al. , 2017 ; Safran & Shamir , 2018 ; Yun et al. , 2018 ) show that spurious local minima can happen even in two-layer network with population or empirical loss , some are specific to two-layer and difficult to generalize to general multilayer cases . Another line of works ( Arora et al. , 2018 ; Allen-Zhu et al. , 2018 ; Du & Hu , 2019 ; Du et al. , 2018 ; Li et al. , 2018 ; Mei et al. , 2018 ) understands the landscape of neural network in an overparameterized setting , discovering benign landscape with or without gradient method . Since modern sparse networks reserve few parameters compared to overparameterization , we still seek a fundamental view of sparsity in contrast . Our standpoint is that spurious local minima can happen when applied with specific sparsity even in linear networks . Sparse networks . Sparse networks ( Han et al. , 2015b ; a ; Zhu & Gupta , 2017 ; Frankle & Carbin , 2018 ; Liu et al. , 2018 ) have a long history , but appears heavily on the experiments , and mainly related to network pruning , which has practical importance for reducing model parameter count and deploying diverse devices . However , training sparse networks ( from scratch ) suffers great difficulty . Frankle & Carbin ( 2018 ) recommend reusing the sparsity pattern found through pruning and train a sparse network from the same initialization as the original training ( ‘ lottery ’ ) to obtain comparable performance and avoid bad solution . Besides , for fixed sparsity patterns , Evci et al . ( 2019 ) attempt to find a decreasing objective path from ‘ bad ’ solutions to the ‘ good ’ ones in the sparse subspace but fail , showing bad local minima can be produced by pruning , while we give more direct view of simple examples to verify this . Moreover , several recent works also give abundant methods ( Molchanov et al. , 2017 ; Louizos et al. , 2017b ; Lee et al. , 2018 ; Carreira-Perpinán & Idelbayev , 2018 ) for choosing weights or sparse network structure while achieving similar performance . In theoretical view , Malach et al . ( 2020 ) prove that a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network , providing guarantee for ‘ good ’ sparse networks . Some works analyze convolutional network ( Shalev-Shwartz et al. , 2017 ; Du et al. , 2018 ) as a specific sparse structure . Brutzkus & Globerson ( 2017 ) analyze non-overlapping and overlapping structure as we do , but with weight sharing to simulate CNN-type structure , and under teacher-student setting with population risk . We do not follow CNN-type network but in general sparse networks , though still linear , to conclude straightforward results . 2 LANDSCAPE OF SHALLOW SPARSE LINEAR NETWORKS . 2.1 PRELIMINARIES AND NOTATION . We use bold-faced letters ( e.g. , w and a ) to denote vectors , capital letters ( e.g. , W = [ wij ] and A = [ aij ] ) for matrices . Let PX be the orthogonal projection to the column space of the matrix X , and λi ( H ) is the i-th smallest eigenvalue of a real symmetric matrix H . We consider the training samples and their outputs as { ( xi , yi ) } ni=1 ⊂ Rdx × Rdy , which may come from unknown distribution D. We form the data matrices X = [ x1 , . . . , xn ] T ∈ Rn×dx and Y = [ y1 , . . . , yn ] T ∈ Rn×dy , respectively . In our analysis in Sections 2 and 3 , we consider a two-layer ( sparse ) linear neural network with squared loss : min W , A L ( W , A ) : = 1 2 ‖Y −XWA‖2F , ( 1 ) where the first layer weight matrix W = [ w1 , . . . , wd ] ∈ Rdx×d , and the second layer weight matrix A = [ a1 , . . . , ad ] T ∈ Rd×dy . After weights pruning or sparsity constraint , many weights parameters become zero and would not be updated during retraining . We adopt Sj : = { k : wkj = 0 } as pruned dimensions in the j-th column ofW , and−Sj : = Scj = [ dx ] \Sj , where [ d ] : = { 1 , . . . , d } . In addition , wj , S denotes the sub-vector of wj choosing the positions in S , XS the sub-matrix of X choosing the column indices in S. We let pj = dx − |Sj | , where |S| is the cardinality of the set S. Then wj , −Sj ∈ Rpj is the remaining j-th column in first layer weight which leaves out pruned dimension set Sj . Similarly , X−Sj ∈ Rn×pj means the remaining data matrix connected to j-th node in the first layer . Finally , for simplicity , we denote X−j = X−Sj , w−j = wj , −Sj , and ( ̃· ) as the pruned layer weight with several zero elements not updated all along , if no ambiguity . Before we begin , a small note on the sparse structure we concern : there may have unnecessary connections and nodes , such as a node with zero out-degree which can be retrieved and excluded from the final layer to the first layer , and other cases are showing in Appendix C. Thus we do not consider them in the subsequent proof and assume each data dimension has valid output connection , i.e. , ∩dj=1Sj = ∅ . 2.2 SCALAR CASE . In the scalar case , assume dy = 1 . We then simplify A = ( a1 , . . . , ad ) T . When pruning any weight ai in the second layer , the output of the i-th node in the first layer contribute zero to final output . Hence wi can also be pruned . Without loss of generality , we assume second layer parameters are not pruned . After pruning several parameters , the original problem becomes min w−i , ai L ( W̃ , A ) : = 1 2 ∥∥∥∥∥∥∥Y − ( X−1 , . . . , X−i , . . . , X−d ) a1w−1 ... adw−d ∥∥∥∥∥∥∥ 2 F . ( 2 ) Theorem 1 For a two-layer linear neural network with scalar output and any sparse structure , every local minimum is a global minimum . Proof : From Eq . ( 2 ) , if a local minimizer satisfies ai = 0 for some 1 ≤ i ≤ d , then based on the second order condition for a local minima , we have ∂2L ∂a2i ∂2L ∂ai∂wT−i ∂L ∂w−i∂ai ∂L ∂w−i∂wT−i 0 , ( 3 ) which implies that wT−iX−iXT−iw−i − ( Y −∑di=1X−iw−iai ) T X−i −XT−i ( Y − ∑d i=1X−iw−iai ) 0 0 . ( 4 ) Then XT−i ( Y − ∑d i=1X−iw−iai ) = 0 , which is the global minimizer condition of w−iai . Otherwise , ai 6= 0 , then from the first-order condition for a local minima , ∂L ∂w−i = aiX T −i ( Y − d∑ i=1 X−iw−iai ) = 0 , showing that XT−i ( Y − ∑d i=1X−iw−iai ) = 0 , which also gives the global minimizer condition of w−iai . Hence every local minimum is a global minimum .

Summary: This paper studies the optimization landscape of (deep) sparse linear networks. The study of sparse neural networks is well motivated: on the one hand, there is a lot of experimental evidence that the loss of the trained network does not decrease much after removing a large subset of the connections; on the other hand, there is little theoretical evidence of what makes this behaviour possible (or how to provably construct sparse networks from dense ones). As a result, an investigation of the optimization landscape of sparse networks, even in the simple case of a linear activation function, is timely and interesting to the ICLR community, since it can potentially shed light on the questions above.