On the Landscape of Sparse Linear Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved remarkable empirical successes in the domains of computer vision , speech recognition , and natural language processing , sparking great interests in the theory behind their architectures and training . However , DNNs are often found to be highly overparameterized , making them computationally expensive with large amounts of memory and computational power . For example , it may take up to weeks on a modern multi-GPU server for large datasets such as ImageNet ( Deng et al. , 2009 ) . Hence , DNNs are often unsuitable for smaller devices like embedded electronics , and there is a pressing demand for techniques to optimize models with reduced model size , faster inference and lower power consumption . Sparse networks , that is , neural networks in which a large subset of the model parameters are zero , have emerged as one of the leading approaches for reducing model parameter count . It has been shown empirically that deep neural networks can achieve state-of-the-art results under high levels of sparsity ( Han et al. , 2015b ; Gale et al. , 2019 ; Louizos et al. , 2017a ) . Modern sparse networks are mainly obtained from network pruning ( Zhu & Gupta , 2017 ; Lee et al. , 2018 ; Liu et al. , 2018 ; Frankle & Carbin , 2018 ) , which has been the subject of a great deal of work in recent years . However , training a sparse network with fixed sparsity patterns is difficult ( Evci et al. , 2019 ) and few theoretical understanding of general sparse networks has been provided . Previous work has already analyze deep neural networks , showing that the non-convexity of the associated loss functions may cause complicated and strange optimization landscapes . However , the property of general sparse networks is poorly understood . Saxe et al . ( 2013 ) empirically showed that the optimization of deep linear models exhibits similar properties as deep nonlinear models , and for theoretical development , it is natural to begin with linear models before studying nonlinear models ( Baldi & Lu , 2012 ) . In addition , several works ( Sun et al. , 2020 ) have show bad minimum exists with nonlinear activation . Hence , it is natural to begin with linear activation to understand the impact of sparsity . In this article , we go further to consider the global landscape of general sparse linear neural networks . We need to emphasize that dense deep linear networks already satisfy that every local minimum is a global minimum under mild conditions ( Kawaguchi , 2016 ; Lu & Kawaguchi , 2017 ) , but findings are different and complicated for sparse linear network . The goal of this paper is to study the relation between sparsity and local minima with the following contributions : • First , we point out that every local minimum is a global minimum in scalar target case with any depths , any widths and any sparse structure . Besides , we also briefly show that similar results hold for non-overlapping filters and orthogonal data feature when sparsity only occurs in the first layer . • Second , we find out that sparse connections would already give sub-optimal local minima in general non-scalar case through analytic and numerical examples built on convergence analyze . The local-min may be produced from two situations : a sub-sparse linear network which owes its minimum as a local-min of the original sparse network ; a rank-deficient solution between different data features due to sparse connections , while both cases verify the fact that sparsity cuts out the decreasing path in original fully-connected networks . Overall , we hope our work contributes to a better understanding of the landscape of sparsity network on simple neural networks , and provide insights for future research . The remainder of our paper is organized as follows . In Section 2 , we derive the positive findings of shallow sparse linear networks , providing similar landscape as dense linear networks . In Section 3 , we give several examples to show the existence of bad local-min for non-scalar case . In section 4 , we briefly generalize the results from shallow to deep sparse linear networks . Some proofs are in Appendix . 1.1 RELATED WORK . There is a rapidly increasing literature on analyzing the loss surface of neural network objectives , surveying all of which is well outside our scope . Thus , we only briefly survey the works most related to ours . Local minima is Global . The landscape of a linear network date back to Baldi & Hornik ( 1989 ) , proving that shallow linear neural networks do not suffer from bad local minima . Kawaguchi ( 2016 ) generalized same results to deep linear neural networks , and subsequent several works ( Arora et al. , 2018 ; Du & Hu , 2019 ; Eftekhari , 2020 ) give direct algorithm-type convergence based on this benign property , though algorithm analysis is beyond the scope of this paper . However , situations are quite complicated with nonlinear activations . Multiple works ( Ge et al. , 2017 ; Safran & Shamir , 2018 ; Yun et al. , 2018 ) show that spurious local minima can happen even in two-layer network with population or empirical loss , some are specific to two-layer and difficult to generalize to general multilayer cases . Another line of works ( Arora et al. , 2018 ; Allen-Zhu et al. , 2018 ; Du & Hu , 2019 ; Du et al. , 2018 ; Li et al. , 2018 ; Mei et al. , 2018 ) understands the landscape of neural network in an overparameterized setting , discovering benign landscape with or without gradient method . Since modern sparse networks reserve few parameters compared to overparameterization , we still seek a fundamental view of sparsity in contrast . Our standpoint is that spurious local minima can happen when applied with specific sparsity even in linear networks . Sparse networks . Sparse networks ( Han et al. , 2015b ; a ; Zhu & Gupta , 2017 ; Frankle & Carbin , 2018 ; Liu et al. , 2018 ) have a long history , but appears heavily on the experiments , and mainly related to network pruning , which has practical importance for reducing model parameter count and deploying diverse devices . However , training sparse networks ( from scratch ) suffers great difficulty . Frankle & Carbin ( 2018 ) recommend reusing the sparsity pattern found through pruning and train a sparse network from the same initialization as the original training ( ‘ lottery ’ ) to obtain comparable performance and avoid bad solution . Besides , for fixed sparsity patterns , Evci et al . ( 2019 ) attempt to find a decreasing objective path from ‘ bad ’ solutions to the ‘ good ’ ones in the sparse subspace but fail , showing bad local minima can be produced by pruning , while we give more direct view of simple examples to verify this . Moreover , several recent works also give abundant methods ( Molchanov et al. , 2017 ; Louizos et al. , 2017b ; Lee et al. , 2018 ; Carreira-Perpinán & Idelbayev , 2018 ) for choosing weights or sparse network structure while achieving similar performance . In theoretical view , Malach et al . ( 2020 ) prove that a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network , providing guarantee for ‘ good ’ sparse networks . Some works analyze convolutional network ( Shalev-Shwartz et al. , 2017 ; Du et al. , 2018 ) as a specific sparse structure . Brutzkus & Globerson ( 2017 ) analyze non-overlapping and overlapping structure as we do , but with weight sharing to simulate CNN-type structure , and under teacher-student setting with population risk . We do not follow CNN-type network but in general sparse networks , though still linear , to conclude straightforward results . 2 LANDSCAPE OF SHALLOW SPARSE LINEAR NETWORKS . 2.1 PRELIMINARIES AND NOTATION . We use bold-faced letters ( e.g. , w and a ) to denote vectors , capital letters ( e.g. , W = [ wij ] and A = [ aij ] ) for matrices . Let PX be the orthogonal projection to the column space of the matrix X , and λi ( H ) is the i-th smallest eigenvalue of a real symmetric matrix H . We consider the training samples and their outputs as { ( xi , yi ) } ni=1 ⊂ Rdx × Rdy , which may come from unknown distribution D. We form the data matrices X = [ x1 , . . . , xn ] T ∈ Rn×dx and Y = [ y1 , . . . , yn ] T ∈ Rn×dy , respectively . In our analysis in Sections 2 and 3 , we consider a two-layer ( sparse ) linear neural network with squared loss : min W , A L ( W , A ) : = 1 2 ‖Y −XWA‖2F , ( 1 ) where the first layer weight matrix W = [ w1 , . . . , wd ] ∈ Rdx×d , and the second layer weight matrix A = [ a1 , . . . , ad ] T ∈ Rd×dy . After weights pruning or sparsity constraint , many weights parameters become zero and would not be updated during retraining . We adopt Sj : = { k : wkj = 0 } as pruned dimensions in the j-th column ofW , and−Sj : = Scj = [ dx ] \Sj , where [ d ] : = { 1 , . . . , d } . In addition , wj , S denotes the sub-vector of wj choosing the positions in S , XS the sub-matrix of X choosing the column indices in S. We let pj = dx − |Sj | , where |S| is the cardinality of the set S. Then wj , −Sj ∈ Rpj is the remaining j-th column in first layer weight which leaves out pruned dimension set Sj . Similarly , X−Sj ∈ Rn×pj means the remaining data matrix connected to j-th node in the first layer . Finally , for simplicity , we denote X−j = X−Sj , w−j = wj , −Sj , and ( ̃· ) as the pruned layer weight with several zero elements not updated all along , if no ambiguity . Before we begin , a small note on the sparse structure we concern : there may have unnecessary connections and nodes , such as a node with zero out-degree which can be retrieved and excluded from the final layer to the first layer , and other cases are showing in Appendix C. Thus we do not consider them in the subsequent proof and assume each data dimension has valid output connection , i.e. , ∩dj=1Sj = ∅ . 2.2 SCALAR CASE . In the scalar case , assume dy = 1 . We then simplify A = ( a1 , . . . , ad ) T . When pruning any weight ai in the second layer , the output of the i-th node in the first layer contribute zero to final output . Hence wi can also be pruned . Without loss of generality , we assume second layer parameters are not pruned . After pruning several parameters , the original problem becomes min w−i , ai L ( W̃ , A ) : = 1 2 ∥∥∥∥∥∥∥Y − ( X−1 , . . . , X−i , . . . , X−d ) a1w−1 ... adw−d ∥∥∥∥∥∥∥ 2 F . ( 2 ) Theorem 1 For a two-layer linear neural network with scalar output and any sparse structure , every local minimum is a global minimum . Proof : From Eq . ( 2 ) , if a local minimizer satisfies ai = 0 for some 1 ≤ i ≤ d , then based on the second order condition for a local minima , we have ∂2L ∂a2i ∂2L ∂ai∂wT−i ∂L ∂w−i∂ai ∂L ∂w−i∂wT−i 0 , ( 3 ) which implies that wT−iX−iXT−iw−i − ( Y −∑di=1X−iw−iai ) T X−i −XT−i ( Y − ∑d i=1X−iw−iai ) 0 0 . ( 4 ) Then XT−i ( Y − ∑d i=1X−iw−iai ) = 0 , which is the global minimizer condition of w−iai . Otherwise , ai 6= 0 , then from the first-order condition for a local minima , ∂L ∂w−i = aiX T −i ( Y − d∑ i=1 X−iw−iai ) = 0 , showing that XT−i ( Y − ∑d i=1X−iw−iai ) = 0 , which also gives the global minimizer condition of w−iai . Hence every local minimum is a global minimum .

Summary: This paper studies the loss landscapes of sparse linear networks. It proves that under squared loss, (1) spurious local minimum does not exist when the output dimension is one, or with separated first layer and orthogonal training data; and (2) for two-layer sparse linear networks, the good property in (1) does not exist anymore when the conditions are violated. The authors also report experimental results to show that two-layer sparse linear networks with two hidden neurons have spurious local minima.