LAYER SPARSITY IN NEURAL NETWORKS

Sparsity has become popular in machine learning , because it can save computational resources , facilitate interpretations , and prevent overfitting . In this paper , we discuss sparsity in the framework of neural networks . In particular , we formulate a new notion of sparsity that concerns the networks ’ layers and , therefore , aligns particularly well with the current trend toward deep networks . We call this notion layer sparsity . We then introduce corresponding regularization and refitting schemes that can complement standard deep-learning pipelines to generate more compact and accurate networks . 1 INTRODUCTION . The number of layers and the number of nodes in each layer are arguably among the most fundamental parameters of neural networks . But specifying these parameters can be challenging : deep and wide networks , that is , networks with many layers and nodes , can describe data in astounding detail , but they are also prone to overfitting and require large memory , CPU , energy , and so forth . The resource requirements can be particularly problematic for real-time applications or applications on fitness trackers and other wearables , whose popularity has surged in recent years . A promising approach to meet these challenges is to fit networks sizes adaptively , that is , to allow for many layers and nodes in principle , but to ensure that the final network is “ simple ” in that it has a small number of connections , nodes , or layers ( Changpinyo et al. , 2017 ; Han et al. , 2016 ; Kim et al. , 2016 ; Liu et al. , 2015 ; Wen et al. , 2016 ) . Popular ways to fit such simple and compact networks include successively augmenting small networks ( Ash , 1989 ; Bello , 1992 ) , pruning large networks ( Simonyan & Zisserman , 2015 ) , or explicit sparsity-inducing regularization of the weight matrices , which we focus on here . An example is the ` 1-norm , which can reduce the number of connections . Another example is the ` 1-norm grouped over the rows of the weight matrices , which can reduce the number of nodes . It has been shown that such regularizers can indeed produce networks that are both accurate and yet have a small number of nodes and connections either in the first layer ( Feng & Simon , 2017 ) or overall ( Alvarez & Salzmann , 2016 ; Liu et al. , 2015 ; Scardapane et al. , 2017 ) . Such sparsity-inducing regularizers also have a long-standing tradition and thorough theoretical underpinning in statistics ( Hastie et al. , 2015 ) . But while sparsity on the level of connections and nodes has been studied in some detail , sparsity on the level of layers is much less understood . This lack of understanding contrasts the current trend to deep network architectures , which is supported by state-of-the-art performances of deep networks ( LeCun et al. , 2015 ; Schmidhuber , 2015 ) , recent approximation theory for ReLU activation networks ( Liang & Srikant , 2016 ; Telgarsky , 2016 ; Yarotsky , 2017 ) , and recent statistical theory ( Golowich et al. , 2017 ; Kohler & Langer , 2019 ; Taheri et al. , 2020 ) . Hence , a better understanding of sparsity on the level of layers seems to be in order . Therefore , we discuss in this paper sparsity with a special emphasis on the networks ’ layers . Our key observation is that for typical activation functions such as ReLU , a layer can be removed if all its parameter values are non-negative . We leverage this observation in the development of a new regularizer that specifically targets sparsity on the level of layers , and we show that this regularizer can lead to more compact and more accurate networks . Our three main contributions are : 1 . We introduce a new notion of sparsity that we call layer sparsity . 2 . We introduce a corresponding regularizer that can reduce network sizes . 3 . We introduce an additional refitting step that can further improve prediction accuracies . In Section 2 , we specify our framework , discuss different notions of sparsity , and introduce our refitting scheme . In Section 3 , we establish a numerical proof of concept . In Section 4 , we conclude with a discussion . 2 SPARSITY IN NEURAL NETWORKS . We first state our framework , then discuss different notions of sparsity , and finally introduce a refitting scheme . 2.1 MATHEMATICAL FRAMEWORK . To fix ideas , we first consider fully-connected neural networks that model data according to yi = f 1 [ W 1f2 [ ... f l [ W lxi ] ] ] + ui , ( 1 ) where i ∈ { 1 , . . . , n } indexes the n different samples , yi ∈ R is the output , xi ∈ Rd is the corresponding input with d the input dimension , l is the number of layers , W j ∈ Rpj×pj+1 for j ∈ { 1 , . . . , l } are the weight matrices with p1 = 1 and pl+1 = d , f j : Rpj → Rpj for j ∈ { 1 , . . . , l } are the activation functions , and ui ∈ R is the random noise . Extensions beyond fully-connected networks are straightforward—see Section 2.5 . We summarize the parameters in W : = ( W 1 , . . . , W l ) ∈ V : = { V = ( V 1 , . . . , V l ) : V j ∈ Rpj×pj+1 } , and we write for ease of notation fV [ xi ] : = f 1 [ V 1f2 [ ... f l [ V lxi ] ] ] ( 2 ) for V ∈ V . Neural networks are usually fitted based on regularized estimators in Lagrange Ŵ ∈ argmin V ∈V { DataFit [ y1 , . . . , yn , x1 , . . . , xn ] + h [ V ] } ( 3 ) or constraint form Ŵ ∈ argmin V ∈V h [ V ] ≤1 { DataFit [ y1 , . . . , yn , x1 , . . . , xn ] } , ( 4 ) where DataFit : Rn × Rn×d is a data-fitting function such as least-squares ∑n i=1 ( yi − fV [ xi ] ) 2 , and h : V → [ 0 , ∞ ) is a regularizer such as the elementwise ` 1-norm ∑ j , k , l | ( V j ) kl| . We are particularly interested in regularizers that induce sparsity . 2.2 STANDARD NOTIONS OF SPARSITY . We first state two regularizers that are known in deep learning and the corresponding notions of sparsity . Connection sparsity Consider the vanilla ` 1-regularizer hC [ V ] : = l∑ j=1 ( rC ) j |||V j |||1 : = l∑ j=1 ( rC ) j pj∑ v=1 pj+1∑ w=1 | ( V j ) vw| , where rC ∈ [ 0 , ∞ ) l is a vector of tuning parameters . This regularizer is the deep learning equivalent of the lasso regularizer in linear regression ( Tibshirani , 1996 ) and has received considerable attention no sparsity connection sparsity node sparsity layer sparsity combined sparsity recently ( Barron & Klusowski , 2018 ; 2019 ; Kim et al. , 2016 ) . The regularizer acts on each individual connection , pruning a full network ( first network from the left in Figure 1 ) to a more sparsely connected network ( second network in Figure 1 ) . We , therefore , propose to speak of connection sparsity . Node sparsity Consider a grouped version of the above regularizer hN [ V ] : = l∑ j=1 ( rN ) j |||V j |||2,1 : = l∑ j=1 ( rN ) j pj∑ v=1 √√√√pj+1∑ w=1 | ( V j ) vw|2 , where rN ∈ [ 0 , ∞ ) l is again a vector of tuning parameters . This regularizer is the deep learning equivalent of the group lasso regularizer in linear regression ( Bakin , 1999 ) and has received some attention recently ( Alvarez & Salzmann , 2016 ; Feng & Simon , 2017 ; Scardapane et al. , 2017 ) . The regularizer acts on all connections that go into a node simultaneously , rendering entire nodes inactive ( third network in Figure 1 ) . We , therefore , propose to speak of node sparsity . 2.3 LAYER SPARSITY . We now complement the two existing regularizers and notions of sparsity with a new , third notion . Layer sparsity Consider the regularizer hL [ V ] : = l−1∑ j=1 ( rL ) j |||V j |||2 , + : = l−1∑ j=1 ( rL ) j √√√√ pj∑ v=1 pj+1∑ w=1 ( neg [ ( V j ) vw ] ) 2 , ( 5 ) where rL ∈ [ 0 , ∞ ) l−1 is a vector of tuning parameters , and neg [ a ] : = min { a , 0 } is the negative part of a real value a ∈ R. This regularizers does not have an equivalent in linear regression , and it is also new in deep learning . We argue that the regularizer can give rise to a new type of sparsity . The regularizer can be disentangled along the layers according to hL [ V ] = l−1∑ j=1 ( rL ) jh L , j [ V j ] with hL , j [ V j ] : = √√√√ pj∑ v=1 pj+1∑ w=1 ( neg [ ( V j ) vw ] ) 2 for j ∈ { 1 , . . . , l − 1 } . We then focus on an individual inner layer , that is , a layer that corresponds to an index j ∈ { 2 , . . . , l− 1 } . To fix ideas , we consider the popular ReLU activation ( Glorot et al. , 2011 ; Hahnloser , 1998 ; Hahnloser et al. , 2000 ; Salinas & Abbott , 1996 ) ; in other words , we consider ( f j ) q [ t ] : = fReLU [ t ] : = max { t , 0 } for all j ∈ { 2 , . . . , l } , q ∈ { 1 , . . . , pj } , and t ∈ R ( the activation of the output layer can be arbitrary ) . It is now easy to show that the regularizer hL indeed induces sparsity on the level of layers . Theorem 1 ( Layer Sparsity ) . Consider j ∈ { 2 , . . . , l − 1 } , and define a merged weight matrix as V j−1 , j : = V j−1V j ∈ Rpj−1×pj+1 . It holds that hL , j [ V j ] = 0 ⇒ f j−1 [ V j−1f j [ V jz ] ] = f j−1 [ V j−1 , jz ] for all z ∈ [ 0 , ∞ ) pj+1 . Proof of Theorem 1 . If hL , j [ V j ] = 0 , then ( V j ) qm ≥ 0 for all q ∈ { 1 , . . . , pj } , m ∈ { 1 , . . . , pj+1 } . Hence , it holds for all q ∈ { 1 , . . . , pj } that ( V jz ) q ≥ 0 and , therefore , that f j [ V jz ] = V jz . The theorem follows then by the fact that V j−1 , j = V j−1V j . A key property of the ReLU function is positive homogeneity . That positive homogeneity can allow for moving weights between layers had been observed in Barron & Klusowski ( 2018 ) ; here , we use the positive homogeneity to merge layers . The idea is as follows : hL , j [ V j ] = 0 means in view of the stated theorem that we can redefine the network of depth l as a network of depth l − 1 by removing the function f j , replacing the weights V j−1 by V j−1 , j , and then removing the jth layer altogether . Theorem 1 can be applied sequentially to neighboring layers ; hence , the regularization can merge not only one but many layers into one . In conclusion , our new regularizer hL acts on all nodes and connections of each layer simultaneously , rendering entire layers inactive ( fourth network in Figure 1 ) . We , therefore , propose to speak of layer sparsity . The concept of layer sparsity and Theorem 1 in particular do not hinge on the exact choice of the regularizer in ( 5 ) : one can take any function hL that can be disentangled along the layers as described and that ensures the fact that hL , j [ V j ] = 0 implies mink , l ( V j ) kl ≥ 0 . We illustrate layer sparsity with two examples . Example 1 ( Identity Activation ) . We first highlight the meaning of layer sparsity in a simplistic setting that does not rely on Theorem 1 . We consider identity activation , that is , ( f j ) q [ t ] = t for all j ∈ { 2 , . . . , l } , q ∈ { 1 , . . . , pj } , and t ∈ R. The networks in ( 2 ) can then be written as fV [ xi ] = f 1 [ V 1 · · ·V lxi ] . In other words , the initial l-layer network can be compressed into a one-layer network with activation function f1 and parameter matrix V 1 · · ·V l ∈ R1×d . This setting with identity activation is , of course , purely academic , but it motivates an important question : can parts of networks be compressed similarly in the case of ReLU ? Theorem 1 gives an answer to this question : if hL , j [ V j ] = 0 , then the jth and ( j − 1 ) th layers can be combined . In the extreme case hL,2 [ V 2 ] = · · · = hL , l [ V l ] = 0 and non-negative input , the network can be condensed into a one-layer network just as in the linear case . In this sense , one can understand our layer regularizer is as a measure for the networks ’ “ distance to linearity. ” We detail this further in the following example . Example 2 ( ReLU Activation ) . We now illustrate how layer sparsity compresses and , therefore , simplifies networks in the case of ReLU activation . We fix an initial network fN parameterized by N ∈ V . We identify the active layers of the network by S ≡ S [ N ] : = { j ∈ { 2 , . . . , l − 1 } : hL , j [ N j ] 6= 0 } ∪ { 1 , l } . ( 6 ) Thus , S and { 1 , . . . , l } \ S contain the indexes of the relevant and irrelevant layers , respectively . ( We always consider the input and output layers as active . ) The level of sparsity , that is , the number of active layers , is s : = |S| ≤ l. Observe first that the theorem ’ s restriction to z ’ s that have non-negative elements makes sense : by the definition of fReLU , the outputs of every ReLU layer are non-negative . We now denote the indexes in S in an orderly fashion : j1 , . . . , js ∈ S such that j1 < · · · < js = l. We then define scaled versions of the corresponding merged matrices : if ji−1 ∈ S or ji ∈ { 1 , l } , we do the “ trivial merge ” M ji : = N ji ∈ Rpji×pji+1 ; otherwise , we do the “ non-trivial merge ” M ji : = N ji−1+1 · · ·N ji ∈ Rpji−1+1×pji+1 . In other words , we merge all irrelevant layers between the ji−1th and jith layers into the jith layer . We can then compress the data-generating model in ( 1 ) into yi = f j1 [ M j1f j2 [ ... f js [ M jsxi ] ] ] + ui with M : = ( M j1 , . . . , M js ) ∈ VS : = { V = ( V 1 , . . . , V s ) : V i ∈ Rpji−1+1×pji+1 } . Formulated differently , we can condense the original network according to fN [ xi ] = fM [ xi ] = f j1 [ M j1f j2 [ ... f js [ M jsxi ] ] ] , that is , we can formulate the initial ReLU activation network with l layers as a new ReLU activation network with s layers . The new network is still a ReLU activation network but has a smaller number of layers if s < l and , consequently , a smaller number of parameters in total : the total number of parameters in the initial network is ∑l j=1 ( pj × pj+1 ) , while the total number of parameters in the transformed network is only ∑s i=1 ( pji−1+1 × pji+1 ) . Our concept for regularizing layers is substantially different from existing ones : our layer-wise regularizer induces weights to be non-negative , whereas existing layer-wise regularizers induce weights to be zero ( Wen et al. , 2016 , Section 3.3 ) . The two main advantages of our approach are that it ( i ) does not require shortcuts to avoid trivial networks and ( ii ) does not implicitly enforce connection or node sparsity . We thus argue that our layer sparsity is a much more natural and appropriate way to capture and regularize network depths . Layer sparsity more closely relates to ResNets ( He et al. , 2016 ) . The recent popularity of ResNets is motivated by two observations : 1 . Solvers seem to struggle with finding good minima of deep networks ; even training accuracies can deteriorate when increasing the number of layers . 2 . Allowing for linear mappings that short-circuit parts of the network seem to help solvers in finding better minima . From our viewpoint here , one can argue that ResNets use these linear mappings to regulate network depths adaptively and , therefore , are related to layer sparsity . But importantly , ResNets are even more complex than the networks they are based on , while our notion simplifies networks . Since , as one can verify again readily , all three regularizers are convex , any combination of them is also convex . Such combinations can be used to obtain networks that are sparse in two or all three aspects ( last network in Figure 1 ) . In this sense , the different notions of sparsity are not competing but rather complementing each other .

Summary: The paper proposes a regularizer enforcing a novel form of sparsity that authors call "layer sparsity". Under certain conditions on layer weights, two consecutive layers in a deep neural network (with certain nonlinear activation functions) can be represented exactly as a single layer. The authors proposed a regularizer that can lead to such layer collapse thus resulting in shallower and more compact models.