Connection- and Node-Sparse Deep Learning: Statistical Guarantees

Neural networks are becoming increasingly popular in applications , but a comprehensive mathematical understanding of their potentials and limitations is still missing . In this paper , we study the prediction accuracies of neural networks from a statistical point of view . In particular , we establish statistical guarantees for deep learning with different types of sparsity-inducing regularization . Our bounds feature a mild dependence on network widths and depths , and , therefore , support the current trend toward wide and deep networks . The tools that we use in our derivations are uncommon in deep learning and , hence , might be of additional interest . 1 INTRODUCTION . Sparsity reduces network complexities and , consequently , lowers the demands on memory and computation , reduces overfitting , and improves interpretability ( Changpinyo et al. , 2017 ; Han et al. , 2016 ; Kim et al. , 2016 ; Liu et al. , 2015 ; Wen et al. , 2016 ) . Three common notions of sparsity are connection sparsity , which means that there is only a small number of nonzero connections between nodes , node sparsity , which means that there is only a small number of active nodes ( Alvarez & Salzmann , 2016 ; Changpinyo et al. , 2017 ; Feng & Simon , 2017 ; Kim et al. , 2016 ; Lee et al. , 2008 ; Liu et al. , 2015 ; Nie et al. , 2015 ; Scardapane et al. , 2017 ; Wen et al. , 2016 ) , and layer sparsity , which means that there is only a small number of active layers ( Hebiri & Lederer , 2020 ) . Approaches to achieving sparsity include augmenting small networks ( Ash , 1989 ; Bello , 1992 ) , pruning large networks ( Simonyan & Zisserman , 2015 ; Han et al. , 2016 ) , constraint estimation ( Ledent et al. , 2019 ; Neyshabur et al. , 2015 ; Schmidt-Hieber , 2020 ) , and statistical regularization ( Taheri et al. , 2020 ) . The many empirical observations of the benefits of sparsity have sparked interest in mathematical support in the form of statistical theories . But such theories are still scarce and , in any case , have severe limitations . For example , statistical guarantees for deep learning with connection-sparse regularization have been established in Taheri et al . ( 2020 ) , but they do not cover node sparsity , which , in view of the removal of entire nodes , has become especially popular . Moreover , their estimator involves an additional parameter , their theory is limited to a single output node , and their results have a suboptimal dependence on the input vectors . Statistical guarantees for constraint estimation over connection- and node-sparse networks follow from combining results in Neyshabur et al . ( 2015 ) and Bartlett & Mendelson ( 2002 ) . But for computational and practical reasons , regularized estimation is typically preferred over constraint estimation in deep learning as well as in machine learning at large ( Hastie et al. , 2015 ) . Moreover , their theory is limited to a single output node and ReLU activation , scales exponentially in the number of layers , and requires bounded loss functions . Statistical prediction guarantees for constraint estimation over connection-sparse networks have been derived in Schmidt-Hieber ( 2020 ) , but their theory is limited to a single output node and ReLU activation and assumes bounded weights . In short , the existing statistical theory for deep learning with connection and node sparsity is still deficient . The goal of this paper is to provide an improved theory for sparse deep learning . We focus on regression-type settings with layered , feedforward neural networks . The estimators under consideration consist of a standard least-squares estimator with additional regularizers that induce connection or node sparsity . We then derive our guarantees by using techniques from high-dimensional statistics ( Dalalyan et al. , 2017 ) and empirical process theory ( van de Geer , 2000 ) . In the case of subgaussian noise , we find the rates√ l ( log [ mnp ] ) 3 n and √ mlp ( log [ mnp ] ) 3 n for the connection-sparse and node-sparse estimators , respectively , where l is the number of hidden layers , m the number of output nodes , n the number of samples , p the total number of parameters , and p the maximal width of the network . The rates suggest that sparsity-inducing approaches can provide accurate prediction even in very wide ( with connection sparsity ) and very deep ( with either type of sparsity ) networks while , at the same time , ensuring low network complexities . These findings underpin the current trend toward sparse but wide and especially deep networks from a statistical perspective . Outline of the paper Section 2 recapitulates the notions of connection and node sparsity and introduces the corresponding deep learning framework and estimators . Section 3 confirms the empirically-observed accuracies of connection- and node-sparse estimation in theory . Section 4 summarizes the key features and limitations of our work . The Appendix contains all proofs . 2 CONNECTION- AND NODE-SPARSE DEEP LEARNING . We consider data ( y1 , x1 ) , . . . , ( yn , xn ) ∈ Rm × Rd that are related via yi = g∗ [ xi ] + ui for i ∈ { 1 , . . . , n } ( 1 ) for an unknown data-generating function g∗ : Rd → Rm and unknown , random noise u1 , . . . , un ∈ Rm . We allow all aspects , namely yi , g∗ , xi , and ui , to be unbounded . Our goal is to model the data-generating function with a feedforward neural network of the form gΘ [ x ] ··= Θlf l [ Θl−1 · · ·f1 [ Θ0x ] ] for x ∈ Rd ( 2 ) indexed by the parameter space M ··= { Θ = ( Θl , . . . , Θ0 ) : Θj ∈ Rp j+1×pj } . The functions f j : Rpj → Rpj are called the activation functions , and p0 ··= d and pl+1 ··= m are called the input and output dimensions , respectively . The depth of the network is l , the maximal width is p ··= maxj∈ { 0 , ... , l−1 } pj+1 , and the total number of parameters is p ··= ∑l j=0 p j+1pj . In practice , the total number of parameters often rivals or exceeds the number of samples : p ≈ n or p n. We then speak of high dimensionality . A common technique for avoiding overfitting in high-dimensional settings is regularization that induces additional structures , such as sparsity . Sparsity has the interesting side-effect of reducing the networks ’ complexities , which can facilitate interpretations and reduce demands on energy and memory . Our first sparse estimator is Θ̂con ∈ arg min Θ∈M1 { n∑ i=1 ∣∣∣∣yi − gΘ [ xi ] ∣∣∣∣22 + rcon|||Θl|||1 } ( 3 ) for a tuning parameter rcon ∈ [ 0 , ∞ ) , a nonempty set of parameters M1 ⊂ { Θ ∈M : max j∈ { 0 , ... , l−1 } |||Θj |||1 ≤ 1 } , and the ` 1-norm |||Θj |||1 ··= pj+1∑ i=1 pj∑ k=1 | ( Θj ) ik| for j ∈ { 0 , . . . , l } , Θj ∈ Rp j+1×pj . This estimator is an analog of the lasso estimator in linear regression ( Tibshirani , 1996 ) . It induces sparsity on the level of connections : the larger the tuning parameter rcon , the fewer connections among the nodes . Deep learning with ` 1-regularization has become common in theory and practice ( Kim et al. , 2016 ; Taheri et al. , 2020 ) . Our estimator ( 3 ) specifies one way to formulate this type of regularization . The estimator is indeed a regularized estimator ( rather than a constraint estimator ) , because the complexity is regulated entirely through the tuning parameter rcon in the objective function ( rather than through a tuning parameter in the set over which the objective function is optimized ) . But ` 1-regularization could also be formulated slightly differently . For example , one could consider the estimators Θcon ∈ arg min Θ∈M { n∑ i=1 ∣∣∣∣yi − gΘ [ xi ] ∣∣∣∣22 + rcon l∏ j=0 |||Θj |||1 } ( 4 ) or Θ̃con ∈ arg min Θ∈M { n∑ i=1 ∣∣∣∣yi − gΘ [ xi ] ∣∣∣∣22 + rcon l∑ j=0 |||Θj |||1 } . ( 5 ) The differences among the estimators ( 3 ) – ( 5 ) are small : for example , our theory can be adjusted for ( 4 ) with almost no changes of the derivations . The differences among the estimators mainly concern the normalizations of the parameters ; we illustrate this in the following proposition . Proposition 1 ( Scaling of Norms ) . Assume that the all-zeros parameter ( 0pl+1×pl , . . . ,0p1×p0 ) ∈ M1 is neither a solution of ( 3 ) nor of ( 5 ) , that rcon > 0 , and that the activation functions are nonnegative homogenous : f j [ ab ] = af j [ b ] for all j ∈ { 1 , . . . , l } , a ∈ [ 0 , ∞ ) , and b ∈ Rpj . Then , ||| ( Θ̂con ) 0|||1 , . . . , ||| ( Θ̂con ) l−1|||1 = 1 ( concerns the inner layers ) for all solutions of ( 3 ) , while ||| ( Θ̃con ) 0|||1 = · · · = ||| ( Θ̃con ) l|||1 ( concerns all layers ) for at least one solution of ( 5 ) . Another way to formulate ` 1-regularization was proposed in Taheri et al . ( 2020 ) : they reparametrize the networks through a scale parameter and a constraint version ofM and then to focus the regularization on the scale parameter only . Our above-stated estimator ( 3 ) is more elegant in that it avoids the reparametrization and the additional parameter . The factor |||Θl|||1 in the regularization term of ( 3 ) measures the complexity of the network over the set M1 , and the factor rcon regulates the complexity of the resulting estimator . This provides a convenient lever for data-adaptive complexity regularization through well-established calibration schemes for the tuning parameter , such as cross-validation . This practical aspect is an advantage of regularized formulations like ours as compared to constraint estimation over sets with a predefined complexity . The constraints in the setM1 of the estimator ( 3 ) can also retain the expressiveness of the full parameterization that corresponds to the setM : for example , assuming again nonnegative-homogeneous activation , one can check that for every Γ ∈M , there is a Γ′ ∈ { Θ ∈M : maxj∈ { 0 , ... , l−1 } |||Θj |||1 ≤ 1 } such that gΓ = gΓ′—cf . Taheri et al . ( 2020 , Proposition 1 ) . In contrast , existing theories on neural networks often require the parameter space to be bounded , which limits the expressiveness of the networks . Our regularization approach is , therefore , closer to practical setups than constraint approaches . The price is that to develop prediction theories , we have to use different tools than those typically used in theoretical deep learning . For example , we can not use established risk bounds such as Bartlett & Mendelson ( 2002 , Theorem 8 ) ( because Rademacher complexities over classes of unbounded functions are unbounded ) or Lederer ( 2020a , Theorem 1 ) ( because our loss function is not Lipschitz continuous ) or established concentration bounds such as McDiarmid ’ s inequality in McDiarmid ( 1989 , Lemma ( 3.3 ) ) ( because that would require a bounded loss ) . We instead invoke ideas from high-dimensional statistics , prove Lipschitz properties for neural networks , and use empirical process theory that is based on chaining ( see the Appendix ) . Our second estimator is Θ̂node ∈ arg min Θ∈M2,1 { n∑ i=1 ∣∣∣∣yi − gΘ [ xi ] ∣∣∣∣22 + rnode|||Θl|||2,1 } ( 6 ) for a tuning parameter rnode ∈ [ 0 , ∞ ) , a nonempty set of parameters M2,1 ⊂ { Θ ∈M : max j∈ { 0 , ... , l−1 } |||Θj |||2,1 ≤ 1 } , and the ` 2/ ` 1-norm |||Θj |||2,1 ··= pj∑ k=1 √√√√pj+1∑ i=1 | ( Θj ) ik|2 for j ∈ { 0 , . . . , l − 1 } , Θj ∈ Rp j+1×pj . This estimator is an analog of the group-lasso estimator in linear regression ( Bakin , 1999 ) . Again , to avoid ambiguities in the regularization , our formulation is slightly different from the standard formulations in the literature , but the fact that group-lasso regularizers leads to node-sparse networks has been discussed extensively before ( Alvarez & Salzmann , 2016 ; Liu et al. , 2015 ; Scardapane et al. , 2017 ) : the larger the tuning parameter rnode , the fewer active nodes in the network . The above-stated comments about the specific form of the connection-sparse estimator also apply to the node-sparse estimator . An illustration of connection and node sparsity is given in Figure 1 . Connection-sparse networks have only a small number of active connections between nodes ( left panel of Figure 1 ) ; node-sparse networks have inactive nodes , that is , completely unconnected nodes ( right panel of Figure 1 ) . The two notions of sparsity are connected : for example , connection sparsity can render entire nodes inactive “ by accident ” ( see the layer that follows the input layer in the left panel of the figure ) . In general , node sparsity is the weaker assumption , because it allows for highly connected nodes ; this observation is reflected in the theoretical guarantees in the following section . The optimal network architecture for given data ( such as the optimal width ) is hardly known beforehand in a data analysis . A main feature of sparsity-inducing regularization is , therefore , that it adjusts parts of the network architecture to the data . In other words , sparsity-inducing regularization is a data-driven approach to adapting the complexity of the network . While versions of the estimators ( 3 ) and ( 6 ) are popular in deep learning , statistical analyses , especially of node-sparse deep learning , are scarce . Such a statistical analysis is , therefore , the goal of the following section .

Summary: This paper studies mean-squared-error bounds for neural networks with small $\ell_1$-norm. The use of $\ell_1$-norm constraint is analogous to the use of LASSO in sparse linear regression. They give a "mean-squared-error" bound because they only analyze a fixed-design setting (where the goal is only to analyze the effect of noise) instead of the random-design setting which requires an additional analysis of generalization to fresh data.