THE EFFICACY OF L1 REGULARIZATION IN NEURAL NETWORKS

1 INTRODUCTION . Neural networks have been successfully applied in modeling nonlinear regression functions in various domains of applications . A critical evaluation metric for a predictive learning model is to measure its statistical risk bound . For example , the L1 or L2 risks of typical parametric models such as linear regressions are at the order of ( d/n ) 1/2 for small d ( Seber & Lee , 2012 ) , where d and n denote respectively the input dimension and number of observations . Obtaining the risk bound for a nonparametric regression model such as neural networks is highly nontrivial . It involves an approximation error ( or bias ) term as well as a generalization error ( or variance ) term . The standard analysis of generalization error bounds may not be sufficient to describe the overall predictive performance of a model class unless the data is assumed to be generated from it . For the model class of two-layer feedforward networks and a rather general data-generating process , Barron ( 1993 ; 1994 ) proved an approximation error bound of O ( r−1/2 ) where r denotes the number of neurons . The author further developed a statistical risk error bound of O ( ( d/n ) 1/4 ) , which is the tightest statistical risk bound for the class of two-layer neural networks up to the authors ’ knowledge ( for d < n ) . This risk bound is based on an optimal bias-variance tradeoff involving an deliberate choice of r. Note that the risk is at a convergence rate much slower than the classical parametric rate . We will tackle the same problem from a different perspective , and obtain a much tighter risk bound . A practical challenge closely related to statistical risks is to select the most appropriate neural network architecture for a particular data domain ( Ding et al. , 2018 ) . For two-layer neural networks , this is equivalent to selecting the number of hidden neurons r. While a small r tends to underfit , researchers have observed that the network is not overfitting even for moderately large r. Nevertheless , recent research has also shown that an overly large r ( e.g. , when r > n ) does cause overfitting with high probability ( Zhang et al. , 2016 ) . It can be shown under some non-degeneracy conditions that a two-layer neural network with more than n hidden neurons can perfectly fit n arbitrary data , even in the presence of noise , which inevitably leads to overfitting . A theoretical choice of r suggested by the asymptotic analysis in ( Barron , 1994 ) is at the order of ( n/d ) 1/2 , and a practical choice of r is often from cross-validation with an appropriate splitting ratio ( Ding et al. , 2018 ) . An alternative perspective that we advocate is to learn from a single neural network with sufficiently many neurons and an appropriate L1 regularization on the neuron coefficients , instead of performing a selection from multiple candidate neural models . A potential benefit of this approach is easier hardware implementation and computation since we do not need to implement multiple models separately . Perhaps more importantly , this perspective of training enables much tighter risk bounds , as we will demonstrate . In this work , we focus on the model class of two-layer feedforward neural networks . Our main contributions are summarized below . First , we prove that L1 regularization on the coefficients of the output layer can produce a risk bound O ( ( d/n ) 1/2 ) ( up to a logarithmic factor ) under the L1 training loss , which approaches the minimax optimal rate . Such a rate has not been established under the L2 training loss so far . The result indicates a potential benefit of using L1 regularization for training a neural network , instead of selecting from a number of neurons . Additionally , a key ingredient of our result is a unique amalgamation of dimension-based and norm-based risk analysis , which may be interesting on its own right . The technique leads to an interesting observation that an excessively large r can reduce approximation error while not increasing generalization error under L1 regularizations . This implies that an explicit regularization can eliminate overfitting even when the specified number of neurons is enormous . Moreover , we prove that the L1 regularization on the input layer can induce sparsity by producing a risk bound that does not involve d , where d may be much larger compared with the true number of significant variables . Related work on neural network analysis . Despite the practical success of neural networks , a systematic understanding of their theoretical limit remains an ongoing challenge and has motivated research from various perspectives . Cybenko ( 1989 ) showed that any continuous function could be approximated arbitrarily well by a two-layer perceptron with sigmoid activation functions . Barron ( 1993 ; 1994 ) established an approximation error bound of using two-layer neural networks to fit arbitrary smooth functions and their statistical risk bounds . A dimension-free Rademacher complexity for deep ReLU neural networks was recently developed ( Golowich et al. , 2017 ; Barron & Klusowski , 2019 ) . Based on a contraction lemma , a series of norm-based complexities and their corresponding generalization errors are developed ( Neyshabur et al. , 2015 , and the references therein ) . Another perspective is to assume that the data are generated by a neural network and convert its parameter estimation into a tensor decomposition problem through the score function of the known or estimated input distribution ( Anandkumar et al. , 2014 ; Janzamin et al. , 2015 ; Ge et al. , 2017 ; Mondelli & Montanari , 2018 ) . Also , tight error bounds have been established recently by assuming that neural networks of parsimonious structures generate the data . In this direction , Schmidt-Hieber ( 2017 ) proved that specific deep neural networks with few non-zero network parameters can achieve minimax rates of convergence . Bauer & Kohler ( 2019 ) developed an error bound that is free from the input dimension , by assuming a generalized hierarchical interaction model . Related work on L1 regularization . The use of L1 regularization has been widely studied in linear regression problems ( Hastie et al. , 2009 , Chapter 3 ) . The use of L1 regularization for training neural networks has been recently advocated in deep learning practice . A prominent use of L1 regularization was to empirically sparsify weight coefficients and thus compress a network that requires intensive memory usage ( Cheng et al. , 2017 ) . The extension of L1 regularization to groupL1 regularization ( Yuan & Lin , 2006 ) has also been extensively used in learning various neural networks ( Han et al. , 2015 ; Zhao et al. , 2015 ; Wen et al. , 2016 ; Scardapane et al. , 2017 ) . Despite the above practice , the efficacy of L1 regularization in neural networks deserves more theoretical study . In the context of two-layer neural networks , we will show that the L1 regularizations in the output and input layers play two different roles : the former for reducing generalization error caused by excessive neurons while the latter for sparsifying input signals in the presence of substantial redundancy . Unlike previous theoretical work , we consider the L1 loss , which ranks among the most popular loss functions in , e.g. , learning from ordinal data ( Pedregosa et al. , 2017 ) or imaging data ( Zhao et al. , 2016 ) , and for which the statistical risk has not been studied previously . In practice , the use of L1 loss for training has been implemented in prevalent computational frameworks such as Tensorflow ( Google , 2016 ) , Pytorch ( Ketkar , 2017 ) , and Keras ( Gulli & Pal , 2017 ) . 2 PROBLEM FORMULATION . 2.1 MODEL ASSUMPTION AND EVALUATION . Suppose we have n labeled observations { ( xi , yi ) } i=1 , ... , n , where yi ’ s are continuously-valued responses or labels . We assume that the underlying data generating model is yi = f∗ ( xi ) + εi for some unknown function f∗ ( · ) , where xi ’ s ∈ X ⊂ Rd are independent and identically distributed , and εi ’ s are independent and identically distributed that is symmetric at zero and E ( ε2i | xi ) ≤ τ2 . ( 1 ) Here , X is a bounded set that contains zero , for example { x : ‖x‖∞ ≤ M } for some constant M . Our goal is learn a regression model f̂n : x 7→ f̂n ( x ) for prediction . The f̂n is obtained from the following form of neural networks r∑ j=1 ajσ ( w > j x+ bj ) + a0 , ( 2 ) where a0 , aj , bj ∈ R , wj ∈ Rd , j = 1 , . . . , r , are parameters to estimate . We let a = [ a0 , a1 , . . . , ar ] T denote the output layer coefficients . An illustration is given Figure 1 . The estimation is typically accomplished by minimizing the empirical risk n−1 ∑n i=1 ` ( yi , f ( xi ) ) , for some loss function l ( · ) plus a regularization term . We first consider the L1 regularization at the output layer . In particular , we search for such f by the empirical risk minimization from the function class FV = { f : Rd → R ∣∣∣f ( x ) = r∑ j=1 ajσ ( w > j x+ bj ) + a0 , ‖a‖1 ≤ V } ( 3 ) where V is a constant . The following statistical risk measures the predictive performance of a learned model f : R ( f ) ∆= E ` ( y , f ( x ) ) − E ` ( y , f∗ ( x ) ) . The loss function ` ( · ) is pre-determined by data analysts , usually the L1 loss defined by ` ( y , ỹ ) = |y − ỹ| or the L2 loss defined by ` 2 ( y , ỹ ) = ( y − ỹ ) 2 . Under the L1 loss , the risk is R ( f ) = E |f∗ ( x ) + ε − f ( x ) | − E |ε| , which is nonnegative for symmetric random variables ε . It is typical to use the same loss function for both training and evaluation . 2.2 NOTATION . Throughout the paper , we use n , d , k , r to denote the number of observations , the number of input variables or input dimension , the number of significant input variables or sparsity level , the number of neurons ( or hidden dimension ) , respectively . We write an & bn , bn . an , or bn = O ( an ) , if |bn/an| < c for some constant c for all sufficiently large n. We write an bn if an & bn as well as an . bn . Let N ( µ , V ) denote Gaussian distribution with mean µ and covariance V . Let ‖ · ‖1 and ‖ · ‖2 denote the common L1 and L2 vector norms , respectively . Let X denote the essential support of X . For any vector z ∈ Rd , we define ‖z‖X ∆ = supx∈X |x > z| , which may or may not be infinity . If X = { x : ‖x‖∞ ≤ M } , ‖z‖X is equivalent to M‖z‖1 . Throughout the paper , f̂n denotes the estimated regression function with n being the number of observations .

Summary: This paper studies the statistical risk bounds for two-layer neural networks with $L_1$-regularization. The authors consider two types of $L_1$-regularization: the $L_1$-regularization on output layer and the $L_1$-regularization on the input layer. For the $L_1$-regularization on output layer, the authors develop nearly minimax statistical risk bounds. For the $L_1$-regularization on input layers, the authors develop bounds with no-dependency on the input dimension. The paper is clearly written and easy to follow.