Derivatives and residual distribution of regularized M-estimators with application to adaptive tuning

This paper studies M-estimators with gradient-Lipschitz loss function regularized1 with convex penalty in linear models with Gaussian design matrix and arbitrary2 noise distribution . A practical example is the robust M-estimator constructed with3 the Huber loss and the Elastic-Net penalty and the noise distribution has heavy-tails.4 Our main contributions are three-fold . ( i ) We provide general formulae for the5 derivatives of regularized M-estimators β̂ ( y , X ) where differentiation is taken with6 respect to both y andX ; this reveals a simple differentiability structure shared by7 all convex regularized M-estimators . ( ii ) Using these derivatives , we characterize8 the distribution of the residual ri = yi−x > i β̂ in the intermediate high-dimensional9 regime where dimension and sample size are of the same order . ( iii ) Motivated10 by the distribution of the residuals , we propose a novel adaptive criterion to select11 tuning parameters of regularized M-estimators . The criterion approximates the12 out-of-sample error up to an additive constant independent of the estimator , so13 that minimizing the criterion provides a proxy for minimizing the out-of-sample14 error . The proposed adaptive criterion does not require the knowledge of the15 noise distribution or of the covariance of the design . Simulated data confirms the16 theoretical findings , regarding both the distribution of the residuals and the success17 of the criterion as a proxy of the out-of-sample error . Finally our results reveal18 new relationships between the derivatives of β̂ ( y , X ) and the effective degrees of19 freedom of the M-estimator , which are of independent interest.20 1 Introduction21 This paper studies properties of robust estimators in linear models y = Xβ∗ + ε with response22 y ∈ Rn , unknown regression vector β∗ whereX is a design matrix with n rows x1 , ... , xn , each row23 xi being a high-dimensional feature vector in Rp with covariance Σ . Throughout , let β̂ = β̂ ( y , X ) 24 be a regularized M -estimator given as a solution of the convex minimization problem25 β̂ ( y , X ) = argminb∈Rp 1 n ∑n i=1 ρ ( yi − x > i b ) + g ( b ) ( 1 ) where ρ : R → R is a convex data-fitting loss function and g : Rp → R a convex penalty . We26 may write β̂ρ , g ( y , X ) for ( 1 ) to emphasize the dependence on the loss-penalty pair ( ρ , g ) ; if the27 argument ( y , X ) is dropped then β̂ is implicitly understood at the observed that ( y , X ) . Typical28 examples of losses include the square loss ρ ( u ) = u2/2 , the Huber loss H ( u ) = ∫ |u| 0 min ( 1 , t ) dt29 or its scaled version ρ = Λ2H ( u/Λ ) for some tuning parameter Λ > 0 , while typical examples of30 penalty functions include the Elastic-Net g ( b ) = λ‖b‖1 + µ‖b‖2/2 for tuning parameters λ , µ ≥ 0.31 The paper introduces the following criterion to select a loss-penalty pair ( ρ , g ) with small out-of-32 sample error ‖Σ1/2 ( β̂ − β∗ ) ‖2 : for a given set of candidate loss-penalty pairs { ( ρ , g ) } and the33 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . corresponding M -estimator β̂ρ , g in ( 1 ) , select the pair ( ρ , g ) that minimizes the criterion34 Crit ( ρ , g ) = ∥∥∥r+ d̂f tr [ V ] ψ ( r ) ∥∥∥ 2 with r = y −Xβ̂ρ , g ∈ Rn , d̂f = tr [ X ( ∂/∂y ) β̂ρ , g ] ∈ R , V = diag { ψ′ ( r ) } ( In −X ( ∂/∂y ) β̂ρ , g ) ∈ Rn×n ( 2 ) where tr [ · ] is the trace , ψ : R → R is the derivative of ρ , ψ′ the derivative of ψ and we extend ψ35 and ψ′ to functions Rn → Rn by componentwise application of the univariate function of the same36 symbol . Above , ( ∂/∂y ) β̂ρ , g ∈ Rp×n denotes the Jacobian of ( 1 ) with respect to y for X fixed,37 at the observed data ( y , X ) . As we will see while studying particular examples , for pairs ( ρ , g ) 38 commonly used in robust high-dimensional statistics such as the square loss , Huber loss with the39 ` 1-penalty or Elastic-Net penalty , the ratio d̂f/ tr [ V ] in ( 2 ) admits simple , closed-form expressions40 and can be computed at a negligible computational cost once β̂ρ , g ( y , X ) itself has been computed.41 The criterion ( 2 ) has an appealing adaptivity property : it does not require any knowledge of the noise42 ε or its distribution , nor any knowledge of the covariance Σ of the design.43 1.1 Contributions44 1 . The end goal of paper is to provide theoretical justification and theoretical guarantees for the45 criterion ( 2 ) in the high-dimensional regime where the ratio p/n has a finite limit and X has46 anisotropic Gaussian distribution . The theoretical results will justify the approximation47 ∥∥r + ( d̂f/ tr [ V ] ) ψ ( r ) ∥∥2/n ≈ ‖ε‖2/n+ ‖Σ1/2 ( β̂ − β∗ ) ‖2 . ( 3 ) Figure 1 illustrates the accuracy of ( 3 ) on simulated data . To study the criterion ( 2 ) and derive the48 approximation ( 3 ) , we develop novel results of independent interest regarding M -estimators in ( 1 ) :49 2 . The paper derives general formula for the derivatives ( ∂/∂yi ) β̂ and ( ∂/∂xij ) β̂ . This sheds light50 on the differentiability structure ofM -estimators for general loss-penalty pairs : for any ρ , g with g51 strongly convex , there exists Â ∈ Rp×p depending on ( y , X ) such that for almost every ( y , X ) ,52 ( ∂/∂yi ) β̂ ( y , X ) = ÂX > eiψ ′ ( ri ) , ( ∂/∂xij ) β̂ ( y , X ) = Âejψ ( ri ) − ÂX > eiψ′ ( ri ) β̂j , for ri = yi − x > i β̂ , ∀i ∈ [ n ] , j ∈ [ p ] where ej ∈ Rp and ei ∈ Rn are canonical basis vectors.53 3 . The paper obtains a stochastic representation for the residual yi−x > i β̂ for some fixed i = 1 , ... , n,54 extending some results of [ 10 ] on unregularized M -estimators to penalized ones as in ( 1 ) . In55 short , for each i = 1 , ... , n the i-th residual satisfies ri = yi − x > i β̂56 ri + ( d̂f/ trV ) ψ ( ri ) ≈ εi + Zi‖Σ1/2 ( β̂ − β∗ ) ‖ ( 4 ) where Zi ∼ N ( 0 , 1 ) is independent of εi . This stochastic representation is the motivation for57 the criterion ( 2 ) as the amplitude of the normal part in the right-hand side is proportional to the58 out-of-sample error ‖Σ1/2 ( β̂ − β∗ ) ‖ that we wish to minimize , while the variance of the noise59 εi does not depend on the choice of ( ρ , g ) .60 Simulated data in Figure 2 confirms that the stochastic representation for the i-th residual ri =61 yi − x > i β̂ is accurate . Our working assumption throughout the paper is the following.62 Assumption 1.1 . For constants γ , µ > 0 independent of n , p we have p/n ≤ γ , the loss ρ : R→ R63 is convex with a unique minimizer at 0 , continuously differentiable and its derivative ψ = ρ′ is64 1-Lipschitz . The design matrix X has iid N ( 0 , Σ ) rows for some invertible covariance Σ and the65 noise ε is independent of X with continuous distribution . The penalty g : Rp → R is µ-strongly66 convex w.r.t . Σ in the sense that b 7→ g ( b ) − ( µ/2 ) b > Σb is convex in b ∈ Rp.67 Throughout the paper , we consider a sequence ( say , indexed by n ) of regression problems with p,68 β∗ , Σ and the loss-penalty pair ( ρ , g ) depending implicitly on n. For some deterministic sequence69 ( an ) , the stochastically bounded notation OP ( an ) in this context may hide constants depending on70 γ , µ only , that is , OP ( an ) denotes a sequence of random variables Wn such that for any ε > 0 there71 exists K depending on ( ε , γ , µ ) satisfying P ( |Wn| ≥ Kan ) ≤ ε.72 Since Assumption 1.1 requires p/n ≤ γ , the Bolzano-Weierstrass theorem lets us extract a subse-73 quence of regression problems such that p/n→ γ′ along this subsequence , for some constant γ. This74 is the asymptotic regime we have in mind throughout the paper , although our results do not require a75 specific limit for the ratio p/n . For some results , we will require the following additional assumption76 which is satisfied by robust loss functions and penalty that shrink towards 0.77 Assumption 1.2 . The penalty is minimized at 0 , that is , g ( 0 ) = minb∈Rp g ( b ) ; the loss is Lipschitz as78 in |ψ| ≤M for some constant M independent of n , p ; the signal is bounded as in ‖Σ1/2β∗‖2 ≤M .79 1.2 Related works80 The context of the present work is the study of M -estimators in the regime pn has a finite limit . This81 literature pioneered in [ 2 , 10 , 9 , 15 ] typically describes the subtle behavior of β̂ in this regime by82 solving a system of nonlinear equations . This system typically depends on a prior distribution for the83 components of β∗ , and either depends on the covariance Σ [ 7 ] or assume Σ = Ip [ 2 , 16 , 6 , among84 many others ] . Solutions to the nonlinear system are a powerful tool to understand β̂ in theory , e.g.,85 to characterize the deterministic limit of ‖Σ1/2 ( β̂ − β∗ ) ‖ , see e.g. , the general results in [ 6 ] for the86 square loss and [ 16 ] for general loss-penalty pairs . However , since the system and its solution depend87 on unobservable quantity ( Σ and prior on β∗ ) , the system solution is not directly usable for practical88 purposes such as parameter tuning.89 The present work distinguishes itself from most of this literature as the goal is to describe the behavior90 of β̂ using observable quantities that only depend on the data ( y , X ) ( and not unobservable ones such91 as Σ or a prior distribution on β∗ that appear in the aforementioned nonlinear system of equations ) .92 As we will see this view lets us perform adaptive tuning of parameters in a fully adaptive manner93 using the criterion ( 2 ) . The criterion ( 2 ) appeared in previous works for the square loss only : [ 1 , 12 ] 94 studied ( 2 ) for the Lasso with Σ = Ip and [ 3 , Section 3 ] for the square loss and general penalty95 ( note that for the square loss ρ ( u ) = u2/2 , ( 2 ) reduces to n2‖r‖2/ ( n− d̂f ) 2 due to ψ ( u ) = u and96 tr [ V ] = n − d̂f . The property ψ ( u ) = u of the square loss hides the subtle interplay between97 r , ψ ( r ) , d̂f and tr [ V ] in ( 2 ) for ρ different than the square loss ) . A criterion different from ( 2 ) is98 studied in [ 12 , 3 ] to estimate the out-of-sample error . That criterion has the drawback to require the99 knowledge of Σ , unlike ( 2 ) which is fully adaptive.100 This work leverages probabilistic results on functions of standard normal random variables [ 4 ] [ 3,101 §6 , §7 ] which are consequences of Stein ’ s formula [ 14 ] . Consequently , the main limitation of our102 work is that it currently requires Gaussian design for the probabilistic results ( on the other hand , the103 differentiability result ( 5 ) is deterministic and does not rely on any probabilistic assumption ) .104 2 Differentiability of regularized M-estimators105 The first step towards the study of the criterion ( 2 ) is to justify the almost sure existence of the106 derivatives of β̂ that appear in ( 2 ) through the scalar scalar d̂f and the matrix V in ( 2 ) . Although the107 criterion ( 2 ) only involves the derivatives of β̂ ( y , X ) with respect to y for a fixedX , the proof of108 our results rely on the interplay between the derivatives with respect to y and with respect toX : this109 differentiability structure of M -estimators is the content of the following result.110 Theorem 2.1 . Let Assumption 1.1 be fulfilled . For almost every ( y , X ) the map ( y , X ) 7→ β̂ ( y , X ) 111 is differentiable at ( y , X ) and there exists a matrix Â ∈ Rp×p with ‖Σ1/2ÂΣ1/2‖op ≤ ( nµ ) −1 s.t.112 ( ∂/∂yi ) β̂ ( y , X ) = ÂX > eiψ ′ ( ri ) , ( ∂/∂xij ) β̂ ( y , X ) = Âejψ ( ri ) − ÂX > eiψ′ ( ri ) β̂j , where ri = yi − x > i β̂ , ( 5 ) ei ∈ Rn , ej ∈ Rp are canonical basis vectors , ψ : = ρ′ and ψ′ denote the derivatives . Furthermore,113 d̂f = tr [ X ( ∂/∂y ) β̂ ] = tr [ XÂX diag { ψ′ ( r ) } ] , ( 6 ) V = diag { ψ′ ( r ) } ( In −X ( ∂/∂y ) β̂ ) = diag { ψ′ ( r ) } − diag { ψ′ ( r ) } XÂX diag { ψ′ ( r ) } . ( 7 ) satisfy 0 ≤ d̂f ≤ n and 0 ≤ tr [ V ] ≤ n.114 Since the same matrix Â appears in both the derivatives with respect to yi and to xij , ( 5 ) provides115 relationship between ( ∂/∂yi ) β̂ and ( ∂/∂xij ) β̂ , for instance ( ∂/∂xij ) β̂ = Âejψ ( ri ) − β̂j ( ∂/∂yi ) β̂.116 Although the matrix Â is not explicit for arbitrary loss-penalty pair , closed-form expressions are117 available for particular examples such as the Elastic-Net penalty as discussed in Section 6.118 Remark 2.1 . For the square loss ρ ( u ) = u2/2 , the differentiability formulae ( 5 ) reduce to119 ( ∂/∂yl ) β̂ ( y , X ) = ÂX > el , ( ∂/∂xij ) β̂ ( y , X ) = Âej ( yi − x > i β̂ ) − ÂX > eiβ̂j for most every ( y , X ) and some matrix Â ∈ Rp×p depending on ( y , X ) , since in this case ψ′ = 1.120 In the simple case where g is twice continuously differentiable , ( 5 ) follows [ 4 ] with121 Â = ( X > diag { ψ′ ( r ) } X + n∇2g ( β̂ ) ) −1 ( 8 ) by differentiating the KKT conditions X > ψ ( y −Xβ̂ ) = n∇g ( β̂ ) . To illustrate why this is true , provided that β̂ ( y , X ) is differentiable , if ( y ( t ) , X ( t ) ) are smooth perturbations of ( y , X ) with ( y ( 0 ) , X ( 0 ) ) = ( y , X ) and ddt ( y ( t ) , X ( t ) ) |t=0 = ( ẏ , Ẋ ) , differentiation of X ( t ) > ψ ( y ( t ) − X ( t ) β̂ ( y ( t ) , X ( t ) ) ) = n∇g ( β̂ ( y ( t ) , X ( t ) ) ) at t = 0 and the chain rule yields Ẋ > ψ ( r ) −X > diag { ψ′ ( r ) } ( ẏ − Ẋβ̂ ( y , X ) ) = Â−1 ddt β̂ ( y ( t ) , X ( t ) ) ∣∣ t=0 with Â in ( 8 ) . This gives ( 5 ) if the penalty g is twice-differentiable . Theorem 2.1 reveals that for122 arbitrary convex penalty functions including non-differentiable ones , the differentiability structure123 ( 5 ) always holds , as in the case of twice differentiable penalty g , even for penalty functions such as124 g ( b ) = µ‖b‖2/2 + λ‖mat ( b ) ‖nuc where mat : Rp → Rd1×d2 is a linear isomorphism to the space of125 d1 × d2 matrices and ‖ · ‖nuc is the nuclear norm : in this case by Theorem 2.1 there exists a matrix126 Â ∈ Rp×p such that ( 5 ) holds although no closed-form expression for Â is known.127 The representation ( 5 ) is a powerful tool as it provides explicit derivatives of quantities of interest128 such as r = y −Xβ̂ , ‖ψ ( r ) ‖2 or ‖Σ1/2 ( β̂ − β∗ ) ‖2 . These explicit derivatives can then be used in129 probabilistic identities and inequalities that involve derivatives , for instance Stein ’ s formulae [ 14 ] ,130 the Gaussian Poincaré inequalty [ 5 , Theorem 3.20 ] , or normal approximations [ 8 , 4 ] .131 Remark 2.2 . Similar derivative formulae hold if an intercept is included in the minimization , as in132 ( β̂0 ( y , X ) , β̂ ( y , X ) ) = argmin b0∈R , b∈Rp 1 n n∑ i=1 ρ ( yi − b0 − x > i b ) + g ( b ) ( 9 ) Let Assumption 1.1 be fulfilled , and assume further ‖ψ′ ( r ) ‖2 > 0 with r : = y− 1nβ̂0−x > i β̂ where133 1n = ( 1 , ... , 1 ) > ∈ Rn . For almost every ( y , X ) the map ( y , X ) 7→ β̂ ( y , X ) is differentiable at134 ( y , X ) , and there exists Â ∈ Rp×p depending on ( y , X ) with ‖Σ1/2ÂΣ1/2‖op ≤ ( nµ ) −1 such that135 136 ( ∂/∂yi ) β̂ ( y , X ) = ÂX > Ψ′ei , ( ∂/∂xij ) β̂ ( y , X ) = Âejψ ( ri ) − ÂX > Ψ′eiβ̂j , ( 10 ) where ei ∈ Rn , ej ∈ Rp are canonical basis vectors , ψ = ρ′ and Ψ′ : = diag { ψ′ ( r ) } −137 ψ′ ( r ) ψ′ ( r ) > / ∑ i∈ [ n ] ψ ′ ( ri ) .138 3 Distribution of individual residuals139 We now turn to the distribution of a single residual ri = yi − x > i β̂ for some fixed observation140 i ∈ { 1 , ... , n } ( for instance , fix i = 1 ) . By leveraging the differentiability structure ( 5 ) and the normal141 approximation from [ 4 ] , the following result provides a clear picture of the distribution of ri.142 Theorem 3.1 . Let Assumption 1.1 be fulfilled and let Â ∈ Rp×p be given by Theorem 2.1 . Then for143 every i = 1 , ... , n there exists Zi ∼ N ( 0 , 1 ) such that144 ∣∣∣ ( ri+tr [ ΣÂ ] ψ ( ri ) ) − ( εi+‖Σ1/2 ( β̂−β∗ ) ‖Zi ) ∣∣∣ ≤ OP ( n−1/4 ) ( |ψ ( εi ) |+‖Σ1/2 ( β̂−β∗ ) ‖ ) ( 11 ) Furthermore , if εi has a fixed distribution F , there exists a bivariate variable ( ε̃ni , Z̃ n i ) converging in145 distribution to the product measure F ⊗N ( 0 , 1 ) such that146 ri + tr [ ΣÂ ] ψ ( ri ) = ε̃ n i + ‖Σ 1/2 ( β̂ − β∗ ) ‖Z̃ni . ( 12 ) If εi has a fixed distribution F and Assumption 1.2 holds then |ψ ( εi ) |+ ‖Σ1/2 ( β̂ − β∗ ) ‖ = OP ( 1 ) .147 Theorem 3.1 is a formal statement regarding the informal normal approximation148 ζi : = ri + tr [ ΣÂ ] ψ ( ri ) − εi ‖Σ1/2 ( β̂ − β∗ ) ‖ ≈ N ( 0 , 1 ) . ( 13 ) Simulations in Figure 2 confirm the normality of ζi for the Huber loss with Elastic-Net penalty149 and four combinations of tuning parameters . For the square loss ρ ( u ) = u2/2 , because ψ ( u ) = u,150 asymptotic normality of the residuals hold in the following form.151 Theorem 3.2 . Let Assumption 1.1 hold with ρ ( u ) = u2/2 and ε ∼ N ( 0 , σ2In ) . Then for i = 1,152 ( σ2 + ‖Σ1/2 ( β̂ − β∗ ) ‖2 ) −1/2 ( 1 + tr [ ΣÂ ] ) ( yi − x > i β̂ ) →d N ( 0 , 1 ) as n→ +∞ . ( 14 ) It is informative to provide a sketch of the proof of Theorem 3.1 explain the appearance of ψ ( ri ) and153 tr [ ΣÂ ] in the normal approximation results ( 11 ) and ( 13 ) . A variant of the normal approximation154 of [ 4 ] proved in the supplement states that for a differentiable function f : Rq → Rq \ { 0 } and155 z ∼ N ( 0 , Iq ) , there exists Z ∼ N ( 0 , 1 ) such hat156 E [ ∣∣∣ f ( z ) > z − ∑q k=1 ( ∂/∂zk ) fk ( z ) ‖f ( z ) ‖ − Z ∣∣∣ 2 ] ≤ C1E [ ∑q k=1 ‖ ( ∂/∂zk ) f ( z ) ‖2 ‖f ( z ) ‖2 ] . ( 15 ) Some technical hurdles aside , the proof sketch is the following : Apply the previous display to q = p,157 z = Σ−1/2xi conditionally on ( ε , ( xl ) l∈ [ n ] \ { i } ) and to f ( z ) = Σ 1/2 ( β̂ − β∗ ) in the simple case158 where β∗ = 0 ( this amounts to performing a change of variable by translation of β̂ to β̂ − β∗ ) . Then159 the right-hand side of the previous display is negligible in probability compared to Z , and in the160 left-hand side f ( z ) > z = x > i ( β̂ − β ∗ ) and ∑q k=1 ( ∂/∂zk ) fk ( z ) ≈ tr [ ΣÂ ] ψ ( ri ) as the second term161 in ( 5 ) is negligible . This completes the sketch of the proof of ( 13 ) .162 Proximal operator representation . From the above asymptotic normality results , a stochastic representation for the i-th residual ri = yi − x > i β̂ can be obtained as follows : With prox [ tρ ] ( u ) the proximal operator of x 7→ tρ ( x ) defined as the unique solution z ∈ R of equation z + tψ ( z ) = u , ri = yi − x > i β̂ = prox [ t̂ρ ] ( ε̃ni + ‖Σ 1/2 ( β̂ − β∗ ) ‖Z̃ni ) with t̂ = tr [ ΣÂ ] where ( ε̃ni , Z̃ n i ) converges in distribution to product measure F ⊗N ( 0 , 1 ) where F is the law of εi.163 4 A proxy of the out-of-sample error if Σ is known164 The approximations of the previous sections for ri + tr [ ΣÂ ] ψ ( ri ) and the fact that εi is independent165 of Zi ∼ N ( 0 , 1 ) in ( 11 ) suggest that ( ri + tr [ ΣÂ ] ψ ( ri ) ) 2 ≈ ε2i + ‖Σ 1/2 ( β̂ − β∗ ) ‖2Z2i ; and166 averaging over { 1 , ... , n } one can hope for the approximation ‖r + tr [ ΣÂ ] ψ ( r ) ‖2/n ≈ ‖ε‖2/n+167 ‖Σ1/2 ( β̂ − β∗ ) ‖2 . The following result makes this heuristic precise.168 Theorem 4.1 . Let Assumption 1.1 be fulfilled and Â be given by Theorem 2.1 . Then169 ‖Σ1/2 ( β̂ − β∗ ) ‖2 + ‖ε‖2/n = ∥∥r + tr [ ΣÂ ] ψ ( r ) ∥∥2/n+OP ( n−1/2 ) Rem , where Rem : = ‖Σ1/2 ( β̂−β∗ ) ‖2 + 1n‖ψ ( r ) ‖ 2 + ( ‖Σ1/2 ( β̂−β∗ ) ‖2 + 1n‖ψ ( r ) ‖ 2 ) 1/2‖ 1√ n ε‖ . Thus ‖Σ1/2 ( β̂ − β∗ ) ‖2 + ‖ε‖2/n = ( 1 +OP ( n−1/2 ) ) ∥∥r + tr [ ΣÂ ] ψ ( r ) ∥∥2/n . Theorem 4.1 provides a first candidate , ∥∥r + tr [ ΣÂ ] ψ ( r ) ∥∥2/n to estimate170 ‖Σ1/2 ( β̂ − β∗ ) ‖2 + ‖ε‖2/n . ( 16 ) Estimation of ( 16 ) is useful as ‖ε‖2/n is independent of the choice of the estimator β̂ and in particular171 independent of the chosen loss-penalty pair in ( 1 ) . Given two or more estimators ( 1 ) , choosing the172 one with smallest ∥∥r + tr [ ΣÂ ] ψ ( r ) ∥∥2 is thus a good proxy for minimizing the out-of-sample error.173 Corollary 4.2 . Let β̂ , β̃ be two M -estimators ( 1 ) Assumption 1.1 with loss-penalty pair ( ρ , g ) and174 ( ρ̃ , g̃ ) respectively . Assume that both satisfy Assumption 1.1 and let ψ = ρ′ and ψ̃ = ρ̃′ . Let175 r = y −Xβ̂ , r̃ = y −Xβ̃ be the residuals , Â , Ã be the corresponding matrices of size p × p176 given by Theorem 2.1 . Further assume that both estimators satisfy Assumption 1.2 and that ε has iid177 coordinates independent with E [ |εi|1+q ] ≤M for constants q ∈ ( 0 , 1 ) , M > 0 independent of n , p.178 Let Ω = { ‖XΣ−1/2‖op ≤ 2 √ n+ √ p } ∩ { ‖ε‖2 ≤ n2/ ( 1+q ) } . Then for any η > 0 independent of179 n , p there exists C ( γ , µ , η , q , M ) > 0 depending only on { γ , µ , η , q , M } such that180 P ( ‖Σ1/2 ( β̂ − β∗ ) ‖2 − ‖Σ1/2 ( β̃ − β∗ ) ‖2 > η , ‖r + tr [ ΣÂ ] ψ ( r ) ‖2 ≤ ‖r̃ + tr [ ΣÃ ] ψ̃ ( r̃ ) ‖2 ) ≤ C ( γ , µ , η , q , M ) n−q/ ( 1+q ) + P ( Ωc ) → 0 . Provided that the noise random variables εi have at least 1 + q moments , Corollary 4.2 implies181 that with probability approaching one given two M -estimators β̂ and β̃ , choosing the estimator182 corresponding to the smallest criteria among ‖r + tr [ ΣÂ ] r‖2 and ‖r̃ + tr [ ΣÃ ] r̃‖2 leads to the183 smallest out-of-sample error , up to any small constant η > 0 . This allows noise random variables εi184 with infinite variance . A similar result can be obtained to select among K different M -estimators ( 1 ) .185 Corollary 4.3 . As in Corollary 4.2 , assume E [ |εi|1+q ] ≤M and let β̂1 , ... , β̂K be M -estimators of186 the form ( 1 ) with loss-penalty pair ( ρk , gk ) satisfying Assumptions 1.1 and 1.2 . For each k = 1 , ... , K,187 let rk = y−Xβ̂k be the residuals and Âk be the corresponding matrix of size p×p from Theorem 2.1.188 Let k̂ ∈ argmink=1 , ... , K ‖rk + tr [ ΣÂk ] ψk ( rk ) ‖ where ψk = ρ′k . Then if ( γ , µ , η , q , M ) are189 constants independent of n , p190 P ( ‖Σ1/2 ( β̂k̂ − β ∗ ) ‖2 > mink=1 , ... , K ‖Σ1/2 ( β̂k − β ∗ ) ‖2 + η ) → 0 if K = o ( nq/ ( 1+q ) ) . Given K different loss-penalty pairs and the corresponding M -estimators in ( 1 ) , minimizing the191 criterion ‖r + tr [ ΣÂ ] r‖ thus provably selects a loss-penalty pair that leads to an optimal out-192 of-sample error , up to an arbitrary small constant η > 0 independent of n , p. The requirement193 K = o ( nq/ ( 1+q ) ) means that the cardinality of the collection of M -estimators to select from should194 grow more slowly than a power of n. This is typically satisfied for default tuning parameter grids in195 popular libraries ( e.g. , sklearn.linear_model.Lasso [ 13 ] ) with tuning parameters evenly spaced196 in a log-scale that consequently have cardinality logarithmic in the parameter range . The major197 drawback of the criterion ‖r + tr [ ΣÂ ] r‖ is the dependence through tr [ ΣÂ ] on the covariance Σ198 of the design , which is typically unknown . The next section introduces an estimator of tr [ ΣÂ ] that199 does not require the knowledge of Σ.200

Summary: This paper studies the derivatives of regularized M-estimators and distribution of the residual.  These results give rises to a new novel adaptive criterion for tuning loss and penalty function. Numerical experiments are conducted on various models to validate the theoretical result.