1 2 O ct 2 01 9 First order expansion of convex regularized estimators

We consider first order expansions of convex penalized estimators in highdimensional regression problems with random designs. Our setting includes linear regression and logistic regression as special cases. For a given penalty function h and the corresponding penalized estimator β̂, we construct a quantity η, the first order expansion of β̂, such that the distance between β̂ and η is an order of magnitude smaller than the estimation error ‖β̂ − β∗‖. In this sense, the first order expansion η can be thought of as a generalization of influence functions from the mathematical statistics literature to regularized estimators in high-dimensions. Such first order expansion implies that the risk of β̂ is asymptotically the same as the risk of η which leads to a precise characterization of the MSE of β̂; this characterization takes a particularly simple form for isotropic design. Such first order expansion also leads to inference results based on β̂. We provide sufficient conditions for the existence of such first order expansion for three regularizers: the Lasso in its constrained form, the lasso in its penalized form, and the Group-Lasso. The results apply to general loss functions under some conditions and those conditions are satisfied for the squared loss in linear regression and for the logistic loss in the logistic model. Introduction. We consider learning problems where one observes observations (X1, Y1), . , (Xn, Yn) with responses Yi and feature vectors Xi ∈ R. The literature of the past two decades has demonstrated the great success of regularized estimators that are commonly defined as solutions to regularized optimization problems of the form β̂ = argminβ∈Rp n −1 ∑n i=1 l(Yi, X T i β) + h(β), (1) where l(·, ·) is referred to as the loss (e.g. squared loss, logistic loss) and h : R → R is a regularization penalty (e.g. the l1-norm for the Lasso, the l2,1 norm for the Group-Lasso). All tuning parameters are included in h(·). The performance of such regularized estimators is measured in terms of prediction error or in terms estimation error ‖β̂ − β∗‖ if the data comes from a model such as Y = Xβ + ε for some noise random variable ε in linear regression or P(Y = 1|X = x) = 1/(1 + exp(xβ)) = 1− P(Y = 0|X = x) in logistic regression, where β is the unknown coefficient vector. For instance, if s = ‖β‖0 is the sparsity of β in the above model, and (Xi, Yi)i=1,. ,n are iid observations with the same distribution as (X,Y ), both the Lasso in linear regression and the logistic Lasso in logistic regression enjoy rate optimality: ‖β̂ − β∗‖2 ≤ s log(ep/s)/n; see [35, 1] or the proof of Proposition 3.4 in Appendix F for self-contained proofs. The latter estimation bound is optimal in a minimax sense and cannot 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. be improved, and the minimax rate s log(ep/s)/n represents the scale below which uncertainty is unavoidable by information theoretic arguments, see for instance [36, Section 5]. We are interested in providing first order expansion of β̂ at scales negligible compared to the minimax estimation rate, e.g. at scales negligible compared to s log(ep/s)/n in the aforementioned sparsity contexts. To be more precise, the results below will construct random first order expansion η such that η is measurable w.r. t. a much smaller sigma algebra than that generated by (Xi, Yi)i=1,. ,n, and ‖η − β̂‖K = op(1)‖β̂ − β‖K for some norm ‖ · ‖K related to the problem at hand, (2) where op(1) is a quantity that converges to 0 in probability. In other words, we provide a firstorder expansion of β̂ similar to an influence function expansion, cf. Section 1. This allows for understanding bias and standard deviation of β̂ at a finer scale than simply showing that β̂ − β converge to zero at the minimax rate. The present paper intends to answer the two questions below regarding such first order expansion. (Q1) How to construct η such that (2) holds for a given convex regularized estimator such as (1)? (Q2) How are such first order expansions useful in high-dimensional learning problems where convex regularized estimators (1) are commonly used? An expansion η satisfying (2) is interesting in and by itself because it describes phenomena at a finer scale than most of the literature in high-dimensional problems which focuses on minimax prediction and estimation bounds. More importantly, we will see in Section 4 that such first-order expansions lead to exact identities for the loss of estimators, and in Section 5 that such first-order expansions can be used for inference (i.e., uncertainty quantification) about the unknown coefficient vector β. Notation. Throughout the paper, C1, C2, C3, . denote positive absolute constants and we write a . b if a ≤ Cb for some absolute constant C > 0. The Euclidean norm in R or in R is denoted by ‖ · ‖. For any positive definite matrix A, we write ‖u‖A = ‖A1/2u‖ for the matrix square-root A. For matrices, ‖ · ‖op and ‖ · ‖F denote the operator norm and Frobenius norm. For any real a, a+ = max(0, a). If S ⊂ {1, . , p}, v ∈ R,M ∈ R then vS is the restriction (vj , j ∈ S) and MS,S is the square submatrix of M made of entries indexed in S × S. 1 Influence functions and Construction of η To answer (Q1), we start with a recap of unregularized estimators that correspond to h(·) ≡ 0, when p is fixed as n → +∞. In this case, it is well-known that certain smoothness assumptions on the loss such as twice differentiability [25, 19] or stochastic equicontinuity [42, 41] imply (for any norm, since all norms are equivalent in R for fixed p):