Localization, Convexity, and Star Aggregation

1 Introduction . A central question in statistical learning theory asks which properties of the empirical risk minimizer can be inferred without regard to the underlying data distribution . In particular , the convergence rate of empirical risk minimization has been shown to depend both on distribution-specific properties , such as Tsybakov ’ s margin condition ( Tsybakov , 2004 ) , as well as distribution-independent or “ global ” properties , such as VC dimension of the class and curvature of the loss ( Mendelson , 2002 ) . The local Rademacher complexities of Bartlett et al . ( 2005 ) address this issue by giving bounds that are data dependent and also adapt to the global properties of the learning problem . More recently , the offset Rademacher complexities of Rakhlin and Sridharan ( 2014 ) and Liang et al . ( 2015 ) have been shown to provide similar bounds for the square loss in several contexts . The offset complexity provides a simplified analysis that transparently relates the geometry of the square loss to statistical properties of the estimator . We extend offset Rademacher complexities to systematically study how curvature of the loss translates to fast rates of convergence in proper and improper statistical learning . We show many results in the literature can be studied in a common geometric framework : from classical results on uniform convexity of the loss ( Bartlett et al. , 2006 ; Mendelson , 2002 ) , to newer results on exponential concavity ( van Erven et al. , 2015 ; Mehta , 2017 ) and self-concordance ( Bilodeau et al. , 2020 ; Marteau-Ferey et al. , 2019 ) . Remarkably , our analysis applies simultaneously to convex empirical risk minimization and to improper learning with the star algorithm of Audibert ( 2008 ) . In each case , excess risk is tightly ∗Supported by the MIT Jerry A. Hausman Graduate Dissertation Fellowship 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . bounded by the offset Rademacher complexity . Thus , we give a clean and unified treatment of fast rates in proper and improper learning with curved loss . Our contributions may be summarized as follows . 1 . We generalize the offset Rademacher complexities of Liang et al . ( 2015 ) to show that when the loss is ( µ , d ) -convex ( defined below ) , excess risk is bounded by a corresponding offset Rademacher process . 2 . We show in this general setting that the “ star algorithm ” of Audibert ( 2008 ) satisfies an upper bound matching the convex case , allowing us to simultaneously treat convex risk minimization and improper learning under general entropy conditions . 3 . We show that ( µ , d ) -convexity captures the behavior of exponentially concave , uniformly convex , and self concordant losses , leading to fast upper bounds for learning under the p-loss , the relative entropy , and the logistic loss . 4 . We observe that generalized linear models ( glms ) may be expressed as learning in a non-convex class subject to the exponentially concave relative entropy loss , and introduce a general procedure—the regularized star estimator—which attains fast rates in arbitrary glms from a mixture of at most two proper estimators . In the case of k-ary logistic regression with bounded features X ∈ Rq and each predictor norm bounded by B , our upper bounds on the excess risk take the form kq n · polylog ( n , k , B , 1/ρ ) ( 1 ) with probability at least 1− ρ . Thus , in the i.i.d . setting , fast rates that depend only logarithmically on B are achievable by a simple , improper variant of empirical risk minimization . 1.1 Relation to Prior Work . This paper examines the localization or fast-rate phenomenon , where the excess loss vanishes at a rate exceeding the model ’ s uniform law of large numbers ( Koltchinskii and Panchenko , 2000 ; Tsybakov , 2004 ) . Recently , Liang et al . ( 2015 ) showed that localization for the square loss is captured by the so-called offset Rademacher complexity . In addition to being analytically straightforward , the offset complexity appears in a minimax analysis of online regression ( Rakhlin and Sridharan , 2014 ) , suggesting a unified treatment of online and statistical learning . In parallel work , van Erven et al . ( 2015 ) have showed that exp-concavity of the loss , often studied in online optimization , implies fast rates in the statistical setting via the probabilistic “ central condition ” ( see also Juditsky et al . ( 2008 ) ; Mehta ( 2017 ) ) . Our work unifies these results . In particular , we show that the offset complexity upper bound can be generalized to arbitrary losses satisfying a certain curvature condition that includes exp-concave losses . These results considerably extend the scope of offset Rademacher complexities in the statistical setting , as we illustrate ; we also believe it can refine the minimax analysis of online regression with exp-concave loss , which we leave to subsequent work . Another noteworthy aspect of our work is its treatment of improper learning—that is , minimizing loss compared to the class F with an estimator that need not belong to F . Unlike prior work , which typically involved aggregation over a finite subset of F ( Rakhlin et al. , 2017 ; Audibert , 2009 ) , we provide a completely unified analysis that simultaneously leads to optimal rates for convex empirical risk minimization and improper learning under general entropy conditions . In particular , we circumvent the issue that passing to the convex hull of certain classes can “ blow up ” their complexity and lead to slower rates ( Mendelson , 2002 ) . We apply our analysis to generalized linear models ( glms ) , such as logistic regression , where standard procedures surprisingly do not attain fast rates ( Hazan et al. , 2014 ) . Inspired by Foster et al . ( 2018 ) , we note that any glm can be recast as learning in a non-convex class subject to the exp-concave relative entropy . The star algorithm therefore attains fast rates for a wide class of generalized linear models , improving on prior results that exploit strong convexity ( see e.g . Rigollet ( 2012 ) ) . Unlike other recent works on improper logistic regression , we do not prove a polynomial time complexity for our procedure , nor do we study the online setting ; these are left as open problems for future work . However , our procedure remains of interest for its simplicity—it outputs a fixed mixture of two predictors as opposed to optimizing at each query—and for its applicability to arbitrary glms ( Foster et al. , 2018 ; Mourtada and Gaïffas , 2020 ; Jézéquel et al. , 2020 ) . 2 Margin inequalities . We begin our discussion with the following definition which extends the classical uniform convexity ( see e.g . Chidume ( 2009 ) ) , giving a quantitative bound on the extent to which the plane tangent to f at ( y , f ( y ) ) lies below the graph of f . Definition 1 . Let µ : R→ R be increasing and convex with µ ( 0 ) = 0 , and let d ( x , y ) be a pseudometric on the vector space B . A differentiable convex function f : B → R with gradient ∇f is ( µ , d ) -convex if for all x , y ∈ B , f ( x ) − f ( y ) − 〈∇f |y , x− y〉 ≥ µ ( d ( x , y ) ) ( 2 ) In particular , this condition implies that the lower sets of f , fx = { z ∣∣∣ f ( z ) ≤ f ( x ) } , are strictly convex , and that for any closed and convex K there is a unique x̂ ∈ arg minx∈K f ( x ) . We adopt the notation that for a subset S ⊂ B , ∂S denotes the boundary of S , S◦ = S \ ∂S denotes the interior , and coS denotes the convex hull . Remark 2 . At this point , we are intentionally vague about the role of the metric , d. We shall subsequently see that the choice of d is crucial for producing good upper bounds . The conventional choice is d ( x , y ) = |x− y| and µ ( z ) = αzp , corresponding to p-uniform convexity ( with modulus α ) . However , our results on self-concordant and exp-concave losses use non-standard choices of d. By a standard argument , given in Lemma 3 below , the condition ( 2 ) on the loss implies that the empirical risk minimizer in a convex class satisfies a geometric margin inequality . Intuitively , if another hypothesis performs nearly as well as the empirical minimizer in the sample , it must also be d-close to the empirical minimizer . Lemma 3 . Let K be a closed , convex subset of B . If f is ( µ , d ) -convex and x̂ minimizes f over K then for all x ∈ K f ( x ) − f ( x̂ ) ≥ µ ( d ( x , x̂ ) ) . ( 3 ) Proof . Let zλ = λx+ ( 1− λ ) x̂ be the line segment interpolating x and x̂ , and put ϕ ( λ ) = f ( zλ ) . By convexity of K , we must have zλ ∈ K , hence ϕ is minimized at λ = 0 by optimality of x̂ . Hence , ϕ ( λ ) − ϕ ( 0 ) ≥ 0 for all λ , and ∇ϕ must satisfy ∇ϕ|λ=0 = 〈∇f |x̂ , x− x̂〉 ≥ 0 . Thus , f ( x ) − f ( x̂ ) ≥ f ( x ) − f ( x̂ ) − 〈∇f |x̂ , x− x̂〉 ≥ µ ( d ( x , x̂ ) ) , proving the lemma . 2.1 The star algorithm . We now introduce our first main result , which extends the above analysis to the problem of learning in a non-convex class . Surprisingly , in the general setting of ( µ , d ) -convexity , an inequality matching ( 3 ) is always attained by a simple , two-step variant of empirical risk minimization . Let us now introduce the procedure . Given x ∈ S , the “ star hull ” of S with respect to x is defined by star ( x , S ) = { λx+ ( 1− λ ) s | s ∈ S , λ ∈ [ 0 , 1 ] } . The procedure may be summarized as follows . 1 : procedure STAR ( S , f ) 2 : Find x̂ that minimizes f over S 3 : Find x̃ that minimizes f over star ( x̂ , S ) 4 : Output x̃ 5 : end procedure Our next goal is to show that if S is an arbitrary ( not necessarily convex ) set , and x̃ is the output of the star algorithm , then f ( x ) − f ( x̃ ) ≥ µ ( 1 3d ( x , x̃ ) ) ( 4 ) whenever f is ( µ , d ) -convex . In particular , if f is the empirical risk and S is the function class , then x̃ is the “ star estimator ” introduced by Audibert ( 2008 ) . Thus , whenever ( µ , d ) -convexity leads to an upper bound for empirical risk minimization in a convex class via the inequality ( 3 ) , the same bound is enjoyed up to constants by the star algorithm in an arbitrary class . To prove ( 4 ) , recall how we proved Lemma 3 : we showed that if the line-segment λ 7→ zλ which linearly interpolates between any two points x and y lies entirely in the set fx \ f◦y , so f ( zλ ) ≥ f ( y ) , then we have that f ( x ) − f ( y ) ≥ µ ( d ( x , y ) ) . This is illustrated in Figure 1 . Unfortunately , the existence of such a line-segment zλ connecting x and x̃ relies on the convexity of S. In order to extend the argument as we want to , we make two observations . Firstly , the line segment λ 7→ zλ can be chosen to be any line segment connecting ∂fx to ∂fy and lying entirely in fx \ f◦y . Then , since ∂fx and ∂fy are level sets , the same argument yields f ( x ) − f ( y ) = f ( z1 ) − f ( z0 ) ≥ µ ( d ( z1 , z0 ) ) . ( 5 ) Secondly , we can show that there exist at most three such line-segments which together comprise a path from x to x̃ . By the triangle inequality , at least one of these segments has length greater than 1 3d ( x̃ , x ) . Combining this with ( 5 ) yields the desired inequality ( 4 ) . We state this as a proposition and prove it formally in Appendix A . A straightforward illustration of the proof , following the preceding discussion , is provided in Figure 2 . Proposition 4 . Let f be ( µ , d ) -convex . Suppose x̂ minimizes f over S and x̃ minimizes f over star ( x̂ , S ) . Then , for any x ∈ S , f ( x ) − f ( x̃ ) ≥ µ ( 1 3d ( x , x̃ ) ) .

Summary: This paper is concerned with the excess risk analysis of empirical risk minimization (ERM) over convex function classes, as well as Audibert's 'improper' Star-shaped aggregation procedure over nonconvex classes (which reduces to ERM in the convex setting), in statistical learning. Sharp bounds have been obtained in the case of least-squares regression, and this paper considers additional settings such as exp-concave statistical learning, as well as losses satisfying different types of uniform convexity (such as p-convexity). From a technical point of view, the analysis relies on an extension of the offset Rademacher complexities (introduced and leveraged for square loss by Liang, Rakhlin and Sridharan to analyze convex ERM and nonconvex Star) to other uniformly convex losses, in particular exp-concave ones; it is shown that an analogue of this offset process also controls the excess risk in these cases. The main outcomes of this analysis are applications to regression with p-loss for p>2, and to improper learning of generalized linear models (such as logistic regression).