Localization, Convexity, and Star Aggregation

1 Introduction . A central question in statistical learning theory asks which properties of the empirical risk minimizer can be inferred without regard to the underlying data distribution . In particular , the convergence rate of empirical risk minimization has been shown to depend both on distribution-specific properties , such as Tsybakov ’ s margin condition ( Tsybakov , 2004 ) , as well as distribution-independent or “ global ” properties , such as VC dimension of the class and curvature of the loss ( Mendelson , 2002 ) . The local Rademacher complexities of Bartlett et al . ( 2005 ) address this issue by giving bounds that are data dependent and also adapt to the global properties of the learning problem . More recently , the offset Rademacher complexities of Rakhlin and Sridharan ( 2014 ) and Liang et al . ( 2015 ) have been shown to provide similar bounds for the square loss in several contexts . The offset complexity provides a simplified analysis that transparently relates the geometry of the square loss to statistical properties of the estimator . We extend offset Rademacher complexities to systematically study how curvature of the loss translates to fast rates of convergence in proper and improper statistical learning . We show many results in the literature can be studied in a common geometric framework : from classical results on uniform convexity of the loss ( Bartlett et al. , 2006 ; Mendelson , 2002 ) , to newer results on exponential concavity ( van Erven et al. , 2015 ; Mehta , 2017 ) and self-concordance ( Bilodeau et al. , 2020 ; Marteau-Ferey et al. , 2019 ) . Remarkably , our analysis applies simultaneously to convex empirical risk minimization and to improper learning with the star algorithm of Audibert ( 2008 ) . In each case , excess risk is tightly ∗Supported by the MIT Jerry A. Hausman Graduate Dissertation Fellowship 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . bounded by the offset Rademacher complexity . Thus , we give a clean and unified treatment of fast rates in proper and improper learning with curved loss . Our contributions may be summarized as follows . 1 . We generalize the offset Rademacher complexities of Liang et al . ( 2015 ) to show that when the loss is ( µ , d ) -convex ( defined below ) , excess risk is bounded by a corresponding offset Rademacher process . 2 . We show in this general setting that the “ star algorithm ” of Audibert ( 2008 ) satisfies an upper bound matching the convex case , allowing us to simultaneously treat convex risk minimization and improper learning under general entropy conditions . 3 . We show that ( µ , d ) -convexity captures the behavior of exponentially concave , uniformly convex , and self concordant losses , leading to fast upper bounds for learning under the p-loss , the relative entropy , and the logistic loss . 4 . We observe that generalized linear models ( glms ) may be expressed as learning in a non-convex class subject to the exponentially concave relative entropy loss , and introduce a general procedure—the regularized star estimator—which attains fast rates in arbitrary glms from a mixture of at most two proper estimators . In the case of k-ary logistic regression with bounded features X ∈ Rq and each predictor norm bounded by B , our upper bounds on the excess risk take the form kq n · polylog ( n , k , B , 1/ρ ) ( 1 ) with probability at least 1− ρ . Thus , in the i.i.d . setting , fast rates that depend only logarithmically on B are achievable by a simple , improper variant of empirical risk minimization . 1.1 Relation to Prior Work . This paper examines the localization or fast-rate phenomenon , where the excess loss vanishes at a rate exceeding the model ’ s uniform law of large numbers ( Koltchinskii and Panchenko , 2000 ; Tsybakov , 2004 ) . Recently , Liang et al . ( 2015 ) showed that localization for the square loss is captured by the so-called offset Rademacher complexity . In addition to being analytically straightforward , the offset complexity appears in a minimax analysis of online regression ( Rakhlin and Sridharan , 2014 ) , suggesting a unified treatment of online and statistical learning . In parallel work , van Erven et al . ( 2015 ) have showed that exp-concavity of the loss , often studied in online optimization , implies fast rates in the statistical setting via the probabilistic “ central condition ” ( see also Juditsky et al . ( 2008 ) ; Mehta ( 2017 ) ) . Our work unifies these results . In particular , we show that the offset complexity upper bound can be generalized to arbitrary losses satisfying a certain curvature condition that includes exp-concave losses . These results considerably extend the scope of offset Rademacher complexities in the statistical setting , as we illustrate ; we also believe it can refine the minimax analysis of online regression with exp-concave loss , which we leave to subsequent work . Another noteworthy aspect of our work is its treatment of improper learning—that is , minimizing loss compared to the class F with an estimator that need not belong to F . Unlike prior work , which typically involved aggregation over a finite subset of F ( Rakhlin et al. , 2017 ; Audibert , 2009 ) , we provide a completely unified analysis that simultaneously leads to optimal rates for convex empirical risk minimization and improper learning under general entropy conditions . In particular , we circumvent the issue that passing to the convex hull of certain classes can “ blow up ” their complexity and lead to slower rates ( Mendelson , 2002 ) . We apply our analysis to generalized linear models ( glms ) , such as logistic regression , where standard procedures surprisingly do not attain fast rates ( Hazan et al. , 2014 ) . Inspired by Foster et al . ( 2018 ) , we note that any glm can be recast as learning in a non-convex class subject to the exp-concave relative entropy . The star algorithm therefore attains fast rates for a wide class of generalized linear models , improving on prior results that exploit strong convexity ( see e.g . Rigollet ( 2012 ) ) . Unlike other recent works on improper logistic regression , we do not prove a polynomial time complexity for our procedure , nor do we study the online setting ; these are left as open problems for future work . However , our procedure remains of interest for its simplicity—it outputs a fixed mixture of two predictors as opposed to optimizing at each query—and for its applicability to arbitrary glms ( Foster et al. , 2018 ; Mourtada and Gaïffas , 2020 ; Jézéquel et al. , 2020 ) . 2 Margin inequalities . We begin our discussion with the following definition which extends the classical uniform convexity ( see e.g . Chidume ( 2009 ) ) , giving a quantitative bound on the extent to which the plane tangent to f at ( y , f ( y ) ) lies below the graph of f . Definition 1 . Let µ : R→ R be increasing and convex with µ ( 0 ) = 0 , and let d ( x , y ) be a pseudometric on the vector space B . A differentiable convex function f : B → R with gradient ∇f is ( µ , d ) -convex if for all x , y ∈ B , f ( x ) − f ( y ) − 〈∇f |y , x− y〉 ≥ µ ( d ( x , y ) ) ( 2 ) In particular , this condition implies that the lower sets of f , fx = { z ∣∣∣ f ( z ) ≤ f ( x ) } , are strictly convex , and that for any closed and convex K there is a unique x̂ ∈ arg minx∈K f ( x ) . We adopt the notation that for a subset S ⊂ B , ∂S denotes the boundary of S , S◦ = S \ ∂S denotes the interior , and coS denotes the convex hull . Remark 2 . At this point , we are intentionally vague about the role of the metric , d. We shall subsequently see that the choice of d is crucial for producing good upper bounds . The conventional choice is d ( x , y ) = |x− y| and µ ( z ) = αzp , corresponding to p-uniform convexity ( with modulus α ) . However , our results on self-concordant and exp-concave losses use non-standard choices of d. By a standard argument , given in Lemma 3 below , the condition ( 2 ) on the loss implies that the empirical risk minimizer in a convex class satisfies a geometric margin inequality . Intuitively , if another hypothesis performs nearly as well as the empirical minimizer in the sample , it must also be d-close to the empirical minimizer . Lemma 3 . Let K be a closed , convex subset of B . If f is ( µ , d ) -convex and x̂ minimizes f over K then for all x ∈ K f ( x ) − f ( x̂ ) ≥ µ ( d ( x , x̂ ) ) . ( 3 ) Proof . Let zλ = λx+ ( 1− λ ) x̂ be the line segment interpolating x and x̂ , and put ϕ ( λ ) = f ( zλ ) . By convexity of K , we must have zλ ∈ K , hence ϕ is minimized at λ = 0 by optimality of x̂ . Hence , ϕ ( λ ) − ϕ ( 0 ) ≥ 0 for all λ , and ∇ϕ must satisfy ∇ϕ|λ=0 = 〈∇f |x̂ , x− x̂〉 ≥ 0 . Thus , f ( x ) − f ( x̂ ) ≥ f ( x ) − f ( x̂ ) − 〈∇f |x̂ , x− x̂〉 ≥ µ ( d ( x , x̂ ) ) , proving the lemma . 2.1 The star algorithm . We now introduce our first main result , which extends the above analysis to the problem of learning in a non-convex class . Surprisingly , in the general setting of ( µ , d ) -convexity , an inequality matching ( 3 ) is always attained by a simple , two-step variant of empirical risk minimization . Let us now introduce the procedure . Given x ∈ S , the “ star hull ” of S with respect to x is defined by star ( x , S ) = { λx+ ( 1− λ ) s | s ∈ S , λ ∈ [ 0 , 1 ] } . The procedure may be summarized as follows . 1 : procedure STAR ( S , f ) 2 : Find x̂ that minimizes f over S 3 : Find x̃ that minimizes f over star ( x̂ , S ) 4 : Output x̃ 5 : end procedure Our next goal is to show that if S is an arbitrary ( not necessarily convex ) set , and x̃ is the output of the star algorithm , then f ( x ) − f ( x̃ ) ≥ µ ( 1 3d ( x , x̃ ) ) ( 4 ) whenever f is ( µ , d ) -convex . In particular , if f is the empirical risk and S is the function class , then x̃ is the “ star estimator ” introduced by Audibert ( 2008 ) . Thus , whenever ( µ , d ) -convexity leads to an upper bound for empirical risk minimization in a convex class via the inequality ( 3 ) , the same bound is enjoyed up to constants by the star algorithm in an arbitrary class . To prove ( 4 ) , recall how we proved Lemma 3 : we showed that if the line-segment λ 7→ zλ which linearly interpolates between any two points x and y lies entirely in the set fx \ f◦y , so f ( zλ ) ≥ f ( y ) , then we have that f ( x ) − f ( y ) ≥ µ ( d ( x , y ) ) . This is illustrated in Figure 1 . Unfortunately , the existence of such a line-segment zλ connecting x and x̃ relies on the convexity of S. In order to extend the argument as we want to , we make two observations . Firstly , the line segment λ 7→ zλ can be chosen to be any line segment connecting ∂fx to ∂fy and lying entirely in fx \ f◦y . Then , since ∂fx and ∂fy are level sets , the same argument yields f ( x ) − f ( y ) = f ( z1 ) − f ( z0 ) ≥ µ ( d ( z1 , z0 ) ) . ( 5 ) Secondly , we can show that there exist at most three such line-segments which together comprise a path from x to x̃ . By the triangle inequality , at least one of these segments has length greater than 1 3d ( x̃ , x ) . Combining this with ( 5 ) yields the desired inequality ( 4 ) . We state this as a proposition and prove it formally in Appendix A . A straightforward illustration of the proof , following the preceding discussion , is provided in Figure 2 . Proposition 4 . Let f be ( µ , d ) -convex . Suppose x̂ minimizes f over S and x̃ minimizes f over star ( x̂ , S ) . Then , for any x ∈ S , f ( x ) − f ( x̃ ) ≥ µ ( 1 3d ( x , x̃ ) ) .

Summary: The paper provides a unified analysis of fast rates in statistical learning. In particular, they provide excess risk bounds for the star-ERM solution for various loss functions by bounding the offset Rademacher complexity. In particular, they identify a general property: the suboptimality gap of a point with respect to the star-ERM solution is lower bound by distance (under a particular metric) between that point and the star-ERM solution. They derive this metric for various assumptions on the loss functions like uniform convexity, exp-concavity, self-concordance, etc. In terms of new results, using these techniques: (a) they can obtain fast rates for l_p losses for p > 2.  (b) they can obtain fast rates for exp-concave losses that depend on the range of the loss instead of its Lipschitz constant.