Surrogate Regret Bounds for Polyhedral Losses

1 Introduction . In supervised learning , our goal is to learn a hypothesis that solves some target problem given labeled data . The problem is often specified by a target loss ` that is discrete in the sense that the prediction lies in a finite set , as with classification , ranking , and structured prediction . As minimizing the target ` over a data set is typically NP-hard , we instead turn to the general paradigm of surrogate risk minimization . Here a surrogate loss function L is chosen and minimized instead of the target ` . A link function ψ then maps the surrogate predictions to target predictions . Naturally , a central question in surrogate risk minimization is how to choose the surrogate L. A minimal requirement is statistical consistency , which roughly says that approaching the best possible surrogate loss will eventually lead to the best possible target loss . A more precise goal is to understand the rate at which this target minimization occurs , called the generalization rate : how quickly the target loss approaches the best possible , in terms of the number of data points . In other words , if we define the regret of a hypothesis to be its performance relative to the Bayes optimal hypothesis , we wish to know how quickly the ` -regret approaches zero . Surrogate regret bounds are a common tool to prove generalization rates for surrogate risk minimization , by bounding the target regret as a function of the surrogate regret . This bound allows generalization rates for surrogate losses—of which there are many , often leveraging properties of L like strong convexity and smoothness—to imply rates for the target problem . Roughly speaking , a surrogate regret bound takes the form ∀ surrogate hypotheses h , Regret ` ( ψ ◦ h ) ≤ ζ ( RegretL ( h ) ) , ( 1 ) where the regret transfer function ζ : R+ → R+ controls how surrogate regret translates to target regret . For example , if we have a sequence of surrogate hypotheses hn achieving a fast rate RegretL ( hn ) = O ( 1/n ) , and we have a regret transfer of ζ ( ) = √ , then we can only guarantee a slower target rate of Regret ` ( ψ◦hn ) = O ( 1/ √ n ) . The ideal regret transfer is linear , i.e . ζ ( ) = C · for some C > 0 , which would have directly translated the fast-rate guarantee to the target . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Much is known about surrogate regret bounds for particular problems , especially for binary classification and related problems like bipartite ranking ; see § 1.1 for a discussion of the literature . General results which span multiple target problems , however , are sparse . Furthermore , we still lack guidance as to which surrogate will lead to the best overall target generalization rate , as this rate depends on both the surrogate generalization rate and the regret transfer function . For example , consider the choice between a polyhedral ( piecewise-linear and convex ) surrogate , like hinge loss , or a smooth surrogate like exponential or squared loss . The surrogate rate for polyhedral losses will likely be a slower O ( 1/ √ n ) , whereas a faster O ( 1/n ) rate is typical for smooth surrogates—but what is the tradeoff when it comes to the regret transfer function ζ , which ultimately decides the target generalization rate ? To answer these types of questions , we need more general results . Our first result is a general surrogate regret bound for polyhedral surrogates . Perhaps surprisingly , we show that every consistent surrogate in this class achieves a linear regret transfer , meaning their surrogate generalization rates translate directly to target rates , up to a constant factor . Theorem 1 ( Linear bound for polyhedral ) . Let L be any polyhedral surrogate and ψ any link function such that ( L , ψ ) is consistent for the target loss ` . Then L satisfies Equation 1 with a linear ζ . The recent embedding framework of Finocchiaro et al . [ 10 ] shows how to construct consistent polyhedral surrogates for any discrete target loss ; Theorem 1 additionally endows these surrogates with linear regret bounds . The same framework shows that every polyhedral loss is a consistent surrogate for some discrete target . Theorem 1 therefore immediately applies to the Weston–Watkins surrogate [ 31 ] , Lovász hinge [ 10 , 33 ] , and other polyhedral surrogates in the literature currently lacking regret bounds.1 We also give a partial converse : If the surrogate is sufficiently “ non-polyhedral ” , i.e . smooth and locally strongly convex , then ζ can not shrink faster than a square root . For these losses , therefore , even fast surrogate generalization rates would likely translate to slow rates for the target . Theorem 2 ( Lower bound for “ non-polyhedral ” ) . Let L be a locally strongly convex surrogate with locally Lipschitz gradient . If L satisfies Equation 1 for a target loss ` , then there exists c > 0 such that , for all small enough ≥ 0 , we have ζ ( ) ≥ c · √ . While it may seem intuitive that ζ = √ · for strongly convex losses , it is well-known that losses like exponential loss for binary classification also have ζ = √ · , even though they are neither strongly convex nor strongly smooth [ 4 ] . Theorem 2 offers an explanation for this phenomenon , and together with Theorem 1 clarifies an intriguing trend in the literature : ζ is typically either linear or square-root . Taken together , our results shed light on the dichotomy expressed above : slower polyhedral surrogate rates of O ( 1/ √ n ) will translate to the target unchanged , while most smooth surrogates , even if achieving a fast surrogate generalization rate of O ( 1/n ) , will likely yield the same target rate of O ( 1/ √ n ) . Our results therefore suggest an inherent tradeoff fast surrogate rates and optimal regret transfer , a phenomenon also observed by Mahdavi et al . [ 17 ] for binary classification . In particular , our results suggest that polyhedral losses may achieve an overall generalization rate to the target problem with the optimal dependence on the number of data points . The proof of our upper bound , Theorem 1 , is nonconstructive . After proving the main results ( Sections 3 and 4 ) , we make it constructive in Section 5 . Specifically , the regret transfer function for a polyhedral loss can be bounded by ζ ( t ) ≤ α · t where the constant is of the form α = α ` · αL · αψ . Each component of α depends only on its respective object ` , L , and ψ . A corollary of the derivation is that a polyhedral loss L and link ψ are consistent for ` if and only if ψ is -separated , a condition introduced in Finocchiaro et al . [ 10 ] . 1.1 Literature on surrogate regret bounds . Surrogate regret bounds were perhaps developed first by Zhang [ 37 ] , as a method to prove statistical consistency for various surrogate losses for binary classification . These results were then generalized by Bartlett et al . [ 4 ] for margin losses , and Reid and Williamson [ 26 ] for composite proper losses . 1Many of these polyhedral surrogates mentioned are actually known to be inconsistent for their intended target problems . Nonetheless , the embedding framework of Finocchiaro et al . [ 10 ] shows that any polyhedral surrogate is consistent for some target problem , namely the one that it embeds . It is to this embedded target loss function , rather than the intended target , that our linear regret bounds apply . These latter works and others give a fairly complete picture for binary classification : a characterization of calibrated surrogates , and precise closed forms for the optimal ζ for each . By extension , close variations on binary classification are also well-understood , such as cost-sensitive binary classification [ 28 ] , binary classification with an abstain/reject option [ 3 , 8 , 34 ] , bipartite ranking [ 1 , 13 , 18 ] , and ordinal regression via reduction to binary classification [ 22 ] . For multiclass classification , we have nice consistency characterizations with respect to 0-1 loss [ 30 ] , with some surrogate regret bounds [ 9 , 23 ] . Beyond binary classification and 0-1 loss , our knowledge drops off precipitously , with few general results . There are some works which provide partial understand for specific target problems . For example , in the problem of multiclass classification with an abstain option , Ramaswamy et al . [ 25 ] give three consistent polyhedral surrogates ( two of which are commonly used as inconsistent surrogates for 0-1 loss ) , along with linear surrogate regret bounds . ( See also Ni et al . [ 19 , Theorems 7 & 8 ] and Charoenphakdee et al . [ 6 , Theorem 8 ] for other surrogate regret bounds for the same problem . ) Ramaswamy et al . [ 24 ] show how to extend these surrogates to nested , hierarchical versions , again providing linear regret bounds . Motivated by these polyhedral surrogates , together with examples arising in top-k prediction [ 16 , 32 ] and other structured prediction problems [ 33 ] , Finocchiaro et al . [ 10 ] introduce an embedding framework for polyhedral surrogates . This framework shows that all polyhedral surrogates are consistent for some discrete target problem , and vice versa , but do not provide surrogate regret bounds . Our results fill this gap , giving linear regret bounds for all polyhedral losses , and thus all discrete target problems . Perhaps closest in spirit to our work are several recent works on general structured prediction problems . Ciliberto et al . [ 7 , Theorem 2 ] and Osokin et al . [ 21 , Theorem 7 ] give surrogate regret bounds for a quadratic surrogate , for a general class of structed prediction problems . Blondel [ 5 , Proposition 5 ] gives surrogate regret bounds for a tighter class of quadratic surrogates based on projection oracles . Nowak-Vila et al . [ 20 , Theorem 4.4 ] give even more general bounds , using a broader class of surrogates based on estimating a vector-valued expectation . All of these works apply to a broad range of discrete target problems . The main difference to our work is that their upper bounds are all for strongly convex surrogates,2 whereas ours are for polyhedral surrogates . Osokin et al . [ 21 , Theorem 8 ] and Nowak-Vila et al . [ 20 , Theorem 4.5 ] also provide lower bounds , in the same spirit as our Theorem 2 but only for the classes of surrogates they study . In particular , our results recover their bounds up to a constant factor . Finally , in the binary classification setting , some works have used surrogate regret bounds to study the relationship between generalization rates for the surrogate and target , as discussed above . Zhang et al . [ 36 ] give detailed results about how surrogate regret bounds actually convert the surrogate rate to the target rate , by carefully analyzing the behavior of the regret tranfer function near 0 . They argue that hinge loss achieves strictly better rates than smooth surrogates like logistic and exponential loss ; notably , though , this claim implicitly assumes that generalization rate is the same for all surrogates under consideration . By contrast , Mahdavi et al . [ 17 ] , as we do , consider that the surrogate generalization rate may depend on the choice of surrogate . They study a particular family of smoothed hinge losses , and show a direct tradeoff : smoother losses yield faster surrogate generalization rates , but “ slower ” regret transfers . Without placing further assumptions on the data , therefore , Mahdavi et al . [ 17 ] observe the same general phenomenon as we do : while smoothness may help surrogate generalization rates , it can hurt the regret transfer just as much . In particular , without further assumptions , polyhedral surrogates appear to achieve optimal overall generalization rates .

Summary: This paper provides general excess risk bounds result for a class of polyhedral losses for a general machine learning problem, where a prediction space and label space lie in a finite set. This covers many fundamental problems, e.g., classification, ranking, classification with rejection.  While such theoretical results have been quite well-studied for a family of strictly proper losses (a loss that can estimate the class probability (p(y|x)), less attention has been paid to other loss families. This paper shows that a polyhedral surrogate loss (i.e., piecewise-linear convex loss such as hinge loss) that is consistent with the target discrete loss has a linear excess risk transfer bound, which is desirable. Furthermore, the paper showed that a large class of non-polyhedral loss (including some strictly proper losses) cannot achieve the same result. On the other hand, such non-polyhedral losses have a slower rate than linear (square root).