Surrogate Regret Bounds for Polyhedral Losses

1 Introduction . In supervised learning , our goal is to learn a hypothesis that solves some target problem given labeled data . The problem is often specified by a target loss ` that is discrete in the sense that the prediction lies in a finite set , as with classification , ranking , and structured prediction . As minimizing the target ` over a data set is typically NP-hard , we instead turn to the general paradigm of surrogate risk minimization . Here a surrogate loss function L is chosen and minimized instead of the target ` . A link function ψ then maps the surrogate predictions to target predictions . Naturally , a central question in surrogate risk minimization is how to choose the surrogate L. A minimal requirement is statistical consistency , which roughly says that approaching the best possible surrogate loss will eventually lead to the best possible target loss . A more precise goal is to understand the rate at which this target minimization occurs , called the generalization rate : how quickly the target loss approaches the best possible , in terms of the number of data points . In other words , if we define the regret of a hypothesis to be its performance relative to the Bayes optimal hypothesis , we wish to know how quickly the ` -regret approaches zero . Surrogate regret bounds are a common tool to prove generalization rates for surrogate risk minimization , by bounding the target regret as a function of the surrogate regret . This bound allows generalization rates for surrogate losses—of which there are many , often leveraging properties of L like strong convexity and smoothness—to imply rates for the target problem . Roughly speaking , a surrogate regret bound takes the form ∀ surrogate hypotheses h , Regret ` ( ψ ◦ h ) ≤ ζ ( RegretL ( h ) ) , ( 1 ) where the regret transfer function ζ : R+ → R+ controls how surrogate regret translates to target regret . For example , if we have a sequence of surrogate hypotheses hn achieving a fast rate RegretL ( hn ) = O ( 1/n ) , and we have a regret transfer of ζ ( ) = √ , then we can only guarantee a slower target rate of Regret ` ( ψ◦hn ) = O ( 1/ √ n ) . The ideal regret transfer is linear , i.e . ζ ( ) = C · for some C > 0 , which would have directly translated the fast-rate guarantee to the target . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Much is known about surrogate regret bounds for particular problems , especially for binary classification and related problems like bipartite ranking ; see § 1.1 for a discussion of the literature . General results which span multiple target problems , however , are sparse . Furthermore , we still lack guidance as to which surrogate will lead to the best overall target generalization rate , as this rate depends on both the surrogate generalization rate and the regret transfer function . For example , consider the choice between a polyhedral ( piecewise-linear and convex ) surrogate , like hinge loss , or a smooth surrogate like exponential or squared loss . The surrogate rate for polyhedral losses will likely be a slower O ( 1/ √ n ) , whereas a faster O ( 1/n ) rate is typical for smooth surrogates—but what is the tradeoff when it comes to the regret transfer function ζ , which ultimately decides the target generalization rate ? To answer these types of questions , we need more general results . Our first result is a general surrogate regret bound for polyhedral surrogates . Perhaps surprisingly , we show that every consistent surrogate in this class achieves a linear regret transfer , meaning their surrogate generalization rates translate directly to target rates , up to a constant factor . Theorem 1 ( Linear bound for polyhedral ) . Let L be any polyhedral surrogate and ψ any link function such that ( L , ψ ) is consistent for the target loss ` . Then L satisfies Equation 1 with a linear ζ . The recent embedding framework of Finocchiaro et al . [ 10 ] shows how to construct consistent polyhedral surrogates for any discrete target loss ; Theorem 1 additionally endows these surrogates with linear regret bounds . The same framework shows that every polyhedral loss is a consistent surrogate for some discrete target . Theorem 1 therefore immediately applies to the Weston–Watkins surrogate [ 31 ] , Lovász hinge [ 10 , 33 ] , and other polyhedral surrogates in the literature currently lacking regret bounds.1 We also give a partial converse : If the surrogate is sufficiently “ non-polyhedral ” , i.e . smooth and locally strongly convex , then ζ can not shrink faster than a square root . For these losses , therefore , even fast surrogate generalization rates would likely translate to slow rates for the target . Theorem 2 ( Lower bound for “ non-polyhedral ” ) . Let L be a locally strongly convex surrogate with locally Lipschitz gradient . If L satisfies Equation 1 for a target loss ` , then there exists c > 0 such that , for all small enough ≥ 0 , we have ζ ( ) ≥ c · √ . While it may seem intuitive that ζ = √ · for strongly convex losses , it is well-known that losses like exponential loss for binary classification also have ζ = √ · , even though they are neither strongly convex nor strongly smooth [ 4 ] . Theorem 2 offers an explanation for this phenomenon , and together with Theorem 1 clarifies an intriguing trend in the literature : ζ is typically either linear or square-root . Taken together , our results shed light on the dichotomy expressed above : slower polyhedral surrogate rates of O ( 1/ √ n ) will translate to the target unchanged , while most smooth surrogates , even if achieving a fast surrogate generalization rate of O ( 1/n ) , will likely yield the same target rate of O ( 1/ √ n ) . Our results therefore suggest an inherent tradeoff fast surrogate rates and optimal regret transfer , a phenomenon also observed by Mahdavi et al . [ 17 ] for binary classification . In particular , our results suggest that polyhedral losses may achieve an overall generalization rate to the target problem with the optimal dependence on the number of data points . The proof of our upper bound , Theorem 1 , is nonconstructive . After proving the main results ( Sections 3 and 4 ) , we make it constructive in Section 5 . Specifically , the regret transfer function for a polyhedral loss can be bounded by ζ ( t ) ≤ α · t where the constant is of the form α = α ` · αL · αψ . Each component of α depends only on its respective object ` , L , and ψ . A corollary of the derivation is that a polyhedral loss L and link ψ are consistent for ` if and only if ψ is -separated , a condition introduced in Finocchiaro et al . [ 10 ] . 1.1 Literature on surrogate regret bounds . Surrogate regret bounds were perhaps developed first by Zhang [ 37 ] , as a method to prove statistical consistency for various surrogate losses for binary classification . These results were then generalized by Bartlett et al . [ 4 ] for margin losses , and Reid and Williamson [ 26 ] for composite proper losses . 1Many of these polyhedral surrogates mentioned are actually known to be inconsistent for their intended target problems . Nonetheless , the embedding framework of Finocchiaro et al . [ 10 ] shows that any polyhedral surrogate is consistent for some target problem , namely the one that it embeds . It is to this embedded target loss function , rather than the intended target , that our linear regret bounds apply . These latter works and others give a fairly complete picture for binary classification : a characterization of calibrated surrogates , and precise closed forms for the optimal ζ for each . By extension , close variations on binary classification are also well-understood , such as cost-sensitive binary classification [ 28 ] , binary classification with an abstain/reject option [ 3 , 8 , 34 ] , bipartite ranking [ 1 , 13 , 18 ] , and ordinal regression via reduction to binary classification [ 22 ] . For multiclass classification , we have nice consistency characterizations with respect to 0-1 loss [ 30 ] , with some surrogate regret bounds [ 9 , 23 ] . Beyond binary classification and 0-1 loss , our knowledge drops off precipitously , with few general results . There are some works which provide partial understand for specific target problems . For example , in the problem of multiclass classification with an abstain option , Ramaswamy et al . [ 25 ] give three consistent polyhedral surrogates ( two of which are commonly used as inconsistent surrogates for 0-1 loss ) , along with linear surrogate regret bounds . ( See also Ni et al . [ 19 , Theorems 7 & 8 ] and Charoenphakdee et al . [ 6 , Theorem 8 ] for other surrogate regret bounds for the same problem . ) Ramaswamy et al . [ 24 ] show how to extend these surrogates to nested , hierarchical versions , again providing linear regret bounds . Motivated by these polyhedral surrogates , together with examples arising in top-k prediction [ 16 , 32 ] and other structured prediction problems [ 33 ] , Finocchiaro et al . [ 10 ] introduce an embedding framework for polyhedral surrogates . This framework shows that all polyhedral surrogates are consistent for some discrete target problem , and vice versa , but do not provide surrogate regret bounds . Our results fill this gap , giving linear regret bounds for all polyhedral losses , and thus all discrete target problems . Perhaps closest in spirit to our work are several recent works on general structured prediction problems . Ciliberto et al . [ 7 , Theorem 2 ] and Osokin et al . [ 21 , Theorem 7 ] give surrogate regret bounds for a quadratic surrogate , for a general class of structed prediction problems . Blondel [ 5 , Proposition 5 ] gives surrogate regret bounds for a tighter class of quadratic surrogates based on projection oracles . Nowak-Vila et al . [ 20 , Theorem 4.4 ] give even more general bounds , using a broader class of surrogates based on estimating a vector-valued expectation . All of these works apply to a broad range of discrete target problems . The main difference to our work is that their upper bounds are all for strongly convex surrogates,2 whereas ours are for polyhedral surrogates . Osokin et al . [ 21 , Theorem 8 ] and Nowak-Vila et al . [ 20 , Theorem 4.5 ] also provide lower bounds , in the same spirit as our Theorem 2 but only for the classes of surrogates they study . In particular , our results recover their bounds up to a constant factor . Finally , in the binary classification setting , some works have used surrogate regret bounds to study the relationship between generalization rates for the surrogate and target , as discussed above . Zhang et al . [ 36 ] give detailed results about how surrogate regret bounds actually convert the surrogate rate to the target rate , by carefully analyzing the behavior of the regret tranfer function near 0 . They argue that hinge loss achieves strictly better rates than smooth surrogates like logistic and exponential loss ; notably , though , this claim implicitly assumes that generalization rate is the same for all surrogates under consideration . By contrast , Mahdavi et al . [ 17 ] , as we do , consider that the surrogate generalization rate may depend on the choice of surrogate . They study a particular family of smoothed hinge losses , and show a direct tradeoff : smoother losses yield faster surrogate generalization rates , but “ slower ” regret transfers . Without placing further assumptions on the data , therefore , Mahdavi et al . [ 17 ] observe the same general phenomenon as we do : while smoothness may help surrogate generalization rates , it can hurt the regret transfer just as much . In particular , without further assumptions , polyhedral surrogates appear to achieve optimal overall generalization rates .

Summary: The paper concerns theory of surrogate losses. The authors present general results concerning surrogate regret bounds which characterize how fast the target loss approaches the best possible solution if a surrogate loss is optimized. The authors point out that families of surrogate losses are characterized by different generalization rates (how fast the surrogate loss approaches optimum as a function of the number of training points) and regret transfer rates (how the surrogate regret translates to target regret). The authors focus on polyhedral losses (piecewise-linear convex loss functions; hinge loss is an examples of such function). Particularly, they show that the regret transfer for consistent polyhedral surrogates (a surrogate loss is consistent if its all optimal solutions are also optimal for the target loss) is linear (which is a desired rate). On the other hand, they show that for "sufficiently non-polyhedral" surrogates (like logistic loss) the transfer rate is no faster than a square root. Nevertheless, the generalization rate for smooth losses can be as fast as $\frac{1}{n}$, while for polyhedral losses is $\frac{1}{\sqrt{n}}$. All together it leads to comparable surrogate regret rates.