Title: A Universal Growth Rate for Learning with Smooth Surrogate Losses
Venue: NeurIPS 2024
Accepted: yes
Keywords: surrogate loss functions

Bayes-consistency

H-consistency bounds

excess error bounds

estimation error bounds

generalization bounds

learning theory
Abstract: This paper presents a comprehensive analysis of the growth rate of $H$-consistency bounds (and excess error bounds) for various surrogate losses used in classification. We prove a square-root growth rate near zero for smooth margin-based surrogate losses in binary classification, providing both upper and lower bounds under mild assumptions. This result also translates to excess error bounds. Our lower bound requires weaker conditions than those in previous work for excess error bounds, and our upper bound is entirely novel. Moreover, we extend this analysis to multi-class classification with a series of novel results, demonstrating a universal square-root growth rate for smooth *comp-sum* and *constrained losses*, covering common choices for training neural networks in multi-class classification. Given this universal rate, we turn to the question of choosing among different surrogate losses. We first examine how $H$-consistency bounds vary across surrogates based on the number of classes. Next, ignoring constants and focusing on behavior near zero, we identify *minimizability gaps* as the key differentiating factor in these bounds. Thus, we thoroughly analyze these gaps, to guide surrogate loss selection, covering: comparisons across different comp-sum losses, conditions where gaps become zero, and general conditions leading to small gaps.  Additionally, we demonstrate the key role of minimizability gaps in comparing excess error bounds and $H$-consistency bounds.

A Universal Growth Rate for Learning with Smooth Surrogate Losses

Abstract

This paper presents a comprehensive analysis of the growth rate of H -consistency bounds (and excess error bounds) for various surrogate losses used in classification. We prove a square-root growth rate near zero for smooth margin-based surrogate losses in binary classification, providing both upper and lower bounds under mild assumptions. This result also translates to excess error bounds. Our lower bound requires weaker conditions than those in previous work for excess error bounds, and our upper bound is entirely novel. Moreover, we extend this analysis to multi-class classification with a series of novel results, demonstrating a universal square-root growth rate for smooth comp-sum and constrained losses , covering common choices for training neural networks in multi-class classification. Given this universal rate, we turn to the question of choosing among different surrogate losses. We first examine how H -consistency bounds vary across surrogates based on the number of classes. Next, ignoring constants and focusing on behavior near zero, we identify minimizability gaps as the key differentiating factor in these bounds. Thus, we thoroughly analyze these gaps, to guide surrogate loss selection, covering: comparisons across different comp-sum losses, conditions where gaps become zero, and general conditions leading to small gaps. Additionally, we demonstrate the key role of minimizability gaps in comparing excess error bounds and H -consistency bounds.

1 Introduction

Learning algorithms frequently optimize surrogate loss functions like the logistic loss, in lieu of the task's true objective, commonly the zero-one loss. This is necessary when the original loss function is computationally intractable to optimize or lacks essential mathematical properties such as differentiability. But, what guarantees can we rely on when minimizing a surrogate loss? This is a fundamental question with significant implications for learning.

The related property of Bayes-consistency of surrogate losses has been extensively studied in the context of binary classification. Zhang [2004a], Bartlett et al. [2006] and Steinwart [2007] established Bayes-consistency for various convex loss functions, including margin-based surrogates. They also introduced excess error bounds (or surrogate regret bounds) for margin-based surrogates. Reid and Williamson [2009] extended these results to proper losses in binary classification.

The Bayes-consistency of several surrogate loss function families in the context of multi-class classification has also been studied by Zhang [2004b] and Tewari and Bartlett [2007]. Zhang [2004b] established a series of results for various multi-class classification formulations, including negative results for multi-class hinge loss functions [Crammer and Singer, 2001], as well as positive results for the sum exponential loss [Weston and Watkins, 1999, Awasthi et al., 2022a], the (multinomial) logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], and the constrained losses [Lee et al., 2004].

Later, Tewari and Bartlett [2007] adopted a different geometric method to analyze Bayes-consistency, yielding similar results for these loss function families. Steinwart [2007] developed general tools to characterize Bayes consistency for both binary and multi-class classification. Additionally, excess error bounds have been derived by Pires et al. [2013] for a family of constrained losses and by Duchi et al. [2018] for loss functions related to generalized entropies.

For a surrogate loss glyph[lscript] , an excess error bound holds for any predictor h and has the form E glyph[lscript] 0 -1 ( h ) -E ∗ glyph[lscript] 0 -1 ≤ Ψ ( E glyph[lscript] ( h ) -E ∗ glyph[lscript] ) , where E glyph[lscript] 0 -1 ( h ) and E glyph[lscript] ( h ) represent the expected losses of h for the zero-one loss and surrogate loss respectively, E ∗ glyph[lscript] 0 -1 and E ∗ glyph[lscript] the Bayes errors for the zero-one and surrogate loss respectively, and Ψ a non-decreasing function. The growth rate of excess error bounds, that is the behavior of function Ψ near zero, has gained attention in recent research [Mahdavi et al., 2014, Zhang et al., 2021, Frongillo and Waggoner, 2021, Bao, 2023]. Mahdavi et al. [2014] examined the growth rate for smoothed hinge losses in binary classification, demonstrating that smoother losses result in worse growth rates. The optimal rate is achieved with the standard hinge loss, which exhibits linear growth. Zhang et al. [2021] tied the growth rate of excess error bounds in binary classification to two properties of the surrogate loss function: consistency intensity and conductivity. These metrics enable comparisons of growth rates across different surrogates. This prompts a natural question: can we establish rigorous lower and upper bounds for excess error growth rates under specific regularity conditions?

Frongillo and Waggoner [2021] pioneered research on this question in binary classification settings. They established a critical square-root lower bound for excess error bounds when a surrogate loss is locally strongly convex and has a locally Lipschitz gradient. Additionally, they demonstrated a linear excess error bound for Bayes-consistent polyhedral loss functions (convex and piecewise-linear) [Finocchiaro et al., 2019] (see also [Lapin et al., 2016, Ramaswamy et al., 2018, Yu and Blaschko, 2018, Yang and Koyejo, 2020]). More recently, Bao [2023] complemented these results by showing that proper losses associated with Shannon entropy, exponential entropy, spherical entropy, squared α -norm entropies and α -polynomial entropies, with α > 1 , also exhibit a square-root lower bound for excess error bounds relative to the glyph[lscript] 1 -distance.

However, while Bayes-consistency and excess error bounds are valuable, they are not sufficiently informative, as they are established for the family of all measurable functions and disregard the crucial role played by restricted hypothesis sets in learning. As pointed out by Long and Servedio [2013], in some cases, minimizing Bayes-consistent losses can result in constant expected error, while minimizing inconsistent losses can yield an expected loss approaching zero. To address this limitation, the authors introduced the concept of realizable H -consistency , further explored by Kuznetsov et al. [2014] and Zhang and Agarwal [2020]. Nonetheless, these guarantees are only asymptotic and rely on a strong realizability assumption that typically does not hold in practice.

Recent research by Awasthi, Mao, Mohri, and Zhong [2022b,a] and Mao, Mohri, and Zhong [2023f,c,e,b] has instead introduced and analyzed H -consistency bounds . These bounds are more informative than Bayes-consistency since they are hypothesis set-specific and non-asymptotic. Their work covers broad families of surrogate losses in binary classication, multi-class classification, structured prediction, and abstention [Mao, Mohri, Mohri, and Zhong, 2023a]. Crucially, they provide upper bounds on the estimation error of the target loss, for example, the zero-one loss in classification, that hold for any predictor h within a hypothesis set H . These bounds relate this estimation error to the surrogate loss estimation error.

Their general form is: E glyph[lscript] 0 -1 ( h ) -E ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) ≤ Γ ( E glyph[lscript] ( h ) -E ∗ glyph[lscript] ( H ) + M glyph[lscript] ( H )) , where E ∗ glyph[lscript] 0 -1 ( H ) and E ∗ glyph[lscript] ( H ) represent the best-in-class expected losses for the zero-one and surrogate loss respectively, Γ is a non-negative concave function and M glyph[lscript] 0 -1 ( H ) and M glyph[lscript] ( H ) are minimizability gaps . The exact definition of these gaps will be detailed later. For now, let us mention that they are non-negative quantities, upper-bounded by the approximation error of their respective loss functions. H -consistency bounds subsume excess error bounds as a special case when the hypothesis set is expanded to include all measurable functions, in which case the minimizability gaps vanish. More generally, an H -consistency bound with a Γ function implies E glyph[lscript] 0 -1 ( h ) -E ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) ≤ Γ ( E glyph[lscript] ( h ) -E ∗ glyph[lscript] ( H )) + Γ ( M glyph[lscript] ( H )) since a concave function Γ with Γ ( 0 ) ≥ 0 is sub-additive over R + . Thus, when the surrogate estimation loss E glyph[lscript] ( h ) -E ∗ glyph[lscript] ( H ) is minimized to glyph[epsilon1] , the zero-one estimation error E glyph[lscript] 0 -1 ( h ) -E ∗ glyph[lscript] 0 -1 ( H ) is bounded by Γ ( glyph[epsilon1] ) + Γ ( M glyph[lscript] ( H )) -M glyph[lscript] 0 -1 ( H ) . Can we characterize the growth rate of H -consistency bounds, that is how quickly the functions Γ increase near zero?

Our results . This paper presents a comprehensive analysis of the growth rate of H -consistency bounds for all margin-based surrogate losses in binary classification, as well as for comp-sum losses and constrained losses in multi-class classification. We establish a square-root growth rate near zero for margin-based surrogate losses glyph[lscript] defined by glyph[lscript] ( h, x, y ) = Φ (-yh ( x )) , assuming only that Φ is convex and twice continuously differentiable with Φ ′ ( 0 )> 0 and Φ ′′ ( 0 )> 0 (Section 4). This includes both upper and lower bounds (Theorem 4.2). These results directly apply to excess error bounds as well. Importantly, our lower bound requires weaker conditions than [Frongillo and Waggoner, 2021, Theorem 4], and our upper bound is entirely novel. This work demonstrates that the H -consistency bound growth rate for these loss functions is precisely square-root, refining the 'at least square-root' finding of these authors (for excess error bounds). It is known that polyhedral losses admit a linear grow rate [Frongillo and Waggoner, 2021]. Thus, a striking dichotomy emerges that reflects previous observations by these authors: H -consistency bounds for polyhedral losses exhibit a linear growth rate in binary classification, while they follow a square-root rate for smooth loss functions.

Moreover, we significantly extend our findings to key multi-class surrogate loss families, including comp-sum losses [Mao et al., 2023f] (e.g., logistic loss or cross-entropy with softmax [Berkson, 1944], sum-losses [Weston and Watkins, 1999], generalized cross entropy loss [Zhang and Sabuncu, 2018]), and constrained losses [Lee et al., 2004, Awasthi et al., 2022a] (Section 5). In Section 5.1, we prove that the growth rate of H -consistency bounds for comp-sum losses is exactly square-root. This applies when the auxiliary function Φ they are based upon is convex and twice continuously differentiable with Φ ′ ( u )< 0 and Φ ′′ ( u )> 0 for all u in ( 0 , 1 2 ] . These conditions hold for all common loss functions used in practice. Further, in Section 5.2, we demonstrate that the square-root growth rate also extends to H -consistency bounds for constrained losses. This requires the auxiliary function Φ to be convex and twice continuously differentiable with Φ ′ ( u )> 0 and Φ ′′ ( u )> 0 for any u ≥ 0 , alongside an additional technical condition. These are satisfied by all constrained losses typically encountered in practice.

These results reveal a universal square-root growth rate for smooth surrogate losses, the predominant choice in neural network training (over polyhedral losses) for both binary and multi-class classification in applications. Given this universal growth rate, how do we choose between different surrogate losses? Section 6 addresses this question in detail. To start, we examine how H -consistency bounds vary across surrogates based on the number of classes. Then, focusing on behavior near zero (ignoring constants), we isolate minimizability gaps as the key differentiating factor in these bounds. These gaps depend solely on the chosen surrogate loss and hypothesis set. We provide a detailed analysis of minimizability gaps, covering: comparisons across different comp-sum losses, conditions where gaps become zero, and general conditions leading to small gaps. These findings help guide surrogate loss selection. Additionally, we demonstrate the key role of minimizability gaps in comparing excess error bounds and H -consistency bounds (Appendix F). Importantly, combining H -consistency bounds with surrogate loss Rademacher complexity bounds allows us to derive zero-one loss (estimation) learning bounds for surrogate loss minimizers (Appendix O).

For a more comprehensive discussion of related work, please refer to Appendix A. We start with the introduction of necessary concepts and definitions.

2 Preliminaries

Notation and definitions. We denote the input space by X and the label space by Y , a finite set of cardinality n with elements { 1 , . . . , n } . D denotes a distribution over X × Y .

We write H all to denote the family of all real-valued measurable functions defined over X × Y and denote by H a subset, H ⊆ H all . The label assigned by h ∈ H to an input x ∈ X is denoted by h ( x ) and defined by h ( x ) = argmax y ∈ Y h ( x, y ) , with an arbitrary but fixed deterministic strategy used for breaking the ties. For simplicity, we fix that strategy to be the one selecting the label with the highest index under the natural ordering of labels.

We will consider general loss functions glyph[lscript] ∶ H × X × Y → R + . For many loss functions used in practice, the loss value at ( x, y ) , glyph[lscript] ( h, x, y ) , only depends on the value h takes at x and not on its values on other points. That is, there exists a measurable function ˆ glyph[lscript] ∶ R n × Y → R + such that glyph[lscript] ( h, x, y ) = ˆ glyph[lscript] ( h ( x ) , y ) , where h ( x ) = [ h ( x, 1 ) , . . . , h ( x, n )] is the score vector of the predictor h . We will then say that glyph[lscript] is a pointwise loss function . We denote by E glyph[lscript] ( h ) the generalization error or expected loss of a hypothesis h ∈ H and by E ∗ glyph[lscript] ( H ) the best-in class error : E glyph[lscript] ( h ) =

E ( x,y )∼ D [ glyph[lscript] ( h, x, y )] , E ∗ glyph[lscript] ( H ) = inf h ∈ H E glyph[lscript] ( h ) . E ∗ glyph[lscript] ( H all ) is also known as the Bayes error . We write p ( y ∣ x ) = D ( Y = y ∣ X = x ) to denote the conditional probability of Y = y given X = x and p ( x ) = ( p ( 1 ∣ x ) , . . . , p ( n ∣ x )) for the conditional probability vector for any x ∈ X . We denote by C glyph[lscript] ( h, x ) the conditional error of h ∈ H at a point x ∈ X and by C ∗ glyph[lscript] ( H , x ) the best-in-class conditional error : C glyph[lscript] ( h, x ) = E y [ glyph[lscript] ( h, x, y ) ∣ x ] = ∑ y ∈ Y p ( y ∣ x ) glyph[lscript] ( h, x, y ) , C ∗ glyph[lscript] ( H , x ) = inf h ∈ H C glyph[lscript] ( h, x ) , and use the shorthand ∆ C glyph[lscript], H ( h, x ) = C glyph[lscript] ( h, x ) -C ∗ glyph[lscript] ( H , x ) for the calibration gap or conditional regret for glyph[lscript] . The generalization error of h can be written as E glyph[lscript] ( h ) = E x [ C glyph[lscript] ( h, x )] . For convenience, we also define, for any vector p = ( p 1 , . . . , p n ) ∈ ∆ n , where ∆ n is the probability simplex of R n , C glyph[lscript] ( h, x, p ) = ∑ y ∈ Y p y glyph[lscript] ( h, x, y ) , C ∗ glyph[lscript], H ( x, p ) = inf h ∈ H C glyph[lscript] ( h, x, p ) and ∆ C glyph[lscript], H ( h, x, p ) = C glyph[lscript] ( h, x, p ) -C ∗ glyph[lscript], H ( x, p ) . Thus, we have ∆ C glyph[lscript], H ( h, x, p ( x )) = ∆ C glyph[lscript], H ( h, x ) .

We will study the properties of a surrogate loss function glyph[lscript] 1 for a target loss function glyph[lscript] 2 . In multi-class classification, glyph[lscript] 2 is typically the zero-one multi-class classification loss function glyph[lscript] 0 -1 defined by glyph[lscript] 0 -1 ( h, x, y ) = 1 h ( x )≠ y . Some surrogate loss functions glyph[lscript] 1 include the max losses [Crammer and Singer, 2001], comp-sum losses [Mao et al., 2023f] and constrained losses [Lee et al., 2004].

Binary classification. The definitions just presented were given for the general multi-class classification setting. In the special case of binary classification (two classes), the standard formulation and definitions are slightly different. For convenience, the label space is typically defined as Y = {-1 , + 1 } . Instead of two scoring functions, one for each label, a single real-valued function is used whose sign determines the predicted class. Thus, here, a hypothesis set H is a family of measurable real-valued functions defined over X and H all is the family of all such functions. glyph[lscript] is pointwise if there exists a measurable function ˆ glyph[lscript] ∶ R × Y → R + such that glyph[lscript] ( h, x, y ) = ˆ glyph[lscript] ( h ( x ) , y ) . The target loss function is typically the binary loss glyph[lscript] 0 -1 , defined by glyph[lscript] 0 -1 ( h, x, y ) = 1 sign ( h ( x ))≠ y , where sign ( h ( x )) = 1 h ( x )≥ 0 -1 h ( x )< 0 . Some widely used surrogate losses glyph[lscript] 1 for glyph[lscript] 0 -1 are margin-based losses, which are defined by glyph[lscript] 1 ( h, x, y ) = Φ (-yh ( x )) , for some non-decreasing convex function Φ ∶ R → R + . Instead of two conditional probabilities, one for each label, a single conditional probability corresponding to the positive class + 1 is used. That is, let η ( x ) = D ( Y = + 1 ∣ X = x ) denote the conditional probability of Y = + 1 given X = x . The conditional error can then be expressed as:

For convenience, we also define, for any p ∈ [ 0 , 1 ] , C glyph[lscript] ( h, x, p ) = pglyph[lscript] ( h, x, + 1 ) + ( 1 -p ) glyph[lscript] ( h, x, -1 ) , C ∗ glyph[lscript], H ( x, p ) = inf h ∈ H C glyph[lscript] ( h, x, p ) and ∆ C glyph[lscript], H ( h, x, p ) = C glyph[lscript] ( h, x, p ) -inf h ∈ H C glyph[lscript] ( h, x, p ) . Thus, we have ∆ C glyph[lscript], H ( h, x, η ( x )) = ∆ C glyph[lscript], H ( h, x ) .

To simplify matters, we will use the same notation for binary and multi-class classification, such as Y for the label space or H for a hypothesis set. We rely on the reader to adapt to the appropriate definitions based on the context.

Estimation, approximation, and excess errors. For a hypothesis h , the difference E glyph[lscript] ( h )-E ∗ glyph[lscript] ( H all ) is known as the excess error . It can be decomposed into the sum of two terms, the estimation error , ( E glyph[lscript] ( h ) -E ∗ glyph[lscript] ( H )) and the approximation error A glyph[lscript] ( H ) = ( E ∗ glyph[lscript] ( H ) -E ∗ glyph[lscript] ( H all )) :

A fundamental result for a pointwise loss function glyph[lscript] is that the Bayes error and the approximation error admit the following simpler expressions. We give a concise proof of this lemma in Appendix B, where we establish the measurability of the function x ↦ C ∗ glyph[lscript] ( H all , x ) .

Lemma 2.1. Let glyph[lscript] be a pointwise loss function. Then, the Bayes error and the approximation error can be expressed as follows: E ∗ glyph[lscript] ( H all ) = E x [ C ∗ glyph[lscript] ( H all , x )] and A glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E x [ C ∗ glyph[lscript] ( H all , x )] .

For restricted hypothesis sets ( H ≠ H all ), the infimum's super-additivity implies that E ∗ glyph[lscript] ( H ) ≥ E x [ C ∗ glyph[lscript] ( H , x )] . This inequality is generally strict, and the difference, E ∗ glyph[lscript] ( H ) -E x [ C ∗ glyph[lscript] ( H , x )] , plays a crucial role in our analysis.

3 H -consistency bounds

A widely used notion of consistency is that of Bayes-consistency given below [Steinwart, 2007].

Definition 3.1 ( Bayes-consistency ) . Aloss function glyph[lscript] 1 is Bayes-consistent with respect to a loss function glyph[lscript] 2 , if for any distribution D and any sequence { h n } n ∈ N ⊂ H all , lim n →+∞ E glyph[lscript] 1 ( h n )-E ∗ glyph[lscript] 1 ( H all ) = 0 implies lim n →+∞ E glyph[lscript] 2 ( h n ) -E ∗ glyph[lscript] 2 ( H all ) = 0 .

Thus, when this property holds, asymptotically, a nearly optimal minimizer of glyph[lscript] 1 over the family of all measurable functions is also a nearly optimal optimizer of glyph[lscript] 2 . But, Bayes-consistency does not supply any information about a hypothesis set H not containing the full family H all , that is a typical hypothesis set used for learning. Furthermore, it is only an asymptotic property and provides no convergence guarantee. In particular, it does not give any guarantee for approximate minimizers. Instead, we will consider upper bounds on the target estimation error expressed in terms of the surrogate estimation error, H -consistency bounds [Awasthi et al., 2022b,a, Mao et al., 2023f], which account for the hypothesis set H adopted.

Definition 3.2 ( H -consistency bounds ) . Given a hypothesis set H , an H -consistency bound relating the loss function glyph[lscript] 1 to the loss function glyph[lscript] 2 for a hypothesis set H is an inequality of the form

that holds for any distribution D , where Γ ∶ R + → R + is a non-decreasing concave function with Γ ≥ 0 [Awasthi et al., 2022b,a]. Here, M glyph[lscript] 1 ( H ) and M glyph[lscript] 2 ( H ) are minimizability gaps for the respective loss functions. The minimizability gap for a hypothesis set H and loss function glyph[lscript] is denoted by M glyph[lscript] ( H ) and defined as: M glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E x [ C ∗ glyph[lscript] ( H , x )] . It quantifies the discrepancy between the best possible expected loss within a hypothesis class and the expected infimum of pointwise expected losses. This gap is always non-negative: M glyph[lscript] ( H ) = inf h ∈ H E x [ C glyph[lscript] ( h, x )] -E x [ inf h ∈ H C glyph[lscript] ( H , x )] ≥ 0 , by the infimum's super-additivity, and is bounded above by the approximation error A glyph[lscript] ( H ) = inf h ∈ H E x [ C glyph[lscript] ( h, x )] -E x [ inf h ∈ H all C glyph[lscript] ( H , x )] . We further study the key role of minimizability gaps in H -consistency bounds and their properties in Section 6 and Appendix D. As shown in Appendix C, under general assumptions, minimizability gaps are essential quantities required in any bound that relates the estimation errors of two loss functions with an arbitrary hypothesis set H .

Thus, an H -consistency bound provides the guarantee that when the surrogate estimation loss E glyph[lscript] ( h ) -E ∗ glyph[lscript] ( H ) is minimized to glyph[epsilon1] , the following upper bound holds for the zero-one estimation error:

where the second inequality follows from the sub-additivity of a concave function Γ over R + . We will demonstrate that, for smooth surrogate losses, Γ ( glyph[epsilon1] ) scales as √ glyph[epsilon1] . Note, however, that, while Γ ( glyph[epsilon1] ) tends to zero when glyph[epsilon1] → 0 for functions Γ derived in H -consistency bounds, the remaining terms in the bound are constant. This is not surprising as, in general, minimizing the surrogate estimation error to zero cannot guarantee that the zero-one estimation error will also converge to zero. This is well-known, for example, in the case of linear models [Ben-David et al., 2012]. Instead, an H -consistency bound provides the tightest possible upper bound on the estimation error for the zero-one loss when the surrogate estimation error is minimized.

The upper bound simplifies to Γ ( glyph[epsilon1] ) when the minimizability gaps are zero, which occurs when either H = H all (the set of all measurable functions) or in realizable cases, which are particularly relevant to the practical use of complex neural networks in applications. In Appendix I, we examine more general cases of small minimizability gaps, taking into account the complexity of H and the distribution.

Our results cover in particular the special case of excess bounds ( H = H all ). Let us emphasize that, for H ≠ H all , H -consistency bounds offer tighter and more favorable guarantees on the estimation error compared to those derived from excess bounds analysis alone (see Appendix F).

When glyph[lscript] 2 = glyph[lscript] 0 -1 , the zero-one loss, we say that T is the H -estimation error transformation function of a surrogate loss glyph[lscript] if the following holds:

and the bound is tight . That is, for any t ∈ [ 0 , 1 ] , there exists a hypothesis h ∈ H and a distribution such that E glyph[lscript] 0 -1 ( h )-E ∗ glyph[lscript] 0 -1 ( H )+ M glyph[lscript] 0 -1 ( H ) = t and E glyph[lscript] ( h )-E ∗ glyph[lscript] ( H )+ M glyph[lscript] ( H ) = T ( t ) . An explicit form of T has been characterized for binary margin-based losses [Awasthi et al., 2022b], as well as compsum losses and constrained losses in multi-class classification [Mao et al., 2023b]. In the following sections, we will prove the property T ( t ) = Θ ( t 2 ) (under mild assumptions), demonstrating a squareroot growth rate for H -consistency bounds. Appendix E provides examples of H -consistency bounds for both binary and multi-class classification. Our analysis also suggests choosing appropriately H and the function Γ to ensure a small minimizability gap and to take into account the number of classes and other properties, as discussed in Section 6.

4 Binary classification

We consider the broad family of margin-based loss functions glyph[lscript] defined for any h ∈ H , and ( x, y ) ∈ X × Y by glyph[lscript] ( h, x, y ) = Φ (-yh ( x )) , where Φ is a non-decreasing convex function upper-bounding the zero-one loss. Margin-based loss functions include most loss functions used in binary classification. As an example, Φ ( u ) = log ( 1 + e u ) for the logistic loss or Φ ( u ) = exp ( u ) for the exponential loss. We say that a hypothesis set H is complete , if for all x ∈ X , we have { h ( x )∶ h ∈ H } = R . As shown by Awasthi et al. [2022b], the transformation T has the following form for complete hypothesis sets:

Here, for any t ∈ [ 0 , 1 ] , f t is defined by: ∀ u ∈ R , f t ( u ) = 1 -t 2 Φ ( u ) + 1 + t 2 Φ (-u ) . The following result is useful for proving the growth rate in binary classification.

Theorem 4.1. Let H be a complete hypothesis set. Assume that Φ is convex and differentiable at zero and satisfies the inequality Φ ′ ( 0 ) > 0 . Then, the transformation T can be expressed as follows:

Proof. By the convexity of Φ , for any t ∈ [ 0 , 1 ] and u ∈ R -, we have

Thus, we can write T ( t ) = inf u ≤ 0 f t ( u )-inf u ∈ R f t ( u ) ≥ Φ ( 0 )-inf u ∈ R f t ( u ) = f t ( 0 )-inf u ∈ R f t ( u ) , where equality is achieved when u = 0 .

Theorem 4.2 (Upper and lower bound for binary margin-based losses) . Let H be a complete hypothesis set. Assume that Φ is convex, twice continuously differentiable, and satisfies the inequalities Φ ′ ( 0 ) > 0 and Φ ′′ ( 0 ) > 0 . Then, the following property holds: T ( t ) = Θ ( t 2 ) ; that is, there exist positive constants C > 0 , c > 0 , and T > 0 such that Ct 2 ≥ T ( t ) ≥ ct 2 , for all 0 < t ≤ T .

Proof sketch First, we demonstrate that, by applying the implicit function theorem, inf u ∈ R f t ( u ) is attained uniquely by a ∗ t , and that a ∗ t is continuously differentiable over [ 0 , glyph[epsilon1] ] for some glyph[epsilon1] > 0 . The minimizer a ∗ t satisfies the following condition: f ′ t ( a ∗ t ) = 1 -t 2 Φ ′ ( a ∗ t ) -1 + t 2 Φ ′ (-a ∗ t ) = 0 . Specifically, at t = 0 , we have Φ ′ ( a ∗ 0 ) = Φ ′ (-a ∗ 0 ) . Then, by the convexity of Φ and monotonicity of the derivative Φ ′ , we must have a ∗ 0 = 0 and since Φ ′ is non-decreasing and Φ ′′ ( 0 ) > 0 , we have a ∗ t > 0 for all t ∈ ( 0 , glyph[epsilon1] ] . Furthermore, since a ∗ t is a function of class C 1 , we can differentiate this condition with respect to t and take the limit t → 0 , which gives the following equality: da ∗ t dt ( 0 ) = Φ ′ ( 0 ) Φ ′′ ( 0 ) > 0 . Since lim t → 0 a ∗ t t = da ∗ t dt ( 0 ) = Φ ′ ( 0 ) Φ ′′ ( 0 ) > 0 , we have a ∗ t = Θ ( t ) . By Theorem 4.1 and Taylor's theorem with an integral remainder, T can be expressed as follows: for any t ∈ [ 0 , glyph[epsilon1] ] , T ( t ) = f t ( 0 ) -inf u ∈ R f t ( u ) = ∫ a ∗ t 0 uf ′′ t ( u ) du = ∫ a ∗ t 0 u [ 1 -t 2 Φ ′′ ( u ) + 1 + t 2 Φ ′′ (-u )] du . Since Φ ′′ ( 0 ) > 0 and Φ ′′ is continuous, there is a non-empty interval [-α, + α ] over which Φ ′′ is positive. Since a ∗ 0 = 0 and a ∗ t is continuous, there exists a sub-interval [ 0 , glyph[epsilon1] ′ ] ⊆ [ 0 , glyph[epsilon1] ] over which a ∗ t ≤ α . Since Φ ′′ is continuous, it admits a minimum and a maximum over any compact set and we can define c = min u ∈[-α,α ] Φ ′′ ( u ) and C = max u ∈[-α,α ] Φ ′′ ( u ) . c and C are both positive since we have Φ ′′ ( 0 ) > 0 . Thus, for t in [ 0 , glyph[epsilon1] ′ ] , the following inequality holds: C ( a ∗ t ) 2 2 = ∫ a ∗ t 0 uC du ≥ T ( t ) = ∫ a ∗ t 0 u [ 1 -t 2 Φ ′′ ( u ) + 1 + t 2 Φ ′′ (-u )] du ≥ ∫ a ∗ t 0 ucdu = c ( a ∗ t ) 2 2 . This implies that T ( t ) = Θ ( t 2 ) . The full proof is included in Appendix K.

Theorem 4.2 directly applies to excess error bounds as well, when H = H all . Importantly, our lower bound requires weaker conditions than [Frongillo and Waggoner, 2021, Theorem 4], and our upper bound is entirely novel. This result demonstrates that the growth rate for these loss functions is precisely square-root, refining the 'at least square-root' finding of these authors. It is known that polyhedral losses admit a linear grow rate [Frongillo and Waggoner, 2021]. Thus, a striking dichotomy emerges: H -consistency bounds for polyhedral losses exhibit a linear growth rate, while they follow a square-root rate for smooth loss functions (see Appendix G for a detailed comparison).

5 Multi-class classification

In this section, we will study two families of surrogate losses in multi-class classification: comp-sum losses and constrained losses, defined in Section 5.1 and Section 5.2 respectively. Comp-sum losses and constrained losses are general and cover all loss functions commonly used in practice. We will consider any hypothesis set H that is symmetric and complete . We say that a hypothesis set is symmetric when it does not depend on a specific ordering of the classes, that is, when there exists a family F of functions f mapping from X to R such that {[ h ( x, 1 ) , . . . , h ( x, n )]∶ h ∈ H } = {[ f 1 ( x ) , . . . , f n ( x )]∶ f 1 , . . . , f n ∈ F } , for any x ∈ X . We say that a hypothesis set H is complete if the set of scores it generates spans R , that is, { h ( x, y )∶ h ∈ H } = R , for any ( x, y ) ∈ X × Y .

5.1 Comp-sum losses

Here, we consider comp-sum losses [Mao, Mohri, and Zhong, 2023f], defined as

where Φ ∶ R → R + is a non-increasing function. For example, Φ can be chosen as the negative log function u ↦ -log ( u ) for the comp-sum losses, which leads to the multinomial logistic loss. As shown by Mao, Mohri, and Zhong [2023b], for symmetric and complete hypothesis sets, the transformation T for the family of comp-sum losses can be characterized as follows.

Theorem 5.1 (Mao et al. [2023b, Theorem 3]) . Let H be a symmetric and complete hypothesis set. Assume that Φ is convex, differentiable at 1 2 and satisfies the inequality Φ ′ ( 1 2 ) < 0 . Then, the transformation T can be expressed as

Next, we will show that as with the binary case, for the comp-sum losses, the properties T ( t ) = Ω ( t 2 ) and T ( t ) = O ( t 2 ) hold. We first introduce a generalization of the classical implicit function theorem where the function takes the value zero over a set of points parameterized by a compact set. We treat the special case of a function F defined over R 3 and denote by ( t, a, τ ) ∈ R 3 its arguments. The theorem holds more generally for the arguments being in R n 1 × R n 2 × R n 3 and with the condition on the partial derivative being non-zero replaced with a partial Jacobian being non-singular.

Theorem 5.2 (Implicit function theorem with a compact set) . Let F ∶ R × R × R → R be a continuously differentiable function in a neighborhood of ( 0 , 0 , τ ) , for any τ in a non-empty compact set C , with F ( 0 , 0 , τ ) = 0 . Then, if ∂F ∂a ( 0 , 0 , τ ) is non-zero for all τ in C , then, there exist a neighborhood O of 0 and a unique function ¯ a defined over O × C that is continuously differentiable and satisfies

Proof. By the implicit function theorem (see for example [Dontchev and Rockafellar, 2009]), for any τ ∈ C , there exists an open set U τ = (-t τ , + t τ ) × ( τ -glyph[epsilon1] τ , τ + glyph[epsilon1] τ ) , ( t τ > 0 and glyph[epsilon1] τ > 0 ), and a unique function ¯ a τ ∶ U τ → R that is in C 1 and such that for all ( t, τ ) ∈ U τ , F ( t, ¯ a τ ( t ) , τ ) = 0 .

By the uniqueness of ¯ a τ , for any τ ≠ τ ′ and ( t 1 , τ 1 ) ∈ U τ ∩ U τ ′ , we have ¯ a τ ( t 1 , τ 1 ) = ¯ a τ ′ ( t 1 , τ 1 ) . Thus, we can define a function ¯ a over U = ⋃ τ ∈ C U τ that is of class C 1 and such that for any ( t, τ ) ∈ U , F ( t, ¯ a ( t, τ ) , τ ) = 0 .

Now, ⋃ τ ∈ C ( τ -glyph[epsilon1] τ , τ + glyph[epsilon1] τ ) is a cover of the compact set C via open sets. Thus, we can extract from it a finite cover ⋃ τ ∈ I ( τ -glyph[epsilon1] τ , τ + glyph[epsilon1] τ ) , for some finite cardinality set I . Define (-t 0 , + t 0 ) = ⋂ τ ∈ I (-t τ , + t τ ) , which is a non-empty open interval as an intersection of (embedded) open intervals containing zero. Then, ¯ a is continously differentiable over (-t 0 , + t 0 ) × C and for any ( t, τ ) ∈ (-t 0 , + t 0 ) × C , we have F ( t, ¯ a ( t, τ ) , τ ) = 0 .

Theorem 5.3 (Upper and lower bound for comp-sum losses) . Assume that Φ is convex, twice continuously differentiable, and satisfies the properties Φ ′ ( u ) < 0 and Φ ′′ ( u ) > 0 for any u ∈ ( 0 , 1 2 ] . Then, the following property holds: T ( t ) = Θ ( t 2 ) .

Proof sketch For any τ ∈ [ 1 n , 1 2 ] , define the function T τ by T τ ( t ) = f t,τ ( 0 ) -inf ∣ u ∣≤ τ f t,τ ( u ) , where f t,τ ( u ) = 1 -t 2 Φ τ ( u ) + 1 + t 2 Φ τ (-u ) , t ∈ [ 0 , 1 ] and Φ τ ( u ) = Φ ( τ + u ) .

We aim to establish a lower and upper bound for inf τ ∈[ 1 n , 1 2 ] T τ ( t ) . For any fixed τ ∈ [ 1 n , 1 2 ] , this situation is parallel to that of binary classification (Theorem 4.1 and Theorem 4.2), since we have Φ ′ τ ( 0 ) = Φ ′ ( τ ) < 0 and Φ ′′ τ ( 0 ) = Φ ′′ ( τ ) > 0 . By Theorem 5.2 and the proof of Theorem 4.2, adopting a similar notation, while incorporating the τ subscript to distinguish different functions Φ τ and f t,τ , we can write ∀ t ∈ [ 0 , t 0 ] , T τ ( t ) = ∫ -a ∗ t,τ 0 u [ 1 -t 2 Φ ′′ τ (-u ) + 1 + t 2 Φ ′′ τ ( u )] du , where a ∗ t,τ verifies a ∗ 0 ,τ = 0 and ∂a ∗ t,τ ∂t ( 0 ) = Φ ′ τ ( 0 ) Φ ′′ τ ( 0 ) = c τ < 0 . Then, by further analyzing this equality, we can show the lower bound inf τ ∈[ 1 n , 1 2 ] -a ∗ t,τ = Ω ( t ) and the upper bound sup τ ∈[ 1 n , 1 2 ] -a ∗ t,τ = O ( t ) for some t ∈ [ 0 , t 1 ] , t 1 > 0 . Finally, using the fact that Φ ′′ reaches its maximum and minimum over a compact set, we obtain that T ( t ) = inf τ ∈[ 1 n , 1 2 ] T τ ( t ) = Θ ( t 2 ) . The full proof is included in Appendix L.

Theorem 5.3 significantly extends Theorem 4.2 to multi-class comp-sum losses, which include the logistic loss or cross-entropy used with a softmax activation function. It shows that the growth rate of H -consistency bounds for comp-sum losses is exactly square-root, provided that the auxiliary function Φ they are based upon is convex, twice continuously differentiable, and satisfies Φ ′ ( u ) < 0 and Φ ′′ ( u ) > 0 for any u ∈ ( 0 , 1 2 ] , which holds for most loss functions used in practice.

5.2 Constrained losses

Here, we consider constrained losses (see [Lee et al., 2004]), defined as

where Φ ∶ R → R + is a non-decreasing function. On possible choice for Φ is the exponential function. As shown by Mao et al. [2023b], for symmetric and complete hypothesis sets, the transformation T for the family of constrained losses can be characterized as follows.

Theorem 5.4 (Mao et al. [2023b, Theorem 11]) . Let H be a symmetric and complete hypothesis set. Assume that Φ is convex, differentiable at zero and satisfies the inequality Φ ′ ( 0 ) > 0 . Then, the transformation T can be expressed as

Next, we will show that for the constrained losses, the properties T ( t ) = Ω ( t ) and T ( t ) = O ( t ) hold as well. Note that by Theorem 5.4, we have

Therefore, to prove T ( t ) = Θ ( t 2 ) , we only need to show

For simplicity, we assume that the infimum over τ ≥ 0 can be reached within some finite interval [ 0 , A ] , A > 0 . This assumption holds for common choices of Φ , as discussed in [Mao et al., 2023b]. Furthermore, as demonstrated in Appendix N, under certain conditions on Φ ′′ , the infimum over τ ∈ [ 0 , A ] is reached at zero for sufficiently small values of t . For specific examples, see [Mao et al., 2023b, Appendix D.3], where Φ ( t ) = e t is considered.

Theorem 5.5 (Upper and lower bound for constrained losses) . Assume that Φ is convex, twice continuously differentiable, and satisfies the properties Φ ′ ( u ) > 0 and Φ ′′ ( u ) > 0 for any u ≥ 0 . Then, for any A > 0 , the following property holds:

Proof sketch For any τ ∈ [ 0 , A ] , define the function T τ by T τ ( t ) = f t,τ ( 0 ) -inf u ∈ R f t,τ ( u ) , where f t,τ ( u ) = 1 -t 2 Φ τ ( u ) + 1 + t 2 Φ τ (-u ) , t ∈ [ 0 , 1 ] and Φ τ ( u ) = Φ ( τ + u ) . We aim to establish a lower and upper bound for inf τ ∈[ 0 ,A ] T τ ( t ) . For any fixed τ ∈ [ 0 , A ] , this situation is parallel to that of binary classification (Theorem 4.1 and Theorem 4.2), since we also have Φ ′ τ ( 0 ) = Φ ′ ( τ ) > 0 and Φ ′′ τ ( 0 ) = Φ ′′ ( τ ) > 0 . By applying Theorem 5.2 and leveraging the proof of Theorem 4.2, adopting a similar notation, while incorporating the τ subscript to distinguish different functions Φ τ and f t,τ , we can write ∀ t ∈ [ 0 , t 0 ] , T τ ( t ) = ∫ a ∗ t,τ 0 u [ 1 -t 2 Φ ′′ τ ( u ) + 1 + t 2 Φ ′′ τ (-u )] du, where a ∗ t,τ verifies a ∗ 0 ,τ = 0 and ∂a ∗ t,τ ∂t ( 0 ) = Φ ′ τ ( 0 ) Φ ′′ τ ( 0 ) = c τ > 0 . Then, by further analyzing this equality, we can show the lower bound inf τ ∈[ 0 ,A ] a ∗ t,τ = Ω ( t ) and the upper bound sup τ ∈[ 0 ,A ] a ∗ t,τ = O ( t ) for some t ∈ [ 0 , t 1 ] , t 1 > 0 . Finally, using the fact that Φ ′′ reaches its maximum and minimum over some compact set, we obtain that T ( t ) = inf τ ∈[ 0 ,A ] T τ ( t ) = Θ ( t 2 ) . The full proof is included in Appendix M.

Theorem 5.5 significantly expands our findings to multi-class constrained losses. It demonstrates that, under some assumptions, which are commonly satisfied by smooth constrained losses used in practice, constrained loss H -consistency bounds also exhibit a square-root growth rate.

6 Minimizability gaps

As shown in Sections 4 and 5, H -consistency bounds for smooth loss functions in both binary and multi-class classification all admit a square-root growth rate near zero. In this section, we start by examining how the number of classes impacts these bounds. We then turn our attention to the minimizability gaps, which are the only distinguishing factors between the bounds.

6.1 Dependency on number of classes

Even with identical growth rates, surrogate losses can vary in their H -consistency bounds due to the number of classes. This factor becomes crucial to consider when the class count is large. Consider the family of comp-sum loss functions glyph[lscript] comp τ with τ ∈ [ 0 , 2 ) , defined as

where Φ τ ( u ) = -log ( u ) 1 τ = 1 + 1 1 -τ ( u τ -1 -1 ) 1 τ ≠ 1 , for any τ ∈ [ 0 , 2 ) . Mao et al. [2023f, Eq. (7) & Theorem 3.1], established the following bound for any h ∈ H and τ ∈ [ 1 , 2 ) ,

where ̃ Γ τ ( t ) = √ 2 n τ -1 t . Thus, while all these loss functions show square-root growth, the number of classes acts as a critical scaling factor.

6.2 Comparison across comp-sum losses

In Appendix H, we compare minimizability gaps cross comp-sum losses. We will see that minimizability gaps decrease as τ increases. This might suggest favoring τ close to 2 . But when accounting for n , glyph[lscript] comp τ with τ = 1 (logistic loss) is optimal since n then vanishes. Thus, both class count and minimizability gaps are essential in loss selection. In Appendix J, we will show that the minimizability gaps can become zero or relatively small under certain conditions in multi-class classification. In such scenarios, the logistic loss is favored, which can partly explain its widespread practical application.

6.3 Small surrogate minimizability gaps

While minimizability gaps vanish in special scenarios (e.g., unrestricted hypothesis sets, best-inclass error matching Bayes error), we now seek broader conditions for zero or small surrogate minimizability gaps to make our bounds more meaningful.

Due to space constraints, we focus on binary classification here, with multi-class results given in Appendix J. We address pointwise surrogate losses which take the form glyph[lscript] ( h ( x ) , y ) for a labeled point

( x, y ) . We write A = { h ( x )∶ h ∈ H } to denote the set of predictor values at x , which we assume to be independent of x . All proofs for this section are presented in Appendix I.

Deterministic scenario . We first consider the deterministic scenario, where the conditional probability p ( y ∣ x ) is either zero or one. For a deterministic distribution, we denote by X + the subset of X over which the label is + 1 and by X -the subset of X over which the label is -1 . For convenience, let glyph[lscript] + = inf α ∈ A glyph[lscript] ( α, + 1 ) and glyph[lscript] -= inf α ∈ A glyph[lscript] ( α, -1 ) .

Theorem 6.1. Assume that D is deterministic and that the best-in-class error is achieved by some h ∗ ∈ H . Then, the minimizability gap is null, M ( H ) = 0 , iff

If further α ↦ glyph[lscript] ( α, + 1 ) and α ↦ glyph[lscript] ( α, -1 ) are injective and glyph[lscript] + = glyph[lscript] ( α + , + 1 ) , glyph[lscript] -= glyph[lscript] ( α -, -1 ) , then, the condition is equivalent to h ∗ ( x ) = α + 1 x ∈ X + + α -1 x ∈ X -Furthermore, the minimizability gap is bounded by glyph[epsilon1] iff p ( E [ glyph[lscript] ( h ∗ ( x ) , + 1 ) ∣ y = + 1 ] -glyph[lscript] + ) + ( 1 -p )( E [ glyph[lscript] ( h ∗ ( x ) , -1 ) ∣ y = -1 ] -glyph[lscript] -) ≤ glyph[epsilon1] . In particular, the condition implies:

The theorem suggests that, under those assumptions, for the surrogate minimizability gap to be zero, the best-in-class hypothesis must be piecewise constant with specific values on X + and X -. The existence of such a hypothesis in H depends both on the complexity of the decision surface separating X + and X -and on that of the hypothesis set H . More generally, when the best-in-class classifier glyph[epsilon1] -approximates α + over X + and α -over X -, then the minimizability gap is bounded by glyph[epsilon1] . As an example, when the decision surface is a hyperplane, a hypothesis set of linear functions combined with a sigmoid activation function can provide such a good approximation (see Figure 1 for an illustration in a simple case).

+

X

+

X

Stochastic scenario . Here, we present a general result that is a direct extension of that of the deterministic scenario. We show that the minimizability gap is zero when there exists h ∗ ∈ H that matches α ∗ ( x ) for all x , where α ∗ ( x ) is the minimizer of the conditional error. We also show that the minimizability gap is bounded by glyph[epsilon1] when there exists h ∗ ∈ H whose conditional error glyph[epsilon1] -approximates best-in-class conditional error for all x .

Theorem 6.2. The best-in-class error is achieved by some h ∗ ∈ H and the minimizability gap is null, M ( H ) = 0 , iff there exists h ∗ ∈ H such that for all x ,

If further α ↦ E y [ glyph[lscript] ( α,y ) ∣ x ] is injective and inf α ∈ A E y [ glyph[lscript] ( α,y ) ∣ x ] = E y [ glyph[lscript] ( α ∗ ( x ) , y ) ∣ x ] , then, the condition is equivalent to h ∗ ( x ) = α ∗ ( x ) a.s. for x ∈ X . Furthermore, the minimizability gap is bounded by glyph[epsilon1] , M ( H ) ≤ glyph[epsilon1] , iff there exists h ∗ ∈ H such that

In deterministic settings, condition (4) coincides with that of Theorem 6.1. However, in stochastic scenarios, the existence of such a hypothesis depends on both decision surface complexity and the conditional distribution's properties. For illustration, see Appendix J.3 where we analyze the exponential, logistic (binary), and multi-class logistic losses.

We thoroughly analyzed minimizability gaps, comparing them across comp-sum losses, and identifying conditions for zero or small gaps, which help inform surrogate loss selection. In Appendix F, we show the crucial role of minimizability gaps in comparing excess bounds with H -consistency bounds. Importantly, combining H -consistency bounds with surrogate loss Rademacher complexity bounds yields zero-one loss (estimation) learning bounds for surrogate loss minimizers (see Appendix O).

7 Conclusion

We established a universal square-root growth rate for the widely-used class of smooth surrogate losses in both binary and multi-class classification. This underscores the minimizability gap as a crucial discriminator among surrogate losses. Our detailed analysis of these gaps can provide guidance for loss selection.

References

Agarwal. Surrogate regret bounds for bipartite ranking via strongly proper losses. The Journal of Machine Learning Research , 15(1):1653-1674, 2014.

Awasthi, N. Frank, A. Mao, M. Mohri, and Y. Zhong. Calibration and consistency of adversarial surrogate losses. Advances in Neural Information Processing Systems , pages 9804-9815, 2021a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. A finer calibration analysis for adversarial robustness. arXiv preprint arXiv:2105.01550 , 2021b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H -consistency bounds. In Advances in neural information processing systems , pages 782-795, 2022a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. H -consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , pages 1117-1174, 2022b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094, 2023a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. DC-programming for neural network optimizations. Journal of Global Optimization , 2023b.

Bao. Proper losses, moduli of convexity, and surrogate regret bounds. In Conference on Learning Theory , pages 525-547, 2023.

L. Bartlett and M. H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research , 9(8), 2008.

L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Ben-David, D. Loker, N. Srebro, and K. Sridharan. Minimizing the misclassification error rate using a surrogate convex loss. In International Coference on International Conference on Machine Learning , pages 83-90, 2012.

Berkson. Application of the logistic function to bio-assay. Journal of the American Statistical Association , 39:357--365, 1944.

Berkson. Why I prefer logits to probits. Biometrics , 7(4):327--339, 1951.

Blondel. Structured prediction with projection oracles. In Advances in neural information processing systems , 2019.

Cao, T. Cai, L. Feng, L. Gu, J. Gu, B. An, G. Niu, and M. Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In Advances in neural information processing systems , 2022.

Charoenphakdee, Z. Cui, Y. Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning , pages 1507-1517, 2021.

D.-R. Chen, Q. Wu, Y. Ying, and D.-X. Zhou. Support vector machine soft margin classifiers: error analysis. The Journal of Machine Learning Research , 5:1143-1175, 2004.

Ciliberto, L. Rosasco, and A. Rudi. A consistent regularization approach for structured prediction. In Advances in neural information processing systems , 2016.

Ciliberto, L. Rosasco, and A. Rudi. A general framework for consistent structured prediction with implicit loss embeddings. The Journal of Machine Learning Research , 21(1):3852-3918, 2020.

Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory , pages 67-82, 2016a.

Cortes, G. DeSalvo, and M. Mohri. Boosting with abstention. In Advances in Neural Information Processing Systems , pages 1660-1668, 2016b.

Cortes, G. DeSalvo, and M. Mohri. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence , 2023.

Cortes, A. Mao, C. Mohri, M. Mohri, and Y. Zhong. Cardinality-aware set prediction and topk classification. In Advances in neural information processing systems , 2024.

Crammer and Y. Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research , 2(Dec):265-292, 2001.

L. Dontchev and R. T. Rockafellar. Robinson's implicit function theorem and its extensions. Math. Program. , 117(1-2):129-147, 2009.

Duchi, K. Khosravi, and F. Ruan. Multiclass classification, information, divergence and surrogate risk. The Annals of Statistics , 46(6B):3246-3275, 2018.

Finocchiaro, R. Frongillo, and B. Waggoner. An embedding framework for consistent polyhedral surrogates. In Advances in neural information processing systems , 2019.

Frongillo and B. Waggoner. Surrogate regret bounds for polyhedral losses. In Advances in Neural Information Processing Systems , volume 34, pages 21569-21580, 2021.

Gao and Z.-H. Zhou. On the consistency of AUC pairwise optimization. In International Joint Conference on Artificial Intelligence , 2015.

Kotlowski, K. J. Dembczynski, and E. Huellermeier. Bipartite ranking through minimization of univariate loss. In International Conference on Machine Learning , pages 1113-1120, 2011.

Kuratowski and C. Ryll-Nardzewski. A general theorem on selectors. Bull. Acad. Pol. Sci., Sér. Sci. Math. Astron. Phys. , 13(8):397-403, 1965.

Kuznetsov, M. Mohri, and U. Syed. Multi-class deep boosting. In Advances in Neural Information Processing Systems , pages 2501-2509, 2014.

Lapin, M. Hein, and B. Schiele. Loss functions for top-k error: Analysis and insights. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1468-1477, 2016.

Lee, Y. Lin, and G. Wahba. Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association , 99(465):67-81, 2004.

Long and R. Servedio. Consistency versus realizable H-consistency for multiclass classification. In International Conference on Machine Learning , pages 801-809, 2013.

Mahdavi, L. Zhang, and R. Jin. Binary excess risk for smooth convex surrogates. arXiv preprint arXiv:1402.1792 , 2014.

Mao, C. Mohri, M. Mohri, and Y. Zhong. Two-stage learning to defer with multiple experts. In Advances in neural information processing systems , 2023a.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds: Characterization and extensions. In Advances in Neural Information Processing Systems , 2023b.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds for pairwise misranking loss surrogates. In International conference on Machine learning , 2023c.

Mao, M. Mohri, and Y. Zhong. Ranking with abstention. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023d.

Mao, M. Mohri, and Y. Zhong. Structured prediction with stronger consistency guarantees. In Advances in Neural Information Processing Systems , 2023e.

Mao, M. Mohri, and Y. Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , 2023f.

Mao, M. Mohri, and Y. Zhong. Principled approaches for learning to defer with multiple experts. In International Symposium on Artificial Intelligence and Mathematics , 2024a.

Mao, M. Mohri, and Y. Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. In International Conference on Algorithmic Learning Theory , 2024b.

Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for scorebased multi-class abstention. In International Conference on Artificial Intelligence and Statistics , 2024c.

Mao, M. Mohri, and Y. Zhong. Enhanced H -consistency bounds. arXiv preprint arXiv:2407.13722 , 2024d.

Mao, M. Mohri, and Y. Zhong. H -consistency guarantees for regression. In International Conference on Machine Learning , pages 34712-34737, 2024e.

Mao, M. Mohri, and Y. Zhong. Multi-label learning with stronger consistency guarantees. In Advances in neural information processing systems , 2024f.

Mao, M. Mohri, and Y. Zhong. Realizable H -consistent and Bayes-consistent loss functions for learning to defer. In Advances in neural information processing systems , 2024g.

Mao, M. Mohri, and Y. Zhong. Regression with multi-expert deferral. In International Conference on Machine Learning , pages 34738-34759, 2024h.

K. Menon and R. C. Williamson. Bayes-optimal scorers for bipartite ranking. In Conference on Learning Theory , pages 68-106, 2014.

Mohri, D. Andor, E. Choi, M. Collins, A. Mao, and Y. Zhong. Learning to reject with a fixed predictor: Application to decontextualization. In International Conference on Learning Representations , 2024.

Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning . MIT Press, second edition, 2018.

Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning , pages 7076-7087, 2020.

Mozannar, H. Lang, D. Wei, P. Sattigeri, S. Das, and D. Sontag. Who should predict? exact algorithms for learning to defer to humans. In International Conference on Artificial Intelligence and Statistics , pages 10520-10545, 2023.

Ni, N. Charoenphakdee, J. Honda, and M. Sugiyama. On the calibration of multiclass classification with rejection. In Advances in Neural Information Processing Systems , pages 2582-2592, 2019.

Nowak, F. Bach, and A. Rudi. Sharp analysis of learning with discrete losses. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1920-1929, 2019.

Nowak, F. Bach, and A. Rudi. Consistent structured prediction with max-min margin markov networks. In International Conference on Machine Learning , pages 7381-7391, 2020.

Nowak, A. Rudi, and F. Bach. On the consistency of max-margin losses. In International Conference on Artificial Intelligence and Statistics , pages 4612-4633, 2022.

Nowak-Vila, F. Bach, and A. Rudi. A general theory for structured prediction with smooth convex surrogates. arXiv preprint arXiv:1902.01958 , 2019.

Osokin, F. Bach, and S. Lacoste-Julien. On structured prediction theory with calibrated convex surrogate losses. In Advances in Neural Information Processing Systems , 2017.

A. Pires, C. Szepesvari, and M. Ghavamzadeh. Cost-sensitive multiclass classification risk bounds. In International Conference on Machine Learning , pages 1391-1399, 2013.

G. Ramaswamy, A. Tewari, and S. Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics , 12(1):530-554, 2018.

D. Reid and R. C. Williamson. Surrogate regret bounds for proper losses. In International Conference on Machine Learning , pages 897-904, 2009.

Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Tewari and P. L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research , 8(36):1007-1025, 2007.

Uematsu and Y. Lee. On theoretically optimal ranking functions in bipartite ranking. Journal of the American Statistical Association , 112(519):1311-1322, 2017.

F. Verhulst. Notice sur la loi que la population suit dans son accroissement. Correspondance mathématique et physique , 10:113--121, 1838.

F. Verhulst. Recherches mathématiques sur la loi d'accroissement de la population. Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles , 18:1--42, 1845.

Verma and E. Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning , pages 22184-22202, 2022.

Verma, D. Barrejón, and E. Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics , pages 11415-11434, 2023.

Weston and C. Watkins. Support vector machines for multi-class pattern recognition. European Symposium on Artificial Neural Networks , 4(6), 1999.

Yang and S. Koyejo. On the consistency of top-k surrogate losses. In International Conference on Machine Learning , pages 10727-10735, 2020.

Yu and M. B. Blaschko. The lovász hinge: A novel convex surrogate for submodular losses. IEEE transactions on pattern analysis and machine intelligence , 42(3):735-748, 2018.

Yuan and M. Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research , 11(1), 2010.

Zhang, T. Liu, and D. Tao. On the rates of convergence from surrogate risk minimizers to the Bayes optimal classifier. IEEE Transactions on Neural Networks and Learning Systems , 33(10): 5766-5774, 2021.

Zhang and S. Agarwal. Bayes consistency vs. H-consistency: The interplay between surrogate loss functions and the scoring function class. In Advances in Neural Information Processing Systems , pages 16927-16936, 2020.

Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics , 32(1):56-85, 2004a.

Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004b.

Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems , 2018.

Contents of Appendix

P

Future work

33

A Related work

The Bayes-consistency of surrogate losses has been extensively studied in the context of binary classification. Zhang [2004a], Bartlett et al. [2006] and Steinwart [2007] established Bayes-consistency for various convex loss functions, including margin-based surrogates. They also introduced excess error bounds (or surrogate regret bounds) for margin-based surrogates. Other works include Chen et al. [2004], which specifically studied the SVM q -norm soft margin loss and established a square-root excess error bound with an optimal constant for that specific family, and Reid and Williamson [2009], which established tight excess error bounds for proper losses in binary classification.

The Bayes-consistency of several surrogate loss function families in the context of multi-class classification has also been studied by Zhang [2004b] and Tewari and Bartlett [2007]. Zhang [2004b] established a series of results for various multi-class classification formulations, including negative results for multi-class hinge loss functions [Crammer and Singer, 2001], as well as positive results for the sum exponential loss [Weston and Watkins, 1999, Awasthi, Mao, Mohri, and Zhong, 2022a], the (multinomial) logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], and the constrained losses [Lee et al., 2004]. Later, Tewari and Bartlett [2007] adopted a different geometric method to analyze Bayes-consistency, yielding similar results for these loss function families. Steinwart [2007] developed general tools to characterize Bayes consistency for both binary and multi-class classification. Additionally, excess error bounds have been derived by Pires et al. [2013] for a family of constrained losses and by Duchi et al. [2018] for loss functions related to generalized entropies.

For a surrogate loss glyph[lscript] , an excess error bound holds for any predictor h and has the form E glyph[lscript] 0 -1 ( h ) -E ∗ glyph[lscript] 0 -1 ≤ Ψ ( E glyph[lscript] ( h ) -E ∗ glyph[lscript] ) , where E glyph[lscript] 0 -1 ( h ) and E glyph[lscript] ( h ) represent the expected losses of h for the zero-one loss and surrogate loss respectively, E ∗ glyph[lscript] 0 -1 and E ∗ glyph[lscript] the Bayes errors for the zero-one and surrogate loss respectively, and Ψ a non-decreasing function.

The growth rate of excess error bounds, that is the behavior of function Ψ near zero, has gained attention in recent research [Mahdavi et al., 2014, Zhang et al., 2021, Frongillo and Waggoner, 2021, Bao, 2023]. Mahdavi et al. [2014] examined the growth rate for smoothed hinge losses in binary classification, demonstrating that smoother losses result in worse growth rates. The optimal rate is achieved with the standard hinge loss, which exhibits linear growth. Zhang et al. [2021] tied the growth rate of excess error bounds in binary classification to two properties of the surrogate loss function: consistency intensity and conductivity. These metrics enable comparisons of growth rates across different surrogates. But, can we establish lower and upper bounds for the growth rate of excess error bounds under specific regularity conditions?

Frongillo and Waggoner [2021] pioneered research on this question in binary classification settings. They established a critical square-root lower bound for excess error bounds when a surrogate loss is locally strongly convex and has a locally Lipschitz gradient. Additionally, they demonstrated a linear excess error bound for Bayes-consistent polyhedral loss functions (convex and piecewise-linear) [Finocchiaro et al., 2019] (see also [Lapin et al., 2016, Ramaswamy et al., 2018, Yu and Blaschko, 2018, Yang and Koyejo, 2020]). More recently, Bao [2023] complemented these results by showing that proper losses associated with Shannon entropy, exponential entropy, spherical entropy, squared α -norm entropies and α -polynomial entropies, with α > 1 , also exhibit a square-root lower bound for excess error bounds relative to the glyph[lscript] 1 -distance.

However, while Bayes-consistency and excess error bounds are valuable, they are not sufficiently informative, as they are established for the family of all measurable functions and disregard the crucial role played by restricted hypothesis sets in learning. As pointed out by Long and Servedio [2013], in some cases, minimizing Bayes-consistent losses can result in constant expected error, while minimizing inconsistent losses can yield an expected loss approaching zero. To address this limitation, the authors introduced the concept of realizable H -consistency , further explored by Kuznetsov et al. [2014] and Zhang and Agarwal [2020]. Nonetheless, these guarantees are only asymptotic and rely on a strong realizability assumption that typically does not hold in practice.

Recent research by Awasthi, Mao, Mohri, and Zhong [2022b,a] and Mao, Mohri, and Zhong [2023f,c,e,b] has instead introduced and analyzed H -consistency bounds . These bounds are more informative than Bayes-consistency since they are hypothesis set-specific and non-asymptotic. Their work covers broad families of surrogate losses in binary classication, multi-class classification, structured prediction, and abstention [Mao, Mohri, Mohri, and Zhong, 2023a]. Crucially, they provide upper bounds on the estimation error of the target loss, for example, the zero-one loss in classification,

that hold for any predictor h within a hypothesis set H . These bounds relate this estimation error to the surrogate loss estimation error. Their general form is: E glyph[lscript] 0 -1 ( h ) -E ∗ glyph[lscript] 0 -1 ( H ) ≤ f ( E glyph[lscript] ( h ) -E ∗ glyph[lscript] ( H )) , where E ∗ glyph[lscript] 0 -1 ( H ) and E ∗ glyph[lscript] ( H ) represent the best-in-class expected losses for the zero-one and surrogate loss respectively, and f is a non-decreasing function continuous at zero. H -consistency bounds imply in particular excess error bounds, when the hypothesis set is taken to be the family of all measurable functions.

The authors have further analyzed H -consistency bounds in structured prediction, ranking, and abstention.

Mao, Mohri, and Zhong [2023e] revealed limitations of existing structured prediction loss functions, demonstrating they lack Bayes-consistency. They introduced new surrogate loss families proven to benefit form H -consistency bounds, thus also establishing Bayes-consistency. This complements earlier negative finding about the Bayes-consistency of Struct-SVM and positive results for quadratic surrogate (QS) losses or some non-smooth polyhedral-type loss functions [Osokin et al., 2017, Ciliberto et al., 2016, Blondel, 2019, Nowak-Vila et al., 2019, Nowak et al., 2019, 2020, Ciliberto et al., 2020, Nowak et al., 2022].

Mao et al. [2023c] showed that there are no meaningful H -consistency bounds for general pairwise ranking and bipartite ranking surrogate losses with equicontinuous hypothesis sets, including linear models and neural networks. They proposed ranking with abstention as a solution. These results demonstrated that although these surrogate loss functions have been shown to be Bayes-consistent in various studies [Kotlowski et al., 2011, Menon and Williamson, 2014, Agarwal, 2014, Gao and Zhou, 2015, Uematsu and Lee, 2017], they are, in fact, not H -consistent.

Mao et al. [2023a] applied H -consistency bounds to two-stage learning to defer scenarios, designing new surrogate losses. This complemented the Bayes-consistent surrogate losses in the single-stage scenario of learning to defer [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022, Verma et al., 2023, Mozannar et al., 2023] or learning with abstention [Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, Cortes et al., 2016a,b, Ramaswamy et al., 2018, Ni et al., 2019, Charoenphakdee et al., 2021, Cao et al., 2022, Cortes et al., 2023].

Mao, Mohri, and Zhong [2024d] recently established enhanced H -consistency bounds based on more general inequalities relating conditional regrets. They derived more favorable distribution- and predictor-dependent bounds in various scenarios including standard multi-class classification, binary and multi-class classification under Tsybakov noise conditions, and bipartite ranking. H -consistency bounds have also been studied in the scenario of adversarial robustness [Awasthi et al., 2021a,b, 2023a,b], bounded regression [Mao et al., 2024e], regression with multi-expert deferral [Mao et al., 2024h], topk classification [Cortes et al., 2024], multi-label learning [Mao et al., 2024f], score-based abstention [Mao et al., 2024c], predictor-rejector abstention [Mao et al., 2024b], learning to abstain with a fixed predictor with application in decontextualization [Mohri et al., 2024], ranking with abstention [Mao et al., 2023d], realizable learning to defer [Mao et al., 2024g], and learning to defer with multiple experts [Mao et al., 2024a].

This papers presents a characterization of the growth rate of H -consistency bounds, that is how quickly the functions f increase near zero, both in binary and multi-class classification.

B Pointwise loss functions - Proof of Lemma 2.1

Lemma 2.1. Let glyph[lscript] be a pointwise loss function. Then, the Bayes error and the approximation error can be expressed as follows: E ∗ glyph[lscript] ( H all ) = E x [ C ∗ glyph[lscript] ( H all , x )] and A glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E x [ C ∗ glyph[lscript] ( H all , x )] .

Proof. By definition, for a pointwise loss function glyph[lscript] , there exists a measurable function ˆ glyph[lscript] ∶ R n × Y → R + such that glyph[lscript] ( h, x, y ) = ˆ glyph[lscript] ( h ( x ) , y ) , where h ( x ) = [ h ( x, 1 ) , . . . , h ( x, n )] is the score vector of the predictor h . Thus, the following inequality holds:

Since ˆ glyph[lscript] ∶ ( α,y ) ↦ R + is measurable, the function ( α,x ) ↦ E y [ ˆ glyph[lscript] ( α,y ) ∣ x ] is also measurable. We now show that the function x ↦ C ∗ glyph[lscript] ( H all , x ) = inf α ∈ R E y [ ˆ glyph[lscript] ( α,y ) ∣ x ] is also measurable.

To do this, we consider for any β > 0 , the set { x ∶ inf α ∈ R E y [ ˆ glyph[lscript] ( α,y ) ∣ x ] < β } which can be expressed as

where Π X is the projection onto X . By the measurable projection theorem, x ↦ inf α ∈ R n E y [ ˆ glyph[lscript] ( α,y ) ∣ x ] is measurable. Then, since a pointwise difference of measurable functions is measurable, for all n ∈ N , the set {( α,x )∶ E y [ ˆ glyph[lscript] ( α,y ) ∣ x ] < inf α ∈ R n E y [ ˆ glyph[lscript] ( α,y ) ∣ x ] + 1 n } is measurable. Thus, by Kuratowski and Ryll-Nardzewski [1965]'s measurable selection theorem, for all n ∈ N , there exists a measurable function h n ∶ x ↦ α ∈ R n such that the following holds:

Therefore, we have

By taking the limit n → +∞ , we obtain E ∗ glyph[lscript] ( H all ) = E x [ C ∗ glyph[lscript] ( H all , x )] . By definition, A glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E ∗ glyph[lscript] ( H all ) = E ∗ glyph[lscript] ( H ) -E x [ C ∗ glyph[lscript] ( H all , x )] . This completes the proof.

C General form of H -consistency bounds

Fix a target loss function glyph[lscript] 2 and a surrogate loss glyph[lscript] 1 . Given a hypothesis set H , a bound relating the estimation errors of these loss functions admits the following form:

where, for any distribution D , Γ D ∶ R + → R + is a non-decreasing function on R + . We will assume that Γ D is concave. In particular, the bound should hold for any point mass distribution δ x , x ∈ X . We will operate under the assumption that the same bound holds uniformly over X and thus that there exists a fixed concave function Γ such that Γ δ x = Γ for all x .

Observe that for any point mass distribution δ x , the conditional loss and the expected loss coincide and therefore that we have E glyph[lscript] 2 ( h ) -E ∗ glyph[lscript] 2 ( H ) = ∆ C glyph[lscript] 2 , H ( H , x ) , and similarly with glyph[lscript] 1 . Thus, we can write:

Therefore, by Jensen's inequality, for any distribution D , we have

Since E x [ ∆ C glyph[lscript] 2 , H ( h, x )] = E glyph[lscript] 2 ( h )-E ∗ glyph[lscript] 2 ( H )+ M glyph[lscript] 2 ( H ) and similarly with glyph[lscript] 1 , we obtain the following bound for all distributions D :

This leads to the general form of H -consistency bounds that we will be considering, which includes the key role of the minimizability gaps.

D Properties of minimizability gaps

By Lemma 2.1, for a pointwise loss function, we have E ∗ glyph[lscript] ( H all ) = E x [ C ∗ glyph[lscript] ( H all , x )] , thus the minimizability gap vanishes for the family of all measurable functions.

Lemma D.1. Let glyph[lscript] be a pointwise loss function. Then, we have M glyph[lscript] ( H all ) = 0 .

Thus, in that case, (6) takes the following simpler form:

In general, however, the minimizabiliy gap is non-zero for a restricted hypothesis set H and is therefore important to analyze. Let I glyph[lscript] ( H ) be the difference of pointwise infima I glyph[lscript] ( H ) = E x [ C ∗ glyph[lscript] ( H , x )-C ∗ glyph[lscript] ( H all , x )] , which is non-negative. Note that, for a pointwise loss function, the minimizability gap can be decomposed as follows in terms of the approximation error and the difference of pointwise infima:

Thus, the minimizabiliy gap can be upper bounded by the approximation error. It is however a finer quantity than the approximation error and can lead to more favorable guarantees. When the difference of pointwise infima can be evaluated or bounded, this decomposition can provide a convenient way to analyze the minimizability gap in terms of the approximation error.

Note that I glyph[lscript] ( H ) can be non-zero for families of bounded functions. Let Y = {-1 , + 1 } and H be a family of functions h with ∣ h ( x )∣ ≤ Λ for all x ∈ X and such that all values in [-Λ , + Λ ] can be reached. Consider for example the exponential-based margin loss: glyph[lscript] ( h, x, y ) = e -yh ( x ) . Let η ( x ) = p (+ 1 ∣ x ) = D ( Y = + 1 ∣ X = x ) . Thus, C glyph[lscript] ( h, x ) = η ( x ) e -h ( x ) + ( 1 -η ( x )) e h ( x ) . Then, it is not hard to see that C ∗ glyph[lscript] ( H all , x ) = 2 √ η ( x )( 1 -η ( x )) for all x but C ∗ glyph[lscript] ( H , x ) depends on Λ with the minimizing value for h ( x ) being: min { 1 2 log η ( x ) 1 -η ( x ) , Λ } if η ( x ) ≥ 1 / 2 , max { 1 2 log η ( x ) 1 -η ( x ) , -Λ } otherwise. Thus, in the deterministic case, I glyph[lscript] ( H ) = e -Λ .

When the best-in-class error coincides with the Bayes error, E ∗ glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H all ) , both the approximation error and minimizability gaps vanish.

Lemma D.2. For any loss function glyph[lscript] such that E ∗ glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H all ) = E x [ C ∗ glyph[lscript] ( H all , x )] , we have M glyph[lscript] ( H ) = A glyph[lscript] ( H ) = 0 .

Proof. By definition, A glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E ∗ glyph[lscript] ( H all ) = 0 . Since we have M glyph[lscript] ( H ) ≤ A glyph[lscript] ( H ) , this implies M glyph[lscript] ( H ) = 0 .

E Examples of H -consistency bounds

Here, we compile some common examples of H -consistency bounds for both binary and multi-class classification. Table 1, 2 and 3 include the examples of H -consistency bounds for binary margin-based losses, comp-sum losses and constrained losses, respectively.

These bounds are due to previous work by Awasthi et al. [2022b] for binary margin-based losses, by Mao et al. [2023f] for multi-class comp-sum losses, and by Awasthi et al. [2022a] and Mao et al. [2023b] for multi-class constrained losses, respectively. We consider complete hypothesis sets for binary classification (see Section 4), and symmetric and complete hypothesis sets for multi-class classification (see Section 5).

F Comparison with excess error bounds

Excess error bounds can be used to derive bounds for a hypothesis set H expressed in terms of the approximation error. Here, we show, however, that, the resulting bounds are looser than H -consistency bounds.

Fix a target loss function glyph[lscript] 2 and a surrogate loss glyph[lscript] 1 . Excess error bounds, also known as surrogate regret bounds , are bounds relating the excess errors of these loss functions of the following form:

where ψ ∶ R + → R + is a non-decreasing and convex function on R + . Recall that as shown in (1), the excess error can be written as the sum of the estimation error and the approximation error. Thus, the excess error bound can be equivalently expressed as follows:

In Section 3, we have shown that the minimizabiliy gap can be upper bounded by the approximation error M glyph[lscript] ( H ) ≤ A ( H ) and is in general a finer quantity for a surrogate loss glyph[lscript] 1 . However, we will show that for a target loss glyph[lscript] 2 that is discrete , the minimizabiliy gap in general coincides with the approximation error.

Definition F.1. We say that a target loss glyph[lscript] 2 is discrete if we can write glyph[lscript] 2 ( h, x, y ) = L ( h ( x ) , y ) for some binary function L ∶ Y × Y → R + .

In other words, a discrete target loss glyph[lscript] 2 is explicitly a function of both the prediction h ( x ) and the true label y , where both belong to the label space Y . Consequently, it can assume at most n 2 distinct discrete values.

Next, we demonstrate that for such discrete target loss functions, if for any instance, the set of predictions generated by the hypothesis set completely spans the label space, then the minimizability gap is precisely equal to the approximation error. For convenience, we denote by H ( x ) the set of predictions generated by the hypothesis set on input x ∈ X , defined as H ( x ) = { h ( x )∶ h ∈ H }

Theorem F.2. Given a discrete target loss function glyph[lscript] 2 . Assume that the hypothesis set H satisfies, for any x ∈ X , H ( x ) = Y . Then, we have I glyph[lscript] 2 ( H ) = 0 and M glyph[lscript] 2 ( H ) = A glyph[lscript] 2 ( H ) .

Proof. As shown in Section 3, the minimizability gap can be decomposed in terms of the approximation error and the difference of pointwise infima:

By definition and the fact that glyph[lscript] 2 is discrete, the conditional error can be written as

Thus, for any x ∈ X , the best-in-class conditional error can be expressed as

By the assumption that H ( x ) = Y , we obtain

By Theorem F.2, for a target loss glyph[lscript] 2 that is discrete and hypothesis sets H modulo mild assumptions, the minimizabiliy gap coincides with the approximation error. In such cases, by comparing an excess error bound (9) with the H -consistency bound (6):

we obtain that the left-hand side of both bounds are equal (since M glyph[lscript] 2 ( H ) = A glyph[lscript] 2 ( H ) ), while the right-hand side of the H -consistency bound is always upper bounded by and can be finer than the right-hand side of the excess error bound (since M glyph[lscript] 1 ( H ) ≤ A glyph[lscript] 1 ( H ) ), which implies that excess error bounds (or surrogate regret bounds) are in general inferior to H -consistency bounds.

G Polyhedral losses versus smooth losses

Since H -consistency bounds subsume excess error bounds as a special case (Appendix F), the linear growth rate of polyhedral loss excess error bounds (Finocchiaro et al. [2019]) also dictates a linear growth rate for polyhedral H -consistency bounds, if they exist. This is illustrated by the hinge loss or ρ -margin loss which have been shown to benefit from H -consistency bounds [Awasthi et al., 2022b].

Here, we compare in more detail polyhedral losses and the smooth losses. Assume that a hypothesis set H is complete and thus H ( x ) = Y for any x ∈ X . By Theorem F.2, we have A glyph[lscript] 0 -1 ( H ) = M glyph[lscript] 0 -1 ( H ) . As shown by Frongillo and Waggoner [2021, Theorem 3], a Bayes-consistent polyhedral loss Φ poly admits the following linear excess error bound, for some β 1 > 0 ,

However, for a smooth loss Φ smooth , if it satisfies the condition of Theorem 4.2, Φ smooth admits the following H -consistency bound:

where T ( t ) = Θ ( t 2 ) . Therefore, our theory offers a principled basis for comparing polyhedral losses (10) and smooth losses (11), which depends on the following factors:

The growth rate: linear for polyhedral losses, while square-root for smooth losses.

The optimization property: smooth losses are more favorable for optimization compared to polyhedral losses, in particular with deep neural networks.

The approximation theory: the approximation error A Φ poly ( H ) appears on the right-hand side of the bound for polyhedral losses, whereas a finer quantity, the minimizability gap M Φ smooth ( H ) , is present on the right-hand side of the bound for smooth losses.

H Comparison of minimizability gaps across comp-sum losses

For glyph[lscript] comp τ loss functions, τ ∈ [ 0 , 2 ) , we can characterize minimizability gaps as follows.

Theorem H.1. Assume that for any x ∈ X , we have {( h ( x, 1 ) , . . . , h ( x, n ))∶ h ∈ H } = [-Λ , + Λ ] n . Then, for comp-sum losses glyph[lscript] comp τ and any deterministic distribution, the minimizability gaps can be expressed as follows:

where f τ ( u ) = log ( 1 + u ) 1 τ = 1 + 1 1 -τ (( 1 + u ) 1 -τ -1 ) 1 τ ≠ 1 and C ∗ glyph[lscript] comp τ = 0 ( H , x ) = e -2Λ ( n -1 ) . Moreover, ̃ M glyph[lscript] comp τ ( H ) is a non-increasing function of τ .

Proof. Since f τ is concave and non-decreasing, and the equality glyph[lscript] τ = f τ ( glyph[lscript] τ = 0 ) holds, the minimizability gaps can be upper bounded as follows, for any τ ≥ 0,

Since the distribution is deterministic, the conditional error can be expressed as follows:

where y max = argmax p ( y ∣ x ) . Using the fact that f τ is increasing for any τ > 0 , the hypothesis h ∗ ∶ ( x, y ) ↦ Λ1 y = y max -Λ1 y ≠ y max achieves the best-in-class conditional error. Thus,

where C ∗ glyph[lscript] comp τ = 0 ( H , x ) = e -2Λ ( n -1 ) . Therefore,

This completes the first part of the proof. Using the fact that τ ↦ f τ ( u 1 )-f τ ( u 2 ) is a non-increasing function of τ for any u 1 ≥ u 2 ≥ 0 , the second proof is completed as well.

The theorem shows that for comp-sum loss functions glyph[lscript] comp τ , the minimizability gaps are nonincreasing with respect to τ . Note that Φ τ satisfies the conditions of Theorem 5.3 for any τ ∈ [ 0 , 2 ) . Therefore, focusing on behavior near zero (ignoring constants), the theorem provides a principled comparison of minimizability gaps and H -consistency bounds across different comp-sum losses.

I Small surrogate minimizability gaps: proof for binary classification

Theorem 6.1. Assume that D is deterministic and that the best-in-class error is achieved by some h ∗ ∈ H . Then, the minimizability gap is null, M ( H ) = 0 , iff

If further α ↦ glyph[lscript] ( α, + 1 ) and α ↦ glyph[lscript] ( α, -1 ) are injective and glyph[lscript] + = glyph[lscript] ( α + , + 1 ) , glyph[lscript] -= glyph[lscript] ( α -, -1 ) , then, the condition is equivalent to h ∗ ( x ) = α + 1 x ∈ X + + α -1 x ∈ X -Furthermore, the minimizability gap is bounded by glyph[epsilon1] iff p ( E [ glyph[lscript] ( h ∗ ( x ) , + 1 ) ∣ y = + 1 ] -glyph[lscript] + ) + ( 1 -p )( E [ glyph[lscript] ( h ∗ ( x ) , -1 ) ∣ y = -1 ] -glyph[lscript] -) ≤ glyph[epsilon1] . In particular, the condition implies:

Proof. By definition of h ∗ , using the shorthand p = P [ y = + 1 ] , we can write

Since the distribution is deterministic, the expected pointwise infimum can be rewritten as follows:

where glyph[lscript] + = inf α ∈ A glyph[lscript] ( α, + 1 ) and glyph[lscript] -= inf α ∈ A glyph[lscript] ( α, -1 ) . Thus, we have

In view of that, since, by definition of glyph[lscript] + and glyph[lscript] -, the expressions within the conditional expectations are non-negative, the equality M ( H ) = 0 holds iff glyph[lscript] ( h ∗ ( x ) , + 1 ) -glyph[lscript] + = 0 almost surely for any x in X + and glyph[lscript] ( h ∗ ( x ) , -1 ) -glyph[lscript] -= 0 almost surely for any x in X -. This completes the first part of the proof. Furthermore, M ( H ) ≤ glyph[epsilon1] is equivalent to

that is

In light of the non-negativity of the expressions, this implies in particular:

This completes the second part of the proof.

Theorem 6.2. The best-in-class error is achieved by some h ∗ ∈ H and the minimizability gap is null, M ( H ) = 0 , iff there exists h ∗ ∈ H such that for all x ,

If further α ↦ E y [ glyph[lscript] ( α,y ) ∣ x ] is injective and inf α ∈ A E y [ glyph[lscript] ( α,y ) ∣ x ] = E y [ glyph[lscript] ( α ∗ ( x ) , y ) ∣ x ] , then, the condition is equivalent to h ∗ ( x ) = α ∗ ( x ) a.s. for x ∈ X . Furthermore, the minimizability gap is bounded by glyph[epsilon1] , M ( H ) ≤ glyph[epsilon1] , iff there exists h ∗ ∈ H such that

Proof. Assume that the best-in-class error is achieved by some h ∗ ∈ H . Then, we can write

The expected pointwise infimum can be rewritten as follows:

Thus, we have

In view of that, since, by the definition of infimum, the expressions within the marginal expectations are non-negative, the condition that M ( H ) = 0 implies that

On the other hand, if there exists h ∗ ∈ H such that the condition (3) holds, then,

Since M ( H ) is non-negative, the inequality is achieved. Thus, we have

If there exists h ∗ ∈ H such that the condition (4) holds, then,

On the other hand, since we have

M ( H ) ≤ glyph[epsilon1] implies that

This completes the proof.

J Small surrogate minimizability gaps: multi-class classification

We consider the multi-class setting with label space [ n ] = { 1 , 2 , . . . , n } . In this setting, the surrogate loss incurred by a predictor h at a labeled point ( x, y ) can be expressed by glyph[lscript] ( h ( x ) , y ) , where h ( x ) = [ h ( x, 1 ) , . . . , h ( x, n )] is the score vector of the predictor h . We denote by A the set of values in R n taken by the score vector of predictors in H at x , which we assume to be independent of x : A = { h ( x )∶ h ∈ H } , for all x ∈ X .

J.1 Deterministic scenario

We first consider the deterministic scenario, where the conditional probability p ( y ∣ x ) is either zero or one. For a deterministic distribution, we denote by X k the subset of X over which the label is k . For convenience, let glyph[lscript] k = inf α ∈ A glyph[lscript] ( α,k ) , for any k ∈ [ n ] .

Theorem J.1. Assume that D is deterministic and that the best-in-class error is achieved by some h ∗ ∈ H . Then, the minimizability gap is null, M ( H ) = 0 , iff

If further α ↦ glyph[lscript] ( α,k ) is injective and glyph[lscript] k = glyph[lscript] ( α k , k ) for all k ∈ [ n ] , then, the condition is equivalent to ∀ k ∈ [ n ] , h ∗ ( x ) = α k a.s. for x ∈ X k . Furthermore, the minimizability gap is bounded by glyph[epsilon1] , M ( H ) ≤ glyph[epsilon1] , iff

In particular, the condition implies:

Proof. By definition of h ∗ , using the shorthand p k = P [ y = k ] for any k ∈ [ n ] , we can write

Since the distribution is deterministic, the expected pointwise infimum can be rewritten as follows:

where glyph[lscript] k = inf α ∈ A glyph[lscript] ( α,k ) , for any k ∈ [ n ] . Thus, we have

In view of that, since, by definition of glyph[lscript] k , the expressions within the conditional expectations are non-negative, the equality M ( H ) = 0 holds iff glyph[lscript] ( h ∗ ( x ) , k ) -glyph[lscript] k = 0 almost surely for any x in X k , ∀ k ∈ [ n ] . Furthermore, M ( H ) ≤ glyph[epsilon1] is equivalent to

that is

In light of the non-negativity of the expressions, this implies in particular:

This completes the proof.

The theorem suggests that, under those assumptions, for the surrogate minimizability gap to be zero, the score vector of best-in-class hypothesis must be piecewise constant with specific values on X k . The existence of such a hypothesis in H depends both on the complexity of the decision surface separating X k and on that of the hypothesis set H . The theorem also suggests that when the score vector of best-in-class classifier glyph[epsilon1] -approximates α k over X k for any k ∈ [ n ] , then the minimizability gap is bounded by glyph[epsilon1] . The existence of such a hypothesis in H depends on the complexity of the decision surface.

J.2 Stochastic scenario

In the previous sections, we analyze instances featuring small minimizability gaps in a deterministic setting. Moving forward, we aim to extend this analysis to the stochastic scenario. We first provide two general results, which are the direct extensions of that in the deterministic scenario. The following result shows that the minimizability gap is zero when there exists h ∗ ∈ H that matches α ∗ ( x ) for all x , where α ∗ ( x ) is the minimizer of the conditional error. It also shows that the minimizability gap is bounded by glyph[epsilon1] when there exists h ∗ ∈ H whose conditional error glyph[epsilon1] -approximates best-in-class conditional error for all x .

Theorem J.2. The best-in-class error is achieved by some h ∗ ∈ H and the minimizability gap is null, M ( H ) = 0 , iff there exists h ∗ ∈ H such that

If further α ↦ E y [ glyph[lscript] ( α,y ) ∣ x ] is injective and inf α ∈ A E y [ glyph[lscript] ( α,y ) ∣ x ] = E y [ glyph[lscript] ( α ∗ ( x ) , y ) ∣ x ] , then, the condition is equivalent to h ∗ ( x ) = α ∗ ( x ) a.s. for x ∈ X . Furthermore, the minimizability gap is bounded by glyph[epsilon1] , M ( H ) ≤ glyph[epsilon1] , iff there exists h ∗ ∈ H such that

Proof. Assume that the best-in-class error is achieved by some h ∗ ∈ H . Then, we can write

The expected pointwise infimum can be rewritten as follows:

Thus, we have

In view of that, since, by the definition of infimum, the expressions within the marginal expectations are non-negative, the condition that M ( H ) = 0 implies that

On the other hand, if there exists h ∗ ∈ H such that the condition (3) holds, then,

Since M ( H ) is non-negative, the inequality is achieved. Thus, we have

If there exists h ∗ ∈ H such that the condition (4) holds, then,

On the other hand, since we have

M ( H ) ≤ glyph[epsilon1] implies that

This completes the proof.

J.3 Examples

Note that when the distribution is assumed to be deterministic, the condition (4) and condition (15) are reduced to the condition of Theorem 6.1 in binary classification and that of Theorem J.1 in multi-class classification, respectively. In the stochastic scenario, the existence of such a hypothesis not only depends on the complexity of the decision surface, but also depends on the distributional assumption on the conditional distribution p ( x ) = ( p ( y ∣ x )) y ∈ Y , where p ( y ∣ x ) = D ( Y = y ∣ X = x ) is the conditional probability of Y = y given X = x . In the binary classification, we have p ( x ) = ( p (+ 1 ∣ x ) , p (-1 ∣ x )) , where p (+ 1 ∣ x ) + p (-1 ∣ x ) = 1 . For simplicity, we use the notation η ( x ) and 1 -η ( x ) to represent p (+ 1 ∣ x ) and p (-1 ∣ x ) respectively. In the multi-class classification with Y = { 1 , . . . , n } , we have p ( x ) = ( p ( 1 ∣ x ) , p ( 2 ∣ x ) , . . . , p ( n ∣ x )) where n is the number of classes. As examples, here too, we examine exponential loss and logistic loss in binary classification and multi-class logistic loss in multi-class classification.

A. Example: binary classification. Let glyph[epsilon1] ∈ [ 0 , 1 2 ] . We denote by X + the subset of X over which η ( x ) = 1 -glyph[epsilon1] and by X -the subset of X over which η ( x ) = glyph[epsilon1] . Let H be a family of functions h with ∣ h ( x )∣ ≤ Λ for all x ∈ X and such that all values in [-Λ , + Λ ] can be reached. Thus, A = [-Λ , Λ ] for any x ∈ X . Consider the exponential loss: glyph[lscript] ( h, x, y ) = e -yh ( x ) . Then, for any x ∈ X and α ∈ A , we have

Thus, it is not hard to see that for any glyph[epsilon1] ≤ 1 e 2Λ + 1 , the infimum inf α ∈ A E y [ glyph[lscript] ( α,y ) ∣ x ] can be achieved by α ∗ ( x ) = { Λ x ∈ X + -Λ x ∈ X -∈ A . Similarly, for the logistic loss glyph[lscript] ( h, x, y ) = log ( 1 + e -yh ( x ) ) , we have that

and for glyph[epsilon1] ≤ 1 e Λ + 1 , the infimum inf α ∈ A E y [ glyph[lscript] ( α,y ) ∣ x ] can be achieved by α ∗ ( x ) = { Λ x ∈ X + -Λ x ∈ X -.

Therefore, by Theorem 6.2, for these distributions and loss functions, when the best-in-class classifier h ∗ glyph[epsilon1] -approximates α + = Λ over X + and α -= -Λ over X -, then the minimizability gap is bounded by glyph[epsilon1] . The existence of such a hypothesis in H depends on the complexity of the decision surface. For example, as previously noted, when the decision surface is characterized by a hyperplane, a hypothesis set of linear functions, coupled with a sigmoid activation function, can offer a highly effective approximation (see Figure 1 for illustration).

B. Example: multi-class classification. Let glyph[epsilon1] ∈ [ 0 , 1 2 ] . We denote by X k the subset of X over which p ( k ∣ x ) = 1 -glyph[epsilon1] and p ( j ∣ x ) = glyph[epsilon1] n -1 for j ≠ k . Let H be a family of functions h with

∣ h ( x, ⋅)∣ ≤ Λ for all x ∈ X and such that all values in [-Λ , + Λ ] can be reached. Thus, A = [-Λ , Λ ] n for any x ∈ X . Consider the multi-class logistic loss: glyph[lscript] ( h, x, y ) = -log [ e h ( x,y ) ∑ y ′ ∈ Y e h ( x,y ′ ) ] . For any

α = [ α 1 , . . . , α n ] ∈ A , we denote by S k = e α k ∑ k ′ ∈[ n ] e αk ′ . Then, for any x ∈ X and α ∈ A ,

Thus, it is not hard to see that for any glyph[epsilon1] ≤ n -1 e 2 λ + n -1 , the infimum inf α ∈ A E y [ glyph[lscript] ( α,y ) ∣ x ] can be achieved by α ∗ ( x ) = [-Λ , . . . , Λ , . . . , -Λ ] , where Λ occupies the k -th position for x ∈ X k . Therefore, by Theorem 6.2, for these distributions and loss functions, when the best-in-class classifier h ∗ glyph[epsilon1] -approximates α k = [-Λ , . . . , Λ k -th , . . . , -Λ ] over X k , then the minimizability gap is bounded by glyph[epsilon1] . The existence of such a hypothesis in H depends on the complexity of the decision surface.

K Proof for binary margin-based losses (Theorem 4.2)

Theorem 4.2 (Upper and lower bound for binary margin-based losses) . Let H be a complete hypothesis set. Assume that Φ is convex, twice continuously differentiable, and satisfies the inequalities Φ ′ ( 0 ) > 0 and Φ ′′ ( 0 ) > 0 . Then, the following property holds: T ( t ) = Θ ( t 2 ) ; that is, there exist positive constants C > 0 , c > 0 , and T > 0 such that Ct 2 ≥ T ( t ) ≥ ct 2 , for all 0 < t ≤ T .

Proof. Since Φ is convex and in C 2 , f t is also convex and differentiable with respect to u . For any t ∈ [ 0 , 1 ] , differentiate f t with respect to u , we have

Consider the function F defined over R 2 by F ( t, a ) = 1 -t 2 Φ ′ ( a )-1 + t 2 Φ ′ (-a ) . Observe that F ( 0 , 0 ) = 0 and that the partial derivative of F with respect to a at ( 0 , 0 ) is Φ ′′ ( 0 ) > 0 :

Consequently, by the implicit function theorem, there exists a continuously differentiable function a such that F ( t, a ( t )) = 0 in a neighborhood [-glyph[epsilon1], glyph[epsilon1] ] around zero. Thus, by the convexity of f t and the definition of F , for t ∈ [ 0 , glyph[epsilon1] ] , inf u ∈ R f t ( u ) is reached by a ( t ) and we can denote it by a ∗ t . Then, a ∗ t is continuously differentiable over [ 0 , glyph[epsilon1] ] . The minimizer a ∗ t satisfies the following equality:

Specifically, at t = 0 , we have Φ ′ ( a ∗ 0 ) = Φ ′ (-a ∗ 0 ) . Since Φ is convex, its derivative Φ ′ is nondecreasing. Therefore, if a ∗ 0 were non-zero, then Φ ′ would be constant over the segment [-∣ a ∗ 0 ∣ , ∣ a ∗ 0 ∣] . This would contradict the condition Φ ′′ ( 0 ) > 0 , as a constant function cannot have a positive second derivative at any point. Thus, we must have a ∗ 0 = 0 and since Φ ′ is non-decreasing and Φ ′′ ( 0 ) > 0 , we have a ∗ t > 0 for all t ∈ ( 0 , glyph[epsilon1] ] . By Theorem 4.1 and Taylor's theorem with an integral remainder, T can be expressed as follows: for any t ∈ [ 0 , glyph[epsilon1] ] ,

Since a ∗ t is a function of class C 1 , we can differentiate (16) with respect to t , which gives the following equality for any t in ( 0 , glyph[epsilon1] ] :

Taking the limit t → 0 yields

This implies that

Since lim t → 0 a ∗ t t = da ∗ t dt ( 0 ) = Φ ′ ( 0 ) Φ ′′ ( 0 ) > 0 , we have a ∗ t = Θ ( t ) .

Since Φ ′′ ( 0 ) > 0 and Φ ′′ is continuous, there is a non-empty interval [-α, + α ] over which Φ ′′ is positive. Since a ∗ 0 = 0 and a ∗ t is continuous, there exists a sub-interval [ 0 , glyph[epsilon1] ′ ] ⊆ [ 0 , glyph[epsilon1] ] over which a ∗ t ≤ α . Since Φ ′′ is continuous, it admits a minimum and a maximum over any compact set and we can define c = min u ∈[-α,α ] Φ ′′ ( u ) and C = max u ∈[-α,α ] Φ ′′ ( u ) . c and C are both positive since we have Φ ′′ ( 0 ) > 0 . Thus, for t in [ 0 , glyph[epsilon1] ′ ] , by (17), the following inequality holds:

L Proof for comp-sum losses (Theorem 5.3)

Theorem 5.3 (Upper and lower bound for comp-sum losses) . Assume that Φ is convex, twice continuously differentiable, and satisfies the properties Φ ′ ( u ) < 0 and Φ ′′ ( u ) > 0 for any u ∈ ( 0 , 1 2 ] . Then, the following property holds: T ( t ) = Θ ( t 2 ) .

Proof. For any τ ∈ [ 1 n , 1 2 ] , define the function T τ by

where

We aim to establish a lower bound for inf τ ∈[ 1 n , 1 2 ] T τ ( t ) . For any fixed τ ∈ [ 1 n , 1 2 ] , this situation is parallel to that of binary classification (Theorem 4.1 and Theorem 4.2), since we have Φ ′ τ ( 0 ) = Φ ′ ( τ ) < 0 and Φ ′′ τ ( 0 ) = Φ ′′ ( τ ) > 0 . Let a ∗ t,τ denotes the minimizer of f t,τ over R . By applying Theorem 5.2 to the function F ∶ ( t, u, τ ) ↦ f ′ t,τ ( u ) = 1 -t 2 Φ ′ τ ( u ) -1 + t 2 Φ ′ τ (-u ) and the convexity of f t,τ with respect to u , a ∗ t,τ exists, is unique and is continuously differentiable over [ 0 , t ′ 0 ] × [ 1 n , 1 2 ] , for some t ′ 0 > 0 . Moreover, by using the fact that f ′ 0 ,τ ( τ ) > 0 and f ′ 0 ,τ (-τ ) < 0 , and the convexity of f 0 ,τ with respect to u , we have ∣ a ∗ 0 ,τ ∣ ≤ τ , ∀ τ ∈ [ 1 n , 1 2 ] . By the continuity of a ∗ t,τ , we have ∣ a ∗ t,τ ∣ ≤ τ over [ 0 , t 0 ] × [ 1 n , 1 2 ] , for some t 0 > 0 and t 0 ≤ t ′ 0 .

Next, we will leverage the proof of Theorem 4.2. Adopting a similar notation, while incorporating the τ subscript to distinguish different functions Φ τ and f t,τ , we can write

where a ∗ t,τ verifies

We first show the lower bound inf τ ∈[ 1 n , 1 2 ] -a ∗ t,τ = Ω ( t ) . Given the equalities (18), it follows that for any τ , the following holds: lim t → 0 (-a ∗ t,τ + c τ t ) = 0 . For any τ ∈ [ 1 n , 1 2 ] , t ↦ (-a ∗ t,τ + c τ t ) is a continuous function over [ 0 , t 0 ] since a ∗ t,τ is a function of class C 1 . Since the infimum over a fixed compact set of a family of continuous functions is continuous, t ↦ inf τ ∈[ 1 n , 1 2 ] {-a ∗ t,τ + c τ t } is continuous. Thus, for any glyph[epsilon1] > 0 , there exists t 1 > 0 , t 1 ≤ t 0 , such that for any t ∈ [ 0 , t 1 ] ,

where c = inf τ ∈[ 1 n , 1 2 ] -c τ . Since Φ ′ τ ( 0 ) and Φ ′′ τ ( 0 ) are positive and continuous functions of τ , this infimum is attained over the compact set [ 1 n , 1 2 ] , leading to c > 0 . Since the lower bound holds uniformly over τ , this shows that for t ∈ [ 0 , t 1 ] , we have inf τ ∈[ 1 n , 1 2 ] -a ∗ t,τ = Ω ( t ) .

Now, since for any τ ∈ [ 1 n , 1 2 ] , -a ∗ t,τ is a function of class C 1 and thus continuous, its supremum over a compact set, sup τ ∈[ 1 n , 1 2 ] -a ∗ t,τ , is also continuous and is bounded over [ 0 , t 1 ] by some a > 0 . For ∣ u ∣ ≤ a and τ ∈ [ 1 n , 1 2 ] , we have 1 2 -a ≤ τ + u ≤ 1 2 + a and 1 2 -a ≤ τ -u ≤ 1 2 + a . Since Φ ′′ is positive and continuous, it reaches its minimum C > 0 over the compact set [ 1 2 -a, 1 2 + a ] . Thus, we can write

Thus, for t ≤ t 1 , we have which implies

which implies

Similarly, we aim to establish an upper bound for inf τ ∈[ 1 n , 1 2 ] T τ ( t ) . We first show the upper bound sup τ ∈[ 1 n , 1 2 ] -a ∗ t,τ = O ( t ) . Given the equalities (18), it follows that for any τ , the following holds: lim t → 0 (-a ∗ t,τ + c τ t ) = 0 . For any τ ∈ [ 1 n , 1 2 ] , t ↦(-a ∗ t,τ + c τ t ) is a continuous function over [ 0 , t 0 ] since a ∗ t,τ is a function of class C 1 . Since the supremum over a fixed compact set of a family of continuous functions is continuous, t ↦ sup τ ∈[ 1 n , 1 2 ] {-a ∗ t,τ + c τ t } is continuous. Thus, for any glyph[epsilon1] > 0 , there exists t 1 > 0 , t 1 ≤ t 0 , such that for any t ∈ [ 0 , t 1 ] ,

where c = sup τ ∈[ 1 n , 1 2 ] -c τ . Since Φ ′ τ ( 0 ) and Φ ′′ τ ( 0 ) are positive and continuous functions of τ , this supremum is attained over the compact set [ 1 n , 1 2 ] , leading to c > 0 . Since the upper bound holds uniformly over τ , this shows that for t ∈ [ 0 , t 1 ] , we have sup τ ∈[ 1 n , 1 2 ] -a ∗ t,τ = O ( t ) .

Now, since for any τ ∈ [ 1 n , 1 2 ] , -a ∗ t,τ is a function of class C 1 and thus continuous, its supremum over a compact set, sup τ ∈[ 1 n , 1 2 ] -a ∗ t,τ , is also continuous and is bounded over [ 0 , t 1 ] by some a > 0 . For ∣ u ∣ ≤ a and τ ∈ [ 1 n , 1 2 ] , we have 1 2 -a ≤ τ + u ≤ 1 2 + a and 1 2 -a ≤ τ -u ≤ 1 2 + a . Since Φ ′′ is

positive and continuous, it reaches its maximum C > 0 over the compact set [ 1 2 -a, 1 2 + a ] . Thus, we can write

Thus, for t ≤ t 1 , we have

This completes the proof.

M Proof for constrained losses (Theorem 5.5)

Theorem 5.5 (Upper and lower bound for constrained losses) . Assume that Φ is convex, twice continuously differentiable, and satisfies the properties Φ ′ ( u ) > 0 and Φ ′′ ( u ) > 0 for any u ≥ 0 . Then, for any A > 0 , the following property holds:

Proof. For any τ ∈ [ 0 , A ] , define the function T τ by

where

We aim to establish a lower bound for inf τ ∈[ 0 ,A ] T τ ( t ) . For any fixed τ ∈ [ 0 , A ] , this situation is parallel to that of binary classification (Theorem 4.1 and Theorem 4.2), since we also have Φ ′ τ ( 0 ) = Φ ′ ( τ ) > 0 and Φ ′′ τ ( 0 ) = Φ ′′ ( τ ) > 0 . Let a ∗ t,τ denotes the minimizer of f t,τ over R . By applying Theorem 5.2 to the function F ∶ ( t, u, τ ) ↦ f ′ t,τ ( u ) = 1 -t 2 Φ ′ τ ( u ) -1 + t 2 Φ ′ τ (-u ) and the convexity of f t,τ with respect to u , a ∗ t,τ exists, is unique and is continuously differentiable over [ 0 , t 0 ] × [ 0 , A ] , for some t 0 > 0 .

Next, we will leverage the proof of Theorem 4.2. Adopting a similar notation, while incorporating the τ subscript to distinguish different functions Φ τ and f t,τ , we can write

where a ∗ t,τ verifies

We first show the lower bound inf τ ∈[ 0 ,A ] a ∗ t,τ = Ω ( t ) . Given the equalities (19), it follows that for any τ , the following holds: lim t → 0 ( a ∗ t,τ -c τ t ) = 0 . For any τ ∈ [ 0 , A ] , t ↦( a ∗ t,τ -c τ t ) is a continuous function over [ 0 , t 0 ] since a ∗ t,τ is a function of class C 1 . Since the infimum over a fixed compact set of a family of continuous functions is continuous, t ↦ inf τ ∈[ 0 ,A ] { a ∗ t,τ -c τ t } is continuous. Thus, for any glyph[epsilon1] > 0 , there exists t 1 > 0 , t 1 ≤ t 0 , such that for any t ∈ [ 0 , t 1 ] ,

which implies

where c = inf τ ∈[ 0 ,A ] c τ . Since Φ ′ τ ( 0 ) and Φ ′′ τ ( 0 ) are positive and continuous functions of τ , this infimum is attained over the compact set [ 0 , A ] , leading to c > 0 . Since the lower bound holds uniformly over τ , this shows that for t ∈ [ 0 , t 1 ] , we have inf τ ∈[ 0 ,A ] a ∗ t,τ = Ω ( t ) .

Now, since for any τ ∈ [ 0 , A ] , a ∗ t,τ is a function of class C 1 and thus continuous, its supremum over a compact set, sup τ ∈[ 0 ,A ] a ∗ t,τ , is also continuous and is bounded over [ 0 , t 1 ] by some a > 0 . For ∣ u ∣ ≤ a and τ ∈ [ 0 , A ] , we have A -a ≤ τ + u ≤ A + a and A -a ≤ τ -u ≤ A + a . Since Φ ′′ is positive and continuous, it reaches its minimum C > 0 over the compact set [ A -a, A + a ] . Thus, we can write

Thus, for t ≤ t 1 , we have which implies

Similarly, we aim to establish an upper bound for inf τ ∈[ 0 ,A ] T τ ( t ) . We first show the upper bound sup τ ∈[ 0 ,A ] a ∗ t,τ = O ( t ) . Given the equalities (19), it follows that for any τ , the following holds: lim t → 0 ( a ∗ t,τ -c τ t ) = 0 . For any τ ∈ [ 0 , A ] , t ↦ ( a ∗ t,τ -c τ t ) is a continuous function over [ 0 , t 0 ] since a ∗ t,τ is a function of class C 1 . Since the supremum over a fixed compact set of a family of continuous functions is continuous, t ↦ sup τ ∈[ 0 ,A ] { a ∗ t,τ -c τ t } is continuous. Thus, for any glyph[epsilon1] > 0 , there exists t 1 > 0 , t 1 ≤ t 0 , such that for any t ∈ [ 0 , t 1 ] ,

where c = sup τ ∈[ 0 ,A ] c τ . Since Φ ′ τ ( 0 ) and Φ ′′ τ ( 0 ) are positive and continuous functions of τ , this supremum is attained over the compact set [ 0 , A ] , leading to c > 0 . Since the upper bound holds uniformly over τ , this shows that for t ∈ [ 0 , t 1 ] , we have sup τ ∈[ 0 ,A ] a ∗ t,τ = O ( t ) .

Now, since for any τ ∈ [ 0 , A ] , a ∗ t,τ is a function of class C 1 and thus continuous, its supremum over a compact set, sup τ ∈[ 0 ,A ] a ∗ t,τ , is also continuous and is bounded over [ 0 , t 1 ] by some a > 0 . For ∣ u ∣ ≤ a and τ ∈ [ 0 , A ] , we have A -a ≤ τ + u ≤ A + a and A -a ≤ τ -u ≤ A + a . Since Φ ′′ is positive and continuous, it reaches its maximum C > 0 over the compact set [ A -a, A + a ] . Thus, we can write

Thus, for t ≤ t 1 , we have

This completes the proof.

N Analysis of the function of τ

Let F be the function defined by

where Φ is a convex function in C 2 with Φ ′ , Φ ′′ > 0 . In light of the analysis of the previous sections, for any ( τ, t ) , there exists a unique function a t,τ solution of the maximization (supremum in F ), a C 1 function over a neighborhood U of ( τ, 0 ) with a 0 ,τ = 0 , a t,τ > 0 for t > 0 , and ∂a t,τ ∂t ( 0 , τ ) = Φ ′ ( τ ) Φ ′′ ( τ ) = c τ . Thus, we have lim t → 0 a t,τ tc τ = 1 . The optimality of a t,τ implies

Thus, the partial derivative of F over the appropriate neighborhood U is given by

Since Φ ′ is continuous, by the mean value theorem, there exists ξ ∈ ( τ, τ + a t,τ ) such that Φ ′ ( τ + a t,τ ) -Φ ′ ( τ ) = a t,τ Φ ′′ ( ξ ) . Thus, we can write

Note that if Φ ′′ is locally non-increasing, then we have Φ ′′ ( ξ ) ≤ Φ ′′ ( τ ) and for t sufficiently small, since Φ ′ is increasing and a t,τ tc τ ∼ 1 :

In that case, for any A > 0 , we can find a neighborhood O of t around zero over which ∂F ∂τ ( t, τ ) is defined for all ( t, τ ) ∈ O ×[ 0 , A ] and ∂F ∂τ ( t, τ ) ≥ 0 . From this, we can conclude that the infimum of F over τ ∈ [ 0 , A ] is reached at zero for t sufficiently small ( t ∈ O ).

O Generalization bounds

Let S = (( x 1 , y 1 ) , . . . , ( x m , y m )) be a sample drawn from D m . Denote by ̂ h S an empirical minimizer within H for the surrogate loss glyph[lscript] : ̂ h S ∈ argmin h ∈ H 1 m ∑ m i = 1 glyph[lscript] ( h, x i , y i ) . Let H glyph[lscript] denote the hypothesis set {( x, y ) ↦ glyph[lscript] ( h, x, y )∶ h ∈ H } and R glyph[lscript] m ( H ) its Rademacher complexity. We also write B glyph[lscript] to denote an upper bound for glyph[lscript] . Then, given the following H -consistency bound:

for any δ > 0 , with probability at least 1 -δ over the draw of an i.i.d. sample S of size m , the following estimation bound holds for ̂ h S :

Proof. By the standard Rademacher complexity bounds [Mohri et al., 2018], for any δ > 0 , with probability at least 1 -δ , the following holds for all h ∈ H :

For any glyph[epsilon1] > 0 , by definition of the infimum, there exists h ∗ ∈ H such that E glyph[lscript] ( h ∗ ) ≤ E ∗ glyph[lscript] ( H ) + glyph[epsilon1] . By the definition of ̂ h S , we obtain

Since the inequality holds for all glyph[epsilon1] > 0 , it implies the following:

Plugging in this inequality in the H -consistency bound (21) completes the proof.

These bounds for surrogate loss minimizers, expressed in terms of minimizability gaps, offer more detailed and informative insights compared to existing bounds based solely on approximation errors. Our analysis of growth rates suggests that for commonly used smooth loss functions, Γ varies near zero with a square-root dependency. Furthermore, this dependency cannot be generally improved for arbitrary distributions.

P Future work

We demonstrated a universal square-root growth rate for smooth surrogate losses commonly used in binary and multi-class classification. This result holds across all data distributions. A promising direction for future research would be to further investigate how incorporating specific distributional assumptions could refine these results.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the section 'Our results' in the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 3, Section 4, Section 5, Section 6, and Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is theoretical in nature and we do not anticipate any immediate negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the growth rate of the H-consistency bounds (which subsume excess risk bounds) for various smooth surrogate losses commonly used in binary and multiclass classification. Specifically, for binary classification, the work establishes a tight square-root growth rate near zero (under mild conditions) for margin-based surrogate losses. For multiclass classification, the work establishes a tight square-root growth rate near zero (under mild conditions) for two families of surrogate losses: comp-sum and constrained losses. Finally, the work also studies how the number of classes affects these bounds, as well as the minimizability gaps in the bounds.\n\nSTRENGTHS:\n**Originality**\n\nA comprehensive analysis of the growth rate of the H-consistency bounds (which subsume excess risk bounds) for various smooth surrogate losses commonly used in binary and multiclass classification:\n-  For binary classification, the work establishes a tight square-root growth rate near zero (under mild conditions) for margin-based surrogate losses (Theorem 4.2). In particular, the lower bound requires weaker conditions than [Frongillo and Waggoner, 2021,\nTheorem 4], and the upper bound is new.\n- For multiclass classification, the work establishes a tight square-root growth rate near zero (under mild conditions) for two families of surrogate losses: comp-sum and constrained losses (Theorems 5.3 and 5.5).\n\nRelated work has been properly cited.\n\n**Quality**\n\nThe work is technically sound. Proofs are given for theoretical results.\n\n**Clarity**\n\nThe paper is clearly written and well-organized. Although it is a bit dry, as a researcher working in the related field, I did not find it very hard to read.\n\n**Significance**\n\nThe comprehensive analysis presented in this work promotes a deeper understanding of different surrogate losses. It is helpful for researchers studying surrogate losses and consistency.\n\nWEAKNESSES:\nI did not find any obvious weaknesses.\n\nQUESTIONS:\n1. Based on this work, do you have any concrete suggestions for practitioners to choose surrogate losses (assuming they do not know about H-consistency at all)?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

comments: SUMMARY:
Since optimizing zero-one loss is intractable and it does not have properties such as differentiability, a common approach in learning theory is to replace it with a surrogate loss function. H-consistency bounds relate the excess error for surrogate loss to zero-one loss. This paper establishes a square-root growth rate near zero for smooth surrogate losses in binary and multi-class classification, providing both upper and lower bounds under mild assumptions.

STRENGTHS:
The paper proves both upper and lower bounds for H-consistency with smooth surrogate losses. Previous results only provided lower bounds, but this paper presents lower bounds with fewer conditions and also shows a matching upper bound of the square root, applicable to both binary classification and multiclass classification, such as Comp-sum losses.

WEAKNESSES:
The paper studies only smooth surrogate losses, not piecewise linear ones such as hinge loss. However, using smooth surrogate loss functions is very common in machine learning applications.

QUESTIONS:
The results are based on the assumption that the hypothesis class is complete. Can you explain why this condition is necessary and what happens to the growth rate when this is not satisfied? It seems that many practical hypothesis classes don't meet this condition.

LIMITATIONS:
The paper does not have any potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
This paper presents a comprehensive analysis of the growth rate of H-consistency bounds (and excess error bounds) for various
surrogate losses for some intractable loss used in classification. The authors prove a square-root growth rate near zero for smooth margin-based surrogate losses in binary classification and comp-sum and constrained loss used in multi-class classification, providing both upper and lower bounds under mild assumptions.

STRENGTHS:
The paper provides solid analysis for the H-consistency and excess error bound and these provide good guidance for selecting good surrogate loss for classification tasks where the target loss is hard to optimize.
The theoretical analysis is novel and provides good insight for handling intractable target loss.

WEAKNESSES:
The paper should provide a more intuitive statement on the motivation and implication of these error bounds and provide more insights about the proof.

QUESTIONS:
1. The error bounds are local bounds. Can the author provide insights on how to have global bounds and the neighborhood for the local bounds to hold?
 2. Can the authors provide more insights on how these bounds help us select surrogate functions?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides an analysis of the growth rate of H-consistency bound for surrogate losses in binary and multi-class classification. The authors prove square root growth rate near zero for smooth margin-based surrogate losses for binary classification as well as  for smooth comp-sum and constrained losses for multiclass classification. Since minimizability gaps makes a big difference between the bounds of different surrogates, the authors also analyze these gaps to guide in selecting better surrogates.  In section , the authors introduce H consistency bounds and they build upon this to derive results for binary and multi-class classification in later sections.\n\nSTRENGTHS:\nI must say, I am not an expert in this area of research. I find this paper pretty well written. In theorem 4.2, the authors prove that the transformation function tau is precisely of the order t^2 for a class of margin based loss function that is smooth and follows some other properties as mentioned in the main statement.  Hence, the growth rate for these loss functions is precisely square-root.  Further, the authors derive similar result for multiclass classification for comp sum losses and constrained losses. Because there is a minimizability gap term in the H consistency bound hence even with identical growth rates, surrogate losses can vary in their H-consistency bounds. The authors show that in the case of multiclass classification, minimizability gap scales with number of classes.\n\nWEAKNESSES:\nI have a few very basic questions, regarding the work.\n1. I understand that H consistency-based bound helps convert a surrogate-based bound to a bound that we require. How is it an improvement over earlier work Bartlett et al. (Convexity, Classification, and Risk Bounds). H consistence based bound contains the term minimizability gap which was not existent in the previous work that I cited. \n\n\n2. I also understand that the faster the bound for surrogate loss will be, the better bound for the actual loss could be obtained. However, I am wondering if we provide a fast rate using small ball method or local Rademacher complexity based method for the surrogates, under what conditions the fast rate for the actual loss can still be recovered or is it lost always ?\n\nIt would be also great if you could explain simply the implications of getting a precise rate for the transformation function as I understand it might not give you a lower bound on the estimation error because of equation 2 is not an eqaulity. Am I missing something?\n\nI am currently giving it a borderline accept and am happy to reconsider it after the rebuttal.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}