Title: $H$-Consistency Bounds: Characterization and Extensions
Venue: NeurIPS 2023
Accepted: yes
Keywords: consistency

H-consistency

characterization

learning theory
Abstract: A series of recent publications by Awasthi et al. have introduced the key notion of *$H$-consistency bounds* for surrogate loss functions.  These are upper bounds on the zero-one estimation error of any predictor in a hypothesis set, expressed in terms of its surrogate loss estimation error. They are both non-asymptotic and hypothesis set-specific and thus stronger and more informative than Bayes-consistency. However, determining if they hold and deriving these bounds have required a specific proof and analysis for each surrogate loss. Can we derive more general tools and characterizations? This paper provides both a general characterization and an extension of $H$-consistency bounds for multi-class classification. We present new and tight $H$-consistency bounds for both the family of constrained losses and that of comp-sum losses, which covers the familiar cross-entropy, or logistic loss applied to the outputs of a neural network. We further extend our analysis beyond the completeness assumptions adopted in previous studies and cover more realistic bounded hypothesis sets.  Our characterizations are based on error transformations, which are explicitly defined for each formulation. We illustrate the application of our general results through several special examples. A by-product of our analysis is the observation that a recently derived multi-class $H$-consistency bound for cross-entropy reduces to an excess bound and is not significant. Instead, we prove a much stronger and more significant guarantee.

H -Consistency Bounds: Characterization and Extensions

Abstract

A series of recent publications by Awasthi, Mao, Mohri, and Zhong [2022b] have introduced the key notion of H -consistency bounds for surrogate loss functions. These are upper bounds on the zero-one estimation error of any predictor in a hypothesis set, expressed in terms of its surrogate loss estimation error. They are both non-asymptotic and hypothesis set-specific and thus stronger and more informative than Bayes-consistency. However, determining if they hold and deriving these bounds have required a specific proof and analysis for each surrogate loss. Can we derive more general tools and characterizations? This paper provides both a general characterization and an extension of H -consistency bounds for multi-class classification. We present new and tight H -consistency bounds for both the family of constrained losses and that of comp-sum losses, which covers the familiar crossentropy, or logistic loss applied to the outputs of a neural network. We further extend our analysis beyond the completeness assumptions adopted in previous studies and cover more realistic bounded hypothesis sets. Our characterizations are based on error transformations, which are explicitly defined for each formulation. We illustrate the application of our general results through several special examples. A by-product of our analysis is the observation that a recently derived multi-class H -consistency bound for cross-entropy reduces to an excess bound and is not significant. Instead, we prove a much stronger and more significant guarantee.

1 Introduction

Bayes-consistency is an important property of surrogate loss functions. It requires that minimizing the surrogate excess error over the family of all measurable functions leads to the minimization of the target error loss in the limit [Steinwart, 2007]. This property applies to a broad family of convex margin-based losses in binary classification [Zhang, 2004a, Bartlett et al., 2006], as well as some extensions in multi-class classification [Tewari and Bartlett, 2007]. However, Bayes-consistency does not apply to the hypothesis sets commonly used for learning, such as the family of linear models or that of neural networks, which of course do not include all measurable functions. Furthermore, it is also only an asymptotic property and does not supply any convergence guarantee.

To address these limitations, a series of recent publications by Awasthi, Mao, Mohri, and Zhong [2022b] introduced the key notion of H -consistency bounds for surrogate loss functions. These are upper bounds on the zero-one estimation error of any predictor in a hypothesis set, expressed in terms of its surrogate loss estimation error. They are both non-asymptotic and hypothesis set-specific and thus stronger and more informative than Bayes-consistency. However, determining the validity of these bounds and deriving them have required a specific proof and analysis for each surrogate loss. Can we derive more general tools and characterizations for H -consistency bounds?

This paper provides both a general characterization and an extension of H -consistency bounds for multi-class classification. Previous approaches to deriving these bounds required the development of new proofs for each specific case. In contrast, we introduce the general concept of an error transformation function that serves as a very general tool for deriving such guarantees with tightness guarantees. We show that deriving an H -consistency bound for comp-sum losses and constrained losses for both complete and bounded hypothesis sets can be reduced to the calculation of their corresponding error transformation function. Our general tools and tight bounds show several remarkable advantages: first, they improve existing bounds for complete hypothesis sets previously proven in [Awasthi et al., 2022b]; second, they encompass all previously comp-sum and constrained losses studied thus far as well as many new ones [Awasthi et al., 2022a, Mao et al., 2023h]; third, they extend beyond the completeness assumption adopted in previous work; fourth, they provide novel guarantees for bounded hypothesis sets; and, finally, they help prove a much stronger and more significant guarantee for logistic loss with linear hypothesis set than [Zheng et al., 2023].

Previous work. Here, we briefly discuss recent studies of H -consistency bounds by Awasthi et al. [2022a,b], Mao et al. [2023h] and Zheng et al. [2023]. Awasthi et al. [2022a] introduced and studied H -consistency bounds in binary classification. They provided a series of tight H -consistency bounds for bounded hypothesis set of linear models and one-hidden-layer neural networks. The subsequent study [Awasthi et al., 2022b] further generalized the framework to multi-class classification and presented an extensive study of H -consistency bounds for diverse multi-class surrogate losses, including negative results for max losses [Crammer and Singer, 2001] and positive results for sum losses [Weston and Watkins, 1998], and constrained losses [Lee et al., 2004]. However, the hypothesis sets examined in their analysis were assumed to be complete, which rules out the bounded hypothesis sets typically used in practice. Moreover, the final bounds derived from [Awasthi et al., 2022b] are based on ad hoc methods and may not be tight. [Mao et al., 2023h] complemented this previous work by studying a wide family of comp-sum losses in the multi-class classification, which generalizes the sum-losses and includes as special cases the logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], the generalized cross-entropy loss [Zhang and Sabuncu, 2018], and the mean absolute error loss [Ghosh et al., 2017]. Here too, the completeness assumption on the hypothesis sets was adopted and their H -consistency bounds do not apply to common bounded hypothesis sets in practice. Recently, Zheng et al. [2023] proved H -consistency bounds for multi-class logistic loss with bounded linear hypothesis sets. However, their bounds require a crucial distributional assumption, under which the minimizability gaps coincide with the approximation errors. Thus, their bounds can be recovered as excess error bounds, which are less significant.

Other related work on H -consistency bounds includes H -consistency bounds for pairwise ranking [Mao, Mohri, and Zhong, 2023d,e]; theoretically grounded surrogate losses and algorithms for learning with abstention supported by H -consistency bounds, including the study of score-based abstention [Mao, Mohri, and Zhong, 2023f], predictor-rejector abstention [Mao, Mohri, and Zhong, 2023c] and learning to abstain with a fixed predictor with application in decontextualization [Mohri, Andor, Choi, Collins, Mao, and Zhong, 2023]; principled approaches for learning to defer with multiple experts that benefit from strong H -consistency bounds, including the single-stage scenario [Mao, Mohri, and Zhong, 2023b] and a two-stage scenario [Mao, Mohri, Mohri, and Zhong, 2023a]; H -consistency theory and algorithms for adversarial robustness [Awasthi et al., 2021a,b, 2023a, Mao et al., 2023h, Awasthi et al., 2023b]; and efficient algorithms and loss functions for structured prediction with stronger H -consistency guarantees [Mao et al., 2023g].

Structure of this paper. We present new and tight H -consistency bounds for both the family of comp-sum losses (Section 4.1) and that of constrained losses (Section 5.1), which cover the familiar cross-entropy, or logistic loss applied to the outputs of a neural network. We further extend our analysis beyond the completeness assumptions adopted in previous studies and cover more realistic bounded hypothesis sets (Section 4.2 and 5.2). Our characterizations are based on error transformations, which are explicitly defined for each formulation. We illustrate the application of our general results through several special examples. A by-product of our analysis is the observation that a recently derived multi-class H -consistency bound for cross-entropy reduces to an excess bound independent of the hypothesis set. Instead, we prove a much stronger and more significant guarantee (Section 4.2).

We give a comprehensive discussion of related work in Appendix A. We start with some basic definitions and notation in Section 2.

2 Preliminaries

Wedenote by X the input space, by Y the output space, and by D a distribution over X × Y . Weconsider the standard scenario of multi-class classification, where Y = { 1 , . . . , n } . Given a hypothesis set H of functions mapping X × Y to R , the multi-class classification problem consists of finding a hypothesis h ∈ H with small generalization error R glyph[lscript] 0 -1 ( h ) , defined by R glyph[lscript] 0 -1 ( h ) = E ( x,y )∼ D [ glyph[lscript] 0 -1 ( h, x, y )] , where glyph[lscript] 0 -1 ( h, x, y ) = 1 h ( x )≠ y is the multi-class zero-one loss with h ( x ) = argmax y ∈ Y h ( x, y ) the prediction of h for the input point x . We also denote by H ( x ) the set of all predictions associated to input x generated by functions in H , that is, H ( x ) = { h ( x )∶ h ∈ H } .

We will analyze the guarantees of surrogate multi-class losses in terms of the zero-one loss. We denote by glyph[lscript] a surrogate loss and by R glyph[lscript] ( h ) its generalization error, R glyph[lscript] ( h ) = E ( x,y )∼ D [ glyph[lscript] ( h, x, y )] . For a loss function glyph[lscript] , we define the best-in-class generalization error within a hypothesis set H as R ∗ glyph[lscript] ( H ) = inf h ∈ H R glyph[lscript] ( h ) , and refer to R glyph[lscript] ( h ) -R ∗ glyph[lscript] ( H ) as the estimation error . We will study the key notion of H -consistency bounds [Awasthi et al., 2022a,b], which are upper bounds on the zero-one estimation error of any predictor in a hypothesis set, expressed in terms of its surrogate loss estimation error, for some real-valued function f that is non-decreasing:

These bounds imply that the zero-one estimation error is at most f ( glyph[epsilon1] ) whenever the surrogate loss estimation error is bounded by glyph[epsilon1] . Thus, the learning guarantees provided by H -consistency bounds are both non-asymptotic and hypothesis set-specific. The function f appearing in these bounds is expressed in terms of a minimizability gap , which is a quantity measuring the difference of bestin-class error R ∗ glyph[lscript] ( H ) and the expected best-in-class conditional error E x [ C ∗ glyph[lscript] ( H , x )] : M glyph[lscript] ( H ) = R ∗ glyph[lscript] ( H ) -E X [ C ∗ glyph[lscript] ( H , x )] , where C glyph[lscript] ( h, x ) = E y ∣ x [ glyph[lscript] ( h, x, y )] and C ∗ glyph[lscript] ( H , x ) = inf h ∈ H C glyph[lscript] ( h, x ) are the conditional error and best-in-class conditional error respectively. We further write ∆ C glyph[lscript], H = C glyph[lscript] ( h, x ) -C ∗ glyph[lscript] ( H , x ) to denote the conditional regret . Note that that the minimizability gap is an inherent quantity depending on a hypothesis set H and the loss function glyph[lscript] .

By Lemma 1, the minimizability gap for the zero-one loss, M glyph[lscript] 0 -1 ( H ) , coincides with its approximation error A glyph[lscript] 0 -1 ( H ) = R ∗ glyph[lscript] 0 -1 ( H ) -R ∗ glyph[lscript] 0 -1 ( H all ) when the set of all possible predictions generated by H covers the label space Y . This holds for typical hypothesis sets used in practice. However, for a surrogate loss glyph[lscript] , the minimizability gap M glyph[lscript] ( H ) is always upper bounded by and in general finer than its approximation error A glyph[lscript] ( H ) = R ∗ glyph[lscript] ( H ) -R ∗ glyph[lscript] ( H all ) since M glyph[lscript] ( H ) = A glyph[lscript] ( H ) -I glyph[lscript] ( H ) , where H all is the family of all measurable functions and I glyph[lscript] ( H ) = E x [ C ∗ glyph[lscript] ( H , x ) -C ∗ glyph[lscript] ( H all , x )] (see Appendix B for a more detailed discussion). Thus, an H -consistency bound, expressed as follows for some increasing function Γ :

is more favorable than an excess error bound expressed in terms of approximation errors R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H ) + A glyph[lscript] 0 -1 ( H ) ≤ Γ ( R glyph[lscript] ( h ) -R ∗ glyph[lscript] ( H ) + A glyph[lscript] ( H )) . Here, Γ is typically linear or the square-root function modulo constants. When H = H all , the family of all measurable functions, an H -consistency bound coincides with the excess error bound and implies Bayes-consistency by taking the limit. It is therefore a stronger guarantee than an excess error bound and Bayes-consistency.

The minimizability gap is always non-negative, since the infimum of the expectation is greater than or equal to the expectation of infimum. Furthermore, as shown in Appendix B, when H is the family of all measurable functions or when the Bayes-error coincides with the best-in-class error, that is, R ∗ glyph[lscript] ( H ) = R ∗ glyph[lscript] ( H all ) , the minimizability gap vanishes. In such cases, (1) implies the H -consistency of a surrogate loss glyph[lscript] with respect to the zero-one loss glyph[lscript] 0 -1 :

In the next sections, we will provide both a general characterization and an extension of H -consistency bounds for multi-class classification. Before proceeding, we first introduce a useful lemma from [Awasthi et al., 2022b] which characterizes the conditional regret of zero-one loss explicitly. We denote by p ( x ) = ( p ( x, 1 ) , . . . , p ( x, n )) as the conditional distribution of y given x .

Lemma 1. For zero-one loss glyph[lscript] 0 -1 , the best-in-class conditional error and the conditional regret for glyph[lscript] 0 -1 can be expressed as follows: for any x ∈ X , we have

3 Comparison with previous work

Here, we briefly discuss previous studies of H -consistency bounds [Awasthi et al., 2022a,b, Zheng et al., 2023, Mao et al., 2023h] in standard binary or multi-class classification and compare their results with those we present.

Awasthi et al. [2022a] studied H -consistency bounds in binary classification. They provided a series of tight H -consistency bounds for the bounded hypothesis set of linear models H bi lin and one-hidden-layer neural networks H bi NN , defined as follows:

where B , W , and Λ are positive constants and where (⋅) + = max (⋅ , 0 ) . We will show that our bounds recover these binary classification H -consistency bounds.

The scenario of multi-class classification is more challenging and more crucial in applications. Recent work by Awasthi et al. [2022b] showed that max losses [Crammer and Singer, 2001], defined as glyph[lscript] max ( h, x, y ) = max y ′ ≠ y Φ ( h ( x, y ) -h ( x, y ′ )) for some convex and non-increasing function Φ , cannot admit meaningful H -consistency bounds, unless the distribution is deterministic. They also presented a series of H -consistency bounds for sum losses [Weston and Watkins, 1998] and constrained losses [Lee et al., 2004] for symmetric and complete hypothesis sets, that is such that:

for some family F of functions mapping from X to R . The completeness assumption rules out the bounded hypothesis sets typically used in practice such as H lin . Moreover, the final bounds derived from [Awasthi et al., 2022b] are based on ad hoc proofs and may not be tight. In contrast, we will study both the complete and bounded hypothesis sets, and provide a very general tool to derive H -consistency bounds. Our bounds are tighter than those of Awasthi et al. [2022b] given for complete hypothesis sets and extend beyond the completeness assumption.

[Mao et al., 2023h] complemented the work of [Awasthi et al., 2022b] by studying a wide family of comp-sum losses in multi-class classification, which generalized the sum-losses and included as special cases the logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], the generalized crossentropy loss [Zhang and Sabuncu, 2018], and the mean absolute error loss [Ghosh et al., 2017]. Here too, the completeness assumption was adopted, thus their H -consistency bounds do not apply to common bounded hypothesis sets used in practice. We illustrate the application of our general results through a broader set of surrogate losses than [Mao et al., 2023h] and significantly generalize the bounds of [Mao et al., 2023h] to bounded hypothesis sets.

Recently, Zheng et al. [2023] proved H -consistency bounds for logistic loss with linear hypothesis sets in the multi-class classification: H lin = { x ↦ w y ⋅ x + b y ∣ ∥ w y ∥ ≤ W, ∣ b y ∣ ≤ B,y ∈ Y } . However, their bounds require a crucial distributional assumption under which, the minimizability gaps M glyph[lscript] 0 -1 ( H lin ) and M glyph[lscript] log ( H lin ) coincide with the approximation errors R glyph[lscript] 0 -1 ( H lin )-R ∗ glyph[lscript] 0 -1 ( H all ) and R glyph[lscript] log ( H lin )-R ∗ glyph[lscript] log ( H all ) respectively (see the note before [Zheng et al., 2023, Appendix F]). Thus, their bounds can be recovered as excess error bounds R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H all ) ≤ √ 2 ( R glyph[lscript] log ( h ) -R ∗ glyph[lscript] log ( H all )) 1 2 , which are less significant. In contrast, our H lin -consistency bound are much finer and take into account the role of the parameter B and that of the number of labels n . Thus, we provide stronger and more significant guarantees for logistic loss with linear hypothesis set than [Zheng et al., 2023].

In summary, our general tools offer the remarkable advantages of deriving tight bounds, which improve upon the existing bounds of Awasthi et al. [2022b] given for complete hypothesis sets, cover the comp-sum and constrained losses considered in [Awasthi et al., 2022a, Mao et al., 2023h] as well as new ones, extend beyond the completeness assumption with novel guarantees valid for bounded hypothesis sets, and are much stronger and more significant guarantees for logistic loss with linear hypothesis sets than those of Zheng et al. [2023].

4 Comp-sum losses

In this section, we present a general characterization of H -consistency bounds for comp-sum losses , a family of loss functions including the logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], the sum exponential loss [Weston and Watkins, 1998, Awasthi et al., 2022b], the generalized cross entropy loss [Zhang and Sabuncu, 2018], the mean absolute error loss [Ghosh et al., 2017], and many other loss functions used in applications.

This is a family of loss functions defined via the composition of a non-negative and non-decreasing function Ψ with the sum exponential losses (see [Mao et al., 2023h]):

This expression can be equivalently written as glyph[lscript] comp ( h, x, y ) = Φ ( e h ( x,y ) ∑ y ′ ∈ Y e h ( x,y ′ ) ) , where Φ ∶ u ↦ Ψ ( 1 -u u ) is a non-increasing auxiliary function from [ 0 , 1 ] to R + ∪{+∞} . As an example, the logistic loss corresponds to the choice Φ ∶ u ↦-log ( u ) and the sum exponential loss to Φ ∶ u ↦ 1 -u u .

4.1 H -consistency bounds

In previous work, deriving H -consistency bounds has required giving new proofs for each instance. The following result provides a very general tool for deriving such bounds with tightness guarantees. We introduce an error transformation function and show that deriving an H -consistency bound for comp-sum losses can be reduced to the calculation of this function.

Theorem 2 ( H -consistency bound for comp-sum losses ) . Assume that H is symmetric and complete and that T comp is convex. Then, the following inequality holds for any hypothesis h ∈ H and any distribution

with T comp an H -estimation error transformation for comp-sum losses defined for all t ∈ [ 0 , 1 ] by T comp ( t ) =

Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and a hypothesis h ∈ H such that R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] comp ( h ) -R ∗ glyph[lscript] comp ( H ) + M glyph[lscript] comp ( H ) = T comp ( t ) .

Thus, Theorem 2 shows that, when T comp is convex, to make these guarantees explicit, all that is needed is to calculate T comp . Moreover, the last statement shows the tightness of the guarantees derived using this function. The constraints in T comp are due to the forms that the conditional probability vector and scoring functions take. These forms become more flexible for n > 2 , leading to intricate constraints. Note that our H -consistency bounds are distribution-independent and we cannot claim tightness across all distributions.

The general expression of T comp in Theorem 2 is complex, but it can be considerably simplified under some broad assumptions, as shown by the following result.

Theorem 3 ( characterization of T comp ) . Assume that Φ is convex, differentiable at 1 2 and Φ ′ ( 1 2 ) < 0 . Then, T comp can be expressed as follows:

The proof of this result as well as that of other theorems in this section are given in Appendix C.

Examples. We now illustrate the application of our theory through several examples. To do so, we compute the H -estimation error transformation T comp for comp-sum losses and present the results in

Table 1. Remarkably, by applying Theorem 2, we are able to obtain the same H -consistency bounds for comp-sum losses with Φ ( t ) = -log ( t ) , 1 t -1 , 1 q ( 1 -t q ) , q ∈ ( 0 , 1 ) and 1 -t as those derived using ad hoc methods in [Mao et al., 2023h], and a novel tight H -consistency bound for the new

The calculation of T comp for all entries of Table 1 is detailed in Appendix C.3. To illustrate the effectiveness of our general tools, here, we show how the error transformation function can be straightforwardly calculated in the case of the new surrogate loss glyph[lscript] sq .

Theorem 4 ( H -consistency bound for a new comp-sum loss ) . Assume that H is symmetric and complete. Then, for all h ∈ H and any distribution, the following tight bound holds.

Proof. For n = 2 , plugging in Φ ( t ) = ( 1 -t ) 2 in Theorem 3, gives

Similarly, for n > 2 , plugging in Φ ( t ) = ( 1 -t ) 2 in Theorem 3 yields

By Theorem 2, the bound obtained is tight, which completes the proof.

4.2 Extension to non-complete/bounded hypothesis sets: comp-sum losses

As pointed out earlier, the hypothesis sets typically used in practice are bounded. Let F be a family of real-valued functions f with ∣ f ( x )∣ ≤ Λ ( x ) for all x ∈ X and such that all values in [-Λ ( x ) , + Λ ( x )] can be reached, where Λ ( x ) > 0 is a fixed function on X . We will study hypothesis sets H in which each scoring function is bounded:

This holds for most hypothesis sets used in practice. The symmetric and complete hypothesis sets studied in previous work correspond to the special case of H where Λ ( x ) = +∞ for all x ∈ X . The hypothesis set of linear models H lin , defined by

is also a special instance of H where Λ ( x ) = W ∥ x ∥ + B . Let us emphasize that previous studies did not establish any H -consistency bound for these general hypothesis sets, H .

Theorem 5 ( H -consistency bound for comp-sum losses ) . Assume that T comp is convex. Then, the following inequality holds for any hypothesis h ∈ H and any distribution:

with T comp the H -estimation error transformation for comp-sum losses defined for all t ∈ [ 0 , 1 ] by T comp ( t ) =

where C = [ max { s min -τ 2 , τ 1 -s max } , min { s max -τ 2 , τ 1 -s min }] , s max = 1 1 +( n -1 ) e -2inf x Λ ( x ) and s min = 1 1 +( n -1 ) e 2inf x Λ ( x ) . Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and h ∈ H such that R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] comp ( h ) -R ∗ glyph[lscript] comp ( H ) + M glyph[lscript] comp ( H ) = T comp ( t ) .

This theorem significantly broadens the applicability of our framework as it encompasses bounded hypothesis sets. The last statement of the theorem further shows the tightness of the H -consistency bounds derived using this error transformation function. We now illustrate the application of our theory through several examples.

A. Example: logistic loss. We first consider the multinomial logistic loss, that is glyph[lscript] comp with Φ ( u ) = -log ( u ) , for which we give the following guarantee.

Theorem 6 ( H -consistency bounds for logistic loss ) . For any h ∈ H and any distribution, we have

The proof of Theorem 6 is given in Appendix E.2. With the help of some simple calculations, we can derive a simpler upper bound:

When the relative difference between s min and s max is small, the coefficient of the linear term in Γ explodes. On the other hand, making that difference large essentially turns Γ into a square-root function for all values. In general, Λ is not infinite since a regularization is used, which controls both the complexity of the hypothesis set and the magnitude of the scores.

Comparison with [Mao et al., 2023h]. For the symmetric and complete hypothesis sets H considered in [Mao et al., 2023h], Λ ( x ) = +∞ , s max = 1 , s min = 0 , Ψ ( t ) = 1 + t 2 log ( 1 + t ) + 1 -t 2 log ( 1 -t ) and Γ ( t ) = √ 2 t . By Theorem 6, this yields an H -consistency bound for the logistic loss.

Corollary 7 ( H -consistency bounds for logistic loss ) . Assume that H is symmetric and complete. Then, for any h ∈ H and any distribution, we have

Corollary 7 recovers the H -consistency bounds of Mao et al. [2023h].

Comparison with [Awasthi et al., 2022a] and [Zheng et al., 2023]. For the linear models H lin = {( x, y ) ↦ w y ⋅ x + b y ∣ ∥ w y ∥ ≤ W, ∣ b y ∣ ≤ B } , we have Λ ( x ) = W ∥ x ∥ + B . By Theorem 6, we obtain H lin -consistency bounds for logistic loss.

Corollary 8 ( H lin -consistency bounds for logistic loss ) . For any h ∈ H lin and any distribution,

with the H lin -estimation error transformation in [Awasthi et al., 2022a]. Thus, Corollary 8 includes as a special case the H lin -consistency bounds given by Awasthi et al. [2022a] for binary classification.

Our bounds of Corollary 8 improves upon the multi-class H lin -consistency bounds of recent work [Zheng et al., 2023, Theorem 3.3] in the following ways: i) their bound holds only for restricted distributions while our bound holds for any distribution; ii) their bound holds only for restricted values of the estimation error R glyph[lscript] log ( h ) -R ∗ glyph[lscript] log ( H lin ) while ours holds for any value in R and more precisely admits a piecewise functional form; iii) under their distributional assumption, the minimizability gaps M glyph[lscript] 0 -1 ( H lin ) and M glyph[lscript] log ( H lin ) coincide with the approximation errors R glyph[lscript] 0 -1 ( H lin ) -R ∗ glyph[lscript] 0 -1 ( H all ) and R glyph[lscript] log ( H lin ) -R ∗ glyph[lscript] log ( H all ) respectively (see the note before [Zheng et al., 2023, Appendix F]). Thus, their bounds can be recovered as an excess error bound √ 1

R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H all ) ≤ 2 [ R glyph[lscript] log ( h ) -R ∗ glyph[lscript] log ( H all )] 2 , which is not specific to the hypothesis set H and thus not as significant. In contrast, our H lin -consistency bound is finer and takes into account the role of the parameter B as well as the number of labels n ; iv) [Zheng et al., 2023, Theorem 3.3] only offers approximate bounds that are not tight; in contrast, by Theorem 5, our bound is tight.

Note that our H -consistency bound in Theorem 6 are not limited to specific hypothesis set forms. They are directly applicable to various types of hypothesis sets including neural networks. For example, the same derivation can be extended to one-hidden-layer neural networks studied in [Awasthi et al., 2022a] and their multi-class generalization by calculating and substituting the corresponding Λ ( x ) . As a result, we can obtain novel and tight H -consistency bounds for bounded neural network hypothesis sets in multi-class classification, which highlights the versatility of our general tools.

B. Example: sum exponential loss . We then consider the sum exponential loss, that is glyph[lscript] comp with Φ ( u ) = 1 -u u . By computing the error transformation in Theorem 5, we obtain the following result.

Theorem 9 ( H -consistency bounds for sum exponential loss ) . For any h ∈ H and any distribution,

The proof of Theorem 9 is given in Appendix E.3. Observe that 1 -√ 1 -t 2 ≥ t 2 / 2 . By Theorem 9, making s min close to zero, that is making Λ close to infinite for any x ∈ X , essentially turns Ψ into a square function for all values. In general, Λ is not infinite since a regularization is used in practice, which controls both the complexity of the hypothesis set and the magnitude of the scores.

C. Example: generalized cross-entropy loss and mean absolute error loss . Due to space limitations, we present the results for these loss functions in Appendix E.

5 Constrained losses

In this section, we present a general characterization of H -consistency bounds for constrained loss , that is loss functions defined via a constraint, as in [Lee et al., 2004]:

with the constraint that ∑ y ∈ Y h ( x, y ) = 0 for a non-negative and non-increasing auxiliary function Φ .

5.1 H -consistency bounds

As in the previous section, we prove a result that supplies a very general tool, an error transformation function for deriving H -consistency bounds for constrained losses. When T cstnd is convex, to make these guarantees explicit, we only need to calculate T cstnd .

Theorem 10 ( H -consistency bound for constrained losses ) . Assume that H is symmetric and complete. Assume that T cstnd is convex. Then, for any hypothesis h ∈ H and any distribution,

with H -estimation error transformation for constrained losses defined on t ∈ [ 0 , 1 ] by T cstnd ( t ) =

Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and a hypothesis h ∈ H such that R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] cstnd ( h ) -R ∗ glyph[lscript] cstnd ( H ) + M glyph[lscript] cstnd ( H ) = T cstnd ( t ) .

Here too, the theorem guarantees the tightness of the bound. This general expression of T cstnd can be considerably simplified under some broad assumptions, as shown by the following result.

Theorem 11 ( characterization of T cstnd ) . Assume that Φ is convex, differentiable at zero and Φ ′ ( 0 ) < 0 . Then, T cstnd can be expressed as follows:

The proof of all the results in this section are given in Appendix D.

Examples. We now compute the H -estimation error transformation for constrained losses and present the results in Table 2. Here, we present the simplified T cstnd by using the lower bound in Theorem 11. Remarkably, by applying Theorem 10, we are able to obtain tighter H -consistency bounds for constrained losses with Φ = Φ hinge , Φ sq -hinge , Φ exp than those derived using ad hoc methods in [Awasthi et al., 2022b], and a novel H -consistency bound for the new constrained loss glyph[lscript] cstnd ( h, x, y ) = ∑ y ′ ≠ y ( 1 + h ( x, y ′ )) 2 with Φ ( t ) = ( 1 -t ) 2 .

5.2 Extension to non-complete or bounded hypothesis sets

As in the case of comp-sum losses, we extend our results beyond the completeness assumption adopted in previous work and establish H -consistency bounds for bounded hypothesis sets. This significantly broadens the applicability of our framework.

Theorem 12 ( H -consistency bound for constrained losses ) . Assume that T cstnd is convex. Then, the following inequality holds for any hypothesis h ∈ H and any distribution:

with T cstnd the H -estimation error transformation for constrained losses defined for all t ∈ [ 0 , 1 ] by T cstnd ( t ) =

where C = [ max { τ 1 , -τ 2 } -Λ min , min { τ 1 , -τ 2 } + Λ min ] and Λ min = inf x ∈ X Λ ( x ) . Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and a hypothesis h ∈ H such that R glyph[lscript] 0 -1 ( h )-R ∗ glyph[lscript] 0 -1 ( H )+ M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] cstnd ( h ) -R ∗ glyph[lscript] cstnd ( H ) + M glyph[lscript] cstnd ( H ) = T cstnd ( t ) .

The proof is presented in Appendix F.1. Next, we illustrate the application of our theory through an example of constrained exponential losses, that is glyph[lscript] cstnd with Φ ( t ) = e -t . By using the error transformation in Theorem 12, we obtain new H -consistency bounds in Theorem 13 (see Appendix F.2 for the proof) for bounded hypothesis sets H .

Theorem 13 ( H -consistency bounds for constrained exponential loss ) . Let Φ ( t ) = e -t . For any h ∈ H and any distribution,

Awasthi et al. [2022b] proves H -consistency bounds for constrained exponential losses when H is symmetric and complete. Theorem 13 significantly generalizes those results to the non-complete hypothesis sets. Different from the complete case, the functional form of our new bounds has two pieces which corresponds to the linear and the square root convergence respectively, modulo the constants. Furthermore, the coefficient of the linear piece depends on the the magnitude of Λ min . When Λ min is small, the coefficient of the linear term in Ψ -1 explodes. On the other hand, making Λ min large essentially turns Ψ -1 into a square-root function.

6 Discussion

Here, we further elaborate on the practical value of our tools and H -consistency bounds. Our contributions include a more general and convenient mathematical tool for proving H -consistency bounds, along with tighter bounds that enable a better comparison of surrogate loss functions and extensions beyond previous completeness assumptions. As mentioned by [Awasthi et al., 2022b], given a hypothesis set H , H -consistency bounds can be used to compare different surrogate loss functions and select the most favorable one, which depends on the functional form of the H -consistency bound; the smoothness of the surrogate loss and its optimization properties; approximation properties of the surrogate loss function controlled by minimizability gaps; and the dependency on the number of classes in the multiplicative constant. Consequently, a tighter H -consistency bound provides a more accurate comparison, as a loose bound might not adequately capture the full advantage of using one surrogate loss. In contrast, Bayes-consistency does not take into account the hypothesis set and is an asymptotic property, thereby failing to guide the comparison of different surrogate losses.

Another application of our H -consistency bounds involves deriving generalization bounds for surrogate loss minimizers [Mao et al., 2023h], expressed in terms of the same quantities previously discussed. Therefore, when dealing with finite samples, a tighter H -consistency bound could also result in a corresponding tighter generalization bound. Moreover, our novel results extend beyond previous completeness assumptions, offering guarantees applicable to bounded hypothesis sets commonly used with regularization. This enhancement provides meaningful learning guarantees. Technically, our error transformation function serves as a very general tool for deriving H -consistency bounds with tightness guarantees. These functions are defined within each class of loss functions including comp-sum losses and constrained losses, and their formulation depends on the structure of the individual loss function class, the range of the hypothesis set and the number of classes. To derive explicit bounds, all that is needed is to calculate these error transformation functions. Under some broad assumptions on the auxiliary function within a loss function, these error transformation functions can be further distilled into more simplified forms, making them straightforward to compute.

7 Conclusion

We presented a general characterization and extension of H -consistency bounds for multi-class classification. We introduced new tools for deriving such bounds with tightness guarantees and illustrated their benefits through several applications and examples. Our proposed method is a significant advance in the theory of H -consistency bounds for multi-class classification. It can provide a general and powerful tool for deriving tight bounds for a wide variety of other loss functions and hypothesis sets. We believe that our work will open up new avenues of research in the field of multi-class classification consistency.

References

Agarwal and S. Agarwal. On consistent surrogate risk minimization and property elicitation. In Conference on Learning Theory , pages 4-22, 2015.

Awasthi, N. Frank, A. Mao, M. Mohri, and Y. Zhong. Calibration and consistency of adversarial surrogate losses. In Advances in Neural Information Processing Systems , pages 9804-9815, 2021a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. A finer calibration analysis for adversarial robustness. arXiv preprint arXiv:2105.01550 , 2021b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. H -consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , pages 1117-1174, 2022a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H -consistency bounds. In Advances in neural information processing systems , 2022b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094, 2023a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. DC-programming for neural network optimizations. Journal of Global Optimization , 2023b.

L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Berkson. Application of the logistic function to bio-assay. Journal of the American Statistical Association , 39:357--365, 1944.

Berkson. Why I prefer logits to probits. Biometrics , 7(4):327--339, 1951.

Blondel. Structured prediction with projection oracles. In Advances in neural information processing systems , 2019.

D.-R. Chen and T. Sun. Consistency of multiclass empirical risk minimization methods based on convex loss. Journal of Machine Learning Research , 7:2435-2447, 2006.

D.-R. Chen and D.-H. Xiang. The consistency of multicategory support vector machines. Advances in Computational Mathematics , 24(1):155-169, 2006.

Ciliberto, L. Rosasco, and A. Rudi. A consistent regularization approach for structured prediction. In Advances in neural information processing systems , 2016.

Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In Algorithmic Learning Theory , pages 67-82, 2016a.

Cortes, G. DeSalvo, and M. Mohri. Boosting with abstention. In Advances in Neural Information Processing Systems , pages 1660-1668, 2016b.

Cortes, G. DeSalvo, and M. Mohri. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence , to appear, 2023.

Crammer and Y. Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research , 2(Dec):265-292, 2001.

Dembczynski, W. Kotlowski, and E. Hüllermeier. Consistent multilabel ranking through univariate losses. arXiv preprint arXiv:1206.6401 , 2012.

Dogan, T. Glasmachers, and C. Igel. A unified view on multi-class support vector classification. Journal of Machine Learning Research , 17:1-32, 2016.

Finocchiaro, R. M. Frongillo, and B. Waggoner. An embedding framework for the design and analysis of consistent polyhedral surrogates. arXiv preprint arXiv:2206.14707 , 2022.

Frongillo and B. Waggoner. Surrogate regret bounds for polyhedral losses. In Advances in Neural Information Processing Systems , pages 21569-21580, 2021.

Gao and Z.-H. Zhou. On the consistency of multi-label learning. In Conference on learning theory , pages 341-358, 2011.

Gao and Z.-H. Zhou. On the consistency of AUC pairwise optimization. In International Joint Conference on Artificial Intelligence , 2015.

Ghosh, H. Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , 2017.

Kuznetsov, M. Mohri, and U. Syed. Multi-class deep boosting. In Advances in Neural Information Processing Systems , pages 2501-2509, 2014.

Lee, Y. Lin, and G. Wahba. Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association , 99(465):67-81, 2004.

Liu. Fisher consistency of multicategory support vector machines. In Artificial intelligence and statistics , pages 291-298, 2007.

Long and R. Servedio. Consistency versus realizable H-consistency for multiclass classification. In International Conference on Machine Learning , pages 801-809, 2013.

Mao, C. Mohri, M. Mohri, and Y. Zhong. Two-stage learning to defer with multiple experts. In Advances in neural information processing systems , 2023a.

Mao, M. Mohri, and Y. Zhong. Principled approaches for learning to defer with multiple experts. arXiv preprint arXiv:2310.14774 , 2023b.

Mao, M. Mohri, and Y. Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. arXiv preprint arXiv:2310.14772 , 2023c.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds for pairwise misranking loss surrogates. In International conference on Machine learning , 2023d.

Mao, M. Mohri, and Y. Zhong. Ranking with abstention. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023e.

Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for scorebased multi-class abstention. arXiv preprint arXiv:2310.14770 , 2023f.

Mao, M. Mohri, and Y. Zhong. Structured prediction with stronger consistency guarantees. In Advances in Neural Information Processing Systems , 2023g.

Mao, M. Mohri, and Y. Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , 2023h.

Mohri, D. Andor, E. Choi, M. Collins, A. Mao, and Y. Zhong. Learning to reject with a fixed predictor: Application to decontextualization. arXiv preprint arXiv:2301.09044 , 2023.

Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning . MIT Press, second edition, 2018.

Narasimhan, H. Ramaswamy, A. Saha, and S. Agarwal. Consistent multiclass algorithms for complex performance measures. In International Conference on Machine Learning , pages 23982407, 2015.

Osokin, F. Bach, and S. Lacoste-Julien. On structured prediction theory with calibrated convex surrogate losses. In Advances in Neural Information Processing Systems , 2017.

Pedregosa, F. Bach, and A. Gramfort. On the consistency of ordinal regression methods. Journal of Machine Learning Research , 18:1-35, 2017.

Á. Pires and C. Szepesvári. Multiclass classification calibration functions. arXiv preprint arXiv:1609.06385 , 2016.

A. Pires, C. Szepesvari, and M. Ghavamzadeh. Cost-sensitive multiclass classification risk bounds. In International Conference on Machine Learning , pages 1391-1399, 2013.

G. Ramaswamy and S. Agarwal. Classification calibration dimension for general multiclass losses. In Advances in Neural Information Processing Systems , 2012.

G. Ramaswamy and S. Agarwal. Convex calibration dimension for multiclass loss matrices. Journal of Machine Learning Research , 17(1):397-441, 2016.

G. Ramaswamy, S. Agarwal, and A. Tewari. Convex calibrated surrogates for low-rank loss matrices with applications to subset ranking losses. In Advances in Neural Information Processing Systems , 2013.

G. Ramaswamy, A. Tewari, and S. Agarwal. Consistent algorithms for multiclass classification with a reject option. arXiv preprint arXiv:1505.04137 , 2015.

Ravikumar, A. Tewari, and E. Yang. On NDCG consistency of listwise ranking methods. In International Conference on Artificial Intelligence and Statistics , pages 618-626, 2011.

Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Tewari and P. L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research , 8(36):1007-1025, 2007.

Thilagar, R. Frongillo, J. J. Finocchiaro, and E. Goodwill. Consistent polyhedral surrogates for top-k classification and variants. In International Conference on Machine Learning , pages 21329-21359, 2022.

Uematsu and Y. Lee. On theoretically optimal ranking functions in bipartite ranking. Journal of the American Statistical Association , 112(519):1311-1322, 2017.

F. Verhulst. Notice sur la loi que la population suit dans son accroissement. Correspondance mathématique et physique , 10:113--121, 1838.

F. Verhulst. Recherches mathématiques sur la loi d'accroissement de la population. Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles , 18:1--42, 1845.

Wang and C. Scott. Weston-Watkins hinge loss and ordered partitions. In Advances in neural information processing systems , pages 19873-19883, 2020.

Wang and C. D. Scott. On classification-calibration of gamma-phi losses. arXiv preprint arXiv:2302.07321 , 2023.

Weston and C. Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.

C. Williamson, E. Vernet, and M. D. Reid. Composite multiclass losses. Journal of Machine Learning Research , 17:1-52, 2016.

Zhang and S. Agarwal. Bayes consistency vs. H-consistency: The interplay between surrogate loss functions and the scoring function class. In Advances in Neural Information Processing Systems , pages 16927-16936, 2020.

Zhang, H. G. Ramaswamy, and S. Agarwal. Convex calibrated surrogates for the multi-label f-measure. In International Conference on Machine Learning , pages 11246-11255, 2020.

Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics , 32(1):56-85, 2004a.

Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004b.

Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems , 2018.

Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. In International Conference on Machine Learning , 2023.

Contents of Appendix

A Related work

The notions of Bayes-consistency (also known as consistency) and calibration have been well studied not only with respect to the binary zero-one loss [Zhang, 2004a, Bartlett et al., 2006, Steinwart, 2007, Mohri et al., 2018], but also with respect to the multi-class zero-one loss [Zhang, 2004b, Tewari and Bartlett, 2007], the general multi-class losses [Ramaswamy and Agarwal, 2012, Narasimhan et al., 2015, Ramaswamy and Agarwal, 2016], the multi-class SVMs [Chen and Sun, 2006, Chen and Xiang, 2006, Liu, 2007, Dogan et al., 2016, Wang and Scott, 2020], the gamma-phi losses [Wang and Scott, 2023], the multi-label losses [Gao and Zhou, 2011, Dembczynski et al., 2012, Zhang et al., 2020], the losses with a reject option [Ramaswamy et al., 2015, Cortes et al., 2016a,b, 2023], the ranking losses [Ravikumar et al., 2011, Ramaswamy et al., 2013, Gao and Zhou, 2015, Uematsu and Lee, 2017], the cost sensitive losses [Pires et al., 2013, Pires and Szepesvári, 2016], the structured losses [Ciliberto et al., 2016, Osokin et al., 2017, Blondel, 2019], the polyhedral losses [Frongillo and Waggoner, 2021, Finocchiaro et al., 2022], the Topk classification losses [Thilagar et al., 2022], the proper losses [Agarwal and Agarwal, 2015, Williamson et al., 2016] and the losses of ordinal regression [Pedregosa et al., 2017].

Bayes-consistency only holds for the full family of measurable functions, which of course is distinct from the more restricted hypothesis set used by a learning algorithm. Therefore, a hypothesis setdependent notion of H -consistency has been proposed by Long and Servedio [2013] in the realizable setting, used by Zhang and Agarwal [2020] for linear models, and generalized by Kuznetsov et al. [2014] to the structured prediction case. Long and Servedio [2013] showed that there exists a case where a Bayes-consistent loss is not H -consistent while inconsistent losses can be H -consistent. Zhang and Agarwal [2020] further investigated the phenomenon in [Long and Servedio, 2013] and showed that the situation of losses that are not H -consistent with linear models can be remedied by carefully choosing a larger piecewise linear hypothesis set. Kuznetsov et al. [2014] proved positive results for the H -consistency of several multi-class ensemble algorithms, as an extension of H -consistency results in [Long and Servedio, 2013].

Recently, Awasthi et al. [2022a,b], Mao et al. [2023h], Zheng et al. [2023] presented a series of results providing H -consistency bounds. These are upper bounds on the zero-one estimation error of any predictor in a hypothesis set, expressed in terms of its surrogate loss estimation error. They are more informative guarantees than similar excess error bounds derived in the literature, which correspond to the special case where H is the family of all measurable functions [Zhang, 2004a, Bartlett et al., 2006, Mohri et al., 2018]. Awasthi et al. [2022a] studied H -consistency bounds in binary classification. They provided a series of tight H -consistency bounds for bounded hypothesis set of linear models and one-hidden-layer neural networks. The subsequent study [Awasthi et al., 2022b] further generalized the framework to multi-class classification, where they presented a extensive study of H -consistency bounds for diverse multi-class surrogate losses including negative results for max losses [Crammer and Singer, 2001] and positive results for sum losses [Weston and Watkins, 1998], and constrained losses [Lee et al., 2004]. However, the hypothesis sets adopted there were assumed to be complete, which rules out the bounded hypothesis sets typically used in practice. Moreover, the final bounds derived from [Awasthi et al., 2022b] are based on ad hoc methods and may not be tight. [Mao et al., 2023h] complemented the previous work by studying a wide family of comp-sum losses in the multi-class classification, which generalized the sum-losses and included as special cases the logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], the generalized cross-entropy loss [Zhang and Sabuncu, 2018], and the mean absolute error loss [Ghosh et al., 2017]. Here too, the completeness assumption on the hypothesis sets was adopted and their H -consistency bounds do not apply to common bounded hypothesis sets in practice. Zheng et al. [2023] proved H -consistency bounds for multi-class logistic loss with bounded linear hypothesis sets. However, their bounds require a crucial distributional assumption under which, the minimizability gaps coincide with the approximation errors. Thus, their bounds can be recovered as excess error bounds, which are less significant.

This paper provides both a general characterization and an extension of H -consistency bounds for multi-class classification. Our general tools and tight bounds show several remarkable advantages: first, they improve existing bounds for complete hypothesis sets previously proven in [Awasthi et al., 2022b], second, they encompass all previously comp-sum and constrained losses studied thus far as well as many new ones [Awasthi et al., 2022a, Mao et al., 2023h], third, they extend beyond the completeness assumption adopted in previous work, fourth, they give novel guarantees for bounded

hypothesis sets, and finally they help prove a much stronger and more significant guarantee for logistic loss with linear hypothesis set than [Zheng et al., 2023].

Other related work on H -consistency bounds includes: H -consistency bounds for pairwise ranking [Mao et al., 2023d,e]; theoretically grounded surrogate losses and algorithms for learning with abstention supported by H -consistency bounds, including the study of score-based abstention [Mao et al., 2023f], predictor-rejector abstention [Mao et al., 2023c] and learning to abstain with a fixed predictor with application in decontextualization [Mohri et al., 2023]; principled approaches for learning to defer with multiple experts that benefit from strong H -consistency bounds, including the single-stage scenario [Mao et al., 2023b] and a two-stage scenario [Mao et al., 2023a]; H -consistency theory and algorithms for adversarial robustness [Awasthi et al., 2021a,b, 2023a, Mao et al., 2023h, Awasthi et al., 2023b]; and efficient algorithms and loss functions for structured prediction with stronger H -consistency guarantees [Mao et al., 2023g].

B Minimizability gap

This is a brief discussion of minimizability gaps and their properties. By definition, for any loss function glyph[lscript] , the minimizability gap is defined by

B.1 Zero minimizability

Lemma 14. Let glyph[lscript] be a surrogate loss such that for ( x, y ) ∈ X × Y and any measurable function h ∈ H all , the loss glyph[lscript] ( h, x, y ) only depends on h ( x ) and y (thus we can write glyph[lscript] ( h, x, y ) = glyph[lscript] ( h ( x ) , y ) for some function glyph[lscript] ). Then, the minimizabiliy gap vanishes: M glyph[lscript] ( H all ) = 0 .

Proof. Fix glyph[epsilon1] > 0 . Then, by definition of the infimum, for any x ∈ X , there exists h x ∈ H all such that

Now, define the function h by h ( x ) = h x ( x ) , for all x ∈ X . h can be shown to be measurable, for example, when X admits a countable dense subset. Then,

Thus, we have

Since the inequality holds for any glyph[epsilon1] > 0 , we have R ∗ glyph[lscript] ( H all ) = inf h ∈ H all E ( x,y )∼ D [ glyph[lscript] ( h, x, y )] ≤ E x [ C ∗ glyph[lscript] ( H all , x )] . This implies equality since the inequality R ∗ glyph[lscript] ( H ) ≥ E x [ C ∗ glyph[lscript] ( H , x )] holds for any H .

B.2 Relationship with approximation error

Let A glyph[lscript] denote the approximation error of a loss function glyph[lscript] and a hypothesis set H : A glyph[lscript] ( H ) = R ∗ glyph[lscript] ( H ) -R ∗ glyph[lscript] ( H all ) . We will denote by I glyph[lscript] ( H ) the difference of pointwise infima I glyph[lscript] ( H ) = E x [ C ∗ glyph[lscript] ( H , x ) -C ∗ glyph[lscript] ( H all , x )] , which is non-negative. The minimizability gap can be decomposed as

follows in terms of the approximation error and the difference of pointwise infima:

The decomposition immediately implies the following result.

Lemma 15. Let glyph[lscript] be a surrogate loss such that for ( x, y ) ∈ X × Y and any measurable function h ∈ H all , the loss glyph[lscript] ( h, x, y ) only depends on h ( x ) and y (thus we can write glyph[lscript] ( h, x, y ) = glyph[lscript] ( h ( x ) , y ) for some function glyph[lscript] ). Then, for any loss function glyph[lscript] and hypothesis set H , we have: M glyph[lscript] ( H ) ≤ A glyph[lscript] ( H ) .

By Lemma 1, when glyph[lscript] is the zero-one loss, I glyph[lscript] ( H ) = 0 when the hypothesis set generates labels that cover all possible outcomes for each input. However, for a surrogate loss function, I glyph[lscript] ( H ) is non-negative, and is generally non-zero.

Take the example of binary classification and denote the conditional distribution as η ( x ) = D ( Y = 1 ∣ X = x ) . Let H be a family of functions h with ∣ h ( x )∣ ≤ Λ for all x ∈ X and such that all values in [-Λ , + Λ ] can be reached. Consider for example the exponential-based margin loss: glyph[lscript] ( h, x, y ) = e -yh ( x ) . Then, C glyph[lscript] ( h, x ) = η ( x ) e -h ( x ) +( 1 -η ( x )) e h ( x ) . Upon observing this, it becomes apparent that the infimum over all measurable functions can be expressed in the following way, for all x :

while the infimum over H , C ∗ glyph[lscript] ( H , x ) , depends on Λ and can be expressed as

Thus, in the deterministic scenario,

B.3 Significance of H -consistency bounds

As shown in the previous section, for target loss glyph[lscript] 0 -1 , the minimizability gap coincides with the approximation error M glyph[lscript] 0 -1 ( H ) = A glyph[lscript] 0 -1 ( H ) when the hypothesis set generates labels that cover all possible outcomes for each input. However, for a surrogate loss glyph[lscript] , the minimizability gap is generally strictly less than the approximation error M glyph[lscript] ( H ) < A glyph[lscript] ( H ) . Thus, an H -consistency bound, expressed as follows for some increasing function Γ :

is more favorable than an excess error bound expressed in terms of approximation errors:

Here, Γ is typically linear or the square-root function modulo constants. When H = H all , the family of all measurable functions, by Lemma 14, the H -consistency bound coincides with the excess error bound and implies Bayes-consistency by taking the limit. It is therefore a stronger guarantee than an excess error bound and Bayes-consistency.

C Proofs for comp-sum losses

Let y max = argmax y ∈ Y p ( x, y ) and h ( x ) = argmax y ∈ Y h ( x, y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy.

C.1 Proof of H -consistency bounds with T comp (Theorem 2)

Theorem 2 ( H -consistency bound for comp-sum losses ) . Assume that H is symmetric and complete and that T comp is convex. Then, the following inequality holds for any hypothesis h ∈ H and any distribution

with T comp an H -estimation error transformation for comp-sum losses defined for all t ∈ [ 0 , 1 ] by

Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and a hypothesis h ∈ H such that R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] comp ( h ) -R ∗ glyph[lscript] comp ( H ) + M glyph[lscript] comp ( H ) = T comp ( t ) .

Proof. For the comp-sum loss glyph[lscript] comp , the conditional glyph[lscript] comp -risk can be expressed as follows:

where we let S h ( x, y ) = e h ( x,y ) ∑ y ′ ∈ Y e h ( x,y ′ ) for any y ∈ Y with the constraint that ∑ y ∈ Y S h ( x, y ) = 1 . For any h ∈ H such that h ( x ) ≠ y max and x ∈ X , we can always find a family of hypotheses { h µ } ⊂ H such that S h,µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y e hµ ( x,y ′ ) take the following values:

Note that S h,µ satisfies the constraint:

Let p 1 = p ( x, y max ) , p 2 = p ( x, h ( x )) , τ 1 = S h ( x, h ( x )) and τ 2 = S h ( x, y max ) to simplify the notation. Then, by the definition of S h,µ , we have for any h ∈ H and x ∈ X ,

where for n = 2 , an additional constraint τ 1 + τ 2 = 1 is imposed and the expression of T comp is simplified. Since T comp is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

For the second part, we first consider n = 2 . For any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 . For any glyph[epsilon1] > 0 , by the definition of infimum, we can take h ∈ H such that S h ( x, 1 ) = τ glyph[epsilon1] ∈ [ 0 , 1 2 ] and satisfies

Then, and

By letting glyph[epsilon1] → 0 , we prove the tightness for n = 2 . The proof for n > 2 directly extends from the case when n = 2 . Indeed, for any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 , p ( x, y ) = 0 , 3 ≤ y ≤ n . For any glyph[epsilon1] > 0 , by the definition

of infimum, we can take h ∈ H such that S h ( x, 1 ) = τ 1 ,glyph[epsilon1] , S h ( x, 2 ) = τ 2 ,glyph[epsilon1] , S h ( x, y ) = 0 , 3 ≤ y ≤ n and satisfies τ 1 ,glyph[epsilon1] + τ 2 ,glyph[epsilon1] = 1 , and

Then, and

By letting glyph[epsilon1] → 0 , we prove the tightness for n > 2 .

C.2 Characterization of T comp (Theorem 3)

Theorem 3 ( characterization of T comp ) . Assume that Φ is convex, differentiable at 1 2 and Φ ′ ( 1 2 ) < 0 . Then, T comp can be expressed as follows:

Proof. For n = 2 , we have

where the equality can be achieved by τ = 1 2 .

For n > 2 , we have

where we let F ( P,τ 1 , τ 2 , µ ) = P + t 2 [ Φ ( τ 2 ) -Φ ( τ 1 -µ )] + P -t 2 [ Φ ( τ 1 ) -Φ ( τ 2 + µ )] . For simplicity, we assume that Φ is differentiable. For general convex Φ , we can proceed by using left and right derivatives, which are non-decreasing. Differentiate F with respect to µ , we have

Using the fact that P ∈ [ 1 n -1 ∨ t, 1 ] , t ∈ [ 0 , 1 ] and Φ ′ is non-decreasing, we obtain that ∂F ∂µ is nonincreasing. Furthermore, Φ ′ is non-decreasing and non-positive, Φ is non-negative, we obtain that

Φ ′ (+∞) = 0 . This implies that ∂F ∂µ (+∞) ≤ 0 and ∂F ∂µ (-∞) ≥ 0 . Therefore, there exists µ 0 ∈ R such that

By taking µ = τ 1 -τ 2 and using the fact that τ 2 ≤ 1 2 , Φ ′ ( 1 2 ) < 0 , we have

Thus, since ∂F ∂µ is non-increasing, we obtain µ 0 < τ 1 -τ 2 . Differentiate F with respect to τ 2 at µ 0 , we have

Since Φ ′ is non-decreasing, we obtain

which implies that the infimum inf τ 1 ≥ max { τ 2 , 1 n } is achieved when τ 2 = τ 1 . Differentiate F with respect to P at µ 0 and τ 1 = τ 2 , by the convexity of Φ , we obtain

which implies that the infimum inf P ∈[ 1 n -1 , 1 ] is achieved when P = 1 . Above all, we obtain

C.3 Computation of examples

Example: Φ ( t ) = -log ( t ) . For n = 2 , plugging in Φ ( t ) = -log ( t ) in Theorem 3, gives

(minimum achieved at µ = -t 2 )

Similarly, for n > 2 , plugging in Φ ( t ) = -log ( t ) in Theorem 3 yields

Example: Φ ( t ) = 1 t -1 . For n = 2 , plugging in Φ ( t ) = 1 t -1 in Theorem 3, gives

Similarly, for n > 2 , plugging in Φ ( t ) = 1 t -1 in Theorem 3 yields

T comp = inf τ ∈[ 1 n , 1 2 ] { 1 τ -inf µ ∈[-τ,τ ] { 1 + t 2 1 τ -µ + 1 + t 2 1 τ + µ }} = inf τ ∈[ 1 n , 1 2 ] 1 2 τ ( 1 -√ 1 -t 2 ) (minimum achieved at µ = ( 1 -t ) 1 2 -( 1 + t ) 1 2 ( 1 + t ) 1 2 +( 1 -t ) 1 2 τ ) = 1 -√ 1 -t 2 . (minimum achieved at τ = 1 2 )

Example: Φ ( t ) = 1 q ( 1 -t q ) , q ∈ ( 0 , 1 ) . For n = 2 , plugging in Φ ( t ) = 1 q ( 1 -t q ) in Theorem 3, gives

Similarly, for n > 2 , plugging in Φ ( t ) = 1 q ( 1 -t q ) in Theorem 3 yields

(minimum achieved at µ = ( 1 -t ) 1 1 -q -( 1 + t ) 1 1 -q ( 1 + t ) 1 1 -q +( 1 -t ) 1 1 -q τ )

Example: Φ ( t ) = 1 -t . For n = 2 , plugging in Φ ( t ) = 1 -t in Theorem 3, gives

Similarly, for n > 2 , plugging in Φ ( t ) = 1 -t in Theorem 3 yields

Example: Φ ( t ) = ( 1 -t ) 2 . For n = 2 , plugging in Φ ( t ) = ( 1 -t ) 2 in Theorem 3, gives

Similarly, for n > 2 , plugging in Φ ( t ) = ( 1 -t ) 2 in Theorem 3 yields

D Proofs for constrained losses

Let y max = argmax y ∈ Y p ( x, y ) and h ( x ) = argmax y ∈ Y h ( x, y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy.

D.1 Proof of H -consistency bounds with T cstnd (Theorem 10)

Theorem 10 ( H -consistency bound for constrained losses ) . Assume that H is symmetric and complete. Assume that T cstnd is convex. Then, for any hypothesis h ∈ H and any distribution,

with H -estimation error transformation for constrained losses defined on t ∈ [ 0 , 1 ] by T cstnd ( t ) =

Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and a hypothesis h ∈ H such that R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] cstnd ( h ) -R ∗ glyph[lscript] cstnd ( H ) + M glyph[lscript] cstnd ( H ) = T cstnd ( t ) .

Proof. For the constrained loss glyph[lscript] cstnd , the conditional glyph[lscript] cstnd -risk can be expressed as follows:

For any h ∈ H and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ R } ⊂ H such that h µ ( x, ⋅) take the following values:

Note that the hypotheses h µ satisfies the constraint:

Let p 1 = p ( x, y max ) , p 2 = p ( x, h ( x )) , τ 1 = h ( x, h ( x )) and τ 2 = h ( x, y max ) to simplify the notation. Then, by the definition of h µ , we have for any h ∈ H and x ∈ X ,

where for n = 2 , an additional constraint τ 1 + τ 2 = 0 is imposed and the expression of T comp is simplified. Since T cstnd is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

For the second part, we first consider n = 2 . For any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 . For any glyph[epsilon1] > 0 , by the definition of infimum, we can take h ∈ H such that h ( x, 2 ) = τ glyph[epsilon1] ≥ 0 and satisfies

Then, and

By letting glyph[epsilon1] → 0 , we conclude the proof. The proof for n > 2 directly extends from the case when n = 2 . Indeed, For any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 , p ( x, y ) = 0 , 3 ≤ y ≤ n . For any glyph[epsilon1] > 0 , by the definition of infimum, we can take h ∈ H such that h ( x, 1 ) = τ 1 ,glyph[epsilon1] , h ( x, 2 ) = τ 2 ,glyph[epsilon1] , h ( x, y ) = 0 , 3 ≤ y ≤ n and satisfies τ 1 ,glyph[epsilon1] + τ 2 ,glyph[epsilon1] = 0 , and

Then, and

D.2 Characterization of T cstnd (Theorem 11)

Theorem 11 ( characterization of T cstnd ) . Assume that Φ is convex, differentiable at zero and Φ ′ ( 0 ) < 0 . Then, T cstnd can be expressed as follows:

Proof. For n = 2 , we have

where the equality can be achieved by τ = 0 .

For n > 2 , we have

where we let F ( P,τ 1 , τ 2 , µ ) = 2 -P -t 2 [ Φ (-τ 2 ) -Φ (-τ 1 + µ )] + 2 -P + t 2 [ Φ (-τ 1 ) -Φ (-τ 2 -µ )] . For simplicity, we assume that Φ is differentiable. For general convex Φ , we can proceed by using left and right derivatives, which are non-decreasing. Differentiate F with respect to µ , we have

Using the fact that P ∈ [ 1 n -1 , 1 ] , t ∈ [ 0 , 1 ] and Φ ′ is non-decreasing, we obtain that ∂F ∂µ is nonincreasing. Furthermore, Φ ′ is non-decreasing and non-positive, Φ is non-negative, we obtain that Φ ′ (+∞) = 0 . This implies that ∂F ∂µ (+∞) ≤ 0 and ∂F ∂µ (-∞) ≥ 0 . Therefore, there exists µ 0 ∈ R such that

By taking µ = τ 1 -τ 2 and using the fact that Φ ′ ( 0 ) < 0 , we have

Thus, since ∂F ∂µ is non-increasing, we obtain µ 0 < τ 1 -τ 2 . Differentiate F with respect to τ 2 at µ 0 , we have

Since Φ ′ is non-decreasing, we obtain

which implies that the infimum inf τ 1 ≥ max { τ 2 , 0 } is achieved when τ 2 = τ 1 . Differentiate F with respect to P at µ 0 and τ 1 = τ 2 , by the convexity of Φ , we obtain

which implies that the infimum inf P ∈[ 1 n -1 , 1 ] is achieved when P = 1 n -1 . Above all, we obtain

D.3 Computation of examples

Example: Φ ( t ) = Φ exp ( t ) = e -t . For n = 2 , plugging in Φ ( t ) = e -t in Theorem 11, gives

(minimum achieved at µ = 1 2 log 1 t 1 + t )

For n > 2 , plugging in Φ ( t ) = e -t in Theorem 11 yields

}} (minimum achieved at τ = 0 ) = --(minimum achieved at µ = 1 2 log 2 -t 2 + t )

Example: Φ ( t ) = Φ hinge ( t ) = max { 0 , 1 -t } . For n = 2 , plugging in Φ ( t ) = max { 0 , 1 -t } in Theorem 11, gives

(minimum achieved at µ = -1 )

For n > 2 , plugging in Φ ( t ) = max { 0 , 1 -t } in Theorem 11 yields

Example: Φ ( t ) = Φ sq -hinge ( t ) = ( 1 -t ) 2 1 t ≤ 1 . For n = 2 , plugging in Φ ( t ) = ( 1 -t ) 2 1 t ≤ 1 in Theorem 11, gives

(minimum achieved at µ = -t )

For n > 2 , plugging in Φ ( t ) = ( 1 -t ) 2 1 t ≤ 1 in Theorem 11 yields

Example: Φ ( t ) = Φ sq ( t ) = ( 1 -t ) 2 . For n = 2 , plugging in Φ ( t ) = ( 1 -t ) 2 in Theorem 11, gives

For n > 2 , plugging in Φ ( t ) = ( 1 -t ) 2 in Theorem 11 yields

E Extensions of comp-sum losses

E.1 Proof of H -consistency bounds with T comp (Theorem 5)

Theorem 5 ( H -consistency bound for comp-sum losses ) . Assume that T comp is convex. Then, the following inequality holds for any hypothesis h ∈ H and any distribution:

with T comp the H -estimation error transformation for comp-sum losses defined for all t ∈ [ 0 , 1 ] by T comp ( t ) =

where C = [ max { s min -τ 2 , τ 1 -s max } , min { s max -τ 2 , τ 1 -s min }] , s max = 1 1 +( n -1 ) e -2inf x Λ ( x ) and s min = 1 1 +( n -1 ) e 2inf x Λ ( x ) . Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and h ∈ H such that R glyph[lscript] 0 -1 ( h ) -R ∗ glyph[lscript] 0 -1 ( H ) + M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] comp ( h ) -R ∗ glyph[lscript] comp ( H ) + M glyph[lscript] comp ( H ) = T comp ( t ) .

Proof. For the comp-sum loss glyph[lscript] comp , the conditional glyph[lscript] comp -risk can be expressed as follows:

where we let S h ( x, y ) = e h ( x,y ) ∑ y ′ ∈ Y e h ( x,y ′ ) for any y ∈ Y with the constraint that ∑ y ∈ Y S h ( x, y ) = 1 . Note that for any h ∈ H ,

Therefore for any ( x, y ) ∈ X × Y , S h ( x, y ) ∈ [ S min , S max ] , where we let S max = 1 1 +( n -1 ) e -2Λ ( x ) and S min = 1 1 +( n -1 ) e 2Λ ( x ) . Furthermore, all values in [ S min , S max ] of S h can be reached for some h ∈ H .

(minimum achieved at µ = -t )

Observe that 0 ≤ S max + S min ≤ 1 . Let y max = argmax y ∈ Y p ( x, y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y max and x ∈ X , we can always find a family of hypotheses { h µ } ⊂ H such that S h,µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y e hµ ( x,y ′ ) take the following values:

Note that S h,µ satisfies the constraint:

Since S h,µ ( x, y ) ∈ [ S min , S max ] , we have the following constraints on µ :

Let p 1 = p ( x, y max ) , p 2 = p ( x, h ( x )) , τ 1 = S h ( x, h ( x )) and τ 2 = S h ( x, y max ) to simplify the notation. Let C = { µ ∈ R ∶ µ verify constraint (7) } . Since S h ( x, h ( x ))-S max ≤ S max -S h ( x, y max ) and S min -S h ( x, y max ) ≤ S h ( x, h ( x )) -S min , C is not an empty set and can be expressed as C = [ max { S min -τ 2 , τ 1 -S max } , min { S max -τ 2 , τ 1 -S min }] .

Then, by the definition of S , we have for any h ∈ H and x ∈ X ,

where C = [ max { s min -τ 2 , τ 1 -s max } , min { s max -τ 2 , τ 1 -s min }] ⊂ C , s max = 1 1 +( n -1 ) e -2inf x Λ ( x ) and s min = 1 1 +( n -1 ) e 2inf x Λ ( x ) . Note that for n = 2 , an additional constraint τ 1 + τ 2 = 1 is imposed and the expression can be simplified as

where we use the fact that s max + s min = 1 and P = 1 when n = 2 . Since T comp is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

For the second part, we first consider n = 2 . For any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 . For any glyph[epsilon1] > 0 , by the definition of infimum, we can take h ∈ H such that S h ( x, 1 ) = τ glyph[epsilon1] ∈ [ 0 , 1 2 ] and satisfies

and

By letting glyph[epsilon1] → 0 , we conclude the proof. The proof for n > 2 directly extends from the case when n = 2 . Indeed, For any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 , p ( x, y ) = 0 , 3 ≤ y ≤ n . For any glyph[epsilon1] > 0 , by the definition of infimum, we can take h ∈ H such that S h ( x, 1 ) = τ 1 ,glyph[epsilon1] , S h ( x, 2 ) = τ 2 ,glyph[epsilon1] and S h ( x, y ) = 0 , 3 ≤ y ≤ n and satisfies τ 1 ,glyph[epsilon1] + τ 2 ,glyph[epsilon1] = 1 , and

Then, and

By letting glyph[epsilon1] → 0 , we conclude the proof.

E.2 Logistic loss

Theorem 6 ( H -consistency bounds for logistic loss ) . For any h ∈ H and any distribution, we have

Proof. For the multinomial logistic loss glyph[lscript] log , plugging in Φ ( t ) = -log ( t ) in Theorem 5, gives T comp

where C = [ max { s min -τ 2 , τ 1 -s max } , min { s max -τ 2 , τ 1 -s min }] . Here, we only compute the expression for n > 2 . The expression for n = 2 will lead to the same result since it can be viewed as a special case of the expression for n > 2 . By differentiating with respect to τ 2 and P , we can see that the infimum is achieved when τ 1 = τ 2 = s min + s max 2 and P = 1 modulo some elementary analysis. Thus, T comp can be reformulated as

where C = [ s min -s max 2 , s max -s min 2 ] and g ( µ ) = 1 + t 2 log ( s min + s max 2 -µ ) + 1 -t 2 log ( s min + s max 2 + µ ) . Since g is continuous, it attains its supremum over a compact set. Note that g is concave and differentiable. In view of that, the maximum over the open set (-∞ , +∞) can be obtained by setting its gradient to zero. Differentiate g ( µ ) to optimize, we obtain

Moreover, by the concavity, g ( µ ) is non-increasing when µ ≥ µ ∗ . Since s max -s min ≥ 0 , we have

In view of the constraint C , if µ ∗ ≥ s min -s max 2 , the maximum is achieved by µ = µ ∗ . Otherwise, if µ ∗ < s min -s max 2 , since g ( µ ) is non-increasing when µ ≥ µ ∗ , the maximum is achieved by µ = s min -s max 2 . Since µ ∗ ≥ s min -s max 2 is equivalent to t ≤ s max -s min s min + s max , the maximum can be expressed as

Computing the value of g at these points yields:

Otherwise, we obtain

Since T comp is convex, by Theorem 5, for any h ∈ H and any distribution,

where

E.3 Sum exponential loss

Theorem 9 ( H -consistency bounds for sum exponential loss ) . For any h ∈ H and any distribution,

Proof. For the sum exponential loss glyph[lscript] exp , plugging in Φ ( t ) = 1 t -1 in Theorem 5, gives T comp

where C = [ max { s min -τ 2 , τ 1 -s max } , min { s max -τ 2 , τ 1 -s min }] . Here, we only compute the expression for n > 2 . The expression for n = 2 will lead to the same result since it can be viewed as a special case of the expression for n > 2 . By differentiating with respect to τ 2 and P , we can see that the infimum is achieved when τ 1 = τ 2 = s min + s max 2 and P = 1 modulo some elementary analysis. Thus, T comp can be reformulated as

where C = [ s min -s max 2 , s max -s min 2 ] and g ( µ ) = -1 + t s min + s max -2 µ -1 -t s min + s max + 2 µ . Since g is continuous, it attains its supremum over a compact set. Note that g is concave and differentiable. In view of that, the maximum over the open set (-∞ , +∞) can be obtained by setting its gradient to zero. Differentiate g ( µ ) to optimize, we obtain

Moreover, by the concavity, g ( µ ) is non-increasing when µ ≥ µ ∗ . Since s max -s min ≥ 0 , we have

In view of the constraint C , if µ ∗ ≥ s min -s max 2 , the maximum is achieved by µ = µ ∗ . Otherwise, if µ ∗ < s min -s max 2 , since g ( µ ) is non-increasing when µ ≥ µ ∗ , the maximum is achieved by µ = s min -s max 2 . Since µ ∗ ≥ s min -s max 2 is equivalent to t ≤ s 2 max -s 2 min s 2 min + s 2 max , the maximum can be expressed as

Computing the value of g at these points yields:

Then, if t ≤ s 2 max -s 2 min s 2 min + s 2 max , we obtain

Otherwise, we obtain

Since T comp is convex, by Theorem 5, for any h ∈ H and any distribution,

where

E.4 Generalized cross-entropy loss

Theorem 16 ( H -consistency bounds for generalized cross-entropy loss ) . For any h ∈ H and any distribution, we have

Proof. For generalized cross-entropy loss glyph[lscript] gce , plugging Φ ( t ) = 1 q ( 1 -t q ) in Theorem 5, gives T comp

where C = [ max { s min -τ 2 , τ 1 -s max } , min { s max -τ 2 , τ 1 -s min }] . Here, we only compute the expression for n > 2 . The expression for n = 2 will lead to the same result since it can be viewed as a special case of the expression for n > 2 . By differentiating with respect to τ 2 and P , we can see that the infimum is achieved when τ 1 = τ 2 = s min + s max 2 and P = 1 modulo some elementary analysis. Thus, T comp can be reformulated as

where C = [ s min -s max 2 , s max -s min 2 ] and g ( µ ) = 1 + t 2 q ( s min + s max 2 -µ ) q + 1 -t 2 q ( s min + s max 2 + µ ) q . Since g is continuous, it attains its supremum over a compact set. Note that g is concave and differentiable. In view of that, the maximum over the open set (-∞ , +∞) can be obtained by setting its gradient to zero. Differentiate g ( µ ) to optimize, we obtain

Moreover, by the concavity, g ( µ ) is non-increasing when µ ≥ µ ∗ . Since s max -s min ≥ 0 , we have

In view of the constraint C , if µ ∗ ≥ s min -s max 2 , the maximum is achieved by µ = µ ∗ . Otherwise, if µ ∗ < s min -s max 2 , since g ( µ ) is non-increasing when µ ≥ µ ∗ , the maximum is achieved by µ = s min -s max 2 . Since µ ∗ ≥ s min -s max 2 is equivalent to t ≤ s 1 -q max -s 1 -q min s 1 -q min + s 1 -q max , the maximum can be expressed as

Computing the value of g at these points yields:

Then, if t ≤ s 1 -q max -s 1 -q min s 1 -q min + s 1 -q max , we obtain

Otherwise, we obtain

Since T comp is convex, by Theorem 5, for any h ∈ H and any distribution,

where

E.5 Mean absolute error loss

Theorem 17 ( H -consistency bounds for mean absolute error loss ) . For any h ∈ H and any distribution, we have

Proof. For mean absolute error loss glyph[lscript] mae , plugging Φ ( t ) = 1 -t in Theorem 5, gives T comp

where C = [ max { s min -τ 2 , τ 1 -s max } , min { s max -τ 2 , τ 1 -s min }] . Here, we only compute the expression for n > 2 . The expression for n = 2 will lead to the same result since it can be viewed as a special case of the expression for n > 2 . By differentiating with respect to τ 2 and P , we can see that the infimum is achieved when τ 1 = τ 2 = s min + s max 2 and P = 1 modulo some elementary analysis. Thus, T comp can be reformulated as

where C = [ s min -s max 2 , s max -s min 2 ] . Since -tµ is monotonically non-increasing, the maximum over C can be achieved by

Since T comp is convex, by Theorem 5, for any h ∈ H and any distribution,

F Extensions of constrained losses

F.1 Proof of H -consistency bound with T cstnd (Theorem 12)

Theorem 12 ( H -consistency bound for constrained losses ) . Assume that T cstnd is convex. Then, the following inequality holds for any hypothesis h ∈ H and any distribution:

with T cstnd the H -estimation error transformation for constrained losses defined for all t ∈ [ 0 , 1 ] by T cstnd ( t ) =

where C = [ max { τ 1 , -τ 2 } -Λ min , min { τ 1 , -τ 2 } + Λ min ] and Λ min = inf x ∈ X Λ ( x ) . Furthermore, for any t ∈ [ 0 , 1 ] , there exist a distribution D and a hypothesis h ∈ H such that R glyph[lscript] 0 -1 ( h )-R ∗ glyph[lscript] 0 -1 ( H )+ M glyph[lscript] 0 -1 ( H ) = t and R glyph[lscript] cstnd ( h ) -R ∗ glyph[lscript] cstnd ( H ) + M glyph[lscript] cstnd ( H ) = T cstnd ( t ) .

Proof. For the constrained loss glyph[lscript] cstnd , the conditional glyph[lscript] cstnd -risk can be expressed as follows:

For any h ∈ H and x ∈ X , by the definition of H , we can always find a family of hypotheses { h µ } ⊂ H such that h µ ( x, ⋅) take the following values:

Note that the hypotheses h µ satisfies the constraint:

Since h µ ( x, y ) ∈ [-Λ ( x ) , Λ ( x )] , we have the following constraints on µ :

Let p 1 = p ( x, y max ) , p 2 = p ( x, h ( x )) , τ 1 = h ( x, h ( x )) and τ 2 = h ( x, y max ) to simplify the notation. Then, the constraint on µ can be expressed as

Since max { τ 1 , -τ 2 } -min { τ 1 , -τ 2 } = ∣ τ 1 + τ 2 ∣ ≤ ∣ τ 1 ∣ + ∣ τ 2 ∣ ≤ 2Λ ( x ) , C is not an empty set. By the definition of h µ , we have for any h ∈ H and x ∈ X ,

Note that for n = 2 , an additional constraint τ 1 + τ 2 = 1 is imposed and the expression can be simplified as

Since T cstnd is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

Let n = 2 . For any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 . For any glyph[epsilon1] > 0 , by the definition of infimum, we can take h ∈ H such that h ( x, 2 ) = τ glyph[epsilon1] ≥ 0 and satisfies

and

By letting glyph[epsilon1] → 0 , we conclude the proof. The proof for n > 2 directly extends from the case when n = 2 . Indeed, for any t ∈ [ 0 , 1 ] , we consider the distribution that concentrates on a singleton { x } and satisfies p ( x, 1 ) = 1 + t 2 , p ( x, 2 ) = 1 -t 2 , p ( x, y ) = 0 , 3 ≤ y ≤ n . For any glyph[epsilon1] > 0 , by the definition of infimum, we can take h ∈ H such that h ( x, 1 ) = τ 1 ,glyph[epsilon1] , h ( x, 2 ) = τ 2 ,glyph[epsilon1] , h ( x, 3 ) = 0 , 3 ≤ y ≤ n and satisfies τ 1 ,glyph[epsilon1] + τ 2 ,glyph[epsilon1] = 0 , and

Then, and

By letting glyph[epsilon1] → 0 , we conclude the proof.

F.2 Constrained exponential loss

Theorem 13 ( H -consistency bounds for constrained exponential loss ) . Let Φ ( t ) = e -t . For any h ∈ H and any distribution,

Proof. For n = 2 , plugging in Φ ( t ) = e -t in Theorem 12, gives

By differentiating with respect to τ , we can see that the infimum is achieved when τ = 0 modulo some elementary analysis. Thus, T cstnd can be reformulated as

where g ( µ ) = -1 -t 2 e -µ -1 + t 2 e µ . Since g is continuous, it attains its supremum over a compact set. Note that g is concave and differentiable. In view of that, the maximum over the open set (-∞ , +∞) can be obtained by setting its gradient to zero. Differentiate g ( µ ) to optimize, we obtain

Moreover, by the concavity, g ( µ ) is non-increasing when µ ≥ µ ∗ . Since µ ∗ ≤ 0 and Λ min ≥ 0 , we have

In view of the constraint, if µ ∗ ≥ -Λ min , the maximum is achieved by µ = µ ∗ . Otherwise, if µ ∗ < -Λ min , since g ( µ ) is non-increasing when µ ≥ µ ∗ , the maximum is achieved by µ = -Λ min . Since µ ∗ ≥ -Λ min is equivalent to t ≤ e 2Λ min -1 e 2Λ min + 1 , the maximum can be expressed as

Computing the value of g at these points yields:

Then, if t ≤ e 2Λ min -1 e 2Λ min + 1 , we obtain

Otherwise, we obtain

For n > 2 , plugging in Φ ( t ) = e -t in Theorem 12, gives

where C = [ max { τ 1 , -τ 2 } -Λ min , min { τ 1 , -τ 2 } + Λ min ] . By differentiating with respect to τ 2 and P , we can see that the infimum is achieved when τ 2 = τ 1 = 0 and P = 1 modulo some elementary analysis. Thus, T cstnd can be reformulated as

where C = [-Λ min , Λ min ] and g ( µ ) = -1 -t 2 e -µ -1 + t 2 e µ . Since g is continuous, it attains its supremum over a compact set. Note that g is concave and differentiable. In view of that, the maximum over the open set (-∞ , +∞) can be obtained by setting its gradient to zero. Differentiate g ( µ ) to optimize, we obtain

Moreover, by the concavity, g ( µ ) is non-increasing when µ ≥ µ ∗ . Since µ ∗ ≤ 0 and Λ min ≥ 0 , we have

In view of the constraint, if µ ∗ ≥ -Λ min , the maximum is achieved by µ = µ ∗ . Otherwise, if µ ∗ < -Λ min , since g ( µ ) is non-increasing when µ ≥ µ ∗ , the maximum is achieved by µ = -Λ min . Since µ ∗ ≥ -Λ min is equivalent to t ≤ e 2Λ min -1 e 2Λ min + 1 , the maximum can be expressed as

Computing the value of g at these points yields:

Then, if t ≤ e 2Λ min -1 e 2Λ min + 1 , we obtain

Otherwise, we obtain

Since T cstnd is convex, by Theorem 12, for any h ∈ H and any distribution,

where

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a general characterization and an extension of H-consistency bounds for multiclass classification. By introducing an error transformation function, the paper provides a general tool for deriving H-consistency bounds with tightness guarantees. The paper demonstrates that the proposed tools and bounds can recover or even improve results from various recent works.\n\nSTRENGTHS:\nThis is good work in the field, original with innovative methodologies and new insights. The paper provides a general characterization and extension of H-consistency bounds for multiclass classification. Focusing on two widely studied types of loss functions in the literature, comp-sum losses and constrained losses, a new tool is introduced to derive H-consistency bounds with tightness guarantees. Before this work, deriving such bounds often required proofs for each instance. The tool proposed in this work, based on error transformations (Theorem 2, 3, 5, 10, 11, 12), is general and can be applied to derive H-consistency bounds (Theorem 4, 6, 9, Corollary 7, 8) for various loss functions and hypothesis sets. The paper demonstrates that the proposed tools and bounds can recover or even improve results from various recent works (Line 201-206, 246-248, 253-255, 261-272, 301-306, 318-322).\n\nThe quality of research is good, with meticulous comparisons with several recent works and detailed analysis leading to convincing conclusions. Well-organized and well-written, the paper adeptly communicates complex concepts in a relatively understandable way. The work is significant, with findings that resonate beyond the specific area of study and are likely to stimulate further research in the field of H-consistency bounds for multiclass classification.\n\nOverall, this paper is a nice addition to the literature.\n\nWEAKNESSES:\nAs far as I am concerned, I do not see any significant weaknesses.\n\nQUESTIONS:\nCould you provide further explanation on the connection between the results presented in this study and the theory of calibration functions as described by Steinwart in 2007?\n\nLIMITATIONS:\nI could not find the location where the authors explicitly described the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUnlike previous work on H-consistency bounds for surrogate loss functions, this paper attempts to analyze them from a more unified perspective.\nFor this purpose, the paper proposed a new error transformation method to present new tight H-consistency bounds for comp-sum losses and constrained losses in multi-class classification. With this method, the authors derived the\nH-consistency bounds in a unified manner for various loss functions, and a new theoretical analysis have developed for non-complete hypothesis sets.\n\nSTRENGTHS:\n# Writing and Organization\n- The paper is well organized such that it is easy to understand the contribution of this paper, especially the technical difference from existing work.\n-  The extensive comparison in Appendix A is very helpful in understanding the recent progress in this field, and I thought this Appendix A should be placed in the main paper if there would be any spaces.\n\n# Technical contribution\n- This paper presents a significant improvement over the existing H-consistency bounds by developing a unified and consistent approach based on error transformation, which seems novel.\n\nWEAKNESSES:\n- I think the argument of section 2 should be improved to understand more easily.\n 1. The notation introduced in Sec 2 is hard to understand at first read. It would be helpful to add the summary of notations in the Appendix so that readers can easily follow the importance of this setting\n 2. I think the current argument in paragraph 106 is hard to follow. I finally understood the paragraph after reading Appendix B.2. It might be better to introduce  $I_l(\\mathcal{H})$ in the main paper, which only appears in the Appendix, better to understand the difference between $M_l$ and $A_l$.\n 3.  I like the explanation in Appendix B.2 and B.3 more than the current main paper since the importance of each concept is discussed mathematically and clearly. \n\n- The statement, especially Theorem 2, is very general and hard to understand. I understand that this is unavoidable because the developed analysis is a kind of unified viewpoint, but it would be better to add a little more discussion of the statements, for example,  under Theorem 2.\n\nTypos\n- In line 107, H_all is used before its formal definition. Currently, it is defined in Line 114, so it should be corrected.\n- Some parentheses are missing in equations from lines 532 to 533 in the Appendix.\n\nQUESTIONS:\n- In line 59, the author explained that the bound in Awasthi 2022b may not be tight due to its ad-hoc analysis. I would like to know what it means by ad-hoc analysis and when it is not tight to clarify the limitation of existing work.\n\n- In Appendix C.1, I could not follow the equation between lines 574 and 575. Could you explain how the condition is derived from the second equality to the third equality, where $\\inf_{\\tau_1 \\leq \\max(\\dots...)}$ appeared. \n\n- I found it hard to understand what is the intuition behind J^{comp} for $n>2$ in Theorem 2 since the constraint is very complex. Is there any clear explanation for that?\n\n- In theorem 2 and 5, the statement says that there exists a distribution D and h such that the equality holds. Could you present an explicit example of that ?\n\nLIMITATIONS:\nThe limitation and assumption of the analysis is presented in detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors propose a general characterization and extension of H-consistency bounds for multi-class classification. They introduce an error transformation function that serves as a general tool for deriving these guarantees with tightness guarantees. The paper demonstrates that calculating the error transformation function enables the derivation of H-consistency bounds for various loss functions and hypothesis sets.

The general tools and tight bounds presented in the paper offer several advantages. They improve existing bounds for complete hypothesis sets, encompass a wide range of previously studied and new loss functions, extend beyond the completeness assumption, provide guarantees for bounded hypothesis sets, and offer a stronger guarantee for logistic loss with linear hypothesis sets compared to previous work.

Overall, this paper contributes to the understanding and derivation of H-consistency bounds, providing a more general framework and tools for analyzing surrogate loss functions in machine learning.

STRENGTHS:
This paper is well presented. It introduced a new general tool for deriving these H-consistency bounds with tightness guarantees. The results are carefully articulated and compared with existing work. I did not have enough time to verify all the math details, but the results look solid.

WEAKNESSES:
See next section

QUESTIONS:
Can the theory be extended to derive tight bounds for more complicated models such as simple neural networks?

LIMITATIONS:
No potential negative societal impact
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nH-consistency bounds for surrogate loss function plays an important part in learning theory.\nThis manuscript provides a comprehensive characterization and extension of H-consistency\nbounds for multi-class classification. The authors introduce a novel error transformation function\nthat enables the derivation of tighter bounds, even under weaker assumptions compared\nto existing related works. Technically, this research presents a systematic approach to H-consistency bounds. Moreover,\nit contributes to the refinement of results concerning the consistency of multi-class\nclassification on a theoretical level.\n\nSTRENGTHS:\nThe authors present a systematic technique for H-consistency analysis and apply it to\nvarious types of loss functions.\n\nThe authors compare the results in the manuscript with those in related works in detail\nand show a significant advance.\n\nWEAKNESSES:\nThis manuscript includes a wealth of work but only limited technical innovations compared\nto [1].\n\n[1] P. Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H-consistency bounds.\nNeurIPS 2022.\n\nQUESTIONS:\n• As stated in the manuscript, H-consistency is a generalization of Bayes consistency,\nand new techniques have been developed in the analysis of H-consistency. Is it possible\nto illustrate the advantages of H-consistency over Bayes consistency with more specific\nexamples?\n\n• The H-consistency analysis for not only the linear hypothesis class but also the neural\nnetwork class has been done in [2]. Can the results in the manuscript be extended to\nthe neural network class as well? As far as I understand, the analysis in the manuscript\nrelies heavily on the explicit form of the hypothesis, does it mean that it is difficult to\nextend these inequalities to the deep neural network class?\n\n• Between Line 256 and 257: Φ−1 → Ψ−1\n\n[2] P. Awasthi, A. Mao, M. Mohri, and Y. Zhong. H-consistency bounds for surrogate loss\nminimizers. ICML 2022.\n\nLIMITATIONS:\nI suggest the authors illustrating more specifically the advantages of H-consistency over Bayes\nconsistency.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an extension of previous works on $\\mathcal{H}$-consistency bounds for multi-classification using a novel method. The strength of the paper lies in proposing a new method and tool that guarantees tight $\\mathcal{H}$-consistency bounds for multi-classification.\n\nSTRENGTHS:\nThe paper proposes a general characterisation $\\mathcal{H}$-consistency bounds for multi-classification via new tools.\n\nWEAKNESSES:\nOne weakness of the paper is the lack of clarity in the discussion of tightness guarantees. While the statement in lines 191-192 suggests the existence of a distribution $\\mathcal{D}$ and a hypothesis for which the upper bound in Theorem 2 is tight, it is not explicitly clarified if this bound holds tight for all data distributions. Therefore, it is recommended to provide further clarification regarding the tightness statement.\n\nTo enhance the comprehension of the paper, it would be beneficial to include an experimental design demonstrating the tightness of the proposed bounds in comparison to previous results.\n\nThe authors mention the application of a tool to derive new results, but it would be helpful to explicitly mention and explain the techniques employed in the main body of the paper. Currently, it is unclear what these tools entail.\n\nIn the proof of Theorem 3, it is advised to provide additional elaboration on the equalities following line 587. It should be noted that in the second equality, the infimum with respect to $\\tau$ is taken over all terms.\n\nIn line 318-319, the authors mentioned \"Next, we illustrate the application of our theory through an example of constrained exponential losses,\". However, the paper is finished.\n\nThe proof of Theorems 2 and 5 for the tightness is limited to n=2. For n>2, there is no discussion.\n\nQUESTIONS:\nPlease refer to the weaknesses section for the identified limitations.\n\nLIMITATIONS:\nPlease refer to the weaknesses section for the identified questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors present some extension study on the H-consistency bounds for various loss functions. They also introduce an error transformation function that the authors claim could be a general tool in deriving H-consistency bound.

STRENGTHS:
The authors seem very well aware of the related work and have solid comparison with them.

WEAKNESSES:
Maybe the authors can simplify the notation and presentation so as the paper can be consumed by more general audience.

QUESTIONS:
1. I understand the importance of introducing a more general mathematical tool in deriving H-consistence bound for a wide range of function (but I did not check the correctness of this part). I don't know what's the value of a tigher H-consistent bound. This bound does not seem really useful (or maybe I am wrong) in pratise to guide people's design decision. Whereas Awasthi et al introduced the H-consistent bound.

2. Essentailly following 1 above, while we pushed the theoretical guranttee forward a little bit, application wise, what are the add-on values that this paper brings compared to the related work?

LIMITATIONS:
A bit hard for general audience to understand.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-08