Title: Multi-Label Learning with Stronger Consistency Guarantees
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-label learning

consistency

surrogate loss

hamming loss

learning theory
Abstract: We present a detailed study of surrogate losses and algorithms for multi-label learning, supported by $H$-consistency bounds. We first show that, for the simplest form of multi-label loss (the popular Hamming loss), the well-known consistent binary relevance surrogate suffers from a sub-optimal dependency on the number of labels in terms of $H$-consistency bounds, when using smooth losses such as logistic losses. Furthermore, this loss function fails to account for label correlations. To address these drawbacks, we introduce a novel surrogate loss, *multi-label logistic loss*,  that accounts for label correlations and benefits from label-independent $H$-consistency bounds. We then broaden our analysis to cover a more extensive family of multi-label losses, including all common ones and a new extension defined based on linear-fractional functions with respect to the confusion matrix. We also extend our multi-label logistic losses to more comprehensive multi-label comp-sum losses, adapting comp-sum losses from standard classification to the multi-label learning. We prove that this family of surrogate losses benefits from $H$-consistency bounds, and thus Bayes-consistency, across any general multi-label loss. Our work thus proposes a unified surrogate loss framework benefiting from strong consistency guarantees for any multi-label loss, significantly expanding upon previous work which only established Bayes-consistency and for specific loss functions. Additionally, we adapt constrained losses from standard classification to multi-label constrained losses in a similar way, which also benefit from $H$-consistency bounds and thus Bayes-consistency for any multi-label loss. We further describe efficient gradient computation algorithms for minimizing the multi-label logistic loss.

Multi-Label Learning with Stronger Consistency Guarantees

Abstract

We present a detailed study of surrogate losses and algorithms for multi-label learning, supported by H -consistency bounds. We first show that, for the simplest form of multi-label loss (the popular Hamming loss), the well-known consistent binary relevance surrogate suffers from a sub-optimal dependency on the number of labels in terms of H -consistency bounds, when using smooth losses such as logistic losses. Furthermore, this loss function fails to account for label correlations. To address these drawbacks, we introduce a novel surrogate loss, multi-label logistic loss , that accounts for label correlations and benefits from label-independent H -consistency bounds. We then broaden our analysis to cover a more extensive family of multi-label losses, including all common ones and a new extension defined based on linear-fractional functions with respect to the confusion matrix. We also extend our multi-label logistic losses to more comprehensive multi-label comp-sum losses, adapting comp-sum losses from standard classification to the multi-label learning. We prove that this family of surrogate losses benefits from H -consistency bounds, and thus Bayes-consistency, across any general multi-label loss. Our work thus proposes a unified surrogate loss framework benefiting from strong consistency guarantees for any multi-label loss, significantly expanding upon previous work which only established Bayes-consistency and for specific loss functions. Additionally, we adapt constrained losses from standard classification to multi-label constrained losses in a similar way, which also benefit from H -consistency bounds and thus Bayes-consistency for any multi-label loss. We further describe efficient gradient computation algorithms for minimizing the multi-label logistic loss.

1 Introduction

Supervised learning methods often assign a single label to each instance. However, real-world data exhibits a more complex structure, with objects belonging to multiple categories simultaneously. Consider a video about sports training, which could be categorized as both 'health' and 'athletics,' or a culinary blog post tagged with 'cooking' and 'nutrition'. As a result, multi-label learning [McCallum, 1999, Schapire and Singer, 2000] has become increasingly important, leading to the development of various interesting and effective approaches, predominantly experimental in nature, in recent years [Elisseeff and Weston, 2001, Deng et al., 2011, Petterson and Caetano, 2011, Kapoor et al., 2012].

Although there is a rich literature on multi-label learning (see [Zhang and Zhou, 2013] and [Bogatinovski et al., 2022] for detailed surveys), only a few studies focus on the theoretical analysis of multi-label learning, particularly the study of the Bayes-consistency of surrogate losses [Zhang, 2004a,b, Bartlett et al., 2006, Tewari and Bartlett, 2007, Steinwart, 2007].

Gao and Zhou [2011] initiated the study of Bayes-consistency in multi-label learning with respect to Hamming loss and (partial) ranking loss. They provided negative results for ranking loss, demonstrating that no convex and differentiable pairwise surrogate loss is Bayes-consistent for that multi-label loss. They also showed that the binary relevance method, which learns an independent binary classifier for each of the l labels, is Bayes-consistent with respect to the Hamming loss. Dembczynski et al. [2011] further demonstrated that under the assumption of conditionally independent labels, the binary relevance method is also Bayes-consistent with respect to the F β measure loss. However, they noted that it can perform arbitrarily poorly when this assumption does not hold. Dembczynski et al. [2012] provided a positive result for the (partial) ranking loss by showing that the simpler univariate variants of smooth surrogate losses are Bayes-consistent with respect to it. Additionally, Zhang et al. [2020] proposed a family of Bayes-consistent surrogate losses for the F β measure by reducing the F β learning problem to a set of binary class probability estimation problems. This approach was motivated by the consistent output coding scheme in [Ramaswamy et al., 2014] for general multiclass problems. Other works have studied generalization bounds in multi-label learning [Yu et al., 2014, Wydmuch et al., 2018, Wu and Zhu, 2020, Wu et al., 2021, 2023, Busa-Fekete et al., 2022].

Another related topic is the characterization of the Bayes classifier and corresponding Bayes-consistent plug-in algorithm in multi-label learning. This includes the characterization of the Bayes classifier for subset 0 / 1 loss and Hamming loss in [Cheng et al., 2010] and the characterization of the Bayes classifier for F 1 measure in [Dembczynski et al., 2011]. Dembczynski et al. [2013], Waegeman et al. [2014] further extended the results in [Dembczynski et al., 2011] by designing a Bayes-consistent plug-in algorithm for the F β measure. Koyejo et al. [2015] characterized the Bayes classifier for general linear fractional losses with respect to the confusion matrix and designed the corresponding plug-in algorithms in the empirical utility maximization (EUM) framework. In this framework, the measures are directly defined as functions of the population, in contrast to a loss function that is defined as a function over a single instance in the decision theoretic analysis (DTA) framework [Ye et al., 2012]. Menon et al. [2019] studied the Bayes-consistency of various reduction methods with respect to Precision@ κ and Recall@ κ in multi-label learning. However, all these publications only established Bayes-consistency for specific loss functions. Can we derive a unified surrogate loss framework that is Bayes-consistent for any multi-label loss?

Furthermore, as Awasthi, Mao, Mohri, and Zhong [2022a,b] pointed out, Bayes-consistency is an asymptotic guarantee and does not provide convergence guarantees. It also applies only to the family of all measurable functions unlike the restricted hypothesis sets typically used in practice. Instead, they proposed a stronger guarantee known as H -consistency bounds , which are both non-asymptotic and account for the hypothesis set while implying Bayes-consistency. These guarantees provide upper bounds on the target estimation error in terms of the surrogate estimation error. Can we leverage this state-of-the-art consistency guarantee when designing surrogate loss functions for multi-label learning?

Moreover, one of the main concerns in multi-label learning is label correlations (see [Dembczy´ nski et al., 2012]). For the simplest form of multi-label loss, the popular Hamming loss, the existing Bayes-consistent binary relevance surrogate fails to account for label correlations. Can we design consistent loss functions that effectively account for label correlations as well?

Our Contributions. This paper directly addresses these key questions in multi-label learning. We present a detailed study of surrogate losses and algorithms for multi-label learning, supported by H -consistency bounds.

In Section 3, we first show that for the simplest form of multi-label loss, the popular Hamming loss, the well-known consistent binary relevance surrogate, when using smooth losses such as logistic losses, suffers from a sub-optimal dependency on the number of labels in terms of H -consistency bounds. Furthermore, this loss function fails to account for label correlations.

To address these drawbacks, we introduce a novel surrogate loss, multi-label logistic loss , that accounts for label correlations and benefits from label-independent H -consistency bounds (Section 4). We then broaden our analysis to cover a more extensive family of multi-label losses, including all common ones and a new extension defined based on linear-fractional functions with respect to the confusion matrix (Section 5).

In Section 6, we also extend our multi-label logistic losses to more comprehensive multi-label comp-sum losses, adapting comp-sum losses from standard classification to the multi-label learning.

We prove that this family of surrogate losses benefits from H -consistency bounds, and thus Bayesconsistency, across any general multi-label loss. Our work thus proposes a unified surrogate loss framework that is Bayes-consistent for any multi-label loss, significantly expanding upon previous work which only established consistency for specific loss functions.

Additionally, we adapt constrained losses from standard classification to multi-label constrained losses in a similar way, which also benefit from H -consistency bounds and thus Bayes-consistency for any multi-label loss (Section 7). We further describe efficient gradient computation algorithms for minimizing the multi-label logistic loss (Section 8).

2 Preliminaries

Multi-label learning. We consider the standard multi-label learning setting. Let X be the input space and Y = {+ 1 , -1 } l the set of all possible labels or tags, where l is a finite number. For example, X can be a set of images, and Y can be a set of l pre-given tags (such as 'flowers', 'shoes', or 'books') that can be associated with each image in the image tagging problem. Let n = ∣ Y ∣ . For any instance x ∈ X and its associated label y = ( y 1 , . . . , y l ) ∈ Y , if y i = + 1 , we say that label i is relevant to x . Otherwise, it is not relevant. Let [ l ] = { 1 , . . . , l } . Given a sample S drawn i.i.d. according to some distribution D over X × Y , the goal of multi-label learning is to learn a hypothesis h ∶ X ×[ l ] → R to minimize the generalization error defined by a multi-label loss function L ∶ H all × X × Y → R ,

where H all is the family of all measurable hypotheses. For convenience, we abusively denote the scoring vector by h ( x ) = ( h ( x, 1 ) , . . . , h ( x, l )) . Given a hypothesis set H ⊂ H all , we denote by R ∗ L ( H ) = inf h ∈ H R ( h ) the best-in-class error. We refer to the difference R L ( h ) -R ∗ L ( H ) as the estimation error , which is termed the excess error when H = H all . Let sign ∶ t ↦ 1 t ≥ 0 -1 t < 0 be the sign function, and let t ∶ X → R + be a threshold function. The target loss function L can be typically given by a function L mapping from Y × Y to real numbers:

where h ( x ) ∶= [ h 1 ( x ) , . . . , h l ( x )] ∈ Y is the prediction for the input x ∈ X and h i ( x ) = sign ( h ( x, i )) for any i ∈ [ l ] . There are many multi-label loss functions, such as Hamming loss, (partial) ranking loss, F 1 and the more general F β measure loss, subset 0 / 1 loss, precision@ κ , recall@ κ , etc. [Zhang and Zhou, 2013]. Among these, several loss functions are defined based on the prediction of the hypothesis h ( x ) , while others are based on the scoring vector h ( x ) . We will specifically consider the first type of multi-label loss in the form given in (2), which is based on some 'distance' between the prediction and the true label. This includes all the loss functions previously mentioned (see Section 5 for a list of several common multi-label losses in this family) but excludes the (partial) ranking loss, which is defined based on pairwise scores. For convenience, we may alternatively refer to L or its induced L as the multi-label loss. Without loss of generality, we assume that L ∈ [ 0 , 1 ] , which can be achieved through normalization. We also denote by L max = max y ′ ,y L ( y ′ , y ) . Our analysis is general and adapts to any multi-label loss L . Note that we adhere to the decision-theoretic analysis (DTA) framework, in which a loss function is defined over a single instance, and the measure is the expected loss, also known as the generalization error (the expectation of the loss function over samples). Another popular framework is empirical utility maximization (EUM), where measures are defined directly as functions of the population, that is, as a function of the expectation over samples.

Surrogate risk minimization and consistency. Minimizing the multi-label loss L directly is computationally hard for most hypothesis sets because it is discrete and non-convex. A common method involves minimizing a smooth surrogate loss function ̃ L ∶ H all × X × Y → R , which is the main focus of this paper. Minimizing a surrogate loss directly leads to an algorithm for multi-label learning. A desirable guarantee for the surrogate loss in multi-label learning is Bayes-consistency [Zhang, 2004a,b, Bartlett et al., 2006, Tewari and Bartlett, 2007, Steinwart, 2007, Gao and Zhou, 2011]. That is, minimizing the surrogate loss over the family of all measurable functions leads to the minimization of the multi-label loss over the same family:

Definition 2.1. A surrogate loss ̃ L is said to be Bayes-consistent with respect to a multi-label loss L if the following holds for any distribution and all given sequences of hypotheses { h n } n ∈ N ⊂ H all :

As pointed out by Awasthi, Mao, Mohri, and Zhong [2022a,b] (see also [Long and Servedio, 2013, Zhang and Agarwal, 2020, Awasthi et al., 2021a,b, Mao et al., 2023d,c,a,b,e, Awasthi et al., 2023, 2024, Mao et al., 2024a,b,c,h,g,e,h,d,f, Mohri et al., 2024, Cortes et al., 2024]), Bayes-consistency is an asymptotic guarantee that cannot provide any guarantee for approximate minimizers; it also applies only to the family of all measurable functions and does not consider the hypothesis sets typically used in practice. Instead, they propose a stronger guarantee known as H -consistency bounds , which are both non-asymptotic and dependent on the hypothesis set, and imply Bayes-consistency when H = H all . These guarantees provide upper bounds on the target estimation error in terms of the surrogate estimation error. In the multi-label learning scenario, they can be formulated as follows:

Definition 2.2. A surrogate loss ̃ L is said to admit an H -consistency bound with respect to a multilabel loss L if the following condition holds for any distribution and for all hypotheses h ∈ H , given a concave function Γ ∶ R + → R + with Γ ( 0 ) = 0 :

The quantities M ̃ L ( H ) appearing in the bounds are called minimizability gaps, which measure the difference between the best-in-class error and the expected best pointwise error for a loss function ̃ L and a hypothesis set H :

These are inherent quantities depending on the distribution and hypothesis set, which we cannot hope to minimize. Since Γ is concave and Γ ( 0 ) = 0 , Γ is sub-additive and an H -consistency bound (4) implies that: R L ( h ) -R ∗ L ( H ) + M L ( H ) ≤ Γ ( R ̃ L ( h ) -R ∗ ̃ L ( H )) + Γ ( M ̃ L ( H )) . Therefore, when the surrogate estimation error ( R ̃ L ( h ) -R ∗ ̃ L ( H )) is minimized to glyph[epsilon1] , the target estimation error ( R L ( h ) -R ∗ L ( H )) is upper bounded by Γ ( glyph[epsilon1] ) + Γ ( M ̃ L ( H )) . The minimizability gaps vanish when H = H all or in more general realizable cases, such as when R ∗ ̃ L ( H ) = R ∗ ̃ L ( H all ) [Steinwart, 2007, Awasthi, Mao, Mohri, and Zhong, 2022b, Mao, Mohri, and Zhong, 2023f]. In these cases, H -consistency bounds imply the H -consistency of a surrogate loss ̃ L with respect to a multi-label loss L : R ̃ L ( h )-R ∗ ̃ L ( H ) ≤ glyph[epsilon1] /Leftrightline⇒ R L ( h )-R ∗ L ( H ) ≤ Γ ( glyph[epsilon1] ) , for any glyph[epsilon1] ≥ 0 . The minimizability gap M ̃ L ( H ) is upper bounded by the approximate error A ̃ L ( H ) = R ∗ ̃ L ( H )-E x [ inf h ∈ H all ( E y ∣ x [ ̃ L ( h, x, , y )])] and is generally a finer quantity [Mao et al., 2023f]. Thus, H -consistency bounds are more informative, more favorable, and stronger than excess error bounds, and they imply these bounds when H = H all .

Next, we will study surrogate loss functions and algorithms for multi-label learning, supported by H -consistency bounds, the state-of-the-art consistency guarantee for surrogate risk minimization.

3 Existing consistent surrogates for the Hamming loss

In the section, we consider the simplest form of multi-label loss, the Hamming loss, defined as:

The existing Bayes-consistent surrogate loss function is to transform the multi-label learning into l independent binary classification tasks [Gao and Zhou, 2011], defined as for all ( h, x, y ) ∈ H × X × Y ,

where Φ ∶ R → R + is a binary margin-based loss function, such as the logistic loss u ↦ log ( 1 + e -u ) . The algorithm that minimizes this surrogate loss is known as binary relevance [Zhang and Zhou, 2013], which learns an independent binary classifier for each of the l labels. Gao and Zhou [2011, Theorem 15] shows that ̃ L br is Bayes-consistent with respect to L ham if Φ is Bayes-consistent with respect to glyph[lscript] 0 -1 ∶ ( f, x, y ) ↦ 1 y ≠ sign ( f ( x )) , the binary zero-one loss. Here, we prove a stronger result that ̃ L br admits an H -consistency bound with respect to L ham with a functional form l Γ ( ⋅ l ) if Φ admits an H -consistency bounds with respect to glyph[lscript] 0 -1 with a functional form Γ (⋅) . Let F be a hypothesis set consisting of functions mapping from X to R .

Theorem 3.1. Let H = F l . Assume that the following F -consistency bound holds in the binary classification, for some concave function Γ ∶ R → R + :

Then, the following H -consistency bound holds in the multi-label learning: for all h ∈ H ,

The proof is included in Appendix A. We say that a hypothesis set F is complete if { f ( x )∶ f ∈ F } = R for all x ∈ X . This notion of completeness is broadly applicable and holds for commonly used hypothesis sets in practice, including linear hypotheses, multi-layer feed-forward neural networks, and all measurable functions. For such complete hypothesis sets F and with smooth functions Φ like the logistic loss function, Γ admits a square root dependency in the binary classification [Awasthi et al., 2022a, Mao et al., 2024h]. Thus, by Theorem 3.1, we obtain the following result.

Corollary 3.2. Let H = F l . Assume that F is complete and Φ ( u ) = log ( 1 + e -u ) . Then, the following H -consistency bound holds in the multi-label learning: for all h ∈ H ,

Since t ↦ t 1 2 is sub-additive, the right-hand side of the H -consistency bound in Corollary 3.2 can be further upper bounded by l 1 2 ( R ̃ L br ( h ) -R ∗ ̃ L br ( H )) 1 2 + l 1 2 ( M ̃ L br ( H )) 1 2 . This implies that when the estimation error of the surrogate loss ̃ L br is reduced to glyph[epsilon1] , the corresponding estimation error of the Hamming loss is upper bounded by l 1 2 glyph[epsilon1] 1 2 + l 1 2 ( M ̃ L br ( H )) 1 2 -M L ham ( H ) . In the nearly realizable cases where minimizability gaps are negligible, this upper bound approximates to

Therefore, as the number of labels l increases, the bound becomes less favorable. Furthermore, the loss function ̃ L br clearly fails to account for the inherent correlations among labels. For instance, 'coffee' and 'mug' are more likely to co-occur than 'coffee' and 'umbrella'. Additionally, ̃ L br is only Bayes-consistent with respect to the Hamming loss and cannot yield risk-minimizing predictions for other multi-label losses such as subset 0 / 1 loss or F β -measure loss [Dembczy´ nski et al., 2012]. To address these drawbacks, we will introduce a new surrogate loss in the next section.

4 Multi-label logistic loss

In this section, we define a new surrogate loss for Hamming loss in multi-label learning that accounts for label correlations and benefits from label-independent H -consistency bounds. This loss function can be viewed as a generalization of the (multinomial) logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], used in standard classification, to multi-label learning. Thus, we will refer to it as multi-label logistic loss . It is defined as follows: for all ( h, x, y ) ∈ H × X × Y ,

This formulation can be interpreted as a weighted logistic loss, where ( 1 -L ham (⋅ , y )) serves as a weight vector. Additionally, the term ∑ l i = 1 ( y ′′ i -y ′ i ) h ( x, i ) represents the difference in the scores between the label y ′ and any other label y ′′ , where these scores account for the correlations among the labels y i within the logarithmic function.

The next result shows that the multi-label logistic loss benefits from a favorable H -consistency bound with respect to L ham , without dependency on the number of labels l . We assume that H = F l and F is complete, conditions that typically hold in practice.

Theorem 4.1. Let H = F l . Assume that F is complete. Then, the following H -consistency bound holds in the multi-label learning: for all h ∈ H ,

Since t ↦ t 1 2 is sub-additive, the right-hand side of the H -consistency bound in Theorem 4.1 can be further upper bounded by 2 ( R ̃ L log ( h ) -R ∗ ̃ L log ( H )) 1 2 + 2 ( M ̃ L log ( H )) 1 2 . This implies that when the estimation error of the surrogate loss ̃ L log is reduced up to glyph[epsilon1] , the corresponding estimation error of the Hamming loss is upper bounded by 2 glyph[epsilon1] 1 2 + 2 ( M ̃ L log ( H )) 1 2 -M L ham ( H ) . In the nearly realizable cases where minimizability gaps are negligible, this upper bound approximates to

Therefore, the bound is independent of the number of labels l . This contrasts with the bound for ̃ L br shown in (11), where a label-dependent factor glyph[lscript] 1 2 replaces the constant factor 2 , making it significantly less favorable.

The proof of Theorem 4.1 is included in Appendix B.2. We first present a general tool (Theorem B.1) in Appendix B.1, which shows that to derive H -consistency bounds in multi-label learning with a concave function Γ , it is only necessary to upper bound the conditional regret of the target multi-label loss by that of the surrogate loss with the same Γ . This generalizes [Awasthi, Mao, Mohri, and Zhong, 2022b, Theorem 2] in standard multi-class classification to multi-label learning. Next, we characterize the conditional regret of the target multi-label loss, such as Hamming loss, in Lemma B.2 found in Appendix B.1, under the given assumption. By using Lemma B.2, we upper bound the conditional regret of L ham by that of the surrogate loss ̃ L log with a concave function Γ ( t ) = 2 √ t .

When H = H all , minimizability gaps M ̃ L log ( H ) and M L ham ( H ) vanish, Theorem 4.1 implies excess error bound and Bayes-consistency of multi-label logistic loss with respect to the Hamming loss. Corollary 4.2. The following excess error bound holds in the multi-label learning: for all h ∈ H all ,

Moreover, ̃ L log is Bayes-consistent with respect to L ham .

Approaches like binary relevance surrogate loss treat each label independently. This overlooks crucial inherent information encoded in the relationships between labels. Our new form of surrogate losses explicitly captures these label correlations. Both the binary relevance surrogate loss and our new surrogate loss are Bayes-consistent, meaning that minimizing them over the family of all measurable functions approximates the Bayes-optimal solution. However, our correlation-aware surrogate losses can converge faster, which is reflected in their more favorable H -consistency bounds, independent of the number of labels.

It is known that ̃ L br is only Bayes-consistent with respect to the Hamming loss and Precision@ κ , and can be arbitrarily bad for other multi-label losses such as F β -measure loss [Dembczynski et al., 2011]. Instead, we will show in the next section that our surrogate loss ̃ L log adapts to and is Bayes-consistent with respect to an extensive family of multi-label losses, including the F β measure loss.

5 Extension: general multi-label losses

In this section, we broaden our analysis to cover a more extensive family of multi-label losses, including all common ones and a new extension defined based on linear-fractional functions with respect to the confusion matrix. Note that several loss functions are defined over the space { 0 , 1 } l , rather than {+ 1 , -1 } l . To accommodate this difference, any pair y, y ′ ∈ Y = {+ 1 , -1 } l can be projected onto { 0 , 1 } l by letting y = y + 1 2 and y ′ = y ′ + 1 2 , where 1 ∈ R l is the vector with all elements equal to 1. Several common multi-label losses are defined as follows.

Hamming loss: L ( y ′ , y ) = ∑ l i = 1 1 y i ≠ y ′ i .

F β -measure loss: L ( y ′ , y ) = 1 -( 1 + β 2 ) y ′ ⋅ y β 2 ∥ y ∥ 1 +∥ y ′ ∥ 1 .

Subset 0 / 1 loss: L ( y ′ , y ) = max i ∈[ l ] 1 y ′ i ≠ y i .

Jaccard distance: L ( y ′ , y ) = 1 -y ′ ⋅ y ∥ y ∥ 1 +∥ y ′ ∥ 1 -y ′ ⋅ y

Precision@ κ : L ( y ′ , y ) = 1 -1 κ ∑ i ∈ T ( y ′ ) 1 y i = 1 subject to y ′ ∈ Y κ , where Y κ = { y ∈ Y ∶ ∥ y ∥ 1 = κ } and T ( y ′ ) = { i ∈ [ l ]∶ y ′ i = 1 } .

More generally, we can define a multi-label loss based on true positives ( TP ), true negatives ( TN ), false positives ( FP ) and false negatives ( FN ) , which can be written explicitly as follows:

Similar to [Koyejo et al., 2014, 2015], we now define a general family of multi-label losses as linear-fractional functions in terms of these four quantities:

It can be shown that the aforementioned Hamming loss, F β -measure loss, Jaccard distance, precision and recall all belong to this family. Note that the previous definitions in [Koyejo et al., 2014, 2015] are within the empirical utility maximization (EUM) framework [Ye et al., 2012], where the measures are directly defined as functions of the population. We generalize their definition to the decision theoretic analysis (DTA) framework, in terms of loss functions defined over y and y ′ .

Moreover, we can consider extending multi-label losses (16) to non-linear fractional functions of these four quantities, or more generally, to any other forms, as long as they are defined over the space Y × Y .

Another important family of multi-label losses is the tree distance loss, used in cases of hierarchical classes. In many practical applications, the class labels exist within a predefined hierarchy. For example, in the image tagging problem, class labels might include broad categories such as 'animals' or 'vehicles', which further subdivide into more specific classes like 'mammals' and 'birds' for animals, or 'cars' and 'trucks' for vehicles. Each of these subcategories can be divided even further, showcasing a clear hierarchical structure.

Tree distance: Let T = ( Y , E, W ) be a tree over the label space Y , with edge set E and positive, finite edge lengths specified by W . Suppose r ∈ Y is designated as the root node. Then, L T ( y ′ , y ) = the shortest path length in T between y and y ′ .

Despite the widespread use of hierarchical classes in practice, to our knowledge, no Bayes-consistent surrogate has been proposed for the tree distance loss in multi-label learning. Next, we will show that our multi-label logistic loss can accommodate all these different loss functions, including the tree distance loss. For any general multi-label loss L , we define the multi-label logistic loss as follows:

Here, L can be chosen as all the multi-label losses mentioned above. Next, we will show that ̃ L log benefits from H -consistency bounds and Bayes consistency with respect to any of these loss functions. Theorem 5.1. Let H = F l . Assume that F is complete. Then, the following H -consistency bound holds in the multi-label learning:

The proof of Theorem 5.1 is basically the same as that of Theorem 4.1, modulo replacing the Hamming loss L ham with a general multi-label loss L . We include it in Appendix B.3 for completeness. When H = H all , minimizability gaps M ̃ L log ( H ) and M L ( H ) vanish, Theorem 4.1 implies excess error bound and Bayes-consistency of multi-label logistic loss with respect to any multi-label loss.

Corollary 5.2. The following excess error bound holds in the multi-label learning: for all h ∈ H all ,

Moreover, ̃ L log is Bayes-consistent with respect to L .

Corollary 5.2 is remarkable, as it demonstrates that a unified surrogate loss, ̃ L log , is Bayes-consistent for any multi-label loss, significantly expanding upon previous work which only established consistency for specific loss functions. Furthermore, Theorem 5.1 provides a stronger guarantee than Bayes-consistency, which is both non-asymptotic and specific to the hypothesis set used.

Minimizing the multi-label logistic loss directly leads to the effective algorithm in multi-label learning. We further discuss the efficiency and practicality of this algorithm in Section 8, where we describe efficient gradient computation.

6 Extension: multi-label comp-sum losses

In this section, we further extend our multi-label logistic losses to more comprehensive multilabel comp-sum losses , adapting comp-sum losses [Mao, Mohri, and Zhong, 2023f] from standard classification to the multi-label learning. As shown by Mao, Mohri, and Zhong [2023f], a comp-sum loss is defined via a composition of the function Ψ and a sum, and includes the logistic loss ( Ψ ( u ) = log ( u ) ) [Verhulst, 1838, 1845, Berkson, 1944, 1951], the sum-exponential loss ( Ψ ( u ) = u -1 ) [Weston and Watkins, 1998, Awasthi et al., 2022b], the generalized cross-entropy loss ( Ψ ( u ) = 1 q ( 1 -1 u q ) , q ∈ ( 0 , 1 ) ) [Zhang and Sabuncu, 2018], and the mean absolute error loss ( Ψ ( u ) = 1 -1 u ) [Ghosh et al., 2017] as special cases.

Given any multi-label loss L , we will define our novel multi-label comp-sum losses as follows:

This formulation can be interpreted as a weighted comp-sum loss, where ( 1 -L (⋅ , y )) serves as a weight vector. Additionally, this formulation accounts for label correlations among the y i s within the function Ψ . Next, we prove that this family of surrogate losses benefits from H -consistency bounds, and thus Bayes-consistency, across any general multi-label loss.

Theorem 6.1. Let H = F l . Assume that F is complete. Then, the following H -consistency bound holds in the multi-label learning:

where Γ ( t ) = 2 √ t when Ψ ( u ) = log ( u ) or u -1 ; Γ ( t ) = 2 √ n q t when Ψ ( u ) = 1 q ( 1 -1 u q ) , q ∈ ( 0 , 1 ) ; and Γ ( t ) = nt when Ψ ( u ) = 1 -1 u .

Corollary 6.2. The following excess error bound holds in the multi-label learning:

where Γ ( t ) = 2 √ t when Ψ ( u ) = log ( u ) or u -1 ; Γ ( t ) = 2 √ n q t when Ψ ( u ) = 1 q ( 1 -1 u q ) , q ∈ ( 0 , 1 ) ; and Γ ( t ) = nt when Ψ ( u ) = 1 -1 u . Moreover, ̃ L comp with these choices of Ψ are Bayes-consistent with respect to L .

The proof of Theorem 6.1 is included in Appendix B.4. Similar to the proof of Theorem 5.1, we make use of Theorem B.1 and Lemma B.2 in Appendix B.1. However, upper bounding the conditional regret of L by that of the surrogate loss ̃ L comp for different choices of Ψ requires a distinct analysis depending on the specific form of the function Ψ , leading to various concave functions Γ . Our proof is inspired by the proof of H -consistency bounds for comp-sum losses in [Mao et al., 2023f] through the introduction of a parameter µ and optimization. However, the novelty lies in the adaptation of µ with a quantity s tailored to multi-label loss functions instead of the score vector h itself.

Note that, as with Ψ ( u ) = log ( u ) shown in Section 5, for Ψ ( u ) = u -1 , the bounds are also independent of the number of labels and are favorable. However, for other choices of Ψ , the bounds exhibit a worse dependency on n , which can be exponential with respect to l .

7 Extension: multi-label constrained losses

In this section, we introduce another novel family of surrogate losses, adapting constrained losses [Lee et al., 2004, Awasthi et al., 2022b] from standard classification to multi-label constrained losses

in a similar way. Given any general multi-label loss L , we define multi-label constrained losses as:

where ∑ y ∈ Y ∑ l i = 1 y i h ( x, i ) = 0 . Next, we show that ̃ L cstnd also benefit from H -consistency bounds and thus Bayes-consistency for any multi-label loss.

Theorem 7.1. Let H = F l . Assume that F is complete Then, the following H -consistency bound holds in the multi-label learning:

Corollary 7.2. The following excess error bound holds in the multi-label learning:

where Γ ( t ) = 2 √ L max t when Φ ( u ) = e -u ; Γ ( t ) = 2 √ t when Φ ( u ) = max { 0 , 1 -u } 2 ; and Γ ( t ) = t when Φ ( u ) = max { 0 , 1 -u } or Φ ( u ) = min { max { 0 , 1 -u / ρ } , 1 } , ρ > 0 . Moreover, ̃ L cstnd with these choices of Φ are Bayes-consistent with respect to L .

The proof of Theorem 7.1 is included in Appendix B.5. As with the proof of Theorem 6.1, we use Theorem B.1 and Lemma B.2 from Appendix B.1, and aim to upper bound the conditional regret of L by that of the surrogate losses ̃ L comp using various concave functions Γ . However, the difference lies in our introduction and optimization of a parameter µ tailored to a quantity z that is specific to the form of the multi-label constrained loss.

These results show that in cases where minimizability gaps vanish, reducing the estimation error of ̃ L cstnd to glyph[epsilon1] results in the estimation error of target multi-label loss L being upper bounded by either √ glyph[epsilon1] or glyph[epsilon1] , modulo a constant that is independent of the number of labels.

8 Efficient Gradient Computation

In this section, we demonstrate the efficient computation of the gradient for the multi-label logistic loss ̃ L log at any point ( x j , y j ) . This loss function is therefore both theoretically grounded in H -consistency bounds and computationally efficient. Consider the labeled pair ( x j , y j ) and a hypothesis h in H . The expression for ̃ L log ( h, x j , y j ) can be reformulated as follows:

Let L 1 ( j ) = ∑ y ∈ Y ( 1 -L ( y, y j )) , which is independent of h and can be pre-computed. It can also be invariant with respect to j and is a fixed constant for many loss functions such as the Hamming loss.

For many commonly used loss functions, the terms involving sums over all possible label combinations can be simplified analytically. To illustrate this, we provide explicit formulae for the Hamming loss and F β measure loss functions:

A similar analysis applies to many other loss functions, thus, these terms do not affect the tractability of our algorithms. Next, we will consider the hypothesis set of linear functions H = { x ↦ w ⋅ Ψ ( x, i )∶ w ∈ R d } , where Ψ is a feature mapping from X × [ l ] to R d . Using the shorthand w for h , we can rewrite ̃ L log at ( x j , y j ) as follows:

where Z w ,j = ∑ y ∈ Y e w ⋅(∑ l i = 1 y i Ψ ( x j ,i )) . Then, we can compute the gradient of ̃ L log at any w ∈ R d :

where q w is a distribution over Y with probability mass function q w ( y ) = e w ⋅(∑ l i = 1 y i Ψ ( x j ,i )) Z w ,j . By rearranging the terms in (27), we obtain the following result.

Lemma 8.1. The gradient of ̃ L log at any w ∈ R d can be expressed as follows:

where L 2 ( i, j ) = ∑ y ∈ Y ( 1 -L ( y, y j )) y i , L 1 ( j ) = ∑ y ∈ Y ( 1 -L ( y, y j )) , Q w ( i ) = ∑ y ∈ Y q w ( y ) y i , q w ( y ) = e w ⋅(∑ l i = 1 y i Ψ ( x j ,i )) Z w ,j , and Z w ,j = ∑ y ∈ Y e w ⋅(∑ l i = 1 y i Ψ ( x j ,i )) . The overall time complexity for gradient computation is O ( l ) .

Here, the evaluation of L 2 ( i, j ) , i ∈ [ l ] and L 1 ( j ) can be computed once and for all, before any gradient computation. For evaluation of Q w ( i ) , note that it can be equivalently written as follows:

where ̃ Ψ ( x j , y ) admits a Markovian property of order 1 [Manning and Schutze, 1999, Cortes et al., 2016]. Thus, as shown by Cortes et al. [2016, 2018], Q w ( i ) can be evaluated efficiently by running two single-source shortest-distance algorithms over the (+ , ×) semiring on an appropriate weighted finite automaton (WFA). More specifically, in our case, the WFA can be described as follows: there are ( l + 1 ) vertices labeled 0 , . . . , l . There are two transitions from k to ( k + 1 ) labeled with + 1 and -1 . The weight of the transition with label + 1 is exp (+ w ⋅ ̃ Ψ ( x j , k )) , and exp (-w ⋅ ̃ Ψ ( x j , k )) for the other. 0 is the initial state, and l the final state. The overall time complexity of computing all quantities Q w ( i ) , i ∈ [ l ] , is O ( l ) .

9 Conclusion

We presented a comprehensive analysis of surrogate losses for multi-label learning, establishing strong consistency guarantees. We introduced a novel multi-label logistic loss that addresses the shortcomings of existing methods and enjoys label-independent consistency bounds. Our proposed family of multi-label comp-sum losses offers a unified framework with strong consistency guarantees for any general multi-label loss, significantly expanding upon previous work. Additionally, we presented efficient algorithms for their gradient computation. While empirical validation is left for future work, our theoretical results demonstrate the potential of these new surrogate losses to advance multi-label learning. This unified framework shows promise for broader applications and paves the way for future research in multi-label learning and related areas.

References

Awasthi, N. Frank, A. Mao, M. Mohri, and Y. Zhong. Calibration and consistency of adversarial surrogate losses. Advances in Neural Information Processing Systems , pages 9804-9815, 2021a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. A finer calibration analysis for adversarial robustness. arXiv preprint arXiv:2105.01550 , 2021b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. H -consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , 2022a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H -consistency bounds. In Advances in neural information processing systems , pages 782-795, 2022b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094, 2023.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. DC-programming for neural network optimizations. Journal of Global Optimization , 2024.

L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Berkson. Application of the logistic function to bio-assay. Journal of the American Statistical Association , 39:357--365, 1944.

Berkson. Why I prefer logits to probits. Biometrics , 7(4):327--339, 1951.

Bogatinovski, L. Todorovski, S. Džeroski, and D. Kocev. Comprehensive comparative study of multi-label classification methods. Expert Systems with Applications , 203:117215, 2022.

Busa-Fekete, H. Choi, K. Dembczynski, C. Gentile, H. Reeve, and B. Szorenyi. Regret bounds for multilabel classification in sparse label regimes. In Advances in Neural Information Processing Systems , pages 5404-5416, 2022.

Cheng, E. Hüllermeier, and K. J. Dembczynski. Bayes optimal multilabel classification via probabilistic classifier chains. In international conference on machine learning , pages 279-286, 2010.

Cortes, V. Kuznetsov, M. Mohri, and S. Yang. Structured prediction theory based on factor graph complexity. In Advances in Neural Information Processing Systems , 2016.

Cortes, V. Kuznetsov, M. Mohri, D. Storcheus, and S. Yang. Efficient gradient computation for structured output learning with rational and tropical losses. In Advances in Neural Information Processing Systems , 2018.

Cortes, A. Mao, C. Mohri, M. Mohri, and Y. Zhong. Cardinality-aware set prediction and topk classification. In Advances in neural information processing systems , 2024.

Dembczynski, W. Waegeman, W. Cheng, and E. Hüllermeier. An exact algorithm for f-measure maximization. In Advances in neural information processing systems , 2011.

Dembczynski, W. Kotłowski, and E. Hüllermeier. Consistent multilabel ranking through univariate loss minimization. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1347-1354, 2012.

Dembczy´ nski, W. Waegeman, W. Cheng, and E. Hüllermeier. On label dependence and loss minimization in multi-label classification. Machine Learning , 88:5-45, 2012.

Dembczynski, A. Jachnik, W. Kotlowski, W. Waegeman, and E. Hüllermeier. Optimizing the f-measure in multi-label classification: Plug-in rule approach versus structured loss minimization. In International conference on machine learning , pages 1130-1138, 2013.

Deng, S. Satheesh, A. Berg, and F. Li. Fast and balanced: Efficient label tree learning for large scale object recognition. In Advances in Neural Information Processing Systems , 2011.

Elisseeff and J. Weston. A kernel method for multi-labelled classification. In Advances in neural information processing systems , 2001.

Gao and Z.-H. Zhou. On the consistency of multi-label learning. In Conference on learning theory , pages 341-358, 2011.

Ghosh, H. Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , 2017.

Kapoor, R. Viswanathan, and P. Jain. Multilabel classification using bayesian compressed sensing. In Advances in neural information processing systems , 2012.

O. Koyejo, N. Natarajan, P. K. Ravikumar, and I. S. Dhillon. Consistent binary classification with generalized performance metrics. In Advances in neural information processing systems , 2014.

O. Koyejo, N. Natarajan, P. K. Ravikumar, and I. S. Dhillon. Consistent multilabel classification. In Advances in Neural Information Processing Systems , 2015.

Lee, Y. Lin, and G. Wahba. Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association , 99(465):67-81, 2004.

Long and R. Servedio. Consistency versus realizable H-consistency for multiclass classification. In International Conference on Machine Learning , pages 801-809, 2013.

Manning and H. Schutze. Foundations of statistical natural language processing . MIT press, 1999.

Mao, C. Mohri, M. Mohri, and Y. Zhong. Two-stage learning to defer with multiple experts. In Advances in neural information processing systems , 2023a.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds: Characterization and extensions. In Advances in Neural Information Processing Systems , 2023b.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds for pairwise misranking loss surrogates. In International conference on Machine learning , 2023c.

Mao, M. Mohri, and Y. Zhong. Ranking with abstention. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023d.

Mao, M. Mohri, and Y. Zhong. Structured prediction with stronger consistency guarantees. In Advances in Neural Information Processing Systems , 2023e.

Mao, M. Mohri, and Y. Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , pages 23803-23828, 2023f.

Mao, M. Mohri, and Y. Zhong. Principled approaches for learning to defer with multiple experts. In International Symposium on Artificial Intelligence and Mathematics , 2024a.

Mao, M. Mohri, and Y. Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. In Algorithmic Learning Theory , 2024b.

Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for scorebased multi-class abstention. In International Conference on Artificial Intelligence and Statistics , 2024c.

Mao, M. Mohri, and Y. Zhong. Enhanced H -consistency bounds. arXiv preprint arXiv:2407.13722 , 2024d.

Mao, M. Mohri, and Y. Zhong. H -consistency guarantees for regression. In International Conference on Machine Learning , pages 34712-34737, 2024e.

Mao, M. Mohri, and Y. Zhong. Realizable H -consistent and Bayes-consistent loss functions for learning to defer. In Advances in neural information processing systems , 2024f.

Mao, M. Mohri, and Y. Zhong. Regression with multi-expert deferral. In International Conference on Machine Learning , pages 34738-34759, 2024g.

Mao, M. Mohri, and Y. Zhong. A universal growth rate for learning with smooth surrogate losses. In Advances in neural information processing systems , 2024h.

K. McCallum. Multi-label text classification with a mixture model trained by em. In AAAI'99 workshop on text learning , 1999.

K. Menon, A. S. Rawat, S. Reddi, and S. Kumar. Multilabel reductions: what is my loss optimising? In Advances in Neural Information Processing Systems , 2019.

Mohri, D. Andor, E. Choi, M. Collins, A. Mao, and Y. Zhong. Learning to reject with a fixed predictor: Application to decontextualization. In International Conference on Learning Representations , 2024.

Petterson and T. Caetano. Submodular multi-label learning. In Advances in Neural Information Processing Systems , 2011.

G. Ramaswamy, B. S. Babu, S. Agarwal, and R. C. Williamson. On the consistency of output code based learning algorithms for multiclass learning problems. In Conference on Learning Theory , pages 885-902, 2014.

E. Schapire and Y. Singer. Boostexter: A boosting-based system for text categorization. Machine learning , 39:135-168, 2000.

Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Tewari and P. L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research , 8(36):1007-1025, 2007.

F. Verhulst. Notice sur la loi que la population suit dans son accroissement. Correspondance mathématique et physique , 10:113--121, 1838.

F. Verhulst. Recherches mathématiques sur la loi d'accroissement de la population. Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles , 18:1--42, 1845.

Waegeman, K. Dembczy´ nski, A. Jachnik, W. Cheng, and E. Hüllermeier. On the bayes-optimality of f-measure maximizers. Journal of Machine Learning Research , 15:3333-3388, 2014.

Weston and C. Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.

Wu and J. Zhu. Multi-label classification: do hamming loss and subset accuracy really conflict with each other? In Advances in Neural Information Processing Systems , pages 3130-3140, 2020.

Wu, C. Li, K. Xu, and J. Zhu. Rethinking and reweighting the univariate losses for multi-label ranking: Consistency and generalization. In Advances in Neural Information Processing Systems , pages 14332-14344, 2021.

Wu, C. Li, and Y. Yin. Towards understanding generalization of macro-auc in multi-label learning. In International Conference on Machine Learning , pages 37540-37570, 2023.

Wydmuch, K. Jasinska, M. Kuznetsov, R. Busa-Fekete, and K. Dembczynski. A no-regret generalization of hierarchical softmax to extreme multi-label classification. In Advances in neural information processing systems , 2018.

Ye, K. M. Chai, W. S. Lee, and H. L. Chieu. Optimizing f-measures: A tale of two approaches. In International Conference on Machine Learning , pages 289-296, 2012.

H.-F. Yu, P. Jain, P. Kar, and I. Dhillon. Large-scale multi-label learning with missing labels. In International conference on machine learning , pages 593-601, 2014.

Zhang and S. Agarwal. Bayes consistency vs. H-consistency: The interplay between surrogate loss functions and the scoring function class. In Advances in Neural Information Processing Systems , 2020.

Zhang, H. G. Ramaswamy, and S. Agarwal. Convex calibrated surrogates for the multi-label f-measure. In International Conference on Machine Learning , pages 11246-11255, 2020.

M.-L. Zhang and Z.-H. Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering , 26(8):1819-1837, 2013.

Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics , 32(1):56-85, 2004a.

Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004b.

Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems , 2018.

Contents of Appendix

A Proof of H -consistency bounds for existing surrogate losses (Theorem 3.1)

Theorem 3.1. Let H = F l . Assume that the following F -consistency bound holds in the binary classification, for some concave function Γ ∶ R → R + :

Then, the following H -consistency bound holds in the multi-label learning: for all h ∈ H ,

Proof. Let p ( y ∣ x ) = P ( Y = y ∣ X = x ) be the conditional probability of Y = y given X = x . Given a multi-label surrogate loss ̃ L and a hypothesis set H , we denote the conditional error by C ̃ L ( h, x ) = E y ∣ x [ ̃ L ( h, x, y )] , the best-in-class conditional error by C ∗ ̃ L ( H , x ) = inf h ∈ H C ̃ L ( h, x ) , and the conditional regret by ∆ C ̃ L , H ( h, x ) = C ̃ L ( h, x ) -C ∗ ̃ L ( H , x ) . We can express the conditional error of the hamming loss and the surrogate loss ̃ L br as follows:

Let q i (+ 1 ∣ x ) = ∑ y ∶ y i =+ 1 p ( y ∣ x ) and q i (-1 ∣ x ) = ∑ y ∶ y i =-1 p ( y ∣ x ) . Let f i = h (⋅ , i ) ∈ F , for all i ∈ [ l ] . Then, it is clear that the conditional regrets of L ham and ̃ L br can be expressed as the corresponding conditional regrets of glyph[lscript] 0 -1 and Φ under this new introduced new distribution:

Since we have ∆ C glyph[lscript] 0 -1 , F ( f i , x ) ≤ Γ ( ∆ C Φ , F ( f i , x )) under the assumption, we obtain

By taking the expectation on both sides and using the Jensen's inequality, we have

This completes the proof.

B Proofs of H -consistency bounds for new surrogate losses

B.1 Auxiliary definitions and results (Theorem B.1 and Lemma B.2)

Before proceeding with the proof, we first introduce some notation and definitions. Given a multilabel surrogate loss ̃ L and a hypothesis set H , we denote the conditional error by C ̃ L ( h, x ) = E y ∣ x [ ̃ L ( h, x, y )] , the best-in-class conditional error by C ∗ ̃ L ( H , x ) = inf h ∈ H C ̃ L ( h, x ) , and the conditional regret by ∆ C ̃ L , H ( h, x ) = C ̃ L ( h, x ) -C ∗ ̃ L ( H , x ) . We then present a general theorem, which shows that to derive H -consistency bounds in multi-label learning with a concave function Γ , it is only necessary to upper bound the conditional regret of the target multi-label loss by that of the surrogate loss with the same Γ .

Theorem B.1. Let L be a multi-label loss and ̃ L be a surrogate loss. Given a concave function Γ ∶ R + → R + . If the following condition holds for all h ∈ H and x ∈ X :

then, for any distribution and for all hypotheses h ∈ H ,

Proof. By the definitions, the expectation of the conditional regrets for L and ̃ L can be expressed as:

Thus, by taking the expectation on both sides of (31) and using Jensen's inequality, we have

This completes the proof.

To derive H -consistency bounds using Theorem B.1, we will characterize the conditional regret of a multi-label loss L . For simplicity, we first introduce some notation. For any x ∈ X , let y ( x ) = argmin y ′ ∈ Y E y ∣ x [ L ( y ′ , y )] ∈ Y , where we choose the label with the lowest index under the natural ordering of labels as the tie-breaking strategy. To simplify the notation further, we will drop the dependency on x . Specifically, we use y to denote y ( x ) and h to denote h ( x ) . Additionally, we define c h = E y ∣ x [ L ( h , y )] , c y = E y ∣ x [ L ( y , y )] and c y ′ = E y ∣ x [ L ( y ′ , y )] , ∀ y ′ ∈ Y .

Lemma B.2. Let H = F l . Assume that F is complete. Then, the conditional regret of a multi-label loss L can be expressed as follows: ∆ C L , H ( h, x ) = c h -c y .

Proof. By definition, the conditional error of L can be expressed as follows:

Since H = F l and F is complete, for any x ∈ X , { h ( x )∶ h ∈ H } = Y . Then, the best-in-class conditional error of L can be expressed as follows:

Next, by using Lemma B.2, we will upper bound the conditional regret of the target multi-label loss L by that of the surrogate loss ̃ L with a concave function Γ .

B.2 Proof of Theorem 4.1

Theorem 4.1. Let H = F l . Assume that F is complete. Then, the following H -consistency bound holds in the multi-label learning: for all h ∈ H ,

Proof. We will use the following notation adapted to the Hamming loss: c h = E y ∣ x [ L ham ( h , y )] , c y = E y ∣ x [ L ( y , y )] and c y ′ = E y ∣ x [ L ham ( y ′ , y )] , ∀ y ′ ∈ Y . We will denote by s ( h, x, y ′ ) = e ∑ l i = 1 y ′ i h ( x,i ) ∑ y ′′ ∈ Y e ∑ l i = 1 y ′′ i h ( x,i ) and simplify notation by using s y ′ , thereby dropping the dependency on h and x .

It is clear that s y ′ ∈ [ 0 , 1 ] . Then, the conditional error of ̃ L log can be expressed as follows:

For any h ≠ y , we define s µ as follows: set s µ y ′ = s y ′ for all y ′ ≠ y and y ′ ≠ h ; define s µ h = s y -µ ; and let s µ y = s h + µ . Note that s µ can be realized by some h ′ ∈ H due to the completeness assumption. Then, we have

Therefore, by Lemma B.2, ∆ C L ham , H ( h, x ) ≤ 2 ( ∆ C ̃ L log , H ( h, x )) 1 2 . By Theorem B.1, we complete the proof.

B.3 Proof of Theorem 5.1

Theorem 5.1. Let H = F l . Assume that F is complete. Then, the following H -consistency bound holds in the multi-label learning:

Proof. The proof is basically the same as that of Theorem 4.1, modulo replacing the Hamming loss L ham with a general multi-label loss L . We adopt the following notation: c h = E y ∣ x [ L ( h , y )] , c y = E y ∣ x [ L ( y , y )] and c y ′ = E y ∣ x [ L ( y ′ , y )] , ∀ y ′ ∈ Y . We will denote by s ( h, x, y ′ ) = e ∑ l i = 1 y ′ i h ( x,i ) ∑ y ′′ ∈ Y e ∑ l i = 1 y ′′ i h ( x,i ) and simplify notation by using s y ′ , thereby dropping the dependency on h and x . It is clear that s y ′ ∈ [ 0 , 1 ] . Then, the conditional error of ̃ L log can be expressed as follows:

For any h ≠ y , we define s µ as follows: set s µ y ′ = s y ′ for all y ′ ≠ y and y ′ ≠ h ; define s µ h = s y -µ ; and let s µ y = s h + µ . Note that s µ can be realized by some h ′ ∈ H under the assumption. Then, we have

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ 2 ( ∆ C ̃ L log , H ( h, x )) 1 2 . By Theorem B.1, we complete the proof.

B.4 Proof of Theorem 6.1

Theorem 6.1. Let H = F l . Assume that F is complete. Then, the following H -consistency bound holds in the multi-label learning:

where Γ ( t ) = 2 √ t when Ψ ( u ) = log ( u ) or u -1 ; Γ ( t ) = 2 √ n q t when Ψ ( u ) = 1 q ( 1 -1 u q ) , q ∈ ( 0 , 1 ) ; and Γ ( t ) = nt when Ψ ( u ) = 1 -1 u .

Proof. Recall that we adopt the following notation: c h = E y ∣ x [ L ( h , y )] , c y = E y ∣ x [ L ( y , y )] and c y ′ = E y ∣ x [ L ( y ′ , y )] , ∀ y ′ ∈ Y . We will denote by s ( h, x, y ′ ) = e ∑ l i = 1 y ′ i h ( x,i ) ∑ y ′′ ∈ Y e ∑ l i = 1 y ′′ i h ( x,i ) and simplify notation by using s y ′ , thereby dropping the dependency on h and x . It is clear that s y ′ ∈ [ 0 , 1 ] . Next, we will analyze case by case.

The case where Φ ( u ) = log ( u ) : See the proof of Theorem 5.1.

The case where Φ ( u ) = u -1 : The conditional error of ̃ L comp can be expressed as follows:

For any h ≠ y , we define s µ as follows: set s µ y ′ = s y ′ for all y ′ ≠ y and y ′ ≠ h ; define s µ h = s y -µ ; and let s µ y = s h + µ . Note that s µ can be realized by some h ′ ∈ H under the assumption. Then, we have

(supremum is attained when µ ∗ = - √ 1 -chsh + √ 1 -cysy √ 1 -cy + √ 1 -ch )

(minimum is attained when s h = s y = 1 )

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ 2 ( ∆ C ̃ L comp , H ( h, x )) 1 2 . By Theorem B.1, we complete the proof.

The case where Φ ( u ) = 1 q ( 1 -1 u q ) , q ∈ ( 0 , 1 ) : The conditional error of ̃ L comp can be expressed as:

For any h ≠ y , we define s µ as follows: set s µ y ′ = s y ′ for all y ′ ≠ y and y ′ ≠ h ; define s µ h = s y -µ ; and let s µ y = s h + µ . Note that s µ can be realized by some h ′ ∈ H under the assumption. Then, we have

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ 2 n q 2 ( ∆ C ̃ L comp , H ( h, x )) 1 2 . By Theorem B.1, we complete the proof.

The case where Φ ( u ) = ( 1 -1 u ) : The conditional error of ̃ L comp can be expressed as:

For any h ≠ y , we define s µ as follows: set s µ y ′ = s y ′ for all y ′ ≠ y and y ′ ≠ h ; define s µ h = s y -µ ; and let s µ y = s h + µ . Note that s µ can be realized by some h ′ ∈ H under the assumption. Then, we have

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ n ∆ C ̃ L comp , H ( h, x ) . By Theorem B.1, we complete the proof.

B.5 Proof of Theorem 7.1

Theorem 7.1. Let H = F l . Assume that F is complete Then, the following H -consistency bound holds in the multi-label learning:

where Γ ( t ) = 2 √ L max t when Φ ( u ) = e -u ; Γ ( t ) = 2 √ t when Φ ( u ) = max { 0 , 1 -u } 2 ; and Γ ( t ) = t when Φ ( u ) = max { 0 , 1 -u } or Φ ( u ) = min { max { 0 , 1 -u / ρ } , 1 } , ρ > 0 .

Proof. Recall that we adopt the following notation: c h = E y ∣ x [ L ( h , y )] , c y = E y ∣ x [ L ( y , y )] and c y ′ = E y ∣ x [ L ( y ′ , y )] , ∀ y ′ ∈ Y . We will also denote by z ( h, x, y ′ ) = ∑ l i = 1 y ′ i h ( x, i ) and simplify notation by using z y ′ , thereby dropping the dependency on h and x . It is clear that the constraint can be expressed as ∑ y ′ ∈ Y z y ′ = 0 . Next, we will analyze case by case.

The case where Φ ( u ) = e -u : The conditional error of ̃ L cstnd can be expressed as follows:

For any h ≠ y , we define z µ as follows: set z µ y ′ = z y ′ for all y ′ ≠ y and y ′ ≠ h ; define z µ h = z y -µ ; and let z µ y = z h + µ . Note that z µ can be realized by some h ′ ∈ H under the assumption. Then, we have

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ 2 ( L max ) 1 2 ( ∆ C ̃ L cstnd , H ( h, x )) 1 2 . By Theorem B.1, we complete the proof.

The case where Φ ( u ) = max { 0 , 1 -u } 2 : The conditional error of ̃ L cstnd can be expressed as follows:

For any h ≠ y , we define z µ as follows: set z µ y ′ = z y ′ for all y ′ ≠ y and y ′ ≠ h ; define z µ h = z y -µ ; and let z µ y = z h + µ . Note that z µ can be realized by some h ′ ∈ H under the assumption. Then, we have

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ ( ∆ C ̃ L cstnd , H ( h, x )) 1 2 . By Theorem B.1, we complete the proof.

The case where Φ ( u ) = max { 0 , 1 -u } : The conditional error of ̃ L cstnd can be expressed as:

For any h ≠ y , we define z µ as follows: set z µ y ′ = z y ′ for all y ′ ≠ y and y ′ ≠ h ; define z µ h = z y -µ ; and let z µ y = z h + µ . Note that z µ can be realized by some h ′ ∈ H under the assumption. Then, we have

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ ∆ C ̃ L cstnd , H ( h, x ) . By Theorem B.1, we complete the proof.

The case where Φ ( u ) = min { max { 0 , 1 -u / ρ } , 1 } , ρ > 0 : The conditional error of ̃ L cstnd can be expressed as:

For any h ≠ y , we define z µ as follows: set z µ y ′ = z y ′ for all y ′ ≠ y and y ′ ≠ h ; define z µ h = z y -µ ; and let z µ y = z h + µ . Note that z µ can be realized by some h ′ ∈ H under the assumption. Then, we have to optimize)

Therefore, by Lemma B.2, ∆ C L , H ( h, x ) ≤ ∆ C ̃ L cstnd , H ( h, x ) . By Theorem B.1, we complete the proof.

C Future work

While our work introduces a unified surrogate loss framework that is Bayes-consistent across any multi-label loss, thereby broadening the scope beyond previous approaches that established consistency only for particular loss functions, there remains an exciting opportunity for empirical comparison with surrogate losses tailored to specific loss functions-a direction we leave for future work. Furthermore, refining surrogate losses to theoretically enhance performance for specific target losses presents another promising avenue for research.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the section 'Our Contributions' in the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 3, Section 4, Section 5, Section 6, Section 7, Appendix A, Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is theoretical in nature and we do not anticipate any immediate negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes an improved approach to multi-label learning using $\mathcal{H}$-consistency bounds by introducing the multi-label logistic loss to effectively handle label correlations. It extends to various multi-label losses, ensuring Bayes-consistency across diverse settings, and includes efficient gradient computation algorithms for minimizing the proposed loss function. This work offers a unified framework with robust consistency guarantees, advancing beyond traditional methods in multi-label learning.

STRENGTHS:
- Introducing the multi-label logistic loss, which effectively addresses label correlations often overlooked by traditional binary relevance surrogates under Hamming loss.

- The paper establishes $\mathcal{H}$-consistency bounds for a wide range of multi-label losses, ensuring Bayes-consistency across diverse multi-label learning scenarios. This extends beyond previous research that primarily focused on specific loss functions.

- It offers a unified framework that accommodates various multi-label losses, including novel extensions and adaptations from standard classification. This is supported by efficient gradient computation algorithms specifically designed for minimizing the proposed multi-label logistic loss.

WEAKNESSES:
- The motivation and background of this paper lack clear logic and hierarchy. It is suggested to first outline the shortcomings of existing methods and then clearly present the research questions addressed in this paper.

QUESTIONS:
Please check the weaknesses.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores surrogate losses and algorithms for multi-label learning, focusing on \\( \\mathcal{H} \\)-consistency bounds. It identifies the limitations of Hamming loss and introduces a new multi-label logistic loss that accounts for label correlations. The study extends this to a broader family of multi-label losses and adapts comp-sum losses from standard classification to multi-label learning. The authors propose a unified framework providing strong consistency guarantees for multi-label losses and describe efficient gradient computation methods for minimizing these losses.\n\nSTRENGTHS:\n1. The authors conduct a detailed analysis of the popular Hamming loss in multi-label learning when using smooth losses. They identify its sub-optimal dependency on the number of labels and its failure to account for label correlations, providing valuable insights into the limitations of existing loss functions. \n1. The authors introduce an improvement by presenting a novel surrogate loss, the multi-label logistic loss, which accounts for label correlations and benefits from label-independent \\( \\mathcal{H} \\)-consistency bounds. This innovation addresses the identified drawbacks of existing loss functions and broadens the analysis to include a more extensive family of multi-label losses, including a new extension based on linear-fractional functions related to the confusion matrix.\n1. The authors extend their work by adapting multi-label logistic losses to more comprehensive multi-label comp-sum losses. By demonstrating that this family of surrogate losses benefits from \\( \\mathcal{H} \\)-consistency bounds and Bayes-consistency across any general multi-label loss, they propose a unified surrogate loss framework. This expands upon previous work that only established consistency for specific loss functions, showcasing the applicability of their approach.\n1. The authors' writing is clear and well-structured, with each theoretical assumption and conclusion articulated distinctly.\n\nWEAKNESSES:\n1. In section 4, although the excellent properties of the proposed multi-label logistic loss are proven, providing a detailed explanation of each component of this loss would further enhance the reader's understanding of its superiority.\n2. If the advantages of this loss could be demonstrated through experimental validation, it would be more intuitive for readers.\n\nQUESTIONS:\n1. Could the authors elaborate on the individual components of the multi-label logistic loss and how each contributes to its overall effectiveness?\n2. Given the detailed nature of this loss function, what is the computational complexity associated with implementing the multi-label logistic loss compared to other traditional loss functions? It is better to be able to verify the advantages and complexity of the algorithm through experiments.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors study surrogate losses and algorithms for multi-label learning via H-consistency bounds and introduce a novel surrogate loss, multi-label logistic loss in this paper. By broadening the H-consistency bounds analyses to more general multi-label losses and extending to multi-label comp-sum losses, the authors provide a unified surrogate loss framework for H-consistency.

STRENGTHS:
1. This paper is well-written and easy to follow.
2. The authors make comprehensive reviews of related works, including their pros and cons.
3. The authors provide rigorous theoretical analyses of the limitations of existing binary relevance loss, the H-consistency of the proposed multi-label logistic loss, and the extensions to more general multi-label losses. The theoretical contribution is important for multi-label learning.
4. The authors demonstrate the efficient computation of the gradient for the proposed multi-label logistic loss and conduct time complexity analyses.

WEAKNESSES:
1. I understand that this is a theoretical work, and experiments of empirical evaluations are not its focus. However, adding experiments to compare the proposed loss with commonly used multi-label losses on standard datasets would make the paper more comprehensive and appealing. Besides, it can also verify whether the proposed loss is effective in practice.
2. There is a typo in line 300.($1-\bar{L}_{ham}(\cdot, y)$).

QUESTIONS:
See above weaknesses.

LIMITATIONS:
Yes, the authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derives H-consistency bounds for binary-relevance style surrogate losses, as well as a new surrogate, for mutli-label learning problems, showing that the proposed multi-label logistic loss whose upper-bound on the Hamming loss is independent of the number of labels.\n\nSTRENGTHS:\nThe $H$-consistency bounds provided in the paper are more informative than existing Bayes-consistency results, as they hold not just in the infinite limit.\n\nThe novel multi-label logistic loss allows upper-bounds that do not depend on the number of labels.\n\nWEAKNESSES:\nThe paper does not provide any experiments. While this is OK for a theory paper, it does mean that the question of whether the new surrogate works better in practice remains unanswered (which should be reflected in the conclusion section, at least), for two reasons:\na) all the theory provides are upper-bounds, which might not be indicative of actual performance\nb) while the theory provides better guarantees for the task loss if the surrogate is reduced to the level $\\epsilon$, it might be that reducing the new surrogate is just much more difficult than optimizing binary relevance. In particular, if the computational cost for reducing the multi-label logistic loss to the same level $\\epsilon$ as binary relevance is larger by at least $\\sqrt{l}$, then, normalized for compute, the advantage of the new surrogate vanishes.\n\nIt is claimed that the gradient of the multi-label logistic loss can be computed efficiently, yet the presented formulas still contain sums over the entire $2^l$ entries of the label space. Even if they can be precomputed once, already at moderate label space sizes of l ~ 100 would these quantities be intractable.\n\nIt is annoying that most equations are unnumbered. Even if they are not referred to in the paper, your readers and reviewers might want to reference them.\n\nthe equation after l. 328 switches between $\\mathbf{\\mathsf{y}}'$ and $y'$; and $y''$ changes to $y$\n\nl. 114: I'm not sure what the point here is of introducing the threshold $t$, if it is set to $0$ in the same sentence? Couldn't $t$ be simply absorbed into $h$?\n\nl. 178-180; 208: Arguably, completeness does _not_ hold in practice, because there is some form of upper-bound (e.g., weights representable in the given floating-point format)\n\nl. 231. Binary relevance is not just Bayes-consistent w.r.t. the Hamming-loss, but also works for precision-at-$k$\n\nIn the equation after line 542, I think $\\bar{L}$ should be $\\bar{L}_\\mathrm{ham}$? \n\nl. 503: I think $q$ should be $q_i$, and there is a weird subscript on that line.\n\nl. 174 consist -> consisting\n\nQUESTIONS:\nIn several places, the paper talks about label correlations, in particular, it claims an advantage of the new surrogate is that it takes into account label correlations. However, it is never specified what exactly that means (conditional correlations, i.e., dependent on the specific instance $x$, or marginal correlations). Further, for many loss functions (such as Hamming-loss), the Bayes-optimal prediction is a function of purely the label marginals $P[Y_i|X]$, so it is not clear to me whether taking into account label correlations actually is an advantage in those cases.\n\nThe paper mentions the decision-theoretic and the empirical-utility framework, but then seems to consider only loss functions that are defined on the level of a single instance. Aren't the two settings that same in that case?\n\nl. 525: Is the argmin unique? Are we breaking ties arbitrarily?\n\nDespite being part of the theorem, $\\mathcal{M}$ does not appear anywhere in the proof of 3.1\n\nI tried going through the proof of 4.1, but I'm not quite sure how to construct the hypothesis $h'$ with that realized $s^{\\mu}$, not do I see why the minimum is achieved for $s_h = s_y$, unless $c_h = c_y$.\n\nLIMITATIONS:\nI'm not sure if the proposed surrogate actually is tractable for label spaces with more than 50 labels.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}