Title: On the Learnability of Multilabel Ranking
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multilabel Ranking

PAC Learning

Online Learning
Abstract: Multilabel ranking is a central task in machine learning. However, the most fundamental question of learnability in a multilabel ranking setting with relevance-score feedback remains unanswered. In this work, we characterize the learnability of multilabel ranking problems in both batch and online settings for a large family of ranking losses. Along the way, we give two equivalence classes of ranking losses based on learnability that capture most losses used in practice.

On the Learnability of Multilabel Ranking

Abstract

Multilabel ranking is a central task in machine learning. However, the most fundamental question of learnability in a multilabel ranking setting with relevancescore feedback remains unanswered. In this work, we characterize the learnability of multilabel ranking problems in both batch and online settings for a large family of ranking losses. Along the way, we give two equivalence classes of ranking losses based on learnability that capture most losses used in practice.

1 Introduction

Multilabel ranking is a supervised learning problem where a learner is presented with an instance x ∈ X and is required to output a ranking of K different labels in decreasing order of relevance to x . This is in contrast with multilabel classification where given an instance x ∈ X , the learner is tasked with predicting a subset of the K labels without any explicit ordering. Multilabel ranking is a canonical learning problem with a wide range of applications to text categorization, genetics, medical imaging, social networks, and visual object recognition [Joachims, 2005, Schapire and Singer, 2000, McCallum, 1999, Clare and King, 2001, Baltruschat et al., 2019, Wang and Sukthankar, 2013, Bucak et al., 2009, Yang et al., 2016]. Recent years have seen a surge in the development of multilabel ranking methods with strong practical and theoretical guarantees [Schapire and Singer, 2000, Dembczynski et al., 2012, Gong et al., 2013, Bucak et al., 2009, Jung and Tewari, 2018, Gao and Zhou, 2011, Koyejo et al., 2015, Zhang and Zhou, 2013, Korba et al., 2018]. A related line of work has studied consistency for the convex surrogates of natural ranking losses [Duchi et al., 2010, Buffoni et al., 2011, Gao and Zhou, 2011, Ravikumar et al., 2011, Calauzenes et al., 2012, Dembczynski et al., 2012]. Despite this vast literature on multilabel ranking, the fundamental question of when a multilabel ranking problem is learnable remains unanswered.

Understanding when a hypothesis class is learnable is a fundamental question in Statistical Learning Theory. For binary classification, the finiteness of the Vapnik-Chervonenkis (VC) dimension is both sufficient and necessary for Probably Approximately Correct (PAC) learning [Vapnik and Chervonenkis, 1974, Valiant, 1984]. Likewise, the finiteness of the Daniely-Shwartz (DS) dimension characterizes multiclass PAC learnability Daniely and Shalev-Shwartz [2014], Brukhim et al. [2022]. In the online setting, the Littlestone dimension [Littlestone, 1987] characterizes the online learnability of a binary hypothesis class and the multiclass Littlestone dimension [Daniely et al., 2011] characterizes online multiclass learnability. Unlike classification, a distinguishing property of multilabel ranking is the mismatch between the predictions the learner makes and the feedback it receives. In particular, a learner is required to produce a permutation that ranks the relevance of the labels but only receives a relevance-score vector as feedback. This feedback model is standard in multilabel ranking since obtaining full permutation feedback is generally costly [Liu et al., 2009]. As a result, unlike the 0-1 loss in classification, there is no canonical loss function in ranking. Together, these two issues

create barriers for existing techniques used to prove learnability, such as the agnostic-to-realizable reductions from Hopkins et al. [2022] and Raman et al. [2023], to readily extend to ranking.

In this paper, we characterize the batch and online learnability of a ranking hypothesis class H ⊆ S X K under relevance-score feedback, where S K is the set of all permutations over [ K ] = { 1 , ..., K } . In doing so, we make the following contributions.

We show that a ranking hypothesis class H embeds K 2 different binary hypothesis classes H j i for i, j ∈ [ K ] , where hypotheses in H j i answer whether the label i should be ranked in the top j . Our main result relates the learnability of H to the learnability of H j i 's.

We define two families of ranking loss functions that capture most if not all ranking losses used in practice. We show that these families are actually equivalence classes - the same characterization of batch and online learnability holds for every loss in that family.

By relating the learnability of H to the learnability of binary hypothesis classes H j i , we show that existing combinatorial dimensions, like the VC and Littlestone dimension, continue to characterize learnability in the multilabel ranking setting. This allows us to prove that linear ranking hypothesis classes are learnable in the batch setting.

A unifying theme throughout the paper is our ability to constructively convert a learning algorithm A for H into a learning algorithm A j i for H j i for each i, j ∈ [ K ] and vice versa. To do so, our proof techniques involve adapting the agnostic-to-realizable reduction for batch and online classification, proposed by Hopkins et al. [2022] and Raman et al. [2023] respectively, to ranking.

2 Preliminaries and Notation

Let X denote the instance space, S K the set of permutations over labels [ K ] := { 1 , ..., K } , and Y = { 0 , 1 , ..., B } K the target space for some K,B ∈ . We highlight that the set of labels [ K ] is fixed beforehand and does not depend on the instance x ∈ X . This is to be contrasted with subset ranking, the set of labels can change depending on the instance x ∈ X .

We refer to an element y ∈ Y as a relevance-score vector that indicates the relevance of each of the K labels, where B indicates the highest relevance and 0 indicates the lowest relevance. Throughout the paper, we treat a permutation π ∈ S K as a vector in { 1 , ..., K } K that induces a ranking of the K labels in decreasing order of relevance. Accordingly, for an index i ∈ [ K ] , we let π i ∈ [ K ] denote the rank of label i . Likewise, given an index i ∈ [ K ] , we let y i denote the relevance of label i . In addition, it will be useful to define a mapping from S K to { 0 , 1 } K . In particular, we define BinRel ( · , · ) : S K × [ K ] → { 0 , 1 } K as an operator that given a permutation (ranking) π ∈ S K and threshold p ∈ [ K ] , outputs a bit string b ∈ { 0 , 1 } K s.t. b i = { π i ≤ p } .

Ranking Equivalences . Our construction of ranking loss families in Section 3 requires different notions of equivalence between permutations (rankings) in S K . To that end, we say that π = ˆ π if and only if for all i ∈ [ K ] , π i = ˆ π i . On the other hand, we say π p = ˆ π if and only if { i : π i ≤ p } = { i : ˆ π i ≤ p } . That is, two rankings are p -equivalent if the set of labels they rank in the topp are equal. Finally, we say π [ p ] = ˆ π if and only if for all j ∈ [ p ] , { i : π i ≤ j } = { i : ˆ π i ≤ j } . That is, two rankings are [ p ] -equivalent if not only the set but also the order of labels they rank in the topp are equal.

Ranking Hypothesis . A ranking hypothesis h ∈ H ⊆ S X K maps instances in X to a ranking (permutation) in S K . Given an instance x ∈ X , one can think of h ( x ) as h 's ranking of the K different labels in decreasing order of relevance. For any ranking hypothesis h , we let h i : X → [ K ] denote its restriction to the i 'th coordinate output. Accordingly, for an instance x ∈ X , h i ( x ) gives the rank that h assigns to label i . Given a ranking hypothesis class H ⊆ S X K and any i, j ∈ [ K ] , we define its binary threshold-restricted hypothesis class H j i = { h j i : h ∈ H} where h j i ( x ) = { h i ( x ) ≤ j } . We can think of hypotheses in H j i as providing binary responses to queries of the form: 'for instance x , should label i ranked in the top j ?" These threshold-restricted classes are central to our characterization of learnability in both the batch and online learning settings.

Batch Learnability. In the batch setting, we are interested in characterizing the learnability of a ranking hypothesis class H under a model similar to the classical PAC model [Valiant, 1984].

Definition 1 (Agnostic Ranking PAC Learnability) . A ranking hypothesis class H ⊆ S X K is agnostic PAC learnable w.r.t. loss /lscript : S K × Y → R ≥ 0 , if there exists a function m : (0 , 1) 2 × N → N and a learning algorithm A : ( X × Y ) /star → S X K with the following property: for every /epsilon1 , δ ∈ (0 , 1) and for every distribution D on X × Y , running algorithm A on n ≥ m ( /epsilon1 , δ , K ) iid samples from D outputs a predictor g = A ( S ) such that with probability at least 1 -δ over S ∼ D n ,

If D is restricted to the class of distributions such that inf h ∈ H E D [ /lscript ( h ( x ) , y )] = 0 , then we say we are in the realizable setting. Note that unlike in classification, realizability in the multilabel ranking setting is loss dependent.

Online Learnability. In the online setting, an adversary plays a sequential game with the learner over T rounds. In each round t ∈ [ T ] , an adversary selects a labeled instance ( x t , y t ) ∈ X × Y and reveals x t to the learner. The learner makes a (potentially randomized) prediction ˆ π t ∈ S K . Finally, the adversary reveals the true relevance-score vector y t , and the learner suffers the loss /lscript (ˆ π t , y t ) , where /lscript is some pre-specified ranking loss function. Given a ranking hypothesis class H ⊆ S X K , the goal of the learner is to output predictions ˆ π t such that its cumulative loss is close to the best possible cumulative loss over hypotheses in H . A hypothesis class is online learnable if there exists an algorithm such that for any sequence of labeled examples ( x 1 , y 1 ) , ..., ( x T , y T ) , the difference in cumulative loss between its predictions and the predictions of the best possible function in H is small.

Definition 2 (Agnostic Online Ranking Learnability) . A ranking hypothesis class H ⊆ S X K is agnostic online learnable w.r.t. loss /lscript , if there exists an (potentially randomized) algorithm A such that for any adaptively chosen sequence of labeled examples ( x t , y t ) ∈ X × Y , the algorithm outputs A ( x t ) ∈ S K at every iteration t ∈ [ T ] such that its expected regret,

is a non-decreasing, sub-linear function of T . Here, the expectation is taken with respect to the randomness of the algorithm A .

If it is further guaranteed that there exists a hypothesis h /star ∈ H such that ∑ T t =1 /lscript ( h /star ( x t ) , y t ) = 0 , then we say we are in the realizable setting. Again, realizability is loss dependent.

3 Ranking Loss Families

In statistical learning theory, we often characterize learnability with respect to a loss function. Unlike the 0-1 loss in classification, there is no canonical loss function in multilabel ranking. Accordingly, we define two general families of ranking loss functions in this section and later characterize learnability with respect to all losses in these families. In Appendix A, we show that many of the ranking metrics used in practice (e.g. Pairwise Rank Loss, Discounted Cumulative Gain, Reciprocal Rank, Average Precision, Precision@p, etc.) fall into one of these two families.

On a high level, we can classify ranking losses into two main groups: (A) those losses that care about both the order and magnitude of the relevance scores within the topp ranked labels and (B) those losses that only care about the magnitude of the relevance scores within the topp ranked labels. Our goal will be to define a loss family for both groups A and B. To do so, we start by identifying a canonical ranking loss that lies in each group. For group A, the normalized sum loss@p,

captures both the order and magnitude of the relevance scores only for the topp ranked labels. Here, Z p y is an appropriately chosen normalization factor that only depends on p and y such that

min π ∈ S K /lscript @ p sum ( π , y ) = 0 . For Group B, the normalized precision loss@p,

cares only about the magnitude of relevance scores in the topp ranked labels. Again, Z p y is an appropriately chosen normalization constant that only depends on p and y such that the minimum loss is 0 . The form of /lscript @ p prec differs from /lscript @ p sum because ∑ K i =1 { π i ≤ p } y i is a gain whereas ∑ K i =1 min( π i , p +1) y i is a loss.

Next, we build loss families around /lscript @ p sum and /lscript @ p prec. For /lscript @ p sum, consider the family:

By definition, L ( /lscript @ p sum ) contains those ranking losses that are (1) zero-matched with /lscript @ p sum and (2) remain unchanged for any two predicted rankings (permutations) that are [ p ] -equivalent. The second constraint is needed to ensure that losses in L ( /lscript @ p sum ) only depend on the order and set of labels that π ranks in the topp . Likewise, we can construct a similar loss family around /lscript @ p prec as follows:

The set L ( /lscript @ p prec ) contains those ranking losses that are (1) zero-matched with /lscript @ p prec and (2) remain unchanged for any two predicted rankings (permutations) that are p -equivalent . The second constraint is needed to ensure that losses in L ( /lscript @ p prec ) only depend on the set of labels that π ranks in the topp . A major contribution of this paper is showing that both L ( /lscript @ p sum ) and L ( /lscript @ p prec ) are actually equivalence classes - the same characterization of learnability holds for every loss in that family.

4 Batch Multilabel Ranking

In this section, we characterize the agnostic PAC learnability of hypothesis classes H ⊆ S X K with respect to both L ( /lscript @ p sum ) and L ( /lscript @ p prec ) . Our main results, stated below as two theorems, relate the learnability of H to the learnability of the threshold-restricted classes H j i .

Theorem 4.1. A hypothesis class H ⊆ S X K is agnostic PAC learnable w.r.t /lscript ∈ L ( /lscript @ p sum ) if and only if for all i ∈ [ K ] and j ∈ [ p ] , H j i is agnostic PAC learnable w.r.t the 0-1 loss.

Theorem 4.2. A hypothesis class H ⊆ S X K is agnostic PAC learnable w.r.t /lscript ∈ L ( /lscript @ p prec ) if and only if for all i ∈ [ K ] , H p i is agnostic PAC learnable w.r.t the 0-1 loss.

Since VC dimension characterizes the learnability of binary hypothesis classes under the 0-1 loss, an important corollary of Theorems 4.1 and 4.2 is that finiteness of VC ( H j i ) 's, for the appropriate i, j ∈ [ K ] × [ p ] , is necessary and sufficient for agnostic ranking PAC learnability. Later on, we use this fact to prove that linear ranking hypothesis classes are agnostic ranking PAC learnable.

We start with the proof of Theorem 4.1, which follows in three steps. First, we show that if for all ( i, j ) ∈ [ K ] × [ p ] , H j i is agnostic PAC learnable w.r.t 0-1 loss, then Empirical Risk Minimization (ERM) is an agnostic PAC learner for H w.r.t /lscript @ p sum. Next, we show that if H is agnostic PAC learnable w.r.t /lscript @ p sum, then H is agnostic PAC learnable w.r.t any loss /lscript ∈ L ( /lscript @ p sum ) . Our proof of the latter uses the realizable to agnostic conversion from Hopkins et al. [2022]. Finally, we prove the necessity direction - if H is agnostic PAC learnable w.r.t an arbitrary /lscript ∈ L ( /lscript @ p sum ) , then for all ( i, j ) ∈ [ K ] × [ p ] , H j i is agnostic PAC learnable w.r.t 0-1 loss. The proof of necessity direction also uses the realizable to agnostic conversion from Hopkins et al. [2022]. The proof of Theorem 4.2 follows exactly the same way as Theorem 4.1 with some minor changes. Thus, we only focus on the proof of Theorem 4.1 in this section and defer all discussion of Theorem 4.2 to Appendix C.3.

We begin with Lemma 4.3, which asserts that if H j i is agnostic PAC learnable for all ( i, j ) ∈ [ K ] × [ p ] , then ERM is an agnostic PAC learner for H w.r.t /lscript @ p sum.

Lemma 4.3. If for all i ∈ [ K ] and j ∈ [ p ] , H j i is agnostic PAC learnable w.r.t the 0-1 loss, then ERM is an agnostic PAC learner for H ⊆ S X K w.r.t /lscript @ p sum .

The proof of Lemma 4.3 exploits the nice structure of /lscript @ p sum by upperbounding the empirical Rademacher complexity of the loss class /lscript @ p sum ◦ H = { ( x, y ) ↦→ /lscript @ p sum ( h ( x ) , y ) : h ∈ H ) } and showing that it vanishes as the sample size n becomes large. Then, standard uniform convergence arguments outlined in Proposition C.1 imply that ERM is an agnostic PAC learner for H w.r.t /lscript @ p sum. The full proof is in Appendix C.

Since arbitrary losses in L ( /lscript @ p sum ) may not have nice analytical forms, Lemma 4.4 relates the learnability of an arbitrary loss /lscript ∈ L ( /lscript @ p sum ) to the learnability of /lscript @ p sum.

Lemma 4.4. If H ⊆ S X K is agnostic PAC learnable w.r.t /lscript @ p sum , then H is agnostic PAC learnable w.r.t any /lscript ∈ L ( /lscript @ p sum ) .

/negationslash

Proof. (of Lemma 4.4) Fix /lscript ∈ L ( /lscript @ p sum ) . Let a = min π ,y { /lscript ( π , y ) | /lscript ( π , y ) = 0 } and b = max π ,y /lscript ( π , y ) . We need to show that if H is agnostic PAC learnable w.r.t /lscript @ p sum, then H is agnostic PAC learnable w.r.t /lscript . We will do so in two steps. First, we will show that if A is an agnostic PAC learner for /lscript @ p sum, then A is also a realizable PAC learner for /lscript . Next, we will show how to convert a realizable PAC learner for /lscript into an agnostic PAC learner for /lscript in a black-box fashion. The composition of these two pieces yields an agnostic PAC learner for H w.r.t /lscript .

Realizable PAC learnability of H w.r.t /lscript . If H is agnostic PAC learnable w.r.t /lscript @ p sum, then there exists a learning algorithm A with sample complexity m ( /epsilon1 , δ , K ) s.t. for any distribution D over X × Y , with probability 1 -δ over a sample S ∼ D n of size n ≥ m ( /epsilon1 , δ , K ) , the output predictor g = A ( S ) achieves D [ /lscript @ p sum ( g ( x ) , y )) ] ≤ inf h ∈ H D [ /lscript @ p sum ( h ( x ) , y )) ] + /epsilon1 . In the realizable setting, we are further guaranteed that there exists a hypothesis h /star ∈ H s.t. D [ /lscript ( h /star ( x ) , y ))] = 0 . Since /lscript ∈ L ( /lscript @ p sum ) , this also implies that D [ /lscript @ p sum ( h /star ( x ) , y )) ] = 0 . Therefore, under realizability and the fact that /lscript ≤ b /lscript @ p sum, we have D [ /lscript ( g ( x ) , y ))] ≤ b /epsilon1 . This completes the first part of the proof as we have shown that A is also a realizable PAC learner for H w.r.t /lscript with sample complexity m ( /epsilon1 b , δ , K ) .

Realizable-to-agnostic conversion . Now, we show how to convert the realizable PAC learner A for /lscript into an agnostic PAC learner for /lscript in a black-box fashion. For this step, we will extend the agnosticto-realizable reduction proposed by Hopkins et al. [2022] to the ranking setting by accommodating the mismatch between the range space of H and the label space Y . In particular, we will show that Algorithm 1 below converts a realizable PAC learner for /lscript into an agnostic PAC learner for /lscript . Note that although input A is a realizable learner, the distribution D may not be realizable.

Algorithm 1 Agnostic PAC learner for H w.r.t. /lscript

Input: Realizable PAC learner A for H , unlabeled and labeled samples S U ∼ D n X and S L ∼ D m 1 For each h , construct a dataset

∈ H | S U

2 Run A over all datasets to get C ( S U ) := { A ( S h U ) | h ∈ H | S U } 3 Return ˆ g ∈ C ( S U ) with the lowest empirical error over S L w.r.t. /lscript .

Let h /star = arg min h ∈ H D [ /lscript ( h ( x ) , y )] denote the optimal predictor in H w.r.t D . Consider the sample S h /star U and let g = A ( S h /star U ) . We can think of g as the output of A run over an i.i.d sample S drawn from D /star , a joint distribution over X × Y defined procedurally by first sampling x ∼ D X , then independently sampling j ∼ Unif ([ p ]) , and finally outputting the labeled sample ( x, BinRel ( h /star ( x ) , j )) . Note that D /star is indeed a realizable distribution (realized by h /star ) w.r.t both /lscript and /lscript @ p sum. Recall that m A ( /epsilon1 b , δ , K ) is the sample complexity of A . Since A is a realizable learner for H w.r.t /lscript , we have that for n ≥ m A ( a /epsilon1 2 b 2 p , δ / 2 , K ) , with probability at least 1 -δ 2 , D /star [ /lscript ( g ( x ) , y )] ≤ a /epsilon1 2 bp .

Next, by Lemma E.1, we have /lscript ( g ( x ) , y ) ≤ /lscript ( h /star ( x ) , y ) + bp a j ∼ Unif ([ p ]) [ /lscript ( g ( x ) , BinRel ( h /star ( x ) , j ))] pointwise. Taking expectations on both sides of the inequality gives

The last inequality follows from the definition of D /star , namely D /star [ /lscript ( g ( x ) , y )] = x ∼ D X j ∼ Unif ([ p ]) [ /lscript ( g ( x ) , BinRel ( h /star ( x ) , j ))] . This shows that C ( S U ) contains a hypothesis g that generalizes well with respect to D . Now we want to show that the predictor ˆ g returned in step 4 also has good generalization. Crucially, observe that C ( S U ) is a finite hypothesis class with cardinality at most 2 nK . By standard Chernoff and union bounds, with probability at least 1 -δ / 2 , the empirical risk of every hypothesis in C ( S U ) on a sample of size ≥ 8 /epsilon1 2 log 4 | C ( S U ) | δ is at most /epsilon1 / 4 away from its true error. So, if m = | S L | ≥ 8 /epsilon1 2 log 4 | C ( S U ) | δ , then with probability at least 1 -δ / 2 ,

Since ˆ g is the ERM on S L over C ( S ) , its empirical risk can be at most D [ /lscript ( h /star ( x ) , y )] + 3 /epsilon1 4 . Given that the population risk of ˆ g can be at most /epsilon1 / 4 away from its empirical risk, we have that

Applying union bounds, the entire process succeeds with probability 1 -δ . We can upper bound the sample complexity of Algorithm 1, denoted n ( /epsilon1 , δ , K ) , as

where we use | C ( S U ) | ≤ 2 Km A ( a /epsilon1 2 b 2 p , δ / 2 ,K ) . This shows that Algorithm 1 is an agnostic PAC learner for H w.r.t /lscript .

Finally, Lemma 4.5 gives the necessity direction of Theorem 4.1.

Lemma 4.5. If a hypothesis class H ⊆ S X K is agnostic PAC learnable w.r.t /lscript ∈ L ( /lscript @ p sum ) , then H j i is agnostic PAC learnable w.r.t the 0-1 loss for all ( i, j ) ∈ [ K ] × [ p ] .

Like the sufficiency proofs, the proof of Lemma 4.5 is constructive. Given an agnostic PAC learner for H w.r.t /lscript , we construct an agnostic PAC learner for H j i w.r.t 0-1 loss using a slight modification of Algorithm 1. We defer the full proof to Appendix C since the analysis is similar to that of Algorithm 1. Together, Lemmas 4.3, 4.4 and 4.5 imply Theorem 4.1.

We conclude this section by giving a concrete application of our characterization. Consider the class of ranking-hypotheses H = { x ↦→ argsort( Wx ) : W ∈ K × d } that compute rankings by sorting scores, in descending order, obtained from a linear function of the input features. Lemma 4.6, whose proof is in Appendix B, computes the VC dimension of H j i for an arbitrary i, j ∈ [ K ] .

Lemma 4.6. Let H = { x ↦→ argsort ( Wx ) : W ∈ K × d } be a linear ranking hypothesis class. Then for all i, j ∈ [ K ] , VC ( H j i ) = ˜ O ( Kd ) , where ˜ O hides logarithmic factors of d and K .

Combining Lemma 4.6 with Theorems 4.1 and 4.2 shows that linear ranking hypothesis classes are agnostic ranking PAC learnable w.r.t to all losses in L ( /lscript @ p sum ) ∪ L ( /lscript @ p prec ) . More generally, in Appendix B we give a dimension-based sufficient condition under which generic score-based ranking hypothesis classes are agnostic ranking PAC learnable.

5 Online Multilabel Ranking

We now move to the online setting and characterize the online learnability of hypothesis classes H ⊆ S X K with respect to both L ( /lscript @ p sum ) and L ( /lscript @ p prec ) . As in the batch setting, our characterization relates the learnability of H to the learnability of the threshold-restricted classes H j i .

Theorem 5.1. A hypothesis class H ⊆ S X K is agnostic online learnable w.r.t /lscript ∈ L ( /lscript @ p sum ) if and only if for all i ∈ [ K ] and j ∈ [ p ] , H j i is agnostic online learnable w.r.t the 0-1 loss.

Theorem 5.2. A hypothesis class H ⊆ S X K is agnostic online learnable w.r.t /lscript ∈ L ( /lscript @ p prec ) if and only if for all i ∈ [ K ] , H p i is agnostic online learnable w.r.t the 0-1 loss.

Since the Littlestone dimension characterizes the online learnability of binary hypothesis classes under the 0-1 loss, an important corollary of Theorems 5.1 and 5.2 is is that finiteness of Ldim ( H j i ) , for the appropriate i, j ∈ [ K ] × [ p ] , is necessary and sufficient for agnostic online ranking learnability.

We now begin the proof of Theorem 5.1. Since the proof of Theorem 5.2 follows a similar trajectory, we defer all discussion of Theorem 5.2 to Appendix D.2. Unlike Theorem 4.1 in the batch setting, we prove the sufficiency and necessity directions of Theorem 5.1 directly. We chose this direct path because, unlike the batch setting, sequential Rademacher analysis does not yield a constructive algorithm [Rakhlin et al., 2015]. On the other hand, our proofs are constructive and use the celebrated Randomized Exponential Weights Algorithm (REWA) [Cesa-Bianchi and Lugosi, 2006]. Moreover, a key ingredient of our proof is the realizable to agnostic conversion from Raman et al. [2023].

/negationslash

Proof. (of sufficiency in Theorem 5.1) Fix /lscript ∈ L ( /lscript @ p sum ) . Let a = min π ,y { /lscript ( π , y ) | /lscript ( π , y ) = 0 } and M = max π ,y /lscript ( π , y ) . Given online learners for H j i for the 0-1 loss, our goal is to construct an online learner Q for H w.r.t /lscript that enjoys sub-linear regret in T . Our strategy will be to construct a set of experts E using the online learners for H j i 's and run REWA using E and an appropriately scaled version of /lscript . Our proof borrows ideas from the realizable-to-agnostic online conversion from Raman et al. [2023] and so we use the same notation whenever possible.

Let ( x 1 , y 1 ) , ..., ( x T , y T ) ∈ ( X × Y ) T denote the stream of points to be observed by the online learner. We will assume an oblivious adversary and thus the stream is fixed before the game starts. A standard reduction (Chapter 4 in Cesa-Bianchi and Lugosi [2006]) allows us to convert oblivious regret bounds to adaptive regret bounds. Since H j i ⊆ { 0 , 1 } X is online learnable w.r.t. 0 -1 loss, we are guaranteed the existence of online learners A j i for H j i .

Constructing Experts . For any bitstring b ∈ { 0 , 1 } T , let φ : { t ∈ [ T ] : b t = 1 } → S K denote a function mapping time points where b t = 1 to rankings (permutations). Let Φ b = S { t ∈ [ T ]: b t =1 } K denote all such functions φ . For every h ∈ H , there exists a φ h b ∈ Φ b such that for all t ∈ { t : b t = 1 } , φ h b ( t ) = h ( x t ) . Let | b | = |{ t ∈ [ T ] : b t = 1 }| . For every b ∈ { 0 , 1 } T and φ ∈ Φ b , we will define an Expert E b, φ . Expert E b, φ , formally presented in Algorithm 2, uses A j i 's to make predictions in each round. However, E b, φ only updates the A j i 's on those rounds where b t = 1 , using φ to compute a labeled instance. For every b ∈ { 0 , 1 } T , let E b = ⋃ φ ∈ Φ b { E b, φ } denote the set of all Experts parameterized by functions φ ∈ Φ b . If b is the bitstring with all zeros, then E b will be empty. Therefore, we will actually define E b = { E 0 } ∪ ⋃ φ ∈ Φ b { E b, φ } , where E 0 is the expert that never updates A j i 's and only uses them for predictions in all t ∈ [ T ] . Note that 1 ≤ |E b | ≤ ( K !) | b | ≤ K K | b | .

Using these experts, Algorithm 3 presents our agnostic online learner Q for H w.r.t /lscript ∈ L ( /lscript @ p sum ) . We now show that Q enjoys sub-linear regret. We highlight that there are three sources of randomness in online learner Q , namely the randomness of sampling B , the internal randomness of A j i 's, and the internal randomness of P . One may think of internal randomness as arising from the sampling step involved in the randomized predictions. Let A be the random variable associated with joint internal randomness of A j i for all ( i, j ) ∈ [ K ] × [ p ] . Similarly, denote P to be the random variable associated with the internal randomness of P . We begin by using the guarantee of REWA.

Algorithm 2 Expert ( b, φ )

Algorithm 3 Agnostic Online Learner Q for H w.r.t. /lscript

Input: Parameter 0 < β < 1

1 Let B ∈ { 0 , 1 } T s.t. B t iid ∼ Bernoulli ( T β T )

2 Construct the set of experts E B = { E 0 } ∪ ⋃ φ ∈ Φ B { E B, φ } according to Algorithm 2 3 Run REWA P using E B and the loss function /lscript M over the stream ( x 1 , y 1 ) , ..., ( x T , y T )

REWA Guarantee . Using Theorem 21.11 in Shalev-Shwartz and Ben-David [2014] and the fact that B,A and P are mutually independent, REWA guarantees almost surely that

Taking an outer expectation gives

Noting that Q ( x t ) = P ( x t ) , we obtain

In the last step, we used the fact that for all b ∈ { 0 , 1 } T and h ∈ H , E b, φ h b ∈ E b . Here, h /star = inf h ∈ H ∑ T t =1 /lscript ( h ( x t ) , y t ) is the optimal function in hindsight. First, note that ln( |E B | ) ≤ K | B | ln( K ) . Using Jensen's inequality gives [ √ 2 T ln( |E B | ) ] ≤ √ 2 T 1+ β K ln K . Thus,

Upperbounding (I). It now suffices to upperbound [ ∑ T t =1 /lscript ( E B, φ h /star B ( x t ) , y t ) ] . Recall that Lemma E.1 gives pointwise

/negationslash where M = max π ,y /lscript ( π , y ) and a = min π ,y { /lscript ( π , y ) | /lscript ( π , y ) = 0 } . Note that, by definition of the constant M , we further get

]

where the equality follows from the fact that /lscript ∈ L ( /lscript @ p sum ) .

In order to upperbound the indicator above, we need to introduce some more notations. Given the realizable online learner A i m for ( i, m ) ∈ [ K ] × [ p ] , an instance x ∈ X , and an ordered finite sequence of labeled examples L ∈ ( X × { 0 , 1 } ) ∗ , let A i m ( x | L ) be the random variable denoting the prediction of A i m on the instance x after running and updating on L . For any b ∈ { 0 , 1 } T , h ∈ H , and t ∈ [ T ] , let L h b <t ( i, m ) = { ( x s , h i m ( x s )) : s < t and b s = 1 } denote the subsequence of the sequence of labeled instances { ( x s , h i m ( x s )) } t -1 s =1 where b s = 1 . Then, for any j ∈ [ p ] , we have

/negationslash

To prove the inequality above, consider the case when ∑ K i =1 ∑ p m =1 {A i m ( x t | L h /star B <t ( i, m )) = h /star ,m i ( x t ) } = 0 because the inequality is trivial otherwise. Then, we must have A i m ( x t | L h /star B <t ( i, m )) = h /star ,m i ( x t ) for all ( i, m ) ∈ [ K ] × [ p ] . Let V t ∈ { 0 , 1 } K × p be a binary vote matrix that E B, φ h /star B constructs in round t . Then, we have V t [ i, m ] = A i m ( x t | L h /star B <t ( i, m )) = h /star ,m i ( x t ) for all ( i, m ) ∈ [ K ] × [ p ] . Since h /star ( x t ) is a permutation, the vote vector V t 1 p must contain p labels with distinct number of non-zero votes, namely p, p -1 , p -2 , . . . , 2 , 1 votes. Similarly, there must be K -p labels with exactly 0 votes. Thus, every ˆ π t ∈ arg min π ∈ S K 〈 π , V t 1 p 〉 must rank label that obtained p votes as 1 , label with p -1 votes as 2 , and so forth. In other words, we must have ˆ π t [ p ] = h /star ( x t ) , and thus /lscript @ p sum (ˆ π t , BinRel ( h /star ( x t ) , j )) = 0 for any j ∈ [ p ] by definition of /lscript @ p sum. Our claim now follows because E B, φ h /star B ( x t ) ∈ arg min π ∈ S K 〈 π , V t 1 p 〉 . Using these two inequalities in equation (2), we obtain

/negationslash

/negationslash

which further implies that

/negationslash

The first term above is the cumulative loss of the best-fixed hypothesis in hindsight.

/negationslash

/negationslash where the last equality follows because E [ { B t = 1 } ] = T β T and the prediction of A i m ( x t | L h /star B <t ( i, m )) on round t only depends on bitstring ( B 1 , . . . , B t -1 ), but is independent of B t . Next, we can use the regret guarantee of algorithm A i m on the rounds it was updated. That is,

/negationslash

Upperbounding (II). It now suffices to show that E [ ∑ T t =1 {A i m ( x t | L h /star B <t ( i, m )) = h /star ,m i ( x t ) } ] is sub-linear for every ( i, m ) ∈ [ K ] × [ p ] . Note that we can write

/negationslash

/negationslash

/negationslash where R i m ( | B | ) is the regret of A i m , a sub-linear function of | B | . In the last step, we use the fact that A i m is a realizable algorithm for H i m and the feedback that the algorithm received was ( x t , h /star ,m i ( x t ))

in the rounds whenever B t = 1 . Next, Lemma 5.17 from Ceccherini-Silberstein et al. [2017] guarantees that there exists a concave sub-linear function ˜ R i m ( | B | ) that upperbounds R i m ( | B | ) . Thus, by Jensen's inequality, B [ R i m ( | B | )] ≤ B [ ˜ R i m ( | B | ) ] ≤ ˜ R i m ( B [ | B | ]) ≤ ˜ R i m ( T β ) , a sub-linear function of T β .

Combining (I) and (II) together, we obtain

Since ˜ R i m ( T β ) is a sublinear function of T β , we have that T T β ˜ R i m ( T β ) is a sublinear function of T . As the sum of sublinear functions is sublinear, the second term above must be a sublinear function of T . The regret is sub-linear for any choice of β ∈ (0 , 1) . This completes our proof as we have shown that the algorithm Q achieves sub-linear regret in T .

The proof of the necessity direction of Theorem 5.1 also involves constructing experts and running the REWA algorithm. Since the argument is similar, we defer details to Appendix D.1.

6 Discussion

In this paper, we characterize the learnability of a multilabel ranking hypothesis class in both the batch and online setting for a wide range of practical ranking losses. In all cases, we show that a ranking hypothesis class is learnable if and only if a sufficient number of its binary-valued threshold restrictions are learnable. Our paper studies two families of ranking loss functions and leaves it open to characterize the learnability of other natural ranking loss functions. One loss function not captured by our families is recall@p.

While we do establish quantitative bounds on the sample complexity and regret, our bounds are not optimal. It may be difficult to improve the sample complexity and regret bound at the highest level of generality for all losses in the families considered here. However, for natural losses such as sum loss, it is an interesting future direction to derive the optimal sample complexity and regret bounds in both the realizable and agnostic settings. In addition, our bounds depend on the number of labels K . Recently, K -free bounds have been achieved for multiclass classification problems in both batch and online settings [Brukhim et al., 2022, Hanneke et al., 2023]. An interesting future direction is to study whether K -free bounds are possible for multilabel ranking.

Finally, the focus of this paper is on characterizing learnability, and thus our algorithms are not computationally efficient. A natural future direction is to construct computationally efficient algorithms for multilabel ranking. Along this direction, since ERM is the most common algorithm used in practice, it is an important future direction to tightly quantify the sample complexity of ERM in the batch setting. Moreover, in learning theory, combinatorial dimensions play an important role in providing a tight quantitative characterization of learnability. Thus, it is an interesting future direction to identify combinatorial dimensions that characterize multilabel ranking learnability for specific loss functions.

References

Noga Alon, Amos Beimel, Shay Moran, and Uri Stemmer. Closure properties for private classification and online prediction. In Conference on Learning Theory , pages 119-152. PMLR, 2020.

Ivo M Baltruschat, Hannes Nickisch, Michael Grass, Tobias Knopp, and Axel Saalbach. Comparison of deep learning approaches for multi-label chest x-ray classification. Scientific reports , 9(1):1-10, 2019.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Nataly Brukhim, Daniel Carmon, Irit Dinur, Shay Moran, and Amir Yehudayoff. A characterization of multiclass learnability, 2022. URL https://arxiv.org/abs/2203.01550 .

Serhat S Bucak, Pavan Kumar Mallapragada, Rong Jin, and Anil K Jain. Efficient multi-label ranking for multi-class learning: application to object recognition. In 2009 IEEE 12th International Conference on Computer Vision , pages 2098-2105. IEEE, 2009.

David Buffoni, Clément Calauzenes, Patrick Gallinari, and Nicolas Usunier. Learning scoring functions with order-preserving losses and standardized supervision. In The 28th International Conference on Machine Learning (ICML 2011) , pages 825-832, 2011.

Clément Calauzenes, Nicolas Usunier, and Patrick Gallinari. On the (non-) existence of convex, calibrated surrogate losses for ranking. Advances in Neural Information Processing Systems , 25, 2012.

Ceccherini-Silberstein, M. Salvatori, and E. Sava-Huss. Groups, Graphs and Random Walks . London Mathematical Society Lecture Note Series. Cambridge University Press, 2017. doi: 10.1017/9781316576571.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Amanda Clare and Ross D King. Knowledge discovery in multi-label phenotype data. In Principles of Data Mining and Knowledge Discovery: 5th European Conference, PKDD 2001, Freiburg, Germany, September 3-5, 2001 Proceedings 5 , pages 42-53. Springer, 2001.

Amit Daniely and Shai Shalev-Shwartz. Optimal learners for multiclass problems. In Conference on Learning Theory , pages 287-316. PMLR, 2014.

Amit Daniely, Sivan Sabato, Shai Ben-David, and Shai Shalev-Shwartz. Multiclass learnability and the erm principle. In Sham M. Kakade and Ulrike von Luxburg, editors, Proceedings of the 24th Annual Conference on Learning Theory , volume 19 of Proceedings of Machine Learning Research , pages 207-232, Budapest, Hungary, 09-11 Jun 2011. PMLR.

Krzysztof Dembczynski, Wojciech Kotlowski, and Eyke Hüllermeier. Consistent multilabel ranking through univariate losses. arXiv preprint arXiv:1206.6401 , 2012.

John C Duchi, Lester W Mackey, and Michael I Jordan. On the consistency of ranking algorithms. In ICML , pages 327-334, 2010.

Richard M Dudley. Central limit theorems for empirical measures. The Annals of Probability , pages 899-929, 1978.

Wei Gao and Zhi-Hua Zhou. On the consistency of multi-label learning. In Proceedings of the 24th annual conference on learning theory , pages 341-358. JMLR Workshop and Conference Proceedings, 2011.

Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. Deep convolutional ranking for multilabel image annotation. arXiv preprint arXiv:1312.4894 , 2013.

Steve Hanneke, Shay Moran, Vinod Raman, Unique Subedi, and Ambuj Tewari. Multiclass online learning and uniform convergence. In The Thirty Sixth Annual Conference on Learning Theory , pages 5682-5696. PMLR, 2023.

Max Hopkins, Daniel M. Kane, Shachar Lovett, and Gaurav Mahajan. Realizable learning is all you need. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 3015-3069. PMLR, 02-05 Jul 2022.

Thorsten Joachims. Text categorization with support vector machines: Learning with many relevant features. In Machine Learning: ECML-98: 10th European Conference on Machine Learning Chemnitz, Germany, April 21-23, 1998 Proceedings , pages 137-142. Springer, 2005.

Young Hun Jung and Ambuj Tewari. Online boosting algorithms for multi-label ranking. In International Conference on Artificial Intelligence and Statistics , pages 279-287. PMLR, 2018.

Anna Korba, Alexandre Garcia, and Florence d'Alché Buc. A structured prediction approach for label ranking. Advances in Neural Information Processing Systems , 31, 2018.

Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, and Inderjit S Dhillon. Consistent multilabel classification. Advances in Neural Information Processing Systems , 28, 2015.

Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes , volume 23. Springer Science & Business Media, 1991.

Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning , 2:285-318, 1987.

Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval , 3(3):225-331, 2009.

Andrew Kachites McCallum. Multi-label text classification with a mixture model trained by em. In AAAI'99 workshop on text learning , 1999.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Sequential complexities and uniform martingale laws of large numbers. Probability theory and related fields , 161:111-153, 2015.

Vinod Raman, Unique Subedi, and Ambuj Tewari. A characterization of multioutput learnability. arXiv cs.LG , 2023. preprint arXiv:2303.17716.

Pradeep Ravikumar, Ambuj Tewari, and Eunho Yang. On ndcg consistency of listwise ranking methods. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 618-626. JMLR Workshop and Conference Proceedings, 2011.

Robert E Schapire and Yoram Singer. Boostexter: A boosting-based system for text categorization. Machine learning , 39:135-168, 2000.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, USA, 2014.

Leslie G. Valiant. A theory of the learnable. In Symposium on the Theory of Computing , 1984.

Vapnik and A. Chervonenkis. Theory of Pattern Recognition [in Russian] . 1974.

Xi Wang and Gita Sukthankar. Multi-label relational neighbor classification using social context features. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 464-472, 2013.

Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, and Jianfei Cai. Exploit bounding box annotations for multi-label object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 280-288, 2016.

Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering , 26(8):1819-1837, 2013.

Peer reviews:
comments: SUMMARY:
This paper is mainly about the learnability of multilabel ranking. The authors show that a ranking hypothesis class embeds $K^2$ different binary hypothesis classes and defines two families of ranking loss functions that capture most, if not all, ranking losses used in practice. The paper addresses the fundamental question of when a multilabel ranking problem is learnable, which remains unanswered despite the vast literature on multilabel ranking.

STRENGTHS:
The paper provides a significant contribution to understanding when a multilabel ranking problem is learnable, addressing a fundamental question that has remained unanswered despite the extensive literature on multilabel ranking. The authors also provide sufficient conditions for the agnostic Probably Approximately Correct learnability of score-based ranking hypothesis classes.

WEAKNESSES:
This paper does not contain any experiments for their theories and algorithms.

QUESTIONS:
I have no questions about this paper.

LIMITATIONS:
This paper did not contain experiments for any proposed algorithm and losses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This work study the learnability of multilabel ranking problems in both batch and online settings for different ranking losses family. They show that the multilabel ranking learnability is equivalent to the learnablities of a family of binary classification problems.

STRENGTHS:
* The paper tackles a well-motivated problem. Besides, since the theory on the learnability of multilabel ranking is blank before this work, it seems that the novelity and originality of this paper can be promised.
*  The paper is overall well-written. The presentation of theoretical results is rigorous.
* The proofs are solid based my judgement.
* The results are surprisingly concise.
* The authors present a example stating that linear ranking hypothesis class is learnable, which shows that theri results are pratical.

WEAKNESSES:
As the authors confirmed, the bounds may be not optimal.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in their work focus on a single theoretical contribution - showing that the multilabel ranking hypothesis class is PAC learnable for a wide range of ranking losses in both batch and online settings.\nThe authors first define two general families of ranking losses that are equivalent to popularly used losses.\n (like Pairwise Rank Loss, Discounted Cumulative Gain, Reciprocal Rank, Average Precision, Precision@p) \nThen they show for the defined losses and both batch and online settings, the hypothesis class of multilabel ranking losses $\\mathcal{H}$ can be expressed using binary hypothesis classes $\\mathcal{H}_i^j$ for $i, j \\in [K]$, where ($K$ is a number of labels), where $\\mathcal{H}_i^j$ tels whether the label $i$ should be ranked in the top $j$ positions. The authors show that binary class $\\mathcal{H}_i^j$ is PAC learnable, and as a result of that, it's also $\\mathcal{H}$, and existing combinatorial dimensions for batch and online learning like VC and Littleston characterized learnability in the considered multilabel setting.\n\nTo be honest, I'm far from the best person to review this work, as I don't know related works that are the most important to this paper except the book of Shalev-Shwartz and Ben-David, which I read some time ago, so I cannot judge it in the context of recent works in this area. Since the proof is very slightly outlined in the main paper, I tried to check the appendix as much as time allowed me. I checked Appendix A, B, and most of C. I did not check Appendix D and E.\n\nAssuming that all the parts are correct, I believe this is a solid contribution that should be accepted.\n\nSTRENGTHS:\n- The paper contributes to the fundamental understanding of statistical learning theory for learning multilabel ranking.\n- The presented proofs are non-trivial but, at the same time, clearly presented in the appendix.\n\nWEAKNESSES:\n- The main proofs are only slightly outlined in the main paper; by slightly, I mean they don't give out any idea behind the proof, only the intermediate results. Because of that, the main paper feels more like an introduction to notation, and then it basically forces the reader to switch to reading a long appendix (or the read can just stop reading and believe the authors the result is valid). The paper would definitely benefit from a longer form. Also, the appendix would benefit from including the original theorems and lemmas to reduce the need for switching between the main paper and the appendix.\n- It seems to me that two families of defined ranking losses capture \"most if not all ranking losses used in practice\" is a bit overstated. The result is not surprising as losses equivalent to these families (listed in Appendix A) are simple losses with (I think) the same form of an optimal hypothesis that predicts top $p$ or $K$ labels in descending order of expected relevance. These families may not capture more complex measures, but I agree that these are very popular.\n- This is a notation-dense paper; as a reader, I would appreciate some more comments/full-sentence explanations on notation in Sections 2 and 3 to confirm my understanding of it before jumping to the main results. For example, it was not clear to me if relevance is descending, meaning B is the most relevant and 0 is the least relevant or its opposite. I could only confirm it by reading definitions of $\\ell_{\\text{sum}}^{@p}$, and $\\ell_{\\text{prec}}^{@p}$ and figuring out which one makes sense, but that was one page later. Again this I believe is a fault of limited space.\n- NIT: The discussion section is not really a discussion, just a short summary.\n- NIT: Lack of equation numbers is inconvenient when reading the paper like this (for example, to make notes on equations, as there is no defined way to address them).\n- NIT: There are equations that go over the right margin, which is a bit unelegant when there is a strict space limit. \n- NIT: Bit string? Why not just binary vector? To me bit string is a compact implementation of a binary vector.\n\nQUESTIONS:\nI don't really have questions.\n\nLIMITATIONS:\nThere are no direct negative social impacts of this work. The limitation of tightness of sample complexity and regret is mentioned in the last section. I do not see any other limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of learnability of label ranking for a large family of ranking losses. The paper is purely theoretical. The main results have been built on the equivalence of realizable and agnostic learnability of the PAC model for binary classification. Using the techniques from Hopkins et al. (2022) and Raman et al. (2023), the authors show the learnability of label ranking in batch and online settings. Furthermore, they prove that ranking based on linear functions is learnable in the batch setting.\n\nSTRENGTHS:\nStrengths:\n- Important theoretical contribution\n- Practical learning problem\n\nThe paper is an important theoretical contribution for a practical learning problem. The result is novel and seems to be correct. Nevertheless, it is rather a straightforward extension of previous results.\n\nWEAKNESSES:\nWeak points:\n- Confusing problem setup\n- No discussion on related work \n- Structure of the paper \n\nThe problem setting seems to be a bit confusing. I would call it rather label ranking (potentially with ties) instead of multi-label ranking. The latter suggests that the feedback information is binary. Alternatively, a wider discussion on differences between learning problems should be given.\n\nThe paper lacks discussion on related work. The obtained results should be discussed in light of the previous theoretical results on multi-label ranking (e.g., Dembczynski et al., 2012, Gao and Zhou, 2011, Koyejo et al., 2015). Unfortunately, the link between those results is not given in the paper. \n\nHaving a page limit, it is always hard to properly divide the results between the main text and the appendix. I appreciate that the authors decided to give the most important proofs in the main text. Unfortunately, the price for it is that the paper lacks the context (clear definition of the problem, discussion of ranking loss functions), wider discussion on the results (examples, related work), and discussion on limitations and practical implications.\n\nQUESTIONS:\nLearnability is an important concept, but consistency is important as well. How do the results from the submitted paper relate to the previous results concerning consistency of multi-label ranking?\n\nLIMITATIONS:\nThe realizable-to-agnostic conversion is not computationally efficient. How does it impact the main findings of the submission? \n\nIt seems there is many other loss functions of \"ranking\" type in multi-label classification. How those results would apply to, for example, recall@p? It also seems that macro-metrics could also be interpreted as a kind of \"ordering\" functions. The abstract says that the results \"capture most, if not all, losses used in practice.\" Is the statement not too strong? What are examples of losses not included in the two families considered in the paper?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}