SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ann Arbor, MI 48104
vkraman@umich.edu
subedi@umich.edu
tewaria@umich.edu
Multilabel ranking is a central task in machine learning. However, the most fundamental question of learnability in a multilabel ranking setting with relevance-score feedback remains unanswered. In this work, we characterize the learnability of multilabel ranking problems in both batch and online settings for a large family of ranking losses. Along the way, we give two equivalence classes of ranking losses based on learnability that capture most losses used in practice.
Multilabel ranking is a supervised learning problem where a learner is presented with an instance  $x \in \mathcal{X}$  and is required to output a ranking of  $K$  different labels in decreasing order of relevance to  $x$ . This is in contrast with multilabel classification where given an instance  $x \in \mathcal{X}$ , the learner is tasked with predicting a subset of the  $K$  labels without any explicit ordering. Multilabel ranking is a canonical learning problem with a wide range of applications to text categorization, genetics, medical imaging, social networks, and visual object recognition [Joachims, 2005, Schapire and Singer, 2000, McCallum, 1999, Clare and King, 2001, Baltruschat et al., 2019, Wang and Sukthankar, 2013, Bucak et al., 2009, Yang et al., 2016]. Recent years have seen a surge in the development of multilabel ranking methods with strong practical and theoretical guarantees [Schapire and Singer, 2000, Dembczynski et al., 2012, Gong et al., 2013, Bucak et al., 2009, Jung and Tewari, 2018, Gao and Zhou, 2011, Koyejo et al., 2015, Zhang and Zhou, 2013, Korba et al., 2018]. A related line of work has studied consistency for the convex surrogates of natural ranking losses [Duchi et al., 2010, Buffoni et al., 2011, Gao and Zhou, 2011, Ravikumar et al., 2011, Calauzenes et al., 2012, Dembczynski et al., 2012]. Despite this vast literature on multilabel ranking, the fundamental question of when a multilabel ranking problem is learnable remains unanswered.
Understanding when a hypothesis class is learnable is a fundamental question in Statistical Learning Theory. For binary classification, the finiteness of the Vapnik-Chervonenkis (VC) dimension is both sufficient and necessary for Probably Approximately Correct (PAC) learning [Vapnik and Chervonenkis, 1974, Valiant, 1984]. Likewise, the finiteness of the Daniely-Shwartz (DS) dimension characterizes multiclass PAC learnability [Daniely and Shalev-Shwartz, 2014], Brukhim et al., 2022]. In the online setting, the Littlestone dimension [Littlestone, 1987] characterizes the online learnability of a binary hypothesis class and the multiclass Littlestone dimension [Daniely et al., 2011] characterizes online multiclass learnability. Unlike classification, a distinguishing property of multilabel ranking is the mismatch between the predictions the learner makes and the feedback it receives. In particular, a learner is required to produce a permutation that ranks the relevance of the labels but only receives a relevance-score vector as feedback. This feedback model is standard in multilabel ranking since obtaining full permutation feedback is generally costly [Liu et al., 2009]. As a result, unlike the 0-1 loss in classification, there is no canonical loss function in ranking. Together, these two issues
create barriers for existing techniques used to prove learnability, such as the agnostic-to-realizable reductions from Hopkins et al. [2022] and Raman et al. [2023], to readily extend to ranking.
In this paper, we characterize the batch and online learnability of a ranking hypothesis class  $\mathcal{H} \subseteq S_K^X$  under relevance-score feedback, where  $S_K$  is the set of all permutations over  $[K] = \{1, \dots, K\}$ . In doing so, we make the following contributions.
- We show that a ranking hypothesis class  $\mathcal{H}$  embeds  $K^2$  different binary hypothesis classes  $\mathcal{H}_i^j$  for  $i, j \in [K]$ , where hypotheses in  $\mathcal{H}_i^j$  answer whether the label  $i$  should be ranked in the top  $j$ . Our main result relates the learnability of  $\mathcal{H}$  to the learnability of  $\mathcal{H}_i^j$ 's.
- We define two families of ranking loss functions that capture most if not all ranking losses used in practice. We show that these families are actually equivalence classes - the same characterization of batch and online learnability holds for every loss in that family.
- By relating the learnability of  $\mathcal{H}$  to the learnability of binary hypothesis classes  $\mathcal{H}_i^j$ , we show that existing combinatorial dimensions, like the VC and Littlestone dimension, continue to characterize learnability in the multilabel ranking setting. This allows us to prove that linear ranking hypothesis classes are learnable in the batch setting.
A unifying theme throughout the paper is our ability to constructively convert a learning algorithm  $\mathcal{A}$  for  $\mathcal{H}$  into a learning algorithm  $\mathcal{A}_i^j$  for  $\mathcal{H}_i^j$  for each  $i, j \in [K]$  and vice versa. To do so, our proof techniques involve adapting the agnostic-to-realizable reduction for batch and online classification, proposed by Hopkins et al. [2022] and Raman et al. [2023] respectively, to ranking.
Let  $\mathcal{X}$  denote the instance space,  $S_K$  the set of permutations over labels  $[K] := \{1, \dots, K\}$ , and  $\mathcal{Y} = \{0, 1, \dots, B\}^K$  the target space for some  $K, B \in \mathbb{N}$ . We highlight that the set of labels  $[K]$  is fixed beforehand and does not depend on the instance  $x \in \mathcal{X}$ . This is to be contrasted with subset ranking, the set of labels can change depending on the instance  $x \in \mathcal{X}$ .
We refer to an element  $y \in \mathcal{Y}$  as a relevance-score vector that indicates the relevance of each of the  $K$  labels, where  $B$  indicates the highest relevance and 0 indicates the lowest relevance. Throughout the paper, we treat a permutation  $\pi \in S_K$  as a vector in  $\{1, \dots, K\}^K$  that induces a ranking of the  $K$  labels in decreasing order of relevance. Accordingly, for an index  $i \in [K]$ , we let  $\pi_i \in [K]$  denote the rank of label  $i$ . Likewise, given an index  $i \in [K]$ , we let  $y^i$  denote the relevance of label  $i$ . In addition, it will be useful to define a mapping from  $S_K$  to  $\{0, 1\}^K$ . In particular, we define  $\mathrm{BinRel}(\cdot, \cdot): S_K \times [K] \to \{0, 1\}^K$  as an operator that given a permutation (ranking)  $\pi \in S_K$  and threshold  $p \in [K]$ , outputs a bit string  $b \in \{0, 1\}^K$  s.t.  $b_i = \mathbb{1}\{\pi_i \leq p\}$ .
Ranking Equivalences. Our construction of ranking loss families in Section  $\boxed{3}$  requires different notions of equivalence between permutations (rankings) in  $S_K$ . To that end, we say that  $\pi = \hat{\pi}$  if and only if for all  $i \in [K]$ ,  $\pi_i = \hat{\pi}_i$ . On the other hand, we say  $\pi \stackrel{p}{=} \hat{\pi}$  if and only if  $\{i : \pi_i \leq p\} = \{i : \hat{\pi}_i \leq p\}$ . That is, two rankings are  $p$ -equivalent if the set of labels they rank in the top- $p$  are equal. Finally, we say  $\pi \stackrel{[p]}{=} \hat{\pi}$  if and only if for all  $j \in [p]$ ,  $\{i : \pi_i \leq j\} = \{i : \hat{\pi}_i \leq j\}$ . That is, two rankings are  $[p]$ -equivalent if not only the set but also the order of labels they rank in the top- $p$  are equal.
Ranking Hypothesis. A ranking hypothesis  $h \in \mathcal{H} \subseteq S_K^\chi$  maps instances in  $\mathcal{X}$  to a ranking (permutation) in  $S_K$ . Given an instance  $x \in \mathcal{X}$ , one can think of  $h(x)$  as  $h$ 's ranking of the  $K$  different labels in decreasing order of relevance. For any ranking hypothesis  $h$ , we let  $h_i : \mathcal{X} \to [K]$  denote its restriction to the  $i$ 'th coordinate output. Accordingly, for an instance  $x \in \mathcal{X}$ ,  $h_i(x)$  gives the rank that  $h$  assigns to label  $i$ . Given a ranking hypothesis class  $\mathcal{H} \subseteq S_K^\chi$  and any  $i, j \in [K]$ , we define its binary threshold-restricted hypothesis class  $\mathcal{H}_i^j = \{h_i^j : h \in \mathcal{H}\}$  where  $h_i^j(x) = \mathbb{1}\{h_i(x) \leq j\}$ . We can think of hypotheses in  $\mathcal{H}_i^j$  as providing binary responses to queries of the form: "for instance  $x$ , should label  $i$  ranked in the top  $j$ ?" These threshold-restricted classes are central to our characterization of learnability in both the batch and online learning settings.
Batch Learnability. In the batch setting, we are interested in characterizing the learnability of a ranking hypothesis class  $\mathcal{H}$  under a model similar to the classical PAC model [Valiant, 1984].
Definition 1 (Agnostic Ranking PAC Learnability). A ranking hypothesis class  $\mathcal{H} \subseteq S_K^\chi$  is agnostic PAC learnable w.r.t. loss  $\ell : \mathcal{S}_K \times \mathcal{Y} \to \mathbb{R}_{\geq 0}$ , if there exists a function  $m : (0,1)^2 \times \mathbb{N} \to \mathbb{N}$  and a learning algorithm  $\mathcal{A} : (\mathcal{X} \times \mathcal{Y})^\star \to S_K^\chi$  with the following property: for every  $\epsilon, \delta \in (0,1)$  and for every distribution  $\mathcal{D}$  on  $\mathcal{X} \times \mathcal{Y}$ , running algorithm  $\mathcal{A}$  on  $n \geq m(\epsilon, \delta, K)$  iid samples from  $\mathcal{D}$  outputs a predictor  $g = \mathcal{A}(S)$  such that with probability at least  $1 - \delta$  over  $S \sim \mathcal{D}^n$ ,
$$
\mathbb {E} _ {\mathcal {D}} [ \ell (g (x), y) ] \leq \inf  _ {h \in \mathcal {H}} \mathbb {E} _ {\mathcal {D}} [ \ell (h (x), y) ] + \epsilon .
$$
If  $\mathcal{D}$  is restricted to the class of distributions such that  $\inf_{h\in \mathcal{H}}\mathbb{E}_{\mathcal{D}}[\ell (h(x),y)] = 0$ , then we say we are in the realizable setting. Note that unlike in classification, realizability in the multilabel ranking setting is loss dependent.
Online Learnability. In the online setting, an adversary plays a sequential game with the learner over  $T$  rounds. In each round  $t \in [T]$ , an adversary selects a labeled instance  $(x_{t}, y_{t}) \in \mathcal{X} \times \mathcal{Y}$  and reveals  $x_{t}$  to the learner. The learner makes a (potentially randomized) prediction  $\hat{\pi}_{t} \in S_{K}$ . Finally, the adversary reveals the true relevance-score vector  $y_{t}$ , and the learner suffers the loss  $\ell(\hat{\pi}_{t}, y_{t})$ , where  $\ell$  is some pre-specified ranking loss function. Given a ranking hypothesis class  $\mathcal{H} \subseteq S_{K}^{\mathcal{X}}$ , the goal of the learner is to output predictions  $\hat{\pi}_{t}$  such that its cumulative loss is close to the best possible cumulative loss over hypotheses in  $\mathcal{H}$ . A hypothesis class is online learnable if there exists an algorithm such that for any sequence of labeled examples  $(x_{1}, y_{1}), \ldots, (x_{T}, y_{T})$ , the difference in cumulative loss between its predictions and the predictions of the best possible function in  $\mathcal{H}$  is small.
Definition 2 (Agnostic Online Ranking Learnability). A ranking hypothesis class  $\mathcal{H} \subseteq S_K^\mathcal{X}$  is agnostic online learnable w.r.t. loss  $\ell$ , if there exists an (potentially randomized) algorithm  $\mathcal{A}$  such that for any adaptively chosen sequence of labeled examples  $(x_t, y_t) \in \mathcal{X} \times \mathcal{Y}$ , the algorithm outputs  $\mathcal{A}(x_t) \in S_K$  at every iteration  $t \in [T]$  such that its expected regret,
$$
R (T, K) := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell (\mathcal {A} (x _ {t}), y _ {t}) - \inf  _ {h \in \mathcal {H}} \sum_ {t = 1} ^ {T} \ell (h (x _ {t}), y _ {t}) \right],
$$
is a non-decreasing, sub-linear function of  $T$ . Here, the expectation is taken with respect to the randomness of the algorithm  $\mathcal{A}$ .
If it is further guaranteed that there exists a hypothesis  $h^\star \in \mathcal{H}$  such that  $\sum_{t=1}^{T} \ell(h^\star(x_t), y_t) = 0$ , then we say we are in the realizable setting. Again, realizability is loss dependent.
3 Ranking Loss Families
In statistical learning theory, we often characterize learnability with respect to a loss function. Unlike the 0-1 loss in classification, there is no canonical loss function in multilabel ranking. Accordingly, we define two general families of ranking loss functions in this section and later characterize learnability with respect to all losses in these families. In Appendix A, we show that many of the ranking metrics used in practice (e.g. Pairwise Rank Loss, Discounted Cumulative Gain, Reciprocal Rank, Average Precision, Precision@p, etc.) fall into one of these two families.
On a high level, we can classify ranking losses into two main groups: (A) those losses that care about both the order and magnitude of the relevance scores within the top- $p$  ranked labels and (B) those losses that only care about the magnitude of the relevance scores within the top- $p$  ranked labels. Our goal will be to define a loss family for both groups A and B. To do so, we start by identifying a canonical ranking loss that lies in each group. For group A, the normalized sum loss@p,
$$
\ell_ {\mathrm {s u m}} ^ {@ p} (\pi , y) = \sum_ {i = 1} ^ {K} \min  (\pi_ {i}, p + 1) y ^ {i} - Z _ {y} ^ {p}
$$
captures both the order and magnitude of the relevance scores only for the top- $p$  ranked labels. Here,  $Z_{y}^{p}$  is an appropriately chosen normalization factor that only depends on  $p$  and  $y$  such that
$\min_{\pi \in S_K}\ell_{\mathrm{sum}}^{\mathbb{Q}p}(\pi ,y) = 0.$  For Group B, the normalized precision loss@p,
$$
\ell_ {\text {p r e c}} ^ {\circledast p} (\pi , y) = Z _ {y} ^ {p} - \sum_ {i = 1} ^ {K} \mathbb {1} \left\{\pi_ {i} \leq p \right\} y ^ {i}
$$
cares only about the magnitude of relevance scores in the top- $p$  ranked labels. Again,  $Z_{y}^{p}$  is an appropriately chosen normalization constant that only depends on  $p$  and  $y$  such that the minimum loss is 0. The form of  $\ell_{\mathrm{prec}}^{\otimes p}$  differs from  $\ell_{\mathrm{sum}}^{\otimes p}$  because  $\sum_{i=1}^{K} \mathbb{1}\{\pi_{i} \leq p\} y^{i}$  is a gain whereas  $\sum_{i=1}^{K} \min(\pi_{i}, p+1) y^{i}$  is a loss.
Next, we build loss families around  $\ell_{\mathrm{sum}}^{\mathbb{Q}p}$  and  $\ell_{\mathrm{prec}}^{\mathbb{Q}p}$ . For  $\ell_{\mathrm{sum}}^{\mathbb{Q}p}$ , consider the family:
$$
\mathcal {L} \left(\ell_ {\text {s u m}} ^ {\circledast p}\right) = \{\ell \in \mathbb {R} ^ {\mathcal {S} _ {K} \times \mathcal {Y}}: \ell = 0 \text {i f a n d o n l y i f} \ell_ {\text {s u m}} ^ {\circledast p} = 0 \} \cap \{\ell \in \mathbb {R} ^ {\mathcal {S} _ {K} \times \mathcal {Y}}: \pi \stackrel {[ p ]} {=} \hat {\pi} \Rightarrow \ell (\pi , y) = \ell (\hat {\pi}, y) \}.
$$
By definition,  $\mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$  contains those ranking losses that are (1) zero-matched with  $\ell_{\mathrm{sum}}^{\otimes p}$  and (2) remain unchanged for any two predicted rankings (permutations) that are  $[p]$ -equivalent. The second constraint is needed to ensure that losses in  $\mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$  only depend on the order and set of labels that  $\pi$  ranks in the top- $p$ . Likewise, we can construct a similar loss family around  $\ell_{\mathrm{prec}}^{\otimes p}$  as follows:
$$
\mathcal {L} \left(\ell_ {\text {p r e c}} ^ {\circledast p}\right) = \left\{\ell \in \mathbb {R} ^ {\mathcal {S} _ {K} \times \mathcal {Y}}: \ell = 0 \text {i f a n d o n l y i f} \ell_ {\text {p r e c}} ^ {\circledast p} = 0 \right\} \cap \left\{\ell \in \mathbb {R} ^ {\mathcal {S} _ {K} \times \mathcal {Y}}: \pi \stackrel {{p}} {{=}} \hat {\pi} \Rightarrow \ell (\pi , y) = \ell (\hat {\pi}, y) \right\}.
$$
The set  $\mathcal{L}(\ell_{\mathrm{prec}}^{\otimes p})$  contains those ranking losses that are (1) zero-matched with  $\ell_{\mathrm{prec}}^{\otimes p}$  and (2) remain unchanged for any two predicted rankings (permutations) that are  $p$ -equivalent. The second constraint is needed to ensure that losses in  $\mathcal{L}(\ell_{\mathrm{prec}}^{\otimes p})$  only depend on the set of labels that  $\pi$  ranks in the top- $p$ . A major contribution of this paper is showing that both  $\mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$  and  $\mathcal{L}(\ell_{\mathrm{prec}}^{\otimes p})$  are actually equivalence classes - the same characterization of learnability holds for every loss in that family.
4 Batch Multilabel Ranking
In this section, we characterize the agnostic PAC learnability of hypothesis classes  $\mathcal{H} \subseteq S_K^{\mathcal{X}}$  with respect to both  $\mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$  and  $\mathcal{L}(\ell_{\mathrm{prec}}^{\otimes p})$ . Our main results, stated below as two theorems, relate the learnability of  $\mathcal{H}$  to the learnability of the threshold-restricted classes  $\mathcal{H}_i^j$ .
Theorem 4.1. A hypothesis class  $\mathcal{H} \subseteq \mathcal{S}_K^{\mathcal{X}}$  is agnostic PAC learnable w.r.t  $\ell \in \mathcal{L}(\ell_{sum}^{\otimes p})$  if and only if for all  $i \in [K]$  and  $j \in [p]$ ,  $\mathcal{H}_i^j$  is agnostic PAC learnable w.r.t the 0-1 loss.
Theorem 4.2. A hypothesis class  $\mathcal{H} \subseteq S_K^\mathcal{X}$  is agnostic PAC learnable w.r.t  $\ell \in \mathcal{L}(\ell_{\text{prec}}^{\otimes p})$  if and only if for all  $i \in [K]$ ,  $\mathcal{H}_i^p$  is agnostic PAC learnable w.r.t the 0-1 loss.
Since VC dimension characterizes the learnability of binary hypothesis classes under the 0-1 loss, an important corollary of Theorems 4.1 and 4.2 is that finiteness of  $\operatorname{VC}(\mathcal{H}_i^j)$ 's, for the appropriate  $i, j \in [K] \times [p]$ , is necessary and sufficient for agnostic ranking PAC learnability. Later on, we use this fact to prove that linear ranking hypothesis classes are agnostic ranking PAC learnable.
We start with the proof of Theorem 4.1, which follows in three steps. First, we show that if for all  $(i,j)\in [K]\times [p],\mathcal{H}_i^j$  is agnostic PAC learnable w.r.t 0-1 loss, then Empirical Risk Minimization (ERM) is an agnostic PAC learner for  $\mathcal{H}$  w.r.t  $\ell_{\mathrm{sum}}^{@p}$ . Next, we show that if  $\mathcal{H}$  is agnostic PAC learnable w.r.t  $\ell_{\mathrm{sum}}^{@p}$ , then  $\mathcal{H}$  is agnostic PAC learnable w.r.t any loss  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{@p})$ . Our proof of the latter uses the realizable to agnostic conversion from Hopkins et al. [2022]. Finally, we prove the necessity direction - if  $\mathcal{H}$  is agnostic PAC learnable w.r.t an arbitrary  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{@p})$ , then for all  $(i,j)\in [K]\times [p],\mathcal{H}_i^j$  is agnostic PAC learnable w.r.t 0-1 loss. The proof of necessity direction also uses the realizable to agnostic conversion from Hopkins et al. [2022]. The proof of Theorem 4.2 follows exactly the same way as Theorem 4.1 with some minor changes. Thus, we only focus on the proof of Theorem 4.1 in this section and defer all discussion of Theorem 4.2 to Appendix C.3.
We begin with Lemma 4.3, which asserts that if  $\mathcal{H}_i^j$  is agnostic PAC learnable for all  $(i,j) \in [K] \times [p]$ , then ERM is an agnostic PAC learner for  $\mathcal{H}$  w.r.t  $\ell_{\mathrm{sum}}^{@p}$ .
Lemma 4.3. If for all  $i \in [K]$  and  $j \in [p]$ ,  $\mathcal{H}_i^j$  is agnostic PAC learnable w.r.t the 0-1 loss, then ERM is an agnostic PAC learner for  $\mathcal{H} \subseteq S_K^\mathcal{X}$  w.r.t  $\ell_{sum}^{\otimes p}$ .
The proof of Lemma 4.3 exploits the nice structure of  $\ell_{\mathrm{sum}}^{\mathbb{Q}p}$  by upper bounding the empirical Rademacher complexity of the loss class  $\ell_{\mathrm{sum}}^{\mathbb{Q}p} \circ \mathcal{H} = \{(x,y) \mapsto \ell_{\mathrm{sum}}^{\mathbb{Q}p}(h(x),y) : h \in \mathcal{H}\}$  and showing that it vanishes as the sample size  $n$  becomes large. Then, standard uniform convergence arguments outlined in Proposition C.1 imply that ERM is an agnostic PAC learner for  $\mathcal{H}$  w.r.t  $\ell_{\mathrm{sum}}^{\mathbb{Q}p}$ . The full proof is in Appendix C.
Since arbitrary losses in  $\mathcal{L}(\ell_{\mathrm{sum}}^{@p})$  may not have nice analytical forms, Lemma 4.4 relates the learnability of an arbitrary loss  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{@p})$  to the learnability of  $\ell_{\mathrm{sum}}^{@p}$ .
Lemma 4.4. If  $\mathcal{H} \subseteq S_K^\mathcal{X}$  is agnostic PAC learnable w.r.t  $\ell_{sum}^{\mathbb{Q}p}$ , then  $\mathcal{H}$  is agnostic PAC learnable w.r.t any  $\ell \in \mathcal{L}(\ell_{sum}^{\mathbb{Q}p})$ .
Proof. (of Lemma 4.4) Fix  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$ . Let  $a = \min_{\pi, y} \{\ell(\pi, y) \mid \ell(\pi, y) \neq 0\}$  and  $b = \max_{\pi, y} \ell(\pi, y)$ . We need to show that if  $\mathcal{H}$  is agnostic PAC learnable w.r.t  $\ell_{\mathrm{sum}}^{\otimes p}$ , then  $\mathcal{H}$  is agnostic PAC learnable w.r.t  $\ell$ . We will do so in two steps. First, we will show that if  $\mathcal{A}$  is an agnostic PAC learner for  $\ell_{\mathrm{sum}}^{\otimes p}$ , then  $\mathcal{A}$  is also a realizable PAC learner for  $\ell$ . Next, we will show how to convert a realizable PAC learner for  $\ell$  into an agnostic PAC learner for  $\ell$  in a black-box fashion. The composition of these two pieces yields an agnostic PAC learner for  $\mathcal{H}$  w.r.t  $\ell$ .
Realizable PAC learnability of  $\mathcal{H}$  w.r.t  $\ell$ . If  $\mathcal{H}$  is agnostic PAC learnable w.r.t  $\ell_{\mathrm{sum}}^{@p}$ , then there exists a learning algorithm  $\mathcal{A}$  with sample complexity  $m(\epsilon, \delta, K)$  s.t. for any distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , with probability  $1 - \delta$  over a sample  $S \sim \mathcal{D}^n$  of size  $n \geq m(\epsilon, \delta, K)$ , the output predictor  $g = \mathcal{A}(S)$  achieves  $\mathbb{E}_{\mathcal{D}}\left[\ell_{\mathrm{sum}}^{@p}(g(x), y)\right] \leq \inf_{h \in \mathcal{H}} \mathbb{E}_{\mathcal{D}}\left[\ell_{\mathrm{sum}}^{@p}(h(x), y)\right] + \epsilon$ . In the realizable setting, we are further guaranteed that there exists a hypothesis  $h^\star \in \mathcal{H}$  s.t.  $\mathbb{E}_{\mathcal{D}}\left[\ell(h^\star(x), y)\right] = 0$ . Since  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{@p})$ , this also implies that  $\mathbb{E}_{\mathcal{D}}\left[\ell_{\mathrm{sum}}^{@p}(h^\star(x), y)\right] = 0$ . Therefore, under realizability and the fact that  $\ell \leq b\ell_{\mathrm{sum}}^{@p}$ , we have  $\mathbb{E}_{\mathcal{D}}\left[\ell(g(x), y)\right] \leq b\epsilon$ . This completes the first part of the proof as we have shown that  $\mathcal{A}$  is also a realizable PAC learner for  $\mathcal{H}$  w.r.t  $\ell$  with sample complexity  $m(\frac{\epsilon}{b}, \delta, K)$ .
Realizable-to-agnostic conversion. Now, we show how to convert the realizable PAC learner  $\mathcal{A}$  for  $\ell$  into an agnostic PAC learner for  $\ell$  in a black-box fashion. For this step, we will extend the agnostic-to-realizable reduction proposed by Hopkins et al. [2022] to the ranking setting by accommodating the mismatch between the range space of  $\mathcal{H}$  and the label space  $\mathcal{V}$ . In particular, we will show that Algorithm below converts a realizable PAC learner for  $\ell$  into an agnostic PAC learner for  $\ell$ . Note that although input  $\mathcal{A}$  is a realizable learner, the distribution  $\mathcal{D}$  may not be realizable.
Algorithm 1 Agnostic PAC learner for  $\mathcal{H}$  w.r.t.  $\ell$
Input: Realizable PAC learner  $\mathcal{A}$  for  $\mathcal{H}$ , unlabeled and labeled samples  $S_U \sim \mathcal{D}_X^n$  and  $S_L \sim \mathcal{D}^m$ . For each  $h \in \mathcal{H}_{|S_U}$ , construct a dataset
$$
S _ {U} ^ {h} = \left\{\left(x _ {1}, \tilde {y} _ {1}\right),..., \left(x _ {n}, \tilde {y} _ {n}\right) \right\} \text {s . t .} \tilde {y} _ {i} \sim \operatorname {U n i f} \left\{\operatorname {B i n R e l} \left(h \left(x _ {i}\right), 1\right),..., \operatorname {B i n R e l} \left(h \left(x _ {i}\right), p\right) \right\}
$$