SlowGuess/NeurIPS_2023_OCR huggingface dataset:
2 Run  $\mathcal{A}$  over all datasets to get  $C(S_U)\coloneqq \{\mathcal{A}\big(S_U^h\big)\mid h\in \mathcal{H}_{|S_U}\}$
3 Return  $\hat{g} \in C(S_U)$  with the lowest empirical error over  $S_L$  w.r.t.  $\ell$ .
Let  $h^{\star} = \arg \min_{h\in \mathcal{H}}\mathbb{E}_{\mathcal{D}}[\ell (h(x),y)]$  denote the optimal predictor in  $\mathcal{H}$  w.r.t  $\mathcal{D}$ . Consider the sample  $S_U^{h^\star}$  and let  $g = \mathcal{A}(S_U^{h^\star})$ . We can think of  $g$  as the output of  $\mathcal{A}$  run over an i.i.d sample  $S$  drawn from  $\mathcal{D}^\star$ , a joint distribution over  $\mathcal{X}\times \mathcal{Y}$  defined procedurally by first sampling  $x\sim \mathcal{D}_X$ , then independently sampling  $j\sim \mathrm{Unif}([p])$ , and finally outputting the labeled sample  $(x,\mathrm{BinRel}(h^{\star}(x),j))$ . Note that  $\mathcal{D}^\star$  is indeed a realizable distribution (realized by  $h^\star$ ) w.r.t both  $\ell$  and  $\ell_{\mathrm{sum}}^{\otimes p}$ . Recall that  $m_{\mathcal{A}}(\frac{\epsilon}{b},\delta ,K)$  is the sample complexity of  $\mathcal{A}$ . Since  $\mathcal{A}$  is a realizable learner for  $\mathcal{H}$  w.r.t  $\ell$ , we have that for  $n\geq m_{\mathcal{A}}(\frac{a\epsilon}{2b^2p},\delta /2,K)$ , with probability at least  $1 - \frac{\delta}{2}$ ,  $\mathbb{E}_{\mathcal{D}^\star}[\ell (g(x),y)]\leq \frac{a\epsilon}{2bp}$ .
Next, by Lemma E.1, we have  $\ell(g(x), y) \leq \ell(h^{\star}(x), y) + \frac{bp}{a} \mathbb{E}_{j \sim \mathrm{Unif}([p])} [\ell(g(x), \mathrm{BinRel}(h^{\star}(x), j))]$  pointwise. Taking expectations on both sides of the inequality gives
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \left[ \ell (g (x), y) \right] \leq \mathbb {E} _ {\mathcal {D}} \left[ \ell \left(h ^ {\star} (x), y\right) \right] + \frac {b p}{a} \mathbb {E} _ {x \sim \mathcal {D}} \left[ \mathbb {E} _ {j \sim \operatorname {U n i f} ([ p ])} \left[ \ell (g (x), \operatorname {B i n R e l} \left(h ^ {\star} (x), j\right)) \right] \right] \\ \leq \mathbb {E} _ {\mathcal {D}} [ \ell (h ^ {\star} (x), y) ] + \frac {\epsilon}{2}. \\ \end{array}
$$
The last inequality follows from the definition of  $\mathcal{D}^{\star}$ , namely  $\mathbb{E}_{\mathcal{D}^{\star}}[\ell(g(x),y)] = \mathbb{E}_{x \sim \mathcal{D}_X} \mathbb{E}_{j \sim \mathrm{Unif}([p])}[\ell(g(x),\mathrm{BinRel}(h^{\star}(x),j))]$ . This shows that  $C(S_U)$  contains a hypothesis  $g$  that generalizes well with respect to  $\mathcal{D}$ . Now we want to show that the predictor  $\hat{g}$  returned in step 4 also has good generalization. Crucially, observe that  $C(S_U)$  is a finite hypothesis class with cardinality at most  $2^{nK}$ . By standard Chernoff and union bounds, with probability at least  $1 - \delta/2$ , the empirical risk of every hypothesis in  $C(S_U)$  on a sample of size  $\geq \frac{8}{\epsilon^2} \log \frac{4|C(S_U)|}{\delta}$  is at most  $\epsilon/4$  away from its true error. So, if  $m = |S_L| \geq \frac{8}{\epsilon^2} \log \frac{4|C(S_U)|}{\delta}$ , then with probability at least  $1 - \delta/2$ ,
$$
\frac {1}{| S _ {L} |} \sum_ {(x, y) \in S _ {L}} \ell (g (x), y) \leq \mathbb {E} _ {\mathcal {D}} [ \ell (g (x), y) ] + \frac {\epsilon}{4} \leq \mathbb {E} _ {\mathcal {D}} [ \ell (h ^ {\star} (x), y) ] + \frac {3 \epsilon}{4}.
$$
Since  $\hat{g}$  is the ERM on  $S_{L}$  over  $C(S)$ , its empirical risk can be at most  $\mathbb{E}_{\mathcal{D}}[\ell (h^{\star}(x),y)] + \frac{3\epsilon}{4}$ . Given that the population risk of  $\hat{g}$  can be at most  $\epsilon /4$  away from its empirical risk, we have that
$$
\mathbb {E} _ {\mathcal {D}} [ \ell (\hat {g} (x), y) ] \leq \mathbb {E} _ {\mathcal {D}} \left[ \ell \left(h ^ {\star} (x), y\right) \right] + \epsilon .
$$
Applying union bounds, the entire process succeeds with probability  $1 - \delta$ . We can upper bound the sample complexity of Algorithm 1, denoted  $n(\epsilon, \delta, K)$ , as
$$
\begin{array}{l} n (\epsilon , \delta , K) \leq m _ {\mathcal {A}} \left(\frac {a \epsilon}{2 b ^ {2} p}, \delta / 2, K\right) + O \left(\frac {1}{\epsilon^ {2}} \log \frac {| C (S _ {U}) |}{\delta}\right) \\ \leq m _ {\mathcal {A}} \left(\frac {a \epsilon}{2 b ^ {2} p}, \delta / 2, K\right) + O \left(\frac {K m _ {\mathcal {A}} (\frac {a \epsilon}{2 b ^ {2} p} , \delta / 2 , K) + \log \frac {1}{\delta}}{\epsilon^ {2}}\right), \\ \end{array}
$$
where we use  $|C(S_U)| \leq 2^{Km_A\left(\frac{a\epsilon}{2b^2p},\delta /2,K\right)}$ . This shows that Algorithm [1] is an agnostic PAC learner for  $\mathcal{H}$  w.r.t  $\ell$ .
Finally, Lemma 4.5 gives the necessity direction of Theorem 4.1.
Lemma 4.5. If a hypothesis class  $\mathcal{H} \subseteq S_K^\mathcal{X}$  is agnostic PAC learnable w.r.t  $\ell \in \mathcal{L}(\ell_{sum}^{\otimes p})$ , then  $\mathcal{H}_i^j$  is agnostic PAC learnable w.r.t the 0-1 loss for all  $(i,j) \in [K] \times [p]$ .
Like the sufficiency proofs, the proof of Lemma 4.5 is constructive. Given an agnostic PAC learner for  $\mathcal{H}$  w.r.t  $\ell$ , we construct an agnostic PAC learner for  $\mathcal{H}_i^j$  w.r.t 0-1 loss using a slight modification of Algorithm 1. We defer the full proof to Appendix C since the analysis is similar to that of Algorithm 1. Together, Lemmas 4.3 and 4.4 imply Theorem 4.1
We conclude this section by giving a concrete application of our characterization. Consider the class of ranking-hypotheses  $\mathcal{H} = \{x \mapsto \operatorname{argsort}(Wx) : W \in \mathbb{R}^{K \times d}\}$  that compute rankings by sorting scores, in descending order, obtained from a linear function of the input features. Lemma [4.6] whose proof is in Appendix B, computes the VC dimension of  $\mathcal{H}_i^j$  for an arbitrary  $i, j \in [K]$ .
Lemma 4.6. Let  $\mathcal{H} = \{x \mapsto \operatorname{argsort}(Wx) : W \in \mathbb{R}^{K \times d}\}$  be a linear ranking hypothesis class. Then for all  $i, j \in [K]$ ,  $VC(\mathcal{H}_i^j) = \tilde{O}(Kd)$ , where  $\tilde{O}$  hides logarithmic factors of  $d$  and  $K$ .
Combining Lemma 4.6 with Theorems 4.1 and 4.2 shows that linear ranking hypothesis classes are agnostic ranking PAC learnable w.r.t to all losses in  $\mathcal{L}(\ell_{\mathrm{sum}}^{\mathbb{Q}p}) \cup \mathcal{L}(\ell_{\mathrm{prec}}^{\mathbb{Q}p})$ . More generally, in Appendix B we give a dimension-based sufficient condition under which generic score-based ranking hypothesis classes are agnostic ranking PAC learnable.
5 Online Multilabel Ranking
We now move to the online setting and characterize the online learnability of hypothesis classes  $\mathcal{H} \subseteq \mathcal{S}_K^{\mathcal{X}}$  with respect to both  $\mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$  and  $\mathcal{L}(\ell_{\mathrm{prec}}^{\otimes p})$ . As in the batch setting, our characterization relates the learnability of  $\mathcal{H}$  to the learnability of the threshold-restricted classes  $\mathcal{H}_i^j$ .
Theorem 5.1. A hypothesis class  $\mathcal{H} \subseteq S_K^X$  is agnostic online learnable w.r.t  $\ell \in \mathcal{L}(\ell_{sum}^{@p})$  if and only if for all  $i \in [K]$  and  $j \in [p]$ ,  $\mathcal{H}_i^j$  is agnostic online learnable w.r.t the 0-1 loss.
Theorem 5.2. A hypothesis class  $\mathcal{H} \subseteq \mathcal{S}_K^{\mathcal{X}}$  is agnostic online learnable w.r.t  $\ell \in \mathcal{L}(\ell_{\text{prec}}^{\otimes p})$  if and only if for all  $i \in [K]$ ,  $\mathcal{H}_i^p$  is agnostic online learnable w.r.t the 0-1 loss.
Since the Littlestone dimension characterizes the online learnability of binary hypothesis classes under the 0-1 loss, an important corollary of Theorems 5.1 and 5.2 is that finiteness of  $\operatorname{Ldim}(\mathcal{H}_i^j)$ , for the appropriate  $i,j\in [K]\times [p]$ , is necessary and sufficient for agnostic online ranking learnability.
We now begin the proof of Theorem 5.1. Since the proof of Theorem 5.2 follows a similar trajectory, we defer all discussion of Theorem 5.2 to Appendix D.2. Unlike Theorem 4.1 in the batch setting, we prove the sufficiency and necessity directions of Theorem 5.1 directly. We chose this direct path because, unlike the batch setting, sequential Rademacher analysis does not yield a constructive algorithm [Rakhlin et al., 2015]. On the other hand, our proofs are constructive and use the celebrated Randomized Exponential Weights Algorithm (REWA) [Cesa-Bianchi and Lugosi, 2006]. Moreover, a key ingredient of our proof is the realizable to agnostic conversion from Raman et al. [2023].
Proof. (of sufficiency in Theorem 5.1) Fix  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$ . Let  $a = \min_{\pi, y} \{\ell(\pi, y) \mid \ell(\pi, y) \neq 0\}$  and  $M = \max_{\pi, y} \ell(\pi, y)$ . Given online learners for  $\mathcal{H}_i^j$  for the 0-1 loss, our goal is to construct an online learner  $\mathcal{Q}$  for  $\mathcal{H}$  w.r.t  $\ell$  that enjoys sub-linear regret in  $T$ . Our strategy will be to construct a set of experts  $\mathcal{E}$  using the online learners for  $\mathcal{H}_i^j$ 's and run REWA using  $\mathcal{E}$  and an appropriately scaled version of  $\ell$ . Our proof borrows ideas from the realizable-to-agnostic online conversion from [Raman et al. 2023] and so we use the same notation whenever possible.
Let  $(x_{1},y_{1}),\ldots ,(x_{T},y_{T})\in (\mathcal{X}\times \mathcal{Y})^{T}$  denote the stream of points to be observed by the online learner. We will assume an oblivious adversary and thus the stream is fixed before the game starts. A standard reduction (Chapter 4 in Cesà-Bianchi and Lugosi [2006]) allows us to convert oblivious regret bounds to adaptive regret bounds. Since  $\mathcal{H}_i^j\subseteq \{0,1\}^\mathcal{X}$  is online learnable w.r.t. 0-1 loss, we are guaranteed the existence of online learners  $\mathcal{A}_i^j$  for  $\mathcal{H}_i^j$
Constructing Experts. For any bitstring  $b \in \{0,1\}^T$ , let  $\phi : \{t \in [T] : b_t = 1\} \to S_K$  denote a function mapping time points where  $b_t = 1$  to rankings (permutations). Let  $\Phi_b = S_K^{\{t \in [T] : b_t = 1\}}$  denote all such functions  $\phi$ . For every  $h \in \mathcal{H}$ , there exists a  $\phi_b^h \in \Phi_b$  such that for all  $t \in \{t : b_t = 1\}$ ,  $\phi_b^h(t) = h(x_t)$ . Let  $|b| = |\{t \in [T] : b_t = 1\}|$ . For every  $b \in \{0,1\}^T$  and  $\phi \in \Phi_b$ , we will define an Expert  $E_{b,\phi}$ . Expert  $E_{b,\phi}$ , formally presented in Algorithm 2, uses  $A_i^j$ 's to make predictions in each round. However,  $E_{b,\phi}$  only updates the  $A_i^j$ 's on those rounds where  $b_t = 1$ , using  $\phi$  to compute a labeled instance. For every  $b \in \{0,1\}^T$ , let  $\mathcal{E}_b = \bigcup_{\phi \in \Phi_b} \{E_{b,\phi}\}$  denote the set of all Experts parameterized by functions  $\phi \in \Phi_b$ . If  $b$  is the bitstring with all zeros, then  $\mathcal{E}_b$  will be empty. Therefore, we will actually define  $\mathcal{E}_b = \{E_0\} \cup \bigcup_{\phi \in \Phi_b} \{E_{b,\phi}\}$ , where  $E_0$  is the expert that never updates  $A_i^j$ 's and only uses them for predictions in all  $t \in [T]$ . Note that  $1 \leq |\mathcal{E}_b| \leq (K!)^{|b|} \leq K^K |b|$ .
Using these experts, Algorithm 3 presents our agnostic online learner  $\mathcal{Q}$  for  $\mathcal{H}$  w.r.t  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{@p})$ . We now show that  $\mathcal{Q}$  enjoys sub-linear regret. We highlight that there are three sources of randomness in online learner  $\mathcal{Q}$ , namely the randomness of sampling  $B$ , the internal randomness of  $\mathcal{A}_i^j$ 's, and the internal randomness of  $\mathcal{P}$ . One may think of internal randomness as arising from the sampling step involved in the randomized predictions. Let  $A$  be the random variable associated with joint internal randomness of  $\mathcal{A}_i^j$  for all  $(i,j) \in [K] \times [p]$ . Similarly, denote  $P$  to be the random variable associated with the internal randomness of  $\mathcal{P}$ . We begin by using the guarantee of REWA.
Algorithm 2 Expert  $(b,\phi)$
Input: Independent copy of realizable learners  $\mathcal{A}_i^j$  of  $\mathcal{H}_i^j$  for each  $(i,j) \in [K] \times [p]$
1 for  $t = 1,\dots,T$  do
2 Receive example  $x_{t}$
3 Define a binary vote matrix  $V_{t}\in \{0,1\}^{K\times p}$  such that  $V_{t}[i,j] = \mathcal{A}_{i}^{j}(x_{t})$
4 | Predict  $\hat{\pi}_t \in \arg \min_{\pi \in S_K} \langle \pi, V_t \mathbf{1}_p \rangle$
5 if  $b_{t} = 1$  then
Let  $\pi = \phi (t)$  and for all  $(i,j)\in [K]\times [p]$ , update  $\mathcal{A}_i^j$  by passing  $(x_{t},\pi_{i}^{j})$
Algorithm 3 Agnostic Online Learner  $\mathcal{Q}$  for  $\mathcal{H}$  w.r.t.  $\ell$
Input: Parameter  $0 < \beta < 1$
1 Let  $B \in \{0, 1\}^T$  s.t.  $B_t \stackrel{\mathrm{iid}}{\sim} \text{Bernoulli}\left(\frac{T^\beta}{T}\right)$ .
2 Construct the set of experts  $\mathcal{E}_B = \{E_0\} \cup \bigcup_{\phi \in \Phi_B}\{E_{B,\phi}\}$  according to Algorithm 2
3 Run REWA  $\mathcal{P}$  using  $\mathcal{E}_B$  and the loss function  $\frac{\ell}{M}$  over the stream  $(x_{1},y_{1}),\dots,(x_{T},y_{T})$
REWA Guarantee. Using Theorem 21.11 in Shalev-Shwartz and Ben-David [2014] and the fact that  $B, A$  and  $P$  are mutually independent, REWA guarantees almost surely that
$$
\sum_ {t = 1} ^ {T} \mathbb {E} [ \ell (\mathcal {P} (x _ {t}), y _ {t}) | B, A ] \leq \inf  _ {E \in \mathcal {E} _ {B}} \sum_ {t = 1} ^ {T} \ell (E (x _ {t}), y _ {t}) + M \sqrt {2 T \ln (| \mathcal {E} _ {B} |)}.
$$
Taking an outer expectation gives
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell (\mathcal {P} (x _ {t}), y _ {t}) \right] \leq \mathbb {E} \left[ \inf  _ {E \in \mathcal {E} _ {B}} \sum_ {t = 1} ^ {T} \ell (E (x _ {t}), y _ {t}) \right] + \mathbb {E} \left[ M \sqrt {2 T \ln (| \mathcal {E} _ {B} |)} \right].
$$
Noting that  $\mathcal{Q}(x_t) = \mathcal{P}(x_t)$ , we obtain
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell \left(\mathcal {Q} \left(x _ {t}\right), y _ {t}\right) \right] \leq \mathbb {E} \left[ \inf  _ {E \in \mathcal {E} _ {B}} \sum_ {t = 1} ^ {T} \ell \left(E \left(x _ {t}\right), y _ {t}\right) \right] + \mathbb {E} \left[ M \sqrt {2 T \ln \left(\left| \mathcal {E} _ {B} \right|\right)} \right] \\ \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell \left(E _ {B, \phi_ {B} ^ {h ^ {\star}}} \left(x _ {t}\right), y _ {t}\right) \right] + M \mathbb {E} \left[ \sqrt {2 T \ln \left(\left| \mathcal {E} _ {B} \right|\right)} \right]. \\ \end{array}
$$
In the last step, we used the fact that for all  $b \in \{0,1\}^T$  and  $h \in \mathcal{H}$ ,  $E_{b,\phi_b^h} \in \mathcal{E}_b$ . Here,  $h^\star = \inf_{h \in \mathcal{H}} \sum_{t=1}^T \ell(h(x_t), y_t)$  is the optimal function in hindsight. First, note that  $\ln(|\mathcal{E}_B|) \leq K|B| \ln(K)$ . Using Jensen's inequality gives  $\mathbb{E}\left[\sqrt{2T \ln(|\mathcal{E}_B|)}\right] \leq \sqrt{2T^{1+\beta}K \ln K}$ . Thus,
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell \left(\mathcal {Q} \left(x _ {t}\right), y _ {t}\right) \right] \leq \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell \left(E _ {B , \phi_ {B} ^ {h ^ {*}}} \left(x _ {t}\right) , y _ {t}\right) \right]} _ {\text {(I)}} + M \sqrt {2 T ^ {1 + \beta} K \ln K}. \tag {1}
$$
Upperbounding (I). It now suffices to upperbound  $\mathbb{E}\left[\sum_{t = 1}^{T}\ell (E_{B,\phi_B^{h\star}}(x_t),y_t)\right]$  . Recall that Lemma E.1 gives pointwise
$$
\ell \left(E _ {B, \phi_ {B} ^ {h ^ {\star}}} \left(x _ {t}\right), y _ {t}\right) \leq \ell \left(h ^ {\star} \left(x _ {t}\right), y _ {t}\right) + \frac {p M}{a} \mathbb {E} _ {j \sim \operatorname {U n i f} ([ p ])} \left[ \ell \left(E _ {B, \phi_ {B} ^ {h ^ {\star}}} \left(x _ {t}\right), \operatorname {B i n R e l} \left(h ^ {\star} \left(x _ {t}\right), j\right)\right) \right] \tag {2}
$$
where  $M = \max_{\pi, y} \ell(\pi, y)$  and  $a = \min_{\pi, y} \{ \ell(\pi, y) \mid \ell(\pi, y) \neq 0 \}$ . Note that, by definition of the constant  $M$ , we further get
$$
\begin{array}{l} \ell \left(E _ {B, \phi_ {B} ^ {h ^ {\star}}} \left(x _ {t}\right), \operatorname {B i n R e l} \left(h ^ {\star} \left(x _ {t}\right), j\right)\right) \leq M \mathbb {1} \left\{\ell \left(E _ {B, \phi_ {B} ^ {h ^ {\star}}} \left(x _ {t}\right), \operatorname {B i n R e l} \left(h ^ {\star} \left(x _ {t}\right), j\right)\right) > 0 \right\} \\ = M \mathbb {1} \left\{\ell_ {\text {s u m}} ^ {\circledast p} \left(E _ {B, \phi_ {B} ^ {h ^ {\star}}} (x _ {t}), \operatorname {B i n R e l} \left(h ^ {\star} (x _ {t}), j\right)\right) > 0 \right\}, \\ \end{array}
$$
where the equality follows from the fact that  $\ell \in \mathcal{L}(\ell_{\mathrm{sum}}^{\otimes p})$
In order to upperbound the indicator above, we need to introduce some more notations. Given the realizable online learner  $\mathcal{A}_i^m$  for  $(i,m)\in [K]\times [p]$ , an instance  $x\in \mathcal{X}$ , and an ordered finite sequence of labeled examples  $L\in (\mathcal{X}\times \{0,1\})^{*}$ , let  $\mathcal{A}_i^m (x|L)$  be the random variable denoting the prediction of  $\mathcal{A}_i^m$  on the instance  $x$  after running and updating on  $L$ . For any  $b\in \{0,1\} ^T$ ,  $h\in \mathcal{H}$ , and  $t\in [T]$ , let  $L_{b_{< t}}^h (i,m) = \{(x_s,h_i^m (x_s)):s < t$  and  $b_{s} = 1\}$  denote the subsequence of the sequence of labeled instances  $\{(x_s,h_i^m (x_s))\}_{s = 1}^{t - 1}$  where  $b_{s} = 1$ . Then, for any  $j\in [p]$ , we have
$$
\mathbb {1} \{\ell_ {\mathrm {s u m}} ^ {@ p} (E _ {B, \phi_ {B} ^ {h ^ {\star}}} (x _ {t}), \operatorname {B i n R e l} (h ^ {\star} (x _ {t}), j)) > 0 \} \leq \sum_ {i = 1} ^ {K} \sum_ {m = 1} ^ {p} \mathbb {1} \{\mathcal {A} _ {i} ^ {m} (x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)) \neq h _ {i} ^ {\star , m} (x _ {t}) \}.
$$
To prove the inequality above, consider the case when  $\sum_{i=1}^{K}\sum_{m=1}^{p}\mathbb{1}\{\mathcal{A}_i^m(x_t\mid L_{B_{<t}}^{h^\star}(i,m))\neq h_i^{\star,m}(x_t)\} = 0$  because the inequality is trivial otherwise. Then, we must have  $\mathcal{A}_i^m(x_t\mid L_{B_{<t}}^{h^\star}(i,m)) = h_i^{\star,m}(x_t)$  for all  $(i,m)\in [K]\times [p]$ . Let  $V_{t}\in \{0,1\}^{K\times p}$  be a binary vote matrix that  $E_{B,\phi_B^{h^\star}}$  constructs in round  $t$ . Then, we have  $V_{t}[i,m] = \mathcal{A}_{i}^{m}(x_{t}\mid L_{B_{<t}}^{h^{\star}}(i,m)) = h_{i}^{\star,m}(x_{t})$  for all  $(i,m)\in [K]\times [p]$ . Since  $h^\star (x_t)$  is a permutation, the vote vector  $V_{t}\mathbf{1}_{p}$  must contain  $p$  labels with distinct numbers of non-zero votes, namely  $p,p - 1,p - 2,\ldots ,2,1$  votes. Similarly, there must be  $K - p$  labels with exactly 0 votes. Thus, every  $\hat{\pi}_t\in \arg \min_{\pi \in S_K}\langle \pi ,V_t\mathbf{1}_p\rangle$  must rank label that obtained  $p$  votes as 1, label with  $p - 1$  votes as 2, and so forth. In other words, we must have  $\hat{\pi}_t\stackrel {[p]}{=}h^\star (x_t)$ , and thus  $\ell_{\mathrm{sum}}^{\otimes p}(\hat{\pi}_t,\mathrm{BinRel}(h^\star (x_t),j)) = 0$  for any  $j\in [p]$  by definition of  $\ell_{\mathrm{sum}}^{\otimes p}$ . Our claim now follows because  $E_{B,\phi_B^{h^\star}}(x_t)\in \arg \min_{\pi \in S_K}\langle \pi ,V_t\mathbf{1}_p\rangle$ . Using these two inequalities in equation (2), we obtain
$$
\ell (E _ {B, \phi_ {B} ^ {h ^ {\star}}} (x _ {t}), y _ {t}) \leq \ell (h ^ {\star} (x _ {t}), y _ {t}) + \frac {p M ^ {2}}{a} \sum_ {i = 1} ^ {K} \sum_ {m = 1} ^ {p} \mathbb {1} \{\mathcal {A} _ {i} ^ {m} (x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)) \neq h _ {i} ^ {\star , m} (x _ {t}) \},
$$
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell (E _ {B, \phi_ {B} ^ {h ^ {\star}}} (x _ {t}), y _ {t}) \right] \leq \sum_ {t = 1} ^ {T} \ell (h ^ {\star} (x _ {t}), y _ {t}) + \frac {p M ^ {2}}{a} \sum_ {i = 1} ^ {K} \sum_ {m = 1} ^ {p} \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {1} \{\mathcal {A} _ {i} ^ {m} (x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i , m)) \neq h _ {i} ^ {\star , m} (x _ {t}) \} \right]} _ {\mathrm {(I I)}}.
$$
The first term above is the cumulative loss of the best-fixed hypothesis in hindsight.
Upper bounding (II). It now suffices to show that  $\mathbb{E}\left[\sum_{t = 1}^{T}\mathbb{1}\{\mathcal{A}_i^m (x_t\mid L_{B_{< t}}^{h^\star}(i,m))\neq h_i^{\star ,m}(x_t)\}\right]$  is sub-linear for every  $(i,m)\in [K]\times [p]$ . Note that we can write
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {1} \{\mathcal {A} _ {i} ^ {m} (x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)) \neq h _ {i} ^ {\star , m} (x _ {t}) \} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} \{\mathcal {A} _ {i} ^ {m} (x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)) \neq h _ {i} ^ {\star , m} (x _ {t}) \} \right] \frac {\mathbb {E} \left[ \mathbb {1} \{B _ {t} = 1 \} \right]}{\mathbb {E} \left[ \mathbb {1} \{B _ {t} = 1 \} \right]} \\ = \frac {T}{T ^ {\beta}} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {1} \{\mathcal {A} _ {i} ^ {m} (x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)) \neq h _ {i} ^ {\star , m} (x _ {t}) \} \mathbb {1} \{B _ {t} = 1 \} \right], \\ \end{array}
$$
where the last equality follows because  $\mathbb{E}\left[\mathbb{1}\{B_t = 1\}\right] = \frac{T^\beta}{T}$  and the prediction of  $A_{i}^{m}(x_{t}\mid L_{B < t}^{h^{\star}}(i,m))$  on round  $t$  only depends on bitstring  $(B_{1},\ldots ,B_{t - 1})$ , but is independent of  $B_{t}$ . Next, we can use the regret guarantee of algorithm  $A_{i}^{m}$  on the rounds it was updated. That is,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathcal {A} _ {i} ^ {m} \left(x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)\right) \mathbb {1} \left\{B _ {t} = 1 \right\} \right] = \mathbb {E} \left[ \sum_ {t: B _ {t} = 1} \mathcal {A} _ {i} ^ {m} \left(x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)\right) \neq h _ {i} ^ {\star , m} \left(x _ {t}\right) \right\} \\ = \mathbb {E} \left[ \mathbb {E} \left[ \sum_ {t: B _ {t} = 1} \mathcal {A} _ {i} ^ {m} (x _ {t} \mid L _ {B _ {<   t}} ^ {h ^ {\star}} (i, m)) \neq h _ {i} ^ {\star , m} (x _ {t}) \right\} \right] \Bigg | B \Bigg ] \leq \mathbb {E} \left[ R _ {i} ^ {m} (| B |) \right], \\ \end{array}
$$
where  $R_{i}^{m}(|B|)$  is the regret of  $\mathcal{A}_i^m$ , a sub-linear function of  $|B|$ . In the last step, we use the fact that  $\mathcal{A}_i^m$  is a realizable algorithm for  $\mathcal{H}_i^m$  and the feedback that the algorithm received was  $(x_{t},h_{i}^{\star ,m}(x_{t}))$
in the rounds whenever  $B_{t} = 1$ . Next, Lemma 5.17 from Ceccherini-Silberstein et al. [2017] guarantees that there exists a concave sub-linear function  $\tilde{R}_i^m (|B|)$  that upperbounds  $R_i^m (|B|)$ . Thus, by Jensen's inequality,  $\mathbb{E}_B\left[R_i^m (|B|)\right]\leq \mathbb{E}_B\left[\tilde{R}_i^m (|B|)\right]\leq \tilde{R}_i^m (\mathbb{E}_B\left[|B|\right])\leq \tilde{R}_i^m (T^\beta)$ , a sub-linear function of  $T^\beta$ .
Combining (I) and (II) together, we obtain
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell (\mathcal {Q} (x _ {t}), y _ {t}) \right] \leq \inf _ {h \in \mathcal {H}} \sum_ {t = 1} ^ {T} \ell (h (x _ {t}), y _ {t}) + \frac {p M ^ {2}}{a} \sum_ {i = 1} ^ {K} \sum_ {m = 1} ^ {p} \frac {T}{T ^ {\beta}} \tilde {R} _ {i} ^ {m} (T ^ {\beta}) + M \sqrt {2 T ^ {1 + \beta} K \ln K}.
$$
Since  $\tilde{R}_i^m (T^\beta)$  is a sublinear function of  $T^{\beta}$ , we have that  $\frac{T}{T^{\beta}}\tilde{R}_i^m (T^\beta)$  is a sublinear function of  $T$ . As the sum of sublinear functions is sublinear, the second term above must be a sublinear function of  $T$ . The regret is sub-linear for any choice of  $\beta \in (0,1)$ . This completes our proof as we have shown that the algorithm  $\mathcal{Q}$  achieves sub-linear regret in  $T$ .
The proof of the necessity direction of Theorem 5.1 also involves constructing experts and running the REWA algorithm. Since the argument is similar, we defer details to Appendix D.1.