SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathrm {T} _ {A} \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left| \bar {f} ^ {\star} \left(x _ {t}\right) - \bar {f} _ {t} \left(x _ {t}\right) \right| > \varepsilon \right\} \cdot \left| \bar {f} ^ {\star} \left(x _ {t}\right) - \bar {f} _ {t} \left(x _ {t}\right) \right|
$$
where the inequality holds because  $|\bar{f}^{\star}(x_{t}) - \bar{f}_{t}(x_{t})| \geq |\bar{f}^{\star}(x_{t})|$  since they have opposite signs. Using the fact that  $\mathbf{1}\{a \geq b\} \leq a / b$  for all  $a, b \geq 0$ , and the bound in (10), we get
$$
\mathrm {T} _ {A} \leq \frac {1}{\varepsilon} \sum_ {t = 1} ^ {T} Z _ {t} \big (\bar {f} ^ {\star} (x _ {t}) - \bar {f} _ {t} (x _ {t}) \big) ^ {2} \lesssim \frac {1}{\varepsilon} \mathrm {R e g} ^ {\mathrm {s q}} (\mathcal {F}; T) + \frac {1}{\varepsilon} \log \big (T / \delta \big),
$$
Gathering all the terms, we get
$$
\operatorname {R e g} _ {T} = \widetilde {\mathcal {O}} \bigg (\varepsilon T _ {\varepsilon} + \frac {1}{\varepsilon} \operatorname {R e g} ^ {\mathrm {s q}} (\mathcal {F}; T) + \frac {1}{\varepsilon} \log (1 / \delta) \bigg).
$$
Query complexity. Plugging in the query rule, and splitting as in the regret bound, we get
$$
\begin{array}{l} N _ {T} = \sum_ {t = 1} ^ {T} Z _ {t} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\left| \bar {f} _ {t} (x _ {t}) \right| \leq \Delta_ {t} (x _ {t}) \right\} \\ \leq \underbrace{\sum_{t = 1}^{T}\mathbf{1}\{| \bar{f}^{\star}(x_{t})| \leq \varepsilon \}}_{= T_{\varepsilon}} + \underbrace{\sum_{t = 1}^{T}\mathbf{1}\{| \bar{f}_{t}(x_{t})| \leq \Delta_{t}(x_{t}),|\bar{f}^{\star}(x_{t})| > \varepsilon ,\Delta_{t}(x_{t})\leq \varepsilon /3\}}_{:= \mathrm{T}_{C}} \\ + \underbrace {\sum_ {t = 1} ^ {T} \mathbf {1} \left\{\left| \bar {f} _ {t} (x _ {t}) \right| \leq \Delta_ {t} (x _ {t}) , \left| \bar {f} ^ {\star} (x _ {t}) \right| > \varepsilon , \Delta_ {t} (x _ {t}) > \varepsilon / 3 \right\}} _ {:= \mathrm {T} _ {D}} \\ \end{array}
$$
$\mathrm{T}_C$  denotes the rounds in which we make a query,  $\Delta_t(x_t) \leq \varepsilon / 3$ , and the margin for  $\bar{f}^*(x_t)$  is larger than  $\varepsilon$ . Since  $|\bar{f}_t(x_t) - \bar{f}^*(x_t)| \leq \Delta_t(x_t)$  (as shown above), we have
$$
\left| \bar {f} ^ {\star} \left(x _ {t}\right) \right| \leq \left| \bar {f} _ {t} \left(x _ {t}\right) - \bar {f} ^ {\star} \left(x _ {t}\right) \right| + \left| \bar {f} _ {t} \left(x _ {t}\right) \right| \leq \Delta_ {t} \left(x _ {t}\right) + \left| \bar {f} _ {t} \left(x _ {t}\right) \right|.
$$
$$
\mathrm {T} _ {C} \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\left| \bar {f} ^ {\star} (x _ {t}) \right| \leq 2 \Delta_ {t} (x _ {t}), \left| \bar {f} ^ {\star} (x _ {t}) \right| > \varepsilon , \Delta_ {t} (x _ {t}) \leq \varepsilon / 3 \right\} = 0.
$$
$\mathrm{T}_D$  is bounded by the number of rounds for which we make a query and  $\Delta_t(x_t) \geq \varepsilon / 3$ . Using the properties of eluder dimension, we get that
$$
\mathrm {T} _ {D} \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \{\Delta_ {t} (x _ {t}) \geq \varepsilon / 3 \} \lesssim \frac {1}{\varepsilon^ {2}} \mathrm {R e g} ^ {\mathrm {s q}} (\mathcal {F}; T) \cdot \mathfrak {E} (\mathcal {F}, \varepsilon / 6; \bar {f} ^ {\star}) + \log (1 / \delta).
$$
Gathering all the terms, we conclude
$$
N _ {T} = \widetilde {\mathcal {O}} \left(T _ {\varepsilon} + \frac {1}{\varepsilon^ {2}} \operatorname {R e g} ^ {\mathrm {s q}} (\mathcal {F}; T) \cdot \mathfrak {E} (\mathcal {F}, \varepsilon / 6; \bar {f} ^ {*}) + \frac {1}{\varepsilon^ {2}} \log (1 / \delta)\right).
$$
In Appendix E.3, we provide the complete proof and show how to generalize it for multiple actions, link function  $\phi$  and corresponding regression oracles w.r.t.  $\ell_{\phi}$ .
Before delving into the proof, we recall the relevant notation. In Algorithm 1,
- The label  $y_{t} \sim \phi(f^{\star}(x_{t}))$ , where  $\phi$  denotes the link-function given in (2).
- The function SelectAction  $(f_{t}(x_{t})) := \operatorname{argmax}_{k} \phi(f_{t}(x_{t}))[k]$ .
- For any vector  $v \in \mathbb{R}^K$ , the margin is given by the gap between the value at the largest and the second largest coordinate, i.e.
$$
\operatorname {M a r g i n} (v) = \phi (v) \left[ k ^ {\star} \right] - \max  _ {k \neq k ^ {\star}} \phi (v) [ k ],
$$
where  $k^{\star}\in \mathrm{argmax}_{k\in [K]}\phi (v)[k]$
- We also define  $T_{\varepsilon} = \sum_{t=1}^{T} \mathbf{1}\left\{\mathrm{Margin}(f^{\star}(x_t)) \leq \varepsilon \right\}$  to denote the number of samples within  $T$  rounds of interaction for which the margin w.r.t.  $f^{\star}$  is smaller than  $\varepsilon$ .
- We define the function  $\operatorname{Gap}:\mathbb{R}^K\times [K]\mapsto \mathbb{R}^+$  as
$$
\operatorname {G a p} (v, k) = \max  _ {k ^ {\prime}} \phi (v) [ k ^ {\prime} ] - \phi (v) [ k ], \tag {12}
$$
to denote the gap between the largest and the  $k$ -th coordinate of  $v$ .
E.3.1 Supporting Technical Results
Lemma 9. For any  $u$ , and  $k' \neq \operatorname{argmax}_k \phi(u)[k]$ ,
$$
\operatorname {M a r g i n} (u) \leq \operatorname {G a p} (u, k ^ {\prime}).
$$
Proof. Let  $k^{\star} = \operatorname{argmax}_{k} \phi(u)[k]$ . By definition,
$$
\begin{array}{l} \operatorname {G a p} (u, k ^ {\prime}) = \phi (u) [ k ^ {\star} ] - \phi (u) [ k ^ {\prime} ] \\ \geq \phi (u) [ k ^ {\star} ] - \max  _ {k ^ {\prime} \neq k} \phi (u) [ k ^ {\prime} ] = \operatorname {M a r g i n} (u). \\ \end{array}
$$
The following technical result establishes a certain favorable property for the function  $f^{\star}$ , whose proof follows from the regret bound of the online oracle used in Algorithm 1.
Lemma 10. With probability at least  $1 - \delta$ , the function  $f^{\star} \in \mathcal{F}$  satisfies the following for all  $t \leq T$ :
$$
\sum_ {s = 1} ^ {t} Z _ {s} \left\| f ^ {\star} \left(x _ {s}\right) - f _ {s} \left(x _ {s}\right) \right\| ^ {2} \leq \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T),
$$
where  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F},T)\coloneqq \frac{4}{\lambda}\mathrm{Reg}^{\ell_{\phi}}(\mathcal{F};T) + \frac{112}{\lambda^2}\log (4\log^2 (T) / \delta).$
Proof. The desired result follows from an application of Lemma 5, where we note that we do not query oracle when  $Z_{s} = 0$ , and thus do not count the time steps for which  $Z_{s} = 0$ .
Throughout the proof, we condition on the  $1 - \delta$  probability event that Lemma 10 holds. The next technical lemma allows us to bound the number of times when we query for the label and  $\Delta_t(x_t) \geq \zeta$  in terms of the eluder dimension (normed version) of the function class  $\mathcal{F}$ . Note that Lemma 11 holds even if the sequence  $\{x_t\}_{t \leq T}$  could be adversarially generated.
Lemma 11. Let  $f^{\star}$  satisfy Lemma 10, and let  $\Delta_t(x_t)$  be defined in (5) in Algorithm 1. Then, for any  $\zeta > 0$ , with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\Delta_ {t} \left(x _ {t}\right) \geq \zeta \right\} \leq \widetilde {O} \left(\frac {\Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} , T)}{\zeta^ {2}} \cdot \mathfrak {E} \left(\mathcal {F}, \zeta / 2; f ^ {\star}\right)\right).
$$
where  $\mathfrak{E}$  denotes the eluder dimension is given in Definition 1.
Proof. Let  $f_{t}^{\star}$  denote the maximizer of (5) at round  $t$  on point  $x_{t}$ . Thus,
$$
\Delta_ {t} \left(x _ {t}\right) = \left\| f _ {t} ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\|, \quad \text {a n d} \quad \sum_ {s = 1} ^ {t - 1} Z _ {s} \left\| f _ {t} ^ {\star} \left(x _ {s}\right) - f _ {s} \left(x _ {s}\right) \right\| ^ {2} \leq \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T). \tag {13}
$$
However, recall that Lemma 10 implies that, with probability at least  $1 - \delta$ , the function  $f^{\star}$  satisfies the bound
$$
\sum_ {s = 1} ^ {t - 1} Z _ {s} \left\| f ^ {\star} \left(x _ {s}\right) - f _ {s} \left(x _ {s}\right) \right\| ^ {2} \leq \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T). \tag {14}
$$
Using (13), (14) and Triangle inequality, we get that
$$
\begin{array}{l} \sum_ {s = 1} ^ {t - 1} Z _ {s} \left\| f _ {t} ^ {\star} (x _ {s}) - f ^ {\star} (x _ {s}) \right\| ^ {2} \leq 2 \sum_ {s = 1} ^ {t - 1} Z _ {s} \left\| f _ {t} ^ {\star} (x _ {t}) - f _ {s} (x _ {s}) \right\| ^ {2} + 2 \sum_ {s = 1} ^ {t - 1} Z _ {s} \left\| f ^ {\star} (x _ {t}) - f _ {s} (x _ {s}) \right\| ^ {2} \\ \leq 4 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T). \tag {15} \\ \end{array}
$$
Next, note that, an application of Triangle inequality implies that  $\| f_t^*(x_t) - f_t(x_t) \| \leq \| f_t^*(x_t) - f^*(x_t) \| + \| f^*(x_t) - f_t(x_t) \|$ . Thus,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\Delta_ {t} \left(x _ {t}\right) \geq \zeta \right\} = \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left\| f _ {t} ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| \geq \zeta \right\} \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left\| f _ {t} ^ {\star} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| + \left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| \geq \zeta \right\} \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left\| f _ {t} ^ {\star} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| \geq \frac {\zeta}{2} \right\} + \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left\| f _ {t} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| \geq \frac {\zeta}{2} \right\} \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\| f _ {t} ^ {\star} (x _ {t}) - f ^ {\star} (x _ {t}) \| \geq \frac {\zeta}{2} \right\} + \frac {4}{\zeta^ {2}} \sum_ {t = 1} ^ {T} Z _ {t} \left(f _ {t} (x _ {t}) - f ^ {\star} (x _ {t})\right) ^ {2} \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\| f _ {t} ^ {\star} (x _ {t}) - f ^ {\star} (x _ {t}) \| \geq \frac {\zeta}{2} \right\} + \frac {4 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} , T)}{\zeta^ {2}}, \tag {16} \\ \end{array}
$$
where in the last line we used Lemma 10 to bound the second term. In the following, we show how to bound the first term. Recall that for any  $t \leq T$ , the function  $f_t^\star$  satisfies (15). Thus, we wish to bound
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\| f _ {t} ^ {\star} (x _ {t}) - f ^ {\star} (x _ {t}) \| \geq \frac {\zeta}{2} \right\} \quad \mathrm {s . t .} \quad \sum_ {s = 1} ^ {t - 1} Z _ {s} (f _ {t} ^ {\star} (x _ {s}) - f ^ {\star} (x _ {s})) ^ {2} \leq 4 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T),
$$
for all  $t \leq T$ . An application of Lemma 8 in the above implies that
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left\| f _ {t} ^ {\star} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| \geq \frac {\zeta}{2} \right\} \leq \frac {1 7 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} , T)}{\zeta^ {2}} \cdot \mathfrak {E} \left(\mathcal {F}, \zeta / 2; f ^ {\star}\right). \tag {17}
$$
where in the last line, we used the fact that  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F},T) / \zeta^{2}\geq 1$  , for our parameter setting.
Plugging in the bound (17) in (16), and using the fact that  $\mathfrak{E}(\mathcal{F},\zeta /2;f^{\star})\geq 1$  , we get that
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\Delta_ {t} \left(x _ {t}\right) \geq \zeta \right\} \leq \frac {2 0 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} , T)}{\zeta^ {2}} \cdot \mathfrak {E} \left(\mathcal {F}, \zeta / 2; f ^ {\star}\right).
$$
The next two technical lemma's relate the margin to the gap between functions, and are useful in the analysis for regret / total number of queries.
Lemma 12. Suppose the functions  $\pi_1$  and  $\pi_2$  are defined such that  $\pi_i(x) = \operatorname{argmax}_{k\in [K]}\phi (f_i(x))[k]$ . Then, for any  $x$  for which  $\pi_1(x)\neq \pi_2(x)$ , we have
$$
\operatorname {M a r g i n} \left(f _ {1} (x)\right) \leq \phi \left(f _ {1} (x)\right) \left[ \pi_ {1} (x) \right] - \phi \left(f _ {1} (x)\right) \left[ \pi_ {2} (x) \right] \leq 2 \gamma \| f _ {1} (x) - f _ {2} (x) \| _ {2},
$$
where  $\gamma$  denotes the Lipschitz parameter of the link function  $\phi$ .
Proof. First note  $\phi(f_2(x))[ \pi_2(x)] \geq \phi(f_2(x))[ \pi_1(x)]$  by the definition of  $\pi_2$ . Thus,
$$
\begin{array}{l} \phi (f _ {1} (x)) [ \pi_ {1} (x) ] - \phi (f _ {1} (x)) [ \pi_ {2} (x) ] \\ \leq \phi (f _ {1} (x)) [ \pi_ {1} (x) ] - \phi (f _ {2} (x)) [ \pi_ {1} (x) ] + \phi (f _ {2} (x)) [ \pi_ {2} (x) ] - \phi (f _ {1} (x)) [ \pi_ {2} (x) ] \\ \leq 2 \| \phi (f _ {1} (x)) - \phi (f _ {2} (x)) \| _ {\infty} \\ \leq 2 \| \phi (f _ {1} (x)) - \phi (f _ {2} (x)) \| _ {2}. \\ \end{array}
$$
Using the fact that  $\phi$  is  $\gamma$ -Lipschitz, we immediately get that
$$
\phi (f _ {1} (x)) [ \pi_ {1} (x) ] - \phi (f _ {1} (x)) [ \pi_ {2} (x) ] \leq 2 \gamma \| f _ {1} (x) - f _ {2} (x) \| _ {2}.
$$
Lemma 13. For any two function  $f_1, f_2 \in \mathcal{F}$ , and  $x \in \mathcal{X}$
$$
\operatorname {M a r g i n} \left(f _ {1} (x)\right) - \operatorname {M a r g i n} \left(f _ {2} (x)\right) \leq 2 \gamma \| f _ {1} (x) - f _ {2} (x) \|.
$$
Proof. For the ease of notation, define
$$
k _ {1} = \underset {k \in [ k ]} {\operatorname {a r g m a x}} \phi (f _ {1} (x)) [ k ] \qquad \text {a n d} \qquad k _ {1} ^ {\prime} = \underset {k ^ {\prime} \neq k _ {1}} {\operatorname {a r g m a x}} \phi (f _ {1} (x)) [ k ^ {\prime} ],
$$
where ties are broken arbitrarily but consistently. Similarly, we define
$$
k _ {2} = \underset {k \in [ k ]} {\operatorname {a r g m a x}} \phi (f _ {2} (x)) [ k ] \quad \text {a n d} \quad k _ {2} ^ {\prime} = \underset {k ^ {\prime} \neq k _ {2}} {\operatorname {a r g m a x}} \phi (f _ {2} (x)) [ k ^ {\prime} ]. \tag {18}
$$
$$
\operatorname {M a r g i n} \left(f _ {1} (x)\right) = \phi \left(f _ {1} (x)\right) \left[ k _ {1} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {1} ^ {\prime} \right],
$$
$$
\operatorname {M a r g i n} \left(f _ {2} (x)\right) = \phi \left(f _ {2} (x)\right) \left[ k _ {2} \right] - \phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right]. \tag {19}
$$
Finally, also note that for any coordinate  $k$ ,
$$
\left. \phi \left(f _ {1} (x)\right) [ k ] - \phi \left(f _ {2} (x)\right) [ k ] \leq \left\| \phi \left(f _ {2} (x)\right) - \phi \left(f _ {1} (x)\right) \right\|. \right. \tag {20}
$$
We now proceed with the proof. Plugging in the form in (19), we get that
$$
\begin{array}{l} \operatorname {M a r g i n} \left(f _ {1} (x)\right) - \operatorname {M a r g i n} \left(f _ {2} (x)\right) \\ = \phi (f _ {1} (x)) [ k _ {1} ] - \phi (f _ {1} (x)) [ k _ {1} ^ {\prime} ] - (\phi (f _ {2} (x)) [ k _ {2} ] - \phi (f _ {2} (x)) [ k _ {2} ^ {\prime} ]) \\ = \left(\phi \left(f _ {1} (x)\right) [ k _ {1} ] - \phi \left(f _ {2} (x)\right) [ k _ {2} ]\right) + \left(\phi \left(f _ {2} (x)\right) [ k _ {2} ^ {\prime} ] - \phi \left(f _ {1} (x)\right) [ k _ {1} ^ {\prime} ]\right) \\ \leq \left(\phi \left(f _ {1} (x)\right) [ k _ {1} ] - \phi \left(f _ {2} (x)\right) [ k _ {1} ]\right) + \left(\phi \left(f _ {2} (x)\right) [ k _ {2} ^ {\prime} ] - \phi \left(f _ {1} (x)\right) [ k _ {1} ^ {\prime} ]\right) \\ \leq \left\| \phi \left(f _ {2} (x)\right) - \phi \left(f _ {1} (x)\right) \right\| + \left(\phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {1} ^ {\prime} \right]\right), \\ \end{array}
$$
where the first inequality uses the fact that  $k_{2}$  is the maximizer coordinate of  $\phi(f_{2}(x))$  and the last inequality uses (20). In the following, we bound the second term in the right hand side above under the following three cases:
- Case 1:  $k_2' \neq k_1$ : Since  $k_2' \neq k_1$ , we note that replacing  $k_1'$  by  $k_2'$  in the second term will only increase the value (see the definition in (18)). Thus,
$$
\begin{array}{l} \phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {1} ^ {\prime} \right] \leq \phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {2} ^ {\prime} \right] \\ \leq \left\| \phi \left(f _ {2} (x)\right) - \phi \left(f _ {1} (x)\right) \right\|, \\ \end{array}
$$
where the last line uses (20).
- Case 2a:  $k_2' = k_1$ ,  $k_2 = k_1'$ : Using definition of  $k_2$  in (18), we note that
$$
\begin{array}{l} \phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {1} ^ {\prime} \right] = \phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {2} \right] \\ \leq \phi (f _ {2} (x)) [ k _ {2} ] - \phi (f _ {1} (x)) [ k _ {2} ] \\ \leq \left\| \phi \left(f _ {2} (x)\right) - \phi \left(f _ {1} (x)\right) \right\|, \\ \end{array}
$$
- Case 2b:  $k_2' = k_1$ ,  $k_2 \neq k_1'$ : Using the fact that  $k_2' = k_1$  and that  $k_2 \neq k_2'$ , we get that  $k_2 \neq k_1$ . Thus using the definition of  $k_1'$  along with the fact that  $k_2 \neq k_1$ , we get that
$$
\begin{array}{l} \phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {1} ^ {\prime} \right] \leq \phi \left(f _ {2} (x)\right) \left[ k _ {2} ^ {\prime} \right] - \phi \left(f _ {1} (x)\right) \left[ k _ {2} \right] \\ \leq \phi (f _ {2} (x)) [ k _ {2} ] - \phi (f _ {1} (x)) [ k _ {2} ] \\ \leq \left\| \phi \left(f _ {2} (x)\right) - \phi \left(f _ {1} (x)\right) \right\|, \\ \end{array}
$$
where the second last line uses definition of  $k_{2}$  and the last line uses (20).
Combining all the above bounds together implies that
$$
\operatorname {M a r g i n} \left(f _ {1} (x)\right) - \operatorname {M a r g i n} \left(f _ {2} (x)\right) \leq 2 \| \phi \left(f _ {2} (x)\right) - \phi \left(f _ {1} (x)\right) \|.
$$
The final statement follows since  $\phi$  is  $\gamma$ -Lipschitz.
E.3.2 Regret Bound
For the ease of notation, for the rest of the proof in this section we define the function  $\pi^{\star}$  such that
$$
\pi^ {\star} (x) = \underset {k} {\operatorname {a r g m a x}} \phi (f ^ {\star} (x)) [ k ].
$$
Additionally, we recall that for any time  $t$ ,  $\widehat{y}_t = \text{SelectAction}(f_t(x_t)) = \arg \max_k \phi(f_t(x))[k]$ . Starting from the definition of the regret, we have
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \Pr \left(\widehat {y} _ {t} \neq y _ {t}\right) - \Pr \left(\pi^ {\star} \left(x _ {t}\right) \neq y _ {t}\right) \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \cdot \left| \Pr \left(y _ {t} = \pi^ {\star} \left(x _ {t}\right)\right) - \Pr \left(y _ {t} = \widehat {y} _ {t}\right) \right| \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \cdot | \phi (f ^ {\star} (x _ {t})) [ \pi^ {\star} (x _ {t}) ] - \phi (f ^ {\star} (x _ {t})) [ \widehat {y} _ {t} ] | \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \cdot \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right), \\ \end{array}
$$
where the second last line uses the probabilistic model from which labels are generated, and the last inequality plugs in the definition of Gap from (45). Let  $\varepsilon > 0$  be a free parameter. We can decompose the above regret bound further as:
$$
\begin{array}{l} \operatorname {R e g} _ {T} \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) \leq \varepsilon \right\} \cdot \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) \\ \end{array}
$$
Using the fact that  $y_{t}(x_{t}) = \operatorname{argmax}_{k\in [K]}\phi (f_{t}(x))[k]$  along with the definition of Gap and Lemma 12, we get that
$$
\begin{array}{l} \operatorname {R e g} _ {T} \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) \leq \varepsilon \right\} \cdot \varepsilon \\ + 2 \gamma \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {*} (x _ {t}), \operatorname {G a p} \left(f ^ {\star} (x _ {t}), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \| \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \cdot \varepsilon \\ + 2 \gamma \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \| \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \cdot \varepsilon \\ \end{array}
$$
$$
\begin{array}{l} + 2 \gamma \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} (x _ {t}), \operatorname {G a p} \left(f ^ {\star} (x _ {t}), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \| \\ + 2 \gamma \sum_ {t = 1} ^ {T} \bar {Z} _ {t} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \cdot \| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \| \tag {21} \\ = T _ {\varepsilon} \cdot \varepsilon + 2 \gamma \cdot T _ {A} + 2 \gamma \cdot T _ {B} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \|, \\ \end{array}
$$
where the second inequality holds because  $\mathrm{Gap}(f^{\star}(x_t),\widehat{y}_t)\leq \varepsilon$  implies that  $\mathrm{Margin}(f^{\star}(x_t))\leq \varepsilon$  whenever  $\widehat{y}_t\neq \pi^{\star}(x_t)$ . In the last line above, we plugged in the definition of  $T_{\varepsilon}$ , and defined  $\mathrm{T}_A$  and  $\mathrm{T}_B$  as the second term and the last term respectively (upto constants). We bound them separately below:
- Bound on  $\mathrm{T}_A$ : We note that
$$
\begin{array}{l} \mathrm {T} _ {A} = \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\widehat {y _ {t}} \neq \pi^ {\star} (x _ {t}), \operatorname {G a p} \left(f ^ {\star} (x _ {t}), \widehat {y _ {t}}\right) > \varepsilon \right\} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \| \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| > \varepsilon / 2 \gamma \right\} \cdot \left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| \\ \end{array}
$$
where the second line follows from Lemma 12 and because  $\widehat{y}_t \neq \pi^\star(x_t)$ . Using the fact that  $\mathbf{1}\{a \geq b\} \leq a / b$  for all  $a, b \geq 0$ , we get that
$$
\mathrm {T} _ {A} \leq 4 \gamma \sum_ {t = 1} ^ {T} Z _ {t} \frac {\left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| ^ {2}}{\varepsilon}. \tag {22}
$$
- Bound on  $\mathbb{T}_B$ : Fix any  $t \leq T$ , and note that Lemma 10 implies that  $\sum_{s=1}^{t} \|f^\star(x_t) - f_t(x_t)\|^2 \leq \Psi_\delta^{\ell_\phi}(\mathcal{F}, T)$ . Thus  $f^\star$  satisfies the constraint in the definition of  $\Delta_t$  in (5) and we must have that
$$
\left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| \leq \Delta_ {t} \left(x _ {t}\right). \tag {23}
$$
Plugging in the definition of  $Z_{t}$ , we note that