SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underset {a \sim p} {\mathbb {E}} \left[ f ^ {\star} (x, \pi_ {f ^ {\star}} (x), a) \right] \leq \frac {\gamma}{4} \underset {a, b \sim p} {\mathbb {E}} \left[ \left(f (x, a, b) - f ^ {\star} (x, a, b)\right) ^ {2} \right] + \frac {5 A}{\gamma}
$$
whenever  $\gamma \geq 2A$
Lemma 12. Assume that for each  $f \in \mathcal{F}$ , there exists an associated function  $r: \mathcal{X} \times \mathcal{A} \to [0,1]$  such that  $f(x, a, b) = r(x, a) - r(x, b)$  for any  $x \in \mathcal{X}$  and  $a, b \in \mathcal{A}$ . In this case, for any context  $x \in \mathcal{X}$ , if we define  $p$  as
$$
p (a) = \left\{ \begin{array}{l l} \frac {1}{A + \gamma \big (r (x , \pi_ {f} (x)) - r (x , a) \big)} & a \neq \pi_ {f} (x) \\ 1 - \sum_ {a \neq \pi_ {f} (x)} p (a) & a = \pi_ {f} (x) \end{array} \right.,
$$
$$
\underset {a \sim p} {\mathbb {E}} \left[ f ^ {\star} (x, \pi_ {f ^ {\star}} (x), a) \right] \leq \gamma \underset {a, b \sim p} {\mathbb {E}} \left[ \left(f (x, a, b) - f ^ {\star} (x, a, b)\right) ^ {2} \right] + \frac {A}{\gamma}
$$
Proof of lemma 12. Fix any  $b \in \mathcal{A}$ . Then, the distribution  $p$  can be rewritten as
$$
p (a) = \left\{ \begin{array}{l l} \Big (A + 2 \gamma \left(\frac {r (x , \pi_ {f} (x)) - r (x , b) + 1}{2} - \frac {r (x , a) - r (x , b) + 1}{2}\right) \Big) ^ {- 1} & a \neq \pi_ {f} (x) \\ 1 - \sum_ {a \neq \pi_ {f} (x)} p (a) & a = \pi_ {f} (x) \end{array} \right..
$$
Therefore, denoting  $f^{\star}(x,a,b) = r^{\star}(x,a) - r^{\star}(x,b)$  for some function  $r^{\star}$ , we have
$$
\begin{array}{l} \underset {a \sim p} {\mathbb {E}} \left[ f ^ {\star} (x, \pi_ {f ^ {\star}} (x), a) \right] = \underset {a \sim p} {\mathbb {E}} \left[ r ^ {\star} (x, \pi_ {f ^ {\star}} (x)) - r ^ {\star} (x, a) \right] \\ = 2 \underset {a \sim p} {\mathbb {E}} \left[ \frac {r ^ {\star} (x , \pi_ {f ^ {\star}} (x)) - r ^ {\star} (x , b) + 1}{2} - \frac {r ^ {\star} (x , a) - r ^ {\star} (x , b) + 1}{2} \right] \\ \leq 2 \cdot 2 \gamma \underset {a \sim p} {\mathbb {E}} \left[ \left(\frac {r (x , a) - r (x , b) + 1}{2} - \frac {r ^ {\star} (x , a) - r ^ {\star} (x , b) + 1}{2}\right) ^ {2} \right] + \frac {A}{\gamma} \\ = \gamma \underset {a \sim p} {\mathbb {E}} \left[ \left(f (x, a, b) - f ^ {\star} (x, a, b)\right) ^ {2} \right] + \frac {A}{\gamma} \\ \end{array}
$$
where for the inequality above we invoked Lemma 3 with  $\hat{y}(a) = (r(x, a) - r(x, b) + 1)/2$  and  $y^{\star}(a) = (r^{\star}(x, a) - r^{\star}(x, b) + 1)/2$ . We note that the above holds for any  $b \in \mathcal{A}$ . Hence, we complete the proof by sampling  $b \sim p$ .
Lemma 13. Assume  $f^{\star} \in \mathcal{F}_t$  for all  $t \in [T]$ . Suppose there exists some  $t' \in [T]$  such that  $\lambda_t = 0$  for all  $t \leq t'$ . Then we have
$$
\sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} w _ {t} \leq 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta))
$$
Proof. Since  $f^{\star} \in \mathcal{F}_t$ , we always have  $\pi_{f^{\star}}(x_t) \in \mathcal{A}_t$  for all  $t \in [T]$ . Hence, whenever  $Z_t$  is zero, we have  $\mathcal{A}_t = \{\pi_{f^{\star}}(x_t)\}$  and thus we do not incur any regret. Hence, we know  $Z_t w_t$  is either 0 or at least  $\Delta$  by Lemma 7. Let us fix an integer  $m > 1 / \Delta$ , whose value will be specified later. We divide the interval  $[\Delta, 1]$  into bins of width  $1 / m$  and conduct a refined study of the sum of  $Z_t w_t$ :
$$
\begin{array}{l} \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} w _ {t} \leq \sum_ {t = 1} ^ {t ^ {\prime}} \sum_ {j = 0} ^ {(1 - \Delta) m - 1} Z _ {t} w _ {t} \cdot \mathbb {1} \left\{Z _ {t} w _ {t} \in \left[ \Delta + \frac {j}{m}, \Delta + \frac {j + 1}{m} \right] \right\} \\ \leq \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \left(\Delta + \frac {j + 1}{m}\right) \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} \mathbb {1} \left\{w _ {t} \geq \Delta + \frac {j}{m} \right\} \\ = \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \left(\Delta + \frac {j + 1}{m}\right) \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} \mathbb {1} \left\{\sup  _ {a, b \in \mathcal {A} _ {t}} \sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a, b) - f ^ {\prime} (x _ {t}, a, b) \geq \Delta + \frac {j}{m} \right\} \\ = \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \left(\Delta + \frac {j + 1}{m}\right) \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} \sup  _ {a, b \in \mathcal {A} _ {t}} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a, b) - f ^ {\prime} (x _ {t}, a, b) \geq \Delta + \frac {j}{m} \right\} \\ \leq \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \left(\Delta + \frac {j + 1}{m}\right) \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} \sum_ {a, b} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a, b) - f ^ {\prime} (x _ {t}, a, b) \geq \left(\Delta + \frac {j}{m}\right) \right\} \\ \leq \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \left(\Delta + \frac {j + 1}{m}\right) A ^ {2} \underbrace {\sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} \underset {a , b \sim p _ {t}} {\mathbb {E}} \mathbb {1} \left\{\sup  _ {f , f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t} , a , b) - f ^ {\prime} (x _ {t} , a , b) \geq \left(\Delta + \frac {j}{m}\right) \right\}} _ {(*)} \\ \end{array}
$$
where in the third inequality we replace the supremum over  $a, b$  by the summation over  $a, b$ , and in the last inequality we further replace it by the expectation. Here recall that  $p_t(a)$  is uniform when  $\lambda_t = 0$ , leading to the extra  $A^2$  factor. To deal with  $(*)$ , we first apply Lemma 4 to recover the empirical  $a_t$  and  $b_t$ , and then apply Lemma 10 to get an upper bound via the eluder dimension:
$$
\begin{array}{l} (*) \leq 2 \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f \left(x _ {t}, a _ {t}, b _ {t}\right) - f ^ {\prime} \left(x _ {t}, a _ {t}, b _ {t}\right) \geq \left(\Delta + \frac {j}{m}\right) \right\} + 8 \log (\delta^ {- 1}) \\ \leq 2 \left(\frac {4 \beta}{\left(\Delta + \frac {j}{m}\right) ^ {2}} + 1\right) \dim_ {E} (\mathcal {F}; \Delta) + 8 \log (\delta^ {- 1}) \\ \leq \frac {1 0 \beta}{\left(\Delta + \frac {j}{m}\right) ^ {2}} \cdot \dim_ {E} \left(\mathcal {F}; \Delta\right) + 8 \log (\delta^ {- 1}) \\ \end{array}
$$
with probability at least  $1 - \delta$ . Plugging  $(*)$  back, we obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} w _ {t} \leq \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \left(\Delta + \frac {j + 1}{m}\right) \cdot \frac {1 0 A ^ {2} \beta}{\left(\Delta + \frac {j}{m}\right) ^ {2}} \cdot \dim_ {E} (\mathcal {F}; \Delta) + 8 m A ^ {2} \log (\delta^ {- 1}) \\ = 1 0 A ^ {2} \beta \cdot \dim_ {E} (\mathcal {F}, \Delta) \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \frac {\Delta + \frac {j + 1}{m}}{\left(\Delta + \frac {j}{m}\right) ^ {2}} + 8 m A ^ {2} \log (\delta^ {- 1}) \\ \leq 1 0 A ^ {2} \beta \cdot \dim_ {E} (\mathcal {F}, \Delta) \left(\frac {\Delta + 1 / m}{\Delta^ {2}} + \sum_ {j = 1} ^ {(1 - \Delta) m - 1} \frac {2}{\Delta + \frac {j}{m}}\right) + 8 m A ^ {2} \log (\delta^ {- 1}) \\ \leq 10A^{2}\beta \cdot \dim_{E}(\mathcal{F},\Delta)\sum_{j = 0}^{(1 - \Delta)m - 1}\frac{2}{\Delta + \frac{j}{m}} +8mA^{2}\log (\delta^{-1}) \\ \leq 2 0 A ^ {2} \beta \cdot \dim_ {E} (\mathcal {F}, \Delta) \sum_ {j = 0} ^ {(1 - \Delta) m - 1} \int_ {j - 1} ^ {j} \frac {1}{\Delta + \frac {x}{m}} d x + 8 m A ^ {2} \log (\delta^ {- 1}) \\ = 2 0 A ^ {2} \beta \cdot \dim_ {E} (\mathcal {F}, \Delta) \int_ {- 1} ^ {(1 - \Delta) m - 1} \frac {1}{\Delta + \frac {x}{m}} d x + 8 m A ^ {2} \log (\delta^ {- 1}) \\ = 2 0 A ^ {2} \beta \cdot \dim_ {E} (\mathcal {F}, \Delta) \cdot m \log \left(\frac {1}{\Delta - m ^ {- 1}}\right) + 8 m A ^ {2} \log (\delta^ {- 1}) \\ \end{array}
$$
where for the second inequality, we use the fact that  $(j + 1) / m \leq 2j / m$  for any  $j \geq 1$ ; for the third inequality, we assume  $m > 1 / \Delta$ . Setting  $m = 2 / \Delta$ , we arrive at
$$
\begin{array}{l} \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} w _ {t} \leq 4 0 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / \Delta) + 1 6 A ^ {2} \log (\delta^ {- 1}) / \Delta \\ \leq 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)), \\ \end{array}
$$
Lemma 14. Whenever
$$
5 6 A ^ {2} \beta \cdot \dim_ {E} (\mathcal {F}, \Delta) \cdot \log (2 / (\delta \Delta)) / \Delta <   \sqrt {A T / \beta},
$$
we have  $\lambda_1 = \lambda_2 = \dots = \lambda_T = 0$  with probability at least  $1 - \delta$ .
Proof of Lemma 14. We prove it via contradiction. Assume the inequality holds but there exists  $t'$  for which  $\lambda_{t'} = 1$ . Without loss of generality, we assume that  $\lambda_t = 0$  for all  $t < t'$ , namely that  $t'$  is the first time that  $\lambda_t$  is 1. Then by definition of  $\lambda_{t'}$ , we have
$$
\sum_ {s = 1} ^ {t ^ {\prime} - 1} Z _ {s} w _ {s} \geq \sqrt {A T / \beta}.
$$
On the other hand, by Lemma 13, we have
$$
\sum_ {s = 1} ^ {t ^ {\prime} - 1} Z _ {s} w _ {s} \leq 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)).
$$
The combination of the above two inequalities contradicts with the conditions.
Proof of Lemma 1. We prove it via contradiction. If no such arm exists, meaning that for any arm  $a$ , there exists an arm  $b$  such that  $f^{\star}(x,a,b) < 0$ . Then we can find a sequence of arms  $(a_{1},a_{2},\ldots ,a_{k})$  such that  $f^{\star}(x,a_i,a_{i + 1}) < 0$  for any  $i = 1,\dots ,k - 1$  and  $f^{\star}(x,a_k,a_1) < 0$ , which contradicts with the transitivity (Assumption 1).
We begin by showing the worst-case regret upper bound.
Lemma 15 (Worst-case regret upper bound). For Algorithm 1, assume  $f^{\star} \in \mathcal{F}_t$  for all  $t \in [T]$ . Then, we have
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} \leq 6 8 \sqrt {A T \beta} \cdot \log (4 \delta^ {- 1})
$$
Proof of Lemma 15. We recall that the regret is defined as
$$
\mathrm {R e g r e t} _ {T} ^ {\mathrm {C B}} = \sum_ {t = 1} ^ {T} \big (f ^ {\star} (x _ {t}, \pi_ {f ^ {\star}} (x _ {t}), a _ {t}) + f ^ {\star} (x _ {t}, \pi_ {f ^ {\star}} (x _ {t}), b _ {t}) \big).
$$
Since  $a_{t}$  and  $b_{t}$  are always drawn independently from the same distribution in Algorithm 1, we only need to consider the regret of the  $a_{t}$  part in the following proof for brevity — multiplying the result by two would yield the overall regret.
We first observe the definition of  $\lambda_{t}$  in Algorithm 1: the left term  $\sum_{s=1}^{t-1} Z_{s} w_{s}$  in the indicator is non-decreasing in  $t$  while the right term remains constant. This means that there exists a particular time step  $t' \in [T]$  dividing the time horizon into two phases:  $\lambda_{t} = 0$  for all  $t \leq t'$  and  $\lambda_{t} = 1$  for all  $t > t'$ . Now, we proceed to examine these two phases individually.
For all rounds before or on  $t'$ , we can compute the expected partial regret as
$$
\sum_ {t = 1} ^ {t ^ {\prime}} \underset {a \sim p _ {t}} {\mathbb {E}} \left[ f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \right] = \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} \underset {a \sim p _ {t}} {\mathbb {E}} \left[ f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \right] \leq \sum_ {t = 1} ^ {t ^ {\prime}} Z _ {t} w _ {t} \leq \sqrt {A T \beta}, \tag {8}
$$
where the equality holds since we have  $\mathcal{A}_t = \{\pi_{f^{\star}}(x_t)\}$  whenever  $Z_{t} = 0$  under the condition that  $f^{\star}\in \mathcal{F}_{t}$ , and thus we don't incur regret in this case. The first inequality is Lemma 8, and the second inequality holds by the definition of  $\lambda_{t}$  and the condition that  $\lambda_{t} = 0$ .
On the other hand, for all rounds after  $t'$ , we have
$$
\begin{array}{l} \sum_ {t = t ^ {\prime} + 1} ^ {T} \underset {a \sim p _ {t}} {\mathbb {E}} \left[ f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \right] \\ = \sum_ {t = t ^ {\prime} + 1} ^ {T} Z _ {t} \underset {a \sim p _ {t}} {\mathbb {E}} \left[ f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \right] \\ \leq \sum_ {t = t ^ {\prime} + 1} ^ {T} Z _ {t} \left(\frac {5 A}{\gamma_ {t}} + \frac {\gamma_ {t}}{4} \underset {a, b \sim p _ {t}} {\mathbb {E}} \left[ \left(f ^ {\star} \left(x _ {t}, a, b\right) - f _ {t} \left(x _ {t}, a, b\right)\right) ^ {2} \right]\right) \\ = \sum_ {t = t ^ {\prime} + 1} ^ {T} Z _ {t} \left(\frac {5 A}{\sqrt {A T / \beta}} + \frac {\sqrt {A T / \beta}}{4} \underset {a, b \sim p _ {t}} {\mathbb {E}} \left[ \left(f ^ {\star} \left(x _ {t}, a, b\right) - f _ {t} \left(x _ {t}, a, b\right)\right) ^ {2} \right]\right) \\ \leq 5 \sqrt {A T \beta} + \frac {\sqrt {A T / \beta}}{4} \sum_ {t = t ^ {\prime} + 1} ^ {T} Z _ {t} \underset {a, b \sim p _ {t}} {\mathbb {E}} \left[ \left(f ^ {\star} (x _ {t}, a, b) - f _ {t} (x _ {t}, a, b)\right) ^ {2} \right] \\ \leq 5 \sqrt {A T \beta} + \frac {\sqrt {A T / \beta}}{2} \sum_ {t = t ^ {\prime} + 1} ^ {T} Z _ {t} \left(f ^ {\star} \left(x _ {t}, a _ {t}, b _ {t}\right) - f _ {t} \left(x _ {t}, a _ {t}, b _ {t}\right)\right) ^ {2} + 8 \sqrt {A T / \beta} \cdot \log \left(4 \delta^ {- 1}\right) \\ \leq 5 \sqrt {A T \beta} + \frac {\sqrt {A T \beta}}{2} + 8 \sqrt {A T / \beta} \cdot \log \left(4 \delta^ {- 1}\right). \tag {9} \\ \end{array}
$$
where the first inequality holds by Lemma 11 (or Lemma 12 for specific function classes), the second equality is by the definition of  $\gamma_{t}$ , the third inequality is by Lemma 4, and the fourth inequality holds by Lemma 9.
Putting the two parts, (8) and (9), together, we arrive at
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {a \sim p _ {t}} \left[ f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \right] \leq 7 \sqrt {A T \beta} + 8 \sqrt {A T / \beta} \cdot \log \left(4 \delta^ {- 1}\right) \leq 1 5 \sqrt {A T \beta} \cdot \log \left(4 \delta^ {- 1}\right).
$$
Now we apply Lemma 4 again. The following holds with probability at least  $1 - \delta /2$
$$
\sum_ {t = 1} ^ {T} f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a _ {t}\right) \leq 2 \sum_ {t = 1} ^ {T} \mathbb {E} _ {a \sim p _ {t}} \left[ f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \right] + 4 \log \left(4 \delta^ {- 1}\right) \leq 3 4 \sqrt {A T \beta} \cdot \log \left(4 \delta^ {- 1}\right).
$$
The above concludes the regret of the  $a_{t}$  part. The regret of the  $b_{t}$  can be shown in the same way. Adding them together, we conclude that
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} = \sum_ {t = 1} ^ {T} \left(f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a _ {t}\right) + f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), b _ {t}\right)\right) \leq 6 8 \sqrt {A T \beta} \cdot \log \left(4 \delta^ {- 1}\right).
$$
Lemma 16 (Instance-dependent regret upper bound). For Algorithm 1, assume  $f^{\star} \in \mathcal{F}_t$  for all  $t \in [T]$ . Then, we have
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} \leq 3 8 0 8 A ^ {2} \beta^ {2} \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log^ {2} (4 / (\delta \Delta))
$$
Proof of Lemma 16. We consider two cases. First, when
$$
5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)) <   \sqrt {A T / \beta}, \tag {10}
$$
we invoke Lemma 14 and get that  $\lambda_t = 0$  for all  $t \in [T]$ . Hence, we have
$$
\begin{array}{l} \operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} = \sum_ {t = 1} ^ {T} \left(f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a _ {t}\right) + f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), b _ {t}\right)\right) \\ \leq 2 \sum_ {t = 1} ^ {T} Z _ {t} w _ {t} \\ \leq 1 1 2 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)) \\ \leq 3 8 0 8 A ^ {2} \beta^ {2} \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log^ {2} (4 / (\delta \Delta)) \\ \end{array}
$$
where the first inequality is by Lemma 8 and the fact that we incur no regret when  $Z_{t} = 0$  since  $f^{\star} \in \mathcal{F}_{t}$ . The second inequality is by Lemma 13.
On the other hand, when the contrary of (10) holds, i.e.,
$$
5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)) \geq \sqrt {A T / \beta}, \tag {11}
$$
applying Lemma 15, we have
$$
\begin{array}{l} \operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} \leq 6 8 \sqrt {A T \beta} \cdot \log (4 \delta^ {- 1}) \\ = 6 8 \beta \cdot \log (4 \delta^ {- 1}) \cdot \sqrt {A T / \beta} \\ \leq 6 8 \beta \cdot \log (4 \delta^ {- 1}) \cdot 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)) \\ \leq 3 8 0 8 A ^ {2} \beta^ {2} \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log^ {2} (4 / (\delta \Delta)) \\ \end{array}
$$
where we apply the condition (11) in the second inequality.
Lemma 17 (Query complexity). For Algorithm 1, assume  $f^{\star} \in \mathcal{F}_t$  for all  $t \in [T]$ . Then, we have
$$
\operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} \leq \min  \left\{T, 3 1 3 6 A ^ {3} \beta^ {3} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \Delta)}{\Delta^ {2}} \cdot \log^ {2} (2 / (\delta \Delta)) \right\}
$$
with probability at least  $1 - \delta$ .
Proof of Lemma 17. We consider two cases. First, when
$$
5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)) <   \sqrt {A T / \beta} \tag {12}
$$
we can invoke Lemma 14 and get that  $\lambda_t = 0$  for all  $t \in [T]$ . Hence,
$$
\begin{array}{l} \operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} = \sum_ {t = 1} ^ {T} Z _ {t} \\ = \sum_ {t = 1} ^ {T} Z _ {t} \mathbb {1} \{w _ {t} \geq \Delta \} \\ = \sum_ {t = 1} ^ {T} Z _ {t} \sup  _ {a, b \in \mathcal {A} _ {t}} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a, b) - f ^ {\prime} (x _ {t}, a, b) \geq \Delta \right\} \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \sum_ {a, b} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t}, a, b) - f ^ {\prime} (x _ {t}, a, b) \geq \Delta \right\} \\ \leq A ^ {2} \underbrace {\sum_ {t = 1} ^ {T} Z _ {t} \underset {a , b \sim p _ {t}} {\mathbb {E}} \mathbb {1} \left\{\sup  _ {f , f ^ {\prime} \in \mathcal {F} _ {t}} f (x _ {t} , a , b) - f ^ {\prime} (x _ {t} , a , b) \geq \Delta \right\}} _ {(*)} \\ \end{array}
$$
where the second equality is by Lemma 7, the second inequality holds as  $p_t(a)$  is uniform for any  $a, b$  when  $\lambda_t = 0$ . We apply Lemma 4 and Lemma 10 to (*) and obtain
$$
\begin{array}{l} (*) \leq 2 \sum_ {t = 1} ^ {T} Z _ {t} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f \left(x _ {t}, a _ {t}, b _ {t}\right) - f ^ {\prime} \left(x _ {t}, a _ {t}, b _ {t}\right) \geq \Delta \right\} + 8 \log \left(\delta^ {- 1}\right) \\ \leq 2 \left(\frac {4 \beta}{\Delta^ {2}} + 1\right) \dim_ {E} (\mathcal {F}; \Delta) + 8 \log (\delta^ {- 1}) \\ \leq \frac {1 0 \beta}{\Delta^ {2}} \cdot \dim_ {E} (\mathcal {F}; \Delta) + 8 \log (\delta^ {- 1}). \\ \end{array}
$$
Plugging this back, we obtain
$$
\begin{array}{l} \operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} \leq \frac {1 0 A ^ {2} \beta}{\Delta^ {2}} \cdot \dim_ {E} (\mathcal {F}; \Delta) + 8 A ^ {2} \log (\delta^ {- 1}) \\ \leq 3 1 3 6 A ^ {3} \beta^ {3} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \Delta)}{\Delta^ {2}} \cdot \log^ {2} (2 / (\delta \Delta)). \\ \end{array}
$$
On the other hand, when the contrary of (12) holds, i.e.,
$$
5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \cdot \log (2 / (\delta \Delta)) \geq \sqrt {A T / \beta}.
$$
Squaring both sides, we obtain
$$
3 1 3 6 A ^ {4} \beta^ {2} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \Delta)}{\Delta^ {2}} \cdot \log^ {2} (2 / (\delta \Delta)) \geq A T / \beta
$$
$$
T \leq 3 1 3 6 A ^ {3} \beta^ {3} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \Delta)}{\Delta^ {2}} \cdot \log^ {2} (2 / (\delta \Delta)).
$$
We note that we always have  $\mathrm{Queries}_T^{\mathrm{CB}} \leq T$ , and thus,
$$
\operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} \leq T \leq 3 1 3 6 A ^ {3} \beta^ {3} \frac {\dim_ {E} ^ {2} (\mathcal {F} , \Delta)}{\Delta^ {2}} \cdot \log^ {2} (2 / (\delta \Delta)).
$$
Hence, we complete the proof.
Having established the aforementioned lemmas, we are now able to advance towards the proof of Theorem 1.
Proof of Theorem 1. By Lemma 9 and the construction of version spaces  $\mathcal{F}_t$  in Algorithm 1, we have  $f^{\star} \in \mathcal{F}_t$  for all  $t \in [T]$  with probability at least  $1 - \delta$ . Then, the rest of the proof follows from Lemmas 15 to 17.
C.4 Proof of Theorem 2
In this section, we will prove the following theorem, which is stronger than Theorem 2.
Theorem 5 (Lower bounds). The following two claims hold:
(1) for any algorithm, there exists an instance that leads to  $\mathrm{Regret}_T^{\mathrm{CB}} = \Omega (\sqrt{AT})$ ;
(2) for any algorithm achieving a worse-case expected regret upper bound in the form of  $\mathbb{E}[\mathrm{Regret}_T^{\mathrm{CB}}] = O(\sqrt{A}\cdot T^{1 - \beta})$  for some  $\beta >0$ , there exists an instance with gap  $\Delta = \sqrt{A}\cdot T^{-\beta}$  that results in  $\mathbb{E}[\mathrm{Regret}_T^{\mathrm{CB}}] = \Omega (A / \Delta) = \Omega (\sqrt{A}\cdot T^{\beta})$  and  $\mathbb{E}[\mathrm{Queries}_T^{\mathrm{CB}}] = \Omega (A / \Delta^2) = \Omega (T^{2\beta})$ .
We observe that Theorem 2 can be considered as a corollary of the above theorem when setting  $\beta = 1 / 2$ .
In what follows, we will first demonstrate lower bounds in the setting of multi-armed bandits (MAB) with active queries and subsequently establish a reduction from it to contextual dueling bandits in
order to achieve these lower bounds. We start by formally defining the setting of MAB with active queries below.
Multi-armed bandits with active queries. We consider a scenario where there exist  $A$  arms. Each arm  $a$  is assumed to yield a binary reward (0 or 1), which is sampled from a Bernoulli distribution  $\mathrm{Bern}(\bar{r}_a)$ , where  $\bar{r}_a$  denotes the mean reward associated with arm  $a$ . The arm with the highest mean reward is denoted by  $a^{\star} \coloneqq \arg \max_{a} \bar{r}_{a}$ . Let  $\Delta_a \coloneqq \bar{r}_{a^{\star}} - \bar{r}_a$  denote the gap of arm  $a \in [A]$ . The interaction proceeds as follows: at each round  $t \in [T]$ , we need to pull an arm but can choose whether to receive the reward signal (denote this choice by  $Z_t$ ). The objective is to minimize two quantities: the regret and the number of queries,
$$
\operatorname {R e g r e t} _ {T} = \sum_ {t = 1} ^ {T} \Delta_ {a _ {t}}, \quad \operatorname {Q u e r i e s} _ {T} = \sum_ {t = 1} ^ {T} Z _ {t}. \tag {13}
$$
Towards the lower bounds, we will start with a bound on the KL divergence over distributions of runs under two different bandits. This result is a variant of standard results which can be found in many bandit literature (e.g., Lattimore and Szepesvári [2020]).
Lemma 18. Let  $I_{1}$  and  $I_{2}$  be two instances of MAB. We define  $p_{1}$  and  $p_{2}$  as their respective distributions over the outcomes of all pulled arms and reward signals when a query is made. Concretely,  $p_{1}$  and  $p_{2}$  are measuring the probability of outcomes (denoted by  $O$ ) in the following form:
$$
O = \left(Z _ {1}, a _ {1}, (r _ {1}), \dots , Z _ {T}, a _ {T}, (r _ {T})\right)
$$
where the reward  $r_t$  is included only when  $Z_t = 1$ , and we added parentheses above to indicate this point. We denote  $\operatorname{Pr}_1$  (resp.  $\operatorname{Pr}_2$ ) as the reward distribution of  $I_1$  (resp.  $I_2$ ). We define  $\bar{n}_a = \sum_{t=1}^{T} Z_t \mathbb{1}\{a_t = a\}$  as the number of times arm  $a$  is pulled when making a query. Then, given any algorithm  $\mathfrak{A}$ , the Kullback-Leibler divergence between  $p_1$  and  $p_2$  can be decomposed in the following way
$$
\operatorname {K L} \left(p _ {1}, p _ {2}\right) = \sum_ {a = 1} ^ {A} \mathbb {E} _ {p _ {1}} [ \bar {n} _ {a} ] \cdot \operatorname {K L} \left(\Pr_ {1} (r \mid a), \Pr_ {2} (r \mid a)\right).
$$