SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Lemma 18. We define the conditional distribution
$$
\overline {{\operatorname {P r}}} _ {1} (r _ {t} \mid Z _ {t}, a _ {t}) \left\{ \begin{array}{l l} \operatorname {P r} _ {1} (r _ {t} \mid a _ {t}) & \text {i f} Z _ {t} = 1 \\ 1 & \text {i f} Z _ {t} = 0 \end{array} , \right.
$$
and similarly for  $\overline{\mathrm{Pr}}_2$ . Additionally, we denote  $\operatorname{Pr}_{\mathfrak{A}}$  as the probability associated with algorithm  $\mathfrak{A}$ . Then, for any outcome  $O$ , we have
$$
p _ {1} (O) = \prod_ {t = 1} ^ {T} \Pr_ {\mathfrak {A}} \left(Z _ {t}, a _ {t} \mid Z _ {1}, a _ {1}, (r _ {1}), \dots , Z _ {t - 1}, a _ {t - 1}, (r _ {t - 1})\right) \overline {{\operatorname* {P r}}} _ {1} (r _ {t} \mid Z _ {t}, a _ {t}),
$$
and we can write  $p_2(O)$  in a similar manner. Hence,
$$
\begin{array}{l} \mathrm {K L} (p _ {1}, p _ {2}) = \underset {O \sim p _ {1}} {\mathbb {E}} \left[ \log \left(\frac {\prod_ {t = 1} ^ {T} \operatorname * {P r} _ {\mathfrak {A}} \left(Z _ {t} , a _ {t} \mid Z _ {1} , a _ {1} , (r _ {1}) , \dots , Z _ {t - 1} , a _ {t - 1} , (r _ {t - 1})\right) \overline {{\operatorname* {P r}}} _ {1} (r _ {t} \mid Z _ {t} , a _ {t})}{\prod_ {t = 1} ^ {T} \operatorname * {P r} _ {\mathfrak {A}} \left(Z _ {t} , a _ {t} \mid Z _ {1} , a _ {1} , (r _ {1}) , \dots , Z _ {t - 1} , a _ {t - 1} , (r _ {t - 1})\right) \overline {{| \operatorname* {P r}}} _ {2} (r _ {t} \mid Z _ {t} , a _ {t})}\right) \right] \\ = \underset {O \sim p _ {1}} {\mathbb {E}} \left[ \sum_ {t = 1} ^ {T} \log \left(\frac {\overline {{\operatorname* {P r}}} _ {1} \left(r _ {t} \mid Z _ {t} , a _ {t}\right)}{\overline {{\operatorname* {P r}}} _ {2} \left(r _ {t} \mid Z _ {t} , a _ {t}\right)}\right) \right] \\ = \underset {O \sim p _ {1}} {\mathbb {E}} \left[ \sum_ {t = 1} ^ {T} Z _ {t} \log \left(\frac {\Pr_ {1} \left(r _ {t} \mid a _ {t}\right)}{\Pr_ {2} \left(r _ {t} \mid a _ {t}\right)}\right) \right] \\ = \underset {O \sim p _ {1}} {\mathbb {E}} \left[ \sum_ {t = 1} ^ {T} Z _ {t} \underset {r _ {t} \sim \Pr_ {1} (\cdot \mid a _ {t})} {\mathbb {E}} \left[ \log \left(\frac {\Pr_ {1} (r _ {t} \mid a _ {t})}{\Pr_ {2} (r _ {t} \mid a _ {t})}\right) \right] \right] \\ = \underset {O \sim p _ {1}} {\mathbb {E}} \left[ \sum_ {t = 1} ^ {T} Z _ {t} \cdot \operatorname {K L} \left(\Pr_ {1} (\cdot | a _ {t}), \Pr_ {2} (\cdot | a _ {t})\right) \right] \\ = \sum_ {a = 1} ^ {A} \underset {\mathcal {O} \sim p _ {1}} {\mathbb {E}} \left[ \bar {n} _ {a} \right] \cdot \operatorname {K L} \left(\Pr_ {1} (\cdot \mid a _ {t}), \Pr_ {2} (\cdot \mid a _ {t})\right) \\ \end{array}
$$
where the third equality holds by the definition of  $\overline{\mathrm{Pr}}_1$  and  $\overline{\mathrm{Pr}}_2$ .
The following lemma establishes lower bounds for MAB with active queries. It presents a trade-off between the regret and the number of queries.
Lemma 19. Let  $\mathcal{I}$  denote the set of all MAB instances. Assume ALG is an algorithm that achieves the following worst-case regret upper bound for some  $C$  and  $\beta$ :
$$
\mathbb {E} \left[ \operatorname {R e g r e t} _ {T} \right] \leq C T ^ {1 - \beta},
$$
for all  $I \in \mathcal{I}$ . Then, for any MAB instance  $I \in \mathcal{I}$ , the regret and the number of queries made by algorithm ALG are lower bounded:
$$
\mathbb {E} \left[ \mathrm {R e g r e t} _ {T} \right] \geq \sum_ {a \neq a ^ {\star}} \frac {\zeta}{\Delta_ {a}} \log \left(\frac {\Delta_ {a}}{4 C T ^ {- \beta}}\right), \quad \mathbb {E} \left[ \mathrm {Q u e r i e s} _ {T} \right] \geq \sum_ {a \neq a ^ {\star}} \frac {\zeta}{\Delta_ {a} ^ {2}} \log \left(\frac {\Delta_ {a}}{4 C T ^ {- \beta}}\right)
$$
where the coefficient  $\zeta = \min_{a}\min \{\bar{r}_{a},1 - \bar{r}_{a}\}$  depends on the instance  $I$ .
Proof of Lemma 19. For any MAB instance  $I$  and any arm  $a^\dagger$ , we define a corresponding MAB instance  $I'$  as follows. Denote  $\bar{r}$  and  $\bar{r}'$  as the mean reward of  $I$  and  $I'$ , respectively. For  $I'$ , we set the mean reward  $\bar{r}_a' = \bar{r}_a$  for any  $a \neq a^\dagger$  and  $\bar{r}_{a\dagger}' = \bar{r}_{a\dagger} + 2\Delta_{a\dagger}$ . Consequently, the optimal arm of  $I'$  is  $a^\dagger$  with margin  $\Delta_{a\dagger}$ . Let  $n_a$  denote the number of times that arm  $a$  is pulled. We define the event
$$
E = \{n _ {a ^ {\dagger}} > T / 2 \}.
$$
$$
\underset {p} {\mathbb {E}} \left[ \mathrm {R e g r e t} _ {T} \right] \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \cdot p (E), \quad \underset {p ^ {\prime}} {\mathbb {E}} \left[ \mathrm {R e g r e t} _ {T} \right] \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \cdot p ^ {\prime} (E ^ {\complement}).
$$
$$
\begin{array}{l} 2 C T ^ {1 - \beta} \geq \underset {p} {\mathbb {E}} \left[ \operatorname {R e g r e t} _ {T} \right] + \underset {p ^ {\prime}} {\mathbb {E}} \left[ \operatorname {R e g r e t} _ {T} \right] \\ \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} (p (E) + p ^ {\prime} (E ^ {\mathbb {C}})) \\ = \frac {T \Delta_ {a ^ {\dagger}}}{2} \left(1 - \left(p ^ {\prime} (E) - p (E)\right)\right) \\ \geq \frac {T \Delta_ {a ^ {\uparrow}}}{2} \left(1 - \operatorname {T V} \left(p, p ^ {\prime}\right)\right) \\ \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \left(1 - \sqrt {1 - \exp \left(- \mathrm {K L} (p , p ^ {\prime})\right)}\right) \\ \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \exp \left(- \frac {1}{2} \cdot \mathrm {K L} (p, p ^ {\prime})\right). \\ \end{array}
$$
By Lemma 18, we have
$$
\begin{array}{l} \operatorname {K L} \left(p, p ^ {\prime}\right) = \sum_ {a = 1} ^ {A} \mathbb {E} _ {p} [ \bar {n} _ {a} ] \cdot \operatorname {K L} \left(\Pr (r \mid a), \Pr^ {\prime} (r \mid a)\right) \\ = \mathbb {E} _ {p} [ \bar {n} _ {a ^ {\dagger}} ] \cdot \operatorname {K L} \left(\Pr (r \mid a ^ {\dagger}), \Pr^ {\prime} (r \mid a ^ {\dagger})\right) \\ \leq \underset {p} {\mathbb {E}} [ \bar {n} _ {a ^ {\dagger}} ] \cdot \Delta_ {a ^ {\dagger}} ^ {2} \cdot 2 / \zeta \\ \end{array}
$$
where the last inequality is by Lemma 6. Putting the above two inequalities together, we arrive at
$$
\underset {p} {\mathbb {E}} [ \bar {n} _ {a ^ {\dagger}} ] \geq \frac {\zeta}{\Delta_ {a ^ {\dagger}} ^ {2}} \log \left(\frac {\Delta_ {a ^ {\dagger}}}{4 C T ^ {- \beta}}\right).
$$
This establishes a query lower bound for arm  $a^\dagger$ . Consequently, we have
$$
\mathbb {E} [ \operatorname {R e g r e t} _ {T} ] \geq \sum_ {a \neq a ^ {\star}} \mathbb {E} _ {p} [ \bar {n} _ {a} ] \cdot \Delta_ {a} \geq \sum_ {a \neq a ^ {\star}} \frac {\zeta}{\Delta_ {a}} \log \left(\frac {\Delta_ {a}}{4 C T ^ {- \beta}}\right),
$$
and similarly,
$$
\mathbb {E} [ \mathrm {Q u e r i e s} _ {T} ] \geq \sum_ {a \neq a ^ {\star}} \mathbb {E} _ {p} [ \bar {n} _ {a} ] \geq \sum_ {a \neq a ^ {\star}} \frac {\zeta}{\Delta_ {a} ^ {2}} \log \left(\frac {\Delta_ {a}}{4 C T ^ {- \beta}}\right).
$$
Now we can proceed with the proof of Theorem 5.
Proof of Theorem 5. We provide a reduction from the multi-armed bandits with active queries to the contextual dueling bandits. Our desired lower bound for the contextual dueling bandit setting thus follows from the above lower bound for Multi-Armed Bandits (MABs). Let ALG denote any algorithm for contextual dueling bandits.
Reduction. Since we focus on the multi-armed bandit where no context is involved, we just ignore the notation of context everywhere for brevity. We will start from an MAB instance, and then simulate a binary feedback and feed it to a dueling bandit algorithm ALG which is used to solve the original MAB instance. Particularly, consider the MAB instance with A-many actions each with an expected reward denoted as  $\bar{r}_a$ .
At the beginning of iteration  $t$  in the MAB instance, the learner calls the dueling algorithm ALG to generate two actions  $a_{t}$  and  $b_{t}$ . The learner plays  $a_{t}$  at iteration  $t$  to receive a reward  $y_{a_t}$ ; the learner then moves to iteration  $t + 1$  to play  $b_{t}$ , and receives reward  $y_{b_t}$ . At the end of iteration  $t + 1$ , the learner simulates a binary feedback by setting  $o = 1$  if  $y_{a_t} > y_{b_t}$ ;  $o = -1$  if  $y_{a_t} < y_{b_t}$ ;  $o$  being 1 or -1 uniform randomly if  $y_{a_t} = y_{b_t}$ . Then, the learner sends  $(a_{t}, b_{t}, o)$  to the dueling algorithm ALG to query for two actions which will be played at iterations  $t + 2$  and  $t + 3$ , respectively.
From the dueling algorithm ALG's perspective, given two actions  $a$  and  $b$ , we can verify that the probability of seeing label 1 is  $(\bar{r}_a - \bar{r}_b + 1)/2$ . So we can just specify the link function to be  $\phi(d) = (d + 1)/2$ . As we verified earlier, the corresponding  $\Phi$  is strongly convex (Example 2). Moreover, since  $f^{\star}(a,b) = \bar{r}_a - \bar{r}_b$ , if we define the gap of the MAB instance as  $\bar{\Delta} := \min_{a \neq a^{\star}} (\bar{r}_{a^{\star}} - \bar{r}_a)$  where  $a^{\star} := \arg \max_{i} \bar{r}_i$ , then we have  $\bar{\Delta} = \Delta$  in this reduction where  $\Delta$  is the definition of the gap in the dueling setting. We further note that the regret of the MAB instance is
$$
\sum_ {t = 1} ^ {T} \left(\bar {r} _ {a ^ {\star}} - \bar {r} _ {a _ {t}}\right) + \sum_ {t = 1} ^ {T} \left(\bar {r} _ {a ^ {\star}} - \bar {r} _ {b _ {t}}\right),
$$
which, by our definition of  $f^{\star}$ , is equivalent to the preference-based regret that occurred to the dueling algorithm ALG. The number of queries is clearly equivalent as well. Thus, the regret and the query complexity of the dueling algorithm ALG can be directly translated to the regret and the query complexity of the MAB instance.
Now, we are ready to prove the two claims in our statement.
Proof of the first claim. We refer the reader to Lattimore and Szepesvári [2020, Theorem 15.2] for a proof of the minimax regret lower bound of  $\Omega(\sqrt{AT})$  for the MAB. Through the reduction outlined above, that lower bound naturally extends to the dueling bandits setting, yielding  $\mathrm{Regret}_T^{\mathrm{CB}} \geq \Omega(\sqrt{AT})$  (otherwise, via the above reduction, we would have achieved an approach that breaks the lower bound of MAB).
Proof of the second claim. We choose an arbitrary MAB for which  $\zeta = \min_{a}\min \{\bar{r}_{a},1 - \bar{r}_{a}\} >0.2$  and the gaps of all arms are equal to  $\Delta$ . Invoking Lemma 19, we have
$$
\mathbb {E} \left[ \operatorname {R e g r e t} _ {T} \right] \geq \frac {0 . 2 (A - 1)}{\Delta} \log \left(\frac {\Delta}{4 C T ^ {- \beta}}\right) \geq \Omega \left(\frac {A}{\Delta}\right),
$$
$$
\mathbb {E} \left[ \text {Q u e r i e s} _ {T} \right] \geq \frac {0 . 2 (A - 1)}{\Delta^ {2}} \log \left(\frac {\Delta}{4 C T ^ {- \beta}}\right) \geq \Omega \left(\frac {A}{\Delta^ {2}}\right).
$$
We further choose  $\Delta = 40CT^{-\beta}$  and  $C = \sqrt{A}$ , leading to
$$
\mathbb {E} \left[ \operatorname {R e g r e t} _ {T} \right] \geq \frac {0 . 2 (A - 1)}{4 0 \sqrt {A}} \cdot T ^ {\beta} = \Omega (\sqrt {A} \cdot T ^ {\beta}),
$$
$$
\mathbb {E} \left[ \operatorname {Q u e r i e s} _ {T} \right] \geq \frac {0 . 2 (A - 1)}{1 6 0 0 A} \cdot T ^ {2 \beta} = \Omega \left(T ^ {2 \beta}\right).
$$
Via the reduction we have shown above, these lower bounds naturally extend to the contextual dueling bandit setting, thereby completing the proof.
C.4.1 Alternative Lower Bounds Conditioning on the Limit of Regret
In this section, we establish an analogue of Theorem 5 but under a different condition. We first introduce the concept of diminishing regret.
Definition 4. We say that an algorithm guarantees a diminishing regret if for all contextual dueling bandit instances and  $p > 0$ , it holds that
$$
\lim _ {T \to \infty} \frac {\mathbb {E} [ \mathrm {R e g r e t} _ {T} ^ {\mathrm {C B}} ]}{T ^ {p}} = 0.
$$
The lower bounds under the assumption of diminishing regret guarantees are stated as follows.
Theorem 6 (Lower bounds). The following two claims hold:
(1) for any algorithm, there exists an instance that leads to  $\mathrm{Regret}_T^{\mathrm{CB}}\geq \Omega (\sqrt{AT})$
(2) for any gap  $\Delta$  and any algorithm achieving diminishing regret, there exists an instance with gap  $\Delta$  that results in  $\mathbb{E}[\mathrm{Regret}_T^{\mathrm{CB}}] \geq \Omega(A / \Delta)$  and  $\mathbb{E}[\mathrm{Queries}_T^{\mathrm{CB}}] \geq \Omega(A / \Delta^2)$  for sufficiently large  $T$ .
We should highlight that the condition of diminishing regret (Theorem 6) and the worst-case regret upper bounds (Theorems 2 and 5) are not comparable in general. However, Theorem 6 is also applicable to our algorithm (Algorithm 1) since our algorithm possesses an instance-dependent regret upper bound that is clearly diminishing.
To prove Theorem 6, we first show the following lemma, which is a variant of Lemma 19.
Lemma 20. Let  $\mathcal{I}$  denote the set of all MAB instances. Assume ALG is an algorithm that achieves diminishing regret for all MAB instances in  $\mathcal{I}$ , i.e., for any  $I \in \mathcal{I}$  and  $p > 0$ , it holds that
$$
\lim  _ {T \to \infty} \frac {\mathbb {E} [ \mathrm {R e g r e t} _ {T} ]}{T ^ {p}} = 0.
$$
Then, for any MAB instance  $I \in \mathcal{I}$ , the regret and the number of queries made by algorithm ALG are lower bounded in the following manner:
$$
\liminf_{T\to \infty}\frac{\mathbb{E}\left[ \mathrm{Regret}_{T} \right]}{\log T}\geq \sum_{a\neq a^{\star}}\frac{\zeta}{\Delta_{a}},\quad \liminf_{T\to \infty}\frac{\mathbb{E}\left[ \mathrm{Queries}_{T} \right]}{\log T}\geq \sum_{a\neq a^{\star}}\frac{\zeta}{\Delta_{a}^{2}}
$$
where the coefficient  $\zeta \coloneqq \min_{a}\min \{\bar{r}_{a},1 - \bar{r}_{a}\}$  depends on the instance  $I$ . Recall that  $\mathrm{Regret}_T$  and  $\mathrm{Queries}_T$  are defined in (13).
Proof of Lemma 20. The proof is similar to Lemma 19. For any MAB instance  $I \in \mathcal{I}$  and any arm  $a^\dagger$ , we define a corresponding MAB instance  $I'$  as follows. Denote  $\bar{r}$  and  $\bar{r}'$  as the mean reward of  $I$  and  $I'$ , respectively. For  $I'$ , we set the mean reward  $\bar{r}_a' = \bar{r}_a$  for any  $a \neq a^\dagger$  and  $\bar{r}_{a^\dagger}' = \bar{r}_{a^\dagger} + 2\Delta_{a^\dagger}$ . Consequently, the optimal arm of  $I'$  is  $a^\dagger$  with margin  $\Delta_{a^\dagger}$ . Let  $n_a$  denote the number of times that arm  $a$  is pulled. We define the event
Let  $p$  and  $p'$  denote the probability of  $I$  and  $I'$ , respectively. Then, we have
$$
\underset {p} {\mathbb {E}} \left[ \operatorname {R e g r e t} _ {T} \right] \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \cdot p (E), \quad \underset {p ^ {\prime}} {\mathbb {E}} \left[ \operatorname {R e g r e t} _ {T} \right] \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \cdot p ^ {\prime} (E ^ {\complement})
$$
where  $E^{\mathbb{C}}$  means the complement of event  $E$ . Hence,
$$
\begin{array}{l} \underset {p} {\mathbb {E}} \left[ \mathrm {R e g r e t} _ {T} \right] + \underset {p ^ {\prime}} {\mathbb {E}} \left[ \mathrm {R e g r e t} _ {T} \right] \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \big (p (E) + p ^ {\prime} (E ^ {\complement}) \big) \\ = \frac {T \Delta_ {a ^ {\dagger}}}{2} \left(1 - \left(p ^ {\prime} (E) - p (E)\right)\right) \\ \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \left(1 - \operatorname {T V} \left(p, p ^ {\prime}\right)\right) \\ \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \left(1 - \sqrt {1 - \exp \left(- \mathrm {K L} (p , p ^ {\prime})\right)}\right) \\ \geq \frac {T \Delta_ {a ^ {\dagger}}}{2} \exp \left(- \frac {1}{2} \cdot \mathrm {K L} (p, p ^ {\prime})\right). \\ \end{array}
$$
Here TV denotes the total variation distance. By Lemma 18, we have
$$
\begin{array}{l} \operatorname {K L} (p, p ^ {\prime}) = \sum_ {a = 1} ^ {A} \mathbb {E} _ {p} [ \bar {n} _ {a} ] \cdot \operatorname {K L} \left(\Pr (r \mid a), \Pr^ {\prime} (r \mid a)\right) \\ = \underset {p} {\mathbb {E}} [ \bar {n} _ {a ^ {\dagger}} ] \cdot \operatorname {K L} \left(\Pr (r \mid a ^ {\dagger}), \Pr^ {\prime} (r \mid a ^ {\dagger})\right) \\ \leq \mathbb {E} _ {p} [ \bar {n} _ {a ^ {\dagger}} ] \cdot \Delta_ {a ^ {\dagger}} ^ {2} \cdot 2 / \zeta \\ \end{array}
$$
where the last inequality is by Lemma 6. Putting it all together, we arrive at
$$
\underset {p} {\mathbb {E}} [ \bar {n} _ {a ^ {\dagger}} ] \geq \frac {\zeta}{\Delta_ {a ^ {\dagger}} ^ {2}} \log \left(\frac {T \Delta_ {a ^ {\dagger}}}{2 \Big (\mathbb {E} _ {p} [ \mathrm {R e g r e t} _ {T} ] + \mathbb {E} _ {p ^ {\prime}} [ \mathrm {R e g r e t} _ {T} ] \Big)}\right).
$$
Taking the limit on both sides yields
$$
\begin{array}{l} \liminf_{T\to \infty}\frac{\mathbb{E}_{p}[\bar{n}_{a^{\dagger}}]}{\log T}\geq \liminf_{T\to \infty}\frac{\zeta}{\Delta_{a^{\dagger}}^{2}}\cdot \frac{\log\left(\frac{T\Delta_{a^{\dagger}}}{2\left(\mathbb{E}_{p}\left[ \mathrm{Regret}_{T}\right] + \mathbb{E}_{p^{\prime}}\left[ \mathrm{Regret}_{T}\right]\right)}\right)}{\log T} \\ = \lim _ {T \to \infty} \inf _ {\Delta_ {a ^ {\dagger}} ^ {2}} \frac {\zeta}{\left. \Delta_ {a ^ {\dagger}} ^ {2} \right.} \cdot \left(1 + \underbrace {\frac {\log (\Delta_ {a ^ {\dagger}} / 2)}{\log T}} _ {\mathrm {(i)}} - \underbrace {\frac {\log \left(\mathbb {E} _ {p} \left[ \mathrm {R e g r e t} _ {T} \right] + \mathbb {E} _ {p ^ {\prime}} \left[ \mathrm {R e g r e t} _ {T} \right]\right)}{\log T}} _ {\mathrm {(i i)}}\right). \\ \end{array}
$$
Here the limit of (i) is clearly 0. For the limit of (ii), we note that by the definition of diminishing regret, for any  $C > 0$ , there exists a  $T'$  such that  $\mathbb{E}[\mathrm{Regret}_T] / T^p \leq C$  for any  $T > T'$ . This implies
$$
\frac {\log \left(\mathbb {E} _ {p} \left[ \operatorname {R e g r e t} _ {T} \right] + \mathbb {E} _ {p ^ {\prime}} \left[ \operatorname {R e g r e t} _ {T} \right]\right)}{\log T} \leq \frac {\log \left(2 C T ^ {p}\right)}{\log T} = \frac {\log (2 C)}{\log T} + p
$$
for any  $p > 0$ . Therefore, the limit of (ii) is also 0. Plugging these back, we obtain
$$
\liminf_{T\to \infty}\frac{\mathbb{E}_{p}[\bar{n}_{a^{\dagger}}]}{\log T}\geq \frac{\zeta}{\Delta_{a^{\dagger}}^{2}}.
$$
$$
\operatorname * {l i m i n f} _ {T \to \infty} \frac {\mathbb {E} [ \mathrm {R e g r e t} _ {T} ]}{\log T} \geq \operatorname * {l i m i n f} _ {T \to \infty} \sum_ {a \neq a ^ {\star}} \frac {\mathbb {E} _ {p} [ \bar {n} _ {a} ] \cdot \Delta_ {a}}{\log T} \geq \sum_ {a \neq a ^ {\star}} \frac {\zeta}{\Delta_ {a}},
$$
$$
\liminf_{T\to \infty}\frac{\mathbb{E}[\mathrm{Queries}_{T}]}{\log T}\geq \liminf_{T\to \infty}\sum_{a\neq a^{\star}}\frac{\mathbb{E}_{p}[\bar{n}_{a}]}{\log T}\geq \sum_{a\neq a^{\star}}\frac{\zeta}{\Delta_{a}^{2}}.
$$
Now, we proceed with the proof of Theorem 6.
Proof of Theorem 6. The proof of the first claim is the same as Theorem 5, so we will omit it here. Let us now focus on the proof of the second claim. By Lemma 20, for any algorithm achieving diminishing regret, the following is true for any MAB instance:
$$
\lim _ {T \to \infty} \inf _ {\zeta \to 0} \frac {\mathbb {E} [ \mathrm {R e g r e t} _ {T} ]}{\log T} \geq \sum_ {a \neq a ^ {\star}} \frac {\zeta}{\Delta_ {a}}, \quad \lim _ {T \to \infty} \inf _ {\zeta \to 0} \frac {\mathbb {E} [ \mathrm {Q u e r i e s} _ {T} ]}{\log T} \geq \sum_ {a \neq a ^ {\star}} \frac {\zeta}{\Delta_ {a} ^ {2}}.
$$
We choose an arbitrary MAB for which  $\zeta \geq 0.2$  and the gaps of all suboptimal arms are equal to  $\Delta$ . Then, for this instance, we have
$$
\liminf _ {T \to \infty} \frac {\mathbb {E} \left[ \mathrm {R e g r e t} _ {T} \right]}{\log T} \geq \frac {0 . 2 (A - 1)}{\Delta}, \liminf _ {T \to \infty} \frac {\mathbb {E} \left[ \mathrm {Q u e r i e s} _ {T} \right]}{\log T} \geq \frac {0 . 2 (A - 1)}{\Delta^ {2}}.
$$
By the definition of limit, when  $T$  is large enough (exceeding a certain threshold), we have
$$
\frac {\mathbb {E} \left[ \operatorname {R e g r e t} _ {T} \right]}{\log T} \geq \frac {0 . 1 (A - 1)}{\Delta}, \quad \frac {\mathbb {E} \left[ \operatorname {Q u e r i e s} _ {T} \right]}{\log T} \geq \frac {0 . 1 (A - 1)}{\Delta^ {2}}.
$$
Via the reduction we have shown in the proof of Theorem 5, these lower bounds naturally extend to the contextual dueling bandit setting, thereby completing the proof.
C.5 Proof of Theorem 3
Proof of Theorem 3. We establish the bounds for regret and the number of queries, consecutively. First, we set an arbitrary gap threshold  $\epsilon >0$ . Since our algorithm is independent of  $\epsilon$ , we can later choose any  $\epsilon$  that minimizes the upper bounds.
Proof of regret. We start with the regret upper bound. By definition, we have
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} = \sum_ {t = 1} ^ {T} \left(f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a _ {t}\right) + f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), b _ {t}\right)\right).
$$
The worst-case regret upper bound presented in Lemma 15 doesn't reply on the gap assumption and thus remains applicable in this setting. Hence, we only need to prove the instance-dependent regret upper bound. To that end, we first need an analogue of Lemma 14.
Lemma 21. Fix any  $\epsilon >0$ . Whenever
$$
2 T _ {\epsilon} + 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon)) <   \sqrt {A T / \beta},
$$
Proof of Lemma 21. The proof is similar to Lemma 14 and is via contradiction. Assume the inequality holds but there exists  $t'$  for which  $\lambda_{t'} = 1$ . Without loss of generality, we assume that  $\lambda_t = 0$  for all  $t < t'$ , namely that  $t'$  is the first time that  $\lambda_t$  is 1. Then by definition of  $\lambda_{t'}$ , we have
$$
\begin{array}{l} \sum_ {s = 1} ^ {t ^ {\prime} - 1} Z _ {s} w _ {s} = \sum_ {s = 1} ^ {t ^ {\prime} - 1} \mathbb {1} \left\{\operatorname {G a p} \left(x _ {t}\right) \leq \epsilon \right\} Z _ {s} w _ {s} + \sum_ {s = 1} ^ {t ^ {\prime} - 1} \mathbb {1} \left\{\operatorname {G a p} \left(x _ {t}\right) > \epsilon \right\} Z _ {s} w _ {s} \\ \leq 2 T _ {\epsilon} + 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon)) \\ \end{array}
$$
where the inequality is by Lemma 13. The above two inequalities contradict with the conditions.
Towards an instance-dependent regret upper bound, we adapt the proof of Lemma 16 to this setting. We consider two cases. First, when
$$
2 T _ {\epsilon} + 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon)) <   \sqrt {A T / \beta}, \tag {14}
$$
we invoke Lemma 21 and get that  $\lambda_t = 0$  for all  $t \in [T]$ . Hence, we have
$$
\begin{array}{l} \operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} = \sum_ {t = 1} ^ {T} \left(f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a _ {t}\right) + f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), b _ {t}\right)\right) \\ \leq 2 \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\operatorname {G a p} \left(x _ {t}\right) \leq \epsilon \right\} Z _ {t} w _ {t} + 2 \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\operatorname {G a p} \left(x _ {t}\right) > \epsilon \right\} Z _ {t} w _ {t} \\ \leq 4 T _ {\epsilon} + 1 1 2 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon)) \\ \leq 1 3 6 \beta \cdot \log (4 \delta^ {- 1}) \cdot T _ {\epsilon} + 3 8 0 8 A ^ {2} \beta^ {2} \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log^ {2} (4 / (\delta \epsilon)) \\ \end{array}
$$
where the first inequality is by Lemma 8 and the fact that we incur no regret when  $Z_{t} = 0$  since  $f^{\star}\in \mathcal{F}_{t}$ . The second inequality is by Lemma 13.
On the other hand, when the contrary of (14) holds, i.e.,
$$
2 T _ {\epsilon} + 5 6 A ^ {2} \beta \cdot \frac {\dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \cdot \log (2 / (\delta \epsilon)) \geq \sqrt {A T / \beta}, \tag {15}
$$