SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$^5$ Policy  $\pi_t$  consists of  $H$  time-dependent policies  $\pi_{t,1},\ldots ,\pi_{t,H}$ , where each  $\pi_{t,h}$  is defined implicitly via AURORA $_h$ , i.e.,  $\pi_{t,h}$  generates action as follows: given  $x_{t,h}$ , AURORA $_h$  recommends  $a_{t,h},b_{t,h}$ , followed by uniformly sampling an action from  $\{a_{t,h},b_{t,h}\}$ .
AS acknowledges support from the Simons Foundation and NSF through award DMS-2031883, as well as from the DOE through award DE-SC0022199. KS acknowledges support from NSF CAREER Award 1750575, and LinkedIn-Cornell grant.
Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
Alekh Agarwal. Selective sampling algorithms for cost-sensitive multiclass prediction. In International Conference on Machine Learning, pages 1220-1228. PMLR, 2013.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In International Conference on Machine Learning, pages 856-864. PMLR, 2014.
Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In  $COLT$ , pages 41-53, 2010.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning, pages 463-474. PMLR, 2020.
Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey. The Journal of Machine Learning Research, 22(1): 278-385, 2021.
Viktor Bengs, Aadirupa Saha, and Eyke Hüllermeier. Stochastic contextual dueling bandits under linear stochastic transitivity models. In International Conference on Machine Learning, pages 1764-1786. PMLR, 2022.
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference on robot learning, pages 519-528. PMLR, 2018.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning, pages 330-359. PMLR, 2020.
Róbert Busa-Fekete, Balázs Szörenyi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine learning, 97:327-351, 2014.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Nicolo Cesa-Bianchi, Gabor Lugosi, and Gilles Stoltz. Minimizing regret with label efficient prediction. IEEE Transactions on Information Theory, 51(6):2152-2162, 2005.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd national conference on Artificial intelligence-Volume 3, pages 1433-1438, 2008.
Maspour Zoghi, Shimon Whiteson, Remi Munos, and Maarten Rijke. Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning, pages 10-18. PMLR, 2014.
A Computational tractability of Algorithm 1
We observe that the computational complexity of the proposed algorithm mainly depends on the computation of the candidate arm set (Line 3) and the width (Line 6). When  $\mathcal{F}$  is a  $d$ -dimensional linear function class, the computational complexity can be  $\tilde{O}(dTA)$  since the version space exhibits an ellipsoid structure and thus both the candidate arm and the width can be computed in  $\tilde{O}(dA)$  time. When  $\mathcal{F}$  is tabular, it can be considered as a special case of linear class with one-hot encoding. In this case, we have  $d = S \times A$ , resulting in a computational complexity of  $\tilde{O}(TSA^2)$ .
For a more general convex function class  $\mathcal{F}$ , we can design an efficient algorithm based on a weighted regression oracle for  $\mathcal{F}$ . To this end, we first note that an approach to efficiently compute the width has been proposed by Foster et al. [2018a], and now we propose the following method to compute the candidate arm set. An arm  $a$  belongs to the candidate arm set at round  $t$  if and only if the following optimization problem (with an constant objective) is feasible
$$
\begin{array}{l} \min  _ {f \in \mathcal {F}, \xi \in \mathbb {R} ^ {A}} 1 \quad \text {s . t .} \quad f (x, a, a ^ {\prime}) = \xi_ {a ^ {\prime}}  , \quad \xi_ {a ^ {\prime}} > 0 \quad (\forall a ^ {\prime} \neq a), \\ \text {a n d} \quad \sum_ {s = 1} ^ {t - 1} Z _ {s} \left(f \left(x _ {s}, a _ {s}, b _ {s}\right) - f _ {t} \left(x _ {s}, a _ {s}, b _ {s}\right)\right) ^ {2} \leq \beta . \\ \end{array}
$$
Here we introduce the slack variable  $\xi$  so that the optimization part for  $f$  can be simply reduced to a weighted regression oracle. Next, we convert the above into Lagrangian formulation and obtain
$$
\begin{array}{l} \min _ {f \in \mathcal {F}, \xi \in \mathbb {R} ^ {A}} \max _ {\alpha \in \mathbb {R} _ {+} ^ {A}, \gamma \in \mathbb {R} _ {+} ^ {A}, \lambda \in \mathbb {R} _ {+}} 1 + \sum_ {a ^ {\prime} \neq a} \alpha_ {a ^ {\prime}} \left(f (x, a, a ^ {\prime}) - \xi_ {a ^ {\prime}}\right) ^ {2} - \sum_ {a ^ {\prime} \neq a} \gamma_ {a ^ {\prime}} \xi_ {a ^ {\prime}} \\ + \lambda \left(\sum_ {s = 1} ^ {t - 1} Z _ {s} \left(f \left(x _ {s}, a _ {s}, b _ {s}\right) - f _ {t} \left(x _ {s}, a _ {s}, b _ {s}\right)\right) ^ {2} - \beta\right) \\ = \max _ {\alpha \in \mathbb {R} _ {+} ^ {A}, \gamma \in \mathbb {R} _ {+} ^ {A}, \lambda \in \mathbb {R} _ {+}} \min _ {f \in \mathcal {F}, \xi \in \mathbb {R} ^ {A}} 1 + \sum_ {a ^ {\prime} \neq a} \alpha_ {a ^ {\prime}} \left(f (x, a, a ^ {\prime}) - \xi_ {a ^ {\prime}}\right) ^ {2} - \sum_ {a ^ {\prime} \neq a} \gamma_ {a ^ {\prime}} \xi_ {a ^ {\prime}} \\ + \lambda \left(\sum_ {s = 1} ^ {t - 1} Z _ {s} \left(f \left(x _ {s}, a _ {s}, b _ {s}\right) - f _ {t} \left(x _ {s}, a _ {s}, b _ {s}\right)\right) ^ {2} - \beta\right). \\ \end{array}
$$
Here we can swap the min and max since the objective is convex in the joint space of  $f$  and  $\xi$ . Then, the inner minimization problem can be solved by updating  $f$  via the regression oracle and updating  $\xi$  via gradient descent; for the outer maximization problem, we can do projected gradient ascent.
B Preliminaries
Lemma 2 (Kakade and Tewari [2008, Lemma 3]). Suppose  $X_{1},\ldots ,X_{T}$  is a martingale difference sequence with  $|X_{t}|\leq b$ . Let
$$
\operatorname {V a r} _ {t} X _ {t} = \operatorname {V a r} \left(X _ {t} \mid X _ {1}, \dots , X _ {t - 1}\right)
$$
Let  $V = \sum_{t=1}^{T} \operatorname{Var}_t X_t$  be the sum of conditional variances of  $X_t$ 's. Further, let  $\sigma = \sqrt{V}$ . Then we have, for any  $\delta < 1/e$  and  $T \geq 3$ ,
$$
\Pr \left(\sum_ {t = 1} ^ {T} X _ {t} > \max  \{2 \sigma , 3 b \sqrt {\ln (1 / \delta)} \} \sqrt {\ln (1 / \delta)}\right) \leq 4 \ln (T) \delta .
$$
Lemma 3 (Foster and Rakhlin [2020, Lemma 3]). For any vector  $\hat{y} \in [0,1]^A$ , if we define  $p$  to be
$$
p (a) = \left\{ \begin{array}{l l} \frac {1}{A + \gamma (\hat {y} (\hat {a}) - \hat {y} (a))} & i f a \neq \hat {a}, \\ 1 - \sum_ {a \neq \hat {a}} p (a) & i f a = \hat {a} \end{array} \right.
$$
where  $\hat{a} = \arg \max_{a}\hat{y} (a)$ , then for any  $y^{\star}\in [0,1]^{A}$  and  $\gamma >0$ , we have
$$
\underset {a \sim p} {\mathbb {E}} \left[ \left(y ^ {\star} (a ^ {\star}) - y ^ {\star} (a)\right) - \gamma \left(\hat {y} (a) - y ^ {\star} (a)\right) ^ {2} \right] \leq \frac {A}{\gamma}.
$$
Lemma 4 (Zhu and Nowak [2022, Lemma 2]). Let  $(Z_{t})_{t\leq T}$  to be real-valued sequence of positive random variables adapted to a filtration  $\mathfrak{F}_t$ . If  $|Z_{t}|\leq B$  almost surely, then with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} Z _ {t} \leq \frac {3}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ Z _ {t} \right] + 4 B \log \left(2 \delta^ {- 1}\right),
$$
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ Z _ {t} \right] \leq 2 \sum_ {t = 1} ^ {T} Z _ {t} + 8 B \log \left(2 \delta^ {- 1}\right).
$$
Lemma 5 (Performance difference lemma [Agarwal et al., 2019]). For any two policies  $\pi$  and  $\pi'$  and any state  $x_0 \in \mathcal{X}$ , we have
$$
V _ {0} ^ {\pi} \left(x _ {0}\right) - V _ {0} ^ {\pi^ {\prime}} \left(x _ {0}\right) = \sum_ {h = 0} ^ {H - 1} \underset {x _ {h}, a _ {h} \sim d _ {x _ {0}, h} ^ {\pi}} {\mathbb {E}} \left[ A _ {h} ^ {\pi^ {\prime}} \left(x _ {h}, a _ {h}\right) \right]
$$
where  $A_h^\pi(x, a) = Q_h^\pi(x, a) - V_h^\pi(x, a)$  and  $d_{x_0, h}^\pi(x, a)$  is the probability of  $\pi$  reaching the state-action pair  $(x, a)$  at time step  $h$  starting from initial state  $x_0$ .
Lemma 6. For any two Bernoulli distributions  $\operatorname{Bern}(x)$  and  $\operatorname{Bern}(y)$  with  $x, y \in [b, 1 - b]$  for some  $0 < b \leq 1/2$ , the KL divergence is bounded as
$$
\operatorname {K L} \left(\operatorname {B e r n} (x), \operatorname {B e r n} (y)\right) \leq \frac {2 (x - y) ^ {2}}{b}.
$$
Proof of Lemma 6. Denote  $\Delta = x - y$ . Then, by definition, we have
$$
\begin{array}{l} \operatorname {K L} \left(\operatorname {B e r n} (x), \operatorname {B e r n} (y)\right) = x \ln \frac {x}{y} + (1 - x) \ln \frac {1 - x}{1 - y} \\ = x \ln \frac {x}{x - \Delta} + (1 - x) \ln \frac {1 - x}{1 - x + \Delta} \\ = x \ln \left(1 + \frac {\Delta}{x - \Delta}\right) + (1 - x) \ln \left(1 - \frac {\Delta}{1 - x + \Delta}\right) \\ \end{array}
$$
Since  $\ln (1 + x)\leq x$  for all  $x > - 1$ , we have
$$
\begin{array}{l} \operatorname {K L} \left(\operatorname {B e r n} (x), \operatorname {B e r n} (y)\right) \leq x \cdot \frac {\Delta}{x - \Delta} - (1 - x) \cdot \frac {\Delta}{1 - x + \Delta} \\ = \Delta \cdot \left(\frac {x}{x - \Delta} - \frac {1 - x}{1 - x + \Delta}\right) \\ = \Delta \cdot \left(\frac {\Delta}{x - \Delta} + \frac {\Delta}{1 - x + \Delta}\right) \\ \leq \Delta^ {2} \cdot \left(\frac {1}{y} + \frac {1}{1 - y}\right) \leq \frac {2 \Delta^ {2}}{b}. \\ \end{array}
$$
C Missing Proofs
C.1 Supporting Lemmas
Definition 3 (Strong convexity). A function  $\Phi : [-1,1] \to \mathbb{R}$  is  $\alpha$ -strongly-convex if for all  $u, u' \in \mathbb{R}$ , we have
$$
\frac {\alpha}{2} \left(u ^ {\prime} - u\right) ^ {2} \leq \Phi \left(u ^ {\prime}\right) - \Phi (u) - \nabla \Phi (u) \left(u ^ {\prime} - u\right).
$$
where  $\nabla \Phi$  means the derivative of  $\Phi$ .
Lemma 7. For any  $t \in [T]$ , if  $f^{\star} \in \mathcal{F}_t$ , then we have  $w_t \geq \Delta$  whenever  $|\mathcal{A}_t| > 1$ .
Proof of lemma 7. When  $|\mathcal{A}_t| > 1$ , we know there exists a function  $f' \in \mathcal{F}_t$  satisfying
$$
a ^ {\prime} := \pi_ {f ^ {\prime}} (x _ {t}) \neq \pi_ {f ^ {\star}} (x _ {t}) =: a _ {t} ^ {\star}.
$$
Then we have  $\Delta \leq f^{\star}(x_t, a_t^{\star}, a') \leq f^{\star}(x_t, a_t^{\star}, a') - f'(x_t, a_t^{\star}, a') \leq w_t$  where the second inequality holds since  $f'(x_t, a_t^{\star}, a') \leq 0$ .
Lemma 8. For any  $t \in [T]$  and any arm  $a \in \mathcal{A}_t$ , we have  $f^{\star}(x_t, \pi_{f^{\star}}(x_t), a) \leq w_t$ .
Proof of Lemma 8. For any  $a \in \mathcal{A}_t$ , by the definition of  $\mathcal{A}_t$ , there must exists a function  $f$  for which  $a = \pi_f(x_t)$ . Hence,
$$
f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \leq f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) - f \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a\right) \leq w _ {t},
$$
where the first inequality holds since  $f(x_{t},\pi_{f^{*}}(x_{t}),a)\leq 0$
The following lemma is adapted from Agarwal [2013, Lemma 2].
Lemma 9. The following holds with probability at least  $1 - \delta$  for any  $T > 3$ ,
$$
\sum_ {t = 1} ^ {T} Z _ {t} \left(f ^ {\star} \left(x _ {t}, a _ {t}, b _ {t}\right) - f _ {t} \left(x _ {t}, a _ {t}, b _ {t}\right)\right) ^ {2} \leq \frac {4 \Upsilon}{\alpha} + \frac {1 6 + 2 4 \alpha}{\alpha^ {2}} \log \left(4 \delta^ {- 1} \log (T)\right).
$$
Proof of Lemma 9. Throughout the proof, we denote  $z_{t} \coloneqq (x_{t}, a_{t}, b_{t})$  for notational simplicity. We define  $D_{\Phi}$  as the Bregman divergence of the function  $\Phi$ :
$$
D _ {\Phi} (u, v) = \Phi (u) - \Phi (v) - \phi (v) (u - v)
$$
where we recall that  $\phi = \Phi'$  is the derivative of  $\Phi$ . Since  $\Phi$  is  $\alpha$ -strong convex, we have  $\alpha(u - v)^2 / 2 \leq D_{\Phi}(u, v)$ , and hence,
$$
\sum_ {t = 1} ^ {T} Z _ {t} \left(f ^ {\star} \left(z _ {t}\right) - f _ {t} \left(z _ {t}\right)\right) ^ {2} \leq \frac {2}{\alpha} \sum_ {t = 1} ^ {T} Z _ {t} D _ {\Phi} \left(f _ {t} \left(z _ {t}\right), f ^ {\star} \left(z _ {t}\right)\right). \tag {2}
$$
Hence, it suffices to derive an upper bound for the Bregman divergence in the right hand side above. Define  $\nu_{t}$  as below:
$$
\begin{array}{l} \nu_ {t} := Z _ {t} \left[ D _ {\Phi} \left(f _ {t} \left(z _ {t}\right), f ^ {\star} \left(z _ {t}\right)\right) - \left(\ell_ {\phi} \left(f _ {t} \left(z _ {t}\right), y _ {t}\right) - \ell_ {\phi} \left(f ^ {\star} \left(z _ {t}\right), y _ {t}\right)\right) \right] \\ = Z _ {t} \left[ D _ {\Phi} \left(f _ {t} \left(z _ {t}\right), f ^ {\star} \left(z _ {t}\right)\right) - \left(\Phi \left(f _ {t} \left(z _ {t}\right)\right) - \left(y _ {t} + 1\right) f _ {t} \left(z _ {t}\right) / 2 - \Phi \left(f ^ {\star} \left(z _ {t}\right)\right) + \left(y _ {t} + 1\right) f ^ {\star} \left(z _ {t}\right) / 2\right) \right] \\ = Z _ {t} \left[ \Phi \left(f _ {t} \left(z _ {t}\right)\right) - \Phi \left(f ^ {\star} \left(z _ {t}\right)\right) - \phi \left(f ^ {\star} \left(z _ {t}\right)\right) \left(f _ {t} \left(z _ {t}\right) - f ^ {\star} \left(z _ {t}\right)\right) \right. \\ \left. - \left(\Phi \left(f _ {t} \left(z _ {t}\right)\right) - \left(y _ {t} + 1\right) f _ {t} \left(z _ {t}\right) / 2 - \Phi \left(f ^ {\star} \left(z _ {t}\right)\right) + \left(y _ {t} + 1\right) f ^ {\star} \left(z _ {t}\right) / 2\right) \right] \\ = Z _ {t} \left(f _ {t} \left(z _ {t}\right) - f ^ {\star} \left(z _ {t}\right)\right) \left(\left(y _ {t} + 1\right) / 2 - \phi \left(f ^ {\star} \left(z _ {t}\right)\right)\right) \\ \end{array}
$$
We note that  $\mathbb{E}_t[(y_t + 1) / 2] = \phi (f^\star (z_t))$ , and thus  $\mathbb{E}_t[\nu_t] = 0$ , which means  $\nu_{t}$  is a martingale difference sequence. Now we bound the value and the conditional variance of  $\nu_{t}$  in order to derive concentration results.
1. Bound the value of  $\nu_{t}$ :
$$
\left| \nu_ {t} \right| \leq \left| \left(y _ {t} + 1\right) / 2 - \phi \left(f ^ {\star} \left(z _ {t}\right)\right) \right| \cdot \left| f _ {t} \left(z _ {t}\right) - f ^ {\star} \left(z _ {t}\right) \right| \leq 1 \cdot 2 = 2.
$$
2. Bound the conditional variance of  $\nu_{t}$ :
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \nu_ {t} ^ {2} \right] = Z _ {t} \mathbb {E} _ {t} \left[ \left(\left(y _ {t} + 1\right) / 2 - \phi \left(f ^ {\star} \left(z _ {t}\right)\right)\right) ^ {2} \left(f _ {t} \left(z _ {t}\right) - f ^ {\star} \left(z _ {t}\right)\right) ^ {2} \right] \\ \leq Z _ {t} \underset {t} {\mathbb {E}} \left[ \left(f _ {t} \left(z _ {t}\right) - f ^ {\star} \left(z _ {t}\right)\right) ^ {2} \right] \\ \leq Z _ {t} \underset {t} {\mathbb {E}} \left[ \frac {2}{\alpha} \cdot D _ {\Phi} \left(f _ {t} \left(z _ {t}\right), f ^ {\star} \left(z _ {t}\right)\right) \right] \\ \leq \frac {2 Z _ {t}}{\alpha} D _ {\Phi} \left(f _ {t} \left(z _ {t}\right), f ^ {\star} \left(z _ {t}\right)\right) \\ \end{array}
$$
where for the last line we note that  $x_{t}, g_{t}$  are measurable at  $t$ .
Now we apply Lemma 2, which yields for any  $\delta < 1/e$  and  $T > 3$ , with probability at least  $1 - 4\delta \log(T)$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \nu_ {t} \leq \max \left\{2 \sqrt {\sum_ {t = 1} ^ {T} \frac {2 Z _ {t}}{\alpha} D _ {\Phi} (f _ {t} (z _ {t}) , f ^ {\star} (z _ {t}))}, 6 \sqrt {\log (1 / \delta)} \right\} \sqrt {\log (1 / \delta)} \\ \leq 2 \sqrt {\sum_ {t = 1} ^ {T} \frac {2 Z _ {t}}{\alpha} D _ {\Phi} \left(f _ {t} \left(z _ {t}\right) , f ^ {\star} \left(z _ {t}\right)\right) \log (1 / \delta)} + 6 \log (1 / \delta) \quad (\text {s i n c e} a, b) \leq a + b) \\ \leq \sum_ {t = 1} ^ {T} \frac {1}{2} Z _ {t} D _ {\Phi} \left(f _ {t} \left(z _ {t}\right), f ^ {\star} \left(z _ {t}\right)\right) + \frac {4 \log (1 / \delta)}{\alpha} + 6 \log (1 / \delta) \tag {AM-GM} \\ \end{array}
$$
Recall the definition of  $\nu_{t}$ , and we conclude that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} Z _ {t} D _ {\Phi} \left(f _ {t} (z _ {t}), f ^ {\star} (z _ {t})\right) - \sum_ {t = 1} ^ {T} Z _ {t} \left(\ell_ {\phi} \left(f _ {t} (z _ {t}), y _ {t}\right) - \ell_ {\phi} \left(f ^ {\star} (z _ {t}), y _ {t}\right)\right) \leq \\ \sum_ {t = 1} ^ {T} \frac {1}{2} Z _ {t} D _ {\Phi} (f _ {t} (z _ {t}), f ^ {\star} (z _ {t})) + \frac {4 \log (1 / \delta)}{\alpha} + 6 \mathrm {l o g} (1 / \delta), \\ \end{array}
$$
$$
\frac {1}{2} \sum_ {t = 1} ^ {T} Z _ {t} D _ {\Phi} \left(f _ {t} \left(z _ {t}\right), f ^ {\star} \left(z _ {t}\right)\right) \leq \sum_ {t = 1} ^ {T} Z _ {t} \left(\ell_ {\phi} \left(f _ {t} \left(z _ {t}\right), y _ {t}\right) - \ell_ {\phi} \left(f ^ {\star} \left(z _ {t}\right), y _ {t}\right)\right) + \frac {4 \log (1 / \delta)}{\alpha} + 6 \log (1 / \delta).
$$
Plugging this upper bound of Bregman divergence into (2), we obtain that, with probability at least  $1 - 4\delta \log(T)$ , for any  $\delta < 1/e$  and  $T > 3$ , we have
$$
\sum_ {t = 1} ^ {T} Z _ {t} \left(f ^ {\star} \left(z _ {t}\right) - f _ {t} \left(z _ {t}\right)\right) ^ {2} \leq \frac {4}{\alpha} \Upsilon + \left(\frac {1 6}{\alpha^ {2}} + \frac {2 4}{\alpha}\right) \log (\delta^ {- 1}) =: \beta
$$
Finally, we finish the proof by adjusting the coefficient  $\delta$  and taking a union bound to obtain the desired result.
The following lemma is a variant of Russo and Van Roy [2013, Proposition 3], with the main difference being that (1) the version space is established using the function produced by the oracle instead of the least squares estimator, and (2) the extra multiplicative factor  $Z_{t}$ .
Lemma 10. For Algorithm 1, it holds that
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {t}} f \left(x _ {t}, a _ {t}, b _ {t}\right) - f ^ {\prime} \left(x _ {t}, a _ {t}, b _ {t}\right) > \epsilon \right\} \leq \left(\frac {4 \beta}{\epsilon^ {2}} + 1\right) \dim_ {E} (\mathcal {F}, \epsilon) \tag {3}
$$
for any constant  $\epsilon > 0$ ,
Proof of Lemma 10. We first define a subsequence consisting only of the elements for which we made a query in that round. Specifically, we define  $((x_{i_1},a_{i_1},b_{i_1}),(x_{i_2},a_{i_2},b_{i_2}),\ldots ,(x_{i_k},a_{i_k},b_{i_k}))$  where  $1\leq i_1 < i_2 < \dots < i_k\leq T$  and  $(x_{t},a_{t},b_{t})$  belongs to the subsequence if and only if  $Z_{t} = 1$ . We further simplify the notation by defining  $z_{j}\coloneqq (x_{i_{j}},a_{i_{j}},b_{i_{j}})$  and  $f(z_{j})\coloneqq f(x_{i_{j}},a_{i_{j}},b_{i_{j}})$ . Then we note that the left-hand side of (3) is equivalent to
$$
\sum_ {j = 1} ^ {k} \mathbb {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {j}} f \left(z _ {j}\right) - f ^ {\prime} \left(z _ {j}\right) > \epsilon \right\}, \tag {4}
$$
and the version space in Algorithm 1 is equal to
$$
\mathcal {F} _ {j} = \left\{f \in \mathcal {F}: \sum_ {s = 1} ^ {j - 1} \left(f \left(z _ {s}\right) - f _ {s} \left(z _ {s}\right)\right) ^ {2} \leq \beta \right\}. \tag {5}
$$
Hence, it suffices to establish the lower bound for (4) under the version space of (5). To that end, we make one more simplification in notation: we denote
$$
w _ {j} ^ {\prime} := \sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {j}} f (z _ {j}) - f ^ {\prime} (z _ {j})
$$
We begin by showing that if  $w_{j}^{\prime} > \epsilon$  for some  $j\in [k]$ , then  $z_{j}$  is  $\epsilon$ -dependent on at most  $4\beta /\epsilon^{2}$  disjoint subsequence of its predecessors. To see this, we note that when  $w_{j}^{\prime} > \epsilon$ , there must exist two functions  $f,f^{\prime}\in \mathcal{F}_{j}$  such that  $f(z_{j}) - f^{\prime}(z_{j}) > \epsilon$ . If  $z_{j}$  is  $\epsilon$ -dependent on a subsequence  $(z_{i_1},z_{i_2},\dots ,z_{i_n})$  of its predecessors, we must have
$$
\sum_ {s = 1} ^ {n} \left(f \left(z _ {i _ {s}}\right) - f ^ {\prime} \left(z _ {i _ {s}}\right)\right) ^ {2} > \epsilon^ {2}.
$$
Hence, if  $z_{j}$  is  $\epsilon$ -dependent on  $l$  disjoint subsequences, we have
$$
\sum_ {s = 1} ^ {j - 1} \left(f \left(z _ {s}\right) - f ^ {\prime} \left(z _ {s}\right)\right) ^ {2} > l \epsilon^ {2}. \tag {6}
$$
For the left-hand side above, we also have
$$
\sum_ {s = 1} ^ {j - 1} \left(f \left(z _ {s}\right) - f ^ {\prime} \left(z _ {s}\right)\right) ^ {2} \leq 2 \sum_ {s = 1} ^ {j - 1} \left(f \left(z _ {s}\right) - f _ {s} \left(z _ {s}\right)\right) ^ {2} + 2 \sum_ {s = 1} ^ {j - 1} \left(f _ {s} \left(z _ {s}\right) - f ^ {\prime} \left(z _ {s}\right)\right) ^ {2} \leq 4 \beta \tag {7}
$$
where the first inequality holds since  $(a + b)^2 \leq 2(a^2 + b^2)$  for any  $a, b$ , and the second inequality holds by (5). Combining (6) and (7), we get that  $l \leq 4\beta/\epsilon^2$ .
Next, we show that for any sequence  $(z_1',\ldots ,z_\tau ')$ , there is at least one element that is  $\epsilon$ -dependent on at least  $\tau /d - 1$  disjoint subsequence of its predecessors, where  $d\coloneqq \dim_{E}(\mathcal{F},\epsilon)$ . To show this, let  $m$  be the integer satisfying  $md + 1\leq \tau \leq md + d$ . We will construct  $m$  disjoint subsequences,  $B_{1},\dots ,B_{m}$ . At the beginning, let  $B_{i} = (z_{i}^{\prime})$  for  $i\in [m]$ . If  $z_{m + 1}^{\prime}$  is  $\epsilon$ -dependent on each subsequence  $B_{1},\ldots ,B_{m}$ , then we are done. Otherwise, we select a subsequence  $B_{i}$  which  $z_{m + 1}^{\prime}$  is  $\epsilon$ -independent of and append  $z_{m + 1}^{\prime}$  to  $B_{i}$ . We repeat this process for all elements with indices  $j > m + 1$  until either  $z_j^\prime$  is  $\epsilon$ -dependent on each subsequence or  $j = \tau$ . For the latter, we have  $\sum_{i = 1}^{m}|B_i|\geq md$ , and since each element of a subsequence  $B_{i}$  is  $\epsilon$ -independent of its predecessors, we must have  $|B_i| = d$  for all  $i$ . Then,  $z_{\tau}$  must be  $\epsilon$ -dependent on each subsequence by the definition of eluder dimension.
Finally, let's take the sequence  $(z_1',\ldots z_\tau ')$  to be the subsequence of  $(z_{1},\dots ,z_{k})$  consisting of elements  $z_{j}$  for which  $w_j^\prime >\epsilon$ . As we have established, we have (1) each  $z_j^\prime$  is  $\epsilon$ -dependent on at most  $4\beta /\epsilon^2$  disjoint subsequences, and (2) some  $z_j^\prime$  is  $\epsilon$ -dependent on at least  $\tau /d - 1$  disjoint subsequences. Therefore, we must have  $\tau /d - 1\leq 4\beta /\epsilon^2$ , implying that  $\tau \leq (4\beta /\epsilon^2 +1)d$ .
The following lemma is adopted from Saha and Krishnamurthy [2022, Lemma 3].
Lemma 11. For any function  $f \in \mathcal{F}$  and any context  $x \in \mathcal{X}$ , the following convex program of  $p \in \Delta(\mathcal{A})$  is always feasible:
$$
\forall a \in \mathcal {A}: \sum_ {b} f (x, a, b) p (b) + \frac {2}{\gamma p (a)} \leq \frac {5 A}{\gamma}.
$$
Furthermore, any solution  $p$  satisfies: