SlowGuess/NeurIPS_2023_OCR huggingface dataset:
T. Lattimore and C. Szepesvari. The end of optimism? an asymptotic analysis of finite-armed linear bandits. In Artificial Intelligence and Statistics, pages 728-737. PMLR, 2017.
T. Lattimore and C. Szepesvári. An information-theoretic approach to minimax regret in partial monitoring. In Conference on Learning Theory, pages 2111-2139. PMLR, 2019.
T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020a.
T. Lattimore and C. Szepesvári. Exploration by optimisation in partial monitoring. In Conference on Learning Theory, pages 2488-2515. PMLR, 2020b.
T. Lattimore, B. Kveton, S. Li, and C. Szepesvári. Toprank: A practical algorithm for online stochastic ranking. In Advances in Neural Information Processing Systems, pages 3949-3958. Curran Associates, Inc., 2018.
T. Lin, B. Abrahao, R. Kleinberg, J. Lui, and W. Chen. Combinatorial partial monitoring game with linear feedback and its applications. In International Conference on Machine Learning, pages 901-909, 2014.
S. Mannor and O. Shamir. From bandits to experts: On the value of side-observations. Advances in Neural Information Processing Systems, 24, 2011.
I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
F. Radlinski, R. Kleinberg, and T. Joachims. Learning diverse rankings with multi-armed bandits. In Proceedings of the 25th International Conference on Machine Learning, pages 784-791. ACM, 2008.
D. Russo and B. Van Roy. Learning to optimize via information-directed sampling. Advances in Neural Information Processing Systems, 27, 2014.
D. Russo and B. Van Roy. An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research, 17(1):2442-2471, 2016.
A. Rustichini. Minimizing regret: The general case. Games and Economic Behavior, 29(1-2): 224-243, 1999.
S. Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning, 4(2):107-194, 2012.
N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proc. International Conference on Machine Learning (ICML), 2010.
W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
Y. Yue and T. Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th International Conference on Machine Learning, pages 1201-1208. ACM, 2009.
A. Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning, pages 10978-10989. PMLR, 2020.
T. Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022.
D. Zhou, Q. Gu, and C. Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021.
Algorithm 2: Exponential Weights Algorithm (EWA) for Density Estimation
Input: Finite model class  $\mathcal{M}$ , data  $\mathcal{D}_t = \{(y_1, \pi_1), \dots, (y_t, \pi_t)\}$ , Learning rate  $\eta > 0$   
1 Define  $L(f) = -\sum_{s=1}^{t} \log M_f(y_s | \pi_s)$   
2 Let  $p(f) \propto \exp(-\eta L(f))$   
3 For convex  $\mathcal{M}$ : Return  $\sum_{f \in \mathcal{M}} p(f) f$   
4 Else: Return  $f \sim p(\cdot)$ .
A Online Density Estimation
For any  $f \in \mathcal{M}$  and  $\pi \in \Pi$ , we denote by  $p(\cdot |\pi, f)$  the density function of the observation distribution  $M_f(\pi)$  w.r.t. a reference measure over the observation space  $\mathcal{O}$ . Consider a finite model class  $\mathcal{M}$  and the KL divergence,
$$
e _ {\pi} I _ {f} e _ {g} = \mathbb {E} _ {y \sim M _ {g} (\pi)} \left[ \log \left(\frac {p (y \mid \pi , g)}{p (y \mid \pi , f)}\right) \right] \tag {13}
$$
In this case, the estimation error can be written as follows:
$$
\begin{array}{l} \operatorname {E s t} _ {n} = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} e _ {\pi_ {t}} I _ {\hat {f} _ {t}} e _ {f ^ {*}} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \log \left(p \left(y _ {t} \mid \pi_ {t}, f ^ {*}\right) / p \left(y _ {t} \mid \pi_ {t}, \hat {f} _ {t}\right) \right. \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \log \left(\frac {1}{p \left(y _ {t} \mid \pi_ {t} , \hat {f} _ {t}\right)}\right) - \sum_ {t = 1} ^ {n} \log \left(\frac {1}{p \left(y _ {t} \mid \pi_ {t} , f ^ {*}\right)}\right) \right] \\ \end{array}
$$
The last line can be understood as the estimation regret of the estimates  $\hat{f}_1, \dots, \hat{f}_n$  under the logarithmic loss. A classical approach to control this term is the exponential weights algorithm (EWA) given in Algorithm 2. For the EWA algorithm, we have the following bound.
Lemma 6 (EWA for Online Density Estimation). For any data stream  $\{y_1, \pi_1, \dots, y_n, \pi_n\}$  the predictions  $\hat{f}_1, \dots, \hat{f}_n$  obtained via Algorithm 2 with  $\eta = 1$  satisfy
$$
\operatorname {E s t} _ {n} \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \log \left(\frac {1}{p \left(y _ {t} \mid \pi_ {t} , \hat {f} _ {t}\right)}\right) - \inf  _ {g \in \mathcal {M}} \sum_ {t = 1} ^ {n} \log \left(\frac {1}{p \left(y _ {t} \mid \pi_ {t} , g\right)}\right) \right] \leq \log (| \mathcal {M} |) \tag {14}
$$
For a proof, see [Cesa-Bianchi and Lugosi, 2006, Proposition 3.1].
A.1 Bounding the Estimation Error of Projected Regularized Least-Squares
In this section, we consider the linear model from Example 2.2. We denote by  $\| \cdot \|$  the Euclidean norm. For simplicity, the observation maps  $M_{\pi} \in \mathbb{R}^{m \times d}$  are assumed to have the same output dimension  $m \in \mathbb{N}$ . The observation distribution is such that  $y_{t} = M_{\pi_{t}}f^{*} + \xi_{t}$ , where  $\xi \in \mathbb{R}^{m}$  is random noise such that  $\mathbb{E}_t[\xi] = 0$  and  $\mathbb{E}_t[\| \xi \|^2] \leq \sigma^2$ . Here,  $\mathbb{E}_t[\cdot] = \mathbb{E}[\cdot | \pi_1, y_1, \ldots, \pi_{t-1}, y_{t-1}, \pi_t]$  is the conditional observation in round  $t$  including the decision  $\pi_t$  chosen in round  $t$ .
We will use the quadratic divergence $^4$ ,  $e_{\pi}I_{f}e_{g} = \frac{1}{2}\| M_{\pi}(g - f)\|^{2}$ . This choice corresponds to the Gaussian KL, but we do not require that the noise distribution is Gaussian is the following. In the linear bandit model, this choice reduces to  $e_{\pi}I_{f}e_{g} = \frac{1}{2}\langle \phi_{\pi},g - f\rangle^{2}$ .
Let  $K \subset \mathbb{R}^d$  be a closed convex set. Our goal is to control the estimation regret for the projected regularized least-squares estimator,
$$
\hat {f} _ {t} = \underset {f \in K} {\arg \min } \sum_ {s = 1} ^ {t - 1} \| M _ {\pi_ {s}} f - y _ {s} \| ^ {2} + \| f \| _ {V _ {0}} ^ {2} = \operatorname {P r o j} _ {V _ {t}} \left(V _ {t} ^ {- 1} \sum_ {s = 1} ^ {t - 1} M _ {\pi_ {s}} ^ {\top} y _ {s}\right) \tag {15}
$$
where  $V_{0}$  is a positive definite matrix,  $V_{t} = \sum_{s = 1}^{t - 1}M_{\pi_{s}}^{\top}M_{\pi_{s}} + V_{0}$  and  $\mathrm{Proj}_{V_t}(\cdot)$  is the orthogonal projection w.r.t. the  $\| \cdot \|_{V_t}$  norm. For  $K = \mathbb{R}^d$  and  $V_{0} = \eta \mathbf{1}_{d}$ , this recovers the standard ridge
4We added a factor of  $\frac{1}{2}$  for convenience.
regression. The projection is necessary to bound the magnitude of the squared loss, and the result will depend on an almost-surely bound on the 'observed' diameter,
$$
\max  _ {f, g \in K} \max  _ {\pi \in \Pi} \| M _ {\pi} (f - g) \| \leq B
$$
Recall that our goal is to bound the estimation error,
$$
\operatorname {E s t} _ {n} = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} e _ {\pi_ {t}} I _ {\hat {f} _ {t}} e _ {f ^ {*}} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \frac {1}{2} \| M _ {\pi_ {t}} \left(f ^ {*} - \hat {f} _ {t}\right) \| ^ {2} \right] \tag {16}
$$
We remark that one can get the following naive bound by applying Cauchy-Schwarz:
$$
\sum_ {t = 1} ^ {n} \left\| M _ {\pi_ {t}} \left(f ^ {*} - \hat {f} _ {t}\right) \right\| ^ {2} \leq \sum_ {t = 1} ^ {n} \left\| M _ {\pi_ {t}} \right\| _ {V _ {t} ^ {- 1}} ^ {2} \| f ^ {*} - \hat {f} _ {t} \| _ {V _ {t}} ^ {2} \leq \mathcal {O} \left(d ^ {2} \log (n) ^ {2}\right) \tag {17}
$$
The last inequality follows from the elliptic potential lemma and standard concentration inequalities [Lattimore and Szepesvári, 2020a, Lemma 19.4 and Theorem 20.5]. However, this will lead to an additional  $d$ -factor in the regret that can be avoided, as we see next.
For  $K = \mathbb{R}^d$ , one-dimensional observations and noise bounded in the range  $[-\bar{B}, \bar{B}]$ , one can also directly apply [Cesa-Bianchi and Lugosi, 2006, Theorem 11.7] to get  $\mathrm{Est}_n \leq \mathcal{O}(\bar{B}^2 d\log (n))$ , thereby improving the naive bound by a factor  $d\log (n)$ . This result is obtained in a more general setting, where no assumptions, other than boundedness, are placed on the observation sequence  $y_{1},\ldots ,y_{n}$ . Here we refine and generalize this result in two directions: First, we allow for the more general feedback model in with multi-dimensional observations (Example 2.2). Second, we directly exploit the stochastic observation model to obtain a stronger result that does not require the observation noise to be bounded.
Theorem 2. Consider the linear observation setting with additive noise and quadratic divergence  $e_{\pi}I_{f}e_{g} = \frac{1}{2}\| M_{\pi}(g - f)\|^{2}$ , as described at the beginning of this section. Assume that  $\max_{f,g\in \mathcal{M},\pi \in \Pi}\| M_{\pi}(f - g)\| \leq B$  and  $\mathbb{E}\big[\| \xi_t\| ^2\big]\leq \sigma^2$ . Then
$$
\operatorname {E s t} _ {n} \leq (\sigma^ {2} + B ^ {2}) \mathbb {E} \left[ \log \left(\frac {\det V _ {n}}{\det V _ {0}}\right) \right]
$$
If in addition  $\| M_{\pi}\| \leq L$  and  $V_{0} = \eta \mathbf{1}_{d}$ , then  $\operatorname{Est}_n \leq (\sigma^2 + B^2) \log \left(1 + \frac{nL^2}{\eta d}\right)$ .
Remark 1. Note that by the [Lattimore and Szepesvári, 2020a, Theorem 19.4],  $\log \left(\frac{\operatorname*{det}V_n}{\operatorname*{det}V_0}\right)$  can further be upper bounded by  $d\log \left(\frac{\text{trace}V_0 + nL^2}{d\operatorname*{det}(V_0)^{1 / d}}\right)$ , which effectively results the desired bound.
Proof. The proof adapts [György et al., 2013, Theorem 19.8] to multi-dimensional observations and takes advantage of the stochastic loss function by taking the expectation.
First, define  $l_{t}(f) = \frac{1}{2}\| M_{\pi_{t}}f - y_{t}\|^{2}$ . Then, using that  $\mathbb{E}_t[y_t] = M_{\pi_t}f^*$ ,
$$
\begin{array}{l} \operatorname {E s t} _ {n} = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \frac {1}{2} \| M _ {\pi_ {t}} (f ^ {*} - \hat {f} _ {t}) \| ^ {2} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \frac {1}{2} \| M _ {\pi_ {t}} \hat {f} _ {t} - y _ {t} \| ^ {2} - \frac {1}{2} \| M _ {\pi_ {t}} f ^ {*} - y _ {t} \| ^ {2} \right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} l _ {t} \left(\hat {f} _ {t}\right) - l _ {t} \left(f ^ {*}\right) \right] \\ \end{array}
$$
Further, by directly generalizing [György et al., 2013, Lemma 19.7], we have that
$$
l _ {t} \left(\hat {f} _ {t + 1}\right) - l _ {t} \left(\hat {f} _ {t}\right) \leq \nabla l _ {t} \left(\hat {f} _ {t}\right) V _ {t} ^ {- 1} \nabla l _ {t} \left(\hat {f} _ {t}\right) = \left(M _ {\pi_ {t}} \hat {f} _ {t} - y _ {t}\right) ^ {\top} M _ {\pi_ {t}} ^ {\top} V _ {t} ^ {- 1} M _ {\pi_ {t}} \left(M _ {\pi_ {t}} \hat {f} _ {t} - y _ {t}\right) \tag {18}
$$
We now start upper bounding the estimation error,
$$
\begin{array}{l} \operatorname {E s t} _ {n} \stackrel {(i)} {\leq} \| f ^ {*} \| ^ {2} + \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \left(l _ {t} \left(w _ {t}\right) - l _ {t} \left(w _ {t + 1}\right)\right) \right] \\ \stackrel {(i i)} {\leq} \| f ^ {*} \| ^ {2} + \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \left(\xi_ {t} + M _ {\pi_ {t}} (f ^ {*} - \hat {f} _ {t - 1})\right) M _ {\pi_ {t}} V _ {t} ^ {- 1} M _ {\pi_ {t}} \left(\xi_ {t} + M _ {\pi_ {t}} (f ^ {*} - \hat {f} _ {t - 1})\right) \right] \\ \stackrel {(i i i)} {=} \| f ^ {*} \| ^ {2} + \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \xi_ {t} M _ {\pi_ {t}} V _ {t} ^ {- 1} M _ {\pi_ {t}} \xi_ {t}) \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \bar {x} _ {t} M _ {\pi_ {t}} V _ {t} ^ {- 1} M _ {\pi_ {t}} \bar {x} _ {t}) \right] \\ \stackrel {(i v)} {\leq} \| f ^ {*} \| ^ {2} + \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \lambda_ {\max } \left(M _ {\pi_ {t}} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right) \| \xi_ {t} \| ^ {2} \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \lambda_ {\max } \left(M _ {\pi_ {t}} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right) \| \bar {x} _ {t} \| ^ {2}\right) \\ \stackrel {(v)} {\leq} \| f ^ {*} \| ^ {2} + \left(\sigma^ {2} + B ^ {2}\right) \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \lambda_ {\max } \left(M _ {\pi_ {t}} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right) \right] \tag {19} \\ \end{array}
$$
The inequality (i) follows from [Shalev-Shwartz et al., 2012, Lemma 2.3]. For (ii) we used Eq. (18). For (iii) we used that  $\mathbb{E}_t[\xi_t] = 0$ . In (iv), we introduce the maximum eigenvalue  $\lambda_{\max}(A)$  for  $A \in \mathbb{R}^{m \times m}$  and denote  $\bar{x}_t = M_{\pi_t}(f^* - f_{t-1})$ . Lastly, in (v) we used that  $\|\bar{x}_t\|^2 \leq B$  and  $\mathbb{E}_t[\|\xi_t\|^2] \leq \sigma^2$ .
We conclude the proof with basic linear algebra. Denote by  $\lambda_{i}(A)$  the  $i$ -th eigenvalue of a matrix  $M \in \mathbb{R}^{m \times m}$ . Using the generalized matrix determinant lemma, we get
$$
\begin{array}{l} \det  \left(V _ {t - 1}\right) = \det  \left(V _ {t} - M _ {\pi_ {t}} ^ {\top} M _ {\pi_ {t}}\right) \\ = \det  (V _ {t}) \det  (I - M _ {\pi_ {t}} ^ {\top} V _ {t} ^ {- 1} M _ {\pi_ {t}}) \\ = \det  (V _ {t}) \prod_ {i = 1} ^ {m} \left(1 - \lambda_ {i} \left(M _ {\pi_ {t}} ^ {\top} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right)\right) \\ \end{array}
$$
Note that  $\lambda_{i}(M_{\pi_{t}}^{\top}V_{t}^{-1}M_{\pi_{t}})\in (0,1]$ . Next, using that  $\log (1 - x)\leq -x$  for all  $x < 1$ , we get that
$$
\log \left(\frac {\det  (V _ {t - 1})}{\det  (V _ {t})}\right) = \sum_ {i = 1} ^ {m} \log \left(1 - \lambda_ {i} \left(M _ {\pi_ {t}} ^ {\top} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right)\right) \leq - \sum_ {i = 1} ^ {m} \lambda_ {i} \left(M _ {\pi_ {t}} ^ {\top} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right)
$$
Rearranging the last display, and bounding the sum by its maximum element, we get
$$
\lambda_ {\max } \left(M _ {\pi_ {t}} ^ {\top} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right) \leq \sum_ {i = 1} ^ {m} \lambda_ {i} \left(M _ {\pi_ {t}} ^ {\top} V _ {t} ^ {- 1} M _ {\pi_ {t}}\right) \leq \log \left(\frac {\det (V _ {t})}{\det (V _ {t - 1})}\right) \tag {20}
$$
The proof is concluded by combining Eqs. (19) and (20).
Remark 2 (Expected Regret). The beauty of Theorem 1 is that the proof uses only in-expectation arguments. This is unlike most previous analysis, that controls the regret via controlling tail-events, and bounds on the expected regret are then derived a-posteriori from high-probability bounds. In the context of linear bandits, Theorem 2 leads to bound on the expected regret that only requires the noise variance to be bounded, whereas most previous work relies on the stronger sub-Gaussian noise assumption [e.g. Abbasi-Yadtori et al., 2011].
Remark 3 (Kernel Bandits / Bayesian Optimization). Using the standard 'kernel-trick', the analysis can further be extended to the non-parametric setting where  $\mathcal{M}$  is an infinite-dimensional reproducing kernel Hilbert space (RKHS).
B PAC to Regret Bounds
Proof. The Lagrangian for Eq. (12) is
$$
\operatorname {p a c - d e c} _ {\epsilon} ^ {a c, f} (f) = \min  _ {\lambda \geq 0} \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \delta_ {f} \nu - \lambda \left(\mu I _ {f} \nu - \epsilon^ {2}\right).
$$
Reparametrize any  $\mu \in \mathcal{P}(\Pi)$  as  $\bar{\mu}(p) = (1 - p)e_{\pi_f^*} + p\mu_2$ . We bound  $\mathrm{dec}_{\epsilon}^{ac,f}$  by a function of  $\mathrm{pac-dec}_{\epsilon}^{ac,f}$ . Starting from Eq. (5), we have
$$
\begin{array}{l} \operatorname{dec}_{\epsilon}^{ac,f}(f) = \min_{\lambda \geq 0}\min_{\mu \in \mathcal{P}(\Pi)}\max_{\nu \in \mathcal{P}(\mathcal{M})}\mu \Delta_{f}\nu -\lambda (\mu I_{f}\nu -\epsilon^{2}) \\ = \min  _ {\lambda \geq 0} \min  _ {\bar {\mu} \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \bar {\mu} \Delta_ {f} \nu - \lambda \left(\bar {\mu} I _ {f} \nu - \epsilon^ {2}\right) \\ = \min  _ {\lambda \geq 0} \min  _ {0 \leq p \leq 1} \min  _ {\mu_ {2} \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \delta_ {f} \nu + p \mu_ {2} \Delta_ {f} e _ {f} - \lambda \bar {\mu} I _ {f} \nu - \lambda \epsilon^ {2} \\ \leq \min  _ {\lambda \geq 0} \min  _ {0 \leq p \leq 1} \min  _ {\mu_ {2} \in \mathscr {P} (\Pi)} \max  _ {\nu \in \mathscr {P} (\mathcal {M})} \delta_ {f} \nu + p \mu_ {2} \Delta_ {f} e _ {f} - \lambda p \mu_ {2} I _ {f} \nu - \lambda \epsilon^ {2} \\ \leq \min  _ {0 \leq p \leq 1} \min  _ {\lambda^ {\prime} \geq 0} \min  _ {\mu_ {2} \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \delta_ {f} \nu - \lambda^ {\prime} (\mu_ {2} I _ {f} \nu - \frac {\epsilon^ {2}}{p}) + p \Delta_ {\max } \\ \leq \min  _ {0 \leq p \leq 1} \operatorname {p a c - d e c} _ {\frac {\epsilon}{\sqrt {p}}} ^ {a c, f} (f) + p \Delta_ {\max } . \\ \end{array}
$$
Proof of Lemma 4. For the first part, note that
$$
\begin{array}{l} \operatorname {p a c - d e c} _ {\epsilon} ^ {a c, f} (f) = \min  _ {\mu \in \mathscr {P} (\Pi)} \min  _ {\lambda \geq 0} \max  _ {\nu \in \mathscr {P} (\mathcal {M})} \delta_ {f} \nu - \lambda \mu I _ {f} \nu + \lambda \epsilon^ {2} \\ = \min  _ {\mu \in \mathcal {P} (\Pi)} \min  _ {\lambda \geq 0} \max  _ {b \in \Pi} \max  _ {g \in \mathcal {M}} \left\langle \phi_ {b}, g \right\rangle - \left\langle \phi_ {\pi_ {f} ^ {*}}, f \right\rangle - \lambda \| g - f \| _ {V (\mu)} ^ {2} + \lambda \epsilon^ {2} \\ \stackrel {(i)} {=} \min  _ {\mu \in \mathscr {P} (\Pi)} \min  _ {\lambda \geq 0} \max  _ {b \in \Pi} \left\langle \phi_ {b} - \phi_ {\pi_ {f} ^ {*}}, f \right\rangle + \frac {1}{4 \lambda} \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} ^ {2} + \lambda \epsilon^ {2} \\ \stackrel {(i i)} {\leq} \min  _ {\mu \in \mathcal {P} (\Pi)} \min  _ {\lambda \geq 0} \max  _ {b \in \Pi} \frac {1}{4 \lambda} \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} ^ {2} + \lambda \epsilon^ {2} \\ = \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {b \in \Pi} \epsilon \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} \\ \stackrel {(i i i)} {\leq} \epsilon \sqrt {d}. \\ \end{array}
$$
Equation (i) follows by computing the maximizer attaining the quadratic form over  $\mathcal{M} = \mathbb{R}^d$ . The inequality (ii) is by definition of  $\pi_f^*$  and the last inequality (iii) by the assumption that the reward is observed, respectively,  $\phi_{\pi}\phi_{\pi}^{\top} \preceq M_{\pi}^{\top}M_{\pi}$ , and the Kiefer-Wolffowitz theorem.
The second part of the statement follows by combining Lemmas 2 and 3.
C Coefficient Relations Results and Proofs
Lemma 7. Assume Assumption 1 holds, i.e.
$$
\left(\mu \left(r _ {f} - r _ {g}\right)\right) ^ {2} \leq \mu I _ {f} e _ {g}. \tag {21}
$$
$$
\left(\sum_ {g} \mu \left(r _ {f} - r _ {g}\right) \nu_ {g}\right) ^ {2} \leq \sum_ {g} \left(\mu \left(r _ {f} - r _ {g}\right)\right) ^ {2} \nu_ {g} \leq \mu I _ {f} \nu .
$$
Proof. First Jensen's inequality, then Eq. (21).
$$
\begin{array}{l} \operatorname {d e c} _ {\epsilon} ^ {a c} (f) = \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \mu \Delta \nu \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2} \\ = \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \mu \Delta_ {f} \nu + \sum_ {g \in \mathcal {M}} \sum_ {\pi \in \Pi} \nu_ {g} \mu_ {\pi} (r _ {f} (\pi) - r _ {g} (\pi)) \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2} \\ \leq \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \mu \Delta_ {f} \nu + \sqrt {\mu I _ {f} \nu} \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2} \\ \leq \epsilon + \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \mu \Delta_ {f} \nu \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2} \\ \leq \epsilon + \operatorname {d e c} _ {\epsilon} ^ {a c, f} (f) \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2}, \\ \end{array}
$$
where the first inequality is by Lemma 7. Also, by lower bounding the sum in the second inequality by  $-\sqrt{\mu I_f \nu}$  we get left inequality.
Proof of Lemma 2. For the first inequality, using the definition of  $\mathrm{dec}_{\epsilon}^{ac}(f,\Delta_f)$  and the AM-GM inequality:
$$
\begin{array}{l} \operatorname{dec}_{\epsilon}^{ac,f}(f) = \min_{\lambda \geq 0}\max_{\nu \in \mathcal{P}(\mathcal{M})}\min_{\mu \in \mathcal{P}(\Pi)}\mu \Delta_{f}\nu -\lambda \mu I_{f}\nu +\lambda \epsilon^{2} \\ \leq \min  _ {\lambda > 0} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \min  _ {\mu \in \mathcal {P} (\Pi)} \frac {\left(\mu \Delta_ {f} \nu\right) ^ {2}}{4 \lambda \mu I _ {f} \nu} + \lambda \epsilon^ {2} (22) \\ = \min  _ {\lambda > 0} \frac {\Psi (f)}{4 \lambda} + \lambda \epsilon^ {2} = \epsilon \sqrt {\Psi (f)}. (23) \\ \end{array}
$$
Further, by Eq. (11) and Assumption 1 we have  $\mu_{\nu}^{\mathrm{TS}}\Delta_f\nu \leq \sqrt{K\sum_{g,h\in\mathcal{M}}\nu_g\nu_he_{\pi_h^*}(r_g - r_f)^2}\leq$ $\sqrt{K\mu_f^{\mathrm{TS}}I_f\nu}$ , which gives  $\Psi (f)\leq K$ . Plugging this into Eq. (23) gives the second inequality.
C.3 Regret bound for Algorithm 1 defined for  $\Delta_f$  and  $\mathrm{dec}_{\epsilon}^{ac,f}$
Lemma 8. If Assumption 1 holds, then the regret of ANYTIME-E2D (Algorithm 1) with  $\Delta$  replaced with  $\Delta_f$  is bounded as follows:
$$
R _ {n} \leq \max  _ {t \in [ n ], f \in \mathcal {M}} \left\{\frac {\operatorname {d e c} _ {\epsilon_ {t}} ^ {a c , f} (f)}{\epsilon_ {t} ^ {2}} \right\} \left(\sum_ {t = 1} ^ {n} \epsilon_ {t} ^ {2} + \operatorname {E s t} _ {n}\right) + \sqrt {n \operatorname {E s t} _ {n}}
$$
Proof. The proof follows along the lines of the proof of Theorem 1. The main difference is that when introducing  $\Delta_f$ , we get a term that captures the reward estimation error:
$$
\begin{array}{l} R _ {n} \leq \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \operatorname {d e c} _ {\epsilon_ {t}} ^ {a c} \left(\hat {f} _ {t}\right) + \lambda_ {t} \left(\mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} - \epsilon_ {t} ^ {2}\right) \right] + \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \mu_ {t} \left(r _ {\hat {f} _ {t}} - r _ {f ^ {*}}\right) \right] (24) \\ \leq \max  _ {t \in [ n ]} \max  _ {f \in \mathcal {M}} \left\{\frac {1}{\epsilon_ {t} ^ {2}} \operatorname {d e c} _ {\epsilon_ {t}} ^ {a c} (f) \right\} \sum_ {t = 1} ^ {n} \left(\epsilon_ {t} ^ {2} + \mathbb {E} \left[ \mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} \right]\right) + \sqrt {n \operatorname {E s t} _ {n}} (25) \\ \end{array}
$$
For the last inequality, we used Cauchy-Schwarz and Assumption 1 to bound the error term,
$$
\sum_ {t = 1} ^ {n} \mathbb {E} \left[ \mu_ {t} \left(r _ {\hat {f} _ {t}} - r _ {f ^ {*}}\right) \right] \leq \sqrt {n \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \left(\mu_ {t} \left(r _ {\hat {f} _ {t}} - r _ {f ^ {*}}\right)\right) ^ {2} \right]} \leq \sqrt {n \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \mu_ {t} I _ {\hat {f} _ {t}} e _ {f} \right]} = \sqrt {n \operatorname {E s t} _ {n}}
$$
C.4 Generalized Information Ratio
The generalized information ratio [Lattimore and Gyorgy, 2021] for  $\mu \in \mathcal{P}(\Pi)$ ,  $\nu \in \mathcal{P}(\mathcal{M})$ , and  $\alpha > 1$  is defined as
$$
\Psi_ {\alpha , f} (\mu , \nu) = \frac {(\mu \Delta_ {f} \nu) ^ {\alpha}}{\mu I _ {f} \nu} \tag {26}
$$
For  $\alpha = 2$ , we get the standard information ratio introduced by Russo and Van Roy [2014] with  $\nu$  as a prior over the model class  $\mathcal{M}$ . Define  $\Psi_{\alpha}(f) = \max_{\nu \in \mathcal{M}} \min_{\mu \in \mathcal{P}(\Pi)} \Psi_{\alpha, f}(\mu, \nu)$ . To upper bound  $\mathrm{dec}_{\epsilon}^{ac}$ , we have the following lemma.
Lemma 9. For the reference model  $f$ , the ac-dec can be upper bounded as