SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Input: Hypothesis class  $\mathcal{M}$ , estimation oracle EST, sequence  $\epsilon_t \geq 0$ , data  $\mathcal{D}_0 = \emptyset$  for  $t = 1, 2, 3, \ldots$  do
Estimate  $\hat{f}_t = \mathrm{EST}(\mathcal{D}_{t - 1})$
3 Compute gap and information matrices,  $\Delta$  and  $I_{\hat{f}_t}\in \mathbb{R}^{\Pi \times \mathcal{M}}$
4  $\mu_t = \arg \min_{\mu \in \mathcal{P}(\Pi)} \max_{\nu \in \mathcal{P}(\mathcal{M})} \{\mu \Delta \nu : \mu I_{\hat{f}_t} \nu \leq \epsilon_t^2\}$
5 Sample  $\pi_t\sim \mu_t$  and observe  $y_{t}\sim M_{f^{*}}(\pi_{t})$
6 Append data  $\mathcal{D}_t = \mathcal{D}_{t - 1}\cup \{\left(\pi_t,y_t\right)\}$
depends on the estimation error (or estimation regret), defined as the sum over divergences of the observation distributions under the estimate  $\hat{f}_t$  and the true model  $f^*$ :
$$
\operatorname {E s t} _ {n} = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} \right] \tag {8}
$$
Intuitively, the estimation error is well-behaved if  $\hat{f}_t\approx f^*$ , since  $\mu_t I_{f^*}e_{f^*} = 0$ . Equation (8) is closely related to the total information gain used in the literature on information-directed sampling [Russo and Van Roy, 2014] and kernel bandits [Srinivas et al., 2010].
To bound the estimation error, Foster et al. [2021] rely on online density estimation (also, online regression or online aggregation) [Cesa-Bianchi and Lugosi, 2006, Chapter 9]. For finite  $\mathcal{M}$ , the default approach is the exponential weights algorithm (EWA), which we provide for reference in Appendix A. When using this algorithm, the estimation error always satisfies  $\mathrm{Est}_n \leq \log(|\mathcal{M}|)$ , see [Cesa-Bianchi and Lugosi, 2006, Proposition 3.1]. While these bounds extend to continuous model classes via standard covering arguments, the resulting algorithm is often not tractable without additional assumptions. For linear feedback models (Examples 2.1 and 2.2), one can rely on the more familiar ridge regression estimator, which, we show, achieves bounded estimation regret  $\mathrm{Est}_n \leq \mathcal{O}(d\log(n))$ . For further discussion, see Appendix A.1.
With this in mind, we state our main result.
Theorem 1. Let  $\lambda_t \geq 0$  be any sequence adapted to the filtration  $\mathcal{F}_t$ . Then the regret of ANYTIME-E2D (Algorithm 1) with input sequence  $\lambda_t$  satisfies for all  $n \geq 1$ :
$$
R _ {n} \leq \operatorname * {e s s} _ {t \in [ n ]} \left\{\frac {\operatorname* {d e c} _ {\epsilon_ {t} , \lambda_ {t}} ^ {a c} (\hat {f} _ {t})}{\epsilon_ {t} ^ {2}} \right\} \left(\sum_ {t = 1} ^ {n} \epsilon_ {t} ^ {2} + \operatorname {E s t} _ {n}\right)
$$
where we defined  $\mathrm{dec}_{\epsilon, \lambda}^{ac}(f) = \min_{\mu \in \mathcal{P}(\Pi)} \max_{\nu \in \mathcal{P}(\mathcal{M})} \mu \Delta \nu - \lambda (\mu I_f \nu - \epsilon^2)$ .
As an immediate corollary, we obtain a regret bound for Algorithm 1 where the sampling distribution  $\mu_t$  is chosen to optimize  $\mathrm{dec}_{\epsilon_t}^{ac}$  for any sequence  $\epsilon_t$ .
Corollary 1. The regret of ANYTIME-E2D (Algorithm 1) with input  $\epsilon_t \geq 0$  satisfies for all  $n \geq 1$ :
$$
R _ {n} \leq \max  _ {t \in [ n ], f \in \mathcal {M}} \left\{\frac {\operatorname {d e c} _ {\epsilon_ {t}} ^ {a c} (f)}{\epsilon_ {t} ^ {2}} \right\} \left(\sum_ {t = 1} ^ {n} \epsilon_ {t} ^ {2} + \operatorname {E s t} _ {n}\right)
$$
Importantly, the regret of Algorithm 1 is directly controlled by the worst-case DEC,  $\max_{f\in \mathcal{M}}\mathrm{dec}_{\epsilon}^{ac}(f)$ , and the estimation error  $\operatorname{Est}_n$ . It remains to set  $\epsilon_t^2$  (respectively  $\lambda_t$ ) appropriately. For a fixed horizon  $n$ , we let  $\epsilon_t^2 = \frac{\operatorname{Est}_n}{n}$ . With the reasonable assumption that  $\max_{f\in \mathcal{M}}\{\epsilon^{-2}\mathrm{dec}_{\epsilon}^{ac}(f)\}$  is non-decreasing in  $\epsilon$ , Corollary 1 reads
$$
R _ {n} \leq 2 n \max  _ {f \in \mathcal {M}} \left\{\operatorname {d e c} ^ {a c} \sqrt {\operatorname {E s t} _ {n} / n} (f) \right\}. \tag {9}
$$
This almost matches the lower bound  $R_{n} \geq \Omega (n\mathrm{dec}_{1 / \sqrt{n}}^{c}(\mathcal{F}))^{3}$  [Foster et al., 2023, Theorem 2.2], up to the estimation error and the beforehand mentioned gap between  $\mathrm{dec}_{\epsilon}^{c}$  and  $\mathrm{dec}_{\epsilon}^{ac}$ .
3Here,  $\mathrm{dec}_{\epsilon}^{c}(\mathcal{F}) = \max_{f\in \mathrm{co}(\mathcal{M})}\min_{\mu \in \mathcal{P}(\Pi)}\max_{g\in \mathcal{M}\cup \{f\}}\{\mu \Delta \nu :\mu I_{f}e_{g}\leq \epsilon^{2}\} .$
Setting	\( \mathbf{dec}_{\gamma}^{o} \)	\( \mathbf{dec}_{\epsilon}^{ac} \)
Multi-Armed Bandits	\( |\Pi|/γ \)	\( 2ε√|\Pi| \)
Linear Bandits	d/4γ	\( ε√d \)
Lipschitz Bandits	\( 2γ^{-\frac{1}{d+1}} \)	\( 2^{\frac{d+1}{d+2}}ε^{\frac{2}{d+2}} \)
Convex Bandits	\( \tilde{O}(d^4/γ) \)	\( \tilde{O}(\epsilon d^2) \)
Table 1: Comparison of  $\mathrm{dec}_{\gamma}^{o}$  and  $\mathrm{dec}_{\epsilon}^{ac}$  for different settings. Bounds between  $\mathrm{dec}_{\gamma}^{o}$  and  $\mathrm{dec}_{\epsilon}^{ac}$  can be converted using Eq. (7). Refined bounds for linear bandits with side-observations are in Lemma 4.
To get an anytime algorithm with essentially the same scaling as in Eq. (9), we set  $\epsilon_t^2 = \log(|\mathcal{M}|)/t$  for finite model classes, and  $\epsilon_t^2 = \frac{\beta_{\mathcal{M}}}{t}$  if  $\mathrm{Est}_t \leq \beta_{\mathcal{M}} \log(t)$  for  $\beta_{\mathcal{M}} > 0$ . For linear bandits,  $\mathrm{dec}_{\epsilon}^{ac} \leq \epsilon \sqrt{d}$  (see Section 3.3), and  $\mathrm{Est}_n \leq d \log(n)$ . Choosing  $\epsilon_t^2 = d/t$  recovers the optimal regret bound  $R_n \leq \tilde{\mathcal{O}}(d\sqrt{n})$  [Lattimore and Szepesvári, 2020a]. Alternatively, one can also choose  $\lambda_t$  by minimizing an upper bound on  $\max_{t \in [n], f \in \mathcal{M}} \left\{\mathrm{dec}_{\epsilon_t, \lambda_t}^{ac}(f) / \epsilon_t^2\right\}$ . For example, in linear bandits,  $\mathrm{dec}_{\epsilon_t, \lambda}^{ac} \leq \frac{d}{4\lambda} + \lambda \epsilon_t^2$  (see Table 1); hence, for  $\epsilon_t^2 = d/t$ , we can set  $\lambda_t = t/4$ . Further discussion and refined upper bound for linear feedback models are in Section 3.3.
Proof of Theorem 1. Let  $\mu_t^*$  and  $\nu_t^*$  be a saddle-point solution to the offset dec,
$$
\operatorname{dec}_{\lambda_{t}}^{o}(\hat{f}_{t}) = \min_{\mu \in \mathscr{P}(\Pi)}\max_{\nu \in \mathscr{P}(\mathcal{M})}\mu \Delta \nu -\lambda_{t}\mu I_{\hat{f}_{t}}\nu
$$
Note that  $\mu_t^*\Delta \nu_t^* -\lambda_t\mu_t^* I_f\nu_t^* \geq \mu_t^*\Delta e_f - \lambda_t\mu_t^* I_f e_f \geq 0$  which implies that  $\lambda_t\epsilon_t^2 \leq \mathrm{dec}_{\epsilon_t,\lambda_t}^{ac}$ . Next,
$$
\begin{array}{l} R _ {n} = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \mu_ {t} \Delta e _ {f ^ {*}} \right] = \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \mu_ {t} \Delta e _ {f ^ {*}} - \lambda_ {t} \left(\mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} - \epsilon_ {t} ^ {2}\right) + \lambda_ {t} \left(\mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} - \epsilon_ {t} ^ {2}\right) \right] \\ \leq \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \max  _ {g \in \mathcal {M}} \mu_ {t} \Delta_ {\hat {f} _ {t}} e _ {g} - \lambda_ {t} \left(\mu_ {t} I _ {\hat {f} _ {t}} e _ {g} - \epsilon_ {t} ^ {2}\right) + \lambda_ {t} \left(\mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} - \epsilon_ {t} ^ {2}\right) \right] \\ = \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \mu \Delta \nu - \lambda_ {t} \left(\mu I _ {\hat {f} _ {t}} \nu - \epsilon_ {t} ^ {2}\right) + \lambda_ {t} \left(\mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} - \epsilon_ {t} ^ {2}\right) \right] \\ \end{array}
$$
So far, we only introduced the saddle point problem by maximizing over  $f^{*}$ . The last equality is by our choice of  $\lambda_{t}$  and  $\mu_t$ , and noting that  $\nu \in \mathcal{P}(\mathcal{M})$  can always be realized as a Dirac. Continuing,
$$
\begin{array}{l} R _ {n} \leq \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \operatorname {d e c} _ {\epsilon_ {t}, \lambda_ {t}} ^ {a c} (\hat {f} _ {t}) + \lambda_ {t} \left(\mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} - \epsilon_ {t} ^ {2}\right) \right] \\ \stackrel {(i)} {\leq} \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \operatorname {d e c} _ {\epsilon_ {t}, \lambda_ {t}} ^ {a c} (\hat {f} _ {t}) + \frac {1}{\epsilon_ {t} ^ {2}} \operatorname {d e c} _ {\epsilon_ {t}, \lambda_ {t}} ^ {a c} (\hat {f} _ {t}) \mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} \right] \\ \stackrel {(i i)} {\leq} \operatorname * {e s s} _ {t \in [ n ]} \max  _ {f \in \mathcal {M}} \left\{\frac {1}{\epsilon_ {t} ^ {2}} \mathrm {d e c} _ {\epsilon_ {t}, \lambda_ {t}} ^ {a c} (f) \right\} \sum_ {t = 1} ^ {n} \left(\epsilon_ {t} ^ {2} + \mathbb {E} \left[ \mu_ {t} I _ {\hat {f} _ {t}} e _ {f ^ {*}} \right]\right) \\ \end{array}
$$
We first drop the negative term in (i) and use the beforehand stated fact that  $\lambda_t\epsilon_t^2 \leq \mathrm{dec}_{\epsilon_t,\lambda_t}^{ac}(\hat{f}_t)$ . The last step, (ii), is taking the maximum out of the sum.
3.3 Certifying Upper Bounds
As shown by Corollary 1, the regret of Algorithm 1 scales directly with the  $\mathrm{dec}_{\epsilon}^{ac}$ . For analysis purposes, it is however useful to compute upper bounds on the  $\mathrm{dec}_{\epsilon}^{ac}$  to verify the scaling w.r.t. parameters of interest. Via the equivalence Eq. (7), bounds on the  $\mathrm{dec}_{\lambda}^{o}$  directly translate to the  $\mathrm{dec}_{\epsilon}^{ac}$  (see Table 1). For a detailed discussion of upper bounds in various models, we refer to Foster et al. [2021]. Below, we highlight three connections that are directly facilitated by the  $\mathrm{dec}_{\epsilon}^{ac}$ .
To this end, we first introduce a variant of the  $\mathrm{dec}_{\epsilon}^{ac}$  where the gap function depends on  $f$ :
$$
\operatorname {d e c} _ {\epsilon} ^ {a c, f} (f) = \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \mu \Delta_ {f} \nu \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2}, \tag {10}
$$
where  $\Delta_f(\pi, g) = r_g(\pi_g^*) - r_f(\pi)$ . We remark that for distributions  $\nu \in \mathcal{P}(\mathcal{M})$  and  $\mu \in \mathcal{P}(\Pi)$ , the gap  $\Delta_f$  can be decoupled,  $\mu \Delta_f \nu = \delta_f \nu + \mu \Delta_f e_f$ , where we defined  $\delta_f(g) = r_g(\pi_g^*) - r_f(\pi_f^*)$ . The following assumption implies that the observations for a decision  $\pi$  are at least as informative as observing the rewards.
Assumption 1 (Reward Data Processing). The rewards and information matrices are related via the following data-processing inequality that holds for any  $\mu \in \mathcal{P}(\Pi)$ :
$$
| \mathbb {E} _ {\pi \sim \mu} [ r _ {f} (\pi) - r _ {g} (\pi) ] | \leq \sqrt {\mathbb {E} _ {\pi \sim \mu} [ D (M _ {f} (\pi) \| M _ {g} (\pi)) ]}
$$
The next lemma shows that under Assumption 1,  $\mathrm{dec}_{\epsilon}^{ac}(f)$  and  $\mathrm{dec}_{\epsilon}^{ac,f}(f)$  are essentially equivalent, at least for the typical worst-case bounds where  $\max_{f\in \mathcal{M}}\mathrm{dec}_{\epsilon}^{ac}(f)\geq \Omega (\epsilon)$ .
Lemma 1. If Assumption 1 holds, then
$$
\operatorname {d e c} _ {\epsilon} ^ {a c, f} (f) - \epsilon \leq \operatorname {d e c} _ {\epsilon} ^ {a c} (f) \leq \operatorname {d e c} _ {\epsilon} ^ {a c, f} (f) + \epsilon
$$
The proof is in Appendix C.1. We remark that Algorithm 1 where the sampling distribution is computed for  $\mathrm{dec}_{\epsilon}^{ac,f}(\hat{f}_t)$  and  $\Delta_f$  achieves a bound analogous to Theorem 1, as long as Assumption 1 holds. For details see Lemma 8 in Appendix C.
Upper Bounds via Decoupling First, we introduce the information ratio,
$$
\Psi_ {f} (\mu , \nu) = \frac {\left(\mu \Delta_ {f} \nu\right) ^ {2}}{\mu I _ {f} \nu}
$$
The definition is closely related to the Bayesian information ratio [Russo and Van Roy, 2016], where  $\nu$  takes the role of a prior over  $\mathcal{M}$ . The Thompson sampling distribution is  $\mu_{\nu}^{\mathrm{TS}} = \sum_{h\in \mathcal{M}}\nu_{h}e_{\pi_{h}^{*}}$ . The decoupling coefficient,  $\mathrm{dc}(f)$ , [Zhang, 2022, Definition 1] is defined as the smallest number  $K\geq 0$ , such that for all distributions  $\nu \in \mathcal{P}(\mathcal{M})$ ,
$$
\mu_ {\nu} ^ {\mathrm {T S}} \Delta_ {f} \nu \leq \inf  _ {\eta \geq 0} \left\{\eta \sum_ {g, h \in \mathcal {M}} \nu_ {g} \nu_ {h} e _ {\pi_ {h} ^ {*}} \left(r _ {g} - r _ {f}\right) ^ {2} + \frac {K}{4 \eta} \right\} = \sqrt {K \sum_ {g , h \in \mathcal {M}} \nu_ {g} \nu_ {h} e _ {\pi_ {h} ^ {*}} \left(r _ {g} - r _ {f}\right) ^ {2}} \tag {11}
$$
The next lemma provides upper bounds on the  $\mathrm{dec}_{\epsilon}^{ac}(f)$  in terms of the information ratio, which is further upper-bounded by the decoupling coefficient.
Lemma 2. With  $\Psi (f) = \max_{\nu \in \mathcal{M}}\min_{\mu \in \mathcal{P}(\Pi)}\Psi_f(\mu ,\nu)$  and Assumption 1 satisfied, we have
$$
\operatorname {d e c} _ {\epsilon} ^ {a c, f} (f) \leq \epsilon \sqrt {\Psi (f)} \leq \epsilon \sqrt {\mathrm {d c} (f)}
$$
The proof follows directly using the AM-GM inequality, see Appendix C.2. By [Zhang, 2022, Lemma 2], this further implies  $\mathrm{dec}_{\epsilon}^{ac,f} \leq \epsilon \sqrt{d}$ . An analogous result for the generalized information ratio [Lattimore and Gyorgy, 2021] that recovers rates  $\epsilon^{\rho}$  for  $\rho \leq 1$  is given in Appendix C.4.
PAC to Regret Another useful way to upper bound the  $\mathrm{dec}_{\epsilon}^{ac,f}$  is via an analogous definition for the PAC setting [c.f. Eq. (10), Foster et al., 2023]:
$$
\operatorname {p a c} - \det  _ {\epsilon} ^ {a c, f} (f) = \min  _ {\mu \in \mathscr {P} (\Pi)} \max  _ {\nu \in \mathcal {M}} \delta_ {f} \nu \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2} \tag {12}
$$
Lemma 3. Under Assumption 1,
$$
\operatorname {d e c} _ {\epsilon} ^ {a c, f} (f) \leq \min  _ {p \in [ 0, 1 ]} \left\{\operatorname {p a c - d e c} _ {\epsilon p ^ {- 1 / 2}} ^ {a c, f} (f) + p \Delta_ {\max } \right\}
$$
The proof is given in Appendix B.1. Lemma 3 combined with Theorem 1 leads to  $\mathcal{O}(n^{2/3})$  upper bounds on the regret that are reminiscent of so-called globally observable games in linear partial monitoring [Kirschner et al., 2023].
Application to Linear Feedback Models To illustrate the techniques introduced, we compute a regret bound for Algorithm 1 for linear bandits with side-observations (Examples 2.1 and 2.2).
Lemma 4. For linear bandits with side-observations and divergence  $I_{f}(\pi ,g) = \| M_{\pi}(g - f)\|^{2}$ ,
$$
\operatorname {p a c - d e c} _ {\epsilon} ^ {a c, f} (f) \leq \min  _ {\mu \in \mathscr {P} (\Pi)} \max  _ {b \in \Pi} \epsilon \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} \leq \epsilon \sqrt {d}
$$
where  $V(\mu) = \sum_{\pi \in \Pi}\mu_{\pi}M_{\pi}M_{\pi}^{\top}$ . Moreover, denoting  $\Omega = \min_{\mu \in \mathcal{P}(\Pi)}\max_{b\in \Pi}\| \phi_b\|_{V(\mu)^{-1}}$
$$
\operatorname {d e c} _ {\epsilon} ^ {a c, f} (f) \leq \min  \left(\epsilon \sqrt {\Psi (f)}, 2 \epsilon^ {2 / 3} \Omega^ {1 / 3} \Delta_ {\max } ^ {1 / 3}\right)
$$
The proof is given in Appendix B.2. While in the worst-case for linear bandits, there is no improvement over the standard  $\mathcal{O}(d\sqrt{n})$  without further refinement or specification of the upper bounds, in the case of linear side-observations there is an improvement whenever  $\Omega \leq \max_{f\in \mathcal{M}}\Psi (f)$ . To exemplify the improvement, consider a semi-bandit with a "revealing" action  $\hat{\pi}$ , e.g.  $M_{\hat{\pi}} = \mathbf{1}_d$ . Here, the regret bound improves to  $R_{n}\leq \min \{d\sqrt{n},d^{1 / 3}n^{2 / 3}\}$ , since then  $\mathrm{pac - dec}_{\epsilon}^{ac,f}(f)\leq \epsilon$ . The corresponding improvement in the regime  $n\leq d^4$  might seem modest, but is relevant in high-dimensional and non-parametric models. Moreover, in (deep) reinforcement learning, high-dimensional models are commonly used and the learner obtains side information in the form of state observations. Therefore, it is plausible that the  $n^{2 / 3}$  rate is dominant even for a moderate horizon. Exploring this effect in reinforcement learning is therefore an important direction for future work.
Notably, this improvement is not observed by upper confidence bound algorithms and Thompson sampling, because both approaches discard informative but suboptimal actions early on [c.f. Latimore and Szepesvari, 2017], including the action  $\hat{\pi}$  in the example above. E2D for a constant offset parameter  $\lambda > 0$ , in principle, attains the better rate, but only if one pre-commits to a fixed horizon. Lastly, we note that a similar effect was observed for information-directed sampling in sparse high-dimensional linear bandits [Hao et al., 2020].
3.4 Computational Aspects
For finite model classes, Algorithm 1 can be readily implemented. Since almost no structure is imposed on the gap and information matrices of size  $|\Pi| \times |\mathcal{M}|$ , avoiding scaling with  $|\Pi| \cdot |\mathcal{M}|$  seems hardly possible without introducing additional assumptions. Even in the finite case, solving Eq. (3) is not immediate because the corresponding Lagrangian is not convex-concave. A practical approach is to solve the inner saddle point for Eq. (5) as a function of  $\lambda$ . Strong duality holds for the inner problem, and one can obtain a solution efficiently by solving the corresponding linear program using standard solvers. It then remains to optimize over  $\lambda \geq 0$ . This can be done, for example, via a grid search over the range  $[0, \max_{f \in \mathcal{M}} \epsilon^{-2} \mathrm{dec}_{\epsilon}^{ac}(f)]$ .
In the linear setting, the above is not satisfactory because most commonly  $\mathcal{M}$  is identified with parameters in  $\mathbb{R}^d$ . As noted before, ridge regression can be used instead of online aggregation while preserving the optimal scaling of the estimation error (see Appendix A.1). The next lemma further shows that the saddle point problem Eq. (3) can be rewritten to only scale with the size of the decision set  $|\Pi|$ .
Lemma 5. Consider linear bandits with side observations,  $\mathcal{M} = \mathbb{R}^d$  and quadratic divergence,  $I_{f}(\pi ,g) = \| M_{\pi}(g - f)\|^{2}$ , and denote  $\phi_{\mu} = \sum_{\pi \in \Pi}\mu_{\pi}\phi_{\pi}$  and  $V(\mu) = \sum_{\pi \in \Pi}\mu_{\pi}M_{\pi}^{\top}M_{\pi}$ . Then
$$
\mathbf {d e c} _ {\epsilon} ^ {a c, f} (\hat {f} _ {t}) = \min  _ {\lambda \geq 0} \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {b \in \Pi} \langle \phi_ {b} - \phi_ {\mu}, \hat {f} _ {t} \rangle + \frac {1}{4 \lambda} \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} ^ {2} + \lambda \epsilon^ {2}
$$
Moreover, the objective is convex in  $\mu \in \mathcal{P}(\Pi)$ .
The proof is a straightforward calculation provided in Appendix C.5. Note that the saddle point expression is analogous to Eq. (5), and in fact, one can linearize the inner maximization over  $\mathcal{P}(\Pi)$  such that the inner saddle point becomes convex-concave. This leads to expressions equivalent to Eqs. (4) and (6), albeit the objective is no longer linear in  $\mu \in \mathcal{P}(\Pi)$ . We use Lemma 5 to employ the same strategy as before: As a function of  $\lambda \geq 0$ , solve the inner problem of the expression in Lemma 5, for example, as a convex program with  $|\Pi|$  variables and  $|\Pi|$  constraints (Appendix D). Then all that remains is to solve a one-dimensional optimization problem over  $\lambda \in [0, \max_{f \in \mathcal{M}} \epsilon^{-2} \mathrm{dec}_{\epsilon}^{ac}(f)]$ . We demonstrate this approach in Appendix E to showcase the performance of E2D on simple examples.
We introduced ANYTIME-E2D, an algorithm based on the estimation-to-decisions framework for sequential decision-making with structured observations. The algorithm optimizes the average-constrained decision-making coefficient, which can be understood as a reparametrization of the corresponding offset version. The reparametrization facilitates an elegant anytime analysis and makes setting all remaining hyperparameters immediate. We demonstrate the improvement with a novel bound for linear bandits with side-observations, that is not attained by previous approaches. Lastly, we discuss how the algorithm can be implemented for finite and linear model classes. Nevertheless, much remains to be done. For example, one can expect the reference model to change very little from round to round, and therefore, it seems wasteful to solve Eq. (3) from scratch repetitively. Preferable instead would be an incremental scheme that iteratively computes updates to the sampling distribution.
Johannes Kirschner gratefully acknowledges funding from the SNSF Early Postdoc.Mobility fellowship P2EZP2_199781.
Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
N. Abe and P. M. Long. Associative reinforcement learning using linear probabilistic concepts. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML '99, pages 3-11, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2.
A. Agarwal and T. Zhang. Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. arXiv preprint arXiv:2206.07659, 2022.
M. G. Azar, I. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
S. Bubeck, O. Dekel, T. Koren, and Y. Peres. Bandit convex optimization:  $\backslash$  sqrtt regret in one dimension. In Conference on Learning Theory, pages 266-278. PMLR, 2015.
N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
F. Chen, S. Mei, and Y. Bai. Unified algorithms for rl with decision-estimation coefficients: No-regret, pac, and reward-free learning. arXiv preprint arXiv:2209.11745, 2022.
R. Combes, S. Magureanu, A. Proutiere, and C. Laroche. Learning to rank: Regret lower bounds and efficient algorithms. In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pages 231-244. ACM, 2015. ISBN 978-1-4503-3486-0.
R. Combes, S. Magureanu, and A. Proutiere. Minimal exploration in structured stochastic bandits. Advances in Neural Information Processing Systems, 30, 2017.
C. Dann, M. Mohri, T. Zhang, and J. Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems, 34:12040-12051, 2021.
R. Degenne, P. Menard, X. Shang, and M. Valko. Gamification of pure exploration for linear bandits. In International Conference on Machine Learning, pages 2432-2442. PMLR, 2020a.
R. Degenne, H. Shao, and W. Koolen. Structure adaptive algorithms for stochastic bandits. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2443-2452. PMLR, 13-18 Jul 2020b. URL https://proceedings.mlr.press/v119/degenne20b.html.