SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shinji Ito
NEC Corporation, RIKEN AIP
i-shinji@nec.com
Kei Takemura
NEC Corporation
kei_takemura@nec.com
In this paper, we consider how to construct best-of-both-worlds linear bandit algorithms that achieve nearly optimal performance for both stochastic and adversarial environments. For this purpose, we show that a natural approach referred to as exploration by optimization [Lattimore and Szepesvári, 2020b] works well. Specifically, an algorithm constructed using this approach achieves  $O(d\sqrt{T\log T})$ -regret in adversarial environments and  $O(\frac{d^2\log T}{\Delta_{\min}})$ -regret in stochastic environments. Symbols  $d$ ,  $T$  and  $\Delta_{\mathrm{min}}$  here represent the dimensionality of the action set, the time horizon, and the minimum sub-optimality gap, respectively. We also show that this algorithm has even better theoretical guarantees for important special cases including the multi-armed bandit problem and multitask bandits.
The linear bandit problem [Abernethy et al., 2008] is a fundamental sequential decision-making problem in which a player chooses a  $d$ -dimensional vector  $a_{t}$  from a given action set  $\mathcal{A} \subseteq \mathbb{R}^{d}$  and incurs loss  $f_{t}$ . The expectation of  $f_{t}$  is assumed to be expressed as an inner product of  $\ell_{t}$  and  $a_{t}$  with an unknown loss vector  $\ell_{t} \in \mathbb{R}^{d}$  in each round  $t = 1, 2, \ldots, T$ . This paper focuses on the (expected) regret defined as  $R_{T} = \max_{a^{*} \in \mathcal{A}} \mathbf{E}[\sum_{t=1}^{T} \langle \ell_{t}, a - a^{*} \rangle]$ . It is known that the order of achievable regret can vary greatly depending on the structure of the environment that generates losses. For example, in stochastic environments, i.e., if the loss vector  $\ell_{t}$  is fixed at  $\ell^{*}$  for all  $t$ , there exists an algorithm that achieves regret of  $O(\log T)$ . By way of contrast, for adversarial environments in which the loss vector changes arbitrarily, the optimal regret bound is  $\Theta(\sqrt{T})$  [Abernethy et al., 2008, Bubeck et al., 2012]. While this clearly suggests the importance of choosing an algorithm that is well-suited to the environment, in real-world applications it is often quite difficult, if not virtually impossible, to know the type of environment in advance.
One promising approach to this difficulty is to apply best-of-both-worlds (BOBW) algorithms [Bubeck and Slivkins, 2012] that work nearly optimally both for stochastic and adversarial environments. The first BOBW algorithm for linear bandit problems was proposed by Lee et al. [2021]; it achieves  $O(\sqrt{dT\log(T)\log(T|\mathcal{A}|)})$ -regret in adversarial environments and  $O(c^{*}\log (T|\mathcal{A}|)\log T)$ -regret in stochastic environments, where  $c^* = c(\mathcal{A},\ell^*)$  represents an instance-dependent quantity characterizing the instance-optimal asymptotic bound for stochastic environments [Lattimore and Szepesvari, 2017]. This algorithm by Lee et al. [2021] also has the notable advantage that these regret guarantees hold with high probability. On the other hand, their regret bound for stochastic environments includes an extra log  $T$  factor. $^1$  This issue of extra log  $T$  factors has been resolved in a recent study by Dann et al. [2023b]. They have proposed algorithms achieving regret bounds with optimal dependency on  $T$ , which are summarized here in Table 1. Their regret bounds for stochastic environments depend
<sup>1</sup>As shown in Appendix D by Lee et al. [2021], this extra log  $T$  factor is inevitable as long as we pursue high-probability BOBW regret bounds.
Table 1: Regret bounds for linear bandits
Reference	Adversarial	Stochastic
Lattimore and Szepesvari [2017]	c* log T + o(log T)
Bubeck et al. [2012]	O(√dT log(|A|))
Lee et al. [2021]	O(√dT log(T) log(T|A|))	O(c* log(T|A|) log T)
Dann et al. [2023b, Cor.7]	O(√d2T log T)	O(d2logT/Δmin)
Dann et al. [2023b, Cor.12]	O(√dT log(|A|))	O(dlog(|A|) log T/Δmin)
[This work, Theorem 1]	O(√αdT log T)	O(βd log T)
Table 2: Bounds on parameters  $\alpha$  and  $\beta$
Setting	Action set	α	β
Corollary 1	General	A ⊆ Rd	O(d)	O(d/Δmin)
Corollary 2	Multi-armed	A = Ad := {e1, . . ., ed}	O(1)	O(1/Δmin)
Corollary 3	Hypercube	A = {0,1}d	O(d)	O(1/Δmin)
Corollary 3	Multitask	A = Ad1 × . . . × Adm	O(m)	O(1/Δmin)
on the minimum sub-optimality gap  $\Delta_{\mathrm{min}}$ . Note that we have  $c^{*} \leq O(d / \Delta_{\mathrm{min}})$  [Lee et al., 2021, Lemma 16] and that this inequality can be arbitrarily loose [Lattimore and Szepesvari, 2017]. The study by Dann et al. [2023b] involves the proposal of a general reduction technique, which has the remarkable advantage of being applicable to a variety of sequential decision-making problems in addition to linear bandit problems, though it does tend to complicate the structure and implementation of the algorithms.
The main contribution of this paper is to show that BOBW algorithms can be constructed via a (conceptually) simple approach referred to as exploration by optimization (EXO) [Lattimore and Szepesvári, 2020b, Lattimore and Gyory, 2021, Foster et al., 2022]. In this approach, we update the reference distribution  $q_{t}$  using the exponential weight method and then compute sampling distribution  $p_{t}$  for action  $a_{t}$  (by modifying  $q_{t}$  moderately) and loss estimator  $g_{t}$  by solving an optimization problem so that an upper bound on regret is minimized. As shown by Lattimore and Gyory [2021] and Foster et al. [2022], this natural approach has a connection to both the information ratio [Russo and Van Roy, 2014] and the decision-estimation coefficient [Foster et al., 2021], and it achieves nearly optimal worst-case regret bounds for some online decision-making problems. These existing studies on EXO, however, focus on adversarial environments, and not much has yet been shown about the potential of its working for stochastic environments. In this paper, we show that the EXO approach also works for stochastic environments to achieve  $O(\log T)$ -regret.
The regret bounds with the EXO approach and in existing studies are summarized in Table 1. This paper shows the regret bounds of  $O(\sqrt{\alpha dT\log T})$  in adversarial environments and of  $O(\beta d\log T)$  in stochastic environments, where  $\alpha$  is a parameter that depends on  $\mathcal{A}$  and  $\beta$  is a parameter depending on  $\mathcal{A}$  and  $\ell^{*}$ . Bounds on  $\alpha$  and  $\beta$  shown in this paper are summarized in Table 2. For general action sets, the EXO approach reproduces the regret bound shown by Dann et al. [2023b, Corollary 7]. Further, for some special cases, such as multi-armed bandits and hypercube linear bandits, it achieves improved regret bounds that are tight in stochastic regimes parametrized by the sub-optimality gap  $\Delta_{\mathrm{min}}$ .
To show BOBW regret bounds, we consider here the EXO approach combined with the continuous exponential weights method [Hoeven et al., 2018], in which we update reference distributions  $q_{t}$  over a convex set. In our regret analysis of the exponential weight, the ratio of the variance of the sampling distribution  $p_{t}$  to the variance of  $q_{t}$  plays an important role: the larger the former, the better the guarantee obtained. To show  $O(\log T)$ -regret bounds for stochastic environments, we exploit the fact that  $q_{t}$  is a log-concave distribution [Lovasz and Vempala, 2007]. Intuitively speaking, as any log-concave distribution has most of its weight concentrated around its mean (see, e.g., [Lovasz and Vempala, 2007, Lemma 5.17]), we may choose sampling distribution  $p_{t}$  so that it has a larger variance than  $q_{t}$  which leads to improved regret bounds.
Our regret analysis uses the self-bounding technique [Wei and Luo, 2018, Zimmert and Seldin, 2021] to show regret bounds for stochastic environments. One advantage of this technique is that a regret bound also leads to stochastic environments with adversarial corruptions or corrupted stochastic environments [Lykouris et al., 2018, Li et al., 2019]. As with existing algorithms based on the self-bounding technique, our algorithm achieves a regret bound of  $O(\alpha dT\log T + \sqrt{C\alpha dT\log T})$  for corrupted stochastic environments with the corruption level  $C \geq 0$ . More generally, these regret bounds are achieved for environments in adversarial regimes with self-bounding constraints [Zimmert and Seldin, 2021], details of which are provided here in Section 3.
Another contribution of this paper is to establish a connection between the EXO approach and the SCRiBLe algorithm [Abernethy et al., 2008, 2012] (and its extensions [Dann et al., 2023b, Ito and Takemura, 2023]). The SCRiBLe algorithm uses a follow-the-regularized-leader (FTRL) method with self-concordant barrier regularization to select a point in the action set, rather than a probability distribution. Abernethy et al. [2008] have shown that this approach achieves  $O(\sqrt{d^3 T})$ -regret bounds for adversarial environments. Recently, Dann et al. [2023b] and Ito and Takemura [2023] have shown that it can also achieve  $O\left(\frac{d^3 \log T}{\Delta_{\min}}\right)$ -regret in stochastic environments, through modification of the sampling distributions and loss estimators. To better understand the connection between these studies and the EXO approach, we propose a variant that we refer to as mean-oriented EXO and which includes EXO with the continuous exponential weight method as a special case. Given this correspondence, existing SCRiBLe-type algorithms can be interpreted as EXO-based methods in which the optimization is roughly solved with approximation ratios  $\geq \Omega(d)$ .
The results of this paper establish a connection between the framework of exploration by optimization and best-of-both-worlds regret guarantees. We would like to emphasize that there is potential to extend this result to a broader class of sequential decision-making beyond linear bandits, such as partial monitoring problems and episodic MDPs. In fact, the EXO framework can be applied to a general model of sequential decision-making with structured observation [Foster et al., 2022] and is known to have a connection to several key concepts in sequential decision-making, such as the Decision-Estimation Coefficients [Foster et al., 2021] and information-directed sampling [Lattimore and Gyorgy, 2021, Russo and Van Roy, 2014], which has been extensively applied to partial monitoring [Lattimore and Szepesvári, 2020b] and linear partial monitoring [Kirschner et al., 2020]. Building further relationships between these generic frameworks and BOBW regret analysis will be an important topic to work on in the future.
For adversarial linear bandit problems, various approaches have been considered, including algorithms based on self-concordant barriers (SCRiBLe) [Abernethy et al., 2008, 2012, Rakhlin and Sridharan, 2013], discrete exponential weights (Geometric Hedge) [Dani et al., 2008, Bubeck et al., 2012, Cesa-Bianchi and Lugosi, 2012], and continuous exponential weights (continuous Geometric Hedge) [Hazan and Karnin, 2016, Bubeck and Eldan, 2019], all of which can be interpreted as methods in the FTRL framework. We can see that continuous exponential weight methods can be interpreted as an FTRL approach with an entropic barrier, a  $d$ -self-concordant barrier [Chewi, 2021], which implies a close relationship between the SCRiBLe algorithm and the continuous Geometric Hedge, as Bubeck and Eldan [2019] have noted. One key difference lies in how the sampling distribution  $p_t$  is chosen; the SCRiBLe algorithm employs a sampling scheme supported on the Dikin ellipsoid while the continuous Geometric Hedge designs  $p_t$  by mixing the reference distribution  $q_t$  with some exploration basis (e.g., a G-optimal design distribution). As a result of this difference, the latter can achieve an  $O(d^{-1/2})$ -times better regret bound than the former. Recent studies by Dann et al. [2023b, Theorem 3] and Ito and Takemura [2023] have proposed new sampling schemes based on Dikin ellipsoid sampling to achieve BOBW regret bounds. This paper takes these approaches a step further, determining a sampling distribution itself on the basis of an optimization problem for minimizing regret, which leads to improved performance.
The self-bounding technique [Wei and Luo, 2018, Zimmert and Seldin, 2021] is known as a promising approach for designing and analyzing BOBW algorithms. The greatest impetus for the widespread use of this technique is considered to come from the Tsallis-INF algorithm [Zimmert et al., 2019] for the multi-armed bandit problem. Since the study on Tsallis-INF, FTRL methods with Tsallis-entropy regularizer (with some modification) have been used successfully in a variety of sequential
decision-making problems with limited (structured) observation, including combinatorial semi-bandits [Zimmert et al., 2019], bandits with switching costs [Rouyer et al., 2021, Amir et al., 2022], bandits with delayed feedback [Masoudian et al., 2022], online learning with feedback graphs [Erez and Koren, 2021, Rouyer et al., 2022].ueling bandits [Saha and Gaillard, 2022], and episodic MDPs [Jin and Luo, 2020, Jin et al., 2021, Dann et al., 2023a]. On the other hand, a few (rather limited number of) studies have constructed BOBW algorithms using regularizer functions other than the Tsallis entropy. For example, Ito [2021] proposed a BOBW multi-armed bandit algorithm by using log-barrier regularization equipped with adaptive learning rates. More recently, Shannon-entropy regularization with some specific adjustment methods for learning rates has been shown to provide BOBW regret guarantees for several problems including online learning with feedback graphs [Ito et al., 2022b], partial monitoring [Tsuchiya et al., 2023], and linear bandits [Kong et al., 2023]. It seems that these Shannon-entropy-based methods have the strength of being versatile while these have the weakness that their regret bounds for stochastic environments include extra  $O(\log T)$ -factors. The approach in this paper may appear to be related to the previous studies using the Shannon entropy [Ito et al., 2022b, Tsuchiya et al., 2023, Kong et al., 2023] as it employs exponential weight methods, but it is more related to the studies using log-barrier regularization [Ito, 2021]. In fact, both log barriers and continuous exponential weight can be captured in the category of self-concordant barriers as we will discuss in Sections 5 and 6. These also have in common the type of regret bounds:  $O(\log T)$  in stochastic environments and  $O(\sqrt{T\log T})$  in adversarial environments.
3 Problem setting
In linear bandit problems, the player is given an action set  $\mathcal{A} \subseteq \mathbb{R}^d$ , which is a closed bounded set of  $d$ -dimensional vectors. Without loss of generality, we assume that  $\mathcal{A}$  spans all vectors in  $\mathbb{R}^d$ . In each round  $t$ , the environment chooses a loss vector  $\ell_t \subseteq \mathbb{R}^d$  while the player selects an action  $a_t \in \mathcal{A}$  without knowing  $\ell_t$ . The player then gets feedback of  $f_t \in [-1,1]$  that is drawn from a distribution  $M_{a_t}$ , where the series of distributions  $M = (M_a)_{a \in \mathcal{A}}$  is in  $\mathcal{M}_{\ell_t} := \{(M_a)_{a \in \mathcal{A}} | \forall a \in \mathcal{A}, \mathbf{E}_{f \sim M_a}[f] = \langle \ell_t, a \rangle\}$ . The condition that the support of  $M_a$  is included in  $[-1,1]$  for all  $a \in \mathcal{A}$  implies that  $\langle \ell_t, a \rangle \in [-1,1]$  holds for all  $a \in \mathcal{A}$  and  $t \in [T]$ . In other words, it is assumed that  $\ell_t \in \mathcal{L}$  for any  $t$ , where we define  $\mathcal{L} \subseteq \mathbb{R}^d$  by  $\mathcal{L} = \{\ell \in \mathbb{R}^d | \forall a \in \mathcal{A}, |\langle \ell, a \rangle| \leq 1\}$ . The performance of the player is evaluated by means of regret defined as
$$
R _ {T} \left(a ^ {*}\right) = \mathbf {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, a _ {t} \right\rangle - \sum_ {t = 1} ^ {T} \left\langle \ell_ {t}, a ^ {*} \right\rangle \right], \quad R _ {T} = \sup  _ {a ^ {*} \in \mathcal {A}} R _ {T} \left(a ^ {*}\right). \tag {1}
$$
Regimes of environments In stochastic environments,  $\ell_t$  is assumed to be round-invariant, i.e., there exists  $\ell^* \in \mathcal{L}$  such that  $\ell_t = \ell^*$  holds for all  $t$ . In adversarial environments,  $\ell_t$  can be chosen in an adversarial manner depending on the history so far  $((\ell_s, a_s))_{s=1}^{t-1}$ . Stochastic environments with adversarial corruptions correspond to an intermediate setting between these two types of environments. These environments are parametrized by the corruption level  $C \geq 0$ , in which the loss vectors  $(\ell_t)$  are assumed to satisfy  $\mathbf{E}\left[\sum_{t=1}^{T} \max_{a \in \mathcal{A}} |\langle \ell_t - \ell^*, a \rangle|\right] \leq C$ . To capture these settings of environments, we define an adversarial regime with a  $(\ell^*, a^*, C, T)$ -self-bounding constraint [Zimmert and Seldin, 2021] in which the environments choose losses so that
$$
R _ {T} \left(a ^ {*}\right) \geq \mathbf {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \ell^ {*}, a _ {t} - a ^ {*} \right\rangle \right] - C \tag {2}
$$
holds for any algorithms. This regime includes (corrupted) stochastic environments. In fact, any corrupted stochastic environments with the true loss  $\ell^{*} \in \mathcal{L}$  and the corruption level  $C \geq 0$  are included in an adversarial regime with a  $(\ell^{*}, a^{*}, C, T)$ -self-bounding constraint. Our regret analysis for (corrupted) stochastic environments relies only on the condition of (2), as in some existing studies [Zimmert and Seldin, 2021, Dann et al., 2023b].
Reduction to convex action set Let  $\mathcal{X} = \operatorname{conv}(\mathcal{A})$  represent the convex hull of  $\mathcal{A}$ . We may consider the action set  $\mathcal{X}$  instead of  $\mathcal{A}$ , i.e., if we can construct an algorithm  $\mathrm{Alg}_{\mathcal{X}}$  for the problem with action set  $\mathcal{X}$ , we can convert it to an algorithm  $\mathrm{Alg}_{\mathcal{A}}$  for the problem with  $\mathcal{A}$ . In fact, for any
output  $a_{t}^{\prime}\in \mathcal{X}$  of  $\mathsf{A}\mathsf{lg}_{\mathcal{X}}$  , we can pick  $a_{t}\in \mathcal{A}$  so that  $\mathbf{E}[a_t|a_t^{\prime}] = a_t^{\prime},$  2 When defining  $a_{t}$  as the output of  $\mathsf{A}\mathsf{lg}_{\mathcal{A}}$  , we have  $\mathbf{E}[f_t|a_t^{\prime}] = \mathbf{E}[\langle \ell_t,a_t\rangle |a_t^{\prime}] = \langle \ell_t,a_t^{\prime}\rangle$  , which means that the expectation of the feedback  $f_{t}$  given  $a_{t}^{\prime}$  is indeed a linear function in  $a_{t}^{\prime}$  and that the regret for  $\mathsf{A}\mathsf{lg}_{\mathcal{A}}$  coincides with that for  $\mathsf{A}\mathsf{lg}_{\mathcal{X}}$
4 Exploration by optimization
4.1 Algorithm framework and regret analysis
Let  $\mathcal{P}(\mathcal{X})$  denote the set of all distributions over  $\mathcal{X}$ . For any distribution  $p\in \mathcal{P}(\mathbb{R}^d)$ , denote  $\mu (p) = \mathbf{E}_{x\sim p}[x]\in \mathbb{R}^d$ ,  $H(p) = \mathbf{E}_{x\sim p}[xx^{\top}]\in \mathbb{R}^{d\times d}$ , and  $V(p) = H(p) - \mu (p)\mu (p)^{\top}\in \mathbb{R}^{d\times d}$ . We consider algorithms that choose  $a_{t}$  following a sampling distribution  $p_t\in \mathcal{P}(\mathcal{X})$  in each round  $t$ . To determine  $p_t$ , we compute an (approximately) unbiased estimator  $g_{t}(x;a_{t},f_{t})$  of the loss function  $\langle \ell_t,x\rangle$ . Let  $\mathcal{G}$  be the set of all estimators  $g$  that are linear in the first variable, i.e.,  $\mathcal{G} = \{g:\mathcal{X}\times \mathcal{X}\times [-1,1]\to \mathbb{R}\mid \exists h:\mathcal{X}\times [-1,1]\to \mathbb{R}^d,g(x;a,f) = \langle h(a,f),x\rangle \}$ . For any distribution  $p\in \mathcal{P}(\mathcal{X})$ , let  $\mathcal{G}^{\mathrm{u}}(p)\subseteq \mathcal{G}$  represent the set of all unbiased estimators:
$$
\mathcal {G} ^ {\mathrm {u}} (p) = \left\{g \in \mathcal {G} \mid \forall \ell \in \mathcal {L}, \forall x \in \mathcal {X}, \forall M \in \mathcal {M} _ {\ell}, \quad \underset {a \sim p, f \sim M _ {a}} {\mathbf {E}} [ g (x; a, f) ] = \langle \ell , x \rangle \right\}. \tag {3}
$$
A typical choice of  $g$  given  $p$  is the estimator defined as follows:
$$
g (x; a, f) = \langle h (a, f), x \rangle , \quad \text {w h e r e} \quad h (a, f) = f \cdot (H (p)) ^ {- 1} a, \tag {4}
$$
where  $H(p)$  is assumed to be non-singular. This is an unbiased estimator, i.e.,  $g$  defined by (4) is an element of  $\mathcal{G}^{\mathrm{u}}$ . In fact, if  $M \in \mathcal{M}_{\ell}$ , we have  $\mathbf{E}_{a \sim p, f \sim M_a}[h(a, f)] = \mathbf{E}_{a \sim p}[(H(p))^{-1}aa^\top \ell] = (H(p))^{-1}H(p)\ell = \ell$ .
Using  $\{g_s\}_{s=1}^t$ , we set a reference distribution  $q_t$  over  $\mathcal{X}$  by the continuous exponential weights method as follows:  $q_t(x) \propto q_0(x) \exp(-\eta_t \sum_{s=1}^{t-1} g_s(x; a_s, f_s))$ , where  $q_0$  is an arbitrary initial distribution over  $\mathcal{X}$  and  $\eta_t$  is a learning rate such that  $\eta_1 \geq \eta_2 \geq \dots \geq \eta_T > 0$ . We may choose  $\eta_t$  depending on the historical actions and observations so far  $((a_s, f_s))_{s=1}^{t-1}$ . Let  $D_{\mathrm{KL}}(q', q)$  represent the Kullback-Leibler divergence of  $q'$  from  $q$ . Note that the exponential weights method can be interpreted as a follow-the-regularized-leader approach over  $\mathcal{P}$  with the regularizer function  $q \mapsto D_{\mathrm{KL}}(q, a_0)$ , i.e., the distribution  $q_t$  is a solution to the following optimization problem:
$$
q _ {t} \in \underset {q \in \mathcal {P} (\mathcal {X})} {\arg \min } \left\{\mathbf {E} _ {x \sim q} \left[ \sum_ {s = 1} ^ {t - 1} g _ {s} \left(x; a _ {s}, f _ {s}\right) \right] + \frac {1}{\eta_ {t}} D _ {\mathrm {K L}} (q, q _ {0}) \right\}. \tag {5}
$$
We may determine the sampling distribution  $p_t$  on the basis of  $q_t$  in an arbitrary way. The regret is then bounded as follows:
Lemma 1. For any distribution  $p^* \in \mathcal{P}(\mathcal{X})$  such that  $D_{\mathrm{KL}}(p^*, q_0) < \infty$ , the expected value of the regret  $R_T(a^*)$  for  $a^* \sim p^*$  is bounded by
$$
\mathbf {E} \left[ \sum_ {t = 1} ^ {T} \left(\langle \ell_ {t}, \mu (p _ {t}) - \mu (q _ {t}) \rangle + \operatorname {b i a s} (g _ {t}, p _ {t}, q _ {t}, \ell_ {t}) + \frac {1}{\eta_ {t}} \operatorname {s t a b} (\eta_ {t} g _ {t} (\cdot ; a _ {t}, f _ {t}), q _ {t})\right) + \frac {D _ {\mathrm {K L}} (p ^ {*} , q _ {0})}{\eta_ {T + 1}} \right],
$$
where  $\begin{array}{r}\mathrm{bias}(g,p,q,\ell) = \sup_{a^{*}\in \mathcal{X},M\in \mathcal{M}_{\ell}}\left\{\mathbf{E}_{a\sim p,x\sim q,f\sim M_a}[\langle \ell ,x - a^*\rangle -g(a^*;a,f) + g(x;a,f)]\right\} , \end{array}$
$$
\operatorname {s t a b} (g, q) = \sup  _ {q ^ {\prime} \in \mathcal {P} (\mathcal {X})} \left\{\mathbf {E} _ {x \sim q} [ g (x) ] - \mathbf {E} _ {x \sim q ^ {\prime}} [ g (x) ] - D _ {\mathrm {K L}} \left(q ^ {\prime}, q\right) \right\}. \tag {6}
$$
This follows from the standard analysis for exponential update methods or FTRL (see, e.g., [Lattimore and Szepesvári, 2020a, Exercise 28.12]), and all omitted proofs can be found in the appendix. For the case of the exponential weights, the stability term  $\operatorname{stab}(g, q)$  can be bounded as
$$
\operatorname {s t a b} (g, q) \leq \inf  _ {\bar {g} \in \mathbb {R}} \mathbf {E} _ {x \sim q} [ \phi (g (x) - \bar {g}) ], \quad \text {w h e r e} \quad \phi (y) = \exp (- y) + y - 1. \tag {7}
$$
<sup>2</sup>Such a distribution of  $a_{t}$  given  $a_{t}^{\prime}$  can be computed efficiently e.g., by the algorithm given in [Schrijver, 1998, Corollary 14.1g] or [Mirrokni et al., 2017].
The bias term  $\mathrm{bias}(g,p,q,\ell)$  corresponds to the bias of the estimator  $g(\cdot ;a,f)$  for the fucntion  $\langle \ell ,\cdot \rangle$  under the condition that  $a\sim p$  and  $\mathbf{E}[f|a] = \langle a,\ell \rangle$  . We can easily see that  $g\in \mathcal{G}^{\mathrm{u}}(p)$  implies bias  $(g,p,q,\ell) = 0$  for all  $q\in \mathcal{P}(\mathcal{X})$  and  $\ell \in \mathcal{L}$