SlowGuess/NeurIPS_2023_OCR huggingface dataset:
dylanfoster@microsoft.com
Noah Golowich
nzg@mit.edu
Jian Qian
We consider the problem of interactive decision making, encompassing structured bandits and reinforcement learning with general function approximation. Recently, Foster et al. [12] introduced the Decision-Estimation Coefficient, a measure of statistical complexity that lower bounds the optimal regret for interactive decision making, as well as a meta-algorithm, Estimation-to-Decisions, which achieves upper bounds in terms of the same quantity. Estimation-to-Decisions is a reduction, which lifts algorithms for (supervised) online estimation into algorithms for decision making. In this paper, we show that by combining Estimation-to-Decisions with a specialized form of optimistic estimation introduced by Zhang [31], it is possible to obtain guarantees that improve upon those of Foster et al. [12] by accommodating more lenient notions of estimation error. We use this approach to derive regret bounds for model-free reinforcement learning with value function approximation, and give structural results showing when it can and cannot help more generally.
The theory of interactive decision making—ranging from bandits to reinforcement learning with function approximation—contains a variety of sufficient conditions for sample-efficient learning, [25, 17, 26, 11, 19, 7, 29, 18, 23, 3, 20, 10, 22, 9, 33], but necessary conditions have been comparatively unexplored. Recently, however, Foster et al. [12] introduced the Decision-Estimation Coefficient (DEC), a measure of statistical complexity which leads to upper and lower bounds on the optimal sample complexity for interactive decision making.
Regret bounds based on the Decision-Estimation Coefficient are achieved by Estimation-to-Decisions (E2D), a meta-algorithm which reduces the problem of interactive decision making to supervised online estimation. While the Decision-Estimation Coefficient leads to tight lower bounds on regret for many problem settings, the upper bounds in Foster et al. [12] can be suboptimal in certain situations due to the need to perform estimation with respect to Hellinger distance, a stringent notion of estimation error. When specialized to reinforcement learning, the guarantees for the E2D meta-algorithm in Foster et al. [12] are only tight for model-based settings (where function approximation is employed to model and estimate transition probabilities), and do not lead to meaningful guarantees for model-free settings with value function approximation. In this paper, we explore the prospect of developing tighter regret bounds suitable for model-free settings.
Contributions. We show that by combining Estimation-to-Decisions with optimistic online estimation, an elegant technique recently introduced by Zhang [31], it is possible to obtain regret bounds that improve upon Foster et al. [12] by accommodating weaker notions of estimation error. Our main contributions are:
- We introduce a new optimistic variant of the Decision-Estimation Coefficient, and show that a variant of Estimation-to-Decisions that incorporates optimistic estimation achieves regret bounds that scale with this quantity (Section 2). Using this approach, we derive the first regret bounds for Estimation-to-Decisions applied to model-free reinforcement learning with bilinear classes [11] (Section 2.3).
- We show that in general, whether or not optimistic estimation leads to improvement depends on the divergence with respect to which estimation is performed: For symmetric divergences, optimistic estimation offers no improvement, but for asymmetric divergences, including those found in reinforcement learning, the improvement can be drastic (Section 3). In addition, we highlight settings in which combining optimistic estimation with Estimation-to-Decisions offers provable improvement over previous approaches that apply the technique with posterior sampling [31].
Perhaps the most important aspect of our work is to elucidate the connection between the DEC framework and optimistic estimation, building foundations for further research into these techniques.
In what follows, we review the Decision-Estimation Coefficient and Estimation-to-Decisions metaalgorithm (Section 1.2), highlighting opportunities for improvement. In Section 2, we present our main results, including our application to model-free reinforcement learning. We close with discussion and structural results, highlighting situations in which optimistic estimation can and cannot help (Section 3).
We adopt the Decision Making with Structured Observations (DMSO) framework of Foster et al. [12], which is a general setting for interactive decision making that encompasses bandit problems (structured, contextual, and so forth) and reinforcement learning with function approximation.
The protocol consists of  $T$  rounds. For each round  $t = 1,\dots ,T$  ..
1. The learner selects a decision  $\pi^t\in \Pi$  , where  $\Pi$  is the decision space.
2. The learner receives a reward  $r^t \in \mathcal{R} \subseteq \mathbb{R}$  and observation  $o^t \in \mathcal{O}$  sampled via  $(r^t, o^t) \sim M^\star(\pi^t)$ , where  $M^\star: \Pi \to \Delta(\mathcal{R} \times \mathcal{O})$  is the underlying model.
Above,  $\mathcal{R}$  is the reward space and  $\mathcal{O}$  is the observation space. The model (conditional distribution)  $M^{\star}$  represents the underlying environment, and is unknown to the learner, but the learner is assumed to have access to a model class  $\mathcal{M} \subset (\Pi \to \Delta(\mathcal{R} \times \mathcal{O}))$  that is flexible enough to capture  $M^{\star}$ .
Assumption 1.1 (Realizability). The learner has access to a model class  $\mathcal{M}$  containing the true model  $M^{\star}$ .
The model class  $\mathcal{M}$  represents the learner's prior knowledge about the decision making problem, and allows one to appeal to estimation and function approximation. For structured bandit problems, models correspond to reward distributions, and  $\mathcal{M}$  encodes structure in the reward landscape. For reinforcement learning problems, models correspond to Markov decision processes (MDPs), and  $\mathcal{M}$  typically encodes structure in value functions or transition probabilities. We refer to Foster et al. [12] for further background.
For a model  $M \in \mathcal{M}$ ,  $\mathbb{E}^{M,\pi}[\cdot]$  denotes the expectation under the process  $(r,o) \sim M(\pi)$ ,  $f^M (\pi) := \mathbb{E}^{M,\pi}[r]$  denotes the mean reward function, and  $\pi_M := \arg \max_{\pi \in \Pi} f^M (\pi)$  denotes the optimal decision. We measure performance in terms of regret, which is given by  $\mathrm{Reg}_{\mathrm{DM}} := \sum_{t = 1}^{T} \mathbb{E}_{\pi^t \sim p^t} [f^{M^\star}(\pi_{M^\star}) - f^{M^\star}(\pi^t)]$ , where  $p^t$  is the learner's randomization distribution for round  $t$ .
Additional notation. For an integer  $n \in \mathbb{N}$ , we let  $[n]$  denote the set  $\{1, \ldots, n\}$ . For a set  $\mathcal{Z}$ , we let  $\Delta(\mathcal{Z})$  denote the set of all probability distributions over  $\mathcal{Z}$ . For a model class  $\mathcal{M}$ ,  $\operatorname{co}(\mathcal{M})$  denotes the convex hull. We write  $f = \widetilde{O}(g)$  to denote that  $f = O(g \cdot \max\{1, \operatorname{polylog}(g)\})$ , and use  $\lesssim$  as shorthand for  $a = O(b)$ .
1.2 Background: Estimation-to-Decisions and Decision-Estimation Coefficient
To motivate our results, this section provides a primer on the Estimation-to-Decisions meta-algorithm and the Decision-Estimation Coefficient. We refer to Foster et al. [12] for further background.
Algorithm 1 Estimation-to-Decisions (E2D) for General Divergences
1: parameters: Estimation oracle  $\mathbf{Alg}_{\mathrm{Est}}$ , Exp. parameter  $\gamma > 0$ , divergence  $D^{\pi}(\cdot \| \cdot)$ .  
2: for  $t = 1, 2, \dots, T$  do  
3: Compute estimate  $\widehat{M}^t = \mathbf{Alg}_{\mathrm{Est}}^t\left(\left(\pi^i, r^i, o^i\right)_{i=1}^{t-1}\right)$ .  
4:  $p^t \gets \arg \min_{p \in \Delta(\Pi)} \sup_{M \in \mathcal{M}} \mathbb{E}_{\pi \sim p}\left[f^M(\pi_M) - f^M(\pi) - \gamma \cdot D^{\pi}\left(\widehat{M}^t \| M\right)\right]$ . // Eq. (2).  
5: Sample decision  $\pi^t \sim p^t$  and update estimation oracle with  $(\pi^t, r^t, o^t)$ .
Online estimation. Estimation-to-Decisions (Algorithm 1) is a reduction that lifts algorithms for online estimation into algorithms for decision making. An online estimation oracle, denoted by  $\mathbf{Alg}_{\mathrm{Est}}$ , is an algorithm that, using knowledge of the class  $\mathcal{M}$ , estimates the underlying model  $M^{\star}$  from data in a sequential fashion. At each round  $t$ , given the data  $\mathcal{H}^{t-1} = (\pi^1, r^1, o^1), \ldots, (\pi^{t-1}, r^{t-1}, o^{t-1})$  observed so far, the estimation oracle computes an estimate  $\widehat{M}^t = \mathbf{Alg}_{\mathrm{Est}}^t \left( \left\{ (\pi^i, r^i, o^i) \right\}_{i=1}^{t-1} \right)$  for the true model  $M^{\star}$ .
To measure the estimation oracle's performance, we make use of a user-specified divergence-like function, which quantifies the discrepancy between models. Formally, we define a divergence-like function (henceforth, "divergence") as any function  $D: \Pi \times \operatorname{co}(\mathcal{M}) \times \operatorname{co}(\mathcal{M}) \to \mathbb{R}_+$ , with  $D^{\pi}(M \parallel M')$  representing the discrepancy between the models  $M$  and  $M'$  at the decision  $\pi$ . Standard choices used in past work [12, 13, 5, 15, 14, 27] include the squared error  $D_{\mathrm{sq}}^{\pi}(M, M') := (f^{M}(\pi) - f^{M'}(\pi))^{2}$  for bandit problems, and squared Hellinger distance<sup>1</sup>  $D_{\mathrm{H}}^{\pi}(M, M') := D_{\mathrm{H}}^{2}(M(\pi), M'(\pi))$  for RL, where for distributions  $\mathbb{P}$  and  $\mathbb{Q}$ ,  $D_{\mathrm{H}}^{2}(\mathbb{P}, \mathbb{Q}) := \int (\sqrt{d\mathbb{P}} - \sqrt{d\mathbb{Q}})^{2}$ . We then measure the estimation oracle's performance in terms of cumulative estimation error with respect to  $D$ , defined as
$$
\mathbf {E s t} ^ {D} := \sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi^ {t} \sim p ^ {t}} \left[ D ^ {\pi^ {t}} \left(\widehat {M} ^ {t} \| M ^ {\star}\right) \right], \tag {1}
$$
where  $p^t$  is the conditional distribution over  $\pi^t$  given  $\mathcal{H}^{t-1}$ . We make the following assumption on the algorithm's performance.
Assumption 1.2. At each time  $t \in [T]$ , the online estimation oracle  $\mathbf{Alg}_{\mathsf{Est}}$  returns, given  $(\pi^1, r^1, o^1), \ldots, (\pi^{t-1}, r^{t-1}, o^{t-1})$  with  $(r^i, o^i) \sim M^\star(\pi^i)$  and  $\pi^i \sim p^i$ , an estimator  $\widehat{M^t} : \Pi \to \Delta(\mathcal{R} \times \mathcal{O})$  such that  $\mathbf{Est}^D \leq \mathbf{Est}^D(T, \delta)$ , with probability at least  $1 - \delta$ , where  $\mathbf{Est}^D(T, \delta)$  is a known upper bound.
For the squared error, one can obtain  $\mathbf{Est}^{\mathbf{sq}}(T,\delta) := \mathbf{Est}^{D_{\mathbf{sq}}}(T,\delta) \lesssim \log(|\mathcal{F}_{\mathcal{M}}| / \delta)$ , where  $\mathcal{F}_{\mathcal{M}} := \{f^M \mid M \in \mathcal{M}\}$ , and for Hellinger distance, it is possible to obtain  $\mathbf{Est}^{\mathsf{H}}(T,\delta) := \mathbf{Est}^{D_{\mathsf{H}}}(T,\delta) \lesssim \log(|\mathcal{M}| / \delta)$ .
Estimation-to-Decisions. A general version of the E2D meta-algorithm is displayed in Algorithm 1. At each timestep  $t$ , the algorithm queries estimation oracle to obtain an estimator  $\widehat{M}^t$  using the data  $(\pi^1, r^1, o^1), \ldots, (\pi^{t-1}, r^{t-1}, o^{t-1})$  observed so far. The algorithm then computes the decision distribution  $p^t$  by solving a min-max optimization problem involving  $\widehat{M}^t$  and  $\mathcal{M}$  (as well as the divergence  $D$ ), and then samples the decision  $\pi^t$  from this distribution.
The Decision-Estimation Coefficient. The min-max optimization problem in Algorithm 1 is derived from the Decision-Estimation Coefficient (DEC), a complexity measure whose value, for a given scale parameter  $\gamma >0$  and reference model  $\overline{M}:\Pi \to \Delta (\mathcal{R}\times \mathcal{O})$ , is given by
$$
\operatorname {d e c} _ {\gamma} ^ {D} (\mathcal {M}, \bar {M}) = \inf  _ {p \in \Delta (\Pi)} \sup  _ {M \in \mathcal {M}} \mathbb {E} _ {\pi \sim p} \left[ f ^ {M} \left(\pi_ {M}\right) - f ^ {M} (\pi) - \gamma \cdot D ^ {\pi} \left(\bar {M} \| M\right) \right], \tag {2}
$$
with  $\mathbf{dec}_{\gamma}^{D}(\mathcal{M}) \coloneqq \sup_{\overline{M} \in \mathrm{co}(\mathcal{M})} \mathbf{dec}_{\gamma}^{D}(\mathcal{M}, \overline{M})$ . Informally, the DEC measures the best tradeoff between suboptimality  $(f^{M}(\pi_{M}) - f^{M}(\pi))$  and information gain (measured by  $D^{\pi}\big(\overline{M} \parallel M\big)$ ) that can be achieved by a decision distribution  $p$  in the face of a worst-case model  $M \in \mathcal{M}$ .
<sup>1</sup> When  $D^{\pi}(\cdot \parallel \cdot)$  is symmetric, we write  $D^{\pi}(\cdot ,\cdot)$  to make this explicit.
The main result of Foster et al. [12] shows that the regret of E2D is controlled by the DEC and the estimation oracle's cumulative error  $\mathbf{Est}^D$ . Let  $\widehat{\mathcal{M}}$  be any set for which  $\widehat{M}^t \in \widehat{\mathcal{M}}$  for all  $t$  almost surely. We have the following guarantee.
Theorem 1.1 (Foster et al. [12]). Algorithm 1 with exploration parameter  $\gamma >0$  guarantees that  $\mathbf{Reg}_{\mathrm{DM}}\leq \sup_{\overline{M}\in \widehat{\mathcal{M}}}\mathsf{dec}_{\gamma}^{D}(\mathcal{M},\overline{M})\cdot T + \gamma \cdot \mathbf{Est}^{D}$  almost surely.
For the special case of Hellinger distance, standard algorithms (exponential weights) achieve  $\mathbf{Est}^{\mathsf{H}}(T,\delta)\lesssim \log (|\mathcal{M}| / \delta)$  with  $\widehat{\mathcal{M}} = \operatorname {co}(\mathcal{M})$ , so that Theorem 1.1 gives (abbreviating  $\mathrm{dec}_{\gamma}^{\mathsf{H}}\equiv$ $\mathrm{dec}_{\gamma}^{D_{\mathsf{H}}}$ ):
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim \operatorname {d e c} _ {\gamma} ^ {\mathrm {H}} (\mathcal {M}) \cdot T + \gamma \cdot \log (| \mathcal {M} | / \delta). \tag {3}
$$
Opportunities for improvement. Foster et al. [12, 15] provide lower bounds on the regret for any decision making problem that have a similar expression to (3), showing that the Decision-Estimation Coefficient with Hellinger distance  $(\mathrm{dec}_{\gamma}^{\mathsf{H}}(\mathcal{M}))$  plays a fundamental role in determining the statistical complexity of decision making. However, these lower bounds do contain the estimation term  $\mathbf{Est}^{\mathsf{H}}(T,\delta) = \log (|\mathcal{M}| / \delta))$  appearing in (3), and thus capture the price of moving from estimation to decision making (as characterized by the DEC), but not the complexity of estimation itself.
In general, the dependence on  $\mathbf{Est}^{\mathsf{H}}(T,\delta) = \log (|\mathcal{M}| / \delta)$  in the upper bound (3) can render the bound loose. In reinforcement learning, working with Hellinger distance necessitates modeling transition probabilities. While this leads to optimal results in some settings, in general the optimal rates for Hellinger estimation error can be prohibitively large, even in settings where model-free (or, value-based) methods, which directly model value functions, are known to succeed; this drawback is shared by all subsequent work based on the DEC [13, 5, 15, 14, 27]. A natural solution is to replace Hellinger distance with a divergence (e.g., based on Bellman error) tailored to value function approximation, but naive choices for  $D$  along these lines render  $\mathrm{dec}_{\gamma}^{D}(\mathcal{M},\overline{M})$  too large to give meaningful guarantees, and Foster et al. [12, 15] left this as an open problem.
2 Estimation-to-Decisions with Optimistic Estimation
To derive improved regret bounds that address the shortcomings described in the prequel, we combine Estimation-to-Decisions with a specialized estimation approach introduced by Zhang [31] (see also Dann et al. [6], Agarwal and Zhang [1, 2], Zhong et al. [32]), which we refer to as optimistic estimation. We then use this approach to derive regret bounds for model-free reinforcement learning.
2.1 Optimistic Estimation
The idea of optimistic estimation is to augment the estimation objective with a bonus that biases the estimator toward models  $M \in \mathcal{M}$  for which the value  $f^{M}(\pi_{M})$  is large. We present a general version of the technique.
Let a divergence  $D^{\pi}(\cdot \parallel \cdot)$  be fixed. Following the development in Section 1.2, an optimistic estimation oracle  $\mathbf{Alg}_{\mathrm{Est}}$  is an algorithm which, at each step  $t$ , given the observations and rewards collected so far, computes an estimate for the underlying model. For technical reasons, it will be useful to consider randomized estimators (Foster et al. [12], Chen et al. [5]) that, at each round, produce a distribution  $\mu^t \in \Delta(\mathcal{M})$  over models. Such estimators take the form  $\mu^t = \mathbf{Alg}_{\mathrm{Est}}^t\left(\left\{(\pi^i, r^i, o^i)\right\}_{i=1}^{t-1}\right)$  where  $\mu^t \in \Delta(\mathcal{M})$ . For a parameter  $\gamma > 0$ , we define the optimistic estimation error as
$$
\mathbf {O p t E s t} _ {\gamma} ^ {D} := \sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi^ {t} \sim p ^ {t}} \mathbb {E} _ {\widehat {M} ^ {t} \sim \mu^ {t}} \left[ D ^ {\pi} \left(\widehat {M} ^ {t} \| M ^ {\star}\right) + \gamma^ {- 1} \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {\widehat {M} ^ {t}} \left(\pi_ {\widehat {M} ^ {t}}\right)\right) \right]. \tag {4}
$$
This quantity is similar to (1), but incorporates a bonus term  $\gamma^{-1}(f^{M^{\star}}(\pi_{M^{\star}}) - f^{\widehat{M}^{t}}(\pi_{\widehat{M}^{t}}))$ , which encourages the estimation algorithm to over-estimate the optimal value  $f^{M^{\star}}(\pi_{M^{\star}})$  for the underlying model  $\widetilde{M}^{\star}$ .
For simpler model classes, it is possible to improve upon (3) by moving from Hellinger distance to lenient notions of estimation error: In bandit problems with Gaussian rewards, it suffices to consider the  $D_{\mathrm{sq}}^{\pi}(M,\overline{M}):= (f^{M}(\pi) - f^{\overline{M}}(\pi))^{2}$ , which leads to upper bounds that scale with  $\log |\mathcal{F}_{\mathcal{M}}|\ll \log |\mathcal{M}|$  [12].
Algorithm 2 Optimistic Estimation-to-Decisions (E2DOPT)
1: params: Estimation oracle  $\mathbf{Alg}_{\mathrm{Est}}$ , Exp. param.  $\gamma > 0$ , divergence  $D$  with suff. stat. space  $\Psi$ .
2: for  $t = 1,2,\dots ,T$  do
3: Receive randomized estimator  $\mu^t\in \Delta (\Psi) = \mathbf{A}\mathbf{lg}_{\mathrm{Est}}^t\big(\big(\big(\pi^i,r^i,o^i\big)\big\}_{i = 1}^{t - 1}\big)$ .
4: Compute
$$
p ^ {t} \leftarrow \underset {p \in \Delta (\Pi)} {\arg \min } \sup  _ {M \in \mathcal {M}} \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {\widehat {\psi} \sim \mu^ {t}} \left[ f ^ {\widehat {\psi}} \left(\pi_ {\widehat {\psi}}\right) - f ^ {M} (\pi) - \gamma \cdot D ^ {\pi} \left(\widehat {\psi} \| M\right) \right] / / \text {E q .} \tag {6}.
$$
5: Sample decision  $\pi^t\sim p^t$  and update estimation oracle with  $(\pi^t,r^t,o^t)$
Assumption 2.1. At each time  $t \in [T]$ , the optimistic estimation oracle  $\mathbf{Alg}_{\mathrm{Est}}$  returns, given  $(\pi^1, r^1, o^1), \ldots, (\pi^{t-1}, r^{t-1}, o^{t-1})$  with  $(r^i, o^i) \sim M^\star(\pi^i)$  and  $\pi^i \sim p^i$ , a randomized estimator  $\mu^t \in \Delta(\mathcal{M})$  such that  $\mathbf{OptEst}_\gamma^D \leq \mathbf{OptEst}_\gamma^D(T, \delta)$ , with w.p.  $1 - \delta$ , where  $\mathbf{OptEst}_\gamma^D(T, \delta)$  is a known upper bound.
For the case of contextual bandits, Zhang [31] proposes an augmented version of the exponential weights algorithm which, for a learning rate parameter  $\eta > 0$ , sets  $\mu(M) \propto \exp\left(-\eta\left(L^{t}(f^{M}) - \gamma^{-1}f^{M}(\pi_{M})\right)\right)$ , where  $L^{t}(f^{M})$  is the squared prediction error for the rewards observed so far. This method achieves  $\mathbb{E}\left[\mathbf{OptEst}_{\gamma}^{\mathbf{sq}}\right] \lesssim \log(|\mathcal{F}_{\mathcal{M}}|) + \sqrt{T\log|\mathcal{F}_{\mathcal{M}}| / \gamma}$ , and Zhang [31] combines this estimator with posterior sampling to achieve optimal contextual bandit regret. Agarwal and Zhang [1], Zhong et al. [32], Agarwal and Zhang [2] extend this development to reinforcement learning, also using posterior sampling as the exploration mechanism. In what follows, we combine optimistic estimation with Estimation-to-Decisions, which provides a universal mechanism for exploration. Beyond giving guarantees which were previously out of reach for E2D (Section 2.3), this approach generalizes and subsumes posterior sampling, and can succeed in situations where posterior sampling fails (Section 3).
Remark 2.1. For the non-optimistic estimation error  $\mathbf{Est}^D$ , it is possible to obtain low error for well-behaved losses such as the square loss and Hellinger distance without the use of randomization by appealing to improper mixture estimators (e.g., Foster et al. [12]). We show in Section 3 that for such divergences, randomization does not lead to statistical improvements. For the optimistic estimation error (4), randomization is essential due to the presence of the term  $\gamma^{-1}(f^{M^{\star}}(\pi_{M^{\star}}) - f^{\widehat{M}^{t}}(\pi_{\widehat{M}^{t}})$ .
Sufficient statistics. Before proceeding, we note that many divergences of interest have the useful property that they depend on the estimated model  $\widehat{M}$  only through a "sufficient statistic" for the model class under consideration. Formally, there exists a sufficient statistic space  $\Psi$  and sufficient statistic  $\psi : \mathcal{M} \to \Psi$  with the property that we can write (overloading notation)
$$
D ^ {\pi} (M \parallel M ^ {\prime}) = D ^ {\pi} (\psi (M) \parallel M ^ {\prime}), \quad f ^ {M} (\pi) = f ^ {\psi (M)} (\pi), \quad \text {a n d} \quad \pi_ {M} = \pi_ {\psi (M)}
$$
for all models  $M, M'$ . In this case, it suffices for the online estimation oracle to directly estimate the sufficient statistic by producing a randomized estimator  $\mu^t \in \Delta(\Psi)$ . We measure performance via
$$
\mathbf {O p t E s t} _ {\gamma} ^ {D} := \sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi^ {t} \sim p ^ {t}} \mathbb {E} _ {\widehat {\psi} ^ {t} \sim \mu^ {t}} \left[ D ^ {\pi^ {t}} \left(\widehat {\psi} ^ {t} \| M ^ {\star}\right) + \gamma^ {- 1} \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {\widehat {\psi} ^ {t}} \left(\pi_ {\widehat {\psi} ^ {t}}\right)\right) \right] \tag {5}
$$
Examples include bandit problems, where one may use squared estimation error  $D_{\mathrm{sq}}^{\pi}(\cdot ,\cdot)$  and take  $\psi (\mathcal{M}) = f^{M}$ , and model-free reinforcement learning, where we show that by choosing the divergence  $D$  appropriately, one can use  $Q$ -value functions as a sufficient statistic. Note that we only focus on sufficient statistics for the first argument to  $D^{\pi}(\cdot \| \cdot)$ , since this is the quantity we wish to estimate.
2.2 Algorithm and Main Result
We provide an optimistic variant of the E2D meta-algorithm (E2D.Opt) in Algorithm 2. At each timestep  $t$ , the algorithm calls the estimation oracle to obtain a randomized estimator  $\mu^t$  using the data
<sup>3</sup>Dann et al. [6] also apply the optimistic estimation idea to model-free reinforcement learning, but do not provide online estimation guarantees.
$(\pi^1, r^1, o^1), \ldots, (\pi^{t-1}, r^{t-1}, o^{t-1})$  collected so far. The algorithm then uses the estimator to compute a distribution  $p^t \in \Delta(\Pi)$  and samples  $\pi^t$  from this distribution, with the main change relative to Algorithm 1 being that the minimax problem in Algorithm 2 is derived from an "optimistic" variant of the DEC, which we refer to as the Optimistic Decision-Estimation Coefficient. For  $\mu \in \Delta(\mathcal{M})$ , define
$$
\mathbf {o} - \operatorname {d e c} _ {\gamma} ^ {D} (\mathcal {M}, \mu) = \inf  _ {p \in \Delta (\Pi)} \sup  _ {M \in \mathcal {M}} \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {\overline {{M}} \sim \mu} \left[ f ^ {\overline {{M}}} \left(\pi_ {\overline {{M}}}\right) - f ^ {M} (\pi) - \gamma \cdot D ^ {\pi} \left(\overline {{M}} \| M\right) \right]. \tag {6}
$$