SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and o-dec $_\gamma^D(\mathcal{M}) = \sup_{\mu \in \Delta(\mathcal{M})} \mathsf{o-dec}_\gamma^D(\mathcal{M}, \mu)$ . The Optimistic DEC has two differences from the original DEC. First, it is parameterized by a distribution  $\mu \in \Delta(\mathcal{M})$  rather than a reference model  $\overline{M}: \Pi \to \Delta(\mathcal{R} \times \mathcal{O})$ , which reflects the use of randomized estimators; the value in (6) takes the expectation over a reference model  $\overline{M}$  drawn from this distribution (this modification also appears in the randomized DEC introduced in Foster et al. [12]). Second, and more critically, the optimal value  $f^M(\pi_M)$  in (2) is replaced by the optimal value  $f^{\overline{M}}(\pi_{\overline{M}})$  for the (randomized) reference model. This seemingly small change is the main advantage of incorporating optimistic estimation, and makes it possible to bound the Optimistic DEC for certain divergences  $D$  for which the value of the unmodified DEC would otherwise be unbounded (cf. Section 3).
Remark 2.2. When the divergence  $D$  admits a sufficient statistic  $\psi : \mathcal{M} \to \Psi$ , for any distribution  $\mu \in \Delta(\mathcal{M})$ , if we define  $\nu \in \Delta(\Psi)$  via  $\nu(\psi) = \mu(\{M \in \mathcal{M} : \psi(M) = \psi\})$ , we have
$$
\mathsf{o - d e c} _ {\gamma} ^ {D} (\mathcal {M}, \mu) = \inf  _ {p \in \Delta (\Pi)} \sup  _ {M \in \mathcal {M}} \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {\psi \sim \nu} \left[ f ^ {\psi} \left(\pi_ {\psi}\right) - f ^ {M} (\pi) - \gamma \cdot D ^ {\pi} (\psi \| M) \right].
$$
In this case, by overloading notation slightly, we may write  $\mathsf{o - dec}_\gamma^D (\mathcal{M}) =$ $\sup_{\nu \in \Delta (\Psi)}\mathsf{o - dec}_{\gamma}^{D}(\mathcal{M},\nu).$
Main result. Our main result shows that the regret of Optimistic Estimation-to-Decisions is controlled by the Optimistic DEC and the optimistic estimation error for the oracle  $\mathbf{Alg}_{\mathrm{Est}}$ .
Theorem 2.1. For any  $\delta >0$ , Algorithm 2 ensures that with probability at least  $1 - \delta$
$$
\operatorname {R e g} _ {\mathrm {D M}} \leq o - \det  _ {\gamma} ^ {D} (\mathcal {M}) \cdot T + \gamma \cdot \operatorname {O p t E s t} _ {\gamma} ^ {D} (T, \delta). \tag {7}
$$
This regret bound has the same structure as Theorem 1.1, with the DEC and estimation error replaced by their optimistic counterparts. In the remainder of the paper, we show that 1) by adopting asymmetric divergences specialized to reinforcement learning, this result leads to the first guarantees for model-free RL with E2D, but 2) for symmetric divergences such as Hellinger distance, the result never improves upon Theorem 1.1.
Estimation with batching. For our application to reinforcement learning, we generalize the results above to accommodate estimation algorithms that draw batches of multiple samples from each distribution  $p^t$ . Given a batch size  $n$ , we break the  $T$  rounds of the decision making protocol into  $K \coloneqq T / n$  contiguous epochs (or, "iterations"). Within each epoch, the learner's distribution  $p^k$  is unchanged (we index by  $k$  rather than  $t$  to reflect this), and we create a batch  $B^k = \{(\pi^{k,l}, r^{k,l}, o^{k,l})\}_{l=1}^n$  by sampling  $\pi^{k,l} \sim p^k$  independently and observing  $(r^{k,l}, o^{k,l}) \sim M^*(\pi^{k,l})$  for each  $l \in [n]$ . We can then appeal to estimation algorithms of the form  $\mu^k = \mathbf{Alg}_{\mathrm{Est}}^k(\{B^k\}_{i=1}^{k-1})$ . Regret bounds for a variant of E2D.Opt with batching are given in Appendix B.1.
2.3 Application to Model-Free Reinforcement Learning
In this section, we use Optimistic Estimation-to-Decisions to provide sample-efficient guarantees for model-free reinforcement learning with bilinear classes [11], a general class of tractable reinforcement learning problems which encompasses many settings [17, 26, 7, 29, 18, 23, 3, 20, 10, 22, 9, 33].
Reinforcement learning preliminaries. To state our results, let us recall how reinforcement learning fits into the DMSO framework. We consider an episodic, finite-horizon reinforcement learning setting. With  $H$  denoting the horizon, each model  $M \in \mathcal{M}$  specifies a non-stationary Markov decision process  $M = \{S, \mathcal{A}, \{P_h^M\}_{h=1}^H, \{R_h^M\}_{h=1}^H, d_1\}$ , where  $S$  is the state space,  $\mathcal{A}$  is the action space,  $P_h^M : S \times \mathcal{A} \to \Delta(S)$  is the probability transition distribution at step  $h$ ,  $R_h^M : S \times \mathcal{A} \to \Delta(\mathbb{R})$  is the reward distribution, and  $d_1 \in \Delta(S_1)$  is the initial state distribution. We allow the reward distribution and transition kernel to vary across models in  $\mathcal{M}$ , but assume that the initial state distribution is fixed.
For a fixed MDP  $M \in \mathcal{M}$ , each episode proceeds under the following protocol. At the beginning of the episode, the learner selects a randomized, non-stationary policy  $\pi = (\pi_1, \dots, \pi_H)$ , where  $\pi_h : \mathcal{S} \to \Delta(\mathcal{A})$ ; we let  $\Pi_{\mathrm{RNS}}$  (for "randomized, non-stationary") denote the set of all such policies. The episode then evolves through the following process, beginning from  $s_1 \sim d_1$ : For  $h = 1, \dots, H$ :  $a_h \sim \pi_h(s_h)$ ,  $r_h \sim R_h^M(s_h, a_h)$ , and  $s_{h+1} \sim P_h^M(\cdot | s_h, a_h)$ . For notational convenience, we take  $s_{H+1}$  to be a deterministic terminal state. We assume for simplicity that  $\mathcal{R} \subseteq [0,1]$  (that is,  $\sum_{h=1}^{H} r_h \in [0,1]$  almost surely). Within the DMSO framework, at each time  $t$ , the learning agent chooses  $\pi^t \in \Pi_{\mathrm{RNS}}$ , then observes the cumulative reward  $r^t = \sum_{h=1}^{H} r_h^t$  and trajectory  $o^t := (s_1^t, a_1^t, r_1^t), \dots, (s_H^t, a_H^t, r_H^t)$  that results from executing  $\pi^t$ .
Value functions. The value for a policy  $\pi$  under  $M$  is given by  $f^{M}(\pi) := \mathbb{E}^{M,\pi}\left[\sum_{h=1}^{H}r_{h}\right]$ , where  $\mathbb{E}^{M,\pi}[\cdot]$  denotes expectation under the process above. For a given model  $M$  and policy  $\pi$ , we define the state-action value function and state value functions via  $Q_{h}^{M,\pi}(s,a) = \mathbb{E}^{M,\pi}\left[\sum_{h'=h}^{H}r_{h'}\mid s_{h}=s,a_{h}=a\right]$ , and  $V_{h}^{M,\pi}(s) = \mathbb{E}^{M,\pi}\left[\sum_{h'=h}^{H}r_{h'}\mid s_{h}=s\right]$ . We define  $\pi_{M}$  as the optimal policy, which maximizes  $Q_{h}^{M,\pi_{M}}(s,a)$  for all states simultaneously. We abbreviate  $Q^{M,\star} \equiv Q^{M,\pi_{M}}$ .
Value function approximation. To apply our results to reinforcement learning, we take a model-free (or, value function approximation) approach, and estimate value functions for the underlying MDP  $M^{\star}$ ; this contrasts with model-based methods, such as those considered in Foster et al. [12], which estimate transition probabilities for  $M^{\star}$  directly. We assume access to a class  $\mathcal{Q}$  of value functions of the form  $Q = (Q_{1},\ldots ,Q_{H})$
Assumption 2.2. The value function class  $\mathcal{Q}$  has  $Q^{M^{\star},\star} \in \mathcal{Q}$ , where  $M^{\star}$  is the underlying model.
For  $Q = (Q_{1},\ldots ,Q_{H}) \in \mathcal{Q}$ , we define  $\pi_Q = (\pi_{Q,1},\dots,\pi_{Q,H})$  via  $\pi_{Q,h}(s) = \arg \max_{a\in \mathcal{A}}Q_h(s,a)$ . We define  $\Pi_{\mathcal{Q}} = \{\pi_Q|Q\in \mathcal{Q}\}$  as the induced policy class. While it is not necessary for our results, we mention in passing that the class  $\mathcal{Q}$ , under Assumption 2.2, implicitly induces a model class via  $\mathcal{M}_{\mathcal{Q}} := \{M\mid Q^{M,\star}\in \mathcal{Q}\}$ .
Bilinear classes. The bilinear class framework [11] gives structural conditions for sample-efficient reinforcement learning that capture most known settings where tractable guarantees are possible. The following is an adaptation of the definition from Du et al. [11].<sup>4</sup>
Definition 2.1 (Bilinear class). An MDP  $M$  is said to be bilinear with dimension  $d$  relative to a class  $\mathcal{Q}$  if:
(i) There exist functions  $W_{h}(\cdot ;M):\mathcal{Q}\to \mathbb{R}^{d}$ ,  $X_{h}(\cdot ;M):\mathcal{Q}\to \mathbb{R}^{d}$  such that for all  $Q\in \mathcal{Q}$  and  $h\in [H]$ ,
$$
\left| \mathbb {E} ^ {M, \pi_ {Q}} \left[ Q _ {h} \left(s _ {h}, a _ {h}\right) - r _ {h} - \max  _ {a ^ {\prime} \in \mathcal {A}} Q _ {h + 1} \left(s _ {h + 1}, a ^ {\prime}\right) \right] \right| \leq | \langle X _ {h} (Q; M), W _ {h} (Q; M) \rangle |. \tag {8}
$$
(ii) Let  $z_{h} \coloneqq (s_{h}, a_{h}, r_{h}, s_{h+1})$ . There exists a collection of estimation policies  $\{\pi_{Q}^{\mathrm{est}}\}_{Q \in \mathcal{Q}}$  and a discrepancy function  $\ell_{h}^{\mathrm{est}}(\cdot; \cdot): \mathcal{Q} \times \mathcal{Z} \to \mathbb{R}$  such that for all  $Q, Q' \in \mathcal{Q}$  and  $h \in [H]$ ,
$$
\left| \left\langle X _ {h} (Q; M), W _ {h} \left(Q ^ {\prime}; M\right) \right\rangle \right| = \left| \mathbb {E} ^ {M, \pi_ {Q} \circ_ {h} \pi_ {Q} ^ {\text {e s t}}} \left[ \ell_ {h} ^ {\text {e s t}} \left(Q ^ {\prime}; z _ {h}\right) \right] \right|. \tag {9}
$$
If  $\pi_Q^{\mathrm{est}} = \pi_Q$ , we say that estimation is on-policy. We assume  $|\mathbb{E}^{M,\pi}[\ell_h^{\mathrm{est}}(Q^{M,\star};z_h)]| = 0$  for all  $\pi$ .
We let  $d_{\mathrm{bi}}(\mathcal{Q};M)$  denote the minimal dimension  $d$  for which the bilinear property holds for  $M$ . For a model class  $\mathcal{M}$ , we define  $d_{\mathrm{bi}}(\mathcal{Q};\mathcal{M}) = \sup_{M\in \mathcal{M}}d_{\mathrm{bi}}(\mathcal{Q};M)$ . We let  $L_{\mathrm{bi}}(\mathcal{Q};M)\geq 1$  denote any almost sure upper bound on  $|\ell_h^{\mathrm{est}}(Q;z_h)|$  under  $M$ .
<sup>4</sup>For the sake of simplicity, we adopt a less general definition than Du et al. [11]: We 1) assume that the "hypothesis class" is parameterized by the  $Q$ -function class  $\mathcal{Q}$ , and 2) limit to discrepancy functions that do not explicitly depend on the function  $Q$  indexing the factor  $X_{h}(Q;M)$ . The results here readily extend to the full definition.
For  $\pi$  and  $\pi'$ ,  $\pi \circ_h \pi'$  denotes the policy that follows  $\pi$  for layers  $1, \ldots, h-1$  and follows  $\pi'$  for layers  $h, \ldots, H$ .
2.4 Guarantees for Bilinear Classes
We now apply our main results to derive regret bounds for bilinear classes. We first provide optimistic estimation guarantees, then bound the Optimistic DEC, and conclude by applying E2D.Opt.
We take  $\Psi = \mathcal{Q}$  as the sufficient statistic space, with  $\psi (M)\coloneqq Q^{M,\star}$ , and define  $f^Q (\pi_Q)\coloneqq \mathbb{E}_{s_1\sim d_1}[Q_1(s_1,\pi_Q(s_1))]$ . For the divergence  $D$ , we appeal to squared discrepancy, in the vein of Jiang et al. [17], Du et al. [11]:
$$
D _ {\mathrm {b i}} ^ {\pi} (Q \| M) = \sum_ {h = 1} ^ {H} \left(\mathbb {E} ^ {M, \pi} \left[ \ell_ {h} ^ {\text {e s t}} \left(Q; z _ {h}\right) \right]\right) ^ {2}. \tag {10}
$$
We abbreviate  $\mathbf{OptEst}_{\gamma}^{\mathrm{bi}} = \mathbf{OptEst}_{\gamma}^{D_{\mathrm{bi}}}$  and o-dec<sup>bi</sup> (M, μ) = o-dec<sup>di</sup> (M, μ).
Estimation. To perform estimation, we approximate the average discrepancy in (14) from samples (drawing a batch of  $n$  samples at each step; cf. Algorithm 3), then appeal to the exponential weights method for online learning, with a bonus to enforce optimism. See Algorithm 6 (deferred to Appendix D for space).
Proposition 2.1. For any batch size  $n$  (with  $K \coloneqq T / n$ ) and parameter  $\gamma \geq 1$ , Algorithm 6, with an appropriate learning rate, ensures that with probability at least  $1 - \delta$ ,
$$
\mathbf {O p t E s t} _ {\gamma} ^ {\mathrm {b i}} \lesssim \frac {\sqrt {K \log | \mathcal {Q} |}}{\gamma} + H L _ {\mathrm {b i}} ^ {2} (\mathcal {Q}; M ^ {\star}) \log (| \mathcal {Q} | K H / \delta) \left(1 + \frac {K}{n}\right), \tag {11}
$$
whenever  $M^{\star}$  is bilinear relative to  $\mathcal{Q}$  and Assumption 2.2 is satisfied.
This result does not actually use the bilinear class structure, and gives a bound on (14) for any choice of  $\ell_h^{\mathrm{est}}$ . Similar to the optimistic estimation result given by Zhang [31] for contextual bandits, this guarantee consists of "slow" term  $\frac{\sqrt{K\log|\mathcal{Q}|}}{\gamma}$  resulting from optimism (which decays as  $\gamma$  grows), and a "fast" term. However, compared to the contextual bandit setting, the fast term,  $\log (|\mathcal{Q}|KH / \delta)\left(1 + \frac{K}{n}\right)$ , scales with the ratio  $\frac{K}{n}$ , which reflects sampling error in the estimated discrepancy, and necessitates a large batch size. Previous algorithms for bilinear classes [17, 11] require large batch sizes for similar reasons (specifically, because the expectation in (14) appears inside the square, it is not possible to form an unbiased estimate for  $D_{\mathrm{bi}}$  directly).
Bounding the DEC. A bound on the Optimistic DEC follows by adapting arguments in Foster et al. [12]; our result has slightly improved dependence on  $H$  compared to the bounds in that work.
Proposition 2.2. Let  $\mathcal{M}$  be any model class for which the bilinear class property holds relative to  $\mathcal{Q}$ . (1) In the on-policy case where  $\pi_Q^{\mathrm{est}} = \pi_Q$ , we have that for all  $\gamma > 0$ , o-dec $_\gamma^{\mathrm{bi}}(\mathcal{M}) \lesssim \frac{H \cdot d_{\mathrm{bi}}(\mathcal{Q}; \mathcal{M})}{\gamma}$ .
(2) In the general case  $(\pi_Q^{\mathrm{est}} \neq \pi_Q)$ , we have that for all  $\gamma \geq H^2 d_{\mathrm{bi}}(\mathcal{Q}; \mathcal{M})$ , o-dec $_\gamma^{\mathrm{bi}}(\mathcal{M}) \lesssim \sqrt{\frac{H^2 \cdot d_{\mathrm{bi}}(\mathcal{Q}; \mathcal{M})}{\gamma}}$ .
Combining Theorem B.1, Proposition 2.1, and Proposition 2.2, we obtain the following result.
Corollary 2.1 (Regret bound for bilinear classes). Let  $\mathcal{Q}$  be given. Assume that  $M^{\star} \in \mathcal{M}$ , where  $\mathcal{M}$  is bilinear relative to  $\mathcal{Q}$ , and that Assumption 2.2 holds. Abbreviate  $d \equiv d_{\mathrm{bi}}(\mathcal{Q}; \mathcal{M})$  and  $L \equiv L_{\mathrm{bi}}(\mathcal{Q}; \mathcal{M}) \coloneqq \sup_{M \in \mathcal{M}} L_{\mathrm{bi}}(\mathcal{Q}; M)$ . For an appropriate choice of  $n$  and  $\gamma$ , Algorithm 3, using the algorithm from Proposition 2.1 as an oracle, enjoys the following guarantees with probability at least  $1 - \delta$ :
(1) In the on-policy case where  $\pi_Q^{\mathrm{est}} = \pi_Q$ :  $\mathbf{Reg}_{\mathsf{DM}} \lesssim (H^2 dL^2 \log (|\mathcal{Q}| TH / \delta))^{1/2} T^{3/4}$ .
(2) In the general case where  $\pi_Q^{\mathrm{est}}\neq \pi_Q$  ..  $\mathbf{Reg}_{\mathrm{DM}}\lesssim (H^6 d^2 L^4\log^3 (|\mathcal{Q}|TH / \delta))^{1 / 6}T^{5 / 6}$
This is the first regret bound for model-free reinforcement learning with the Estimation-to-Decisions meta-algorithm. Importantly, the result scales only with the horizon, the dimension  $d$ , and the capacity  $\log |\mathcal{Q}|$ . Improving the dependence on  $T$  in Corollary 2.1 is an interesting question: currently, there are no algorithms for general bilinear classes that achieve  $\sqrt{T}$  regret without additional assumptions. Let us emphasize that regret bounds for bilinear classes can already be achieved by a number of existing methods [11, 19]. The contribution here is to show that such guarantees can be achieved through the general DEC framework, thereby placing this line of research on stronger foundations.
Tighter guarantees under Bellman completeness. In Appendix B.2 (deferred for space), we adapt techniques from Agarwal and Zhang [2] to derive tighter estimation guarantees when  $\mathcal{Q}$  satisfies a Bellman completeness property (e.g., Zanette et al. [30], Jin et al. [19]), by appealing to refined algorithms tailored to squared Bellman error. We use this to derive tighter regret bounds for bilinear classes ( $T^{2/3}$  instead of  $T^{3/4}$ ).
3 Understanding the Role of Optimistic Estimation
We close with discussion and interpretation of our results.
When does optimistic estimation help? Perhaps the most pressing question at this point is to understand when the regret bound for E2D.Opt (Theorem 2.1) improves upon the corresponding regret bound for vanilla E2D (Theorem 1.1). In what follows, we show that: (1) For any divergence  $D$ , the Optimistic DEC is equivalent to a variant of the original DEC which incorporates randomized estimators [12], but with the arguments to the divergence flipped; (2) For divergences  $D$  that satisfy a triangle inequality, this randomized DEC is equivalent to the original DEC itself. Together these results show that the improvement given by the Optimistic DEC is limited to asymmetric divergences such as the bilinear divergence in Section 2.3; for more traditional divergences such as Hellinger distance and squared error, the optimistic approach offers no improvement. Our results use the following regularity assumption, satisfied by all standard divergences.
Assumption 3.1. For all  $M, \overline{M} \in \operatorname{co}(\mathcal{M})$ ,  $(f^{\overline{M}}(\pi) - f^{M}(\pi))^{2} \leq L_{\mathrm{lip}}^{2} \cdot D^{\pi}\left(\overline{M} \parallel M\right)$  for a constant  $L_{\mathrm{lip}} > 0$ .
Given a divergence  $D$ , we define the flipped divergence, which swaps the first and second arguments, by  $\check{D}^{\pi}\big(\overline{M}\parallel M\big):= D^{\pi}\big(M\parallel \overline{M}\big)$ . We define the Decision-Estimation Coefficient for randomized estimators [12, 5] as  $\underline{\mathrm{dec}}_{\gamma}^{D}(\mathcal{M},\mu) = \inf_{p\in \Delta (\Pi)}\sup_{M\in \mathcal{M}}\mathbb{E}_{\pi \sim p}\big[f^{M}(\pi_{M}) - f^{M}(\pi) - \gamma \cdot$ $\mathbb{E}_{\overline{M}\sim \mu}\big[D^{\pi}\big(\overline{M}\parallel M\big)\big]\big],$  with  $\underline{\mathrm{dec}}_{\gamma}^{D}(\mathcal{M}):= \sup_{\mu \in \Delta (\mathcal{M})}\underline{\mathrm{dec}}_{\gamma}^{D}(\mathcal{M},\mu)$ . This definition is identical to (2), but allows  $\overline{M}$  to be randomized.
Proposition 3.1. Whenever Assumption 3.1 holds, we have that for all  $\gamma >0$
$$
\underline {{\operatorname {d e c}}} _ {3 \gamma / 2} ^ {\check {D}} (\mathcal {M}) - \frac {L _ {\mathrm {l i p}} ^ {2}}{2 \gamma} \leq \mathrm {o} - \operatorname {d e c} _ {\gamma} ^ {D} (\mathcal {M}) \leq \underline {{\operatorname {d e c}}} _ {\gamma / 2} ^ {\check {D}} (\mathcal {M}) + \frac {L _ {\mathrm {l i p}} ^ {2}}{2 \gamma}. \tag {12}
$$
For settings in which there exists an estimation oracle for which the flipped estimation error  $\mathbf{Est}^{\check{D}} = \sum_{t=1}^{T} \mathbb{E}_{\pi \sim p^t} \mathbb{E}_{\widehat{M}^t \sim \mu^t} \left[ D^{\pi^t} \left( M^\star \| \widehat{M}^t \right) \right]$  is controlled, this result shows that to match the guarantee in Theorem 2.1, optimism is not required, and it suffices to run a variant of vanilla E2D that incorporates randomized estimators (cf. Foster et al. [12], Section 4.3).
We now turn to the role of randomization. When  $D$  is convex in the first argument, we have  $\underline{\mathrm{dec}}_{\gamma}^{D}(\mathcal{M}) \leq \sup_{\overline{M} \in \mathrm{co}(\mathcal{M})} \mathrm{dec}_{\gamma}^{D}(\mathcal{M}, \overline{M}) = \mathrm{dec}_{\gamma}^{D}(\mathcal{M})$ , but it is not immediately apparent whether the opposite direction of this inequality holds, and one might hope that working with the randomized DEC in Proposition 3.1 would lead to improvements over the non-randomized counterpart in Theorem 1.1. The next result shows that this is not the case: Under mild assumptions on the divergence  $D$ , randomization offers no improvement.
Proposition 3.2. Let  $D$  be any bounded divergence such that for all  $M, M', \overline{M}$  and  $\pi \in \Pi$ ,  $D^{\pi}(M \parallel M') \leq C \left(D^{\pi}(\overline{M} \parallel M) + D^{\pi}(\overline{M} \parallel M')\right)$ . Then for all  $\gamma > 0$ ,  $\sup_{\overline{M}} \mathsf{dec}_{\gamma}^{D}(\mathcal{M}, \overline{M}) \leq \underline{\mathsf{dec}}_{\gamma/(2C)}^{D}(\mathcal{M})$ .
Implication for Hellinger distance. Squared Hellinger distance is symmetric, satisfies Assumption 3.1 with  $L_{\mathrm{lip}} = 1$  whenever  $\mathcal{R} \subseteq [0,1]$ , and satisfies the condition in Proposition 3.2 with  $C = 2$ . Hence, by combining Proposition 3.1 with Proposition 3.2, we obtain the following corollary.
Corollary 3.1. If  $\mathcal{R} \subseteq [0,1]$ , then  $\mathrm{o - dec}_{2\gamma}^{\mathsf{H}}(\mathcal{M}) - \frac{1}{\gamma} \leq \sup_{\overline{M}} \mathrm{dec}_{\gamma}^{\mathsf{H}}(\mathcal{M}, \overline{M}) \leq \mathrm{o - dec}_{\gamma/6}^{\mathsf{H}}(\mathcal{M}) + \frac{3}{\gamma} \forall \gamma > 0$ .
This shows that for Hellinger distance—at least from a statistical perspective—there is no benefit to using the Optimistic DEC or randomized DEC compared to the original version. In particular, this implies that regret bounds based on the randomized DEC with Hellinger distance (such as those found
in recent work of Chen et al. [5]) do not offer improvement over the guarantees for vanilla E2D in Foster et al. [12]. One caveat, though, is that working with the Optimistic DEC, as well as randomized estimators, has potential to give computational improvement, as computing a distribution  $p \in \Delta(\Pi)$  that minimizes  $o\mathsf{dec}_{\gamma}^{\mathsf{H}}(\mathcal{M}, \mu)$  might be simpler than computing a corresponding distribution for  $\mathsf{dec}_{\gamma}^{\mathsf{H}}(\mathcal{M}, \overline{M})$  with  $\overline{M} \in \mathrm{co}(\mathcal{M})$ . We are not currently aware of any examples where such an improvement occurs, as even maintaining a distribution  $\mu \in \Delta(\mathcal{M})$  requires intractably large memory for most classes of interest.
Implication for model-free RL. The bilinear divergence  $D_{\mathrm{bi}}^{\pi}(Q \parallel M) = \sum_{h=1}^{H} (\mathbb{E}^{M, \pi}[\ell_h^{\mathrm{est}}(Q; z_h)])^2$  that we adopt in Section 2.3 is asymmetric, as are closely related divergences such as squared Bellman error. Here, there are two reasons why optimistic estimation offers meaningful advantages.
(1) By Proposition 3.1 ( $D_{\mathrm{bi}}$  satisfies Assumption 3.1 with  $L_{\mathrm{lip}} = O(H)$ ), a natural alternative to optimistic estimation is to estimate with respect to the flipped divergence  $\check{D}_{\mathrm{bi}}$ , then appeal to Algorithm 3 of Foster et al. [12]. The issue with this approach is that minimizing the flipped estimation error, which takes the form
$$
\mathbf {E s t} ^ {\check {D} _ {\mathrm {b i}}} = \sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi \sim p ^ {t}} \mathbb {E} _ {\widehat {M} ^ {t} \sim \mu^ {t}} \left[ D _ {\mathrm {b i}} ^ {\pi^ {t}} \left(Q ^ {M ^ {\star}, \star} \| \widehat {M} ^ {t}\right) \right] = \sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi \sim p ^ {t}} \mathbb {E} _ {\widehat {M} ^ {t} \sim \mu^ {t}} \left[ \sum_ {h = 1} ^ {H} \left(\mathbb {E} ^ {\widehat {M} ^ {t}, \pi^ {t}} \left[ \ell_ {h} ^ {\mathrm {e s t}} (Q ^ {M ^ {\star}, \star}; z _ {h}) \right]\right) ^ {2} \right],
$$
is challenging in model-free settings; we are not aware of any algorithms that accomplish this.6
(2) Alternatively, a second choice is to perform estimation with respect to the un-flipped divergence  $D_{\mathrm{bi}}$  (which can be accomplished with Proposition 2.1 by taking  $\gamma \rightarrow \infty$ ), and appeal to vanilla E2D (either Algorithm 1, or Algorithm 3 of Foster et al. [12] if one wishes to incorporate randomized estimators). However, the following result shows that unlike the Optimistic DEC, the original DEC with the divergence  $D_{\mathrm{bi}}$  does not admit a favorable bound, even for tabular reinforcement learning.
Proposition 3.3. Let  $\mathcal{M}$  be the class of all horizon- $H$  tabular MDPs with  $|\mathcal{S}| = 2$  and  $|\mathcal{A}| = 2$ . Consider the discrepancy function  $\ell_h^{\mathrm{est}}(Q;z_h) = (Q_h(s_h,a_h) - r_h - \max_{a'\in \mathcal{A}}Q_{h + 1}(s_{h + 1},a'))$ . Then we have o-dec $_\gamma^{\mathrm{bi}}(\mathcal{M}) \lesssim \frac{H}{\gamma}$ , yet there exists  $\overline{M} \in \mathcal{M}$  for which  $\mathrm{dec}_{\gamma}^{\mathrm{bi}}(\mathcal{M},\overline{M}) \gtrsim \frac{2^H}{\gamma} \wedge 1$ .