SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Insufficiency of posterior sampling. For contextual bandits, where  $\mathsf{o - dec}_{\gamma}^{\mathrm{bi}}(\mathcal{M}) \lesssim \frac{|\mathcal{A}|}{\gamma}$  [31], and bilinear classes, where  $\mathsf{o - dec}_{\gamma}^{\mathrm{bi}}(\mathcal{M}) \lesssim \frac{H \cdot d_{\mathrm{bi}}(\mathcal{M})}{\gamma}$  (Proposition 2.2), a strategy that achieves the bound on the Optimistic DEC is posterior sampling (this is also the approach taken in Agarwal and Zhang [1, 2], Zhong et al. [32]). That is, given a distribution  $\mu \in \Delta(\mathcal{M})$ , choosing  $p(\pi) = \mu(\{M \in \mathcal{M} \mid \pi_M = \pi\})$  in (6) certifies the desired bound on  $\mathsf{o - dec}_{\gamma}^{D}(\mathcal{M}, \mu)$  for these examples. Optimistic Estimation-to-Decisions subsumes and generalizes posterior sampling, but in light of the fact that this simple strategy succeeds for large classes of problems, it is reasonable to ask if there is a sense in which posterior sampling is universal, and whether it can achieve the value of the Optimistic DEC for any model class. This would be desirable, since it would indicate that solving the minimax problem in Algorithm 2 is not necessary. The following sample shows that this is not the case: there are model classes (specifically, MDPs with a constant number of actions) for which the regret of posterior sampling is exponentially large compared to the regret of Algorithm 2.
Proposition 3.4. Consider the divergence  $D_{\mathsf{H}}^{\pi}(\cdot ,\cdot)$ . For any  $S\in \mathbb{N}$  and  $H\geq \log_2(S)$ , there exists a class of horizon-H MDPs  $\mathcal{M}$  with  $|\mathcal{S}| = S$  and  $|\mathcal{A}| = 3$  that satisfies the following properties:
- There exists an estimation oracle with  $\mathbf{OptEst}_{\gamma}^{\mathrm{H}} \lesssim \log(S/\delta)$  w.p. at least  $1 - \delta$  for all  $\gamma > 0$ .
- Posterior sampling, which sets  $p^t(\pi) = \mu^t(\{M \mid \pi_M = \pi\})$ , has  $\mathbb{E}[\mathbf{Reg}_{\mathbf{DM}}] \gtrsim S \wedge 2^{\Omega(H)}$ .
- Algorithm 2 with divergence  $D = D_{\mathsf{H}}^{\pi}(\cdot, \cdot)$  has  $\mathbb{E}[\mathbf{Reg}_{\mathsf{DM}}] \leq \widetilde{O}\big(\sqrt{T\log(S)}\big)$ .
This shows that posterior sampling does not provide a universal mechanism for exploration, and highlights the need for deliberate strategies such as E2D.
<sup>6</sup>While Foster et al. [12] do give regret bounds for model-free RL using this divergence, they only bound Bayesian regret, and do not provide an explicit algorithm for the frequentist setting.
NG is supported by a Fannie & John Hertz Foundation Fellowship and an NSF Graduate Fellowship. AR acknowledges support from the NSF through award DMS-2031883, from the ARO through award W911NF-21-1-0328, and from the DOE through award DE-SC0022199.
[19] C. Jin, Q. Liu, and S. Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. Neural Information Processing Systems, 2021.
[20] A. Krishnamurthy, A. Agarwal, and J. Langford. PAC reinforcement learning with rich observations. In Advances in Neural Information Processing Systems, pages 1840-1848, 2016.
[21] T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[22] L. Li. A unifying framework for computational reinforcement learning theory. Rutgers, The State University of New Jersey—New Brunswick, 2009.
[23] A. Modi, N. Jiang, A. Tewari, and S. Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics, pages 2010–2020. PMLR, 2020.
[24] I. Osband and B. Van Roy. On lower bounds for regret in reinforcement learning. arXiv preprint arXiv:1608.02732, 2016.
[25] D. Russo and B. Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems, pages 2256-2264, 2013.
[26] W. Sun, N. Jiang, A. Krishnamurthy, A. Agarwal, and J. Langford. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Conference on learning theory, pages 2898-2933. PMLR, 2019.
[27] A. Wagenmaker and D. J. Foster. Instance-optimality in interactive decision making: Toward a non-asymptotic theory. Conference on Learning Theory (COLT), 2023.
[28] T. Xie, D. J. Foster, Y. Bai, N. Jiang, and S. M. Kakade. The role of coverage in online reinforcement learning. International Conference on Learning Representations (ICLR), 2023.
[29] L. Yang and M. Wang. Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning, pages 6995-7004. PMLR, 2019.
[30] A. Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning, pages 10978-10989. PMLR, 2020.
[31] T. Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022.
[32] H. Zhong, W. Xiong, S. Zheng, L. Wang, Z. Wang, Z. Yang, and T. Zhang. A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962, 2022.
[33] D. Zhou, Q. Gu, and C. Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021.
A Additional Related Work 13
B Additional Results 13
B.1 Optimistic Estimation-to-Decisions with Batching 13
B.2 Model-Free RL: Tighter Guarantees under Bellman Completeness 14
B.3 Proofs from Appendix B.2 15
C Technical Tools 17
C.1 Preliminaries 17
C.2 Basic Online Learning Results 17
C.3 Online Learning with Completeness 18
D Proofs from Section 2 and Section 3 30
D.1 Proofs from Section 2 30
D.2 Proofs from Section 3 35
In this section we discuss additional related work not already covered.
Chen et al. [5] build on the DEC framework by giving regret bounds for a E2D that incorporates randomized estimators, but not optimism. For the case of finite classes, their guarantees scale as roughly
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim \underline {{\operatorname {d e c}}} _ {\gamma} ^ {D _ {\mathrm {H}}} (\mathcal {M}) \cdot T + \gamma \cdot \log (| \mathcal {M} | / \delta),
$$
where  $\underline{\mathrm{dec}}_{\gamma}^{D_{\mathrm{H}}}(\mathcal{M})$  is the DEC for randomized estimators defined in Section 3. As discussed in Section 3, this regret bound cannot improve upon the guarantees for the original E2D method in Foster et al. [12] beyond constants, as we have  $\sup_{\overline{M}}\mathsf{dec}_{4\gamma}^{\mathrm{H}}(\mathcal{M},\overline{M})\leq \underline{\mathsf{dec}}_{\gamma}^{D_{\mathrm{H}}}(\mathcal{M})$ . In addition, since the algorithm does not incorporate optimism, it cannot be directly applied to model-free reinforcement learning settings.
Foster et al. [15] give upper and lower bounds on optimal regret based on a variant of the DEC called the constrained Decision-Estimation Coefficient. These results tighten the original regret bounds in Foster et al. [12], but the upper bounds still scale with  $\mathbf{Est}^{\mathrm{H}}(T,\delta) = \log (|\mathcal{M}| / \delta)$ , rendering them unsuitable for model-free RL. Nonetheless it would be interesting to explore whether the techniques in this work can be combined with optimistic estimation.
B.1 Optimistic Estimation-to-Decisions with Batching
Algorithm 3 Optimistic Estimation-to-Decisions (E2D.Opt) with Batching
1: parameters:
Online estimation oracle  $\mathbf{Alg}_{\mathrm{Est}}$  with batch size  $n$ .
Exploration parameter  $\gamma > 0$ .
Divergence  $D(\cdot \parallel \cdot)$  with sufficient statistic space  $\Psi$ .
2: Let  $K \coloneqq T / n$ .
3: for  $k = 1,2,\dots ,K$  do
4: Receive randomized estimator  $\mu^k\in \Delta (\Psi) = \mathbf{A}\mathbf{l}\mathbf{g}_{\mathrm{Est}}^t\big((B^i)_{i = 1}^{k - 1}\big)$ .
5: Get  $p^k \gets \arg \min_{p \in \Delta(\Pi)} \sup_{M \in \mathcal{M}} \mathbb{E}_{\pi \sim p} \mathbb{E}_{\widehat{\psi} \sim \mu^k}\left[f^{\widehat{\psi}}(\pi_{\widehat{\psi}}) - f^M(\pi) - \gamma \cdot D^\pi\left(\widehat{\psi} \| M\right)\right]$ . // Eq. (6).
6: Sample batch  $B^{k} = \{(\pi^{k,l}, r^{k,l}, o^{k,l})\}_{l=1}^{n}$  where  $\pi^{k,l} \sim p^{k}$  and  $(r^{k,l}, o^{k,l}) \sim M^{\star}(\pi^{k,l})$ , and update estimation oracle with  $B^{k}$ .
For our application to reinforcement learning, it will be useful to generalize E2D. Opt to accommodate estimation algorithms that draw batches of multiple samples from each distribution  $p^t$ . Given a batch
size  $n$ , we break the  $T$  rounds of the decision making protocol into  $K \coloneqq T / n$  contiguous epochs (or, "iterations"). Within each epoch, the learner's distribution  $p^k$  is unchanged (we index by  $k$  rather than  $t$  to reflect this), and we create a batch  $B^k = \{(\pi^{k,l}, r^{k,l}, o^{k,l})\}_{l=1}^n$  by sampling  $\pi^{k,l} \sim p^k$  independently and observing  $(r^{k,l}, o^{k,l}) \sim M^\star(\pi^{k,l})$  for each  $l \in [n]$ . We consider estimation oracles of the form  $\mu^k = \mathbf{A}\mathbf{l}\mathbf{g}_{\mathrm{Est}}^k\left(\{B^k\}_{i=1}^{k-1}\right)$ , and measure estimation error via
$$
\mathbf {O p t E s t} _ {\gamma} ^ {D} := \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {\widehat {M} ^ {k} \sim \mu^ {k}} \left[ D ^ {\pi} \left(\widehat {M} ^ {k} \| M ^ {\star}\right) + \gamma^ {- 1} \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {\widehat {M} ^ {t}} \left(\pi_ {\widehat {M} ^ {k}}\right)\right) \right]. \tag {13}
$$
We assume that the estimation oracle ensures that with probability at least  $1 - \delta$ ,  $\mathbf{OptE s t}_{\gamma}^{D}\leq$ $\mathbf{OptE s t}_{\gamma}^{D}(K,n,\delta)$ , where  $\mathbf{OptE s t}_{\gamma}^{D}(K,n,\delta)$  is a known upper bound.
Algorithm 3 is a variant of E2D. Opt that incorporates batching. The algorithm updates the distribution  $p^k$  in the same fashion as its non-batched counterpart, but does so only at the beginning of each epoch. The main guarantee for this method as follows.
Theorem B.1. Let  $T \in \mathbb{N}$  be given, and let  $n$  be the batch size. For any  $\delta > 0$ , Algorithm 3 ensures that with probability at least  $1 - \delta$ ,  $\mathbf{Reg}_{\mathrm{DM}} \leq o\text{-dec}_\gamma^D(\mathcal{M}) \cdot T + \gamma n \cdot \mathbf{OptEst}_\gamma^D(T/n, n, \delta)$ .
See Appendix D.1 for the proof. When working with divergences for which unbiased estimates are unavailable, this approach can lead to stronger guarantees than Theorem 2.1. We refer to the proof of Proposition 2.1 for a concrete example.
B.2 Model-Free RL: Tighter Guarantees under Bellman Completeness
In this section, we show how to derive tighter estimation guarantees (and consequently tighter regret bounds) when  $\mathcal{Q}$  satisfies a Bellman completeness assumption [30, 19]. For a given MDP  $M$ , let  $\mathcal{T}_h^M [f](s,a)\coloneqq \mathbb{E}^M [r_h + \max_{a'}f(s_{h + 1},a')\mid s_h = s,a_h = a]$  denote the Bellman operator for layer  $h$ .
Assumption B.1 (Completeness). We assume that  $\mathcal{Q} = \mathcal{Q}_1 \times \dots \times \mathcal{Q}_H$  is a product class, and that for all  $h$  and  $Q_{h+1} \in \mathcal{Q}_{h+1}$ ,  $[\mathcal{T}_h^{M^*} Q_{h+1}] \in \mathcal{Q}_h$ .
As before, we take  $\Psi = \mathcal{Q}$ ,  $\psi(M) := Q^{M,\star}$ , and  $f^Q(\pi_Q) := \mathbb{E}_{s_1 \sim d_1}[Q_1(s_1, \pi_Q(s_1))]$ . For the divergence  $D$ , we now appeal to squared Bellman error (e.g., [19, 28]):
$$
D _ {\mathbf {s b e}} ^ {\pi} (Q \| M) = \sum_ {h = 1} ^ {H} \mathbb {E} ^ {M, \pi} \left[ \left(Q _ {h} \left(s _ {h}, a _ {h}\right) - \left[ \mathcal {T} _ {h} ^ {M} Q _ {h + 1} \right] \left(s _ {h}, a _ {h}\right)\right) ^ {2} \right]. \tag {14}
$$
We abbreviate  $\mathbf{OptEst}_{\gamma}^{\mathrm{sbe}} = \mathbf{OptEst}_{\gamma}^{D_{\mathrm{sbe}}}$  and  $\mathsf{o - dec}_{\gamma}^{\mathrm{sbe}}(\mathcal{M},\mu) = \mathsf{o - dec}_{\gamma}^{D_{\mathrm{sbe}}}(\mathcal{M},\mu)$ .
Estimation. Algorithm 4 performs optimistic online estimation with squared Bellman error. The algorithm is an adaptation of a two-timescale exponential weights strategy originally introduced by Agarwal and Zhang [2] within an optimistic posterior sampling algorithm referred to as TS3. We show that this technique leads to a self-contained online estimation guarantee outside the context of the TS3 algorithm.
Proposition B.1 (Estimation for square Bellman error). Assume that  $\mathcal{Q}$  satisfies completeness relative to  $M^{\star}$ . Moreover assume  $\sum_{h=1}^{H} \sup_{s,a} r_h(s,a) \leq 1$  and  $\sup_{Q,h,s,a} Q_h(s,a) \leq 1$ . Then for any  $\gamma \geq 1$  and  $\eta \leq 1/(2^{16} (\log(|\mathcal{Q}|K/\delta) + 1))$ , with batch size  $n = H$  ( $K := T/n$ ),  $\lambda = 1/8$ ,  $\beta = (12\gamma H)^{-1}$  and  $\delta > 0$ , Algorithm 4 ensures that with probability at least  $1 - \delta$ ,
$$
\mathbf {O p t E s t} _ {\gamma} ^ {\mathrm {s b e}} \lesssim \frac {H \log | \mathcal {Q} |}{\eta} + \frac {\eta \log (| \mathcal {Q} | K / \delta) K}{\gamma} + \frac {K}{\gamma^ {2} H}. \tag {15}
$$
whenever  $\mathcal{Q}$  satisfies completeness relative to  $M^{\star}$ .<sup>7</sup>
Note that Proposition B.1 does not make use of the bilinear class assumption, and only requires that  $\mathcal{Q}$  satisfies completeness. As such, we expect that this result will find use more broadly.
<sup>7</sup>Agarwal and Zhang [2] give a tighter estimation error bound of roughly OptEst<sup>sbe</sup>  $\lesssim \log^2(|\mathcal{Q}|HK) + \frac{K}{\gamma^2}$ , but this result takes advantage of a Bellman rank assumption on the underlying MDP. The estimation error bound we state here does not require any structural assumptions on the MDP under consideration, but gives a worse rate.
Algorithm 4 Two-Timescale Exponential Weights for Bellman Complete Value Function Classes
1: Initialize  $S^0 = \emptyset$
2: for  $k = 1, \dots, K$  do
3: For any  $Q, Q' \in \mathcal{Q}$  and  $h \in [H]$ , define
$$
\Delta_ {h} ^ {k} (Q ^ {\prime}, Q) := Q ^ {\prime} \left(s _ {h} ^ {k, h}, a _ {h} ^ {k, h}\right) - r _ {h} ^ {k, h} - Q \left(s _ {h + 1} ^ {k, h}\right),
$$
$$
q ^ {k} \left(Q ^ {\prime} | Q\right) := q ^ {k} \left(Q ^ {\prime} | Q, S ^ {k - 1}\right) \propto \exp \left(- \lambda \cdot \frac {1}{H} \sum_ {s = 1} ^ {k - 1} \sum_ {h = 1} ^ {H} \Delta_ {h} ^ {s} \left(Q ^ {\prime}, Q\right) ^ {2}\right),
$$
$$
L ^ {k} (Q) := \frac {1}{H} \sum_ {h = 1} ^ {H} \Delta_ {h} ^ {k} (Q, Q) ^ {2} + \frac {1}{\lambda} \log \mathbb {E} _ {Q ^ {\prime} \sim q ^ {k} (\cdot | Q)} \left[ \exp \left(- \lambda \cdot \frac {1}{H} \sum_ {h = 1} ^ {H} \Delta_ {h} ^ {k} (Q ^ {\prime}, Q) ^ {2}\right) \right],
$$
$$
\mu^ {k} (Q) := \mu^ {k} (Q | S ^ {k - 1}) \propto \exp \left(- \eta \sum_ {s = 1} ^ {k - 1} \left(L ^ {s} (Q) - \beta \cdot \frac {1}{H} \sum_ {h = 1} ^ {H} \max  _ {a} Q \left(s _ {1} ^ {s, h}, a\right)\right)\right).
$$
4: Predict  $\mu^k$
5: for  $l = 1, \ldots, H$  do
6: Play  $\pi^{k,l} \sim p^k$  and obtain the trajectory  $o^{k,l} = (s_1^{k,l}, a_1^{k,l}, r_1^{k,l}), \ldots, (s_H^{k,l}, a_H^{k,l}, r_H^{k,l})$ , where  $p^k \in \Delta(\Pi)$  is a decision distribution produced by any batched algorithm (e.g., Algorithm 3) that selects a decision adaptively based on  $\mu^k$ .
7: Update  $S^k \gets S^{k-1} \cup \bigcup_{l=1}^{H} \{s_l^{k,l}, a_l^{k,l}, r_l^{k,l}, s_{l+1}^{k,l}\} \cup \bigcup_{l=1}^{H} \{s_1^{k,l}\}$ .
Regret bound for bilinear classes. To provide regret bounds, we assume that  $M^{\star}$  satisfies the bilinear class property relative to  $\mathcal{Q}$  as in Section 2.3. In addition to assuming that  $M^{\star}$  is bilinear, we make the following restrictions: (1)  $\pi_Q^{\mathrm{est}} = \pi_Q$ , i.e. estimation is on policy, (2)  $\ell_h^{\mathrm{est}}(Q;z_h) = Q_h(s_h,a_h) - r_h - \max_{a'}Q_{h + 1}(s_{h + 1},a')$ , so that  $\mathbb{E}^{M,\pi}[\ell_h^{\mathrm{est}}(Q;z_h)]$  is the average Bellman error for  $Q$  under  $M$ . With this discrepancy function, Jensen's inequality implies that o-dec $_\gamma^{\mathrm{sbe}}(\mathcal{M}) \leq$  o-dec $_\gamma^{\mathrm{bi}}(\mathcal{M})$ , so combining Theorem B.1, Proposition B.1, and Proposition 2.2, we obtain the following result.
Corollary B.1 (Regret bound under completeness). Let  $\mathcal{Q}$  be given. Assume that  $M^{\star} \in \mathcal{M}$ , where  $\mathcal{M}$  is bilinear relative to  $\mathcal{Q}$ , and that completeness holds. Moreover assume  $\sum_{h=1}^{H} \sup_{s,a} r_h(s,a) \leq 1$  and  $\sup_{Q,h,s,a} Q_h(s,a) \leq 1$ . Abbreviate  $d \equiv d_{\mathrm{bi}}(\mathcal{Q};\mathcal{M})$ . For an appropriate choice of  $n$  and  $\gamma$ , Algorithm 3, using Algorithm 4 (with appropriate parameter choice) as an oracle, enjoys the following guarantees with probability at least  $1 - \delta$ :
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim H d ^ {1 / 3} \left(\log \left(| \mathcal {Q} | K / \delta\right)\right) ^ {4 / 5} T ^ {2 / 3}. \tag {16}
$$
This improves upon the  $T^{3/4}$ -type rate in Corollary 2.1.
B.3 Proofs from Appendix B.2
Proposition B.1 is an application of more general results given in Appendix C.3, which analyze a generalization of Algorithm 4 for a more general online learning setting. To Proposition B.1, we simply apply these results to the reinforcement learning framework.
Proof of Proposition B.1. Let the batch size  $n = H$  be fixed, and let  $K \coloneqq T / n$  be the number of epochs. Recall that for each step  $k \in [K]$ , the estimation oracle is given a batch of examples  $B^{k} = \{(\pi^{k,l}, r^{k,l}, o^{k,l})\}_{l=1}^{n}$  where  $\pi^{k,l} \sim p^{k}$  and  $(r^{k,l}, o^{k,l}) \sim M^{\star}(\pi^{k,l})$ . Each observation (trajectory) takes the form  $o^{k,l} = (s_{1}^{k,l}, a_{1}^{k,l}, r_{1}^{k,l}), \ldots, (s_{H}^{k,l}, a_{H}^{k,l}, r_{H}^{k,l})$ . We abbreviate  $Q^{\star} = Q^{M^{\star},\star}$ .
Estimation algorithm. For each step  $k$ , the randomized estimator  $\mu^k$  selected as described in Algorithm 4. This algorithm is an instantiation of Algorithm 5 in the general online learning setting described in Appendix C.3, with  $\mathcal{G} = \mathcal{Q}$  and for all  $h \in [H]$ ,  $\mathcal{X}_h = \mathcal{S} \times \mathcal{A}$ ,  $\mathcal{Y}_h = \mathbb{R} \times \mathcal{S}$  and
These restrictions correspond to restricting attention to  $Q$ -type Bellman rank, a special case of the bilinear class property [19, 12].
$\mathcal{W} = \mathcal{S}^H$ . The unknown kernels are the transition distributions for the corresponding layers of the MDP  $M^{\star}$ , and the loss functions are
$$
\ell_ {h, 1} \left(\left(s _ {h}, a _ {h}\right), Q\right) := Q _ {h} \left(s _ {h}, a _ {h}\right),
$$
$$
\ell_ {h, 2} \bigl ((r _ {h}, s _ {h + 1}), Q \bigr) := r _ {h} + \max  _ {a} Q _ {h} (s _ {h + 1}, a),
$$
$$
\ell_ {3} (\{s _ {1} ^ {l} \} _ {l \in [ H ]}) := - \frac {1}{H} \sum_ {l = 1} ^ {H} \max  _ {a} Q _ {1} (s _ {1} ^ {l}, a)
$$
Finally, take  $x_h^k = (s_{h}^{k,h}, a_{h}^{k,h})$ ,  $y_h^k = (r_{h}^{k,h}, s_{h+1}^{k,h})$  and  $w^k = \{s_1^{k,h}\}_{h \in [H]}$ . It is important to note that  $s_{h}^{k,h}, a_{h}^{k,h}, r_{h}^{k,h}, s_{h+1}^{k,h}$  are taken from different trajectories for  $h \in [H]$ , so  $y_h^k \mid x_h^k$  are independent from one other for  $h \in [H]$ . Moreover, note that the distributions  $p^k \in \Delta(\Pi)$  play the role of nature: the distribution of the tuple  $(x_h^k, y_h^k, w^k)$  for  $h \in [H]$  is determined by running a policy  $\pi^k \sim p^k$  in the ground-truth MDP  $M^\star$ . With this configuration, observe that in the notation of Appendix C.3, we have, for any  $Q$ ,
$$
\begin{array}{l} \mathbb {E} _ {x _ {1: H} ^ {k}} \mathcal {E} (Q, Q, x _ {1: H} ^ {k}) ^ {2} = \frac {1}{H} \sum_ {h = 1} ^ {H} \mathbb {E} _ {x _ {h} ^ {k}} (\ell_ {h, 1} (x _ {h} ^ {k}, Q) - \mathbb {E} [ \ell_ {h, 2} (y _ {h}, Q) | x _ {h} ^ {k} ]) ^ {2} \\ = \frac {1}{H} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {k, h} \sim p ^ {k}} \mathbb {E} ^ {M ^ {\star}, \pi^ {k, h}} \left[ \left(Q _ {h} \left(s _ {h}, a _ {h}\right) - \left[ \mathcal {T} _ {h} ^ {M ^ {\star}} Q _ {h + 1} \right] \left(s _ {h}, a _ {h}\right)\right) ^ {2} \right] \\ = \frac {1}{H} \mathbb {E} _ {\pi \sim p ^ {k}} \mathbb {E} ^ {M ^ {\star}, \pi} \left[ \sum_ {h = 1} ^ {H} \left(Q _ {h} \left(s _ {h}, a _ {h}\right) - \left[ T _ {h} ^ {M ^ {\star}} Q _ {h + 1} \right] \left(s _ {h}, a _ {h}\right)\right) ^ {2} \right] \\ = \frac {1}{H} \mathbb {E} _ {\pi \sim p ^ {k}} D _ {\mathrm {s b e}} ^ {\pi} (Q \| M ^ {\star}). \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {x _ {1: H} ^ {k}, w ^ {k}} t ^ {k} (Q) = \frac {1}{H} \sum_ {l = 1} ^ {H} \mathbb {E} _ {\pi^ {k, h} \sim p ^ {k}} \mathbb {E} ^ {M ^ {\star}, \pi^ {k, h}} \left[ \max  _ {a} Q _ {1} ^ {*} \left(s _ {1} ^ {k, l}, a\right) - \max  _ {a} Q _ {1} \left(s _ {1} ^ {k, l}, a\right) \right] \\ = f ^ {M ^ {\star}} (\pi_ {M ^ {\star}}) - f ^ {Q} (\pi_ {Q}). \\ \end{array}
$$
Estimation error bound. We take  $\alpha = 12\beta$ , so that Theorem C.1 implies that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} \left(\frac {1}{H} \mathbb {E} _ {\pi \sim p ^ {k}} D _ {\mathbf {s b e}} ^ {\pi} (Q \| M ^ {\star}) + \alpha \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {Q} \left(\pi_ {Q}\right)\right)\right) \\ \lesssim \eta \alpha \log (| \mathcal {Q} | K / \delta) K + \log | \mathcal {Q} | / \eta + \alpha^ {2} K. \\ \end{array}
$$
Then by taking  $\alpha = \frac{1}{\gamma H}$ , this further implies that with probability at least  $1 - \delta$
$$
\begin{array}{l} \mathbf {O p t E s t} _ {\gamma} ^ {\mathrm {s b e}} = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi \sim p ^ {k}} \mathbb {E} _ {Q \sim \mu^ {k}} \left(D _ {\mathrm {s b e}} ^ {\pi} (Q \| M ^ {\star}) + \frac {1}{\gamma} \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {Q} \left(\pi_ {Q}\right)\right)\right) \\ \lesssim H (\eta \alpha \log (| \mathcal {Q} | K / \delta) K + \log | \mathcal {Q} | / \eta + \alpha^ {2} K) \\ \lesssim \frac {H \log | \mathcal {Q} |}{\eta} + \frac {\eta \log (| \mathcal {Q} | K / \delta) K}{\gamma} + \frac {K}{\gamma^ {2} H}. \\ \end{array}
$$
Proof of Corollary B.1. We choose  $n = H$  and apply Algorithm 4 as the estimation oracle. We first consider the "trivial" parameter regime in which  $Hd^{1/3}(\log(|\mathcal{Q}|K/\delta))^{-1/5}T^{-1/3} \geq 1/(2^{16}(\log(|\mathcal{Q}|K/\delta) + 1))$ . Here,  $T \lesssim Hd^{1/3}(\log(|\mathcal{Q}|K/\delta))^{4/5}T^{2/3}$ , and thus
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim H d ^ {1 / 3} (\log (| \mathcal {Q} | K / \delta)) ^ {4 / 5} T ^ {2 / 3}.
$$
When the case above, does not hold, we proceed as in the theorem statement, choosing  $\eta = Hd^{1/3}(\log(|\mathcal{Q}|K/\delta))^{-1/5}T^{-1/3} \leq 1/(2^{16}(\log(|\mathcal{Q}|K/\delta)+1))$ . Combining Theorem B.1 and Proposition B.1 then gives
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim \mathfrak {o} - \operatorname {d e c} _ {\gamma} ^ {\mathrm {s b e}} (\mathcal {M}) \cdot T + \gamma \frac {H ^ {2} \log | \mathcal {Q} |}{\eta} + \eta \log (| \mathcal {Q} | K / \delta) T + \frac {K}{\gamma^ {2}}
$$
with probability at least  $1 - \delta$ . Next, using Proposition 2.2 to bound  $\mathsf{o - dec}_{\gamma}^{\mathrm{sb}}(\mathcal{M})$  in the above display, it follows that
$$
\mathbf {R e g} _ {\mathrm {D M}} \lesssim \frac {H d T}{\gamma} + \gamma \frac {H ^ {2} \log | \mathcal {Q} |}{\eta} + \eta \log (| \mathcal {Q} | K / \delta) T + \frac {K}{\gamma^ {2}}.
$$
We choose  $\gamma = d^{2/3} (\log (|\mathcal{Q}| K / \delta))^{-2/5} T^{1/3}$  to obtain
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim H d ^ {1 / 3} (\log (| \mathcal {Q} | K / \delta)) ^ {4 / 5} T ^ {2 / 3}
$$
C Technical Tools
Lemma C.1. For all  $x \in [0,1]$ , we have
$$
e ^ {- x} \leq 1 - (1 - 1 / e) x \leq 1 - x / 2, \quad a n d \quad e ^ {x} \leq 1 + (e - 1) x \leq 1 + 2 x.
$$
Lemma C.2. For all  $x \geq -1/8$ , we have  $e^{-x} \leq 1 - x + x^2$ .