SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ e ^ {6 \eta \delta^ {t} (f)} \mid x _ {1: H} ^ {t} \right] = \mathbb {E} _ {t - 1} \left[ \exp \left(6 \eta \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \delta^ {t} (g, f)\right) \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp (6 \eta \delta^ {t} (g, f)) \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp \left(6 \eta (1 + 1 2 \eta) \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}\right) \mid x _ {1: H} ^ {t} \right] (Lemma C.6) \\ \leq 1 + 1 6 \eta \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] \quad (Lemma C.1) \\ \leq 1 + 6 4 \eta \mathbb {E} _ {t - 1} \left[ Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right]. (Lemma C.8) \\ \end{array}
$$
- Term (d): Using Lemma C.2, we have
$$
\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} e ^ {6 \eta \beta \iota^ {t} (f ^ {\prime})} \mid x _ {1: H} ^ {t} \right] \leq 1 + 6 \eta \beta + 3 6 \eta^ {2} \beta^ {2}.
$$
Term (e):
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} e ^ {6 \eta \delta^ {t} (f ^ {\prime}, f ^ {\prime})} \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} \exp (6 \eta (1 + 1 2 \eta) \mathcal {E} \left(f ^ {\prime}, f ^ {\prime}, x _ {1: H} ^ {t}\right) ^ {2}) \mid x _ {1: H} ^ {t} \right] (Lemma C.6) \\ \leq 1 + 1 6 \eta \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} \mathcal {E} \left(f ^ {\prime}, f ^ {\prime}, x _ {1: H} ^ {t}\right) ^ {2} \mid x _ {1: H} ^ {t} \right]. (Lemma C.1) \\ \end{array}
$$
- Term (f):
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} e ^ {- 6 \eta Z ^ {t} \left(f ^ {\prime}\right)} \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} 1 - 6 \eta Z ^ {t} \left(f ^ {\prime}\right) + 3 6 \eta^ {2} \left(Z ^ {t} \left(f ^ {\prime}\right)\right) ^ {2} \mid x _ {1: H} ^ {t} \right] \quad (\text {L e m m a C . 2}) \\ \leq 1 + \left(- 6 \eta + 1 4 4 0 \eta^ {2}\right) \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} Z ^ {t} \left(f ^ {\prime}\right) \mid x _ {1: H} ^ {t} \right] (Lemma C.15) \\ \leq 1. (Lemma C.8) \\ \end{array}
$$
Proof of Lemma C.12. Combining the preceding lemmas, we have with probability at least  $1 - \delta$  over the draw of  $S^T$ , the event  $A^t$  defined in Lemma C.14 holds for all  $t \leq T$  simultaneously, and thus
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left(e ^ {- \eta \left[ \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(1 6 \eta \mathbb {E} _ {t - 1} \left[ Z ^ {t} (f) + \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} \mathcal {E} \left(f ^ {\prime}, f ^ {\prime}, x _ {1: H} ^ {t}\right) ^ {2} \mid x _ {1: H} ^ {t} \right] + 1 2 \eta^ {2} \beta^ {2} + 2 \eta \beta\right) \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \right] \\ \leq 1 6 \eta \mathbb {E} _ {t \sim p ^ {t}} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(Z ^ {t} (f) + \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2}\right) \cdot 2 \log (| \mathcal {G} | T / \delta) \right] \mid x _ {1: H} ^ {t} \right] \\ + \left(2 4 \eta^ {2} \beta^ {2} + 4 \eta \beta\right) \log (| \mathcal {G} | T / \delta), \\ \end{array}
$$
where the first inequality is derived from Lemma C.17, and the second inequality follows from Lemma C.8 and Lemma C.14.
C.3.7 Proof of Lemma C.13 (Bound on Term II)
Proof of Lemma C.13. We compute
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(e ^ {- \eta \left[ \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) Z ^ {t} (f) \right] \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(e ^ {\eta \left(\beta | \iota^ {t} (f) | + | \delta^ {t} (f, f) | + | Z ^ {t} (f) | + | Z ^ {t} |\right)} - 1\right) | Z ^ {t} (f) | \right] \mid x _ {1: H} ^ {t} \right] \\ \leq 2 \eta \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left[ \beta | t ^ {t} (f) | + | \delta^ {t} (f, f) | + | Z ^ {t} (f) | + | Z ^ {t} | \right] | Z ^ {t} (f) | \right] \mid x _ {1: H} ^ {t} \right] \\ \leq \eta \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ | \delta^ {t} (f, f) | ^ {2} + 5 | Z ^ {t} (f) | ^ {2} + | Z ^ {t} | ^ {2} \right] \mid x _ {1: H} ^ {t} \right] + \eta \beta^ {2} \\ \leq \eta \mathbb {E} _ {t \sim p ^ {t}} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ 1 5 \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} + 2 8 0 Z ^ {t} (f) \right] \mid x _ {1: H} ^ {t} \right] + 4 \eta \beta^ {2}, \\ \end{array}
$$
where the second inequality applies Lemma C.1 (together with the fact that  $\eta \in (0, 1/240)$ ), the third inequality uses Young's inequality and the fact that  $|\iota^t(f)| \leq 1$  for all  $f$ , and the final inequality applies Lemma C.15 (to bound  $\mathbb{E}_{t-1}[|Z^t(f)|^2]$ ), Lemma C.16 (to bound  $\mathbb{E}_{t-1}[|Z^t|^2]$ ), and Lemma C.7 (to bound  $\mathbb{E}_{t-1}[\delta^t(f, f) \mid x_{1:H}^t]$ ).
D Proofs from Section 2 and Section 3
D.1 Proofs from Section 2
Proof of Theorem 2.1 and Theorem B.1. We prove Theorem B.1; Theorem 2.1 is the special case of this result in which  $n = 1$ . Observe that for Algorithm 3, we have
$$
\mathbf {R e g} _ {\mathsf {D M}} = n \cdot \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \left[ f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {M ^ {\star}} \left(\pi^ {k}\right) \right].
$$
We can rewrite this sum as
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \left[ f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {M ^ {\star}} \left(\pi^ {k}\right) \right] \\ = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {\widehat {\psi} ^ {k} \sim \mu^ {k}} \left[ f ^ {\widehat {\psi} ^ {k}} \left(\pi_ {\widehat {\psi} ^ {k}}\right) - f ^ {M ^ {\star}} \left(\pi^ {k}\right) \right] + \mathbb {E} _ {\widehat {\psi} ^ {k} \sim \mu^ {k}} \left[ \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {\widehat {\psi} ^ {k}} \left(\pi_ {\widehat {\psi} ^ {k}}\right)\right) \right] \\ = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {\widehat {\psi} ^ {k} \sim \mu^ {k}} \left[ f ^ {\widehat {\psi} ^ {k}} \left(\pi_ {\widehat {\psi} ^ {k}}\right) - f ^ {M ^ {\star}} \left(\pi^ {k}\right) - \gamma \cdot D ^ {\pi^ {k}} \left(\widehat {\psi} ^ {k} \| M ^ {\star}\right) \right] \\ + \mathbb {E} _ {\widehat {\psi} ^ {k} \sim \mu^ {k}} \left[ \gamma \cdot D ^ {\pi^ {k}} \left(\widehat {\psi} ^ {k} \| M ^ {\star}\right) + \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {\widehat {\psi} ^ {k}} \left(\pi_ {\widehat {\psi} ^ {k}}\right)\right) \right] \\ = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {\widehat {\psi} ^ {k} \sim \mu^ {k}} \left[ f ^ {\widehat {\psi} ^ {k}} \left(\pi_ {\widehat {\psi} ^ {k}}\right) - f ^ {M ^ {\star}} \left(\pi^ {k}\right) - \gamma \cdot D ^ {\pi^ {k}} \left(\widehat {\psi} ^ {k} \| M ^ {\star}\right) \right] + \gamma \cdot \mathbf {O p t E s t} _ {\gamma} ^ {D}. \\ \end{array}
$$
For each step  $k$ , by the choice of  $p^k$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {\widehat {\psi} ^ {k} \sim \mu^ {k}} \left[ f ^ {\widehat {\psi} ^ {k}} \left(\pi_ {\widehat {\psi} ^ {k}}\right) - f ^ {M ^ {\star}} \left(\pi^ {k}\right) - \gamma \cdot D ^ {\pi^ {k}} \left(\widehat {\psi} ^ {k} \| M ^ {\star}\right) \right] \\ \leq \sup  _ {M \in \mathcal {M}} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {\widehat {\psi} ^ {k} \sim \mu^ {k}} \left[ f ^ {\widehat {\psi} ^ {k}} \left(\pi_ {\widehat {\psi} ^ {k}}\right) - f ^ {M} \left(\pi^ {k}\right) - \gamma \cdot D ^ {\pi^ {k}} \left(\widehat {\psi} ^ {k} \| M\right) \right] \\ = \inf  _ {p \in \Delta (\Pi)} \sup  _ {M \in \mathcal {M}} \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {\widehat {\psi} \sim \mu^ {k}} \left[ f ^ {\widehat {\psi}} \left(\pi_ {\widehat {\psi}}\right) - f ^ {M} (\pi) - \gamma \cdot D ^ {\pi} \left(\widehat {\psi} \| M\right) \right] \\ = \mathbf {o} - \operatorname {d e c} _ {\gamma} ^ {D} (\mathcal {M}, \mu^ {k}) \leq \mathbf {o} - \operatorname {d e c} _ {\gamma} ^ {D} (\mathcal {M}). \\ \end{array}
$$
Finally, we use probability at least  $1 - \delta$ ,  $\mathbf{OptE s t}_{\gamma}^{D}\leq \mathbf{OptE s t}_{\gamma}^{D}(K,n,\delta)$ .
We next prove Proposition 2.1, which gives a bound on
$$
\mathbf {O p t E s t} _ {\gamma} ^ {\mathrm {b i}} = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {\widehat {M} ^ {k} \sim \mu^ {k}} \left[ \sum_ {h = 1} ^ {H} \left(\mathbb {E} ^ {M ^ {\star}, \pi} \left[ \ell_ {h} ^ {\mathrm {e s t}} \left(\widehat {Q} ^ {k}; z _ {h}\right) \right]\right) ^ {2} + \gamma^ {- 1} \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {\widehat {M} ^ {t}} \left(\pi_ {\widehat {M} ^ {k}}\right)\right) \right].
$$
Algorithm 6 Optimistic Estimation for Bilinear Classes
Number of rounds  $T$ , Batch size  $n$
- Learning rate  $\eta > 0$ .
- Discrepancy function  $\ell_h^{\mathrm{est}}(Q;z_h^{k,l})$  Exploration parameter  $\gamma >0$
2: Let  $K = T / n$  and  $B^0 \coloneqq \varnothing$ .
3: for  $k = 1,2,\dots ,K$  do
4: Form the randomized estimator  $\mu^k$  via  $\mu^k(Q) \propto \exp\left(-\eta \sum_{i < k} \ell^i(Q)\right)$ , where
$$
\ell^ {i} (Q) := \sum_ {h = 1} ^ {H} \left(\frac {1}{n} \sum_ {l = 1} ^ {n} \ell_ {h} ^ {\mathrm {e s t}} \left(Q; z _ {h} ^ {i, l}\right)\right) ^ {2} - \frac {1}{8 \gamma} \cdot f ^ {Q} (\pi_ {Q})
$$
5: Receive batch of samples  $B^{k} = \{(\pi^{k,l}, r^{k,l}, o^{k,l})\}_{l=1}^{n}$  where  $\pi^{k,l} \sim p^{k}$  and  $(r^{k,l}, o^{k,l}) \sim M^{\star}(\pi^{k,l})$ .
//  $p^k$  is the decision distribution produced by E2DOPT (cf. Algorithm 3).
6: Let  $z_{h}^{k,l} = (s_{h}^{k,l},a_{h}^{k,l},r_{h}^{k,l},s_{h + 1}^{k,l})$ , where we recall  $o^{k,l} = (s_1^{k,l},a_1^{k,1},r_1^{k,l}),\ldots ,(s_H^{k,l},a_H^{k,1},r_H^{k,l})$
Proof of Proposition 2.1. Throughout this proof, we use the batched estimation notation from Appendix B.1. Let  $n$  be fixed, and let  $K := T / n$  be the number of epochs. Recall that for each step  $k \in [K]$ , the estimation oracle is given a batch of examples  $B^{k} = \{(\pi^{k,l}, r^{k,l}, o^{k,l})\}_{l=1}^{n}$  where  $\pi^{k,l} \sim p^{k}$  and  $(r^{k,l}, o^{k,l}) \sim M^{\star}(\pi^{k,l})$ . Each observation (trajectory) takes the form  $o^{k,l} = (s_{1}^{k,l}, a_{1}^{k,l}, r_{1}^{k,l}), \ldots, (s_{H}^{k,l}, a_{H}^{k,l}, r_{H}^{k,l})$ . Throughout the proof, we use the notation
$$
\mathcal {E} _ {h} ^ {k} (Q) = \mathbb {E} ^ {M ^ {\star}, \pi^ {k}} \left[ \ell_ {h} ^ {\text {e s t}} (Q; z _ {h}) \right],
$$
and abbreviate  $Q^{\star} = Q^{M^{\star},\star}$
Estimation algorithm. Define
$$
\widehat {\mathcal {E}} _ {h} ^ {k} (Q) = \frac {1}{n} \sum_ {l = 1} ^ {n} \ell_ {h} ^ {\mathrm {e s t}} (Q; z _ {h} ^ {k, l}),
$$
where  $z_{h}^{k,l} \coloneqq (s_{h}^{k,l}, a_{h}^{k,l}, r_{h}^{k,l}, s_{h+1}^{k,l})$ . Let  $\eta > 0$  and  $\alpha > 0$  be parameters whose values will be chosen at the end of the proof. Defining
$$
\ell^ {k} (Q) := \sum_ {h = 1} ^ {H} \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2} - \alpha f ^ {Q} \left(\pi_ {Q}\right),
$$
Algorithm 6 chooses
$$
\mu^ {k} (Q) \propto \exp \left(- \eta \sum_ {i <   k} \ell^ {i} (Q)\right)
$$
as the randomized estimator for epoch  $k$ .
Estimation error bound. Let us abbreviate  $L = L_{\mathrm{bi}}(\mathcal{Q};M^{\star})$ . Observe that for all  $k$ ,  $|\ell^{k}(Q)| \leq R \coloneqq HL^{2} + \alpha$  almost surely. Hence, Lemma C.3 implies that as long as  $\eta \leq 1 / 2R$ , we have
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} [ \ell^ {k} (Q) ] - \sum_ {k = 1} ^ {K} \ell^ {k} (Q ^ {\star}) \leq 4 \eta \sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ (\ell^ {k} (Q)) ^ {2} \right] + \frac {\log | Q |}{\eta} \tag {25}
$$
For each  $k\in [K]$  , we have that for all  $Q\in \mathcal{Q}$
$$
\left(\ell^ {k} (Q)\right) ^ {2} \leq 2 \left(\sum_ {h = 1} ^ {H} \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2}\right) ^ {2} + 2 \alpha^ {2} \leq 2 H L ^ {2} \sum_ {h = 1} ^ {H} \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2} + 2 \alpha^ {2}.
$$
As a result, (25) implies that
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2} \right] + \alpha \sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {Q} \left(\pi_ {Q}\right) \right. \right] (26) \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q ^ {\star})\right) ^ {2} + 8 \eta H L ^ {2} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2} \right] + 8 \eta \alpha^ {2} K + \frac {\log | Q |}{\eta}. (27) \\ \end{array}
$$
Whenever  $\eta \leq \frac{1}{16HL^2}$ , rearranging gives
$$
\begin{array}{l} \frac {1}{2} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2} \right] + \alpha \sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {Q} \left(\pi_ {Q}\right) \right] \right. (28) \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\widehat {\mathcal {E}} _ {h} ^ {k} \left(Q ^ {\star}\right)\right) ^ {2} + 8 \eta \alpha^ {2} K + \frac {\log | Q |}{\eta}. (29) \\ \end{array}
$$
We now appeal to the following lemma.
Lemma D.1. With probability at least  $1 - \delta$ , it holds that for all  $k \in [K]$ ,  $h \in [H]$ , and  $Q \in \mathcal{Q}$
$$
\frac {1}{2} \left(\mathcal {E} _ {h} ^ {k} (Q)\right) ^ {2} - \varepsilon_ {\text {c o n c}} ^ {2} (n) \leq \left(\hat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2} \leq 2 \left(\mathcal {E} _ {h} ^ {k} (Q)\right) ^ {2} + 2 \varepsilon_ {\text {c o n c}} ^ {2} (n), \tag {30}
$$
where  $\varepsilon_{\mathrm{conc}}(n)\coloneqq L\sqrt{\frac{2\log(|Q|KH / \delta)}{n}}$
Going forward, we condition on the event in Lemma D.1. Applying the inequality (30) within (29), we have
$$
\begin{array}{l} \frac {1}{4} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(\mathcal {E} _ {h} ^ {k} (Q)\right) ^ {2} \right] + \alpha \sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {Q} \left(\pi_ {Q}\right) \right] \right. (31) \\ \leq 2 \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathcal {E} _ {h} ^ {k} \left(Q ^ {\star}\right)\right) ^ {2} + 8 \eta \alpha^ {2} K + \frac {\log | \mathcal {Q} |}{\eta} + 3 H K \varepsilon_ {\mathrm {c o n c}} ^ {2} (n) \\ = 8 \eta \alpha^ {2} K + \frac {\log | Q |}{\eta} + 3 H K \varepsilon_ {\text {c o n c}} ^ {2} (n), (32) \\ \end{array}
$$
where the last equality uses that  $\mathcal{E}_h^k (Q^\star) = 0$  for all  $h\in [H]$ , and  $k\in [K]$ , which is a consequence of Definition 2.1. Next, we recall that
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ \left(\mathcal {E} _ {h} ^ {k} (Q)\right) ^ {2} \right] = \sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ D _ {\mathrm {b i}} ^ {\pi^ {k}} (Q \| M ^ {\star}) \right].
$$
A standard application of Freedman's inequality (c.f Lemma A.3 in Foster et al. [12]) implies that with probability at least  $1 - \delta$ ,
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ D _ {\mathrm {b i}} ^ {\pi^ {k}} (Q \parallel M ^ {\star}) \right] \geq \frac {1}{2} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi \sim p ^ {k}} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ D _ {\mathrm {b i}} ^ {\pi^ {k}} (Q \parallel M ^ {\star}) \right] - O (H L ^ {2} \log (H / \delta)).
$$
Putting everything together, we have that
$$
\begin{array}{l} \frac {1}{8} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k} \sim p ^ {k}} \mathbb {E} _ {Q \sim \mu^ {k}} \left[ D _ {\mathrm {b i}} ^ {\pi^ {k}} (Q \| M ^ {\star}) + 8 \alpha \left(f ^ {M ^ {\star}} \left(\pi_ {M ^ {\star}}\right) - f ^ {Q} \left(\pi_ {Q}\right)\right) \right] \\ \lesssim \eta \alpha^ {2} K + \frac {\log | \mathcal {Q} |}{\eta} + H K \varepsilon_ {\mathrm {c o n c}} ^ {2} (n) + H L ^ {2} \log (H / \delta). \\ \end{array}
$$
Choosing  $\alpha = \frac{1}{8\gamma}$ , this gives
$$
\begin{array}{l} \mathbf {O p t E s t} _ {\gamma} ^ {\mathrm {b i}} \lesssim \eta \alpha^ {2} K + \frac {\log | \mathcal {Q} |}{\eta} + H K \varepsilon_ {\text {c o n c}} ^ {2} (n) + H L ^ {2} \log (H / \delta) \\ \lesssim \eta \alpha^ {2} K + \frac {\log | \mathcal {Q} |}{\eta} + H L ^ {2} \log (| \mathcal {Q} | K H / \delta) \left(1 + \frac {K}{n}\right) \\ \end{array}
$$
We choose  $\eta = \sqrt{\frac{\log|\mathcal{Q}|}{\alpha^2K}}\wedge \frac{1}{16R}$ , which satisfies the constraints described earlier in the proof, and gives
$$
\begin{array}{l} \mathbf {O p t E s t} _ {\gamma} ^ {\mathrm {b i}} \lesssim \frac {\sqrt {K \log | \mathcal {Q} |}}{\gamma} + R \log | \mathcal {Q} | + H L ^ {2} \log (| \mathcal {Q} | K H / \delta) \left(1 + \frac {K}{n}\right) \\ \lesssim \frac {\sqrt {K \log | \mathcal {Q} |}}{\gamma} + H L ^ {2} \log (| \mathcal {Q} | K H / \delta) \bigg (1 + \frac {1}{\gamma} + \frac {K}{n} \bigg). \\ \end{array}
$$
Proof of Lemma D.1. For any fixed  $k \in [K]$  and  $Q \in \mathcal{Q}$ , Hoeffding's inequality implies that with probability at least  $1 - \delta$ ,
$$
\left| \mathcal {E} _ {h} ^ {k} (Q) - \widehat {\mathcal {E}} _ {h} ^ {k} (Q) \right| \leq L \sqrt {\frac {2 \log (1 / \delta)}{n}}.
$$
By a standard union bound it follows that with probability at least  $1 - \delta$ , for all  $k \in [K]$ ,  $h \in [H]$ , and  $Q \in \mathcal{Q}$  simultaneously,
$$
\left| \mathcal {E} _ {h} ^ {k} (Q) - \widehat {\mathcal {E}} _ {h} ^ {k} (Q) \right| \leq L \sqrt {\frac {2 \log (| \mathcal {Q} | K H / \delta)}{n}} =: \varepsilon_ {\mathrm {c o n c}} (n).
$$
Whenever this event occurs, the AM-GM inequality implies that
$$
\left(\mathcal {E} _ {h} ^ {k} (Q)\right) ^ {2} \leq 2 \left(\widehat {\mathcal {E}} _ {h} ^ {k} (Q)\right) ^ {2} + 2 \varepsilon_ {\text {c o n c}} ^ {2} (n)
$$
and likewise  $(\widehat{\mathcal{E}}_h^k (Q))^2\leq 2(\mathcal{E}_h^k (Q))^2 +2\varepsilon_{\mathrm{conc}}^2 (n)$
Proof of Proposition 2.2. Let  $\mu \in \Delta(\mathcal{Q})$  be fixed. Fix  $\alpha \in (0,1)$ , and let  $\pi_Q^\alpha$  be the randomized policy that—for each  $h$ —independently plays  $\pi_{Q,h}$  with probability  $1 - \alpha / H$  and  $\pi_{Q,h}^{\mathrm{est}}$  with probability  $\alpha / H$ . Let  $p \in \Delta(\Pi)$  be the distribution induced by sampling  $Q \in \mu$  and playing  $\pi_Q^\alpha$ . Translated to our notation, Foster et al. [12] (cf. Proof of Theorem 7.1, Eq. (152)) shows that for all MDPs  $M$ , this strategy guarantees that for all  $\eta > 0$ ,
$$
\begin{array}{l} \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {Q \sim \mu} \left[ f ^ {Q} (\pi_ {Q}) - f ^ {M} (\pi) \right] \leq \alpha + \frac {H \cdot d _ {\mathrm {b i}} (Q ; M)}{2 \eta} + \frac {\eta}{2} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q, Q ^ {\prime} \sim \mu} \left[ \left\langle X _ {h} (Q; M), W _ {h} (Q ^ {\prime}; M) \right\rangle^ {2} \right] \\ = \alpha + \frac {H \cdot d _ {\mathrm {b i}} (\mathcal {Q} ; M)}{2 \eta} + \frac {\eta}{2} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q, Q ^ {\prime} \sim \mu} \left[ \left(\mathbb {E} ^ {M, \pi_ {Q} \circ_ {h} \pi_ {Q} ^ {\mathrm {e s t}}} \left[ \ell_ {h} ^ {\mathrm {e s t}} (Q ^ {\prime}; z _ {h}) \right]\right) ^ {2} \right]. \\ \end{array}
$$
In the on-policy case in which  $\pi_Q^{\mathrm{est}} = \pi_Q$ , it suffices to set  $\alpha = 0$ , which gives
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q, Q ^ {\prime} \sim \mu} \left[ \left(\mathbb {E} ^ {M, \pi_ {Q} \circ_ {h} \pi_ {Q} ^ {\mathrm {e s t}}} \left[ \ell_ {h} ^ {\mathrm {e s t}} \left(Q ^ {\prime}; z _ {h}\right) \right]\right) ^ {2} \right] \\ = \sum_ {h = 1} ^ {H} \mathbb {E} _ {Q, Q ^ {\prime} \sim \mu} \left[ \left(\mathbb {E} ^ {M, \pi_ {Q}} \left[ \ell_ {h} ^ {\text {e s t}} \left(Q ^ {\prime}; z _ {h}\right) \right]\right) ^ {2} \right] \\ = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {Q ^ {\prime} \sim \mu} \left[ \left(\mathbb {E} ^ {M, \pi} \left[ \ell_ {h} ^ {\text {e s t}} \left(Q ^ {\prime}; z _ {h}\right) \right]\right) ^ {2} \right] \\ = \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {Q \sim \mu} \left[ D _ {\mathrm {b i}} ^ {\pi} (Q \| M) \right], \\ \end{array}
$$
where the last equality relabels  $Q^{\prime} \gets Q$ . Setting  $\eta = 2\gamma$  yields the result.
In the general case where  $\pi_Q^{\mathrm{est}}\neq \pi_Q$  , we have that for all  $Q^{\prime}\in \mathcal{Q},$
$$
\mathbb {E} _ {\pi \sim p} \Big [ \big (\mathbb {E} ^ {M, \pi} \big [ \ell_ {h} ^ {\mathrm {e s t}} (Q ^ {\prime}; z _ {h}) \big ] \big) ^ {2} \Big ] \geq \frac {\alpha}{H} (1 - \alpha / H) ^ {H - 1} \mathbb {E} _ {Q \sim \mu} \Big [ \Big (\mathbb {E} ^ {M, \pi_ {Q} \circ_ {h} \pi_ {Q} ^ {\mathrm {e s t}}} \big [ \ell_ {h} ^ {\mathrm {e s t}} (Q ^ {\prime}; z _ {h}) \big ] \Big) ^ {2} \Big ].
$$
We have  $\frac{\alpha}{H} (1 - \alpha / H)^{H - 1} \geq \frac{\alpha}{2H}$  whenever  $\alpha \leq 1/2$ , which gives
$$
\begin{array}{l} \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {Q \sim \mu} \left[ f ^ {Q} \left(\pi_ {Q}\right) - f ^ {M} (\pi) \right] \leq \alpha + \frac {H \cdot d _ {\mathrm {b i}} (Q ; M)}{2 \eta} + \frac {\eta H}{\alpha} \cdot \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {Q \sim \mu} \left[ D _ {\mathrm {b i}} ^ {\pi} (Q \| M) \right] \\ = \alpha + \frac {H ^ {2} \cdot d _ {\mathrm {b i}} (Q ; M)}{2 \gamma \alpha} + \gamma \cdot \mathbb {E} _ {\pi \sim p} \mathbb {E} _ {Q \sim \mu} [ D _ {\mathrm {b i}} ^ {\pi} (Q \| M) ], \\ \end{array}
$$
where the last line chooses  $\eta = \gamma \alpha / H$ . To conclude, we set  $\alpha = \sqrt{H^2 d_{\mathrm{bi}}(\mathcal{Q}; \mathcal{M}) / 4\gamma}$ , which is admissible whenever  $\gamma \geq H^2 d_{\mathrm{bi}}(\mathcal{Q}; \mathcal{M})$ .
Proof of Corollary 2.1. In both of the cases in the theorem statement (on-policy and off-policy), whenever  $\gamma \geq 1$  and  $n \leq \sqrt{T}$  (to ensure that  $K / n \geq 1$ ), combining Theorem B.1 and Proposition 2.1 gives
$$
\mathbf {R e g} _ {\mathrm {D M}} \lesssim \mathsf {o - d e c} _ {\gamma} ^ {\mathrm {b i}} (\mathcal {M}) \cdot T + \gamma H L ^ {2} \log (| \mathcal {Q} | K H / \delta) \cdot \frac {T}{n} + \sqrt {n T \log | \mathcal {Q} |}.
$$
We choose  $n = \sqrt{T}$ , which is admissible whenever  $T$  is sufficiently large, and gives
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim \mathfrak {o} - \operatorname {d e c} _ {\gamma} ^ {\mathrm {b i}} (\mathcal {M}) \cdot T + \gamma H L ^ {2} \log (| \mathcal {Q} | K H / \delta) \sqrt {T} + \sqrt {\log | \mathcal {Q} |} \cdot T ^ {3 / 4}.
$$
In the on-policy case, we have, from Proposition 2.2, that
$$
\operatorname {R e g} _ {\mathrm {D M}} \lesssim \frac {H d T}{\gamma} + \gamma H L ^ {2} \log (| \mathcal {Q} | K H / \delta) \sqrt {T} + \sqrt {\log | \mathcal {Q} |} \cdot T ^ {3 / 4},
$$
and in the off-policy case, we have
$$
\mathbf {R e g} _ {\mathrm {D M}} \lesssim \sqrt {\frac {H ^ {2} d}{\gamma}} T + \gamma H L ^ {2} \log (| \mathcal {Q} | K H / \delta) \sqrt {T} + \sqrt {\log | \mathcal {Q} |} \cdot T ^ {3 / 4}.
$$
Choosing  $\gamma$  to balance yields the result.
Note that this result uses that the quantity  $\mathbb{E}^{M,\pi}\left[\ell_h^{\mathrm{est}}(Q';z_h)\right]$  only depends on the policy  $\pi$  through  $a_1,\ldots ,a_h$ .
D.2 Proofs from Section 3
Proof of Proposition 3.1. We begin with the upper bound. First, note that by Assumption 3.1 and the AM-GM inequality, we have