SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} Z _ {t} = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \widehat {\Phi} (f, \tilde {\pi}; \mathcal {D}) + \lambda \Delta f _ {1} (s _ {1}, \tilde {\pi}) + \ln \hat {p} (f | \mathcal {D}, \tilde {\pi}) \right] \\ = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ \widehat {\Phi} (f, \tilde {\pi}; \mathcal {D}) + \lambda \Delta f _ {1} (s _ {1}, \tilde {\pi}) + \ln p (f) \right] \\ \leq \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ \ln \frac {p (f)}{p _ {0} (f)} + \alpha \gamma \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) + \lambda \Delta f _ {1} \left(s _ {1}, \tilde {\pi}\right) \right] \\ + \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ \alpha \sum_ {h = 1} ^ {H} \ln \mathbb {E} _ {\tilde {f} _ {h} \sim p _ {0}} \exp \left(- \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} \left(\tilde {f} _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right)\right) \right] \\ \end{array}
$$
where the second equality uses the fact that  $D_{\mathrm{KL}}[p\| \hat{p} ]\geq 0$  with the minimum occurring at  $p = \hat{p}$  and the inequality uses the triangle inequality. The first term is bounded by Lemma E.9 and Lemma E.11, and the second term is bounded by Lemma E.10
It remains to state and prove Lemma E.9, Lemma E.11 and Lemma E.10.
The following lemma bounds the in-expectation of the loss  $\Delta L_{\bar{\pi}}$  by the in-expectation of the squared Bellman error.
Lemma E.9. For any distribution  $p$  over  $\mathcal{F}$ , for any  $\epsilon, \delta > 0$ ,  $\gamma > 0$ , any  $t \in [T]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \alpha \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h} ^ {k}) \right] \\ \leq \gamma \left(\alpha + \frac {3 (e - 2)}{\alpha}\right) \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} \mathcal {E} _ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}) (s _ {h} ^ {k}, a _ {h} ^ {k}) ^ {2} \right] \\ + 3 0 \alpha \gamma b (b + 2) K \epsilon + 1 3 \alpha \gamma b ^ {2} \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)\right) + 9 \alpha \gamma K b ^ {2} \delta . \\ \end{array}
$$
Proof of Lemma E.9. For simplicity, define
$$
x _ {k} := \alpha \gamma \Delta L _ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right),
$$
$$
e _ {k} := \mathcal {E} _ {h} ^ {\bar {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right).
$$
$$
\mathbb {E} [ x _ {k} ] = \alpha \gamma e _ {k} ^ {2},
$$
$$
\mathbb {E} \left[ x _ {k} ^ {2} \right] \leq 3 6 b ^ {2} \alpha^ {2} \gamma^ {2} e _ {k} ^ {2}.
$$
Thus, by Lemma B.2, for any  $\delta > 0$ , with probability at least  $1 - \delta$ , for any  $t \in [0, \frac{1}{13\alpha\gamma b^2}]$  we have
$$
\sum_ {k = 1} ^ {K} x _ {k} \leq \sum_ {k = 1} ^ {K} \mathbb {E} [ x _ {k} ] + t (e - 2) \sum_ {k = 1} ^ {K} \mathbb {E} [ x _ {k} ^ {2} ] + \frac {\ln (1 / \delta)}{t}
$$
$$
\leq \left(\alpha \gamma + t (e - 2) 3 6 b ^ {2} \gamma^ {2}\right) \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + \frac {\ln (1 / \delta)}{t}.
$$
Using the discretization argument and the union bound, we have that: For any  $\epsilon > 0$ ,  $\delta > 0$ , we have
$$
\Pr (E) \geq 1 - \delta , \text {t h u s} \Pr (E ^ {c}) \leq \delta ,
$$
where  $E$  denotes that event that for any  $t \in [0, \frac{1}{13\alpha\gamma b^2}]$ ,
$$
\sum_ {k = 1} ^ {K} x _ {k} \leq 3 0 \alpha \gamma b (b + 2) K \epsilon + \left(\alpha \gamma + t (e - 2) 3 6 b ^ {2} \alpha^ {2} \gamma^ {2}\right) \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + \frac {2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)}{t},
$$
any  $f_{h}\in \mathcal{F}_{h},f_{h + 1}\in \mathcal{F}_{h + 1},\tilde{\pi}\in \Pi_{h}^{soft}(T)$  . Thus, we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} x _ {k} \right] = \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} x _ {k} \right] + \mathbb {E} _ {\mathcal {D}} 1 \{E ^ {c} \} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} x _ {k} \right] \\ \leq \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ 3 0 \alpha \gamma b (b + 2) K \epsilon \right. \\ \left. + \left(\alpha \gamma + t (e - 2) 3 6 \alpha^ {2} b ^ {2} \gamma^ {2}\right) \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + \frac {2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)}{t} \right] + 9 \alpha \gamma K b ^ {2} \delta \\ \leq \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ 3 0 b (b + 2) K \epsilon \right. \\ \left. + \left(\alpha \gamma + t (e - 2) 3 6 \alpha^ {2} b ^ {2} \gamma^ {2}\right) \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + \frac {2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)}{t} \right] + 9 \alpha \gamma K b ^ {2} \delta . \\ \end{array}
$$
Picking  $t = \frac{1}{13\alpha\gamma b^2}$  completes the proof.
The following lemma bounds the in-expectation negation of the loss proxy  $\Delta L_{\bar{\pi}}$
Lemma E.10. For any  $\delta > 0, \epsilon > 0, \gamma > 0$ , any  $\tilde{f}_h \in \mathcal{F}_h$ , any  $t \in [T]$ , and any distribution  $p$  over  $\mathcal{F}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ - \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} \left(\tilde {f} _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) \right] \leq 3 6 (e - 2) \gamma b ^ {2} \left(d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)\right) \\ + 9 \gamma K b ^ {2} \delta + 3 0 \gamma b (b + 2) K \epsilon . \\ \end{array}
$$
Proof of Lemma E.10. For simplicity, define
$$
y _ {k} := - \gamma \Delta L _ {\bar {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right),
$$
$$
\mathbb {E} [ y _ {k} ] = - \gamma e _ {k} ^ {2},
$$
$$
\mathbb {E} \left[ y _ {k} ^ {2} \right] \leq 3 6 b ^ {2} \gamma^ {2} e _ {k} ^ {2}.
$$
Thus, by Lemma B.2, for any  $\delta > 0$ , with probability at least  $1 - \delta$ , for any  $t \in [0, \frac{1}{13\gamma b^2}]$  we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} y _ {k} \leq \sum_ {k = 1} ^ {K} \mathbb {E} [ y _ {k} ] + t (e - 2) \sum_ {k = 1} ^ {K} \mathbb {E} [ y _ {k} ^ {2} ] + \frac {\ln (1 / \delta)}{t} \\ \leq - \gamma \left(1 - 3 6 t (e - 2) b ^ {2} \gamma\right) \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + \frac {\ln (1 / \delta)}{t}. \\ \end{array}
$$
Setting  $t = \frac{1}{36(e - 2)b^2\gamma} < \frac{1}{13\gamma b^2}$  in the above inequality, we obtain
$$
\sum_ {k = 1} ^ {k} y _ {k} \leq 3 6 (e - 2) b ^ {2} \gamma \ln (1 / \delta).
$$
where  $E$  denotes that event,
$$
\sum_ {k = 1} ^ {K} y _ {k} \leq 3 0 \gamma b (b + 2) K \epsilon + 3 6 (e - 2) \gamma b ^ {2} \left(d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)\right),
$$
any  $f_{h + 1}\in \mathcal{F}_{h + 1},\tilde{\pi}\in \Pi_h^{soft}(T)$  . Thus, we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} y _ {k} \right] = \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} y _ {k} \right] + \mathbb {E} _ {\mathcal {D}} 1 \{E ^ {c} \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} y _ {k} \right] \\ \leq 3 0 \gamma b (b + 2) K \epsilon + 3 6 (e - 2) \gamma b ^ {2} \left(d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)\right) + 9 \gamma K b ^ {2} \delta . \\ \end{array}
$$
The following lemma bounds the in-expectation squared Bellman errors with the regularization term and the data distribution term, under the infimum realization of the data distribution  $p$ .
Lemma E.11. For any  $\epsilon > 0, \beta \geq 0$ , any  $t \in [T]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ \lambda \Delta f _ {1} (s _ {1}, \tilde {\pi}) + \ln \frac {p (f)}{p _ {0} (f)} + \beta \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \mathcal {E} _ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}) (s _ {h} ^ {k}, a _ {h} ^ {k}) ^ {2} \right] \\ \leq \lambda \epsilon - \inf  _ {\tilde {\pi} \in \Pi^ {s o f t} (T)} \ln p _ {0} (\mathcal {F} _ {h} (\epsilon ; Q _ {h} ^ {\tilde {\pi}})) + 4 \beta H K \epsilon^ {2}. \\ \end{array}
$$
where recall that  $\mathcal{F}_h(\epsilon ;f_{h + 1})$  is defined in Definition 2.
Proof of Lemma E.11. For any  $f\in \mathcal{F}(\epsilon ;Q^{\tilde{\pi}})$  , we have
$$
\left\| f _ {h} - Q _ {h} ^ {\tilde {\pi}} \right\| _ {\infty} \leq \epsilon , \forall h.
$$
$$
\begin{array}{l} \left| \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) (s, a) \right| \leq \left\| \mathbb {T} _ {h} ^ {\tilde {\pi}} f _ {h} - f _ {h + 1} \right\| _ {\infty} = \left\| \mathbb {T} _ {h} ^ {\tilde {\pi}} f _ {h} - \mathbb {T} _ {h} ^ {\tilde {\pi}} Q _ {h} ^ {\tilde {\pi}} - f _ {h + 1} + Q _ {h + 1} ^ {\tilde {\pi}} \right\| _ {\infty} \\ \leq \left\| \mathbb {T} _ {h} ^ {\bar {\pi}} f _ {h} - \mathbb {T} _ {h} ^ {\bar {\pi}} Q _ {h} ^ {\bar {\pi}} \right\| _ {\infty} + \left\| f _ {h + 1} - Q _ {h + 1} ^ {\bar {\pi}} \right\| _ {\infty} \\ \leq 2 \epsilon . \\ \end{array}
$$
Thus, by choosing
$$
p (f) = \frac {p _ {0} (f) 1 \{f \in \mathcal {F} (\epsilon ; \tilde {\pi}) \}}{p _ {0} (\mathcal {F} (\epsilon ; \tilde {\pi}))},
$$
$$
\begin{array}{l} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ \lambda \Delta f _ {1} (s _ {1}, \tilde {\pi}) + \ln \frac {p (f)}{p _ {0} (f)} + \beta \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {2} \right] \\ \leq \lambda \epsilon - \ln p _ {0} \left(\mathcal {F} _ {h} (\epsilon ; \tilde {\pi})\right) + 4 \beta H K \epsilon^ {2}. \\ \end{array}
$$
We by now have everything needed to prove Proposition 2.
E.3.3 Proof of Proposition 2
Proof of Proposition 2. By Proposition 3, we have
$$
\begin{array}{l} Z _ {t} \geq \lambda \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \Delta f _ {1} (s _ {1}, \tilde {\pi}) \\ + 0. 1 2 5 \alpha \gamma \sum_ {h = 1} ^ {H} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\mu^ {k}} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] \\ - 0. 5 \alpha H \left(1 2 0 \gamma b (b + 2) K \epsilon + 6 4 0 (e - 2) \gamma b ^ {2} \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)\right)\right) - 1 3 \alpha \gamma b ^ {2} K H \delta \\ - 0. 2 5 \alpha \gamma H \left(b (b + 2) K \epsilon + \frac {3 2}{3} b ^ {2} \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \frac {\ln \ln 4 K b ^ {2}}{\delta}\right) + 1 + 2 K b ^ {2} \delta\right) \\ - \sum_ {h = 1} ^ {H} \max  _ {\tilde {\pi} \in \Pi^ {s o f t} (T)} \kappa_ {h} (\alpha , \epsilon , \tilde {\pi}) - \gamma \alpha 6 b K H \epsilon . \\ \end{array}
$$
By Proposition 4, we have
$$
\begin{array}{l} Z _ {t} \leq \lambda \epsilon - \inf  _ {\tilde {\pi} \in \Pi^ {s o f t} (T)} \sum_ {h = 1} ^ {H} \ln p _ {0} \left(\mathcal {F} _ {h} (\epsilon ; \tilde {\pi})\right) + 4 \gamma \left(\alpha + \frac {3 (e - 2)}{\alpha}\right) H K \epsilon^ {2} \\ + 6 0 \alpha \gamma b (b + 2) K H \epsilon + \alpha b ^ {2} \gamma H (1 3 + 3 6 (e - 2)) (2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)) + 1 8 \alpha \gamma K H b ^ {2} \delta . \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ 0. 1 2 5 \alpha \gamma K \sum_ {h = 1} ^ {H} \mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] + \lambda \Delta f _ {1} \left(s _ {1}, \tilde {\pi}\right) \right] \\ \lesssim \lambda \epsilon + \alpha \gamma H b ^ {2} \cdot \max  \{d _ {\mathcal {F}} (\epsilon), d _ {\Pi} (\epsilon , T), \ln \frac {\ln K b ^ {2}}{\delta} \} + \alpha \gamma b ^ {2} K H \cdot \max  \{\epsilon , \delta \} + \gamma H K \frac {\epsilon^ {2}}{\alpha} \\ + \sum_ {h = 1} ^ {H} \max  _ {\tilde {\pi} _ {h} \in \Pi_ {h} ^ {s o f t}} \kappa_ {h} (\alpha , \epsilon , \tilde {\pi} _ {h}) + \sup  _ {\tilde {\pi} \in \Pi^ {s o f t} (T)} \sum_ {h = 1} ^ {H} \ln \frac {1}{p _ {0} \left(\mathcal {F} _ {h} \left(\epsilon ; Q _ {h} ^ {\tilde {\pi} _ {h}}\right)\right)}. \\ \end{array}
$$
Appendix F Proof of Proposition 1
In this appendix, we prove Proposition 1, which is a simple reduction from Theorem 1, Theorem 2, and Theorem 3.
Proof of Proposition 1. We recall that Proposition 1 consists of two parts of statements: Part (i) – the simplified bounds of all three algorithms into one unified form under no misspecification, and Part (ii) – the specialization of the unified bound into the special cases of finite function classes and linear function classes.
Part (i): The unified sub-optimality bounds for VS, RO, and PS
We recall that the first part of Proposition 1 is that:
$$
\forall \hat {\pi} \in \left\{\hat {\pi} ^ {v s}, \hat {\pi} ^ {r o}, \hat {\pi} ^ {p s} \right\}, \mathbb {E} _ {\mathcal {D}} \operatorname {S u b O p t} _ {\pi} (\hat {\pi}) = \tilde {\mathcal {O}} \left(\frac {H b}{\sqrt {K}} \sqrt {\tilde {d} (1 / K) \cdot \mathcal {C} (\pi ; 1 / \sqrt {K})} + \frac {H b \sqrt {\ln \operatorname {V o l} (\mathcal {A})}}{T}\right), \tag {19}
$$
$$
\tilde {d} (1 / K) = \left\{ \begin{array}{l l} \tilde {d} _ {o p t} (1 / K, T) & \text {i f} \hat {\pi} \in \{\hat {\pi} ^ {v s}, \hat {\pi} ^ {r o} \}, \\ \tilde {d} _ {p s} (1 / K, T) & \text {i f} \hat {\pi} = \hat {\pi} ^ {p s}, \end{array} \right.
$$
where we recall in Section 4.2 that
$$
\tilde {d} _ {\text {o p t}} (\epsilon , T) := \max  \left\{d _ {\mathcal {F}} (\epsilon), d _ {\Pi} (\epsilon , T) \right\},
$$
$$
\tilde {d} _ {p s} (\epsilon , T) := \max  \{d _ {\mathcal {F}} (\epsilon), d _ {\Pi} (\epsilon , T), \frac {d _ {0} (\epsilon)}{\gamma H b ^ {2}}, \frac {d _ {0} ^ {\prime} (\epsilon)}{\gamma H b ^ {2}} \},
$$
and  $d_{\mathcal{F}}(\epsilon), d_{\Pi}(\epsilon, T), d_0(\epsilon)$ , and  $d_0'(\epsilon)$  are defined in Section 2.4. Also recall that for Proposition 1, we assume that there is no misspecification, i.e.,  $\xi_h = \nu_h = 0, \forall h \in [H]$ .
For  $\hat{\pi}^{vs}$ . It follows from Theorem 1, where we choose  $\epsilon_c = 1 / \sqrt{K}$ , and  $\epsilon = 1 / K$  that with probability at least  $1 - 2\delta$ , we have
$$
\begin{array}{l} \mathrm {S u b O p t} _ {\pi} (\hat {\pi} ^ {v s}) \lesssim \sqrt {K ^ {- 1} \cdot H \cdot \mathcal {C} (\pi ; 1 / \sqrt {K}) (H b ^ {2} \max \{\tilde {d} _ {o p t} (1 / K , T) , \ln (H / \delta) \} + b ^ {2} H)} + H / \sqrt {K} + \zeta_ {o p t} \\ \lesssim \sqrt {K ^ {- 1} \cdot H ^ {2} b ^ {2} \cdot \mathcal {C} (\pi ; 1 / \sqrt {K}) \max \{\tilde {d} _ {o p t} (1 / K , T) , \ln (H / \delta) \}} + \frac {H b \sqrt {\ln \mathrm {V o l} (\mathcal {A})}}{T} \\ \end{array}
$$
$$
\operatorname {S u b O p t} _ {\pi} \left(\hat {\pi} ^ {v s}\right) = \mathcal {O} \left(\frac {H b}{\sqrt {K}} \sqrt {\mathcal {C} (\pi ; 1 / \sqrt {K}) \cdot \max  \left\{\tilde {d} _ {o p t} (1 / K , T) , \ln (H / \delta) \right\}} + \frac {H b \sqrt {\ln \operatorname {V o l} (\mathcal {A})}}{T}\right). \tag {20}
$$
For  $\hat{\pi}^{ro}$ . The sub-optimality bound for  $\hat{\pi}^{ro}$  is obtained from Theorem 2 with the same parameter setting as that for  $\hat{\pi}^{vs}$ , where we set  $\epsilon = 1 / K$ ,  $\epsilon_{c} = 1 / \sqrt{K}$ , and  $T \geq K \ln \operatorname{Vol}(\mathcal{A})$ . Additionally, we shall need to set the regularization parameter  $\lambda$ . Since the bound in Theorem 2 holds for any  $\lambda > 0$ , we shall minimize this bound with respect to  $\lambda > 0$ , which results in the optimal  $\lambda$  as
$$
\lambda_ {*} = \sqrt {\frac {2 K H b ^ {2} \cdot \max \{\tilde {d} _ {o p t} (1 / K , T) , \ln (H / \delta) \}}{H \cdot \mathcal {C} (\pi , 1 / \sqrt {K})}}.
$$
and the sub-optimality bound as
$$
\operatorname {S u b O p t} _ {\pi} \left(\hat {\pi} ^ {r o}\right) = \mathcal {O} \left(\frac {H b}{\sqrt {K}} \sqrt {\mathcal {C} (\pi ; 1 / \sqrt {K}) \cdot \max  \{\tilde {d} _ {o p t} (1 / K , T) , \ln (H / \delta) \}} + \frac {H b \sqrt {\ln \operatorname {V o l} (\mathcal {A})}}{T}\right). \tag {21}
$$
For  $\hat{\pi}^{ps}$ . We specialize the sub-optimality of  $\hat{\pi}^{ps}$  from Theorem 3. Similar to the case of  $\hat{\pi}^{vs}$  and  $\hat{\pi}^{ro}$ , we set:  $\epsilon = 1 / K$ ,  $\epsilon_c = 1 / \sqrt{K}$ , and  $T \geq K \ln \operatorname{Vol}(\mathcal{A})$ . Additionally, we need to set the failure probability  $\delta \in [0,1]$ , the learning rate  $\gamma \in [0, \frac{1}{144(e - 2)b^2}]$  and the regularization parameter  $\lambda > 0$ . For  $\delta$ , we set  $\delta = 1 / K$ . For  $\lambda$ , we minimize the bound in Theorem 3 with respect to  $\lambda$ , which results into  $\lambda = \lambda_*$  which is given as
$$
\lambda_ {*} = \gamma \sqrt {\frac {K H b ^ {2} \cdot \max \{\tilde {d} _ {p s} (1 / K , T) , \ln (K \ln (K b ^ {2})) \}}{H \cdot \mathcal {C} (\pi , 1 / \sqrt {K})}},
$$
turns the sub-optimality bound into
$$
\mathbb {E} _ {\mathcal {D}} \operatorname {S u b O p t} _ {\pi} \left(\hat {\pi} ^ {p s}\right) = \mathcal {O} \left(\frac {H b}{\sqrt {K}} \sqrt {\mathcal {C} \left(\pi ; 1 / \sqrt {K}\right) \cdot \max  \left\{\tilde {d} _ {p s} \left(1 / K , T\right) , \ln \left(K \ln \left(K b ^ {2}\right)\right) \right\}} + \frac {H b \sqrt {\ln \operatorname {V o l} (\mathcal {A})}}{T}\right).
$$
Finally, we choose  $\gamma \in [0,\frac{1}{144(e - 2)b^2} ]$  to minimize  $\tilde{d}_{ps}(\epsilon ,T) = \max \{d_{\mathcal{F}}(\epsilon),d_{\Pi}(\epsilon ,T),\frac{d_0(\epsilon)}{\gamma Hb^2},\frac{d_0'(\epsilon)}{\gamma Hb^2}\}$ , which occurs at  $\gamma = \frac{1}{144(e - 2)b^2}$ , and thus
$$
\tilde {d} _ {p s} (\epsilon , T) = \max  \left\{d _ {\mathcal {F}} (\epsilon), d _ {\Pi} (\epsilon , T), \frac {1 4 4 (e - 2) d _ {0} (\epsilon)}{H}, \frac {1 4 4 (e - 2) d _ {0} ^ {\prime} (\epsilon)}{H} \right\}. \tag {23}
$$
Overall, we have that Equation (20), Equation (21), and Equation (22) can be unified into Equation (19).
Part (ii): Specializing to the finite function classes and linear function classes
We consider two common cases.
Case 1. Finite function class. We consider the case that  $\mathcal{F}_h$  and  $\Pi_h^{\text{soft}}(T)$  have finite elements for all  $h\in [H]$ . Then we have  $\tilde{d} (\epsilon) = \mathcal{O}(\max_{h\in [H]}\max \{\ln |\mathcal{F}_h|,\ln |\Pi_h^{\text{soft}}(T)|\}),\forall \epsilon$ , due to that  $d_0^\prime (\epsilon)\leq d_0(\epsilon)\leq H\max_{h\in [H]}\ln |\mathcal{F}_h|$  and Equation (23).
Case 2. Linear function class. We consider the case that the function class  $\mathcal{F}_h$  is linear in some (known) feature map  $\phi_h: S \times \mathcal{A} \to \mathbb{R}^d$ . Concretely, the corresponding function class and the policy class defined in Section 2.3 are simplified into:
$$
\mathcal {F} _ {h} = \left\{(s, a) \mapsto \langle \phi_ {h} (s, a), w \rangle : \| w \| _ {2} \leq b \right\},
$$
$$
\Pi_ {h} ^ {\text {s o f t}} (T) := \left\{(s, a) \mapsto \frac {\exp (\langle \phi_ {h} (s , a) , \theta \rangle)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp (\langle \phi_ {h} (s , a ^ {\prime}) , \theta \rangle)}: \| \theta \| _ {2} \leq \eta T \right\}.
$$
$$
d _ {\mathcal {F}} (\epsilon) \leq d \ln \left(1 + \frac {2 b}{\epsilon}\right),
$$
$$
d _ {\Pi} (\epsilon , T) \leq d \ln (1 + \frac {1 6 \eta T}{\epsilon}),
$$
$$
d _ {0} ^ {\prime} (\epsilon) \leq d _ {0} (\epsilon) \leq c _ {1} d H \ln (c _ {2} / \epsilon),
$$
where the first two inequalities use [Zanette et al., 2021, Lemma 6] and the last inequality follows the discussion in Section 2.4. Note that  $d_{\Pi}(\epsilon, T)$  depends only logarithmically in  $T$ .
Appendix G Support Lemmas
In this section, for convenience, we present some simple yet useful lemmas that our proofs above often refer to.
The following lemma establishes the variance condition for the squared loss, which is typically used along with Bernstein's inequality.
Lemma G.1. Consider any real-valued function class  $\mathcal{F}$ . Consider the squared loss  $L(f(x),y) = (f(x) - y)^2$ . Assume bounded loss  $L(f(x),y)\leq M^2$  for any  $f\in \mathcal{F}$ , for some  $M > 0$ . Let  $f_{*}(x) = \mathbb{E}[y|x]$  and assume that  $L(f_{*}(x),y)\leq B^{2}$  for some  $B > 0$  (we do not require that  $f_{*}\in \mathcal{F}$ ). Let  $z = (x,y)$  and define
$$
\mathcal {G} = \{\phi (\cdot): \phi (z) = L (f (x), y) - L (f _ {*} (x), y), f \in \mathcal {F} \}.
$$
Then, for all  $\phi \in \mathcal{G}$ , we have
$$
\mathbb {E} _ {y} [ \phi (z) ^ {2} ] \leq 2 (M ^ {2} + B ^ {2}) \mathbb {E} _ {y} [ \phi (z) ], \forall x.
$$
where  $\mathbb{E}_y$  is the expectation taken over  $y$  given  $x$ .
Proof of Lemma G.1. Consider any  $\phi \in \mathcal{G}$  (with the corresponding  $f\in \mathcal{F}$ ). For any  $x$ , we have
$$
\mathbb {E} _ {y} [ \phi (z) ] = (f (x) - f _ {*} (x)) ^ {2}.
$$
$$
\begin{array}{l} \phi (z) ^ {2} = \left(f (x) - f _ {*} (x)\right) ^ {2} \left(f (x) + f _ {*} (x) - 2 y\right) ^ {2} \\ \leq (f (x) - f _ {*} (x)) ^ {2} 2 \left[ (f (x) - y) ^ {2} + (f _ {*} (x) - y) ^ {2} \right] \\ \leq 2 \left(M ^ {2} + B ^ {2}\right) \left(f (x) - f _ {*} (x)\right) ^ {2}. \\ \end{array}
$$
The first inequality uses Cauchy-Schwartz. The second inequality uses that  $f_* \in \mathcal{F}$  and  $L(f(x), y) \leq M^2, \forall f \in \mathcal{F}$ . Thus, for any  $x$ , we have
$$
\mathbb {E} _ {y} [ \phi (z) ^ {2} ] \leq 2 (M ^ {2} + B ^ {2}) (f (x) - f _ {*} (x)) ^ {2} = 2 (M ^ {2} + B ^ {2}) \mathbb {E} _ {y} [ \phi (z) ].
$$
The equation uses that  $\mathbb{E}_y[\phi (z)] = (f(x) - f_*(x))^2$
The following lemma is a simple decomposition of the value gap in the initial state, typically known as the performance difference lemma in the RL literature.
Lemma G.2 (Performance difference lemma). For any policy  $\pi, \widetilde{\pi}$ , we have
$$
V _ {1} ^ {\pi} (s _ {1}) - V _ {1} ^ {\widetilde {\pi}} (s _ {1}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ Q _ {h} ^ {\widetilde {\pi}} (s _ {h}, a _ {h}) - V _ {h} ^ {\widetilde {\pi}} (s _ {h}) \right],
$$
where  $\mathbb{E}_{\pi}$  denotes the expectation over the random trajectory  $(s_1, a_1, \ldots, s_h, a_h)$  generated by  $\pi$  (and the underlying MDP).
Proof of Lemma G.2. We simply expand  $V_{1}^{\pi}(s_{1}) = \mathbb{E}_{a_{1}, s_{2}|s_{1}, \pi}[r_{1}(s_{1}, a_{1}) + V_{2}^{\pi}(s_{2})]$  and use recursion to obtain the lemma.
The following lemma presents a simple connection from a form of a log partition function to the expectation under the infimum realization of the sampling distribution.
Lemma G.3. For any density functions  $p$  and  $p_0$  and any function  $f$ , we have