SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname{Proj}_{\mathcal{F}}(Q^{\tilde{\pi}}):= \operatorname *{arg  min}_{f\in \mathcal{F}}\left\{|f_{h}(s_{h},a_{h}) - Q^{\tilde{\pi}}_{h}(s_{h},a_{h})|,\forall h\in [H],(s_{h},a_{h})\in \operatorname {supp}(d_{h}^{\mu})\right\} .
$$
By Assumption 2.1, we have
$$
\left| \operatorname {P r o j} _ {\mathcal {F}} \left(Q ^ {\bar {\pi}}\right) \left(s _ {h}, a _ {h}\right) - Q _ {h} ^ {\bar {\pi}} \left(s _ {h}, a _ {h}\right) \right| \leq \xi_ {h}, \forall h \in [ H ], \left(s _ {h}, a _ {h}\right) \in \operatorname {s u p p} \left(d _ {h} ^ {\mu}\right).
$$
Definition 5 (Projection of Bellman operations). For any  $f \in \mathcal{F}$  and  $\tilde{\pi} \in \Pi^{soft}(T)$  for some  $T$ , we define the projection of the Bellman operation  $\mathbb{T}^{\tilde{\pi}}f$  onto  $\mathcal{F}$  as
$$
\operatorname{Proj}_{\mathcal{F}}(\mathbb{T}^{\tilde{\pi}}f):= \operatorname *{arg  min}_{f^{\prime}\in \mathcal{F}}\left\{\| f_{h}^{\prime} - \mathbb{T}_{h}^{\tilde{\pi}}f_{h + 1}\|_{\infty},\forall h\in [H]\right\} .
$$
By Assumption 2.2, we have
$$
\left\| \mathrm {P r o j} _ {\mathcal {F}} \big (\mathbb {T} ^ {\tilde {\pi}} f \big) - \mathbb {T} _ {h} ^ {\tilde {\pi}} f _ {h + 1} \right\| _ {\infty} \leq \nu_ {h}, \forall h \in [ H ].
$$
B.3 Induced MDPs
We now introduce the notion of induced MDPs which is originally used in [Zanette et al., 2021].
Definition 6 (Induced MDPs). For any policy  $\pi \in \Pi^{all}$  and any sequence of functions  $Q = \{Q_h\}_{h \in [H]} \in \{\mathcal{S} \times \mathcal{A} \to \mathbb{R}\}^H$ , the  $(Q, \pi)$ -induced MDPs, denoted by  $M(Q, \pi)$  is the MDP that is identical to the original MDP  $M$  except only that the expected reward of  $M(Q, \pi)$  is given by  $\{r_h^{\pi, Q}\}_{h \in [H]}$ , where
$$
r _ {h} ^ {\pi , Q} (s, a) := r _ {h} (s, a) - \mathcal {E} _ {h} ^ {\pi} (f _ {h}, f _ {h + 1}) (s, a).
$$
By definition of  $M(\pi, Q)$ ,  $Q$  is the fixed point of the Bellman equation  $Q_{h} = \mathbb{T}_{h,M(\pi, Q)}^{\pi}Q_{h + 1}$ .
Lemma B.3. For any  $\pi \in \Pi^{all}$  and any sequence of functions  $Q = \{Q_h\}_{h\in [H]}\in \{\mathcal{S}\times \mathcal{A}\to \mathbb{R}\} ^H$  we have
$$
Q _ {M (\pi , Q)} ^ {\pi} = Q,
$$
where  $M(\pi, Q)$  is the induced MDP given in Definition 6.
B.4 Error decomposition
The key starting point for the proofs of all of the three main theorems is the following error decomposition that decomposes the sub-optimality into three sources of errors: the Bellman error under the comparator policy  $\pi$ , the gap values in the initial states, and the online-regret term due to the induced MDPs. In online RL, the sub-optimality of a greedy policy against an optimal policy can be decomposed into the sub-optimality in the Bellman errors and the error in the initial states [Dann et al., 2021], using the standard value-function error decomposition in [Jiang et al., 2017, Lemma 1]. However, in our setting, we compete against an arbitrary policy  $\pi$  (not necessarily an optimal policy) and the learned policy  $\pi^t$  is not greedy with respect to the current action-value function  $Q^t$  - thus [Jiang et al., 2017, Lemma 1] cannot apply here. Instead, we develop an error decomposition - Lemma B.4 which generalizes what was implicit in Zanette et al. [2021].
Lemma B.4 (Error decomposition). For any action-value functions  $Q \in \{\mathcal{S} \times \mathcal{A} \to \mathbb{R}\}^H$  and any policies  $\pi, \tilde{\pi} \in \Pi^{all}$ , we have
$$
\mathrm {S u b O p t} _ {\pi} ^ {M} (\tilde {\pi}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} [ \mathcal {E} _ {h} ^ {\tilde {\pi}} (Q _ {h}, Q _ {h + 1}) (s _ {h}, a _ {h}) ] + Q _ {1} (s _ {1}, \tilde {\pi} _ {1}) - V _ {1} ^ {\tilde {\pi}} (s _ {1}) + \mathrm {S u b O p t} _ {\pi} ^ {M (Q, \tilde {\pi})} (\tilde {\pi}).
$$
Proof of Lemma B.4. We have
$$
\begin{array}{l} \operatorname {S u b O p t} _ {\pi} ^ {M} (\tilde {\pi}) = V _ {1} ^ {\pi} (s _ {1}) - V _ {1} ^ {\tilde {\pi}} (s _ {1}) \\ = \left(V _ {1} ^ {\pi} (s _ {1}) - V _ {1, M (Q, \tilde {\pi})} ^ {\pi} (s _ {1})\right) + \left(V _ {1, M (Q, \tilde {\pi})} ^ {\tilde {\pi}} (s _ {1}) - V _ {1} ^ {\tilde {\pi}} (s _ {1})\right) + \left(V _ {1, M (Q, \tilde {\pi})} ^ {\pi} (s _ {1}) - V _ {1, M (Q, \tilde {\pi})} ^ {\tilde {\pi}} (s _ {1})\right) \\ = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(Q _ {h}, Q _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) \right] + Q _ {1} \left(s _ {1}, \tilde {\pi} _ {1}\right) - V _ {1} ^ {\tilde {\pi}} \left(s _ {1}\right) + \operatorname {S u b O p t} _ {\pi} ^ {M (Q, \tilde {\pi})} (\tilde {\pi}), \\ \end{array}
$$
where in the last equality, for the first term, we use, by Definition 6, that
$$
V _ {1} ^ {\pi} (s _ {1}) - V _ {1, M (Q, \tilde {\pi})} ^ {\pi} (s _ {1}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ r _ {h} (s _ {h}, a _ {h}) - r _ {h} ^ {\tilde {\pi}, Q} (s _ {h}, a _ {h}) \right] = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} [ \mathcal {E} _ {h} ^ {\tilde {\pi}} (Q _ {h}, Q _ {h + 1}) (s _ {h}, a _ {h}) ],
$$
for the second term, we use, by Lemma B.3, that
$$
V _ {1, M (Q, \tilde {\pi})} ^ {\pi} (s _ {1}) = Q _ {1, M (Q, \tilde {\pi})} ^ {\pi} (s _ {1}, \tilde {\pi} _ {1}) = Q _ {1} (s _ {1}, \tilde {\pi}),
$$
and for the last term, we use the definition of value sub-optimality in Equation (1).
B.5 Decoupling lemma
One of the central tools for our proofs is the following decoupling lemma. The decoupling lemma essentially decouples the Bellman residuals under the  $\pi$ -induced state-action distribution into the squared Bellman residuals under the  $\mu$ -induced state-action distribution and the new data diversity measure in Definition 3 and additive terms of low order.
Lemma B.5 (Decoupling argument). Under Assumption 2.2, for any  $f \in \mathcal{F}$ , any  $\tilde{\pi} \in \Pi^{soft}(T)$  for some  $T$ , any  $\pi \in \Pi^{all}$ , any  $\lambda > 0$ , and any  $\epsilon \geq 0$ , we have
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) \right] \leq \frac {1}{2 \lambda} \sum_ {h = 1} ^ {H} \left(\sum_ {k = 1} ^ {K} \mathbb {E} _ {\mu^ {k}} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] + K \nu_ {h} ^ {2} + 4 b K \nu_ {h}\right) \\ + \frac {\lambda H \cdot \mathcal {C} (\pi ; \epsilon)}{2 K} + H \epsilon + \sum_ {h = 1} ^ {H} v _ {h}, \\ \end{array}
$$
where  $\mathcal{C}(\pi ;\epsilon)$  is defined in Definition 3.
Proof of Lemma B.5. We have
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {E} _ {h} ^ {\bar {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) \right] = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \left(\mathbb {T} _ {h} ^ {\bar {\pi}} f _ {h + 1} - f _ {h}\right) \left(s _ {h}, a _ {h}\right) \right] \\ \leq \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \left(\operatorname {P r o j} _ {\mathcal {F} _ {h}} \left(\mathbb {T} _ {h} ^ {\tilde {\pi}} f _ {h + 1}\right) - f _ {h}\right) \left(s _ {h}, a _ {h}\right) \right] + \bar {\nu} \\ \leq \sum_ {h = 1} ^ {H} \sqrt {\mathcal {C} (\pi , \epsilon) \mathbb {E} _ {\mu} \left[ \left(\operatorname {P r o j} _ {\mathcal {F} _ {h}} \left(\mathbb {T} _ {h} ^ {\tilde {\pi}} f _ {h + 1}\right) - f _ {h}\right) (s _ {h} , a _ {h}) ^ {2} \right]} + H \epsilon + \bar {\nu} \\ \leq \sum_ {h = 1} ^ {H} \sqrt {\mathcal {C} (\pi , \epsilon) \left(\mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h} , f _ {h + 1}\right) \left(s _ {h} , a _ {h}\right) ^ {2} \right] + \nu_ {h} ^ {2} + 4 b \nu_ {h}\right)} + H \epsilon + \bar {\nu} \\ \leq \sqrt {H C (\pi , \epsilon) \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\bar {\pi}} \left(f _ {h} , f _ {h + 1}\right) \left(s _ {h} , a _ {h}\right) ^ {2} \right] + \nu_ {h} ^ {2} + 4 b \nu_ {h}\right)} + H \epsilon + \bar {\nu} \\ \leq \frac {K}{2 \lambda} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] + \nu_ {h} ^ {2} + 4 b \nu_ {h}\right) + \frac {\lambda H \mathcal {C} (\pi , \epsilon)}{2 K} + H \epsilon + \bar {\nu}, \\ \end{array}
$$
where the first inequality uses Assumption 2.2, the second inequality uses the definition of  $C_{\pi}(\epsilon)$ , the third inequality uses Assumption 2.2 (again), the fourth inequality uses Cauchy-Schwartz inequality, and the last inequality uses the AM-GM inequality  $\sqrt{xy} \leq \frac{K}{2\lambda} x + \frac{\lambda}{2K} y$ .
B.6 Regret of the multiplicative weights algorithm for the actors
Now we establish the regret bound for the online-regret term due to the induced MDPs. The result in the following lemma is quite standard and can be readily generalized from a similar result in Zanette et al. [2021]. We present the proof here for completeness.
Lemma B.6. Consider an arbitrary sequence of value functions  $\{Q^t\}_{t\in [T]}$  such that  $\max_{h,t}\| Q_h^t\|_\infty \leq b$  and define the following sequence of policies  $\{\pi^t\}_{t\in [T + 1]}$  where
$$
\pi^ {1} (\cdot | s) = \operatorname {U n i f o r m} (\mathcal {A}), \forall s,
$$
$$
\pi_ {h} ^ {t + 1} (a | s) \propto \pi_ {h} ^ {t} (a | s) \exp (\eta Q _ {h} ^ {t} (s, a)), \forall (s, a, h, t).
$$
Suppose  $\eta = \sqrt{\frac{\ln\mathrm{Vol}(\mathcal{A})}{4(e - 2)b^2T}}$  and  $T\geq \frac{\ln\mathrm{Vol}(\mathcal{A})}{(e - 2)}$ , where  $\mathrm{Vol}(\mathcal{A})$  denotes the volume of the action set  $\mathcal{A}$ . For an arbitrary policy  $\pi \in \Pi^{all}$ , we have
$$
\sum_ {t = 1} ^ {T} \left(V _ {1, M (\pi^ {t}, Q ^ {t})} ^ {\pi} (s _ {1}) - V _ {1, M (\pi^ {t}, Q ^ {t})} ^ {\pi^ {t}} (s _ {1})\right) \leq 4 H b \sqrt {T \ln \operatorname {V o l} (\mathcal {A})}.
$$
<sup>8</sup>When  $|\mathcal{A}| < \infty$ ,  $\operatorname{Vol}(\mathcal{A}) = |\mathcal{A}|$ .
Proof of Lemma B.6. The proof for this lemma is quite standard as shown in Zanette et al. [2021]. We rewrite the proof with a slight modification for completeness. For simplicity, we write  $M_{t} \coloneqq M(\pi^{t}, Q^{t})$ . We will see that the key property that enables this lemma is that  $Q_{h}^{t} = Q_{h,M_{t}}^{\pi^{t}}$  (Lemma B.3), which allows us to relate the value difference lemma to the log policy ratio. Using the value difference lemma (Lemma G.2), we have
$$
V _ {1, M _ {t}} ^ {\pi} (s _ {1}) - V _ {1, M _ {t}} ^ {\pi^ {t}} (s _ {1}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} A _ {h, M _ {t}} ^ {\pi^ {t}} (s _ {h}, a _ {h}),
$$
where  $\mathbb{E}_{\pi}$  is the expectation over the random trajectory  $(s_1, a_1, \ldots, s_H, a_H)$  generated by  $\pi$  (and the underlying MDP  $M_t^9$ ). For any  $V: \mathcal{S} \to \mathbb{R}$ , it follows from the definition of  $\{\pi^t\}$  update that we have
$$
\begin{array}{l} \log \frac {\pi_ {h} ^ {t + 1} (a | s)}{\pi_ {h} ^ {t} (a | s)} = \eta Q _ {h} ^ {t} (s, a) - \log \left(\mathbb {E} _ {a \sim \pi_ {h} ^ {t} (\cdot | s)} [ \exp (\eta Q _ {h} ^ {t} (s, a)) ]\right) \\ = \eta (Q _ {h} ^ {t} (s, a) - V (s)) - \log \left(\mathbb {E} _ {a \sim \pi_ {h} ^ {t} (\cdot | s)} \left[ \exp \left(\eta (Q _ {h} ^ {t} (s, a) - V (s))\right) \right]\right). \\ \end{array}
$$
In the equation above, noting that  $Q_h^t = Q_{h,M_t}^\pi$  (Lemma B.3) and replacing  $V(s)$  by  $V_{h,M_t}^{\pi^t}$ , we have
$$
\log \frac {\pi_ {h} ^ {t + 1} (a | s)}{\pi_ {h} ^ {t} (a | s)} = \eta A _ {h, M _ {t}} ^ {\pi^ {t}} (s, a) - \log \left(\mathbb {E} _ {a \sim \pi_ {h} ^ {t} (\cdot | s)} \left[ \exp \left(\eta A _ {h, M _ {t}} ^ {\pi^ {t}} (s, a)\right) \right]\right), \tag {6}
$$
where we define the advantage function  $A_{M}^{\pi} = \{A_{h,M}^{\pi}\}_{h\in [H]}$  as
$$
A _ {h, M} ^ {\pi} (s, a) := Q _ {h, M} ^ {\pi} (s, a) - V _ {h, M} ^ {\pi} (s), \forall (s, a, h).
$$
Note that  $|A_{h,M_t}^\pi (s,a)|\leq 2b$ . By choosing  $\eta \in (0,1 / (2b))$ , we have
$$
\begin{array}{l} \left. \log \left(\mathbb {E} _ {a \sim \pi_ {h} ^ {t} (\cdot | s)} \left[ \exp \left(\eta A _ {h, M _ {t}} ^ {\pi^ {t}} (s, a)\right) \right]\right)\right) \\ \leq \log \left(\mathbb {E} _ {a \sim \pi_ {h} ^ {t} (\cdot | s)} \left[ 1 + \eta A _ {h, M _ {t}} ^ {\pi^ {t}} (s, a) + (e - 2) \eta^ {2} A _ {h, M _ {t}} ^ {\pi^ {t}} (s, a) ^ {2} \right]\right) \\ = \log \left(\mathbb {E} _ {a \sim \pi_ {h} ^ {t} (\cdot | s)} \left[ 1 + (e - 2) \eta^ {2} A _ {h, M _ {t}} ^ {\pi^ {t}} (s, a) ^ {2} \right]\right) \\ \leq \log (1 + (e - 2) \eta^ {2} 4 b ^ {2}) \\ \leq 4 (e - 2) b ^ {2} \eta^ {2} \tag {7} \\ \end{array}
$$
where the first inequality uses that  $e^x \leq 1 + x + (e - 2)x^2, \forall x \leq 1$  and  $|\eta A_{h,M_t}^\pi(s,a)| \leq 1, \forall (s,a)$ , the first equality uses that  $\mathbb{E}_{a \sim \pi_h^t(\cdot|s)}\left[A_{h,M_t}^\pi(s,a)\right] = 0$ , the second inequality uses that  $|A_{h,M_t}^\pi(s,a)| \leq 2b$ , and the last inequality uses that  $\log(1 + x) \leq x, \forall x \geq 0$ . Combining Equation (7) and Equation (6), we have
$$
A _ {h, M _ {t}} ^ {\pi^ {t}} (s, a) \leq \frac {1}{\eta} \log \frac {\pi_ {h} ^ {t + 1} (a | s)}{\pi_ {h} ^ {t} (a | s)} + 4 (e - 2) b ^ {2} \eta .
$$
Thus, for any  $h\in [H]$  , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi} A _ {h, M _ {t}} ^ {\pi^ {t}} (s _ {h}, a _ {h}) \\ \leq \frac {1}{\eta} \sum_ {t = 1} ^ {T} \left( \right.\mathbb {E} _ {\pi} \left[ K L [ \pi_ {h} (\cdot | s _ {h}) \| \pi_ {h} ^ {t} (\cdot | s _ {h}) ] \right] - \mathbb {E} _ {\pi} \left[ \right. K L [ \pi_ {h} (\cdot | s _ {h}) \| \pi_ {h} ^ {t + 1} (\cdot | s _ {h}) ] ]\left. \right) + 4 (e - 2) T b ^ {2} \eta \\ \end{array}
$$
Note that  $\operatorname*{Pr}((s_1, a_1, \ldots, s_H, a_H) | \pi, M) = \operatorname*{Pr}((s_1, a_1, \ldots, s_H, a_H) | \pi, M_t)$  since  $M_t$  and  $M$  have identical transition kernels.
$$
\begin{array}{l} = \frac {1}{\eta} \left(\mathbb {E} _ {\pi} \left[ K L [ \pi_ {h} (\cdot | s _ {h}) \| \pi_ {h} ^ {1} (\cdot | s _ {h}) ] \right] - \mathbb {E} _ {\pi} \left[ K L [ \pi_ {h} (\cdot | s _ {h}) \| \pi_ {h} ^ {T + 1} (\cdot | s _ {h}) ] \right]\right) + 4 (e - 2) T b ^ {2} \eta \\ \leq \frac {1}{\eta} \mathbb {E} _ {\pi} \left[ K L [ \pi_ {h} (\cdot | s _ {h}) \| \pi_ {h} ^ {1} (\cdot | s _ {h}) ] \right] + 4 (e - 2) T b ^ {2} \eta \\ \leq \frac {1}{\eta} \log (\operatorname {V o l} (\mathcal {A})) + 4 (e - 2) T b ^ {2} \eta , \\ \end{array}
$$
where the second inequality uses the non-negativity of KL divergence, and the last inequality uses that  $KL[\pi_h(\cdot |s_h)\| \pi_h^1 (\cdot |s_h)] = -H[\pi_h(\cdot |s_h)] + \log (\mathrm{Vol}(\mathcal{A}))\leq \log (\mathrm{Vol}(\mathcal{A}))$  where  $\pi^1$  is uniform over  $\mathcal{A}$  and  $\operatorname {Vol}(\mathcal{A})$  denotes the volume over the compact set  $\mathcal{A}$ . Combining all pieces together, we have
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi} A _ {h, M _ {t}} ^ {\pi^ {t}} \left(s _ {h}, a _ {h}\right) \leq \frac {1}{\eta} H \log (\operatorname {V o l} (\mathcal {A})) + 4 (e - 2) T H b ^ {2} \eta .
$$
Minimizing the RHS of the above equation with respect to  $\eta$  yields  $\eta = \sqrt{\frac{\log\operatorname{Vol}(\mathcal{A})}{4(e - 2)b^2T}}$  and
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {\pi} A _ {h, M _ {t}} ^ {\pi^ {t}} \left(s _ {h}, a _ {h}\right) \leq 2 H \sqrt {4 (e - 2) b ^ {2} T \log \operatorname {V o l} (\mathcal {A})} \leq 4 H b \sqrt {T \log \operatorname {V o l} (\mathcal {A})}.
$$
Finally, we need that
$$
\eta = \sqrt {\frac {\log \operatorname {V o l} (\mathcal {A})}{4 (e - 2) b ^ {2} T}} \leq \frac {1}{2 b},
$$
which implies  $T \geq \frac{\ln \operatorname{Vol}(\mathcal{A})}{(e - 2)}$ .
We are now ready to establish the proofs of our three main theorems.
Appendix C Proof of Theorem 1
To construct our proof for Theorem 1, we first establish two following lemmas. The first lemma, Lemma C.1 establishes that the in-distribution squared Bellman residuals are bounded by the unbiased proxy of the squared Bellman error  $\mathcal{L}_{\tilde{\pi}}(f)$ , up to some estimation and approximation errors. The second lemma, Lemma C.2, asserts that the unbiased proxy of the squared Bellman error at the projection of  $Q^{\tilde{\pi}}$  is close to zero, up to some estimation and approximation errors.
Lemma C.1. For any  $\delta > 0, \epsilon > 0$  and any  $T \in \mathbb{N}$ , under Assumption 2.2, with probability at least  $1 - \delta$ , it holds uniformly over all  $f \in \mathcal{F}$  and  $\tilde{\pi} \in \Pi^{soft}(T)$  that
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \mathbb {E} _ {k} \left[ \tilde {\mathcal {E}} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] \leq 2 \mathcal {L} _ {\tilde {\pi}} (f) + 4 0 b (b + 2) K H \epsilon + 1 2 b K \sum_ {h = 1} ^ {H} \nu_ {h} \\ + 1 4 4 (e - 2) b ^ {2} H \left[ d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta) \right], \\ \end{array}
$$
where  $\mathcal{L}_{\tilde{\pi}}(f)\coloneqq \sum_{h = 1}^{H}\hat{L}_{\tilde{\pi}}(f_h,f_{h + 1}) - \inf_{g\in \mathcal{F}}\sum_{h = 1}^{H}\hat{L}_{\tilde{\pi}}(g_h,f_{h + 1}).$
Lemma C.2. Under Assumption 2.1, for any  $T \in \mathbb{N}$ , with probability at least  $1 - \delta$ , it holds uniformly for any  $\tilde{\pi} \in \Pi^{soft}(T)$  that
$$
\mathcal {L} _ {\tilde {\pi}} \left(\operatorname {P r o j} _ {\mathcal {F}} \left(Q ^ {\tilde {\pi}}\right)\right) \leq 3 6 (e - 2) b ^ {2} H \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln \frac {H}{\delta}\right) + 6 b (3 b + 4) \epsilon K H + 1 5 b K \sum_ {h = 1} ^ {H} \xi_ {h},
$$
where  $\operatorname{Proj}_{\mathcal{F}}(Q^{\tilde{\pi}})$  is the projection of  $Q^{\tilde{\pi}}$  onto  $\mathcal{F}$ , formally defined in Definition 4.
With the two lemmas above, we are ready to prove Theorem 1. This proof is also laying a foundational step for our proofs of Theorem 2 and Theorem 3 that we shall present shortly. The proofs for the two lemmas above are presented immediately after the proof of Theorem 1.
Proof of Theorem 1. Using Lemma B.4, we have
$$
\operatorname {S u b O p t} _ {\pi} ^ {M} (\pi^ {t}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {E} _ {h} ^ {\pi^ {t}} \left(\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}\right) \left(s _ {h}, a _ {h}\right) \right] + \Delta_ {1} \underline {{Q}} _ {1} \left(s _ {1}, \pi_ {1} ^ {t}\right) + \operatorname {S u b O p t} _ {\pi} ^ {M _ {t}} (\pi^ {t})
$$
where we denote
$$
M _ {t} := M \left(Q ^ {t}, \pi^ {t}\right),
$$
$$
\Delta_ {1} \underline {{Q}} _ {1} (s _ {1}, \pi^ {t}) := \underline {{Q}} _ {1} (s _ {1}, \pi^ {t}) - V _ {1} ^ {\pi^ {t}} (s _ {1}).
$$
Bounding  $\sum_{t=1}^{T} \mathrm{SubOpt}_{\pi}^{M_t}(\pi^t)$ . Note that  $\sum_{t=1}^{T} \mathrm{SubOpt}_{\pi}^{M_t}(\pi^t)$  can be controlled by standard tools from online learning (Lemma B.6); thus it remains to control the first  $H + 1$  terms.
Bounding  $\Delta_1\underline{Q}_1(s_1,\pi_1^t)$ . Due to Lemma C.2, the event that  $\mathrm{Proj}_{\mathcal{F}}(Q^{\pi^t})\in \mathcal{F}(\beta ;\pi^t)$  holds occurs at probability at least  $1 - \delta$ . Furthermore, under this event, we have
$$
\begin{array}{l} \Delta_ {1} \underline {{Q}} _ {1} (s _ {1}, \pi_ {1} ^ {t}) = \underline {{Q}} _ {1} (s _ {1}, \pi^ {t}) - V _ {1} ^ {\pi^ {t}} (s _ {1}) \\ \leq \operatorname {P r o j} _ {\mathcal {F} _ {1}} \left(Q _ {1} ^ {\pi^ {t}}\right) - V _ {1} ^ {\pi^ {t}} \left(s _ {1}\right) \\ \leq \xi_ {1}, \\ \end{array}
$$
where the first inequality exploits Line 2 of Algorithm 2, and the last inequality uses Assumption 2.1.
Bounding  $\sum_{h=1}^{H} \mathbb{E}_{\pi}[\mathcal{E}_h^{\pi^t}(\underline{Q}_h^t, \underline{Q}_{h+1}^t)(s_h, a_h)]$ . It follows from Lemma B.5 that
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {E} _ {h} ^ {\pi^ {t}} \left(\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}\right) \left(s _ {h}, a _ {h}\right) \right] \leq \sqrt {H C (\pi ; \epsilon) \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(\underline {{Q}} _ {h} ^ {t} , \underline {{Q}} _ {h + 1} ^ {t}\right) \left(s _ {h} , a _ {h}\right) ^ {2} \right] + \nu_ {h} ^ {2} + 4 b \nu_ {h}\right)} \\ + H \epsilon + \bar {\nu}. \\ \end{array}
$$
The term  $\sum_{h=1}^{H} \mathbb{E}_{\mu}[\mathcal{E}_{h}^{\tilde{\pi}}(\underline{Q}_{h}^{t}, \underline{Q}_{h+1}^{t})(s_{h}, a_{h})^{2}]$  is bounded by Lemma C.1, with notice that  $\mathcal{L}_{\pi^{t}}(\underline{Q}^{t}) \leq \beta$  (due to the definition of  $\mathcal{F}(\beta; \pi^{t})$  in Algorithm 2).
Combining the three steps above via the union bound completes our proof.
We now prove the two support lemmas.
C.2 Proof of Lemma C.1
Proof of Lemma C.1. Let us consider any fixed  $f \in \mathcal{F}$  and any  $\pi \in \Pi^{all}$ . By Lemma B.1, we have
Combining with Lemma B.2, we have that with probability at least  $1 - \delta$ , for any  $\iota \in [0, \frac{1}{13b^2}]$ ,
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \mathbb {E} _ {k} \left[ \mathcal {E} _ {h} ^ {\pi} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] - \sum_ {k = 1} ^ {K} \Delta L _ {\pi} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) \\ \leq 3 6 (e - 2) b ^ {2} \iota \sum_ {k = 1} ^ {K} \mathbb {E} _ {k} \left[ \mathcal {E} _ {h} ^ {\pi} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] + (1 / \iota) \log (1 / \delta). \\ \end{array}
$$
By setting  $\iota = \frac{1}{72(e - 2)b^2}$ , the above inequality becomes
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {k} [ \mathcal {E} _ {h} ^ {\pi} (f _ {h}, f _ {h + 1}) (s _ {h}, a _ {h}) ^ {2} ] \leq 2 \sum_ {k = 1} ^ {K} \Delta L _ {\pi} (f _ {h}, f _ {h + 1}; z _ {h} ^ {k}) + 1 4 4 (e - 2) b ^ {2} \ln (1 / \delta).
$$
For any  $\epsilon > 0$ , and for any  $f \in \mathcal{F}$ ,  $\pi \in \Pi^{soft}(T)$ , by definition of  $\epsilon$ -covering, there exist  $f'$  and  $\pi'$  in the  $\epsilon$ -cover of  $\mathcal{F}$  and  $\Pi^{soft}(T)$ , i.e.,
$$
\left\| f _ {h} - f _ {h} ^ {\prime} \right\| _ {\infty} \leq \epsilon , \left\| \pi_ {h} - \pi_ {h} ^ {\prime} \right\| _ {1, \infty} \leq \epsilon .
$$
By simple calculations, we have
$$
\begin{array}{l} \left. \right.\left| \mathcal {E} _ {h} ^ {\pi} \left(f _ {h}, f _ {h + 1}\right)\left(s _ {h}, a _ {h}\right) ^ {2} - \mathcal {E} _ {h} ^ {\pi^ {\prime}} \left(f _ {h} ^ {\prime}, f _ {h + 1} ^ {\prime}\right)\left(s _ {h}, a _ {h}\right) ^ {2} \right| \leq 4 b (b + 2) \epsilon , \\ | \Delta L _ {\pi} (f _ {h}, f _ {h + 1}; z _ {h} ^ {k}) - \Delta L _ {\pi^ {\prime}} (f _ {h} ^ {\prime}, f _ {h + 1} ^ {\prime}; z _ {h} ^ {k}) | \leq 1 8 b (b + 2) \epsilon . \\ \end{array}
$$
Thus, by the union bound, we have with probability at least  $1 - \delta$ , it holds uniformly over all  $f \in \mathcal{F}, \pi \in \Pi^{soft}(T)$  that
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \mathbb {E} _ {k} \left[ \mathcal {E} _ {h} ^ {\pi} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] \leq 2 \sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \Delta L _ {\pi} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) + 4 0 b (b + 2) K H e \\ + 1 4 4 (e - 2) b ^ {2} \sum_ {h = 1} ^ {H} \ln (N (\epsilon ; \mathcal {F} _ {h}, \| \cdot \| _ {\infty}) N (\epsilon ; \Pi_ {h} ^ {s o f t} (T), \| \cdot \| _ {1, \infty}) / \delta). \\ \end{array}
$$
Finally, notice that
$$
\left| l _ {\pi} (\mathbb {T} _ {h} ^ {\pi} f _ {h + 1}, f _ {h + 1}; z _ {h}) - l _ {\pi} (\mathrm {P r o j} _ {\mathcal {F} _ {h}} (\mathbb {T} _ {h} ^ {\pi} f _ {h + 1}), f _ {h + 1}; z _ {h}) \right| \leq 6 b \nu_ {h}.
$$
$$
\sum_ {h = 1} ^ {H} \sum_ {k = 1} ^ {K} \Delta L _ {\pi} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) \leq \mathcal {L} _ {\pi} (f) + 6 b K \sum_ {h = 1} ^ {H} \nu_ {h}.
$$
We can then conclude our proof.
C.3 Proof of Lemma C.2
In order to prove Lemma C.2, we shall first prove the following lemma, which establishes the confidence radius of the empirical squared Bellman errors that we used to establish the version space in Algorithm 2.
Lemma C.3. Consider any  $\delta > 0, \epsilon > 0, T \in \mathbb{N}$ , let