SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
A _ {t} \geq \lambda \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \Delta f _ {1} (s _ {1}, \tilde {\pi}).
$$
Proof of Lemma E.2. It simply follows from that:
$$
\mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ (1 - 0. 5 \alpha) \ln \frac {\hat {p} (f _ {1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {1})} \right] = (1 - 0. 5 \alpha) D _ {\mathrm {K L}} [ \hat {p} (\cdot | \mathcal {D}, \tilde {\pi}) \| p _ {0} ] \geq 0.
$$
Bounding  $B_{h,t}$
Lemma E.3. For any  $f, \tilde{\pi}, 0 \leq \gamma \leq \frac{1}{72(e - 2)b^2}$ , and  $h \in [H]$ , we have
$$
\ln \mathbb {E} _ {(s _ {h + 1}, r _ {h}) \sim P _ {h} (\cdot | s _ {h}, a _ {h})} \exp \big (- 2 \gamma \Delta L _ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h}) \big) \leq - 2 \gamma (1 - 7 2 (e - 2) \gamma b ^ {2}) \mathcal {E} _ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}) (s _ {h}, a _ {h}) ^ {2}.
$$
Proof of Lemma E.3. For simplicity, we write  $\mathbb{E} = \mathbb{E}_{(s_{h + 1},r_h)}\sim P_h(\cdot |_{s_h,a_h})$  . We have
$$
\begin{array}{l} \ln \mathbb {E} \exp (- 2 \gamma \Delta L _ {\bar {\pi}} (f _ {h}, f _ {h + 1}; z _ {h})) \leq \mathbb {E} \exp (- 2 \gamma \Delta L _ {\bar {\pi}} (f _ {h}, f _ {h + 1}; z _ {h})) - 1 \\ \leq - 2 \gamma \mathbb {E} \Delta L _ {\bar {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h}\right) + (e - 2) 4 \gamma^ {2} \mathbb {E} \Delta L _ {\bar {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h}\right) ^ {2} \\ \leq - 2 \gamma (1 - (e - 2) 2 \gamma 3 6 b ^ {2}) \mathcal {E} _ {h} ^ {2} \left(f _ {h}, f _ {h + 1}, \tilde {\pi}\right) \left(s _ {h}, a _ {h}\right) \\ \end{array}
$$
where the first inequality uses  $\ln x\leq x - 1,\forall x\geq 0$  , the second inequality uses  $e^x\leq 1 + x + (e-$ $2)x^{2},\forall |x|\leq 1$  and  $|2\gamma \mathbb{E}\Delta L_{\tilde{\pi}}(f_h,f_{h + 1};z_h)|\leq 18\gamma b^2\leq 1$  , the third inequality uses Lemma B.1 and  $\gamma \leq \frac{1}{72(e - 2)b^2}$
Lemma E.4. Define the random variable
$$
\xi_ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h}) := - 2 \gamma \Delta L _ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h}) - \ln \mathbb {E} _ {(s _ {h + 1}, r _ {h}) \sim P _ {h} (\cdot | s _ {h}, a _ {h})} \exp (- 2 \gamma \Delta L _ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h})).
$$
For any  $\gamma \in [0, \frac{1}{144(e - 2)b^2}]$ ,  $t \in [0, \frac{1}{26\gamma b^2}]$ ,  $\epsilon > 0$ ,  $\delta > 0$ ,  $T \in \mathbb{N}$  with probability at least  $1 - \delta$ , it holds uniformly over all  $\tilde{\pi} \in \Pi^{soft}(T)$ ,  $f_h \in \mathcal{F}_h$ ,  $f_{h+1} \in \mathcal{F}_{h+1}$  that
$$
\sum_ {k = 1} ^ {K} \xi_ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h} ^ {k}) \leq D + c \sum_ {k = 1} ^ {K} e _ {k} ^ {2},
$$
$$
\left\{ \begin{array}{l l} D & := 1 2 0 \gamma b (b + 2) K \epsilon + \frac {2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)}{t}, \\ c & := 3 2 0 b ^ {2} \gamma^ {2} (e - 2) t, \\ e _ {k} & := \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right). \end{array} \right. \tag {13}
$$
Proof of Lemma E.4. For simplicity, denote
$$
\left\{ \begin{array}{l l} u _ {k} & := \ln \mathbb {E} _ {\left(s _ {h + 1}, r _ {h}\right) \sim P _ {h} \left(\cdot \mid s _ {h}, a _ {h}\right)} \exp \left(- 2 \gamma \Delta L _ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right)\right), \\ v _ {k} & := - 2 \gamma \Delta L _ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right), \\ w _ {k} & := v _ {k} - u _ {k}, \\ e _ {k} & := \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right). \end{array} \right. \tag {14}
$$
$$
u _ {k} \geq \mathbb {E} _ {(s _ {h + 1}, r _ {h}) \sim P _ {h} (\cdot | s _ {h}, a _ {h})} \ln \exp \left(- 2 \gamma \Delta L _ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h} ^ {k})\right) = - 2 \gamma e _ {k} ^ {2},
$$
where the first inequality uses Jensen's inequality for concave function  $\ln (\cdot)$  and the equality uses Lemma B.1. Now using Lemma E.3 with  $\gamma \leq \frac{1}{144(e - 2)b^2}$ , we have
$$
u _ {k} \leq - \gamma e _ {k} ^ {2}. \tag {15}
$$
We also have  $\mathbb{E}v_{k} = -2\gamma e_{k}^{2}$  by Lemma B.1. Thus, we have
$$
\left| u _ {k} \right| \leq 2 \gamma e _ {k} ^ {2}, \text {a n d} \mathbb {E} \left[ w _ {k} \right] = - 2 \gamma e _ {k} ^ {2} - u _ {k} \leq 0.
$$
$$
\begin{array}{l} \mathbb {E} w _ {k} ^ {2} = \mathbb {E} (v _ {k} - u _ {k}) ^ {2} \\ \leq 2 \mathbb {E} \left(v _ {k} ^ {2} + u _ {k} ^ {2}\right) \\ \leq 2 8 8 b ^ {2} \gamma^ {2} e _ {k} ^ {2} + 8 \gamma^ {2} e _ {k} ^ {4} \\ \leq 3 2 0 b ^ {2} \gamma^ {2} e _ {k} ^ {2} \\ \end{array}
$$
where the first inequality uses Cauchy-Schwartz inequality, the second inequality uses Lemma B.1 and that  $|u_k| \leq 2\gamma e_k^2$ , and the last inequality uses that  $|e_k| \leq 2b$ . Also note that  $|w_k| \leq |v_k| + |u_k| \leq 2\gamma (9b^2) + 2\gamma (4b^2) = 26\gamma b^2$ . Thus, by Lemma B.2, for any  $\delta > 0$ , for any  $t \in [0, \frac{1}{26\gamma b^2}]$ , with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} w _ {k} \leq \sum_ {k = 1} ^ {K} \mathbb {E} w _ {k} + (e - 2) t \cdot \mathbb {E} \sum_ {k = 1} ^ {K} w _ {k} ^ {2} + \frac {\ln (1 / \delta)}{t} \\ \leq 3 2 0 b ^ {2} \gamma^ {2} (e - 2) t \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + \frac {\ln (1 / \delta)}{t}. \\ \end{array}
$$
We apply the discretization argument and the union bound to obtain that: For any  $\delta > 0, \epsilon > 0$ ,  $T \in \mathbb{N}$  it holds uniformly over all  $\tilde{\pi} \in \Pi_h^{soft}(T)$ ,  $f_h \in \mathcal{F}_h$ ,  $f_{h+1} \in \mathcal{F}_{h+1}$  that
$$
\sum_ {k = 1} ^ {K} w _ {k} \leq 1 2 0 \gamma b (b + 2) K \epsilon + 3 2 0 b ^ {2} \gamma^ {2} (e - 2) t \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + \frac {2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)}{t}.
$$
Lemma E.5. For any  $\gamma \in [0, \frac{1}{144(e - 2)b^2}]$ ,  $\epsilon > 0$ ,  $\delta > 0$ , we have
$$
\begin{array}{l} B _ {h, t} \geq 0. 5 \gamma \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \bar {\pi})} \left[ \sum_ {k = 1} ^ {K} \mathcal {E} _ {h} ^ {\bar {\pi}} (f _ {h}, f _ {h + 1}) (s _ {h} ^ {k}, a _ {h} ^ {k}) ^ {2} \right] \\ \geq - 1 2 0 \gamma b (b + 2) K \epsilon - 6 4 0 (e - 2) b ^ {2} \gamma \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)\right) - 2 6 \gamma b ^ {2} K \delta . \\ \end{array}
$$
Proof of Lemma E.5. Define the random variables  $u_{k}, v_{k}, w_{k}, e_{k}$  as Equation (14). Recall  $D, c$  are defined in Equation (13) for any  $t \in [0, \frac{1}{26\gamma b^2}]$ . Define the event  $E$  such that the inequality
$$
\sum_ {k = 1} ^ {K} \xi_ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right) \leq \underbrace {3 2 0 b ^ {2} \gamma^ {2} (e - 2) t} _ {c} \sum_ {k = 1} ^ {K} e _ {k} ^ {2} + D, \tag {16}
$$
holds uniformly over all  $\tilde{\pi} \in \Pi^{soft}(T)$ ,  $f_h \in \mathcal{F}_h$ ,  $f_{h+1} \in \mathcal{F}_{h+1}$ . By Lemma E.4, we have
$$
\Pr (E) \geq 1 - \delta , \text {t h u s} \Pr (E ^ {c}) \leq \delta .
$$
$$
\mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \sum_ {k = 1} ^ {K} (- w _ {k} + c e _ {k} ^ {2}) + \ln \frac {\hat {p} (f _ {h} , f _ {h + 1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {h} , f _ {h + 1})} \right]
$$
$$
\begin{array}{l} \geq \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \inf  _ {p} \mathbb {E} _ {f \sim p} \left[ \sum_ {k = 1} ^ {K} (- w _ {k} + c e _ {k} ^ {2}) + \ln \frac {p \left(f _ {h} , f _ {h + 1}\right)}{p _ {0} \left(f _ {h} , f _ {h + 1}\right)} \right] \\ = - \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \ln \mathbb {E} _ {f _ {h}, f _ {h + 1} \sim p _ {0}} \exp \left(\sum_ {k = 1} ^ {K} \left(w _ {k} - c e _ {k} ^ {2}\right)\right) \\ = - \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \ln \mathbb {E} _ {f _ {h}, f _ {h + 1} \sim p _ {0}} \exp \left(\sum_ {k = 1} ^ {K} \left(w _ {k} - c e _ {k} ^ {2}\right)\right) \\ - \mathbb {E} _ {\mathcal {D}} 1 \{E ^ {c} \} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \ln \mathbb {E} _ {f _ {h}, f _ {h + 1} \sim p _ {0}} \exp \left(\sum_ {k = 1} ^ {K} (w _ {k} - c e _ {k} ^ {2})\right) \\ \geq - \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \ln \mathbb {E} _ {f _ {h}, f _ {h + 1} \sim p _ {0}} \exp \left(\sum_ {k = 1} ^ {K} \left(w _ {k} - c e _ {k} ^ {2}\right)\right) - 2 6 \gamma b ^ {2} K \delta \\ \geq - D - 2 6 \gamma b ^ {2} K \delta , \tag {17} \\ \end{array}
$$
where the first equality uses Lemma G.3, the second inequality uses that  $\operatorname{Pr}(E^c) \leq \delta$  and  $\sum_{k=1}^{K} (w_k - ce_k^2) \leq \sum_{k=1}^{K} w_k \leq 26\gamma b^2 K$  and the last inequality uses Equation (16). Thus, using the same notations as Lemma Lemma E.4, we have
$$
\begin{array}{l} B _ {h, t} = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ - \sum_ {k = 1} ^ {K} v _ {k} + \ln \frac {\hat {p} (f _ {h} , f _ {h + 1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {h} , f _ {h + 1})} \right] \\ = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \sum_ {k = 1} ^ {K} (- w _ {k} + c e _ {k} ^ {2}) + \ln \frac {\hat {p} (f _ {h} , f _ {h + 1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {h} , f _ {h + 1})} \right] \\ + \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \sum_ {k = 1} ^ {K} (- u _ {k} - c e _ {k} ^ {2}) \right] \\ \geq - D - 2 6 \gamma b ^ {2} K \delta + (\gamma - c) \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \sum_ {k = 1} ^ {K} e _ {k} ^ {2} \right] \\ \end{array}
$$
where the inequality uses Equation (17) and Equation (15). Finally, setting
$$
t = \frac {1}{6 4 0 b ^ {2} (e - 2) \gamma} <   \frac {1}{1 3 b ^ {2} \gamma}
$$
completes our proof.
From squared Bellman errors to in-expectation squared Bellman errors and fixing a nonrigorous argument of Dann et al. [2021]. Lemma E.5 only bounds  $B_h$  with the squared Bellman errors  $\sum_{k=1}^{K} \mathcal{E}_h^{\tilde{\pi}}(f_h, f_{h+1})(s_h^k, a_h^k)^2$  while the in-expectation squared Bellman errors  $\sum_{k=1}^{K} \mathbb{E}_{\mu^k}[\mathcal{E}_h^{\tilde{\pi}}(f_h, f_{h+1})(s_h, a_h)^2]$  are what we need for showing Proposition 2. There is no an immediate path to go from the squared Bellman error to the in-expectation squared Bellman errors as the order of  $\mathbb{E}_{\mathcal{D}}$  and  $\mathbb{E}_{\tilde{\pi} \sim P_t(\cdot|\mathcal{D})} \mathbb{E}_{f \sim \hat{p}(\cdot|\mathcal{D}, \tilde{\pi})}$  are not exchangeable, i.e.,
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \sum_ {k = 1} ^ {K} \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {2} \right] \\ \neq \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \sum_ {k = 1} ^ {K} \mathbb {E} _ {\mu^ {k}} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] \right]. \tag {18} \\ \end{array}
$$
A similar caveat arises in the online setting in Dann et al. [2021] as well. In particular, a non-rigorous argument of [Dann et al., 2021, Lemma 8] is that they conclude (an online analogue of) the LHS of Equation (18) is equal to (an online analogue of) its RHS.
To fix this issue without ultimately incurring a sub-optimality rate that is slower than  $1 / \sqrt{K}$ , we need to change the squared Bellman error into the in-expectation squared Bellman error, up to some estimation error that scales faster than  $K^{\alpha}$  for any  $\alpha > 0$ . Note that, a standard Azuma-Hoeffding inequality (and the union bound) give an estimation error that scales with  $K^{1/2}$ . To achieve the logarithmic dependence on  $K$ , the following lemma exploits the non-negativity of the squared Bellman error and uses the localization argument of Bartlett et al. [2005] to obtain an estimation error rate that scales polylogarithmic with  $K$ .
Lemma E.6 (Improved online-to-batch argument for non-negative R.V.s [Nguyen-Tang et al., 2023]). Let  $\{X_k\}$  be any real-valued stochastic process adapted to the filtration  $\{\mathcal{F}_k\}$ , i.e.  $X_k$  is  $\mathcal{F}_k$ -measurable. Suppose that for any  $k$ ,  $X_k \in [0, H]$  almost surely for some  $H > 0$ . For any  $K > 0$  with probability at least  $1 - \delta$ , we have:
$$
\sum_ {k = 1} ^ {K} \mathbb {E} \left[ X _ {k} \mid \mathcal {F} _ {k - 1} \right] \leq 2 \sum_ {k = 1} ^ {K} X _ {k} + \frac {1 6}{3} H \log \left(\log_ {2} (K H) / \delta\right) + 2.
$$
With Lemma E.6, we now actually make a connection from the squared Bellman error to the in-expectation squared Bellman error in the following lemma, which incorporates the uniform convergence argument into the posterior sampling in the same spirit with our earlier argument in Section E.3.
Lemma E.7. For any  $\delta, \epsilon > 0$  and  $T \in \mathbb{N}$ , any  $t \in [T]$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \sum_ {k = 1} ^ {K} \mathcal {E} _ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}) (s _ {h} ^ {k}, a _ {h} ^ {k}) ^ {2} \\ \geq 0. 5 \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\mu^ {k}} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] \\ - b (b + 2) K \epsilon - \frac {3 2}{3} b ^ {2} \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln \frac {\ln 4 K b ^ {2}}{\delta}\right) - 1 - 2 K b ^ {2} \delta . \\ \end{array}
$$
Proof of Lemma E.7. For simplicity, we denote  $X(f,\mathcal{D})\coloneqq \sum_{k = 1}^{K}\mathcal{E}_{h}^{\tilde{\pi}}(f_{h},f_{h + 1})(s_{h}^{k},a_{h}^{k})^{2}$  and  $X(f)\coloneqq \mathbb{E}_{\mathcal{D}}[X(f,\mathcal{D})]$ , and  $\Delta \coloneqq 8bK\epsilon +\frac{64}{3} b^{2}\left(2d_{\mathcal{F}}(\epsilon) + d_{\Pi}(\epsilon ,T) + \frac{\ln\ln 4Kb^{2}}{\delta}\right) + 2$ . We define the event:
$$
E = \left\{\mathcal {D}: X (f) \leq 2 X (f, \mathcal {D}) + \Delta , \forall f _ {h} \in \mathcal {F} _ {h}, f _ {h + 1} \in \mathcal {F} _ {h + 1}, \tilde {\pi} \in \Pi_ {h} ^ {s o f t} (T) \right\}.
$$
Due to the non-negativity of  $\mathcal{E}_h^{\tilde{\pi}}(f_h,f_{h + 1})(s,a)^2$ , Lemma E.6 and the union bound, we have
$$
\Pr (E) \geq 1 - \delta \text {a n d} \Pr (E ^ {c}) \leq \delta .
$$
$$
\begin{array}{l} 2 \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} X (f, \mathcal {D}) \\ = 2 \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \bar {\pi})} X (f, \mathcal {D}) 1 \{E \} + 2 \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \bar {\pi})} X (f, \mathcal {D}) 1 \{E ^ {c} \} \\ \geq 2 \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} X (f, \mathcal {D}) 1 \{E \} \\ \geq \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} (X (f) - \Delta) \\ = \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} X (f) - \Delta \Pr (E) \\ \geq \mathbb {E} _ {\mathcal {D}} 1 \{E \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} X (f) - \Delta \\ = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} X (f) - \Delta - \mathbb {E} _ {\mathcal {D}} 1 \{E ^ {c} \} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} X (f) \\ \geq \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} X (f) - \Delta - \mathbb {E} _ {\mathcal {D}} 1 \{E ^ {c} \} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} 4 K b ^ {2} \\ = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \bar {\pi})} X (f) - \Delta - 4 K b ^ {2} \Pr \left(E ^ {c}\right) \\ \geq \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \bar {\pi})} X (f) - \Delta - 4 K b ^ {2} \delta \\ \end{array}
$$
where the fourth inequality uses  $|X(f)| \geq 4Kb^2$  and the last inequality uses  $\operatorname{Pr}(E^c) \leq \delta$ .
Bounding  $C_{h,t}$
Lemma E.8. For any  $\epsilon > 0$  and  $T \in \mathbb{N}$ , we have
$$
C _ {h, t} \geq - \max  _ {\tilde {\pi} \in \Pi_ {h} ^ {s o f t} (T)} \kappa_ {h} (\alpha , \epsilon , \tilde {\pi}) - \gamma \alpha 6 b K \epsilon .
$$
where  $\kappa_h(\alpha, \epsilon, \tilde{\pi})$  is defined in Equation (11).
Proof of Lemma E.8. We have
$$
\begin{array}{l} C _ {h, t} = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\bar {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \bar {\pi})} \left[ \alpha \ln \mathbb {E} _ {f _ {h} ^ {\prime} \sim p _ {0}} \exp \left(- \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} (f _ {h} ^ {\prime}, f _ {h + 1}; z _ {h} ^ {k})\right) + (1 - \alpha) \ln \frac {\hat {p} (f _ {h + 1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {h + 1})} \right] \\ = (1 - \alpha) \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \frac {\alpha}{1 - \alpha} \ln \mathbb {E} _ {f _ {h} ^ {\prime} \sim p _ {0}} \exp \left(- \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} (f _ {h} ^ {\prime}, f _ {h + 1}; z _ {h} ^ {k})\right) + \ln \frac {\hat {p} (f _ {h + 1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {h + 1})} \right] \\ \geq - (1 - \alpha) \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \ln \mathbb {E} _ {f _ {h + 1} \sim p _ {0}} \left(\mathbb {E} _ {f _ {h} ^ {\prime} \sim p _ {0}} \exp \left(- \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} \left(f _ {h} ^ {\prime}, f _ {h + 1}; z _ {h} ^ {k}\right)\right)\right) ^ {\frac {- \alpha}{1 - \alpha}} \\ \geq -\max_{\tilde{\pi}\in \Pi^{soft}(T)}\kappa_{h}(\alpha ,\epsilon ,\tilde{\pi}) - \gamma \alpha 6bK\epsilon . \\ \end{array}
$$
where the first inequality uses Lemma G.3 and the last inequality uses the following inequalities: For any  $f_h \in \mathcal{F}_h(\epsilon, f_{h+1}, \tilde{\pi})$ , we have
$$
\left| \Delta L _ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h}\right) \right| \leq 6 b \left| \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \right| \leq 6 b \epsilon ; \text {t h u s}
$$
$$
\mathbb {E} _ {f _ {h} ^ {\prime} \sim p _ {0}} \exp \left(- \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} (f _ {h} ^ {\prime}, f _ {h + 1}; z _ {h})\right) \geq p _ {0, h} (\mathcal {F} _ {h} ^ {\tilde {\pi}} (\epsilon , f _ {h + 1})) \cdot \exp (- \gamma 6 b K \epsilon).
$$
We are now ready to state the complete form of the lower bound of  $Z$ .
Proposition 3. For any  $\gamma \in [0, \frac{1}{144(e - 2)b^2}]$ ,  $\epsilon > 0$ ,  $\delta > 0$ , and any  $t \in [T]$ , we have,
$$
\begin{array}{l} Z \geq \lambda \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \Delta f _ {1} (s _ {1}, \tilde {\pi}) \\ + 0. 1 2 5 \alpha \gamma \sum_ {h = 1} ^ {H} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\mu^ {k}} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] \\ - 0. 5 \alpha H (1 2 0 \gamma b (b + 2) K \epsilon + 6 4 0 (e - 2) \gamma b ^ {2} (2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta))) - 1 3 \alpha \gamma b ^ {2} K H \delta \\ - 0. 2 5 \alpha \gamma H \left(b (b + 2) K \epsilon + \frac {3 2}{3} b ^ {2} \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln \frac {\ln 4 K b ^ {2}}{\delta}\right) + 1 + 2 K b ^ {2} \delta\right) \\ - \sum_ {h = 1} ^ {H} \max  _ {\tilde {\pi} \in \Pi^ {s o f t} (T)} \kappa_ {h} (\alpha , \epsilon , \tilde {\pi}) - \gamma \alpha 6 b K H \epsilon . \\ \end{array}
$$
Proof of Proposition 3. Using Lemma E.1, it suffices to bound terms  $A$ ,  $B_h$ , and  $C_h$  defined in Lemma E.1. For this purpose, we use
- Lemma E.2: To bound  $A_{t}$ ,
- Lemma E.5 and Lemma E.7: To bound  $B_{h,t}$ ,
- Lemma E.8: To bound  $C_{h,t}$ .
The result is then simply a direct combination of the above lemmas.
E.3.2 Upper-bounding log-partition function
In this appendix, we upper bound the log-partition function  $Z$ . While we follow the proof flow in Dann et al. [2021], due to the statistical dependence in the actor-critic framework of our algorithm, we require different technical arguments to establish this result. In particular, Lemma E.9 and Lemma E.10 are our new technical lemmas.
Proposition 4. For any  $\epsilon, \delta > 0$ ,  $\gamma > 0$ , and  $t \in [T]$ , we have
$$
\begin{array}{l} Z _ {t} \leq \lambda \epsilon - \inf  _ {\tilde {\pi} \in \Pi^ {s o f t} (T)} \sum_ {h = 1} ^ {H} \ln p _ {0} \left(\mathcal {F} _ {h} (\epsilon ; \tilde {\pi})\right) + 4 \gamma \left(\alpha + \frac {3 (e - 2)}{\alpha}\right) H K \epsilon^ {2} \\ + 6 0 \alpha \gamma b (b + 2) K H \epsilon + \alpha \gamma b ^ {2} H \left(1 3 + 3 6 (e - 2)\right) \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (1 / \delta)\right) + 1 8 \alpha \gamma K H b ^ {2} \delta . \\ \end{array}
$$
where recall that  $Z_{t}$  is defined in Equation (12).
Proof of Proposition 4. We have