SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\beta_ {\epsilon} := 3 6 (e - 2) b ^ {2} \left(2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) + \ln (H / \delta)\right) + 6 b (3 b + 4) \epsilon K.
$$
With probability at least  $1 - \delta$ , it holds uniformly over any  $\pi \in \Pi^{soft}(T)$ ,  $f \in \mathcal{F}$ , and  $h \in [H]$  that
$$
\sum_ {k = 1} ^ {K} \left(\mathbb {T} _ {h} ^ {\pi} f _ {h + 1} \left(x _ {h} ^ {k}\right) - r _ {h} ^ {k} - f _ {h + 1} \left(s _ {h + 1} ^ {k}, \pi_ {h + 1}\right)\right) ^ {2} \leq \inf  _ {g _ {h} \in \mathcal {F} _ {h}} \sum_ {k = 1} ^ {K} \left(g _ {h} \left(x _ {h} ^ {k}\right) - r _ {h} ^ {k} - f _ {h + 1} \left(s _ {h + 1} ^ {k}, \pi_ {h + 1}\right)\right) ^ {2} + \beta_ {\epsilon}.
$$
Proof of Lemma C.3. Let us fix any  $h \in [H]$ . For any  $(f, g, \pi) \in \mathcal{F} \times \mathcal{F} \times \Pi^{\text{soft}}(T)$  and any  $k \in [K]$ , define the following random variable
$$
Z _ {k, h} (f, g, \pi) := \left(g _ {h} \left(x _ {h} ^ {k}\right) - r _ {h} ^ {k} - f _ {h + 1} \left(s _ {h + 1} ^ {k}, \pi_ {h + 1}\right)\right) ^ {2} - \left(\mathbb {T} _ {h} ^ {\pi} f _ {h + 1} \left(x _ {h} ^ {k}\right) - r _ {h} ^ {k} - f _ {h + 1} \left(s _ {h + 1} ^ {k}, \pi_ {h + 1}\right)\right) ^ {2}.
$$
$$
\mathbb {E} _ {k, h} [ \cdot ] := \mathbb {E} \left[ \cdot \left| \left\{z _ {h} ^ {i} \right\} _ {h \in [ H ]} ^ {i \in [ k - 1 ]}, s _ {1} ^ {k}, a _ {1} ^ {k}, r _ {1} ^ {k}, \dots , s _ {h - 1} ^ {k}, a _ {h - 1} ^ {k}, r _ {h - 1} ^ {k}, s _ {h} ^ {k}, a _ {h} ^ {k} \right]. \right.
$$
By Lemma B.1, we have
$$
\mathbb {E} _ {k, h} \left[ Z _ {k, h} (f, g, \pi) \right] = \mathcal {E} _ {h} ^ {\pi} \left(g _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {2},
$$
$$
\mathbb {E} _ {k, h} \left[ Z _ {k, h} ^ {2} (f, g, \pi) \right] \leq 3 6 b ^ {2} \mathcal {E} _ {h} ^ {\pi} \left(g _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {2}.
$$
Thus, combing with Lemma B.2, for any  $(f,g,\pi)\in \mathcal{F}\times \mathcal{F}\times \Pi^{soft}(T)$ , with probability at least  $1 - \delta$ , for any  $\iota \in [0,\frac{1}{13b^2}]$
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {k, h} \left[ Z _ {k, h} (f, g, \pi) \right] - \sum_ {k = 1} ^ {K} Z _ {k, h} (f, g, \pi) \leq 3 6 (e - 2) b ^ {2} \iota \sum_ {k = 1} ^ {K} \mathcal {E} _ {h} ^ {\pi} (g _ {h}, f _ {h + 1}) (s _ {h} ^ {k}, a _ {h} ^ {k}) ^ {2} + \frac {\ln (1 / \delta)}{\iota}.
$$
By setting  $\iota = 1 / (36(e - 2)b^{2})$ , the above inequality becomes
$$
- \sum_ {k = 1} ^ {K} Z _ {k, h} (f, g, \pi) \leq 3 6 (e - 2) b ^ {2} \ln (1 / \delta). \tag {8}
$$
For any  $\epsilon > 0$ , let  $\mathcal{F}^{\epsilon}$  and  $\Pi^{\epsilon}$  be  $\epsilon$ -covers of  $\mathcal{F}$  and  $\Pi^{soft}(T)$ , respectively, with respect to  $\|\cdot\|_{\infty}$  and  $\|\cdot\|_{\infty,1}$ , respectively, where  $\|u - v\|_{\infty} := \sup_{(s,a)} |u(s,a) - v(s,a)|$  and  $\|\pi - \pi'\|_{\infty,1} := \sup_{s} \sum_{a \in \mathcal{A}} |\pi(a|s) - \pi'(a|s)|$ . Using the union bound, it follows from Equation (8) that with probability at least  $1 - \delta$ , it holds uniformly over any  $h \in [H]$  and any  $(f,g,\pi) \in \mathcal{F}^{\epsilon} \times \mathcal{F}^{\epsilon} \times \Pi^{\epsilon}$  that
$$
- \sum_ {k = 1} ^ {K} Z _ {k, h} (f, g, \pi) \leq 1 8 (e - 2) b ^ {2} [ \ln (H / \delta) + 2 d _ {\mathcal {F}} (\epsilon) + d _ {\Pi} (\epsilon , T) ].
$$
For any  $(f,g,\pi)\in \mathcal{F}\times \mathcal{F}\times \Pi^{soft}(T)$ , there exist  $(f_{\epsilon},g_{\epsilon},\pi_{\epsilon})\in \mathcal{F}^{\epsilon}\times \mathcal{F}^{\epsilon}\times \Pi^{\epsilon}$  such that
$$
\left\| f _ {h} - \left(f _ {\epsilon}\right) _ {h} \right\| _ {\infty} \leq \epsilon , \left\| g _ {h} - \left(g _ {\epsilon}\right) _ {h} \right\| _ {\infty} \leq \epsilon , \left\| \pi_ {h} - \left(\pi_ {\epsilon}\right) _ {h} \right\| _ {\infty , 1} \leq \epsilon , \forall h \in [ H ].
$$
It is easy to compute the discretization error that
$$
Z _ {k, h} (f, g, \pi) - Z _ {k, h} \left(f _ {\epsilon}, g _ {\epsilon}, \pi_ {\epsilon}\right) \leq 1 8 b (b + 1) \epsilon .
$$
Using the discretization argument and the union bound complete our proof.
We are now ready to prove Lemma C.2.
Proof of Lemma C.2. Consider the event that the inequality in Lemma C.3 holds. Under this event, for any  $\tilde{\pi} \in \Pi^{soft}(T)$ , we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} l _ {\tilde {\pi}} \left(\operatorname {P r o j} _ {\mathcal {F} _ {h}} \left(Q _ {h} ^ {\tilde {\pi}}\right), \operatorname {P r o j} _ {\mathcal {F} _ {h + 1}} \left(Q _ {h + 1} ^ {\tilde {\pi}}\right); z _ {h} ^ {k}\right) \leq \sum_ {k = 1} ^ {K} l _ {\tilde {\pi}} \left(Q _ {h} ^ {\pi^ {t}}, Q _ {h + 1} ^ {\tilde {\pi}}; z _ {h} ^ {k}\right) + 6 b K \xi_ {h} \\ = \sum_ {k = 1} ^ {K} l _ {\tilde {\pi}} \left(\mathbb {T} _ {h} ^ {\tilde {\pi}} Q _ {h + 1} ^ {\tilde {\pi}}, Q _ {h + 1} ^ {\tilde {\pi}}; z _ {h} ^ {k}\right) + 6 b K \xi_ {h} \\ \leq \sum_ {k = 1} ^ {K} l _ {\tilde {\pi}} \left(\mathbb {T} _ {h} ^ {\tilde {\pi}} \operatorname {P r o j} _ {\mathcal {F} _ {h + 1}} \left(Q _ {h + 1} ^ {\tilde {\pi}}\right), \operatorname {P r o j} _ {\mathcal {F} _ {h + 1}} \left(Q _ {h + 1} ^ {\tilde {\pi}}\right); z _ {h} ^ {k}\right) + 1 2 b K \xi_ {h} \\ \leq \sum_ {k = 1} ^ {K} l _ {\tilde {\pi}} \left(g _ {h}, \operatorname {P r o j} _ {\mathcal {F} _ {h + 1}} \left(Q _ {h + 1} ^ {\tilde {\pi}}\right); z _ {h} ^ {k}\right) + \beta_ {\epsilon} + 1 2 b K \xi_ {h} \quad (\text {f o r a n y} g _ {h} \in \mathcal {F} _ {h}) \\ \leq \sum_ {k = 1} ^ {K} l _ {\tilde {\pi}} \left(g _ {h}, Q _ {h + 1} ^ {\tilde {\pi}}; z _ {h} ^ {k}\right) + \beta_ {\epsilon} + 1 5 b K \xi_ {h}, \\ \end{array}
$$
where we use Assumption 2.1 for the first, second, and last inequalities, the third inequality uses Lemma C.3, and the equality uses  $Q_{h + 1}^{\tilde{\pi}} = \mathbb{T}_h^{\tilde{\pi}}Q_{h + 1}^{\tilde{\pi}}$ . Rearranging the last inequality completes our proof.
Appendix D Proof of Theorem 2
In this appendix, we present our complete argument to establish Theorem 2. In order to prove Theorem 2, the key is to establish a connection from the squared Bellman error under the data distribution  $\mu$  to the regularized objective in Algorithm 3. This key idea should become clear in the following proof.
Proof of Theorem 2. Similar to the proof of Theorem 1, our starting point is using Lemma B.4:
$$
\mathrm {S u b O p t} _ {\pi} ^ {M} (\pi^ {t}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} [ \mathcal {E} _ {h} ^ {\pi^ {t}} (\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}) (s _ {h}, a _ {h}) ] + \Delta_ {1} \underline {{Q}} _ {1} (s _ {1}, \pi_ {1} ^ {t}) + \mathrm {S u b O p t} _ {\pi} ^ {M _ {t}} (\pi^ {t})
$$
and we bound  $\sum_{t=1}^{T} \operatorname{SubOpt}_{\pi}^{M_t}(\pi^t)$  using Lemma B.6. We now bound the remaining terms.
For any  $\gamma > 0$ , we have
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} [ \mathcal {E} _ {h} ^ {\pi^ {t}} (\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}) (s _ {h}, a _ {h}) ] \\ \leq \frac {K}{2 \lambda} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\pi^ {t}} \left(\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] + \nu_ {h} ^ {2} + 4 b \nu_ {h}\right) + \frac {\lambda H C (\pi , \epsilon)}{2 K} + H \epsilon + \bar {\nu} \\ \leq \frac {\mathcal {L} _ {\pi^ {t}} (\underline {{Q}}) + 0 . 5 \iota_ {1} + 0 . 5 K \bar {\nu} ^ {2} + 2 b K \bar {\nu}}{\lambda} + \frac {\lambda H \mathcal {C} (\pi , \epsilon)}{2 K} + H \epsilon + \bar {\nu}, \\ \end{array}
$$
where the first inequality uses Lemma B.5 and the second inequality uses Lemma C.1, and here  $\iota_{1} := 40b(b + 2)KH\epsilon + 12bK\sum_{h = 1}^{H}\nu_{h} + 144(e - 2)b^{2}H[d_{\mathcal{F}}(\epsilon) + d_{\Pi}(\epsilon, T) + \ln(1 / \delta)]$ . Thus, we have
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {E} _ {h} ^ {\pi^ {t}} \left(\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}\right) \left(s _ {h}, a _ {h}\right) \right] + \Delta_ {1} \underline {{Q}} _ {1} \left(s _ {1}, \pi_ {1} ^ {t}\right) \\ \leq \frac {\mathcal {L} _ {\pi^ {t}} (\underline {{Q}}) + \lambda \Delta_ {1} \underline {{Q}} _ {1} \left(s _ {1} , \pi_ {1} ^ {t}\right) + 0 . 5 \iota_ {1} + 0 . 5 K \sum_ {h = 1} ^ {H} \nu_ {h} ^ {2} + 2 b K \sum_ {h = 1} ^ {H} \nu_ {h}}{\lambda} + \frac {\lambda H C (\pi , \epsilon)}{2 K} \\ + H \epsilon + \sum_ {h = 1} ^ {H} \nu_ {h} \\ \leq \frac {\mathcal {L} _ {\pi^ {t}} \left(\operatorname {P r o j} _ {\mathcal {F}} \left(Q ^ {\pi^ {t}}\right)\right) + \lambda \Delta_ {1} \operatorname {P r o j} _ {\mathcal {F} _ {1}} \left(Q _ {1} ^ {\pi^ {t}}\right) \left(s _ {1} , \pi_ {1} ^ {t}\right) + 0 . 5 \iota_ {1} + \sum_ {h = 1} ^ {H} \nu_ {h} ^ {2} + 2 b K \sum_ {h = 1} ^ {H} \nu_ {h}}{\lambda} \\ + \frac {\lambda H C (\pi , \epsilon)}{2 K} + H \epsilon + \sum_ {h = 1} ^ {H} \nu_ {h} \\ \leq \frac {\iota_ {2} + \lambda \xi_ {1} + 0 . 5 \iota_ {1} + \sum_ {h = 1} ^ {H} \nu_ {h} ^ {2} + 2 b K \sum_ {h = 1} ^ {H} \nu_ {h}}{\lambda} + \frac {\lambda H \mathcal {C} (\pi , \epsilon)}{2 K} + H \epsilon + \sum_ {h = 1} ^ {H} \nu_ {h}, \\ \end{array}
$$
where the second inequality uses the fact that  $\underline{Q}_h^t$  is a minimizer over  $\mathcal{F} \ni \mathrm{Proj}_{\mathcal{F}}(Q^{\pi^t})$  of  $\mathcal{L}_{\pi^t}(f) + \lambda f_1(s_1, \pi_1^t)$  (which has the same minimizer as  $\mathcal{L}_{\pi^t}(f) + \lambda \Delta_1 f_1(s_1, \pi_1^t)$ ), and the last inequality uses Lemma C.2, and here we define  $\iota_2 := 36(e - 2)b^2 H \left(2d_{\mathcal{F}}(\epsilon) + d_{\Pi}(\epsilon, T) + \ln \frac{H}{\delta}\right) + 6b(3b + 4)\epsilon KH + 15bK \sum_{h=1}^{H} \xi_h$ .
Appendix E Proof of Theorem 3
In this appendix, we give our complete proof for Theorem 3. In order to develop our argument for proving Theorem 3, we shall start with a generalized form of posterior sampling in Section E.1 and develop our key support result in Proposition 2. We then use Proposition 2 and the similar machinery developed in Section D to complete our argument for proving Theorem 3.
E.1 Generalized form of posterior and Proposition 2
We start with recalling the posterior distribution defined in Line 1 of Algorithm 4 as
$$
\hat {p} (f | \mathcal {D}, \pi) \propto \exp \left(- \lambda f _ {1} \left(s _ {1}, \pi_ {1}\right)\right) p _ {0} (f) \prod_ {h \in [ H ]} \frac {\exp \left(- \gamma \hat {L} _ {\pi} \left(f _ {h} , f _ {h + 1}\right)\right)}{\mathbb {E} _ {f _ {h} ^ {\prime} \sim p _ {0 , h}} \exp \left(- \gamma \hat {L} _ {\pi} \left(f _ {h} ^ {\prime} , f _ {h + 1}\right)\right)}. \tag {9}
$$
Similar to the proof strategy in Dann et al. [2021], we now consider a slightly more general form of the posterior distribution with an extra parameter  $\alpha \in [0,1]$  and in an equivalent but more useful form. In concrete, consider any  $\alpha \in [0,1]$  and define the potential functions:
$$
\begin{array}{l} \widehat {\Phi} _ {h} (f, \pi ; \mathcal {D}) := - \ln p _ {0} (f _ {h}) + \alpha \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} (f _ {h}, f _ {h + 1}; z _ {h} ^ {k}) \\ + \alpha \ln \mathbb {E} _ {\tilde {f} _ {h} \sim p _ {0}} \exp \left(- \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}; z _ {h} ^ {k}\right)\right), \\ \end{array}
$$
$$
\widehat {\Phi} (f, \pi ; \mathcal {D}) := \sum_ {h = 1} ^ {H} \widehat {\Phi} _ {h} (f, \pi ; \mathcal {D}),
$$
$$
\Delta_ {1} f _ {1} \left(s _ {1}, \pi\right) := f _ {1} \left(s _ {1}, \pi\right) - V _ {1} ^ {\pi} \left(s _ {1}\right).
$$
where recall that  $\Delta L_{\tilde{\pi}}(f_h, f_{h+1}; z_h^k)$  is defined in Table 3. Define the generalized posterior distribution
$$
\hat {p} (f | \mathcal {D}, \pi) \propto \exp \left(- \widehat {\Phi} (f, \pi ; \mathcal {D}) - \lambda \Delta f _ {1} (s _ {1}, \pi)\right), \tag {10}
$$
where it is equivalent to the posterior defined in Equation (9) when  $\alpha = 1$ . We shall use Equation (10) for the posterior for the rest of this section. We shall also define the complexity measure of this generalized posterior - a counterpart to that of the canonical posterior form in Definition 2.
Definition 7. Define
$$
\kappa_ {h} (\alpha , \epsilon , \tilde {\pi}) := (1 - \alpha) \ln \mathbb {E} _ {f _ {h + 1} \sim p _ {0}} \left[ p _ {0, h} \left(\mathcal {F} _ {h} ^ {\tilde {\pi}} (\epsilon ; f _ {h + 1})\right) ^ {- \alpha / (1 - \alpha)} \right],
$$
where recall that  $\mathcal{F}_h^{\tilde{\pi}}(\epsilon; f_{h+1}) = \{f' \in \mathcal{F}_h : \sup_{s,a} |\mathcal{E}_h^{\tilde{\pi}}(f', f_{h+1})(s,a)| \leq \epsilon\}$  which is defined in Definition 2. Define the complexity measure
$$
d _ {0} (\epsilon , \alpha) := \sup  _ {T \in \mathbb {N}, \tilde {\pi} \in \Pi^ {\text {s o f t}} (T)} \sum_ {h = 1} ^ {H} \kappa_ {h} (\alpha , \epsilon , \tilde {\pi}). \tag {11}
$$
$$
\lim  _ {\alpha \to 1 ^ {-}} d _ {0} (\epsilon , \alpha) = d _ {0} (\epsilon).
$$
We now state our key milestone result - Proposition 2 to support the argument for proving Theorem 3. The proof of Proposition 2 is deferred to Section E.3.
Notation  $\mathbb{E}_{\tilde{\pi}\sim P_t(\cdot |\mathcal{D})}$ . Note that in Algorithm 4, each policy  $\pi^t$  for  $t\in [T]$  is a random variable that depends on both the offline data  $\mathcal{D}$  and the randomization of sampling from the posteriors. That is, when conditioned on the offline data  $\mathcal{D}$ , each  $\pi_t$  is still a random variable. We denote  $P_{t}(\cdot |\mathcal{D})$  as the posterior distribution of  $\pi^t$  conditioned on  $\mathcal{D}$ . Note that for any  $\tilde{\pi}\sim P_t(\cdot |\mathcal{D})$  and any  $t\in [T]$ , we have  $\tilde{\pi}\in \Pi^{soft}(T)$ .
Proposition 2. For any  $\gamma \in [0, \frac{1}{144(e - 2)b^2}]$ ,  $\epsilon > 0$ ,  $\delta > 0$ ,  $\alpha \in (0,1]$ ,  $T \in \mathbb{N}$ , and any  $t \in [T]$  and  $\lambda > 0$ , we have,
$$
\mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ 0. 1 2 5 \alpha \gamma K \sum_ {h = 1} ^ {H} \mathbb {E} _ {\mu} [ \mathcal {E} _ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}) (s _ {h}, a _ {h}) ^ {2} ] + \lambda \Delta f _ {1} (s _ {1}, \tilde {\pi}) \right]
$$
$$
\begin{array}{l} \lesssim \lambda \epsilon + \alpha \gamma H b ^ {2} \cdot \max  \{d _ {\mathcal {F}} (\epsilon), d _ {\Pi} (\epsilon , T), \ln \frac {\ln K b ^ {2}}{\delta} \} + \alpha \gamma b ^ {2} K H \cdot \max  \{\epsilon , \delta \} + \gamma H K \frac {\epsilon^ {2}}{\alpha} \\ + \sum_ {h = 1} ^ {H} \sup  _ {\tilde {\pi} _ {h} \in \Pi_ {h} ^ {s o f t} (T)} \kappa_ {h} (\alpha , \epsilon , \tilde {\pi} _ {h}) + \sup  _ {\tilde {\pi} \in \Pi^ {s o f t} (T)} \sum_ {h = 1} ^ {H} \ln \frac {1}{p _ {0} \left(\mathcal {F} _ {h} \left(\epsilon ; Q _ {h} ^ {\tilde {\pi} _ {h}}\right)\right)}. \\ \end{array}
$$
We now have all main components needed to construct our argument for proving Theorem 3.
E.2 Proof of Theorem 3
Proof of Theorem 3. We start with the error decomposition argument.
Step 1: Error decomposition. Similar to the first step of the proof of Theorem 2, using Lemma B.4, we have
$$
\operatorname {S u b O p t} _ {\pi} ^ {M} (\pi^ {t}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} [ \mathcal {E} _ {h} ^ {\pi^ {t}} (\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}) (s _ {h}, a _ {h}) ] + \Delta_ {1} \underline {{Q}} _ {1} (s _ {1}, \pi_ {1} ^ {t}) + \operatorname {S u b O p t} _ {\pi} ^ {M _ {t}} (\pi^ {t})
$$
where we denote  $M_{t} \coloneqq M(\underline{Q}^{t},\pi^{t})$  and  $\Delta_1\underline{Q}_1(s_1,\pi^t)\coloneqq \underline{Q}_1(s_1,\pi^t) - V_1^{\pi^t}(s_1)$ . Since term  $\sum_{t = 1}^{T}\mathrm{SubOpt}_{\pi}^{M_t}(\pi^t)$  can be controlled Lemma B.6, it remains to control
$$
\begin{array}{l} J := \mathbb {E} _ {\mathcal {D}} \left[ \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} [ \mathcal {E} _ {h} ^ {\pi^ {t}} (\underline {{Q}} _ {h} ^ {t}, \underline {{Q}} _ {h + 1} ^ {t}) (s _ {h}, a _ {h}) ] + \Delta_ {1} \underline {{Q}} _ {1} (s _ {1}, \pi_ {1} ^ {t}) \right] \\ = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \tilde {\pi}, \mathcal {D})} \left[ \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} [ \mathcal {E} _ {h} ^ {\tilde {\pi}} (f _ {h}, f _ {h + 1}) (s _ {h}, a _ {h}) ] + \Delta_ {1} f _ {1} (s _ {1}, \tilde {\pi} _ {1}) \right]. \\ \end{array}
$$
Step 2: Decoupling argument. Using Lemma B.5, we have
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) \right] + \Delta_ {1} f _ {1} \left(s _ {1}, \tilde {\pi} _ {1}\right) \\ \leq \frac {0 . 1 2 5 K \gamma}{\lambda} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h}, a _ {h}\right) ^ {2} \right] + \nu_ {h} ^ {2} + 4 b \nu_ {h}\right) + \frac {0 . 5 \lambda H C (\pi , \epsilon_ {c})}{K \gamma} + \Delta_ {1} f _ {1} \left(s _ {1}, \tilde {\pi} _ {1}\right) \\ + H \epsilon_ {c} + \sum_ {h = 1} ^ {H} \nu_ {h} \\ = \frac {0 . 1 2 5 K \gamma \sum_ {h = 1} ^ {H} \mathbb {E} _ {\mu} \left[ \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h} , f _ {h + 1}\right) \left(s _ {h} , a _ {h}\right) ^ {2} \right] + \lambda \Delta_ {1} f _ {1} \left(s _ {1} , \tilde {\pi} _ {1}\right) + \iota_ {1}}{\lambda} + \frac {0 . 5 \lambda H C (\pi , \epsilon_ {c})}{K \gamma} \\ + H \epsilon_ {c} + \sum_ {h = 1} ^ {H} \nu_ {h} \\ \end{array}
$$
where  $\iota_{1}:= 0.125K\gamma \left(\sum_{h = 1}^{H}\nu_{h}^{2} + 4b\sum_{h = 1}^{H}\nu_{h}\right)$
Applying Proposition 2, taking the limit  $\alpha \to 1^{-}$ , and re-organizing the terms complete our proof.
It remains to prove Proposition 2, which is the focus of the remaining appendix.
Our proof strategy for Proposition 2 builds upon Dann et al. [2021] where the central idea in the proof is to upper and lower bound the log-partition function – which in our case is as follows:
$$
Z _ {t} := \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} \left[ \widehat {\Phi} (f, \tilde {\pi}; \mathcal {D}) + \lambda \Delta f _ {1} (s _ {1}, \tilde {\pi}) + \ln \hat {p} (f | \mathcal {D}, \tilde {\pi}) \right], \tag {12}
$$
for any  $t \in [T]$  and any  $T \in \mathbb{N}$ . The key technical distinction is that we need to handle the statistical dependence induced by  $\mathbb{E}_{\tilde{\pi} \sim P_t(\cdot|\mathcal{D})} -$  which is absent in Dann et al. [2021]. In concrete, when  $\tilde{\pi}$  depends on  $\mathcal{D}$ , then
$$
\mathbb {E} \Delta L _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \neq \mathcal {E} _ {h} ^ {\tilde {\pi}} \left(f _ {h}, f _ {h + 1}\right) \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {2},
$$
since  $\tilde{\pi}$  depends on  $(s_h^k, a_h^k)$ . We develop an machinery to handle such issue in posterior sampling by carefully controlling the variance of the variable of interest (thus we can leverage the variance-dependent concentration inequality in Lemma B.2) and integrating it into posterior sampling using a uniform convergence argument. Roughly speaking, several milestone results during the process of developing our proof argument, we need to bound the form of
$$
\mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} [ S (f, \tilde {\pi}, \mathcal {D}) ]
$$
where  $S(f, \tilde{\pi}, \mathcal{D})$  is a function of  $f, \tilde{\pi}, \mathcal{D}$ . It is useful to view  $S(f, \tilde{\pi}, \mathcal{D})$  as a stochastic process indexed by  $(f, \tilde{\pi})$ . In our machinery, we shall first construct an upper bound on the variance of the random process, namely
$$
V (f, \tilde {\pi}) \geq \mathbb {E} _ {\mathcal {D}} [ S (f, \tilde {\pi}, \mathcal {D}) ^ {2} ].
$$
Using a discretization argument, the union bound and Lemma B.2, we have with probability at least  $1 - \delta$ , for any  $f \in \mathcal{F}$ ,  $\tilde{\pi} \in \Pi^{soft}(T)$ , for any  $t \in [0, \frac{1}{\sup S(f, \tilde{\pi}, \mathcal{D})}]$ , we have
$$
S (f, \tilde {\pi}, \mathcal {D}) \leq O _ {K} (1) + \mathbb {E} _ {\mathcal {D}} [ S (f, \tilde {\pi}, \mathcal {D}) ] + (e - 2) t \mathbb {E} _ {\mathcal {D}} [ S (f, \tilde {\pi}, \mathcal {D}) ^ {2} ] + \frac {\ln (N / \delta)}{t}
$$
where  $O_K(1)$  is a discretization error that can be controlled, and  $N$  is a covering number of  $\mathcal{F} \times \Pi^{soft}(T)$ . Note that  $S(f, \tilde{\pi}, \mathcal{D})$  often involves the squared loss which satisfies the Bernstein condition (see Lemma G.1) – thus we can roughly bound  $\mathbb{E}_{\mathcal{D}}[S(f, \tilde{\pi}, \mathcal{D})^2] \leq \alpha |\mathbb{E}_{\mathcal{D}}[S(f, \tilde{\pi}, \mathcal{D})]|$  for some constant  $\alpha$ . To integrate the high-probability bound into in-expected bound, we use the argument:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D}, \tilde {\pi})} [ S (f, \tilde {\pi}, \mathcal {D}) ] \leq O _ {K} (1) + \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {\mathcal {D}} [ S (f, \tilde {\pi}, \mathcal {D}) ] \\ + (e - 2) t \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {\mathcal {D}} [ S (f, \tilde {\pi}, \mathcal {D}) ^ {2} ] + \frac {\ln (N / \delta)}{t} + \delta \sup S (f, \tilde {\pi}, \mathcal {D}). \\ \end{array}
$$
E.3.1 Lower-bounding log-partition function.
In this appendix, we give a lower bound of the log-partition function defined in Equation (12). The final lower bound is presented in Proposition 3. In order to establish such a lower bound, we first present a series of support lemmas that will culminate into Proposition 3.
The following lemma decomposes the log-partition function  $Z$  into different terms that we shall control separately.
Lemma E.1. For any  $t \in [T]$  and any  $T \in \mathbb{N}$ , we have
$$
\begin{array}{l} Z _ {t} \geq \underbrace {\mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D} , \tilde {\pi})} \left[ \lambda \Delta f _ {1} (s _ {1} , \tilde {\pi}) + (1 - 0 . 5 \alpha) \ln \frac {\hat {p} (f _ {1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {1})} \right]} _ {A _ {t}} \\ + 0. 5 \alpha \sum_ {h = 1} ^ {H} \underbrace {\mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D} , \tilde {\pi})} \left[ 2 \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} \left(f _ {h} , f _ {h + 1} ; z _ {h} ^ {k}\right) + \ln \frac {\hat {p} \left(f _ {h} , f _ {h + 1} | \mathcal {D} , \tilde {\pi}\right)}{p _ {0} \left(f _ {h} , f _ {h + 1}\right)} \right]} _ {B _ {h, t}} \\ + \sum_ {h = 1} ^ {H} \underbrace {\mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\tilde {\pi} \sim P _ {t} (\cdot | \mathcal {D})} \mathbb {E} _ {f \sim \hat {p} (\cdot | \mathcal {D} , \tilde {\pi})} \left[ \alpha \ln \mathbb {E} _ {f _ {h} ^ {\prime} \sim p _ {0}} \exp \left(- \gamma \sum_ {k = 1} ^ {K} \Delta L _ {\tilde {\pi}} (f _ {h} ^ {\prime} , f _ {h + 1} ; z _ {h} ^ {k})\right) + (1 - \alpha) \ln \frac {\hat {p} (f _ {h + 1} | \mathcal {D} , \tilde {\pi})}{p _ {0} (f _ {h + 1})} \right]} _ {C _ {h, t}}. \\ \end{array}
$$
Proof of Lemma E.1. This is a simple adaptation of the decomposition in [Dann et al., 2021, Lemma 6].  $\square$
We now control each term of the above decomposition of  $Z$  separately – where a majority of these steps are where our technical arguments depart from those in Dann et al. [2021]. In particular, Lemma E.4, Lemma E.5, and Lemma E.7 are our new technical results.
Bounding  $A_{t}$
Lemma E.2. We have