SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given a historical dataset containing a collection of statistically independent sample trajectories, Li et al. (2022) came up with a model-based approach that enjoys provable minimax optimality. This approach first employs a two-fold subsampling trick in order to decouple the statistical dependency across different steps of a single trajectory. After this subsampling step, this approach resorts to the principle of pessimism in the face of uncertainty, which employs value iteration but penalizes the updates via proper variance-aware penalization (i.e., Bernstein-style lower confidence bounds). Details can be found in Algorithm 6.
Algorithm 5: Subroutine for computing the desired online exploration policy (Li et al., 2023).
1 Initialize:  $\mu_{\mathsf{b}}^{(0)} = \delta_{\pi_{\mathrm{init}}}$  for an arbitrary policy  $\pi_{\mathrm{init}} \in \Pi$ ,  $T_{\max} = \lfloor 50SAH\log (KH)\rfloor$
2 for  $t = 0$  to  $T_{\mathrm{max}}$  do
/* find the optimal policy */
Compute the optimal deterministic policy  $\pi^{(t),\mathbf{b}}$  of the MDP
$\mathcal{M}_{\mathrm{b}} = (\mathcal{S}\cup \{s_{\mathrm{aug}}\} ,\mathcal{A},H,\widetilde{P}^{\mathrm{aug}},r_{\mathrm{b}})$  , where  $s_{\mathrm{aug}}$  is an augmented state,
$$
r _ {\mathrm {b}, h} (s, a) = \left\{ \begin{array}{l l} \frac {1}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu_ {\mathrm {b}} ^ {(t)}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}, & \text {i f} (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ], \\ 0, & \text {i f} (s, a, h) \in \left\{s _ {\mathrm {a u g}} \right\} \times \mathcal {A} \times [ H ], \end{array} \right. \tag {48}
$$
$$
\widehat {P} _ {h} ^ {\mathbf {a u g}} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} \in \mathcal {S} \\ 1 - \sum_ {s ^ {\prime} \in \mathcal {S}} \widehat {P} _ {h} \left(s ^ {\prime} \mid s, a\right), & \text {i f} s ^ {\prime} = s _ {\mathbf {a u g}} \end{array} \quad \text {f o r a l l} (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ]; \right. \tag {49a}
$$
(49b)
Compute // choose the stepsize
$$
\alpha_ {t} = \frac {\frac {1}{S A H} g \left(\pi^ {(t)} , \widehat {d} , \mu_ {\mathbf {b}} ^ {(t)}\right) - 1}{g \left(\pi^ {(t)} , \widehat {d} , \mu_ {\mathbf {b}} ^ {(t)}\right) - 1}, \quad \text {w h e r e} \quad g (\pi , \widehat {d}, \mu) = \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \frac {\frac {1}{K ^ {\mathrm {o n}} H} + \widehat {d} _ {h} ^ {\pi} (s , a)}{\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi \sim \mu} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}.
$$
Here,  $\widehat{d}_h^\pi (s,a)$  is computed via (42) for  $h = 1$ , and (43) for  $h\geq 2$
If  $g(\pi^{(t)},\widehat{d},\mu_{\mathsf{b}}^{(t)})\leq 2HSA$  then exit for-loop. // stopping rule
Update // Frank-Wolfe update
$$
\mu_ {\mathfrak {b}} ^ {(t + 1)} = \left(1 - \alpha_ {t}\right) \mu_ {\mathfrak {b}} ^ {(t)} + \alpha_ {t} \mathbb {1} _ {\pi^ {(t)}}.
$$
10 Output: the exploration policy  $\pi^{\mathrm{explore}} = \mathbb{E}_{\pi \sim \mu_{\mathrm{b}}^{(t)}}[\pi]$  and the associated weight  $\mu^{\mathrm{explore}} = \mu_{\mathrm{b}}^{(t)}$ .
Algorithm 6: A pessimistic model-based offline RL algorithm (Li et al., 2022).
1 Input: a dataset  $\mathcal{D}$ ; reward function  $r$ . Let  $K_{0}$  denote the number of sample trajectories in  $\mathcal{D}$ .
2 Subsampling: run the following procedure to generate the subsampled dataset  $\mathcal{D}^{\mathrm{trim}}$ .
1) Data splitting. Split  $\mathcal{D}$  into two halves:  $\mathcal{D}^{\mathrm{main}}$  (which contains the first  $K_{0}/2$  trajectories), and  $\mathcal{D}^{\mathrm{aux}}$  (which contains the remaining  $K_{0}/2$  trajectories); we let  $N_{h}^{\mathrm{main}}(s)$  (resp.  $N_{h}^{\mathrm{aux}}(s)$ ) denote the number of sample transitions in  $\mathcal{D}^{\mathrm{main}}$  (resp.  $\mathcal{D}^{\mathrm{aux}}$ ) that transition from state  $s$  at step  $h$ .
2) Lower bounding  $\{N_h^{\mathrm{main}}(s)\}$  using  $\mathcal{D}^{\mathrm{aux}}$ . For each  $s \in S$  and  $1 \leq h \leq H$ , compute
$$
N _ {h} ^ {\operatorname {t r i m}} (s) := \max  \left\{N _ {h} ^ {\mathrm {a u x}} (s) - 1 0 \sqrt {N _ {h} ^ {\mathrm {a u x}} (s) \log \frac {H S}{\delta}}, 0 \right\}; \tag {50}
$$
3) Random subsampling. Let  $\mathcal{D}^{\mathrm{main}}$  be the set of all sample transitions (i.e., the quadruples taking the form  $(s, a, h, s')$ ) from  $\mathcal{D}^{\mathrm{main}}$ . Subsample  $\mathcal{D}^{\mathrm{main}}$  to obtain  $\mathcal{D}^{\mathrm{trim}}$ , such that for each  $(s, h) \in S \times [H]$ ,  $\mathcal{D}^{\mathrm{trim}}$  contains  $\min \{N_h^{\mathrm{trim}}(s), N_h^{\mathrm{main}}(s)\}$  sample transitions randomly drawn from  $\mathcal{D}^{\mathrm{main}}$ . (We shall also let  $N_h^{\mathrm{trim}}(s, a)$  denote the number of samples that visits  $(s, a, h)$  in  $\mathcal{D}^{\mathrm{trim}}$ .)
3 Run VI-LCB: set  $\mathcal{D}_0 = \mathcal{D}^{\mathrm{trim}}$ ; run Algorithm 7 to compute a policy  $\widehat{\pi}$ .
4   
5   
6   
7   
8   
9
Algorithm 7: Offline value iteration with lower confidence bounds (Li et al., 2022).
1 Input: dataset  $\mathcal{D}_0$  ; reward function  $r$  target success probability  $1 - \delta$    
2 Initialization:  $\widehat{V}_{H + 1} = 0$    
3 for  $h = H,\dots ,1$  do   
4 compute the empirical transition kernel  $\widehat{P}_h$  as   
 $\begin{array}{r}\widehat{P}_h(s'|s,a) = \left\{\frac{1}{N_h(s,a)}\sum_{i = 1}^N\mathbb{1}\big\{(s_i,a_i,h_i,s_i') = (s,a,h,s')\big\} ,\right.\\ \frac{1}{S},\qquad \mathrm{else,} \end{array}$  (51)   
where  $N_{h}(s,a)\coloneqq \sum_{i = 1}^{N}\mathbb{1}\big\{(s_{i},a_{i},h_{i}) = (s,a,h)\big\}$  and   
 $N_{h}(s)\coloneqq \sum_{i = 1}^{N}\mathbb{1}\big\{(s_{i},h_{i}) = (s,h)\big\} .$    
for  $s\in S,a\in \mathcal{A}$  do   
6 compute the penalty term  $b_{h}(s,a)$  as   
7  $\forall (s,a,h)\in \mathcal{S}\times \mathcal{A}\times [H]:b_h(s,a) = \min \left\{\sqrt{\frac{c_b\log\frac{K}{\delta}}{N_h(s,a)}\mathsf{Var}_{\widehat{P}_h(\cdot|s,a)}(\widehat{V}_{h + 1})} + c_bH\frac{\log\frac{K}{\delta}}{N_h(s,a)},H\right\}$    
for some universal constant  $c_{\mathrm{b}} > 0$  (e.g.,  $c_{\mathrm{b}} = 16$  ); set   
8  $\widehat{Q}_h(s,a) = \max \left\{r_h(s,a) + \widehat{P}_{h,s,a}\widehat{V}_{h + 1} - b_h(s,a),0\right\} .$    
9 for  $s\in S$  do   
10 set  $\widehat{V}_h(s) = \max_a\widehat{Q}_h(s,a)$  and  $\widehat{\pi}_h(s)\in \arg \max_a\widehat{Q}_h(s,a)$
E Analysis of Theorem 1
In this section, we present the proof for our main result in Theorem 1. Throughout the proof, we let  $\{\mathcal{G}_h\}_{1\leq h\leq H}$  denote a sequence of subsets obeying
$$
\max  _ {1 \leq h \leq H} \max  _ {(s, a) \in \mathcal {G} _ {h}} \frac {d _ {h} ^ {\pi^ {\star}} (s , a)}{d _ {h} ^ {\mathrm {o f f}} (s , a)} = C ^ {\star} (\sigma) \quad \text {a n d} \quad \frac {1}{H} \sum_ {h = 1} ^ {H} \sum_ {(s, a) \notin \mathcal {G} _ {h}} d _ {h} ^ {\pi^ {\star}} (s, a) \leq \sigma , \tag {52}
$$
as motivated by Definition 2. As it turns out, if  $K^{\mathrm{on}} \geq c_1 \frac{H^3 SA}{\varepsilon^2} \log \frac{K}{\delta}$  for some large enough constant  $c_1 > 0$ , then the claimed result in Theorem 1 follows immediately from the main theory in Li et al. (2023) developed for pure online exploration. As a result, it suffices to prove the theorem by replacing Condition (24b) with
$$
K ^ {\mathrm {o n}} \geq c _ {1} \frac {H ^ {4} S A \sigma}{\varepsilon^ {2}} \log \frac {K}{\delta} \tag {53}
$$
throughout this section.
On a high level, our proof comprises the following three steps:
- Establish the proximity of  $\widehat{d}^{\mathrm{off}}$  (resp.  $\widehat{d}^{\pi}$ ) and  $d^{\mathrm{off}}$  (resp.  $d^{\pi}$ ).
- Show that the mixed policy  $\pi^{\text{imitate}}$  is able to mimic and strengthen the offline dataset  $\mathcal{D}^{\text{off}}$ , while the mixed policy  $\pi^{\text{explore}}$  is capable of exploring the part of the state-action space that has not been adequately visited by  $\mathcal{D}^{\text{off}}$ .
- Derive the sub-optimality of the policy returned by the offline RL algorithm (i.e., Algorithm 6) when applied to the hybrid dataset  $\mathcal{D} = \mathcal{D}^{\mathrm{off},2} \cup \mathcal{D}_{\mathrm{imitate}}^{\mathrm{on}} \cup \mathcal{D}_{\mathrm{explore}}^{\mathrm{on}}$ .
In the sequel, we shall elaborate on these three steps.
E.1 Step 1: establishing the proximity of  $\widehat{d^{\pi}}$  (resp.  $\widehat{d^{\mathrm{off}}}$ ) and  $d^{\pi}$  (resp.  $d^{\mathrm{off}}$ )
To begin with, the goodness of the occupancy distribution estimators  $\widehat{d^{\pi}}$  (cf. Algorithm 3) has been analyzed in Li et al. (2023, Lemma 4), which come with the following performance guarantees.
Lemma 1 (Li et al. (2023)). Recall that  $\xi = c_{\xi}H^{3}S^{3}A^{3}\log \frac{HSA}{\delta}$  for some large enough constant  $c_{\xi} > 0$ . With probability at least  $1 - \delta$ , the estimated occupancy distributions specified in Algorithm 3 satisfy
$$
\frac {1}{2} \widehat {d} _ {h} ^ {\pi} (s, a) - \frac {\xi}{4 N} \leq d _ {h} ^ {\pi} (s, a) \leq 2 \widehat {d} _ {h} ^ {\pi} (s, a) + 2 e _ {h} ^ {\pi} (s, a) + \frac {\xi}{4 N} \tag {54}
$$
simultaneously for all  $(s,a,h)\in \mathcal{S}\times \mathcal{A}\times [H]$  and all deterministic policy  $\pi \in \Pi$ , provided that
$$
K ^ {\text {o n}} \geq C _ {N} H ^ {1 8} S ^ {1 4} A ^ {1 4} \log^ {2} \frac {H S A}{\delta} \tag {55}
$$
for some large enough constant  $C_N > 0$ . Here,  $\{e_h^\pi (s,a)\}$  is some non-negative sequence satisfying
$$
\sum_ {s, a} e _ {h} ^ {\pi} (s, a) \leq \frac {2 S A}{K ^ {\mathrm {o n}}} + \frac {1 3 S A H \xi}{N} \quad \text {f o r a l l} h \in [ H ] \text {a n d a l l d e t e r m i n i s t i c M a r k o v p o l i c y} \pi . \tag {56}
$$
We now turn to the estimator  $\widehat{d}^{\mathrm{off}}$  (cf. (15)) for the occupancy distribution of the offline dataset, for which we begin with the following lemma concerning the proximity of  $d_h^{\mathrm{off}}$  and  $\widehat{d}_h^{\mathrm{off}}$ . The proof of this lemma is deferred to Section G.1.
Lemma 2. Suppose that  $c_{\mathrm{off}} \geq 48$ . With probability at least  $1 - \delta / 3$ , one has
$$
\frac {1}{3} \widehat {d} _ {h} ^ {\text {o f f}} (s, a) \leq d _ {h} ^ {\text {o f f}} (s, a) \leq \widehat {d} _ {h} ^ {\text {o f f}} (s, a) + 5 c _ {\text {o f f}} \left\{\frac {\log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K ^ {\text {o n}}} \right\} \tag {57}
$$
simultaneously for all  $(s,a,h)\in S\times \mathcal{A}\times [H]$
This lemma implies that: when  $d_h^{\mathrm{off}}(s,a) \lesssim \frac{\log\frac{HSA}{\delta}}{K^{\mathrm{off}}} + \frac{H^4S^4A^4\log\frac{HSA}{\delta}}{N} + \frac{SA}{K^{\mathrm{on}}}$ , the estimator  $\widehat{d}_h^{\mathrm{off}}(s,a)$  might be unable to track  $d_h^{\mathrm{off}}(s,a)$  in a faithful manner. This motivates us to single out the following two subsets of state-action pairs for which  $\widehat{d}_h^{\mathrm{off}}(s,a)$  might become problematic at step  $h$ :
- the set  $\mathcal{G}_h^c$  (see (11) for the definition of  $\mathcal{G}_h$ ), which corresponds to the set of optimal state-action pairs that even the true data distribution  $d_h^{\mathrm{off}}$  cannot cover adequately;
another set  $\mathcal{T}_h^{\mathrm{small}}$  defined as
$$
\mathcal {T} _ {h} ^ {\text {s m a l l}} := \left\{(s, a): d _ {h} ^ {\text {o f f}} (s, a) \leq 1 0 c _ {\text {o f f}} \left(\frac {\log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K ^ {\text {o n}}}\right) \right\}, \tag {58}
$$
comprising those state-action pairs for which  $\widehat{d}_h^{\mathrm{off}}(s,a)$  might not be a faithful estimator of  $d_h^{\mathrm{off}}(s,a)$ .
In what follow, we shall adopt the notation:
$$
\mathcal {T} _ {h} := \mathcal {G} _ {h} ^ {\mathrm {c}} \cup \mathcal {T} _ {h} ^ {\text {s m a l l}}. \tag {59}
$$
It is straightforward to demonstrate that:
- For any  $(s, a) \notin T_h^{\text{small}}$ , it is seen from Lemma 2 that
$$
d _ {h} ^ {\text {o f f}} (s, a) \leq \widehat {d} _ {h} ^ {\text {o f f}} (s, a) + \frac {1}{2} d _ {h} ^ {\text {o f f}} (s, a) \quad \Longleftrightarrow \quad d _ {h} ^ {\text {o f f}} (s, a) \leq 2 \widehat {d} _ {h} ^ {\text {o f f}} (s, a). \tag {60}
$$
- For any  $(s, a) \in \mathcal{G}_h$ , Condition (52) tells us that
$$
d _ {h} ^ {\pi^ {\star}} (s, a) \leq C ^ {\star} (\sigma) d _ {h} ^ {\text {o f f}} (s, a). \tag {61}
$$
As a consequence, any  $(s,a)\notin T_h$  necessarily obeys
$$
d _ {h} ^ {\pi^ {*}} (s, a) \leq C ^ {\star} (\sigma) d _ {h} ^ {\mathrm {o f f}} (s, a) \leq 2 C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\mathrm {o f f}} (s, a). \tag {62}
$$
Another useful observation that we can readily make is as follows:
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {T} _ {h}} d _ {h} ^ {\pi^ {*}} (s, a) \leq \sum_ {h = 1} ^ {H} \sum_ {(s, a) \notin \mathcal {G} _ {h}} d _ {h} ^ {\pi^ {*}} (s, a) + \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {G} _ {h} \cap \mathcal {T} _ {h} ^ {\text {s m a l l}}} d _ {h} ^ {\pi^ {*}} (s, a) \\ \leq H \sigma + \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {G} _ {h} \cap \mathcal {T} _ {h} ^ {\text {s m a l l}}} d _ {h} ^ {\pi^ {*}} (s, a) \\ \leq H \sigma + C ^ {\star} (\sigma) \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {T} _ {h} ^ {\text {s m a l l}}} d _ {h} ^ {\text {o f f}} (s, a) \mathbb {1} \left(a = \pi^ {\star} (s)\right) \\ \leq H \sigma + C ^ {\star} (\sigma) H S \cdot 1 0 c _ {\text {o f f}} \left(\frac {\log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K ^ {\text {o n}}}\right) \\ \leq H \sigma + 1 0 c _ {\text {o f f}} \left(\frac {C ^ {\star} (\sigma) H S \log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {4 C ^ {\star} (\sigma) H ^ {6} S ^ {5} A ^ {4} \log \frac {H S A}{\delta}}{K ^ {\text {o n}}}\right) =: \widehat {\sigma}. \tag {63} \\ \end{array}
$$
Here, the second and the third lines arise from Condition (52), the penultimate line invokes the definition (58) of  $\mathcal{T}_h^{\mathrm{small}}$ , whereas the last line is valid since  $N = K^{\mathrm{on}} / (3H)$  (see (14)).
E.2 Step 2: showing that  $\pi^{\mathrm{imitate}}$  (resp.  $\pi^{\mathrm{explore}}$ ) covers  $\widehat{d}^{\mathrm{off}}$  (resp.  $d^{\pi^*}$ ) adequately
In this step, we aim to demonstrate the quality of the two exploration policies  $\pi^{\mathrm{imitate}}$  and  $\pi^{\mathrm{explore}}$ , designed for different purposes.
Goodness of  $\pi^{\text{imitate}}$ . We begin by assessing the quality of the exploration policy  $\pi^{\text{imitate}}$ . Towards this, we first make note of the following crude bound:
$$
\frac {\widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)}{\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {t}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \leq \frac {\widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)}{\frac {1}{K ^ {\mathrm {o n}} H}} \leq K ^ {\mathrm {o n}} H =: L.
$$
In view of the convergence guarantees for FTRL (Shalev-Shwartz, 2012, Corollary 2.16), we see that: if  $\eta = \sqrt{\frac{\log A}{2T_{\max}L^2}} = \sqrt{\frac{\log A}{2T_{\max}(K^{\mathrm{on}}H)^2}}$ , then running FTRL for  $T_{\max}$  iterations results in
$$
\begin{array}{l} \max  _ {a \in \mathcal {A}} \frac {1}{T _ {\max }} \sum_ {t = 1} ^ {T _ {\max }} \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {t}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} - \frac {1}{T _ {\max }} \sum_ {t = 1} ^ {T _ {\max }} \sum_ {a \in \mathcal {A}} \pi_ {h} ^ {t} (a \mid s) \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {t}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \\ \leq K ^ {\text {o n}} H \sqrt {\frac {2 \log A}{T _ {\max}}} \tag {64} \\ \end{array}
$$
for all  $s \in S$  and  $1 \leq h \leq H$ . Therefore, recalling that  $\mu^{\text{imitate}} = \frac{1}{T_{\text{max}}} \sum_{t=1}^{T_{\text{max}}} \mu^t$  and applying Jensen's inequality yield
$$
\begin{array}{l} \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \frac {\widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)}{\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {\mathrm {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \\ \leq \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \frac {1}{T _ {\operatorname* {m a x}}} \sum_ {t = 1} ^ {T _ {\operatorname* {m a x}}} \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {t}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \\ \leq \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \frac {1}{T _ {\max }} \sum_ {t = 1} ^ {T _ {\max }} \pi_ {h} ^ {t} (a \mid s) \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {t}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} + K ^ {\text {o n}} H ^ {2} S \sqrt {\frac {2 \log A}{T _ {\max }}} \tag {65} \\ \end{array}
$$
where the second inequality results from (64). In addition, it follows from the stopping rule (37) that
$$
\sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \pi_ {h} ^ {t} (a \mid s) \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {t}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \leq 1 0 8 S H. \tag {66}
$$
As a consequence, combining (65) and (66) yields
$$
\sum_ {h \in [ H ]} \sum_ {s \in \mathcal {S}} \max  _ {a \in \mathcal {A}} \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \leq 1 0 8 S H + K ^ {\text {o n}} H ^ {2} S \sqrt {\frac {2 \log A}{T _ {\operatorname* {m a x}}}} \leq 1 0 9 S H, \tag {67}
$$
provided that  $T_{\mathrm{max}} \geq 2(K^{\mathrm{on}}H)^2 \log A$ . The fact that the left-hand side of (67) is well-controlled suggests that  $\pi^{\mathrm{imitate}}$  is able to cover  $\hat{d}^{\mathrm{off}}$  adequately, a crucial fact we shall rely on in the subsequent analysis.
Goodness of  $\pi^{\mathrm{explore}}$ . Next, we turn attention to the other exploration policy  $\pi^{\mathrm{explore}}$ , computed via Algorithm 5. The following performance guarantees have been established in Li et al. (2023, Section 3.2).
Lemma 3. The distribution  $\mu^{\mathrm{explore}}\in \Delta (\Pi)$  returned by Algorithm 5 satisfies
$$
\max _ {\pi} \sum_ {h = 1} ^ {H} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \frac {\widehat {d} _ {h} ^ {\pi} (s , a)}{\frac {1}{K ^ {\mathrm {o n}} H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {\mathrm {e x p l o r e}}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]} \leq 2 H S A.
$$
In light of the performance bound (17) for the subsequent offline RL approach, Lemma 3 suggests that  $\pi^{\mathrm{explore}}$  is able to explore well with regards to the visitation of any policy  $\pi$  — including the optimal policy  $\pi^{\star}$ .
E.3 Step 3: establishing the performance of offline RL
Now, we can readily proceed to analyze the performance of the model-based offline procedure described in Algorithm 6. In this subsection, we abuse the notation  $\widehat{P}$  to represent the empirical transition kernel constructed within the offline subroutine in Algorithm 7. Additionally, we introduce a  $S$ -dimensional vector  $d_h^{\pi^*} \coloneqq [d_h^{\pi^*}(s)]_{s \in S}$ .
E.3.1 Step 3.1: error decomposition
To begin with, we convert the sub-optimality gap of the policy estimate  $\widehat{\pi}$  into several terms that shall be controlled separately. The following two preliminary facts, which have been established in Li et al. (2022), prove useful for this purpose.
Lemma 4. With probability exceeding  $1 - \delta /3$ , one has
$$
N _ {h} ^ {\text {m a i n}} (s, a) \geq N _ {h} ^ {\text {t r i m}} (s, a), \quad \forall (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ]
$$
$$
\left\langle d _ {j} ^ {\pi^ {\star}}, V _ {j} ^ {\star} - V _ {j} ^ {\widehat {\pi}} \right\rangle \leq 2 \sum_ {h: h \geq j} \sum_ {s, a} d _ {h} ^ {\pi^ {\star}} (s, a) b _ {h} (s, a), \quad 1 \leq h \leq H,
$$
where  $b_{h}(s,a)$  is defined in line 7 of Algorithm 7.
In view of Lemma 4, we can derive, for all  $j \in [H]$ ,
$$
\begin{array}{l} \left\langle d _ {j} ^ {\pi^ {\star}}, V _ {j} ^ {\star} - V _ {j} ^ {\widehat {\pi}} \right\rangle \leq 2 \sum_ {h: h \geq j} \sum_ {s, a} d _ {h} ^ {\pi^ {\star}} (s, a) b _ {h} (s, a) = 2 \sum_ {h: h \geq j} \sum_ {s} d _ {h} ^ {\pi^ {\star}} \left(s, \pi_ {h} ^ {\star} (s)\right) b _ {h} \left(s, \pi_ {h} ^ {\star} (s)\right) \\ \leq 2 \sum_ {h: h \geq j} \sum_ {s: (s, \pi^ {\star} (s)) \notin \mathcal {T} _ {h}} \sqrt {2 d _ {h} ^ {\pi^ {\star}} (s , \pi_ {h} ^ {\star} (s)) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\mathrm {o f f}} (s , \pi_ {h} ^ {\star} (s))} b _ {h} (s, \pi_ {h} ^ {\star} (s)) + 2 \sum_ {h: h \geq j} \sum_ {(s, a) \in \mathcal {T} _ {h}} d _ {h} ^ {\pi^ {\star}} (s, a) b _ {h} (s, a) \\ \leq 2 \sum_ {h: h \geq j} \sum_ {s: (s, \pi^ {\star} (s)) \notin \mathcal {T} _ {h}} \sqrt {2 d _ {h} ^ {\pi^ {\star}} (s , \pi_ {h} ^ {\star} (s)) C ^ {\star} (\sigma) \widehat {d _ {h} ^ {\mathrm {o f f}}} (s , \pi_ {h} ^ {\star} (s))} b _ {h} (s, \pi_ {h} ^ {\star} (s)) \\ \end{array}
$$
$$
+ 4 \sum_ {h: h \geq j} \sum_ {(s, a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {*}} (s, a) b _ {h} (s, a) + \frac {8 H ^ {2} S A}{K ^ {\mathrm {o n}}} + \frac {5 3 c _ {\xi} H ^ {6} S ^ {4} A ^ {4}}{N} \log \frac {H S A}{\delta}.
$$
Here, the second line comes from (62), whereas the third line is due to Lemma 1 and the basic fact that  $b_h(s,a) \leq H$  (see line 7 of Algorithm 7). Substituting the definition of  $b_h$  (see line 7 of Algorithm 7) into the above display and applying Lemma 4, we arrive at
$$
\begin{array}{l} \left\langle d _ {j} ^ {\pi^ {\star}}, V _ {j} ^ {\star} - V _ {j} ^ {\widehat {\pi}} \right\rangle \leq \sum_ {h: h \geq j} \sum_ {s} \max _ {a: (s, a) \notin \mathcal {T} _ {h}} \left\{\sqrt {8 d _ {h} ^ {\pi^ {\star}} (s , a) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)}. \right. \\ \left. \min  \left\{\sqrt {\frac {c _ {\mathrm {b}} \log \frac {K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a)}} \mathsf {V a r} _ {\widehat {P} _ {h} (\cdot | s, a)} \big (\widehat {V} _ {h + 1} \big) + \frac {c _ {\mathrm {b}} H \log \frac {K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a)}, H \right\} \right\} \\ + 4 H \sum_ {h: h \geq j} \sum_ {(s, a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {*}} (s, a) \sqrt {\frac {c _ {\mathrm {b}} \log \frac {K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a) + 1}} + \frac {8 H ^ {2} S A}{K ^ {\mathrm {o n}}} + \frac {5 3 c _ {\xi} H ^ {6} S ^ {4} A ^ {4}}{N} \log \frac {K}{\delta}, \tag {68} \\ \end{array}
$$
where we recall that  $c_{\mathrm{b}} > 0$  is also an absolute constant used to specify  $b_{h}(s,a)$ .
It is worth noting that the right-hand side of (68) involves a variance term  $\mathsf{Var}_{\widehat{P}_h(\cdot |s,a)}(\widehat{V}_{h + 1})$  w.r.t. the empirical model  $\widehat{P}$ . As it turns out, the following lemma established in Li et al. (2022, Lemma 8) makes apparent the intimate connection between  $\mathsf{Var}_{\widehat{P}_h(\cdot |s,a)}(\widehat{V}_{h + 1})$  and  $\mathsf{Var}_{P_h(\cdot |s,a)}(\widehat{V}_{h + 1})$ .
Lemma 5. With probability exceeding  $1 - \delta /3$ , we have, for all  $(s,a,h)\in S\times \mathcal{A}\times [H]$
$$
\mathsf {V a r} _ {\widehat {P} _ {h} (\cdot | s, a)} \big (\widehat {V} _ {h + 1} \big) \leq 2 \mathsf {V a r} _ {P _ {h} (\cdot | s, a)} \big (\widehat {V} _ {h + 1} \big) + \frac {1 0 H ^ {2} \log \frac {K}{\delta}}{3 N _ {h} ^ {\mathrm {t r i m}} (s , a)}.
$$
Substituting the result of Lemma 5 into (68) leads to