SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the worst case, the data distribution  $\mu$  at a certain  $(s,a)$  pair can be zero when  $d^{\pi}(s,a) > 0$ , which implies the concentrability coefficient  $C^{*} = \infty$ . Here,  $\pi$  is an arbitrary comparator policy. In this case, either classical offline RL algorithm or our policy finetuning algorithm cannot guarantee an  $\varepsilon$ -optimal policy compared to the global optimal policy. However, we can still output a locally  $\varepsilon$ -optimal policy, compared to the optimal policy on the sparsified MDP.
In order to compare  $\pi_{final}$  to any policy on the original MDP, we have the corollary 5.2, which will be proved in appendix B.3.
The statement in corollary 5.2 is a quite direct consequence of theorem 5.1, and it expresses the sub-optimality gap of  $\pi_{final}$  with respect to any comparator policy  $\pi$  on the original MDP  $\mathcal{M}$ . It can also be written in terms of the sub-optimality: If we fix a comparator policy  $\pi$ , then with probability at least  $1 - \delta$ , for any reward function  $r$ , the policy  $\pi_{final}$  returned by algorithm 2 satisfies:
$$
\begin{array}{l} V_{1}\left(s_{1};\mathbb{P},r,\pi\right) - V_{1}\left(s_{1};\mathbb{P},r,\pi_{final}\right) = \widetilde{O}\Big(\underbrace{\frac{H|\mathcal{S}| \sqrt{|\mathcal{A}|}}{\sqrt{K_{de}}}}_{\text{Online error}} + \underbrace{\frac{H^{4}|\mathcal{S}|}{N}\sum_{s,a}\frac{d^{\pi}(s,a)}{\mu(s,a)}}_{\text{Offline error}}\Big) \\ = \widetilde {O} \left(\frac {H | \mathcal {S} | \sqrt {| \mathcal {A} |}}{\sqrt {K _ {d e}}} + \frac {H ^ {4} | \mathcal {S} | ^ {2} | \mathcal {A} |}{N} \sup _ {s, a} \frac {d ^ {\pi} (s , a)}{\mu (s , a)}\right), \\ \end{array}
$$
where  $K_{de}$  is the number of online episodes and  $N$  is the number of state-action pairs in offline data. Here, the sub-optimality depends on an online error as well as on an offline error. The online error is the one that also arises in the statement of theorem 5.1. It is an error that can be reduced by collecting more online samples, i.e., by increasing  $K$ , with the typical inverse square-root dependence  $1 / \sqrt{K}$ .
However, the upper bound suggests that even in the limit of infinite online data, the value of  $\pi_{final}$  will not approach that of  $\pi_*$  because of a residual error due to the offline dataset  $\mathcal{D}$ . Such residual error depends on certain concentrability factor expressed as  $\sum_{s,a} \frac{d^{\pi}(s,a)}{\mu(s,a)} \leq |\mathcal{S}||\mathcal{A}| \sup_{s,a} \frac{d^{\pi}(s,a)}{\mu(s,a)}$ , whose presence is intuitive: if a comparator policy  $\pi$  is not covered well, our algorithm does not have enough information to navigate to the area that  $\pi$  tends to visit, and so it is unable to refine its estimates there. However the dependence on the number of offline samples  $N = |\mathcal{D}|$  is through its inverse, i.e.,  $1/N$  as opposed to the more typical  $1/\sqrt{N}$ : such gap represents the improvable performance when additional online data are collected non-reactively.
It is useful to compare corollary 5.2 with what is achievable by using a minimax-optimal online algorithm[Xie et al., 2021b]. In this latter case, one can bound the sub-optimality gap for any comparator policy  $\pi$  with high probability as
$$
V _ {1} \left(s _ {1}; \mathbb {P}, r ^ {\dagger}, \pi\right) - V _ {1} \left(s _ {1}; \mathbb {P}, r ^ {\dagger}, \pi_ {f i n a l}\right) \leq \widetilde {O} \left(\sqrt {\frac {H ^ {3} | \mathcal {S} |}{N} \sup  _ {s , a} \frac {d ^ {\pi} (s , a)}{\mu (s , a)}}\right). \tag {B.2}
$$
B.3 Proof of corollary 5.2
Let's denote  $\mathcal{D} = \{(s_i, a_i, s_i')\}_{i \in [N]}$  as the offline dataset, where  $N$  as the total number of tuples. We keep the notation the same as in the main text. We use  $N(s, a)$  and  $N(s, a, s')$  to denote the counter of  $(s, a)$  and  $(s, a, s')$  in the offline data  $\mathcal{D}$ . The state-action pairs are sampled i.i.d. from  $\mu(s, a)$  and the subsequent states are sampled from the transition dynamics. We fix a comparator policy  $\pi$  and assume  $\mu(s, a) > 0$  for any  $(s, a)$  such that  $d^{\pi}(s, a) > 0$ , which implies a finite concentrability constant  $C^{*}$ . Here,  $d^{\pi}(s, a)$  is the occupancy probability of  $(s, a)$  when executing policy  $\pi$ , averaged over all stages  $h \in [H]$ .
Similar to the Section B.1, we have
$$
\begin{array}{l} V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi_ {\text {f i n a l}}\right) \\ \leq V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi\right) + \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi\right) - V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {*} ^ {\dagger}\right)} _ {\leq 0} \\ + V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi_ {*} ^ {\dagger}\right) - V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi_ {f i n a l}\right) + \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {f i n a l}\right) - V _ {1} \left(s _ {1} , \mathbb {P} , r , \pi_ {f i n a l}\right)} _ {\leq 0} \\ \lesssim \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} , r , \pi\right) - V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi\right)} _ {\text {A p p r o x i m a t i o n E r r o r}} + \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {*} ^ {\dagger}\right) - V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {\text {f i n a l}}\right)} _ {\text {E s t i m a t i o n E r r o r}}. \tag {B.3} \\ \end{array}
$$
where  $\pi_{*}^{\dagger}$  and  $\pi_{final}$  are the optimal policy on the sparsified MDP and the policy output by our algorithm, respectively, and  $\pi$  is the fixed comparator policy. Here, the second term is non-positive from the definition of  $\pi_{*}^{\dagger}$ , the last term is non-positive from the definition of the sparsified MDP. This is because, for any state-action pair  $(s,a)$  and any fixed policy  $\pi$ , the probability of reaching  $(s,a)$  under  $\mathbb{P}^{\dagger}$  will not exceed that under the true transition probability  $\mathbb{P}$ . If we denote the visiting probability under  $\pi$  and  $\mathbb{P}$  (or  $\mathbb{P}^{\dagger}$  resp.) as  $d_{\pi,h}(s,a)$  ( $d_{\pi,h}^{\dagger}(s,a)$  resp.), then we have
$$
d _ {\pi , h} ^ {\dagger} (s, a) \leq d _ {\pi , h} (s, a)
$$
for any  $h \in [H]$ ,  $s \in S$ ,  $a \in \mathcal{A}$ . Note that, for  $s = s^{\dagger}$ , this does not hold necessarily. Them, for any policy  $\pi$ , we have
$$
\begin{array}{l} V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi\right) = \sum_ {h = 1} ^ {H} \sum_ {s, a} d _ {\pi , h} ^ {\dagger} (s, a) r (s, a) \quad \text {(d e f i n i t i o n o f p o l i c y v a l u e)} \\ = \sum_ {h = 1} ^ {H} \sum_ {s \neq s ^ {\dagger}, a} d _ {\pi , h} ^ {\dagger} (s, a) r (s, a) \quad \left(r \left(s ^ {\dagger}, a\right) = 0 \text {f o r a n y} a\right) \\ \leq \sum_ {h = 1} ^ {H} \sum_ {s \neq s ^ {\dagger}, a} d _ {\pi , h} (s, a) r (s, a) = V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right). \\ \end{array}
$$
Therefore, we get  $V_{1}\left(s_{1},\mathbb{P}^{\dagger},r,\pi_{final}\right) - V_{1}\left(s_{1},\mathbb{P},r,\pi_{final}\right)\leq 0$  when we take  $\pi = \pi_{final}$ .
From the main result (Theorem 5.1), we know the estimation error is bounded as
$$
\underbrace {V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {*} ^ {\dagger}\right) - V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {f i n a l}\right)} _ {\text {E s t i m a t i o n E r r o r}} \lesssim \widetilde {O} \left(\frac {H | \mathcal {S} | \sqrt {| \mathcal {A} |}}{\sqrt {K _ {d e}}}\right), \tag {B.4}
$$
where  $K_{de}$  is the number of online episodes. Therefore, to make the right hand side of (B.4) less than  $\varepsilon / 2$ , one needs at least  $\widetilde{O}\left(\frac{H^2|\mathcal{S}||\mathcal{A}|}{\varepsilon^2}\right)$  online episodes. This is exactly what the main result shows.
So it suffices to bound the approximation error term. From the value difference lemma (Lemma D.11), we have
$$
\begin{array}{l} \left| V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi\right) \right| \\ = \left| \mathbb {E} _ {\mathbb {P}, \pi} \left[ \sum_ {i = 1} ^ {H} \left(\mathbb {P} ^ {\dagger} (\cdot \mid s _ {i}, a _ {i}) - \mathbb {P} (\cdot \mid s _ {i}, a _ {i})\right) ^ {\top} V _ {i + 1} \left(\cdot ; \mathbb {P} ^ {\dagger}, r, \pi\right) \middle | s _ {1} = s \right] \right| \\ = \left| \sum_ {i = 1} ^ {H} \sum_ {s _ {i}, a _ {i}} d _ {\pi , h} \left(s _ {i}, a _ {i}\right) \left(\mathbb {P} ^ {\dagger} (\cdot \mid s _ {i}, a _ {i}) - \mathbb {P} (\cdot \mid s _ {i}, a _ {i})\right) ^ {\top} V _ {i + 1} \left(\cdot ; \mathbb {P} ^ {\dagger}, r, \pi\right) \right| \\ \end{array}
$$
(by expanding the expectation)
$$
\leq H \left| \mathcal {S} \right| \cdot \sum_ {i = 1} ^ {H} \sum_ {s _ {i}, a _ {i}} d _ {\pi , h} (s _ {i}, a _ {i}) \left(\sup _ {s ^ {\prime}} \left| \mathbb {P} ^ {\dagger} (s ^ {\prime} \mid s _ {i}, a _ {i}) - \mathbb {P} (s ^ {\prime} \mid s _ {i}, a _ {i}) \right|\right).
$$
$(V_{i + 1}\leq H$  and the inner product has  $|\mathcal{S}|$  terms)
$$
d _ {\pi} (s, a) = \frac {1}{H} \sum_ {h = 1} ^ {H} d _ {\pi , h} (s, a) \tag {B.5}
$$
as the average visiting probability. Then, we have
$$
\left| V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi\right) \right| \leq | \mathcal {S} | H ^ {2} \sum_ {s, a} \left[ \sup  _ {s ^ {\prime}} \left| \mathbb {P} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) - \mathbb {P} \left(s ^ {\prime} \mid s, a\right) \right| d _ {\pi} (s, a) \right], \tag {B.6}
$$
So it suffices to upper bound  $\left|\mathbb{P}^{\dagger}(s' \mid s, a) - \mathbb{P}(s' \mid s, a)\right|$ . Notice here we only consider  $s \neq s^{\dagger}$  and  $s' \neq s^{\dagger}$ , since the value function starting from  $s^{\dagger}$  is always zero.
For  $(s,a,s')$ , if  $N(s,a,s') \geq \Phi$ , it holds that  $\mathbb{P}^{\dagger}(s' \mid s,a) = \mathbb{P}(s' \mid s,a)$ . Otherwise, from the definition, we know  $\mathbb{P}^{\dagger}(s' \mid s,a) = 0$ , so it suffices to bound  $\mathbb{P}(s' \mid s,a)$  in this case. From lemma B.1, we know that with probability at least  $1 - \delta / 2$ , for any  $(s,a,s')$  such that  $N(s,a,s') < \Phi$ , it holds that
$$
\mathbb {P} (s ^ {\prime} \mid s, a) \leq \frac {2 N (s , a , s ^ {\prime}) + 2 \log \left(2 | \mathcal {S} | ^ {2} | \mathcal {A} | / \delta\right)}{N (s , a)}.
$$
Then we deal with two cases. When  $N\mu (s,a)\geq 6\Phi$  , from lemma B.2 we have
$$
\mathbb {P} (s ^ {\prime} \mid s, a) \leq \frac {4 N (s , a , s ^ {\prime}) + 2 \log \left(4 | \mathcal {S} | ^ {2} | \mathcal {A} | / \delta\right)}{N \mu (s , a) - 2 \log \left(2 | \mathcal {S} | | \mathcal {A} | / \delta\right)} \leq \frac {4 \Phi + 2 \log \left(4 | \mathcal {S} | ^ {2} | \mathcal {A} | / \delta\right)}{N \mu (s , a) - 2 \log \left(2 | \mathcal {S} | | \mathcal {A} | / \delta\right)}.
$$
From the definition of  $\Phi$ , we know that  $2\log \left(4|\mathcal{S}|^2 |\mathcal{A}| / \delta\right) \leq \Phi$  and  $2\log \left(2|\mathcal{S}||\mathcal{A}| / \delta\right) \leq \Phi$ , which implies
$$
\mathbb {P} (s ^ {\prime} \mid s, a) \leq \frac {5 \Phi}{N \mu (s , a) - \Phi} \leq \frac {6 \Phi}{N \mu (s , a)}.
$$
The last inequality comes from our assumption for  $N\mu (s,a)\geq 6\Phi$
In the other case, when  $N\mu (s,a) < 6\Phi$  , it holds that
$$
\mathbb {P} (s ^ {\prime} \mid s, a) \leq 1 \leq \frac {6 \Phi}{N \mu (s , a)}
$$
for any  $(s,a,s')$ . Therefore, for any for any  $(s,a,s')$  such that  $N(s,a,s') < \Phi$ , we have
$$
\mathbb {P} \left(s ^ {\prime} \mid s, a\right) \leq \frac {6 \Phi}{N \mu (s , a)}. \tag {B.7}
$$
Combining equations (B.7) and (B.6), we know for any comparator policy  $\pi$ , it holds that
$$
\underbrace{\left|V_{1}\left(s_{1},\mathbb{P},r,\pi\right) - V_{1}\left(s_{1},\mathbb{P}^{\dagger},r,\pi\right)\right|}_{\text{Approximation Error}}\lesssim \frac{\Phi H^{2}\left|\mathcal{S}\right|}{N}\sum_{s,a}\frac{d_{\pi}(s,a)}{\mu(s,a)}\lesssim \widetilde{O}\left(\frac{H^{4}\left|\mathcal{S}\right|}{N}\sum_{s,a}\frac{d_{\pi}(s,a)}{\mu(s,a)}\right),
$$
where  $N$  is the total number of transitions in the offline data. In order to make the right hand side of last display less than  $\varepsilon /2$ , one needs at least
$$
\widetilde {O} \Big (\frac {H ^ {4} | \mathcal {S} |}{\varepsilon} \sum_ {s, a} \frac {d _ {\pi} (s , a)}{\mu (s , a)} \Big) \leq \widetilde {O} \Big (\frac {H ^ {4} | \mathcal {S} | ^ {2} | \mathcal {A} | C ^ {*}}{\varepsilon} \Big),
$$
offline transitions. Combining the proof for estimation error and approximation error, we conclude.
B.4 Proof for lemma B.1 and lemma B.2
Lemma B.1. With probability at least  $1 - \delta / 2$ , for any  $(s, a, s') \in S \times \mathcal{A} \times S$ , it holds that
$$
N (s, a, s ^ {\prime}) \geq \frac {1}{2} N (s, a) \mathbb {P} \left(s ^ {\prime} \mid s, a\right) - \log \left(\frac {2 \left| \mathcal {S} \right| ^ {2} \left| \mathcal {A} \right|}{\delta}\right).
$$
Proof. We fixed  $(s, a, s')$  and denote  $I$  as the index set where  $(s_i, a_i) = (s, a)$  for  $i \in I$ . We range the indexes in  $I$  as  $i_1 < i_2 < \ldots < i_{N(s, a)}$ . For  $j \leq N(s, a)$ , we denote  $X_j = \mathbb{I}\left(s_{i_j}' = s'\right)$ , which is the indicator of whether the next state is  $s'$  when we encounter  $(s, a)$  the  $j$ -th time. When  $j \geq N(s, a)$ , we denote  $X_j$  as independent Bernoulli random variables with successful rate  $\mathbb{P}(s' \mid s, a)$ . Then, we know  $X_j$  for all  $j \in \mathbb{N}$  are i.i.d. sequence of Bernoulli random variables. From Lemma D.1, we know with probability at least  $1 - \delta/2$ , for any positive integer  $n$ , it holds that
$$
\sum_ {j = 1} ^ {n} X _ {j} \geq \frac {1}{2} \sum_ {j = 1} ^ {n} \mathbb {P} (s ^ {\prime} \mid s, a) - \log \left(\frac {2}{\delta}\right).
$$
We take  $n = N(s, a)$  (although  $N(s, a)$  is random, we can still take it because for any  $n$  the inequality above holds) to get
$$
N (s, a, s ^ {\prime}) = \sum_ {j = 1} ^ {N (s, a)} X _ {j} \geq \frac {1}{2} N (s, a) \mathbb {P} (s ^ {\prime} \mid s, a) - \log \left(\frac {2}{\delta}\right).
$$
Applying a union bound for all  $(s, a, s')$ , we conclude.
Lemma B.2. With probability at least  $1 - \delta / 2$ , for any  $(s, a) \in S \times \mathcal{A}$ , it holds that
$$
N (s, a) \geq \frac {1}{2} N \mu (s, a) - \log \left(\frac {2 | \mathcal {S} | | \mathcal {A} |}{\delta}\right).
$$
Proof. If  $\mu(s, a) = 0$ , this is trivial. We fixed an  $(s, a)$  such that  $\mu(s, a) > 0$ . For  $j \leq N$ , we denote  $X_j = \mathbb{I}(s_j = s, a_j = a)$ , which is the indicator of whether the  $j$ -th state-action pair we encounter in the offline dataset is  $(s, a)$ . When  $j \geq N$ , we denote  $X_j$  as independent Bernoulli random variables with successful rate  $\mu(s, a)$ . Then, we know  $X_j$  for all  $j \in \mathbb{N}$  are i.i.d. sequence of Bernoulli random variables. From Lemma D.1, we know with probability at least  $1 - \delta/2$ , for any positive integer  $n$ , it holds that
$$
\sum_ {j = 1} ^ {n} X _ {j} \geq \frac {1}{2} \sum_ {j = 1} ^ {n} \mu (s, a) - \log \left(\frac {2}{\delta}\right).
$$
We take  $n = N$  to get
$$
N (s, a) = \sum_ {j = 1} ^ {N} X _ {j} \geq \frac {1}{2} N \mu (s, a) - \log \left(\frac {2}{\delta}\right).
$$
Applying a union bound for all  $(s,a)$  such that  $\mu (s,a) > 0$ , we conclude.
C Lower bound
In this section we briefly discuss the optimality of the algorithm. Although the following considerations are also mentioned in the main text, here we mention how they naturally lead to a lower bound.
Lower bound for reward-free exploration Consider the MDP class  $\mathcal{M}$  defined in the proof of the lower bound of Theorem 4.1 in [Jin et al., 2020b]. Assume that the dataset arises from a logging policy  $\pi_{\text{log}}$  which induces the condition  $N(s, a, s') \geq \Phi$  for all  $(s, a, s') \in S \times \mathcal{A}$  for every instance of the class. In this case, every MDP instance  $\mathcal{M} \in \mathcal{M}$  and its sparsified version  $\mathcal{M}^{\dagger}$  coincide. Then the concatenation of the logging policy  $\pi_{\text{log}}$  and of the policy  $\pi_{\text{final}}$  produced by our algorithm (i.e., algorithm 3) can be interpreted as a reactive policy, which must comply with the reward free lower bound established in Theorem 4.1 of [Jin et al., 2020b]. More precisely, the reward-free sample complexity lower bound established in Theorem 4.1 in [Jin et al., 2020b] is
$$
\Omega \left(\frac {\left| \mathcal {S} \right| ^ {2} \left| \mathcal {A} \right| H ^ {2}}{\varepsilon^ {2}}\right) \tag {C.1}
$$
trajectories. This matches the sample complexity of theorem 5.1. Notice that the number of samples originally present in the dataset can be
$$
\left| \mathcal {S} \right| ^ {2} | \mathcal {A} | \times \widetilde {\mathcal {O}} \left(H ^ {2}\right) = \widetilde {\mathcal {O}} \left(H ^ {2} \left| \mathcal {S} \right| ^ {2} \mid \mathcal {A} |\right), \tag {C.2}
$$
a term independent of the accuracy  $\varepsilon$ . Given that when  $\epsilon \leq 1$  we have
$$
\widetilde {O} (H ^ {2} | \mathcal {S} | ^ {2} | \mathcal {A} |) + \Omega \Big (\frac {| \mathcal {S} | ^ {2} | \mathcal {A} | H ^ {2}}{\varepsilon^ {2}} \Big) = \Omega \Big (\frac {| \mathcal {S} | ^ {2} | \mathcal {A} | H ^ {2}}{\varepsilon^ {2}} \Big),
$$
our algorithm is unimprovable beyond constant terms and logarithmic terms in a minimax sense.
Lower bound for non-reactive exploration Consider the MDP class  $\mathcal{M}$  defined in the proof of the lower bound in Theorem 1 of [Xiao et al., 2022]. It establishes an exponential sample complexity for non-reactive exploration when no prior knowledge is available. In other words, in absence of any data about the MDP, non-reactive exploration must suffer an exponential sample complexity. In such case, our theorem 5.1 (correctly) provides vacuous guarantees, because the sparsified MDP  $\mathcal{M}$  is degenerate (all edges lead to the absorbing state).
Combining the two constructions It is possible to combine the MDP class  $\mathcal{M}_1$  from the paper [Xiao et al., 2022] with the MDP class  $\mathcal{M}_2$  from the paper [Jin et al., 2020b]. In lieu of a formal proof, here we provide only a sketch of the construction that would induce a lower bound. More precisely, consider a starting state  $s_1$  where only two actions— $a = 1$  and  $a = 2$ —are available. Taking  $a = 1$  leads to the start state of an instance of the class  $\mathcal{M}_1$ , while taking  $a = 2$  leads to the start state of an instance of the class  $\mathcal{M}_2$ ; in both cases the transition occurs with probability one and zero reward is collected.
Furthermore, assume that the reward function given over the MDPs in  $\mathcal{M}_1$  is shifted such that the value of a policy that takes  $a = 1$  in  $s_1$  and then plays optimally is 1 and that the reward functions on  $\mathcal{M}_2$  is shifted such that the value of a policy which takes  $a = 2$  initially and then plays optimally is  $2\varepsilon$ .
In addition, assume that the dataset arises from a logging policy  $\pi_{\text{log}}$  which takes  $a = 2$  initially and then visits all  $(s, a, s')$  uniformly.
Such construction and dataset identify a sparsified MDP which coincide with  $\mathcal{M}_2$  with the addition of  $s_1$  (and its transition to  $\mathcal{M}_2$  with zero reward). Intuitively, a policy with value arbitrarily close to 1 must take action  $a = 1$  which leads to  $\mathcal{M}_1$ , which is the portion of the MDP that is unexplored in the dataset. In this case, unless the agent collects exponentially many trajectories in the online phase, the lower bound from [Xiao et al., 2022] implies that it is not possible to discover a policy with value close to 1 (e.g., larger than  $1/2$ ). On the other hand, our theorem 5.1 guarantees that  $\pi_{final}$  has a value at least  $\varepsilon$ , because  $\pi_{final}$  is  $\varepsilon$ -optimal on the sparsified MDP—i.e.,  $\varepsilon$ -optimal when restricted to an instance on  $\mathcal{M}_2$ —with high probability using at most  $\sim H^2 |\mathcal{S}|^2 |\mathcal{A}|/\varepsilon^2$  trajectories. This value is unimprovable given the lower bound of Jin et al. [2020b], which applies to the class  $\mathcal{M}_2$ .
For completeness, in the next sub-section we refine the lower bound of [Xiao et al., 2022] to handle mixture policies.
D Technical lemmas and proofs
Lemma D.1 (Lemma F.4 in [Dann et al., 2017]). Let  $\mathcal{F}_i$  for  $i = 1\ldots$  be a filtration and  $X_{1},\ldots X_{n}$  be a sequence of Bernoulli random variables with  $\mathbb{P}(X_i = 1\mid \mathcal{F}_{i - 1}) = P_i$  with  $P_{i}$  being  $\mathcal{F}_{i - 1}$ -measurable and  $X_{i}$  being  $\mathcal{F}_i$  measurable. It holds that
$$
\mathbb {P} \left(\exists n: \sum_ {t = 1} ^ {n} X _ {t} <   \sum_ {t = 1} ^ {n} P _ {t} / 2 - W\right) \leq e ^ {- W}.
$$
Lemma D.2. Let  $\mathcal{F}_i$  for  $i = 1\ldots$  be a filtration and  $X_{1},\ldots X_{n}$  be a sequence of Bernoulli random variables with  $\mathbb{P}(X_i = 1\mid \mathcal{F}_{i - 1}) = P_i$  with  $P_{i}$  being  $\mathcal{F}_{i - 1}$ -measurable and  $X_{i}$  being  $\mathcal{F}_i$  measurable. It holds that
$$
\mathbb {P} \left(\exists n: \sum_ {t = 1} ^ {n} X _ {t} > \sum_ {t = 1} ^ {n} 2 P _ {t} + W\right) \leq e ^ {- W}.
$$
Proof. Notice that  $\frac{1}{u^2}[\exp(u) - u - 1]$  is non-decreasing on  $\mathbb{R}$ , where at zero we continuously extend this function. For any  $t \in \mathbb{N}$ , since  $X_t - P_t \leq 1$ , we have  $\exp(X_t - P_t) - (X_t - P_t) - 1 \leq (X_t - P_t)^2(e - 2) \leq (X_t - P_t)^2$ . Taking expectation conditional on  $\mathcal{F}_{t-1}$  and noticing that  $P_t - X_t$  is a Martingale difference sequence w.r.t. the filtration  $\mathcal{F}_t$ , we have
$$
\mathbb {E} \left[ \exp \left(X _ {t} - P _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \leq 1 + \mathbb {E} \left[ \left(X _ {t} - P _ {t}\right) ^ {2} \mid \mathcal {F} _ {t - 1} \right] \leq \exp \left[ \mathbb {E} \left[ \left(X _ {t} - P _ {t}\right) ^ {2} \mid \mathcal {F} _ {t - 1} \right] \right] \leq \exp \left(P _ {t}\right),
$$
where the last inequality comes from the fact that conditional on  $\mathcal{F}_{t - 1}$ ,  $X_{t}$  is a Bernoulli random variable. We define  $M_{n} \coloneqq \exp \left[\sum_{t = 1}^{n}(X_{t} - 2P_{t})\right]$ , which is a supermartingale from our derivation above. We define now the stopping time  $\tau = \min \left\{t \in \mathbb{N}: M_{t} > e^{W}\right\}$  and the sequence  $\tau_{n} = \min \left\{t \in \mathbb{N}: M_{t} > e^{W} \vee t \geq n\right\}$ . Applying the convergence theorem for nonnegative supermartingales (Theorem 4.2.12 in [Durrett, 2019]), we get that  $\lim_{t \to \infty} M_{t}$  is well-defined almost surely. Therefore,  $M_{\tau}$  is well-defined even when  $\tau = \infty$ . By the optional stopping theorem for non-negative supermartingales (Theorem 4.8.4 in [Durrett, 2019], we have  $\mathbb{E}[M_{\tau_n}] \leq \mathbb{E}[M_0] \leq 1$  for all  $n$  and applying Fatou's lemma, we obtain  $\mathbb{E}[M_{\tau}] = \mathbb{E}[\lim_{n \to \infty} M_{\tau_n}] \leq \lim \inf_{n \to \infty} \mathbb{E}[M_{\tau_n}] \leq 1$ . Using Markov's inequality, we can finally bound
$$
\mathbb {P} \left(\exists n: \sum_ {t = 1} ^ {n} X _ {t} > 2 \sum_ {t = 1} ^ {n} P _ {t} + W\right) > \mathbb {P} (\tau <   \infty) \leq \mathbb {P} \left(M _ {\tau} > e ^ {W}\right) \leq e ^ {- W} \mathbb {E} \left[ M _ {\tau} \right] \leq e ^ {- W}.
$$
Lemma D.3 (Empirical Bernstein Inequality, Theorem 11 in [Maurer and Pontil, 2009]). Let  $n \geq 2$  and  $x_{1}, \dots, x_{n}$  be i.i.d random variables such that  $|x_{i}| \leq A$  with probability 1. Let  $\bar{x} = \frac{1}{n}\sum_{i=1}^{n}x_{i}$ , and  $\widehat{V}_{n} = \frac{1}{n}\sum_{i=1}^{n}(x_{i} - \bar{x})^{2}$ , then with probability  $1 - \delta$  we have
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} - \mathbb {E} [ x ] \right| \leq \sqrt {\frac {2 \widehat {V} _ {n} \log (2 / \delta)}{n}} + \frac {1 4 A}{3 n} \log (2 / \delta)
$$