SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma A.16. There exists a universal constant  $C \geq 1$  such that under the high probability event  $\mathcal{E}$ , when  $3K_{ucb} \geq K_{de}$ , we have for any  $(h, s, a) \in [H] \times S \times \mathcal{A}$ , it holds that
$$
X _ {h} (s, a) \leq C H \frac {K _ {u c b}}{K _ {d e}} \phi \left(n ^ {K _ {u c b}} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot \mid s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right).
$$
Proof. Here, the universal constant may vary from line to line. Under  $\mathcal{E}$ , we have
$$
\begin{array}{l} X _ {h} (s, a) \\ \leq C H \phi (m (s, a)) + \left(1 + \frac {1}{H}\right) \widetilde {\mathbb {P}} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} X _ {h + 1} (\cdot , a ^ {\prime})\right). (definition) \\ \leq C H \min  \left\{1, \phi (m (s, a)) \right\} + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) \quad (Lemma A.15) \\ \leq C H \phi \left(K _ {d e} \sum_ {h \in [ H ]} w _ {h} ^ {\text {m i x}} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) (Lemma A.18) \\ = C H \phi \left(\frac {K _ {d e}}{K _ {u c b}} \sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h \in [ H ]} d _ {\pi^ {k}, h} ^ {\dagger} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) (definition) \\ \leq C H \phi \left(\frac {K _ {d e}}{3 K _ {u c b}} \sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) (Lemma A.22 and A.17) \\ \leq C H \frac {K _ {u c b}}{K _ {d e}} \phi \left(\sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} \left(\cdot , a ^ {\prime}\right) \right\}\right) (Lemma A.17) \\ \end{array}
$$
Since  $X_{h}(s,a) \leq (H - h) \leq CH\frac{K_{ucb}}{K_{de}}$ , we can modify the last display as
$$
\begin{array}{l} X _ {h} (s, a) \leq C H \frac {K _ {u c b}}{K _ {d e}} \min  \left\{1, \phi \left(\sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a)\right) \right\} + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) \\ \leq C H \frac {K _ {u c b}}{K _ {d e}} \phi \left(n ^ {K _ {u c b}} (s, a)\right) + \left(1 + \frac {2}{H}\right) \mathbb {P} ^ {\dagger} (\cdot | s, a) ^ {\top} \left(\max  _ {a ^ {\prime}} \left\{X _ {h + 1} (\cdot , a ^ {\prime}) \right\}\right) \tag {Lemma A.19} \\ \end{array}
$$
Therefore, we conclude.
A.3.4 Proof for lemma A.17, lemma A.18, lemma A.19, lemma A.20 (properties of bonus function)
For our bonus function, we have the following basic property.
Lemma A.17.  $\phi (x)$  is non-increasing when  $x > 0$ . For any  $\alpha \leq 1$ , we have  $\phi (\alpha x)\leq \frac{1}{\alpha}\phi (x)$ .
Proof. We define  $f(x) \coloneqq \frac{1}{x} \left[ C + D \log \left( e \left( 1 + \frac{x}{D} \right) \right) \right]$ , where  $C, D \geq 1$ . Then, for  $x > 0$ ,
$$
f ^ {\prime} (x) = - \frac {C + D \log \left(e \left(1 + \frac {x}{D}\right)\right)}{x ^ {2}} + \frac {D}{x (D + x)} \leq - \frac {C + D \log \left(1 + \frac {x}{D}\right)}{x ^ {2}} \leq 0.
$$
Taking  $C = \log (6H|\mathcal{S}||\mathcal{A}|\delta)$  and  $D = |\mathcal{S}|$ , we conclude the first claim. For the second claim, it is trivial since the logarithm function is increasing.
Lemma A.18. Under  $\mathcal{E}$ , we have for any  $(s, a) \in S \times \mathcal{A}$ ,
$$
\min  \left\{1, \phi (m (s, a)) \right\} \leq 4 \phi \left(K _ {d e} \sum_ {h = 1} ^ {H} w _ {h} ^ {m i x} (s, a)\right).
$$
Proof. For fixed  $(s,a)\in \mathcal{S}\times \mathcal{A}$ , when  $H\ln (6H|\mathcal{S}||\mathcal{A}| / \delta)\leq \frac{1}{4}\left(K_{de}\sum_{h\in [H]}w_h^{mix}(s,a)\right)$ , we know that  $\mathcal{E}^5$  implies  $m(s,a)\geq \frac{1}{4}\sum_{h\in [H]}K_{de}w_h^{mix}(s,a)$ , From Lemma A.17, we know
$$
\phi \left(m (s, a)\right) \leq \phi \left(\frac {1}{4} \sum_ {h \in [ H ]} K _ {d e} w _ {h} ^ {m i x} (s, a)\right) \leq 4 \phi \left(\sum_ {h \in [ H ]} K _ {d e} w _ {h} ^ {m i x} (s, a)\right).
$$
When  $H \ln (6H|\mathcal{S}||\mathcal{A}| / \delta) > \frac{1}{4}\left(K_{de}\sum_{h\in [H]}w_h^{mix}(s,a)\right)$ , simple algebra shows that
$$
\min  \left\{1, \phi (m (s, a)) \right\} \leq 1 \leq \frac {4 H \ln (6 H | \mathcal {S} | | \mathcal {A} | / \delta)}{K _ {d e} \sum_ {h \in [ H ]} w _ {h} ^ {m i x} (s , a)} \leq 4 \phi \left(K _ {d e} \sum_ {h \in [ H ]} w _ {h} ^ {m i x} (s, a)\right).
$$
Lemma A.19. Under  $\mathcal{E}$ , we have for any  $(s, a) \in S \times \mathcal{A}$ ,
$$
\min  \left\{1, \phi \left(\sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h = 1} ^ {H} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a)\right) \right\} \leq 4 \phi \left(n ^ {K _ {u c b}} (s, a)\right).
$$
Proof. We consider under the event  $\mathcal{E}^3$ , it holds that for any  $(s, a) \in S \times \mathcal{A}$ ,
$$
\sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h = 1} ^ {H} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a) \geq \frac {1}{2} n ^ {K _ {u c b}} (s, a) - \frac {1}{2} H \ln \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right).
$$
If  $H\ln \left(6H|\mathcal{S}||\mathcal{A}| / \delta\right)\leq \frac{1}{2} n^{K_{ucb}}(s,a)$ , then we have  $\sum_{k = 1}^{K_{ucb}}\sum_{h = 1}^{H}\widehat{d}_{\pi^k,h}^\dagger (s,a)\geq \frac{1}{4} n^{K_{ucb}}(s,a)$  which implies
$$
\phi \left(\sum_ {k = 1} ^ {K _ {u c b}} \sum_ {h = 1} ^ {H} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a)\right) \leq \phi \left(\frac {1}{4} n ^ {K _ {u c b}} (s, a)\right) \leq 4 \phi \left(n ^ {K _ {u c b}} (s, a)\right).
$$
Otherwise, if  $H \ln (6H|\mathcal{S}||\mathcal{A}| / \delta) > \frac{1}{2} n^{K_{ucb}}(s,a)$ , then we have
$$
\min  \left\{1, \phi \left(\sum_ {k = 1} ^ {K _ {u c b}} \widehat {d} _ {\pi^ {k}, h} ^ {\dagger} (s, a)\right) \right\} \leq 1 \leq \frac {2 H \ln \left(6 H | \mathcal {S} | | \mathcal {A} | / \delta\right)}{n ^ {K _ {u c b}} (s , a)} \leq 4 \phi \left(n ^ {K _ {u c b}} (s, a)\right).
$$
Combining the two cases above, we conclude.
Lemma A.20. Under  $\mathcal{E}$ , we have for any  $k\in [K_{ucb}]$  and any  $(s,a)\in S\times \mathcal{A}$
$$
\begin{array}{l} \min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} \\ \leq \frac {4 H}{\max \left\{1 , \sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i} , h} ^ {\dagger} (s , a) \right\}} \left[ \log \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right) + | \mathcal {S} | \log \left(e \left(1 + \frac {\sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i} , h} ^ {\dagger} (s , a)}{| \mathcal {S} |}\right)\right) \right]. \\ \end{array}
$$
Proof. Under  $\mathcal{E}^2$ , it holds that
$$
n ^ {k} (s, a) \geq \frac {1}{2} \sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a) - H \ln \left(\frac {6 H | \mathcal {S} | | \mathcal {A} |}{\delta}\right).
$$
If  $H$ $\ln (6H|\mathcal{S}||\mathcal{A}| / \delta)\leq \frac{1}{4}\sum_{i <   k}\sum_{h\in [H]}\widehat{d}_{\pi^i,h}^\dagger (s,a)$  then  $n^k (s,a)\geq \frac{1}{4}\sum_{i <   k}\sum_{h\in [H]}\widehat{d}_{\pi^i,h}^\dagger (s,a)$  and hence,
$$
\min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} \leq \phi \left(n ^ {k} (s, a)\right) \leq \phi \left(\frac {1}{4} \sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a)\right) \leq 4 \phi \left(\sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i}, h} ^ {\dagger} (s, a)\right).
$$
This result equals to the right hand side in the lemma, because  $\sum_{i < k} \sum_{h \in [H]} \widehat{d}_{\pi^i, h}^\dagger(s, a) \geq 4H \ln (6H|\mathcal{S}||\mathcal{A}|/\delta) \geq 1$  (so taking maximum does not change anything). Otherwise, if  $H \ln (6H|\mathcal{S}||\mathcal{A}|/\delta) > \frac{1}{4} \sum_{i < k} \sum_{h \in [H]} \widehat{d}_{\pi^i, h}^\dagger(s, a)$ , then
$$
\min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} \leq 1 \leq \frac {4 H \ln \left(H | \mathcal {S} | | \mathcal {A} | / \delta^ {\prime}\right)}{\sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i} , h} ^ {\dagger} (s , a)}.
$$
Since  $1 \leq 4H \ln (6H|\mathcal{S}||\mathcal{A}| / \delta)$ , we have
$$
\min  \left\{1, \phi \left(n ^ {k} (s, a)\right) \right\} \leq 1 \leq \frac {4 H \ln \left(H | \mathcal {S} | | \mathcal {A} | / \delta^ {\prime}\right)}{\max  \left\{1 , \sum_ {i <   k} \sum_ {h \in [ H ]} \widehat {d} _ {\pi^ {i} , h} ^ {\dagger} (s , a) \right\}} \leq \mathrm {R H S}
$$
The last inequality comes from simple algebra. Therefore, we conclude.
A.3.5 Proof for lemma A.21 and lemma A.22 (properties of empirical sparsified MDP)
In this section, we state two important properties of the empirical sparsified MDP and prove them. We remark that we do not include  $\mathbb{P}\left(s^{\dagger} \mid s, a\right)$  in these two lemmas, since by definition  $s^{\dagger} \notin S$ .
Lemma A.21 (Multiplicative Accuracy). We set
$$
\Phi \geq 6 H ^ {2} \log \left(\frac {1 2 | \mathcal {S} | ^ {2} | \mathcal {A} |}{\delta}\right).
$$
Then, when  $H \geq 2$ , under  $\mathcal{E}^P$ , we have for any  $(s, a, s') \in S \times \mathcal{A} \times S$ ,
$$
\left(1 - \frac {1}{H}\right) \widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \leq \mathbb {P} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \leq \left(1 + \frac {1}{H}\right) \widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right).
$$
Proof. For  $N(s, a, s') < \Phi$ , both sides are zero. For  $N(s, a, s') \geq \Phi$ , recall  $\widehat{\mathbb{P}}^{\dagger}(s' \mid s, a) = \frac{N(s, a, s')}{N(s, a)}$ , then from lemma A.10, with probability at least  $1 - \delta$ ,
(lemma A.10 and definition of  $\mathcal{E}^P$ )
$$
\begin{array}{l} \left| \widehat {\mathbb {P}} ^ {\dagger} (s ^ {\prime} \mid s, a) - \mathbb {P} ^ {\dagger} (s ^ {\prime} \mid s, a) \right| \\ \leq \sqrt {\frac {2 \widehat {\mathbb {P}} ^ {\dagger} (s ^ {\prime} \mid s , a)}{N (s , a)} \log \left(\frac {1 2 | \mathcal {S} | ^ {2} | \mathcal {A} |}{\delta}\right)} + \frac {1 4}{3 N (s , a)} \log \left(\frac {1 2 | \mathcal {S} | ^ {2} | \mathcal {A} |}{\delta}\right) \\ = \left[ \sqrt {\frac {2}{N (s , a , s ^ {\prime})} \log \left(\frac {1 2 | \mathcal {S} | ^ {2} | \mathcal {A} |}{\delta}\right)} + \frac {1 4}{3 N (s , a , s ^ {\prime})} \log \left(\frac {1 2 | \mathcal {S} | ^ {2} | \mathcal {A} |}{\delta}\right) \right] \cdot \widehat {\mathbb {P}} ^ {\dagger} (s ^ {\prime} \mid s, a) \\ (\widehat {\mathbb {P}} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) = \frac {N (s , a , s ^ {\prime})}{N (s , a)}) \\ \leq \left[ \sqrt {\frac {1}{3 H ^ {2}}} + \frac {7}{9 H ^ {2}} \right] \cdot \widehat {\mathbb {P}} ^ {\dagger} (s ^ {\prime} \mid s, a) \leq \frac {\widehat {\mathbb {P}} ^ {\dagger} (s ^ {\prime} \mid s , a)}{H}, \\ \end{array}
$$
where the second line comes from the lower bound on  $\Phi$ . We conclude.
Lemma A.22 (Bound on Ratios of Visitation Probability). For any deterministic policy  $\pi$  and any  $(h,s,a)\in [H]\times S\times \mathcal{A}$ , it holds that
$$
\frac {1}{4} d _ {\pi , h} ^ {\dagger} (s, a) \leq \widehat {d} _ {\pi , h} ^ {\dagger} (s, a) \leq 3 d _ {\pi , h} ^ {\dagger} (s, a).
$$
Here, recall that we denote  $d_{\pi,h}^{\dagger}(s,a)$  and  $\widehat{d}_{\pi,h}^{\dagger}(s,a)$  as the occupancy measure of  $(s,a)$  at stage  $h$  under policy  $\pi$ , on  $\mathbb{P}^{\dagger}$  (the transition dynamics in the sparsified MDP) and  $\widehat{\mathbb{P}}^{\dagger}$  (the transition dynamics in the empirical sparsified MDP) respectively.
We remark that for  $s^{\dagger} \notin S$  the inequality does not necessarily hold.
Proof. We denote  $T_{h,s,a}$  as all truncated trajectories  $(s_1, a_1, s_2, a_2, \dots, s_h, a_h)$  up to stage  $h$  such that  $(s_h, a_h) = (s, a)$ . Notice that if  $\tau_h = (s_1, a_1, s_2, a_2, \dots, s_h, a_h) \in T_{h,s,a}$ , then it holds that  $s_i \neq s^\dagger$  for  $1 \leq i \leq h - 1$ . We denote  $\mathbb{P}(\cdot; \mathbb{P}', \pi)$  as the probability under a specific transition dynamics  $\mathbb{P}'$  and policy  $\pi$ . For any fixed  $(h, s, a) \in [H] \times S \times A$  and any fixed  $\tau \in T_{h,s,a}$ , we apply Lemma A.21 to get
$$
\begin{array}{l} \mathbb {P} \left[ \tau ; \widehat {\mathbb {P}} ^ {\dagger}, \pi \right] = \prod_ {i = 1} ^ {h} \pi_ {i} \left(a _ {i} \mid s _ {i}\right) \prod_ {i = 1} ^ {h - 1} \widehat {\mathbb {P}} ^ {\dagger} \left(s _ {i + 1} \mid s _ {i}, a _ {i}\right) \\ \leq \left(1 + \frac {1}{H}\right) ^ {H} \prod_ {i = 1} ^ {h} \pi_ {i} (a _ {i} \mid s _ {i}) \prod_ {i = 1} ^ {h - 1} \mathbb {P} ^ {\dagger} (s _ {i + 1} \mid s _ {i}, a _ {i}) \leq 3 \mathbb {P} [ \tau ; \mathbb {P} ^ {\dagger}, \pi ] \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} \left[ \tau ; \widehat {\mathbb {P}} ^ {\dagger}, \pi \right] = \prod_ {i = 1} ^ {h} \pi_ {i} \left(a _ {i} \mid s _ {i}\right) \prod_ {i = 1} ^ {h - 1} \widehat {\mathbb {P}} ^ {\dagger} \left(s _ {i + 1} \mid s _ {i}, a _ {i}\right) \\ \geq \left(1 - \frac {1}{H}\right) ^ {H} \prod_ {i = 1} ^ {h} \pi_ {i} \left(a _ {i} \mid s _ {i}\right) \prod_ {i = 1} ^ {h - 1} \mathbb {P} ^ {\dagger} \left(s _ {i + 1} \mid s _ {i}, a _ {i}\right) \geq \frac {1}{4} \mathbb {P} \left[ \tau ; \mathbb {P} ^ {\dagger}, \pi \right]. \\ \end{array}
$$
We conclude by rewriting the visiting probability as
$$
d _ {\pi , h} ^ {\dagger} (s, a) = \sum_ {\tau \in T _ {h, s, a}} \mathbb {P} \left[ \tau ; \mathbb {P} ^ {\dagger}, \pi \right]; \quad \widehat {d} _ {\pi , h} ^ {\dagger} (s, a) = \sum_ {\tau \in T _ {h, s, a}} \mathbb {P} \left[ \tau ; \widehat {\mathbb {P}} ^ {\dagger}, \pi \right].
$$
B Additional comparisons
B.1 Comparison with other comparator policy
Our main result compares the sub-optimality of the policy  $\pi_{final}$  against the optimal policy on the sparsified MDP. We can further derive the sub-optimality of our output with respect to any comparator policy on the original MDP  $\mathcal{M}$ . If we denote  $\pi_*,\pi_*^\dagger$  and  $\pi_{final}$  as the global optimal policy, the optimal policy on the sparsified MDP and the policy output by our algorithm, respectively, and denote  $\pi$  as the comparator policy, we have
$$
\begin{array}{l} V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi_ {\text {f i n a l}}\right) \\ \leq V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi\right) + \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi\right) - V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {*} ^ {\dagger}\right)} _ {\leq 0} \\ + \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {*} ^ {\dagger}\right) - V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {f i n a l}\right)} _ {\lesssim \varepsilon} + \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {f i n a l}\right) - V _ {1} \left(s _ {1} , \mathbb {P} , r , \pi_ {f i n a l}\right)} _ {\leq 0} \\ \lesssim \underbrace {V _ {1} \left(s _ {1} , \mathbb {P} , r , \pi_ {*}\right) - V _ {1} \left(s _ {1} , \mathbb {P} ^ {\dagger} , r , \pi_ {*}\right)} _ {\text {A p p r o x i m a t i o n E r r o r}} + \varepsilon . \tag {B.1} \\ \end{array}
$$
Here, the second term is non-positive from the definition of  $\pi_{*}^{\dagger}$ , the third term is upper bounded by  $\varepsilon$  due to our main theorem (Theorem 5.1), and the last term is non-positive from the definition of the sparsified MDP. Since the connectivity graph of the sparsified MDP is a sub-graph of the original MDP, for any policy, the policy value on the sparsified MDP must be no higher than that on the original MDP.
At a high level, the  $\varepsilon$  term in the last line of (B.1) represents the error from the finite online episodes, while the approximation error term  $V_{1}(s_{1},\mathbb{P},r,\pi) - V_{1}(s_{1},\mathbb{P}^{\dagger},r,\pi)$  measures the policy value difference of  $\pi$  on the original MDP and the sparsified one, representing the coverage quality of the offline dataset. If the dataset covers most of what  $\pi$  covers, then this gap should be small. When  $\pi$  is the global optimal policy  $\pi_{*}$ , this means the data should cover the state-action pairs where optimal policy covers. The approximation error here plays a similar role as the concentrability coefficient in the offline reinforcement learning.
B.2 Comparison with offline reinforcement learning
Our algorithm leverages some information from the offline dataset, so it is natural to ask how we expect that offline dataset to be, compared to traditional offline reinforcement learning does. In offline RL, we typically require the concentrability condition, namely good coverage for the offline dataset, in order to achieve a polynomial sample complexity. Specifically, if we assume the offline data are sampled by first sampling  $(s,a)$  i.i.d. from  $\mu$  and then sampling the subsequent state from the transition dynamics, then the concentrability condition says the following constant  $C^*$  is well-defined and finite.
$$
C ^ {*} := \sup  _ {(s, a)} \frac {d ^ {\pi_ {*}} (s , a)}{\mu (s , a)} <   \infty .
$$
The concentrability coefficient can be defined in several alternative ways, either for a set of policies or with respect to a single policy [Chen and Jiang, 2019, Zhan et al., 2022, Xie et al., 2021b, Zanette et al., 2021b]. Here, we follow the definition in [Xie et al., 2021b]. This means, the sampling distribution must covers the region where the global optimal policy covers, which is a very similar intuition obtained from our setting.
[Xie et al., 2021b] also gave optimal sample complexity (in terms of state-action pairs) for an offline RL algorithm is
$$
N = \widetilde {O} \left(\frac {C ^ {*} H ^ {3} | \mathcal {S} |}{\varepsilon^ {2}} + \frac {C ^ {*} H ^ {5 . 5} | \mathcal {S} |}{\varepsilon}\right),
$$
which is minimax optimal up to logarithm terms and higher order terms. Similar sample complexity were also given in several literature [Yin and Wang, 2020, Yin et al., 2020, Xie and Jiang, 2020b].
Uniform data distribution For simplicity, we first assume  $\mu$  to be uniform on all state-action pairs and the reward function to be given. Consider we have  $N$  state-action pairs in the offline data, which are sampled i.i.d. from the distribution  $\mu$ . Notice that here, the global optimal policy  $\pi_{*}$  still differs from the optimum on the sparsified MDP  $\pi_{*}^{\dagger}$ , since even if we get enough samples from each  $(s,a)$  pairs, we might not get enough samples for every  $(s,a,s')$  and hence, not all  $(s,a,s')$  will be included in the set of known tuples.
Concretely, if we consider the case when we sample each state-action pair for  $N / (|\mathcal{S}| |\mathcal{A}|)$  times and simply treat the transition frequency as the true probability, then for any  $N(s,a,s') < \Phi$ , it holds that  $\mathbb{P}(s' \mid s,a) = \frac{N(s,a,s')}{N(s,a)} = \frac{N(s,a,s')|\mathcal{S}||\mathcal{A}|}{N} \leq \frac{\Phi|\mathcal{S}||\mathcal{A}|}{N}$ . So for any any  $N(s,a,s') \geq \Phi$ , we know  $\mathbb{P}(s' \mid s,a) = \mathbb{P}^{\dagger}(s' \mid s,a)$ ; while for any  $N(s,a,s') < \Phi$ , we have  $\mathbb{P}(s' \mid s,a) \leq \frac{\Phi|\mathcal{S}||\mathcal{A}|}{N}$  and  $\mathbb{P}^{\dagger}(s' \mid s,a) = 0$ . Therefore, we have
$$
\left| \mathbb {P} \left(s ^ {\prime} \mid s, a\right) - \mathbb {P} ^ {\dagger} \left(s ^ {\prime} \mid s, a\right) \right| \leq \frac {\Phi \left| S \right| \left| A \right|}{N}
$$
From the value difference lemma (lemma D.11), we can upper bound the approximation error by
$$
\begin{array}{l} V _ {1} \left(s _ {1}, \mathbb {P}, r, \pi\right) - V _ {1} \left(s _ {1}, \mathbb {P} ^ {\dagger}, r, \pi\right) \\ = \mathbb {E} _ {\mathbb {P}, \pi} \left[ \sum_ {h = 1} ^ {H} \sum_ {s _ {h + 1}} \left(\mathbb {P} ^ {\dagger} \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right) - \mathbb {P} \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right)\right) \cdot V _ {h} \left(s _ {h + 1}, \mathbb {P}, r, \pi\right) \Bigg | s _ {h} = s \right] \tag {lemma D.11} \\ \leq \mathbb {E} _ {\mathbb {P}, \pi} \left[ \sum_ {h = 1} ^ {H} \sum_ {s _ {h + 1}} \frac {\Phi | S | | \mathcal {A} |}{N} \cdot H \mid s _ {h} = s \right] \quad (\text {t h e v a l u e f u n c t i o n i s u p p e r b o u n d e d b y} H) \\ \leq \frac {\Phi H ^ {2} | \mathcal {S} | ^ {2} | \mathcal {A} |}{N} \quad (\text {s u m m a t i o n o v e r h \in [ H ] a n d s _ {h + 1} \in \mathcal {S}}) \\ \asymp \widetilde {O} \left(\frac {H ^ {4} | \mathcal {S} | ^ {2} | \mathcal {A} |}{N}\right). \quad (\text {d e f i n i t i o n o f} \Phi \text {i n (5 . 1)}) \\ \end{array}
$$
Therefore, to get an  $\varepsilon$ -optimal policy compared to the global optimal one, we need the number of state-action pairs in the initial offline dataset  $\mathcal{D}$  to be
$$
N = \widetilde {O} \left(\frac {H ^ {4} | \mathcal {S} | ^ {2} | \mathcal {A} |}{\varepsilon}\right).
$$
From the theorem 5.1, the offline data size we need here is actually significantly smaller than what we need for an offline algorithm. As long as  $\sup_{(s,a)}d^{\pi_*(s,a)}$  is not too small, for instance, larger than  $H^{-1.5}$ , then we shave off the whole  $1 / \varepsilon^2$  term. The order of offline sample complexity here is actually  $O(1 / \varepsilon)$  instead of  $O(1 / \varepsilon^2)$  typical in offline RL, and this is significantly smaller in small  $\varepsilon$  regime. To compensate the smaller offline sample size, actually we need more online sample to obtain an globally  $\varepsilon$ -optimal policy, and we summarize the general requirement for offline and online sample size in corollary 5.2.
Non-uniform data distribution Assume the data generating distribution  $\mu$  is not uniform but still supported on all  $(s,a)$  pairs such that  $d^{\pi_*(s,a)} > 0$ , so that the concentrability coefficient in offline RL is still well defined. We simply consider the case when each state-action pair  $(s,a)$  is sampled by  $N\mu(s,a)$  times and treat the transition frequency as the true underlying probability. Then, following a very similar argument as in the last paragraph, the number of state-action pairs needed in the initial offline dataset in order to extract an  $\varepsilon$ -globally optimal policy is
$$
N = \widetilde {O} \left(\frac {H ^ {4} | \mathcal {S} |}{\varepsilon} \sum_ {s, a} \frac {d _ {\pi_ {*}} ^ {\prime} (s , a)}{\mu (s , a)}\right).
$$
Here, the quantity
$$
C ^ {\dagger} := \sum_ {s, a} \frac {d ^ {\pi_ {*}} (s , a)}{\mu (s , a)}
$$
plays a similar role of classical concentrability coefficient and also measures the distribution shift between two policies. In the worst case, this coefficient can be  $|\mathcal{S}| |\mathcal{A}| C^*$ , resulting in an extra  $|\mathcal{S}| |\mathcal{A}|$  factor compared to the optimal offline sample complexity. However, we still shave off the entire  $1 / \varepsilon^2$  term and also shave off  $H^{1.5}$  in the  $1 / \varepsilon$  term.
Partial coverage data Under partial coverage, we expect the output policy  $\pi_{final}$  to be competitive with the value of the best policy supported in the region covered by the offline dataset. In such case, theorem 5.1 provides guarantees with the best comparator policy on the sparsified MDP  $\mathcal{M}^{\dagger}$ . In order to gain further intuition, it is best to 'translate' such guarantees into guarantees on  $\mathcal{M}$ .