SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\langle d _ {j} ^ {\pi^ {\star}}, V _ {j} ^ {\star} - V _ {j} ^ {\widehat {\pi}} \right\rangle \leq \gamma_ {1} + \gamma_ {2} + \frac {8 H ^ {2} S A}{K ^ {\text {o n}}} + \frac {5 3 c _ {\xi} H ^ {6} S ^ {4} A ^ {4}}{N} \log \frac {K}{\delta}, \tag {69}
$$
$$
\begin{array}{l} \gamma_{1}:= \sum_{h:h\geq j}\sum_{s}\max_{a:(s,a)\notin \mathcal{T}_{h}}\Bigg\{2\sqrt{2d_{h}^{\pi^{\star}}(s,a)C^{\star}(\sigma)\widehat{d}_{h}^{\mathrm{off}}(s,a)}\min \left\{\sqrt{\frac{2c_{\mathrm{b}}\log\frac{K}{\delta}}{N_{h}^{\mathrm{trim}}(s,a)}\mathsf{Var}_{P_{h}(\cdot |s,a)}\big(\widehat{V}_{h + 1}\big)} + \frac{4c_{\mathrm{b}}H\log\frac{K}{\delta}}{N_{h}^{\mathrm{trim}}(s,a)},  H\right\} \Bigg\}; \\ \gamma_ {2} := 4 H \sum_ {h: h \geq j} \sum_ {(s, a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {\star}} (s, a) \sqrt {\frac {c _ {\mathrm {b}} \log \frac {K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a) + 1}}. \\ \end{array}
$$
This leaves us with two terms to bound, which we shall accomplish separately in the ensuing two steps.
E.3.2 Step 3.2: controlling  $\gamma_{1}$  in (69)
Regarding the first term  $\gamma_{1}$  on the right-hand side of (69), let us first define the set  $\mathcal{I}_h$  as follows:
$$
\mathcal {I} _ {h} := \left\{(s, a): \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s, a) \right] \geq \frac {\xi}{N} \right\}, \tag {70}
$$
where we remind the reader that  $\xi = c_{\xi}H^{3}S^{3}A^{3}\log \frac{HSA}{\delta}$  for some constant  $c_{\xi} > 0$ . Armed with this set, we can deduce that
$$
\begin{array}{l} \sum_ {h: h \geq j} \sum_ {s} \max _ {a: (s, a) \notin \mathcal {I} _ {h} \cup \mathcal {T} _ {h}} \sqrt {2 d _ {h} ^ {\pi^ {\star}} (s , a) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)} \\ \leq \sum_ {h: h \geq j} \sum_ {s} 2 \max  _ {a: (s, a) \notin \mathcal {I} _ {h}} C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\text {o f f}} (s, a) \\ \end{array}
$$
$$
\begin{array}{l} \leq 2 C ^ {\star} (\sigma) \left(\frac {1}{K ^ {\text {o n}} H} + \frac {\xi}{N}\right) \sum_ {h: h \geq j} \sum_ {s} \max  _ {a} \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K ^ {\text {o n}} H} + \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \\ \leq 2 1 8 H S C ^ {\star} (\sigma) \left(\frac {\xi}{N} + \frac {1}{K ^ {\text {o n}} H}\right), \\ \end{array}
$$
where the first inequality arises from (62), the penultimate line utilizes the definition (70) of  $\mathcal{I}_h$ , and the last line comes from (67). This in turn allows us to upper bound  $\gamma_{1}$  as follows:
$$
\gamma_ {1} \leq \sum_ {h: h \geq j} \sum_ {s} 2 \max _ {a: (s, a) \in \mathcal {I} _ {h}} \sqrt {2 d _ {h} ^ {\pi^ {*}} (s , a) C ^ {*} (\sigma) \widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)} \min \left\{\sqrt {\frac {2 c _ {\mathrm {b}} \log \frac {H K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a)} \mathsf {V a r} _ {P _ {h} (\cdot | s , a)} \big (\widehat {V} _ {h + 1} \big)} + \frac {4 c _ {\mathrm {b}} H \log \frac {K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a)}, H \right\}
$$
$$
+ \sum_ {h: h \geq j} \sum_ {s} 2 \max _ {a: (s, a) \notin \mathcal {I} _ {h} \cup \mathcal {T} _ {h}} \sqrt {2 d _ {h} ^ {\pi^ {\star}} (s , a) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)} \cdot H
$$
$$
\begin{array}{l} \leq \sum_ {h: h \geq j} \sum_ {s} 2 \max _ {a: (s, a) \in \mathcal {I} _ {h}} \sqrt {2 d _ {h} ^ {\pi^ {\star}} (s , a) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)} \min  \left\{\sqrt {\frac {2 c _ {\mathrm {b}} \log \frac {H K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a)}} \mathsf {V a r} _ {P _ {h} (\cdot | s, a)} (\widehat {V} _ {h + 1}) + \frac {4 c _ {\mathrm {b}} H \log \frac {K}{\delta}}{N _ {h} ^ {\mathrm {t r i m}} (s , a)}, H \right\} \\ + 4 3 6 H ^ {2} S C ^ {\star} (\sigma) \left(\frac {\xi}{N} + \frac {1}{K ^ {\text {o n}} H}\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq 16c_{\mathrm{b}}\sum_{h:h\geq j}\sum_{s}\max_{a:(s,a)\in \mathcal{I}_{h}}\sqrt{2d_{h}^{\pi^{\star}}(s,a)C^{\star}(\sigma)\widehat{d}_{h}^{\text{off}}(s,a)}\sqrt{\frac{\operatorname{Var}_{P_{h}(\cdot|s,a)}(\widehat{V}_{h + 1}) + H}{N_{h}^{\text{trim}}(s,a) + 1 / H}\log^{2}\frac{K}{\delta}} \\ + 4 3 6 H ^ {2} S C ^ {\star} (\sigma) \left(\frac {\xi}{N} + \frac {1}{K ^ {\text {o n}} H}\right), \tag {71} \\ \end{array}
$$
where the last line makes use of the elementary fact that  $\min \left\{\frac{x}{y}, \frac{u}{w}\right\} \leq \frac{x + u}{y + w}$  for any  $x, y, u, w > 0$ .
In addition, note that for any  $s$  obeying  $\mathbb{E}_{\pi \sim \mu^{\mathrm{imitate}}}\left[d_h^\pi (s)\right] \geq \xi / N$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ N _ {h} ^ {\mathrm {a u x}} (s) \right] = \frac {1}{4} K ^ {\mathrm {o f f}} d _ {h} ^ {\mathrm {o f f}} (s) + \frac {1}{6} K ^ {\mathrm {o n}} \mathbb {E} _ {\pi \sim \mu^ {\mathrm {i m i t a t e}}} \left[ d _ {h} ^ {\pi} (s) \right] + \frac {1}{6} K ^ {\mathrm {o n}} \mathbb {E} _ {\pi \sim \mu^ {\mathrm {e x p l o r e}}} \left[ d _ {h} ^ {\pi} (s) \right] \\ \geq \frac {1}{6} K ^ {\mathrm {o n}} \mathbb {E} _ {\pi \sim \mu^ {\mathrm {i m i t a t e}}} \left[ d _ {h} ^ {\pi} (s) \right] \geq \frac {1}{6} K ^ {\mathrm {o n}} \cdot \frac {\xi}{N} = \frac {1}{2} c _ {\xi} H ^ {4} S ^ {3} A ^ {3} \log \frac {H S A}{\delta}, \\ \end{array}
$$
where the last line invokes the definition of  $\mathcal{I}_h$  and the choice  $NH = \frac{1}{3} K^{\mathrm{on}}$ . It can then be straightforwardly justified using elementary concentration inequalities (see, e.g., Alon and Spencer (2016, Appendix A.1)) that: with probability exceeding  $1 - \delta / 10$ ,
$$
N _ {h} ^ {\mathrm {a u x}} (s) \geq \frac {1}{2} \mathbb {E} \left[ N _ {h} ^ {\mathrm {a u x}} (s) \right] \geq \frac {1}{4} c _ {\xi} H ^ {4} S ^ {3} A ^ {3} \log \frac {H S A}{\delta}
$$
holds simultaneously for all  $(s,h)\in S\times [H]$  , and as a result,
$$
N _ {h} ^ {\mathrm {t r i m}} (s) \geq N _ {h} ^ {\mathrm {a u x}} (s) - 1 0 \sqrt {N _ {h} ^ {\mathrm {a u x}} (s) \log \frac {H S}{\delta}} \geq \frac {1}{2} N _ {h} ^ {\mathrm {a u x}} (s) \geq \frac {1}{4} \mathbb {E} \left[ N _ {h} ^ {\mathrm {a u x}} (s) \right] \geq \frac {1}{2 4} K ^ {\mathrm {o n}} \mathbb {E} _ {\pi \sim \mu^ {\mathrm {i m i t a t e}}} \left[ d _ {h} ^ {\pi} (s) \right].
$$
Moreover, for any  $(s,a)\in \mathcal{I}_h$  (cf. (70)), one can invoke Lemma 2 to obtain
$$
\mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ d _ {h} ^ {\pi} (s) \right] \geq \frac {1}{3} \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ \widehat {d _ {h} ^ {\pi}} (s) \right] \geq \frac {\xi}{3 N}.
$$
Applying the same concentration of measurement argument as above further reveals that:
$$
N _ {h} ^ {\operatorname {t r i m}} (s, a) \geq \frac {1}{2 4} K ^ {\operatorname {o n}} \mathbb {E} _ {\pi \sim \mu^ {\operatorname {i m i t a t e}}} \left[ d _ {h} ^ {\pi} (s, a) \right] \geq \frac {1}{7 2} K ^ {\operatorname {o n}} \mathbb {E} _ {\pi \sim \mu^ {\operatorname {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s, a) \right]
$$
any  $(s,a)\in \mathcal{I}_h$ . Substitution into (71) then gives
$$
\gamma_ {1} \leq 1 6 c _ {\mathrm {b}} \sum_ {h: h \geq j} \sum_ {s} \max  _ {a: (s, a) \in \mathcal {I} _ {h}} \sqrt {2 d _ {h} ^ {\pi^ {\star}} (s , a) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\text {o f f}} (s , a)} \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot | s , a)} \left(\widehat {V} _ {h + 1}\right) + H}{1 / H + \frac {1}{7 2} K ^ {\text {o n}} \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \log^ {2} \frac {K}{\delta}}
$$
$$
+ 4 3 6 H ^ {2} S C ^ {\star} (\sigma) \left(\frac {\xi}{N} + \frac {1}{K ^ {\text {o n}} H}\right). \tag {72}
$$
By virtue of the Cauchy-Schwarz inequality, we can further derive
$$
\begin{array}{l} \sum_ {h: h \geq j} \sum_ {s} \max  _ {a} \sqrt {d _ {h} ^ {\pi^ {\star}} (s , a) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\text {o f f}} (s , a)} \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot | s , a)} \left(\widehat {V} _ {h + 1}\right) + H}{1 / H + \frac {1}{7 2} K ^ {\text {o n}} \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}} \\ \leq \sqrt {\sum_ {h : h \geq j} \sum_ {s , a} d _ {h} ^ {\pi^ {\star}} (s , a) \left(\operatorname {V a r} _ {P _ {h} (\cdot | s , a)} \left(\widehat {V} _ {h + 1}\right) + H\right)} \cdot \sqrt {\sum_ {h : h \geq j} \sum_ {s} \max  _ {a} \frac {C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{1 / H + \frac {1}{7 2} K ^ {\text {o n}} \mathbb {E} _ {\pi \sim \mu^ {\text {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}}. \tag {73} \\ \end{array}
$$
To further control this term, we resort to the following lemma, whose proof is deferred to Section G.2.
Lemma 6. With probability at least  $1 - \delta / 6$ , we have, for all  $j \in [H]$
$$
\sum_ {h: h \geq j} \sum_ {s, a} d _ {h} ^ {\pi^ {\star}} (s, a) \mathsf {V a r} _ {P _ {h} (\cdot | s, a)} \left(\widehat {V} _ {h + 1}\right) \leq 5 H ^ {2},
$$
provided that
$$
K ^ {\text {o n}} \geq c _ {1 1} \left(H ^ {1 8} S ^ {1 4} A ^ {1 4} + H ^ {5} S ^ {4} A ^ {3} C ^ {\star} (\sigma)\right) \log^ {2} \frac {K}{\delta}
$$
$$
K ^ {\mathrm {o f f}} \geq c _ {1 1} H S \big (C ^ {\star} (\sigma) + A \big) \log \frac {K}{\delta}
$$
for some sufficiently large constant  $c_{11} > 0$ .
Putting Lemma 6 together with (67) and (73), we obtain
$$
\sum_ {h: h \geq j} \sum_ {s} \max  _ {a} \sqrt {d _ {h} ^ {\pi^ {\star}} (s , a) C ^ {\star} (\sigma) \widehat {d} _ {h} ^ {\text {o f f}} (s , a)} \sqrt {\frac {\operatorname {V a r} _ {P _ {h} (\cdot | s , a)} \left(\widehat {V} _ {h + 1}\right) + H}{1 / H + \frac {1}{7 2} K ^ {\text {o n}} \mathbb {E} _ {\pi \in \mu^ {\text {i m i t a t e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}} \lesssim \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma)}{K ^ {\text {o n}}}}. \tag {74}
$$
Substitution into (72) results in
$$
\gamma_ {1} \leq \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma) \log^ {2} \frac {K}{\delta}}{K ^ {\text {o n}}}} + H ^ {2} S C ^ {\star} (\sigma) \left(\frac {\xi}{N} + \frac {1}{K ^ {\text {o n}} H}\right). \tag {75}
$$
Akin to (72) and (75), we can also focus on the offline dataset and obtain
$$
\gamma_ {1} \lesssim \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma)}{K ^ {\text {o f f}}} \log^ {2} \frac {K}{\delta}} + H ^ {2} S C ^ {\star} (\sigma) \left(\frac {\xi}{N} + \frac {1}{K ^ {\text {o f f}} H}\right). \tag {76}
$$
Combine (75) and (76) to arrive at
$$
\begin{array}{l} \gamma_ {1} \lesssim \min \left\{\sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma)}{K ^ {\mathrm {o n}}} \log^ {2} \frac {K}{\delta}}, \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma)}{K ^ {\mathrm {o f f}}} \log^ {2} \frac {K}{\delta}} \right\} + H ^ {2} S C ^ {\star} (\sigma) \Big (\frac {\xi}{N} + \frac {1}{\min \{K ^ {\mathrm {o n}} , K ^ {\mathrm {o f f}} \} H} \Big) \\ \lesssim \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma)}{\operatorname* {m a x} \left\{K ^ {\text {o n}} , K ^ {\text {o f f}} \right\}} \log^ {2} \frac {K}{\delta}} + H ^ {2} S C ^ {\star} (\sigma) \Big (\frac {\xi}{N} + \frac {1}{\operatorname* {m i n} \left\{K ^ {\text {o n}} , K ^ {\text {o f f}} \right\} H} \Big) \\ \lesssim \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma)}{K ^ {\text {o n}} + K ^ {\text {o f f}}} \log^ {2} \frac {K}{\delta}} + H ^ {2} S C ^ {\star} (\sigma) \left(\frac {\xi}{N} + \frac {1}{\min  \left\{K ^ {\text {o n}} , K ^ {\text {o f f}} \right\} H}\right). \tag {77} \\ \end{array}
$$
E.3.3 Step 3.3: controlling  $\gamma_{2}$  in (69)
We now turn attention to the term  $\gamma_{2}$  on the right-hand side of (69). Akin to (72), we can deduce that
$$
\gamma_ {2} \leq 1 6 c _ {\mathrm {b}} H \sum_ {h: h \geq j} \sum_ {(s, a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {\star}} (s, a) \sqrt {\frac {\log \frac {K}{\delta}}{1 + \frac {1}{7 2} K ^ {\text {o n}} \mathbb {E} _ {\pi \sim \mu^ {\text {e x p l o r e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}} + 4 3 6 H ^ {2} S A \left(\frac {\xi}{N} + \frac {1}{K ^ {\text {o n}} H}\right). \tag {78}
$$
The Cauchy-Schwarz inequality then tells us that
$$
\begin{array}{l} \sum_ {h: h \geq j} \sum_ {(s, a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {\star}} (s, a) \sqrt {\frac {1}{1 + \frac {1}{7 2} K ^ {\mathrm {o n}} \mathbb {E} _ {\pi \sim \mu^ {\mathrm {e x p l o r e}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]}} \\ \leq \sqrt {\sum_ {h : h \geq j} \sum_ {(s , a) \in \mathcal {T} _ {h}} \frac {\widehat {d} _ {h} ^ {\pi^ {\star}} (s , a)}{1 + \frac {1}{7 2} K ^ {\mathrm {o n}} \mathbb {E} _ {\pi \sim \mu^ {\mathrm {e x p l o r e}}} [ \widehat {d} _ {h} ^ {\pi} (s , a) ]}} \cdot \sqrt {\sum_ {h : h \geq j} \sum_ {(s , a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {\star}} (s , a)} \\ \leq 6 \sqrt {\frac {2 H S A}{K ^ {\mathrm {o n}}}} \cdot \sqrt {\sum_ {h : h \geq j} \sum_ {(s , a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {\star}} (s , a)} \\ \leq 6 \sqrt {\frac {2 H S A (2 \widehat {\sigma} + H S \xi / N)}{K ^ {\text {o n}}}}, \\ \end{array}
$$
where  $\widehat{\sigma}$  is defined in (63). Here, the penultimate line invokes Lemma 3, and the last line is valid since (according to Lemma 1 and (63))
$$
\begin{array}{l} \sum_ {h: h \geq j} \sum_ {(s, a) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {\star}} (s, a) = \sum_ {h: h \geq j} \sum_ {s: (s, \pi^ {\star} (s)) \in \mathcal {T} _ {h}} \widehat {d} _ {h} ^ {\pi^ {\star}} (s, \pi^ {\star} (s)) \\ \leq 2 \sum_ {h: h \geq j} \sum_ {s: (s, \pi^ {\star} (s)) \in \mathcal {T} _ {h}} d _ {h} ^ {\pi^ {\star}} \left(s, \pi^ {\star} (s)\right) + \frac {H S \xi}{N} \\ \leq 2 \widehat {\sigma} + \frac {H S \xi}{N}. \\ \end{array}
$$
Substitution of the above inequality into (78) yields
$$
\gamma_ {2} \leq 9 6 c _ {\mathrm {b}} \sqrt {\frac {2 H ^ {3} S A (2 \widehat {\sigma} + H S \xi / N)}{K ^ {\mathrm {o n}}} \log \frac {H K}{\delta}} + 4 3 6 H ^ {2} S A \left(\frac {\xi}{N} + \frac {1}{K ^ {\mathrm {o n}} H}\right). \tag {79}
$$
E.3.4 Step 3.4: putting all pieces together
To finish up, combining (69),(77) and (79) reveals that: with probability at least  $1 - \delta$ , one has
$$
\begin{array}{l} V _ {1} ^ {\star} (\rho) - V ^ {\hat {\pi}} (\rho) = \left\langle d _ {1} ^ {\pi^ {\star}}, V _ {1} ^ {\star} - V ^ {\hat {\pi}} \right\rangle \\ \lesssim \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma) \log^ {2} \frac {K}{\delta}}{K ^ {\mathrm {o n}} + K ^ {\mathrm {o f f}}}} + \sqrt {\frac {H ^ {4} S A \sigma \log \frac {K}{\delta}}{K ^ {\mathrm {o n}}}} + \sqrt {\frac {H ^ {4} S ^ {2} A C ^ {\star} (\sigma) \log^ {2} \frac {K}{\delta}}{K ^ {\mathrm {o f f}} K ^ {\mathrm {o n}}}} \\ + \sqrt {\frac {H ^ {8} S ^ {6} A ^ {5} C ^ {\star} (\sigma) \log^ {2} \frac {K}{\delta}}{N K ^ {\text {o n}}}} + \sqrt {\frac {H ^ {4} S ^ {3} A ^ {2} C ^ {\star} (\sigma) \log \frac {K}{\delta}}{K K ^ {\text {o n}}}} \\ + \frac {H ^ {6} S ^ {4} A ^ {4} + H ^ {5} S ^ {4} A ^ {3} C ^ {\star} (\sigma)}{N} \log \frac {K}{\delta} + \frac {H ^ {2} S (C ^ {\star} (\sigma) + A)}{\min  \left\{K ^ {\text {o n}} , K ^ {\text {o f f}} \right\}} \\ \lesssim \sqrt {\frac {H ^ {3} S C ^ {\star} (\sigma) \log^ {2} \frac {K}{\delta}}{K ^ {\text {o n}} + K ^ {\text {o f f}}}} + \sqrt {\frac {H ^ {4} S A \sigma \log \frac {K}{\delta}}{K ^ {\text {o n}}}} + \frac {H ^ {6} S ^ {4} A ^ {4} + H ^ {5} S ^ {4} A ^ {3} C ^ {\star} (\sigma)}{K ^ {\text {o n}}} \log^ {2} \frac {K}{\delta} + \frac {H ^ {2} S (C ^ {\star} (\sigma) + A)}{K ^ {\text {o f f}}}, \tag {80} \\ \end{array}
$$
where the last inequality holds true as long as  $\min \{K^{\mathrm{off}}, K^{\mathrm{on}}\} \gtrsim HSA$ . Taking the right-hand side of (80) to be no larger than  $\varepsilon$ , we immediately establish Theorem 1 under the sample complexity assumption in this theorem.
F Proof for the stopping criterion and the iteration complexity for solving (20b)
Feasibility of the stopping rule (37). We first demonstrate that the stopping rule (37) can be satisfied by some mixed policy, namely,
$$
\min  _ {\mu \in \Delta (\Pi)} \sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \mathbb {E} _ {a \sim \pi_ {h} ^ {t + 1} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K H} + \mathbb {E} _ {\pi \sim \mu} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \right] \leq 1 0 8 S H. \tag {81}
$$
Towards this end, we focus attention on analyzing a specific choice of the mixed policy  $\mu^{\mathrm{off}}$  — the one that represents the mixed policy that generates the offline dataset. Making use of the definition (15) of  $\widehat{d}^{\mathrm{off}}$  gives
$$
\begin{array}{l} \widehat {d} _ {h} ^ {\text {o f f}} (s, a) = \frac {2 N _ {h} ^ {\text {o f f}} (s , a)}{K ^ {\text {o f f}}} \mathbb {1} \left(\frac {N _ {h} ^ {\text {o f f}} (s , a)}{K ^ {\text {o f f}}} \geq c _ {\text {o f f}} \left\{\frac {\log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K} \right\}\right) \\ \leq 3 d _ {h} ^ {\text {o f f}} (s, a) \mathbb {1} \left(\frac {3}{2} d _ {h} ^ {\text {o f f}} (s, a) \geq c _ {\text {o f f}} \left\{\frac {\log \frac {H S A}{\delta}}{K ^ {\text {o f f}}} + \frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K} \right\}\right) \\ \leq 3 d _ {h} ^ {\text {o f f}} (s, a) \mathbb {1} \left(d _ {h} ^ {\text {o f f}} (s, a) \geq \frac {2}{3} c _ {\text {o f f}} \left\{\frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K} \right\}\right), \tag {82} \\ \end{array}
$$
where the second line relies on (94). This combined with Lemma 1 results in
$$
\begin{array}{l} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \pi_ {h} ^ {t} (a \mid s) \frac {\widehat {d} _ {h} ^ {\mathrm {o f f}} (s , a)}{\frac {1}{K H} + \mathbb {E} _ {\pi \sim \mu^ {\mathrm {o f f}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \pi_ {h} ^ {t} (a \mid s) \frac {3 d _ {h} ^ {\text {o f f}} (s , a) \mathbb {1} \left(d _ {h} ^ {\text {o f f}} (s , a) \geq \frac {2}{3} c _ {\text {o f f}} \left\{\frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K} \right\}\right)}{\frac {1}{K H} + \frac {1}{2} \mathbb {E} _ {\pi \sim \mu^ {\text {o f f}}} \left[ d _ {h} ^ {\pi} (s , a) - 2 e _ {h} ^ {\pi} (s , a) - \frac {\xi}{4 N} \right]}. \tag {83} \\ \end{array}
$$
Moreover, inequality (56) tells us that: when  $d_h^{\mathrm{off}}(s,a)\geq \frac{2}{3} c_{\mathrm{off}}\Big(\frac{H^4S^4A^4\log\frac{HSA}{\delta}}{N} +\frac{SA}{K^{\mathrm{on}}}\Big)$  for some large enough constant  $c_{\mathrm{off}} > 0$ , we have
$$
\mathbb {E} _ {\pi \sim \mu^ {\text {o f f}}} \left[ d _ {h} ^ {\pi} (s, a) - 2 e _ {h} ^ {\pi} (s, a) - \frac {\xi}{4 N} \right] \geq \mathbb {E} _ {\pi \sim \mu^ {\text {o f f}}} \left[ d _ {h} ^ {\pi} (s, a) - \frac {4 S A}{K ^ {\text {o n}}} - \frac {2 7 c _ {\xi} S ^ {4} A ^ {4} H ^ {4} \log \frac {H S A}{\delta}}{N} \right] \geq \frac {1}{2} d _ {h} ^ {\text {o f f}} (s, a). \tag {84}
$$
In turn, this implies that
$$
\begin{array}{l} \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \pi_ {h} ^ {t} (a \mid s) \frac {3 d _ {h} ^ {\mathrm {o f f}} (s , a) \mathbb {1} \left(d _ {h} ^ {\mathrm {o f f}} (s , a) \geq \frac {2}{3} c _ {\mathrm {o f f}} \left\{\frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K} \right\}\right)}{\frac {1}{K H} + \frac {1}{2} \mathbb {E} _ {\pi \sim \mu^ {\mathrm {o f f}}} \left[ d _ {h} ^ {\pi} (s , a) - 2 e _ {h} ^ {\pi} (s , a) - \frac {\xi}{4 N} \right]} \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \pi_ {h} ^ {t} (a \mid s) \frac {3 d _ {h} ^ {\text {o f f}} (s , a) \mathbb {1} \left(d _ {h} ^ {\text {o f f}} (s , a) \geq \frac {2}{3} c _ {\text {o f f}} \left\{\frac {H ^ {4} S ^ {4} A ^ {4} \log \frac {H S A}{\delta}}{N} + \frac {S A}{K} \right\}\right)}{\frac {1}{K H} + \frac {1}{4} d _ {h} ^ {\text {o f f}} (s , a)} \\ \leq \sum_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \pi_ {h} ^ {t} (a \mid s) \frac {3 d _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{4} d _ {h} ^ {\text {o f f}} (s , a)} = 1 2 S. \tag {85} \\ \end{array}
$$
Consequently, combine (83) and (85) to yield
$$
\sum_ {h = 1} ^ {H} \sum_ {s \in \mathcal {S}} \mathbb {E} _ {a \sim \pi_ {h} ^ {t + 1} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K H} + \mathbb {E} _ {\pi \sim \mu^ {\text {o f f}}} \left[ \widehat {d} _ {h} ^ {\pi} (s , a) \right]} \right] \leq 1 2 S H, \tag {86}
$$
which clearly validates the claim (81) (with an even better pre-constant).
Before moving forward, we single out one useful property that arises from the above arguments:
$$
\mathbb {E} _ {\pi \sim \mu^ {\text {o f f}}} \left[ \widehat {d} _ {h} ^ {\pi} (s, a) \right] \geq \frac {1}{1 2} \widehat {d} _ {h} ^ {\text {o f f}} (s, a). \tag {87}
$$
To prove the validity of this claim (87), it suffices to make the following two observations:
- When  $d_h^{\mathrm{off}}(s, a) \geq \frac{2}{3} c_{\mathrm{off}} \left( \frac{H^4 S^4 A^4 \log \frac{HSA}{\delta}}{N} + \frac{SA}{K^{\mathrm{on}}} \right)$ , it has been shown in (83) and (84) in conjunction with (82) that
$$
\mathbb {E} _ {\pi \sim \mu^ {\mathrm {o f f}}} \left[ \widehat {d} _ {h} ^ {\pi} (s, a) \right] \geq \frac {1}{4} d _ {h} ^ {\mathrm {o f f}} (s, a) \geq \frac {1}{1 2} \widehat {d} _ {h} ^ {\mathrm {o f f}} (s, a). \tag {88}
$$
- When  $d_h^{\mathrm{off}}(s,a) < \frac{2}{3} c_{\mathrm{off}}\left(\frac{H^4S^4A^4\log\frac{HSA}{\delta}}{N} +\frac{SA}{K^{\mathrm{on}}}\right)$ , one sees from (82) that  $\widehat{d}_h^{\mathrm{off}}(s,a) = 0$  and hence (87) holds true trivially.
Iteration complexity. Suppose that the stopping criterion (37) is not yet met in the  $k$ -th iteration, namely,
$$
\sum_ {h = 1} ^ {H} \sum_ {s \in S} \mathbb {E} _ {a \sim \pi_ {h} ^ {t + 1} (\cdot | s)} \left[ \frac {\widehat {d} _ {h} ^ {\text {o f f}} (s , a)}{\frac {1}{K H} + \mathbb {E} _ {\pi^ {\prime} \sim \mu^ {(k)}} \left[ \widehat {d} _ {h} ^ {\pi^ {\prime}} (s , a) \right]} \right] > 1 0 8 S H. \tag {89}
$$
It can be easily seen that