SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {t \in [ H ], i \in [ m ]} \sum_ {l k j \in \mathbf {J} ^ {i} (t)} \min  \left\{1, \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {l k j} (t) ^ {- 1}} ^ {2} \right\} \geq \sum_ {t \in [ H ], i \in [ m ]} \min  \left\{1, \frac {1}{2} \sum_ {l k j \in \mathbf {J} ^ {i} (t)} \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {i t} ^ {- 1}} ^ {2} \right\} \\ \geq \sum_ {i \in [ m ]} \min  \left\{1, \frac {1}{2} \sum_ {t \in [ H ]} \sum_ {l k j \in \mathbf {J} ^ {i} (t)} \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {i t} ^ {- 1}} ^ {2} \right\} \\ \geq \sum_ {i \in [ m ]} \min  \left\{1, \frac {1}{2 H n} \left(\sum_ {t \in [ H ]} \sum_ {l k j \in \mathbf {J} ^ {i} (t)} \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {i t} ^ {- 1}}\right) ^ {2} \right\} \\ \geq \sum_ {i \in [ m ]} \min  \left\{1, \frac {1}{2 H n} \left(n \sum_ {k \in [ H ]} \bar {\sigma} _ {k} ^ {i}\right) ^ {2} \right\} \\ \end{array}
$$
Whenever an iteration finishes without returning in Line 17,  $\sum_{k\in [H]}\bar{\sigma}_k^m >\varepsilon /(dH^2\beta \omega)$ . Therefore,
$$
\begin{array}{l} 2 d H \log \left(1 + \frac {H m n L _ {1} ^ {2}}{d \lambda}\right) \geq \sum_ {i \in [ m ]} \min  \left\{1, \frac {1}{2 H n} \left(n \sum_ {k \in [ H ]} \bar {\sigma} _ {k} ^ {i}\right) ^ {2} \right\} \\ \geq \sum_ {i \in [ m ]} \min  \left\{1, \frac {1}{2 H} n \left(\frac {\varepsilon}{d H ^ {2} \beta \omega}\right) ^ {2} \right\} \\ \geq \sum_ {i \in [ m ]} \min  \left\{1, H d / \beta^ {2} \right\} = m H d / \beta^ {2}, \\ \end{array}
$$
Therefore, even for the iteration that returns in Line 17,
$$
m \leq \beta^ {2} \log \left(1 + \frac {H m n L _ {1} ^ {2}}{d \lambda}\right) + 1 = m _ {\max } .
$$
J Deferred proofs for Appendix E.3
Proof of Lemma E.8. For notational simplicity we drop the subscripts  $(\hat{G},\bar{\theta})$ . We first use the usual high-probability bounds on the least squares predictor and Hoeffding's inequality on the empirical mean quantities, to prove that with probability at least  $1 - 3\zeta$ , during the execution of SKIPPYE-LEANOR whenever Line 16 is executed, for all  $k\in [H]$
$$
\underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k, H + 1} C \left(S _ {p (k)}\right) \leq \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k)} ^ {H} R _ {u} + \tilde {c} _ {k, H + 1} E ^ {\rightarrow} \left(S _ {p (k) + 1}, \dots , R _ {H}\right) + 2 \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 4 \frac {\varepsilon}{H}. \tag {44}
$$
The proof of this is presented as Lemma J.1.
Next, to prove the statement for  $k \in [H]$ , assume by induction that Eq. (22) holds for  $i \in [k + 1:H]$ .
Observe that SKIPPYPOLICY performs a rollout with policy  $\pi^0$  for the rest of the episode starting from stage  $p(k) + 1$ , that is,  $1 = A_{p(k) + 1} = \dots = A_H$ . Therefore, the law of the random variables  $S_{p(k) + 1}, \ldots, R_H$ , given  $(S_{p(k)}, A_{p(k)})$  is fully determined by the dynamics of the MDP, and is independent of the values of  $p(k + 1), \ldots, p(H)$ . Therefore,
$$
\begin{array}{l} \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k + 1, H + 1} C \left(S _ {p (k + 1)}\right) = \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k + 1, H + 1} D \left(S _ {p (k + 1)}, \dots , R _ {H}\right) + \sum_ {u = p (k + 1)} ^ {H} R _ {u} \tag {45} \\ = \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k, H + 1} E ^ {\rightarrow} (S _ {p (k) + 1}, \dots , R _ {H}) + \sum_ {u = p (k + 1)} ^ {H} R _ {u}  , \\ \end{array}
$$
where we use Eq. (9), and that  $\pi^{mk}$  (SKIPPYPOLICY) is in phase II after stage  $p(k)$ , but defines the mapping  $p(\cdot)$  independently of whether the policy is in phase I or phase II, in such a way that for any  $H \geq j > p(k)$ ,
$$
\mathcal {P} _ {\pi^ {m k}, s _ {1}} \left[ p (k + 1) = j \mid p (k), S _ {p (k)}, A _ {p (k)} \right] = \mathcal {P} _ {\pi^ {m k}, s _ {1}} \left[ \tau (S _ {j}) \prod_ {j ^ {\prime} = p (k) + 1} ^ {j - 1} \left(1 - \tau (S _ {j ^ {\prime}})\right) \mid p (k), S _ {p (k)}, A _ {p (k)} \right].
$$
Combining Eq. (45) with Eq. (44),
$$
\underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k, H + 1} C (S _ {p (k)}) \leq \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k)} ^ {p (k + 1) - 1} R _ {u} + \tilde {c} _ {k + 1, H + 1} C (S _ {p (k + 1)}) + 2 \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 4 \frac {\varepsilon}{H}.
$$
By Remark E.7,  $\mathbb{E}_{\pi^{mk},s_1}\tilde{c}_{k + 1,H + 1}C(S_{p(k + 1)}) = \mathbb{E}_{\pi^{m,k + 1},s_1}\tilde{c}_{k + 1,H + 1}C(S_{p(k + 1)}) = \bar{C}^{k + 1}$ . Therefore, combining with the inductive hypothesis,
$$
\begin{array}{l} \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k, H + 1} C (S _ {p (k)}) \leq \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k)} ^ {p (k + 1) - 1} R _ {u} + \bar {C} ^ {k + 1} + 2 \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 4 \frac {\varepsilon}{H} \\ \leq \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k)} ^ {p (k + 1) - 1} R _ {u} + \underset {\pi^ {m H}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k + 1)} ^ {H} R _ {u} + 2 \sum_ {i = k} ^ {H} \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 4 (H - k + 1) \frac {\varepsilon}{H} \\ = \underset {\pi^ {m H}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k)} ^ {H} R _ {u} + 2 \sum_ {i = k} ^ {H} \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 4 (H - k + 1) \frac {\varepsilon}{H} \\ \end{array}
$$
where the last equation uses Remark E.7 again, finishing the induction.
Lemma J.1. Adopt the notation of Lemma E.8. With probability at least  $1 - 3\zeta$ , during the execution of SKIPPYELEANOR, whenever Line 16 is executed, for all  $k \in [H]$
$$
\underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k, H + 1} C (S _ {p (k)}) \leq \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k)} ^ {H} R _ {u} + \tilde {c} _ {k, H + 1} E ^ {\rightarrow} (S _ {p (k) + 1}, \dots , R _ {H}) + 2 \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 4 \frac {\varepsilon}{H}.
$$
Proof. We refer as  $\hat{\theta}$  to the value of the argument of Optimization Problem 4.10 recorded in Line 5. For  $k\in [H]$ , recall the definition of  $\bar{\sigma}_k^m$  (Eq. (18)), along with the fact that unless  $c_{k,H + 1}^j = 0$ ,  $\left\| \varphi_{\mathrm{p}(k)}^{mkj}\right\|_{X_{m,\mathrm{p}(mkj)}^{-1}} < 2(\beta \omega dH)^{-1}$ , we get a useful bound on the average norm of the features under consideration:
$$
\frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left\| \varphi_ {\mathrm {p} (k)} ^ {m k j} \right\| _ {X _ {m, \mathrm {p} (m k j)} ^ {- 1}} \leq \bar {\sigma} _ {k} ^ {m}. \tag {46}
$$
If Line 16 is executed, the consistency check passed, and therefore for all  $k \in [H - 1], i \in [k + 1:H]$
$$
\operatorname {T r} \left(y ^ {k i} - \hat {F} ^ {k i}\right) \leq \bar {\sigma} _ {k} ^ {m} \beta \omega d + 3 \frac {\varepsilon}{H ^ {2}} \tag {47}
$$
For  $t \in [H]$  let the least-squares predictor of rewards sums under the policy  $\pi^0$  be
$$
\check {\theta} ^ {t, H + 1} = X _ {m t} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \sum_ {u = t} ^ {H} R _ {u} ^ {l k j}.
$$
For  $k \in [H]$  and  $j \in [n]$  let us introduce the shorthand
$$
R _ {k \rightarrow} ^ {m k j} = \sum_ {u = \mathrm {p} (m k j)} ^ {H} R _ {u} ^ {m k j},
$$
and similarly when the trajectory is clear from context:  $R_{k\rightarrow} = \sum_{u = p(k)}^{H}R_{u}$ . For  $k\in [H]$  let
$$
\hat {E} ^ {k} = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left(E ^ {\rightarrow} (S _ {\mathrm {p} (k) + 1} ^ {m k j}, \dots , R _ {H} ^ {m k j}) + R _ {k \rightarrow} ^ {m k j}\right)
$$
$$
\hat {C} ^ {k} = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} C (S _ {\mathrm {p} (k)} ^ {m k j})
$$
$$
y ^ {k, H + 1} = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \check {\theta} ^ {\mathrm {p} (m k j), H + 1} \right\rangle
$$
$$
z ^ {k, H + 1} = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} R _ {k \rightarrow} ^ {m k j}
$$
For  $t\in [H - 1],i\in [t + 1:H]$  , along with  $\check{\theta}^{t,H + 1}$  , let
$$
\check {\theta} ^ {t i} = X _ {m t} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \operatorname {T r} (F (S _ {i} ^ {l k j}, \dots , R _ {H} ^ {l k j})) = X _ {m t} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} E (S _ {i} ^ {l k j}, \dots , R _ {H} ^ {l k j}),
$$
where the second equality is by Eq. (32). Observe that for any  $\nu \in \mathbb{R}^d$ ,  $\mathrm{Tr}(v^\top \hat{\theta}^{ti}) = \langle \nu, \check{\theta}^{ti} \rangle$ . Therefore, for  $k \in [\bar{H}]$ ,
$$
y ^ {k, H + 1} + \sum_ {i = k + 1} ^ {H} \operatorname {T r} (y ^ {k i}) = \frac {1}{n} \sum_ {j \in [ n ]} \sum_ {i = k + 1} ^ {H + 1} c _ {k i} ^ {j} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \check {\theta} ^ {\mathrm {p} (m k j), i} \right\rangle = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \sum_ {i = \mathrm {p} (m k j) + 1} ^ {H + 1} \check {\theta} ^ {\mathrm {p} (m k j), i} \right\rangle
$$
For any  $t\in [H]$  , by the definitions,
$$
\begin{array}{l} \sum_ {i = t + 1} ^ {H + 1} \check {\theta} ^ {t i} = X _ {m t} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \left(\sum_ {i = t + 1} ^ {H} E \left(S _ {i} ^ {l k j}, \dots , R _ {H} ^ {l k j}\right) + \sum_ {u = t} ^ {H} R _ {u} ^ {m k j}\right) \\ = X _ {m t} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \left(E ^ {\rightarrow} (S _ {t + 1} ^ {l k j}, \ldots , R _ {H} ^ {l k j}) + \sum_ {u = t} ^ {H} R _ {u} ^ {m k j}\right) = \hat {\theta} _ {t} \\ \end{array}
$$
Plugging this into the previous calculation,
$$
\begin{array}{l} y ^ {k, H + 1} + \sum_ {i = k + 1} ^ {H} \operatorname {T r} \left(y ^ {k i}\right) = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \hat {\theta} _ {\mathrm {p} (m k j)} \right\rangle \\ \geq \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \bar {\theta} _ {\mathrm {p} (m k j)} \right\rangle \\ - \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left\| \varphi_ {\mathrm {p} (k)} ^ {m k j} \right\| _ {X _ {m, \mathrm {p} (m k j)} ^ {- 1}} \left\| \bar {\theta} _ {\mathrm {p} (m k j)} - \hat {\theta} _ {\mathrm {p} (m k j)} \right\| _ {X _ {m, \mathrm {p} (m k j)}} \tag {48} \\ \geq \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \bar {\theta} _ {\mathrm {p} (m k j)} \right\rangle - \bar {\sigma} _ {k} ^ {m} \beta H \\ \geq \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \operatorname {c l i p} _ {[ 0, H ]} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \bar {\theta} _ {\mathrm {p} (m k j)} \right\rangle - \bar {\sigma} _ {k} ^ {m} \beta H \\ = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} C (S _ {\mathrm {p} (k)} ^ {m k j}) - \bar {\sigma} _ {k} ^ {m} \beta H = \hat {C} ^ {k} - \bar {\sigma} _ {k} ^ {m} \beta H, \\ \end{array}
$$
where the first inequality uses Cauchy-Schwarz. The second inequality bounds the average of the first norm by Eq. (46), and the bound on the second norm (for any  $j$ ) is by definition of Optimization Problem 4.10. The third inequality relies on the fact that  $c_{k,H+1}^j = 0$  if the clipped inner product is negative, and the final equality is due to the definition of  $C$  along with the fact that  $A_{\mathrm{p}(k)}^{mk_j} = \pi^+(S_{\mathrm{p}(k)}^{mk_j})$ , as this is the last state in the trajectory where SKIPPPOLICY takes the inner-product maximizing action  $(\pi^+)$  before rolling out with  $\pi^0$ .
By Eqs. (12) and (32), we have that
$$
\begin{array}{l} z ^ {k, H + 1} + \sum_ {i \in [ k + 1: H ]} \operatorname {T r} \left(\hat {F} ^ {k i}\right) = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left(\sum_ {i = \mathrm {p} (m k j) + 1} ^ {H + 1} E \left(S _ {i} ^ {m k j}, \dots , R _ {H} ^ {m k j}\right) + R _ {k \rightarrow} ^ {m k j}\right) \tag {49} \\ = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left(E ^ {\rightarrow} (S _ {\mathrm {p} (k) + 1} ^ {m k j}, \dots , R _ {H} ^ {m k j}) + R _ {k \rightarrow} ^ {m k j}\right) = \hat {E} ^ {k} \\ \end{array}
$$
Combining Eqs. (48) and (49),
$$
\begin{array}{l} \hat {C} ^ {k} - \hat {E} ^ {k} \leq \bar {\sigma} _ {k} ^ {m} \beta H + \left(y ^ {k, H + 1} - z ^ {k, H + 1}\right) + \sum_ {i \in [ k + 1: H ]} \operatorname {T r} \left(y ^ {k i} - \hat {F} ^ {k i}\right) \\ \leq \bar {\sigma} _ {k} ^ {m} \beta H + \left(\left| \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} c _ {k, H + 1} R _ {k \rightarrow} - z ^ {k, H + 1} \right| + \left| y ^ {k, H + 1} - \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} c _ {k, H + 1} R _ {k \rightarrow} \right|\right) + \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 3 \frac {\varepsilon}{H} \tag {50} \\ \end{array}
$$
where the sum (last term) is bounded by Eq. (47), and we apply a triangle inequality on the second term. To continue bounding this term, we apply Hoeffding's inequality on the independent random variables  $c_{k,H+1}^j R_{k\to}$  (for  $j \in [n]$ ) that have range  $[0,H]$ , along with a union bound over the iteration  $m' \in [m_{\max}'']$  and  $k \in [H]$ , to get that with probability at least  $1 - \zeta$ ,
$$
\left| _ {\pi^ {m k}, s _ {1}} \mathbb {E} c _ {k, H + 1} R _ {k \rightarrow} - z ^ {k, H + 1} \right| \leq \frac {H}{\sqrt {n}} \sqrt {\log \frac {2 m _ {\operatorname* {m a x}} ^ {\prime} H}{\zeta}} \leq \frac {\varepsilon}{d H ^ {2} \omega}. \tag {51}
$$
The remaining term  $\left|y^{k,H + 1} - \mathbb{E}_{\pi^{mk},s_1}c_{k,H + 1}R_{k\to}\right|$  is bounded using the realizability of  $q^{\pi^0}$  (Definition 3.2) as follows. Take any  $t\in [H]$ . By definition there exists  $\theta_t^\star \in \Theta_t^Q\subseteq \mathcal{B}(L_2)$ , such that for all  $s\in S_t$  and  $a\in [\mathcal{A}]$ ,  $q^{\pi^0}(s,a)\approx_\eta \left\langle \varphi (s,a),\theta_t^\star \right\rangle$ . Take the sequence  $A$  formed of  $\varphi_t^{lkj}$  (for  $lkj\in \mathbf{I}^m (t)$ , in the order that these random variables are observed), and the sequence  $X$  formed of  $R_{k\to}^{mkj}$  (for  $lkj\in \mathbf{I}^m (t)$ , in the same order), and the sequence  $\Delta$  formed of  $q^{\pi^0}(S_t^{lkj},A_t^{lkj}) - \left\langle \varphi_t^{lkj},\theta_t^\star \right\rangle$  (for  $lkj\in \mathbf{I}^m (t)$ , in the same order). Then the sequences  $A,X$ , and  $\Delta$  satisfy the conditions of Lemma M.4 with a subgaussianity parameter  $\sigma = H$ . Due to this lemma, applied with a union bound over  $m'\in [m_{\max}'']$  and  $t\in [H]$ , with probability at least  $1 - \zeta$ ,
$$
\begin{array}{l} \left\| \check {\theta} ^ {t, H + 1} - \theta_ {t} ^ {\star} \right\| _ {X _ {m t}} <   \sqrt {\lambda} \left\| \theta_ {t} ^ {\star} \right\| _ {2} + \| \Delta \| _ {\infty} \sqrt {| \mathbf {I} ^ {m} (t) |} + H \sqrt {2 \log \left(\frac {m _ {\operatorname* {m a x}} ^ {\prime} H}{\zeta}\right) + \log \left(\frac {\det X _ {m t}}{\lambda^ {d}}\right)} \\ \leq 2 + H \sqrt {2 \log \frac {m _ {\operatorname* {m a x}} ^ {\prime} H}{\zeta} + \log \left(\frac {\det X _ {m t}}{\lambda^ {d}}\right)} \leq \beta , \\ \end{array}
$$
by Eq. (41). Therefore by Cauchy-Schwarz and Eq. (46),
$$
\begin{array}{l} \left| y ^ {k, H + 1} - \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} c _ {k, H + 1} R _ {k \rightarrow} \right| \leq \frac {1}{n} \sum_ {j \in [ n ]} c _ {k, H + 1} ^ {j} \left(\left\| \varphi_ {\mathrm {p} (k)} ^ {m k j} \right\| _ {X _ {m, \mathrm {p} (m k j)} ^ {- 1}} \left\| \check {\theta} ^ {\mathrm {p} (m k j), H + 1} - \theta_ {\mathrm {p} (m k j)} ^ {\star} \right\| _ {X _ {m t}} + \eta\right) \\ \leq \bar {\sigma} _ {k} ^ {m} \beta + \eta \leq \bar {\sigma} _ {k} ^ {m} \beta + \frac {\varepsilon}{d H ^ {2} \omega}. \\ \end{array}
$$
Combining this with Eqs. (50) and (51),
$$
\hat {C} ^ {k} - \hat {E} ^ {k} \leq 1. 5 \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 3 \frac {\varepsilon}{H} + 2 \frac {\varepsilon}{d H ^ {2} \omega}. \tag {52}
$$
We introduce the following notation for  $j \in [n]$ ,  $k \in [H + 1]$ ,  $i \in [H + 1]$ :
$$
\begin{array}{l} \bar {c} _ {k i} ^ {j} = \mathbb {1} \left\{\mathrm {p} (m k j) <   i \text {a n d} \left\| \varphi_ {\mathrm {p} (k)} ^ {m k j} \right\| _ {X _ {m, \mathrm {p} (m k j)} ^ {- 1}} \geq 2 (\beta \omega d H) ^ {- 1} \text {a n d} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \bar {\theta} _ {\mathrm {p} (m k j)} \right\rangle \geq 0 \right\} \\ \hat {c} _ {k i} ^ {j} = \mathbb {1} \left\{\mathrm {p} (m k j) <   i \text {a n d} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \bar {\theta} _ {\mathrm {p} (m k j)} \right\rangle <   0 \right\}, \\ \end{array}
$$
such that for all  $j$
$$
\tilde {c} _ {k i} ^ {j} = c _ {k i} ^ {j} + \bar {c} _ {k i} ^ {j} + \hat {c} _ {k i} ^ {j}. \tag {53}
$$
Continuing from Eq. (52), as  $E \to (s_i \to) + \sum_{u = i}^{H} r_u \geq 0$  by Eq. (31), and if  $\hat{c}_{k,H+1}^j = 1$  then  $C(S_{\mathrm{p}(k)}^{mk,j}) = 0$ , we have that
$$
\frac {1}{n} \sum_ {j \in [ n ]} (c _ {k, H + 1} ^ {j} + \hat {c} _ {k, H + 1} ^ {j}) \left(C (S _ {\mathrm {p} (k)} ^ {m k j}) - \left(E ^ {\rightarrow} (S _ {\mathrm {p} (k) + 1} ^ {m k j}, \dots , R _ {H} ^ {m k j}) + R _ {k \rightarrow} ^ {m k j}\right)\right) \leq 1. 5 \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 3 \frac {\varepsilon}{H} + 2 \frac {\varepsilon}{d H ^ {2} \omega}.
$$
As (even if  $\bar{c}_{k,H + 1}^j = 1$ )  $C(S_{\mathrm{p}(k)}^{mkj}) \leq H$
$$
\frac {1}{n} \sum_ {j \in [ n ]} \bar {c} _ {k, H + 1} ^ {j} C (S _ {p (k)}) \leq H \bar {\sigma} _ {k} ^ {m} / (2 (\beta \omega d H) ^ {- 1}) = \frac {1}{2} \bar {\sigma} _ {k} ^ {m} \beta \omega d H,
$$
which combined with the previous inequality and Eq. (53) yields
$$
\frac {1}{n} \sum_ {j \in [ n ]} \tilde {c} _ {k, H + 1} ^ {j} \left(C (S _ {\mathrm {p} (k)} ^ {m k j}) - \left(E ^ {\rightarrow} (S _ {\mathrm {p} (k) + 1} ^ {m k j}, \dots , R _ {H} ^ {m k j}) + R _ {k \rightarrow} ^ {m k j}\right)\right) \leq 2 \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 3 \frac {\varepsilon}{H} + 2 \frac {\varepsilon}{d H ^ {2} \omega}.
$$
Observe that the random variables  $\tilde{c}_{k,H+1}^{j}\left(C(S_{\mathrm{p}(k)}^{mkj}) - \left(E^{\rightarrow}(S_{\mathrm{p}(k)+1}^{mkj},\dots,R_{H}^{mkj}) + R_{k\rightarrow}^{mkj}\right)\right)$  are independent (for  $j\in [n]$ ) with range  $[-2H,H]$  (Eq. (31)). By Hoeffding's inequality, with probability at least  $1 - \zeta$ , for all iteration  $m'\in [m_{\max}']$  (this includes the entire execution of SKIPPYLEANOR) and  $k\in [H]$ ,
$$
\begin{array}{l} \left| \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k, H + 1} \left(C \left(S _ {p (k)}\right) - \left(E ^ {\rightarrow} \left(S _ {p (k) + 1}, \dots , R _ {H}\right) + R _ {k \rightarrow}\right)\right)\right. \\ \left. - \frac {1}{n} \sum_ {j \in [ n ]} \tilde {c} _ {k, H + 1} ^ {j} \left(C (S _ {\mathrm {p} (k)} ^ {m k j}) - \left(E ^ {\rightarrow} (S _ {\mathrm {p} (k) + 1} ^ {m k j}, \dots , R _ {H} ^ {m k j}) + R _ {k \rightarrow} ^ {m k j}\right)\right)\right| \\ \leq \frac {4 H}{\sqrt {n}} \sqrt {\log \frac {2 m _ {\operatorname* {m a x}} ^ {\prime} H}{\zeta}} \leq \frac {\varepsilon}{d H ^ {2} \omega}. \\ \end{array}
$$
Combining with the previous bound, under the intersection of the high-probability events referred to above, which by a union bound has a probability of at least  $1 - 3\zeta$ , we have that for all  $k \in [H]$
K Deferred proofs for Appendix E.4
Proof of Lemma E.10. Let  $m$  be the current iteration. Unlike in previous lemmas, here we introduce  $(\hat{G},\bar{\theta})$  that does not refer to the outcome of Optimization Problem 4.10. Instead, let  $\hat{G} = (\vartheta_h^i)_{h\in [2:H],i\in [d_0]}\in \mathbf{G}$  be the correct guess. For  $h = H,\ldots ,1,\bar{\theta}_h$  is defined in sequence along with the behavior of a policy  $\pi$  on stage  $h$ .
For  $h = H, \dots, 1$ , assuming that this process already defined  $\bar{\theta}_{h+1}, \dots, \bar{\theta}_H$  (in Eq. (55)), let  $\pi$  be the policy that, for any  $t > h$  and  $s \in S_t$ , takes action on  $s$  as  $\pi_{\hat{G}\bar{\theta}}^+(s)$  with probability  $\tau_{\hat{G}\bar{\theta}}(s)$ , and action 1 with probability  $1 - \tau_{\hat{G}\bar{\theta}}(s)$  ( $\tau$  is defined in Eq. (8)). Simultaneously, using the second part of Corollary 4.11, define  $\tilde{\theta}_{hi} \in \mathcal{B}(4d_0L_2 / \alpha)$  for  $i \in [h + 1:H]$  to satisfy for all  $s \in S_h$ ,  $a \in [\mathcal{A}]$ :
$$
\underset {\pi^ {0}, s, a} {\mathbb {E}} \operatorname {T r} (\bar {F} _ {\hat {G} \bar {\theta}} (S _ {i})) \approx_ {\eta_ {0}} \left\langle \varphi (s, a), \tilde {\theta} _ {h i} \right\rangle .
$$
We also define  $\tilde{\theta}_{h,H + 1}\in \mathcal{B}(L_2)$  to satisfy for all  $s\in S_h$ $a\in [\mathcal{A}]$
$$
\underset {\pi^ {0}, s, a} {\mathbb {E}} \sum_ {u = h} ^ {H} R _ {u} \approx_ {\eta} \left\langle \varphi (s, a), \tilde {\theta} _ {h, H + 1} \right\rangle .
$$
By Eq. (32),
$$
\underset {\pi^ {0}, s, a} {\mathbb {E}} \sum_ {i \in [ h + 1: H ]} \operatorname {T r} \left(\bar {F} _ {\hat {G} \bar {\theta}} \left(S _ {i}\right)\right) + \sum_ {u = h} ^ {H} R _ {u} = \underset {\pi^ {0}, s, a} {\mathbb {E}} E _ {\hat {G} \bar {\theta}} ^ {\rightarrow} \left(S _ {h + 1}, \dots , R _ {H}\right) + \sum_ {u = h} ^ {H} R _ {u} \approx_ {H \eta_ {0}} \left\langle \varphi (s, a), \bar {\theta} _ {h} \right\rangle , \tag {54}
$$
$$
\bar {\theta} _ {h} = \sum_ {i \in [ h + 1: H + 1 ]} \tilde {\theta} _ {h i}. \tag {55}
$$
We first show that  $(\hat{G},\bar{\theta})$  is feasible for Optimization Problem 4.10. Clearly,  $\left\| \bar{\theta}_h\right\| _2\leq 4d_0HL_2 / \alpha$  For any  $i\in [h + 1:H]$  , let
$$
\hat {\theta} _ {h i} = X _ {m h} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (h)} \varphi_ {h} ^ {l k j} \mathrm {T r} (F _ {\hat {G} \bar {\theta}} (S _ {h + 1} ^ {l k j}, \dots , R _ {H} ^ {l k j})),
$$
$$
\hat {\theta} _ {h, H + 1} = X _ {m h} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (h)} \varphi_ {h} ^ {l k j} \sum_ {u = h} ^ {H} R _ {u} ^ {l k j}.
$$
Then,  $\hat{\theta}$  of Optimization Problem 4.10 satisfies for all  $h\in [H]$ , by Eq. (32),
$$
\hat {\theta} _ {h} = \sum_ {i \in [ h + 1: H + 1 ]} \hat {\theta} _ {h i}.
$$
To show that  $(\hat{G},\bar{\theta})$  is feasible, it thus suffices to show for all  $h\in [H]$ $i\in [h + 1:H + 1]$ , that  $\left\| \tilde{\theta}_{hi} - \hat{\theta}_{hi}\right\|_{X_{mh}}\leq \beta$
Fix any  $h \in [H]$  and  $i \in [h + 1 : H + 1]$ . Take the sequence  $A$  formed of  $\varphi_t^{lkj}$  (for  $lkj \in \mathbf{I}^m(h)$ , in the order that these random variables are observed). For  $i < H + 1$  take the sequence  $X$  formed of  $\mathrm{Tr}(F_{\hat{G}\bar{\theta}}(S_i^{lkj},\ldots,R_H^{lkj}))$  (for  $lkj \in \mathbf{I}^m(h)$ , in the same order), and the sequence  $\Delta$  formed of  $\mathbb{E}_{\pi^0,S_h^{lkj},A_h^{lkj}}\mathrm{Tr}(\bar{F}_{\hat{G}\bar{\theta}}(S_i)) - \left\langle \varphi_h^{lkj},\tilde{\theta}_{hi}\right\rangle$  (for  $lkj \in \mathbf{I}^m(h)$ , in the same order). For  $i = H + 1$ , the sequence  $X$  is formed of  $\sum_{u=h}^{H}R_u^{lkj}$ , and  $\Delta$  is formed of  $q^{\pi^0}(S_h^{lkj},A_h^{lkj}) - \left\langle \varphi_h^{lkj},\tilde{\theta}_{hi}\right\rangle$ . Then the sequences  $A, X$ , and  $\Delta$  satisfy the conditions of Lemma M.4 with a subgaussianity parameter  $\sigma = H$ . Due to this lemma, applied with a union bound over  $m' \in [m_{\max}'',t]$ , and  $i$ , with probability at least  $1 - \zeta$ ,
$$
\begin{array}{l} \left\| \hat {\theta} _ {h i} - \tilde {\theta} _ {h i} \right\| _ {X _ {m h}} <   \sqrt {\lambda} \left\| \tilde {\theta} _ {h i} \right\| _ {2} + \| \Delta \| _ {\infty} \sqrt {| \mathbf {I} ^ {m} (t) |} + H \sqrt {2 \log \left(\frac {m _ {\operatorname* {m a x}} ^ {\prime} H ^ {2}}{\zeta}\right) + \log \left(\frac {\det X _ {m t}}{\lambda^ {d}}\right)} \\ \leq 2 + H \sqrt {2 \log \frac {m _ {\mathrm {m a x}} ^ {\prime} H ^ {2}}{\zeta} + \log \left(\frac {\det X _ {m t}}{\lambda^ {d}}\right)} \leq \beta , \\ \end{array}
$$
by Eq. (41).