SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof of Theorem 4.1 is presented in Appendix E. It is composed of the following main steps: First, we bound the number of times the consistency check can fail (i.e., Line 13 is executed) by Lemma 4.4. Combining this with Lemma E.5, an elliptical potential argument bounding the number of times the average uncertainty can be large (these are the only two ways that the main iteration can continue) implies a sample-complexity result for SKIPPYELEANOR (Corollary E.6). Having limited the number of times the consistency check can fail, we derive guarantees regarding the performance of the policy returned by the algorithm: Via an induction argument (Lemma E.8) we show Corollary E.9, which shows that with high probability the difference between the optimization value of Optimization Problem 4.10,  $C_{\hat{G}, \bar{\theta}}(s_1)$  and  $v^{\pi^{mH}}$  scales with the average uncertainty term  $\sum_{i=1}^{H} \bar{\sigma}_k^m$ . Thus, they are close when SKIPPYELEANOR returns in Line 17. This is complemented with the optimism property proved in Lemma E.10, stating that the optimization value  $C_{\hat{G}, \bar{\theta}}(s_1)$  is close to  $v^{\star}(s_1)$ . Combined, this proves Theorem 4.1.
6 Future work
Since we are not aware of a computationally efficient implementation of SKIPPYELEANOR, it remains an open question whether the problem of learning near-optimal policies from online interactions with a  $q^{\pi}$ -realizable MDP (Problem 3.3) is possible if the computational resources as well as the sample complexity are bounded by a polynomial in the relevant parameters. One approach is to replace ELEANOR with LSVI-UCB as the underlying algorithm, as the latter, despite having worse sample complexity, has a computationally efficient implementation [Jin et al., 2020b]. The challenge is to compute the optimal solution for the parameter  $\hat{G}$  in Optimization Problem 4.10. This parameter interacts with the least-squares targets in a highly nonlinear way. We have been unable to derive a computationally efficient approximation that has an additive instead of a multiplicative approximation error (additive errors increase linearly in  $H$ , while multiplicative errors increase exponentially). Alternatively, it may be possible to show a computational hardness result for Problem 3.3 by e.g., reducing it to the satisfiability problem. These are left for future work. Our work on the realizability of auxiliary functions (Section 4.2) may be of independent interest for designing provably efficient algorithms for related problem settings, e.g., the setting of  $q^{\pi}$ -realizability in batch RL, where the data collection is not controlled.
Simon S Du, Sham M Kakade, Ruosong Wang, and Lin F Yang. Is a good representation sufficient for sample efficient reinforcement learning? In International Conference on Learning Representations, 2019.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137-2143, 2020a.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137-2143. PMLR, 2020b.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
T. Lattimore and Cs. Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
Tor Lattimore, Csaba Szepesvári, and Gellert Weisz. Learning with good feature representations in bandits and in RL with a generative model. In ICML, pages 9464-9472, 2020.
Michael J Todd. Minimum-volume ellipsoids: Theory and algorithms. SIAM, 2016.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Andrew Wagenaaker, Yifang Chen, Max Simchowitz, Simon S Du, and Kevin Jamieson. Reward-free RL is no harder than reward-aware RL in linear Markov decision processes. arXiv preprint arXiv:2201.11206, 2022.
Gellert Weisz, András György, Tadashi Kozuno, and Csaba Szepesvari. Confident approximate policy iteration for efficient local planning in  $q^{\pi}$ -realizable MDPs. In Advances in Neural Information Processing Systems, 2022.
Dong Yin, Botao Hao, Yasin Abbasi-Yadkori, Nevena Lazic, and Csaba Szepesvári. Efficient local planning with linear function approximation. In International Conference on Algorithmic Learning Theory, pages 1165-1192. PMLR, 2022.
Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent Bellman error. In International Conference on Machine Learning, pages 10978-10989. PMLR, 2020.
As usual, we use  $\mathbb{R},\mathbb{N}$ , and  $\mathbb{N}^+$  to denote the set of reals, non-negative and positive integers, respectively. For  $i\in \mathbb{N}^{+}$ , let  $[i] = \{1,\ldots ,i\}$ ; for another positive integer  $j$ , let  $[i:j] = \{i,\dots ,j\}$  if  $i\leq j$ , and  $[i:j] = \{\}$  otherwise. For  $a,b,x\in \mathbb{R}$ , let  $\mathrm{clip}_{[a,b]}(x) = \min \{\max \{x,a\} ,b\}$  and let  $[x]$  denote the smallest integer  $i$  such that  $i\geq x$ . Let  $\mathbf{0}$  be the all-0 vector in  $\mathbb{R}^d$  and  $I$  the  $d$ -dimensional identity matrix. For a (square) matrix  $V$ , let  $V^{\dagger}$  denote its Moore-Penrose inverse, and  $\operatorname {Tr}(V)$  denote its trace. Let PD (and PSD) denote the set of positive definite (and positive semi-definite, respectively) matrices in  $\mathbb{R}^{d\times d}$ . For some  $A\in \mathrm{PSD}$  let  $A^{\frac{1}{2}}$  denote the unique matrix  $B\in \mathrm{PSD}$  such that  $A = BB$ . For  $V\in \mathrm{PD}$  and  $x\in \mathbb{R}^d$ , let  $\| x\| _V^2 = x^\top Gx$ . For matrices  $A$  and  $B$ , we say that  $A\succeq B$  (or  $A\succeq B$ ) if  $B - A$  (or  $A - B$ , respectively) is positive semidefinite.  $\operatorname {Ker}(A)$  and  $\operatorname {Im}(A)$  are the kernel (or null space), and image, respectively, of matrix  $A$ . For compatible vectors  $x,y$ , let  $\langle x,y\rangle$  be their inner product:  $\langle x,y\rangle = x^{\top}y$ . We write  $y\otimes A$  for the tensor product between  $y$  and matrix  $A$ , and then  $\langle x,y\otimes A\rangle = \langle x,y\rangle A$ . Where  $Q$  and  $h$  are obvious from the context, we write  $\nu_{\parallel}$  and  $\nu_{\perp}$  for  $\nu_{\parallel (Q,h)}$  and  $\nu_{\perp (Q,h)}$ , respectively. Throughout the paper, we omit commas between quantities in subscripts or superscript for clarity of presentation, for example, by writing  $A_{bc}$  for  $A_{b,c}$ .
For the big-Oh notation  $O$ , we introduce its counterpart  $\tilde{O}$  that hides logarithmic factors of the problem parameters  $(d, H, \varepsilon^{-1}, \zeta^{-1}, L_1, L_2)$ .
B Parameters of Algorithm 1
$$
n = \tilde {O} \left(d ^ {5} H ^ {6} / \varepsilon^ {2}\right) \quad (\text {f o r p r e c i s e v a l u e s e e E q . (4 2)})
$$
$$
\omega = 7 \left(d _ {1} + 1\right) + 7 / 3 = \tilde {O} (d) \tag {13}
$$
$$
\gamma^ {- 1} = 8 d = \tilde {O} (d) \tag {14}
$$
$$
\beta = \tilde {O} \left(H ^ {1. 5} d\right) \quad (\text {f o r p r e c i s e v a l u e s e e E q . (3 6)}) \tag {15}
$$
$$
\alpha^ {- 1} = \frac {\sqrt {2 d} \sqrt {d _ {1} + 1} H ^ {2}}{\sqrt {\gamma} \varepsilon} = \tilde {O} \left(d ^ {1. 5} H ^ {2} / \varepsilon\right) \tag {16}
$$
$$
\lambda^ {- 1} = \left(4 d _ {0} L _ {2} / \alpha\right) ^ {2} \tag {17}
$$
$$
m _ {\max } = \beta^ {2} \log \left(1 + \frac {H m n L _ {1} ^ {2}}{d \lambda}\right) + 1 = \tilde {O} \left(H ^ {3} d ^ {2}\right)
$$
$$
m _ {\mathrm {m a x}} ^ {\prime} = m _ {\mathrm {m a x}} + H d _ {1} = \tilde {O} (H ^ {3} d ^ {2})
$$
$$
\bar {\sigma} _ {k} ^ {m} = \frac {1}{n} \sum_ {j \in [ n ]} \tilde {c} _ {k, H + 1} ^ {j} \min  \left\{2 \left(\beta \omega d H\right) ^ {- 1}, \left\| \varphi_ {\mathrm {p} (k)} ^ {m k j} \right\| _ {X _ {m, \mathrm {p} (m k j)} ^ {- 1}} \right\} \quad \text {f o r} k \in [ H ] \tag {19}
$$
$$
c _ {k i} ^ {j} = \mathbb {1} \left\{\mathrm {p} (m k j) <   i \text {a n d} \left\| \varphi_ {\mathrm {p} (k)} ^ {m k j} \right\| _ {X _ {m, \mathrm {p}} ^ {- 1} (m k j)} <   2 (\beta \omega d H) ^ {- 1} \text {a n d} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \bar {\theta} _ {\mathrm {p}} (m k j) \right\rangle \geq 0 \right\} \tag {20}
$$
$$
\text {a v e r a g e} _ {\text {u n c e r t a i n t y}} = \sum_ {k = 1} ^ {H} \bar {\sigma} _ {k} ^ {m}
$$
$$
\text {u n c e r t a i n t y \_ t h r e s h o l d} = \varepsilon / (d H ^ {2} \beta \omega)
$$
$$
\text {d i s c r e p a n c y \_ t h r e s h o l d} = \bar {\sigma} _ {k} ^ {m} \beta \omega + 3 \frac {\varepsilon}{d H ^ {2}}
$$
Assumption on the maximum discrepancy:
$$
\eta \leq \frac {\alpha}{1 0 d _ {0}} \min  \left\{\varepsilon / \left(d H ^ {3} \omega\right), 1 / \sqrt {m _ {\max } ^ {\prime} n H} \right\} = \tilde {O} \left(\frac {\varepsilon^ {2}}{d ^ {6} H ^ {8}}\right) \tag {21}
$$
C Proof of Proposition 3.4
Proof of Proposition 3.4, and the MDP conversion argument. First, for  $(i)$ , we show the linearity of rewards with  $\theta_{1},\ldots ,\theta_{H}$ . For this take any  $h\in [H]$ . Fix any policy  $\pi \in \Pi$  and let  $\bar{\theta}_h\in \mathcal{B}(L_2)$  be such that for all  $(s,a)\in S_h\times [\mathcal{A}]$ ,  $q^{\pi}(s,a)\approx_{\eta}\left\langle \varphi (s,a),\bar{\theta}_h\right\rangle$  (the existence of such a  $\bar{\theta}$  follows from Definition 3.2). If  $h = H$ ,  $\mathbb{E}_R\sim_{\mathcal{R}(s,a)}[R] = q^{\pi}(s,a)$ , so  $\theta_H = \bar{\theta}_H$  satisfies Definition 3.1. For  $h < H$ , let  $f:S_{h + 1}\to \mathbb{R}$  be defined as  $f(s) = v^{\pi}(s)$ . Fix an arbitrary  $Q\in \mathrm{PD}^{H}$ , e.g.,  $Q = (I,\dots,I)$ . Since  $v^{\pi}(s)\in [0,H]$  and  $\mathrm{range}_Q(s)\geq \mathrm{range}(s) / \sqrt{2d}\geq \alpha /\sqrt{2d}$  by Proposition 4.5,  $f$  is  $\alpha /(\sqrt{2d} H)$ -admissible, and therefore by Lemma 4.7 we can take  $\tilde{\theta}_h\in \mathcal{B}(4Hd_0\sqrt{2d} L_2 / \alpha)$  such that for all  $(s,a)\in S_h\times [\mathcal{A}]$ ,
$$
\mathbb {E} \left(\nu^ {\pi} \left(S _ {h + 1}\right) \mid s, a\right) \approx_ {\sqrt {2 d} H \eta_ {0}} \left\langle \varphi (s, a), \tilde {\theta} _ {h} \right\rangle ,
$$
where, as before,  $\eta_0 = 5d_0\eta /\alpha$  . Since
$$
\underset {R \sim \mathcal {R} (s, a)} {\mathbb {E}} (R) = q ^ {\pi} (s, a) - \mathbb {E} (v ^ {\pi} (S _ {h + 1}) \mid s, a),
$$
letting  $\theta_h = \bar{\theta}_h - \tilde{\theta}_h$  satisfies (i) of Definition 3.1 with  $\kappa = \eta + \sqrt{2d} H\eta_0 = \eta + 5H\sqrt{2d} d_0\eta / \alpha$ .
To show (ii), take any  $f: \mathcal{S} \to [0, H]$  and  $h \in [H - 1]$ . As before,  $f$  is  $\alpha / (\sqrt{2d} H)$ -admissible, therefore Lemma 4.7 immediately provides  $\theta_h'$  satisfying the required conditions.
Therefore, the MDP is shown to be linear with misspecification  $\eta +\sqrt{2d} H\eta_0$  , and parameter bound  $L_{2}(4Hd_{0}\sqrt{2d} /\alpha +1)$
Sketch of the  $q^{\pi}$ -to-linear MDP conversion argument. We elaborate on the conversion to linear MDP mechanism presented in Section 3. As the basis of this argument is that an idealistic range-determining oracle is present, we note that this argument only serves as intuition and is otherwise tangential to our proof. Instead of a direct approach of learning this oracle, our proof argues that learning about this oracle happens whenever there is a need (performance shortfall) for it. A formal reduction to linear MDPs given this oracle however is fairly straightforward but cumbersome, with the caveat that the linear MDP will end up with  $dH$  (instead of  $d$ ) dimensional features. One would proceed by copying the features of each state  $s$  in stage  $h$  into the  $h^{\mathrm{th}}$  chunk of size  $d$  of this vector of size  $dH$  (the rest of the vector remains zero). A similar transformation is applied to all  $\theta_h(\pi)$ . Then,  $H$  copies are made of each high-enough-range state, with all possible stages (but keeping the feature vectors). These will be the states of the new MDP we construct. When a transition from state  $s$  leads to skipped states, the linear MDP returns with the copy of the first non-skipped state that has a stage counter of  $\mathrm{stage}(s) + 1$ , so that in this linear MDP the stage numbers are consecutive (as required by our definitions).  $q^{\pi}$ -realizability of this modified MDP is easy to show, and (as it has no low-range states) Proposition 3.4 can be used to show that the modified MDP is linear. To account for the fact that this new MDP may finish an episode in fewer than  $H$  steps due to the skips, we add a special, zero-reward, self-transitioning state called "episode-over". To ensure that the MDP stays linear, we extend the feature vectors of each state by a scalar 1, and a scalar indicator of being in this state, with all original features of the "episode-over" state defined to be zero. It is easy to see that this construction leads to a linear MDP with the desired action-value functions.
D Intuition behind our method and proof strategy from the perspective of ELEANOR [Zanette et al., 2020]
The starting point of our method is the ELEANOR algorithm, which is designed for linear MDPs. Similarly to SKIPPYELEANOR, ELEANOR solves an optimistic optimization problem inside a loop. The optimization problem computes optimistic estimates  $\bar{\theta}_t$  of the parameters of the MDP simultaneously for all  $t \in [H]$ , and in each iteration of the loop, more data is collected according to the policy that is optimal for the MDP defined by the estimated parameters. Initial estimates  $\hat{\theta}_t$  are computed via solving least-squares problems whose covariates are the features corresponding to state-action pairs  $(S_t, A_t)$  from all the data collected so far, while the corresponding least-squares targets are computed as the sum of the immediate reward  $R_t$  and the estimated value for  $S_{t+1}$ , computed from  $\bar{\theta}_{t+1}$ .  $\bar{\theta}_t$  is then optimistically chosen as the solution of the optimization problem, in the neighborhood (confidence ellipsoid) of  $\hat{\theta}_t$ , the solution to this least-squares problem. It is shown that this
optimistic choice of estimates results in an optimistic estimate of the value of  $\nu^{\star}$  of the initial state, and the regret is upper bounded in terms of the sum of elliptic potentials of the covariates.
This argument appears in our analysis too, with minor modifications due to our PAC-like setting (instead of aiming to bound the regret), leading to our final-iteration condition of Line 16 in Algorithm 1. Our Optimization Problem 4.10 is similar to ELEANOR's, and the parameters  $\bar{\theta}_t$  and  $\hat{\theta}_t$  have the same meaning. A key difference between the optimization problems of ELEANOR and SKIPPYELEANOR are how the least-squares targets are determined. For ELEANOR, it is the sum of the immediate reward  $R_{t}$  and its estimated value for  $S_{t + 1}$ ; with this target, only one on-policy rollout is required for each episode in order to get the least-squares parameter estimate for all  $H$  stages. In contrast, our least-squares targets are formed as the sum of  $R_{t} + \ldots + R_{t + i}$  and the estimated value for  $S_{t + i + 1}$ , where  $i$ , the number of stages "skipped", depends on the guess  $\hat{G}$ . The guess  $\hat{G}$  is selected only in Optimization Problem 4.10, and we do not know its value at the time of data collection, so we cannot know which stages will have to be skipped for each rollout. Therefore, (i) we need access to the rewards of the current policy at any stage (similarly to ELEANOR), and hence we run the current policy to any stage (including the last one); and (ii) perform rollouts with the fixed policy  $\pi^0$  (from any stage) to be able to estimate the reward  $R_{t} + \ldots + R_{t + i}$  collected while skipping over  $i$  stages (for any  $i$ ). To ensure this happens for every stage, we start Phase II from every stage  $k$ , resulting in the additional for loop in Line 6 of Algorithm 1 compared to ELEANOR. Finally, the randomization in Phase I is applied to make the optimization problem smooth, as described in Section 4.
One could analyze this algorithm similarly to the analysis of ELEANOR if it were not for the fact that the least-squares targets we just introduced are not realizable in general. We can, however, prove the realizability of certain components of the matrix-valued version of these targets,  $F$  (Lemma 4.9 and Corollary 4.11). This enables us to detect when the realizability of our least-squares targets fail, measure the direction (component) of the largest error, and learn from that. This is the job of Optimization Problem 4.12:  $\hat{F}_{\hat{G}\bar{\theta}}^{ki}$  corresponds to the matrix-valued empirical measurements of  $F$ , while the  $y_{\hat{G}\bar{\theta}}^{ki}$  are the average predictions of the same quantities. If the targets are realizable, which happens if we manage to skip the right number of stages), these matrices are very close; if not, the direction of their largest discrepancy tells us something about  $\bot (Q,i)$ , and allows us to learn.
Optimism ties all this together: either there is no shortfall between predicted and measured  $q$ -values (and we are done) or we grow the elliptical potential of  $X$  (the two cases present in the analysis of ELEANOR, Zanette et al. [2020]), or we grow the elliptical potential of  $Q$  (the new case due to the lack of realizability guarantees).
In this section we present the proof of Theorem 4.1. Recall that some quantities are defined in Appendix B.
E.1 Checking consistency
We introduce some lemmas to establish the required guarantees of the consistency checker. Their proofs, which rely on the usual least squares analysis techniques and covering arguments, are presented in Appendix H.
Lemma E.1. Let  $(k,i,\nu)$  be the outcome of Optimization Problem 4.12 any time during the execution of SKIPPYELEANOR, and let  $w = \mathrm{Proj}_{Z(Q,i)}\nu$  as in the algorithm. Then,
$$
w ^ {\top} \left(y _ {\hat {G} \bar {\theta}} ^ {k i} - \hat {F} _ {\hat {G} \bar {\theta}} ^ {k i}\right) w \geq v ^ {\top} \left(y _ {\hat {G} \bar {\theta}} ^ {k i} - \hat {F} _ {\hat {G} \bar {\theta}} ^ {k i}\right) v - \frac {\varepsilon}{d H ^ {2} \omega}
$$
Lemma E.2. There is an event  $\mathcal{E}_1$  that happens with probability at least  $1 - \zeta$ , such that under  $\mathcal{E}_1$ , during the execution of SKIPPYLEANOR, when the beginning of any iteration (Line 5) is executed, for any  $t \in [H-1]$ ,  $i \in [t+1:H]$ , for any  $\hat{G} \in \mathbf{G}$ ,  $\bar{\theta} \in \bar{\Theta}$ , and  $\nu, w \in \mathcal{B}(1)$ , for all  $(s, a) \in S_t \times [\mathcal{A}]$ ,
$$
\left| v _ {\parallel} ^ {\top} \left(\varphi (s, a) ^ {\top} \hat {\theta} _ {\hat {G} \bar {\theta}} ^ {t i} - \underset {\pi^ {0}, s, a} {\mathbb {E}} \bar {F} _ {\hat {G} \bar {\theta}} (S _ {i})\right) w \right| \leq \| \varphi (s, a) \| _ {X _ {m t} ^ {- 1}} \beta + \frac {\varepsilon}{d H ^ {2} \omega},
$$
where  $\bullet_{\parallel}$  denotes  $\bullet_{\parallel (Q,i)}$
The next lemma uses the average least-squares predictions' (capped) uncertainty term  $\bar{\sigma}_k^m$  (defined in Eq. (18)), where the average is taken over predictions from the state-action pair where Phase I of SKIPPYPOLICY  $(\cdot, \cdot, k)$  ends.
Lemma E.3. There is an event  $\mathcal{E}_2$  with probability at least  $1 - \zeta$ , such that under  $\mathcal{E}_1 \cap \mathcal{E}_2$ , during the execution of SKIPPYLEANOR, when Optimization Problem 4.12 is solved (Line 10), for  $(\hat{G}, \bar{\theta})$  as recorded in Line 5 for all  $k \in [H - 1]$ ,  $i \in [k + 1:H]$ , and  $\nu, w \in \mathcal{B}(1)$
$$
\left| v _ {\parallel} ^ {\top} \left(y _ {\hat {G} \bar {\theta}} ^ {k i} - \hat {F} _ {\hat {G} \bar {\theta}} ^ {k i}\right) w \right| \leq \bar {\sigma} _ {k} ^ {m} \beta + 3 \frac {\varepsilon}{d H ^ {2} \omega}
$$
Together, these lemmas can be used to show that the vector  $w$  derived from Line 10 in SKIPPYE-LEANOR is sufficiently aligned with both  $Z(Q,\cdot)$  and the subspace  $\mathrm{Proj}_{\perp (Q,\cdot)}$  projects to, which leads to the following important result:
Lemma E.4. Under the  $\mathcal{E}_1\cap \mathcal{E}_2$  , if Line 13 is executed any time during the execution of SKIPPYELEANOR (i.e., when the consistency check fails), then the resulting  $\dot{Q}$  continues to be a valid preconditioning.
From now on, our lemmas assume the high-probability events of Lemmas E.2 and E.3 hold, and therefore  $Q$  is a valid preconditioning at any time during the execution by Lemma E.4.
E.2 Sample complexity bounds
We bound the number of iterations of  $m$  that SKIPPYELEANOR can execute. The proof of the following lemma is presented in Appendix I:
Lemma E.5. Throughout the execution of SKIPPYELEANOR,  $m \leq m_{max}$ .
Note that throughout the execution of SKIPPYELEANOR,  $m' \leq m_{\max}'$ . As  $m' - m$  equals the number of times Line 13 is executed, i.e., the sum of sequence lengths corresponding to  $Q$ , by Lemma 4.4,
Corollary E.6. Under  $\mathcal{E}_1\cap \mathcal{E}_2$ , SKIPPYELEANOR returns with a policy before exiting the while loop of Line 3, and as each iteration executes Hn trajectories in Line 8, the number of interactions of SKIPPYELEANOR with the MDP is bounded by  $\tilde{\mathcal{O}}\left(H^{11}d^7 /\varepsilon^2\right)$ .
E.3 Performance guarantee
We next consider the  $m^{\mathrm{th}}$  iteration of SKIPPYELEANOR under the assumption that the consistency check passes, that is, Line 16 is executed. We intend to guarantee the performance of  $\pi^{mH}$  in terms of  $\sum_{t=1}^{H} \bar{\sigma}_k^m$ , given that the optimization value  $x$  satisfies  $x \leq \bar{\sigma}_k^m \beta \omega + 3 \frac{\varepsilon}{dH^2}$  (which follows from the execution reaching Line 16). Next we introduce variants of  $c_{ki}^j$  and  $\tilde{c}_{ki}^j$  (Eq. (20)) which act, instead of the data collected during the execution of the algorithm, on a trajectory  $(S_h, A_h, R_h)_{h \in [H]}$  and corresponding stage mapping  $p$  obtained by an independent run of SKIPPYPOLICY, which will be clear from the context:  $\tilde{c}_{ki} = \mathbb{1}\{p(k) < i\}$ , and
$$
c _ {k i} = \mathbb {1} \left\{p (k) <   i \text {a n d} \left\| \varphi \left(S _ {p (k)}, A _ {p (k)}\right) \right\| _ {X _ {m, p (k)} ^ {- 1}} <   2 (\beta \omega d H) ^ {- 1} \text {a n d} \left\langle \varphi \left(S _ {p (k)}, A _ {p (k)}\right), \bar {\theta} _ {p (k)} \right\rangle \geq 0 \right\}.
$$
Remark E.7. In our analysis we rely on the obvious fact that the laws of the trajectories of SKIPPYPOLICY( $\hat{G}, \bar{\theta}, k$ ) and SKIPPYPOLICY( $\hat{G}, \bar{\theta}, k + 1$ ) are the same until stage  $p(k + 1)$  (as the policies are the same until then), for any parameters  $\hat{G}$  and  $\bar{\theta}$ . This includes  $S_{p(k + 1)}$  but not  $A_{p(k + 1)}$  if  $p(k + 1) \leq H$ , and includes the whole trajectory ending with  $R_H$  otherwise.
We prove the following using induction on  $k = H, \dots, 1$  in Appendix J:
Lemma E.8. There is an event  $\mathcal{E}_3$  with probability at least  $1 - 3\zeta$ , such that under  $\mathcal{E}_1 \cap \mathcal{E}_2 \cap \mathcal{E}_3$ , during the execution of SKIPPYELEANOR, whenever Line 16 is executed, for  $(\hat{G}, \bar{\theta})$  as recorded in Line 5 of the current iteration, for  $k \in [H]$ ,
$$
\bar {C} ^ {k} := \underset {\pi^ {m k}, s _ {1}} {\mathbb {E}} \tilde {c} _ {k, H + 1} C _ {\hat {G} \bar {\theta}} \left(S _ {p (k)}\right) \leq \underset {\pi^ {m H}, s _ {1}} {\mathbb {E}} \sum_ {u = p (k)} ^ {H} R _ {u} + 2 \sum_ {i = k} ^ {H} \bar {\sigma} _ {k} ^ {m} \beta \omega d H + 4 \left(H - k + 1\right) \frac {\varepsilon}{H}. \tag {22}
$$
As  $S_{1} = s_{1}$  is fixed and  $\tau(s_{1}) = 1$ , we get the following corollary, which shows that the value  $C_{\hat{G}\bar{\theta}}$  of the solution  $(\hat{G},\bar{\theta})$  of Optimization Problem 4.10 can be used as a lower bound on the value of the policy  $\pi^{mH}$  up to the uncertainty and some  $\varepsilon$  terms: