SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, we check if the data collected is consistent with our estimates  $\hat{G}$  and  $\bar{\theta}$ , by calculating the maximal discrepancy of the estimates of the action-value difference at the last non-skipped state of  $\pi^{mk} = \text{SKIPPYPOLICY}(\hat{G},\bar{\theta},k)$  and that of the fixed skipping policy  $\pi^0$  in different directions in the parameter space. If the discrepancy is too large for any  $k$ , we add the discrepancy-maximizing direction to  $Q$  and throw away the data collected in this (i.e., the  $m^{th}$ ) iteration; this is achieved by reducing the iteration counter  $m$  by 1. On the other hand, if the discrepancy is small enough, we can guarantee that the gap between the value of  $\pi^{mH}$  and  $\nu^{\star}(s_1)$  scales with how much new information we collected, thus the algorithm can terminate returning this policy if this term is sufficiently small (which it eventually has to be).
The following theorem shows that with high probability, SKIPPYELEANOR finds a near-optimal policy after polynomially many interactions with the MDP. Rhe proof sketch is provided in Section 5, while our method and proof strategy is explained from the perspective of ELEANOR in Appendix D.
Theorem 4.1. With probability at least  $1 - \zeta$ , SKIPPYELEANOR interacts with the MDP for at most  $\tilde{O}\left(H^{11}d^7 /\varepsilon^2\right)$  many steps, before returning a policy  $\pi$  that satisfies  $\nu^{\star}(s_1)\leq \nu^{\pi}(s_1) + \varepsilon$ .
4.1 Preconditioning: the enclosing ellipsoid
In this section we give the technical details about the effects of using the matrix  $Q_{h}$  describing an enclosing ellipsoid for  $\Theta_h$  (see Lemma 4.3) as preconditioning the features.
Definition 4.2 (Valid preconditioning).  $\mathcal{Q} = (Q_h)_{h\in [H]}$  is a valid preconditioning matrix sequence if for all  $h\in [H]$
$$
Q _ {h} = \left(L _ {2} ^ {- 2} I + \sum_ {v \in C _ {h}} v v ^ {\top}\right) ^ {- 1 / 2} \tag {4}
$$
for some sequence  $C_h = (v_1, \ldots, v_n)$  of vectors in  $\mathbb{R}^d$  such that for all  $1 \leq i \leq n$ ,
$$
\sup  _ {\theta \in \Theta_ {h}} | \langle \theta , v _ {i} \rangle | \leq 1 \quad a n d \quad \left\| \left(L _ {2} ^ {- 2} I + \sum_ {j = 1} ^ {i - 1} v _ {j} v _ {j} ^ {\top}\right) ^ {- \frac {1}{2}} v _ {i} \right\| _ {2} ^ {2} \geq \frac {1}{2} \quad a n d \quad \| v \| _ {2} \leq L _ {3}, \tag {5}
$$
where  $L_{3}$  is some fixed polynomial of the problem parameters  $(d,H,\varepsilon^{-1},\log \zeta^{-1},\log L_1,\log L_2)$ . (see Eq. (35) for its precise value).
For a valid preconditioning  $Q$  and some  $h \in [H]$ , let  $Z(Q, h)$  be the linear subspace spanned by those eigenvectors of  $Q$  whose corresponding eigenvalues are at least  $L_3^{-2}$ . Let  $\mathrm{Proj}_{Z(Q, h)}$  be the orthogonal projection matrix onto this subspace.
Sometimes it will be convenient to precondition the features and parameters so that the enclosing ellipsoid is transformed to a ball of controlled radius (as Lemma 4.3 will show). To this end, introduce for all  $h \in [H]$  and  $(s, a, b) \in S_h \times [\mathcal{A}] \times [\mathcal{A}]$  the following:
$$
\varphi_ {Q} (s, a) = Q _ {h} \varphi (s, a), \quad \varphi_ {Q} (s, a, b) = Q _ {h} \varphi (s, a, b)
$$
$$
\theta_ {h} ^ {Q} (\pi) = Q _ {h} ^ {- 1} \theta_ {h} (\pi), \quad \Theta_ {h} ^ {Q} = \left\{\theta_ {h} ^ {Q} (\pi): \pi \in \Pi \right\} = \left\{Q _ {h} ^ {- 1} \theta : \theta \in \Theta_ {h} \right\} \tag {6}
$$
$$
\hat {q} ^ {\pi} (s, a) = \left\langle \varphi (s, a), \theta_ {h} (\pi) \right\rangle = \left\langle \varphi_ {Q} (s, a), \theta_ {h} ^ {Q} (\pi) \right\rangle \text {f o r a l l} \pi \in \Pi .
$$
The next lemma (proved in Appendix F) shows that for all  $h \in [H]$ ,  $Q_h$  defines an enclosing ellipsoid for  $\Theta_h$ ; that is,  $\Theta_h \subset \{\theta : \| \theta \|_{Q_h^{-2}} \leq \sqrt{d_1 + 1}\}$ .
3Note that  $Q_{h},h\in [H]$  is invertible by construction.
Lemma 4.3. Let  $d_{1} = 4d\log (1 + 16L_{3}^{4}L_{2}^{4}) = \tilde{O} (d)$ . Then, for any valid preconditioning  $Q$  and  $h\in [H]$ ,
$$
\sup  _ {\theta \in \Theta_ {h}} \| \theta \| _ {Q _ {h} ^ {- 2}} = \sup  _ {\theta \in \Theta_ {h} ^ {Q}} \| \theta \| _ {2} \leq \sqrt {d _ {1} + 1}.
$$
Clearly, every time a new vector is added to  $C_h$ , the enclosing ellipsoid  $\{\theta : \| \theta \|_{Q_h^{-2}} \leq \sqrt{d_1 + 1}\}$  shrinks (as a positive semidefinite matrix is added to  $Q_h^{-2}$ ). The following lemma (also proved in Appendix F) uses an elliptical potential argument to bound the number of times this can happen.
Lemma 4.4. For any valid preconditioning  $Q$ , for all  $h \in [H]$ , the length of sequence  $C_h$  corresponding to  $Q_h$  according to Definition 4.2 is at most  $d_1$ .
Near-optimal design for  $\Theta_h^Q$ . As  $Q_{h}$  only provides an enclosing ellipsoid for  $\Theta_h$ , we introduce an (unknown) ellipsoid that aligns better with  $\Theta_h^Q$ . For all  $h \in [H]$ , fix a set  $G_h^Q$  of policies of size  $d_0 := 4d \log \log(d) + 16$ , together with a probability distribution  $\rho_h^Q$  on  $G_h^Q$ , such that  $(G_h^Q, \rho_h^Q)$  is a near-optimal design for  $\Theta_h^Q$  (i.e., satisfying Definition F.1). The existence of such a near-optimal design follows from [Todd, 2016, Part (ii) of Lemma 3.9].
We apply  $G_h^Q$  to define a cruder version of range that depends only on a small set of policies, and can therefore be succinctly parametrized to inform SKIPPYPOLICY:
$$
\operatorname {r a n g e} _ {Q} (s) = \max  _ {\pi \in G _ {h} ^ {Q}} \max  _ {i, j \in [ \mathcal {A} ]} \left\langle \varphi (s, i, j), \theta_ {h} (\pi) \right\rangle \quad \text {f o r a l l} h \in [ H ], s \in S _ {h}. \tag {7}
$$
range $_Q$  is easy to estimate, and can be used to bound the range function (proved in Appendix F): Proposition 4.5. For all  $s \in S$  and  $Q \in PD^H$ ,  $\mathrm{range}(s) \leq \sqrt{2}d\mathrm{range}_Q(s)$ .
4.2 Linearly realizable functions
$q^{\pi}$ -realizability (Definition 3.2) implies the linearity of many more functions than the action-value functions. In this section we characterize an interesting set of such functions, whose (approximate) linearity plays a crucial role in our algorithm and analysis, as their parameters can be conveniently estimated by least squares using the features. We rely on functions  $f: S_h \to \mathbb{R}$  (for some  $h \in [H]$ ) being small for all states, relative to the states' range  $O$ -value:
Definition 4.6. For any  $h \in [H]$ ,  $f: S_h \to \mathbb{R}$  is  $\alpha$ -admissible for some  $\alpha > 0$  if for all  $s \in S_h$ ,  $|f(s)| \leq \mathrm{range}_Q(s) / \alpha$ .
The key observation is that expected (admissible)  $f$  values are linearly realizable.
Lemma 4.7 (Admissible-realizability). If  $f: S_h \to \mathbb{R}$  is  $\alpha$ -admissible then it is realizable, that is, for all  $t \in [h - 1]$  and  $\pi \in \Pi$ , there exists some  $\tilde{\theta} \in \mathbb{R}^d$  with  $\| \tilde{\theta} \|_2 \leq 4d_0L_2 / \alpha$  such that for all  $(s, a) \in S_t \times [\mathcal{A}]$ ,
$$
\underset {\pi , s, a} {\mathbb {E}} f (S _ {h}) \approx_ {\eta_ {0}} \left\langle \varphi (s, a), \tilde {\theta} \right\rangle \quad \text {w h e r e} \eta_ {0} = 5 d _ {0} \eta / \alpha .
$$
The proof relies on constructing a set of policies that at states  $s \in S_h$  take a higher value action as opposed to a lower one with a certain probability, configured such that the expected action-value difference of some pairs within the set of policies is (approximately) proportional to  $f(s)$ . Thus, a linear combination of the action-values of policies in this set are also (approximately) proportional to  $f(s)$ . The statement of the lemma then follows from setting  $\tilde{\theta}$  to the corresponding linear combination of the policies' parameters. The full proof is presented in Appendix G.
Next, we define matrix-valued functions with a special admissibility guarantee even when the underlying scalar-valued function does not satisfy any non-trivial admissibility criterion. We introduce a guess on the near-optimal design parameters that define range  $Q$  (Eq. (7)) for some valid preconditioning  $Q$ :
Definition 4.8. For  $h \in [2:H]$ , fix some arbitrary order of the policies in the set  $G_h^Q$  (recall that this set is the support of the near-optimal design for  $\Theta_h^Q$ ). Let the parameter of the  $i^{th}$  policy in  $G_h^Q$  be  $\vartheta_h^i$  for  $i \in [d_0]$ . Call a "guess" of these parameters  $\hat{G} = (\hat{G}_h)_{h \in [2:H]} = (\hat{\vartheta}_h^i)_{h \in [2:H], i \in [d_0]}$  "valid", if for all  $h \in [2:H], i \in [d_0]$ ,  $\hat{\vartheta}_h^i \in \mathcal{B}(\sqrt{d_1 + 1})$ . Let the set of valid guesses be  $\mathbf{G}$ . By Lemma 4.3,  $(\vartheta_h^i)_{h \in [2:H], i \in [d_0]} \in \mathbf{G}$ , that is, it is a valid guess, and we call this the "correct" guess.
Note that while  $G_h^Q$  contains policies,  $\mathbf{G}$  and its elements (commonly denoted by  $\hat{G}$ ) contain policy parameter vectors.
From a guess  $\hat{G} = (\hat{\vartheta}_h^i)_{h\in [2:H],i\in [d_0]}$  we can calculate corresponding guesses of the range  $Q$  -values:
$$
\mathrm {r a n g e} _ {Q} ^ {\hat {G}} (s) = \max _ {k \in [ d _ {0} ]} \max _ {i, j \in [ \mathcal {A} ]} \left\langle \varphi_ {Q} (s, i, j), \hat {\vartheta} _ {\mathrm {s t a g e} (s)} ^ {k} \right\rangle \qquad \mathrm {f o r a l l} h \in [ 2: H ], s \in \mathcal {S} _ {h}.
$$
Note that for any  $h \in [2:H]$  and  $s \in S_h$ ,  $\mathrm{range}_Q^{\hat{G}}(s) = \mathrm{range}_Q(s)$  if  $\hat{G}$  is the correct guess for stage  $h$ .
Let  $\bar{\varphi}_Q(s)$  be the unit vector in the direction of the largest feature difference between actions in  $s$  and the zero vector if all feature vectors are the same (see Eq. (27) for a formal definition). Then, for any  $\hat{G} \in \mathbf{G}$ ,  $h \in [2:H]$ , and  $f: S_h \to [-H,H]$ , let
$$
\mathbf {f} (s) = \bar {\varphi} _ {Q} (s) \bar {\varphi} _ {Q} (s) ^ {\top} \min  \left\{1, \operatorname {r a n g e} _ {Q} ^ {\hat {G}} (s) \frac {\sqrt {2 d} H}{\varepsilon} \right\} f (s) \quad \text {f o r} s \in \mathcal {S} _ {h}.
$$
For such  $\mathbf{f}:S_h\to \mathbb{R}^{d\times d}$ , we adopt the notation  $a^\top \mathbf{f}b$  for any  $a,b\in \mathbb{R}^d$  to denote the function  $s\in S_h\mapsto a^\top \mathbf{f}(s)b$ , and similarly,  $\operatorname {Tr}(\mathbf{f})$  to denote the function  $s\in S_h\mapsto \operatorname {Tr}(\mathbf{f}(s))$ .
Let  $\mathrm{Proj}_{\| (Q,h)}$  be the projection matrix onto the linear subspace spanned by those eigenvectors of the design matrix  $V(G_h^Q,\rho_h^Q)$  (defined in Eq. (25)) whose corresponding eigenvalues are at least  $\gamma$  (for some  $\gamma >0$  specified in Appendix B). Intuitively, this is the subspace where  $\Theta_h^Q$  has a sufficiently large width. Let  $\mathrm{Proj}_{\perp (Q,h)}$  be the projection to the orthogonal complement subspace. For any  $\nu \in \mathbb{R}^d$ , we write  $\nu_{\| (Q,h)}$  and  $\nu_{\perp (Q,h)}$  for  $\mathrm{Proj}_{\| (Q,h)}\nu$  and  $\mathrm{Proj}_{\perp (Q,h)}\nu$ , respectively.
We are now ready to state our special admissibility guarantee, which is proved in Appendix G. Let  $\alpha = \tilde{O} (\varepsilon /(d^{1.5}H^2))$  be as in Eq. (16).
Lemma 4.9. For any  $h \in [2:H]$ ,  $\hat{G} \in \mathbf{G}$ , any function  $\mathbf{f}$  constructed as above from some  $f: S_h \to [-H, H]$ , and any  $v, w \in \mathcal{B}(1)$ ,  $v_{\|(Q,h)}^{\top} \mathbf{f}w$  is  $\alpha$ -admissible. Furthermore, if  $\hat{G} = (\vartheta_h^i)_{h \in [2:H], i \in [d_0]}$  (the correct guess),  $\operatorname{Tr}(\mathbf{f})$  is also  $\alpha$ -admissible.
4.3 Least-squares targets and Optimization Problem 4.10
Recall that SKIPPYELEANOR estimates action-values of states by first adding the estimated one-step reward and the estimated value-function of the next state in the reduced MDP (where low-range states are skipped). Due to the linearity of  $q^{\pi}$ -values, these can be used as target variables of a least-squares estimator to estimate the policy parameters. This estimator is only guaranteed to be accurate if the right (low-range) states are skipped; otherwise, we will argue in Section 4.4 that a discrepancy is detected and it is handled by changing the preconditioning  $Q$ . Finally, to ensure optimism, we select parameter estimates that lead to the largest estimated policy values. The whole estimation process leads to Optimization Problem 4.10, which we define in this section along with the functions that it uses as least-square targets. Each estimation is for a particular stage  $h$  and may use the estimates  $\bar{\theta}_i$  of Optimization Problem 4.10 for stages  $i > h$ . In this subsection, we consider the  $m^{\text{th}}$  iteration of the optimization called by SKIPPYELEANOR, and consider  $Q$  fixed. As a shorthand, we introduce the following notation for  $l \in [m]$ ,  $j \in [n]$ ,  $k \in [H]$ :
$p(lkj) = p^{lkj}(k)$  as recorded in Line 9 of Algorithm 1, and
$$
S _ {\mathrm {p} (k)} ^ {l k j} = S _ {p ^ {l k j} (k)} ^ {l k j}, A _ {\mathrm {p} (k)} ^ {l k j} = A _ {p ^ {l k j} (k)} ^ {l k j}, R _ {\mathrm {p} (k)} ^ {l k j} = R _ {p ^ {l k j} (k)} ^ {l k j}, \varphi_ {t} ^ {l k j} = \varphi (S _ {t} ^ {l k j}, A _ {t} ^ {l k j}), \varphi_ {\mathrm {p} (k)} ^ {l k j} = \varphi (S _ {\mathrm {p} (k)} ^ {l k j}, A _ {\mathrm {p} (k)} ^ {l k j}).
$$
We collect the set of  $(l,k,j)$  tuples for which the  $k^{\mathrm{th}}$  skippy step lands at stage  $t$ , for  $t\in [H]$ , as
$$
\mathbf {I} ^ {m} (t) = \{(l, k, j): l \in [ m - 1 ], j \in [ n ], k \in [ H ], \mathrm {p} (l k j) = t \}
$$
Note in particular that here  $l \in [m - 1]$ , so  $\mathbf{I}^m$  only considers data collected prior to iteration  $m$ .
To estimate the parameters  $\hat{G}$  and  $\bar{\theta}$ , we consider (simulated) trajectories of SKIPPYPOLICY starting from stage  $t$ . For simplicity, we suppress the dependence of quantities on  $\hat{G}$  and  $\bar{\theta}$ , which will be brought back later. The skipping probability  $1 - \tau$ , the policy  $\pi^{+}$  (to be also used in SKIPPYPOLICY), and corresponding clipped action-value estimates are defined as
$$
\tau (s) = \min  \left\{1, \operatorname {r a n g e} _ {Q} ^ {\hat {G}} (s) \frac {\sqrt {2 d} H}{\varepsilon} \right\} \quad \text {i f} \operatorname {s t a g e} (s) > 1, \text {a n d} \tau \left(s _ {1}\right) = 1; \tag {8}
$$
$$
\pi^{+}(s_{i}) = \operatorname *{arg  max}_{a\in [  \mathcal{A}  ]}\left\langle \varphi (s_{i},a),\bar{\theta}_{i}\right\rangle ,\qquad \qquad C(s_{i}) = \operatorname {clip}_{[0,H]}\left\langle \varphi (s_{i},\pi^{+}(s_{i})),\bar{\theta}_{i}\right\rangle .
$$
Let  $s_i \rightarrow = (s_i, a_i, r_i, \dots, s_H, a_H, r_H) \in S_i \times [\mathcal{A}] \times [0,1] \times \dots \times [0,1]$  be any ending of a trajectory. For  $s_{t+1} \rightarrow$ , let  $I$  be the (random) index of the first state that is not skipped by SKIPPYPOLICY with the above  $\tau$  (or  $H + 1$ , if such an index does not exist). Then the estimated policy value of SKIPPYPOLICY from stage  $t$  is
$$
\mathbb {E} _ {I} \left[ \sum_ {u = t} ^ {I - 1} r _ {u} + \mathbb {1} \left\{I <   H + 1 \right\} C (s _ {I}) \right],
$$
the sum of rewards along the skipped states plus the policy-value estimate from stage  $I$ . It follows from Corollary 4.11 below (proved based on Lemma 4.9) that if  $\mathrm{range}_Q^G$  is an accurate estimate of  $\mathrm{range}_Q$ , then this quantity decomposes into terms that are linearly expressible using the features. Therefore, we use such quantities as least-square targets. Indeed, writing out the expectation, we can re-express the estimated policy value as the sum of all rewards  $\sum_{u=t}^{H} r_u$  plus a correction term  $E^{\rightarrow}(s_{t+1}\rightarrow)$  defined as
$$
E ^ {\rightarrow} \left(s _ {i} \rightarrow\right) = \sum_ {j = i} ^ {H} D \left(s _ {j} \rightarrow\right) \tau \left(s _ {j}\right) \prod_ {j ^ {\prime} = i} ^ {j - 1} \left(1 - \tau \left(s _ {j ^ {\prime}}\right)\right) \text {w h e r e} D \left(s _ {i} \rightarrow\right) = C \left(s _ {i}\right) - \sum_ {u = i} ^ {H} r _ {u} \quad \text {f o r} i > 1. \tag {9}
$$
The next optimization problem aims to find optimistic parameters yielding the largest estimated action-value function for  $s_1$ , where  $\bar{\theta}$  is in the confidence ellipsoid of the least-squares estimates  $\hat{\theta}$ .
Optimization Problem 4.10 (for iteration  $m$ ). For input state  $s$ , with  $\beta$  defined in Appendix B (emphasizing the dependence of functions defined above on  $\hat{G}$  and  $\bar{\theta}$  by adding them as subscripts):
$$
\underset {\hat {G} \in \mathbf {G}, \bar {\theta} _ {t} \in \mathcal {B} (4 d _ {0} H L _ {2} / \alpha) f o r t \in [ H ]} {\arg \max } C _ {\hat {G} \bar {\theta}} (s _ {1}) \quad s u b j e c t t o, f o r a l l t \in [ H ]
$$
$$
X _ {m t} = \lambda I + \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \varphi_ {t} ^ {l k j ^ {\top}}, \left\| \bar {\theta} _ {t} - \hat {\theta} _ {t} \right\| _ {X _ {m t}} \leq \beta H, \hat {\theta} _ {t} = X _ {m t} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \Big (\underbrace {E _ {\hat {G} \bar {\theta}} ^ {\rightarrow} (S _ {t + 1} ^ {l k j} , \ldots , R _ {H} ^ {l k j}) + \sum_ {u = t} ^ {H} R _ {u} ^ {l k j}} _ {\text {l e a s t - s q u a r e s t a r g e t}} \Big).
$$
Since our realizability results in Section 4.2 only apply to functions defined at a given stage (as only memoryless policies are  $q^{\pi}$ -realizable), to be able to show that the least-squares targets are linearly realizable, we first decompose  $E^{\rightarrow}(s_{i\rightarrow})$  ( $i \in [2:H]$ ) to directly express the effect of each stage in the trajectory (backwards): defining  $E(s_{i\rightarrow}) = E^{\rightarrow}(s_{i\rightarrow}) - E^{\rightarrow}(s_{i + 1\rightarrow})$  (for convenience, we use the notation  $E^{\rightarrow}(s_{H + 1\rightarrow}) = 0$ ), we easily obtain
$$
E ^ {\rightarrow} \left(s _ {i} \rightarrow\right) = \sum_ {j = i} ^ {H} E \left(s _ {j} \rightarrow\right) \quad \text {a n d} \quad E \left(s _ {i} \rightarrow\right) = \tau \left(s _ {i}\right)\left(D \left(s _ {i} \rightarrow\right) - E ^ {\rightarrow} \left(s _ {i + 1} \rightarrow\right)\right). \tag {10}
$$
Next we define matrix-valued functions, whose trace equals  $E(s_i \rightarrow)$ , that have the same form as  $\mathbf{f}$  in Section 4.2, for which Lemma 4.9 applies. This is crucial in establishing optimism of Optimization Problem 4.10, as well as learning from instances where we detect that  $E \rightarrow$  is not realizable in Optimization Problem 4.12. To this end, let
$$
F (s _ {i} \rightarrow) = \bar {\varphi} _ {Q} (s _ {i}) \bar {\varphi} _ {Q} (s _ {i}) ^ {\top} E (s _ {i} \rightarrow) \quad \text {a n d} \quad \bar {F} (s _ {i}) = \mathbb {E} _ {\pi^ {0}, S _ {i}} \left[ F (s _ {i}, A _ {i}, \dots , R _ {H}) \right] \quad \text {f o r} s _ {i} \in S _ {i}.
$$
Let  $\bar{\Theta} = (\mathcal{B}(4d_0HL_2 / \alpha))^H$  denote the base set for the variables  $\bar{\theta}_t$  in Optimization Problem 4.10. As  $\bar{F}$  is of the same form as  $\mathbf{f}$ , we can apply Lemma 4.9 and then Lemma 4.7 to arrive at the following:
Corollary 4.11. For any  $\hat{G} \in \mathbf{G}$ ,  $\bar{\theta} \in \bar{\Theta}$ ,  $\nu, w \in \mathcal{B}(1)$ , and for any  $t \in [H - 1]$ ,  $i \in [t + 1:H]$ , there exists some  $\tilde{\theta}_{ti} \in \mathbb{R}^d$  with  $\left\| \tilde{\theta}_{ti} \right\|_2 \leq 4d_0L_2 / \alpha = 1/\sqrt{\lambda}$  such that for all  $(s,a) \in S_t \times [\mathcal{A}]$ , where  $\eta_0$  is defined in Lemma 4.7.
$$
\mathbb {E} _ {\pi^ {0}, s, a} \left[ v _ {\| (Q, i)} ^ {\top} \bar {F} _ {\hat {G} \bar {\theta}} (S _ {i}) w \right] \approx_ {\eta_ {0}} \left\langle \varphi (s, a), \tilde {\theta} _ {t i} \right\rangle . \tag {11}
$$
Furthermore, if  $\hat{G}$  is the correct guess, there exists some  $\tilde{\theta}_{ti}^{\prime} \in \mathbb{R}^{d}$  with  $\left\| \tilde{\theta}_{ti}^{\prime} \right\|_{2} \leq 4d_{0}L_{2} / \alpha$  such that for all  $(s,a) \in S_{t} \times [\mathcal{A}], \mathbb{E}_{\pi^{0},s,a}[E_{\hat{G}\bar{\theta}}(S_{i},\dots,R_{H}))] = \mathbb{E}_{\pi^{0},s,a}[\mathrm{Tr}(\bar{F}_{\hat{G}\bar{\theta}}(S_{i}))] \approx_{\eta_{0}}\bigl{\langle}\varphi (s,a),\tilde{\theta}_{ti}^{\prime}\bigr{\rangle}$ .
4.4 Checking consistency
Considering the  $m^{\text{th}}$  iteration of SKIPPYELEANOR, we want to verify if the estimated targets of Optimization Problem 4.10 are accurate (and learn if a discrepancy is detected), by using Corollary 4.11 on the targets' decomposition into  $F$ -functions. We filter the data collected in the  $m^{\text{th}}$  iteration with the indicator  $\tilde{c}_{ki}^{j} = \mathbb{1}\left\{\mathrm{p}(mkj) < i\right\}$  for  $j \in [n]$ ,  $k \in [H + 1]$ ,  $i \in [H + 1]$ , and further constrain
this by another indicator  $c_{ki}^{j}$  (defined in Appendix B) that requires the data-point's least-squares uncertainty term to be sufficiently low, and the prediction non-negative (the contribution of the rest of the data will be analyzed separately). Next, we define the least-squares solution for estimating the matrix-valued  $F$ , as well as the empirical average prediction and realization of  $F$  on the data collected in the  $m^{\text{th}}$  round. For any  $i \in [2:H]$ ,  $k \in [i-1]$  (recall that  $\otimes$  denotes the tensor product):
$$
\hat {\theta} _ {\hat {G} \bar {\theta}} ^ {t i} = X _ {m t} ^ {- 1} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \otimes F _ {\hat {G} \bar {\theta}} \left(S _ {i} ^ {l k j}, \dots , R _ {H} ^ {l k j}\right) \quad \text {f o r} t \in [ i - 1 ]
$$
$$
y _ {\hat {G} \bar {\theta}} ^ {k i} = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} \varphi_ {\mathrm {p} (k)} ^ {m k j ^ {\top}} \hat {\theta} _ {\hat {G} \bar {\theta}} ^ {\mathrm {p} (m k j), i} \quad \hat {F} _ {\hat {G} \bar {\theta}} ^ {k i} = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} F _ {\hat {G} \bar {\theta}} \left(S _ {i} ^ {m k j}, \dots , R _ {H} ^ {m k j}\right) \tag {12}
$$
In Appendix E.1, it is established via the usual least-squares analysis techniques and covering arguments, that with high probability the norm of the product of the matrix  $y_{\hat{G}\bar{\theta}}^{ki} - \hat{F}_{\hat{G}\bar{\theta}}^{ki}$  and the projection matrix  $\mathrm{Proj}_{\| (Q,i)}$  is small (Lemmas E.2 and E.3). The next optimization problem tests if this is true in arbitrary directions:
Optimization Problem 4.12 (Consistency check). Input:  $(\hat{G},\bar{\theta})$
$$
\operatorname *{arg  max}_{k\in [H - 1],i\in [k + 1:H],v\in \mathbb{R}^{d}:||v||_{2} = 1}}v^{\top}\left(y_{\hat{G}\bar{\theta}}^{ki} - \hat{F}_{\hat{G}\bar{\theta}}^{ki}\right)v
$$
Lemma E.1 shows that the projection  $w = \operatorname{Proj}_{Z(Q,i)} \nu$  is close to  $\nu$ , where  $\nu$  is the outcome of Optimization Problem 4.12. Also, Lemmas E.1-E.3 imply that if the consistency check fails (i.e., Line 13 is executed because the value of Optimization Problem 4.12 is large), then  $w$  aligns well with the subspace  $\operatorname{Proj}_{\perp(Q,i)}$  projects to, and therefore  $Q$  stays a valid preconditioning after appending  $w$  to the list of values  $\bar{Q}$  is calculated from (Lemma E.4). Thus,  $Q$  is always a valid preconditioning.
5 Proof overview