SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary E.9. Under  $\mathcal{E}_1\cap \mathcal{E}_2\cap \mathcal{E}_3$  , the value of Optimization Problem 4.10 with the solution  $(\hat{G},\bar{\theta})$  satisfies
$$
C _ {\hat {G} \bar {\theta}} (s _ {1}) = \bar {C} ^ {1} \leq \underset {\pi^ {m H}, s _ {1}} {\mathbb {E}} \sum_ {u = 1} ^ {H} R _ {u} + 2 \sum_ {i = 1} ^ {H} \bar {\sigma} _ {k} ^ {m} \beta \omega d H ^ {2} + 4 \varepsilon = v ^ {\pi^ {m H}} (s _ {1}) + 2 \sum_ {i = 1} ^ {H} \bar {\sigma} _ {k} ^ {m} \beta \omega d H ^ {2} + 4 \varepsilon .
$$
E.4 Optimism of Optimization Problem 4.10
The following establishes the optimistic property, that is, that the value of Optimization Problem 4.10 competes with  $\nu^{\star}(s_1)$ . The proof relies on the fact that the correct guess  $\hat{G}$  and a good choice of  $\bar{\theta}$  are feasible for the optimization problem, combined with the fact that this  $\bar{\theta}$  induces a policy  $\pi = \text{SKIPPYPOLICY}(\hat{G},\bar{\theta},H)$  that takes action-value maximizing actions according to a very accurate approximation of action-values almost everywhere. In fact, it only skips states whose range is at most  $\varepsilon / H$ . The proof is presented in Appendix K.
Lemma E.10. There is an event  $\mathcal{E}_4$  with probability at least  $1 - \zeta$ , such that under  $\mathcal{E}_1 \cap \mathcal{E}_2 \cap \mathcal{E}_4$ , throughout the execution of SKIPPYLEANOR, the value of Optimization Problem 4.10 is at least  $v^{\star}(s_1) - 2\varepsilon$ .
Proof of Theorem 4.1. We combine Lemma E.10 with Corollary E.9, Corollary E.6, and the fact that the condition of Line 16 is satisfied when SKIPPYELEANOR returns with a policy, to get that under  $\mathcal{E}_1\cap \mathcal{E}_2\cap \mathcal{E}_3\cap \mathcal{E}_4$  , that is, with probability at least  $1 - 6\zeta$  , SKIPPYELEANOR interacts with the MDP for at most  $\tilde{\mathcal{O}}\left(H^{11}d^7 /\varepsilon^2\right)$  many steps, before returning with the policy  $\pi^{mH}$  that satisfies
$$
v ^ {\star} (s _ {1}) \leq C _ {\hat {G} \bar {\theta}} (s _ {1}) + 2 \varepsilon \leq v ^ {\pi^ {m H}} (s _ {1}) + 2 \sum_ {i = 1} ^ {H} \bar {\sigma} _ {k} ^ {m} \beta \omega d H ^ {2} + 6 \varepsilon \leq v ^ {\pi^ {m H}} (s _ {1}) + 8 \varepsilon ,
$$
where the final inequality follows from the fact that when SKIPPYELEANOR returns in Line 17,  $\sum_{k=1}^{H} \bar{\sigma}_k^m \leq \varepsilon / (\beta \omega dH^2)$ . By scaling the parameters, this finishes the proof of Theorem 4.1.
F Deferred definitions and proofs for Section 4.1
Proof of Lemma 4.3. For any  $\theta \in \Theta_h^Q$ , it holds that  $\theta = Q_h^{-1}\hat{\theta}$  for some  $\hat{\theta} \in \Theta_h$ . Since  $\left\| \hat{\theta} \right\|_2 \leq L_2$  and writing  $Q_h$  as in Definition 4.2,
$$
\| \theta \| _ {2} ^ {2} = \hat {\theta} ^ {\top} \left(L _ {2} ^ {- 2} I + \sum_ {v \in C _ {h}} v v ^ {\top}\right) \hat {\theta} \leq L _ {2} ^ {- 2} L _ {2} ^ {2} + | C _ {h} | \leq 1 + d _ {1},
$$
where we used Definition 4.2 and Lemma 4.4. Finally, we conclude that  $\| \theta \| _2\leq \sqrt{d_1 + 1}$
Definition F.1.  $(G_h^Q,\rho_h^Q)$  is a near-optimal design for  $\Theta_h^Q$ , if for any  $\theta \in \Theta_h^Q$
$$
\langle v, \theta \rangle = 0 \quad f o r a l l v \in \operatorname {K e r} \left(V \left(G _ {h} ^ {Q}, \rho_ {h} ^ {Q}\right)\right), a n d \tag {23}
$$
$$
\left\| \theta \right\| _ {V \left(G _ {h} ^ {Q}, \rho_ {h} ^ {Q}\right) ^ {\dagger}} ^ {2} \leq 2 d, \tag {24}
$$
$$
\text {w h e r e} V \left(G _ {h} ^ {Q}, \rho_ {h} ^ {Q}\right) = \sum_ {\pi \in G _ {h} ^ {Q}} \rho_ {h} ^ {Q} (\pi) \left(\theta_ {h} ^ {Q} (\pi)\right) \left(\theta_ {h} ^ {Q} (\pi)\right) ^ {\top}. \tag {25}
$$
An important corollary of the above definition is that if  $M = \operatorname{Proj}_{\operatorname{Im}(V(G_h^Q, \rho_h^Q))}$ , then  $V(G_h^Q, \rho_h^Q)^{\dagger}^{\frac{1}{2}} V(G_h^Q, \rho_h^Q)^{\frac{1}{2}} M\nu = M\nu$ , and  $\langle \theta, M\nu \rangle = \langle \theta, \nu \rangle$  due to Eq. (23), and so
$$
\theta^ {\top} v = \theta^ {\top} V \left(G _ {h} ^ {Q}, \rho_ {h} ^ {Q}\right) ^ {\dagger \frac {1}{2}} V \left(G _ {h} ^ {Q}, \rho_ {h} ^ {Q}\right) ^ {\frac {1}{2}} v \quad \text {f o r a l l} \theta \in \Theta_ {h} ^ {Q} \text {a n d} v \in \mathbb {R} ^ {d}. \tag {26}
$$
Proof of Proposition 4.5. Take any  $h \in [H]$ ,  $s \in S_h$ , and  $Q \in \mathrm{PD}^H$ . Take  $i, j \in [\mathcal{A}]$  such that  $\text{range}(s) = \sup_{\theta \in \Theta_h} \langle \varphi(s, i, j), \theta \rangle$ . Then,
$$
\begin{array}{l} \operatorname {r a n g e} (s) ^ {2} = \sup  _ {\theta \in \Theta_ {h}} \left\langle \varphi (s, i, j), \theta \right\rangle^ {2} = \sup  _ {\theta \in \Theta_ {h} ^ {Q}} \left\langle \varphi_ {Q} (s, i, j), \theta \right\rangle^ {2} \\ \leq \sup _ {\theta \in \Theta_ {h} ^ {Q}} \| \theta \| _ {V (G _ {h} ^ {Q}, \rho_ {h} ^ {Q}) ^ {\dagger}} ^ {2} \left\| \varphi_ {Q} (s, i, j) \right\| _ {V (G _ {h} ^ {Q}, \rho_ {h} ^ {Q})} ^ {2} \\ \leq 2 d \varphi_ {Q} (s, i, j) ^ {\top} \left(\sum_ {\pi \in G _ {h} ^ {Q}} \rho_ {h} ^ {Q} (\pi) \left(\theta_ {h} ^ {Q} (\pi)\right) \left(\theta_ {h} ^ {Q} (\pi)\right) ^ {\top}\right) \varphi_ {Q} (s, i, j) \\ = 2 d \varphi (s, i, j) ^ {\top} \left(\sum_ {\pi \in G _ {h} ^ {Q}} \rho_ {h} ^ {Q} (\pi) \left(\theta_ {h} (\pi)\right) \left(\theta_ {h} (\pi)\right) ^ {\top}\right) \varphi (s, i, j) \\ \leq 2d \max  _ {\pi \in G _ {h} ^ {Q}} \left\langle \varphi (s, i, j) ^ {\top}, \theta_ {h} (\pi) \right\rangle^ {2} \leq 2 d \operatorname {r a n g e} _ {Q} (s) ^ {2}, \\ \end{array}
$$
where the first inequality uses Eq. (26) and the Cauchy-Schwarz inequality, and the second inequality follows by substituting the definition of  $V(G_h^Q,\rho_h^Q)$  and using Eq. (24). Finally, the first inequality in the last line holds as we replace the weighted sum from the previous line with the maximum operator. We therefore get that  $\mathrm{range}(s)\leq \sqrt{2d}\mathrm{range}_Q(s)$ , finishing the proof.
Proof of Lemma 4.4. Take any  $h \in [H]$  and the sequence  $C_h$  corresponding to  $Q$ . Assume that this sequence is of length  $l$ , and let  $\Sigma_{h,i} = L_2^{-2}I + \sum_{j=1}^i v_j v_j^\top$  for  $i \in [l]$ . By the second part of Eq. (5),
$$
l = \sum_ {i = 1} ^ {l} \min  \left\{1, 2 \left\| \left(L _ {2} ^ {- 2} I + \sum_ {j = 1} ^ {i - 1} v _ {j} v _ {j} ^ {\top}\right) ^ {- \frac {1}{2}} v _ {i} \right\| _ {2} ^ {2} \right\} \leq 2 \sum_ {i = 1} ^ {l} \min  \left\{1, \| v _ {i} \| _ {\Sigma_ {h, i - 1} ^ {- 1}} ^ {2} \right\}.
$$
Applying the elliptical potential lemma (Lemma L.1),
$$
l \leq 2 \sum_ {i = 1} ^ {l} \min  \left\{1, \| v _ {i} \| _ {\Sigma_ {h, i - 1} ^ {- 1}} ^ {2} \right\} \leq 4 d \log \left(\frac {\operatorname {T r} (\Sigma_ {h , 0}) + l L _ {3} ^ {2}}{d \det  (\Sigma_ {h , 0}) ^ {1 / d}}\right) = 4 d \log \left(1 + \frac {l L _ {3} ^ {2}}{L _ {2} ^ {- 2} d}\right).
$$
where  $\Sigma_{h,0} = L_2^{-2}I$  by definition. Using that  $\log (1 + x)\leq \sqrt{x}$  for  $x\geq 0$ , we have  $l\leq 4d\sqrt{lL_3^2L_2^2 / d}$  which implies  $l\leq 16dL_3^2 L_2^2$ . Substituting this into the previous bound yields
$$
l \leq 4 d \log \left(1 + \frac {\left(1 6 d L _ {3} ^ {2} L _ {2} ^ {2}\right) L _ {3} ^ {2}}{L _ {2} ^ {- 2} d}\right) = 4 d \log \left(1 + 1 6 L _ {3} ^ {4} L _ {2} ^ {4}\right) = d _ {1}.
$$
G Deferred proofs for Section 4.2
For any vector  $\nu \in \mathbb{R}^d$ , we define  $\overline{\nu} = \nu / \| \nu \|_2$  as the unit vector in the direction of  $\nu$  if  $\nu \neq 0$  and  $\overline{\mathbf{0}} = 0$  otherwise. For any  $h \in [2:H]$ ,  $s \in S_h$ , the normalized version of the largest preconditioned feature difference is denoted by
$$
\bar {\varphi} _ {Q} (s) = \overline {{\varphi_ {Q} (s , i , j)}} \quad \text {w h e r e} (i, j) = \underset {i ^ {\prime}, j ^ {\prime} \in [ \mathcal {A} ]} {\arg \max } \left\| \varphi_ {Q} \left(s, i ^ {\prime}, j ^ {\prime}\right) \right\| _ {2}. \tag {27}
$$
Proof of Lemma 4.7. Fix \( h \in [H] \), \( \alpha \)-admissible \( f: S_h \to \mathbb{R} \), \( t \in [h - 1] \), and \( \pi \in \Pi \). Our aim is to construct policies \( \pi_k^+, \pi_k^- \in \Pi \) for \( k \in [d_0] \), such that for all \( (s, a) \in S_t \times [\mathcal{A}] \), \( \sum_{k \in [d_0]} (q^{\pi_k^+}(s, a) - q^{\pi_k^-}(s, a)) \) is approximately proportional to the desired \( \mathbb{E}_{\pi, s, a} f(S_h) \). Let \( G_{h,1}^Q, G_{h,2}^Q, \ldots \) denote the policies in \( G_h^Q \) underlying the near-optimal design of \( \Theta_h^Q \), and for any \( s \in S_h \), denote by \( \operatorname{ord}(s) \in
$[d_0]$  the index of the policy maximizing the range of the action-value function in state  $s$ , that is,  $G_{h,\mathrm{ord}(s)}^Q = \arg \max_{\pi \in G_h^Q}\max_{i,j\in [\mathcal{A}]}(q^\pi (s,i) - q^\pi (s,j))$ ; to simplify notation, we define  $\tilde{G} (s) = G_{h,\mathrm{ord}(s)}^Q$ . For  $s\in S_h$  let
$$
(a ^ {+} (s), a ^ {-} (s)) = \left\{ \begin{array}{l l} \arg \max  _ {i, j \in [ \mathcal {A} ]} \hat {q} ^ {\tilde {G} (s)} (s, i) - \hat {q} ^ {\tilde {G} (s)} (s, j) & \text {i f} f (s) \geq 0 \\ \arg \min  _ {i, j \in [ \mathcal {A} ]} \hat {q} ^ {\tilde {G} (s)} (s, i) - \hat {q} ^ {\tilde {G} (s)} (s, j) & \text {o t h e r w i s e .} \end{array} \right.
$$
By Eq. (7) and Definition 4.6 have that
$$
\left| \hat {q} ^ {\tilde {G} (s)} \left(s, a ^ {+} (s)\right) - \hat {q} ^ {\tilde {G} (s)} \left(s, a ^ {-} (s)\right) \right| = \operatorname {r a n g e} _ {Q} (s) \geq \alpha | f (s) | \geq 0.
$$
Since  $q^{\tilde{G}(s)}(s, a^{+}(s)) - q^{\tilde{G}(s)}(s, a^{-}(s)) \approx_{2\eta} \hat{q}^{\tilde{G}(s)}(s, a^{+}(s)) - \hat{q}^{\tilde{G}(s)}(s, a^{-}(s))$ , if  $\alpha | f(s)| \geq 4\eta$ , we have
$$
q ^ {\tilde {G} (s)} \left(s, a ^ {+} (s)\right) - q ^ {\tilde {G} (s)} \left(s, a ^ {-} (s)\right) \geq \alpha f (s) - 2 \eta \geq \frac {\alpha}{2} f (s) > 0 \quad \text {i f} f (s) \geq 0 \tag {28}
$$
$$
q ^ {\tilde {G} (s)} \left(s, a ^ {+} (s)\right) - q ^ {\tilde {G} (s)} \left(s, a ^ {-} (s)\right) \leq \alpha f (s) + 2 \eta \leq \frac {\alpha}{2} f (s) <   0 \quad \text {o t h e r w i s e}.
$$
Let us define  $f': S_h \to \mathbb{R}$  as
$$
f ^ {\prime} (s) = \left\{ \begin{array}{l l} \frac {\alpha f (s) / 2}{q ^ {\hat {G} (s)} (s , a ^ {+} (s)) - q ^ {\hat {G} (s)} (s , a ^ {-} (s))} & \text {i f} \alpha | f (s) | \geq 4 \eta \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
By Eq. (28), there can be no division by zero in the above definition, and  $0 \leq f'(s) \leq 1$ .
Now we are ready to define  $\pi_k^+$  and  $\pi_k^-$ . Both policies follow  $\pi$  up to stage  $h - 1$ , when they switch to  $G_{h,k}^{Q}$ , except if at stage  $h$  a state  $s \in S_h$  is such that  $G_{h,k}^{Q}$  has the maximal action-value function range. In this case  $\pi_k^+$  selects  $a^+(s)$  with probability  $f'(s)$  and  $a^-(s)$  with probability  $1 - f'(s)$ , while  $\pi_k^-$  always selects  $a^-(s)$ . Formally, for  $k \in [d_0]$ , we define for  $s \in S$
$$
\pi_ {k} ^ {+} (s) = \left\{ \begin{array}{l l} \pi (s) & \text {i f s t a g e} (s) <   h; \\ a ^ {+} (s) \text {w . p .} f ^ {\prime} (s), \text {a n d} a ^ {-} (s) \text {w . p .} 1 - f ^ {\prime} (s) & \text {i f s t a g e} (s) = h \text {a n d o r d} (s) = k; \\ G _ {h, k} ^ {Q} (s), & \text {o t h e r w i s e}, \end{array} \right.
$$
where w.p. stands for with probability. Similarly,
$$
\pi_ {k} ^ {-} (s) = \left\{ \begin{array}{l l} \pi (s) & \text {i f s t a g e} (s) <   h; \\ a ^ {-} (s) \text {w . p .} 1 & \text {i f s t a g e} (s) = h \text {a n d o r d} (s) = k; \\ G _ {h, k} ^ {Q} (s) & \text {o t h e r w i s e}. \end{array} \right.
$$
Note that  $\pi_k^+ \in \Pi$  and  $\pi_k^- \in \Pi$ , as desired. Since for all  $k \in [d_0]$ , the policies follow  $G_{h,k}$  for  $s \in S_{t'}$  for  $t' > h$ , therefore for all  $k \in [d_0]$ ,
$$
v ^ {\pi_ {k} ^ {-}} (s) = v ^ {\pi_ {k} ^ {+}} (s) = v ^ {G _ {h, k} ^ {Q}} (s) \quad \text {f o r a l l} s \in S _ {h + 1}, \text {a n d} \tag {29}
$$
$$
q ^ {\pi_ {k} ^ {-}} (s, a) = q ^ {\pi_ {k} ^ {+}} (s, a) = q ^ {G _ {h, k} ^ {Q}} (s, a) \quad \text {f o r a l l} (s, a) \in \mathcal {S} _ {h} \times [ \mathcal {A} ]. \tag {30}
$$
Also, for any  $s \in S$  with stage(s)  $< h$  and any  $a \in [\mathcal{A}]$ ,
$$
\begin{array}{l} \sum_ {k \in [ d _ {0} ]} \left(q ^ {\pi_ {k} ^ {+}} (s, a) - q ^ {\pi_ {k} ^ {-}} (s, a)\right) = \underset {\pi , s, a} {\mathbb {E}} \sum_ {k \in [ d _ {0} ]} \left(v ^ {\pi_ {k} ^ {+}} \left(S _ {h}\right) - v ^ {\pi_ {k} ^ {-}} \left(S _ {h}\right)\right) \\ = \mathop{\mathbb{E}}_{\pi ,s,a}\left(v^{\pi_{\operatorname {ord}(S_{h})}^{+}}(S_{h}) - v^{\pi_{\operatorname {ord}(S_{h})}^{-}}(S_{h})\right) \\ = \underset {\pi , s, a} {\mathbb {E}} \left(q ^ {\tilde {G} (S _ {h})} (S _ {h}, a ^ {+} (S _ {h})) f ^ {\prime} (S _ {h}) + q ^ {\tilde {G} (S _ {h})} (S _ {h}, a ^ {-} (S _ {h})) (1 - f ^ {\prime} (S _ {h})) \right. \\ \left. - q ^ {\tilde {G} (S _ {h})} \left(S _ {h}, a ^ {-} (S _ {h})\right)\right) \\ = \underset {\pi , s, a} {\mathbb {E}} \left(f ^ {\prime} (S _ {h}) \left(q ^ {\tilde {G} (S _ {h})} \left(S _ {h}, a ^ {+} (S _ {h})\right) - q ^ {\tilde {G} (S _ {h})} \left(S _ {h}, a ^ {-} (S _ {h})\right)\right)\right) \\ = \underset {\pi , s, a} {\mathbb {E}} \left. \left\{\alpha \right| f (S _ {h}) \right| \geq 4 \eta \left\{\frac {\alpha}{2} f (S _ {h}) \approx_ {2 \eta} \frac {\alpha}{2} \underset {\pi , s, a} {\mathbb {E}} f (S _ {h}), \right. \\ \end{array}
$$
where the first line is due to both  $q^{\pi_k^+}$  and  $q^{\pi_k^-}$  following  $\pi$  on states with stage less than  $h$ , the second line follows from the fact that for any  $s \in S_h$ ,  $\pi_k^+(s) = \pi_k^-(s)$  for any  $k \neq \operatorname{ord}(s)$ ; combining this with Eq. (29) leads to all  $k \neq \operatorname{ord}(s)$  terms of the sum to cancel. The third line follows from expanding the definition of the policies and Eq. (30).
Let  $\tilde{\theta} = \frac{2}{\alpha}\sum_{k\in [d_0]}\left(\theta_t(\pi_k^+) - \theta_t(\pi_k^-)\right)$ . Since  $\| \theta_t(\cdot)\| _2\leq L_2$  by definition, we have  $\left\| \tilde{\theta}\right\| _2\leq 4d_0L_2 / \alpha$ . By Definition 3.2, for all  $(s,a)\in \mathcal{S}_t\times [\mathcal{A}]$
$$
\left\langle \varphi (s, a), \frac {\alpha}{2} \tilde {\theta} \right\rangle \approx_ {2 d _ {0} \eta} \sum_ {k \in [ d _ {0} ]} q ^ {\pi_ {k} ^ {+}} (s, a) - q ^ {\pi_ {k} ^ {-}} (s, a) \approx_ {2 \eta} \frac {\alpha}{2} \underset {\pi , s, a} {\mathbb {E}} f (S _ {h}),
$$
$$
\left\langle \varphi (s, a), \tilde {\theta} \right\rangle \approx_ {4 (d _ {0} + 1) \eta / \alpha} \underset {\pi , s, a} {\mathbb {E}} f (S _ {h}).
$$
Since  $4(d_0 + 1)\eta /\alpha \leq \eta_0 = 5d_0\eta /\alpha$  as  $d_0\geq 4$  by definition, this completes the proof.
Proof of Lemma 4.9. Take any  $s \in S_h$ . For the correct guess,  $\mathrm{range}_{Q}^{\tilde{G}}(s) = \mathrm{range}_{Q}(s)$ . Then, using that  $\left\| \bar{\varphi}_Q(\cdot)\right\|_2 \leq 1$ ,  $\mathrm{Tr}(\mathbf{f}(s)) \leq \mathrm{range}_Q(s)\frac{\sqrt{2d}H^2}{\varepsilon}$ , proving the second claim of the lemma (as  $\gamma \leq 1$ ).
For the first claim, take any  $\hat{G} = (\hat{\vartheta}_h^i)_{h\in [2:H],i\in [d_0]}\in \mathbf{G}$ . Let  $\varphi^\prime$  be the unnormalized version of  $\bar{\varphi}_{Q}(s)$  of Eq. (27), that is,  $\varphi^{\prime} = \varphi_{Q}(s,i,j)$  for the same  $i,j$  as in Eq. (27) (i.e., with the largest  $\ell_2$ -norm). Then, using that  $\hat{G}\in \mathbf{G}$
$$
\mathrm {r a n g e} _ {Q} ^ {\hat {G}} (s) = \max _ {k \in [ d _ {0} ]} \max _ {i, j} \left\langle \varphi_ {Q} (s, i, j), \hat {\vartheta} _ {h} ^ {k} \right\rangle \leq \| \varphi^ {\prime} \| _ {2} \max _ {k \in [ d _ {0} ]} \left\| \hat {\vartheta} _ {h} ^ {k} \right\| _ {2} \leq \| \varphi^ {\prime} \| _ {2} \sqrt {d _ {1} + 1}.
$$
Using that above in combination with  $|f(s)| \leq H, \nu, w \in \mathcal{B}(1), \left\| \bar{\varphi}_{Q}(s) \right\|_{2} \leq 1$ , we obtain
$$
\begin{array}{l} \left| v _ {\parallel} ^ {\top} \mathbf {f} (s) w \right| \leq \left| \left\langle \bar {\varphi} _ {Q} (s), v _ {\parallel} \right\rangle \left\langle \bar {\varphi} _ {Q} (s), w \right\rangle \right| \operatorname {r a n g e} _ {Q} ^ {\hat {G}} (s) \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} \\ \leq \left\| \bar {\varphi} _ {Q} (s) _ {\parallel} \right\| _ {2} \| \varphi^ {\prime} \| _ {2} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} = \left\| \varphi_ {\parallel} ^ {\prime} \right\| _ {2} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon}. \\ \end{array}
$$
As the eigenvalues of  $V(G_h^Q, \rho_h^Q) = \sum_{\pi \in G_h^Q} \rho_h^Q(\pi)(\theta_h^Q(\pi)) (\theta_h^Q(\pi))^{\top}$  corresponding to the subspace in which  $\varphi_{\parallel}'$  lies are by definition at least  $\gamma$ , we can write
$$
(\mathrm {r a n g e} _ {Q} (s)) ^ {2} \geq \max _ {\pi \in G _ {h} ^ {Q}} \left\langle \varphi^ {\prime}, \theta_ {h} ^ {Q} (\pi) \right\rangle^ {2} \geq \varphi^ {\prime} ^ {\top} V (G _ {h} ^ {Q}, \rho_ {h} ^ {Q}) \varphi^ {\prime} \geq \varphi_ {\parallel} ^ {\top} V (G _ {h} ^ {Q}, \rho_ {h} ^ {Q}) \varphi_ {\parallel} ^ {\prime} \geq \left\| \varphi_ {\parallel} ^ {\prime} \right\| _ {2} ^ {2} \gamma .
$$
Combining with the previous result, we get that
$$
\operatorname {r a n g e} _ {Q} (s) \geq \sqrt {\gamma} \left\| \varphi_ {\parallel} ^ {\prime} \right\| _ {2} \geq \frac {\sqrt {\gamma} \varepsilon}{\sqrt {2 d} \sqrt {d _ {1} + 1} H ^ {2}} | v _ {\parallel} ^ {\top} \mathbf {f} (s) w | = \alpha | v _ {\parallel} ^ {\top} \mathbf {f} (s) w |,
$$
H Deferred proofs for Appendix E.1
The definitions (Eqs. (9) and (10)) immediately give rise to the following facts:
$$
D \left(s _ {i} \rightarrow\right) \in \left[ - \sum_ {u = i} ^ {H} r _ {u}, H \right] \subseteq [ - H, H ] \text {a n d} \tau (\cdot) \in [ 0, 1 ], \text {i m p l y i n g}
$$
$$
E ^ {\rightarrow} \left(s _ {i} \rightarrow\right) \in \left[ - \sum_ {u = i} ^ {H} r _ {u}, H \right] \subseteq [ - H, H ], \text {i m p l y i n g} \tag {31}
$$
$$
E \left(s _ {i} \rightarrow\right) \in \left[ - 2 \tau \left(s _ {i}\right) H, 2 \tau \left(s _ {i}\right) H \right] \subseteq \left[ - 2 H, 2 H \right].
$$
Furthermore, since either  $\tau(s_i) = 0$  or  $\left\| \bar{\varphi}_Q(s_i) \right\|_2 = 1$  (as  $\left\| \bar{\varphi}_Q(s_i) \right\| = 0$  implies that  $\text{range}_Q(s_i)$  and hence  $\tau(s_i)$  are both zero), we have
$$
\operatorname {T r} \left(F \left(s _ {i \rightarrow}\right)\right) = E \left(s _ {i \rightarrow}\right), \tag {32}
$$
which was used to establish the last part of Corollary 4.11.
Proof of Lemma E.1. We drop the subscripts  $(\hat{G},\bar{\theta})$ . Let  $(\hat{\vartheta}_h^i)_{h\in [2:H],i\in [d_0]} = \hat{G}\in \mathbf{G}$ . Let  $z = v - w$  be the projection of  $\nu$  to the subspace orthogonal to  $Z(Q,i)$ , denoted by  $Z(Q,i)^{\perp}$ . In other words,  $z = \mathrm{Proj}_{Z(Q,i)}\nu$ . Let  $\mathbf{M} = y^{ki} - \hat{F}^{ki}$ . By the symmetry of  $\mathbf{M}$ ,
$$
v ^ {\top} \mathbf {M} v = z ^ {\top} \mathbf {M} (v + w) + w ^ {\top} \mathbf {M} w.
$$
It is enough to prove therefore that
$$
\frac {\varepsilon}{d H ^ {2} \omega} \geq z ^ {\top} \mathbf {M} (v + w).
$$
As  $\| v \|_2 \leq 1$  and  $\| v + w \|_2 \leq 2$ , and using the definitions and Eq. (31), for any input  $(s_i \to)$
$$
\begin{array}{l} \left| z ^ {\top} F (s _ {i} \rightarrow) (v + w) \right| = \left|\left\langle z, \bar {\varphi} _ {Q} (s _ {i}) \right\rangle\left\langle v + w, \bar {\varphi} _ {Q} (s _ {i}) \right\rangle E (s _ {i} \rightarrow) \right| \\ \leq 4 H \tau (s _ {i}) | \left\langle z, \bar {\varphi} _ {Q} (s _ {i}) \right\rangle | \leq 4 \operatorname {r a n g e} _ {Q} ^ {\hat {G}} (s _ {i}) \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} | \left\langle z, \bar {\varphi} _ {Q} (s _ {i}) \right\rangle | \\ \leq 4 \left| \left\langle z, \bar {\varphi} _ {Q} (s _ {i}) \right\rangle \right| \max  _ {a, b, k \in [ d _ {0} ]} \left\langle \varphi_ {Q} (s, a, b), \hat {\vartheta} _ {h} ^ {k} \right\rangle \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} \\ \leq 4 \left\| \operatorname {P r o j} _ {Z (Q, i) ^ {\perp}} \bar {\varphi} _ {Q} (s) \right\| _ {2} \| \varphi^ {\prime} \| _ {2} \max  _ {k \in [ d _ {0} ]} \left\| \hat {\vartheta} _ {h} ^ {k} \right\| _ {2} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} \\ \leq 4 \left\| \operatorname {P r o j} _ {Z (Q, i) ^ {\perp}} \varphi^ {\prime} \right\| _ {2} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon}, \\ \end{array}
$$
where  $\varphi^{\prime}$  is the unnormalized version of  $\bar{\varphi}_Q(s_i)$  of Eq. (27), that is,  $\varphi^{\prime} = \varphi_{Q}(s_{i},a,b)$  for the same  $a,b$  as in Eq. (27) (i.e., with the largest  $\ell_2$ -norm).
As  $\mathrm{Proj}_{Z(Q,i)^{\perp}}\varphi^{\prime} = \mathrm{Proj}_{Z(Q,i)^{\perp}}(\varphi_{Q}(s,a) - \varphi_{Q}(s,b)) = \mathrm{Proj}_{Z(Q,i)^{\perp}}Q_{i}(\varphi (s,a) - \varphi (s,b))$  for some  $s\in S_i,a,b\in [\mathcal{A}]$ , and by definition  $\mathrm{Proj}_{Z(Q,i)^{\perp}}Q_{i}\preceq L_{3}^{-2}I,\left\| \mathrm{Proj}_{Z(Q,i)^{\perp}}\varphi^{\prime}\right\|_{2}\leq L_{3}^{-2}\| \varphi (s,a) - \varphi (s,b)\|_{2}\leq 2L_{3}^{-2}L_{1}$ , so
$$
\left| z ^ {\top} F \left(s _ {i} \rightarrow\right) (v + w) \right| \leq 8 L _ {3} ^ {- 2} L _ {1} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon}, \tag {33}
$$
$$
\left| z ^ {\top} \hat {F} ^ {k i} (v + w) \right| \leq 8 L _ {3} ^ {- 2} L _ {1} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon}. \tag {34}
$$
To bound  $\left|z^{\top}y^{ki}(\nu + w)\right|$ , note that by the definition  $y^{ki}$
$$
z ^ {\top} y ^ {k i} (v + w) = \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} \left\langle \varphi_ {\mathrm {p} (k)} ^ {m k j}, \check {\theta} ^ {\mathrm {P} (m k j), i} \right\rangle
$$
where  $\check{\theta}^{ti} = X_{mt}^{-1}\sum_{lkj\in \mathbf{I}^m (t)}\varphi_t^{lkj}\left(z^\top F(S_i^{lkj},\ldots ,R_H^{lkj})(v + w)\right)$  for  $t\in [i - 1]$
$$
\left| z ^ {\top} y ^ {k i} (v + w) \right| \leq \max  _ {t \in [ i - 1 ], s \in \mathcal {S} _ {t}, a \in [ \mathcal {A} ]} \left\langle \varphi (s, a), \check {\theta} ^ {t i} \right\rangle .
$$
Fix any  $t \in [i - 1], s \in S_t, a \in [\mathcal{A}]$ . By repeated application of the Cauchy-Schwarz inequality, the fact that  $X_{mt} \succeq \lambda I$ , the triangle inequality, and using Eq. (33),
$$
\begin{array}{l} \left| \left\langle \varphi (s, a), \check {\theta} ^ {t i} \right\rangle \right| \leq \| \varphi (s, a) \| _ {X _ {m t} ^ {- 1}} \left\| \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \varphi_ {t} ^ {l k j} \left(z ^ {\top} F \left(S _ {i} ^ {l k j}, \dots , R _ {H} ^ {l k j}\right) (v + w)\right) \right\| _ {X _ {m t} ^ {- 1}} \\ \leq \| \varphi (s, a) \| _ {2} \lambda^ {- 1 / 2} \cdot 8 L _ {3} ^ {- 2} L _ {1} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} \sum_ {l k j \in \mathbf {I} ^ {m} (t)} \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {m t} ^ {- 1}} \\ \leq 8 L _ {3} ^ {- 2} L _ {1} ^ {2} \lambda^ {- 1 / 2} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} \sqrt {| \mathbf {I} ^ {m} (t) |} \sqrt {\sum_ {l k j \in \mathbf {I} ^ {m} (t)} \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {m t} ^ {- 1}} ^ {2}} \\ \leq 8 L _ {3} ^ {- 2} L _ {1} ^ {2} \lambda^ {- 1 / 2} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} \sqrt {m _ {\mathrm {m a x}} n H d}, \\ \end{array}
$$
where we use that  $|\mathbf{I}^m (t)|\leq mnH$ $m\leq m_{\mathrm{max}}$  by Lemma E.5, and that
$$
\sqrt {\sum_ {l k j \in \mathbf {I} ^ {m} (t)} \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {m t} ^ {- 1}} ^ {2}} = \sqrt {\sum_ {l k j \in \mathbf {I} ^ {m} (t)} \operatorname {T r} \left(X _ {m t} ^ {- 1} \varphi_ {t} ^ {l k j} \varphi_ {t} ^ {l k j ^ {\top}}\right)} \leq \sqrt {\operatorname {T r} X _ {m t} ^ {- 1} X _ {m t}} = \sqrt {d}.
$$
Combining with Eq. (34), with an appropriate choice of  $L_{3}$ , we obtain
$$
\left| z ^ {\top} \mathbf {M} (v + w) \right| \leq 8 L _ {3} ^ {- 2} L _ {1} \sqrt {d _ {1} + 1} \frac {\sqrt {2 d} H ^ {2}}{\varepsilon} \left(1 + L _ {1} \lambda^ {- 1 / 2} \sqrt {m _ {\max } n H d}\right) \leq \frac {\varepsilon}{d H ^ {2} \omega} \tag {35}
$$
Proof of Lemma E.2. Choose  $\beta$