SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\beta \leq 2 + 2 H \sqrt {2 d H \left(d _ {0} + 1\right) \log \frac {1 2 d _ {0} H L _ {2}}{\alpha \xi} + 2 \log \frac {m _ {\max } ^ {\prime} H ^ {2}}{\zeta} + d \log \left(\lambda + m _ {\max } ^ {\prime} n H L _ {1} ^ {2} / d\right)}, \tag {36}
$$
satisfying  $\beta = \tilde{O} (H^{3 / 2}d)$  as given in Eq. (15), and define
$$
\xi = \frac {\varepsilon}{5 \sqrt {2 d} (H + 1) ^ {3} L _ {1}} \left(\min  \left\{\varepsilon / (d H ^ {2} \omega), 1 / \sqrt {m _ {\max} ^ {\prime} n H} \right\} - \eta_ {0}\right).
$$
Note that subtracting  $\eta_0$  keeps  $\xi$  positive, and of the same order, by our assumption that  $\eta$  is small enough:  $\eta_0 \leq \frac{1}{2} \min \left\{\varepsilon / (dH^2\omega), 1 / \sqrt{m_{\max}'nH}\right\}$ , which follows from Eq. (21).
We start with a covering argument for the set of functions of the form  $v_{\parallel}^{\top}\bar{F}_{\hat{G}\bar{\theta}}w$ , for different choices of  $\hat{G}, \bar{\theta}, \nu,$  and  $w$ . By [Vershynin, 2018, Corollary 4.2.13], there is a set  $C_{\xi} \subset \mathcal{B}(1)$  with  $|C_{\xi}| \leq (3/\xi)^d$  such that for all  $x \in \mathcal{B}(1)$  there exists a  $y \in C_{\xi}$  with  $\|x - y\|_2 \leq \xi$ . Therefore, there is a set  $C_{\xi}^\times \subset \left(\bigvee_{h \in [2:H], k \in [d_0]} \mathcal{B}(\sqrt{d_1 + 1})\right) \times \left(\bigvee_{h \in [2:H]} \mathcal{B}(4d_0HL_2 / \alpha)\right) \times \mathcal{B}(1) \times \mathcal{B}(1)$  with  $|C_{\xi}^\times| \leq (12d_0HL_2 / (\alpha\xi))^{dH(d_0 + 1)}$  such that for any  $\hat{G} = (\hat{\vartheta}_h^i)_{h \in [2:H], i \in [d_0]} \in \mathbf{G}$ ,  $\bar{\theta} \in \bar{\Theta}$ , and  $v, w \in \mathcal{B}(1)$ , there exists a  $y \in C_{\xi}^\times$ , such that if we let  $\tilde{G} = (\tilde{\vartheta}^i)_{h \in [2:H], i \in [d_0]} = (y_{(h-1)d_0+i})_{h \in [2:H], i \in [d_0]}$ ,  $\tilde{\theta} = (\tilde{\theta}_h)_{h \in [2:H]} = (y_{(H-1)d_0+h})_{h \in [2:H]}$ , and  $a = y_{(H-1)(d_0+1)+1}$ ,  $b = y_{(H-1)(d_0+1)+2}$ , then  $\tilde{G} \in \mathbf{G}$ ,  $\tilde{\theta} \in \bar{\Theta}$ ,  $a, b \in \mathcal{B}(1)$ , and
$$
\left\| a - v \right\| _ {2} \leq \xi \quad \text {a n d} \quad \left\| b - w \right\| _ {2} \leq \xi , \quad \text {a n d}
$$
$$
\left\| \hat {\theta} _ {h} ^ {i} - \tilde {\theta} ^ {i} \right\| _ {2} \leq \xi \text {a n d} \left\| \bar {\theta} _ {h} - \tilde {\theta} _ {h} \right\| _ {2} \leq \xi \text {f o r a l l} h \in [ 2: H ], i \in [ d _ {0} ].
$$
As a result, for all  $s \in S \setminus S_1$ ,  $|\mathrm{range}_Q^{\tilde{G}}(s) - \mathrm{range}_Q^{\tilde{G}}(s)| \leq 2L_1\xi$ , and therefore  $|\tau_{\hat{G}\bar{\theta}}(s) - \tau_{\tilde{G}\bar{\theta}}(s)| \leq 2\sqrt{2d} HL_1\xi/\varepsilon$ . Furthermore,  $|D_{\hat{G}\bar{\theta}}(s,\dots,r_H) - D_{\tilde{G}\bar{\theta}}(s,\dots,r_H)| \leq L_1\xi$ . Combining these with the facts that in either case,  $\tau(\cdot) \in [0,1]$ ,  $D(\cdot) \in [-H,H]$ , and  $E^{\rightarrow}(\cdot) \in [-H,H]$  (Eq. (31)), and using the definition of  $E$  and  $E^{\rightarrow}$ , we have that for any  $i \in [H+1]$  and inputs,
$$
\begin{array}{l} \left| E _ {\hat {G} \bar {\theta}} \left(s _ {i} \rightarrow\right) - E _ {\tilde {G} \tilde {\theta}} \left(s _ {i} \rightarrow\right)\right| \leq 4 \sqrt {2 d} H ^ {2} L _ {1} \xi / \varepsilon + L _ {1} \xi + \left| E _ {\hat {G} \bar {\theta}} ^ {\rightarrow} \left(s _ {i + 1} \rightarrow\right) - E _ {\tilde {G} \bar {\theta}} ^ {\rightarrow} \left(s _ {i + 1} \rightarrow\right)\right| \\ = 4 \sqrt {2 d} H ^ {2} L _ {1} \xi / \varepsilon + L _ {1} \xi + \sum_ {j = i + 1} ^ {H} | E _ {\hat {G} \bar {\theta}} (s _ {j} \rightarrow) - E _ {\tilde {G} \bar {\theta}} (s _ {j} \rightarrow) | \\ \leq (H + 1) 5 \sqrt {2 d} H ^ {2} L _ {1} \xi / \varepsilon , \\ \end{array}
$$
where the first inequality sums over the contributions of  $\tau$ ,  $D$ , and  $E^{\rightarrow}$ , and the second applies induction. By combining this bound with the bounds on  $\|v - a\|_2$  and  $\|w - b\|_2$ , and that  $E(\cdot) \in [-2H, 2H]$  (Eq. (31)) implying that  $\bar{F}(\cdot) \in [-2H, 2H]$ , for all  $s \in S \setminus S_1$ , we have that
$$
\begin{array}{l} \left| v _ {\parallel} ^ {\top} \bar {F} _ {\hat {G} \bar {\theta}} (s) w - a _ {\parallel} ^ {\top} \bar {F} _ {\tilde {G} \tilde {\theta}} b \right| (s) \leq 6 H \xi + (H + 1) 5 \sqrt {2 d} H ^ {2} L _ {1} \xi / \varepsilon \tag {37} \\ \leq 5 \sqrt {2 d} (H + 1) ^ {3} L _ {1} \xi / \varepsilon = \min  \{\varepsilon / (d H ^ {2} \omega), 1 / \sqrt {m _ {\max } ^ {\prime} n H} \} - \eta_ {0} \\ \end{array}
$$
Take any  $m' \in [m_{\max}']$  (this includes the entire execution of SKIPPYELEANOR), and let the quantities of Section 4.3 (such as  $F$ ) be calculated with the value of  $Q$  at the beginning iteration  $m'$
(Line 5). Take any  $t \in [H - 1]$ ,  $i \in [t + 1:H]$ . Take any  $y \in C_{\xi}^{\times}$  and assign values to  $a, b, \tilde{G}$ , and  $\tilde{\theta}$  based on  $y$  as above. For any  $lkj \in \mathbf{I}^{m}(t)$ , observe that given all the history of SKIPPYELEANOR interacting with the MDP up to (and including)  $S_{t}^{lkj}, A_{t}^{lkj}$ , the trajectory  $S_{t+1}^{lkj}, A_{t+1}^{lkj}, \ldots, R_{H}^{lkj}$  is an independent rollout with policy  $\pi^{0}$ , with its law given by  $\mathcal{P}_{\pi^{0}, S_{t}^{lkj}, A_{t}^{lkj}}$ . The random variable  $a_{\parallel}^{\top} F_{\tilde{G}\tilde{\theta}}(S_{i}^{lkj}, \ldots, R_{H}^{lkj})b$  has range  $[-2H, 2H]$  and expectation (conditioned on this history)  $\mathbb{E}_{\pi^{0}, S_{t}^{lkj}, A_{t}^{lkj}} a_{\parallel}^{\top} \bar{F}_{\tilde{G}\tilde{\theta}}(S_{i})b$ . Let  $\check{\theta}_{ti}$  be  $\tilde{\theta}_{ti}$  from Corollary 4.11, satisfying  $\| \check{\theta}_{ti} \|_2 \leq 1 / \sqrt{\lambda}$  and Eq. (11) for  $a_{\parallel}, b, \tilde{G}$ , and  $\tilde{\theta}$  instead of  $v_{\parallel}, w, \hat{G}$ , and  $\tilde{\theta}$ :
$$
\underset {\pi^ {0}, s, a} {\mathbb {E}} a _ {\parallel} ^ {\top} \bar {F} _ {\tilde {G} \tilde {\theta}} (S _ {i}) b \approx_ {\eta_ {0}} \left\langle \varphi (s, a), \check {\theta} _ {t i} \right\rangle . \tag {38}
$$
Take the sequence  $A$  formed of  $\varphi_t^{lkj}$  (for  $lkj \in \mathbf{I}^m(t)$ , in the order that these random variables are observed), and the sequence  $X$  formed of  $\nu_{\parallel}F_{\hat{G}\bar{\theta}}(S_i^{lkj},\ldots,R_H^{lkj})w$  (for  $lkj \in \mathbf{I}^m(t)$ , in the same order), and the sequence  $\Delta$  formed of  $\mathbb{E}_{\pi^0,S_t^{lk_j},A_t^{lk_j}}\nu_{\parallel}\bar{F}_{\hat{G}\bar{\theta}}(S_i)w - \left\langle \varphi_t^{lkj},\check{\theta}_{ti}\right\rangle$  (for  $lkj \in \mathbf{I}^m(t)$ , in the same order, for any  $\nu,w,\hat{G}$ , and  $\bar{\theta}$  as in the statement of this lemma). Then the sequences  $A,X$  and  $\Delta$  satisfy the conditions of Lemma M.4 with a subgaussianity parameter  $\sigma = 2H$ . Due to this lemma, with probability at least  $1 - \zeta / (m_{\max}^{\prime}H^{2}|C_{\xi}^{\times}|)$ , for any choice of  $\nu,w,\hat{G}$ , and  $\bar{\theta}$  (as above),
$$
\left\| \tilde {\theta} _ {t i} - \check {\theta} _ {t i} \right\| _ {X _ {m t}} <   \sqrt {\lambda} \left\| \check {\theta} _ {t i} \right\| _ {2} + \| \Delta \| _ {\infty} \sqrt {| \mathbf {I} ^ {m} (t) |} + 2 H \sqrt {2 \log \left(\frac {m _ {\operatorname* {m a x}} ^ {\prime} H ^ {2} | C _ {\xi} ^ {\times} |}{\zeta}\right) + \log \left(\frac {\det X _ {m t}}{\lambda^ {d}}\right)} \tag {39}
$$
where  $\tilde{\theta}_{ti} = X_{mt}^{-1}\sum_{lkj\in \mathbf{I}^m (t)}\varphi_t^{lkj}v_\parallel^\top F_{\hat{G}\bar{\theta}}(S_i^{lkj},\ldots ,R_H^{lkj})w$
A union bound over all  $m' \in [m_{\max}']$ ,  $t, i$ , and  $y \in C_{\xi}^{\times}$  guarantees with probability at least  $1 - \zeta$ , the above holds for all choice of these variables, any time beginning of any iteration (Line 5) is executed. Note that we need the union bound over  $m$  because the value of  $Q$  underlying the targets of least-squares estimations can potentially change between iterations.
To finish the proof, under this high-probability event, take any  $m, t, i, \hat{G}$ , and  $\bar{\theta}$  as in the statement of this lemma, and choose  $y \in C_{\xi}^{\times}$  as before, to satisfy Eq. (37). Combined with Eq. (38), this immediately implies that the sequence  $\Delta$  formed of quantities with absolute value
$$
\begin{array}{l} \left| \underset {\pi^ {0}, S _ {t} ^ {l k j}, A _ {t} ^ {l k j}} {\mathbb {E}} v _ {\parallel} \bar {F} _ {\hat {G} \bar {\theta}} (S _ {i}) w - \left\langle \varphi_ {t} ^ {l k j}, \check {\theta} _ {t i} \right\rangle \right| \\ \leq \left| \mathbb {E} _ {\pi^ {0}, S _ {t} ^ {l k j}, A _ {t} ^ {l k j}} v _ {\parallel} \bar {F} _ {\hat {G} \tilde {\theta}} (S _ {i}) w - a _ {\parallel} \bar {F} _ {\tilde {G} \tilde {\theta}} (S _ {i}) b \right| + \left| a _ {\parallel} \bar {F} _ {\tilde {G} \tilde {\theta}} (S _ {i}) b - \left\langle \varphi_ {t} ^ {l k j}, \check {\theta} _ {t i} \right\rangle \right| \tag {40} \\ \leq \min  \left\{\varepsilon / \left(d H ^ {2} \omega\right), 1 / \sqrt {m _ {\max } ^ {\prime} n H} \right\} - \eta_ {0} + \eta_ {0} \\ \end{array}
$$
satisfies  $\| \Delta \|_{\infty} \leq \min \{\varepsilon / (dH^{2}\omega), 1 / \sqrt{m_{\max}^{\prime}nH}\}$ . Take any  $(s, a) \in S_{t} \times [\mathcal{A}]$ , and let  $\tilde{\theta}_{ti}$  and  $\check{\theta}_{ti}$  be as above (in Eq. (39)) for  $\nu_{\parallel}, w, \hat{G}$ , and  $\bar{\theta}$ . Note that
$$
v _ {\parallel} ^ {\top} \varphi (s, a) ^ {\top} \hat {\theta} _ {\hat {G} \bar {\theta}} ^ {t i} w = \left\langle \varphi (s, a), \tilde {\theta} _ {t i} \right\rangle ,
$$
By the triangle inequality, using Cauchy-Schwarz, and Eqs. (39) and (40),
$$
\begin{array}{l} \left| v _ {\parallel} ^ {\top} \left(\varphi (s, a) ^ {\top} \hat {\theta} _ {\hat {G} \bar {\theta}} ^ {t i} - \underset {\pi^ {0}, s, a} {\mathbb {E}} \bar {F} _ {\hat {G} \bar {\theta}} (S _ {i})\right) w \right| \\ \leq \left| \left\langle \varphi (s, a), \check {\theta} _ {t i} - \check {\theta} _ {t i} \right\rangle \right| + \left| \underset {\pi^ {0}, s, a} {\mathbb {E}} v _ {\parallel} ^ {\top} \bar {F} _ {\hat {G} \bar {\theta}} (S _ {i}) w - \left\langle \left\langle \varphi (s, a), \check {\theta} _ {t i} \right\rangle \right\rangle \right| \\ \leq \| \varphi (s, a) \| _ {X _ {m t} ^ {- 1}} \left(\sqrt {\lambda} \left\| \check {\theta} _ {t i} \right\| _ {2} + \frac {\sqrt {| \mathbf {I} ^ {m} (t) |}}{\sqrt {m _ {\max} ^ {\prime} n H}} + 2 H \sqrt {2 \log \left(\frac {m _ {\max} ^ {\prime} H ^ {2} | C _ {\xi} ^ {\times} |}{\zeta}\right) + \log \left(\frac {\det X _ {m t}}{\lambda^ {d}}\right)}\right) + \frac {\varepsilon}{d H ^ {2} \omega} \\ \leq \| \varphi (s, a) \| _ {X _ {m t} ^ {- 1}} \left(2 + 2 H \sqrt {2 d H (d _ {0} + 1) \log \frac {1 2 d _ {0} H L _ {2}}{\alpha \xi} + 2 \log \frac {m _ {\operatorname* {m a x}} ^ {\prime} H ^ {2}}{\zeta} + d \log (\lambda + m _ {\operatorname* {m a x}} ^ {\prime} n H L _ {1} ^ {2} / d)}\right) + \frac {\varepsilon}{d H ^ {2} \omega} \\ \leq \left\| \varphi (s, a) \right\| _ {X _ {m t} ^ {- 1}} \beta + \frac {\varepsilon}{d H ^ {2} \omega}, \tag {41} \\ \end{array}
$$
where in the fourth line we used that  $|\mathbf{I}^m (t)|\leq m_{\max}^\prime nH,|C_\xi^\times |\leq (12d_0HL_2 / (\alpha \xi))^{dH(d_0 + 1)}$  and we used the inequality of arithmetic and geometric means to bound det  $X_{mt}\leq \left(\frac{1}{d}\operatorname {Tr}X_{mt}\right)^{d}\leq$ $\left(\frac{\mathrm{Tr}\lambda I + |\mathbf{I}^m(t)|L_1^2}{d}\right)^d.$
Proof of Lemma E.3. Choose  $n$  to satisfy
$$
n = \left[ 6 4 \frac {\left(d H ^ {2} \omega\right) ^ {2}}{\varepsilon^ {2}} H ^ {2} \left(2 d \log \frac {1 8 d H ^ {3}}{\varepsilon} + \log \frac {2 m _ {\max } ^ {\prime} H ^ {2}}{\zeta}\right) \right]. \tag {42}
$$
This leads to  $n = \tilde{O}(d^5 H^6 / \varepsilon^2)$ .
Similarly to the proof of Lemma E.2, we start with a covering argument. This time, as  $\hat{G}$  and  $\bar{\theta}$  are fixed, we only consider  $\nu$  and  $w$ , to cover  $\nu_{\parallel}^{\top}\bar{F}_{t^{\prime}}^{(j)}w$  and  $\nu_{\parallel}\hat{F}_{t^{\prime}}^{(j)}w$ . Let  $\xi^{\prime} = \frac{\varepsilon}{12dH^{3}}$ . There is a set  $C_{\xi^{\prime}}^{+}\subset \mathcal{B}(1)\times \mathcal{B}(1)$  with  $|C_{\xi^{\prime}}|\leq (3 / \xi^{\prime})^{2d}$  such that for all  $\nu ,w\in \mathcal{B}(1)$ , there exists an  $(a,b)\in C_{\xi^{\prime}}^{+}$  with  $\| v - a\| _2\leq \xi '$  (and therefore  $\left\| v_{\parallel} - a_{\parallel}\right\| _2\leq \xi ')),$  and  $\| w - b\| _2\leq \xi '$ . Take such a choice of  $(a,b)$  for any  $(\nu ,w)$ . As  $E(\cdot)\in [-2H,2H]$  by Eq. (31), and  $\left\| \bar{\varphi}_Q(\cdot)\right\| _2\leq 1$ , For  $i\in [2:H]$  and any input,
$$
\left| v _ {\parallel} ^ {\top} F (s _ {i} \rightarrow) w - a _ {\parallel} ^ {\top} F (s _ {i} \rightarrow) b \right| \leq 6 H \xi^ {\prime} = \frac {\varepsilon}{2 d H ^ {2}},
$$
and therefore for any  $s \in S \setminus S_1$ ,  $\left| v_{\parallel}^{\top} \bar{F}(s) w - a_{\parallel}^{\top} \bar{F}(s) b \right| \leq \varepsilon / (2dH^2)$ . For  $j \in [n]$  let
$$
\tilde {F} _ {j} ^ {k i} = \underset {\pi^ {0}, S _ {\mathrm {p} (k)} ^ {m k j}, A _ {\mathrm {p} (k)} ^ {m k j}} {\mathbb {E}} F _ {\hat {G} \bar {\theta}} (S _ {i} ^ {m k j}, \ldots , R _ {H} ^ {m k j}) = \underset {\pi^ {0}, S _ {\mathrm {p} (k)} ^ {m k j}, A _ {\mathrm {p} (k)} ^ {m k j}} {\mathbb {E}} \bar {F} _ {\hat {G} \bar {\theta}} (S _ {i} ^ {m k j})
$$
By the triangle inequality, for any  $k \in [H - 1]$ ,  $i \in [k + 1:H]$
$$
\begin{array}{l} \left| v _ {\parallel} ^ {\top} \left(y _ {\hat {G} \bar {\theta}} ^ {k i} - \hat {F} _ {\hat {G} \bar {\theta}} ^ {k i}\right) w \right| \\ \leq \left| \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} v ^ {\top} \left(\varphi_ {\mathrm {p} (k)} ^ {m k j ^ {\top}} \hat {\theta} _ {\hat {G} \bar {\theta}} ^ {\mathrm {p} (m k j), i} - \tilde {F} _ {j} ^ {k i}\right) w \right| + \left| \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} v ^ {\top} \left(\tilde {F} _ {j} ^ {k i} - F _ {\hat {G} \bar {\theta}} (S _ {i} ^ {m k j}, \dots , R _ {H} ^ {m k j})\right) w \right| \\ \leq \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} \left\| \varphi_ {\mathrm {p} (k)} ^ {m k j} \right\| _ {X _ {m, \mathrm {p} (m k j)} ^ {- 1}} \beta + \frac {\varepsilon}{d H ^ {2} \omega} + \frac {\varepsilon}{d H ^ {2} \omega} + \left| \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} a ^ {\top} \left(\tilde {F} _ {j} ^ {k i} - F _ {\hat {G} \bar {\theta}} \left(S _ {i} ^ {m k j}, \dots , R _ {H} ^ {m k j}\right)\right) b \right|, \tag {43} \\ \end{array}
$$
where the second inequality uses Lemma E.2 and applies the triangle inequality twice again. Observe that for all  $j \in [n]$ , given all the history of SKIPPYELEANOR interacting with the MDP
up to (and including)  $S_{\mathrm{p}(k)}^{mkj}, A_{\mathrm{p}(k)}^{mkj}$  (which also includes the value of  $c_{ki}^j$  for  $i \in [H + 1]$ ), the trajectory  $S_{\mathrm{p}(k) + 1}^{mkj}, A_{\mathrm{p}(k) + 1}^{mkj}, \ldots, R_H^{mkj}$  is an independent rollout with policy  $\pi^0$ , with its law given by  $\mathcal{P}_{\pi^0, S_{\mathrm{p}(k)}^{mkj}, A_{\mathrm{p}(k)}^{mkj}}$ . Therefore, for any fixed  $(a, b) \in C_{\xi'}^+$ ,  $c_{ki}^j a^\top \left( \tilde{F}_j^{ki} - F_{\hat{G}\bar{\theta}}(S_i^{mkj}, \ldots, R_H^{mkj}) \right) b$  are independent zero-mean random variables with range  $[-4H, 4H]$ . Applying Hoeffding's inequality with a union bound over  $m', k, i, a$ , and  $b$ , with probability at least  $1 - \zeta$ , for any of the  $m' \in [m_{\max}'']$  times the beginning of the iteration (Line 5) is executed (this includes the entire execution of SKIP-PYELEANOR),
$$
\begin{array}{l} \left| \frac {1}{n} \sum_ {j \in [ n ]} c _ {k i} ^ {j} a ^ {\top} \left(\tilde {F} _ {j} ^ {k i} - F _ {\hat {G} \bar {\theta}} \left(S _ {i} ^ {m k j}, \dots , R _ {H} ^ {m k j}\right)\right) b \right| \leq \frac {8 H}{\sqrt {n}} \sqrt {\log \frac {2 m _ {\operatorname* {m a x}} ^ {\prime} H ^ {2} \left| C _ {\xi^ {\prime}} ^ {+} \right|}{\zeta}} \\ = \frac {8 H}{\sqrt {n}} \sqrt {2 d \log \frac {1 8 d H ^ {3}}{\varepsilon} + \log \frac {2 m _ {\mathrm {m a x}} ^ {\prime} H ^ {2}}{\zeta}} \leq \frac {\varepsilon}{d H ^ {2} \omega}, \\ \end{array}
$$
where we used Eq. (42). To finish, note that unless  $c_{ki}^{j} = 0$ ,  $\left\| \varphi_{\mathrm{p}(k)}^{mk_j} \right\|_{X_{m,\mathrm{p}(mk_j)}^{-1}} < 2(\beta \omega dH)^{-1}$ , so we can continue from Eq. (43) by bounding the average feature-norm by  $\bar{\sigma}_k^m$  as
$$
\left| v _ {\parallel} ^ {\top} \left(y _ {\hat {G} \bar {\theta}} ^ {k i} - \hat {F} _ {\hat {G} \bar {\theta}} ^ {k i}\right) w \right| \leq \bar {\sigma} _ {k} ^ {m} \beta + 3 \frac {\varepsilon}{d H ^ {2} \omega}.
$$
Proof of Lemma E.4. Recall that  $(k,i,\nu)$  are the arguments and  $x$  the value of Optimization Problem 4.12. Throughout the proof we write  $Q$  to refer to its value just before Line 13 is executed. We write  $\bullet_{\parallel}$  for  $\bullet_{\parallel (Q,i)}$ , and  $\bullet_{\perp}$  for  $\bullet_{\perp (Q,i)}$ . Let  $\mathbf{M} = y^{ki}_{\hat{G}\bar{\theta}} - \hat{F}^{ki}_{\hat{G}\bar{\theta}}$ . Therefore,  $\nu^{\top}\mathbf{M}\nu = x > \bar{\sigma}_k^m\beta \omega +3\frac{\varepsilon}{dH^2}$ , and by Lemma E.1,  $w^{\top}\mathbf{M}w > \bar{\sigma}_k^m\beta \omega +2\frac{\varepsilon}{dH^2}$ .
Line 13 changes  $Q_{i}$  by appending  $Q_{i}^{-1}w$  to the sequence  $C_i$  of vectors from which  $Q$  is calculated according to Eq. (4). Eq. (5) lists the conditions on the new sequence  $C_i$  that need to be satisfied for  $Q$  to stay a valid preconditioning. Consider the third condition, i.e.,  $\| Q_{i}^{-1}w\|_{2}\leq L_{3}$ . Observe that  $Q_{i}^{-1}\mathrm{Proj}_{Z(Q,i)}\leq L_3^2 I$  and  $\| v\| _2 = 1$ , therefore  $\| Q_{i}^{-1}w\|_{2} = \| Q_{i}^{-1}\mathrm{Proj}_{Z(Q,i)}v\|_{2}\leq L_{3}$ .
Now consider the second condition. To prove that it holds, we need to show that  $\left\| Q_iQ_i^{-1}w\right\|_2 = \| w\|_2 \geq \frac{1}{2}$ . Let  $x = \| w\|_2^{-1}$ . Since  $\nu$  was the argument of the optimization problem, and using Lemma E.1,
$$
x ^ {2} w ^ {\top} \mathbf {M} w \leq v ^ {\top} \mathbf {M} v \leq w ^ {\top} \mathbf {M} w + \frac {\varepsilon}{d H ^ {2} \omega} \leq w ^ {\top} \mathbf {M} w (1 + 1 / 2)
$$
Therefore,  $\| w\| _2^2\geq \frac{2}{3}$  . We immediately get that
$$
\left\| Q _ {i} Q _ {i} ^ {- 1} w \right\| _ {2} ^ {2} \geq \frac {2}{3},
$$
satisfying the second condition.
It remains to prove that the first condition also holds. First, noting that  $\mathbf{M}$  is symmetric, we can decompose  $w^{\top} \mathbf{M} w$  as
$$
w ^ {\top} \mathbf {M} w = w _ {\parallel} ^ {\top} \mathbf {M} w + w _ {\parallel} ^ {\top} \mathbf {M} w _ {\perp} + w _ {\perp} ^ {\top} \mathbf {M} w _ {\perp}.
$$
Applying Lemma E.3 on the first two terms,
$$
w ^ {\top} \mathbf {M} w \leq 2 \bar {\sigma} _ {k} ^ {m} \beta + 6 \frac {\varepsilon}{d H ^ {2} \omega} + w _ {\perp} ^ {\top} \mathbf {M} w _ {\perp}.
$$
Due to  $\omega > 3$  and  $w^{\top}\mathbf{M}w > \bar{\sigma}_{k}^{m}\beta\omega + 2\frac{\varepsilon}{dH^{2}}$  and the above,  $w_{\perp} \neq \mathbf{0}$ . Let  $w' = w_{\perp} / \|w_{\perp}\|_{2}$ . Since  $\nu$  was the argument of the optimization problem, have that  $\nu^{\top}\mathbf{M}\nu \geq w'^{\top}\mathbf{M}w'$ . Putting this together,
$$
\| w _ {\perp} \| _ {2} ^ {- 2} w _ {\perp} ^ {\top} \mathbf {M} w _ {\perp} = w ^ {\prime \top} \mathbf {M} w ^ {\prime} \leq v ^ {\top} \mathbf {M} v \leq w ^ {\top} \mathbf {M} w + \frac {\varepsilon}{d H ^ {2} \omega} \leq 2 \bar {\sigma} _ {k} ^ {m} \beta + 7 \frac {\varepsilon}{d H ^ {2} \omega} + w _ {\perp} ^ {\top} \mathbf {M} w _ {\perp},
$$
Since  $v^{\top}\mathbf{M}v > \bar{\sigma}_{k}^{m}\beta \omega + 3\frac{\varepsilon}{dH^{2}}$ ,  $w_{\perp}^{\top}\mathbf{M}w_{\perp} \geq (\omega - 7/3)\left(\bar{\sigma}_{k}^{m}\beta + 3\frac{\varepsilon}{dH^{2}\omega}\right) > 0$  and therefore dividing the above by  $w_{\perp}^{\top}\mathbf{M}w_{\perp}$ ,
$$
\left\| w _ {\perp} \right\| _ {2} ^ {- 2} \leq \frac {7 / 3}{\omega - 7 / 3} + 1
$$
$$
\left\| w _ {\perp} \right\| _ {2} ^ {2} \geq \frac {1}{1 + c} \quad \text {f o r} c = \frac {7 / 3}{\omega - 7 / 3}
$$
$$
\left\| w _ {\parallel} \right\| _ {2} ^ {2} \leq 1 - \frac {1}{1 + c} \quad \text {a s} \| w \| _ {2} \leq 1.
$$
Now to prove that the first condition also holds,
$$
\begin{array}{l} \sup  _ {\theta \in \Theta_ {i}} \left| \left\langle \theta , Q _ {i} ^ {- 1} w \right\rangle \right| = \sup  _ {\theta \in \Theta_ {i} ^ {Q}} | \langle \theta , w \rangle | \leq \sup  _ {\theta \in \Theta_ {i} ^ {Q}} \| \theta \| _ {2} \left\| w _ {\parallel} \right\| _ {2} + \sup  _ {\theta \in \Theta_ {i} ^ {Q}} | \langle \theta , w _ {\perp} \rangle | \\ \leq \sqrt {d _ {1} + 1} \sqrt {1 - \frac {1}{1 + c}} + \sup  _ {\theta \in \Theta_ {i} ^ {Q}} \| \theta \| _ {V (G _ {h} ^ {Q}, \rho_ {h} ^ {Q}) ^ {\dagger}} \| w _ {\perp} \| _ {V (G _ {h} ^ {Q}, \rho_ {h} ^ {Q})} \\ \leq \sqrt {d _ {1} + 1} \sqrt {1 - \frac {1}{1 + c}} + \sqrt {2 d w _ {\perp} ^ {\top} (\gamma I) w _ {\perp}} \\ \leq \sqrt {d _ {1} + 1} \sqrt {1 - \frac {1}{1 + c}} + \sqrt {2 d \gamma} = \sqrt {d _ {1} + 1} \sqrt {1 - \frac {1}{1 + c}} + \frac {1}{2}, \\ \end{array}
$$
where in the second line we used Lemma 4.3 to bound  $\sup_{\theta \in \Theta_i^Q} \| \theta \|_2$ , and for the second term we used Eq. (26) with Cauchy-Schwarz. In the third line we used Eq. (24), and the definition of  $\mathrm{Proj}_{\perp}$ . Finally in the last line we use that  $w_\perp$  is perpendicular to  $a_i$  for  $i \leq d'$  (by definition) and that  $\lambda_i \leq \gamma$  for  $i > d'$ . It is left to prove that  $\sqrt{d_1 + 1} \sqrt{1 - \frac{1}{1 + c}} \leq \frac{1}{2}$ . This holds if  $c \geq 1 / (4(d_1 + 1) - 1)$ , which is satisfied as  $c = 1 / (3(d_1 + 1))$ , due to  $\omega = 7(d_1 + 1) + 7/3$  (Eq. (13)).
I Deferred proofs for Appendix E.2
Proof of Lemma E.5. The features  $\varphi_{\mathrm{p}(k)}^{lkj}$  are observed by SKIPPYELEANOR in the order of increasing  $l$ , within that increasing  $k$ , and within that, increasing  $j$ . Each time the next  $\varphi_{\mathrm{p}(k)}^{lkj}$  is observed, we sum the elliptic potential as follows.
For  $i \in [m], r \in [H], u \in [n], t \in [H]$ , let the set of indices observed before  $\varphi_{\mathrm{p}(r)}^{iru}$  whose Phase II (rollout phase) starts at some stage  $t$  be:
$$
\mathbf {I} ^ {i r u} (t) = \{l \in [ i ], k \in [ H ], j \in [ n ]: l H n + k n + j <   i H n + r n + u \text {a n d} p (l k j) = t \}
$$
Let a version of this where only the whole iteration  $i$ 's data is included be
$$
\mathbf {J} ^ {i} (t) = \{l = i, k \in [ H ], j \in [ n ]: p (l k j) = t \}
$$
$$
X _ {i r u} (t) = \lambda I + \sum_ {l k j \in \mathbf {I} ^ {i r u} (t)} \varphi_ {\mathrm {p} (k)} ^ {l k j} \varphi_ {\mathrm {p} (k)} ^ {l k j} ^ {\top}
$$
Observe that  $X_{it}$ , defined in Optimization Problem 4.10, is the version of this that only updates at the start of each iteration  $i$ , that is,
$$
X _ {i t} = X _ {i 1 1} (t).
$$
The total elliptic potential, observed by the end of iteration  $m$  is, writing  $k = \mathrm{p}(iru)$  on the left hand side:
$$
\sum_ {i \in [ m ], r \in [ H ], u \in [ n ]} \mathbb {1} \left\{k <   H + 1 \right\} \min  \left\{1, \left\| \varphi_ {k} ^ {i r u} \right\| _ {X _ {i r u} (k) ^ {- 1}} ^ {2} \right\} = \sum_ {i \in [ m ], t \in [ H ]} \sum_ {l k j \in J ^ {i} (t)} \min  \left\{1, \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {l k j} (t) ^ {- 1}} ^ {2} \right\}.
$$
Applying the elliptical potential lemma (Lemma L.1)  $H$  times for  $t \in [H]$ , this can be bounded as
$$
\sum_ {t \in [ H ], i \in [ m ]} \sum_ {l k j \in \mathbf {J} ^ {i} (t)} \min  \left\{1, \left\| \varphi_ {t} ^ {l k j} \right\| _ {X _ {l k j} (t) ^ {- 1}} ^ {2} \right\} \leq 2 d H \log \left(1 + \frac {H m n L _ {1} ^ {2}}{d \lambda}\right)
$$
On the other hand, by Lemma L.2, then switching to an  $\ell_1$ -bound, then observing that by definition,  $\sum \bar{\sigma}_k^i$  sums the same quantities but caps them by some threshold,