SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widetilde {w} ^ {(t)} := \sum_ {i \in \mathcal {I} ^ {(t)}} \beta_ {i} w ^ {(\ell_ {i})} = \sum_ {i \in \mathcal {I} ^ {(t)}} \beta_ {i} v _ {i} ^ {(t - 1)}, \tag {26}
$$
and note that by the triangle inequality and the fact that  $w^{(t)} = \sum_{i=1}^{d} \beta_i v_i^{(t-1)}$ , we have
$$
\left\| \widetilde {w} ^ {(t)} - w ^ {(t)} \right\| \leq (1 + C) \varepsilon_ {\text {s t a t}}. \tag {27}
$$
Finally, with the notation in (22), define
$$
\widehat {w} _ {t} ^ {(t)} := \sum_ {i \in \mathcal {I} ^ {(t)}} \beta_ {i} \widehat {w} _ {\ell_ {i}} ^ {(t)}, \tag {28}
$$
$$
\widehat {w} _ {t} ^ {(t)} \in (1 + C) \mathcal {B} (2 d ^ {3 / 2}),
$$
since  $|\beta_i| \leq (1 + C)$  for all  $i \in [d]$ ,  $|\mathcal{I}^{(t)}| \leq d$ , and  $\widehat{w}_{\ell}^{(t)} \in \mathcal{B}(2\sqrt{d})$ , for all  $\ell \in [t - 1]$ . Now, by Lemma E.1, in particular (23), we have
$$
\sum_ {i \in \mathcal {I} ^ {(t)}} \mathbb {E} _ {\pi \sim P} \mathbb {E} ^ {\pi \circ_ {h} \pi_ {\mathrm {u n i f}}} \left[ \left(\phi^ {(t)} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} \widehat {w} _ {\ell_ {i}} ^ {(t)} - \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w ^ {(\ell_ {i})}\right) ^ {2} \right] \leq t \bar {\varepsilon} _ {\text {s t a t}} ^ {2}, \tag {29}
$$
where  $\tilde{\varepsilon}_{\mathrm{stat}}$  is as in (24). Using the expressions in Eqs. (26) and (28) with (29) and Jensen's inequality, we have that under  $\mathcal{E}^{(t)}$ ,
$$
\begin{array}{l} \mathbb {E} _ {\pi \sim P} \mathbb {E} ^ {\pi \circ_ {h} \pi_ {\mathrm {u n i f}}} \left[ \left(\phi^ {(t)} (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) ^ {\top} \widetilde {w} _ {t} ^ {(t)} - \phi_ {h} ^ {\star} (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) ^ {\top} \widetilde {w} ^ {(t)}\right) ^ {2} \right] \\ \leq \left(\sum_ {j \in \mathcal {I} ^ {(t)}} | \beta_ {j} |\right) \cdot \sum_ {i \in \mathcal {I} ^ {(t)}} \mathbb {E} _ {\pi \sim P} \mathbb {E} ^ {\pi \circ_ {h} \pi_ {\mathrm {u n i f}}} \left[ \left(\phi^ {(t)} (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) ^ {\top} \widehat {w} _ {\ell_ {i}} ^ {(t)} - \phi_ {h} ^ {\star} (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) ^ {\top} w ^ {(\ell_ {i})}\right) ^ {2} \right], \\ \leq (1 + C) d t \tilde {\varepsilon} _ {\mathrm {s t a t}} ^ {2}. \\ \end{array}
$$
Now, using (27) and the facts that  $(a + b)^2\leq 2a^2 +2b^2$  and  $\| \phi_h^\star \| _2\leq 1$  , we have that
$$
\begin{array}{l} \mathbb {E} _ {\pi \sim P} \mathbb {E} ^ {\pi \circ_ {h} \pi_ {\mathrm {u n i f}}} \left[ \left(\phi^ {(t)} (\pmb {x} _ {h}, \pmb {a} _ {h}) ^ {\top} \widehat {w} _ {t} ^ {(t)} - \phi_ {h} ^ {\star} (\pmb {x} _ {h}, \pmb {a} _ {h}) ^ {\top} w ^ {(t)}\right) ^ {2} \right] \leq 2 (1 + C) ^ {2} \varepsilon^ {2} + 2 (1 + C) d t \bar {\varepsilon} _ {\mathrm {s t a t}} ^ {2}, \\ \leq 2 (1 + C) ^ {2} \tilde {\varepsilon} _ {\mathrm {s t a t}} ^ {2} + 2 (1 + C) d t \tilde {\varepsilon} _ {\mathrm {s t a t}} ^ {2}. \\ \end{array}
$$
Using that  $C = 1/2$ , we conclude that the right-hand side of this inequality is bounded by  $8dt\tilde{\varepsilon}_{\mathrm{stat}}^2$  which is a contradiction, since  $\widehat{w}_t^{(t)} \in (1 + C)\mathcal{B}(2d^{3/2}) = \mathcal{B}(3d^{3/2})$  and by Lemma E.1, we must have
$$
\inf  _ {w \in \mathcal {B} \left(3 d ^ {3 / 2}\right)} \mathbb {E} _ {\pi \sim P} \mathbb {E} ^ {\pi \circ_ {h} \pi_ {\mathrm {u n i f}}} \left[ \left(\phi^ {(t)} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w - \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w ^ {(t)}\right) ^ {2} \right] > 8 t \tilde {\varepsilon} _ {\text {s t a t}}
$$
if RepLearn does not terminate at round  $t$ . Therefore, we have that  $\mathcal{J}^{(t)} \neq \emptyset$ , for any iteration  $t \in [2..N]$  where RepLearn does not terminate.
We now bound the iteration count and prove that the guarantee in Eq. (21) holds at termination. Note that whenever  $\mathcal{J}^{(\ell)}\neq \emptyset$  for  $\ell >1$ , we have by construction:
$$
| \det  (V ^ {(\ell)}) | > 3 / 2 \cdot | \det  (V ^ {(\ell - 1)}) |.
$$
Thus, if RepLearn runs for  $t \in [2..N]$  iterations, then
$$
| \det (V ^ {(t)}) | > (3 / 2) ^ {t - 1} \cdot | \det (V ^ {(1)}) |. \tag {30}
$$
On the other hand, since the determinant of a matrix is bounded by the product of the norms of its columns and  $v_{1:d}^{(t)} \in \mathcal{B}(2\sqrt{d})$ , we have
$$
| \det  (V ^ {(t)}) | \leq 2 ^ {d} d ^ {d / 2}.
$$
Note also that  $|\operatorname{det}(V^{(0)})| = (\varepsilon / d)^d$ . Plugging this into (30), we conclude that
$$
(3 / 2) ^ {t - 1} <   (2 d ^ {3 / 2} / \varepsilon) ^ {d}.
$$
Taking the logarithm on both sides and rearranging yields
$$
t <   1 + d \log_ {3 / 2} (2 d ^ {3 / 2} / \varepsilon) \leq N.
$$
Thus, the algorithm must terminate after at most  $N - 1$  iterations. Furthermore, by [35, Lemma 14], we have that with probability at least  $1 - \frac{\delta}{2N}$ , if the algorithm terminates at iteration  $t$ , then
$$
\begin{array}{l} \max  _ {f \in \mathcal {F}} \inf  _ {w \in \mathcal {B} (3 d ^ {3 / 2})} \mathbb {E} _ {\pi \sim P} \mathbb {E} ^ {\pi \circ_ {h} \pi_ {\text {u n i f}}} \left[ \left(w ^ {\top} \phi^ {(t)} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) - w _ {f} ^ {\top} \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right)\right) ^ {2} \right] \leq 3 2 t \tilde {\varepsilon} _ {\text {s t a t}} ^ {2}, \\ \leq 3 2 (N - 1) \hat {\varepsilon} _ {\mathrm {s t a t}} ^ {2}, \\ \leq 3 2 T \bar {\varepsilon} _ {\text {s t a t}} ^ {2}. \\ \end{array}
$$
Applying a union bound completes the proof.
F Analysis
In this section, we prove the main guarantee for SpanRL (Theorem 3.2). First, we outline our proof strategy in Appendix F.1. Then, in Appendix F.2 and Appendix F.3, we present guarantees for the instances of PSDP (Algorithm 3) and RobustSpanner (Algorithm 2) used within SpanRL. We then combine these results in Appendix F.5 to complete the proof of Theorem 3.2. A self-contained guarantee for RobustSpanner(Lemma 3.1) is given in Appendix F.6.
F.1 Proof Strategy
Like our algorithm, our analysis is inductive. For fixed  $h$ , we assume that the policy set  $\Psi^{(1:h+1)}$  produced by SpanRL satisfies the property:
$$
\Psi^ {(1)}, \dots \Psi^ {(h + 1)} \text {a r e} \left(\frac {1}{4 A d}, 0\right) \text {- p o l i c y c o v e r s f o r l a y e r s 1 t h r o u g h} h + 1, \text {a n d} \max  _ {t \in [ h + 1 ]} | \Psi^ {(t)} | \leq d. \tag {31}
$$
Conditioned on this claim, we show that with high probability, the set  $\Psi^{(h + 2)}$  is a  $(\frac{1}{4Ad},0)$ -policy cover for layer  $h + 2$ . To prove this, we use the inductive assumption to show that PSDP acts as an approximate linear optimization oracle over  $\mathcal{W} = \{\mathbb{E}^{\pi}\left[\phi^{(h)}(\pmb{x}_h,\pmb{a}_h)\right] \mid \pi \in \Pi_{\mathsf{M}}\}$  (Appendix F.2). Using this, we then instantiate the guarantee of RobustSpanner from Lemma F.3 with LinOpt and LinEst instantiated with PSDP and EstVec. To conclude the proof of the inductive step, we the main guarantee for RobustSpanner together with the main guarantee for RepLearn (Theorem E.1), along with a change of measure argument enabled by the assumption that  $\Psi^{(1:h)}$  are policy covers (i.e. (31)).
F.2 Guarantee for PSDP as a Subroutine for RobustSpanner
We begin by showing that PSDP instantiates the approximate linear optimization oracle required by RobustSpanner. In particular, we fix a layer  $h$  and assume that  $\Psi^{(1:h+1)}$  satisfy (31) and apply the results of Appendix D.
More precisely, we need to show that, for any  $\theta \in \mathbb{R}^d\setminus \{0\}$  and  $\phi \in \Phi$  ,PSDP approximately solves
$$
\max  _ {\pi \in \Pi_ {M}} \theta^ {\top} \mathbb {E} ^ {\pi} [ \phi (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) ]. \tag {32}
$$
We can equivalently formulate (32) as, for fixed  $\theta \in \mathbb{R}^d\setminus \{0\}$  and  $\phi \in \Phi$  , maximizing the sum of the reward functions  $r_{1:h}(\cdot ,\cdot ;\theta ,\phi)$  given by:
$$
\forall (x, a) \in \mathcal {X} \times \mathcal {A}, \quad r _ {t} (x, a; \theta , \phi) := \left\{ \begin{array}{l l} \phi (x, a) ^ {\top} \frac {\theta}{\| \theta \|}, & \text {f o r} t = h, \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {33}
$$
Note that this matches the choice of reward functions in SpanRL (Algorithm 1) at iteration  $h$  with  $\phi = \hat{\phi}^{(h)}$ , the feature map returned by RepLearn in Line 8. With these reward functions and the function class
$$
\mathcal {G} := \left\{g: (x, a) \mapsto \phi (x, a) ^ {\top} w \mid \phi \in \Phi , w \in \mathcal {B} (\sqrt {d}) \right\}, \tag {34}
$$
we show that the output  $\hat{\pi} = \mathrm{PSDP}(h, r_{1:h}(\cdot, \cdot; \theta, \phi), \mathcal{G}, \Psi^{(1:h)}, n)$  approximately solves (32) with high probability if  $n \geq 1$  is sufficiently large. We first verify that the class  $\mathcal{G}$  realizes the reward functions specified in (33) in the sense of Definition D.1.
Lemma F.1. Under Assumption 2.2, the function class  $\mathcal{G}$  in (34) realizes the reward functions in (33), for any  $\phi \in \Phi$  and  $\theta \in \mathbb{R}^d \setminus \{0\}$ . Furthermore, we have that functions in  $\mathcal{G}$  are uniformly bounded by  $h \leq H$ , and  $\log \mathcal{N}_{\mathcal{G}}(\varepsilon) \leq \log |\Phi| + d\log (H / \varepsilon)$ , where we recall that  $\mathcal{N}_{\mathcal{G}}(\varepsilon)$  denotes the  $\varepsilon$ -covering number of  $\mathcal{G}$  in  $\ell_{\infty}$  distance.
Proof. Fix  $\phi \in \Phi$  and  $\theta \in \mathbb{R}^d \setminus \{0\}$ , and let  $r_\ell(\cdot, \cdot) \equiv r_\ell(\cdot, \cdot; \theta, \phi)$ , for  $\ell \in [h]$ . For  $t = h$ , we clearly have that for any  $\pi \in \Pi_{\mathfrak{M}}^{h:h}$ ,  $Q_t^\pi(\cdot, \cdot) = r_t(\cdot, \cdot) \in \mathcal{G}$ . For  $t < h$  and  $\pi \in \Pi_{\mathfrak{M}}^{t:h}$ , we have by the low-rank structure that
$$
\begin{array}{l} Q _ {t} ^ {\pi} (x, a) = \int_ {\mathcal {X} _ {t + 1}} \mathbb {E} ^ {\pi} \left[ r _ {h} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \mid \boldsymbol {x} _ {t + 1} = y, \boldsymbol {a} _ {t + 1} = \pi (y) \right] \cdot \phi_ {t} ^ {\star} (x, a) ^ {\top} \mu_ {t + 1} ^ {\star} (y) \mathrm {d} \nu (y), \\ = \phi_ {t} ^ {\star} (x, a) ^ {\top} \left(\int_ {\mathcal {X} _ {t + 1}} \mathbb {E} ^ {\pi} \left[ r _ {h} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \mid \boldsymbol {x} _ {t + 1} = y, \boldsymbol {a} _ {t + 1} = \pi (y) \right] \cdot \mu_ {t + 1} ^ {\star} (y) \mathrm {d} \nu (y)\right). \tag {35} \\ \end{array}
$$
Now, by the fact that  $\mathbb{E}^{\pi}[r_h(\pmb{x}_h,\pmb{a}_h)\mid \pmb{x}_{t + 1} = y,\pmb{a}_{t + 1} = \pi (y)]\in [-1,1]$ , for all  $y\in \mathcal{X}_{t + 1}$  (since  $\phi (\cdot ,\cdot)\in \mathcal{B}(1)$ , for all  $\phi \in \Phi$ ), and the normalizing assumption made on  $(\mu_h^\star)_{h\in [H]}$  in Section 2.2 (i.e. that for all  $g:\mathcal{X}_{t + 1}\to [0,1]$ ,  $\| \int_{\mathcal{X}_{t + 1}}\mu_{t + 1}^{\star}(y)g(y)\mathrm{d}\nu (y)\| \leq \sqrt{d}$ ), we have that
$$
w _ {t} := \int_ {\mathcal {X} _ {t + 1}} \mathbb {E} ^ {\pi} \left[ r _ {h} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \mid \boldsymbol {x} _ {t + 1} = y, \boldsymbol {a} _ {t + 1} = \pi (y) \right] \cdot \mu_ {t + 1} ^ {\star} (y) \mathrm {d} \nu (y) \in \mathcal {B} (\sqrt {d}).
$$
This together with (35) and the fact that  $\phi_t^\star \in \Phi$  (by Assumption 2.2), we have that  $Q_{t}^{\pi}\in \mathcal{G}$ . Combining Lemma F.1 with Theorem D.2 results in the following bound on the quality of PSDP as an approximate linear optimization oracle over the space of policies.
Corollary F.1. Let  $\varepsilon, \delta \in (0,1)$  and  $h \in [H]$ . Further, let  $\theta \in \mathbb{R}^d \setminus \{0\}$ ,  $\phi \in \Phi$ , and  $\hat{\pi} = \mathrm{PSDP}(h, r_{1:h}(\cdot, \cdot; \theta, \phi), \mathcal{G}, \Psi^{(1:h)}, n)$ , where
- The reward functions  $r_{1:h}(\cdot, \cdot; \theta, \phi)$  are as in (33).
- The function class  $\mathcal{G}$  is as in (34).
- The collection of policies  $\Psi^{(1:h)}$  satisfy (31).
- The parameter  $n$  is chosen such that  $cHACd^2 \cdot \varepsilon_{\mathrm{stat}}(n,\delta / H) \leq \varepsilon$ , where  $\varepsilon_{\mathrm{stat}}(n,\delta') \coloneqq \sqrt{dn^{-1} \cdot (d\log(nH) + \log(|\Phi| / \delta'))}$  and  $c > 0$  is some large enough absolute constant.
Then, under Assumption 2.2, with probability at least  $1 - \delta$ , we have that
$$
\max  _ {\pi \in \Pi_ {\mathbb {M}}} \theta^ {\top} \mathbb {E} ^ {\pi} \left[ \phi \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right] \leq \theta^ {\top} \mathbb {E} ^ {\hat {\pi}} \left[ \phi \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right] + \varepsilon / 2.
$$
We emphasize that the inductive assumption that  $\Psi^{(1:h)}$  is a policy cover of bounded size enters only in the statement of Theorem D.2. We now give a guarantee for RobustSpanner as used in SpanRL.
Algorithm 7 EstVec  $(h,F,\pi ,n)$  : Estimate  $\mathbb{E}^{\pi}[F(\pmb{x}_h,\pmb{a}_h)]$  for given policy  $\pi$  and function  $F$ .
1:  $\mathcal{D}\gets \emptyset$
2: for  $n$  times do
3: Sample  $(\pmb{x}_h, \pmb{a}_h) \sim \pi$
4: Update dataset:  $\mathcal{D}\gets \mathcal{D}\cup \{(\pmb {x}_h,\pmb {a}_h)\} .$
5: Return:  $\bar{F} = \frac{1}{n}\sum_{(x,a)\in \mathcal{D}}F(x,a)$
F.3 Guarantee for RobustSpanner as a Subroutine for SpanRL
In this section, we prove a guarantee for the instantiation of RobustSpanner in SpanRL, which we require in the proof of the main theorem (Theorem 3.2). We first show that the LinEst subroutine passed to RobustSpanner can be taken to be EstVec (Algorithm 7), which simply estimates the expected feature imbedding of  $(\boldsymbol{x}_h,\boldsymbol{a}_h)$  under policy  $\pi$  by sampling sufficiently many trajectories and taking the empirical mean.
Lemma F.2 (Guarantee of EstVec). Let  $\delta \in (0,1)$  and  $\varepsilon > 0$ . For  $h \in [H]$ ,  $\phi \in \Phi$ ,  $\pi \in \Pi_{M}$ , and  $n \in \mathbb{N}$  such that  $n \geq \frac{c}{\varepsilon^2} \log(d / \delta)$  for some large enough absolute constant  $c > 0$ , the output  $\bar{\phi}_h = \text{EstVec}(h, \phi, \pi, n)$  (Algorithm 7) satisfies, with probability at least  $1 - \delta$ ,
$$
\left\| \bar {\phi} _ {h} - \mathbb {E} ^ {\pi} \left[ \phi \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right] \right\| \leq \varepsilon / 2.
$$
Proof. By Hoeffding's inequality (see for example [22, Corollary 7]) and the fact that  $\| \phi (x,a)\| \leq 1$  for all  $x\in \mathcal{X}$  and  $a\in \mathcal{A}$ , there exists an absolute constant  $c > 0$  such that with probability at least  $1 - \delta$ ,
$$
\left\| \bar {\phi} _ {h} - \mathbb {E} ^ {\pi} \left[ \phi \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right] \right\| \leq c \cdot \sqrt {\frac {\log (d / \delta)}{n}}.
$$
Setting  $n$  as in the statement of the theorem concludes the proof.
In SpanRL, we instantiate RobustSpanner passing PSDP as LinOpt and EstVec as LinEst. Combining Corollary F.1 and Lemma F.2 with the general guarantee of RobustSpanner in Proposition C.1, we have the following result.
Lemma F.3. Consider iteration  $h \in [H]$  of SpanRL  $(\Phi, \varepsilon, \mathfrak{c}, \delta)$  (Algorithm 1) with  $\varepsilon, \mathfrak{c} > 0$ ,  $\delta \in (0,1)$ , and feature class  $\Phi$  satisfying Assumption 2.2. Further, let  $\hat{\phi}^{(h)}$  denote the feature map returned by RepLearn in Algorithm 1 at iteration  $h$ . If  $\Psi^{(1:h)}$  satisfy (31) and  $\mathfrak{c} = \mathrm{polylog}(A,d,H,\log(|\Phi| / \delta))$  is large enough, then there is an event  $\mathcal{E}_h$  with probability at least  $1 - \frac{\delta}{2H}$  such that
- The number of iterations of RobustSpanner in Line 12 of Algorithm 1 is at most  $N = \left\lceil \frac{d}{2} \log_2 \left( \frac{100d}{\varepsilon} \right) \right\rceil < \infty$ , and
- The output  $(\pi_1, \ldots, \pi_d)$  of RobustSpanner has the property that for all  $\pi \in \Pi_{\mathbb{M}}$ , there exist  $\beta_1, \ldots, \beta_d \in [-2, 2]$  such that
$$
\left\| \hat {\phi} ^ {(h), \pi} - \sum_ {i = 1} ^ {d} \beta_ {i} \hat {\phi} ^ {(h), \pi_ {i}} \right\| \leq 3 d \varepsilon , \quad w h e r e \quad \hat {\phi} ^ {(h), \pi^ {\prime}} := \mathbb {E} ^ {\pi^ {\prime}} \left[ \hat {\phi} ^ {(h)} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right].
$$
Proof. By Proposition C.1, on the event that the instances of PSDP and EstVec used by RobustSpanner satisfy Assumption C.1 with  $\varepsilon' = \frac{\varepsilon}{2}$ , the two desiderata of the lemma hold. $^{10}$  We
<sup>10</sup>Here, we instantiate the guarantee in Proposition C.1 with  $C = 2$ ; this is what  $C$  is set to in Algorithm 1.
claim that each call to PSDP and to EstVec satisfies Assumption C.1 with probability at least  $1 - \frac{\delta}{8dNH}$ . Because each of PSDP and EstVec get called at most  $4dN$  times per iteration of RobustSpanner, a union bound concludes the proof contingent on the above claim.
We now prove the claim. First, note that the instance of PSDP that RobustSpanner uses within Algorithm 1 is of the form:
$$
\operatorname {P S D P} \left(h, r _ {1: h} (\cdot , \cdot , \theta), \mathcal {G}, \Psi^ {1: h}, n _ {\text {P S D P}}\right)
$$
with  $r_{1:h}$  and  $\mathcal{G}$  as in Algorithm 1; this matches the form in Corollary F.1 (PSDP's guarantee) with  $\phi = \hat{\phi}^{(h)}$ . Thus, by choosing
$$
n _ {\text {P S D P}} = \mathfrak {c} \cdot \frac {A ^ {2} d ^ {5} H ^ {2} \cdot (d \log (H) + \log (8 d H ^ {2} N | \Phi | / \delta))}{\varepsilon^ {2}},
$$
for  $\mathfrak{c} = \mathrm{polylog}(A,d,H,\log (|\Phi | / \delta))$  sufficiently large, the conditions of Corollary F.1 are satisfied, and its conclusion implies the claim for the PSDP instance used by RobustSpanner. Similarly, the choice of  $n_{\mathrm{EstVec}}$  in Algorithm 1 ensures that the claim holds for the instance of EstVec that RobustSpanner uses by Lemma F.2. The result follows.
F.4 Guarantee for RepLearn as a Subroutine for SpanRL
In this section, we prove a guarantee for the invocation of RepLearn within SpanRL
Recall that  $P^{(h)} = \mathrm{unif}(\Psi^{(h)})$  is the distribution over policies that SpanRL passes to RepLearn at iteration  $h \in [H - 2]$  to compute feature map  $\phi^{(h)}$ . Thus, by invoking Theorem E.1 in Appendix E and using the choice of  $n_{\mathrm{RepLearn}}$  in Algorithm 1, we immediately obtain the following corollary.
Corollary F.2. Let  $\delta, \varepsilon \in (0,1)$ , and  $\mathcal{F}$  be as in Algorithm 1, and fix  $h \in [H - 2]$ . Suppose that the feature class  $\Phi$  satisfies Assumption 2.2. Then, with probability at least  $1 - \frac{\delta}{2H}$ , the instance of RepLearn in Line 9 of Algorithm 1 runs for  $t \leq \mathfrak{c} \cdot d$  iterations for  $\mathfrak{c} = \mathrm{polylog}(A,d,H,\log(|\Phi| / \delta))$  sufficiently large, and returns output  $\phi^{(h)}$  such that for all  $f \in \mathcal{F}$ , there exists  $w_f^{(h)} \in \mathcal{B}(3d^{3/2})$  satisfying
$$
\mathbb {E} ^ {\text {u n i f} (\Psi^ {(h)})} \left[ \sum_ {a \in \mathcal {A}} \left(\phi^ {(h)} \left(\boldsymbol {x} _ {h}, a\right) ^ {\top} w _ {f} ^ {(h)} - \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, a\right) ^ {\top} w _ {f}\right) ^ {2} \right] \leq \frac {\eta^ {2}}{6 4 A ^ {2} d ^ {2}},
$$
where  $w_{f}\coloneqq \int_{\mathcal{X}_{h + 1}}f(y)\mu_{h + 1}^{\star}(y)\mathrm{d}\nu (y).$
F.5 Concluding the Proof of Theorem 3.2
In this section, we conclude the proof of the main guarantee (Theorem 3.2). We derive the guarantee from the following inductive claim.
Theorem F.1. Consider iteration  $h \in [H]$  of SpanRL  $(\Phi, \varepsilon, \mathfrak{c}, \delta)$  (Algorithm 1) with parameters  $\varepsilon, \mathfrak{c} > 0$ ,  $\delta \in (0,1)$  and a feature class  $\Phi$  satisfying Assumption 2.2. Further, assume that:
- The collection of policies  $\Psi^{(1:h+1)}$  at the start of the  $h$ th iteration of SpanRL satisfy (31).
Assumption 2.1 (reachability) holds with  $\eta >0$
The input parameter  $\varepsilon$  to SpanRL is set to  $\varepsilon = \frac{\eta}{36d^{5 / 2}}$ .
- The input parameter  $\mathfrak{c} = \mathrm{polylog}(A, d, H, \log(|\Phi| / \delta))$  is sufficiently large.
Then, with probability at least  $1 - \frac{\delta}{H}$ , the set of policies  $\Psi^{(h + 2)}$  produced by SpanRL  $(\Phi ,\varepsilon ,\mathfrak{c},\delta)$  at the end of iteration  $h$  is an  $(\frac{1}{4Ad},0)$ -policy cover for layer  $h + 2$ .
With this, we can now prove Theorem 3.2.
Proof of Theorem 3.2. Note that it suffices to prove that (31) holds for  $h = H - 1$  with probability at least  $1 - \delta$ . To do this, we proceed by induction over  $h = 1, \ldots, H - 1$ . The base case of  $h = 1$  trivially holds because  $\Psi^{(1)} = \emptyset$  and  $\Psi^{(2)} = \{\pi_{\mathrm{unif}}\}$ . The induction step now follows by Theorem F.1 and the union bound (see e.g. [30, Lemma I.2]).
The number of trajectories used by SpanRL is dominated by calls to PSDP. Since PSDP is called  $O(d\log (d / \varepsilon))$  times at each iteration of SpanRL (Lemma F.3), and each call to PSDP requires at most  $Hn_{\mathrm{PSDP}}$  trajectories, the total number of trajectories after  $H$  iterations of SpanRL is bounded by  $\widetilde{O} (H^{2}dn_{\mathrm{PSDP}})$ . By plugging the choices for  $n_{\mathrm{PSDP}}$  and  $\varepsilon$  from the theorem statement, we obtain the claimed sample complexity.
Before proving Theorem F.1, we make the following simple observation.
Lemma F.4. For any  $\pi \in \Pi_{\mathsf{M}}$ ,  $h \in [H - 1]$ , any  $x \in \mathcal{X}_{h + 1}$ , we have
$$
\mu_ {h + 1} ^ {\star} (x) ^ {\top} \mathbb {E} ^ {\pi} \left[ \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right] = d ^ {\pi} (x) \geq 0.
$$
Proof of Lemma F.4. The equality follows by construction. The non-negativity of  $d^{\pi}(x)$  follows by definition of a probability density.
We now prove Theorem F.1.
Proof of Theorem F.1. Let  $\mathcal{E}_h$  and  $\mathcal{E}_h^\prime$  denote the success events in Lemma F.3 and Corollary F.2, respectively, and note that by the union bound, we have  $\mathbb{P}[\mathcal{E}_h\cap \mathcal{E}_h^{\prime}]\geq 1 - \delta /H$ . For the rest of this proof, we will condition on  $\mathcal{E}\coloneqq \mathcal{E}_h\cap \mathcal{E}_h^\prime$ .
Throughout, we denote
$$
\phi_ {h} ^ {\star , \pi} := \mathbb {E} ^ {\pi} \left[ \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right], \quad \forall h \in [ H ], \forall \pi \in \Pi_ {\mathfrak {M}}.
$$
Because  $\Psi^{(1:h + 1)}$  satisfy (31) (i.e., are a policy cover) it holds by Lemma F.4 that for all  $x\in \mathcal{X}_h$
$$
\max  _ {\pi \in \Psi^ {(h)}} \mu_ {h} ^ {\star} (x) ^ {\top} \phi_ {h - 1} ^ {\star , \pi} \geq \alpha \cdot \sup  _ {\pi \in \Pi_ {\mathbb {M}}} \mu_ {h} ^ {\star} (x) ^ {\top} \phi_ {h - 1} ^ {\star , \pi}, \quad \text {f o r} \quad \alpha := \frac {1}{4 A d}. \tag {36}
$$
We will show that with probability at least  $1 - \frac{\delta}{H}$ , the policy set  $\Psi^{(h + 2)}$  has the same property for layer  $h + 2$ ; that is, for all  $x \in \mathcal{X}_{h + 1}$
$$
\max  _ {\pi \in \Psi (h + 2)} \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star , \pi} \geq \alpha \cdot \sup  _ {\pi \in \Pi_ {\mathfrak {M}}} \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star , \pi}. \tag {37}
$$
Again, by Lemma F.4 this is equivalent to the statement that  $\Psi^{(h + 2)}$  is an  $(\frac{1}{4Ad},0)$ -policy cover for layer  $h + 2$ .
For the remainder of the proof, we will fix  $x \in \mathcal{X}_{h+2}$  and let  $\pi_x \in \arg \max_{\pi \in \Pi_M} \mu_{h+2}^\star(x)^\top \phi_{h+1}^{\star,\pi}$ . Our goal is to show that the inequality Eq. (37) holds for  $x$ .
Preliminaries. Note that since  $x \in \mathcal{X}_{h + 2}$ , we have  $\| \mu_{h + 2}^{\star}(x) \| > 0$ . It will be convenient to introduce a function  $f: \mathcal{X}_{h + 1} \to \mathbb{R}$  defined by