SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f (y) := \theta_ {x} ^ {\top} \phi_ {h + 1} ^ {\star} (y, \pi_ {x} (y)), \quad \text {w h e r e} \quad \theta_ {x} := \frac {\mu_ {h + 2} ^ {\star} (x)}{\| \mu_ {h + 2} ^ {\star} (x) \|}.
$$
Further, we define
$$
w _ {x} := \int_ {\mathcal {X} _ {h + 1}} f (y) \mu_ {h + 1} ^ {\star} (y) \mathrm {d} \nu (y). \tag {38}
$$
By definition of  $\pi_x$ , we have that for all  $y \in \mathcal{X}_{h+1}$ ,
$$
\theta_ {x} ^ {\top} \phi_ {h + 1} ^ {\star} (y, \pi_ {x} (y)) = \max  _ {a \in \mathcal {A}} \theta_ {x} ^ {\top} \phi_ {h + 1} ^ {\star} (y, a).
$$
This together with the fact that  $\| \theta_x\| = 1$  implies that
$$
f \in \mathcal {F} = \left\{x \mapsto \max  _ {a \in \mathcal {A}} \theta^ {\top} \phi (x, a) \mid \theta \in \mathcal {B} (1), \phi \in \Phi \right\}; \tag {39}
$$
the discriminator class in Line 4 of SpanRL. Note also that since  $x \in \mathcal{X}_{h+2}$ , we have by reachability that
$$
w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {x}} = \theta_ {x} ^ {\top} \phi_ {h + 1} ^ {\star , \pi_ {x}} = \frac {1}{\left\| \mu_ {h + 2} ^ {\star} (x) \right\|} \max  _ {\pi \in \Pi_ {\mathbb {M}}} \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star , \pi} \geq \eta > 0. \tag {40}
$$
Applying the guarantee for RepLearn. Moving forward, let  $\phi^{(h)}$  be the feature map returned by RepLearn at the  $h$ th iteration of Algorithm 1, and define  $\phi^{(h),\pi} \coloneqq \mathbb{E}^{\pi}[\phi^{(h)}(\pmb{x}_h,\pmb{a}_h)]$ , for any  $\pi \in \Pi_{\mathsf{M}}$ . Further, let  $w_x^{(h)}$  be the vector  $w_f^{(h)}$  in Corollary F.2 with  $f = f_x$ , and note that
$$
\left\| w _ {x} ^ {(h)} \right\| \leq 3 d ^ {3 / 2}. \tag {41}
$$
By Jensen's inequality, we compute
$$
\begin{array}{l} \left(\langle w _ {x} ^ {(h)} \rangle \phi^ {(h), \pi_ {x}} - \langle w _ {x} \rangle \phi_ {h} ^ {\star , \pi_ {x}}\right) ^ {2} \\ \leq \mathbb {E} ^ {\pi_ {x}} \left[ \left(\phi^ {(h)} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w _ {x} ^ {(h)} - \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w _ {x}\right) ^ {2} \right], \quad (\text {J e s s e n ' s i n e q u a l i t y}) \\ = \int_ {\mathcal {X} _ {h}} \left(\phi^ {(h)} (y, \pi_ {x} (y)) ^ {\top} w _ {x} ^ {(h)} - \phi_ {h} ^ {\star} (y, \pi_ {x} (y)) ^ {\top} w _ {x}\right) ^ {2} \mu_ {h} ^ {\star} (y) ^ {\top} \phi_ {h - 1} ^ {\star , \pi_ {x}} d \nu (y), \quad (\text {L o w - R a n k M D P}) \\ \leq \alpha^ {- 1} \max  _ {\tilde {\pi} \in \Psi^ {(h)}} \int_ {\mathcal {X} _ {h}} \left(\phi^ {(h)} (y, \pi_ {x} (y)) ^ {\top} w _ {x} ^ {(h)} - \phi_ {h} ^ {\star} (y, \pi_ {x} (y)) ^ {\top} w _ {x}\right) ^ {2} \mu_ {h} ^ {\star} (y) ^ {\top} \phi_ {h - 1} ^ {\star , \tilde {\pi}} \mathrm {d} \nu (y), \quad (\text {b y}) \\ \leq \alpha^ {- 1} \sum_ {\tilde {\pi} \in \Psi^ {(h)}} \int_ {\mathcal {X} _ {h}} \left(\phi^ {(h)} (y, \pi_ {x} (y)) ^ {\top} w _ {x} ^ {(h)} - \phi_ {h} ^ {\star} (y, \pi_ {x} (y)) ^ {\top} w _ {x}\right) ^ {2} \mu_ {h} ^ {\star} (y) ^ {\top} \phi_ {h - 1} ^ {\star , \tilde {\pi}} d \nu (y), \quad (\text {b y}) \\ \leq \alpha^ {- 1} \sum_ {\tilde {\pi} \in \Psi^ {(h)}} \sum_ {a \in \mathcal {A}} \int_ {\mathcal {X} _ {h}} \left(\phi^ {(h)} (y, a) ^ {\top} w _ {x} ^ {(h)} - \phi_ {h} ^ {\star} (y, a) ^ {\top} w _ {x}\right) ^ {2} \mu_ {h} ^ {\star} (y) ^ {\top} \phi_ {h - 1} ^ {\star , \tilde {\pi}} \mathrm {d} \nu (y), \\ = A \alpha^ {- 1} d \cdot \mathbb {E} ^ {\text {u n i f} (\Psi^ {(h)})} \left[ \left(\phi^ {(h)} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w _ {x} ^ {(h)} - \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} w _ {x}\right) ^ {2} \right], \tag {42} \\ \end{array}
$$
where the last step follows by the definition of  $\Psi^{(h)}$  in Algorithm 1 and that  $|\Psi^{(h)}| = d$ . Now, since  $w_{x} = \int_{\mathcal{X}_{h + 1}}f(y)\mu_{h + 1}^{\star}(y)\mathrm{d}\nu (y)$  (see (38)) and  $f\in \mathcal{F}$  (see (39)); the guarantee for RepLearnin Corollary F.2 together with (42) implies that (conditioned on the event  $\mathcal{E}$ )
$$
\left| \langle w _ {x} ^ {(h)} \rangle \phi^ {(h), \pi_ {x}} - \langle w _ {x} \rangle \phi_ {h} ^ {\star , \pi_ {x}} \right| \leq \sqrt {\frac {A d \eta^ {2}}{6 4 \alpha A ^ {2} d ^ {2}}} \leq \frac {\eta}{4}. \tag {43}
$$
Applying the guarantee for RobustSpanner. Letting  $\pi_1, \ldots, \pi_d$  be the policies returned by RobustSpanner at iteration  $h$  of SpanRL, the guarantee of RobustSpanner in Lemma F.3 implies that there exist  $\beta_1, \ldots, \beta_d \in [-2, 2]$  such that
$$
\left\| \phi^ {(h), \pi_ {x}} - \sum_ {i = 1} ^ {d} \beta_ {i} \phi^ {(h), \pi_ {i}} \right\| \leq 3 d \varepsilon \leq \frac {\eta}{1 2 d ^ {3 / 2}}, \tag {44}
$$
where the last inequality follows by the fact that  $\varepsilon = \frac{\eta}{36d^{5/2}}$ . Combining (44) with (43) and using the triangle inequality, we get that
$$
\begin{array}{l} w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {x}} \leq \sum_ {i = 1} ^ {d} \beta_ {i} w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}} + \| w _ {x} ^ {(h)} \| \cdot \frac {\eta}{1 2 d ^ {3 / 2}} + \frac {\eta}{4}, \\ \leq \sum_ {i = 1} ^ {d} \beta_ {i} w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}} + \frac {\eta}{4} + \frac {\eta}{4}, \quad (\text {b y (4 1)}) \\ \leq 2 d \max _ {i \in [ d ]} w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}} + \frac {\eta}{2}. \\ \end{array}
$$
Combining this with (40) and rearranging implies
$$
w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {x}} \leq 4 d \cdot \max  _ {i \in [ d ]} w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}}. \tag {45}
$$
On the other hand, by definition of  $w_{x}$ , we have
$$
\begin{array}{l} \max  _ {i \in [ d ]} w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}} = \max  _ {i \in [ d ]} \theta_ {x} ^ {\top} \phi_ {h + 1} ^ {\star , \pi_ {i} o _ {h + 1} \pi_ {x}}, \\ = \frac {1}{\left\| \mu_ {h + 2} ^ {\star} (x) \right\|} \max _ {i \in [ d ]} \mathbb {E} ^ {\pi_ {i} \circ_ {h + 1} \pi_ {x}} \left[ \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star} (\boldsymbol {x} _ {h + 1}, \boldsymbol {a} _ {h + 1}) \right], \\ \leq \frac {A}{\left\| \mu_ {h + 2} ^ {\star} (x) \right\|} \max  _ {i \in [ d ]} \mathbb {E} ^ {\pi_ {i} \circ_ {h + 1} \pi_ {\text {u n i f}}} \left[ \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star} \left(\boldsymbol {x} _ {h + 1}, \boldsymbol {a} _ {h + 1}\right) \right], \quad (\text {s e e b e l o w}) \\ = \frac {A}{\left\| \mu_ {h + 2} ^ {\star} (x) \right\|} \max  _ {\pi \in \Psi^ {(h + 2)}} \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star , \pi}, \tag {46} \\ \end{array}
$$
where the inequality follows from the non-negativity of  $\mu_{h + 1}^{\star}(\cdot)^{\top}\phi_{h + 1}^{\star}(x,a)$ , for all  $(x,a)\in \mathcal{X}_h\times \mathcal{A}$  (due to Lemma F.4), and (46) follows from the definition of  $\Psi^{(h + 2)}$  in Line 13 of Algorithm 1. Combining (45) and (46) then implies that
$$
\begin{array}{l} \frac {1}{\left\| \mu_ {h + 2} ^ {\star} (x) \right\|} \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star , \pi_ {x}} = \theta_ {x} ^ {\top} \phi_ {h + 1} ^ {\star , \pi_ {x}} = w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {x}} \leq 4 d \cdot \max _ {i \in [ d ]} w _ {x} ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}}, \\ \leq \frac {4 A d}{\left\| \mu_ {h + 2} ^ {\star} (x) \right\|} \max  _ {\pi \in \Psi^ {(h + 2)}} \mu_ {h + 2} ^ {\star} (x) ^ {\top} \phi_ {h + 1} ^ {\star , \pi}. \\ \end{array}
$$
This, together with Lemma F.4, implies that (37) holds. Since this argument holds uniformly for all  $x \in \mathcal{X}_{h+2}$ , this completes the proof.
F.6 Proof of Lemma 3.1
By definition for  $x \in \mathcal{X}_{h+1}$ , we have  $d^{\pi}(x) = \mathbb{E}^{\pi}\left[\mu_{h+1}^{\star}(x)^{\top} \phi_h^{\star}(\boldsymbol{x}_h, \boldsymbol{a}_h)\right]$ . Let  $\pi_x$  denote the policy maximizing  $d^{\pi}(x)$  (if no such maximizer exists, we may pass to a maximizing sequence) and let  $\Psi = \{\pi_1, \dots, \pi_d\}$ . Then, we have for some  $\beta_1, \dots, \beta_d \in [-C, C]$ ,
$$
\begin{array}{l} d ^ {\pi_ {x}} (x) = \mu_ {h + 1} ^ {\star} (x) ^ {\top} \left(\sum_ {i = 1} ^ {d} \beta_ {i} \phi_ {h} ^ {\star , \pi_ {i}}\right) + \mu_ {h + 1} ^ {\star} (x) ^ {\top} \left(\phi_ {h} ^ {\star , \pi_ {x}} - \sum_ {i = 1} ^ {d} \beta_ {i} \phi_ {h} ^ {\star , \pi_ {i}}\right), \\ \leq C d \cdot \max  _ {i \in [ d ]} \mu_ {h + 1} ^ {\star} (x) ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}} + \varepsilon \cdot \| \mu_ {h + 1} ^ {\star} (x) \|, \quad (\text {C a u c h y - S c h w a r z}) \\ \leq C d \cdot \max _ {i \in [ d ]} \mu_ {h + 1} ^ {\star} (x) ^ {\top} \phi_ {h} ^ {\star , \pi_ {i}} + \frac {1}{2} d ^ {\pi_ {x}} (x), \\ \end{array}
$$
where the inequality follows by the fact that Assumption 2.1 holds with  $\varepsilon \leq \eta /2$ . The result now follows by rearranging.
G Application to Reward-Based RL
In this section, we explain how the output  $\Psi^{(1:H)}$  of SpanRL (Algorithm 1) can be used to optimize downstream reward functions  $r_{1:H}$ ; our treatment is standard. Since the output of SpanRL is a policy cover, one way to optimize the sum of rewards  $S_H \coloneqq \sum_{h=1}^H r_h$  is by first generating trajectories using policies in  $\Psi^{(1:H)}$ , then applying an offline RL algorithm, e.g. Fitted Q-Iteration (FQI) [16], to optimize  $S_H$ . It is also possible to use PSDP with the policy cover  $\Psi^{(1:H)}$  to achieve the same goal. We will showcase the latter approach since we have already stated a guarantee for PSDP.
As in Appendix D, we assume access to a function class  $\mathcal{G} \subseteq \{g : \mathcal{X} \times \mathcal{A} \to \mathbb{R}\}$  that realizes the rewards  $r_{1:H}$  in the following sense: for all  $h \in [H]$  and all  $\pi \in \Pi_{\mathfrak{M}}^{h:H}$ ,
$$
Q _ {h} ^ {\pi} \in \mathcal {G}, \quad \text {w h e r e} \quad Q _ {h} ^ {\pi} (x, a) := r _ {h} (x, a) + \mathbb {E} ^ {\pi} \left[ \sum_ {t = h + 1} ^ {H} r _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {a} _ {t}) \Bigg | \boldsymbol {x} _ {h} = x, \boldsymbol {a} _ {h} = a \right].
$$
Note that when the reward functions  $r_{1:H}$  are linear in the feature map  $\phi_h^\star$ ; that is, when for all  $h \in [H]$  and  $(x, a) \in \mathcal{X}_h \times \mathcal{A}$ ,  $r_h(x, a) = \theta_h^\top \phi_h^\star(x, a)$  for some  $\theta_h \in \mathcal{B}(1)$  (a common assumption in the context of RL in Low-Rank MDPs [32, 31, 49, 35]), then the function class
$$
\mathcal {G} := \left\{g: (x, a) \mapsto \phi (x, a) ^ {\top} w \mid \phi \in \Phi , w \in \mathcal {B} (2 H \sqrt {d}) \right\},
$$
realizes  $r_{1:H}$ . Note that  $\mathcal{G}$  is the same function class we used for the PSDP subroutine in Algorithm 3, albeit with a larger ball for the  $w$ 's. For the sake of generality, we state the next result (which shows how to use a policy cover to optimize a downstream reward function) for general  $r_{1:H}$  and a function class  $\mathcal{G}$  that realizes  $r_{1:H}$ .
Theorem G.1. Let  $\varepsilon > 0$  and  $\delta \in (0,1)$ . Suppose reward functions  $r_{1:H} : \mathcal{X} \times \mathcal{A} \to \mathbb{R}$ , a collection of policies  $\Psi^{(1:H)}$ , and a parameter  $n \geq 1$  satisfy the following:
- The function class  $\mathcal{G}$  realizes the reward functions  $r_{1:H}$  (in the sense of Definition D.1), and  $\lim_{n\to \infty}n^{-1}\cdot \log \mathcal{N}_{\mathcal{G}}(1 / n) = 0$ , where  $\mathcal{N}_{\mathcal{G}}(1 / n)$  to denote the  $\frac{1}{n}$ -covering number of  $\mathcal{G}$  in the supremum norm. Furthermore, we suppose that functions in  $\mathcal{G}$  are uniformly bounded by  $H\sqrt{d}$ .
- For some  $0 < \alpha \leq 1$ , for each  $1 \leq h \leq H$ , it holds that  $\Psi^{(h)}$  is an  $\alpha$ -policy cover for layer  $h$  and moreover  $|\Psi^{(h)}| \leq d$ .
- The parameter  $n$  is chosen such that  $cdH\alpha^{-1} \cdot \varepsilon_{\mathrm{stat}}(n, \delta / H) \leq \varepsilon$ , where  $\varepsilon_{\mathrm{stat}}(n, \delta') \coloneqq \sqrt{dH^2n^{-1} \cdot (\log\mathcal{N}_{\mathcal{G}}(1/n) + \log(1/\delta))}$  and  $c > 0$  is a large enough absolute constant.
Then, with probability at least  $1 - \delta$ , the policy  $\hat{\pi} = \mathsf{PSDP}\big(H,r_{1:H},\mathcal{G},P^{(1:H)},n\big)$  (where  $P^{(t)} \coloneqq \mathrm{unif}(\Psi^{(t)})$ , for each  $t \in [h]$ ) coming from Algorithm 3, satisfies the following guarantee:
$$
\max _ {\pi \in \Pi_ {n}} \mathbb {E} ^ {\pi} \left[ \sum_ {h = 1} ^ {H} r _ {h} (\pmb {x} _ {h}, \pmb {a} _ {h}) \right] \leq \mathbb {E} ^ {\hat {\pi}} \left[ \sum_ {h = 1} ^ {H} r _ {h} (\pmb {x} _ {h}, \pmb {a} _ {h}) \right] + \varepsilon .
$$
Moreover, the number of episodes used by PSDP in this case is
$$
\widetilde {O} \left(\frac {A ^ {2} d ^ {5} H ^ {5} (\log \mathcal {N} _ {\mathcal {G}} (\varepsilon) + \log (1 / \delta))}{\varepsilon^ {2}}\right).
$$
Proof. This is simply a restatement of Theorem D.2 with  $h = H$ . The number of trajectories follows by the fact that each call to PSDP requires  $Hn$  trajectories.
H Properties of Reachability Assumption
In this section, we compare  $\eta$ -reachability (Assumption 2.1) to different reachability assumptions used in the literature in the context of RL in Low-Rank MDPs and show that ours is the weakest among those commonly assumed. In Appendix H.1, we demonstrate an exponential separation between our notion of reachability and that considered with respect to the popular latent variable model [1, 35]. In Appendix H.2, we consider a number of other reachability assumptions made outside the latent variable model and show how they imply Assumption 2.1.
H.1 Comparison to Latent Variable Model
In this subsection, we show that our reachability assumption is implied a reachability assumption used by [1, 35] in the latent variable/non-negative feature model, and show that our reachability assumption can hold even when the best possible latent variable embedding dimension is exponential in the dimension  $d$ . We begin by defining the latent variable model.
Definition H.1 (Latent variable representation). Given a transition operator  $T: \mathcal{X} \times \mathcal{A} \to \Delta(\mathcal{X})$ , a latent variable representation consists of a countable latent space  $\mathcal{Z}$  and functions  $\psi: \mathcal{X} \times \mathcal{A} \to \Delta(\mathcal{Z})$  and  $q: \mathcal{Z} \to \Delta(\mathcal{X})$ , such that  $T(\cdot | x, a) = \sum_{z \in \mathcal{Z}} q(\cdot | z) \psi(z | x, a)$ . The latent variable dimension of  $T$ , denoted  $d_{\mathrm{LV}}$  is the cardinality of smallest latent space  $\mathcal{Z}$  for which  $T$  admits a latent variable representation.
The interpretation for the latent variable model is as follows:
1. Each  $(x,a)$  pair induces a distribution  $\psi (x,a)\in \Delta (\mathcal{Z})$  over  $z\in \mathcal{Z}$
2. The latent variable is sampled as  $z \sim \psi(x, a)$ .
3. The next state is sampled as  $x' \sim q(\cdot | z)$ .
Note that in discrete state spaces, all transition operators admit a trivial latent variable representation, as we may take  $\psi(x,a) = T(\cdot | x,a)$ , but the dimension of such a representation is potentially infinite. A latent variable representation certifies that there exists a factorization  $T(x' | x,a) = \psi(x,a)^{\top} q(x')$  with embedding dimension  $|\mathcal{Z}|$ , and so  $d_{\mathrm{LV}}$ , and hence gives an upper bound on the rank of the transition operator. On the other hand, compared with the general Low-Rank factorization, the latent variable factorization additionally requires that  $\psi(x,a)$  and  $q(\cdot | z)$  are probability distributions, and thus non-negative, for all  $z \in \mathcal{Z}$  and  $(x,a) \in \mathcal{X} \times \mathcal{A}$ , implying that  $d_{\mathrm{LV}}$  is equivalent to the non-negative rank [1] of the transition operator.
Assuming that a latent variable representation exists, [1, 35] consider the following notion of reachability.
Definition H.2 (Reachability in latent variable model). There exists  $\eta > 0$  such that
$$
\forall h \in [ H - 1 ], \forall z \in \mathcal {Z} _ {h + 1}, \sup  _ {\pi \in \Pi_ {\mathfrak {M}}} \mathbb {P} ^ {\pi} \left[ \boldsymbol {z} _ {h + 1} = z \right] \geq \eta . \tag {47}
$$
We first show the latent variable reachability condition above implies our more general assumption.
Lemma H.1. Consider a Low-Rank MDP  $\mathcal{M}$  with rank  $d\geq 1$ . Under the latent variable model in Definition H.1, if the latent variable reachability condition in (47) is satisfied for some  $\eta >0$ , then, for all  $h\in [H]$ , the transition kernel  $T_{h}$  in  $\mathcal{M}$  admits a factorization  $T_{h}(\cdot |x,a) = \mu_{h + 1}^{\star}(\cdot)^{\top}\phi_{h}^{\star}(x,a)$  where  $\mu_{h + 1}^{\star}(\cdot)\in \mathbb{R}^{d_{\mathrm{LV}}}$  and  $\phi_h^\star (\cdot ,\cdot)\in \mathbb{R}^{d_{\mathrm{LV}}}$ , such that  $d_{\mathrm{LV}}\leq dA^2 /\eta^2$  and  $\frac{\eta^2}{A\sqrt{d}}$  -reachability (in the sense of Assumption 2.1) is satisfied.
Proof of Lemma H.1. Suppose that Assumption 2.1 ( $\eta$ -reachability) holds. By Agarwal et al. [1, Proposition 4], the non-negative rank of  $\mathcal{M}$  is bounded as  $d_{\mathrm{LV}} \leq dA^2 / \eta^2$ .
Letting  $q$  and  $\psi$  be as in the definition of the latent variable representation in Definition H.1, we define  $\mu_{h+1}^{\star}$  and  $\phi_h^{\star}$  as: for all  $h \in [H-1]$ ,
$$
\mu_ {h + 1} ^ {\star} (\cdot) := \left(q (\cdot \mid z)\right) _ {z \in \mathcal {Z}} \in \mathbb {R} ^ {d _ {\mathrm {L V}}}, \quad \text {a n d} \quad \phi_ {h} ^ {\star} (\cdot , \cdot) := \left(\psi (z \mid \cdot , \cdot)\right) _ {z \in \mathcal {Z}} \in \mathbb {R} ^ {d _ {\mathrm {L V}}}.
$$
Now, fix  $h \in [H - 1]$  and  $x \in \mathcal{X}_{h + 1}$ . For  $z_0 \in \arg \max_{z \in \mathcal{Z}_{h + 1}} q(x \mid z)$ , we have
$$
\begin{array}{l} \sup  _ {\pi \in \Pi_ {\mathfrak {M}}} d ^ {\pi} (x) = \mathbb {P} ^ {\pi} [ \boldsymbol {x} _ {h + 1} = x ] = \sup  _ {\pi \in \Pi_ {\mathfrak {M}}} \sum_ {z \in \mathcal {Z} _ {h + 1}} q (x \mid z) \cdot \mathbb {E} ^ {\pi} [ \psi (z \mid \boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) ], \\ = \sup  _ {\pi \in \Pi_ {M}} q (x \mid z _ {0}) \cdot \mathbb {E} ^ {\pi} [ \psi (z _ {0} \mid x _ {h}, \boldsymbol {a} _ {h}) ], \\ = \left\| \mu_ {h + 1} ^ {\star} (x) \right\| _ {\infty} \cdot \sup _ {\pi \in \Pi_ {\mathbb {M}}} \mathbb {P} ^ {\pi} \left[ z _ {h + 1} = z _ {0} \right], \\ \geq \eta \cdot \| \mu_ {h + 1} ^ {\star} (x) \| _ {\infty}, \quad (\text {u s i n g}) \\ \geq \frac {\eta}{\sqrt {d _ {\mathrm {L V}}}} \cdot \| \mu_ {h + 1} ^ {*} (x) \|. \\ \end{array}
$$
We now complement the result above by showing that there exists low-rank MDPs for which our notion of reachability (Assumption 2.1) is satisfied with  $\eta$  polynomially small, yet the best possible latent variable embedding has dimension  $d_{\mathsf{LV}} = 2^{\Omega(d)}$ . This contrasts the results in [1, Proposition 2], which show that latent variable reachability implies a polynomial bound on the latent variable dimension.
Theorem H.3. There exists a one-step Low-Rank-MDP of rank  $d \geq 1$ , where  $\eta$ -reachability (Assumption 2.1) is satisfied with  $\eta = \frac{1}{2\sqrt{d}}$ , but where the non-negative rank satisfies  $d_{\mathrm{LV}} = 2^{\Omega(d)}$ .
Proof of Theorem H.3. Let  $n \in \mathbb{N}$  and  $d \coloneqq \binom{n}{2} + 1$ . As shown in the proof of Agarwal et al. [1, Proposition 2], there exists a horizon-two MDP  $\mathcal{M}$  with the following properties:
- The state spaces  $\mathcal{X}_1$  and  $\mathcal{X}_2$  at layers 1 and 2, respectively, are finite.
- The cardinality of  $\mathcal{A}$  is  $d$ ; i.e.  $\mathcal{A} = \{a_1, \ldots, a_d\}$ .<sup>11</sup>
- The transition kernel  $T_{1}$  admits the factorization:
$$
T _ {1} (\cdot \mid x, a) = \mu_ {2} ^ {\star} (\cdot) ^ {\top} \phi_ {1} ^ {\star} (x, a) \in \Delta (\mathcal {X} _ {2}), \quad \forall (x, a) \in \mathcal {X} _ {1} \times \mathcal {A},
$$
where for all  $x' \in \mathcal{X}_2$ ,  $\mu_2^\star(x') \in \mathbb{R}_{\geq 0}^d$ , and for all  $(x, a) \in \mathcal{X}_1 \times \mathcal{A}$ ,  $\phi_1^\star(x, a) \in \mathbb{R}_{\geq 0}^d$ .
- The non-negative rank of  $\mathcal{M}$  is  $d_{\mathrm{LV}} = 2^{\Omega(d)}$ .
We augment this MDP by adding an extra state  $x_0$ , and let  $\overline{\mathcal{X}}_1 \coloneqq \mathcal{X}_1 \cup \{x_0\}$ . We define  $\bar{\phi}_1^\star : \overline{\mathcal{X}}_1 \times \mathcal{A} \to \mathbb{R}_{\geq 0}^d$  be the extension of  $\phi_1^\star$  given by
$$
\forall i \in [ d ], \quad \bar {\phi} _ {1} ^ {\star} (x _ {0}, a _ {i}) = e _ {i}, \quad \text {a n d} \quad \forall x \in \mathcal {X} _ {1}, \quad \bar {\phi} _ {1} ^ {\star} (x, a _ {i}) = \phi_ {1} ^ {\star} (x, a _ {i}),
$$
Technically, the example in the proof of [1, Proposition 2] does not explicitly specify the number of actions. Instead, the example assigns a number of state-action pairs to vectors in  $\mathbb{R}^d$ , without specifying the number of actions. The number of actions in their example is a degree of freedom, which we set to  $d$  here without loss of generality.
where  $e_i$  is the  $i$ th basis element in  $\mathbb{R}^d$ . We define the initial state distribution to have  $\rho(x_0) = \frac{1}{2}$  and  $\rho(x) = \frac{1}{2|\mathcal{X}_1|}$ , for all  $x \in \mathcal{X}_1$ .<sup>12</sup> We let  $\overline{\mathcal{M}} = (\mathcal{X}_1 \cup \mathcal{X}_2, \mathcal{A}, \overline{\phi}_1^\star, (\mu_h^\star)_{h \in [2]}, \rho)$  denote the resulting MDP. Note that adding an extra state at layer 1 in this fashion only adds  $d$  additional rows to the transition matrix  $T$  (viewed as a  $(|\mathcal{X}_1 \times \mathcal{A}|) \times |\mathcal{X}_2|$  matrix). Therefore, the non-negative rank of  $\overline{\mathcal{M}}$  is as least that of  $\mathcal{M}$ .
We now show that reachability is satisfied in  $\overline{\mathcal{M}}$ . Let  $\pi_i$  the policy that always plays action  $a_i$ . With this, we have that for any  $x' \in \mathcal{X}_2$ ,
$$
\begin{array}{l} \sup  _ {\pi \in \Pi_ {\mathsf {M}}} d ^ {\pi} \left(x ^ {\prime}\right) \geq \max  _ {i \in [ d ]} d ^ {\pi_ {i}} \left(x ^ {\prime}\right), \\ = \max  _ {i \in [ d ]} \mu_ {2} ^ {\star} \left(x ^ {\prime}\right) ^ {\top} \mathbb {E} \left[ \bar {\phi} _ {1} ^ {\star} \left(\boldsymbol {x} _ {1}, a _ {i}\right) \right], \\ = \max  _ {i \in [ d ]} \left\{\mathbb {E} \left[ \mathbb {I} \left\{\boldsymbol {x} _ {1} = x _ {0} \right\} \cdot \mu_ {2} ^ {\star} \left(x ^ {\prime}\right) ^ {\top} \bar {\phi} _ {1} ^ {\star} \left(\boldsymbol {x} _ {1}, a _ {i}\right) \right] + \mathbb {E} \left[ \mathbb {I} \left\{\boldsymbol {x} _ {1} \neq x _ {0} \right\} \cdot \mu_ {2} ^ {\star} \left(x ^ {\prime}\right) ^ {\top} \bar {\phi} _ {1} ^ {\star} \left(\boldsymbol {x} _ {1}, a _ {i}\right) \right] \right\}, \\ \geq \max  _ {i \in [ d ]} \rho \left(x _ {0}\right) \mu_ {2} ^ {\star} \left(x ^ {\prime}\right) ^ {\top} \bar {\phi} _ {1} ^ {\star} \left(x _ {0}, a _ {i}\right). \tag {48} \\ \end{array}
$$
where the last inequality follows by the fact that, for all  $(x,a)\in \mathcal{X}_1\times \mathcal{A}$ ,  $\mu_2^{\star}(\cdot)^{\top}\bar{\phi}_{1}^{\star}(x,a) = \mu_{2}^{\star}(x^{\prime})^{\top}\phi_{1}^{\star}(x,a)\geq 0$  (since  $\mu_2^\star (x')^\top \phi_1^\star (x,a)$  is a conditional density). On the other hand, from the construction of  $\bar{\phi}_1^*$  and the fact that  $\mu_2^{\star}(x^{\prime})\in \mathbb{R}_{\geq 0}^{d}$ , we have
$$
\max _ {i \in [ d ]} \mu_ {2} ^ {\star} (x ^ {\prime}) ^ {\top} \bar {\phi} _ {1} ^ {\star} (x _ {0}, a _ {i}) = \| \mu_ {2} ^ {\star} (x ^ {\prime}) \| _ {\infty} \geq \| \mu_ {2} ^ {\star} (x ^ {\prime}) \| / \sqrt {d}.
$$
Combining this with (48) and using that  $\rho(x_0) = 1/2$  implies that reachability  $1/(2\sqrt{d})$  is satisfied in  $\overline{\mathcal{M}}$ .
H.2 Relation to Other Reachability Assumptions
In this subsection, we show that Assumption 2.1 is implied by a notion of feature coverage used in the context of transfer learning in Low-Rank MDPs [5], as well as a notion of explorability used in the context of reward-free RL in linear MDPs [46].
H.2.1 Feature Coverage
We first consider coverage condition used by Agarwal et al. [5], which involves the second moments of the feature map  $\phi_h^*$ .