SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition H.4 ( $\eta$ -feature coverage). We say that the linear MDP with featurization  $\phi_h^\star$  satisfies  $\eta$ -feature coverage if for all  $h \in [H]$ ,
$$
\sup  _ {\pi \in \Pi_ {\mathbb {M}}} \lambda_ {\min } \left(\mathbb {E} ^ {\pi} \left[ \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) ^ {\top} \right]\right) \geq \eta .
$$
We show that  $\eta$ -feature coverage implies  $(\eta / 2)^{3/2}$ -reachability. Thus, up to polynomial dependence,  $\eta$ -feature coverage is a special case of Assumption 2.1.
Lemma H.2. Suppose that an MDP satisfies  $\eta$ -feature coverage as in Definition H.4 for some  $\eta > 0$ . If  $\phi_h^\star(x, a) \in \mathcal{B}(1)$  for all  $x, a$ , then the MDP satisfies  $(\eta/2)^{3/2}$ -reachability in the sense of Assumption 2.1.
Proof of Lemma H.2. Let  $h \in [H]$  and  $x \in \mathcal{X}_{h+1}$  be given and define
$$
\theta := \frac {\mu_ {h + 1} ^ {\star} (x)}{\| \mu_ {h + 1} ^ {\star} (x) \|}.
$$
To keep notation compact, we define  $\phi_h^\star \coloneqq \phi_h^\star (\pmb{x}_h,\pmb{a}_h)$ . By  $\eta$ -feature coverage, there exists  $\pi \in \Pi_{\mathbb{M}}$  such that
$$
\begin{array}{l} \eta \leq \mathbb {E} ^ {\pi} \left[ \left(\theta^ {\top} \phi_ {h} ^ {\star}\right) ^ {2} \right] = \mathbb {E} ^ {\pi} \left[ \mathbb {I} \left\{\left(\theta^ {\top} \phi_ {h} ^ {\star}\right) ^ {2} <   \eta / 2 \right\} \cdot \left(\theta^ {\top} \phi_ {h} ^ {\star}\right) ^ {2} \right] + \mathbb {E} ^ {\pi} \left[ \mathbb {I} \left\{\left(\theta^ {\top} \phi_ {h} ^ {\star}\right) ^ {2} \geq \eta / 2 \right\} \cdot \left(\theta^ {\top} \phi_ {h} ^ {\star}\right) ^ {2} \right], \\ \leq \eta / 2 + \mathbb {P} ^ {\pi} \left[ \left(\theta^ {\top} \phi_ {h} ^ {\star}\right) ^ {2} \geq \eta / 2 \right], \tag {49} \\ \end{array}
$$
<sup>12</sup>We note that [1] did not specify the initial distribution, which is not needed for the conclusion of their result.
where we have used that  $\| \theta \| = 1$  and  $\| \phi_h^\star (x,a)\| \leq 1$  for all  $(x,a)\in \mathcal{X}_h\times \mathcal{A}$ . Rearranging (49) and using that  $\theta^{\top}\phi_h^{\star}\geq 0$  (it is a scaled conditional density), have
$$
\mathbb {P} ^ {\pi} \left[ \theta^ {\top} \phi_ {h} ^ {\star} \geq \sqrt {\eta / 2} \right] = \mathbb {P} ^ {\pi} \left[ \left(\theta^ {\top} \phi_ {h} ^ {\star}\right) ^ {2} \geq \eta / 2 \right] \geq \eta / 2.
$$
Now, by Markov's inequality, we have that
$$
\theta^ {\top} \phi_ {h} ^ {\star , \pi} = \mathbb {E} ^ {\pi} \left[ \theta^ {\top} \phi_ {h} ^ {\star} \right] \geq \sqrt {\eta / 2} \cdot \mathbb {P} ^ {\pi} \left[ \theta^ {\top} \phi_ {h} ^ {\star} \geq \sqrt {\eta / 2} \right] \geq (\eta / 2) ^ {3 / 2},
$$
where we have once more used that  $\theta^{\top}\phi_h^\star \geq 0$  almost surely.
H.2.2 Explorability
We now consider the explorability assumption of [46], which involves the first moment of the feature map  $\phi_h^\star$ . This notion is defined as follows.
Definition H.5 ( $\eta$ -explorability). We say that a linear MDP satisfies  $\eta$ -explorability if for any  $h \in [H]$  and any  $\theta \in \mathbb{R}^d \setminus \{0\}$  it holds that
$$
\sup  _ {\pi \in \Pi_ {\mathfrak {M}}} \left| \theta^ {\top} \mathbb {E} ^ {\pi} \left[ \phi_ {h} ^ {\star} \left(\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}\right) \right] \right| \geq \eta \cdot \| \theta \|.
$$
We now show that  $\eta$ -explorability is a special case of  $\eta$ -reachability:
Lemma H.3. Suppose that the explorability condition in Definition H.5 is satisfied with  $\eta > 0$ . Then,  $\eta$ -reachability is satisfied.
Proof of Lemma H.3. Let  $x \in \mathcal{X}_{h+1}$  and define  $\theta \coloneqq \mu_{h+1}^*(x)$ . By explorability, we have that
$$
\begin{array}{l} \sup  _ {\pi \in \Pi_ {\mathfrak {M}}} d ^ {\pi} (x) = \sup  _ {\pi \in \Pi_ {\mathfrak {M}}} \mathbb {E} ^ {\pi} \left[ \mu_ {h + 1} ^ {\star} (x) ^ {\top} \phi_ {h} ^ {\star} (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) \right], \\ = \sup  _ {\pi \in \Pi_ {M}} | \mathbb {E} ^ {\pi} \left[ \mu_ {h + 1} ^ {\star} (x) ^ {\top} \phi_ {h} ^ {\star} (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) \right], \quad \left(\mu_ {h + 1} ^ {\star} (\cdot) ^ {\top} \phi_ {h} ^ {\star} (x, a) \text {i s a c o n d i t i o n l a w}\right) \\ = \sup  _ {\pi \in \Pi_ {\mathfrak {M}}} | \theta^ {\top} \mathbb {E} ^ {\pi} [ \phi_ {h} ^ {\star} (\boldsymbol {x} _ {h}, \boldsymbol {a} _ {h}) ] |, \\ \geq \eta \cdot \| \theta \|, \quad (\text {b y}) \\ = \eta \cdot \| \mu_ {h + 1} ^ {\star} (x) \|. \\ \end{array}
$$
This shows that Assumption 2.1 is satisfied with parameter  $\eta$ .