SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Using the definition of independent sequences established so far, we define the  $\ell_p$  distributional eluder dimension.
Definition G.3 ( $\ell_p$ -distributional eluder dimension). For any  $p$ , define the  $\ell_p$ -distributional eluder dimension (denoted by  $\mathrm{DE}_p(\Psi, \mathcal{D}, \varepsilon)$ ) as the length of the longest sequence  $\{d^{(1)}, \ldots, d^{(d)}\} \subset \mathcal{D}$  such that there exists  $\varepsilon' \geq \varepsilon$ , such that for all  $t \in [d]$ ,  $d^{(t)}$  is  $(\varepsilon', \ell_p)$ -independent of  $d^{(1)}, \ldots, d^{(t-1)}$ .
Of particular interest to us is the  $\ell_1$  case. We show that the  $\ell_1$  eluder dimension is dominated by the  $\ell_2$  eluder dimension of Jin et al. [2021a].
Proof. Since  $\sqrt{\sum_{i}x_{i}^{2}}\leq \sum_{i}|x_{i}|$ , we have that any witness (long independent sequence) for  $\ell_1$  is also a witness for  $\ell_2$ . So, the maximum length of the  $\ell_2$  witnesses is longer than the  $\ell_1$  witnesses. Liu et al. [2022, Proposition 19] obtains an analogous result for the non-distributional eluder dimension of Russo and Van Roy [2013].
We now prove the key pigeonhole result for the  $\ell_1$  distributional eluder dimension.
Proof. For any  $\Gamma \subset \mathcal{D}$ ,  $\nu \in \mathcal{D}$ , and  $0 < \varepsilon \leq 1$ , let  $L(\nu, \Gamma, \varepsilon)$  denote the number of disjoint subsets of  $\Gamma$  such that each subset is  $\varepsilon$ -dependent of  $\nu$ , i.e., for all such disjoint subsets of  $\Gamma$ , it is not the case that  $\nu$  is  $(\varepsilon, \ell_1)$ -independent of each subset.
Fact 1: For any  $\varepsilon$ , if  $\left|\mathbb{E}_{d^{(k)}}f^{(k)}\right| > \varepsilon$  for some  $k\in [K]$ , then  $L(d^{(k)},d^{(1:k - 1)},\varepsilon) < \beta /\varepsilon$ .
By definition of  $L \coloneqq L(d^{(k)}, d^{(1:k-1)}, \varepsilon)$ , there exist disjoint subsequences  $\mathfrak{G}^{(1)}, \ldots, \mathfrak{G}^{(L)}$  of  $d^{(1:k-1)}$  such that each subsequence  $\mathfrak{G}^{(i)}$  satisfies  $\sum_{d \in \mathfrak{G}^{(i)}} \left| \mathbb{E}_d f^{(k)} \right| > \varepsilon$ . Therefore, summing over all subsequences, we have  $L\varepsilon < \sum_{i=1}^{k-1} \left| \mathbb{E}_{d^{(i)}} f^{(k)} \right| \leq \beta$ , where the  $\beta$  inequality comes from the premise. This proves Fact 1.
Fact 2: For any  $\varepsilon$  and any sequence  $\{\nu^{(1)},\ldots ,\nu^{(\kappa)}\} \subset \mathcal{D}$ , there exists  $j\in [\kappa ]$  such that  $L(\nu^{(j)},\nu^{(1:j - 1)},\varepsilon)\geq J\coloneqq \left\lfloor (\kappa -1) / \mathrm{DE}_1(\Psi ,\mathcal{D},\varepsilon)\right\rfloor .$
If  $J = 0$ , the claim is vacuously true. Otherwise, consider the following algorithm for finding the  $j$ :
Step 1) Initialize  $\mathfrak{G}^{(1)} = [\nu^{(1)}],\ldots ,\mathfrak{G}^{(J)} = [\nu^{(J)}]$  and let  $j = J + 1$
Step 2) If  $\nu^{(j)}$  is  $\varepsilon$ -dependent on all of  $\mathfrak{G}^{(i)}, i \in [J]$ , then the claim is proven and terminate.
Step 3) Otherwise, there exists some  $\mathfrak{G}^{(i)}, i \in [J]$  such that  $\nu^{(j)}$  is  $\varepsilon$ -independent of it. Append  $\nu^{(j)}$  to  $\mathfrak{G}^{(i)}, i.e., \mathfrak{G}^{(i)} = \mathfrak{G}^{(i)} + [\nu^{(j)}]$ . Increment  $j = j + 1$  and go back to Step 2.
Hence, we need to argue this process terminates at Step 2 before  $j$  gets to  $\kappa + 1$ . We prove this by contradiction: assume  $j$  gets to  $\kappa + 1$ . Let  $i \in [J]$  be such that  $\mathfrak{G}^{(i)}$  has the most elements (break ties arbitrarily). Since  $\kappa = \sum_{i=1}^{J} \left| \mathfrak{G}^{(i)} \right| \leq J \left| \mathfrak{G}^{(i)} \right|$ , we have that  $\left| \mathfrak{G}^{(i)} \right| \geq \kappa / J \geq \frac{\kappa}{\kappa - 1} \mathrm{DE}_1(\Psi, \mathcal{D}, \varepsilon) > \mathrm{DE}_1(\Psi, \mathcal{D}, \varepsilon)$ , where we've also used the definition of  $J$ . By construction,  $\mathfrak{G}^{(i)}$  is an  $\varepsilon$ -eluder sequence, i.e., it is a sequence such that each element is  $\varepsilon$ -independent of its predecessors. However, this is a contradiction because its size is greater than  $\mathrm{DE}_1(\Psi, \mathcal{D}, \varepsilon)$ . Therefore, this process terminates at Step 2 for some  $j$ , which is the witness for proving Fact 2.
Fact 3: For any  $\varepsilon$  and  $k\in [K]$ , we have  $\sum_{t = 1}^{k}\mathbb{I}\big[\big|\mathbb{E}_{d^{(t)}}f^{(t)}\big| > \varepsilon \big]\leq \big(\beta \varepsilon^{-1} + 1\big)\mathrm{DE}_1(\Psi ,\mathcal{D},\varepsilon) + 1$ . Fix any  $\varepsilon$  and  $k\in [K]$ . Let  $\{d^{(i_1)},\ldots ,d^{(i_\kappa)}\}$  be all the elements of  $d^{(1:k)}$  such that  $\mathbb{E}_{d^{(t)}}f^{(t)} > \varepsilon$  for  $t = i_1,\dots ,i_\kappa$ . By Fact 2, there exists  $j\in [\kappa ]$  such that  $L(d^{(i_j)},d^{(i_{1:j - 1})},\varepsilon)\geq \lfloor (\kappa -1) / \mathrm{DE}_1(\Psi ,\mathcal{D},\varepsilon)\rfloor$ . By Fact 1, we have  $L(d^{(i_j)},d^{(1:i_j)},\varepsilon)\leq \beta /\varepsilon$ . Finally notice that  $L(d^{(i_j)},d^{(i_{1:j - 1})},\varepsilon)\leq L(d^{(i_j)},d^{(1:i_j)},\varepsilon)$  since adding more elements can only create more  $\varepsilon$ -dependent-of- $\nu$  disjoint subsets. Thus, combining these inequalities, we have  $\lfloor (\kappa - 1) / \mathrm{DE}_1(\Psi ,\mathcal{D},\varepsilon)\rfloor < \beta /\varepsilon$ . This implies  $\kappa \leq (\beta \varepsilon^{-1} + 1)\mathrm{DE}_1(\Psi ,\mathcal{D},\varepsilon) + 1$ , which proves Fact 3.
Finishing the proof
Fix any  $k \in [K]$  and  $\omega > 0$ . We have
$$
\begin{array}{l} \sum_ {t = 1} ^ {k} \left| \mathbb {E} _ {d ^ {(t)}} f ^ {(t)} \right| = \sum_ {t = 1} ^ {k} \int_ {0} ^ {C} \mathbb {I} \left[ \left| \mathbb {E} _ {d ^ {(t)}} f ^ {(t)} \right| > y \right] d y \\ \leq k \omega + \sum_ {t = 1} ^ {k} \int_ {\omega} ^ {C} \mathbb {I} \left[ \left| \mathbb {E} _ {d ^ {(t)}} f ^ {(t)} \right| > y \right] d y \\ = k \omega + \int_ {\omega} ^ {C} \sum_ {t = 1} ^ {k} \mathbb {I} \left[ \left| \mathbb {E} _ {d ^ {(t)}} f ^ {(t)} \right| > y \right] d y \\ \leq k \omega + \int_ {\omega} ^ {C} \left\{\left(\beta / y + 1\right) \mathrm {D E} _ {1} (\Psi , \mathcal {D}, y) + 1 \right\} \mathrm {d} y \tag {Fact3} \\ \leq k \omega + \int_ {\omega} ^ {C} \left\{\left(\beta / y + 1\right) \mathrm {D E} _ {1} (\Psi , \mathcal {D}, \omega) + 1 \right\} \mathrm {d} y \quad \text {(M o n o t o n i c i t y o f D E} _ {1}) \\ \leq k \omega + (d + 1) C + d \beta \log (C / \omega). \quad (d := \mathrm {D E} _ {1} (\Psi , \mathcal {D}, \omega)) \\ \end{array}
$$
G.1 Bounding V-type  $\ell_2$  eluder dimension in low-rank MDPs
Theorem G.4 (Bound of  $\ell_2$  distributional eluder for low-rank MDPs). Suppose the MDP is a low-rank MDP. Let  $\Psi \subset \mathcal{X} \to [0,1]$  be any class of functions mapping  $\mathcal{X}$  to  $[0,1]$ . Suppose
$\mathcal{D} = \{x\mapsto d_h^\pi (x):\pi \in \Pi \}$  for some  $h\in [H]$ . Then, we have
$$
\mathrm {D E} _ {2} (\Psi , \mathcal {D}, \varepsilon) \leq \mathcal {O} (d \log (d / \varepsilon)). \tag {7}
$$
Proof. If  $h = 1$ , then  $\mathcal{D}$  is a singleton. Hence,  $\mathrm{DE}_2(\Psi, \mathcal{D}, \varepsilon) \leq 1$ . Hence, suppose  $h \geq 2$ ; set  $h := h - 1$  and we will focus on  $d_{h + 1}^{\pi}$  in the remainder. Suppose  $\left\{d^{(k)}, f^{(k)}\right\}_{k \in [T]}$  is any sequence such for all  $k \in [T]$ , we have that  $(d^{(k)}, f^{(k)})$  is  $(\varepsilon, \ell_2)$ -independent of its predecessors. For any  $k$ , set  $\Sigma_k = \sum_{i=1}^{k-1} \mathbb{E}_{d^{(i)}}[\phi_h^\star(x_h, a_h)] \mathbb{E}_{d^{(i)}}[\phi_h^\star(x_h, a_h)]^\top + \lambda I$ . Then, we have
$$
\begin{array}{l} \mathbb {E} _ {d ^ {(k)}} f ^ {(k)} (x _ {h + 1}) = \mathbb {E} _ {d ^ {(k)}} \int_ {x _ {h}} \phi_ {h} ^ {\star} (x _ {h}, a _ {h}) ^ {\top} \mathrm {d} \mu_ {h} ^ {\star} (x _ {h + 1}) f ^ {(k)} (x _ {h + 1}) \\ = \mathbb {E} _ {d ^ {(k)}} \phi_ {h} ^ {\star} (x _ {h}, a _ {h}) ^ {\top} \int_ {x _ {h + 1}} f ^ {(k)} (x _ {h + 1}) \mathrm {d} \mu_ {h} ^ {\star} (x _ {h + 1}). \\ \leq \left\| \mathbb {E} _ {d ^ {(k)}} \phi_ {h} ^ {\star} \left(x _ {h}, a _ {h}\right) \right\| _ {\Sigma_ {k} ^ {- 1}} \| \int_ {x _ {h + 1}} f ^ {(k)} \left(x _ {h + 1}\right) \mathrm {d} \mu_ {h} ^ {\star} \left(x _ {h + 1}\right) \| _ {\Sigma_ {k}}. \\ \end{array}
$$
Focusing on the second term,
$$
\left\| \int_ {x _ {h + 1}} f ^ {(k)} \left(x _ {h + 1}\right) \mathrm {d} \mu_ {h} ^ {\star} \left(x _ {h + 1}\right) \right\| _ {\Sigma_ {k}} ^ {2} = \sum_ {i = 1} ^ {k - 1} \left(\mathbb {E} _ {d ^ {(i)}} \left[ f ^ {(k)} \left(x _ {h + 1}\right) \right]\right) ^ {2} + \lambda d
$$
$$
\mathbb {E} _ {d ^ {(k)}} f ^ {(k)} (x _ {h + 1}) \leq \| \mathbb {E} _ {d ^ {(k)}} \phi_ {h} ^ {\star} (x _ {h}, a _ {h}) \| _ {\Sigma_ {k} ^ {- 1}} \sqrt {\sum_ {i = 1} ^ {k - 1} \left(\mathbb {E} _ {d ^ {(i)}} \left[ f ^ {(k)} (x _ {h + 1}) \right]\right) ^ {2} + \lambda d}.
$$
Then, by the independent sequence assumption, we have
$$
\begin{array}{l} T \varepsilon <   \sum_ {k = 1} ^ {T} \mathbb {E} _ {d ^ {(k)}} f ^ {(k)} (x _ {h + 1}) \leq \sum_ {k = 1} ^ {T} \| \mathbb {E} _ {d ^ {(k)}} \phi_ {h} ^ {\star} (x _ {h}, a _ {h}) \| _ {\Sigma_ {k} ^ {- 1}} \sqrt {\sum_ {i = 1} ^ {k - 1} \left(\mathbb {E} _ {d ^ {(i)}} [ f ^ {(k)} (x _ {h + 1}) ]\right) ^ {2} + \lambda d} \\ \leq \sum_ {k = 1} ^ {T} \| \mathbb {E} _ {d ^ {(k)}} \phi_ {h} ^ {\star} (x _ {h}, a _ {h}) \| _ {\Sigma_ {k} ^ {- 1}} (\varepsilon + \sqrt {\lambda d}) \quad \left(\sqrt {\sum_ {i = 1} ^ {k - 1} \left(\mathbb {E} _ {d ^ {(i)}} [ f ^ {(k)} (x _ {h + 1}) ]\right) ^ {2}} \leq \varepsilon\right) \\ \leq 2 \varepsilon \sum_ {k = 1} ^ {T} \left\| \mathbb {E} _ {d ^ {(k)}} \phi_ {h} ^ {\star} \left(x _ {h}, a _ {h}\right) \right\| _ {\Sigma_ {k} ^ {- 1}} \quad (\lambda = \varepsilon^ {2} / d) \\ \leq 2 \varepsilon \sqrt {T} \sqrt {\sum_ {k = 1} ^ {T} \| \mathbb {E} _ {d ^ {(k)}} \phi_ {h} ^ {\star} (x _ {h} , a _ {h}) \| _ {\Sigma_ {k} ^ {- 1}} ^ {2}} \\ \leq 2 \varepsilon \sqrt {T} \sqrt {d \log (1 + T / d \lambda)} \quad (\text {e l l i p t i c a l p o t e n t i a l}) \\ \leq 2 \varepsilon \sqrt {T} \sqrt {d \log \left(1 + ^ {T} / _ {\varepsilon^ {2}}\right)}. \quad (\lambda = \varepsilon^ {2} / d) \\ \end{array}
$$
For a reference of the elliptical potential, see Uehara et al. [2021, Lemmas 19&20]. Rearranging, we have  $\sqrt{T} < 2\sqrt{d\log(1 + T / \varepsilon^2)}$ , which implies
$$
T \leq 4 d \log \left(1 + T / \varepsilon^ {2}\right).
$$
By applying Lemma G.5, we have  $T \leq 24d\log(1 + {}^{4d} / \varepsilon^2)$ . This concludes the proof.
Lemma G.5. Let  $c_{1}, c_{2} \geq 1$  be constants. Let  $x \geq 0$  be a solution to  $x \leq c_{1} \log (1 + c_{2}x)$ . Then, we necessarily have  $x \leq 6c_{1} \log (1 + c_{1}c_{2})$ .
Proof. Using change of variables  $B = \frac{x}{c_1}$ , we have the inequality is equivalent to  $B \leq \log(1 + B \cdot c_1c_2)$ . Take  $\exp$  of both sides to get  $\exp(B) \leq \alpha B + 1$  where  $\alpha = c_1c_2$ . From Step 3 of the proof of Russo and Van Roy [2013, Proposition 6], we have  $B \leq \frac{e}{e-1}\frac{e}{e-1}(\log(1 + \alpha) + \log(e/(e-1))) \leq 3(\log(1 + c_1c_2) + 1)$ . Hence,  $x \leq c_1 \cdot 3(\log(1 + c_1c_2) + 1)$ .
G.2 Bounding Q-type  $\ell_2$  eluder dimension in tabular MDPs
Theorem G.6 (Bound of  $\ell_2$  distributional eluder for tabular MDPs). Suppose the MDP is a tabular MDP. Let  $\Psi \subset \mathcal{X} \times \mathcal{A} \to [0,1]$  be any class of functions mapping  $\mathcal{X} \times \mathcal{A}$  to  $[0,1]$ . Suppose  $\mathcal{D}$  be any set of distributions. Then, we have
$$
\mathrm {D E} _ {2} (\Psi , \mathcal {D}, \varepsilon) \leq \mathcal {O} (S A \log (S A / \varepsilon)). \tag {8}
$$
Proof. Suppose  $\{d^{(k)},f^{(k)}\}_{k\in [T]}$  is any sequence such for all  $k\in [T]$ , we have that  $(d^{(k)},f^{(k)})$  is  $(\varepsilon ,\ell_{2})$ -independent of its predecessors. Since the MDP is tabular, we can interpret  $d^{(k)},f^{(k)}$  as SA-dimensional vectors. For any  $k$ , set  $\Sigma_k = \sum_{i = 1}^{k - 1}d^{(i)}(d^{(i)})^\top +\lambda I$ . Then, we have
$$
\mathbb {E} _ {d ^ {(k)}} f ^ {(k)} (x, a) = \left(d ^ {(k)}\right) ^ {\top} f ^ {(k)} \leq \left\| d ^ {(k)} \right\| _ {\Sigma_ {k} ^ {- 1}} \left\| f ^ {(k)} \right\| _ {\Sigma_ {k}}.
$$
Focusing on the second term, we have
$$
\| f ^ {(k)} \| _ {\Sigma_ {k}} ^ {2} = \sum_ {i = 1} ^ {k - 1} \left(\left(d ^ {(i)}\right) ^ {\top} f ^ {(k)}\right) ^ {2} + \lambda S A.
$$
$$
\begin{array}{l} T \varepsilon <   \sum_ {k = 1} ^ {T} \mathbb {E} _ {d ^ {(k)}} f ^ {(k)} (x, a) \leq \sum_ {k = 1} ^ {T} \| d ^ {(k)} \| _ {\Sigma_ {k} ^ {- 1}} \sqrt {\sum_ {i = 1} ^ {k - 1} \left(\mathbb {E} _ {d ^ {(i)}} [ f ^ {(k)} (x , a) ]\right) ^ {2} + \lambda S A} \\ \leq \sum_ {k = 1} ^ {T} \| d ^ {(k)} \| _ {\Sigma_ {k} ^ {- 1}} \left(\varepsilon + \sqrt {\lambda S A}\right) \\ \leq 2 \varepsilon \sum_ {k = 1} ^ {T} \| d ^ {(k)} \| _ {\Sigma_ {k} ^ {- 1}} \quad (\lambda = \varepsilon^ {2} / S A) \\ \leq 2 \varepsilon \sqrt {T} \sqrt {\sum_ {k = 1} ^ {T} \| d ^ {(k)} \| _ {\Sigma_ {k} ^ {- 1}} ^ {2}} \\ \leq 2 \varepsilon \sqrt {T} \sqrt {S A \log (1 + T / \varepsilon^ {2})}. \quad (\text {e l l i p t i c a l p o t e n t i a l}) \\ \end{array}
$$
Rearranging, we have  $\sqrt{T} < 2\sqrt{SA\log(1 + T / \varepsilon^2)}$ , which implies  $T \leq 4SA\log (1 + T / \varepsilon^2)$ . Then by applying Lemma G.5, we have  $T \leq 24SA\log (1 + 4SA / \varepsilon^2)$ . This concludes the proof.
H Proofs for Online RL
H.1 Preliminary Lemmas
Lemma H.1. For any policy  $\pi$ , conditional distribution  $d$  and  $h \in [H]$ , we have
$$
\overline {{\mathcal {T} _ {h} ^ {\pi , D} d (x , a)}} = \mathcal {T} _ {h} ^ {\pi} \bar {d} (x, a),
$$
$$
\overline {{\mathcal {T} _ {h} ^ {\star , D} d (x , a)}} = \mathcal {T} _ {h} ^ {\star} \bar {d} (x, a).
$$
$$
\begin{array}{l} \overline {{\mathcal {T} _ {h} ^ {\pi , D} d (x , a)}} = \mathbb {E} _ {y \sim \mathcal {T} _ {h} ^ {\pi , D} d (x, a)} [ y ] \\ = \mathbb {E} _ {c \sim C _ {h} (x, a), x ^ {\prime} \sim P _ {h} (x, a), a ^ {\prime} \sim \pi_ {h + 1} (x ^ {\prime}), y ^ {\prime} \sim d (x ^ {\prime}, a ^ {\prime})} [ c + y ^ {\prime} ] \\ = \bar {C} _ {h} (x, a) + \mathbb {E} _ {x ^ {\prime} \sim P _ {h} (x, a), a ^ {\prime} \sim \pi_ {h + 1} (x ^ {\prime}), y ^ {\prime} \sim d (x ^ {\prime}, a ^ {\prime})} [ y ^ {\prime} ] \\ = \bar {C} _ {h} (x, a) + \mathbb {E} _ {x ^ {\prime} \sim P _ {h} (x, a), a ^ {\prime} \sim \pi_ {h + 1} \left(x ^ {\prime}\right)} \left[ \bar {d} \left(x ^ {\prime}, a ^ {\prime}\right) \right] \\ = \mathcal {T} _ {h} ^ {\pi} \bar {d} (x, a). \\ \end{array}
$$
$$
\begin{array}{l} \overline {{\mathcal {T} _ {h} ^ {\star , D} d (x , a)}} = \mathbb {E} _ {y \sim \mathcal {T} _ {h} ^ {\star , D} d (x, a)} [ y ] \\ = \mathbb {E} _ {c \sim C _ {h} (x, a), x ^ {\prime} \sim P _ {h} (x, a), a ^ {\prime} = \arg \min  _ {\tilde {a}} \bar {d} \left(x ^ {\prime}, \tilde {a}\right), y ^ {\prime} \sim d \left(x ^ {\prime}, a ^ {\prime}\right)} [ c + y ^ {\prime} ] \\ = \bar {C} _ {h} (x, a) + \mathbb {E} _ {x ^ {\prime} \sim P _ {h} (x, a), a ^ {\prime} = \arg \min  _ {\tilde {a}} \bar {d} \left(x ^ {\prime}, \tilde {a}\right), y ^ {\prime} \sim d \left(x ^ {\prime}, a ^ {\prime}\right)} \left[ y ^ {\prime} \right] \\ = \bar {C} _ {h} (x, a) + \mathbb {E} _ {x ^ {\prime} \sim P _ {h} (x, a), a ^ {\prime} = \arg \min  _ {\widetilde {a}} \bar {d} \left(x ^ {\prime}, \widetilde {a}\right)} \left[ \bar {d} \left(x ^ {\prime}, a ^ {\prime}\right) \right] \\ = \bar {C} _ {h} (x, a) + \mathbb {E} _ {x ^ {\prime} \sim P _ {h} (x, a)} \left[ \min  _ {a ^ {\prime}} \bar {d} \left(x ^ {\prime}, a ^ {\prime}\right) \right] \\ = \mathcal {T} _ {h} ^ {\star} \bar {d} (x, a). \\ \end{array}
$$
Lemma H.2 (Performance Difference Lemma (PDL)). For any  $f: (\mathcal{X} \times \mathcal{A} \to \mathbb{R})^H$  and policies  $\pi, \pi'$ , we have
$$
V ^ {\pi} - \mathbb {E} _ {a \sim \pi^ {\prime} \left(x _ {1}\right)} \left[ f _ {1} \left(x _ {1}, a\right) \right] = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi} \left[ \mathcal {T} _ {h} ^ {\pi^ {\prime}} f _ {h + 1} \left(x _ {h}, a _ {h}\right) - f _ {h} \left(x _ {h}, \pi^ {\prime}\right) \right]. \tag {9}
$$
Proof. We proceed by inducting on the following claim: for all  $h = H + 1, H, \ldots, 1$ ,
$$
V _ {h} ^ {\pi} \left(x _ {h}\right) - f _ {h} \left(x _ {h}, \pi^ {\prime}\right) = \sum_ {t = h} ^ {H} \mathbb {E} _ {\pi , x _ {h}} \left[ \mathcal {T} _ {t} ^ {\pi^ {\prime}} f _ {t + 1} \left(x _ {t}, a _ {t}\right) - f _ {t} \left(x _ {t}, \pi^ {\prime}\right) \right].
$$
The base case of  $H + 1$  is trivially true as everything is 0. Now fix any  $h$  and suppose the IH at  $h + 1$  is true. Then
$$
\begin{array}{l} V _ {h} ^ {\pi} \left(x _ {h}\right) - f _ {h} \left(x _ {h}, \pi^ {\prime}\right) \\ = \mathbb {E} _ {\pi , x _ {h}} \left[ c _ {h} + V _ {h + 1} ^ {\pi} \left(x _ {h + 1}\right) - f _ {h + 1} \left(x _ {h + 1}, \pi^ {\prime}\right) + f _ {h + 1} \left(x _ {h + 1}, \pi^ {\prime}\right) - f _ {h} \left(x _ {h}, \pi^ {\prime}\right) \right] \\ = \mathbb {E} _ {\pi , x _ {h}} \left[ V _ {h + 1} ^ {\pi} \left(x _ {h + 1}\right) - f _ {h + 1} \left(x _ {h + 1}, \pi^ {\prime}\right) \right] + \mathbb {E} _ {\pi , x _ {h}} \left[ c _ {h} + f _ {h + 1} \left(x _ {h + 1}, \pi^ {\prime}\right) - f _ {h} \left(x _ {h}, \pi^ {\prime}\right) \right]. \\ \end{array}
$$
By the IH, the first term is equal to  $\sum_{t = h + 1}^{H}\mathbb{E}_{\pi ,x_h}\left[\mathcal{T}_t^{\pi '}f_{t + 1}(x_t,a_t) - f_t(x_t,\pi ')\right]$ . The second term is exactly  $\mathbb{E}_{\pi ,x_h}\left[\mathcal{T}_h^{\pi '}f_{h + 1}(x_h,a_h) - f_h(x_h,\pi ')\right]$ , which concludes the proof.
H.2 Proof of Small-Loss Regret and PAC Bounds
Recall that we defined the function class and distribution class, for each  $h$ , as
$$
\mathcal {D} _ {h} (\Pi) = \left\{(x, a) \mapsto d _ {h} ^ {\pi} (x, a): \pi \in \Pi \right\} \tag {10}
$$
$$
\Psi_ {h} = \left\{(x, a) \mapsto D _ {\triangle} (f (x, a) \| \mathcal {T} ^ {\star , D} f (x, a)): f \in \mathcal {F} \right\}.
$$
Also, define the  $V$ -type' analogs as follows, which will be useful for PAC instead of regret bounds.
$$
\mathcal {D} _ {h, v} (\Pi) = \{x \mapsto d _ {h} ^ {\pi} (x): \pi \in \Pi \} \tag {11}
$$
$$
\Psi_ {h, v} = \left\{x \mapsto \mathbb {E} _ {a \sim \operatorname {U n i f} (\mathcal {A})} [ D _ {\triangle} (f (x, a) \| \mathcal {T} ^ {\star , D} f (x, a)) ]: f \in \mathcal {F} \right\}.
$$
Let us also overload notation for the eluder dimensions as
$$
\mathrm {D E} _ {1} (\varepsilon) := \max  _ {h} \mathrm {D E} _ {1} (\Psi_ {h}, \mathcal {D} _ {h} (\Pi), \varepsilon),
$$
$$
\mathrm {D E} _ {1, v} (\varepsilon) := \max  _ {h} \mathrm {D E} _ {1} \left(\Psi_ {h, v}, \mathcal {D} _ {h, v} (\Pi), \varepsilon\right).
$$
Before we prove the following main theorem, a couple of remarks are in order:
1. Recall that by Theorem G.6, we have  $\mathrm{DE}_1(\varepsilon) \leq \mathcal{O}(SA\log (SA / \varepsilon))$  and by Theorem G.4, we have  $\mathrm{DE}_{1,v}(\varepsilon) \leq \mathcal{O}(d\log (d / \varepsilon))$ . This shows that the Eluder dimension in terms in Theorem 5.5 are appropriately bounded.
2. In Appendix D, we showed that distributional BC (Assumption 5.1) is satisfied in low-rank MDPs and the log bracketing number is bounded by  $\mathcal{O}(dM\log (d / \varepsilon) + \log |\Phi |)$  where  $\Phi$  is a realizable class for  $\phi^{\star}$ . This shows that the BC assumption of Theorem 5.5 is satisfied and  $\beta$  is appropriately bounded for low-rank MDPs.
Taken together, these two points imply that we have a small-loss PAC bound for low-rank MDPs: concretely, we have  $V^{\bar{\pi}} - V^{\star} \leq \widetilde{\mathcal{O}}\left(dH\sqrt{\frac{AV^{\star}\log|\Phi|}{K}} + \frac{d^{2}H^{2}A\log|\Phi|}{K}\right)$ .
We now prove the our main result for online RL: Theorem 5.5. We will prove the result with general function classes, so we will replace the  $|\mathcal{F}|$  by its  $\ell_{\infty}$  bracketing number, i.e.,  $\beta = \log (HKN_{\mathbb{I}}(1 / K,\mathcal{F},\ell_{\infty}) / \delta)$ .
$$
\operatorname {R e g r e t} _ {\mathrm {O - D I S C O}} (K) \leq 1 6 0 H \sqrt {K V ^ {\star} \mathrm {D E} _ {1} (1 / K) \log (K) \beta} + 1 8 0 0 0 H ^ {2} \mathrm {D E} _ {1} (1 / K) \log (K) \beta .
$$
Proof. For shorthand, let  $\delta_{h,k}(x,a) := D_{\triangle}(f_h^{(k)}(x,a)) \parallel \mathcal{T}_h^{\star,D}f_{h+1}^{(k)}(x,a))$  and  $\Delta_k := \sum_{h=1}^{H}\mathbb{E}_{\pi^k}[\delta_{h,k}(x_h,a_h)]$ . Notice that since  $\pi_{h+1}^k(x) = \arg \min_a\bar{f}_{h+1}^{(k)}(x,a)$ , we have  $\mathcal{T}_h^{\pi^k,D}f_{h+1}^{(k)}(x,a) = \mathcal{T}_h^{\star,D}f_{h+1}^{(k)}(x,a)$ , so  $\delta_{h,k}(x,a) = D_{\triangle}(f_h^{(k)}(x,a)) \parallel \mathcal{T}_h^{\pi^k,D}f_{h+1}^{(k)}(x,a))$  as well.
By Theorem F.2, we have the following two facts for all  $k \in [K]$ ,
(i) Optimism:  $\min_{a}\bar{f}_{1}^{(k)}(x_{1},a)\leq V^{\star}$  (since  $Z^{\star}\in \mathcal{F}_k$  ) and (ii) Low training error: for all  $h$  , we have
$$
\text {I f} \mathrm {U A E} = \text {F A L S E}. \sum_ {i <   k} \mathbb {E} _ {\pi^ {i}} [ \delta_ {h, k} (s _ {h}, a _ {h}) ] \leq 2 4 0 \beta .
$$
$$
\text {I f} \mathrm {U A E} = \text {T R U E}. \sum_ {i <   k} \mathbb {E} _ {\pi^ {i}} \left[ \mathbb {E} _ {a ^ {\prime} \sim \operatorname {u n i f} (\mathcal {A})} \left[ \delta_ {h, k} \left(s _ {h}, a _ {h}\right) \right] \right] \leq 2 4 0 \beta .
$$
The 240 comes from the constants of Theorem F.2 and the fact that  $D_{\triangle}(a,b) \leq 4H^{2}(a,b)$  for all distributions  $a, b$ .
Now, fix any episode  $k \in [K]$ .
(Lemma H.3 and  $\mathbb{E}_{\pi}[Q_h^\pi (s_h,a_h)]\leq V^\pi$
$$
\begin{array}{l} V ^ {\pi^ {k}} - V ^ {\star} \\ \leq V ^ {\pi^ {k}} - \min  _ {a} \bar {f} _ {1} ^ {(k)} \left(x _ {1}, a\right) (Fact(i)) \\ = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {k}} \left[ \mathcal {T} _ {h} ^ {\pi^ {k}} \bar {f} _ {h + 1} ^ {(k)} \left(x _ {h}, a _ {h}\right) - \bar {f} _ {h} ^ {(k)} \left(x _ {h}, \pi_ {h} ^ {k} \left(x _ {h}\right)\right) \right] (PDLLemmaH.2) \\ = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {k}} \left[ \overline {{\mathcal {T} _ {h} ^ {\pi^ {k} , D} f _ {h + 1} ^ {(k)}}} \left(x _ {h}, a _ {h}\right) - \bar {f} _ {h} ^ {(k)} \left(x _ {h}, a _ {h}\right) \right] (Lemma H.1) \\ \leq \sum_ {h = 1} ^ {H} \sqrt {\mathbb {E} _ {\pi^ {k}} \left[ 4 \bar {f} _ {h} ^ {(k)} \left(x _ {h} , a _ {h}\right) + \delta_ {h , k} \left(x _ {h} , a _ {h}\right) \right]} \cdot \sqrt {\mathbb {E} _ {\pi^ {k}} \left[ \delta_ {h , k} \left(x _ {h} , a _ {h}\right) \right]} \quad (\text {E q .} (\triangle_ {2})) \\ \leq \sum_ {h = 1} ^ {H} \sqrt {4 e V ^ {\pi^ {k}} + 1 7 H \sum_ {t = h} ^ {H} \mathbb {E} _ {\pi^ {k}} [ \delta_ {t , k} (x _ {t} , a _ {t}) ]} \cdot \sqrt {\mathbb {E} _ {\pi^ {k}} [ \delta_ {h , k} (x _ {h} , a _ {h}) ]} \\ \leq \sqrt {4 e V ^ {\pi^ {k}} + 1 7 H \Delta_ {k}} \cdot \sqrt {H \Delta_ {k}} (★) \\ \leq \sqrt {4 e H V ^ {\pi^ {k}} \Delta_ {k}} + 5 H \Delta_ {k} \\ \leq 2 \sqrt {H} \eta^ {- 1} V ^ {\pi^ {k}} + 2 \sqrt {H} \eta \Delta_ {k} + 5 H \Delta_ {k}. \\ \end{array}
$$
In  $\star$ , we used Cauchy Schwartz. Setting  $\eta = 4\sqrt{H}$  and rearranging, we have
$$
V ^ {\pi^ {k}} \leq 2 V ^ {\star} + 1 6 H \Delta_ {k} + 1 0 H \Delta_ {k} \leq 2 V ^ {\star} + 2 6 H \Delta_ {k}.
$$
Plugging this into  $\star$ , and noting  $104e + 17 \leq 300$ , we have
$$
V ^ {\pi^ {k}} - V ^ {\star} \leq \sqrt {8 e V ^ {\star} + 3 0 0 H \Delta_ {k}} \sqrt {H \Delta_ {k}}.
$$
Thus, summing the instantaneous regrets over all episodes, we get
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} V ^ {\pi^ {k}} - V ^ {\star} \leq \sum_ {k = 1} ^ {K} \sqrt {8 e V ^ {\star} + 3 0 0 H \Delta_ {k}} \sqrt {H \Delta_ {k}} \\ \leq \sqrt {8 e K V ^ {\star} + 3 0 0 H \sum_ {k} \Delta_ {k}} \sqrt {H \sum_ {k} \Delta_ {k}} \quad (\text {C a u c h y - S c h w a r t z}) \\ \leq 5 \sqrt {H K V ^ {\star} \sum_ {k} \Delta_ {k}} + 1 8 H \sum_ {k} \Delta_ {k}. \\ \end{array}
$$