SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition E.1 (Bracketing Number). Let  $\mathcal{G}$  be a set of functions mapping  $\mathcal{X} \to \mathbb{R}$ . Given two functions  $l, u$  such that  $l(x) \leq u(x)$  for all  $x \in \mathcal{X}$ , the bracket  $[l, u]$  is the set of functions  $g \in \mathcal{G}$  such that  $l(x) \leq g(x) \leq u(x)$  for all  $x \in \mathcal{X}$ . We call  $[l, u]$  an  $\varepsilon$ -bracket if  $\|u - l\| \leq \varepsilon$ . Then, the  $\varepsilon$ -bracketing number of  $\mathcal{G}$  with respect to  $\|\cdot\|$ , denoted by  $N_{\mathbb{I}}(\varepsilon, \mathcal{G}, \| \cdot\|)$  is the minimum number of  $\varepsilon$ -brackets needed to cover  $\mathcal{G}$ .
Since the triangular discrimination is equivalent to squared Hellinger up to universal constants, we now prove MLE generalization bounds in terms of squared Hellinger.
Lemma E.2. Let  $f_{1}:\mathcal{X}\to \Delta (\mathcal{Y})$  and  $f_{2}:\mathcal{X}\times \mathcal{Y}\rightarrow \mathbb{R}_{+}$  satisfying  $\sup_{x\in \mathcal{X}}\int_{\mathcal{Y}}f_2(x,y)\mathrm{d}y\leq s$ , then for any distribution  $\mathcal{D}\in \Delta (\mathcal{X})$ , we have
$$
\mathbb {E} _ {x \sim \mathcal {D}} \left[ H ^ {2} \left(f _ {1} (x) \| f _ {2} (x, \cdot)\right) \right] \leq (s - 1) - 2 \log \mathbb {E} _ {x \sim \mathcal {D}, y \sim f _ {1} (x)} \exp \left(- \frac {1}{2} \log \left(f _ {1} (x, y) / f _ {2} (x, y)\right)\right).
$$
Proof. This follows from the proof of Wu et al. [2023, Lemma C.1].
Lemma E.3. Fix  $\delta \in (0,1)$ . Then w.p. at least  $1 - \delta$ , for any  $f \in \mathcal{F}$ , we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} \mathbb {E} _ {x \sim \mathcal {D} _ {i}} \left[ H ^ {2} (f (x, \cdot) \| f ^ {\star} (x, \cdot)) \right] \\ \leq 6 n \epsilon | \mathcal {Y} | + 2 \sum_ {i = 1} ^ {n} \log \left(f ^ {\star} \left(x _ {i}, y _ {i}\right) / f \left(x _ {i}, y _ {i}\right)\right) + 8 \log \left(N _ {\llbracket} (\epsilon , \mathcal {F}, \| \cdot \| _ {\infty}) / \delta\right). \tag {4} \\ \end{array}
$$
Rearranging, we also have
$$
\sum_ {i = 1} ^ {n} \log \left(f \left(x _ {i}, y _ {i}\right) / f ^ {\star} \left(x _ {i}, y _ {i}\right)\right) \leq 3 n \epsilon | \mathcal {Y} | + 4 \log \left(N _ {[ ]} (\epsilon , \mathcal {F}, \| \cdot \| _ {\infty}) / \delta\right). \tag {5}
$$
Proof. We take an  $\epsilon$ -bracketing of  $\mathcal{F}$ ,  $\{[l_i, u_i] : i = 1, 2, \ldots\}$ , and denote  $\widetilde{\mathcal{F}} = \{u_i : i = 1, 2, \ldots\}$ . Applying Lemma 24 of Agarwal et al. [2020] to function class  $\widetilde{\mathcal{F}}$  and using Chernoff method, w.p. at least  $1 - \delta$ , for all  $\tilde{f} \in \widetilde{\mathcal{F}}$ , we have
$$
\underbrace {- \log_ {\mathbb {E} _ {D ^ {\prime}}} \exp \left(L \left(\tilde {f} (D) , D ^ {\prime}\right)\right)} _ {\text {(i)}} \leq \underbrace {- L \left(\tilde {f} (D) , D\right) + 2 \log \left(N _ {[ ]} \left(\epsilon , \mathcal {F} , \| \cdot \| _ {\infty}\right) / \delta\right)} _ {\text {(i i)}}. \tag {6}
$$
Now, fix any  $f \in \mathcal{F}$  and pick  $\tilde{f} \in \widetilde{\mathcal{F}}$  as the upper bracket, i.e.,  $f \leq \tilde{f}$ . Now set  $L(f, D) = \sum_{i=1}^{n} -1/2 \log (f^*(x_i, y_i) / f(x_i, y_i))$ . Then the right hand side of (6) is
$$
\begin{array}{l} \left(\mathrm {i i}\right) = \frac {1}{2} \sum_ {i = 1} ^ {n} \log \left(f ^ {\star} \left(x _ {i}, y _ {i}\right) / \tilde {f} \left(x _ {i}, y _ {i}\right)\right) + 2 \log \left(N _ {\mathbb {I}} \left(\epsilon , \mathcal {F}, \| \cdot \| _ {\infty}\right) / \delta\right) \\ \leq \frac {1}{2} \sum_ {i = 1} ^ {n} \log \left(f ^ {\star} \left(x _ {i}, y _ {i}\right) / f \left(x _ {i}, y _ {i}\right)\right) + 2 \log \left(N _ {\mathbb {I}} \left(\epsilon , \mathcal {F}, \| \cdot \| _ {\infty}\right) / \delta\right). \\ \end{array}
$$
On the other hand, since  $H$  is a metric, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} \underset {x \sim \mathcal {D} _ {i}} {\mathbb {E}} H ^ {2} (f (x, \cdot), f ^ {\star} (x, \cdot)) \leq \sum_ {i = 1} ^ {n} \underset {x \sim \mathcal {D} _ {i}} {\mathbb {E}} \left(H (f (x, \cdot), \tilde {f} (x, y)\right) + H (\tilde {f} (x, y), f ^ {\star} (x, \cdot))\left. \right) ^ {2} \\ \leq 2 \underbrace {\sum_ {i = 1} ^ {n} \underset {x \sim \mathcal {D} _ {i}} {\mathbb {E}} H ^ {2} \left(f (x , \cdot) , \tilde {f} (x , y)\right)} _ {\mathrm {(i i i)}} + 2 \underbrace {\sum_ {i = 1} ^ {n} \underset {x \sim \mathcal {D} _ {i}} {\mathbb {E}} H ^ {2} \left(\tilde {f} (x , y) , f ^ {\star} (x , \cdot)\right)} _ {\mathrm {(i v)}}. \\ \end{array}
$$
For (iii), by the definition, we have  $\tilde{f}(x,y) - f(x,y) \in [0,\epsilon]$  for all  $x$ , so
$$
\text{(iii)} = \sum_{i = 1}^{n}\underset {x\sim \mathcal{D}_{i}}{\mathbb{E}}H^{2}\left(f(x,\cdot),\tilde{f} (x,y)\right) \leq \sum_{i = 1}^{n}\underset {x\sim \mathcal{D}_{i}}{\mathbb{E}}2\int_{y}\Big|f(x,y) - \tilde{f} (x,y)\Big|\mathrm{d}y\leq 2n\epsilon |\mathcal{Y}|.
$$
For (iv), we apply Lemma E.2 with  $f_{1} = f^{\star}$  and  $f_{2} = \tilde{f}$  (thus  $s = 1 + \epsilon |\mathcal{V}|$ ) and get
$$
\begin{array}{l} \left(\mathrm {i v}\right) = n \epsilon | \mathcal {Y} | - 2 \sum_ {i = 1} ^ {n} \log \underset {x, y \sim f ^ {\star} (x, \cdot)} {\mathbb {E}} \exp \left(- \frac {1}{2} \log \left(f ^ {\star} (x, y) / \tilde {f} (x, y)\right)\right) \\ = n \epsilon | \mathcal {Y} | - 2 \sum_ {i = 1} ^ {n} \log_ {x, y \sim \mathcal {D} _ {i}} \mathbb {E} \exp \left(- \frac {1}{2} \log \left(f ^ {\star} (x, y) / \tilde {f} (x, y)\right)\right) \\ = n \epsilon | \mathcal {Y} | - 2 \log \underset {x, y \sim \mathcal {D} ^ {\prime}} {\mathbb {E}} \left[ \exp \left(\sum_ {i = 1} ^ {n} - \frac {1}{2} \log \left(f ^ {\star} (x, y) / \tilde {f} (x, y)\right)\right) \Bigg | D \right] \\ = n \epsilon | \mathcal {Y} | + 2 \cdot (i). \\ \end{array}
$$
By plugging (iii) and (iv) back we get
$$
\sum_ {i = 1} ^ {n} \underset {x \sim \mathcal {D} _ {i}} {\mathbb {E}} H ^ {2} \left(f (x, \cdot), f ^ {\star} (x, \cdot)\right) \leq 6 n \epsilon | \mathcal {Y} | + 4 \cdot (\mathrm {i}).
$$
Notice that (i)  $\leq$  (ii), so we complete the proof by plugging (ii) into the above.
We first state the MLE generalization result for finite  $\mathcal{F}$ .
Theorem E.4. Suppose  $\mathcal{F}$  is finite. Fix any  $\delta \in (0,1)$ , set  $\beta = \log (|\mathcal{F}| / \delta)$  and define
$$
\widehat {\mathcal {F}} = \left\{f \in \mathcal {F}: \sum_ {i = 1} ^ {n} \log f (x _ {i}, y _ {i}) \geq \max _ {\widetilde {f} \in \mathcal {F}} \sum_ {i = 1} ^ {n} \widetilde {f} (x _ {i}, y _ {i}) - 4 \beta \right\}.
$$
Then w.p. at least  $1 - \delta$ , the following holds:
(1) The true distribution is in the version space, i.e.,  $f^{\star} \in \widehat{\mathcal{F}}$ .
(2) Any function in the version space is close to the ground truth data-generating distribution, i.e., for all  $f \in \hat{\mathcal{F}}$
$$
\sum_ {i = 1} ^ {n} \mathbb {E} _ {x \sim \mathcal {D} _ {i}} \left[ H ^ {2} (f (x, \cdot) \| f ^ {\star} (x, \cdot)) \right] \leq 2 2 \beta .
$$
Proof. These two claims follow from Lemma E.3 with  $\epsilon = 0$ , and so  $N_{\mathbb{I}}(\epsilon, \mathcal{F}, \| \cdot \|_{\infty}) = |\mathcal{F}|$ . For (1), apply Eq. (5) to  $f = \widehat{f}_{\mathrm{MLE}}$  to see that  $f^{\star} \in \widehat{\mathcal{F}}$ . For (2), apply Eq. (4) and note that the sum term is at most  $4\beta$ . Thus, the right hand side of Eq. (4) is at most  $(6 + 8 + 8)\beta = 22\beta$ .
We now state the result for infinite  $\mathcal{F}$  using bracketing entropy.
Theorem E.5. Fix any  $\delta \in (0,1)$ , set  $\beta = \log (N_{\mathbb{I}}((n|\mathcal{Y}|)^{-1},\mathcal{F},\| \cdot \|_{\infty}) / \delta)$  and define
$$
\widehat {\mathcal {F}} = \left\{f \in \mathcal {F}: \sum_ {i = 1} ^ {n} \log f (x _ {i}, y _ {i}) \geq \max _ {\widetilde {f} \in \mathcal {F}} \sum_ {i = 1} ^ {n} \widetilde {f} (x _ {i}, y _ {i}) - 7 \beta \right\}.
$$
(1) The true distribution is in the version space, i.e.,  $f^{\star} \in \widehat{\mathcal{F}}$ .
(2) Any function in the version space is close to the ground truth data-generating distribution, i.e., for all  $f \in \widehat{\mathcal{F}}$
$$
\sum_ {i = 1} ^ {n} \mathbb {E} _ {x \sim \mathcal {D} _ {i}} \left[ H ^ {2} (f (x, \cdot) \| f ^ {\star} (x, \cdot)) \right] \leq 2 8 \beta .
$$
Proof. These two claims follow from Lemma E.3 with  $\epsilon = 1 / n|\mathcal{Y}|$ . For (1), apply Eq. (5) to  $f = \widehat{f}_{\mathrm{MLE}}$  to see that  $f^{\star}\in \widehat{\mathcal{F}}$ . For (2), apply Eq. (4) and note that the sum term is at most  $7\beta$ . Thus, the right hand side of Eq. (5) is at most  $(6 + 14 + 8)\beta = 28\beta$ .
F Confidence set construction with general function class
In this section, we extend the confidence set construction of O-DISCO and P-DISCO to general  $\mathcal{F}$ , which can be infinite. Our procedure constructs the confidence set by performing the thresholding scheme on an  $\varepsilon$ -net of  $\mathcal{F}$ . While constructing an  $\varepsilon$ -net for  $\mathcal{F}$  is admittedly a computationally hard procedure, this is still information theoretically possible and our focus in O-DISCO and P-DISCO is to show that distributional RL information-theoretically leads to small-loss bounds.
We first define some notations. Let  $\mathcal{F}^{\downarrow}$  and  $\mathcal{F}^{\uparrow}$  denote a lower and upper  $\varepsilon$ -bracketing of  $\mathcal{F}$ , i.e., for any  $f \in \mathcal{F}$ , there exists an  $\varepsilon$ -bracket  $[f^{\downarrow}, f^{\uparrow}]$  such that for all  $h$ ,  $f_h^{\downarrow} \leq f_h \leq f_h^{\uparrow}$  with  $f^{\downarrow} \in \mathcal{F}^{\downarrow}$ ,  $f^{\uparrow} \in \mathcal{F}^{\uparrow}$ . Recall that a lower bracket  $g \in \mathcal{F}^{\downarrow}$  may not be a valid distribution, but since elements of  $\mathcal{F}$  map to non-negative values, we can assume  $g$  has non-negative entries as well. Also, we have  $\alpha_h^g(x, a) := \int g_h(z \mid x, a) \geq 1 - \varepsilon$ , so for  $\varepsilon$  small enough,  $g$  is normalizable. Hence, define  $\widetilde{g}(z \mid x, a) = \alpha_h^g(x, a)^{-1} g(z \mid x, a)$  as the normalized version, which is a valid distribution that we can sample from.
Now, consider any martingale  $\{x_{h,i},a_{h,i},c_{h,i}\}_{i\in [n],h\in [H]}$ , which could be the online data up to episode  $k$  or the offline data (consisting of  $N$  i.i.d. samples). We define the MLE with respect to a lower bracket element as follows. For any  $h\in [H],g\in \mathcal{F}^{\downarrow},\pi \in \Pi$ , sample  $y_{h,i}^{g,\pi}\sim \widetilde{g}_{h + 1}(x_{h,i}^{\prime},\pi (x_{h,i}^{\prime}))$ , and  $z_{h,i}^{g,\pi} = c_{h,i} + y_{h,i}^{g,\pi}$ , define the MLE solution for  $(g,\pi)$  at time  $h$  as,
$$
\mathrm {M L E} _ {h} ^ {g, \pi} = \underset {f \in \mathcal {F}} {\arg \max } \sum_ {i = 1} ^ {n} \log f _ {h} \left(z _ {h, i} ^ {g, \pi} \mid x _ {h, i}, a _ {h, i}\right).
$$
Also, define the version space with respect to the above MLE as,
$$
\mathcal {F} _ {g, \pi , h} = \left\{f \in \mathcal {F}: \sum_ {i = 1} ^ {n} \log f _ {h} \left(z _ {h, i} ^ {g, \pi} \mid x _ {h, i}, a _ {h, i}\right) \geq \sum_ {i = 1} ^ {n} \log \mathrm {M L E} _ {h} ^ {g, \pi} \left(z _ {h, i} ^ {g, \pi} \mid x _ {h, i}, a _ {h, i}\right) - \beta \right\}.
$$
We now prove a key result that implies that  $\mathcal{T}_h^\pi f_{h + 1}^\downarrow$  falls into the confidence set  $\mathcal{F}_{f^{\downarrow},\pi ,h}$
Theorem F.1. For any  $\delta \in (0,1)$  and suppose  $n\geq 2$ . Then, w.p. at least  $1 - \delta$ , for any  $h\in [H],g\in \mathcal{F},f^{\downarrow}\in \mathcal{F}^{\downarrow},\pi \in \Pi$ , we have
$$
\sum_ {i = 1} ^ {n} \log g _ {h} (z _ {h, i} ^ {f ^ {\downarrow}, \pi} \mid x _ {h, i}, a _ {h, i}) - \log \mathcal {T} _ {h} ^ {\pi} f _ {h + 1} ^ {\downarrow} (z _ {h, i} ^ {f ^ {\downarrow}, \pi} \mid x _ {h, i}, a _ {h, i}) \leq \log (e ^ {4} N _ {[ ]} (n ^ {- 1}, \mathcal {F}, \| \cdot \| _ {\infty}) ^ {2} | \Pi | / \delta).
$$
where  $z_{h,i}^{f^{\downarrow},\pi} = c_{h,i} + y_{h,i}^{f^{\downarrow},\pi}$  and  $y_{h,i}^{f^{\downarrow},\pi} \sim \widetilde{f}_{h+1}^{\downarrow}(\cdot \mid x_{h,i}', \pi_{h+1}(x_{h,i}'))$ .
Proof of Theorem F.1. Consider a  $\varepsilon$ -bracketing of  $\mathcal{F}$  where  $\varepsilon \leq 1/n \leq 1/2$ ; we will study each element and conclude with a union bound. For any lower bracket  $l$  and upper bracket  $u$  in the bracketing (note  $l, u$  need not correspond to the same bracket). Recall that  $\alpha_{h+1}^l(x, a) := \int l_{h+1}(z|x, a)$ , so we have  $1 - \varepsilon \leq \alpha_{h+1}^l \leq 1$  since  $l$  is a lower  $\varepsilon$ -bracket of distributions. Therefore, we have
$$
\mathbb {E} \left[ \exp \sum_ {i = 1} ^ {n} \log \left(\frac {u _ {h} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}{\mathcal {T} _ {h} ^ {\pi} l _ {h + 1} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}\right) \right] = \prod_ {i = 1} ^ {n} \mathbb {E} _ {\nu_ {h, i}} \left[ \frac {u _ {h} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}{\mathcal {T} _ {h} ^ {\pi} l _ {h + 1} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)} \right],
$$
where  $\nu_{h,i}$  is the distribution of data from  $i$ -th round and time  $h$ . Note that  $\nu_{h,i}(x,a,c,x') = d_{h,i}(x,a)C_h(c|x,a)P_h(x'|x,a)$  for some distribution  $d_{h,i}(x,a)$ . Now focus on each  $i$ , so for all  $i$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\nu_ {h, i}} \left[ \frac {u _ {h} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}{\mathcal {T} _ {h} ^ {\pi} l _ {h + 1} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)} \right] \\ = \int_ {x, a, c, x ^ {\prime}, y} \nu_ {h, i} (x, a, c, x ^ {\prime}) \widetilde {l} _ {h + 1} (y | x ^ {\prime}, \pi (x ^ {\prime})) \frac {u _ {h} (c + y | x , a)}{\int_ {c , x ^ {\prime}} \nu_ {h , i} (c , x ^ {\prime} | x , a) l _ {h + 1} (y | x ^ {\prime} , \pi (x ^ {\prime}))} \\ = \int_ {x, a, z} d _ {h, i} (x, a) \int_ {z} u _ {h} (z \mid x, a) \\ \times \int_ {c, x ^ {\prime}} \nu_ {h, i} (c, x ^ {\prime} \mid x, a) \widetilde {l} _ {h + 1} (z - c \mid x ^ {\prime}, \pi (x ^ {\prime})) \frac {1}{\int_ {c , x ^ {\prime}} \nu_ {h , i} (c , x ^ {\prime} \mid x , a) l _ {h + 1} (z - c \mid x ^ {\prime} , \pi (x ^ {\prime}))} \\ = \int_ {x, a, z} d _ {h, i} (x, a) \int_ {z} u _ {h} (z \mid x, a) \alpha_ {h + 1} ^ {l} (x, a) ^ {- 1} \\ \leq \frac {1 + \varepsilon}{1 - \varepsilon} = 1 + \frac {2 \varepsilon}{1 - \varepsilon} \leq 1 + \frac {4}{n}. \\ \end{array}
$$
$$
\mathbb {E} \left[ \exp \sum_ {i = 1} ^ {n} \log \left(\frac {u _ {h} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}{\mathcal {T} _ {h} ^ {\pi} l _ {h + 1} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}\right) \right] \leq (1 + 4 / n) ^ {n} \leq e ^ {4}.
$$
Thus, by Markov's inequality, w.p. at least  $1 - \delta$ , we have
$$
\sum_ {i = 1} ^ {n} \log \left(\frac {u _ {h} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}{\mathcal {T} _ {h} ^ {\pi} l _ {h + 1} \left(z _ {h , i} ^ {l , \pi} \mid x _ {h , i} , a _ {h , i}\right)}\right) \leq \ln \left(e ^ {4} / \delta\right).
$$
To conclude, apply union bound to get this result for all brackets.
For the remainder of this section, we assume the policy class  $\Pi$  is finite. However, it is possible to extend our results using policy covers in the Hamming distance; in that case,  $\log |\Pi|$  would be replaced by the log covering number or entropy integral of  $\Pi$  [as in Zhou et al., 2023, Kallus et al., 2022]. We note that for the online case, we rely on the assumption that for any  $f \in \mathcal{F}$  we have  $\pi^f \in \Pi$ , where recall that  $\pi_h^f(x) = \arg \min_a \bar{f}_h(x, a)$ . This is because  $\mathcal{T}^{\star,D}$  is not a contraction so we cannot operate with  $\mathcal{T}^{\star,D}$  directly and instead operate with  $\mathcal{T}^{\pi^f,D}$ . We highlight that this assumption is automatically satisfied in tabular MDPs, since the whole policy space is finite, and  $\log |\Pi| = \mathcal{O}(X \log (A))$  is lower order compared to  $\log$  of the bracketing entropy of  $\mathcal{F}_{tab}$ , which is  $\mathcal{O}(X^2 A^2)$ . In contrast, in non-distributional methods such as GOLF, the regular Bellman optimality operator is a contraction so standard Lipschitz arguments for covering go through. We note that it is also possible to construct covers of  $\mathcal{F}$  in the Hellinger distance, but the metric entropy of  $\mathcal{F}_{tab}$  seems to be on the same order as its bracketing entropy.
We now describe the version space construction for general  $\mathcal{F}$ , first for the online setting. Fix any  $k$ , and define the set
$$
\mathcal {F} _ {f ^ {\downarrow}, \pi , h} = \left\{f \in \mathcal {F}: \sum_ {i = 1} ^ {k} \log f _ {h} \left(z _ {h, i} ^ {f ^ {\downarrow}, \pi} \mid x _ {h, i}, a _ {h, i}\right) \geq \sum_ {i = 1} ^ {k} \log \mathrm {M L E} _ {h} ^ {f ^ {\downarrow}, \pi} \left(z _ {h, i} ^ {f ^ {\downarrow}, \pi} \mid x _ {h, i}, a _ {h, i}\right) - \beta \right\}
$$
Then, construct the version space as
$$
\mathcal {F} _ {k} = \left\{f \in \mathcal {F}: f _ {h} \in \mathcal {F} _ {f ^ {\downarrow}, \pi^ {f}, h}, \forall h \in [ H ] \right\}.
$$
Theorem F.2. Fix any  $\delta \in (0,1)$  and suppose Assumption 5.1. Set  $\beta = \log (KH\cdot N_{\mathbb{I}}(K^{-1},\mathcal{F},\| \cdot \|\infty))|\Pi | / \delta)$ . Then, w.p. at least  $1 - \delta$ , the following holds:
(1) The optimal cost distribution is in the version space, i.e.,  $Z^{\star}\in \mathcal{F}_{k}$
(2) For all  $f \in \mathcal{F}_k$  and  $h \in [H]$ ,
$$
\sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \left[ H ^ {2} \left(f _ {h} \left(x _ {h}, a _ {h}\right) \parallel \mathcal {T} _ {h} ^ {\star , D} f _ {h + 1} \left(x _ {h}, a _ {h}\right)\right) \right] \leq 6 0 \beta .
$$
Proof. First, we want to verify that  $Z^{\star} \in \mathcal{F}_k$ . Let  $f^{\downarrow}$  be the lower bracket of  $Z^{\star}$  and set  $g = \mathrm{MLE}_{h}^{f^{\downarrow},\pi^{\star}} \in \mathcal{F}$ ; note  $\pi^{\star} = \pi^{Z^{\star}}$ . By Theorem F.1, we have  $\sum_{i=1}^{k} \log \mathrm{MLE}_{h}^{f^{\downarrow},\pi^{\star}}(z_{h,i}^{f^{\downarrow},\pi^{\star}}|x_{h,i},a_{h,i}) - \log \mathcal{T}_{h}^{\pi^{\star},D}f_{h+1}^{\downarrow}(z_{h,i}^{f^{\downarrow},\pi^{\star}}|x_{h,i},a_{h,i}) \leq \mathcal{O}(\beta)$ . Therefore, noting that  $Z_h^\star = \mathcal{T}_h^{\pi^\star ,D}Z_{h + 1}^\star \geq \mathcal{T}_h^{\pi^\star ,D}f_{h + 1}^\downarrow$  shows that  $Z_h^\star \in \mathcal{F}_{f^\downarrow,\pi^\star,h}$  for every  $h$ , implying that  $Z^{\star} \in \mathcal{F}_k$ .
For the second claim, fix any  $f \in \mathcal{F}_k$  and  $h \in [H]$ . Then,
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \left[ H ^ {2} \left(f _ {h} \left(x _ {h}, a _ {h}\right) \parallel \mathcal {T} _ {h} ^ {\star , D} f _ {h + 1} \left(x _ {h}, a _ {h}\right)\right) \right] \\ = \sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \left[ H ^ {2} \left(f _ {h} \left(x _ {h}, a _ {h}\right) \parallel \mathcal {T} _ {h} ^ {\pi^ {f}, D} f _ {h + 1} \left(x _ {h}, a _ {h}\right)\right) \right] \\ \leq 2 \sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \left[ H ^ {2} \left(f _ {h} \left(x _ {h}, a _ {h}\right) \| \mathcal {T} _ {h} ^ {\pi^ {f}, D} \widetilde {f} _ {h + 1} ^ {\downarrow} \left(x _ {h}, a _ {h}\right)\right) + H ^ {2} \left(\mathcal {T} _ {h} ^ {\pi^ {f}, D} \widetilde {f} _ {h + 1} ^ {\downarrow} \left(x _ {h}, a _ {h}\right) \| \mathcal {T} _ {h} ^ {\pi^ {f}, D} f _ {h + 1} \left(x _ {h}, a _ {h}\right)\right) \right] \\ \leq 2 (2 8 \beta + 3 k \varepsilon). \\ \end{array}
$$
The  $\beta$  comes from Theorem E.5, and for  $\varepsilon$ , we used the fact that  $H^2 \leq H \leq TV$ , and
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \left[ T V \left(\mathcal {T} _ {h} ^ {\pi^ {f}, D} \widetilde {f} _ {h + 1} ^ {\downarrow} \left(x _ {h}, a _ {h}\right) \| \mathcal {T} _ {h} ^ {\pi^ {f}, D} f _ {h + 1} \left(x _ {h}, a _ {h}\right)\right) \right] \\ = \sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \int_ {z} \left| \mathcal {T} _ {h} ^ {\pi^ {f}, D} \widetilde {f} _ {h + 1} ^ {\downarrow} (z \mid x _ {h}, a _ {h}) - \mathcal {T} _ {h} ^ {\pi^ {f}, D} f _ {h + 1} (z \mid x _ {h}, a _ {h})) \right| \\ = \sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \int_ {z} \sum_ {c, x ^ {\prime}} \nu (c, x ^ {\prime} \mid x _ {h}, a _ {h}) \left| \widetilde {f} _ {h + 1} ^ {\downarrow} (z - c \mid x ^ {\prime}, \pi^ {f} (x ^ {\prime})) - f _ {h + 1} (z - c \mid x ^ {\prime}, \pi^ {f} (x ^ {\prime})) \right| \\ \leq \sum_ {i = 1} ^ {k} 3 \varepsilon = 3 k \varepsilon , \\ \end{array}
$$
since for any  $x, a$ , we have  $\int_{z} \left| \widetilde{f}_{h+1}^{\downarrow}(z \mid x, a) - f_{h+1}(z \mid x, a) \right| \leq 3\varepsilon$ . There are two cases. If  $\widetilde{f}_{h+1}^{\downarrow}(z \mid x, a) \geq f_{h+1}(z \mid x, a)$ , then  $\widetilde{f}_{h+1}^{\downarrow}(z \mid x, a) - f_{h+1}(z \mid x, a) \leq (1 - \varepsilon)^{-1} f_{h+1}^{\downarrow}(z \mid x, a) - f_{h+1}(z \mid x, a) \leq 2\varepsilon f_{h+1}(z \mid x, a)$  since  $(1 - \varepsilon)^{-1} \leq 1 + 2\varepsilon$ . If  $\widetilde{f}_{h+1}^{\downarrow}(z \mid x, a) < f_{h+1}(z \mid x, a)$ , then  $f_{h+1}(z \mid x, a) - \widetilde{f}_{h+1}^{\downarrow}(z \mid x, a) \leq f_{h+1}(z \mid x, a) - f_{h+1}^{\downarrow}(z \mid x, a) \leq \varepsilon$ . Thus,  $\int_{z} \max(2\varepsilon f_{h+1}(z \mid x, a), \varepsilon) \leq \int_{z} 2\varepsilon f_{h+1}(z \mid x, a) + \varepsilon = 3\varepsilon$ . Thus, setting  $\varepsilon = 1 / K$  gives
$$
\sum_ {i = 1} ^ {k} \mathbb {E} _ {\pi^ {i}} \left[ H ^ {2} \left(f _ {h} \left(x _ {h}, a _ {h}\right) \| \mathcal {T} _ {h} ^ {\star , D} f _ {h + 1} \left(x _ {h}, a _ {h}\right)\right) \right] \leq 5 9 \beta .
$$
For the offline setting, fix any  $\pi$  and define its general version space as,
$$
\mathcal {F} _ {\pi} = \left\{f \in \mathcal {F}: f _ {h} \in \mathcal {F} _ {f ^ {\downarrow}, \pi , h}, \forall h \in [ H ] \right\}.
$$
Theorem F.3. Fix any  $\delta \in (0,1)$  and suppose Assumption 5.1. Set  $\beta = \log (H|\Pi| \cdot N_{\mathbb{I}}((n|\mathcal{Y}|)^{-1},\mathcal{F},\| \cdot \|_{\infty}) / \delta$ . Then, w.p. at least  $1 - \delta$ , the following holds for all policies  $\pi \in \Pi$ :
(1) The policy cost distribution is in the version space, i.e.,  $Z^{\pi} \in \mathcal{F}_{\pi}$ .
(2) Any function in the version space has bounded triangular discrimination with the ground truth data-generating distribution, i.e., for all  $f \in \mathcal{F}_{\pi}$  and  $h \in [H]$ ,
$$
\mathbb {E} _ {\nu_ {h}} \left[ H ^ {2} \left(f _ {h} \left(x _ {h}, a _ {h}\right) \parallel \mathcal {T} _ {h} ^ {\pi , D} f _ {h + 1} \left(x _ {h}, a _ {h}\right)\right) \right] \leq 6 0 \beta N ^ {- 1}.
$$
Proof. The proof is the same as in Theorem F.2, but instead of  $\pi^f$ , we fix any  $\pi$ .
G The  $\ell_p$  distributional eluder dimension
Let  $S$  denote any input space (for example, we will later instantiate  $\mathcal{S} = \mathcal{X}$  or  $\mathcal{S} = \mathcal{X} \times \mathcal{A}$ ). Let  $\Psi$  denote a set of functions mapping from  $S \to \mathbb{R}$ . Let  $\mathcal{D}$  be a set of distributions on  $S$ .
Recall the definition of  $\varepsilon$ -independent sequence (of distributions) from Jin et al. [2021a].
Definition G.1 ( $\ell_2$ -independent sequence). A distribution  $\nu \in \mathcal{D}$  is  $(\varepsilon, \ell_2)$ -independent of a sequence  $\{d^{(1)}, \ldots, d^{(n)}\} \subset \mathcal{D}$  if there exists  $\psi \in \Psi$  such that  $|\mathbb{E}_{\nu}\psi| > \varepsilon$  and also  $\sqrt{\sum_{i=1}^{n}(\mathbb{E}_{d^{(i)}}\psi)^2} \leq \varepsilon$ .
Note that the definition is on sequences of distributions, which generalizes the original definition on sequences of points from Russo and Van Roy [2013].
We now generalize the above definition for the general  $\ell_p$  norm.
Definition G.2 ( $\ell_p$ -independent sequence). A distribution  $\nu \in \mathcal{D}$  is  $(\varepsilon, \ell_p)$ -independent of a sequence  $\{d^{(1)}, \ldots, d^{(n)}\} \subset \mathcal{D}$  if there exists  $\psi \in \Psi$  such that  $|\mathbb{E}_{\nu}\psi| > \varepsilon$  and also  $\sum_{i=1}^{n} |\mathbb{E}_{d^{(i)}}\psi|^p \leq \varepsilon^p$ .