SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the analysis around maximum likelihood estimation, we largely follow the proving idea of Agarwal et al. (2020); Liu et al. (2022), which is inspired by Zhang (2006).
The next lemma shows that the ground truth model  $M^{\star}$  has a comparable log-likelihood compared with MLE solution.
Lemma 5. Let  $M^{\star}$  be the ground truth model. Then, with probability at least  $1 - \delta$ , we have
$$
\max _ {M \in \mathcal {M}} \log \ell_ {\mathcal {D}} (M) - \log \ell_ {\mathcal {D}} (M ^ {\star}) \leq \log (| \mathcal {M} | / \delta).
$$
Proof of Lemma 5. The proof of this lemma is obtained by a standard argument of MLE (see, e.g., van de Geer, 2000). For any  $M \in \mathcal{M}$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(\log \ell_ {\mathcal {D}} (M) - \log \ell_ {\mathcal {D}} (M ^ {\star})\right) \right] = \mathbb {E} \left[ \frac {\ell_ {\mathcal {D}} (M)}{\ell_ {\mathcal {D}} (M ^ {\star})} \right] \\ = \mathbb {E} \left[ \frac {\prod_ {(s , a , r , s ^ {\prime}) \in \mathcal {D}} P _ {M} \left(s ^ {\prime} \mid s , a\right)}{\prod_ {(s , a , r , s ^ {\prime}) \in \mathcal {D}} P _ {M ^ {*}} \left(s ^ {\prime} \mid s , a\right)} \right] \\ = \mathbb {E} \left[ \prod_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} \frac {P _ {M} (s ^ {\prime} \mid s , a)}{P _ {M ^ {\star}} (s ^ {\prime} \mid s , a)} \right] \\ = \mathbb {E} \left[ \prod_ {(s, a) \in \mathcal {D}} \mathbb {E} \left[ \frac {P _ {M} \left(s ^ {\prime} \mid s , a\right)}{P _ {M ^ {\star}} \left(s ^ {\prime} \mid s , a\right)} \mid s, a \right] \right] \\ = \mathbb {E} \left[ \prod_ {(s, a) \in \mathcal {D}} \sum_ {s ^ {\prime}} P _ {M} (s ^ {\prime} \mid s, a) \right] \\ = 1. \tag {9} \\ \end{array}
$$
Then by Markov's inequality, we obtain
$$
\begin{array}{l} \mathbb {P} \left[ \left(\log \ell_ {\mathcal {D}} (M) - \log \ell_ {\mathcal {D}} (M ^ {\star})\right) > \log (1 / \delta) \right] \\ \leq \underbrace {\mathbb {E} \left[ \exp \left(\log \ell_ {\mathcal {D}} (M) - \log \ell_ {\mathcal {D}} (M ^ {\star})\right) \right]} _ {= 1 \text {b y E q . (9)}} \cdot \exp \left[ - \log (1 / \delta) \right] = \delta . \\ \end{array}
$$
Therefore, taking a union bound over  $\mathcal{M}$ , we obtain
$$
\mathbb {P} \left[ (\log \ell_ {\mathcal {D}} (M) - \log \ell_ {\mathcal {D}} (M ^ {\star})) > \log (| \mathcal {M} | / \delta) \right] \leq \delta .
$$
The following lemma shows that, the on-support error of any model  $M \in \mathcal{M}$  can be captured via its log-likelihood (by comparing with the MLE solution).
Lemma 6. For any model  $M$ , we have with probability at least  $1 - \delta$
$$
\mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} \big (P _ {M} (\cdot | s, a), P _ {M ^ {\star}} (\cdot | s, a) \big) ^ {2} \right] \leq \mathcal {O} \left(\frac {\log \ell_ {\mathcal {D}} (M ^ {\star}) - \log \ell_ {\mathcal {D}} (M) + \log (| \mathcal {M} | / \delta)}{n}\right),
$$
where  $\ell_{\mathcal{D}}(\cdot)$  is defined in Eq. (8).
Proof of Lemma 6. By Agarwal et al. (2020, Lemma 25), we have
$$
\mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} \left(P _ {M} (\cdot \mid s, a), P _ {M ^ {\star}} (\cdot \mid s, a)\right) ^ {2} \right] \leq - 2 \log \mathbb {E} _ {\mu \times P _ {M ^ {\star}}} \left[ \exp \left(- \frac {1}{2} \log \left(\frac {P _ {M ^ {\star}} \left(s ^ {\prime} \mid s , a\right)}{P _ {M} \left(s ^ {\prime} \mid s , a\right)}\right)\right) \right], \tag {10}
$$
where  $\mu \times P_{M^*}$  denote the ground truth offline joint distribution of  $(s, a, s')$ .
Let  $\widetilde{\mathcal{D}} = \{(\widetilde{s}_i, \widetilde{a}_i, \widetilde{r}_i, \widetilde{s}_i')\}_{i=1}^n \sim \mu$  be another offline dataset that is independent to  $\mathcal{D}$ . Then,
$$
\begin{array}{l} - n \cdot \log \mathbb {E} _ {\mu \times P _ {M ^ {\star}}} \left[ \exp \left(- \frac {1}{2} \log \left(\frac {P _ {M ^ {\star}} (s ^ {\prime} \mid s , a)}{P _ {M} (s ^ {\prime} \mid s , a)}\right)\right) \right] \\ = - \sum_ {i = 1} ^ {n} \log \mathbb {E} _ {\left(\widetilde {s} _ {i}, \widetilde {a} _ {i}, \widetilde {s} _ {i} ^ {\prime}\right) \sim \mu} \left[ \exp \left(- \frac {1}{2} \log \left(\frac {P _ {M ^ {\star}} \left(\widetilde {s} _ {i} ^ {\prime} \mid \widetilde {s} _ {i} , \widetilde {a} _ {i}\right)}{P _ {M} \left(\widetilde {s} _ {i} ^ {\prime} \mid \widetilde {s} _ {i} , \widetilde {a} _ {i}\right)}\right)\right) \right] \\ = - \log \mathbb {E} _ {\widetilde {\mathcal {D}} \sim \mu} \left[ \exp \left(\sum_ {i = 1} ^ {n} - \frac {1}{2} \log \left(\frac {P _ {M ^ {*}} \left(\widetilde {s} _ {i} ^ {\prime} \mid \widetilde {s} _ {i} , \widetilde {a} _ {i}\right)}{P _ {M} \left(\widetilde {s} _ {i} ^ {\prime} \mid \widetilde {s} _ {i} , \widetilde {a} _ {i}\right)}\right)\right) \Bigg |   \mathcal {D} \right] \\ = - \log \mathbb {E} _ {\widetilde {\mathcal {D}} \sim \mu} \left[ \exp \left(\sum_ {(s, a, s ^ {\prime}) \in \widetilde {\mathcal {D}}} - \frac {1}{2} \log \left(\frac {P _ {M ^ {\star}} \left(s ^ {\prime} \mid s , a\right)}{P _ {M} \left(s ^ {\prime} \mid s , a\right)}\right)\right) \mid \mathcal {D} \right]. \tag {11} \\ \end{array}
$$
We use  $\ell_{M}(s,a,s^{\prime})$  as the shorthand of  $-\frac{1}{2}\log \left(\frac{P_{M^{\star}}(s^{\prime}|s,a)}{P_{M}(s^{\prime}|s,a)}\right)$ , for any  $(s,a,s^{\prime})\in S\times A\times S$ . By Agarwal et al. (2020, Lemma 24) (see also Liu et al., 2022, Lemma 15), we know
$$
\mathbb {E} _ {\mathcal {D} \sim \mu} \left[ \exp \left(\sum_ {(s, a, s ^ {\prime}) \in \mathcal {D}} \ell_ {M} (s, a, s ^ {\prime}) - \log \mathbb {E} _ {\widetilde {\mathcal {D}} \sim \mu} \left[ \exp \left(\sum_ {(s, a, s ^ {\prime}) \in \widetilde {\mathcal {D}}} \ell_ {M} (s, a, s ^ {\prime})\right) \mid \mathcal {D} \right] - \log | \mathcal {M} |\right) \right] \leq 1.
$$
Thus, we can use Chernoff method as well as a union bound on the equation above to obtain the following exponential tail bound: with probability at least  $1 - \delta$ , we have for all  $(P,R) = M \in \mathcal{M}$
$$
- \log \mathbb {E} _ {\widetilde {\mathcal {D}} \sim \mu} \left[ \exp \left(\sum_ {(s, a, s ^ {\prime}) \in \widetilde {\mathcal {D}}} \ell_ {M} (s, a, s ^ {\prime})\right) \mid \mathcal {D} \right] \leq - \sum_ {(s, a, s ^ {\prime}) \in \mathcal {D}} \ell_ {M} (s, a, s ^ {\prime}) + 2 \log (| \mathcal {M} | / \delta). \tag {12}
$$
Plugging back the definition of  $\ell_M$  and combining Eqs. (10) to (12), we obtain
$$
n \cdot \mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} \left(P (\cdot \mid s, a), P _ {M ^ {\star}} (\cdot \mid s, a)\right) ^ {2} \right] \leq \frac {1}{2} \sum_ {(s, a, s ^ {\prime}) \in \mathcal {D}} \log \left(\frac {P _ {M ^ {\star}} (s ^ {\prime} \mid s , a)}{P (s ^ {\prime} \mid s , a)}\right) + 2 \log (| \mathcal {M} | / \delta).
$$
Therefore, we obtain
$$
\begin{array}{l} n \cdot \mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} (P (\cdot | s, a), P _ {M ^ {\star}} (\cdot | s, a)) ^ {2} \right] \\ \lesssim \sum_ {(s, a, s ^ {\prime}) \in \mathcal {D}} \log \left(\frac {P _ {M ^ {\star}} (s ^ {\prime} \mid s , a)}{P (s ^ {\prime} \mid s , a)}\right) + \log (| \mathcal {M} | / \delta) \\ = \log \ell_ {\mathcal {D}} (M ^ {\star}) - \log \ell_ {\mathcal {D}} (M) + \log (| \mathcal {M} | / \delta). \quad (\ell_ {\mathcal {D}} (\cdot) \text {i s d e f i n e d i n E q .} \tag {8} \\ \end{array}
$$
A.3 Guarantees about Model Fitting Loss
Lemma 7. Let  $M^{\star}$  be the ground truth model. Then, with probability at least  $1 - \delta$ , we have
$$
\mathcal {E} _ {\mathcal {D}} (M ^ {\star}) - \min  _ {M \in \mathcal {M}} \mathcal {E} _ {\mathcal {D}} (M) \leq \mathcal {O} \left(\log \left(| \mathcal {M} | / \delta\right)\right),
$$
where  $\mathcal{E}_{\mathcal{D}}$  is defined in Eq. (3).
Proof of Lemma 7. By defition, we know
$$
\mathcal {E} _ {\mathcal {D}} (M) = - \log \ell_ {\mathcal {D}} (M) + \left(R _ {M} (s, a) - r\right) ^ {2} / V _ {\max } ^ {2}
$$
By Lemma 5, we know
$$
\max  _ {M \in \mathcal {M}} \log \ell_ {\mathcal {D}} (M) - \log \ell_ {\mathcal {D}} \left(M ^ {\star}\right) \leq \log \left(\left| \mathcal {M} \right| / \delta\right). \tag {13}
$$
In addition, by Xie et al. (2021a, Theorem A.1) (with setting  $\gamma = 0$ ), we know w.p.  $1 - \delta$
$$
\sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} \left(R ^ {\star} (s, a) - r\right) ^ {2} - \min  _ {M \in \mathcal {M}} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} \left(R _ {M} (s, a) - r\right) ^ {2} \lesssim \log (| \mathcal {M} | / \delta). \tag {14}
$$
Combining Eqs. (13) and (14) and using the fact of  $V_{\mathrm{max}} \geq 1$ , we have w.p.  $1 - \delta$
$$
\begin{array}{l} \mathcal {E} _ {\mathcal {D}} (M ^ {\star}) - \min  _ {M \in \mathcal {M}} \mathcal {E} _ {\mathcal {D}} (M) \\ \leq \max  _ {M \in \mathcal {M}} \log \ell_ {\mathcal {D}} (M) - \min  _ {M \in \mathcal {M}} \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (R _ {M} (s, a) - r) ^ {2} / V _ {\max } ^ {2} + \mathcal {E} _ {\mathcal {D}} (M ^ {\star}) \\ \lesssim \log (| \mathcal {M} | / \delta). \\ \end{array}
$$
Lemma 8. For any  $M \in \mathcal{M}$ , we have with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} \left(P _ {M} (\cdot | s, a), P _ {M ^ {\star}} (\cdot | s, a)\right) ^ {2} + \left(R _ {M} (s, a) - R ^ {\star} (s, a)\right) ^ {2} / V _ {\max } ^ {2} \right] \\ \leq \mathcal {O} \left(\frac {\mathcal {E} _ {\mathcal {D}} (M) - \mathcal {E} _ {\mathcal {D}} (M ^ {\star}) + \log (| \mathcal {M} | / \delta)}{n}\right), \\ \end{array}
$$
Proof of Lemma 8. By Lemma 6, we have w.p.  $1 - \delta$
$$
n \cdot \mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} \left(P _ {M} (\cdot | s, a), P _ {M ^ {\star}} (\cdot | s, a)\right) ^ {2} \right] \lesssim \log \ell_ {\mathcal {D}} (M ^ {\star}) - \log \ell_ {\mathcal {D}} (M) + \log (| \mathcal {M} | / \delta). \tag {15}
$$
Also, we have
$$
\begin{array}{l} n \cdot \mathbb {E} _ {\mu} \left[ \left(R _ {M} (s, a) - R ^ {\star} (s, a)\right) ^ {2} \right] \tag {16} \\ = n \cdot \mathbb {E} _ {\mu} \left[ \left(R _ {M} (s, a) - r\right) ^ {2} \right] - n \cdot \mathbb {E} _ {\mu} \left[ \left(R ^ {\star} (s, a) - r\right) ^ {2} \right] \\ (\text {s e e}, \text {e . g .}, \text {X i e} \text {e t a l . , 2 0 2 1 a , E q . (A . 1 0) w i t h} \gamma = 0) \\ \end{array}
$$
$$
\lesssim \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (R _ {M} (s, a) - r) ^ {2} - \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (R ^ {\star} (s, a) - r) ^ {2} + \log (| \mathcal {M} | / \delta),
$$
where the last inequality is a direct implication of Xie et al. (2021a, Lemma A.4). Combining Eqs. (15) and (16) and using the fact of  $V_{\mathrm{max}} \geq 1$ , we obtain
$$
\begin{array}{l} n \cdot \mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} \left(P _ {M} (\cdot | s, a), P _ {M ^ {\star}} (\cdot | s, a)\right) ^ {2} + \left(R _ {M} (s, a) - R ^ {\star} (s, a)\right) ^ {2} / V _ {\max } ^ {2} \right] \\ \lesssim \log \ell_ {\mathcal {D}} (M ^ {\star}) - \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (R ^ {\star} (s, a) - r) ^ {2} / V _ {\max } ^ {2} - \log \ell_ {\mathcal {D}} (M) + \sum_ {(s, a, r, s ^ {\prime}) \in \mathcal {D}} (R _ {M} (s, a) - r) ^ {2} / V _ {\max } ^ {2} + \log (| \mathcal {M} | / \delta) \\ = \mathcal {E} _ {\mathcal {D}} (M) - \mathcal {E} _ {\mathcal {D}} (M ^ {\star}) + \log (| \mathcal {M} | / \delta). \\ \end{array}
$$
A.4 Proof of Main Theorems
Proof of Theorem 2. By the optimality of  $\widehat{\pi}$  (from Eq. (1)), we have
$$
\begin{array}{l} J (\pi^ {\dagger}) - J (\widehat {\pi}) = J (\pi^ {\dagger}) - J (\pi_ {\mathrm {r e f}}) - \left[ J (\widehat {\pi}) - J (\pi_ {\mathrm {r e f}}) \right] \\ \leq J (\pi^ {\dagger}) - J (\pi_ {\mathrm {r e f}}) - \min  _ {M \in \mathcal {M} _ {\alpha}} [ J _ {M} (\widehat {\pi}) - J _ {M} (\pi_ {\mathrm {r e f}}) ] \quad (\star) \\ \leq J \left(\pi^ {\dagger}\right) - J \left(\pi_ {\text {r e f}}\right) - \min  _ {M \in \mathcal {M} _ {\alpha}} \left[ J _ {M} \left(\pi^ {\dagger}\right) - J _ {M} \left(\pi_ {\text {r e f}}\right) \right], \tag {17} \\ \end{array}
$$
where step  $(\star)$  follows from Lemma 5 so that we have  $M^{\star} \in \mathcal{M}_{\alpha}$ , and the last step is because of  $\pi^{\dagger} \in \Pi$ . By the simulation lemma (Lemma 4), we know for any policy  $\pi$  and any  $M \in \mathcal{M}_{\alpha}$ ,
$$
\begin{array}{l} | J (\pi) - J _ {M} (\pi) | \leq \frac {V _ {\max}}{1 - \gamma} \mathbb {E} _ {d ^ {\pi}} \left[ D _ {\mathrm {T V}} \left(P _ {M} (\cdot \mid s, a), P _ {M ^ {\star}} (\cdot \mid s, a)\right) \right] + \frac {1}{1 - \gamma} \mathbb {E} _ {d ^ {\pi}} \left[ | R _ {M} (s, a) - R ^ {\star} (s, a) | \right] \\ \leq \frac {V _ {\mathrm {m a x}}}{1 - \gamma} \sqrt {\mathbb {E} _ {d ^ {\pi}} \left[ D _ {\mathrm {T V}} \left(P _ {M} (\cdot | s , a) , P _ {M ^ {\star}} (\cdot | s , a)\right) ^ {2} \right]} + \frac {V _ {\mathrm {m a x}}}{1 - \gamma} \sqrt {\mathbb {E} _ {d ^ {\pi}} \left[ (R _ {M} (s , a) - R ^ {\star} (s , a)) ^ {2} / V _ {\mathrm {m a x}} ^ {2} \right]} \\ \lesssim \frac {V _ {\mathrm {m a x}}}{1 - \gamma} \sqrt {\mathbb {E} _ {d ^ {\pi}} \left[ D _ {\mathrm {T V}} (P _ {M} (\cdot | s , a) , P _ {M ^ {\star}} (\cdot | s , a)) ^ {2} + (R _ {M} (s , a) - R ^ {\star} (s , a)) ^ {2} / V _ {\mathrm {m a x}} ^ {2} \right]} \\ (a \lesssim b \text {m e a n s} a \leq \mathcal {O} (b)) \\ \leq \frac {V _ {\mathrm {m a x}} \sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi)}}{1 - \gamma} \sqrt {\mathbb {E} _ {\mu} \left[ D _ {\mathrm {T V}} \left(P _ {M} (\cdot \mid s , a) , P _ {M ^ {\star}} (\cdot \mid s , a)\right) ^ {2} + (R _ {M} (s , a) - R ^ {\star} (s , a)) ^ {2} / V _ {\mathrm {m a x}} ^ {2} \right]} \\ \lesssim \frac {V _ {\max } \sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi)}}{1 - \gamma} \sqrt {\frac {\mathcal {E} _ {\mathcal {D}} (M) - \mathcal {E} _ {\mathcal {D}} (M ^ {\star}) + \log (| \mathcal {M} | / \delta)}{n}} \quad (by Lemma 8) \\ \lesssim \frac {V _ {\max } \sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi)}}{1 - \gamma} \sqrt {\frac {\mathcal {E} _ {\mathcal {D}} (M) - \min  _ {M ^ {\prime} \in \mathcal {M}} \mathcal {E} _ {\mathcal {D}} \left(M ^ {\prime}\right) + \log \left(\left| \mathcal {M} \right| / \delta\right)}{n}} (‡) \\ \lesssim \frac {V _ {\max } \sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi)}}{1 - \gamma} \sqrt {\frac {\log (| \mathcal {M} | / \delta)}{n}} (18) \\ \end{array}
$$
where the step  $(\ddagger)$  follows from the assumption of  $M^{\star} \in \mathcal{M}$ , and last step is because  $\mathcal{E}_{\mathcal{D}}(M) - \min_{M' \in \mathcal{M}} \mathcal{E}_{\mathcal{D}}(M') \leq \alpha = \mathcal{O}(\log (|\mathcal{M}| / \delta))$  by Eq. (2).
Combining Eqs. (17) and (18), we obtain
$$
J (\pi^ {\dagger}) - J (\widehat {\pi}) \lesssim \left[ \sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi^ {\dagger})} + \sqrt {\mathfrak {C} _ {\mathcal {M}} (\pi_ {\mathrm {r e f}})} \right] \cdot \frac {V _ {\max}}{1 - \gamma} \sqrt {\frac {\log (| \mathcal {M} | / , \delta)}{n}}.
$$
Note that, over the proof above, only steps  $(\star)$  and  $(\ddagger)$  have used the realizability assumption of  $M^{\star} \in \mathcal{M}$ . To extend that to the misspecification case, where there only exists an  $\widetilde{M}^{\star} \in \mathcal{M}$  such that  $\widetilde{M}^{\star}$  is close to  $M^{\star}$  up to some misspecification error, we just need the following straightforward accommodations:
(1) A variant of Lemma 5—to ensure that  $\widetilde{M}^{\star}$  is included in the version space  $\mathcal{M}_{\alpha}$ . By doing so, the misspecification error should also be included in the radius of the version space.
(2) Upper bound  $|J(\pi) - J_{\widetilde{M}^{\star}}(\pi)|$  for any  $\pi$  using misspecification error. This is a standard argument, and by combining with the item above,  $(\star)$  becomes  $J(\widehat{\pi}) - J(\pi_{\mathrm{ref}}) \geq \min_{M \in \mathcal{M}_{\alpha}} [J_M(\widehat{\pi}) - J_M(\pi_{\mathrm{ref}})] -$  misspecification error.
(3) Upper bound difference in model-fitting error  $\mathcal{E}_{\mathcal{D}}(\widetilde{M}^{\star}) - \mathcal{E}_{\mathcal{D}}(M^{\star})$  using misspecification error. Then, step  $(\ddagger)$  becomes, for  $M \in \mathcal{M}_{\alpha}$ ,
$$
\begin{array}{l} \mathcal {E} _ {\mathcal {D}} (M) - \mathcal {E} _ {\mathcal {D}} (M ^ {\star}) = \mathcal {E} _ {\mathcal {D}} (M) - \mathcal {E} _ {\mathcal {D}} (\widetilde {M} ^ {\star}) + \mathcal {E} _ {\mathcal {D}} (\widetilde {M} ^ {\star}) - \mathcal {E} _ {\mathcal {D}} (M ^ {\star}) \\ \leq \mathcal {E} _ {\mathcal {D}} (M) - \min  _ {M ^ {\prime} \in \mathcal {M}} \mathcal {E} _ {\mathcal {D}} \left(M ^ {\prime}\right) + \text {m i s s p e c i f i c a t i o n e r r o r} \\ \lesssim \log (| \mathcal {M} | / \delta) + \text {m i s s p e c i f i c a t i o n e r r o r}. \\ \end{array}
$$
Due to the unboundedness of the likelihood, we conjecture that naively defining misspecification error using total variation without any accommodation on the MLE loss may be insufficient for the steps above. To resolve that, we may adopt an alternate misspecification definition, e.g.,  $\left|\log P_{M^{\star}}(s^{\prime}\mid s,a) - \log P_{\widetilde{M}^{\star}}(s^{\prime}\mid s,a)\right|\leq \varepsilon$ $\forall (s,a,s^{\prime})\in S\times \mathcal{A}\times \mathcal{S},$  or add extra smoothing to the MLE loss with regularization.
Proof of Theorem 3.
$$
\begin{array}{l} J \left(\pi_ {\text {r e f}}\right) - J (\widehat {\pi}) = J \left(\pi_ {\text {r e f}}\right) - J \left(\pi_ {\text {r e f}}\right) - \left[ J (\widehat {\pi}) - J \left(\pi_ {\text {r e f}}\right) \right] \\ \leq - \min  _ {M \in \mathcal {M} _ {\alpha}} \left[ J _ {M} (\widehat {\pi}) - J _ {M} \left(\pi_ {\text {r e f}}\right) \right] \quad \text {(b y} M ^ {\star} \in \mathcal {M} _ {\alpha}) \\ = - \max  _ {\pi \in \Pi} \min  _ {M \in \mathcal {M} _ {\alpha}} \left[ J _ {M} (\pi) - J _ {M} \left(\pi_ {\text {r e f}}\right) \right] \quad \left(\text {b y}\right) \\ \leq - \min  _ {M \in \mathcal {M} _ {\alpha}} \left[ J _ {M} \left(\pi_ {\text {r e f}}\right) - J _ {M} \left(\pi_ {\text {r e f}}\right) \right] \quad \left(\pi_ {\text {r e f}} \in \Pi\right) \\ = 0. \\ \end{array}
$$
A misspecified version of Theorem 3 can be derived similarly to what we discussed about that of Theorem 2. If the policy class is also misspecified, where there exists only  $\widetilde{\pi}_{\mathrm{ref}} \in \Pi$  that is close to  $\pi_{\mathrm{ref}}$  up to some misspecification error, the second last step of the proof of Theorem 3 becomes  $-\min_{M \in \mathcal{M}_{\alpha}} [J_M(\widetilde{\pi}_{\mathrm{ref}}) - J_M(\pi_{\mathrm{ref}})] \leq$  misspecification error by simply applying the performance difference lemma on the difference between  $\widetilde{\pi}_{\mathrm{ref}}$  and  $\pi_{\mathrm{ref}}$ .
B Proofs for Section 6
Proof of Lemma 9. We prove the result by contradiction. First notice  $\min_{M\in \mathcal{M}}J_M(\pi^{\prime}) - J_M(\pi^{\prime}) = 0$ . Suppose there is  $\overline{\pi}\in \Pi$  such that  $\min_{M\in \mathcal{M}_{\alpha}}J_M(\bar{\pi}) - J_M(\pi^{\prime}) > 0$ , which implies that  $J_{M}(\bar{\pi}) > J_{M}(\pi^{\prime}),\forall M\in \mathcal{M}_{\alpha}$ . Since  $\mathcal{M}\subseteq \mathcal{M}_{\alpha}$ , we have
$$
\min  _ {M \in \mathcal {M}} J _ {M} (\bar {\pi}) + \psi (M) > \min  _ {M \in \mathcal {M}} J _ {M} (\pi^ {\prime}) + \psi (M) = \max  _ {\pi \in \Pi} \min  _ {M \in \mathcal {M}} J _ {M} (\pi) + \psi (M)
$$
which is a contradiction of the maximin optimality. Thus  $\max_{\pi \in \Pi} \min_{M \in \mathcal{M}_{\alpha}} J_M(\bar{\pi}) - J_M(\pi') = 0$ , which means  $\pi'$  is a solution.
For the converse statement, suppose  $\pi$  is a fixed point. We can just let  $\psi(M) = -J_M(\pi)$ . Then this pair of  $\pi$  and  $\psi$  by definition of the fixed point satisfies Eq. (19).
C Related Work
There has been an extensive line of works on reinforcement with offline/batch data, especially for the case with the data distribution is rich enough to capture the state-action distribution for any given policy (Munos, 2003; Antos et al., 2008; Munos and Szepesvári, 2008; Farahmand et al., 2010; Lange et al., 2012; Chen and Jiang, 2019; Liu et al., 2020a; Xie and Jiang, 2020, 2021). However, this assumption is not practical since the data distribution is typically restricted by factors such as the quality of available policies, safety concerns, and existing system constraints, leading to narrower coverage. As a result, recent offline RL works in both theoretical and empirical literature have focused on systematically addressing datasets with inadequate coverage.
Modern offline reinforcement learning approaches can be broadly categorized into two groups for the purpose of learning with partial coverage. The first type of approaches rely on behavior regularization, where the learned policy is encouraged to be close to the behavior policy in states where there is insufficient data (e.g., Fujimoto et al., 2018; Laroche et al., 2019; Kumar et al., 2019; Siegel et al., 2020). These algorithms ensure that the learned policy performs at least as well as the behavior policy while striving to improve it when possible, providing a form of safe policy improvement guarantees. These and other studies (Wu et al., 2019; Fujimoto and Gu, 2021; Kostrikov et al., 2021) have provided compelling empirical evidence for the benefits of these approaches.
The second category of approaches that has gained prevalence relies on the concept of pessimism under uncertainty to construct lower-bounds on policy performance without explicitly constraining
the policy. Recently, there have been several model-free and model-based algorithms based on this concept that have shown great empirical performance on high dimensional continuous control tasks. Model-free approaches operate by constructing lower bounds on policy performance and then optimizing the policy with respect to this lower bound (Kumar et al., 2020; Kostrikov et al., 2021). The model-based counterparts first learn a world model and the optimize a policy using model-based rollouts via off-the-shelf algorithms such as Natural Policy Gradient (Kakade, 2001) or Soft-Actor Critic (Haarnoja et al., 2018). Pessimism is introduced by either terminating model rollouts using uncertainty estimation from an ensemble of neural network models (Kidambi et al., 2020) or modifying the reward function to penalize visiting uncertain regions (Yu et al., 2020). Yu et al. (2021) propose a hybrid model-based and model-free approach that integrates model-based rollouts into a model-free algorithm to construct tighter lower bounds on policy performance. On the more theoretical side, the offline RL approaches built upon the pessimistic concept (e.g., Liu et al., 2020b; Jin et al., 2021; Rashidinejad et al., 2021; Xie et al., 2021a; Zanette et al., 2021; Uehara and Sun, 2021; Shi et al., 2022) also illustrate desired theoretical efficacy under various of setups.