SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the finite function class  $\mathcal{F}$  and an uninformative prior  $p_{0,h}(f_h) = 1 / |\mathcal{F}_h|$ , under a stronger version of Assumption 2.2, i.e.,  $\nu_h = 0$ ,  $\forall h$ , we have  $d_0(\epsilon) \leq \sum_{h=1}^{H} \ln |\mathcal{F}_h| = \ln |\mathcal{F}|$ . For a parametric model, where each  $f_h = f_h^\theta$  is represented by a  $d$ -dimensional parameter  $\theta \in \Omega_h^\theta \subset \mathbb{R}^d$ , a prior over  $\Omega^\theta$  induces a prior over  $\mathcal{F}$ . If each  $\Omega_h^\theta$  is compact, we can generally assume the prior that satisfies  $\sup_\theta \ln \frac{1}{p_{0,h}(\theta' : \| \theta - \theta' \| \leq \epsilon)} \leq d \ln (c_0 / \epsilon)$  for some constant  $c_0$ . If  $f_h = f_h^\theta$  is Lipschitz in  $\theta$ , we can assume that  $\sup_\theta \ln \frac{1}{p_{0,h}(\theta' : \| \theta - \theta' \| \leq \epsilon)} \leq c_1 d \ln (c_2 / \epsilon)$  for some constants  $c_1, c_2$ . Overall, we can assume that  $d_0(\epsilon) \leq c_1 H d \ln (c_2 / \epsilon)$ . A similar discussion can be found in Dann et al. [2021].
Next, we present concrete instances of PS-based, RO-based, and VS-based algorithms. The RO-based and VS-based algorithms presented here are slight refinements of their original versions in Xie et al. [2021]. The PS-based algorithm is novel. All three algorithms resemble the actor-critic style update, inspired by Zanette et al. [2021]. We refer to this generic framework as GOPO (Generic Offline Policy Optimization) presented in Algorithm 1. At each round  $t$ , a critic estimates the value  $\underline{Q}_h^t$  of the actor (i.e., policy  $\pi^t$ ) using the procedure
Algorithm 1 GOPO(D,F,η,T,CriticCompute): Generic Offline Policy Optimization Framework
Input: Offline data  $\mathcal{D}$ , function class  $\mathcal{F}$ , learning rate  $\eta > 0$ , and iteration number  $T$
1: Uniform policy  $\pi^1 = \{\pi_h^1\}_{h\in [H]}$
2: for  $t = 1, \dots, T$  do
3:  $\underline{Q}^t = \boxed{\mathrm{CriticCompute}} (\pi^t, \mathcal{D}, \mathcal{F}, \ldots)$
4:  $\pi_h^{t + 1}(a|s)\propto \pi_h^t (a|s)\exp (\eta Q_h^t (s,a)),\forall (s,a,h)$
5: end for
Output:  $\hat{\pi} \sim \mathrm{Uniform}(\{\pi^t\}_{t \in [T]})$
CriticCompute on Line 3, and the actor improves the policy using a multiplicative weights update [Arora et al., 2012] (Line 4). After  $T$  iterations, GOPO returns a policy  $\hat{\pi}$  that is sampled uniformly from the set of the obtained policies  $\{\pi^t\}_{t\in [T]}$ .
To incorporate the pessimism principle, a critic should generate pessimistic estimates of the value of the actor  $\pi^t$  in Line 3. This is where the three approaches differ - each invokes a different method to compute the critic. Here, we provide a detailed description of the critic module for each approach. To aid the presentation, we introduce the total temporal difference (TD) loss  $\hat{L}_{\tilde{\pi}}$ , defined as  $\hat{L}_{\tilde{\pi}}(f_h,f_{h + 1})\coloneqq \sum_{k = 1}^{K}l_{\tilde{\pi}}(f_h,f_{h + 1};z_h^k)$ , where  $z_{h}\coloneqq (s_{h},a_{h},r_{h},s_{h + 1})$ ,  $z_{h}^{k}\coloneqq (s_{h}^{k},a_{h}^{k},r_{h}^{k},s_{h + 1}^{k})$ , and  $l_{\tilde{\pi}}(f_h,f_{h + 1};z_h)\coloneqq (f_h(s_h,a_h) - r_h - f_{h + 1}(s_{h + 1},\tilde{\pi}))^2$ .
Version Space-based Critic (VSC) (Algorithm 2). Given the actor  $\pi^t$ , at each step  $h \in [H]$ , VSC directly maintains a local regression constraint using the offline data:  $\hat{L}_{\pi^t}(f_h, f_{h+1}) \leq \inf_{g \in \mathcal{F}} \hat{L}_{\pi^t}(g_h, f_{h+1}) + \beta$ , where  $\beta$  is a confidence parameter and  $\hat{L}_{\pi^t}(\cdot, \cdot)$  is serving as a proxy to the squared Bellman residual at step  $h$ . By taking the function that minimizes the initial value, VSC then finds the most pessimistic value function  $\underline{Q^t}$  from the version space  $\mathcal{F}(\beta; \pi^t) \subseteq \mathcal{F}$ . In general, the constrained optimization in Line 2 is computationally intractable. Note that a minimax variant of GOPO+VSC first appeared in Xie et al. [2021], where they directly perform an (intractable) search over the policy space, instead of using the multiplicative weights algorithm (Line 4) of Algorithm 1.
Regularized Optimization-based Critic (ROC) (Algorithm 3). Instead of solving the global constrained optimization in VSC, ROC solves  $\arg \inf_{f\in \mathcal{F}}\{\lambda f_1(s_1,\pi_1^t) + \mathcal{L}_{\pi^t}(f)\}$ , where  $\lambda$  is a regularization parameter and  $\mathcal{L}_{\pi^t}(f)$ , defined in Line 1 of Algorithm 3. Note that in ROC, pessimism is implicitly encouraged through the regularization term  $\lambda f_1(s_1,\pi^t)$ . We remark that, unlike VSC, ROC admits tractable approximations that use adversarial training and work competitively in practice [Cheng et al., 2022]. Note that a discounted variant of GOPO-ROC first appears in [Xie et al., 2021] in discounted MDPs.
Algorithm 2 VSC(D, F,  $\pi^t, \beta$ ): Version Space-based Critic
$$
\begin{array}{l} 1: \mathcal {F} (\beta ; \pi^ {t}) := \{f \in \mathcal {F}: \hat {L} _ {\pi^ {t}} \left(f _ {h}, f _ {h + 1}\right) \leq \\ \left. \inf  _ {g \in \mathcal {F}} \hat {L} _ {\pi^ {t}} \left(g _ {h}, f _ {h + 1}\right) + \beta , \forall h \in [ H ] \right\} \\ 2: \underline {{Q}} ^ {t} \in \arg \min  _ {f \in \mathcal {F} (\beta ; \pi^ {t})} f _ {1} (s _ {1}, \pi^ {t}) \\ \end{array}
$$
Output:  $\underline{Q}^t$
Algorithm 3 ROC  $(\mathcal{D},\mathcal{F},\pi^t,\lambda)$  Regularized Optimization-based Critic
$$
\begin{array}{l} \overline {{1 : \mathcal {L} _ {\pi^ {t}} (f) := \sum_ {h = 1} ^ {H} \hat {L} _ {\pi^ {t}} \left(f _ {h} , f _ {h + 1}\right)}} \\ - \inf  _ {g \in \mathcal {F}} \sum_ {h = 1} ^ {H} \hat {L} _ {\pi^ {t}} \left(g _ {h}, f _ {h + 1}\right) \\ \end{array}
$$
$$
2: \underline {{Q}} ^ {t} \leftarrow \arg \inf  _ {f \in \mathcal {F}} \left\{\lambda f _ {1} \left(s _ {1}, \pi^ {t}\right) + \mathcal {L} _ {\pi^ {t}} (f) \right\}
$$
Posterior sampling-based critic (PSC) in Algorithm 4. Instead of solving a regularized minimax optimization, PSC samples the value function  $Q_h^t$  from the data posterior  $\hat{\pi}(f|\mathcal{D},\pi^t) \propto \tilde{p}_0(f) \cdot p(\mathcal{D}|f,\pi^t)$ , where  $\tilde{p}_0(f)$  is the prior over  $\mathcal{F}$  and  $p(\mathcal{D}|f,\pi^t)$  is the likelihood function of the offline data  $\mathcal{D}$ . To formulate the likelihood function  $p(\mathcal{D}|f,\pi^t)$ , we make use of the squared TD error  $\hat{L}_{\pi^t}(\cdot ,\cdot)$  and normalization method in [Dann et al., 2021] to construct an unbiased proxy of the squared Bellman errors. In particular,  $p(\mathcal{D}|f,\pi^t) = \prod_{h\in [H]}\frac{\exp(-\gamma\hat{L}_{\pi^t}(f_h,f_{h + 1}))}{\mathbb{E}_{f_h'}\sim p_{0,h}}\frac{\exp(-\gamma\hat{L}_{\pi^t}(f_h',f_{h + 1}))}{\exp(-\gamma\hat{L}_{\pi^t}(f_h',f_{h + 1}))}$ , where  $\gamma$  is a learning rate and  $p_0$  is an (unregularized) prior over  $\mathcal{F}$ . A value function sampled from the posterior with this likelihood function is encouraged to have small squared TD errors. The key ingredient in our algorithmic design is the "pessimistic" prior  $\tilde{p}_0(f) = \exp (-\lambda f_1(s_1,\pi_1))p_0(f)$  where we add a new regularization term  $\exp (-\lambda f_1(s_1,\pi_1))$ , with  $\lambda$  being a regularization parameter - which is inspired by the optimistic prior in the online setting [Zhang, 2022, Dann et al., 2021]. This pessimistic prior encourages the value function sampled from the posterior to have a small value in the initial state, implicitly enforcing pessimism. We remark that PSC requires a sampling oracle and expectation oracle (to compute the normalization term in the posterior distribution), which could be amenable to tractable approximations, including replacing expectation oracle with a sampling oracle [Agarwal and Zhang, 2022] while the sampling oracle can be implemented via first-order sampling methods [Welling and Teh, 2011] or ensemble methods [Osband et al., 2016].
Algorithm 4 PSC  $(\mathcal{D},\mathcal{F},\pi^t,\lambda ,\gamma ,p_0)$  : Posterior Sampling-based Critic
$$
1: \underline {{Q}} ^ {t} \sim \hat {p} (f | \mathcal {D}, \pi^ {t}) \propto \exp (- \lambda f _ {1} (s _ {1}, \pi^ {t})) p _ {0} (f) \prod_ {h \in [ H ]} \frac {\exp (- \gamma \hat {L} _ {\pi^ {t}} (f _ {h} , f _ {h + 1}))}{\mathbb {E} _ {f _ {h} ^ {\prime} \sim p _ {0 , h}} \exp (- \gamma \hat {L} _ {\pi^ {t}} (f _ {h} ^ {\prime} , f _ {h + 1}))}
$$
Output:  $Q^t$
In this section, we shall present the upper bounds of the sub-optimality of the policies executed by GOPO-VSC, GOPO-ROC, and GOPO-PSC. Our upper bounds are expressed in terms of a new notion of data diversity.
4.1 Data diversity
We now introduce the key notion of data diversity for offline RL. Since the offline learner does not have direct access to the trajectory of a comparator policy  $\pi \in \Pi^{all}$ , they can only observe partial information about the goodness of  $\pi$  channeled through the "transferability" with the behavior policy  $\mu$ . The transferability from  $\mu$  to  $\pi$  depends on how diverse the offline data induced by  $\mu$  can be in supporting the extrapolation to  $\pi$ . Many prior works require uniform diversity where  $\mu$  covers all feasible scenarios of all comparator policies  $\pi$ . The data diversity can be essentially captured by how well the Bellman error under the state-action distribution induced by  $\mu$  can predict the counterpart quantity under the state-action distribution induced by  $\pi$ . Our notion of data diversity, which is inspired by the notion of task diversity in transfer learning literature [Tripuraneni et al., 2020, Watkins et al., 2023], essentially encodes the ratio of some proxies of expected Bellman errors induced by  $\mu$  and  $\pi$ , and is defined as follows.
Definition 3. For any comparator policy  $\pi \in \Pi^{all}$ , we measure the data diversity of the behavior policy  $\mu$  with respect to a target policy  $\pi$  by
$$
\mathcal {C} (\pi ; \epsilon) := \max  _ {h \in [ H ]} \chi_ {\left(\mathcal {F} _ {h} - \mathcal {F} _ {h}\right)} \left(\epsilon ; d _ {h} ^ {\pi}, d _ {h} ^ {\mu}\right), \forall \epsilon \geq 0, \tag {2}
$$
where  $\mathcal{F}_h - \mathcal{F}_h$  is the Minkowski difference between the function class  $\mathcal{F}_h$  and itself, i.e.,  $\mathcal{F}_h - \mathcal{F}_h \coloneqq \{f_h - f_h': f_h, f_h' \in \mathcal{F}\}$ , and  $\chi_{\mathcal{Q}}(\epsilon; q, p)$  is the discrepancy between distributions  $q$  and  $p$  under the witness of function class  $\mathcal{Q}$  defined as
$$
\chi_ {\mathcal {Q}} (\epsilon ; q, p) = \inf  \left\{C \geq 0: \left(\mathbb {E} _ {q} [ g ]\right) ^ {2} \leq C \cdot \mathbb {E} _ {p} [ g ^ {2} ] + \epsilon , \forall g \in \mathcal {Q} \right\}
$$
with  $\mathcal{Q}$  being a function class and  $p$  and  $q$  being two distributions over the same domain.
Up to a small additive error  $\epsilon$ , a finite  $\mathcal{C}(\pi; \epsilon)$  ensures that a proxy of the Bellman error under the  $\pi$ -induced state-action distribution is controlled by that under the  $\mu$ -induced state-action distribution.
Despite the abstraction in the definition of this data diversity, it is always upper bounded by the single-policy concentrability coefficient [Liu et al., 2019, Rashidinejad et al., 2021] and the relative condition number [Agarwal et al., 2021, Uehara et al., 2022b, Uehara and Sun, 2022] that are both commonly used in many prior offline RL works. We further discuss our data diversity measure in more detail in Section 4.2.
4.2 Offline learning guarantees
We now utilize data diversity to give learning guarantees of the considered algorithms for extrapolation to an arbitrary comparator policy  $\pi \in \Pi^{all}$ . To aid the representation, in all of the following theorems we are about to present, we shall set  $\eta = \sqrt{\frac{\ln\operatorname{Vol}(\mathcal{A})}{4(e - 2)b^2T}}$  in Algorithm 1, where  $\operatorname{Vol}(\mathcal{A})$  is the volume of the action set  $\mathcal{A}$  (e.g.,  $\operatorname{Vol}(\mathcal{A}) = |\mathcal{A}|$  for finite  $\mathcal{A}$ ), and define, for simplicity, the misspecification errors  $\zeta_{msp} \coloneqq K\sum_{h=1}^{H}\left(\nu_h^2 + b\nu_h\right)$ ,  $\tilde{\zeta}_{msp} \coloneqq \zeta_{msp} + bK\sum_{h=1}^{H}\xi_h$ ,  $\bar{\nu} \coloneqq \sum_{h=1}^{H}\nu_h$ , the optimization error  $\zeta_{opt} \coloneqq Hb\sqrt{T^{-1}\ln\operatorname{Vol}(\mathcal{A})}$ , and the complexity measures  $\tilde{d}_{opt}(\epsilon,T) \coloneqq \max\{d_{\mathcal{F}}(\epsilon), d_{\Pi}(\epsilon,T)\}$ , and  $\tilde{d}_{ps}(\epsilon,T) \coloneqq \max\{d_{\mathcal{F}}(\epsilon), d_{\Pi}(\epsilon,T), \frac{d_0(\epsilon)}{\gamma Hb^2}, \frac{d_0'(\epsilon)}{\gamma Hb^2}\}$ .
Theorem 1 (Guarantees for GOPO-VSC). Let  $\hat{\pi}^{vs}$  be the output of Algorithm 1 invoked with  $\boxed{CriticCompute}$  being  $VSC(\mathcal{D},\mathcal{F},\pi^t,\beta)$  (Algorithm 2) with  $\beta = \mathcal{O}(Hb^2\max \{\tilde{d}_{opt}(\epsilon ,T),\ln (H / \delta)\} +b^2 K\epsilon +bK\max_{h\in [H]}\xi_h)$ . Fix any  $\delta \in (0,1]$ . Under Assumption 2.1-2.2, with probability at least  $1 - 2\delta$  (over the randomness of the offline data), for any  $\epsilon ,\epsilon_c,\lambda >0$ , and any  $\pi \in \Pi^{all}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {S u b O p t} _ {\pi} \left(\hat {\pi} ^ {v s}\right) | \mathcal {D} \right] \lesssim \frac {H b ^ {2} \cdot \max  \left\{\tilde {d} _ {o p t} (\epsilon , T) , \ln (H / \delta) \right\} + b ^ {2} K H \epsilon + \tilde {\zeta} _ {m s p}}{\lambda} + \frac {\lambda H \cdot \mathcal {C} \left(\pi ; \epsilon_ {c}\right)}{2 K} \\ + H \epsilon_ {c} + \xi_ {1} + \bar {\nu} + \zeta_ {o p t}. \\ \end{array}
$$
Theorem 2 (Guarantees for GOPO-ROC). Let  $\hat{\pi}^{ro}$  be the output of Algorithm 1 invoked with  $\boxed{CriticCompute}$  being  $ROC(\mathcal{D},\mathcal{F},\pi^t,\lambda)$  (Algorithm 3). Fix any  $\delta \in (0,1]$ . Under Assumption 2.1-2.2, with probability at least  $1 - 2\delta$  (over the randomness of the offline data), for any  $\epsilon ,\epsilon_c,\lambda >0$ , and any  $\pi \in \Pi^{all}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {S u b O p t} _ {\pi} \left(\hat {\pi} ^ {r o}\right) | \mathcal {D} \right] \lesssim \frac {H b ^ {2} \cdot \max  \left\{\tilde {d} _ {o p t} (\epsilon , T) , \ln \frac {H}{\delta} \right\} + b ^ {2} K H \epsilon + \tilde {\zeta} _ {m s p}}{\lambda} + \frac {\lambda H \cdot \mathcal {C} \left(\pi ; \epsilon_ {c}\right)}{2 K} \\ + H \epsilon_ {c} + \xi_ {1} + \bar {\nu} + \zeta_ {o p t}. \\ \end{array}
$$
Theorem 3 (Guarantees for GOPO-PSC). Let  $\hat{\pi}^{ps}$  be the output of Algorithm 1 invoked with  $\boxed{CriticCompute}$  being  $PSC(\mathcal{D},\mathcal{F},\pi^t,\lambda ,\gamma ,p_0)$  (Algorithm 4). Under Assumption 2.2, for any  $\gamma \in [0,\frac{1}{144(e - 2)b^2} ]$  and  $\epsilon ,\epsilon_c,\delta ,\lambda >0$  and any  $\pi \in \Pi^{all}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathrm {S u b O p t} _ {\pi} (\hat {\pi} ^ {p s}) \right] \lesssim \frac {\gamma H b ^ {2} \cdot \max \{\tilde {d} _ {p s} (\epsilon , T) , \ln \frac {\ln K b ^ {2}}{\delta} \} + \gamma b ^ {2} K H \cdot \max \{\epsilon , \delta \} + \gamma \zeta_ {m s p}}{\lambda} \\ + \frac {\lambda H \cdot \mathcal {C} (\pi ; \epsilon_ {c})}{K \gamma} + H \epsilon_ {c} + \epsilon + \bar {\nu} + \zeta_ {o p t}. \\ \end{array}
$$
Our results provide a family of upper bounds on the sub-optimality of each of  $\{\hat{\pi}^{vs},\hat{\pi}^{ro},\hat{\pi}^{ps}\}$ , indexed by our choices of the comparator  $\pi$  with the data diversity  $\mathcal{C}(\pi ;\epsilon_c)$ , additive (extrapolation) error  $\epsilon_{c}$ , the discretization level  $\epsilon$  in log-covering numbers, the "failure" probability  $\delta$ , and other algorithm-dependent parameters ( $\lambda$  for  $\hat{\pi}^{ro}$  and  $(\lambda ,\gamma)$  for  $\hat{\pi}^{ps}$ ). Note that the optimization error  $\zeta_{opt}$  captures the error rate of the actor and can be made arbitrarily small with large iteration number  $T$  whereas  $\zeta_{msp},\tilde{\zeta}_{msp},\bar{\nu}$ , and  $\xi_{1}$  are simply misspecification errors aggregated over all stages. Also note that our bound does not scale with the complexity of the comparator policy class  $\Pi^{all}$ . We next highlight the key characteristics of our main results in comparison with existing work.
(I) Tight characterization of data diversity. Our bounds in all the above theorems are expressed in terms of  $\mathcal{C}(\pi; \epsilon_c)$ . Several remarks are in order. First,  $\mathcal{C}(\pi; \epsilon_c)$  is a non-increasing function of  $\epsilon_c$ ; thus  $\mathcal{C}(\pi; \epsilon_c)$  is always smaller or at least equal to  $\mathcal{C}(\pi; 0)$ . In fact, it is possible that  $\mathcal{C}(\pi; 0) = \infty$  yet  $\mathcal{C}(\pi; \epsilon_c) < \infty$  for some  $\epsilon > 0$ . For instance, if there exists  $g \in \mathcal{Q}$  such that  $g(x) = 0, \forall x \in \operatorname{supp}(p)$  and  $\{x : g(x) \neq 0\}$  has a positive measure under  $q$ , then  $\chi_{\mathcal{Q}}(0; q, p) = \infty$  while  $\chi_{\mathcal{Q}}(\sup_{g \in \mathcal{Q}} \mathbb{E}_q[g]^2; q, p) = 0$ . Second,  $\mathcal{C}(\pi; 0)$  is always bounded from above by (often substantially smaller than) the single-policy concentrability coefficient between the  $\pi$ -induced and  $\mu$ -induced state-action distribution [Liu et al., 2019, Rashidinejad et al., 2021], which been used extensively in recent offline RL works [Yin and Wang, 2021, Nguyen-Tang et al., 2022a, 2023, Jin et al., 2022, Zhan et al., 2022, Nguyen-Tang and Arora, 2023, Zhao et al., 2023]. This is essentially because  $d^{\pi}$  can cover the region that is not covered by  $d^{\mu}$  but still the integration of functions in  $\mathcal{F}_h - \mathcal{F}_h$  over two distributions are close to each other. Third,  $\mathcal{C}(\pi; 0)$  is always upper bounded by the relative condition numbers used in [Agarwal et al., 2021,
Uehara et al., 2022b, Uehara and Sun, 2022]. Our data diversity at  $\epsilon = 0$  is similar to the notion of distribution mismatch in Duan et al. [2020], Ji et al. [2022], though our notion is motivated by transfer learning and discovered naturally from our decoupling argument. Our data diversity measure at  $\epsilon = 0$  is smaller than the Bellman residual ratio measure used in Xie et al. [2021] (follows using Jensen's inequality). Finally, the concurrent work of Di et al. [2023] proposed a notion of  $D^2$ -divergence to capture the data disparity of a data point to the offline data. Our data diversity is in general less restricted as we only need to ensure the diversity between two data distributions (of the target policy and the behavior policy), not necessarily between each of their individual data points.
In summary,  $\mathcal{C}(\pi ;\epsilon_c)$ , to the best of our knowledge, provides the tightest characterization of distribution mismatch compared to the prior data coverage notions. We sketch the relationships of the discussed notions in Figure 1, where with our data diversity notion, we show that the scenarios for the offline data in which offline RL is learnable are enlarged compared to the picture depicted by the prior data coverage notions.
(II) Competing with all comparator policies simultaneously. Similar to some recent results in offline RL, our offline RL algorithms compete with all comparator policies that are supported by offline data in some sense. In particular, the choice of the comparator  $\pi$  provides the flexibility to automatically compete with the best policy within a certain diversity level of our choice. For instance, if we want to limit the level  $\mathcal{C}(\pi; \epsilon_c) \leq C$  for some arbitrary  $C > 0$ , our bound automatically competes with  $\pi = \arg \max_{\pi \in \Pi^{all}} \{V_1^\pi(s): \mathcal{C}(\pi; \epsilon_c) \leq C\}$ . This is immensely meaningful since the offline data might not support extrapolation to an optimal policy in practice.
(III) State-of-the-art bounds for standard assumptions. We compare our bounds with other recent guarantees of similar assumptions. To ease comparison, we assume for simplicity, that there is no misspecification, i.e.,  $\nu_{h} = \xi_{h} = 0, \forall h \in [H]$ , and  $T \geq K \ln \operatorname{Vol}(\mathcal{A})$ , and we minimize the bounds in Theorem 2 and Theorem 3 with respect to  $\lambda$ . The three theorems can then be simplified into a unified result presented in Proposition 1.
Proposition 1 (A unified guarantee for VS, RO and PS). Under Assumption 2.1-2.2 with no misspecification, i.e.,  $\nu_{h} = \xi_{h} = 0, \forall h \in [H]$ ,  $\forall \hat{\pi} \in \{\hat{\pi}^{vs}, \hat{\pi}^{ro}, \hat{\pi}^{ps}\}$ ,  $\mathbb{E}[\mathrm{SubOpt}_{\pi}(\hat{\pi})] = \tilde{\mathcal{O}}(Hb\sqrt{\tilde{d}(1/K,T)} \cdot \mathcal{C}(\pi;1/\sqrt{K})/K + \xi_{opt})$ , where  $\tilde{d}(1/K,T) = \tilde{d}_{opt}(1/K,T)$  if  $\hat{\pi} \in \{\hat{\pi}^{vs}, \hat{\pi}^{ro}\}$  and  $\tilde{d}(1/K,T) = \tilde{d}_{ps}(1/K,T)$  if  $\hat{\pi} = \hat{\pi}^{ps}$ . In addition,
Figure 1: The relations of sample-efficient offline RL classes under different data coverage measures. Given the same MDP and a target policy (e.g., an optimal policy of the MDP), each data coverage measure induces a corresponding set of behavior policies (represented by the rectangle labelled by the data coverage measure) from which the target policy is offline-learnable.
<sup>6</sup>Recent primal-dual methods achieve favorable guarantees for offline RL. However, these guarantees are not directly comparable to the guarantees of our value-based methods due to a different set of assumptions. Nonetheless, we make a detailed discussion in Section A.2.
- If  $\mathcal{F}_h$  and  $\Pi_h^{soft}(T)$  have finite elements for all  $h \in [H]$ ,  $\tilde{d}(1/K, T) = \mathcal{O}(\max_{h \in [H]} \max\{\ln |\mathcal{F}_h|, \ln |\Pi_h^{soft}(T)|\})$ ;
- If  $\mathcal{F}_h = \{(s, a) \mapsto \langle \phi_h(s, a), w \rangle : \|w\|_2 \leq b\}$  is a linear model, where  $\phi_h: \mathcal{S} \times \mathcal{A} \to \mathbb{R}^d$  is a known feature map and w.l.o.g.  $\max_h \| \phi_h\|_{\infty} \leq 1$ ,  $\tilde{d}(1/K, T) = \mathcal{O}(d \log(1 + KTb)), \forall T$ .
Proposition 1 essentially asserts that VS-based, RO-based, and PS-based algorithms obtain comparable guarantees for offline RL in the realizable case. We now compare our results to related work in various instantiation of function classes.