SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thanh Nguyen-Tang
nguyent@cs.jhu.edu
Raman Arora
arora@cs.jhu.edu
We seek to understand what facilitates sample-efficient learning from historical datasets for sequential decision-making, a problem that is popularly known as offline reinforcement learning (RL). Further, we are interested in algorithms that enjoy sample efficiency while leveraging (value) function approximation. In this paper, we address these fundamental questions by (i) proposing a notion of data diversity that subsumes the previous notions of coverage measures in offline RL and (ii) using this notion to unify three distinct classes of offline RL algorithms based on version spaces (VS), regularized optimization (RO), and posterior sampling (PS). We establish that VS-based, RO-based, and PS-based algorithms, under standard assumptions, achieve comparable sample efficiency, which recovers the state-of-the-art sub-optimality bounds for finite and linear model classes with the standard assumptions. This result is surprising, given that the prior work suggested an unfavorable sample complexity of the RO-based algorithm compared to the VS-based algorithm, whereas posterior sampling is rarely considered in offline RL due to its explorative nature. Notably, our proposed model-free PS-based algorithm for offline RL is novel, with sub-optimality bounds that are frequentist (i.e., worst-case) in nature.
Learning from previously collected experiences is a vital capability for reinforcement learning (RL) agents, offering a broader scope of applications compared to online RL. This is particularly significant in domains where interacting with the environment poses risks or high costs. However, effectively extracting valuable policies from historical datasets remains a considerable challenge, especially in high-dimensional spaces where the ability to generalize across various scenarios is crucial. In this paper, our objective is to comprehensively examine the efficiency of offline RL in the context of (value) function approximation. We aim to analyze this within the broader framework of general data collection settings.
The problem of learning from historical datasets for sequential decision-making, commonly known as offline RL or batch RL, originated in the early 2000s [Ernst et al., 2005, Antos et al., 2006, Lange et al., 2012] and has recently regained significant attention [Levine et al., 2020, Uehara et al., 2022a]. In offline RL, where direct interaction with environments is not possible, our goal is to learn an effective policy by leveraging pre-collected datasets, typically obtained from different policies known as behavior policies. The sample efficiency of an offline RL algorithm is measured by the sub-optimality of the policies it executes compared to a "good" comparator policy, which may or may not be an optimal policy. Due to the lack of exploration inherent in offline RL, designing an algorithm with low sub-optimality requires employing the fundamental principle of pessimistic extrapolation. This means that the agent extrapolates from the offline data while considering the worst-case scenarios
that are consistent with that data. Essentially, the diversity present in the offline data determines the agent's ability to construct meaningful extrapolations. Hence, a suitable notion of data diversity plays a crucial role in offline RL.
To address the issue of data diversity, several prior methods have made the assumption that the offline data is uniformly diverse – this implies that the data should cover the entire trajectory space with some probability that is bounded from below [Munos and Szepesvári, 2008, Chen and Jiang, 2019, Nguyen-Tang et al., 2022b]. This assumption is often too strong and not feasible in many practical scenarios. In more recent approaches [Jin et al., 2021b, Xie et al., 2021, Uehara and Sun, 2022, Chen and Jiang, 2022, Rashidinejad et al., 2023], the stringent assumption of uniform diversity has been relaxed to only require partial diversity in the offline data. Various measures have been proposed to capture this partial diversity, such as single-policy concentrability coefficients [Liu et al., 2019, Rashidinejad et al., 2021, Yin and Wang, 2021], relative condition numbers [Agarwal et al., 2021, Uehara and Sun, 2022], and Bellman residual ratios [Xie et al., 2021]. These measures aim to quantify the extent to which the data captures diverse states and behaviors. However, it should be noted that in some practical scenarios, these measures may become excessive or may not hold at all.
In terms of algorithmic approaches, existing sample-efficient offline RL algorithms explicitly construct pessimistic estimates of models or value functions to effectively learn from datasets with partial diversity. This is typically achieved through the construction of lower confidence bounds (LCBs) [Jin et al., 2021b, Rashidinejad et al., 2021] or version spaces (VS) [Xie et al., 2021, Zanette et al., 2021]. LCB-based algorithms incorporate a bonus term subtracted from the value estimates to enforce pessimism across all state-action pairs and stages. However, it has been observed that LCB-based algorithms tend to impose unnecessarily aggressive pessimism, leading to sub-optimal bounds [Zanette et al., 2021]. On the other hand, VS-based algorithms search through the space of consistent hypotheses to identify the one with the smallest value in the initial states. These algorithms have demonstrated state-of-the-art bounds [Zanette et al., 2021, Xie et al., 2021].
In contrast to LCB-based and VS-based algorithms, regularized (minimax) optimization (RO) and posterior sampling (PS) are more amenable to tractable implementations but are relatively new in the offline RL literature. The RO-based algorithm initially introduced by [Xie et al., 2021, Algorithm 1] incorporates pessimism implicitly through a regularization term that promotes pessimism in the initial state. This approach eliminates the need for an intractable search over the version space. However, Xie et al. [2021] demonstrate that the RO-based algorithm exhibits a significantly slower sub-optimality rate than standard VS-based algorithms. Specifically, the RO-based algorithm achieves a sub-optimality rate of  $K^{-1/3}$ , whereas VS-based algorithms achieve a faster rate of  $K^{-1/2}$ , where  $K$  represents the number of episodes in the offline data.
On the other hand, posterior sampling (PS) [Thompson, 1933, Russo and Van Roy, 2014], a popular and successful method in online RL, is rarely explored in the context of offline RL. PS involves sampling from a constructed posterior distribution over the model or value function and acting accordingly. However, PS is less commonly considered in offline RL due to its explorative nature, which stems from the randomness of the posterior distribution. This randomness is well-suited for addressing the exploration challenge in online RL tasks [Zhang, 2022, Dann et al., 2021, Zhong et al., 2022, Agarwal and Zhang, 2022]. The only work that considers PS for offline RL is Uehara and Sun [2022], where they maintain a posterior distribution over Markov decision process (MDP) models. However, this model-based PS approach is limited to small-scale problems where computing the optimal policy from an MDP model is computationally feasible. In addition, this work only provides a weak form of guarantees via Bayesian bounds.
In the context of (value) function approximation, achieving sample-efficient offline RL relies on certain conditions that facilitate effective learning. The identification of the minimum condition required for sample efficiency, as well as the algorithms that can exploit such conditions, is an important research question that we aim to address here. We advance our understanding by making the following contributions: (I) We introduce a new notion of data diversity that subsumes and expands all the prior distribution shift measures in offline RL, and (II) We show that all VS-based, RO-based and PS-based algorithms are in fact (surprisingly) competitive to each other, i.e., under standard assumptions, they achieve the same sub-optimality bounds (up to constant and log factors). We summarize our key results in comparison with related work in Table 1. Our results further expand the class of sample-efficient offline RL problems (Figure 1) and provide more choices of offline RL algorithms with competitive guarantees and tractable approximations for practitioners to choose from.
Algorithms	Sub-optimality Bound	Data
VS in [Xie et al., 2021]	Hb√C2(π)·ln(|F|·|Πall|)·K-1/2	I
RO in [Xie et al., 2021]	Hb√C2(π)·3√ln(|F|·|Πsoft(T)|)·K-1/3 + Hb/√T	I
MBPS in [Uehara and Sun, 2022]	Hb√CBayes·ln|M|·K-1/2 (Bayesian)	I
VS in Algorithm 2	Hb√C(π;1/√K)·ln(|F|·|Πsoft(T)|)·K-1/2 + Hb/√T	A
RO in Algorithm 3	Hb√C(π;1/√K)·ln(|F|·|Πsoft(T)|)·K-1/2 + Hb/√T	A
MFPS in Algorithm 4	Hb√C(π;1/√K)·ln(|F|·|Πsoft(T)|)·K-1/2 + Hb/√T (frequentist)	A
Table 1: Comparison of our bounds with SOTA bounds for offline RL under partial coverage and function approximation, where gray cells mark our contributions. Algorithms: VS = version space, RO = regularized optimization, MBPS = model-based posterior sampling, and MFPS = model-free posterior sampling. Sub-optimality bound:  $K =$  #number of episodes,  $\pi =$  an arbitrary comparator policy,  $H =$  horizon,  $b =$  boundedness,  $T =$  the number of algorithmic updates,  $\ln |\mathcal{F}|, \ln |\Pi^{soft}(T)|, \ln |\Pi^{all}|, \ln |\mathcal{M}|$ : complexity measures of some value function class  $\mathcal{F}$ , "induced" policy class  $\Pi^{soft}(T)$ , the class of all comparator policies  $\Pi^{all}$ , and model class  $\mathcal{M}$ , where typically  $\Pi^{soft}(T) \subset \Pi^{all}, \forall T$ . Data: I = independent episodes, A = adaptively collected data. Here  $\mathcal{C}(\pi; 1/\sqrt{K})$  and  $C_2(\pi)$  are some measures of extrapolation from the offline data to target policy  $\pi$ .
For establishing (II), we need to construct concrete VS-based, RO-based and PS-based algorithms. While the key components of the VS-based and RO-based algorithms appear in the literature [Xie et al., 2021], we propose a novel, a first-of-its-kind, model-free posterior sampling algorithm for offline RL. The algorithm contains two new ingredients: a pessimistic prior that encourages pessimistic value functions when being sampled from the posterior distribution and integration of posterior sampling with the actor-critic framework that incrementally updates the learned policy.
Overview of Techniques. Our analysis method presents a "decoupling" argument tailored for the batch setting, drawing inspiration from recent decoupling arguments in the online RL setting [Foster et al., 2021, Jin et al., 2021a, Zhang, 2022, Dann et al., 2021, Zhong et al., 2022, Agarwal and Zhang, 2022]. The core idea behind our decoupling argument is to establish a relationship between the Bellman error under any comparator policy  $\pi$  and the squared Bellman error under the behavior policy. This relationship is mediated through our novel concept of data diversity, denoted as  $\mathcal{C}(\pi; \epsilon_c)$ , which is defined in detail in Definition 3. This allows to separate the sub-optimality of a learned policy into two main sources of errors: the extrapolation error, which captures the out-of-distribution (OOD) generalization from the behavior policy to a target policy, and the in-distribution error, which focuses on generalization within the same behavior distribution. The OOD error is effectively managed by controlling the data diversity  $\mathcal{C}(\pi; \epsilon_c)$ , while the in-distribution error is carefully addressed by utilizing the algorithmic structures and the martingale counterpart to Bernstein's inequality (i.e., Freedman's inequality).
In the process of bounding the in-distribution error of our proposed PS algorithm that we built upon the technique of Dann et al. [2021], we correct a non-rigorous argument of Dann et al. [2021] (which we discuss in detail in Section E.3.1) and develop a new technical argument to handle the statistical dependence induced by the data-dependent target policy in the actor-critic framework. Our new argument carefully incorporates the uniform convergence argument into the in-expectation bounds of PS. We give a detailed description of this argument in Section E.3. As an immediate application, our technique fixes a technical mistake involving how to handle the statistical dependence induced by the min player in the self-play posterior sampling algorithm of Xiong et al. [2022].
2.1 Episodic Time-inhomogenous Markov Decision Process
Let  $S$  and  $\mathcal{A}$  denote Lebesgue-measurable state and action spaces (possibly infinite), respectively. Let  $\mathcal{P}(S)$  denote the space of all probability distributions over  $S$ . We consider an episodic time-inhomogeneous Markov decision process  $M = (S, \mathcal{A}, P, r, H)$ , where  $P = \{P_h\}_{h \in [H]} \in$
$\{\mathcal{S}\times \mathcal{A}\to \mathcal{P}(\mathcal{S})\} ^H$  are the transition probabilities (where  $[H]\coloneqq \{1,\ldots ,H\}$ $r = \{r_h\}_{h\in [H]}\in$ $\{\mathcal{S}\times \mathcal{A}\to \mathbb{R}\} ^H$  is the mean reward functions, and  $H\in \mathbb{N}$  is the length of the horizon for each episode. For any policy  $\pi = \{\pi_h\}_{h\in [H]}\in \{\mathcal{S}\rightarrow \mathcal{P}(\mathcal{A})\} ^H$  , the action-value functions and the value functions under policy  $\pi$  are defined, respectively, as  $Q_{h,M}^{\pi}(s,a) = \mathbb{E}_{\pi}[\sum_{i = h}^{H}r_{i}(s_{i},a_{i})|$ $(s_h,a_h) = (s,a)]$  , and  $V_{h,M}^{\pi}(s) = \mathbb{E}_{\pi}[\sum_{i = h}^{H}r_{i}(s_{i},a_{i})|s_{h} = s]$  . Here  $\mathbb{E}_{\pi}[\cdot ]$  denotes the expectation with respect to the randomness of the trajectory  $(s_h,a_h,\dots,s_H,a_H)$  , with  $a_i\sim \pi_i(\cdot |s_i)$  and  $s_{i + 1}\sim P_i(\cdot |s_i,a_i)$  for all  $i$  . For any policy  $\pi$  , we define the visitation density probability functions  $d_M^\pi = \{d_{h,M}^\pi \}_{h\in [H]}\in \{\mathcal{S}\times \mathcal{A}\to \mathbb{R}_+\} ^H$  as  $d_h^\pi (s,a)\coloneqq \frac{d\operatorname*{Pr}((s_h,a_h) = (s,a)|\pi,M)}{d\rho(s,a)}$  where  $\rho$  is the Lebesgue measure on  $\mathcal{S}\times \mathcal{A}$  and  $\mathrm{Pr}((s_h,a_h) = (s,a)|\pi ,M)$  is the probability of policy  $\pi$  reaching state-action pair  $(s,a)$  at timestep  $h$  . The Bellman operator  $\mathbb{T}_h^\pi$  is defined as  $[\mathbb{T}_h^\pi Q](s,a)\coloneqq r_h(s,a) + \mathbb{E}_{s'\sim P_h(\cdot |s,a),a'\sim \pi_{h + 1}(\cdot |s')}[Q(s',a')]$  , for any  $Q:S\times A\to \mathbb{R}$  . Let  $\pi^*$  be an optimal policy, i.e.,  $Q_{h}^{\pi^{*}}(s,a)\geq Q_{h}^{\pi}(s,a),\forall (s,a,h,\pi)\in S\times A\times [H]\times \Pi^{all}$  , where  $\Pi^{all}\coloneqq \{\mathcal{S}\to \mathcal{P}(\mathcal{A})\} ^H$  is the set of all possible policies. For simplicity, we assume that the initial state  $s_1$  is deterministic across all episodes. We also assume that there is some  $b > 0$  such that for any trajectory  $(s_1,a_1,r_1,\dots,s_H,a_H,r_H)$  generated under any policy,  $|r_h|\leq b,\forall h$  and  $|\sum_{h = 1}^{H}r_{h}|\leq b$  almost surely. This boundedness assumption is standard and subsumes the boundedness conditions in the previous works, e.g., Zanette et al. [2021] set  $b = 1$  and Jin et al. [2021b] use  $b = H$  (and further assume that  $r_h\in [0,1],\forall h)$ . Without loss of generality, we assume that  $b\geq 1$
Additional Notation. For any  $u: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  and any  $\pi: \mathcal{S} \to \mathcal{P}(\mathcal{A})$ , we overload the notation  $u(s, \pi) := \mathbb{E}_{a \sim \pi(\cdot | s)}[u(s, a)]$ . For any  $f: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ , denote the supremum norm  $\| f \|_{\infty} = \max_{(s, a) \in \mathcal{S} \times \mathcal{A}} |f(s, a)|$ . We write  $\mathbb{E}[g]^2 := (\mathbb{E}[g])^2$ . For a probability measure  $\nu$  on some measurable space  $(\Omega, \mathcal{B})$ , we denote by  $\operatorname{supp}(\nu)$  the support of  $\nu$ ,  $\operatorname{supp}(\nu) := \{B \in \mathcal{B}: \nu(B) > 0\}$ . We denote  $x \lesssim y$  to mean that  $x = \mathcal{O}(y)$ .
2.2 Offline Data Generation
Denote the pre-collected dataset by  $\mathcal{D} := \{(s_h^t, a_h^t, r_h^t)\}_{h \in [H]}$ , where  $s_{h+1}^t \sim P_h(\cdot | s_h^t, a_h^t)$  and  $\mathbb{E}[r_h^t | s_h^t, a_h^t] = r_h(s_h^t, a_h^t)$ . We consider the adaptively collected data setting where the offline data is collected by time-varying behavior policies  $\{\mu^k\}_{k \in [K]}$ , concretely, defined as follows.
Definition 1 (Adaptively collected data<sup>4</sup>).  $\mu^k$  is a function of  $\{(s_h^i, a_h^i, r_h^i)\}_{h \in [H]}^{i \in [k-1]}$ ,  $\forall k \in [K]$ .
For simplicity, we denote  $\mu = \frac{1}{K}\sum_{k=1}^{K}\mu^k$ ,  $d^\mu = \frac{1}{K}\sum_{k=1}^{K}d^{\mu^k}$ , and  $\mathbb{E}_{\mu}[\cdot] = \frac{1}{K}\sum_{k=1}^{K}\mathbb{E}_{\mu^k}[\cdot]$ . The setting of adaptively collected data covers a common practice where the offline data is collected by using some adaptive experimentation [Zhan et al., 2023]. When  $\mu^1 = \dots = \mu^K$ , it recovers the setting of independent episodes in Duan et al. [2020].
Value sub-optimality. The goodness of a learned policy  $\hat{\pi} = \hat{\pi}(\mathcal{D})$  against a comparator policy  $\pi$  for the underlying MDP  $M$  is measured by the (value) sub-optimality defined as
$$
\operatorname {S u b O p t} _ {\pi} ^ {M} (\hat {\pi}) := V _ {1} ^ {\pi} \left(s _ {1}\right) - V _ {1} ^ {\hat {\pi}} \left(s _ {1}\right). \tag {1}
$$
Whenever the context is clear, we drop  $M$  in  $Q_{M}^{\pi}$ ,  $V_{M}^{\pi}$ ,  $d_{M}^{\pi}$ , and  $\mathrm{SubOpt}_{\pi}^{M}(\hat{\pi})$ .
2.3 Policy and function classes
Next, we define the policy space and the action-value function space over which we optimize the value sub-optimality. We consider a (Cartesian product) function class  $\mathcal{F} = \mathcal{F}_1 \times \dots \times \mathcal{F}_H \in \{\mathcal{S} \times \mathcal{A} \to [-b, b]\}^H$ . The function class  $\mathcal{F}$  induces the following (Cartesian product) policy class
<sup>1</sup>This assumption is merely for the sake of clean presentation which does not affect any results.
Note that we allow the reward samples to be negative.
<sup>3</sup>We can replace the condition  $|r_h| \leq b, \forall h$  with 1-sub-Gaussian condition:  $r_h \sim R_h(s_h, a_h)$  wherein  $R_h(s_h, a_h)$  is sub-Gaussian with mean  $r_h(s_h, a_h)$  - which replaces  $b$  in our main theorems by  $b + \ln(KH / \delta)$ .
It is essentially the "measurability" condition in Zanette et al. [2021] and "compliance" condition in Jin et al. [2021b].
$\Pi^{soft}(T) = \Pi_1^{soft}(T)\times \dots \times \Pi_H^{soft}(T)$ , where  $\Pi_h^{soft}(T)\coloneqq \{\pi_h(a|s)\propto \exp (\eta \sum_{i = 1}^t g_i(s,a)): t\in [T],g_i\in \mathcal{F}_h,\forall i\in [t],\eta \in [0,1]\}$  for any  $T\in \mathbb{N}$ . The motivation for the induced policy class  $\Pi^{soft}(T)$  is from the soft policy iteration (SPI) update where we incrementally update the policy.
We now discuss a set of assumptions that we impose on the policy and function classes.
Assumption 2.1 (Approximate realizability). There exist  $\{\xi_h\}_{h\in [H]}$  where  $\xi_{h}\geq 0$  such that,
$$
\sup  _ {T \in \mathbb {N}, \pi \in \Pi^ {s o f t} (T), (s _ {h}, a _ {h}) \in \operatorname {s u p p} (d _ {h} ^ {\mu})} \inf  _ {f \in \mathcal {F}} | f _ {h} (s _ {h}, a _ {h}) - Q _ {h} ^ {\pi} (s _ {h}, a _ {h}) | \leq \xi_ {h}, \forall h \in [ H ].
$$
Assumption 2.1 establishes that  $\mathcal{F}$  can realize  $Q^{\pi}$  for any  $\pi \in \Pi^{soft}(T)$  up to some error  $\xi \in \mathbb{R}^H$  in the supremum norm over the  $\mu$ -feasible state-action pairs. It strictly generalizes the assumption in Zanette et al. [2021] which restricts  $\xi_h = 0$ ,  $\forall h$  (i.e., assume realizability) and the assumption in Xie et al. [2021] which constrains the approximation error under any feasible state-action distribution.
The realizability in value functions alone is known to be insufficient for sample-efficient offline RL [Wang et al., 2021]; thus, one needs to impose a stronger assumption for polynomial sample complexity of model-free methods. In this paper, we impose an assumption on the closedness of the Bellman operator.
Assumption 2.2 (General Restricted Bellman Closedness). There exists  $\nu \in \mathbb{R}^H$  such that
$$
\sup  _ {T \in \mathbb {N}, f _ {h + 1} \in \mathcal {F} _ {h + 1}, \tilde {\pi} \in \Pi^ {s o f t} (T)} \inf  _ {f _ {h} ^ {\prime} \in \mathcal {F} _ {h}} \| f _ {h} ^ {\prime} - \mathbb {T} _ {h} ^ {\tilde {\pi}} f _ {h + 1} \| _ {\infty} \leq \nu_ {h}, \forall h \in [ H ].
$$
Assumption 2.2 ensures that the value function space  $\mathcal{F}$  and the induced policy class  $\Pi^{soft}(T)$  for any  $T\in \mathbb{N}$  are closed under the Bellman operator up to some error  $\nu \in \mathbb{R}^H$  in the supremum norm. This assumption is a direct generalization of the Linear Restricted Bellman Closedness in Zanette et al. [2020] from a linear function class to a general function class. As remarked by Zanette et al. [2021], the Linear Restricted Bellman Closedness is already strictly more general than the low-rank MDPs [Yang and Wang, 2019, Jin et al., 2020].
2.4 Effective sizes of policy and function classes
When the function class and the policy class have finite elements, we use their cardinality  $|\mathcal{F}_h|$  and  $|\Pi_h^{soft}(T)|$  to measure their sizes [Jiang et al., 2017, Xie et al., 2021]. When they have infinite elements, we use log-covering numbers, defined as
$$
d _ {\mathcal {F}} (\epsilon) := \max  _ {h \in [ H ]} \ln N (\epsilon ; \mathcal {F} _ {h}, \| \cdot \| _ {\infty}), \text {a n d} d _ {\Pi} (\epsilon , T) := \max  _ {h \in [ H ]} \ln N (\epsilon ; \Pi_ {h} ^ {s o f t} (T), \| \cdot \| _ {1, \infty}),
$$
where  $\| \pi - \pi' \|_{1,\infty} = \sup_{s \in S} \int_{\mathcal{A}} |\pi(a|s) - \pi'(a|s)| d\rho(a)$  for any  $\pi, \pi' \in \{\mathcal{S} \to \mathcal{P}(\mathcal{A})\}$  and  $N(\epsilon; \mathcal{X}, \| \cdot \|)$  denotes the covering number of a pseudometric space  $(\mathcal{X}, \| \cdot \|)$  with metric  $\| \cdot \|$  [Zhang, 2023, e.g. Definition 4.1].
We also define a complexity measure that depends on a prior distribution  $p_0$  over  $\mathcal{F}$  that we employ to favor certain regions of the function space. Our notion, presented in Definition 2, is simply a direct adaptation of a similar notation of Dann et al. [2021] to the actor-critic setting.
Definition 2. For any function  $f' \in \mathcal{F}_{h+1}$  and any policy  $\tilde{\pi} \in \Pi^{all}$ , we define  $\mathcal{F}_h^{\tilde{\pi}}(\epsilon; f') := \{f \in \mathcal{F}_h : \| f - \mathbb{T}_h^{\tilde{\pi}}f'\|_{\infty} \leq \epsilon\}$ , for any  $\epsilon \geq 0$ , and subsequently define
$$
d_{0}(\epsilon):= \sup_{T\in \mathbb{N},f\in \mathcal{F},\tilde{\pi}\in \Pi^{soft}(T)}\sum_{h = 1}^{H}\ln \frac{1}{p_{0,h}(\mathcal{F}_{h}^{\tilde{\pi}}(\epsilon;f_{h + 1}))},d_{0}^{\prime}(\epsilon):= \sup_{T\in \mathbb{N},\tilde{\pi}\in \Pi^{soft}(T)}\sum_{h = 1}^{H}\ln \frac{1}{p_{0,h}(\mathcal{F}_{h}^{\tilde{\pi}}(\epsilon;Q_{h + 1}^{\tilde{\pi}}))}.
$$
The quantity  $d_0(\epsilon)$  and  $d_0'(\epsilon)$  measures the concentration of the prior  $p_0$  over all functions  $f \in \mathcal{F}$  that are  $\epsilon$ -close (element-wise) under  $\mathbb{T}^{\tilde{\pi}}$  and  $\epsilon$ -close (element-wise) to  $Q_h^{\tilde{\pi}}$ , respectively. If a stronger version of Assumption 2.1 is met, i.e.,  $Q_h^{\tilde{\pi}} \in \mathcal{F}_h$ ,  $\forall \tilde{\pi} \in \Pi_h^{all}$ ,  $h \in [H]$ , we have  $d_0'(\epsilon) \leq d_0(\epsilon), \forall \epsilon$ . For
A stronger form of realizability is sufficient for polynomial sample complexity, e.g., realizability for a density ratio w.r.t. the behavior state-action distribution in dual-primal methods [Zhan et al., 2022, Chen and Jiang, 2022, Rashidinejad et al., 2023] or realizability for the underlying MDP in model-based methods [Uehara and Sun, 2022]. Instead, we pursue model-free value-based methods.