SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenzhuo Zhou
University of California Irvine
wenzhuz3@uci.edu
We study offline reinforcement learning (RL) which seeks to learn a good policy based on a fixed, pre-collected dataset. A fundamental challenge behind this task is the distributional shift due to the dataset lacking sufficient exploration, especially under function approximation. To tackle this issue, we propose a bi-level structured policy optimization algorithm that models a hierarchical interaction between the policy (upper-level) and the value function (lower-level). The lower level focuses on constructing a confidence set of value estimates that maintain sufficiently small weighted average Bellman errors, while controlling uncertainty arising from distribution mismatch. Subsequently, at the upper level, the policy aims to maximize a conservative value estimate from the confidence set formed at the lower level. This novel formulation preserves the maximum flexibility of the implicitly induced exploratory data distribution, enabling the power of model extrapolation. In practice, it can be solved through a computationally efficient, penalized adversarial estimation procedure. Our theoretical regret guarantees do not rely on any data-coverage and completeness-type assumptions, only requiring realizability. These guarantees also demonstrate that the learned policy represents the "best effort" among all policies, as no other policies can outperform it. We evaluate our model using a blend of synthetic, benchmark, and real-world datasets for offline RL, showing that it performs competitively with state-of-the-art methods.
Offline reinforcement learning (RL) is a task to learn a good policy using only a pre-collected, fixed dataset, without further exploration with the environment. This distinctive characteristic positions offline RL as a promising approach for solving real-world sequential decision-making problems in healthcare [35, 61], financial marketing [46], robotics [47] and education [32], as acquiring diverse or expert-quality data in these fields can be costly or practically unattainable.
Arguably, two of the biggest challenges in offline RL are the distributional shift between the data-generating distribution and those induced by candidate policies, and the stringent requirements on the properties of function approximation [28]. It has been observed that, in practice, the distributional mismatch often results in unsatisfactory performance of many existing algorithms, and even amplifying with function approximation [18, 27]. Many prior works [39, 13, 2, 14] crucially rely on a global data-coverage assumption and completeness-type function approximation condition in a technical sense. The former necessitates that the dataset to contain any state-action pair with a lower bounded probability so that the distributional shift can be well calibrated. The latter requires the function class to be closed under Bellman updates. Both assumptions are particularly strong and are likely to be violated in practice [58]. Consequently, algorithms that depend on these assumptions may experience performance degradation and instability [52]. Therefore, it is crucial to develop novel algorithms that relax these assumptions, offering robust and widely applicable solutions for real-world scenarios.
To address the aforementioned challenges in offline settings, one fundamental principle is the concept of pessimism, which aims to maximize rewards in the worst possible MDP consistent with the offline dataset [18, 53]. In practice, these methods have generally been shown to be more robust when coverage assumptions are violated [27].
Although many such pessimistic algorithms have been developed, very few works can tackle data-coverage and function approximation issues simultaneously, while establishing strong regret guarantees. For instance, deep offline RL algorithms [18, 25, 23] exhibit impressive empirical performance, but their theoretical consistency guarantees are limited to tabular Markov decision processes (MDPs). The works of [30, 41, 22, 53, 56, 58, 10] relax the global coverage to a partial coverage condition, wherein the offline data only covers a single comparator policy. However, all of these methods require Bellman completeness for the function class. The most recent works [9, 59] take a significant step towards relaxing Bellman completeness to realizability, that the function class can capture the target ground-truth function. Nonetheless, these algorithms are unable to provide a meaningful regret guarantee without any data-coverage assumption (when both global and partial coverage fails), and also empirical evaluations are absent. Even without additional conditions, the learned policies of these algorithms can only compete with the (Bellman flow) optimal policy, resulting in a lack of robustness when the optimal policy is not covered by data, a situation that frequently occurs. Due to page limit, we have only discussed the closest related work here, and the rest is deferred to Appendix.
Our contribution. In this paper, we develop a provably sample-efficient offline RL framework. Our information-theoretic algorithm is designed based on the concept of bi-level (upper and lower level) structured optimization, which leads to a hierarchical interpretation and naturally enjoys learning stability and algorithmic convergence from a game-theoretic perspective. In particular, at the lower level, one component is to construct a confidence set with consistent value estimates regarding the appropriately small weighted average Bellman error, effectively preventing overly pessimistic evaluation. Meanwhile, the second component, which deals with uncertainty control, implicitly enhances the power of model extrapolation. In addition to the information-theoretic algorithm, we also develop a computationally efficient counterpart that is solved by a penalized adversarial estimation algorithm with proximal-mapping updating, allowing both non-linear and linear function approximation. From a theoretical standpoint, we establish a strong regret guarantee for both information-theoretical and practical algorithms under only realizability without requiring any data-coverage (neither global nor partial coverage) and completeness-type assumptions. As a special case study, we further refine our developed mixture density ratio-based concentrability coefficient to a relative condition number in linear MDP settings. The sample complexity of our regret bound improves or at least matches the prior results in the fully exploratory or partial coverage settings where the Bellman-completeness holds. Notably, compared with existing works, either focusing on theoretical or empirical development, we provide a comprehensive theoretical analysis of the proposed framework and also conduct synthetic, benchmark, and real data experiments for empirical evaluation.
Markov decision process. We consider an infinite-horizon discounted MDP  $\mathcal{M} = \{\mathcal{S},\mathcal{A},\mathbb{P},\gamma ,r,s^0\}$  [40], where  $S$  is the state space,  $\mathcal{A}$  is the action space,  $\mathbb{P}:S\times \mathcal{A}\to \Delta (\mathcal{S})$  is the Markov transition kernel for some probabilistic simplex  $\Delta$ ,  $r:S\times \mathcal{A}\rightarrow [0,\bar{R} ]$  is the reward function for  $\bar{R}\geq 0$ $\gamma \in [0,1)$  is the discounted factor and  $s^0$  is the initial state. A policy  $\pi :S\to \Delta (\mathcal{A})$  induces a distribution of the trajectory  $s^0,a^0,r^0,s^1,\ldots$ , where  $a^t\sim \pi (\cdot |s^t)$ ,  $r^t = r(s^t,a^t)$ ,  $s^{t + 1}\sim \mathbb{P}(\cdot |s^t,a^t)$  for any  $t\geq 0$ . The expected discounted return of a policy is defined as  $J(\pi) = \mathbb{E}[\sum_{t = 0}^{\infty}\gamma^{t}r^{t}|\pi ]$ . The discounted return when the trajectory starts with  $(s,a)$  and all remaining actions are taken according to  $\pi$  is called  $q$ -function  $q^{\pi}:S\times \mathcal{A}\to [0,\bar{V} ]$ . The  $q^{\pi}$  is the unique fixed point of the Bellman operator  $\mathcal{B}^{\pi}$ , satisfying the Bellman equation [45]:  $\mathcal{B}^{\pi}q(s,a)\coloneqq r(s,a) + \gamma \mathbb{E}_{s'\sim \mathbb{P}(\cdot |s,a)}[q(s',\pi)]$ . Here  $q(s',\pi)$  is denoted as shorthand for  $\mathbb{E}_{a'\sim \pi (\cdot |s')}[q(s',a')]$ , and we define  $\mathbb{P}^{\pi}q(s,a)\coloneqq \mathbb{E}_{s'\sim \mathbb{P}(\cdot |s,a)}[q(s',\pi)]$ . Additionally, it is helpful to remember that  $J(\pi) = q^{\pi}(s^{0},\pi)$ . Another important notion is the normalized discounted visitation of  $\pi$ , defined as  $d_{\pi}(s,a)\coloneqq (1 - \gamma)\sum_{t = 0}^{\infty}\gamma^{t}d_{\pi ,t}(s,a)$ , where  $d_{\pi ,t}$  is the marginal state-action distribution at the time-step  $t$ .
Offline RL under function approximation. In the offline RL setting, there exists an unknown offline data-generating distribution  $\mu$  induced by behavior policies. Despite the unknowns of  $\mu$ , we can observe a set of transition pairs, as offline dataset  $\mathcal{D}_{1:n} \coloneqq \{s_i, a_i, r_i, s_i'\}_{i=1}^n$  sampling from  $\mu$ . For a given policy  $\pi$ , the density-ratio (importance-weight),  $\tau_{d_{\pi}/\mu}(s, a) = d_{\pi}(s, a)/\mu(s, a)$ , measures how
effectively  $\mu$  covers the visitation induced by  $\pi$ . The primary objective of offline policy optimization is to learn an optimal policy that maximizes the return,  $J(\pi)$ , using the offline dataset. Under the function approximation setting, we assume access to two function classes  $\mathcal{Q}: S \times \mathcal{A} \to \mathbb{R}$  and  $\Omega: S \times \mathcal{A} \to \mathbb{R}$ , which are utilized to capture  $q^{\pi}$  and  $\tau_{d_{\pi} / \mu}$ , respectively.
Exploration and coverage. In general, when saying an offline dataset is well-explored, it means that a well-designed behavior policy has been executed, allowing for comprehensive exploration of the MDP environment. As a result, the dataset is likely to contain possibly all state-action pairs. This implicitly requires  $\mu$  has the global coverage [18, 49]. In this context, the global coverage means that the density ratio-based concentrability coefficient,  $\sup_{s,a}\{d_{\pi}(s,a) / \mu (s,a)\}$ , is upper-bounded by a constant  $c\in \mathbb{R}^{+}$  for all policies  $\pi \in \Pi$ , where  $\Pi$  is some policy class. This condition is frequently employed in offline RL [2, 8, 12]. However, in practice, this assumption may not hold true, as devising an exploratory policy is a challenging task for large-scale RL problems. Instead, our goal is to learn a good policy with strong theoretical guarantees that can compete against any arbitrarily covered comparator policy under much weaker conditions than the global coverage.
3 Bi-Level Offline Policy Optimization Algorithm
In this section, we introduce our bi-level offline policy optimization framework. The development of the framework consists of three major steps.
Step 1: robust interval learning. In this step, we aim to provide a robust off-policy interval evaluation. The major advantage of this interval formulation is its robustness to the model-misspecification of the importance-weight class  $\Omega$ , and the encoding of distributional-shift information in the policy evaluation process. First, we define a detection function  $\mathbb{D}(\cdot)$ , which is used to measure the degree of the distributional-shift in terms of density ratio.
Definition 3.1. For  $x, c_1, c_2, C \in \mathbb{R}^+$  and  $C \geq 1$ , the detection function  $\mathbb{D}(\cdot)$  satisfies the following conditions: (1) 1-minimum:  $\mathbb{D}(1) = 0$ . (2) Non-negativity:  $\mathbb{D}(x) \geq 0$ . (3) Boundedness on first-order derivative:  $|\mathbb{D}'(x)| \leq c_2$  if  $x \in [0, C]$ . (4) Boundedness on value:  $|\mathbb{D}(x)| \leq c_1$  for  $x \in [0, C]$ . (5) Strong convexity:  $\mathbb{D}(x)$  is  $M$ -strongly convex with respect to  $x$ .
The family of Rényi entropy [42], Bhattacharyya distance [11], and simple quadratic form functions [60] all satisfy the conditions outlined in Definition 3.1. Under this definition, it can easily observe that  $\mathbb{D}$  has a convex conjugate function [6],  $\mathbb{D}_*$  with  $\mathbb{D}_*(x_*) = \sup_x\{x\cdot x_*(-\mathbb{D}(x))\}$ , that satisfies  $\mathbb{D}_*(0) = 0$ . It follows from Bellman equation  $\mathcal{B}^{\pi}q^{\pi}(s,a) = q^{\pi}(s,a)$  for any  $s,a$ , then  $J(\pi) = q^{\pi}(s^{0},\pi) + \mathbb{E}_{\mu}[\lambda \mathbb{D}_{*}((\mathcal{B}^{\pi}q^{\pi}(s,a) - q^{\pi}(s,a) / \lambda)) / (1 - \gamma)]$  for  $\lambda \geq 0$ . Applying Fenchel-Legendre transformation [36, 21], and model  $x$  in a restricted importance weight class  $\Omega$  for any  $s,a$ , we obtain
$$
\begin{array}{l} J (\pi) = q ^ {\pi} \left(s ^ {0}, \pi\right) + \mathbb {E} _ {\mu} [ \sup  _ {x} x \cdot \left(\mathcal {B} ^ {\pi} q ^ {\pi} (s, a) - q ^ {\pi} (s, a)\right) - \lambda \mathbb {D} (x) ] / (1 - \gamma) (1) \\ \geq q ^ {\pi} \left(s ^ {0}, \pi\right) + \mathbb {E} _ {\mu} \left[ \tau (s, a) \left(r (s, a) + \gamma q ^ {\pi} \left(s ^ {\prime}, \pi\right) - q ^ {\pi} (s, a)\right) - \lambda \mathbb {D} (\tau (s, a)) \right] / (1 - \gamma). (2) \\ \end{array}
$$
Suppose  $q^{\pi}$  is well-sepcified, i.e.,  $q^{\pi} \in \mathcal{Q}$ , we can find a lower bound of (2), which is valid for any  $\tau \in \Omega$ , via replacing  $q^{\pi}$  with  $\inf_{q \in \mathcal{Q}}$  as follows:
$$
\begin{array}{l} J(\pi)\geq \inf_{q\in \mathcal{Q}}\Big\{\underbrace{\left(\mathbb{E}_{\mu}\left[ \tau(s,a)(r(s,a) + \gamma q \left(s^{\prime},\pi\right) - q(s,a))\right] + q(s^{0},\pi)\right) / (1 - \gamma)}_{:= H(\tau ,q,\pi)} \\ - \underbrace {\lambda / (1 - \gamma) \mathbb {E} _ {\mu} [ \mathbb {D} (\tau (s , a)) ]} _ {:= \lambda \xi (\mathbb {D}, \tau)} \Big \}. \\ \end{array}
$$
After following a similar derivation, we can establish an upper bound for  $J(\pi)$  as well, and thus construct a value interval for  $J(\pi)$ . This interval holds for any  $\tau$  and is therefore robust against model-misspecification of  $\Omega$ . In order to obtain a tighter interval, we can shrink the interval width by maximizing the lower bound and minimizing the upper bound, both with respect to  $\tau$ . This procedure can be interpreted as searching for some good  $\tau \in \Omega$  to minimize the function approximation error.
$$
J (\pi) \in \left[ \sup  _ {\tau \in \Omega} \inf  _ {q \in \mathcal {Q}} H (\tau , q, \pi) - \lambda \xi (\mathbb {D}, \tau), \inf  _ {\tau \in \Omega} \sup  _ {q \in \mathcal {Q}} H (\tau , q, \pi) + \lambda \xi (\mathbb {D}, \tau) \right], \tag {3}
$$
While the interval offers a robust method for dealing with the bias introduced by function approximation when estimating  $J(\pi)$ , it lacks a crucial and non-trivial step for handling statistical uncertainty.
Step 2: uncertainty quantification. In this step, we quantify the uncertainty of the interval (3), and establish a non-asymptotic confidence interval (CI) for  $J(\pi)$  which integrates bias and uncertainty quantifications in a single interval inspired by [60]. Given offline data  $\mathcal{D}_{1:n}$ , our formal result for quantifying sampling uncertainty in order to establish the CI for  $J(\pi)$ .
Theorem 3.1 (Non-asymptotic confidence interval). For a target policy  $\pi$ , the return  $J(\pi)$  is within a CI for any  $\tau \in \Omega$  with probability at least  $1 - \delta$ , i.e.,  $J(\pi) \in [\widehat{J}_n^- (\pi ;\tau),\widehat{J}_n^+ (\pi ;\tau)]$  for
$$
\widehat {J} _ {n} ^ {-} (\pi ; \tau) := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {r _ {i} \tau (s _ {i} , a _ {i})}{1 - \gamma} - \sup _ {q \in \mathcal {Q}} \widehat {M} _ {n} (- q, \tau) - \lambda \xi_ {n} (\mathbb {D}, \tau) - \sigma_ {n},
$$
$$
\widehat {J} _ {n} ^ {+} (\pi ; \tau) := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {r _ {i} \tau \left(s _ {i} , a _ {i}\right)}{1 - \gamma} + \sup  _ {q \in \mathcal {Q}} \widehat {M} _ {n} (q, \tau) + \lambda \xi_ {n} (\mathbb {D}, \tau) + \sigma_ {n}, \tag {4}
$$
if the uncertainty deviation  $\sigma_{n}$  satisfies
$$
P \Bigg (\sup  _ {\tau \in \Omega} \Big | \frac {1}{n (1 - \gamma)} \sum_ {i = 1} ^ {n} \tau (s _ {i}, a _ {i}) (r _ {i} + \gamma q ^ {\pi} (s _ {i} ^ {\prime}, \pi) - q ^ {\pi} (s _ {i}, a _ {i})) - \lambda \xi_ {n} (\mathbb {D}, \tau) \Big | \leq \sigma_ {n} \Bigg) \geq 1 - \delta ,
$$
where  $\widehat{M}_n(q,\tau)\coloneqq \sum_{i = 1}^n\tau (s_i,a_i)(\gamma q(s_i',\pi) - q(s_i,a_i)) / (1 - \gamma)n + q(s^0,\pi).$
Similar to the value interval, the CI  $\left[\widehat{J}_n^{-}(\pi ;\tau),\widehat{J}_n^{+}(\pi ;\tau)\right]$  also holds for any  $\tau \in \Omega$ . Therefore, we can optimize the confidence lower and upper bounds in (4) over  $\tau \in \Omega$  to tighten the CI, and obtain:
$$
P \Big (J (\pi) \in [ \sup  _ {\tau \in \Omega} \widehat {J} _ {n} ^ {-} (\pi ; \tau), \inf  _ {\tau \in \Omega} \widehat {J} _ {n} ^ {+} (\pi ; \tau) ] \subseteq [ \widehat {J} _ {n} ^ {-} (\pi ; \tau), \widehat {J} _ {n} ^ {+} (\pi ; \tau) \Big) \geq 1 - \delta .
$$
Step 3: bridge policy evaluation to policy optimization. In this step, we aim to formulate a policy optimization based on the derived high-confidence policy evaluation from the previous steps. Given the consistent CI estimation of  $J(\pi)$ , we can naturally incorporate the pessimism principle, i.e., using the CI lower bounds of  $J(\pi)$  as the value estimate of the policy evaluation of  $\pi$  [22]. With such a procedure, our objective is to maximize these lower bounds over some family II of policies:
$$
\max  _ {\pi \in \Pi} \left\{\sup  _ {\tau \in \Omega} \widehat {J} _ {n} ^ {-} (\pi ; \tau) \right\}. \tag {5}
$$
Although (5) is algorithmically feasible for obtaining a policy solver  $\widehat{\pi}$ , it lacks direct interpretation without taking advantage of the bi-level optimization structure in hindsight. Therefore, we propose to reformulate (5) via a dual-to-prime conversion (shown in Theorem 3.2), which naturally lends itself to lower-upper optimization with guaranteed convergence. Specifically, we formulate (5) as a bi-level framework problem:
(Upper Level)  $\min_{\pi \in \Pi} - \underline{q}^{\pi}(s^{0},\pi),$  (6)
(Lower Level) s.t.  $\underline{q}^{\pi}\in \operatorname *{arg  min}_{q\in \mathcal{Q}_{\varepsilon_n}}q(s^0,\pi),$  (7)
$\underline{\text{Consistency}}: \mathcal{Q}_{\varepsilon_n} = \left\{q \in \mathcal{Q}: \sup_{\tau \in \widetilde{\Omega}_{\widetilde{\sigma}_n}} \left| n^{-1} \sum_{i=1}^{n} \tau(s_i, a_i) (r_i + \gamma q(s_i', \pi) - q(s_i, a_i)) \right| \leq \varepsilon_n \right\}$ ,
Uncertainty Control:  $\widetilde{\Omega}_{\widetilde{\sigma}_n} = \left\{\tau_{\circ} / \sup_{\tau_{\circ}\in \Omega}\| \tau_{\circ}\|_{\Omega}\text{for}\tau_{\circ}\in \Omega :\xi_n(\mathbb{D},\tau_{\circ})\right.\leq \widetilde{\sigma}_n\Bigg\} .$
At the upper level, the learned policy  $\widehat{\pi}$  attempts to maximize the value estimate of  $\underline{q}^{\pi}$  over some policy class  $\Pi$ , while at the lower level,  $q^{\pi}$  is to seek the  $q$ -function with the pessimistic policy evaluation value from the confidence set  $\overline{\mathcal{Q}}_{\varepsilon_n}$  with consistency guarantee and uncertainty control. For consistency, whenever  $q^{\pi}$  or its good approximator is included in  $\mathcal{Q}$  (realizability for  $\mathcal{Q}$  class is satisfied), the set  $\mathcal{Q}_{\varepsilon_n}$  ensures the estimation consistency of  $q^{\pi}$  in terms of "sufficiently small" weighted average Bellman error. For uncertainty control, the constrained set  $\widetilde{\Omega}_{\widetilde{\sigma}_n}$  attempts to control the uncertainty arising from distributional shift via a user-specific thresholding hyperparameter  $\widetilde{\sigma}_n$ . The feasible (uncertainty controllable) candidates  $\tau \in \widetilde{\sigma}_n$  are used as weights for the average Bellman error, helping to construct the consistent set  $\mathcal{Q}_{\varepsilon_n}$ . Risk-averse users can specify a lower value for the thresholding hyperparameter or consider a higher  $\widetilde{\sigma}_n$  to tolerate a larger distribution shift. In other words, the chosen value of  $\widetilde{\sigma}_n$  depends on the degree of pessimism users want to incorporate in the policy optimization.
Theorem 3.2. There must exist some threshold values  $\varepsilon_{n}$  and  $\widetilde{\sigma}_{n}$ , the return policy of (5)  $\widehat{\pi}$  satisfies the minimization problem in (6), indicating the solution of the optimization (5) and (6) is equivalent.
Interestingly, the new form in (6) characterizes our policy optimization framework as a two-player general-sum game [16], which is a sequential game involving two players. Each player aims to maximize their own payoffs while considering the decisions of other players. Our bi-level optimization framework has been demonstrated to improve learning stability and ensure algorithmic convergence, benefiting from the existence of a local equilibrium [51].
To close this section, we remark that the establishment of consistency with respect to the weighted average Bellman error is the key point for us to relax the completeness-type assumptions. In the famous API/AVI-type algorithms [13, 14, 8], they target to minimize a squared or minimax Bellman error for finding  $q \in \mathcal{Q}$  so that  $\| q - \mathcal{B}^{\pi}q\|_{L_2(\mu)}^2 \approx 0$  to obtain  $q \approx q^{\pi}$ . Unfortunately, even with the infinite amount of data, the empirical estimate of  $\| q - \mathcal{B}^{\pi}q\|_{L_2(\mu)}^2$ , i.e., squared empirical Bellman error) is biased due to the appearance of unwanted conditional variance, i.e., the double sampling issue [3]. The API/AVI-type algorithms need a separate helper function class  $\widetilde{\mathcal{Q}}$  for modeling  $\mathcal{B}^{\pi}q$ , and [8] has shown that when the class  $\widetilde{\mathcal{Q}}$  realizes the Bayes optimal regressor  $\mathcal{B}q$  (Bellman-completeness condition), the estimation is consistent and unbiased. In contrast, thanks to not using the squared loss, our weighted average Bellman error can be estimated from an unbiased estimate without concern about the double sampling issue, and thus no need for any completeness-type conditions.
4 Information-Theoretic Results
In this section, we provide theoretical analyses of our algorithm, which reveals the advantages of the proposed policy optimization method from a technical standpoint.
Notably, to the best of our knowledge, Theorem 4.1 is the first result of regret guarantee under only realizability without requiring any data coverage or completeness-type assumptions. Additionally, in contrast to most existing works that assume finite function classes, we carefully quantify the space complexities for infinite function classes (e.g., a class of real-valued functions generated by neural networks) using Pollard's pseudo-dimension [38]. The formal definition is provided in Appendix. It notices that the pseudo-dimension is a generalization of the well-known VC dimension [50]. In the following, we first introduce the necessary assumptions before presenting the guarantees for our algorithm.
Assumption 1 (Realizability for  $q$ -function class). For any policy  $\pi \in \Pi$ , we have  $q^{\pi} \in \mathcal{Q}$ . When this assumption holds approximately, we measure violation by  $\inf_{q \in \mathcal{Q}} \sup_{\rho} \mathbb{E}_{\rho}[(q(s,a) - \mathcal{B}^{\pi}q(s,a))^2] \leq \varepsilon_{\mathcal{Q}}$ , where  $\varepsilon_{\mathcal{Q}} \geq 0$  and  $\rho$  is some data distribution such that  $\rho \in \{d_{\widetilde{\pi}} : \widetilde{\pi} \in \Pi\}$ .
We would like to emphasize that we do not require Bellman-completeness condition [53, 58], which is much stronger than the realizability condition. In addition, we do not impose realizability on the importance-weight class  $\Omega$ , thereby allowing model misspecification on  $\Omega$ . Having stated the major assumptions, we now turn to the routine ones on boundedness before presenting the main results.
Assumption 2 (Boundedness on  $\mathcal{Q}$ ). There exists a non-negative constant  $\bar{V} < \infty$ , the function  $q(s, a) \in [0, \bar{V}]$ ,  $\forall q \in \mathcal{Q}$ ,  $s \in S$ ,  $a \in \mathcal{A}$ .