SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanlin Zhu  
EECS, UC Berkeley  
hanlinzhu@berkeley.edu
Paria Rashidinejad  
EECS, UC Berkeley  
paria.rashidinejad@berkeley.edu
Jiantao Jiao  
EECS and Statistics, UC Berkeley  
jiantao@berkeley.edu
We propose A-Crab (Actor-Critic Regularized by Average Bellman error), a new practical algorithm for offline reinforcement learning (RL) in complex environments with insufficient data coverage. Our algorithm combines the marginalized importance sampling framework with the actor-critic paradigm, where the critic returns evaluations of the actor (policy) that are pessimistic relative to the offline data and have a small average (importance-weighted) Bellman error. Compared to existing methods, our algorithm simultaneously offers a number of advantages: (1) It achieves the optimal statistical rate of  $1 / \sqrt{N}$  —where  $N$  is the size of offline dataset—in converging to the best policy covered in the offline dataset, even when combined with general function approximators. (2) It relies on a weaker average notion of policy coverage (compared to the  $\ell_{\infty}$  single-policy concentrability) that exploits the structure of policy visitations. (3) It outperforms the data-collection behavior policy over a wide range of specific hyperparameters. We provide both theoretical analysis and experimental results to validate the effectiveness of our proposed algorithm. The code is available at https://github.com/zhuh198/ACrab.
Offline reinforcement learning (RL) algorithms aim at learning a good policy based only on historical interaction data. This paradigm allows for leveraging previously-collected data in learning policies while avoiding possibly costly and dangerous trial and errors and finds applications in a wide range of domains from precision medicine (Tang et al., 2022) to robotics (Sinha et al., 2022) to climate (Rolnick et al., 2022). Despite wide applicability, offline RL has yet to achieve success stories akin to those observed in online settings that allow for trials and errors (Mnih et al., 2013; Silver et al., 2016; Ran et al., 2019; Mirhoseini et al., 2020; Oh et al., 2020; Fawzi et al., 2022; Degrave et al., 2022).
Enabling offline RL for complex real-world problems requires developing algorithms that first, handle complex high-dimensional observations and second, have minimal requirements on the data coverage and "best" exploit the information available in data. Powerful function approximators such as deep neural networks are observed to be effective in handling complex environments and deep RL algorithms have been behind the success stories mentioned above. This motivates us to investigate provably optimal offline RL algorithms that can be combined with general function approximators and have minimal requirements on the coverage and size of the offline dataset.
In RL theory, the data coverage requirements are often characterized by concentrability definitions (Munos, 2007; Scherrer, 2014). For a policy  $\pi$ , the ratio of the state-action occupancy distribution  $d^{\pi}$  of  $\pi$  to the dataset distribution  $\mu$ , denoted by  $w^{\pi} = d^{\pi} / \mu$ , is used to define concentrability. The most widely-used definition is  $\ell_{\infty}$  concentrability, defined as the infinite norm of  $w^{\pi}$ , i.e.,  $C_{\ell_{\infty}}^{\pi} = \| w^{\pi}\|_{\infty}$ . Many earlier works on offline RL require all-policy  $\ell_{\infty}$  concentrability (i.e.,  $C_{\ell_{\infty}}^{\pi}$  is bounded for all candidate policy  $\pi$ ) (Scherrer, 2014; Liu et al., 2019a; Chen and Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a) or stronger assumptions such as a uniform lower bound on  $\mu(a|s)$  (Xie and Jiang, 2021). However, such all-policy concentrability assumptions are often violated in practical scenarios, and in most cases, only partial dataset coverage is guaranteed.
To deal with partial data coverage, recent works use conservative algorithms, which try to avoid policies not well-covered by the dataset, to learn a good policy with much weaker dataset coverage requirements (Kumar et al., 2020; Jin et al., 2021). In particular, algorithms developed based on the principle of pessimism in the face of uncertainty are shown to find the best covered policy (or sometimes they require coverage of the optimal policy) (e.g., Rashidinejad et al. (2021, 2022); Zhan et al. (2022); Chen and Jiang (2022)). However, most of these works use  $\ell_{\infty}$  concentrability to characterize the dataset coverage. This could still be restrictive even if we only require single-policy concentrability, since the  $\ell_{\infty}$  definition characterizes coverage in terms of the worst-case maximum ratio over all states and actions. Other milder variants of the single-policy concentrability coefficient are proposed by Xie et al. (2021); Uehara and Sun (2021) which consider definitions that exploit the structure of the function class to reduce the coverage requirement and involve taking a maximum over the functions in the hypothesis class instead of all states and actions. However, as we show in Section 2.4, when the function class is very expressive, these variants will degenerate to  $\ell_{\infty}$  concentrability. Moreover, previous algorithms requiring milder variants of single-policy concentrability are either computationally intractable (Xie et al., 2021; Uehara and Sun, 2021) or suffer a suboptimal rate of suboptimality (Cheng et al., 2022). Therefore, a natural and important question is raised:
Is there a computationally efficient and statistically optimal algorithm that can be combined with general function approximators and have minimal requirements on dataset coverage?
We answer this question affirmatively by proposing a novel algorithm named A-Crab (Actor-Critic Regularized by Average Bellman error). We also discuss more related works in Appendix A.
Table 1: Comparison of provable offline RL algorithms with general function approximation.
Algorithm	Computation	Any covered policy	Coverage assumption	Policy improvement	Suboptimality
Xie et al. (2021)	Intractable	Yes	single-policy, CπBellman	Yes	O(1/√N)
Uehara and Sun (2021)	Intractable	Yes	single-policy, ℓ∞ and ℓ2	Yes	O(1/√N)
Chen and Jiang (2022)	Intractable	No	single-policy, ℓ∞	No	O(1/√Ngap(Q*)
Zhan et al. (2022)	Efficient	Yes	two-policy, ℓ∞	Yes	O(1/N1/6)
Cheng et al. (2022)	Efficient	Yes	single-policy, CπBellman	Yes & Robust	O(1/N1/3)
Rashidinejad et al. (2022)	Efficient	No	single-policy, ℓ∞	No	O(1/√N)
Ozdaglar et al. (2022)	Efficient	No	single-policy, ℓ∞	No	O(1/√N)
A-Crab (this work)	Efficient	Yes	single-policy, ℓ2	Yes & Robust	O(1/√N)
In this paper, we build on the adversarially trained actor-critic (ATAC) algorithm of Cheng et al. (2022) and combine it with the marginalized importance sampling (MIS) framework (Xie et al., 2019; Chen and Jiang, 2022; Rashidinejad et al., 2022). In particular, we replace the squared Euclidean norm of the Bellman-consistency error term in the critic's objective of the ATAC algorithm with an importance-weighted average Bellman error term. We prove that this simple yet critical modification of the ATAC algorithm enjoys the properties highlighted below (see Table 1 for comparisons with previous works).
Figure 1: An intuitive comparison of  $C_{\ell_{\infty}}^{\pi}$ ,  $C_{\ell_2}^{\pi}$  and  $C_{\mathrm{Bellman}}^{\pi}$  for a fixed policy  $\pi$ . Note that  $C_{\ell_{\infty}}^{\pi}$ ,  $C_{\ell_2}^{\pi}$  are independent of the function class  $\mathcal{F}$ , and  $C_{\ell_2}^{\pi}$  is always smaller than  $C_{\ell_{\infty}}^{\pi}$ .  $C_{\mathrm{Bellman}}^{\pi}$  grows as  $\mathcal{F}$  gets richer and converges to  $C_{\ell_{\infty}}^{\pi}$ .
Notation. We use  $\Delta(\mathcal{X})$  to denote the probability simplex over a set  $\mathcal{X}$  and use  $\mathrm{Unif}(\mathcal{X})$  to denote the uniform distribution over  $\mathcal{X}$ . We denote by  $\| \cdot \|_{2,\mu} = \sqrt{\mathbb{E}_{\mu}[(\cdot)^2]}$  the Euclidean norm weighted
by distribution  $\mu$ . We use the notation  $x \lesssim y$  when there exists a constant  $c > 0$  such that  $x \leq cy$  and  $x \gtrsim y$  if  $y \lesssim x$  and denote  $x \asymp y$  if  $x \lesssim y$  and  $y \lesssim x$  hold simultaneously. We also use standard  $O(\cdot)$  notation to hide constants and use  $\tilde{O}(\cdot)$  to suppress logarithmic factors.
An infinite-horizon discounted MDP is described by a tuple  $M = (\mathcal{S},\mathcal{A},P,R,\gamma ,\rho)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $P:S\times \mathcal{A}\to \Delta (\mathcal{S})$  is the transition kernel,  $R:S\times \mathcal{A}\rightarrow \Delta ([0,1])$  encodes a family of reward distributions with  $r:S\times \mathcal{A}\rightarrow [0,1]$  as the expected reward function,  $\gamma \in [0,1)$  is the discount factor and  $\rho :S\rightarrow [0,1]$  is the initial state distribution. We assume  $\mathcal{A}$  is finite while allowing  $\mathcal{S}$  to be arbitrarily complex. A stationary (stochastic) policy  $\pi :S\rightarrow \Delta (\mathcal{A})$  specifies a distribution over actions in each state. Each policy  $\pi$  induces a (discounted) occupancy density over state-action pairs  $d^{\pi}:S\times \mathcal{A}\to [0,1]$  defined as  $d^{\pi}(s,a)\coloneqq (1 - \gamma)\sum_{t = 0}^{\infty}\gamma^{t}P_{t}(s_{t} = s,a_{t} = a;\pi)$ , where  $P_{t}(s_{t} = s,a_{t} = a;\pi)$  denotes the visitation probability of state-action pair  $(s,a)$  at time step  $t$ , starting at  $s_0\sim \rho (\cdot)$  and following  $\pi$ . We also write  $d^{\pi}(s) = \sum_{a\in \mathcal{A}}d^{\pi}(s,a)$  to denote the (discounted) state occupancy, and use  $\mathbb{E}_{\pi}[\cdot ]$  as a shorthand of  $\mathbb{E}_{(s,a)\sim d^{\pi}}[\cdot ]$  or  $\mathbb{E}_{s\sim d^{\pi}}[\cdot ]$ .
The value function of a policy  $\pi$  is the discounted cumulative reward gained by executing that policy  $V^{\pi}(s) \coloneqq \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} r_{t} \mid s_{0} = s, a_{t} \sim \pi(\cdot \mid s_{t}), \forall t \geq 0\right]$  starting at state  $s \in S$  where  $r_{t} = R(s_{t}, a_{t})$ . Similarly, we define  $Q$  function of a policy as the expected cumulative reward gained by executing that policy starting from a state-action pair  $(s, a)$ , i.e.,  $Q^{\pi}(s, a) \coloneqq \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} r_{t} \mid s_{0} = s, a_{0} = a, a_{t} \sim \pi(\cdot \mid s_{t}), \forall t > 0\right]$ . We write  $J(\pi) \coloneqq (1 - \gamma) \mathbb{E}_{s \sim \rho}[V^{\pi}(s)] = \mathbb{E}_{(s, a) \sim d^{\pi}}[r(s, a)]$  to represent the (normalized) average value of policy  $\pi$ . We denote by  $\pi^{\star}$  the optimal policy that maximizes the above objective and use the shorthand  $V^{\star} \coloneqq V^{\pi^{\star}}$ ,  $Q^{\star} \coloneqq Q^{\pi^{\star}}$  to denote the optimal value function and optimal  $Q$  function respectively.
2.2 Function approximation
In modern RL, the state space  $S$  is usually large or infinite, making the classic tabular RL algorithms not scalable since their sample complexity depends on the cardinality of  $S$ . Therefore, (general) function approximation is necessary for real-world scenarios with huge state space. In this paper, we assume access to three function classes: a function class  $\mathcal{F} \subseteq \{f : S \times \mathcal{A} \to [0, V_{\max}]\}$  that models the (approximate) Q-functions of policies, a function class  $\mathcal{W} \subseteq \{w : S \times \mathcal{A} \to [0, B_w]\}^1$  that represents marginalized importance weights with respect to data distribution, and a policy class  $\Pi \subseteq \{\pi : S \to \Delta(\mathcal{A})\}$  consisting of candidate policies. Our framework combines the marginalized importance sampling framework (e.g., Zhan et al. (2022); Rashidinejad et al. (2022)) with actor-critic methods (Xie et al., 2021; Cheng et al., 2022), which improves and selects among a set of candidate policies by successive computation of their Q-functions.
For any function  $f \in \mathcal{F}$  and any policy  $\pi \in \Pi$ , we denote  $f(s, \pi) = \sum_{a \in \mathcal{A}} \pi(a|s)f(s, a)$  for any  $s \in S$  and denote Bellman operator  $T^{\pi}: \mathbb{R}^{S \times \mathcal{A}} \to \mathbb{R}^{S \times \mathcal{A}}$  as
$$
\left(\mathcal {T} ^ {\pi} f\right) (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ f (s ^ {\prime}, \pi) ]. \tag {1}
$$
Note that solving the fixed point equation (1) for  $f$  finds the  $Q$ -function of policy  $\pi$ .
We make the following assumption on the expressivity of our function classes.
Assumption 1 (Approximate Realizability). Assume there exists  $\epsilon_{\mathcal{F}} \geq 0$ , s.t. for any policy  $\pi \in \Pi$ ,  $\min_{f \in \mathcal{F}} \max_{admissible \nu} \| f - T^{\pi} f \|_{2,\nu}^{2} \leq \epsilon_{\mathcal{F}}$ , where admissible  $\nu$  is defined by  $\nu \in \{d^{\pi} | \pi \in \Pi\}$ .
This assumption is also required for Xie et al. (2021); Cheng et al. (2022). Note that when  $\| f - \mathcal{T}^{\pi}f\|_{2,\nu}$  is small for all admissible  $\nu$ , we have  $f \approx Q^{\pi}$ . Therefore, Assumption 1 assumes that for any policy  $\pi$ ,  $Q^{\pi}$  is "approximately" realized in  $\mathcal{F}$ . In particular, when  $\epsilon_{\mathcal{F}} = 0$ , Assumption 1 is equivalent to  $Q^{\pi} \in \mathcal{F}$  for any  $\pi \in \Pi$ .
2.3 Offline reinforcement learning
In this paper, we study offline RL where we assume access only to a previously-collected and fixed dataset of interactions  $\mathcal{D} = \{(s_i,a_i,r_i,s_i')\}_{i=1}^N$ , where  $r_i \sim R(s_i,a_i)$ ,  $s_i' \sim P(\cdot \mid s_i,a_i)$ .
Without loss of generality, we always assume that the all-one function is contained in  $\mathcal{W}$ .
To streamline the analysis, we assume that  $(s_i,a_i)$  pairs are generated i.i.d. according to a data distribution  $\mu \in \Delta (\mathcal{S}\times \mathcal{A})$ . We make the common assumption that the dataset is collected by a behavior policy, i.e.,  $\mu$  is the discounted visitation probability of a behavior policy, which we also denote by  $\mu$ . For convenience, we assume the behavior policy  $\mu \in \Pi$ . The goal of offline RL is to learn a good policy  $\hat{\pi}$  (a policy with a high  $J(\hat{\pi})$ ) using the offline dataset. Also, for any function  $f$  that takes  $(s,a,r,s')$  as input, we define the expectation w.r.t. the dataset  $\mathcal{D}$  (or empirical expectation) as  $\mathbb{E}_{\mathcal{D}}[f] = \frac{1}{N}\sum_{(s_i,a_i,r_i,s_i')\in \mathcal{D}}f(s_i,a_i,r_i,s_i')$ .
Marginalized importance weights. We define the marginalized importance weights of any policy  $\pi$  to be the ratio between the discounted state-action occupancy of  $\pi$  and the data distribution  $w^{\pi}(s,a) \coloneqq \frac{d^{\pi}(s,a)}{\mu(s,a)}$ . Such weights have been defined in prior works on theoretical RL (Xie and Jiang, 2020; Zhan et al., 2022; Rashidinejad et al., 2022; Ozdaglar et al., 2022) as well as practical RL algorithms (Nachum et al., 2019a,b; Zhang et al., 2020b,c; Lee et al., 2021).
2.4 Coverage of offline dataset
We study offline RL with access to a dataset with partial coverage. We measure the coverage of policy  $\pi$  in the dataset using the weighted  $\ell_2$  single-policy concentrability coefficient defined below.
Definition 1 ( $\ell_2$  concentrability). Given a policy  $\pi$ , define  $C_{\ell_2}^{\pi} = \| w^{\pi}\|_{2,\mu} = \| d^{\pi} / \mu \|_{2,\mu}$ .
This definition is much weaker than the all-policy concentrability conventionally used in offline RL (Scherrer, 2014; Liu et al., 2019a; Chen and Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a), which requires the ratio  $\frac{d^{\pi}(s,a)}{\mu(s,a)}$  to be bounded for all  $s\in S$  and  $a\in \mathcal{A}$  as well as all policies  $\pi$ . The following proposition compares two variants of single-policy concentrability definition that appeared in recent works Rashidinejad et al. (2021); Xie et al. (2021) with the  $\ell_2$  variant defined in Definition 1; see Appendix A.1 for more discussion on different concentrability definitions in prior work. To our knowledge, the  $\ell_2$  version of concentrability definition has been only used in offline RL with all-policy coverage (Farahmand et al., 2010; Xie and Jiang, 2020). In the context of partial coverage, Uehara and Sun (2021) used  $\ell_2$  version in a model-based setting, but their algorithms are computationally intractable. Recent works Xie et al. (2021); Cheng et al. (2022) use another milder version of concentrability than  $\ell_{\infty}$ , and we compare different concentrability versions in Proposition 2.1. An intuitive comparison is presented in Figure 1.
Proposition 2.1 (Comparing concentrability definitions). Define the  $\ell_{\infty}$  single-policy concentrability (Rashidinejad et al., 2021) as  $C_{\ell_{\infty}}^{\pi} = \| d^{\pi} / \mu \|_{\infty}$  and the Bellman-consistent single-policy concentrability (Xie et al., 2021) as  $C_{Bellman}^{\pi} = \max_{f\in \mathcal{F}}\frac{\|f - \mathcal{T}^{\pi}f\|_{2,d}^{2}}{\|f - \mathcal{T}^{\pi}f\|_{2,\mu}^{2}}$ . Then, it always holds  $(C_{\ell_2}^{\pi})^2\leq C_{\ell_{\infty}}^{\pi}$ ,  $C_{\ell_2}^{\pi}\leq C_{\ell_{\infty}}^{\pi}$  and there exist offline RL instances where  $(C_{\ell_2}^{\pi})^2\leq C_{Bellman}^{\pi}$ ,  $C_{\ell_2}^{\pi}\leq C_{Bellman}^{\pi}$ .
A proof for Proposition 2.1 is presented in Appendix B. It is easy to show that the  $\ell_2$  variant is bounded by  $\ell_{\infty}$  variant of concentrability as the former requires  $\mathbb{E}_{d^{\pi}}[w^{\pi}(s,a)]$  to be bounded while the latter requires  $w^{\pi}(s,a)$  to be bounded for any  $s\in S$  and  $a\in \mathcal{A}$ . Example 1 provides a concrete example that  $C_{\ell_2}^{\pi}$  is bounded by a constant while  $C_{\ell_{\infty}}^{\pi}$  could be arbitrarily large.
Example 1 (Arbitrarily large  $\ell_{\infty}$  concentrability with a constant  $\ell_{2}$  concentrability). Consider the simplest two-arm bandit settings, where the dataset distribution is  $\mu(a_{1}) = 1 - \epsilon^{2}$ ,  $\mu(a_{2}) = \epsilon^{2}$  for an arbitrarily small  $\epsilon > 0$ . Let  $\pi$  be a policy s.t.  $\pi(a_{1}) = d^{\pi}(a_{1}) = 1 - \epsilon$ ,  $\pi(a_{2}) = d^{\pi}(a_{2}) = \epsilon$ . Then one can calculate that  $w^{\pi}(a_{1}) = \frac{1 - \epsilon}{1 - \epsilon^{2}} \leq 1$  and  $w^{\pi}(a_{2}) = \frac{1}{\epsilon}$ . Therefore,  $C_{\ell_{2}}^{\pi} \leq \sqrt{2}$  while  $C_{\ell_{\infty}}^{\pi} = \frac{1}{\epsilon}$  can be arbitrarily large.
Furthermore, the Bellman-consistent variant can exploit the structure in the Q-function class  $\mathcal{F}$  for a smaller concentrability coefficient. However, in situations where the class  $\mathcal{F}$  is highly expressive,  $C_{\mathrm{Bellman}}^{\pi}$  could be close to  $C_{\ell_{\infty}}^{\pi}$  and thus possibly larger than  $C_{\ell_2}^{\pi}$ .
Finally, we make a boundedness assumption on our marginalized importance weight function class  $\mathcal{W}$  in terms of  $\ell_2$  concentrability and a single-policy realizability assumption.
Assumption 2 (Boundedness in  $\ell_2$  norm of  $\mathcal{W}$ ). Assume  $\| w\|_{2,\mu} \leq C_{\ell_2}^{\star}$  for all  $w \in \mathcal{W}$ .
Assumption 3 (Single-policy realizability of  $w^{\pi}$ ). Assume  $w^{\pi} \in \mathcal{W}$  for some policy  $\pi \in \Pi$  that we aim to compete with.
The definition of  $C_{\ell_2}^{\star}$  is similar to Xie and Jiang (2020) but they need  $w^{\pi} \in \mathcal{W}$  for all  $\pi \in \Pi$ , which is much stronger than our single-policy realizability assumption for  $\mathcal{W}$ .
3 Actor-Critic Regularized by Average Bellman Error
In this section, we introduce our main algorithm named A-Crab (Actor-Critic Regularized by Average Bellman error, Algorithm 1), and compare it with the previous ATAC algorithm (Cheng et al., 2022). In Section 4, we will provide theoretical guarantees of A-Crab and discuss its advantages.
3.1 From Actor-Critic to A-Crab
Our algorithm design builds upon the actor-critic method, in which we iteratively evaluate a policy and improve the policy based on the evaluation. Consider the following actor-critic example:
$$
\hat {\pi} ^ {*} \in \arg \max  _ {\pi \in \Pi} f ^ {\pi} (s _ {0}, \pi), \quad f ^ {\pi} \in \arg \min  _ {f \in \mathcal {F}} \mathbb {E} _ {\mu} [ ((f - \mathcal {T} ^ {\pi} f) (s, a)) ^ {2} ],
$$
where we assume  $s_0$  is the fixed initial state in this example and recall that  $f(s, \pi) = \sum_{a \in \mathcal{A}} \pi(a|s)f(s,a)$ . Here, the policy is evaluated by the function that minimizes the squared Bellman error. However, insufficient data coverage may lead the critic to give an unreliable evaluation of the policy. To address this, the critic can compute a Bellman-consistent pessimistic evaluation of  $\pi$  (Xie et al., 2021), which picks the most pessimistic  $f \in \mathcal{F}$  that approximately satisfies the Bellman equation. Introducing a hyperparameter  $\beta \geq 0$  to tradeoff between pessimism and Bellman consistency yields the following criteria for the critic:
$$
f ^ {\pi} \in \arg \min  _ {f \in \mathcal {F}} f (s _ {0}, \pi) + \beta \mathbb {E} _ {\mu} [ ((f - \mathcal {T} ^ {\pi} f) (s, a)) ^ {2} ].
$$
Cheng et al. (2022) argue that instead of the above absolute pessimism, a relative pessimism approach of optimizing the performance of  $\pi$  relative to the behavior policy, results in an algorithm that improves over the behavior policy for any  $\beta \geq 0$  (i.e., robust policy improvement). Incorporating relative pessimism in the update rule gives the ATAC algorithm (Cheng et al., 2022):
$$
\hat {\pi} ^ {*} \in \arg \max _ {\pi \in \Pi} \mathbb {E} _ {\mu} [ f ^ {\pi} (s, \pi) - f ^ {\pi} (s, a) ],
$$
$$
f ^ {\pi} \in \arg \min  _ {f \in \mathcal {F}} \mathbb {E} _ {\mu} [ f (s, \pi) - f (s, a) ] + \beta \mathbb {E} _ {\mu} [ ((f - \mathcal {T} ^ {\pi} f) (s, a)) ^ {2} ].
$$
Finally, we introduce the importance weights  $w(s,a)$  and change the squared Bellman regularizer to an importance-weighted average Bellman error to arrive at:
$$
\hat {\pi} ^ {\star} \in \arg \max  _ {\pi \in \Pi} \mathcal {L} _ {\mu} (\pi , f ^ {\pi}), \quad \text {s . t .} f ^ {\pi} \in \arg \min  _ {f \in \mathcal {F}} \mathcal {L} _ {\mu} (\pi , f) + \beta \mathcal {E} _ {\mu} (\pi , f), \tag {2}
$$
$$
\mathcal {L} _ {\mu} (\pi , f) = \mathbb {E} _ {\mu} [ f (s, \pi) - f (s, a) ], \tag {3}
$$
$$
\mathcal {E} _ {\mu} (\pi , f) = \max  _ {w \in \mathcal {W}} | \mathbb {E} _ {\mu} [ w (s, a) (f - \mathcal {T} ^ {\pi} f) (s, a) ] |. \tag {4}
$$
Maximization over  $w$  in the importance-weighted average Bellman regularizer in (4) ensures that the Bellman error is small when averaged over measure  $\mu \cdot w$  for any  $w \in \mathcal{W}$ , which turns out to be sufficient to control the suboptimality of the learned policy as the performance difference decomposition Lemma 1 shows.
Squared Bellman error vs. importance-weighted average Bellman error. Unlike our approach, the ATAC algorithm in Cheng et al. (2022) uses squared Bellman error, wherein direct empirical approximation leads to overestimating the regularization term. To obtain an unbiased empirical estimator, Cheng et al. (2022) uses  $\mathbb{E}_{\mathcal{D}}\left[(f(s,a) - r - \gamma f(s',\pi))^{2}\right] - \min_{g\in \mathcal{F}}\mathbb{E}_{\mathcal{D}}\left[(g(s,a) - r - \gamma f(s',\pi))^{2}\right]$  as the empirical estimator which subtracts the overestimation. Yet, as we later see in Proposition 4.1, even with this correction, ATAC fails to achieve the optimal statistical rate of  $1 / \sqrt{N}$  in certain offline learning instances. In contrast, the importance-weighted average Bellman error in our algorithm is unbiased (as it involves no non-linearity). This makes our theoretical analysis much simpler and leads to achieving an optimal statistical rate of  $1 / \sqrt{N}$  as shown in Theorem 1.
<sup>2</sup>Such importance-weighted minimax formulations of Bellman error have been used in prior work on off-policy evaluation (Uehara et al., 2020) and offline RL with all-policy coverage (Xie and Jiang, 2020).
This is closely related to the infamous double-sampling issue; see Section 3.1 in Chen and Jiang (2019) for a detailed discussion.
3.2 Main algorithms