Title: Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: offline RL

actor-critic

l_2 single-policy concentrability

average bellman error
Abstract: We propose A-Crab (Actor-Critic Regularized by Average Bellman error), a new practical algorithm for offline reinforcement learning (RL) in complex environments with insufficient data coverage. Our algorithm combines the marginalized importance sampling framework with the actor-critic paradigm, where the critic returns evaluations of the actor (policy) that are pessimistic relative to the offline data and have a small average (importance-weighted) Bellman error. Compared to existing methods, our algorithm simultaneously offers a number of advantages:
(1) It achieves the optimal statistical rate of $1/\sqrt{N}$---where $N$ is the size of offline dataset---in converging to the best policy covered in the offline dataset, even when combined with general function approximators.
(2) It relies on a weaker \textit{average} notion of policy coverage (compared to the $\ell_\infty$ single-policy concentrability) that exploits the structure of policy visitations.
(3) It outperforms the data-collection behavior policy over a wide range of specific hyperparameters.
 We provide both theoretical analysis and experimental results to validate the effectiveness of our proposed algorithm. The code is available at https://github.com/zhuhl98/ACrab.

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Abstract

We propose A-Crab (Actor-Critic Regularized by Average Bellman error), a new practical algorithm for offline reinforcement learning (RL) in complex environments with insufficient data coverage. Our algorithm combines the marginalized importance sampling framework with the actor-critic paradigm, where the critic returns evaluations of the actor (policy) that are pessimistic relative to the offline data and have a small average (importance-weighted) Bellman error. Compared to existing methods, our algorithm simultaneously offers a number of advantages: (1) It achieves the optimal statistical rate of 1 / √ N -where N is the size of offline dataset-in converging to the best policy covered in the offline dataset, even when combined with general function approximators. (2) It relies on a weaker average notion of policy coverage (compared to the ℓ ∞ single-policy concentrability) that exploits the structure of policy visitations. (3) It outperforms the data-collection behavior policy over a wide range of specific hyperparameters. We provide both theoretical analysis and experimental results to validate the effectiveness of our proposed algorithm. The code is available at https://github.com/zhuhl98/ACrab .

1 Introduction

Offline reinforcement learning (RL) algorithms aim at learning a good policy based only on historical interaction data. This paradigm allows for leveraging previously-collected data in learning policies while avoiding possibly costly and dangerous trial and errors and finds applications in a wide range of domains from precision medicine (Tang et al., 2022) to robotics (Sinha et al., 2022) to climate (Rolnick et al., 2022). Despite wide applicability, offline RL has yet to achieve success stories akin to those observed in online settings that allow for trials and errors (Mnih et al., 2013; Silver et al., 2016; Ran et al., 2019; Mirhoseini et al., 2020; Oh et al., 2020; Fawzi et al., 2022; Degrave et al., 2022).

Enabling offline RL for complex real-world problems requires developing algorithms that first, handle complex high-dimensional observations and second, have minimal requirements on the data coverage and 'best' exploit the information available in data. Powerful function approximators such as deep neural networks are observed to be effective in handling complex environments and deep RL algorithms have been behind the success stories mentioned above. This motivates us to investigate provably optimal offline RL algorithms that can be combined with general function approximators and have minimal requirements on the coverage and size of the offline dataset.

In RL theory, the data coverage requirements are often characterized by concentrability definitions (Munos, 2007; Scherrer, 2014). For a policy π , the ratio of the state-action occupancy distribution d π of π to the dataset distribution µ , denoted by w π = d π /µ , is used to define concentrability. The most widely-used definition is ℓ ∞ concentrability, defined as the infinite norm of w π , i.e., C π ℓ ∞ = ∥ w π ∥ ∞ . Many earlier works on offline RL require all-policy ℓ ∞ concentrability (i.e., C π ℓ ∞ is bounded for all candidate policy π ) (Scherrer, 2014; Liu et al., 2019a; Chen and Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a) or stronger assumptions such as a uniform lower bound on µ ( a | s ) (Xie and Jiang, 2021). However, such all-policy concentrability assumptions are often violated in practical scenarios, and in most cases, only partial dataset coverage is guaranteed.

To deal with partial data coverage, recent works use conservative algorithms, which try to avoid policies not well-covered by the dataset, to learn a good policy with much weaker dataset coverage requirements (Kumar et al., 2020; Jin et al., 2021). In particular, algorithms developed based on the principle of pessimism in the face of uncertainty are shown to find the best covered policy (or sometimes they require coverage of the optimal policy) (e.g., Rashidinejad et al. (2021, 2022); Zhan et al. (2022); Chen and Jiang (2022)). However, most of these works use ℓ ∞ concentrability to characterize the dataset coverage. This could still be restrictive even if we only require singlepolicy concentrability, since the ℓ ∞ definition characterizes coverage in terms of the worst-case maximum ratio over all states and actions. Other milder variants of the single-policy concentrability coefficient are proposed by Xie et al. (2021); Uehara and Sun (2021) which consider definitions that exploit the structure of the function class to reduce the coverage requirement and involve taking a maximum over the functions in the hypothesis class instead of all states and actions. However, as we show in Section 2.4, when the function class is very expressive, these variants will degenerate to ℓ ∞ concentrability. Moreover, previous algorithms requiring milder variants of single-policy concentrability are either computationally intractable (Xie et al., 2021; Uehara and Sun, 2021) or suffer a suboptimal rate of suboptimality (Cheng et al., 2022). Therefore, a natural and important question is raised:

Is there a computationally efficient and statistically optimal algorithm that can be combined with general function approximators and have minimal requirements on dataset coverage?

We answer this question affirmatively by proposing a novel algorithm named A-Crab (Actor-Critic Regularized by Average Bellman error). We also discuss more related works in Appendix A.

1.1 Contributions

In this paper, we build on the adversarially trained actor-critic (ATAC) algorithm of Cheng et al. (2022) and combine it with the marginalized importance sampling (MIS) framework (Xie et al., 2019; Chen and Jiang, 2022; Rashidinejad et al., 2022). In particular, we replace the squared Euclidean norm of the Bellman-consistency error term in the critic's objective of the ATAC algorithm with an importance-weighted average Bellman error term. We prove that this simple yet critical modification of the ATAC algorithm enjoys the properties highlighted below (see Table 1 for comparisons with previous works).

F

complexity of

concentrability

C

π

ℓ

∞

C

π

ℓ

2

C

π

Bellman

1. Optimal statistical rate in competing with the best covered policy: In Theorem 1, we prove that our A-Crab algorithm, which uses average Bellman error, enjoys an optimal statistical rate of 1 / √ N . In contrast, we prove in Proposition 4.1 that the ATAC algorithm, which uses the squared Bellman error, fails to achieve the optimal rate in certain offline learning instances. As Cheng et al. (2022) explains, the squared Bellman-error regularizer appears to be the culprit behind the suboptimal rate of ATAC being 1 /N 1 / 3 . Moreover, our algorithm improves over any policy covered in the data. This is in contrast to the recent work Rashidinejad et al. (2022), which proposes an algorithm based on the MIS framework that achieves the 1 / √ N rate only when the optimal policy (i.e. the policy with the highest expected rewards) is covered in the data.

2. A weaker notion of data coverage that exploits visitation structure: As we discussed earlier, ℓ ∞ concentrability notion is used in many prior works (Rashidinejad et al., 2021; Chen and Jiang, 2022; Ozdaglar et al., 2022; Rashidinejad et al., 2022). Our importance-weighted average Bellman error as well as using Bernstein inequality in the proof relies on guarantees in terms of an ℓ 2 single-policy concentrability notion that is weaker than the ℓ ∞ variant. In particular, we have C π ℓ ∞ = ∥ w π ∥ ∞ and C π ℓ 2 = ∥ w π ∥ 2 ,µ , where ∥ · ∥ 2 ,µ is the weighted 2-norm w.r.t. the dataset distribution µ . The latter implies that the coverage coefficient only matters as much as it is covered by the dataset. Moreover, by the definition of w π , we can obtain that ( C π ℓ 2 ) 2 = E d π [ w π ( s, a )] , which provides another explanation of ℓ 2 concentrability that the coverage coefficient only matters as much as it is actually visited by the policy. There are also other notions of single-policy concentrability exploiting function approximation structures to make the coverage coefficient smaller (e.g., C π Bellman in Xie et al. (2021)). However, these notions degenerate to C π ℓ ∞ as the function class gets richer (see Figure 1 for an intuitive comparison of different notions of concentrability and Section 2.4 for a rigorous proof).

Robust policy improvement: Policy improvement (PI) refers to the property that an offline RL algorithm (under a careful choice of specific hyperparameters) can always improve upon the behavior policy that is used to collect the data. In particular, robust policy improvement (RPI) means the PI property holds under a wide range of the choice of specific hyperparameters (Cheng et al., 2022; Xie et al., 2022). Similar to the ATAC algorithm in Cheng et al. (2022), our approach enjoys the robust policy improvement guarantee as shown in Theorem 2.

4. Inheriting many other benefits of adversarially-trained actor-critic: Since our algorithm is based on ATAC with a different choice of regularizer, it can be easily implemented as ATAC to be applied to practical scenarios, and we provide experimental results in Section 6. Also, our algorithm is robust to model misspecification and does not require the completeness assumption (Assumption 2 in Cheng et al. (2022)) on the value function class, which makes it more practical. Moreover, our algorithm can learn a policy that outperforms any other policies well covered by the dataset.

2 Background

Notation. We use ∆( X ) to denote the probability simplex over a set X and use Unif ( X ) to denote the uniform distribution over X . We denote by ∥ · ∥ 2 ,µ = √ E µ [( · ) 2 ] the Euclidean norm weighted

by distribution µ . We use the notation x ≲ y when there exists constant c > 0 such that x ≤ cy and x ≳ y if y ≲ x and denote x ≍ y if x ≲ y and y ≲ x hold simultaneously. We also use standard O ( · ) notation to hide constants and use ˜ O ( · ) to suppress logarithmic factors.

2.1 Markov decision process

An infinite-horizon discounted MDP is described by a tuple M = ( S , A , P, R, γ, ρ ) , where S is the state space, A is the action space, P : S ×A → ∆( S ) is the transition kernel, R : S ×A → ∆([0 , 1]) encodes a family of reward distributions with r : S × A → [0 , 1] as the expected reward function, γ ∈ [0 , 1) is the discount factor and ρ : S → [0 , 1] is the initial state distribution. We assume A is finite while allowing S to be arbitrarily complex. A stationary (stochastic) policy π : S → ∆( A ) specifies a distribution over actions in each state. Each policy π induces a (discounted) occupancy density over state-action pairs d π : S × A → [0 , 1] defined as d π ( s, a ) := (1 -γ ) ∑ ∞ t =0 γ t P t ( s t = s, a t = a ; π ) , where P t ( s t = s, a t = a ; π ) denotes the visitation probability of state-action pair ( s, a ) at time step t , starting at s 0 ∼ ρ ( · ) and following π . We also write d π ( s ) = ∑ a ∈A d π ( s, a ) to denote the (discounted) state occupancy, and use E π [ · ] as a shorthand of E ( s,a ) ∼ d π [ · ] or E s ∼ d π [ · ] .

The value function of a policy π is the discounted cumulative reward gained by executing that policy V π ( s ) := E [ ∑ ∞ t =0 γ t r t | s 0 = s, a t ∼ π ( ·| s t ) , ∀ t ≥ 0] starting at state s ∈ S where r t = R ( s t , a t ) . Similarly, we define Q function of a policy as the expected cumulative reward gained by executing that policy starting from a state-action pair ( s, a ) , i.e., Q π ( s, a ) := E [ ∑ ∞ t =0 γ t r t | s 0 = s, a 0 = a, a t ∼ π ( ·| s t ) , ∀ t > 0] . We write J ( π ) := (1 -γ ) E s ∼ ρ [ V π ( s )] = E ( s,a ) ∼ d π [ r ( s, a )] to represent the (normalized) average value of policy π . We denote by π ⋆ the optimal policy that maximizes the above objective and use the shorthand V ⋆ := V π ⋆ , Q ⋆ := Q π ⋆ to denote the optimal value function and optimal Q function respectively.

2.2 Function approximation

In modern RL, the state space S is usually large or infinite, making the classic tabular RL algorithms not scalable since their sample complexity depends on the cardinality of S . Therefore, (general) function approximation is necessary for real-world scenarios with huge state space. In this paper, we assume access to three function classes: a function class F ⊆ { f : S × A → [0 , V max ] } that models the (approximate) Q-functions of policies, a function class W ⊆ { w : S × A → [0 , B w ] } 1 that represents marginalized importance weights with respect to data distribution, and a policy class Π ⊆ { π : S → ∆( A ) } consisting of candidate policies. Our framework combines the marginalized importance sampling framework (e.g., Zhan et al. (2022); Rashidinejad et al. (2022)) with actor-critic methods (Xie et al., 2021; Cheng et al., 2022), which improves and selects among a set of candidate policies by successive computation of their Q-functions.

For any function f ∈ F and any policy π ∈ Π , we denote f ( s, π ) = ∑ a ∈A π ( a | s ) f ( s, a ) for any s ∈ S and denote Bellman operator T π : R S×A → R S×A as

Note that solving the fixed point equation (1) for f finds the Q -function of policy π .

We make the following assumption on the expressivity of our function classes.

Assumption 1 (Approximate Realizability) . Assume there exists ϵ F ≥ 0 , s.t. for any policy π ∈ Π , min f ∈F max admissible ν ∥ f -T π f ∥ 2 2 ,ν ≤ ϵ F , where admissible ν is defined by ν ∈ { d π | π ∈ Π } .

This assumption is also required for Xie et al. (2021); Cheng et al. (2022). Note that when ∥ f -T π f ∥ 2 ,ν is small for all admissible ν , we have f ≈ Q π . Therefore, Assumption 1 assumes that for any policy π , Q π is 'approximatly' realized in F . In particular, when ϵ F = 0 , Assumption 1 is equivalent to Q π ∈ F for any π ∈ Π .

2.3 Offline reinforcement learning

In this paper, we study offline RL where we assume access only to a previously-collected and fixed dataset of interactions D = { ( s i , a i , r i , s ′ i ) } N i =1 , where r i ∼ R ( s i , a i ) , s ′ i ∼ P ( · | s i , a i ) .

To streamline the analysis, we assume that ( s i , a i ) pairs are generated i.i.d. according to a data distribution µ ∈ ∆( S × A ) . We make the common assumption that the dataset is collected by a behavior policy, i.e., µ is the discounted visitation probability of a behavior policy, which we also denote by µ . For convenience, we assume the behavior policy µ ∈ Π . The goal of offline RL is to learn a good policy ˆ π (a policy with a high J (ˆ π ) ) using the offline dataset. Also, for any function f that takes ( s, a, r, s ′ ) as input, we define the expectation w.r.t. the dataset D (or empirical expectation) as E D [ f ] = 1 N ∑ ( s i ,a i ,r i ,s ′ i ) ∈D f ( s i , a i , r i , s ′ i ) .

Marginalized importance weights. We define the marginalized importance weights of any policy π to be the ratio between the discounted state-action occupancy of π and the data distribution w π ( s, a ) := d π ( s,a ) µ ( s,a ) . Such weights have been defined in prior works on theoretical RL (Xie and Jiang, 2020; Zhan et al., 2022; Rashidinejad et al., 2022; Ozdaglar et al., 2022) as well as practical RL algorithms (Nachum et al., 2019a,b; Zhang et al., 2020b,c; Lee et al., 2021).

2.4 Coverage of offline dataset

We study offline RL with access to a dataset with partial coverage. We measure the coverage of policy π in the dataset using the weighted ℓ 2 single-policy concentrability coefficient defined below.

Definition 1 ( ℓ 2 concentrability) . Given a policy π , define C π ℓ 2 = ∥ w π ∥ 2 ,µ = ∥ d π /µ ∥ 2 ,µ .

This definition is much weaker than the all-policy concentrability conventionally used in offline RL (Scherrer, 2014; Liu et al., 2019a; Chen and Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a), which requires the ratio d π ( s,a ) µ ( s,a ) to be bounded for all s ∈ S and a ∈ A as well as all policies π . The following proposition compares two variants of single-policy concentrability definition that appeared in recent works Rashidinejad et al. (2021); Xie et al. (2021) with the ℓ 2 variant defined in Definition 1; see Appendix A.1 for more discussion on different concentrability definitions in prior work. To our knowledge, the ℓ 2 version of concentrability definition has been only used in offline RL with all-policy coverage (Farahmand et al., 2010; Xie and Jiang, 2020). In the context of partial coverage, Uehara and Sun (2021) used ℓ 2 version in a model-based setting, but their algorithms are computationally intractable. Recent works Xie et al. (2021); Cheng et al. (2022) use another milder version of concentrability than ℓ ∞ , and we compare different concentrability versions in Proposition 2.1. An intuitive comparison is presented in Figure 1.

Proposition 2.1 (Comparing concentrability definitions) . Define the ℓ ∞ single-policy concentrability (Rashidinejad et al., 2021) as C π ℓ ∞ = ∥ d π /µ ∥ ∞ and the Bellman-consistent single-policy concentrability (Xie et al., 2021) as C π Bellman = max f ∈F ∥ f -T π f ∥ 2 2 ,d π ∥ f -T π f ∥ 2 2 ,µ . Then, it always holds ( C π ℓ 2 ) 2 ≤ C π ℓ ∞ , C π ℓ 2 ≤ C π ℓ ∞ and there exist offline RL instances where ( C π ℓ 2 ) 2 ≤ C π Bellman , C π ℓ 2 ≤ C π Bellman .

A proof for Proposition 2.1 is presented in Appendix B. It is easy to show that the ℓ 2 variant is bounded by ℓ ∞ variant of concentrability as the former requires E d π [ w π ( s, a )] to be bounded while the latter requires w π ( s, a ) to be bounded for any s ∈ S and a ∈ A . Example 1 provides a concrete example that C π ℓ 2 is bounded by a constant while C π ℓ ∞ could be arbitrarily large.

Example 1 (Arbitrarily large ℓ ∞ concentrability with a constant ℓ 2 concentrability) . Consider the simplest two-arm bandit settings, where the dataset distribution is µ ( a 1 ) = 1 -ϵ 2 , µ ( a 2 ) = ϵ 2 for an arbitrarily small ϵ > 0 . Let π be a policy s.t. π ( a 1 ) = d π ( a 1 ) = 1 -ϵ, π ( a 2 ) = d π ( a 2 ) = ϵ. Then one can calculate that w π ( a 1 ) = 1 -ϵ 1 -ϵ 2 ≤ 1 and w π ( a 2 ) = 1 ϵ . Therefore, C π ℓ 2 ≤ √ 2 while C π ℓ ∞ = 1 ϵ can be arbitrarily large.

Furthermore, the Bellman-consistent variant can exploit the structure in the Q-function class F for a smaller concentrability coefficient. However, in situations where the class F is highly expressive, C π Bellman could be close to C π ℓ ∞ and thus possibly larger than C π ℓ 2 .

Finally, we make a boundedness assumption on our marginalized importance weight function class W in terms of ℓ 2 concentrability and a single-policy realizability assumption.

Assumption 2 (Boundedness in ℓ 2 norm of W ) . Assume ∥ w ∥ 2 ,µ ≤ C ⋆ ℓ 2 for all w ∈ W .

Assumption 3 (Single-policy realizability of w π ) . Assume w π ∈ W for some policy π ∈ Π that we aim to compete with.

The definition of C ⋆ ℓ 2 is similar to Xie and Jiang (2020) but they need w π ∈ W for all π ∈ Π , which is much stronger than our single-policy realizability assumption for W .

3 Actor-Critic Regularized by Average Bellman Error

In this section, we introduce our main algorithm named A-Crab ( A ctorC ritic R egularized by A verage B ellman error, Algorithm 1), and compare it with the previous ATAC algorithm (Cheng et al., 2022). In Section 4, we will provide theoretical guarantees of A-Crab and discuss its advantages.

3.1 From Actor-Critic to A-Crab

Our algorithm design builds upon the actor-critic method, in which we iteratively evaluate a policy and improve the policy based on the evaluation. Consider the following actor-critic example:

where we assume s 0 is the fixed initial state in this example and recall that f ( s, π ) = ∑ a ∈A π ( a | s ) f ( s, a ) . Here, the policy is evaluated by the function that minimizes the squared Bellman error. However, insufficient data coverage may lead the critic to give an unreliable evaluation of the policy. To address this, the critic can compute a Bellman-consistent pessimistic evaluation of π (Xie et al., 2021), which picks the most pessimistic f ∈ F that approximately satisfies the Bellman equation. Introducing a hyperparameter β ≥ 0 to tradeoff between pessimism and Bellman consistency yields the following criteria for the critic:

Cheng et al. (2022) argue that instead of the above absolute pessimism , a relative pessimism approach of optimizing the performance of π relative to the behavior policy, results in an algorithm that improves over the behavior policy for any β ≥ 0 (i.e., robust policy improvement). Incorporating relative pessimism in the update rule gives the ATAC algorithm (Cheng et al., 2022):

Finally, we introduce the importance weights w ( s, a ) and change the squared Bellman regularizer to an importance-weighted average Bellman error to arrive at:

where

Maximization over w in the importance-weighted average Bellman regularizer in (4) ensures that the Bellman error is small when averaged over measure µ · w for any w ∈ W , which turns out to be sufficient to control the suboptimality of the learned policy as the performance difference decomposition Lemma 1 shows. 2

Squared Bellman error v.s. importance-weighted average Bellman error. Unlike our approach, the ATAC algorithm in Cheng et al. (2022) uses squared Bellman error, wherein direct empirical approximation leads to overestimating the regularization term. 3 To obtain an unbiased empirical estimator, Cheng et al. (2022) uses E D [ ( f ( s, a ) -r -γf ( s ′ , π )) 2 ] -min g ∈F E D [ ( g ( s, a ) -r -γf ( s ′ , π )) 2 ] as the empirical estimator which subtracts the overestimation. Yet, as we later see in Proposition 4.1, even with this correction, ATAC fails to achieve the optimal statistical rate of 1 / √ N in certain offline learning instances. In contrast, the importanceweighted average Bellman error in our algorithm is unbiased (as it involves no non-linearity). This makes our theoretical analysis much simpler and leads to achieving an optimal statistical rate of 1 / √ N as shown in Theorem 1.

3.2 Main algorithms

Since we do not have direct access to the dataset distribution µ , our algorithm instead solves an empirical version of (2), which can be formalized as

where

Algorithm 1 A ctorC ritic R egularized by A verage B ellman error (A-CRAB)

1: Input: Dataset D = { ( s i , a i , r i , s ′ i ) } N i =1 , value function class F , importance weight function class W , no-regret policy optimization oracle PO (Definition 2).

2: Initialization: π 1 : uniform policy, β : hyperparameter.

3: for k = 1 , 2 , . . . , K do

4: f k ← arg min f ∈F L D ( π k , f ) + β E D ( π k , f ) , where L D and E D are defined in (6), (7)

5: π k +1 ← PO ( π k , f k , D ) .

6: end for

7: Output: ¯ π = Unif ( { π k } K k =1 ) .

Similar to Cheng et al. (2022), we view program (5) as a Stackelberg game and solve it using a no-regret oracle as shown in Algorithm 1. At each step k , the critic minimizes the objective defined by (7) w.r.t. π k , and π k +1 is generated by a no-regret policy optimization oracle, given below.

Definition 2 (No-regret policy optimization oracle) . An algorithm PO is defined as a no-regret policy optimization oracle if for any (adversarial) sequence of functions f 1 , f 2 , . . . , f K ∈ F where f k : S × A → [0 , V max ] , ∀ k ∈ [ K ] , the policy sequence π 1 , π 2 , . . . , π K produced by PO satisfies that for any policy π ∈ Π , it holds that ϵ π opt ≜ 1 K ∑ K k =1 E π [ f k ( s, π ) -f k ( s, π k )] = o (1) .

Among the well-known instances of the above no-regret policy optimization oracle is natural policy gradient (Kakade, 2001) of the form π k +1 ( a | s ) ∝ π k ( a | s ) exp( ηf k ( s, a )) with η = √ log |A| 2 V 2 max K (EvenDar et al., 2009; Agarwal et al., 2021; Xie et al., 2021; Cheng et al., 2022). A detailed discussion of the above policy optimization oracle can be found in Cheng et al. (2022). Utilizing the no-regret oracle in solving the Stackelberg optimization problem in (5) yields Algorithm 1.

A remark on critic's optimization problem. In our algorithm, for any given π , the critic needs to solve a min f ∈F max w ∈W optimization problem, whereas in ATAC (Cheng et al., 2022), the critic needs to solve a min f ∈F max g ∈F problem. Since we only assume single-policy realizability for the class W (Assumption 3) but assume all-policy realizability for F (Assumption 1) (and Cheng et al. (2022) even requires the Bellman-completeness assumption over F which is much stronger), in general, the cardinality of W could be much smaller than F , which makes the optimization region of the critic's optimization problem in our algorithm F × W smaller than F × F in ATAC.

4 Theoretical Analysis

In this section, we show the theoretical guarantee of our main algorithm (Algorithm 1), which is statistically optimal in terms of N .

4.1 Performance guarantee of the A-Crab algorithm

We first formally present our main theorem, which provides a theoretical guarantee of our A-Crab algorithm (Algorithm 1). A proof sketch is provided in Section 5 and the complete proof is deferred to Appendix C.3.

Theorem 1 (Main theorem) . Under Assumptions 1 and 2 and let π ∈ Π be any policy satisfying Assumption 3, then with probability at least 1 -δ ,

where ϵ stat ≍ V max C ⋆ ℓ 2 √ log( |F|| Π ||W| /δ ) N + V max B w log( |F|| Π ||W| /δ ) N , and ¯ π is returned by Algorithm 1 with the choice of β = 2 .

Below we discuss the advantages of our approach as shown in the above theorem.

Optimal statistical rate and computational efficiency. When ϵ F = 0 (i.e., there is no model misspecification), and when π = π ⋆ is one of the optimal policies, the output policy ¯ π achieves O (1 / √ N ) suboptimality rate which is optimal in N dependence (as long as K is large enough). This improves the O (1 /N 1 / 3 ) rate of the previous algorithm (Cheng et al., 2022). Note that the algorithm of Xie et al. (2021) can also achieve the optimal O (1 / √ N ) rate but their algorithm involves hard constraints of squared ℓ 2 Bellman error and thus is computationally intractable. Cheng et al. (2022) convert the hard constraints to a regularizer, making the algorithm computationally tractable while degenerating the statistical rate. Our algorithm is both statistically optimal and computationally efficient, which improves upon both Xie et al. (2021); Cheng et al. (2022) simultaneously.

Competing with any policy. Another advantage of our algorithm is that it can compete with any policy π ∈ Π as long as w π = d π /µ is contained in W . In particular, the importance ratio of the behavior policy w µ = d µ /µ = µ/µ ≡ 1 is always contained in W , which implies that our algorithm satisfies robust policy improvement (see Theorem 2 for details).

Robustness to model misspecification. Theorem 1 also shows that our algorithm is robust to model misspecification on realizability assumption. Note that our algorithm does not need a completeness assumption, while Xie et al. (2021); Cheng et al. (2022) both require the (approximate) completeness assumption.

Removal of the completeness assumption on F . Compared to our algorithm, Cheng et al. (2022) additionally need a completeness assumption on F , which requires that for any f ∈ F and π ∈ Π , it approximately holds that T π f ∈ F . They need this completeness assumption because they use the estimator E D [ ( f ( s, a ) -r -γf ( s ′ , π )) 2 ] -min g ∈F E D [ ( g ( s, a ) -r -γf ( s ′ , π )) 2 ] to address the over-estimation issue caused by their squared ℓ 2 Bellman error regularizer, and to make this estimator accurate, they need min g ∈F E D [ ( g ( s, a ) -r -γf ( s ′ , π )) 2 ] to be small, which can be implied by the (approximate) completeness assumption. In our algorithm, thanks to the nice property of the weighted average Bellman error regularizer which can be estimated by a simple and unbiased estimator, we can get rid of this strong assumption.

4.2 A-Crab for robust policy improvement

Robust policy improvement (RPI) refers to the property of an offline RL algorithm that the learned policy (almost) always improves upon the behavior policy used to collect data over a wide range of the choice of some specific hyperparameters (in this paper, the hyperparameter is β ) (Cheng et al., 2022). Similar to ATAC in Cheng et al. (2022), our A-Crab also enjoys the RPI property. Theorem 2 implies that as long as β = o ( √ N ) , our algorithm can learn a policy with vanishing suboptimality compared to the behavior policy with high probability. The proof is deferred to Appendix D.

Theorem 2 (Robust policy improvement) . Under Assumptions 1 and 2, with probability at least 1 -δ ,

where ϵ stat ≍ V max C ⋆ ℓ 2 √ log( |F|| Π ||W| /δ ) N + V max B w log( |F|| Π ||W| /δ ) N , and ¯ π is returned by Algorithm 1 with the choice of any β ≥ 0 .

4.3 Suboptimality of squared l 2 norm of Bellman error as regularizers

The ATAC algorithm of Cheng et al. (2022) suffers suboptimal statistical rate O (1 /N 1 / 3 ) due to the squared ℓ 2 Bellman error regularizer. Intuitively, in Cheng et al. (2022), they use Lemma 1 to

decompose the performance difference and use ∥ f -T π f ∥ 2 ,µ to upper bound E µ [( f -T π f )( s, a )] , which causes suboptimality since in general the former could be much larger than the latter. To overcome this suboptimal step, in our algorithm, we use a weighted version of E µ [( f -T π f )( s, a )] as our regularizer instead of ∥ f -T π f ∥ 2 ,µ . Proposition 4.1 shows that ATAC is indeed statistically suboptimal even under their optimal choice of the hyperparameter β = Θ( N 2 / 3 ) . The proof is deferred to Appendix E.1.

Proposition 4.1 (Suboptimality of ATAC) . If we change the regularizer s.t.

where L ( g, f, π, D ) = E D [( g ( s, a ) -r -γf ( s ′ , π )) 2 ] , then even under all policy realizability ( Q π ∈ F for all π ∈ Π ) and completeness assumption ( T π f ∈ F for all π ∈ Π and f ∈ F ), with their optimal choice of β = Θ( N 2 / 3 ) (Cheng et al., 2022), there exists an instance s.t. the suboptimality of the returned policy of (5) (i.e., the output policy by ATAC) is Ω(1 /N 1 / 3 ) with at least constant probability.

5 Proof Sketch

We provide a proof sketch of our main theorem (Theorem 1) in this section. The key lemma of the proof is presented in Lemma 1.

Lemma 1 (Performance difference decomposition, Lemma 12 in Cheng et al. (2022)) . For any π, ˆ π ∈ Π , and any f : S × A → R , we can decompose J ( π ) -J (ˆ π ) as

Note that the first two terms in the RHS of the decomposition are average Bellman errors of f and ˆ π w.r.t. µ and d π . Based on this lemma, we can directly use the average Bellman error as the regularizer instead of the squared Bellman error, which could be much larger and causes suboptimality.

Proof sketch of Theorem 1. For simplicity, we assume realizability of Q π , i.e., ϵ F = 0 . By Lemma 1 and the definition of ¯ π , we have J ( π ) -J (¯ π ) = 1 K ∑ K k =1 ( J ( π ) -J ( π k )) , which equals to

By the concentration argument, with high probability, we have E µ ( π, f ) = E D ( π, f ) ± ϵ stat and L µ ( π, f ) = L D ( π, f ) ± ϵ stat for all π ∈ Π and f ∈ F . Combining the fact that d π /µ ∈ W , one can show that ( a ) + ( b ) ≤ 2 E µ ( π k , f k ) ≤ 2 E D ( π k , f k ) + O ( ϵ stat ) . Therefore,

where the third inequality holds by the optimality of f k , and the last equality holds since the Bellman error of Q π w.r.t. π is 0. Therefore, with high probability,

6 Experiments

In this section, we conduct experiments of our proposed A-Crab algorithm (Algorithm 1) using a selection of the Mujoco datasets (v2) from D4RL offline RL benchmark (Fu et al., 2020). In particular, we compare the performances of A-Crab and ATAC, since ATAC is the state-of-the-art algorithm on a range of continuous control tasks (Cheng et al., 2022).

A more practical version of weighted average Bellman error. Recall the definition of our proposed weighted average Bellman error regularizer

Since the calculation of E D ( π, f ) requires solving an optimization problem w.r.t. importance weights w , for computational efficiency, we choose W = [0 , C ∞ ] S×A as in Hong et al. (2023), and thus

E app D ( π, f ) = C ∞ max { E D [( f ( s, a ) -r -γf ( s ′ , π )) + ] , E D [( r + γf ( s ′ , π ) -f ( s, a )) + ] } , (8) where ( · ) + = max {· , 0 } and C ∞ can be viewed as a hyperparameter. We also observed that using a combination of squared Bellman error and our average Bellman error achieves better performance in practice, and we conjecture the reason is that the squared Bellman error regularizer is computationally more efficient and statistically suboptimal, while our average Bellman error regularizer is statistically optimal while computationally less efficient, and thus the combination of these two regularizers can benefit the training procedure.

The practical implementation of our algorithm is nearly identical to (a lightweight version of) ATAC (Cheng et al., 2022) , except that we choose

as the regularizer, while ATAC uses β E D [(( f - T π f )( s, a )) 2 ] . All hyperparameters are the same as ATAC, including β . For the additional hyperparamter C ∞ , we do a grid search on { 1 , 2 , 5 , 10 , 20 , 50 , 100 , 200 } .

Figure 2 compares the performance of ACrab and ATAC during training. It shows that our A-Crab has higher returns and smaller deviations than ATAC in various settings ( walker2d-random, halfcheetahmedium-replay, hopper-medium-expert ). We provide more details and results in Appendix F. We also observed that in most settings, A-Crab has a smaller variance, which shows that the training procedure of A-Crab is more stable. We provide the choice of β and C ∞ for each setting in Table F.3. Note that we use the same value of β as in ATAC.

0

200

400

600

800

1000

steps (x 10^3)

1

2

3

4

5

6

7

8

return

walker2d-rand

ATAC

ACrab

(a) walker2d-rand

0

200

400

600

800

1000

steps (x 10^3)

34

36

38

40

42

44

46

48

return

halfcheetah-med-replay

ATAC

ACrab

(b) halfcheetah-med-replay

0

200

400

600

800

1000

steps (x 10^3)

40

50

60

70

80

90

100

110

return

hopper-med-exp

ATAC

ACrab

(c) hopper-med-exp

7 Discussion

Wepresent a new offline RL algorithm called A-Crab (Algorithm 1) that can be combined with general function approximators and handle datasets with partial coverage. A-Crab is an actor-critic method, where the critic finds a relatively pessimistic evaluation of the actor while minimizing an importanceweighted average Bellman error. We prove that A-Crab achieves the optimal statistical rate of 1 / √ N converging to the best policy 'covered' in the data. Importantly, the notion of coverage here is a weaker ℓ 2 variant of the single-policy concentrability, which only requires the average marginalized importance weights over visitations of the target policy to be bounded. Also, A-Crab enjoys robust policy improvement that consistently improves over the data-collection behavior policy. Moreover, we empirically validated the effectiveness of A-Crab in the D4RL benchmark. Interesting avenues for future work include combining A-Crab's offline learning with an online fine-tuning algorithm with a limited trial-and-error budget and developing new measures for single-policy coverage that leverage both the visitation and hypothesis class structures.

References

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res. , 22(98): 1-76, 2021.

Andras Antos, Rémi Munos, and Csaba Szepesvari. Fitted Q-iteration in continuous action-space mdps. In Neural Information Processing Systems , 2007.

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with Bellmanresidual minimization based fitted policy iteration and a single sample path. Machine Learning , 71 (1):89-129, 2008.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. arXiv preprint arXiv:1905.00360 , 2019.

Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: The power of gaps. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning , volume 162, 2022.

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897):414-419, 2022.

Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Online markov decision processes. Mathe-

matics of Operations Research , 34(3):726-736, 2009.

Amir Massoud Farahmand, Rémi Munos, and Csaba Szepesvári. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems , 2010.

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53, 2022.

Yihao Feng, Lihong Li, and Qiang Liu. A kernel loss for solving the Bellman equation. arXiv preprint arXiv:1905.10506 , 2019.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , pages 2052-2062. PMLR, 2019.

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expectedmax Q-learning operator for simple yet effective offline and online RL. arXiv preprint arXiv:2007.11091 , 2020.

Kaiyang Guo, Yunfeng Shao, and Yanhui Geng. Model-based offline reinforcement learning with pessimism-modulated dynamics belief. arXiv preprint arXiv:2210.06692 , 2022.

Kihyuk Hong, Yuhang Li, and Ambuj Tewari. A primal-dual-critic algorithm for offline constrained reinforcement learning. arXiv preprint arXiv:2306.07818 , 2023.

Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 , 2019.

Nan Jiang. On value functions and the agent-environment boundary. arXiv preprint arXiv:1905.13341 , 2019.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Modelbased offline reinforcement learning. arXiv preprint arXiv:2005.05951 , 2020.

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with Fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949 , 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779 , 2020.

Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. Should I run offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations , 2021.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR, 2021.

Peng Liao, Zhengling Qi, and Susan Murphy. Batch policy learning in average reward Markov decision processes. arXiv preprint arXiv:2007.11771 , 2020.

Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. In Neural Information Processing Systems , 2019a.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprint arXiv:1904.08473 , 2019b.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforcement learning without great exploration. arXiv preprint arXiv:2007.08202 , 2020.

Jiˇ rí Matoušek and Jan Vondrák. The probabilistic method. Lecture Notes, Department of Applied Mathematics, Charles University, Prague , 2001.

Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang, YoungJoon Lee, Eric Johnson, Omkar Pathak, Sungmin Bae, et al. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004.10746 , 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Rémi Munos. Performance bounds in ℓ p -norm for approximate value iteration. SIAM journal on control and optimization , 46(2):541-561, 2007.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems , pages 2315-2325, 2019a.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. AlgaeDICE: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019b.

Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems , 33:1060-1070, 2020.

Asuman Ozdaglar, Sarath Pattathil, Jiawei Zhang, and Kaiqing Zhang. Revisiting the linearprogramming framework for offline RL with general function approximation. arXiv preprint arXiv:2212.13861 , 2022.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Yongyi Ran, Han Hu, Xin Zhou, and Yonggang Wen. Deepee: Joint optimization of job scheduling and cooling control for data center energy efficiency using deep reinforcement learning. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) , pages 645-655. IEEE, 2019.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline RL with general function approximation via augmented Lagrangian. arXiv preprint arXiv:2211.00716 , 2022.

Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 8106-8114, 2022.

Marc Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581 , 2022.

David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. Tackling climate change with machine learning. ACM Computing Surveys (CSUR) , 55(2):1-96, 2022.

Bruno Scherrer. Approximate policy iteration schemes: A comparison. In International Conference on Machine Learning , pages 1314-1322, 2014.

Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767 , 2022.

Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 , 2020.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4RL: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning , pages 907-917. PMLR, 2022.

Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. arXiv preprint arXiv:2210.06718 , 2022.

Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research , 16(1):1731-1755, 2015.

Csaba Szepesvári and Rémi Munos. Finite time bounds for sampling based fitted value iteration. In Proceedings of the 22nd international conference on Machine learning , pages 880-887, 2005.

Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems , 2022.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations , 2021.

Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and Q-function learning for offpolicy evaluation. In International Conference on Machine Learning , pages 9659-9668. PMLR, 2020.

Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline RL in low-rank MDPs. arXiv preprint arXiv:2110.04652 , 2021.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations , 2019.

Xinqi Wang, Qiwen Cui, and Simon S Du. On gap-dependent bounds for offline reinforcement learning. arXiv preprint arXiv:2206.00177 , 2022.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Tengyang Xie and Nan Jiang. Q ⋆ approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence , pages 550-559. PMLR, 2020.

Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning , pages 11404-11413. PMLR, 2021.

Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. Advances in Neural Information Processing Systems , 32, 2019.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Tengyang Xie, Mohak Bhardwaj, Nan Jiang, and Ching-An Cheng. Armor: A model-based framework for improving arbitrary baseline policies with offline data. arXiv preprint arXiv:2211.04538 , 2022.

Yuling Yan, Gen Li, Yuxin Chen, and Jianqing Fan. The efficacy of pessimism in asynchronous Q-learning. arXiv preprint arXiv:2203.07368 , 2022.

Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34:4065-4078, 2021.

Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05804 , 2022.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239 , 2020.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363 , 2021.

Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34:1362613640, 2021.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. arXiv preprint arXiv:2007.02151 , 2020a.

Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. GenDICE: Generalized offline estimation of stationary values. In International Conference on Learning Representations , 2020b.

Shantong Zhang, Bo Liu, and Shimon Whiteson. GradientDICE: Rethinking generalized offline estimation of stationary values. arXiv preprint arXiv:2001.11113 , 2020c.

Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 5757-5773. PMLR, 2022.

A Related Work

In this section, we review additional related literature not covered in the introductions.

A.1 Dataset Coverage Assumptions

One central challenge of offline RL is the insufficient coverage of the dataset. In RL theory, concentrability is often used to characterize dataset coverage (Munos, 2007; Scherrer, 2014). For example, many earlier works require all-policy ℓ ∞ concentrability (Scherrer, 2014; Liu et al., 2019a; Chen and Jiang, 2019; Jiang, 2019; Wang et al., 2019; Liao et al., 2020; Zhang et al., 2020a). Some works even require the ratio between occupancy probability induced by polices and the dataset distribution to be bounded for every time step (Szepesvári and Munos, 2005; Munos, 2007; Antos et al., 2008; Farahmand et al., 2010; Antos et al., 2007). The work Xie and Jiang (2021) makes stronger assumptions such as requiring lower bound on conditionals µ ( a | s ) .

Since the all-policy concentrability assumption is strong and can often be violated in practice, recent algorithms requiring only partial data coverage assumptions are developed based on single-policy ℓ ∞ concentrability (Rashidinejad et al., 2021; Zhan et al., 2022; Rashidinejad et al., 2022; Chen and Jiang, 2022; Ozdaglar et al., 2022). However, This could still be restrictive even if only single-policy concentrability is required since the ℓ ∞ concentrability is an upper bound of density ratios over all state-action pairs.

Milder versions of ℓ ∞ concentrability have been studied in both all-policy concentrability framework (Xie and Jiang, 2020, 2021; Feng et al., 2019; Uehara et al., 2020) or and single-policy concentrability (Uehara and Sun, 2021; Xie et al., 2021; Song et al., 2022; Cheng et al., 2022). However, these works based on milder versions of ℓ ∞ single-policy concentrability are either computationally intractable (Uehara and Sun, 2021; Xie et al., 2021) or suffer a suboptimal statistical rate (Cheng et al., 2022).

Our work uses ℓ 2 concentrability version, which also appears in Xie and Jiang (2020). In particular, they also use weighted average Bellman error in their algorithm. However, their algorithm requires all-policy concentrability assumptions and thus cannot deal with partial dataset coverage. To the best of our knowledge among previous works, only Uehara and Sun (2021) used ℓ 2 single-policy concentrability to characterize data coverage, but their algorithm is designed for model-based settings and are computationally intractable. Another closely related work is Uehara et al. (2020), which also uses weighted average Bellman error. However, their algorithm is in the off-policy evaluation (OPE) framework, and they use ℓ ∞ concentrability version to characterize dataset coverage.

A.2 Conservative offline reinforcement learning

To address partial dataset coverage in offline RL, a line of recent applied works studies conservative algorithms, which can be divided into several categories.

The first category enforces the learned policy to be close to the behavior policy (or equivalently, dataset), which ensures that candidate policies not well covered by the dataset are eliminated. This can be accomplished by either adding constraints explicitly (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Jaques et al., 2019; Siegel et al., 2020; Ghasemipour et al., 2020; Fujimoto and Gu, 2021), implicitly (Peng et al., 2019; Nair et al., 2020), or by importance sampling with bounded ratio (Swaminathan and Joachims, 2015; Liu et al., 2019b; Nachum et al., 2019b; Zhang et al., 2020c,b; Lee et al., 2021).

The second category consists of model-based methods such as adversarial model learning (Rigter et al., 2022), learning pessimistic models (Kidambi et al., 2020; Guo et al., 2022), using model ensembles to form penalties (Yu et al., 2020), or combining model and values (Yu et al., 2021).

The last category aims to learn conservative values such as fitted Q-iteration using conservative update (Liu et al., 2020), conservative Q-learning (CQL) (Kumar et al., 2020), critic regularization (Kostrikov et al., 2021), and subtracting penalties (Rezaeifar et al., 2022).

On the theoretical side, many works use some form of uncertainty quantification to design to ensure pessimism (Yin and Wang, 2021; Kumar et al., 2021; Uehara et al., 2021; Yin et al., 2022; Zhang et al., 2022; Yan et al., 2022; Shi and Chi, 2022; Wang et al., 2022). Except for uncertainty

quantification, in linear function approximation settings, Zanette et al. (2021) uses value function perturbation combined with the actor-critic method. Recent advances in the theory of pessimistic algorithms include MIS (Zhan et al., 2022; Chen and Jiang, 2022) and adversarially trained actor-critic (ATAC) (Cheng et al., 2022). In particular, our algorithm is built based on MIS combined with the actor-critic method.

B Proof of Proposition 2.1

The first part of the proposition is easy to see:

For the second part, consider the case where for a fixed policy π and for any s ∈ S , a ∈ A , there exists f ∈ F such that ( f - T π f )( s, a ) is non-zero only at ( s, a ) and is zero otherwise. The Bellman-consistent concentrability defines C π Bellman to be the smallest constant that

which makes it equal to the ℓ ∞ variant. On the other hand, the ℓ 2 variant only requires the average importance weights to be bounded:

C Theoretical Analysis of the Main Theorem

In this section, we provide theoretical proof of our main theorem (Theorem 1). We first present two key lemmas in Appendix C.1 and then prove the main theorem in Appendix C.3. For convenience, we always assume Assumptions 1 to 3 hold.

C.1 Key lemmas

The first lemma shows that with high probability, the population version of our weighted average Bellman error regularizer is close to the empirical version.

Lemma 2 (Concentration of the empirical regularizer) . With probability at least 1 -δ , for any f ∈ F , π ∈ Π , we have

Proof. We condition on the high probability event in Lemma 4. For any f ∈ F and π ∈ Π , define w ∗ π,f = arg max w ∈W E µ ( π, f ) = arg max w ∈W | E µ [ w ( s, a )( f -T π f )( s, a )] | and define ˆ w π,f =

0

-

ϵ

stat

=

-

ϵ

stat

,

where the inequality holds by the optimality of w ∗ π,f and Lemma 4. Similarly,

where the inequality holds by the optimality of ˆ w π,f and Lemma 4.

The next lemma provides a high-probability upper bound of the empirical weighted average Bellman error of f π w.r.t. π , where f π is the (approximate) Q -function of π .

Lemma 3 (Empirical weighted average Bellman error of approximate Q function) . With probability at least 1 -δ , for any π ∈ Π , we have

where f π = arg min f ∈F max admissible ν ∥ f -T π f ∥ 2 2 ,ν .

Proof. We condition on the high probability event in Lemma 2. Since

where the first inequality is by Cauchy-Schwarz inequality and the second inequality is by the definition of f π and Assumption 1, we can immediately obtain that

C.2 Complementary lemmas

We provide two complementary lemmas in this section, which are both high-probability concentration inequalities.

Lemma 4 (Concentration of weighted average Bellman error) . With probability at least 1 -δ , for any f ∈ F , π ∈ Π and w ∈ W , we have

Proof. It suffices to bound

for any fixed f ∈ F , π ∈ Π , w ∈ W , and a union bound and the triangle inequality conclude. Note that

Then by Bernstein's inequality, we have that with probability at least 1 -δ ,

Since

Var µ [( f -T π f ) w ] ≤ E µ [( f -T π f ) 2 w 2 ] ≤ O ( V 2 max ∥ w ∥ 2 2 ,µ ) ≤ O ( V 2 max ( C ⋆ ℓ 2 ) 2 ) , we can obtain that

which implies the result.

Lemma 5 (Concentration of the actor's objective) . With probability at least 1 -δ , for any f ∈ F , π ∈ Π , we have

where ϵ stat is defined in Lemma 4.

Proof. Note that E µ [ L D ( π, f )] = L µ ( π, f ) and | f ( s, π ) -f ( s, a ) | ≤ O ( V max ) . Applying a Hoeffding's inequality for any fixed f , π and a union bound over all f ∈ F , π ∈ Π , we can obtain that with probability at least 1 -δ , it holds that

for all f ∈ F and π ∈ Π .

C.3 Proof of Theorem 1

Now we are able to prove our main theorem equipped with lemmas in previous sections.

Proof of Theorem 1. By Lemma 1 and the definition of ¯ π , we have

Now we condition on the high probability event in Lemma 2 and Lemma 5 simultaneously and rescale δ by 1 / 2 and apply a union bound. Note that d π /µ ∈ W , which implies that ( a ) + ( b ) ≤ 2 E µ ( π k , f k ) ≤ 2 E D ( π k , f k ) + 2 ϵ stat, where the last inequality holds by Lemma 2. By Lemma 13 of Cheng et al. (2022), we can obtain that

|L µ ( π k , Q π k ) -L µ ( π k , f π k ) | ≤ ∥ f π k -T π k f π k ∥ 2 ,µ + ∥ f π k -T π k f π k ∥ 2 ,d π k ≤ O ( √ ϵ F ) , where f π k ≜ arg min f ∈F max admissible ν ∥ f -T π f ∥ 2 2 ,ν . Also, by Lemma 5, we have |L µ ( π k , f k ) -L D ( π k , f k ) | + |L µ ( π k , f π k ) -L D ( π k , f π k ) | ≤ O ( ϵ stat ) .

Therefore,

where the third inequality holds by the optimality of f k , and the last inequality holds by Lemma 3. Therefore,

D Analysis of Robust Policy Improvement

Proof of Theorem 2. By Lemma 1 and the definition of ¯ π , we have

Now we condition on the high probability event in Lemma 2 and Lemma 5 simultaneously and rescale δ by 1 / 2 and apply a union bound. Note that ( a ) + ( b ) = 0 . By Lemma 13 of Cheng et al. (2022), we can obtain that

where f π ≜ arg min f ∈F max ν ∥ f -T π f ∥ 2 . Also, by Lemma 5, we have

Therefore,

Therefore,

E Analysis of the Suboptimality of ATAC

In this section, we prove Proposition 4.1 in Appendix E.1. For convenience, we use Bern ( p ) to denote a Bernoulli variable with parameter p , and use Bin ( n, p ) to denote a binomial variable with parameters n and p .

E.1 Proof of Proposition 4.1

Construction of a two-arm bandit example. Assume there are two arms a 1 , a 2 . Assume β = ˜ Θ( N α ) where α > 1 2 . Note that this is a more general case than β = Θ( N 2 / 3 ) as stated in the proposition. Let ∆ = min { β/N, 1 / 10 } . Assume the reward of the first arm is deterministic, i.e., r ( a 1 ) = 1 / 2 + ∆ , and r ( a 2 ) = Bern (1 / 2) . Let Π = { π 1 , π 2 } , where π 1 ( a 1 ) = 1 , π 1 ( a 2 ) = 0 , and π 2 ( a 1 ) = 0 , π 2 ( a 2 ) = 1 . Also, let F = { f 1 , f 2 } , where f 1 ( a 1 ) = 1 / 2 + ∆ , f 1 ( a 2 ) = 1 / 2 and f 2 ( a 1 ) = 1 / 2 + ∆ , f 2 ( a 2 ) = 1 / 2 + 2∆ . Finally, the dataset distribution µ satisfies µ ( a 2 ) = 1 N ∆ 2 , µ ( a 1 ) = 1 -µ ( a 2 ) . Note that in bandit settings, for any policy π and any function f , we have Q π = T π f = r . Therefore, the example above satisfies both completeness and realizability assumptions.

Proof of the suboptimality of ATAC. The ATAC algorithm is simplified to

in bandit settings.

For convenience, we assume that P µ ( a = a 1 ) = P D ( a = a 1 ) . Note that by anti-concentration of the binomial distribution (Lemma 6), we have that with constant probability, it holds that ˆ r ( a 2 ) ≥ 1 2 + 2 √ Nµ ( a 2 ) = 1 2 +2∆ .

Conditioned on this event, we compute f π 1 and f π 2 separately. We have

and

Also, we have

and

Since β/N ≥ ∆ , we can obtain that L D ( π 2 , f 1 )+ β E D ( π 2 , f 1 ) > L D ( π 2 , f 2 )+ β E D ( π 2 , f 2 ) , which implies that f π 2 = f 2 . Finally, note that

we have ˆ π = π 2 by ATAC algorithm. Note that J ( π 1 ) -J (ˆ π ) = r ( π 1 ) -r ( π 2 ) = ∆ ≫ ˜ Ω(1 / √ N ) . Therefore, for any α > 1 / 2 , there exists an instance s.t. ATAC cannot achieve the optimal rate O (1 / √ N ) . In particular, when β = Θ( N 2 / 3 ) , we have J ( π 1 ) -J (ˆ π ) = ∆ = Ω(1 /N 1 / 3 ) .

E.2 Complementary lemma

Lemma 6 (Anti-concentration of Binomial distribution, adapted from Proposition 7.3.2 of Matoušek and Vondrák (2001)) . Let X ∼ Bin ( n, 1 2 ) be a binomial random variable with mean µ = n 2 . Then we have that for any t ∈ [0 , 1 8 ] and universal constants c 1 , c 2 ,

F Additional Experimental Details and Results

F.1 More implementation details

Our implementation is nearly identical to ATAC, except that we use a different regularizer for the critic. More details can be found in Cheng et al. (2022).

0

200

400

600

800

1000

steps (x 10^3)

65

70

75

80

85

90

return

walker2d-med

ATAC

ACrab

(a) walker2d-med

0

200

400

600

800

1000

steps (x 10^3)

20

30

40

50

60

70

80

90

return

walker2d-med-replay

ATAC

ACrab

(b) walker2d-med-replay

0

200

400

600

800

1000

steps (x 10^3)

80

85

90

95

100

105

110

115

return

walker2d-med-exp

ATAC

ACrab

(c) walker2d-med-exp

0

200

400

600

800

1000

steps (x 10^3)

2.2

2.4

2.6

2.8

3.0

3.2

return

halfcheetah-rand

ATAC

ACrab

(d) halfcheetah-rand

0

200

400

600

800

1000

steps (x 10^3)

42

44

46

48

50

52

return

halfcheetah-med

ATAC

ACrab

(e) halfcheetah-med

0

200

400

600

800

1000

steps (x 10^3)

50

60

70

80

90

return

halfcheetah-med-exp

ATAC

ACrab

(f) halfcheetah-med-exp

0

200

400

600

800

1000

steps (x 10^3)

5

10

15

20

return

hopper-rand

ATAC

ACrab

(g) hoper-rand

0

200

400

600

800

1000

steps (x 10^3)

50

60

70

80

90

100

return

hopper-med

ATAC

ACrab

(h) hopper-med

0

200

400

600

800

1000

steps (x 10^3)

0

20

40

60

80

100

return

hopper-med-replay

ATAC

ACrab

(i) hopper-med-replay

F.2 Comparision of A-Crab and ATAC in different settings

In this section, we show comparisons of A-Crab and ATAC in more different settings in Figure 3.

F.3 Value of β and C ∞ for each setting

In this section, we show our choices of β and C ∞ for each setting in Table F.3. Note that we directly choose the same value of β as in ATAC, and select C ∞ by a grid search over { 1 , 2 , 5 , 10 , 20 , 50 , 100 , 200 } .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces A-Crab, an offline RL algorithm derived from ATAC, which incorporates a modified loss function for the Q-function. Instead of employing the square Bellman error used in ATAC, A-Crab utilizes the importance-weighted Bellman error. With this modified loss function, A-Crab effectively addresses some disadvantages of ATAC. These include alleviating the visitation coverage assumption, enhancing the suboptimality rate, and simplifying the minimax optimization process. Additionally, A-Crab inherits several benefits from ATAC. Theoretical findings presented in this paper demonstrate the superior advantages of A-Crab in comparison to ATAC.\n\nSTRENGTHS:\n1. This paper is well-organized, presenting a table that compares provable offline RL algorithms, outlining the key distinctions of the proposed algorithm, A-Crab, in contrast to ATAC, and highlighting the advantages brought about by these modifications.\n2. By simply incorporating the importance sampling ratio from the loss function of ATAC, A-Crab exhibits significantly improved characteristics, namely: 1) relaxation of the visitation coverage assumption, 2) enhancement of the suboptimal rate, and 3) simplification of the minimax problem into a maximization problem. These advancements result from non-trivial steps and can be accomplished by making only a few adjustments to the terms in the loss functions of ATAC.\n\nWEAKNESSES:\n1. Despite the noteworthy novelty of this paper in terms of presenting theoretical advantages of A-Crab, the absence of empirical results diminishes its impact. I acknowledge that it is common to solely provide theoretical results in the realm of provable offline RL. However, considering that A-Crab is built upon ATAC, which offers both theoretical and numerical results, it would be feasible to compare the performance of A-Crab with that of ATAC. I believe that including such results would significantly enhance the paper.\n2. The importance sampling ratio in offline RL can become excessively large due to the limited coverage of the state-action space in the offline dataset and substantial differences between the current policy and the dataset policy. Consequently, the A-Crab algorithm may encounter instability when attempting to learn a robust policy in practical scenarios. However, if this paper addresses this issue, it would substantially strengthen its credibility.\n\nQUESTIONS:\n1. [Regarding Weakness 1] Would it be possible for you to present empirical results on D4RL datasets that compare the performance of A-Crab with that of ATAC?\n\nLIMITATIONS:\nThe only limitation of this paper is the absence of empirical results. However, the implementation of A-Crab appears to be relatively straightforward, as it can be achieve by replacing the squared Bellman error in ATAC's implementation with the importance-weighted averaged Bellman error. Given that the implementation of ATAC is already available on GitHub, there is an opportunity to enhance this paper by presenting empirical results that compare the performance of A-Crab with that of ATAC.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an offline reinforcement learning algorithm called Actor-Critic Regularized by Average Bellman error (A-Crab). A-Crab modifies the pessimistic offline RL framework by replacing the usual squared TD error with an importance sampled TD error. Due to the linearity of the importance sampled TD error, overestimation does not occur, allowing the removal of the correction term. A-Crab achieves the optimal suboptimality rate with weaker assumptions and is computationally efficient. An improvement over the behavior policy is also guaranteed under a wide range of hyperparameters.\n\nSTRENGTHS:\n### Originality\nThe authors introduced a novel technique using the importance sampled TD error instead of the squared TD error. Due to its linearity, the correction for overestimation is unnecessary, allowing better suboptimality bounds and weaker assumptions. The analysis also becomes very simple with the help of Cauchy–Schwarz inequality.\n### Quality\nI could not find any technical flaws in the arguments presented in the paper.\n### Clarity\nThe paper is overall well-written and easy to understand.\n### Significance\nA-Crab achieves the optimal statistical rate of $1/\\sqrt{N}$ with assumptions that are weaker compared to other provable offline RL algorithms. Also, using importance sampling instead of squaring can be useful in areas other than offline RL theory.\n\nWEAKNESSES:\n1. The authors unrealistically assume the action space, the policy space, the importance sampling weight function space, and the value function space to be finite.\n2. Experimental verification of the performance bounds on a simple toy example would be interesting.\n\nQUESTIONS:\nLine 70 states that the A-Crab enjoys an optimal statistical rate of $1/\\sqrt{N}$. Does this mean the optimality of $O(1/\\sqrt{N})$ is theoretically proved? Or does this just mean that it is the best suboptimality rate to be discovered?\n\nLIMITATIONS:\nThe authors do not address the limitations of their work. Assumptions on the finiteness of the action space, the policy space, the importance sampling weight function space, and the value function space can be viewed as limitations of this work. I believe this work does not have a potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper introduces A-Crab, which combines marginalized importance sampling with the actor-critic paradigm to achieve optimal statistical rate in offline RL. Fm theoretical analysis, this algorithm is also more computationally efficient and relies on a weaker average notion of policy coverage compared to prior work.

STRENGTHS:
1. The paper is well-written, effectively summarizing prior work and building upon it to propose the new algorithm. The notations are clear and consistent.
2. The algorithm itself exhibits strong theoretical properties, such as optimal statistical rate and efficiency, making it promising for offline RL problems.

WEAKNESSES:
1. The empirical evaluation of the algorithm is a major concern. Since the theory only requires general function approximators, the gap between theory and practice may not be substantial. Experimental results help demonstrate the algorithm's effectiveness on computation cost and learning speed.

QUESTIONS:
1. In Line 224, where does the overestimation come from? Please provide clarification.
2. In Line 248, why does ATAC need to optimize two functions, f and g? The previous mention in Line 214 only refers to f.
3. In Line 280, how does the theorem also demonstrate robustness against model mismatch? Please elaborate on this point.

LIMITATIONS:
Please refer to the weaknesses section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm called A-Crab (Actor-Critic Regularized by Average Bellman Error) for offline reinforcement learning (RL) in complex environments with insufficient data coverage. The algorithm combines the marginalized importance sampling framework with the actor-critic paradigm and addresses the challenges of handling high-dimensional observations and minimal data coverage. The paper presents sufficient theoretical analysis to demonstrate the advantages of the proposed algorithm over existing methods.\n\nSTRENGTHS:\n- The paper proposes a novel algorithm that combines various techniques to address the challenges of offline RL in complex environments. The use of marginalized importance sampling and the average Bellman error regularization in the critic's objective are innovative and practical.\n- The theoretical analysis provides insights into the statistical properties of the proposed algorithm, with a focus on achieving the optimal statistical rate in converging to the best policy covered in the offline dataset.\n- The paper provides a comparison with existing provable offline RL algorithms, highlighting the strengths and advantages of the proposed A-Crab algorithm.\n\nWEAKNESSES:\n- The paper provides no empirical evaluations or demonstrations of the proposed algorithm. Neither does it shed light on the design of practical algorithms.\n- There remain some issues unsolved in the paper. See the questions for details.\n- Some related works [1,2] use a similar optimization problem as Eq. (2) and also focus on computing an prioritization weight $w$ for Bellman error. They should be cited and discussed.\n\nQUESTIONS:\n- There seems to be a lack of comparison between $C_{\\text{bellman}}$ and $C_{l_2}$ : How to show $C_{l_2}$ is a weaker assumption than $C_{\\text{bellman}}$?\n- How can the A-Crab algorithm shed light on the design of practical algorithms?\n\nLIMITATIONS:\nReferences\n\n[1] Kumar, Aviral, Abhishek Gupta, and Sergey Levine. \"Discor: Corrective feedback in reinforcement learning via distribution correction.\" Advances in Neural Information Processing Systems 33 (2020): 18560-18572.\n\n[2] Liu, Xu-Hui, et al. \"Regret minimization experience replay in off-policy reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021): 17604-17615.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}