Title: Adversarial Model for Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: model based

offline

reinforcement learning

adversarial training
Abstract: We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with any admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model-based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.

Adversarial Model for Offline Reinforcement Learning

Abstract

We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worstcase performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with any admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and modelbased RL algorithms and can robustly improve the reference policy over various hyperparameter choices. 2

1 Introduction

Offline reinforcement learning (RL) is a technique for learning decision-making policies from logged data (Lange et al., 2012; Levine et al., 2020; Jin et al., 2021; Xie et al., 2021a). In comparison with alternate learning techniques, such as off-policy RL and imitation learning (IL), offline RL reduces the data assumption needed to learn good policies and does not require collecting new data. Theoretically, offline RL can learn the best policy that the given data can explain: as long as the offline data includes the scenarios encountered by a near-optimal policy, an offline RL algorithm can learn such a near-optimal policy, even when the data is collected by highly sub-optimal policies and/or is not diverse. Such robustness to data coverage makes offline RL a promising technique for solving real-world problems, as collecting diverse or expert-quality data in practice is often expensive or simply infeasible.

The fundamental principle behind offline RL is the concept of pessimism, which considers worstcase outcomes for scenarios without data. In algorithms, this is realized by (explicitly or implicitly) constructing performance lower bounds in policy learning which penalizes uncertain actions.

hopper-med

pen-cloned

hammer-cloned

0

50

100

ORL

REF

0.01

0.05

0.1

0.5

1.0

Various designs have been proposed to construct such lower bounds, including behavior regularization (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021), point-wise pessimism based on negative bonuses or truncation (Kidambi et al., 2020; Jin et al., 2021), value penalty (Kumar et al., 2020; Yu et al., 2020), or two-player games (Xie et al., 2021a; Uehara and Sun, 2021; Cheng et al., 2022). Conceptually, the tighter the lower bound is, the better the learned policy would perform; see a detailed discussion of related work in Appendix C.

Despite these advances, offline RL still has not been widely adopted to build learning-based decision systems beyond academic research. One important factor we posit is the issue of performance degradation: Usually, the systems we apply RL to have currently running policies, such as an engineered autonomous driving rule or a heuristic-based system for diagnosis, and the goal of applying a learning algorithm is often to further improve upon these baseline reference policies . As a result, it is imperative that the policy learned by the algorithm does not degrade the base performance. This criterion is especially critical for applications where poor decision outcomes cannot be tolerated.

However, running an offline RL algorithm based on pessimism, in general, is not free from performance degradation. While there have been algorithms with policy improvement guarantees (Laroche et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020; Fujimoto and Gu, 2021; Cheng et al., 2022), such guarantees apply only to the behavior policy that collects the data, which might not necessarily be the reference policy. In fact, quite often these two policies are different. For example, in robotic manipulation, it is common to have a dataset of activities different from the target task. In such a scenario, comparing against the behavior policy is meaningless, as these policies do not have meaningful performance in the target task.

In this work, we propose a novel model-based offline RL framework, called Advesarial Model for Offline Rinforcement Learning (ARMOR), which can robustly learn policies that improve upon an arbitrary reference policy by adversarially training a Markov decision process (MDP) model, regardless of the data quality. ARMOR is designed based on the concept of relative pessimism (Cheng et al., 2022), which aims to optimize for the worst-case relative performance over uncertainty. In theory, we prove that, owing to relative pessimism, the ARMOR policy never degrades the performance of the reference policy for a range of hyperparameters which is given beforehand, a property known as Robust Policy Improvement (RPI) (Cheng et al., 2022). In addition, when the right hyperparameter is chosen, and the reference policy is covered by the data, we prove that the ARMOR policy can also compete with any policy covered by the data in an absolute sense. To our knowledge, RPI property of offline RL has so far been limited to comparing against the data collection policy (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022). In ARMOR, by adversarially training an MDP model, we extend the technique of relative pessimism to achieve RPI with arbitrary reference policies, regardless of whether they collected the data or not (Fig. 1).

In addition to theory, we design a scalable deep-learning implementation of ARMOR to validate these claims that jointly trains an MDP model and the state-action value function to minimize the estimated performance difference between the policy and the reference using model-based rollouts. Our implementation achieves state-of-the-art (SoTA) performance on D4RL benchmarks (Fu et al., 2020), while using only a single model (in contrast to ensembles used in existing model-based offline RL works). This makes ARMOR a better framework for using high-capacity world models (e.g.(Hafner et al., 2023)) for which building an ensemble is too expensive. We also empirically validate the RPI property of our implementation.

2 Preliminaries

Markov Decision Process We consider learning in the setup of an infinite-horizon discounted Markov Decision Process (MDP). An MDP M is defined by the tuple ⟨S , A , P M , R M , γ ⟩ , where S is the state space, A is the action space, P M : S × A → ∆( S ) is the transition dynamics, R M : S × A → [0 , 1] is a scalar reward function and γ ∈ [0 , 1) is the discount factor. A policy π is a mapping from S to a distribution on A . For π , we let d π M ( s, a ) denote the discounted state-action distribution obtained by running π on M from an initial state distribution d 0 , i.e d π M ( s, a ) = (1 -γ ) E π,M [ ∑ ∞ t =0 γ t ✶ ( s t = s, a t = a )] . Let J M ( π ) = E π,M [ ∑ ∞ t =0 γ t r t ] be the expected discounted return of policy π on M starting from d 0 , where r t = R M ( s t , a t ) . We define the value function as V π M ( s ) = E π,M [ ∑ ∞ t =0 γ t r t | s 0 = s ] , and the state-action value function (i.e., Q-function) as Q π M ( s, a ) = E π,M [ ∑ ∞ t =0 γ t r t | s 0 = s, s 0 = a ] . By this definition, we note J M ( π ) = E d 0 [ V π M ( s )] = E d 0 ,π [ Q π M ( s, a )] . We use [0 , V max ] to denote the range of value functions, where V max ≥ 1 . We denote the ground truth MDP as M ⋆ , and J = J M ⋆

Offline RL The aim of offline RL is to find the policy that maximizes J ( π ) , while using a fixed dataset D collected by a behavior policy µ . We assume the dataset D consists of { ( s n , a n , r n , s n +1 ) } N n =1 , where ( s n , a n ) is sampled from d µ M ⋆ and r n , s n +1 follow M ⋆ ; for simplicity, we also write µ ( s, a ) = d µ M ⋆ ( s, a ) .

We assume that the learner has access to a Markovian policy class Π and an MDP model class M . Assumption 1 (Realizability) . We assume the ground truth model M ⋆ is in the model class M .

In addition, we assume that we are provided a reference policy π ref . In practice, such a reference policy represents a baseline whose performance we want to improve with offline RL and data.

Assumption 2 (Reference policy) . We assume access to a reference policy π ref , which can be queried at any state. We assume π ref is realizable, i.e., π ref ∈ Π .

If π ref is not provided, we can still run ARMOR as a typical offline RL algorithm, by first performing behavior cloning on the data and setting the cloned policy as π ref . In this case, ARMOR has RPI with respect to the behavior policy.

Robust Policy Improvement RPI is a notion introduced in Cheng et al. (2022), which means that the offline algorithm can learn to improve over the behavior policy, using hyperparameters within a known set. Algorithms with RPI are more robust to hyperparameter choices, and they are often derived from the principle of relative pessimism (Cheng et al., 2022). In this work, we extend the RPI concept to compare with an arbitrary reference (or baseline) policy, which can be different from the behavior policy and can take actions outside data support.

3 Adversarial Model for Offline Reinforcement Learning (ARMOR)

ARMOR is a model-based offline RL algorithm designed with relative pessimism. The goal of ARMOR is to find a policy ̂ π that maximizes the performance difference J ( ̂ π ) -J ( π ref ) to a given reference policy π ref , while accounting for the uncertainty due to limited data coverage. ARMOR achieves this by solving a two-player game between a learner policy and an adversary MDP model:

based on a version space of MDP models

where we define the model fitting loss as

and α ≥ 0 is a bound on statistical errors such that M ⋆ ∈ M α . In this two-player game, ARMOR is optimizing a lower bound of the relative performance J ( π ) -J ( π ref ) . This is due to the construction that M ⋆ ∈ M α , which ensures min M ∈M α J M ( π ) -J M ( π ref ) ≤ J M ⋆ ( π ) -J M ⋆ ( π ref ) .

One interesting property that follows from optimizing the relative performance lower bound is that ̂ π is guaranteed to always be no worse than π ref , for a wide range of α and regardless of the relationship between π ref and the data D .

TrueMDP

ar,

ar

ar

PossibleDynamicsinUnknownRegion

+0.1

+1.0

al

a1

al

DatasetTransitions

ar

ar

ar

+0.1

+1.0

+0.1

+1.0

a1

a1

al

a1

a1

a1

+0.1

a1

a1

+0.1

Ta

ar

+1.0

+0.1

ar

ar

+1.0

ReferencePolicy

a1

al

al

al

al

al

ar

ARMORchoosesworst-casemodels

+0.1

ar

广一

ar

+1.0

+0.1

+1.0

al

al

a1

Proposition 1. For any α large enough such that M ⋆ ∈ M α , it holds that J ( ̂ π ) ≥ J ( π ref ) .

This fact can be easily reasoned: Since π ref ∈ Π , we have max π ∈ Π min M ∈M α J M ( π ) -J M ( π ref ) ≥ min M ∈M α J M ( π ref ) -J M ( π ref ) = 0 . In other words, ARMOR achieves the RPI property with respect to any reference policy π ref and offline dataset D .

This RPI property of ARMOR is stronger than the RPI property in the literature. In comparison, previous algorithms with RPI (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022) are only guaranteed to be no worse than the behavior policy that collected the data. In Section 3.2, we will also show that when α is set appropriately, ARMOR can provably compete with the best data covered policy as well, as prior offline RL works (e.g., Xie et al., 2021a; Uehara and Sun, 2021; Cheng et al., 2022).

3.1 An Illustrative Toy Example

Why does ARMOR have the RPI property, even when the reference policy π ref is not covered by the data D ? While we will give a formal analysis soon in Section 3.2, here we provide some intuitions as to why this is possible. First, notice that ARMOR has access to the reference policy π ref . Therefore, a trivial way to achieve RPI with respect to π ref is to just output π ref . However, this naïve algorithm while never degrading π ref cannot learn to improve from π ref . ARMOR achieves these two features simultaneously by 1) learning an MDP Model, and 2) adversarially training this MDP model to minimize the relative performance difference to π ref during policy optimization.

We illustrate this by a one-dimensional discrete MDP example with five possible states as shown in Figure 2. The dynamic is deterministic, and the agent always starts in the center cell. The agent receives a lower reward of 0.1 in the left-most state and a high reward of 1.0 upon visiting the rightmost state. Say, the agent only has access to a dataset from a sub-optimal policy that always takes the left action to receive the 0.1 reward. Further, let's say we have access to a reference policy that demonstrates optimal behavior on the true MDP by always visiting the right-most state. However, it is unknown a priori that the reference policy is optimal. In such a case, typical offline RL methods can only recover the sub-optimal policy from the dataset as it is the best-covered policy in the data.

ARMOR can learn to recover the expert reference policy in this example by performing rollouts with the adversarially trained MDP model. From the realizability assumption (Assumption 1), we know that the version space of models contains the true model (i.e., M ⋆ ∈ M α ). The adversary can then choose a model from this version space where the reference policy π ref maximally outperforms the learner. In this toy example, the model selected by the adversary would be the one allowing the expert policy to reach the right-most state. Now, optimizing relative performance difference with respect to this model will ensure that the learner can recover the expert behavior, since the only way for the learner to stay competitive with the reference policy is to mimic the reference policy in the region outside data support. In other words, the reason why ARMOR has RPI to π ref is that

its adversarial model training procedure can augment the original offline data with new states and actions that would cover those generated by running the reference policy . 3

3.2 Theoretical Analysis

Now we make the above discussions formal and give theoretical guarantees on ARMOR's absolute performance and RPI property. To this end, we introduce a single-policy concentrability coefficient, which measures the distribution shift between a policy π and the data distribution µ .

Definition 1 (Generalized Single-policy Concentrability) . We define the generalized singlepolicy concentrability for policy π , model class M and offline data distribution µ as C M ( π ) := sup M ∈M E d π [ E ⋆ ( M )] E µ [ E ⋆ ( M )] , where E ⋆ ( M ) = D TV ( P M ( · | s, a ) , P M ⋆ ( · | s, a )) 2 + ( R M ( s,a ) -R ⋆ ( s,a )) 2 / V 2 max .

Note that C M ( π ) is always upper bounded by the standard single-policy concentrability coefficient ∥ d π /µ ∥ ∞ (e.g., Jin et al., 2021; Rashidinejad et al., 2021; Xie et al., 2021b), but it can be smaller in general with model class M . It can also be viewed as a model-based analog of the one in Xie et al. (2021a). A detailed discussion around C M ( π ) can be found in Uehara and Sun (2021).

First, we present the absolute performance guarantee of ARMOR, which holds for a well-tuned α .

Theorem 2 (Absolute performance) . Under Assumption 1, there is an absolute constant c such that for any δ ∈ (0 , 1] , if we set α = c · (log( |M| / δ )) in Eq. (2), then for any reference policy π ref and comparator policy π † ∈ Π , with probability 1 -δ , the policy ̂ π learned by ARMOR in Eq. (1) satisfies that J ( π † ) -J ( ̂ π ) is upper bounded by

Roughly speaking, Theorem 2 shows that ̂ π learned by ARMOR can compete with any policy π † with a large enough dataset, as long as the offline data µ has good coverage on π † (good coverage over π ref can be automatically satisfied if we simply choose π ref = µ , which yields C M ( π ref ) = 1 ). Compared to the closest model-based offline RL work (Uehara and Sun, 2021), if we set π ref = µ (data collection policy), Theorem 2 leads to almost the same guarantee as Uehara and Sun (2021, Theorem 1) up to constant factors.

In addition to absolute performance, below we show that, under Assumptions 1 and 2, ARMOR has the RPI property to π ref : it always improves over J ( π ref ) for a wide range of parameter α . Compared with the model-free ATAC algorithm in Cheng et al. (2022, Proposition 6), the threshold for α in Theorem 3 does not depend on sample size N due to the model-based nature of ARMOR.

Theorem 3 (Robust strong policy improvement) . Under Assumptions 1 and 2, there exists an absolute constant c such that for any δ ∈ (0 , 1] , if: i) α ≥ c · (log( |M| / δ )) in Eq. (2); ii) π ref ∈ Π , then with probability 1 -δ , the policy ̂ π learned by ARMOR in Eq. (1) satisfies J ( ̂ π ) ≥ J ( π ref ) .

The detailed proofs of Theorems 2 and 3, as well as the discussion on how to relax Assumptions 1 and 2 to the misspecified model and policy classes are deferred to Appendix A.

4 Practical Implementation

In this section, we present a scalable implementation of ARMOR (Algorithm 1) that approximately solves the two-player game in Eq. (1). We first describe the overall design principle and then the algorithmic details.

4.1 A Model-based Actor Critic Approach

For computational efficiency, we take a model-based actor critic approach and solve a regularized version of Eq. (1). We construct this regularized version by relaxing the constraint M ∈ M α

Algorithm 1 ARMOR (Adversarial Model for Offline Reinforcement Learning)

Input: Batch data D real, policy π , MDP model M , critics f 1 , f 2 , horizon H , constants β, λ ≥ 0 , τ ∈ [0 , 1] , w ∈ [0 , 1] ,

1: Initialize target networks ¯ f 1 ← f 1 , ¯ f 2 ← f 2 and D model = ∅

2: for k = 0 , . . . , K -1 do

3: Sample minibatch D mini real from dataset D real and minibatch D mini model from dataset D model.

4: Construct transition tuples using model predictions

5: Update the adversary networks; for i = 1 , 2 ,

6: Update actor network with respect to the first critic network and the reference policy

7: If k % H = 0 , then reset model state: ¯ S π ←{ s ∈ D mini real } and ¯ S π ref ←{ s ∈ D mini real }

8: Query the MDP model to expand D model and update model state

9: end for

in the inner minimization of Eq. (1) to a regularization term and introducing an additional critic function. To clearly elaborate this, we first present the regularized objective in its complete form, and subsequently derive it from Eq. (1).

Let F : { f : S ×A → [0 , V max ] }

where E D ( M ) = ∑ D -log P M ( s ′ | s, a ) + ( R M ( s,a ) -r ) 2 / V 2 max is the model-fitting error, L d π ref M ( π, f ) := E d π ref M [ f ( s, π ) -f ( s, π ref )] is equal to the performance difference (1 -γ )( J M ( π ) -J M ( π ref )) , E ρ π ref ,π ( π, f, M ) denotes the squared Bellman error on the distribution ρ π ref ,π that denotes the distribution generated by first running π ref and then rolling out π in M (with a switching time sampled from a geometric distribution of γ ), and β, λ act as the Lagrange multipliers.

This regularized formulation in Eq. (6) can be derived as follows. Assuming Q π M ∈ F , and using the facts that J M ( π ) = E d 0 [ Q π M ( s, π )] and the Bellman equation Q π M ( s, a ) = r M ( s, a ) + γ E s ′ ∼ P M ( s,a ) [ Q π M ( s ′ , π )] , we can rewrite Eq. (1) as

We then convert the constraints in Eq. (7) into regularization terms in the inner minimization by introducing Lagrange multipliers ( β , λ ), following (Xie et al., 2021a; Cheng et al., 2022), and drop the constants not affected by M,f,π , which results in Eq. (6).

4.2 Algorithm Details

Algorithm 1 is an iterative solver for approximating the solution to Eq. (6). Here we further approximate d π ref M and ρ π ref ,π in Eq. (6) using samples from the state-action buffer D model. We want ensure that D model has a larger coverage than both d π ref M and ρ π ref ,π . We do so heuristically, by constructing the model replay buffer D model through repeatedly rolling out π and π ref with the adversarially trained MDP model M , such that D model contains a diverse training set of state-action tuples.

Specifically, the algorithm takes as input an offline dataset D real, a policy π , an MDP model M and two critic networks f 1 , f 2 . At every iteration, the algorithm proceeds in two stages. First, the adversary is optimized to find a data-consistent model that minimizes the performance difference with the reference policy. We sample mini-batches of only states and actions D mini real and D mini model from the real and model-generated datasets respectively (Line 4). The MDP model M is queried on these mini-batches to generate next-state and reward predictions. The adversary then updates the model and Q-functions (Line 5) using the gradient of the loss described in Eq. (4), where

L D M is the pessimistic loss term that forces the f to predict a lower value for the learner than the reference on the sampled states. E w D M is the Bellman surrogate to encourage the Q-functions to be consistent with the model-generated data D M . We use the double Q residual algorithm loss similar to Cheng et al. (2022), which is defined as a convex combination of the temporal difference losses with respect to the critic and the delayed target networks, E td D ( f, f ′ , M, π ) := E D [ ( f ( s, a ) -r -γf ′ ( s ′ , π )) 2 ] . E D ( M ) is the model-fitting loss that ensures the model is dataconsistent. β and λ control the effect of the pessimistic loss, by constraining Q-functions and models the adversary can choose. Once the adversary is updated, we update the policy (Line 6) to maximize the pessimistic loss as defined in Eq. (5). Similar to Cheng et al. (2022), we choose one Q-function and a slower learning rate for the policy updates ( η fast ≫ η slow ).

We remark that E w D M not only affects f 1 , f 2 , but also M , i.e., it forces the model to generate transitions where the Q-function is Bellman consistent. This allows the pessimistic loss to indirectly affect the model learning, thus making the model adversarial. Consider the special case where λ = 0 in the loss of Line 4. The model here is no longer forced to be data consistent, and the adversary can now freely update the model via E w D M such that the Q-function is always Bellman consistent. As a consequence, the algorithm becomes equivalent to IL on the model-generated states. We empirically study this behavior in our experiments (Section 5).

Lines 7 and 8 describe our model-based rollout procedure. We incrementally rollout both π and π ref from states in D mini real for a horizon H , and add the generated transitions to D model. The aim of this strategy is to generate a distribution with large coverage for training the adversary and policy, and we discuss this in detail in the next section.

Finally, it is important to note the fact that neither the pessimistic nor the Bellman surrogate losses uses the real transitions; hence our algorithm is completely model-based from a statistical point of view, that the value function f is solely an intermediate variable that helps in-model optimization and not directly fit from data.

5 Experiments

We test the efficacy of ARMOR on two major fronts: (1) performance comparison to existing offline RL algorithms, and (2) robust policy improvement over a reference policy that is not covered by the dataset, a novel setting that is not applicable to existing works 4 . We use the D4RL (Fu et al., 2020) continuous control benchmarks datasets for all our experiments and the code will be made public.

Experimental Setup: We parameterize π, f 1 , f 2 and M using feedforward neural networks, and set η fast = 5 e -4 , η slow = 5 e -7 , w = 0 . 5 similar to Cheng et al. (2022). In all our experiments, we vary only the β and λ parameters which control the amount of pessimism; others are fixed. Importantly, we set the rollout horizon to be the max episode horizon defined in the environment.

The dynamics model is pre-trained for 100k steps using model-fitting loss on the offline dataset. ARMOR is then trained for 1M steps on each dataset. Refer to Appendix F for more details.

5.1 Comparison with Offline RL Baselines

By setting the reference policy to the behavior-cloned policy on the offline dataset, we can use ARMOR as a standard offline RL algorithm. Table 1 shows a comparison of the performance of ARMOR against SoTA model-free and model-based offline RL baselines. In the former category, we consider ATAC (Cheng et al., 2022), CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2021), and for the latter we consider MoREL (Kidambi et al., 2020), MOPO (Yu et al., 2020), and RAMBO(Rigter et al., 2022). We also compare against COMBO (Yu et al., 2021) which is a hybrid model-free and model-based algorithm. In these experiments, we initially warm start the optimization for 100k steps, by training the policy and Q-function using behavior cloning and temporal difference learning respectively on the offline dataset to ensure the learner policy is initialized to be the same as the reference. Overall, we observe that ARMOR consistently outperforms or is competitive with the best baseline algorithm on most datasets. Specifically, compared to other purely modelbased baselines (MoREL, MOPO and RAMBO), there is a marked increase in performance in the walker2d-med, hopper-med-exp and walker2d-med-exp datasets. We would like to highlight two crucial elements about ARMOR, in contrast to other model-based baselines - (1) ARMOR achieves SoTA performance using only a single neural network to model the MDP, as opposed to complex network ensembles employed in previous model-based offline RL methods (Kidambi et al., 2020; Yu et al., 2021, 2020; Rigter et al., 2022), and (2) to the best of our knowledge, ARMOR is the only purely model-based offline RL algorithm that has shown performance comparable with modelfree algorithms on the high-dimensional Adroit environments. The lower performance compared to RAMBO on halfcheetah-med and halfcheetah-med-replay may be attributed to that the much larger computational budget used by RAMBO is required for convergence on these datasets.

5.2 Robust Policy Improvement

Next, we test whether the practical version of ARMOR demonstrates RPI of the theoretical version. We consider a set of 14 datasets comprised of the medium and medium-replay versions of D4RL locomotion tasks, as well as the human and cloned versions of the Adroit tasks, with the reference policy set to be the stochastic behavior cloned policy on the expert dataset. We chose these combi-

hopper

walker2d

halfcheetah

med

med-replay

med

med-replay

med

med-replay

0

50

100

Normalized Returns

ORL

REF

0.01

0.05

0.1

0.5

1.0

pen

hammer

door

relocate

human

cloned

human

cloned

human

cloned

human

cloned

0

50

100

Normalized Returns

nations of dataset quality and reference, to ensure that the reference policy takes out-of-distribution actions with respect to the data. Unlike Sec. 5.1 here the reference policy is a black-box given as a part of the problem definition. This opens the question of how the learner should be initialized, since we can not trivially initialize the learner to be the reference as in the previous experiments. 6 In a similar spirit to Sec. 5.1, one might consider initializing the learner close to the reference by behavior cloning the reference policy on the provided dataset during warmstart, i.e, by replacing the dataset actions with reference actions. However, when the reference chooses out of support actions, this procedure will not provide a good global approximation of the reference policy, which can make the optimization problem harder. Instead, we propose to learn a residual policy where the learned policy outputs an additive correction to the reference (Silver et al., 2018). This is an appropriate choice since ARMOR does not make any restrictive assumptions about the structure of the policy class. Figure 3 shows the normalized return achieved by ARMOR for different β , with fixed values for remaining hyperparameters. We observe that ARMOR is able to achieve performance comparable or better than the reference policy for a range of β values uniformly across all datasets, thus verifying the RPI property in practice. Specifically, there is significant improvement via RPI in the hammer , door and relocate domains, where running ARMOR as a pure offline RL algorithm(Section 5.1) does not show any progress 7 . Overall, we note the following metrics:

In 14/14 datasets, ARMOR shows RPI (i.e., ARMOR policy is no worse than the reference when measured by overlap of confidence intervals). Further, considering the difference between ORL and REF as a rough indication of whether the reference is within data support, we note that in 12/14 cases REF is strictly better than ORL, and in all those cases ARMOR demonstrates RPI.

In 5/14 datasets, the ARMOR policy is strictly better than the reference. (Criterion: the lower confidence of ARMOR performance is better than upper confidence of REF). It is important to note that this metric is highly dependent on the quality of the reference policy. Since the reference is near-expert, it can be hard for some environments to improve significantly over it.

6 Discussion

The RPI of ARMOR is highly valuable as it allows easy tuning of the pessimism hyperparameter without performance degradation. We believe that leveraging this property can pave the way for real-world deployment of offline RL. Thus, we next present a discussion of RPI. 8

When does RPI actually improve over the reference policy?

Given ARMOR's ability to improve over an arbitrary policy, the following question naturally arises: Can ARMOR nontrivially improve the output policy of other offline algorithms, including itself? If this were true, can we repeatedly run ARMOR to improve over itself and obtain the best policy any algorithm can learn offline? Unfortunately, the answer is negative. Not only can ARMOR not

improve over itself, but it also cannot improve over a variety of algorithms (e.g., absolute pessimism or minimax regret). In fact, the optimal policy of an arbitrary model in the version space M α is provably unimprovable ( Corollary 10; Appendix D). With a deep dive into when RPI gives nontrivial improvement (Appendix D), we found some interesting observations, which we highlight here.

Return maximization and regret minimization are different in offline RL These objectives generally produce different policies, even though they are equivalent in online RL. Their equivalence in online RL relies on the fact that online exploration can eventually resolve any uncertainty. In offline RL with an arbitrary data distribution, there will generally be model uncertainty that cannot be resolved, and the worst-case reasoning over such model uncertainty (i.e., M α ) leads to definitions that are no longer equivalent. Moreover, it is impossible to compare return maximization and regret minimization and make a claim about which is better. They are not simply an algorithm design choice, but are definitions of the learning goals and guarantees themselves -and are thus incomparable: if we care about obtaining a guarantee for the worst-case return, the return maximization is optimal by definition; if we are more interested in a guarantee for the worst-case regret, then regret minimization is optimal. We also note that analyzing algorithms under a metric that is different from the one they are designed for can lead to unusual conclusions, e.g., Xiao et al. (2021) show that optimistic/neutral/pessimistic algorithms are equally minimax-optimal in terms of their regret guarantees in offline multi-armed bandits. However, the algorithms they consider are optimistic/pessimistic with respect to the return (as commonly considered in the offline RL literature) not the regret which is the performance metric they are interested in analyzing.

π ref is more than a hyperparameter-it defines the performance metric and learning goal Corollary 10 in Appendix D shows that ARMOR has many different fixed points: when π ref is chosen from these fixed points, the solution to Eq. (1) is also π ref . Furthermore, some of them may seem quite unreasonable for offline learning (e.g., the greedy policy to an arbitrary model in M α or even the optimistic policy). This is not a defect of the algorithm. Rather, because of the unresolvable uncertainty in the offline setting, there are many different performance metrics/learning goals that are generally incompatible/incomparable, and the agent designer must make a conscious choice among them and convey the intention to the algorithm. In ARMOR, such a choice is explicitly conveyed by π ref , which makes ARMOR subsume return maximization and regret minimization as special cases.

7 Conclusion

We have presented a model-based offline RL framework, ARMOR, that can improve over arbitrary reference policies regardless of data coverage, by using the concept of relative pessimism. ARMOR provides strong theoretical guarantees with general function approximators, and exhibits robust policy improvement over the reference policy for a wide range of hyper-parameters. We have also presented a scalable deep learning instantiation of the theoretical algorithm. Empirically, we demonstrate that ARMOR indeed enjoys the RPI property, and has competitive performance with several SoTA model-free and model-based offline RL algorithms, while employing a simpler model architecture (a single MDP model) than other model-based baselines that rely on ensembles. This also opens the opportunity to leverage high-capacity world models (Hafner et al., 2023) with offline RL in the future. However, there are also some limitations . While RPI holds for the pessimism parameter, the others still need to be tuned. In practice, the non-convexity of the optimization can also make solving the two-player game challenging. For instance, if the adversary is not strong enough (i.e., far from solving the inner minimization), RPI would break. Further, runtime of ARMOR is slightly slower than model-free algorithms owing to extra computations for model rollouts.

Acknowledgments and Disclosure of Funding

Nan Jiang acknowledges funding support from NSF IIS-2112471 and NSF CAREER IIS-214178.

References

Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in Neural Information Processing Systems , 33:20095-20107, 2020.

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellmanresidual minimization based fitted policy iteration and a single sample path. Machine Learning , 71(1):89-129, 2008.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051, 2019.

Xiong-Hui Chen, Yang Yu, Zheng-Mao Zhu, Zhihua Yu, Zhenjun Chen, Chenghe Wang, Yinan Wu, Hongqiu Wu, Rong-Jun Qin, Ruijin Ding, et al. Adversarial counterfactual environment model learning. arXiv preprint arXiv:2206.04890 , 2022.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. International Conference on Machine Learning , 2022.

Amir Massoud Farahmand, Rémi Munos, and Csaba Szepesvári. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems , 2010.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems , 34:20132-20145, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International Conference on Machine Learning , pages 1587-1596. PMLR, 2018.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , pages 2052-2062, 2019.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Sham M Kakade. A natural policy gradient. Advances in Neural Information Processing Systems , 14, 2001.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Modelbased offline reinforcement learning. In Advances in Neural Information Processing Systems , 2020.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations , 2015.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit qlearning. arXiv preprint arXiv:2110.06169 , 2021.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy qlearning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32:11784-11794, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforcement learning: State-of-the-art , pages 45-73, 2012.

Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning , pages 3652-3661. PMLR, 2019.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforcement learning not scary? In Conference on Learning Theory , volume 178, pages 5175-5220. PMLR, 2022.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with stationary distribution correction. In Uncertainty in Artificial Intelligence , pages 1180-1190. PMLR, 2020a.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in Neural Information Processing Systems , 33:1264-1274, 2020b.

Rémi Munos. Error bounds for approximate policy iteration. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning , pages 560-567, 2003.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems , 35:16082-16097, 2022.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning , pages 19967-20025. PMLR, 2022.

Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 , 2020.

Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning. arXiv preprint arXiv:1812.06298 , 2018.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations , 2021.

Sara A van de Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Chenjun Xiao, Yifan Wu, Jincheng Mei, Bo Dai, Tor Lattimore, Lihong Li, Csaba Szepesvari, and Dale Schuurmans. On the optimality of batch policy optimization algorithms. In International Conference on Machine Learning , pages 11362-11371. PMLR, 2021.

Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence , pages 550-559. PMLR, 2020.

Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In International Conference on Machine Learning , pages 11404-11413. PMLR, 2021.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems , 34:6683-6694, 2021a.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems , 34:27395-27407, 2021b.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems , 34:28954-28967, 2021.

Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in Neural Information Processing Systems , 34, 2021.

Tong Zhang. From ε -entropy to kl-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics , 34(5):2180-2210, 2006.

A Proofs for Section 3

A.1 Technical Tools

Lemma 4 (Simulation lemma) . Consider any two MDP model M and M ′ , and any π : S → ∆( A ) , we have

Lemma 4 is the standard simulation lemma in model-based reinforcement learning literature, and its proof can be found in, e.g., Uehara and Sun (2021, Lemma 7).

A.2 MLEGuarantees

We use ℓ D ( M ) to denote the likelihood of model M = ( P, R ) with offline data D , where

For the analysis around maximum likelihood estimation, we largely follow the proving idea of Agarwal et al. (2020); Liu et al. (2022), which is inspired by Zhang (2006).

The next lemma shows that the ground truth model M ⋆ has a comparable log-likelihood compared with MLE solution.

Lemma 5. Let M ⋆ be the ground truth model. Then, with probability at least 1 -δ , we have

Proof of Lemma 5. The proof of this lemma is obtained by a standard argument of MLE (see, e.g., van de Geer, 2000). For any M ∈ M ,

Then by Markov's inequality, we obtain

Therefore, taking a union bound over M , we obtain

This completes the proof.

The following lemma shows that, the on-support error of any model M ∈ M can be captured via its log-likelihood (by comparing with the MLE solution).

Lemma 6. For any model M , we have with probability at least 1 -δ ,

where ℓ D ( · ) is defined in Eq. (8).

Proof of Lemma 6. By Agarwal et al. (2020, Lemma 25), we have

where µ × P M ⋆ denote the ground truth offline joint distribution of ( s, a, s ′ ) .

Let ˜ D = { ( ˜ s i , ˜ a i , ˜ r i , ˜ s ′ i ) } n i =1 ∼ µ be another offline dataset that is independent to D . Then,

We use ℓ M ( s, a, s ′ ) as the shorthand of -1 2 log ( P M⋆ ( s ′ | s,a ) P M ( s ′ | s,a ) ) , for any ( s, a, s ′ ) ∈ S × A × S . By Agarwal et al. (2020, Lemma 24) (see also Liu et al., 2022, Lemma 15), we know

Thus, we can use Chernoff method as well as a union bound on the equation above to obtain the following exponential tail bound: with probability at least 1 -δ , we have for all ( P, R ) = M ∈ M ,

Plugging back the definition of ℓ M and combining Eqs. (10) to (12), we obtain

Therefore, we obtain

This completes the proof.

A.3 Guarantees about Model Fitting Loss

Let M be the ground truth model. Then, with probability at least 1 -δ , we have

Lemma 7. ⋆ , where E D is defined in Eq. (3).

Proof of Lemma 7. By defition, we know

In addition, by Xie et al. (2021a, Theorem A.1) (with setting γ = 0 ), we know w.p. 1 -δ ,

Combining Eqs. (13) and (14) and using the fact of V max ≥ 1 , we have w.p. 1 -δ ,

This completes the proof.

Lemma 8. For any M ∈ M , we have with probability at least 1 -δ ,

where E D is defined in Eq. (3).

Proof of Lemma 8. By Lemma 6, we have w.p. 1 -δ ,

Also, we have

where the last inequality is a direct implication of Xie et al. (2021a, Lemma A.4). Combining Eqs. (15) and (16) and using the fact of V max ≥ 1 , we obtain

This completes the proof.

A.4 Proof of Main Theorems

Proof of Theorem 2. By the optimality of ̂ π (from Eq. (1)), we have

where step ( ⋆ ) follows from Lemma 5 so that we have M ⋆ ∈ M α , and the last step is because of π † ∈ Π . By the simulation lemma (Lemma 4), we know for any policy π and any M ∈ M α ,

where the step ( ‡ ) follows from the assumption of M ⋆ ∈ M , and last step is because E D ( M ) -min M ′ ∈M E D ( M ′ ) ≤ α = O (log( |M| / δ ) by Eq. (2).

Combining Eqs. (17) and (18), we obtain

This completes the proof.

Note that, over the proof above, only steps ( ⋆ ) and ( ‡ ) have used the realizability assumption of M ⋆ ∈ M . To extend that to the misspecification case, where there only exists an ˜ M ⋆ ∈ M such that ˜ M ⋆ is close to M ⋆ up to some misspecification error, we just need the following straightforward accommodations:

A variant of Lemma 5-to ensure that ˜ M ⋆ is included in the version space M α . By doing so, the misspecification error should also be included in the radius of the version space.

Upper bound | J ( π ) -J ˜ M ⋆ ( π ) | for any π using misspecification error. This is a standard argument, and by combining with the item above, ( ⋆ ) becomes J ( ̂ π ) -J ( π ref ) ≥ min M ∈M α [ J M ( ̂ π ) -J M ( π ref )] -misspecification error .

Upper bound difference in model-fitting error E D ( ˜ M ⋆ ) -E D ( M ⋆ ) using misspecification error. Then, step ( ‡ ) becomes, for M ∈ M α ,

Due to the unboundedness of the likelihood, we conjecture that naively defining misspecification error using total variation without any accommodation on the MLE loss may be insufficient for the steps above. To resolve that, we may adopt an alternate misspecification definition, e.g., ∣ ∣ log P M ⋆ ( s ′ | s, a ) -log P ˜ M ⋆ ( s ′ | s, a ) ∣ ∣ ≤ ε, ∀ ( s, a, s ′ ) ∈ S × A × S , or add extra smoothing to the MLE loss with regularization.

Proof of Theorem 3.

A misspecified version of Theorem 3 can be derived similarly to what we discussed about that of Theorem 2. If the policy class is also misspecified, where there exists only ˜ π ref ∈ Π that is close to π ref up to some misspecification error, the second last step of the proof of Theorem 3 becomes -min M ∈M α [ J M ( ˜ π ref ) -J M ( π ref )] ≤ misspecification error by simply applying the performance difference lemma on the difference between ˜ π ref and π ref .

B Proofs for Section 6

Proof of Lemma 9. We prove the result by contradiction. First notice min M ∈M J M ( π ′ ) -J M ( π ′ ) = 0 . Suppose there is π ∈ Π such that min M ∈M α J M (¯ π ) -J M ( π ′ ) > 0 , which implies that J M (¯ π ) > J M ( π ′ ) , ∀ M ∈ M α . Since M⊆M α , we have

which is a contradiction of the maximin optimality. Thus max π ∈ Π min M ∈M α J M (¯ π ) -J M ( π ′ ) = 0 , which means π ′ is a solution.

For the converse statement, suppose π is a fixed point. We can just let ψ ( M ) = -J M ( π ) . Then this pair of π and ψ by definition of the fixed point satisfies Eq. (19).

C Related Work

There has been an extensive line of works on reinforcement with offline/batch data, especially for the case with the data distribution is rich enough to capture the state-action distribution for any given policy (Munos, 2003; Antos et al., 2008; Munos and Szepesvári, 2008; Farahmand et al., 2010; Lange et al., 2012; Chen and Jiang, 2019; Liu et al., 2020a; Xie and Jiang, 2020, 2021). However, this assumption is not practical since the data distribution is typically restricted by factors such as the quality of available policies, safety concerns, and existing system constraints, leading to narrower coverage. As a result, recent offline RL works in both theoretical and empirical literature have focused on systematically addressing datasets with inadequate coverage.

Modern offline reinforcement learning approaches can be broadly categorized into two groups for the purpose of learning with partial coverage. The first type of approaches rely on behavior regularization, where the learned policy is encouraged to be close to the behavior policy in states where there is insufficient data (e.g., Fujimoto et al., 2018; Laroche et al., 2019; Kumar et al., 2019; Siegel et al., 2020). These algorithms ensure that the learned policy performs at least as well as the behavior policy while striving to improve it when possible, providing a form of safe policy improvement guarantees. These and other studies (Wu et al., 2019; Fujimoto and Gu, 2021; Kostrikov et al., 2021) have provided compelling empirical evidence for the benefits of these approaches.

The second category of approaches that has gained prevalence relies on the concept of pessimism under uncertainty to construct lower-bounds on policy performance without explicitly constraining

the policy. Recently, there have been several model-free and model-based algorithms based on this concept that have shown great empirical performance on high dimensional continuous control tasks. Model-free approaches operate by constructing lower bounds on policy performance and then optimizing the policy with respect to this lower bound (Kumar et al., 2020; Kostrikov et al., 2021). The model-based counterparts first learn a world model and the optimize a policy using model-based rollouts via off-the-shelf algorithms such as Natural Policy Gradient (Kakade, 2001) or Soft-Actor Critic (Haarnoja et al., 2018). Pessimism is introduced by either terminating model rollouts using uncertainty estimation from an ensemble of neural network models (Kidambi et al., 2020) or modifying the reward function to penalize visiting uncertain regions (Yu et al., 2020). Yu et al. (2021) propose a hybrid model-based and model-free approach that integrates model-based rollouts into a model-free algorithm to construct tighter lower bounds on policy performance. On the more theoretical side, the offline RL approaches built upon the pessimistic concept (e.g., Liu et al., 2020b; Jin et al., 2021; Rashidinejad et al., 2021; Xie et al., 2021a; Zanette et al., 2021; Uehara and Sun, 2021; Shi et al., 2022) also illustrate desired theoretical efficacy under various of setups.

Another class of approaches employs an adversarial training framework, where offline RL is posed a two player game between an adversary that chooses the worst-case hypothesis (e.g., a value function or an MDP model) from a hypothesis class, and a policy player that tried to maximize the adversarially chosen hypothesis. Xie et al. (2021a) propose the concept of Bellman-consistent pessimism to constrain the class of value functions to be Bellman consistent on the data. Cheng et al. (2022) extend this framework by introducing a relative pessimism objective which allows for robust policy improvement over the data collection policy µ for a wide range of hyper-parameters. Our approach can be interpreted as a model-based extension of Cheng et al. (2022). These approaches provide strong theoretical guarantees even with general function approximators while making minimal assumptions about the function class (realizability and Bellman completeness). Chen et al. (2022) provide an adversarial model learning method that uses an adversarial policy to generate a data-distribution where the model performs poorly and iteratively updating the model on the generated distribution. There also exist model-based approaches based on the same principle (Uehara and Sun, 2021; Rigter et al., 2022) for optimizing the absolute performance. Of these, Rigter et al. (2022) is the closest to our approach, as they also aim to find an adversarial MDP model that minimizes policy performance. They use a policy gradient approach to train the model, and demonstrate great empirical performance. However, their approach is based on absolute pessimism and does not enjoy the same RPI property as ARMOR.

D A Deeper Discussion of Robust Policy Improvement

D.1 How to formally define RPI?

Improving over some reference policy has been long studied in the literature. To highlight the advantage of ARMOR, we formally give the definition of different policy improvement properties.

Definition 2 (Robust policy improvement) . Suppose ̂ π is the learned policy from an algorithm. We say the algorithm has the policy improvement (PI) guarantee if J ( π ref ) -J ( ̂ π ) ≤ o ( N ) / N is guaranteed for some reference policy π ref with offline data D ∼ µ , where N = |D| . We use the following two criteria w.r.t. π ref and µ to define different kinds PI:

(i) The PI is strong if π ref can be selected arbitrarily from policy class Π regardless of the choice data-collection policy µ ; otherwise, PI is weak (i.e., π ref ≡ µ is required).

(ii) The PI is robust if it can be achieved by a range of hyperparameters with a known subset.

Weak policy improvement is also known as safe policy improvement in the literature (Fujimoto et al., 2019; Laroche et al., 2019). It requires the reference policy to be also the behavior policy that collects the offline data. In comparison, strong policy improvement imposes a stricter requirement, which requires policy improvement regardless of how the data were collected. This condition is motivated by the common situation where the reference policy is not the data collection policy. Finally, since we are learning policies offline, without online interactions, it is not straightforward to tune the hyperparameter directly. Therefore, it is desirable that we can design algorithms with these properties in a robust manner in terms of hyperparameter selection. Formally, Definition 2 requires the policy improvement to be achievable by a set of hyperparameters that is known before learning.

Theorem 3 indicates the robust strong policy improvement of ARMOR. On the other hand, algorithms with robust weak policy improvement are available in the literature (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022); this is usually achieved by designing the algorithm to behave like IL for a known set of hyperparameter (e.g., behavior regularization algorithms have a weight that can turn off the RL behavior and regress to IL). However, deriving guarantees of achieving the best data-covered policy of the IL-like algorithm is challenging due to its imitating nature. To our best knowledge, ATAC (Cheng et al., 2022) is the only algorithm that achieves both robust (weak) policy improvement as well as guarantees absolute performance.

D.2 When RPI actually improves?

Given ARMOR's ability to improve over an arbitrary policy, the following questions naturally arise: Can ARMOR nontrivially improve the output policy of other algorithms (e.g., such as those based on absolute pessimism (Xie et al., 2021a)), including itself? Note that outputting π ref itself always satisfies RPI, but such result is trivial. By 'nontrivially' we mean a non-zero worst-case improvement. If the statement were true, we would be able to repeatedly run ARMOR to improve over itself and then obtain the best policy any algorithm can learn offline.

Unfortunately, the answer is negative. Not only ARMOR cannot improve over itself, but it also cannot improve over a variety of algorithms. In fact, the optimal policy of an arbitrary model in the version space is unimprovable (see Corollary 10)! Our discussion reveals some interesting observations (e.g., how equivalent performance metrics for online RL can behave very differently in the offline setting) and their implications (e.g., how we should choose π ref for ARMOR). Despite their simplicity, we feel that many in the offline RL community are not actively aware of these facts (and the unawareness has led to some confusion), which we hope to clarify below.

Setup We consider an abstract setup where the learner is given a version space M α that contains the true model and needs to choose a policy π ∈ Π based on M α . We use the same notation M α as before, but emphasize that it does not have to be constructed as in Eqs. (2) and (3). In fact, for the purpose of this discussion, the data distribution, sample size, data randomness, and estimation procedure for constructing M α are all irrelevant , as our focus here is how decisions should be made with a given M α . This makes our setup very generic and the conclusions widely applicable.

To facilitate discussion, we define the fixed point of ARMOR's relative pessimism step:

Definition 3. Consider Eq. (1) as an operator that maps an arbitrary policy π ref to ̂ π . A fixed point of this relative pessimism operator is, therefore, any policy π ∈ Π such that π ∈ argmax π ′ ∈ Π min M ∈M α J M ( π ′ ) -J M ( π ) .

Given the definition, relative pessimism cannot improve over a policy if it is already a fixed point. Below we show a sufficient and necessary condition for being a fixed point, and show a number of concrete examples (some of which may be surprising) that are fixed points and thus unimprovable.

Lemma 9 (Fixed-point Lemma) . For any M⊆M α and any ψ : M→ R , consider the policy

Then π is a fixed point in Definition 3. Conversely, for any fixed point π in Definition 3, there is a ψ : M→ R such that π is a solution to Eq. (19).

Corollary 10. The following are fixed points of relative pessimism (Definition 3):

Absolute-pessimism policy, i.e., ψ ( M ) = 0 .

Relative-pessimism policy for any reference policy, i.e., ψ ( M ) = -J M ( π ref ) .

Regret-minimization policy, i.e., ψ ( M ) = -J M ( π ⋆ M ) , where π ⋆ M ∈ argmax π ∈ Π J M ( π ) .

Optimal policy of an arbitrary model M ∈ M α , π ⋆ M , i.e., M = { M } . This would include the optimistic policy, that is, argmax π ∈ Π ,M ∈M α J M ( π )

Return maximization and regret minimization are different in offline RL We first note that these four examples generally produce different policies, even though some of them optimize for

objectives that are traditionally viewed as equivalent in online RL (the 'worst-case over M α ' part of the definition does not matter in online RL), e.g., absolute pessimism optimizes for J M ( π ) , which is the same as minimizing the regret J M ( π ⋆ M ) -J M ( π ) for a fixed M . However, their equivalence in online RL relies on the fact that online exploration can eventually resolve any model uncertainty when needed, so we only need to consider the performance metrics w.r.t. the true model M = M ⋆ . In offline RL with an arbitrary data distribution (since we do not make any coverage assumptions), there will generally be model uncertainty that cannot be resolved, and worst-case reasoning over such model uncertainty (i.e., M α ) separates apart the definitions that are once equivalent.

Moreover, it is impossible to compare return maximization and regret minimization and make a claim about which one is better. They are not simply an algorithm design choice, but are definitions of the learning goals and the guarantees themselves-thus incomparable: if we care about obtaining a guarantee for the worst-case return , the return maximization is optimal by definition; if we are more interested in obtaining a guarantee for the worst-case regret , then again, regret minimization is trivially optimal. We also note that analyzing algorithms under a metric that is different from the one they are designed for can lead to unusual conclusions. For example, Xiao et al. (2021) show that optimistic/neutral/pessimistic algorithms 9 are equally minimax-optimal in terms of their regret guarantees in offline multi-armed bandits. However, the algorithms they consider are optimistic/pessimistic w.r.t. the return-as commonly considered in the offline RL literature-not w.r.t. the regret which is the performance metric they are interested in analyzing.

π ref is more than a hyperparameter-it defines the performance metric and learning goal Corollary 10 shows that ARMOR (with relative pessimism) has many different fixed points, some of which may seem quite unreasonable for offline learning, such as greedy w.r.t. an arbitrary model or even optimism (#4). From the above discussion, we can see that this is not a defect of the algorithm. Rather, in the offline setting with unresolvable model uncertainty, there are many different performance metrics/learning goals that are generally incompatible/incomparable with each other, and the agent designer must make a choice among them and convey the choice to the algorithm. In ARMOR, such a choice is explicitly conveyed by the choice of π ref , which subsumes return maximization and regret minimization as special cases (#2 and #3 in Corollary 10)

E A More Comprehensive Toy Example for RPI

We illustrate with a simple toy example why ARMOR intuitively demonstrates the RPI property even when π ref is not covered by the data D . ARMOR achieves this by 1) learning an MDP Model, and 2) adversarially training this MDP model to minimize the relative performance difference to π ref during policy optimization. Consider a one-dimensional discrete MDP with five possible states as shown in Figure 4. The dynamics is deterministic, and the agent always starts in the center cell. The agent receives a lower reward of 0.1 in the left-most state and a high reward of 1.0 upon visiting the right-most state. Say, the agent only has access to a dataset from a sub-optimal policy that always takes the left action to receive the 0.1 reward. Further, let's say we have access to a reference policy that demonstrates optimal behavior on the true MDP by always choosing the right action to visit the right-most state. However, it is unknown a priori that the reference policy is optimal. In such a case, typical offline RL methods can only recover the sub-optimal policy from the dataset as it is the best-covered policy in the data. Now, for the sake of clarity, consider the current learner policy is same as the behavior policy, i.e it always takes the left action.

ARMORcanlearn to recover the expert reference policy in this example by performing rollouts with the adversarially trained MDP model. From the realizability assumption we know that the version space of models contains the true model (i.e., M ⋆ ∈ M α ). The adversary can then choose a model from this version space where the reference policy π ref maximally outperforms the learner. Note, that ARMOR does not require the true reward function to be known. In this toy example, the model selected by the adversary would be the one that not only allows the expert policy to reach the rightmost state, but also predicts the highest reward for doing so. Now, optimizing to maximize relative performance difference with respect to this model will ensure that the learner can recover the expert behavior, since the only way for the learner to stay competitive with the reference policy is to mimic the reference policy in the region outside data support. In other words, the reason why ARMOR has RPI to π ref is that its adversarial model training procedure can augment the original offline data with

Possible Dynamics and Rewards in Unknown Region

+0.1

+0.1

al

al

al

ar

ar

+0.1

+0.9

+0.1

al

al

al

ar

ar

ar

ARMOR chooses worst-case models

+0.1

+1.0

al

al

al

ar

ar

ar

+0.1

+1.0

al

al

ar

ar

ar

True MDP

Dataset Transitions

+0.1

al

al

ar

+0.1

+1.0

ar

ar

al

al

al

+0.1

10.0

ar

al

al

al

ar

ar

+0.1

+10

al

al

al

ar

ar

ar

al

+0.1

+5.0

+5.0

al

al

al

ar

ar

ar

new states and actions that would cover those generated by running the reference policy in the true environment, even though ARMOR does not have knowledge of M ⋆ .

F Further Experimental Details

F.1 Experimental Setup and Hyper-parameters

Werepresent our policy π , Q-functions f 1 , f 2 and MDP model M as standard fully connected neural networks. The policy is parameterized as a Gaussian with a state-dependent covariance, and we use a tanh transform to limit the actions to the action space bound similar to Haarnoja et al. (2018). The MDP model learns to predict the next state distribution, rewards and terminal states, where the reward and next-state distributions part are parameterized as Gaussians with state-dependent covariances. The model fitting loss consists of negative log-likelihood for the next-state and reward and binary cross entropy for the terminal flags. In all our experiments we use the same model architecture and a fixed value of λ . We use Adam optimizer (Kingma and Ba, 2015) with fixed learning rates η fast and η slow similar to Cheng et al. (2022). Also similar to prior work (Kidambi et al., 2020), we let the MDP model network predict delta differences to the current state. The rollout horizon is always set to the maximum episode steps per environment. A complete list of hyper-parameters can be found in Table 3.

Compute: Each run of ARMOR has access to 4CPUs with 28GB RAM and a single Nvidia T4 GPU with 16GB memory. With these resources each run tasks around 6-7 hours to complete. Including all runs for 4 seeds, and ablations this amounts to approximately 2500 hours of GPU compute.

F.2 Detailed Performance Comparison and RPI Ablations

In Table 4 we show the performance of ARMOR compared to model-free and model-based offline RL baselines with associate standard deviations over 8 seeds. For ablation, here we also include ARMOR † , which is running ARMOR in Algorithm 1 but without the model optimizing for the Bellman error (that is, the model is not adversarial). Although ARMOR † does not have any theoretical guarantees (and indeed in the worst case its performance can be arbitrarily bad), we found that ARMOR † in these experiments is performing surprisingly well. Compared with ARMOR, ARMOR † has less stable performance when the dataset is diverse (e.g. -med-replay datasets) and larger learning variance. Nonetheless, ARMOR † using a single model is already pretty competitive with other algorithms. We conjecture that this is due to that Algorithm 1 also benefits from pessimism due to adversarially trained critics. Since the model buffer would not cover all states and actions (they are continuous in these problems), the adversarially trained critic still controls the pessimism for actions not in the model buffer, as a safe guard. As a result, the algorithm can tolerate the model quality more.

F.3 Effect of Residual Policy

In Figure 5, we show the effect on RPI of different schemes for initializing the learner for several D4RL datasets. Specifically, we compare using a residual policy( Section 5) versus behavior cloning the reference policy on the provided offline dataset for learner initialization. Note that this offline dataset is the suboptimal one used in offline RL and is different from the expert-level dataset used to train and produce the reference policy. We observe that using a residual policy (purple) consistently shows RPI across all datasets. However, with behavior cloning initialization (pink), there is a large variation in performance across datasets. While RPI is achieved with behavior cloning initialization on hopper , walker2d and hammer datasets, performance can be arbitrarily bad compared to the reference on other problems. As an ablation, we also study the effect of using a residual policy in

med

med-replay

0

50

100

Normalized Returns

(a) Hopper

med

med-replay

0

50

100

Normalized Returns

(b) Walker2d

med

med-replay

0

50

100

Normalized Returns

(c) Halfcheetah

human

cloned

0

50

100

Normalized Returns

(d) Pen

human

cloned

0

50

100

Normalized Returns

(e) Hammer

human

cloned

0

50

100

Normalized Returns

(f) Door

human

cloned

0

50

100

Normalized Returns

(g) Relocate

the offline RL case where no explicit reference is provided, and the behavior cloning policy is used as the reference similar to Section 5.1. We include the results in Table 4 as ARMOR re , where we observe that using a residual policy overall leads to worse performance across all datasets. This lends evidence to the fact that using a residual policy is a compromise in instances where initializing the learner exactly to the reference policy is not possible.

F.4 Connection to Imitation Learning

As mentioned in Section 4.2, IL is a special case of ARMOR with λ = 0 . In this setting, the Q-function can fully affect the adversarial MDP model, so the best strategy of the policy is to mimic the reference. We test this on the expert versions of the D4RL locomotion tasks in Table 5, and observe that ARMOR can indeed perform IL to match expert performance.

F.5 Ablation Study: RPI for Different Reference Policies

Here we provide ablation study results for robust policy improvement under different reference policies for a wide range of β values (pessimism hyper-parameter). For all the considered reference policies we present average normalized scroes for ARMOR and reference (REF) over multiple random seeds, and observe that ARMOR can consistently outperform the reference for a large range of β values.

Random Dataset Reference We use a reference policy obtained by running behavior cloning on the RANDOM versions of different datasets. This is equivalent to using a randomly initialized neural network as the reference.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA fundamental principle in offline RL is pessimism, which however is not free from the performance degradation with respect to the baseline reference policies, i.e., the currently running policies in the system.\nIn viewing this issue, this paper proposes Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies that improve upon an arbitrary reference policy, regardless of the data quality.\nSpecifically, in ARMOR, the authors extend the technique of relative pessimism by adversarially training an MDP model, to achieve robust policy improvement (RPI) with arbitrary reference policies, regardless of whether they collected the data or not.\nBased on the theory, a scalable implementation of ARMOR is developed, which achieves competitive performance on D4RL benchmarks, while using only a single model rather than the widely-used ensembles.\n\nSTRENGTHS:\n1. The proposed method is theoretically supported.\n2. Practical implementation, including the selection of key hyperparameters, is discussed in details.\n3. Achieving competitive performance while using only a single model worth praising, since this is important for utilizing more sophisticated dynamics model.\n\nWEAKNESSES:\n1. The assumptions, in particular the realizability of the model class $\\mathcal{M}$ and $\\mathcal{M_\\alpha}$, and the policy class $\\Pi$, maybe demanding. It is unclear if or how these assumptions holds in practice. It is also unclear if the theoretical results still hold with a misspecified model and/or policy class, e.g., $M^*  \\notin \\mathcal{M}$, $M^*  \\notin \\mathcal{M_\\alpha}$ , and/or $\\pi_{\\mathrm{ref}} \\notin \\Pi$.  \n2. The authors draw a distinction between the reference policy and the data-collecting behavior policy. But in the experiments, the reference policy is simply set as the behavior-cloned policy on the same or a very similar offline dataset, e.g., expert v.s. medium. Therefore, this conceptual distinction seems artificial. \n    * In practice/experiments, can the learned policy still improve upon the reference when the behavior policy (offline dataset) behaves significantly different from the reference policy? \n    * And more importantly, why couldn't we simply learn from the data generated by the \"reference policy\"?\n3. The authors claim that the proposed method is robust to hyperparameters within a known set, i.e., the RPI property. But there is no experiment directly showing this. I would love to see more evidence and details pertaining to, for examples,\n    * How to select this set?    \n    * How large can this set be? \n    * How many hyperparameters can this set incorporate? \n4. Except the proposed method, the practical implementation (Section 4) shows many differences from the implementations of the baseline methods, such as MOPO, COMBO, CQL, IQL and so on. It is therefore unclear if the performance gain is due to the proposed method or due to these different choices of implementation details. This can be seen from the fact that the closely-related model-free method ATAC has performance similar to ARMOR and better than baselines on several datasets.\n\nQUESTIONS:\n1. Can existing model-based methods, such as MOPO, MoREL or COMBO, utilize the reference policy as well? For example, a quick (and naive) thought may be adding an extra behavior-cloning term towards the reference policy on model-generated states (similar to L153-155). I am curious what preventing them from doing so? Otherwise, with such an augmentation, are they still underperform ARMOR?\n2. Could you elaborate more on the distinction between the reference policy and behavior policy? Are there any practical setting where the reference policy is significantly different from the policy that collect the offline dataset?  \n3. In Eqn. (1) and (2), why do you introduce the $\\alpha$ parameter? Maybe I miss something, but I don't find $\\alpha$ being used later on, except something like \"for $\\alpha$ large enough\".\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ARMOR (Adversarial Model for Offline Reinforcement Learning), a novel model-based framework for offline reinforcement learning (RL) that addresses the challenge of performance degradation. Offline RL allows learning decision-making policies from logged data without requiring new data collection. ARMOR robustly learns policies that improve upon a reference policy by adversarially training a Markov decision process (MDP) model. The framework optimizes policies for worst-case performance relative to the reference policy, ensuring robust policy improvement regardless of data coverage. The authors provide theoretical proofs that ARMOR competes with the best policy within data coverage and never degrades the performance of the reference policy, even when the reference policy is not covered by the dataset.\n\nTo validate their claims, the authors present a scalable implementation of ARMOR that achieves state-of-the-art performance on D4RL benchmarks without using model ensembles. This makes ARMOR suitable for using high-capacity world models. The empirical results support the robust policy improvement property of ARMOR.\n\nSTRENGTHS:\nOverall, ARMOR offers a promising solution for offline RL, providing robust policy improvement over a broad range of hyperparameter choices and regardless of data coverage. The framework has practical implications for real-world problems where collecting diverse or expert-quality data is expensive or infeasible. The paper contributes theoretical analysis, a scalable implementation, and empirical validation.\n\nWEAKNESSES:\n1. Inadequate Discussion of Existing Limitations: The paper briefly mentions that offline RL algorithms have not been widely adopted beyond academic research, but it fails to provide a comprehensive discussion of the existing limitations and challenges that hinder their practical adoption. Without a clear identification and discussion of the existing limitations, the paper's motivation may appear less compelling.\n\n2. Conflicting results:  There is uncertainty regarding the claim that the policy learned by the proposed approach outperforms different hyper-parameter settings. In Figure 1, significant variations in performance can be observed for the different hyper-parameter choices. It is important to address and discuss this variation in more detail to provide a clearer understanding of the robustness and reliability of the proposed approach across different hyper-parameter settings.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose a new model-based offline RL framework, called ARMOR, which can robustly learn policies that improve upon an reference policy by adversarially training a Markov decision process (MDP) model.  ARMOR aims to optimize for the worst-case relative performance over uncertainty. In experiment, ARMOR implementation achieves good performance on D4RL benchmarks using only a single model.\n\nSTRENGTHS:\n1. The idea is novel. The paper give us a new perspective to learn a dynamics model in offline RL, which is valuable to the community;\n2. The paper is overall well-written and easy to follow for me;\n3. The implementation is generally reasonble in intuition.\n\nWEAKNESSES:\n1. The description in Section 4.1 is quite confusing, with many details tucked away in the appendix. This arrangement is not particularly reader-friendly. I suggest that the authors better link the following points: (1) why the objective in Line 228 can substitute for Eq. (1); (2) how Line 228 is transformed into the goal of Line 199 and Algorithm 1. You don't need to elaborate on the relationship, but at least some intuition or motivation should be provided to the reader. Additionally, an optional suggestion: the authors might consider restructuring the overall arrangement by first introducing the part of Line 228 and then describing how it is implemented, which may result in a more natural discourse.\n\n2. The notation used for expressing errors is confusing here, with many instances of $\\mathcal{E}$ and $\\mathcal{L}$ representing loss, lacking in distinctiveness. A classic example is the symbols in Line 228, $\\mathcal{E}\\_{D}$  and $\\mathcal{E}_{\\hat D}$, both error symbols share the same structure, only the parameters differ. But they represent completely different losses, with the former being the maximum likelihood, and the latter the Bellman error. I suggest the authors revise the notations used, making it clear to readers what kind of loss it is without having to inspect the parameter differences.\n\n\n3. Figure 2 could be further polished: Without reading the main text, readers would not understand why ARMOR chooses the two models on the right, as the figure (including the caption) does not demonstrate that the reference policy is actually moving rightward. I believe the authors could add this information, allowing readers to better grasp the authors' motivation.\n\n3. Although D4RL is a fairly popular benchmark, I think the tasks from D4RL do not adequately highlight the advantages of the proposed algorithm. As the authors mentioned, this algorithm is motivated by real-world application scenarios, like \"Usually, the systems we apply RL to have currently running policies, such as an engneeded autonomous driving rule or a heuristic-based system for diagnosis, and the goal of applying a learning algorithm is often to further improve upon these baseline reference policies...\". However, the data in the selected D4RL datasets mostly come from mixed policies. The medium dataset aligns with this scenario, but the baselines already perform well, making it hard to discern the advantage of the proposed method. The recently released neorl [1] dataset might be more suitable for this work, as it was specifically designed for the scenarios the authors proposed, with all datasets collected by a single working policy, a concept similar to the authors' reference policy. \n\n\n\n[1] NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning\n\nQUESTIONS:\nThe implementation of Equation (4) somewhat contradicts my intuition. Specifically,\n- $M$'s initial goal was to enhance $\\pi_{\\rm ref}$'s $J$ and reduce $\\pi$'s $J$. However, in the final implementation, none of $M$'s optimization aspects involve $\\pi_{\\rm ref}$. This makes it unclear how the losses employed in practice can meet the original optimization objective of Equation (1).\n- I'm unclear on why the optimization process for the critic necessitates a pessimistic loss, e.g., $\\mathcal{L}_{D_M}$? The need for pessimism about the critic's estimates isn't evident in the objective Equation (1).\n\nIn Table 1, I'm curious on why several cloned datasets don't work by ARMOR? This seems to be the datasets very suitable for ARMOR to show better performance.\n\nIn Figure 3, the algorithm shows sensitivity to $\\beta$. It would be beneficial if the authors could elucidate the underlying reasons for this and provide any principles to guide researchers.\n\nThe experiments in Section 5.2 could be improved. The authors claim that \"ARMOR can robustly learn policies that improve upon an arbitrary reference policy...\" which is important. However, in the experiments, only performance improvements under a specific reference policy are demonstrated. It would be more constructive to verify this claim by picking various policies as reference policies and conduct experiments under one or two environments. I think DOPE [1] can be referenced as a work to conduct this experiment, which preserves many policies that can be used as reference policies.\n\nRegarding related work, a discussion could be made with the work [2] that utilizes adversarial approaches for offline environment learning. The optimization objective of this work coincidently using an opposite objective to the Equation (1) of this paper for model learning, which is \"$\\max_M \\min_\\pi \\ell(M, \\pi)$\".\n\nAlso suddenly inspired by the work [2], I would like to propose an open question for discussion: Can we also consider using adversarial policies further with ARMOR, such as $\\max_\\pi \\min_M \\max_{\\pi_{\\rm ref}}$? Since constructing $\\pi_{\\rm ref}$ from the offline data isn't a necessity, following Theorem 3, if we can create a sufficient number of $\\pi_{\\rm ref}$ in an adversarial manner, could we theoretically ensure that the final $\\hat \\pi$ approximates the optimal policy in any situation?\n\n[1]Benchmarks for Deep Off-Policy Evaluation.\n\n[2] Adversarial Counterfactual Environment Model Learning.\n\nLIMITATIONS:\nNAN", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Adversarial Model for Offline Reinforcement Learning (ARMOR), a model-based offline RL framework. ARMOR uses adversarial training to robustly learn and improve policies over any given reference policy, regardless of data quality. The framework utilizes the concept of 'relative pessimism' for worst-case optimization, ensuring that it either maintains or improves upon the performance of the reference policy, a property known as Robust Policy Improvement (RPI). The authors have also shown, theoretically and empirically, that ARMOR is robust to hyperparameter choices, and can outperform or be on par with state-of-the-art offline RL methods.\n\nSTRENGTHS:\n* The paper is well-written and technically sound. The RPI in this paper is stronger than the RPI property in the literature, which only guarantees to be no worse than the behavior policy that collected the data. \n* The experimental results show the effectiveness of the proposed method and it outperforms existing baselines in many environments.\n\nWEAKNESSES:\n* I think the proposed method highly depends on the quality of the MDP model $M$. For the results in Table 1, in some environments, the proposed method does not outperform baselines. I think some explanation and intuitions are needed. I am wondering if the $M$'s structure is not optimal in those cases.\n* The model fitting loss in Equation (3) is not well-defined. Specifically, $r$ is not mentioned. Is it the same as $R^*$ as in Definition 1?\n* Minor: Please refrain from only using color to distinguish bars in Figure 3, as it is not friendly to readers with color blindness. Also, in Figure 2, not all illustrations are valid MDPs. I suggest the authors re-draw those toy examples.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}