Adversarial Model for Offline Reinforcement Learning
Authors: Mohak Bhardwaj
Tengyang Xie
Byron Boots
Nan Jiang
Ching-An Cheng
Venue: NeurIPS 2023 poster (2023)
Keywords: model based
offline
reinforcement learning
adversarial training
TL;DR: A model-based offline RL framework that uses adversarial training to robustly improve over arbitrary reference policies regardless of data coverage, has strong theoretical guarantees and empirical performance without ensembles for dynamics modelling

Abstract: We propose a novel model-based offline Reinforcement Learning (RL) framework, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy regardless of data coverage. ARMOR is designed to optimize policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. In theory, we prove that ARMOR, with a well-tuned hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with any admissible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design a scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensembles in contrast to typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model-based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.

Adversarial Model for Offline Reinforcement Learning
Mohak Bhardwaj' University Washington mohakbêcs washington. edu
Tengyang Xie" Microsoft Research UW-Madison txêcs. wisc.edu
Byron Boots
Nan Jiang
Ching-An Cheng Microsoft Research, Redmond chingancemicrosoft.. com
University of Washington bbootsêcs. www.:esttsssssssso.oooooooo edu
UIUC
sanjiang@illiniss.edd
Abstract
We propose a novel model-based offline Reinforcement Learning (RI.) frame- work, called Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies to improve upon an arbitrary reference policy re- gardless of data coverage. ARMOR is designed to optimize policies for the worst- case performance relative to reference policy through adversarially training Markov decision process model. In theory, well-tumed hyperparameter, can compete with the best policy within data coverage when the reference policy is supported by the data. At the same time, ARMOR is robust to hyperparameter choices: the policy learned by ARMOR, with any ad- missible hyperparameter, would never degrade the performance of the reference policy, even when the reference policy is not covered by the dataset. To validate these properties in practice, we design scalable implementation of ARMOR, which by adversarial training, can optimize policies without using model ensem- bles in contrast typical model-based methods. We show that ARMOR achieves competent performance with both state-of-the-art offline model-free and model- based RL algorithms and can robustly improve the reference policy over various hyperparameter choices.
prove that ARMOR, with
Introduction
Offline reinforcement learning (RL) is technique for learning decision-making policies from logged data (Lange et al., 2012; Levine et al., 2020; Jin et al., 2021; Xie et al., 2021a). In com- parison with alternate learning techniques, such as off-policy RL and imitation learning (IL). offline RL reduces the data assumption needed to learn good policies and does not require collecting new data. Theoretically, offline RI. can learn the best policy that the given data can explain: as long as the offline data includes the scenarios encountered by near-optimal policy, offline RI. algorithm can learn such near-optimal policy, even when the data is collected by highly sub-optimal policies and/or is not diverse. Such robustness to data coverage makes offline RL promising technique for solving real-world problems, as collecting diverse or expert-quality data in practice is often expen- sive or simply infeasible.
The fundamental principle behind offline RL is the concept pessimism, which considers worst- case outcomes for scenarios without data. In algorithms, this is realized by (explicitly or implic- itly) constructing performance lower bounds in policy learning which penalizes uncertain actions.
Equual contribution
2 Open source code is available at:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Normalized Retums
DRL REF 0.01 0.05
0.1
0.5 1.0
II
50-
hopper-med
pen-cloned
hammer-cloned
Figure 1: Robust Policy Improvement: ARMOR can improve performance over the reference policy (REF) over broad range of pessimism hyperparameter (purple) regardless of data coverage. ORI. denotes best offline RL policy without using the reference policy, and reference is obtained by behavior cloning on expert dataset.
Various designs have been proposed to construct such lower bounds, including behavior regulariza- tion (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021), point-wise pessimism based on negative bonuses or truncation (Kidambi et al., 2020; Jin et al., 2021), value penalty (Kumar et al., 2020; Yu et al., 2020), or two-player games (Xie et al., aa; Uehara and Sun, 2021; Cheng et al., 2022). Conceptually, the tighter the lower bound is, the better the learned policy would perform; see a detailed discussion of related work in Appendix C.
Despite these advances, offline RL still has not been widely adopted to build learning- sion systems beyond academic research. One important factor we posit is the issue of performance degradation: Usually, the systems we apply RL to have currently running policies, such as an engi- neered autonomous driving rule or a heuristic-baset system for diagnosis, and the goal of applying a learning algorithm often to further improve upon these baseline reference policies. As a result, it is imperative that the policy learned by the algorithm does not degrade the base performance. This criterion is especially critical for applications where poor decision outcomes cannot be tolerated.
bausing-based based deci-
However, running an offline RL algorithm based on pessimism, in general, is not free from perfor- mance degradation. While there have been algorithms with policy improvement guarantees (Laroche et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020; Fujimoto and Gu, 2021; Cheng et al., 2022), such guarantees apply only to the behavior policy that collects the data, which might not necessar- ily be the reference policy. In fact, quite often these two policies are different. For example, in robotic manipulation, it is common have dataset of activities different from the target task. In such a scenario, comparing against the behavior policy is meaningless, as these policies do not have meaningful performance in the target task.
In this work, we propose a novel model-based offline RL framework, called Advesarial Model for Offline Rinforcement Learning (ARMOR), which can robustly learn policies that improve upon an arbitrary reference policy by adversarially training a Markov decision process (MDP) model, re- gardless of the data quality. ARMOR is designed based on the concept relative pessimism (Cheng et al., 2022), which aims to optimize for the worst-case relative performance over uncertainty. In theory, we prove that, owing to relative pessimism, the ARMOR policy never degrades the perfor- mance of the reference policy for range of hyperparameters which is given beforehand, a property known as Robust Policy Improvement (RPI) (Cheng et al., 2022). In addition, when the right hy- perparameter is chosen, and the reference policy is covered by the data, we prove that the ARMOR policy can also compete with any policy covered by the data in an absolute sense. To our knowl- edge, RPI property of offline RL has so far been limited to comparing against the data collection policy (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022). In ARMOR, by adversarially training an MDP model, we extend the technique of relative pessimism to achieve RPI with arbitrary reference policies, regardless of whether they collected the data not (Fig. 1).
In addition to theory, we design scalable deep-learning implementation of ARMOR to validate these claims that jointly trains an MDP model and the state-action value function to minimize the estimated performance difference between the policy and the reference using model-based based rollouts. Our implementation achieves state-of-the-ar (SoTA) performance on D4RL benchmarks (Fu et al., 2020), while using only single model (in contrast to ensembles used in existing model-based offline RL works). This makes ARMOR a better framework for using high-capacity world models (e.g. (Haferr al., 2023)) for which building an ensemble too expensive. We also empirically validate the RPI property of our implementation.

---

2 Preliminaries
Markoy Decision Process We consider learning in the setup of
discounted
Markov Decision Process (MDP). An MDP M is defined by the tuple (S,A, Pas. RM: 7), where S is the state space, A is the action space, PM : S x A
A (S) is the transition dynam- ics, RM : S x A - [0, 1] is a scalar reward function and Y E [0. 1) is the discount factor. A policy x is mapping from S distribution on A. For n, we let (s, denote the dis- counted state-action distribution obtained by running т on M from an initial state distribution do- ie di,
(1 - 7) E,,, Eي== 7 (st = 8,0, = a]]] Let (M( = Er,M En= Xrr,] be the expected discounted return of policy on M starting from do. where r2 = (st a.). We de- fine the value function as (s) E.,. M D= y'relso = s). and the state-action value function (i.e., Q-function) as 2T (s, a) E. Di=o 7 T'ry|so = 8, 80 = a). By this definition, we note (T) = Ed (s)] Edg. 1h (s, a)]. We use [0, Viax to denote the range of value functions, where Vmaax 2 We denote the ground truth MDP as M and J JM
a) (s,
Offiin RL The aim of offline RL is to find the policy that maximizes J(T),, while using a fixed dataset DD collected by behavior policy ft. We assume the dataset 'D consists of {s an Tn, $n+1)] where (sn, is sampled from d',- and Iwy Sn+1 follow M"; for sim- plicity, we also write [l(s, a) d (s)
We assume that the learner has access to a Markovian policy class and an MDP model class M. Assumption (Realizabilitt). We assume the ground truth model M is in the model class M.
In addition, we assume that we are provided reference policy Te In practice, such a reference policy represents a baseline whose performance we want improve with offline RL and data. Assumption 2 (Reference policy). We assume access to a reference policy Treef which can be queried at any state. We assume ree is realizable, i.e., Tref E III
If ree is not provided, we can still run ARMOR as a typical offline RI. algorithm, by first performing behavior cloning on the data and setting the cloned policy as Tref- In this case, ARMOR has RPI with respect to the behavior policy.
Robust Policy Improvement RPI is notion introduced in Cheng et al. (2022), which means that the offline algorithm can learn to improve over the behavior policy, using hyperparameters within a known set. Algorithms with RPI are more robust to hyperparameter choices, and they are often derived from the principle of relative pessimism (Cheng et al., 2022). In this work, we extend the RPI concept to compare with an arbitrary reference (or baseline) policy, which can be different from the behavior policy and can take actions outside data support.
3 Adversarial Model for Offline Reinforcement Learning (ARMOR)
ARMOR is model-based offline RL algorithm designed with relative pessimism. The goal of ARMOR is to find a policy से that maximizes the performance difference J( - (TTee) to a given reference policy Tref, while accounting for the uncertainty due to limited data coverage. ARMOR achieves this by solving a two-player game between a learner policy and an adversary MDP model:
argmax min (M)( JM (Fref)
well MEM.
based on version space of MDP models
M. = (M E M: ED(M) - min MEE
EM(')) S sa),
(2)
where we define the model fitting loss as
{M(M) == Ep log (ss s, s, + (Bar(s,a)-)) -\////
(3)
and a 2 0 is a bound on statistical errors such that M* E MM. In this two-player game, ARMOR is optimizing lower bound of the relative performance J(T) J (TTet). This is due to the construction that M E M,, which ensures minMeM. ( Jaa (Tret) S JM. JM- (Teer).
One interesting property that follows from optimizing the relative performance lower bound is that î guaranteed to always be no worse than Reef for a wide range of a and regardless of the relationship between re and the data D.
3

---

True MDP
Possible Dynamics in Unknown Region
-0.1
-1.
Dataset Transitions
TG.L
+1.0
-0.1
10.3
+10
$1.0
Reference Policy
ARMOR chooses worsh-case models
110
an
110
Figure 2: A toy MDP illustrating the RPI property of ARMOR. (Top) The true MDP has determin- istic dynamics where taking the left (ai) or right (a,) actions takes the agent to corresponding states; start state is in yellow. The suboptimal behavior policy visits only the left part of the state space, and the reference policy demonstrates optimal behavior by always choosing dy- (Bottom) A subset of possible data-consistent MDP models in the version space. The adversary always chooses the MDP that makes the reference maximally outperform the learner. In response, the learner will learn mimic the reference outside data support be competitive.
Proposition 1. For any a large enough such that M E Ma. it holds that ( 2 J (TTret..
This fact can be easily reasoned: Since Tref E IT, we have max=en minMEM., JM((T) Jast (Tret) > minMEM, Ja Jar(Ted (Tret) - JM (Tret) 0. other words, ARMOR achieves the RPI property with respect to any reference policy Tee and offline dataset 'D.
This RPI property of ARMOR is stronger than the RPI property in the literature. In comparison, previous algorithms with RPI (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022) are only guaranteed be worse than the behavior policy that collected the data. In Section 3.2, we will also show that when is set appropriately, ARMOR can provably compete with the best data covered policy as well, as prior offline RL works (e.g., Xie et al., 2021a; Uchara and Sun, 2021; Cheng et al., 2022).
3.1 An Illustrative Toy Example
Why does ARMOR have the RPI property, even when the reference policy Tee is not covered by the data D? While we will give a formal analysis soon in Section 3.2, here provide some intuitions as to why this possible. First, notice that ARMOR has access to the reference policy Tiet- Therefore, trivial way achieve RPI with respect to Tref is to just output Teef- However, this naïve algorithm while never degrading Traf cannot learn improve from Tref- ARMOR achieves these two features simultancously by 1) learning MDP Model, and 2) adversarially training this MDP model to minimize the relative performance difference to Tref during policy optimization.
We illustrate this by one-dimensional discrete MDP example with five possible states as shown Figure 2. The dynamic deterministic, and the agent always starts in the center cell. The agent receives lower reward of 0.1 in the left-most state and a high reward of 1.0 upon visiting the right- most state. Say, the agent only has access to a dataset from sub-optimal policy that always takes the left action to receive the 0.1 reward. Further, let's say have access to reference policy that demonstrates optimal behavior on the true MDP by always visiting the right-most state. However, is unknown a priori that the reference policy is optimal. In such case, typical offline RI. methods can only recover the sub-optimal policy from the dataset as it is the best-covered policy in the data.
ARMOR can learn to recover the expert reference policy in this example by performing rollouts with the adversarially trained MDP model. From the realizability assumption (Assumption 1), we know that the version space of models contains the true model (i.e., M MM)). The adversary can then choose a model from this version space where the reference policy Tref maximally outperforms the leamner. In this toy example, the model selected by the adversary would be the one allowing the expert policy to reach the right-most state. Now, optimizing relative performance difference with respect to this model will ensure that the learner can recover the expert behavior, since the only way for the learner to stay competitive with the reference policy to mimic the reference policy in the region outside data support. In other words, the reason why ARMOR has RPI to re is that

---

its adversarial model training procedure can augment the original offline data with new states and actions that would cover those generated by running the reference policy."
3.2 Theoretical Analysis
Now we make the above discussions formal and give theoretical guarantees on ARMOR's absolute performance and RPI property. To this end, we introduce a single-policy concentrability coefficient, which measures the distribution shift between a policy and the data distribution μr.
Definition 1 (Generalized Single-policy Concentrability). We define the generalized single- policy concentrability for policy T, model class M and offline data distribution I as where E*(M) = Drv (PMa [s,0), P.. (0,88,0 a,)) +
Bar (E-))
(M(T) = SuPMEM F [-())) (Rar(s,a)-R" (*.a)) /V
Note that (T) is always upper bounded by the standard single-policy concentrability coefficient /dd/pllse (c.g., Jin et al., 2021: 1: Rashidinejad et al., 2021; Xie et al., 202 but it can be smaller in general with model class M. It can also be viewed model-based analog of the one in Xie et al. (2021a), detailed discussion around (T) can found Uehara and Sun (2021).
First, we present the absolute performance guarantee of ARMOR, which holds for a well-tuned a.
Theorem (Absolute performance). Under Assumption 1, there is an absolute constant such that for any d (0, 1]. we a c (log(IMl//) in Eq. (2), then for any reference policy Teef and comparator policy nt E II. with probability 1 6, the policy ir learned by ARMOR in Eq. (1) satisfies that ( - (f) upper bounded by
(VeM(TT)+ + (EM(Tret)
Vina I-y
log(l MM/4
Roughly speaking, Theorem 2 shows that से learned by ARMOR can compete with any policy at with a large enough dataset, as long as the offline data μ has good coverage on o (good coverage over Tref can be automatically satisfied if we simply choose Tref μ. which yields (TTe)) Compared to the closest model-based offline RL work (Uehara and Sun, 2021), if we set Tref = μ (data collection policy), Theorem 2 leads almost the same guarantee as Uehara and Sun (2021, Theorem 1) up to constant factors.
1).
In addition to absolute performance, below we show that, under Assumptions 1 and 2, ARMOR has the RPI property to Tref: always improves over [Freet for wide range of parameter a. Compared with the model-free ATAC algorithm in Cheng et al. (2022, Proposition 6). the threshold for a in Theorem does not depend on sample size N due to the model- nature of ARMOR.
Theorem (Robust strong policy improvement). Under Assumptions 1 and 2, there exists an abso- lute constant such that for any ô E (0, 11, if: i) a 2 c- (log(IMi/a)) in Eq. (2); ii) Tref E 11, then with probability
the policy it learned by ARMOR in Eq. (1) satisfies J( 2 J J(Tret).
The detailed proof's of Theorems 2 and 3, well the discussion how to relax Assumptions and 2 to the misspecified model and policy classes are deferred to Appendix A.
Practical Implementation
In this section, we present scalable implementation of ARMOR (Algorithm 1) that approximately solves the two-player game in Eq. (1). We first describe the overall design principle and then the algorithmic details.
4.1 A Model-based Actor Critic Approach
For computational efficiency, we take model-based actor critic approach and solve regularized version of Eq. (1). We construct this regularized version by relaxing the constraint M E M,
NNoe that ARMOR does not depend on knowledge of the true reward function and similar arguments hold in the case of learned rewards illustrate in Appendix E.
5

---

Algorithm ARMOR (Adversarial Model for Offline Reinforcement Learning)
Input: Batch data Dreat, policy и, MDP model M, crities f1, f2, horizon H, constants 3, 0, r E [0, 11. w E [0, 1].
1: Initialize target networks fi + f2 t fe and D moddel = 0
2: for k 0. ................................ K do
3: Sample minibatch Dreai from dataset Dreal and minibatch Diini model from dataset D model
4: Construct transition tuples using model predictions
DM == {(,, (ss, : TM Rad (s, a), 5'M ~ Parl 18, a), (8,
Dmini real
U Dimin 1
5: Update the adversary networks; for 1,2,
padversary (S, M) CDss (S, कT, Tire)) E (,, R) (e
(fi, + V M (advveessryy (f2, M)) fi + p Tins V f. (adversary
M + M- - Mass M padversury
M)) and fi + (1- -) T
6: Update actor network with respect to the first critic network and the reference policy
(actor())) = CDss (f1, "I, Tref)
(5)
Projn (TT 7/slow (((ucto(()))
7: 1%% = then reset model state: S + {s E Drinn } and S + {s E D
8: Query the MDP model expand 'mmddel and update model state
Ax = fa: a ~ (()), E 3x} and Anw = faaa: fa:a~ Tref (s), Smmi
D == Dinotel U U(S, Ax U (S Anm S t fs' |s' ~ detach |s a))) 8x, E Ax)
STm t- {s' s' detach |s s,))), Smn, E Ara)
9: end for
in the inner minimization of Eq. (1) to a regularization term and introducing an additional critic function. To clearly elaborate this, we first present the regularized objective its complete form, and subsequently derive it from Eq. (1).
J0, Vinax] | be class of critie functions. The regularized objective is given as Let F: :f : S A
>
f E argmax Lazg (T, (T. (T, +
(6)
TEll
s.t. 1 E argmin MEM.JE
Lagp (),
AED(M))
8, a) + (ri/ is the model-fitting error. Lazge (T, f) == BA [[( [(8, Trer)] is equal to the performance difference (1 (Jv()() Ja (Tref), Epera. (R, J. .)) denotes the squared Bellman error on the distribution PTw.. that de- notes the distribution generated by first running ree and then rolling out ı in M (with switching time sampled from a geometric distribution of 7), and 3, X act as the Lagrange multipliers.
where (D(M) = Ep log Par(s'
This regularized formulation in Eq. (6) can be derived follows. Assuming Qi F, and us- ing the facts that (M(T) = Ta,, 1Q fy (8, T) and the Bellman equation (8, = rMd (8,a) + yi 12 (s', })), we can rewrite Eq. (1) as
min well MEM.JEF
Egzs [f(s, m) f(8, Tree))
s.t.
ED(M) S a
min ED(M')
M'EM
Ys, a E supp(P f(s,a)= (sa
We then convert the constraints in Eq. (7) into regularization terms in the inner minimization by introducing Lagrange multipliers (3, A), following (Xie et al., 2021a; Cheng et al., 2022), and drop the constants not affected by M, f. which results in Eq. (6).

---

4.2 Algorithm Details
Algorithm is an iterative solver for approximating the solution to Eq. (6). Here we further approx- imate d and Pr in Eq. (6) using samples from the state-action buffer moddi- We want ensure that medde has a larger coverage than both d'g and Pa.... We do so heuristically, by construct- ing the model replay buffer Dmotel through repeatedly rolling out æ and ree with the adversarially trained MDP model M, such that Dmodel contains diverse training set of state-action tuples.
Specifically, the algorithm takes as input an offiin dataset reall a policy x, an MDP model M and two critic networks f2. At every iteration, the algorithm proceeds in two stages. First, the adversary is optimized to find data-consistent model that minimizes the performance difference with the reference policy. We sample mini-batches of only states and actions D and mmmdel from datasets respectively (Line 4). The MDP model M is queried on these
the real and mini-batcher to generate next-state and reward predictions. The adversary then updates the model and Q- functions (Line 5) using the gradient of the loss described in Eq. (4), where
Lox (J, T, meet) == ED ((s) [(s, Teee (s)) BBn, (J. == = (1- -1- w) (S M, x) + wEb (f: (f.J, a,
&Duil (M) = LE P's (s' s, s. + (Rxr(s.))))////a_
502
CD is the pessimistic loss term that forces the f to predict a lower value for the learner than the reference on the sampled states. E% is the Bellman surrogate to encourage the Q-functions to be consistent with the model-generated data Das. We use the double Q residual algorithm loss similar to Cheng et al. (2022), which is defined as a convex combination of the temporal difference losses with respect to the critic and the delayed target networks, J M, == ED [(f(s, a) - r - ff TT) (M) is the model- fitting loss that ensures the model is data- consistent. 3 and À control the effect of the pessimistic loss, by constraining Q-functions and models the adversary can choose. Once the adversary is updated, we update the policy (Line 6) to maximize the pessimistic loss as defined in Eq. (5). Similar to Cheng et al. (2022), we choose one Q-function and a slower learning rate for the policy updates (Mtst > 7slow).
We remark that EY, not only affects f2. but also M. i.e., it forces the model to generate transi- tions where the Q-function is Bellman consistent. This allows the pessimistic loss to indirectly affect the model learning, thus making the model adversarial. Consider the special case where À = 0 in the loss of Line 4. The model here no longer forced to be data consistent, and the adversary can now freely update the model via E such that the Q-function is always Bellman consistent. As consequence, the algorithm becomes equivalent to IL on the model-generated states. We empirically study this behavior in our experiments (Section 5).
Lines 7 and 8 describe our model- rollout procedure. We incrementally rollout both and Tree from states in Dmini for a horizon H, and add the generated transitions to 2 meodet- The aim of this strategy to generate a distribution with large coverage for training the adversary and policy, and we discuss this in detail in the next section.
Finally, it is important to note the fact that neither the pessimistic nor the Bellman surrogate losses uses the real transitions; hence our algorithm is completely model-based from a statistical point of view, that the value function is solely an intermediate variable that helps in-model optimization and not directly fit from data.
5 Experiments
We test the efficacy of ARMOR on two major fronts: (1) performance comparison to existing offline RL algorithms, and (2) robust policy improvement over a reference policy that is not covered by the dataset, a novel setting that is not applicable to existing works*. We use the D4RL (Fu et al., 2020) continuous control benchmarks datasets for all our experiments and the code will be made public.
Experimental Setup: We parameterize æ, f1. and M using feedforward neural networks, and set 7ast = 5e- - 4, 7/loo 5e - 7, 0.5 similar to Cheng et al. (2022). In all our experiments, we vary only the B and À parameters which control the amount of pessimism; others are fixed. Importantly, we set the rollout horizon to be the max episode horizon defined in the environment.
In Appendix F we empirically show how imitation learning can be obtained as a special case of ARMOR
7

---

Dataset
ARMOR
MoREL
MOPO
RAMBO
COMBO
ATAC
CQL
IQL
BC
hopper-med
101.4
95.4
28.0
92.8
97.2
85.6
86.6
66.3
29.0
walker2d-med
90.7
77.8
17.8
86.9
81.9
89.6
74.5
78.3
6.6
halfchectah-med
54.2
42.1
42.3
77.6
54.2
53.3
44.4
47.4
36.1
hopper-med-replay
97.1
93.6
67.5
96.6
89.5
102.5
48.6
94.7
11.8
walker2d-med-reee meppllllay
85.6
49.8
39.0
85.0
56.0
92.5
32.6
73.9
11.3
halfcheetah- replry
50.5
40.2
53.1
68.9
55.1
48.0
46.2
44.2
38.4
Supper-med-ext
103.4
108.7
23.7
83.3
111.1
111.9
111.0
91.5
111.9
walker2d-med m-de----
112.2
95.6
44.6
103.3
114.2
98.7
109.6
6.4
halfcheetah-meddent
93.5
53.3
63.3
93.7
90.0
94.8
62.4
86.7
35.8
pen-human
72.8
37.5
71.5
34.4
hammer-humin human
1.5
1.5
door-human
2.5
9.9
0.5
relocate-humant
0,4
0.1
0.2
0.1
0.0
pen-cloned
51.4
43.7
39.2
37.3
56.9
hammer-cloned
0.7
1.1
2.1
2.1
0.8
door-cloned
-0.1
3.7
0.4
1.6
-0.1
relocate-cloned
-0.0
0.2
-0.1
-0.2
-0.1
pen-exp
112.2
136.2
107.0
85.1
hammer-exp 118.8
126.9
86.7
125.6
door-exp
98.7
99.3
101.5
34.9
relocate-exp
96.0
99.4
95.0
101.3
Table 1: Performance comparison of ARMOR against baselines the D4RL datasets. The values for ARMOR denote last iteration performance averaged over random seeds, and baseline values taken from their respective papers. The values denote normalized returns based on random and expert policy returns similar to Fu et al. (2020). Boldface denotes performance within 10% of the best performing algorithm. We report results with standard deviations in Appendix F.
The dynamics model is pre- pre-trained for 100k steps using model-fitting loss on the offline dataset. ARMOR is then trained for IM steps on each dataset. Refer to Appendix F for more details.
5.1 Comparison with Offline RL Baselines
By setting the reference policy to the behavior-cloned policy on the offline dataset, we can use ARMOR as standard offline RL algorithm. Table shows comparison of the performance of ARMOR against SoTA model-free and model-hased offline RI. baselines. In the former cate- gory, consider ATAC (Cheng al., 2022), CQL (Kumar et al., 2020) and IQL (Kostrikov al., 2021), and for the latter we consider MoREL (Kidambi et al., 2020), MOPO (Yu et al., 2020), and RAMBO (Rigter et al., 2022). We also compare against COMBO (Yu al., 2021) which is hybrid model-free and model-based algorithm. In these experiments, we initially warm start the optimiza- tion for 100k steps, by training the policy and Q-function using behavior cloning and temporal dif- ference learning respectively on the offline dataset to ensure the learner policy is initialized to be the same as the reference. Overall, we observe that ARMOR consistently outperforms or competitive with the best baseline algorithm on most datasets. Specifically, compared to other purely model- based baselines (MoREL, MOPO and RAMBO), there is marked increase performance in the walker2d-med, hopper-med-exp and walker2d-med--et datasets. We would like to highlight two crucial elements about ARMOR, in contrast to other model-basee baselines (1) ARMOR achieves SoTA performance using only single neural network to model the MDP, as opposed to complex network ensembles employed in previous model-based offline RL methods (Kidambi et al., 2020; Yu et al., 2021, 2020; Rigter et al., 2022), and (2) to the best of our knowledge, ARMOR is the only purely model-based offline RL algorithm that has shown performance comparable with model- free algorithms the nigh-dimensione.
Adroit environments. The lower performance compared to RAMBO on halfcheetah -med and halfcheetah- med replayy may be attributed to that the much larger computational budget used by RAMBO is required for convergence on these datasets.
5.2 Robust Policy Improvement
Next, we test whether the practical version of ARMOR demonstrates RPI of the theoretical version. We consider set of datasets comprised of the medium and medium-replay versions of D4RI. locomotion tasks, well as the human and cloned versions of the Adroit tasks, with the reference policy set to be the stochastic behavior cloned policy on the expert dataset. We chose these combi-
'The variation in performance of the reference for different dataset qualities in the same environment is owing different random seeds.

---

ORL
RBF
0.81
0.85
0.1
0.5
2.00
50
medreephy
ma melimppyy
hepper
halfcheotah
Narmal 200
human
conedd
eloned
pen
hammer
doo
relocate
Figure 3: Verification of RPI the reference policy for different 3 (purple). ORL denotes the performance of offline RL with ARMOR Tabble 1), and REF is the performance of reference policy."
nations of dataset quality and reference, to ensure that the reference policy takes out-of cnt-of-diitributtoon actions with respect to the data. Unlike Sec. 5.1 here the reference policy is a black-box given as a part of the problem definition. This opens the question since we can not trivially initialize the learner to be the reference as in the previous experiments. In a similar spirit to Sec. 5.1, one might consider initializing the learner close to the reference by behavior cloning the reference policy the provided dataset during warmstart, i.e. by replacing dataset actions with reference actions. However, when the reference chooses out of support ac- tions, this procedure will not provide a good global approximation of the reference policy, which can make the optimization problem harder. Instead, we propose to learn a residual policy where the learned policy outputs additive correction the reference (Silver et al., 2018). This is an appropriate choice since ARMOR does not make any restrictive assumptions about the structure of the policy class. Figure 3 shows the normalized return achieved by ARMOR for different 3, with fixed values for remaining hyperparameters. We observe that ARMOR able to achieve perfor- mance comparable datasets, thus verifying the RPI property in practice. Specifically, there is significant improvement via RPI in the hammer, door and relocate domains, where running ARMOR pure offline RL algorithm(Section 5.1) does not show any progress Overall, we note the following metrics:
how the learner should be initialized,
better than the reference policy for range of 3 values uniformly across all
In 14/14 datasets, ARMOR shows RPI (i.e., ARMOR policy is worse than the reference when measured by overlap of confidence intervals). Further, considering the difference be- tween ORL and REF rough indication of whether the reference is within data support, we note that in 12/14 4 cases REF is strictly better than ORL, and all those cases ARMOR demonstrates RPI.
In 5/14 datasets, the ARMOR policy is strictly better than the reference. (Criterion: the lower confidence of ARMOR performance is better than upper confidence of REF). It is important to note that this metric is highly dependent the quality of the reference pol- icy. Since the reference is near-expert, it can be hard for some environments to improve significantly over it.
Discussion
The RPI of ARMOR is highly valuable it allows easy tuning of the pessimism hyperparameter without performance degradation. We believe that leveraging this property can pave the way for real-world deployment of offline RL. Thus, next present discussion of RPI
When does RPI actually improve over the reference policy?
Given ARMOR's ability to improve over an arbitrary policy, the following question naturally arises: Can ARMOR nontrivially improve the output policy of other offline algorithms, including itself? this were true, can we repeatedly run ARMOR to improve over itself and obtain the best policy any algorithm can learn offline? Unfortunately, the answer is negative. Not only can ARMOR not
II Appendix F.5 we provide further experiments for different choices of reference policies.
7 We provide comparisons when using behavior cloning initialization learner in Appendix F. Due space limit, we defer the complete discussion Appendix and only provide salient points here.

---

improve over itself, but it also cannot improve over a variety of algorithms (e.g., absolute pessimism or minimax regret). In fact, the optimal policy of an arbitrary model in the version space Ma is provably unimprovable ( Corollary 10; Appendix D). With a deep dive into when RPI gives nontrivial improvement (Appendix D), we found some interesting observations, which we highlight here.
Return maximization and regret minimization are different in offline RL These objectives generally produce different policies, even though they are equivalent in online RL. Their equíva- lence in online RL relies on the fact that online exploration can eventually resolve any uncertainty. In offline RL with an arbitrary data distribution, there will generally be model uncertainty that cannot be resolved, and the worst-case reasoning over such model uncertainty (i.c.. Ma) leads to defini- tions that are longer equivalent. Moreover, it is impossible to compare return maximization and regret minimization and make a claim about which is better. They are not simply an algorithm design choice, but definitions of the learning goals and guarantees themselves- and are thus incomparable: we care about obtaining a guarantee for the worst-case return, the return maxi- mization is optimal by definition; if we more interested in guarantee for the worst-case regret, then regret minimization is optimal. We also note that analyzing algorithms under a metric that is different from the one they are designed for can lead to unusual conclusions, e.g., Xiao et al. (2021) algorithms are equally minimax-optimal in terms of their
show that patnmiiiiiieuuurpppssssssiees regret guarantees in offline multi-armed bandits. However, the algorithms they consider are opti- mistic/pessimistic with respect to the return (as commonly considered in the offline RL literature) not the regret which is the performance metric they are interested in analyzing.
Tref is more than a hyperparameter-itt defines the performance metric and learning goal Corollary 10 in Appendix D shows that ARMOR has many different fixed points: when ref is chosen from these fixed points, the solution to Eq. (1) is also Tref. Furthermore, some of them may seem quite unreasonable for offline learning (e.g.. the greedy policy to an arbitrary model in Ma or even the optimistic policy). This is not a defect of the algorithm. Rather, because of the unresolv- able uncertainty in the offline setting, there are many different performance metrics/learning goals that are generally incompatibleincomprabbe and the agent designer must make a conscious choice among them and convey the intention to the algorithm. In ARMOR, such choice is explicitly conveyed by reef which makes ARMOR subsume return maximization and regret minimization as special cases.
Conclusion
We have presented model-based offline RL framework, ARMOR, that can improve over arbi- trary reference policies regardless of data coverage, by using the concept of relative pessimism. ARMOR provides strong theoretical guarantees with general function approximators, and exhibits robust policy improvement over the reference policy for wide range of hyper-parameters. We have also presented a scalable deep learning instantiation of the theoretical algorithm. Empirically, we demonstrate that ARMOR indeed enjoys the RPI property, and has competitive performance with several SoTA model-free and model-basee offline RL algorithms, while employing simpler model architecture (a single MDP model) than other model- based baselines that rely on ensembles. This also opens the opportunity to leverage high-capacity world models (Hafner et al., 2023) with offline RL in the future. However, there are also some limitations. While RPI holds for the pessimism pa- rameter, the others still need to be tuned. In practice, the non-convexity of the optimization can also make solving the two-player game challenging. For instance, if the adversary is not strong enough (i.e., far from solving the inner minimization), RPI would break. Further, runtime of ARMOR is slightly slower than model-free algorithms owing to extra computations for model rollouts.
Acknowledgments and Disclosure of Funding
Nan Jiang acknowledges funding support from NSF IIS-2112477 and NSF CAREER IIS-214178,
References
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complex- ity and representation learning of low rank mdps. Advances Neural Information Processing Systems, 33:20095-20107, 2020.
10

---

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman- residual minimization based fitted policy iteration and single sample path. Machine Learning, 71(1):89-129 2008.
Jinglin Chen and Nan Jiang. Information- considerations in batch reinforcement learning. In International Conference on Machine Learning pages 1042-1051, 2019.
Xiong- Hui Chen, Yang Yu, Zheng-Mao Zhu, Zhihua Yu, Zhenjun Chen, Chenghe Wang, Yinan Wu, Hongqiu Wu, Rong-Jun Qin, Ruijin Ding, et al. Adversarial counterfactual environment model learning. arXiv preprint 22t:::2000048900 2022.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning, International Conference Machine Learning, 2022.
Amir Massoud Farahmand, Rémi Munos, and Csaba Szepesvári. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems, 2010.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pages 2052-2062, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-eritic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pages 1861-1870. PMLR, 2018.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 2tt1::3300.001444, 2023.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
Sham M Kakade. A natural policy gradient. Advances in Neural Information Processing Systems, 14, 2001.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, 2015.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning, arXiv preprint 16X.v:2110.06169, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy 4- learning via bootstrapping 32:1 12.11784-11944 2019.
reduction. Advances in Neural Information Processing Systems,
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforcement learning: State-of-the-ate pages 45-73, 2012.

---

Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. International Conference on Machine Learning, pages 3652-3661. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforce- ment learning not scary? In Conference on Learning Theory, volume 178, pages 5175-5220. PMLR, 2022.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with stationary distribution correction. In Uncertainty in Artificial Intelligence, pages 180--190. PMLR, 2020a.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off off-policy reinforcement learning without great exploration. Advances in Neural Information Processing Systems, 33:1264-1277 2020b.
Rémi Munos. Error bounds for approximate policy iteration. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, pages 560-567, 2003.
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 14:11702-11716 2021.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 35:16082-16097, 2022.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning, pages 19967-20025. PMLR, 2022.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint wrXiv:2002.08336, 2020.
Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning. arXiv preprint arXiv:1812.062000 2018
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations, 2021.
Sara A van de Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 191 1361, 2019.
Chenjun Xiao, Yifan Wu, Jincheng Mei, Bo Dai, Tor Lattimore, Lihong Li, Csaba Szepesvari, and Dale Schuurmans. On the optimality of batch policy optimization algorithms. In International Conference on Machine Learning, pages 1 11322-11311. PMLR, 2021.
Tengyang Xie and Nan Jiang. Qa approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference Uncertainty in Artificial Intelligence, pages 550-559. PMLR, 2020.
Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In Inter- national Conference on Machine Learning, pages 11404-11413, PMLR, 2021.
12

---

Tengyang Xie, Ching- An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Sys- lems, 34:6683-6694, 2021a.
Tengyang Xie, Nan Jiang, Huan Wang. Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforeement learning. Advances in Neural Information Processing Systems, 14:27395-274077 2021b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129--14144 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34:28954 28967, 2021.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic meth- ods for offline reinforcement learning, Advances in Neural Information Processing Systems, 34, 2021.
Tong Zhang. From E-entropy to kl-entropy: Analysis of minimum information complexity density estimation. The Annals of Statistics, 34(5):2180-2211 2006.
13

---

A Proofs for Section
A.1 Technical Tools
Lemma (Simulation lemma), Consider any two MDP model M and M', and any π S - A(A).
we have
(x) JAr(T)) Vmax Ed (Drv (PM 8, a), Pa (ss Ex [R Rsr a)]]
Lemma 4 the standard simulation lemma model-based reinforcement learning literature, and its proof can be found in, e.g., Uehara and Sun (2021, Lemma 7).
A.2 MLE Guarantees
We LD (M) to denote the likelihood of model M (P, R) with offline data D, where
(D(M)
II
Pas(s' s,a).
(8)
(s.0,
For the analysis around maximum likelihood estimation, we largely follow the proving idea of Agar- wal et al. (2020); Liu et al. (2022), which inspired by Zhang (2006).
The next lemma shows that the ground truth model M* has a comparable log legllikelihood compared with MLE solution.
Lemma 5. Let M* be the ground truth model. Then, with probability least 1 8, we have
max log (D(M) - log (D()) I log(lMl/s).
MEM
Proof of Lemma 5. The proof of this lemma is obtained by standard argument of MLE (see, e.g., van de Geer, 2000). For any M E M,
(M) (D(M*)
lexp (log (M) log lo (M*))) =
I(aan)) Pas(s' (s,a)
E
[lsaana PM- (s' ss,a)
PMrl' 18,8)
E
PM. (s' (s,a)
Par (s' ssa)
E
E
8,0
Par (8'
(8,a)
(e.a)ED
E
II
Σ
Pas s,a)
(s.a)ED
8'
1.
Then by Markov's incquality, we obtain
IP [log (MM) - log (M*)) > log(1/)
s E (exp (log (D(M) log ( (M*)) exp |- log(1/8) 5.
=1 by Eq. (9)
Therefore, taking a union bound over M, we obtain
P [[log (D(M) log (M()) log(lM1/6) $ 8.
This completes the proof.
14

---

The following lemma shows that, the on-supp error of any model M E M can be captured via its log- likelihood (by comparing with the MLE solution). Lemma 6. For any model M, we have with probability at least 1 8,
By Drv (Par 8, Par- (.18,a)) So
11 (MM) - log (D(M) + log((MM/6))
log (log
where (((.)) is defined in Eq. (8).
Proof of Lemma 6. By Agarwal et al. (2020, Lemma 25), we have
Ey Drv (Parl: 8, 8, PM (.18,)))2 S 2 log Bux Pas-
exp (-3
1
log
(as- (s,a) PM 8,
(10)
where μ x Pas- denote the ground truth offline joint distribution of (s, s').
Let D {(5;, di Ti, 30) l be another offline dataset that is independent to D. Then,
PM (8' 8,a) PM
log E Px PMr
exp
log
2
s,
PMs- (5)
Sg, a,)
log B(
exp
-
log
PM (3% 8 (4)
Par- (8)
Bi, (.)
)
D |)
log E
Dayj
exp
2
log
PM (3,
i=1
PMM (s') (s,a)
Σ
= log Day
exp
log
(11)
2
Prs(s' |s,a)
(e,0,
We use (M(s,,, s') as the shorthand of - log PPr- (d for any (s,a,s') E S x A x S.
Phe(s')e.a)
By Agarwal et al. (2020, Lemma 24) (see also Liu et al., 2022, Lemma 15), we know
EDaH exp Σ (arss,,ss)) - log log Eng
exp
Σ
(x)(),a,)))
D
log LM)
$1.
(e,0,,')E
Thus, we can use Chernoff method as well as union bound on the equation above to obtain the following exponential tail bound: with probability at least 1 5, have for all (P, R) M M,
- log E
D
exp
Σ (a((s,a,5))
D -
Σ (s.a,e")ED
(m(s,a,s') + MM)
(12)
Plugging back the definition of f and combining Eqs. (10) to (12), we obtain
Ey DDv (P(: a) Par (1 s,a)))]] S
Σ
log
(Par (s' s,
P(s' a) ) 2lo
(ogg(M/b)) /8).
(s,0,)))ED
Therefore, we obtain
n En Drv (P(: (().)sss)..... a. s,a))?]
(s s,a) + log(lM//))
log
(S.. P(s' s,
log (MM*) - log (MM + lMM()))/8).
(Lx(.) is defined in Eq. (8))
This completes the proof.
15

---

A.3 Guarantees about Model Fitting Loss
Lemma 7. Let M be the ground truth model. Then, with probability least 1 6, we have
Ep(M*) min MEM
(D(M - O (log(lM)/6)),
where &p is defined in Eq. (3).
Proof of Lemma 7. By defition, we know
(D log (M() + (Rar(s,a)-)) -r) /r//aa
By Lemma 5, we know
max log (D(M) - log (D(M)) - log(lM1/8). MEM
(13)
In addition, by Xie et al. (2021a, Theorem A.1) (with setting 7 0), we know w.p. 1 8,
Σ (s a) (.l0)rr) r)²
min MEM
Σ
(RM r)² z log(LM//6).
(14)
(S.0,r,)")E
(E,D,T,0")E
Combining Eqs. (13) and (14) and using the fact of Vmax 2 L, we have w.p. 6,
ED(M*) min (M) MEM
< max log log( - min MEM
Σ
(Rx((t.)-1)//12 (D(M))
MEM (s.a,,,")ED
$ log((MM/8).
This completes the proof. Lemma 8. For any M E M, we have with probability at least 1 5,
Ey Drv (Par(: s,a), Par- (- 8, u))² + (Rar(s..)) ne (x.a)) (aaa))//va /va maax (D(M) ED(M*) + log(lM1/8)
n
where Ed is defined in Eq. (3).
Proof of Lemma 8. By Lemma 6, have w.p. 1 1-6. 5,
n Ep Drv (PM(- 8, a, PM-( s, a))) z log (M(') - logep(M) + log(M))))). (15)
Also, we have
n Ep (RM(s,a) - R²' (s,
(16)
n Ep (RM((s - (
n Ep (s, - -r)² (see, e.g. Xie et al., 2021a, Eq. (A.10) with 7 0)
24
Σ (RM(s, a) - -)
Σ (R* r)² + +og(( Ml/s).
(s.0.r,"")ED
(s,a.r,e))E
where the last inequality is direct implication of Xie et al. (2021a, Lemma A.4). Combining Eqs. (15) and (16) and using the fact of V Vax 2 1, we obtain
Ep [Div (PM(-) s,a), PM (: (s, +
(s.a)))) /va
$ lM
Σ
(s(a))))))/// v log (M) +
Σ
(Rar(s,0) + +log(IM//))
(DDD
(s,D,T,")ED
E (D(')) + log(lM//) /8).
This completes the proof.
16

---

A.4 Proof of Main Theorems
Proof of Theorem 2. By the optimality of के (from Eq. (1)), we have
) - J(F) = J(T') - ((Tret) [J - (
- J(T') - [
min MEM.
[J - JN (Trer))
(+)
S ((T)) - [(Trett min
[JAs(TT] (Tre)t
(17)
MEMMa
where step (x) follows from Lemma so that have M Ma, and the last step is because of rt IT. By the simulation lemma (Lemma 4), we know for any policy a and any M E Ma.
V Vmaa
[J(T) - Jar(T))
Ed
[Drv (PM (,.,))))))........).). (-s ss
1-7 Edd [| a) (s)
V max
Vinax
<
Edz= Drv (Par s, Pa- (: s, s,)))
+ I- V Ed (RRar(s))-R (s.a))) max
Viaa
Ext= Drv (Par PM (- +
(a S means S (b()
Vima v (M(TT)
VE, By Drv (PM( (,,)) Pasa a,)))² (Rar(s,a) -t (v.a))
VINA V (M))))
(DDMM) - (D(M)) + log(lMI/A)
(by Lemma 8)
1-7 V v (M((T)
ED(M) minw ED(')) log( (1l55
24
(1)
n
Vmax (MM(T))
log (M/6)
(18)
1-7
where the step ($) follows from the assumption of M E M, and last step is because ED(M)
minM'eM (M') & a Ollog( Mi/s) by Eq. (2).
Combining Eqs. (17) and (18), we obtain
( [Vemlrt)+ + (
Vmax
log(l
7
This completes the proof.
Note that, over the proof above, only steps (+) and (9) have used the realizability assumption of M E M. To extend that to the misspecification case, where there only exists an M* . such that M is close to M up some misspecification error, we just need the following straightforward accommodations:
A variant of Lemma 5-to ensure that M* is included in the version space Ma. By doing so, the misspecification error should also be included in the radius of the version space.
(2) Upper bound [J(TT) Jiz- (T] for any т using misspecification error. This is a stan- dard argument, and by combining with the item above, (+) becomes J( - J(Tret) 2 minMeM, [J (f) - JMM (Trer) misspecification error.
(3) Upper bound difference in model-fitting error (D(') - ED(M)) using misspecification error. Then, step (t) becomes, for M Ma.
((M))&p(M'))
-
= ED(M) ED(M' + (D(M - ED(M)
s ED((M) min (M' + misspecification error M'EM
< log (LM/6) + misspecification error.
17

---

Due to the unboundedness of the likelihood, we conjecture that naively defining misspecifica- tion error using total variation without any accommodation on the MLE loss may be insufficient for the steps above. To resolve that, we may adopt altemnaee misspecification definition, e.g., log Par (s') 8, a) - log 2.. (s' 8, S E, V(s a()) S A x S, or add extra smoothing to the MLE loss with regularization.
Proof of Theorem 3.
J (TTee) -(a) = [ - (Trett - [J( - J [TTer)]
S min [JM - JM (Tret)
(by Lemma 5, we have M E Maa
MEM.
= max min [JM(T) (Tror)]
(by the optimality of म from Eq. (1))
Fell MEM.
min [Ja((Tret) JM (Trrrt)
(Tree E II)
MEM.
0.
A misspecified version of Theorem can be derived similarly to what we discussed about that of Theorem 2. If the policy class is also misspecified, where there exists only मee E II that is close to Tref up to some misspecification error, the second last step of the proof of Theorem 3 becomes -minMe... [Jas (Tf JM (Fret) misspecification error by simply applying the performance difference lemma on the difference between Fref and Tref-
B Proofs for Section
Proof of Lemma 9. We prove the result by contradiction. First notice minMeM Jar('')
Ja (xr) = 0. Suppose there is ग E II such that minMeM. (x) JM((T') > O, which im- plies that ( > JM(TT) VM M, Since M Ma. we have
min ( + (M) > min JM + +MM) = max min Jas (T) + x(M) MEM
MEM
aElf MEM
which is a contradiction of the maximin optimality. Thus maxæen minMeM. JM( - JM O. which means ní is solution.
For the converse statement, suppose и is a fixed point. We can just let (MM) = (T) Then this pair of and v by definition of the fixed point satisfies Eq. (19).
C Related Work
There has been an extensive line works on reinforcement with offline/batch data, especially for the case with the data distribution is rich enough capture the state-action distribution for any given policy (Munos, 2003; Antos et al., 2008; Munos and Szepesvári, 2008; Farahmand et al., 2010: Lange al., 2012; Chen and Jiang, 2019; Liu al., 2020a; Xie and Jiang, 2020, 2021). However, this assumption not practical since the data distribution is typically restricted by factors such the quality of available policies, safety concerns, and existing system constraints, leading to narrower coverage. As result, recent offline RL works in both theoretical and empirical literature have focused systematically addressing datasets with inadequate coverage.
Modern offline reinforcement learning approaches can be broadly categorized into two groups for the purpose of learning with partial coverage. The first type of approaches rely on behavior regu- larization, where the learned policy is encouraged to be close to the behavior policy in states where there is insufficient data (e.g., Fujimoto et al., 2018; Laroche et al., 2019; Kumar et al., 2019; Siegel et al., 2020). These algorithms ensure that the learned policy performs at least as well as the behav- ior policy while striving to improve it when possible, providing form of safe policy improvement guarantees. These and other studies (Wu et al., 2019; Fujimoto and Gu, 2021; Kostrikov et al., 2021) have provided compelling empirical evidence for the benefits of these approaches.
The second category of approaches that has gained prevalence relies on the concept of pessimism under uncertainty to construct lower-bounds on policy performance without explicitly constraining
18

---

the policy. Recently, there have been several model-free and model based algorithms based on this concept that have shown great empirical performance on high dimensional continuous control tasks. Model-free approaches operate by constructing lower bounds on policy performance and then optimizing the policy with respect to this lower bound (Kumar et al., 2020; Kostrikov et al., 2021). The model-based counterparts first learn world model and the optimize policy using model-based rollouts via off-the-shelf algorithms such as Natural Policy Gradient (Kakade, 2001) or Soft-Actor Critic (Haarnoja et al., 2018). Pessimism is introduced by either terminating model rollouts using uncertainty estimation from an ensemble of neural network models (Kidambi et al., 2020) or modifying the reward function to penalize visiting uncertain regions (Yu et al., 2020). Yu et al. (2021) propose a hybrid model-based and model-free approach that integrates model-based rollouts into a model-free algorithm to construct tighter lower bounds on policy performance. On the more theoretical side, the offline RI. approaches built upon the pessimistic concept (e.g., Liu et al., 2020b; Jín et al., 2021; Rashidinejad et al., 2021; Xie al., 2021a; Zanette et al., 2021; Uehara and Sun, 2021; Shi et al., 2022) also illustrate desired theoretical efficacy under various of setups.
Another class of approaches employs an adversarial training framework, where offline RL is posed a two player game between an adversary that chooses the worst-case hypothesis (e.g. a value func- tion or an MDP model) from hypothesis class, and policy player that tried to maximize the adversarially chosen hypothesis. Xie et al. (2021a) propose the concept of Bellman-consistent pes- simism to constrain the class of value functions to be Bellman consistent on the data. Cheng et al. (2022) extend this framework by introducing a relative pessimism objective which allows for robust policy improvement over the data collection policy μ for wide range of hyper-parameee approach can be interpreted as a model-based extension of Cheng et al. (2022). These approaches provide strong theoretical guarantees even with general function approximators while making min- imal assumptions about the function class (realizability and Bellman completeness). Chen et al. (2022) provide an adversarial model learning method that uses an adversarial policy to generate data-distribution where the model performs poorly and iteratively updating the model on the gener- ated distribution. There also exist model-based approaches based on the same principle (Uehara and Sun, 2021; Rigter et al., 2022) for optimizing the absolute performance. Of these, Rigter et al. (2022) is the closest to our approach, as they also aim to find an adversarial MDP model that mini- mizes policy performance. They use a policy gradient approach to train the model, and demonstrate great empirical performance. However, their approach is based on absolute pessimism and does not enjoy the same RPI property as ARMOR.
Our
D
A Deeper Discussion of Robust Policy Improvement
D.1 How to formally define RPI?
Improving over some reference policy has been long studied in the literature. To highlight the advantage of ARMOR, we formally give the definition of different policy improvement properties.
Definition 2 (Robust policy improvement). Suppose के is the learned policy from an algoritium.
We say the algorithm has the policy improvement (PI) guarantee if ((ret) - J(ā) < o/NY/N is guaranteed for some reference policy kief with offline data D ~ μ,, where N = D. We use the following two criteria w.r.t. Teet and j to define different kinds Pl:
(i) The PI is strong if Teef can be selected arbitrarily from policy class 11 regardless of the choice data-collection policy μú otherwise, PI is weak (i.e., Tref = p is required).
(ii) The PI is robust if it can be achieved by range of hyperparameters with known subset.
Weak policy improvement is also known as safe policy improvement in the literature (Fujimoto al., 2019; Laroche et al., 2019). It requires the reference policy to be also the behavior policy that collects the offline data. In comparison, strong policy improvement imposes a stricter requirement, which requires policy improvement regardless of how the data were collected. This condition is motivated by the common situation where the reference policy is not the data collection policy. Finally, since we are learning policies offline, without online interactions, it is not straightforward to tune the hyperparameter directly. Therefore, it is desirable that we can design algorithms with these properties a robust manner in terms of hyperparameter selection. Formally, Definition requires the policy improvement to be achievable by set of hyperparameters that is known before learning.
19

---

Theorem 3 indicates the robust strong policy improvement of ARMOR. On the other hand, algo- rithms with robust weak policy improvement are available in the literature (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019; Laroche et al., 2019; Fujimoto and Gu, 2021; Cheng et al., 2022); this is usually achieved by designing the algorithm to behave like II. for a known set of hyperpa- rameter (e.g., behavior regularization algorithms have a weight that can turn off the RL. behavior and regress to IL). However, deriving guarantees of achieving the best data-covered policy of the IL-like algorithm is challenging due to its imitating nature. To our best knowledge, ATAC (Cheng et al., 2022) is the only algorithm that achieves both robust (weak) policy improvement as well as guarantees absolute performance.
D.2
When RPI actually improves?
Given ARMOR's ability to improve over an arbitrary policy, the following questions naturally arise: Can ARMOR nontrivially improve the output policy of other algorithms (e.g., such as those based on absolute pessimism (Xie et al., 202Ia)), including itself? Note that outputting ref itself always satisfies RPI, but such result is trivial. By "nontrivially" we mean a non-zero worst-case improve- ment. If the statement were true, we would be able to repeatedly run ARMOR to improve over itself and then obtain the bes/ policy any algorithm can learn offline.
Unfortunately, the answer is negative. Not only ARMOR cannot improve over itself, but it also cannot improve over a variety of algorithms. In fact, the optimal policy of an arbitrary model in the version space is unimprovable (see Corollary 10Y! Our discussion reveals some interesting observations (e.g., how equivalent performance metrics for online RL can behave very differently in the offline setting) and their implications (e.g.. how we should choose Tee for ARMOR). Despite their simplicity, we feel that many in the offline RL community are not actively aware of these facts (and the unawareness has led to some confusion), which we hope to clarify below.
Setup We consider an abstract setup where the learner is given version space Ma that contains the true model and needs to choose a policy E II based on Ma. We use the same notation M, as before, but emphasize that it does not have to be constructed as in Eqs. (2) and (3). In fact, for the purpose of this discussion, the data distribution, sample size, data randomness, and estimation procedure for constructing Ma are all irrelevant, as our focus here is how decisions should be made with given Ma. This makes our setup very generic and the conclusions widely applicable.
To facilitate discussion, we define the fixed point of ARMOR's s relative pessimism step:
Definition 3. Consider Eq. (I) as an operator that maps an arbitrary policy Tref to म. A fixed point of this relative pessimism operator is, therefore, any policy æ E II such that π E argmax,'en minMeM, JMr (5²) Jas(T)
Given the definition, relative pessimism cannot improve over policy if it is already a fixed point. Below we show sufficient and necessary condition for being fixed point, and show number of concrete examples (some of which may be surprising) that are fixed points and thus unimprovable.
Lemma (Fixed-point Lemma). For any M C Ma and any vb M + R. consider the policy
π E argmax min - *'Ell MEM
(19)
Then x is a fixed point in Definition 3. Conversely, for any fixed point π in Definition 3, there is ui: M + RR such that x is solution to Eq. (19).
Corollary 10. The following are fixed points of relative pessimism (Definition 3):
1. absolute-pessimisn policy, i.e., (MM) O.
2. Relative-pessimism policy for any reference policy, i.e. +(M) = -J (Tret)
3. Regret-minimization policy, ie., (b(M) = - (Ji() where xMM E argmaxxen JM
4. Optimal policy of an arbitrary model M E MMa The ie., M = {M This would include the optimistic policy, that is, ar.Max,E (M)
Return maximization and regret minimization are different in offline RL We first note that these four examples generally produce different policies, even though some of them optimize for
20

---

objectives that are traditionally viewed as equivalent in online RL (the "worst-case over Ma part of the definition does not matter in online RL), e.g. absolute pessimism optimizes for JM (T)) which is the same as minimizing the regret Jar (Ti)) JAr(n (TT) for a fixed M. However, their equivalence in online RI. relies on the fact that online exploration can eventually resolve any model uncertainty when needed, so we only need to consider the performance metrics w.r.t the true model M = M In offline RL with an arbitrary data distribution (since we do not make any coverage assumptions), there will generally be model uncertainty that cannot be resolved, and worst-case reasoning over such model uncertainty (i.e., Ma) separates apart the definitions that are once equivalent.
Moreover, it is impossible to compare return maximization and regret minimization and make a claim about which one better. They are not simply an algorithm design choice, but are definitions of the learning goals and the guarantees themselves- thus incomparable: if we care about obtaining a guarantee for the worst-case return, the return maximization is optimal by definition; if we are more interested in ohtaining a guarantee for the worst-case regret, then again, regret minimization is trivially optimal. We also note that analyzing algorithms under a metric that is different from the one they are designed for can lead to unusual conclusions. For example, Xiao et al. (2021) show that op- paaasticnnuutralpsssmsstt algorithms" are equally minimax-optimal in terms of their regret guaran- tees in offline multi-armed bandits. However, the algorithms they consider are optimistic/pessimistie w.r.t. the return- as commonly considered in the offline RL literature- not w.r.t. the regret which is the performance metric they interested in analyzing.
Tee more than a hyperparameter-itt defines the performance metric and learning goal Corollary 10 shows that ARMOR (with relative pessimism) has many different fixed points, some of which may seem quite unreasonable for offline learning, such as greedy w.r.t. an arbitrary model or even optimism (#4). From the above discussion, we can see that this is not a defect of the algorithm. Rather, in the offline setting with unresolvable model uncertainty, there are many different perfor- mance metrics/learning goals that are generally ircompatibe//ncooparabbbl with each other. and the agent designer must make a choice among them and convey the choice to the algorithm. In ARMOR, such a choice is explicitly conveyed by the choice of Tef which subsumes return maximization and regret minimization as special cases (#2 and #3 in Corollary 10)
E A More Comprehensive Toy Example for RPI
We illustrate with simple toy example why ARMOR intuitively demonstrates the RPI property even when Fref is not covered by the data D. ARMOR achieves this by /) learning an MDP Model, and 2) adversarially training this MDP model to minimize the relative performance difference rhe during policy optimization. Consider a one-dimensional discrete MDP with five possible states as shown in Figure 4. The dynamics is deterministic, and the agent always starts in the center cell. The agent receives a lower reward of 0. in the left-most state and a high reward of 1.0 upon visiting the right-mor most state. Say, the agent only has access takes the left action to receive the 0.1 reward. Further, let's say we have access to reference policy that demonstrates optimal behavior on the true MDP by always choosing the right action visit the right-most state. However, it is unknown a priori that the reference policy is optimal. In such case, typical offline RI. methods can only recover the sub-optimal policy from the dataset as it the best-covered policy in the data. Now, for the sake of clarity, consider the current learner policy is same as the behavior policy, i.e always takes the left action.
dataset from a sub-optimal policy that always
ARMOR can learn to recover the expert reference policy in this example by performing rollouts with the adversarially trained MDP model. From the realizability assumption we know that the version space of models contains the true model (i.e., M E MM). The adversary can then choose a model from this version space where the reference policy Reef maximally outperforms the learner. Note, that ARMOR does not require the true reward function to be known. In this toy example, the model selected by the adversary would be the one that not only allows the expert policy to reach the right- most state, but also predicts the highest reward for doing so. Now, optimizing to maximize relative performance difference with respect to this model will ensure that the learner can recover the expert behavior, since the only way for the learner to stay competitive with the reference policy is to mimic the reference policy in the region outside data support. In other words, the reason why ARMOR has RPI to Tee is that its adversarial model training procedure can augment the original offline data with
Inncidentally, optimistic/neutral policies correspond to #4 in Corollary 10.
21

---

True MDP
-11
+10
Dynamies
Region
True MDP
10.1
+83
+10
10.1
+1.0
31
+0.
-i11
Inn
Reference Policy
+10
ARMOR chooses word--cae models
Figure 4: A toy MDP illustrating the RPI property of ARMOR. (Top) The true MDP has determin- istic dynamics where taking the left (aj) or right (a,) actions takes the agent to corresponding states; start state is in yellow. The suboptimal behavior policy only visits only the left part of the state space, and the reference policy demonstrates optimal behavior by always choosing ar- (Bottom) A subset of possible data-consistent MDP models (dynamics rewards) in the version space. The adversary always chooses the MDP that makes the reference maximally outperform the learner. In response, the learner will learn to mimic the reference outside data support to be competitive.
new states and actions that would cover those generated by running the reference policy in the true environment, even though ARMOR does not have knowledge of M*.
F
Further Experimental Details
F1 Experimental Setup and Hyper-parameters
We represent our policy ग, Q-functions fi, and MDP model M as standard fully connected neural networks. The policy parameterized as a Gaussian with state-dependent covariance, and we use tanh transform limit the actions to the action space bound similar to Haarnoja al. (2018). The MDP model learns to predict the next state distribution, rewards and terminal states, where the reward and next-state distributions part are parameterized Gaussians with state-dependent covariances. The model fitting loss consists of negative log-likelihood for the next-state and reward and binary cross entropy for the terminal flags. In all experiments use the same model architecture and fixed value of X. We use Adam optimizer (Kingma and Ba, 2015) with fixed learning rates fast and Tslow similar Cheng et al. (2022). Also similar to prior work (Kidambi et al., 2020), we let the MDP model network predict delta differences to the current state. The rollout horizon is always set to the maximum episode steps per environment. complete list of hyper-paramee parametrsss can found Table
Compute: Each run of ARMOR has access to 4CPUs with 28GB RAM and single Nvidia T4 GPU with 16GB memory. With these resources each run tasks around 6-7 hours to complete. Including all runs for 4 seeds, and ablations this amounts to approximately 2500 hours of GPU compute.
22

---

Hyperparameter
Value
Hyperparameter
Value
model num layers
critic learning rate fasst
5e-4
model _iiddehidden__ size
512
policy learning rate 7/stow
5e-7
model nonlinearity
swish
discount factor
0.99
policy __y___nm_____ layers
rollout horizon
max episode steps
silicy_hidden__ee size
256
model buffer size batch size model batch size num warmstart steps
106
relu
125
f @aum_layery
125
f [hhideensize
256
109
f [_noninearity
relu
5e 3
Table 2: Model Architecture Details
Table 3: List of Hyperparameters.
F.2 Detailed Performance Comparison and RPI Ablations
In Table 4 we show the performance of ARMOR compared to model-free and model-based offline RL baselines with associate standard deviations over 8 seeds. For ablation, here we also include ARMOR which is running ARMOR in Algorithm but without the model optimizing for the Bellman error (that is, the model not adversarial). Although ARMOR does not have any theo- retical guarantees (and indeed in the worst case its performance can be arbitrarily bad), we found that ARMORT in these experiments performing surprisingly well. Compared with ARMOR, ARMORT has less stable performance when the dataset is diverse (e.g. -med-replay datasets) and larger learning variance. Nonetheless, ARMOR using a single model already pretty competi- tive with other algorithms. We conjecture that this is due to that Algorithm 1 also benefits from pessimism due to adversarially trained critics. Since the model buffer would not cover all states and actions (they are continuous in these problems), the adversarially trained critic still controls the pessimism for actions not in the model buffer, as a safe guard. As a result, the algorithm can tolerate the model quality more.
Dumnset
ARMUR
ARMOR
ARMOK
MoKEL
MOPU
RAMBO
COMBO
AIAL
CQL
IQL
BC
hupper med
101.4
1 0.3
100.4 1 1.7
65.3 1 1.8
95.4
28.0 - 12.4
92.8 1 6.0
91.3 1 2.2
K5.6
36.5
66.3
29.0
walkerr2dmedd med
90.7
:: 4.4
91.0 = 10.4
79.0 = 2.2
77.8
17.8
19.3
86.9 2.7
$1.9 + 2.8
89.6
74.5
78.5
6.6
hulfchoetah-mod med
54.2 0::1::22 2.4
56.3 0.5
45.2 0.2
42.1
42.3 1.6
77.6 1.5
54.2 1:
53.3
44.4
47.4
36.1
hopee--al--epay
4.8
82.7
34.7
80.5
102.5
wallser2d-med replay
85.0
haffiteetub-mmod--ilaay
36.8
40.2
55.1
48.0
38.4
hopper-mod-exp
103.4
100.1 10.0
89.3 +
108.7
23.7
60
83.3 9.1
111.1 2.9
111.9
111.0
91.5
111.9
waller?d--medsspp
112.2 + 1.7
110.5 1.4
105.8
1.4
95.6
44.6
12.9
68.3 + 1500
103.3
5.6
114.2
98-7
109.6
644
hallicheerah med exp
93.5 01.5
93.4
0.3
61.8
3.75
53.3
63.3
38.0
93.7 1 10.5
90,0 1 5.6
94.8
62.4
86.7
35.8
hes-human
72.8 = 13.9
50.0 = 15.6
62.3
8:35
53.1
37.5
71.5
34.4
hammer-hummn
1.9
1,1=10 1.4
3.1
1.9
1.5
4.4
1.4
1.5
door-human
2.4
2.75
0.5
relocate-hie humsm
0.25
pen- pen-cloned
51.4 15.5
45.2 15.8
29.2
56.9
hammer-cloned
0.15
2.1
0.8
door cloned
-0.] 0.0
-0.1 1.1
0.5 0.4
3.7
0.1
1.5
-0.1
relucate-choned
-0.0 1-0:000 0.0
-0.0 1 0.0
-0.0 1 0.0
0.2
-0.1
-022
-0.1
112.2
2: 6.3
1130
11.8
92.8 2 9.25
136.2
107.0
85.1
hommer-cxxp
1188 + 5.6
115.3
+ 9,3
51.0 + 11.05
126.9
86,7
125.6
dooc-exp
98.7 4.1
97.1 4.9
88.4 3.05
99.3
101.5
34,9
relocase-opp
Table 4: Performance comparison of ARMOR against baselines on the D4RL datasets. The values for ARMOR denote last iteration performance averaged over 4 random seeds along with standard deviations, and baseline values were taken from their respective papers. Boldface denotes perfor- mance within 10% of the best performing algorithm.
F.3 Effect of Residual Policy
In Figure 5, we show the effect on RPI of different schemes for initializing the learner for several D4RL datasets. Specifically, we compare using residual policy( Section 5) versus behavior cloning the reference policy on the provided offline dataset for learner initialization. Note that this offline dataset is the suboptimal one used in offline RL and is different from the expert-level dataset used to train and produce the reference policy. We observe that using residual policy (purple) consistently shows RPI across all datasets. However, with behavior cloning initialization (pink), there is large variation in performance across datasets. While RPI is achieved with behavior cloning initialization on hopper, walker2d and hammer datasets, performance can be arbitrarily bad compared to the reference on other problems. As an ablation, we also study the effect of using residual policy in
23

---

ORL
0.05
0.5
1.0
REF
0.01
0.05
0.1
0.5
1.0
mestmppy
and
(a) Hopper
(b) Walker2d
med
meddroplay
Iusson
closed
(c) Halfcheetah
(d) Pen
(c) Hammer
(f) Door
(g) Relocate
Figure 5: Comparison of different policy initializations for RPI with varying pessimism hyper- parameter ORL denotes the performance of offline RL with ARMOR Table 1), and REF the performance of reference policy. Purple represents residual policy initialization and pink is initialization using behavior cloning of the reference the suboptimal offline RL dataset.
the offline RL case where no explicit reference is provided, and the behavior cloning policy used the reference similar to Section 5.1. We include the results in Table 4 as ARMORT where observe that using a residual policy overall leads to worse performance across all datasets. This lends evidence the fact that using residual policy is compromise in instances where initializing the learner exactly to the reference policy is not possible.
F.4
Connection Imitation Learning
As mentioned in Section 4.2, IL is special case of ARMOR with À fully affect the adversarial MDP model, the best strategy of the policy is to mimic the reference. We test this on the expert versions of the D4RL locomotion tasks in Table 5, and observe that ARMOR can indeed perform IL to match expert performance.
Dataset
ARMOR-IL
BC
In this setting, the Q-function
hopper-exp
111.6
111.7
walker2d-exp
108.1
108.5
halfcheetah-exp 93.9
94.7
Table 5: ARMOR-IL on expert datasets. By setting 0, 3 > we recover II..
E.5 Ablation Study: RPI for Different Reference Policies
Here we provide ablation study results for robust policy improvement under different reference poli- cies for wide range of 3 values (pessimism syper-parameter.. For all the considered reference policies we present average normalized seroes for ARMOR and reference (REF) over multiple ran- dom seeds, and observe that ARMOR consistently outperform the reference for a large range of values.
Random Dataset Reference We use reference policy obtained by running behavior cloning on the RANDOM versions of different datasets. This equivalent using a randomly initialized neural network as the reference.
24

---

Dataset
0.01
0.05
0.1
0.5
1.0
10.0
100.0
200.0
500.0
1000.0
REF
hopper-med
1.3
1.5
4.7
20.4
34.8
25.8
40.8
25.6
28.9
1.2
walker2d-med
0.0
0.0
0.2
4.0
17.4
23.6
12.4
12.1
20.1
0.0
halfcheetah-mead
0.0
0.1
1.2
-0.1
-0.7
-0.3
-0.1
hopper-men med-replay
1.3
3.0
8.2
13.3
39.5
57.6
48.0
34.4
51.0
32,9
1.2
walker2d-man med-replay
0.0
0.0
0.1
4.0
5.9
13.1
10.8
14.1
16.5
16.6
0.0
haledhlctab-mmeddreplaay
-0.2
0.4
0.3
0.7
0.9
7.7
5.8
8.2
4.9
6.1
-0.2
Hand-designed Reference In this experiment we use hand-designed reference policy called RAN- DOMBANGBAN that selects either the minimum maximum action in the data with 0.5 proba- bility each.
Dataset
0.01
0.05
0.1
0.5
1.0
10.0
100.0
200.0
500.0
1000.0
REF
hopper-med
15.0
15.8
5.0
11.1
37.0
19.6
12.4
25.6
1.2
walker2d-med
31.6
12.0
55.9
33.6
36.6
49.6
0.1
halfchectah-med
10.6
19.1
11.7
15.6
14.3
8.3
11.1
-1.0
hopper-med-replay
33.9
54.8
62.6
66.7
55.3
57.0
67.7
79.5
70.7
62.0
1.3
walker2d-med -eppayy
11.6
35.5
47.5
40.4
42.7
47.6
64.8
49.9
44.7
33.0
0.1
halfcheetah- med-replay
13.1
10.1
11.3
9.7
12.4
17.6
9.0
14.5
11.1
9.1
-1.3
25


Reviews:
[{"review_id": "uLo70h5YYi", "reviewer": "Reviewer_V62y", "summary": "The paper introduces the Adversarial Model for Offline Reinforcement Learning (ARMOR), a model-based offline RL framework. ARMOR uses adversarial training to robustly learn and improve policies over any given reference policy, regardless of data quality. The framework utilizes the concept of 'relative pessimism' for worst-case optimization, ensuring that it either maintains or improves upon the performance of the reference policy, a property known as Robust Policy Improvement (RPI). The authors have also shown, theoretically and empirically, that ARMOR is robust to hyperparameter choices, and can outperform or be on par with state-of-the-art offline RL methods.", "questions": "Please refer to the weakness section.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and technically sound. The RPI in this paper is stronger than the RPI property in the literature, which only guarantees to be no worse than the behavior policy that collected the data. \n* The experimental results show the effectiveness of the proposed method and it outperforms existing baselines in many environments.\n", "weaknesses": "* I think the proposed method highly depends on the quality of the MDP model $M$. For the results in Table 1, in some environments, the proposed method does not outperform baselines. I think some explanation and intuitions are needed. I am wondering if the $M$'s structure is not optimal in those cases.\n* The model fitting loss in Equation (3) is not well-defined. Specifically, $r$ is not mentioned. Is it the same as $R^*$ as in Definition 1?\n* Minor: Please refrain from only using color to distinguish bars in Figure 3, as it is not friendly to readers with color blindness. Also, in Figure 2, not all illustrations are valid MDPs. I suggest the authors re-draw those toy examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We will incorporate them in the revision. We have addressed your comments below.\n\n**Weakness 1**\n\nWe would like to clarify that ARMOR (conceptually) maintains a set of models (i.e., the version space), not just a single one. We also assume that the model class is rich enough in theory such that it includes the true model $M^*$ (Assumption 1, realizability). Note that ARMOR doesn't need to explicitly recover the true $M^*$;  by maintaining a model set that contains $M^*$ and performing worst-case optimization over the set, it automatically enjoys the robustness guarantees (i.e., not being worse than the reference policy). However, robustness here comes at a cost, as worst-case reasoning over *a set of models* may lead to a conservative policy that does not aggressively optimize the return. Hence, there is a fundamental robustness vs. performance trade-off. In Table 1, we see ARMOR is comparable to other baselines across most datasets, with the only exception being halfcheetah domains when compared with RAMBO (in halfcheetah, ARMOR still performs better than other non-RAMBO baselines).\n\n**Weakness 2**\n \nIn Eq (3), $R_M$ denotes the predictions made by the model $M$ and $r$ denotes the reward labels in the data, as per the definitions in Sec. 2. We will make this clearer in the final draft.\n\n**Weakness 3** \n\nFig. 3: Thank you for pointing this out. We will update in the final draft.\n\nFig. 2: We are unsure what the reviewer means by “in Figure 2, not all illustrations are valid MDPs.” It would be helpful if the reviewer could elaborate on this so we can make the appropriate changes if required.\n"}, {"review_id": "rdTAsCqpPW", "reviewer": "Reviewer_suos", "summary": "The paper propose a new model-based offline RL framework, called ARMOR, which can robustly learn policies that improve upon an reference policy by adversarially training a Markov decision process (MDP) model.  ARMOR aims to optimize for the worst-case relative performance over uncertainty. In experiment, ARMOR implementation achieves good performance on D4RL benchmarks using only a single model.", "questions": "The implementation of Equation (4) somewhat contradicts my intuition. Specifically,\n- $M$'s initial goal was to enhance $\\pi_{\\rm ref}$'s $J$ and reduce $\\pi$'s $J$. However, in the final implementation, none of $M$'s optimization aspects involve $\\pi_{\\rm ref}$. This makes it unclear how the losses employed in practice can meet the original optimization objective of Equation (1).\n- I'm unclear on why the optimization process for the critic necessitates a pessimistic loss, e.g., $\\mathcal{L}_{D_M}$? The need for pessimism about the critic's estimates isn't evident in the objective Equation (1).\n\nIn Table 1, I'm curious on why several cloned datasets don't work by ARMOR? This seems to be the datasets very suitable for ARMOR to show better performance.\n\nIn Figure 3, the algorithm shows sensitivity to $\\beta$. It would be beneficial if the authors could elucidate the underlying reasons for this and provide any principles to guide researchers.\n\nThe experiments in Section 5.2 could be improved. The authors claim that \"ARMOR can robustly learn policies that improve upon an arbitrary reference policy...\" which is important. However, in the experiments, only performance improvements under a specific reference policy are demonstrated. It would be more constructive to verify this claim by picking various policies as reference policies and conduct experiments under one or two environments. I think DOPE [1] can be referenced as a work to conduct this experiment, which preserves many policies that can be used as reference policies.\n\nRegarding related work, a discussion could be made with the work [2] that utilizes adversarial approaches for offline environment learning. The optimization objective of this work coincidently using an opposite objective to the Equation (1) of this paper for model learning, which is \"$\\max_M \\min_\\pi \\ell(M, \\pi)$\".\n\nAlso suddenly inspired by the work [2], I would like to propose an open question for discussion: Can we also consider using adversarial policies further with ARMOR, such as $\\max_\\pi \\min_M \\max_{\\pi_{\\rm ref}}$? Since constructing $\\pi_{\\rm ref}$ from the offline data isn't a necessity, following Theorem 3, if we can create a sufficient number of $\\pi_{\\rm ref}$ in an adversarial manner, could we theoretically ensure that the final $\\hat \\pi$ approximates the optimal policy in any situation?\n\n[1]Benchmarks for Deep Off-Policy Evaluation.\n\n[2] Adversarial Counterfactual Environment Model Learning.", "limitations": "NAN", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is novel. The paper give us a new perspective to learn a dynamics model in offline RL, which is valuable to the community;\n2. The paper is overall well-written and easy to follow for me;\n3. The implementation is generally reasonble in intuition.", "weaknesses": "1. The description in Section 4.1 is quite confusing, with many details tucked away in the appendix. This arrangement is not particularly reader-friendly. I suggest that the authors better link the following points: (1) why the objective in Line 228 can substitute for Eq. (1); (2) how Line 228 is transformed into the goal of Line 199 and Algorithm 1. You don't need to elaborate on the relationship, but at least some intuition or motivation should be provided to the reader. Additionally, an optional suggestion: the authors might consider restructuring the overall arrangement by first introducing the part of Line 228 and then describing how it is implemented, which may result in a more natural discourse.\n\n2. The notation used for expressing errors is confusing here, with many instances of $\\mathcal{E}$ and $\\mathcal{L}$ representing loss, lacking in distinctiveness. A classic example is the symbols in Line 228, $\\mathcal{E}\\_{D}$  and $\\mathcal{E}_{\\hat D}$, both error symbols share the same structure, only the parameters differ. But they represent completely different losses, with the former being the maximum likelihood, and the latter the Bellman error. I suggest the authors revise the notations used, making it clear to readers what kind of loss it is without having to inspect the parameter differences.\n\n\n3. Figure 2 could be further polished: Without reading the main text, readers would not understand why ARMOR chooses the two models on the right, as the figure (including the caption) does not demonstrate that the reference policy is actually moving rightward. I believe the authors could add this information, allowing readers to better grasp the authors' motivation.\n\n3. Although D4RL is a fairly popular benchmark, I think the tasks from D4RL do not adequately highlight the advantages of the proposed algorithm. As the authors mentioned, this algorithm is motivated by real-world application scenarios, like \"Usually, the systems we apply RL to have currently running policies, such as an engneeded autonomous driving rule or a heuristic-based system for diagnosis, and the goal of applying a learning algorithm is often to further improve upon these baseline reference policies...\". However, the data in the selected D4RL datasets mostly come from mixed policies. The medium dataset aligns with this scenario, but the baselines already perform well, making it hard to discern the advantage of the proposed method. The recently released neorl [1] dataset might be more suitable for this work, as it was specifically designed for the scenarios the authors proposed, with all datasets collected by a single working policy, a concept similar to the authors' reference policy. \n\n\n\n[1] NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback on improving the readability and constructive comments. We hope that our responses below would help resolve the remaining concerns you might have.\n\n**W 1**: We thank the reviewer for pointing this out.\n\n[How Eq.(1) → L228]:\nWe have a detailed discussion regarding this in Appendix C due to the space limit. The high-level overview is: (1) Relax constrained optimization to regularized optimization. (2) Utilize the learned model to generate augmented data to perform relative pessimism properly (this is why ARMOR could have stronger RPI than the model-free opponents, e.g., ATAC).\n\n[How L228 → L199 and Algorithm 1]:\nWe would like to clarify that the goal of Algorithm 1 is essentially optimizing L229, whereas L199 denotes how each term on L228 is estimated in Algorithm 1.\n\n**W 2**: Currently, we use $\\mathcal{L}$ to denote the term that the policy and the model compete with, and $\\mathcal{E}$ to denote all other losses for enforcing data-consistency on the model. Different $\\mathcal{E}$ losses have different function signatures and we always include the input parameters the loss takes to make it clear. We will make them more visually different, such as by adding additional superscripts.\n\n**W 3**: Please refer to the Global Response and the updated figure in our uploaded PDF.\n\n**W 4**: Thanks for pointing this out. NeoRL seems like a great suite to test our algorithm in the future. However, it is relatively new and getting infrastructure setup to run experiments along with baselines takes a non-trivial amount of time. Also please note that our experiments are not limited to classic D4RL experiments. Here we have also constructed modified problems from D4RL to test RPI (sec 5.2 and Fig. 3): We specifically chose an expert reference policy that is beyond what the data can cover to showcase the benefit of RPI. In almost all cases, ARMOR using the expert reference can outperform the offline RL baseline and sometimes even the expert reference. \n\n**Q 1**: The model and pessimistic losses are connected to each other; as a result, the model is learned to maximize the performance difference between the two policies. This is implemented in Alg. 1 by *jointly* optimizing the critic and the model to be pessimistic. We let both the critic and the model minimize the Bellman error (the critic: pessimistic loss+ *Bellman error*; the model: *Bellman error*+ model fitting loss). So when the critic minimizes the pessimistic loss (to maximize the performance difference), the model also becomes pessimistic due to the link of Bellman error. We explained this in Lines 210-216 of the main paper, where we discuss how setting the weight on model fitting loss to be zero makes ARMOR equivalent to Imitation Learning (this is also verified empirically in Appendix G). We chose to implement ARMOR this way since it is more computationally efficient than directly optimizing Eq.(1) which would require back-propagation through full length rollouts generated by the model for every policy update.\n\n**Q Table 1**: The performance on the cloned datasets is on-par with the baselines. While in some cloned datasets ARMOR is not the best, no algorithm gives meaningful scores there (please see the scale in the -exp version). Since we implement the approximate version with neural networks, it is hard to always ensure perfect consistency between the theory and empirical results.\n\n**Q Fig 3**: We would like to point out that the results in Fig. 1 and Sec. 5.2 in fact show insensitivity to hyper-parameters for the small $\\beta$ regime, *consistent* with the theory. The theory states that we can improve upon the reference policy for a range of beta values. It does not assert that the learned policy with each beta would be the same, or have similar performance, just that they should all be no worse than the reference policy. In Fig 1, ARMOR is consistently able to outperform the reference policy for a range of beta, demonstrating insensitivity to $\\beta$.\n\n**Q Sec. 5.2**: Due to space constraints, we chose the hardest reference to compete with, i.e., a near-expert policy that lies outside the data support. This is because we know that running standard offline RL cannot achieve the expert level performance; as a result, if ARMOR achieves the expert performance, it must be due to RPI. Based on the review, we have conducted more experiments to show that ARMOR can achieve RPI with respect to arbitrary policies for a very wide range of $\\beta$ values. Please refer to the Global Response for more details.\n\n**Q Related Work [2]**: [2] considers a worst-case model fitting error (their Eq (3)), $ \\min_M \\max_{d^{\\pi_\\beta}} [ l_M( s,a,s’) ]$, where $l_M(s,a,s’)$ is the model fitting loss (e.g., the term in the sum of our Eq (3)), and then use the learned model for policy optimization. In contrast, ARMOR finds an adversarial model for each policy, and optimizes the policy that performs well under its adversarial model. We highlight that when the offline data does not have full coverage, their approach could lead to a poorly performing policy, because it does not consider the quality of model predictions outside data support, where the model can be arbitrarily bad. This makes their method inapplicable to offline RL under partial coverage setting here.\n\n**Q Open question**: We think your proposal is quite interesting. We suppose you meant modifying our objective in Eq. 1 to $ \\hat{\\pi} = \\arg\\max_{\\pi} \\min_M  \\max_{\\pi_{ref}} J_M(\\pi) - J_M(\\pi_{ref})$. We actually have a long, dedicated section on this in Appendix E (see line 641) and we call this approach Regret Minimization (as it tries to find the policy that has the smallest worst-case regret). Its worst case regret can be analyzed in a similar fashion as ARMOR. However, this algorithm can be overly conservative and does not necessarily have RPI to a given reference policy. We can discuss more in the final draft due to space limitation in rebuttal.\n"}, {"review_id": "e0Pr97CoHj", "reviewer": "Reviewer_pF8E", "summary": "This paper introduces ARMOR (Adversarial Model for Offline Reinforcement Learning), a novel model-based framework for offline reinforcement learning (RL) that addresses the challenge of performance degradation. Offline RL allows learning decision-making policies from logged data without requiring new data collection. ARMOR robustly learns policies that improve upon a reference policy by adversarially training a Markov decision process (MDP) model. The framework optimizes policies for worst-case performance relative to the reference policy, ensuring robust policy improvement regardless of data coverage. The authors provide theoretical proofs that ARMOR competes with the best policy within data coverage and never degrades the performance of the reference policy, even when the reference policy is not covered by the dataset.\n\nTo validate their claims, the authors present a scalable implementation of ARMOR that achieves state-of-the-art performance on D4RL benchmarks without using model ensembles. This makes ARMOR suitable for using high-capacity world models. The empirical results support the robust policy improvement property of ARMOR. ", "questions": "None", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, ARMOR offers a promising solution for offline RL, providing robust policy improvement over a broad range of hyperparameter choices and regardless of data coverage. The framework has practical implications for real-world problems where collecting diverse or expert-quality data is expensive or infeasible. The paper contributes theoretical analysis, a scalable implementation, and empirical validation.", "weaknesses": "1. Inadequate Discussion of Existing Limitations: The paper briefly mentions that offline RL algorithms have not been widely adopted beyond academic research, but it fails to provide a comprehensive discussion of the existing limitations and challenges that hinder their practical adoption. Without a clear identification and discussion of the existing limitations, the paper's motivation may appear less compelling.\n\n2. Conflicting results:  There is uncertainty regarding the claim that the policy learned by the proposed approach outperforms different hyper-parameter settings. In Figure 1, significant variations in performance can be observed for the different hyper-parameter choices. It is important to address and discuss this variation in more detail to provide a clearer understanding of the robustness and reliability of the proposed approach across different hyper-parameter settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reviewing the paper and providing valuable feedback. We hope our answers below can address your concerns.\n\n**Weakness 1**: “[the paper] fails to provide a comprehensive discussion of the existing limitations and challenges that hinder [offline RL’s] practical adoption”\n\nAs with any ML methodology, real-world adoption is always a complicated multi-faceted problem that comes with many challenges. For offline RL, there are many challenges, such as the difficulty of offline model selection, possible confounding in dataset, real-world non-stationarity, and the list can go on; it is virtually impossible to have a “comprehensive discussion” of all these issues in a technical paper (this is perhaps better suited for a survey/position paper). \n\nAmong all these challenges, we isolate a single important challenge and address it, namely \nthe risk of performance degradation from the current baseline policy. To illustrate, consider using offline RL to improve a recommendation system. The product team has a running policy (i.e., the baseline policy) right now and has some offline data, which is collected by the current running policy as well as other previous policies. Naturally, we want the policy learned by offline RL to be at least no worse than the current policy under deployment; otherwise, we should just continue running the current one. Such a scenario of learning to improve a baseline policy using mixed offline data is common in offline RL applications. Since these applications are often where making mistakes is costly (e.g., losing real money in this example), performance degradation is not allowed.\n\nThe extension of robust policy improvement (RPI) to improvement over any given reference/baseline policy is exactly motivated by this. However, no existing offline RL algorithm has this guarantee to our knowledge (that is, running them with an incorrectly picked hyperparameter can produce a policy worse than the baseline policy). \n\n\n**Weakness 2**: Conflicting Results\n\nWe would like to point out that the empirical results in Fig. 1 and Sec. 5.2 demonstrate that ARMOR can match or outperform the reference policy for a range of $\\beta$ values, **consistent** with the theoretical results. The theoretical analysis only states that we can improve upon the reference policy for a range of $\\beta$ values. It does not say the policy learned with each $\\beta$ would be the same or have similar performance, just that they should all be no worse than the reference policy. In Fig. 1 and Sec 5.2, ARMOR is consistently able to outperform the reference policy for a range of $\\beta$, demonstrating insensitivity to $\\beta$. Further, we would like to point the reviewer to our global response and the uploaded PDF,  where we provide further empirical evidence to support that the RPI property holds for different reference policies as well. "}, {"review_id": "XwHQJrd7LD", "reviewer": "Reviewer_sNsB", "summary": "A fundamental principle in offline RL is pessimism, which however is not free from the performance degradation with respect to the baseline reference policies, i.e., the currently running policies in the system.\nIn viewing this issue, this paper proposes Adversarial Model for Offline Reinforcement Learning (ARMOR), which can robustly learn policies that improve upon an arbitrary reference policy, regardless of the data quality.\nSpecifically, in ARMOR, the authors extend the technique of relative pessimism by adversarially training an MDP model, to achieve robust policy improvement (RPI) with arbitrary reference policies, regardless of whether they collected the data or not.\nBased on the theory, a scalable implementation of ARMOR is developed, which achieves competitive performance on D4RL benchmarks, while using only a single model rather than the widely-used ensembles.\n\n\n", "questions": "1. Can existing model-based methods, such as MOPO, MoREL or COMBO, utilize the reference policy as well? For example, a quick (and naive) thought may be adding an extra behavior-cloning term towards the reference policy on model-generated states (similar to L153-155). I am curious what preventing them from doing so? Otherwise, with such an augmentation, are they still underperform ARMOR?\n2. Could you elaborate more on the distinction between the reference policy and behavior policy? Are there any practical setting where the reference policy is significantly different from the policy that collect the offline dataset?  \n3. In Eqn. (1) and (2), why do you introduce the $\\alpha$ parameter? Maybe I miss something, but I don't find $\\alpha$ being used later on, except something like \"for $\\alpha$ large enough\".\n\n", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is theoretically supported.\n2. Practical implementation, including the selection of key hyperparameters, is discussed in details.\n3. Achieving competitive performance while using only a single model worth praising, since this is important for utilizing more sophisticated dynamics model.\n\n\n\n", "weaknesses": "1. The assumptions, in particular the realizability of the model class $\\mathcal{M}$ and $\\mathcal{M_\\alpha}$, and the policy class $\\Pi$, maybe demanding. It is unclear if or how these assumptions holds in practice. It is also unclear if the theoretical results still hold with a misspecified model and/or policy class, e.g., $M^*  \\notin \\mathcal{M}$, $M^*  \\notin \\mathcal{M_\\alpha}$ , and/or $\\pi_{\\mathrm{ref}} \\notin \\Pi$.  \n2. The authors draw a distinction between the reference policy and the data-collecting behavior policy. But in the experiments, the reference policy is simply set as the behavior-cloned policy on the same or a very similar offline dataset, e.g., expert v.s. medium. Therefore, this conceptual distinction seems artificial. \n    * In practice/experiments, can the learned policy still improve upon the reference when the behavior policy (offline dataset) behaves significantly different from the reference policy? \n    * And more importantly, why couldn't we simply learn from the data generated by the \"reference policy\"?\n3. The authors claim that the proposed method is robust to hyperparameters within a known set, i.e., the RPI property. But there is no experiment directly showing this. I would love to see more evidence and details pertaining to, for examples,\n    * How to select this set?    \n    * How large can this set be? \n    * How many hyperparameters can this set incorporate? \n4. Except the proposed method, the practical implementation (Section 4) shows many differences from the implementations of the baseline methods, such as MOPO, COMBO, CQL, IQL and so on. It is therefore unclear if the performance gain is due to the proposed method or due to these different choices of implementation details. This can be seen from the fact that the closely-related model-free method ATAC has performance similar to ARMOR and better than baselines on several datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nThank you for the detailed questions. We hope that our answers below would address your concerns and clarify the importance of the contribution we are making. \n\n\n**Weakness 1**\n\nThank you for bringing it up. We would like to emphasize that realizability is a standard assumption in the literature (see, e.g., [Uehara and Sun, 2021]). Furthermore, it is straightforward to handle the case of misspecified model and policy classes by additional additive terms. This is why we made the assumption. We will add this discussion in the camera-ready version.\n\n**Weakness 2**\n\nThe distinction between the reference and the behavior policy is more than just conceptual. There are multiple practical scenarios where we can’t learn from data collected by the reference alone: \n\n1. The data might be collected by a more exploratory policy. A priori, it's unknown whether the reference is good or safe.\nData comes from multiple policies (e.g., a system that iterates with the environment for a couple of rounds). The behavior policy is their mixture, but ARMOR allows us to set the reference to be the best.\n\n2. A concrete scenario is the following. Consider using offline RL to improve a recommendation system. The product team has a running policy (i.e., the baseline policy) right now and has some offline data, which is collected by previous policies and perhaps some from the current policy. Naturally, we want the policy learned by offline RL to be at least no worse than the current policy under deployment (i.e. the reference policy).  Such a scenario of learning to improve a baseline/reference policy using mixed offline data is common in offline RL applications. \n\nIn Sec 5.2, we explicitly construct experiments to simulate the scenario where the reference policy is distinct from the behavior policy, and demonstrate that robust policy improvement (RPI) still holds. We found that the learned policy can indeed improve over the reference even when the reference policy (the expert) behaves differently from the behavior policy (mediocre policies); see the gap in returns. There is significant improvement especially in the halfcheetah and adroit domains. In fact, we would argue that these are the major results of the paper, and Sec 5.1, where the reference is set to the behavior cloned policy is just a demonstration of ARMOR being used as a standard offline RL algorithm when no explicit reference policy is provided.\n\n\n\n**Weakness 3**\n\nAs per the theoretical results and main claims of the paper, ARMOR has robustness to the pessimism hyper-parameter $\\beta$ and not all hyperparameters. Our results in Sec 5.2 validate this RPI hypothesis. The theory suggests that RPI should hold for small values of $\\beta$. In practice, hence a good general strategy is to start with small $\\beta$ values, and gradually increase it to find the best performing policy. \n\n\n**Weakness 4**\n \nWe agree with the reviewer that this is an important question. However, in general it is hard to directly compare model-based and model-free methods over implementation details. More generally, it is difficult to study the effects of implementation details and conceptual ideas in isolation when comparing two very different deep RL algorithms. Each algorithm’s code can make different low-level design choices. Consequently, performing a strict comparison between the conceptual ideas is often impossible, unless we reimplement one algorithm based on another’s low-level details (but one might call this reimplementation yet another new deep RL algorithm?). Therefore, it’s hard to do such an isolated and direct comparison between ARMOR and other baselines like MOPO, COMBO, CQL, IQL as the reviewer mentioned.\n\n\nNonetheless, we would like to highlight some (beneficial) effects due to the conceptual difference ARMOR introduces beyond what implementation details can provide.\n\n1. Existing model-based methods such as MOPO and MOReL \"have to\" use ensembles of dynamics models for uncertainty quantification. But ARMOR does not, because it is based on adversarial training. As the reviewer mentioned, by *not* using an ensemble of models, ARMOR can be more suitable to train large models, when hosting ensembles of large models is too computationally expensive.\n\n2. While ATAC and ARMOR share the same low-level implementation details, we show that they have different properties. ATAC only demonstrates RPI with respect to the behavior policy whereas ARMOR, due to its model-based nature has RPI with respect to arbitrary reference policies including those that are not covered by the dataset. \n\n**Question 1** \n\nThis is an interesting idea but it is unclear how it can be done in a principled manner. The direction the reviewer suggested seems intuitive, but there are missing details (e.g., what the states to define the BC term in are and how to generate them) that can greatly affect the performance. Further it is unclear whether this design would have the same RPI guarantees as ARMOR in theory, as our current proof relies on the properties of adversarial training.\n\n\n**Question 2** \n\nPlease refer to the response for Weakness 2.\n\n**Question 3**\n\nThis $\\alpha$ parameter is to account for the finite-sample errors arising from estimating the true expectation with samples. It's a standard slackness parameter in forming version space (see, e.g., $\\xi$ in Eq (1) of [Uehara and Sun, 2021]). We note that if we choose $\\alpha =0$, we are only considering the model with the best fit empirically, which may not be the true model. To make sure we include the true model, we need to set $\\alpha$ to be a non-zero value (\"large enough\"), which can be informally thought of as a \"variance\" term that depends on sample size and the complexity of model class.\n"}]

Author rebuttal:
We would like to thank all the reviewers for taking the time to review our work and providing their constructive feedback. Here we provide responses to some of the common concerns


**RPI Experiments**

Reviewers “suos”, “pF8E” and “sNsB” had questions about the RPI experiments, specifically regarding sensitivity to hyper-parameter values and the nature of the reference policy. We believe that the primary source of confusion could be a misunderstanding of the theoretical results and the main claims of the paper. 

First, the theoretical analysis in Sec 3.2 and discussion in Sec. 6 state that ARMOR can improve upon the reference policy for a range of values for only the pessimism hyper-parameter $\beta$. They do not assert that the policy learned with each $\beta$ would be the same or have similar performance, just that they should all be no worse than the reference policy. Our results in Sec 5.2 and Fig. 1 clearly demonstrate this. 

Second, we chose a policy cloned on the expert dataset as the reference in Sec 5.2 because of space limitations and the fact that it is the hardest policy to show RPI against - a near-expert policy that lies outside the data support. Based on reviewer feedback, we have conducted further experiments to demonstrate that RPI holds for other reference policies as well. In our uploaded PDF, we provide empirical evidence for RPI against a policy cloned on the random dataset as the reference, as well as a hand-designed bang-bang control policy for an even wider range of $\beta$ values. These new experiments further bolster the point that ARMOR can obtain RPI with respect to arbitrary reference policies (which might be out of support) and is robust to the pessimism hyper-parameter $\beta$.


**Illustrative Example Figure**

Reviewer “suos”  pointed out the lack of clarity about the reference policy in Figure 2 and that the caption was not self contained. We have followed up on this input and have provided an updated figure and caption as part of our response PDF, to clearly denote what the reference policy is (changes in red). We also point the reviewers to a more comprehensive example in the appendix where we show that the same holds even when the reward function is learned.



We hope that our responses and new empirical results have alleviated the concerns the reviewers raised and they will consider updating their ratings accordingly.  


Meta-review: This paper addresses offline Reinforcement Learning by optimizing policies for the worst-case performance relative to the reference policy through adversarially training a Markov decision process model. It is theoretically shown to compete with the best policy within data coverage when the reference policy is supported by the data. With a scalable implementation, ARMOR is shown competitive on D4RL benchmarks using a single model.

The paper addresses an interesting problem, and the method is novel.  While the reviewers had a few concerns such as sensitivity to hyper-parameters in RPI experiments, extensive back-and-forth discussion between authors and reviewers finally brings the paper to the accepted side.  Please make sure to incorporate into the revised paper the clarifications in the rebuttal.
Decision: Accept (poster)