RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
Authors: Marc Rigter
Bruno Lacerda
Nick Hawes
Venue: NeurIPS 2022 Accept (2022)
Keywords: offline reinforcement learning
model-based reinforcement learning
deep reinforcement learning
robust reinforcement learning
adversarial learning

Abstract: Offline reinforcement learning (RL) aims to find performant policies from logged data without further environment interaction. Model-based algorithms, which learn a model of the environment from the dataset and perform conservative policy optimisation within that model, have emerged as a promising approach to this problem. In this work, we present Robust Adversarial Model-Based Offline RL (RAMBO), a novel approach to model-based offline RL. We formulate the problem as a two-player zero sum game against an adversarial environment model. The model is trained to minimise the value function while still accurately predicting the transitions in the dataset, forcing the policy to act conservatively in areas not covered by the dataset. To approximately solve the two-player game, we alternate between optimising the policy and adversarially optimising the model. The problem formulation that we address is theoretically grounded, resulting in a probably approximately correct (PAC) performance guarantee and a pessimistic value function which lower bounds the value function in the true environment. We evaluate our approach on widely studied offline RL benchmarks, and demonstrate that it outperforms existing state-of-the-art baselines.

RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
Marc Rigter, Bruno Lacerda, Nick Hawes Oxford Robotics Institute University of Oxford
fmrigter, bruno, nickh)&robots ox
Abstract
Offline reinforcement learning (RL) aims find performant policies from logged data without further environment interaction. Model-based algorithms, which learn model the environment from the dataset and perform conservative policy optimisation within that model, have emerged as promising approach to this problem. In this work, we present Robust Adversarial Model-Based Offline RL (RAMBO), a novel approach to model-based offline RL. We formulate the problem as two-player zero sum game against an adversarial environment model. The model is trained to minimise the value function while still accurately predicting the transitions the dataset, forcing the policy to act conservatively areas not covered by the dataset. To approximately solve the two-player game, we alternate between optimising the policy and adversarially optimising the model. The problem formulation that we address is theoretically grounded, resulting in a probably approximately correct (PAC) performance guarantee and pessimistic value function which lower bounds the value function in the true environment. We evaluate our approach on widely studied offline RL benchmarks, and demonstrate that it outperforms existing state-of- fofte-of-the-ee baselines.
Introduction
Reinforcement learning (RL) [6] has achieved state-of-the-an performance many sequential decision decision-making problems 40 45 59.. However, the need for extensive exploration prohibits the application of RL many real world domains where such exploration is costly dangerous, Offline overcomes this limitation by learning policies from static, pre-recorded datasets.
RI. 33
Online RL algorithms perform poorly in the offline setting due to the distributional shift between the state-action pairs in the dataset and those taken by the learnt policy. Thus, an important aspect of offline RL is to introduce conservatism prevent the learnt policy from executing state-action pairs which are out of distribution. Model-free offline RL algorithms [1] 23 27 29 31 72) train a policy from only the data present in the fixed dataset, and incorporate conservatism either into the value function or by directly constraining the policy.
use the dataset to learn a model of the environment, and train policy using additional synthetic data generated from that model. By training additional synthetic data, model-based algorithms can potentially generalise better to states not present in the dataset, or to solving new tasks. Previous approaches to model-based offline RL incorporate conservatism by estimating the uncertainty in the model and applying reward penalties for state-action pairs that have high uncertainty [76] 25). However, uncertainty estimation can be unreliable for neural network models 775 37 Like recent work (75), propose approach for offline model-based RL which does not require uncertainty estimation.
On the other hand, model-based offline RI. algorithms 76
75
25
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work we present Robust Adversarial Model- Baaeed Offline (RAMBO) RL, new algorithm for model-based offline RL. RAMBO incorporates conservatism by modifying the transition dynamics of the learnt Markov decision process (MDP) model in an adversarial manner. We formulate the problem of offline RI. as a zero-sum game against an adversarial environment. To solve the resulting maximin optimisation problem, we alternate between optimising the agent and optimising the adversary in the style of Robust Adversarial RL (RARL) [50]. Unlike existing RARL approaches, our model- based approach forgoes the need to define and train adversary policy, and instead only learns an adversarial model of the MDP. We train the agent policy with an actor-critic algorithm using synthetic data generated from the model in addition to data sampled from the dataset, similar to Dyna [60] and a number of recent methods [22 76 25 75). We update the environment model so that it reduces the value function for the agent policy, while still accurately predicting the transitions in the dataset. As a result, our approach introduces conservatism by generating pessimistic synthetic transitions for state-action pairs which out-of-distribttion. The theoretical formulation of offline RI. that our algorithm addresses yields a PAC bound for the performance gap with respect to any policy covered by the dataset, and a pessimistic value function that lower bounds the value function in the true environment.
In summary, the main contributions of this work are:
model-based offline RL algorithm which enforces conservatism by training an adversarial dynamics model.
RAMBO, a novel and theoretically-
Adapting the Robust Adversarial RL approach to model-based offline RL by proposing a new formulation of RARL, where instead of defining and training an adversary policy, we directly train the model adversarially.
In our experiments we demonstrate that RAMBO outperforms current state-of-the-art algorithms on the D4RL benchmarks [14] Furthermore, we provide ablation results which show that training the model adversarially is crucial the strong performance of RAMBO.
2 Related Work
Offline RL: Offline RL addresses the problem of learning policies from fixed datasets, and has been applied to domains such as healthcare 42 577 natural language processing 124 23], and
robotics 330 38 51. Model-free offlin RL. algorithms do not require a learnt model. Approaches for
model-free offline RL include importance sampling algorithms [36 41, constraining the learnt policy
to be similar to the behaviour policy [16] 28
72 23 [88 [5]. incorporating conservatism into the value
function during training 2 27 31 73,, using uncertainty quantification to generate more robust value
estimates 22. 229, or applying only single iteration of policy iteration 7 49]. In contrast, model-
based approaches learn model of the environment and generate synthetic data from that model (60)
to optimise policy using either planning 4 RL algorithms [25] 76 75). By training a policy additional synthetic data, model-based approaches have the potential for broader generalisation and for solving new tasks 66 766. A simple approach to ensuring conservatism is to constrain the policy to be similar to the behaviour policy in the same fashion as some model-free approaches [ 39 [3
Another approach is apply reward penalties for executing ftateacction pairs with high uncertainty in the environment model 225 [74] (76). However, this requires explicit uncertainty estimates which may be unreliable for neural network models 37 46 75). COMBO [75 obviates the need for uncertainty estimation in model- offline RL by adapting model-free techniques [31] to regularise the value function for dusto--distribution samples. Like COMBO, our approach does not require uncertainty estimation.
Most approaches to model-based offline RL use maximum likelihood estimates (MLE) of the MDP trained using standard supervised learning 4 39
76 75]. However, other methods have been proposed to learn models which are more suitable for offline policy optimisation. One approach is to reweight the loss function to ensure the model is accurate under the state-action distribution generated by the policy [34 533 20) In contrast, our approach produces pessimistic synthetic transitions when duttof-disttibuttio..
63
Most related to our work is recent paper [66] which introduces the maximin formulation of offline RL that we address. This existing work motivates our approach theoretically by showing that the problem formulation obtains probably approximately correct (PAC) guarantees. However, [66]
2

---

64 70 find the policy with the best worst-case performance over set of possible MDPs. Typically, it is assumed that the uncertainty set of MDPs is specified a priori. To eliminate the need to specify the set of possible MDPs, model-free approaches to Robust MDPs 688 551 instead assume that samples can be drawn from a misspecified we assume that we have a only addresses the theoretical aspects of the problem formulation and does not propose practical algorithm. In this work, we propose a practical RL algorithm solve the maximin formulation of model-based offline RI.
Robust RL: Algorithms for Robust MDPs [ 12 21
54
MDP which is similar to the true MDP. As our work addresses offline RL, fixed dataset from the true MDP.
Our approach is conceptually similar to Robust Adversarial RL (RARL) [50, method proposed to improve the robustness of RI. policies in the online setting. RARL is posed as a two-player zero-sum game where the agent plays against adversary which perturbs the environment. Formulations of model-free RARL differ in how they define the action space of the adversary. Options include allowing the adversary apply perturbation forces to the simulator [50], add noise to the agent's actions [65], or periodically take over control [48). A model-based approach to RARL is proposed in 13. which learns an optimistic and pessimistic model encourage online exploration. However, this existing approach requires uncertainty estimation as well as an adversarial policy to be learnt in addition to the model. Our work follows the paradigm of RARL and alternates between agent and adversarial updates in maximin formulation. We adapt model- RARL to the offline setting and propose an alternative formulation: we eliminate the need learn an adversary policy and instead directly modify the MDP model adversarially.
3 Preliminaries
Nn. 7). S and A MDPs and Offline RL: An MDP is defined by the tuple, M = R, denote the state and action spaces respectively, R(s, a) is the reward function, T(s'ss, a) is the transition function, μl is the initial state distribution, and y E (0, the discount factor. In this work we consider Markovian policies, E II, which map from each state to distribution over actions. We denote the (improper) discounted state visitation distribution of a policy by (s) == Et=o 7" Pr(s, s|æ,, M), where Pr(s, S/TT MM is the probability of reaching state at time by executing policy π M. The improper state-action visitation distribution di, (8, a) = x (als)) -d) (s). We also denote the normalised state-action visitation distribution by dā (s, = (1 7) (5, a).
The value function, V (5), represents the expected discounted return from executing æ from state s in M: (s) = EE,M] [i0 7 R(8,, ()((s, at)]. We write V to indicate the value function under the initial state distribution, i.c. Vi== = =..E VT (s) The standard objective for MDPs is to find the policy which maximises V The state-action value function, (ī, a). is the expected discounted cumulative reward from taking action a at state and then executing ा thereafter.
In offline RL we only have access to fixed dataset of transitions from the MDP, D {(s;, 0i Tr, Ti, [P) The goal of offline RL is to find the best possible policy using the fixed dataset.
Model-Based Offline RL Algorithms: Model-based approaches to offline RL model of the MDP to help train a policy. The dataset is used to learn a dynamics model, T. which is typically trained via maximum likelihood estimation: ming E<e,,'')<< - log T 8, a)) A model of the reward function, A(s, a), can also be learnt if it is unknown. The estimated MDP, M (S, A, R. (o: has the same state and action space as the true MDP but uses the learnt transition and reward functions. Thereafter, any planning or RL algorithm can be used to recover optimal policy in the learnt model, क arg max VX.
Unfortunately, directly applying this approach to the offline RL setting does not perform well due to distributional shift. In particular, if the dataset does not cover the entire state-action space, the model will inevitably be inaccurate for some state-action pairs. Thus, naive policy optimisation on a learnt model in the offline setting can result in model exploitation [22 32 53). To mitigate this issue, we propose the novel approach of enforcing conservatism by adversarially modifying the transition dynamics of M.
3

---

In line with existing works 76 75
we use model-based policy optimisation (MBPO) 22 to
learn the optimal policy for M. MBPO utilises standard actor-critic RL algorithm. However, the value function is trained using an augmented dataset D U D where Dūr is synthetic data generated by simulating rollouts in the learnt model. To generate the synthetic data, MBPO performs s-step rollouts in M starting from states 8 E 'D, and adds this data to Di7- To train the policy, minibatches of data are drawn from 'D U D TT- where each datapoint is sampled from the real data, D, with probability f. and from Djr x7 with probability 1 f-
Robust Adversarial Reinforcement Learning: RARL addresses the problem of finding a robust agent policy, æ, in the online RL setting by posing the problem as a two-player zero sum game against adversary policy,
arg max min V Tell #en
where VM the expected value from executing π and æ in environment M. Different approaches define the action space for गे in different ways, discussed in Section For scalable approximation to the optimisation problem in Equation [ algorithms for RARL alternate between applying steps of stochastic gradient ascent to the agent's s policy to increase the expected value, and stochastic gradient descent to the adversary's 's policy to decrease the expected value. In our work, we follow the RARL paradigm of alternating between agent and adversarial updates and adapt it to the model-based offline setting. Instead of defining a separate adversary policy, we treat the model itself as the policy to be adversarially trained.
Problem Formulation
For the sake of generality, we assume that both the transition function and reward function are unknown. Hereafter, we write F to denote both the learnt dynamics and reward function, where T(s', r|s, a) is the probability of receiving reward and transitioning s' after executing (s, a). We address the maximin formulation of offline RL recently proposed by [66]:
Problem 1. For some dataset, D, and some fixed constant & > 0, find the policy defined by
arg max min V
where
TEll TEMO
MD - {
ED [TV a),
SE].
where (V P2) is the total variation distance between distributions P1 and Pz, and TALE denotes the maximum likelihood estimate of the MDP given the offline dataset, D.
Thus, the set defined in Equation 3 contains MDPs which are similar to the maximum likelihood estimate under state-action pairs in D. However, because the expectation in Equation 3 is taken
under D, there is restriction on T for regions of the state-action space not covered by D. We present a brief overview of the theoretical guarantees from [66] in the following subsection.
Remark 1. Note that Problem T differs from the pessimistic MDP formulations introduced by MOPO [76) and MOReL [25). Problem [ooblmmmcoonsst considers the worst-case transition dynamics, while the pessimistic MDPs constructed by MOPO and MOReL only modify the reward function by applying reward penalties for state-action pairs with high uncertainty.
4.1 Theoretical Motivation
The theoretical analysis from [66] shows that solving Problem I outputs a policy that with high probability approximately as good as any policy with a state- acte-action distribution that covered by the dataset. This is formally stated in the following theorem.
Theorem (PAC guarantee from (66), Theorem 5). Denote the true MDP transition function by T. and let M denote a hypothesis class of MDP models such that T E MM. Let x denote the solution to Problem 7 for dataset D. Then with probubility for any policy, x² E II, we have
Gm. + GGM, + + + In(c/6)
V
V < (1 - 7)=
where
[D]
4

---

E ((aa))
a))
C..
max T'EM B( a)),, a), T(- a))²
where p is the state-action distribution from which DD was sampled, and and e₁ are universal constants. We refer the reader to Appendix A of [66 for the definitions of GMy GM2 and En-
The quantity CT- is upper bounded by the maximum density ratio between the comparator policy, n², and the offline distribution, i.e. C.- S ma)(s.a) di (s, a)/ (s, a). It represents the discrepancy between the distribution data in the dataset compared to the visitation distribution policy n². Theorem shows that if we find policy by solving Problem [ the performance gap of that policy is bounded with respect to any other policy π that has a state-action distribution which is covered by the dataset.
Furthermore, the value function under the worst-case model in the set defined by Problem a lower bound on the value function in the true environment, as stated by Proposition]] |
Proposition 1 (Pessimistic value function). Let T denote the true transition function for some MDP, and let MD be the set of MDP models defined in Equation 3 1 we have that
Then for any policy т, with probability
min V S V PEMD
Proposition follows from the fact that T E MD with high probability, which proven in [66 (Appendix E.2). Proposition [ shows that we can expect the performance of any policy in the true MDP be least as good as the value in the worst-case model defined in Problem [
While [66] provides the theoretical motivation for solving Problem I, it does not propose a practical algorithm. In this work, we focus on developing a practical approach to solving Problem
5 RAMBO-RL
In this section, we present Robust Adversarial Model-Based Offline RL (RAMBO), our algorithm for solving Problem [ The main difficulty with solving Problem i that it is unclear how to find the worst-case MDP in the set defined in Equation To arrive at a scalable solution, we propose a novel approach which is in the spirit of RARL. We alternate between optimising the agent policy to increase the expected value, and adversarially optimising the model to decrease the expected value. In this section, we first describe how compute the gradient to adversarially train the model. Then, we discuss how to ensure that the model remains approximately within the constraint set defined in Finally, we present our overall algorithm.
Problem I
5.1 Model Gradient
We propose a policy cradient-inspired approach to adversarially optimise the model. Typically, policy gradient algorithms are used to modify the distribution over actions taken by policy at each state 662 71. contrast, the update that we propose modifies the likelihood of the successor states and rewards in the MDP model.
We assume that the MDP model defined by parameters, Φ, and we write T' to indicate this. We denote by VT the value function for policy æ in model Tg. To approximately find ming.eMa EMM V as required by Problem
| via gradient descent, we wish to compute the gradient of the model parameters that reduces the value of the policy within the model, i.e. V.V.
Proposition 2 (Model Gradient). Let & denote the parameters of parametric MDP model T and let V denote the value function for policy n in To Then:
eedd aa (6' P.
[(r (V)) V log (s', a))
(4)
The proof of Proposition 2 given in Appendix A We can subtract the baseline Q3(s, a) without biasing the gradient estimate (see Appendix A. for details):
V E Foadd [(r
7 (z a)) V log Tol r/8,
(5)
5

---

The Model Gradient differs from the standard policy gradient in that a) it is used to update the likelihood of successor states in the model, rather than actions policy, and b) the "advantage" term r + VV (5") Q% (s,a) compares the utility of receiving reward r and transitioning to 8' to the expected value of state action pair (s, a). In contrast, the standard advantage term, Q(s, a)- (s) (56), compares the value of executing state-action pair (8, a) to the value at 8. To estimate V and Q we use the critic learnt by the actor-critic algorithm used for policy optimisation. Thus, we use the critic both for training the policy and adversarially training the model.
Remark 2. The Model Gradient can be thought of as a specific instantiation of the policy gradient if we view To as adversarial "policy" on an augmented MDP, M+, i.e. Fg : S+ - Dist The augmented state space, S = S x includes the state in the original MDP augmented by the action taken by the agent. The augmented action space consists of the reward applied and the successor state, A+ = 5 x [miin R. Thus, we can think of this approach as an instantiation of RARL in which the adversary policy (# in Equation that train is the model itself.
5.2 Adversarial Model Training
If we were to update the model using Equation Eauation5Jaaoone alaone, this would allow the model to be modified arbitrarily such that the value function in the model is reduced. However, the set of plausible MDPs given by Equation 3 states that over the dataset, D, the model To should be close to the maximum likelihood estimate, TMPE. Specifically, in the inner optimisation of Problem we wish to find a solution to the constrained optimisation problem
min Vi, s.l. ED [TV To
s.
(6)
The Lagrangian relaxation leads to the unconstrained problem
(L(T,A) == VT + X(ED [TV a), Ta((ss, a)) -)))
max min X20 To
(7)
where À is the Lagrange multiplier. Rather than optimising the Lagrange multiplier, we find that in practice fixing À to apply a constant weighting between the two terms works well with minimal tuning. To facilitate easier tuning of the learning rate, in our implementation we apply the weighting constant to the value function term rather than the model term, which is equivalent up to a scaling factor. This leads to
(AV) Φ +ED T3
min
We aim to make our algorithm efficient and simple implement. Therefore, rather than minimising the TV distance between the model and MIE model as prescribed by Equation 8 we directly optimise the standard MLE loss. This leads the final loss function:
C. AV )( log rlss.
Thus, the loss function for the model in Equation 9 simply adds the adversarial term to the standard MLE loss. By minimising the loss function in Equation 9 the model is trained to a) predict the transitions within the dataset, and b) reduce the value function of the policy, with À determining the tradeoff between these two objectives. Choosing À to be small ensures that the MLE term dominates for transitions within D. ensuring that the model fits the dataset accurately. Because the MLE term is only computed over D. the adversarial term dominates outside of the dataset meaning that the model is modified adversarially for transitions outside of the dataset.
To estimate the gradient of the loss function in Equation 9 for stochastic gradient descent, we sample a minibatch of transitions from D to estimate the MLE term. The gradient for the value function term is computed using the Model Gradient. The transitions used to compute the Model Gradient term must be sampled under the current policy and model (Equation Therefore, to estimate the Model Gradient term, we generate a minibatch of transitions by simulating the current policy in T..
Like previous works
we represent the dynamics model using an ensemble of neural networks. Each neural network produces a Gaussian distribution over the next state and reward:
10)
75

---

ro(s',rls,)) = (Huo(s, a), Es(s,a)), a)). A visualisation of the result of adversarially training the dynamics model can be found in Appendix
Normalisation The composite loss function in Equation comprises two terms which may have different magnitudes across domains depending on the scale of the states and rewards. To enable easier tuning of the adversarial loss weighting, A, across different domains we perform the following normalisation procedure. Prior training, we normalise the states in D in the manner proposed in [1]]. by subtracting the mean and dividing by the standard deviation of each state dimension in the dataset. Additionally, when computing the gradient in Equation [ we normalise the advantage terms, r + VV (s') (s, a). according to the mean and standard deviation across each minibatch.
Advantage normalisation common practice in policy gradient RI. implementations
3 52
5.3 Algorithm
We are now ready to present our overall approach in Algorithm The first step of RAMBO is to pretrain the environment dynamics model using standard MLE (Line Thereafter, the algorithm follows the format of RARL.. At each iteration, we apply gradient updates to the agent to increase the expected value, followed by gradient updates to the model to decrease the expected value.
Prior each agent update, we generate synthetic k-step rollouts starting from states in D by simulating rollouts in the current MDP model Tø. This data is added to the synthetic dataset Dj, (Line
Following previous approaches 776
we only store data from recent iterations Djj- The agent's policy and value functions are trained with an off-policy actor-critic algorithm using samples from D Dr, (). In our implementation, we use soft actor-critic (SAC) [99 for agent training.
75 22
To update the model to minimise the loss in Equation (Line we sample data from D to estimate the gradient for the MLE component. To compute the adversarial component we generate samples by simulating the current policy and model, and utilise the value function learnt by the agent to compute the gradient according Equation
Algorithm RAMBO-RL
Require: Normalised dataset, D;
To < -LLE dynamics model.
2: for i = ...,..., Pliter do
3:
Generate synthetic k-step rollouts. Add transition data to D..
4: Agent update: Update π and Q3 with an actor critic algorithm, using samples from D U Dr.
5: Adversarial model update: Update Ta according to Eq. using samples from D for the MLE component, and the current critic 2 and synthetic data sampled from and Te for the adversarial component.
Experiments
In our experiments, we aim to: a) evaluate how well RAMBO performs compared to state-of- the-art baselines, b) examine whether RAMBO can be tuned offline, c) determine the impact of adversarial training on the performance of the algorithm, and d) investigate the difference between RAMBO and COMBO, the most similar prior algorithm. The code for experiments is available We evaluate our approach on the following domains.
at haathub.com/maac-ritter/aa///l/
MuJoCo There three different environments representing different robots (HalfCheetah, Hopper, Walker2D), each with 4 datasets (Random, Medium, Medium RRelay, Medium Medium-Expert). Random contains transitions collected by random policy. Medium contains transitions collected by an early-stopped SAC policy. Medium-Repla consists the replay buffer generated while training the Medium policy. policy. The Medium- Expert dataset contains mixture of suboptimal and expert data.
AntMaze The agent controls a robot and navigates to reach a goal, receiving a sparse reward only the goal is reached. There are three different layouts of maze (Umaze, Medium, Large), and different dataset types (Fixed, Play, Diverse) which differ in terms of the variety of start and goal locations used to collect the datasct. The MuJoCo and AntMaze benchmarks from D4RL [4].
7

---

Hyperparameter Details The base hyperparameters that we use for RAMBO mostly follow those used SAC [9] and COMBO 75). We find that the performance of RAMBO is sensitive the choice of rollout length, kt consistent with findings in previous works 22 The other critical parameter for RAMBO the choice of the adversarial weighting, A.
For each dataset, we choose the rollout length and the adversarial weighting from one of three possible configurations: (k, A) E {(2. 3c-4), (5, 3c-4), (5, 0)}. We included (k, A) (5, 0) as we found that an adversarial weighting of O worked well for some datasets. For the MuJoCo datasets we performed model rollouts using the current policy, and initialised the policy using behaviour cloning (BC) which is a common practice in offiin RL (25 74). Ablation results in Appendix (C.4 indicate that the BC initialisation results in small improvement. For the AntMaze datasets we used a random rollout policy and randomly initialised policy as we found that this performed better. Further details about the hyperparameters are Appendix B
Evaluation We present two different evaluations of our approach: RAMBO and RAMBOOFF For RAMBO, we each of the three hyperparameter configurations for five seeds each, and report the best performance across the three configurations. Thus, our evaluation of RAMBO utilises limited online tuning which is the most common practice among existing model-based offline RL algorithms [25 37
39 [6]. The performance obtained for each of the hyperparameter configurations
included Appendix C.2
Offline hyperparameter selection an important topic in offline RL (7). Therefore, we present additional results for RAMBOOFF where select between the three choices of hyperparameters offline using a simple heuristic (details in Appendix [B.5) based on the magnitude and stability of the Q-values during offline training. We first select the hyperparameters using the heuristic, and then rerun each dataset for 5 seeds to generate the results for RAMBOOFF
Baselines We compare RAMBO against state-of-the-arr model-based (COMBO 175.. RepB- SDE (34), MOReL (25), and MOPO (76)) and model-free (CQL [31]. IQL (28]. and TD3+BC (15) offline RL algorithms. We provide results for all algorithms for the MuJoCo-v2 D4RL datasets and the AntMaze-vO datasets (details in Appendix B.7).
6.1 Results
The results in Table show that RAMBO achieves the best total score for the MuJoCo locomotion domains, outperforming existing state-of-the-art methods. Furthermore, RAMBO obtains the best overall score both the Medium and Medium--Rplan dataset types. For the random datasets, RAMBO is outperformed only by MORel.. This shows that RAMBO performs very well for datasets that either noisy consist of suboptimal data.
D4RL Performance
For the Medium- -Exuert- datasets, RAMBO is outperformed by most of the baseline algorithms, suggesting that RAMBO is less suitable for high-quality datasets. However, for the Medium-Exx Expert datasets, simpler approaches such as performing behaviour cloning on the best 10% of trajectories be used to achieve stronger performance than offline RL methods 28). Therefore, the suboptimal performance of RAMBO on the Medium Cediuum-Expert datasets is less of a concern, as applying offline RL algorithms may not be the most suitable approach for these high-quality datasets.
For AntMaze, the model-based algorithms perform considerably less well than the model-free approaches. Unlike the other model-based approaches, RAMBO at least scores greater than zero for most of the datasets. Our results echo previous findings that model-based approaches struggle to perform well in the AntMaze domains, potentially because model-based algorithms are too aggressive and collide with walls (67). Recent work [67] showed that reverse rollouts can lead to stronger performance for model-based methods in these domains. In future work, wish to investigate whether combining RAMBO with reverse rollouts improves the performance for AntMaze.
Offline Tuning In Table we also present results for RAMBOOFF where the final hyperparameters
are chosen using the heuristic described in Appendix B.5 We see that there is a slight degradation in the performance relative to RAMBO, which uses online tuning. However, RAMBOOFF still achieves comparable performance to the best existing approaches on the MuloCo datasets. This suggests that suitable hyperparameters for RAMBO can reliably be chosen using our offline heuristic.
8

---

Table 1:
Results for the D4RL benchmark using the normalisation procedure proposed by [1. We
report the normalised performance during the last 10 iterations of training averaged over seeds. = captures the standard deviation over seeds. Highlighted numbers indicate results within 2% of the most performant algorithm. indicates the total without random datasets.
Ours
Model- baselines
Model- free baselines
RAMBO
RAMBOURY
RepB-SDE COMBO MOPO MOReL
CQL
IQL
TD3+BC
BC
Medium Random
HalfCheetah
40.0 + 2.3
33.5 + 2.6
32,9
38.8
35.4
25.6
19.6
11.0
2.1
Hopper
21.6
15.5
8.6
4.1
53.6
6.7
8.5
9.8
Walker2D
11.5 10.5
0.2
21.1
7.0
4.2
37.3
2.4
1.6
1.6
HalfChhetah
77.6 + 1.5
71.0 + 3.0
49.1
54.2
69.5
42.1
49.0
47.4
48.3
36.1
Hopper
92.8 6.0
91.2 16.3
34.0
94.9
48.0
95.4
66.6
66.3
59.3
29.0
Walker2D
86.9 2.7
89.1 + 2.7
72.1
75.5
-0.2
77.8
83.8
78.3
83.7
6.6
Medium Replay
HalfCheetah
68.9 + 2.3
67.0 + 1.5
57.5
55.1
68.2
40.2
47.1
44.2
44.6
38.4
Hopper
96.6 + 7.0
97.6 + 3.4
62.2
73.1
39.1
93.6
97.0
94.7
60.9
11.8
Walker2D
85.0 :: 15.0
88.5 4 4.0
49.8
56.0
69.4
49.8
88.2
73.9
81.8
11.3
Medium Expert
HalfCheetuh
93.7 + 10.5
79.3 2.9
55.4
90.0
72.7
53.3
90.8
86.7
90.7
35.8
Hopper
83.3 +: 9.1
89.5 11.1
82.6
111.1
3.3
108.7
106.8
98.0
111.9
Walker2D
68.3 20.6
63.1 31.3
88.8
96.1
-0.3
95.6
109.4
109.6
110.1
6.4
Mu.loCo-22 Total:
826.2 + 33.8
785.5 + 40.4
614.1
769.7
413.4
773.0
767.4
692.6*
698.5
300.8
25.0 +: 12.0
23.8 15.0
80.3
0.0
78.6
AntMare
Medium-Play
16.4 + 17.9
5.6 10.9
0.0
0.0
0.0
61.2
71.2
3.0
0.0
Large-Play
0.0 0.0
0.0
0.0
0.0
0.0
0.0
15.8
39.6
0.0
0.0
Umaze-Diverse
0,0 0.0
0.0
0.0
57.3
0.0
0.0
84.0
62.2
71.4
55.0
Medium-Divee
23.2 + 14.2
9.4
0.0
0.0
0.0
0.0
53.7
70.0
10.6
0.0
Large-Diverse
2.4
0.0
0.0
0.2
AntMaze-rO Total:
67.0 14.9 37.8 ::
12.4
0.0
137.6
0.0
0.0
303.6
378.0
163.8
120.0
Table 2:
Ablation of the adversarial updates for RAMBO. These results use the same rollout length for
Table 3:
Comparison between RAMBO and COMBO for the Single Transition Ex- ample. We use 20 seeds and = captures the standard deviation over seeds. RAMBO outperforms COMBO (p 0.005).
each dataset as RAMBO but with no adversarial updates (i.e. 0). The scores are averaged over 5 seeds.
RAMBO (No Adversarial Training)
MuJoCo-v2 Total: 694.4 56.5 AntMaze-vi Total: 45.8 21.8
RAMBO: 1.49 + 0.05 COMBO: 1.41 0.12
Ablation of Adversarial Training In Table we present results for RAMBO with no adversarial updates. These results demonstrate that overall performance degrades if the adversarial training is removed. This parallels previous findings that mitigating the issue of model exploitation crucial to obtaining strong performance in model-based offline RL. Interestingly however, for some specific datasets we obtain the best performance with no adversarial training (Appendix C.2). This suggests that potential direction for future work could be trying to identify which types of problems do not require regularisation for a successful policy be trained offline with model-based RL.
We focus especially on comparing our approach to COMBO, as it is the most similar existing algorithm. We compare RAMBO and COMBO on the Single Transition toy example which is described in detail in Appendix C.I This domain has one-dimensional state and action space, and several distinct regions of the action space are covered by the datasct. We use this domain to investigate whether the policies optimised by RAMBO and COMBO tend to become stuck in local optima.
Comparison to COMBO
Table 3 compares the performance of RAMBO and COMBO on the Single Transition Example. Further analysis in Appendix shows that for this problem, the pessimistic value function updates used by COMBO create local maxima in the Q-function which are present throughout training. Policy optimisation can become stuck in these local maxima. On the other hand, the value function produced by RAMBO is initially optimistic, and pessimism is introduced into the value function gradually as the transition function is modified adversarially. Adversarial modification of the transition function visualised in Appendix
C,3 As result of this gradual introduction of pessimism, we observe that the

---

policy produced by RAMBO is less likely to become stuck in poor local maxima, and better overall performance is obtained in Table 3 This observation may help to explain why RAMBO is able to achieve consistently strong performance relative to existing algorithms in the MuJoCo domains. Gradually increasing the level of pessimism could be useful modification for existing offline RI. algorithms to be investigated in future work.
7 Conclusion and Future Directions
RAMBO is a promising new approach to offline RL which imposes conservatism by adversarially modifying the transition dynamics of learnt model. Our approach is theoretically justified, and achieves state-of-the-an performance on standard benchmarks.
There are number of possible extensions to RAMBO, some of which we have already discussed. In addition, we would like apply RAMBO image-space domains by using deep latent variable models to compress the state space [8] 5 and adversarially perturbing the transition dynamics in the latent space representation. Another direction that we would like to investigate is the use of adversarially trained models to aid interpretability in deep RL 44] by generating imagined worst-case trajectories. Finally, we wish to investigate applying the ideas developed in this work to the online RL setting to improve robustness.
Acknowledgements
This work was supported by a Programme Grant from the Engineering and Physical Sciences Research Council (EP/V000748/1), the Clarendon Fund at the University of Oxford, and gift from Amazon Web Services. Additionally, this project made use of time on Tier 2 HPC facility JADE2, funded by the Engineering and Physical Sciences Research Council (EP/TO22205/1),
The authors would like to thank Raunak Bhattacharyya, Paul Duckworth, and Matthew Budd for their feedback earlier draft of this work.
10

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pages 104-114. PMLR, 2020.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified Qensembbe.. Advances in Neural Information Processing 5ystemss,4.7436-7777 3477736-744,, 2021.
[3] Marcin Andrychowicz, Anton Raichuk, Piotr Staüczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? A large-scale study. In International Conference Learning Representations, 2021.
[4] Arthur Argenson and Gabriel Dulac-Arold. Model- offline planning. In International Conference on Learning Representations, 2021.
[5] J Andrew Bagnell, Andrew Y Ng, and Jeff G Schneider. Solving uncertain Markov decision processes. Technical report, 2001.
[6] Philip Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In International Conference on Machine Learning, pages 619-629, PMLR, 2021.
[7] David Brandfonbrener, Will Whimey, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4998 2021.
[8] Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline RL. In Deep RL. Workshop NeurIPS 2021, 2021.
[9] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. 2022.
[10] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
[11] Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-augmeented actor-critic: Backpropagating through paths. International Conference on Learning Representaations, 2020.
[12] Michael K Cohen and Marcus Hutter. Pessimism about unknown unknowns inspires conser- vatism. In Conference Learning Theory, pages 1344-1377 PMLR, 2020.
[13] Sebastian Curi, Ilija Bogumovic, and Andreas Krause. Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning. In International Conference on Machine Learning, pages
PMLR, 2021.
[14] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/99 2020.
[15] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[16] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062
PMLR, 2019.
[17] Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty deep neural networks. arXiv preprint arXtv:2107.03342 2021.
[18] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Ad- vances in Neural Information Processing Systems, 31, 2018.
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.

---

[20] Toru Hishinuma and Kei Senda. Weighted model estimation for offline model-based reinforce- ment learning, Advances in Neural Information Processing Systems, 34, 2021.
[21] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
[22] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
[23] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint M.Niv:9907.0044 2019.
[24] Kirthevasan Kandasamy, Yoram Bachrach, Ryota Tomioka, Daniel Tarlow, and David Carter. Batch policy gradient methods for improving neural conversation models. International Confer- ence on Learning Representations, 2017.
[25] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-based offline reinforcement learning. Advances in neural information processing systems, 18:21810-2182,, 2020.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 2015.
[27] lyy Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with Fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[28] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations, 2022.
[29] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy Q-learnig via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[30] Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. Conference on Robot Learning, 2021.
[31] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 1191, 2020.
[32] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbecl. Model-ensemblle trust-region policy optimization. In International Conference on Learning Representations, 2018.
[33] Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[34] Byung-Jun Lee, Jongmin Lee, and Kee-Eung Kim. Representation balancing offline model- based reinforcement learning. In International Conference on Learning Representations, 2020.
[35] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 1.Xiv:2005.0164: 2020.
[36] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. International Conference on Machine Learning RL4RealLife Workshop, 2019.
[37] Cong Lu, Philip Ball, Jack Parker-Holder Michael Osborne, and Stephen J Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations, 2022.
[38] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEFE, 2020.
[39] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efficient reinforcement learning via model-based offline optimization. In- ternational Conference on Learning Representations, 2021.
12

---

[40] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
[41] Ofir Nachum, Bo Dai, llya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gacDICE: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02072 2019.
[42]
Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning when-to-treat policies. Journal of the American Statistical Association, 116(533):392-4D)) 2021.
[43] Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5)7880-798, 2005.
[44] Michael Oberst and David Sontag. Counterfactual off-policy evaluation with gumbel-max structural causal models. International Conference on Machine Learning, pages 4881-4890 PMLR, 2019.
[45] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand. arXiv preprint, 2019.
[46] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in Neural Information Process- ing Systems, 32, 2019.
[47] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement 2020.
learning. arXiv preprint arXiv:2007.0900
[48] Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforce- ment learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 8522-8528. IEFE, 2019.
[49] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
[50] Lerrel Pinto. James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826.
PMLR, 2017.
[51] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pages 1154-1168. PMLR, 2021.
[52] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. babll-- Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
[53] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. game theoretic framework for model based reinforcement learning. In International Conference Machine Learning, pages 7953-7963. PMIR, 2020.
[54] Mare Rigter, Bruno Lacerda, and Nick Hawes. Minimax regret optimisation for robust planning in uncertain Markov decision processes. In Proceedings the AAAI Conference Artificial Intelligence, volume 35, pages 11930-1 1938. Association for the Advancement of Artificial Intelligence, 2021.
[55] Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. Advances in Neural Information Processing Systems, 30, 2017.
[56] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. International Confer- ence on Learning Representationss 2016.
[57] Susan M Shortreed, Eric Laber, Daniel J Lizotte, T Scott Stroup, Joelle Pineau, and Susan A Murphy. Informing sequential clinical decision-making through reinforcement learning: an empirical study. Machine learning, 84(1):109-136, 2011.
13

---

[58] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki. Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
[59]
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of Go without human knowledge. Nature, 550076766:354-359, 2017.
[60] Richard S Sutton. Dyna, integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 204):160-163, 1991.
[61] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[62] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
[63] Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104: 104:16436, 2021.
[64] Aviv Tamar. Shic Mannor, and Huan Xu. Scaling up robust MDPs using function approximation. In International Conference on Machine Learning, pages 181-189. PMLR, 2014.
[65] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages PMLR, 2019.
[66] Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. International Conference on Learning Representations, 2022.
[67] Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34, 2021.
[68] Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 34, 2021.
[69] Paul Werbos. Backpropagation through time: what does and how to do it. Proceedings of the IEEE, 78(10): 1990.
[70] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Mathematics of Operations Research, 38 38(1):153-183 2013.
[71] Ronald J Williams. Simple statistical soudiient-folloiing algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229 256, 1992.
[72] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1161,, 2019.
[73] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[74] Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-based offline reinforcement learning. In International Conference on Learning Representations, 2021.
[75] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
[76] Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 13:14129-14442, 2020.
[77] Siyuan Zhang and Nan Jiang. Towards hyperparameter-free policy selection for offline rein- forcement learning, Advances in Neural Information Processing Systems, 34, 2021.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No] or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We included results for the
AntMaze datasets where our approach performs less well than model-free methods.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] We include complete proof of Proposition For the theoretical results from [66. provide references to the appropriate parts of that paper.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] A URL the code is provided.
(b) Did you specify all the training details (e.g., g.. data splits, hyperparameters, how they were chosen)? [Yes] Detailed information about implementation details and hyperparameters in the appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Standard deviation with respect to seeds is reported for all results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details of compute used per run and total compute used for full evaluation is in Appendix B.8
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the D4RL benchmark datasets [4]
(b) Did you mention the license of the assets? [Yes] In Appendix B.7
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code for our work is in included via URL.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
15

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "io2yKENun2P", "reviewer": "Reviewer_xriK", "summary": "The authors tackle the generic Offline RL problem and propose a model-based approach to address it. They introduce a novel algorithm (RAMBO), consisting in adversarially training the model in order to minimize the trained policy performance estimate. The justification of the work builds on a previously published theoretical paper. The empirical validation of the algorithm is limited to MuJoCo-v2 domain where it performs very well, and the AntMaze-v0 where it achieves better than other model-based baselines, but much worse that model-free baselines.", "questions": "In addition to addressing the weakness points above, I would be appreciative that the authors answer the following questions:\n* Equation 9: why use the *standard* MLE loss instead of the TV loss as the theory prescribes?\n* Remark 1: I think that this point is important to make stronger. I fail to see why Problem 1 does not reduce to pessimistic reward modification. My intuition is that the worst possible transition outcome consists in transiting to a state with minimal value, which is equivalent to applying a pessimistic reward modification of this amplitude.\n\nMinor points:\n* Line 1: find near-optimal => near-optimality is intractable in general in Offline RL, so this is not the right objective for Offline RL.\n* Line 6: achieve conservatism => enforce conservatism.\n* I don't like the use of conservatism to refer to any kind of regularisation with respect to the dataset, including pessimism. Generally, conservatism is the regularization of the policy to remain close to that of the behavioural policy, which is generally made in opposition to pessimism which intends to find a lower bound to the value function and does not guarantee to remain close to the behavioural.\n", "limitations": "As I have said before, I am worried that over-stating theoretical findings may lead to improper use of them in the future. Other than that, I do not have any concern with respect to the societal impact of this submission.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The authors propose a novel algorithm for model-based Offline RL, which makes a lot of sense and is potentially high impact.\n* RAMBO performs reasonably well in the experiments in the benchmark.\n* The paper is clear overall.\n\nWeaknesses:\n* The formalization/theory is not rigorous enough:\n  * The theoretical claims are vaguely implying an overstatement of what they are. Near optimality is intractable in Offline RL [Foster2021,Xiao2022] so lines 48-50 makes claims that are too strong. The PAC bounds of [69] are loose in the general case. Similarly, lines 189-190 are much too strong. The constants are not discussed at all, which prevents the reader to understand the true nature of the offered guarantees.\n  * Equation 1: what does it mean to have 2 policies in your setting? It is laudable to anchor your work into the literature, but I find it very confusing here, while Problem 1 is much clearer right after.\n  * Equation 3: $\\widehat{T}_{MLE}$ has been defined, and the concept of it is not well defined in large/continuous environments. I understand that the authors, once again, try to make connection with the literature, but this is again confusing. I would have expected to see here the TV with respect to the dataset, instead, which we learn much later to be what the practical algorithm is doing (section 5.2).\n  * Theorem 1: $(1-\\gamma)^2$ should be $(1-\\gamma)^{-2}$.\n  * Proposition 2 is wrongly formalized: $s$ is used both in $V_\\phi^\\pi(s)$ and in the sampling in the expectation. I have looked into the proof in Appendix A and found that the same imprecision is in the proof. It should actually be instead $z\\sim d_\\phi^\\pi(s)$ where $d_\\phi^\\pi(s)$ is the state distribution starting from $s$.\n* The empirical validation is limited:\n  * Some simpler problems (multi-arm bandits, gridworld with finite state-action space) might be useful for a better analysis of what it does better than other model-based Offline RL algorithms.\n  * Some harder problems with visual input (Atari for instance) would be useful too as it is where model-based approaches typically fail.\n\n[Foster2021] Offline reinforcement learning: Fundamental barriers for value function approximation, Foster, Dylan J and Krishnamurthy, Akshay and Simchi-Levi, David and Xu, Yunzong (NeurIPS, Offline RL workshop 2021)\n\n[Xiao2022] The Curse of Passive Data Collection in Batch Reinforcement Learning, Xiao, Chenjun and Lee, Ilbin and Dai, Bo and Schuurmans, Dale and Szepesvari, Csaba (AISTATS 2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bFDbYxLQNqP", "reviewer": "Reviewer_dsJ3", "summary": "To remedy with the extrapolation error in offline setting, this paper formulates the offline RL as an adversarial process between the agent and the environment. The environment is modeled as a neural network and trained with both MLE objective and conservative objective which is essentially minimizing the state value of the agent. Previous research [1] proved that this formulation enjoys a PAC performance guarantee, and the experiments do demonstrate some performance improvement on certain D4RL tasks. \n\n[1] Masatoshi Uehara and Wen Sun. Pessimistic Model-Based Offline Reinforcement Learning under Partial Coverage. ICLR 2022.", "questions": "Just one question about COMBO baseline. In the original paper of COMBO, it was not clarified which version of the dataset (v0 or v2) was used in the experiment, so I suppose COMBO used v0 datasets as CQL did. So did you re-implement COMBO and tested it with v2 dataset? \n\nAlso, it is interesting that the performance of COMBO reported in this paper is precisely identical to the original paper, except for walker2d-medium-expert, hopper-medium-replay, hopper-medium and walker2d-medium. So would you elaborate more on how did you obtain the result of COMBO? \n\nPlease correct me if there is any mis-understanding. \n", "limitations": "The authors do mention some limitations of their work at the conclusion section, such as the computational cost. However I don't think this is a major concern. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I enjoyed reading this paper. \n+ This paper is well organized and comprehensive in presentation. The authors manage to convey their idea in concise but informative language.\n+ The main argument that adversarial training can relieve value overestimation and model defect is supported by an illustrative example in the appendix. This is a persuasive argument to justify the effectiveness of the proposed method. \n+ The authors provide the detail about their hyper-parameter configuration and strategy in the appendix. I think the strategy of deciding on RAMBO's hyper-parameters is acceptable, for no excessive online fine-tuning is involved in training stage. \n\nSome weaknesses:\n+ The baselines, no matter model-based ones or model-free ones, seem to be old at this time. For model-free ones, there are [2] and [3]; for model-based algorithms, there is [4] as far as I can recall. \n+ The analysis of RAMBO is too brief. From table 2 I can tell that RAMBO does better on medium-replay datasets while (comparatively) worse on medium-expert datasets, but the authors did not provide in-depth analysis of this phenomenon. More discussions, such as whether this phenomenon is related to adversarial training and how, is expected. \n\n[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. NeurIPS 2021. \n\n[3] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, Ofir Nachum. Offline Reinforcement Learning with Fisher Divergence Critic Regularization. ICML 2021.\n\n[4] Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, Yuhui Shi. Pareto Policy Pool for Model-based Offline Reinforcement Learning. ICLR 2022. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RmrGhq9kxC1", "reviewer": "Reviewer_j6Y8", "summary": "The authors propose RAMBO to offline RL which imposes conservatism by adversarially modifying the transition dynamics of a learned model. RAMBO is built on a formulation for which PAC bounds are guaranteed. The experiments are conducted in the D4RL benchmark and several SOTA baselines are taken for comparison.", "questions": "The article is well-written and soundness to me. I have no further questions.", "limitations": "\nAs mentioned above, I think further comparison should be considered to clarify the advantages/disadvantages/scopes of the two regularizations.If it is done and we can find more insights in the two regularizations after that,  I will consider increasing the score ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n1. The article is well-written and easy to follow.\n2. The solution is simple to implement and sounds reasonable.\n\n\nWeaknesses\n\n1. RAMBO is similar to COMBO: COMBO learns a conservative Q through regularization based on the policy while RAMBO reduces unreasonable V through regularization (Adversarial training) based on the model. The results in Table 1 also show the similarity in performance. It is OK to give a novel regularization from other perspectives. But I think further comparison should be considered to clarify the advantages/disadvantages/scopes of the two regularizations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BahsH0d7H4D", "reviewer": "Reviewer_pkCT", "summary": "This paper proposes RAMBO, an offline model-based algorithm that formulates the optimization of models and policies as a two-player zero-sum game. Specifically, RAMBO trains models with an extra objective that minimizes the value function to achieve conservative policy optimization. The authors provide a PAC performance guarantee and prove that the learned value function lower bounds the true value function. Experiments demonstrate that RAMBO achieves state-of-the-art performance.", "questions": "Please refer to the \"Strengths And Weaknesses\" part.", "limitations": "The authors have addressed parts of the limitations of RAMBO. Nevertheless, I still have concerns about the sensitivity of RAMBO to the hyperparameters.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \n\nThe idea of RAMBO is novel. This paper introduces the idea of RARL into offline model-based RL, which can provide a pessimistic value estimation and alleviate the distribution-shift issue. \n\nQuality: \n\nThe theory part is solid.\n1. The authors provide a PAC bound that guarantees the accuracy of the learned value function.\n2. The author proves that the learned value function is a lower bound of the value function in a real environment.\n\nNevertheless, the authors may want to provide more details on the algorithm and experiment parts.\n1. I can not understand how the gradient is computed according to (5), especially if $s^\\prime$ is a terminal state. \n2. Appendix shows that RAMBO is sensitive to hyperparameters, especially in Hopper and Walker2D. I wonder whether the gradient in (5) has high variance if $s^\\prime$ is a terminal state, and then the optimization of models becomes unstable.\n3. I wonder whether the model training phase in RAMBO leads to high computational cost. The authors may want to compare RAMBO and COMBO in wall-clock time. \n4. I wonder whether RAMBO learns a lower bound of the true value function in authors' empirical studies. If the authors can provide analyses similar to those in TD3 [1] and REDQ [2], the authors' claims would be better supported.\n\nClarity:\n\nThis paper is well written and easy to follow.\n\nSignificance:\n\nThis paper provides a new idea for learning a pessimistic value function. However, the performance improvement is insignificant, as COMPO outperforms RAMBO^OFF as shown in Appendix. \n\n[1] Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//International conference on machine learning. PMLR, 2018: 1587-1596.\n\n[2]Chen X, Wang C, Zhou Z, et al. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model[C]//International Conference on Learning Representations. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept