COMBO: Conservative Offline Model-Based Policy Optimization
Authors: Tianhe Yu
Aviral Kumar
Rafael Rafailov
Aravind Rajeswaran
Sergey Levine
Chelsea Finn
Venue: NeurIPS 2021 Poster (2021)
Keywords: offline reinforcement learning
model-based reinforcement learning
deep reinforcement learning

Abstract: Model-based reinforcement learning (RL) algorithms, which learn a dynamics model from logged experience and perform conservative planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating conservatism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We empirically find that uncertainty estimation is not accurate and leads to poor performance in certain scenarios in offline model-based RL. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that trains a value function using both the offline dataset and data generated using rollouts under the model while also additionally regularizing the value function on out-of-support state-action tuples generated via model rollouts. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. Theoretically, we show that COMBO satisfies a policy improvement guarantee in the offline setting. Through extensive experiments, we find that COMBO attains greater performance compared to prior offline RL on problems that demand generalization to related but previously unseen tasks, and also consistently matches or outperforms prior offline RL methods on widely studied offline RL benchmarks, including image-based tasks.

COMBO: Conservative Offline Model-Based Policy Optimization
Tianhe Yu* Aviral Kumar* Rafael Rafailov', Aravind Rajeswaran',
Sergey Levine", Chelsea Finn'
Sttanford University, UC Berkeley, Facebook AI Research ("Equal Contribution)
tianheyu8cs wtw.eicds.saaoorr...oooooooo edu, aviralk@berkeley,cedk
Abstract
Model-based reinforcement learning (RL) algorithms, which learn dynamics model from logged experience and perform conservative planning under the learned model, have emerged as promising paradigm for offline reinforcement learning (offline RI.). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating conservatism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We empirically find that uncertainty estimation is not accurate and leads to poor performance in certain scenarios in offline model-based RL. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that trains a value function using both the offline dataset and data generated using rollouts under the model while also additionally regularizing the value function on out-of- state-action tuples generated via model rollouts. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. Theoretically, we show that COMBO satisfies policy improvement guarantee in the offline setting. Through extensive experiments, we find that COMBO attains greater performance compared to prior offline RL on problems that demand generalization to related but previously unseen tasks, and also consistently matches or outperforms prior offline RL methods on widely studied offline RL benchmarks, including image-based tasks.
Introduction
Offline reinforcement learning (offline RL) [30, 34] refers to the setting where policies are trained using static, previously collected datasets. This presents an attractive paradigm for data reuse and safe policy learning in many applications, such as healthcare [62]. autonomous driving [65], roboties [25, 48), and personalized recommendation systems [59]. Recent studies have observed that RL algorithms originally developed for the online or interactive paradigm perform poorly in the offline case [14, 28, 26]. This primarily attributed to the distribution shift that arises over the course of learning between the offline dataset and the learned policy. Thus, development of algorithms specialized for offline RL is paramount importance benefit from the offline data available in aformentioned applications. In this work, we develop a principled model-based offline RL algorithm that matches or exceeds the performance of prior offline RI. algorithms in benchmark tasks.
A major paradigm for algorithm design in offline RL is to incorporate conservatism or regularization into online RL algorithms. Model-free offline RL algorithms [15, 28, 63, 21, 29, 27] directly incorporate conservatism into the policy or value function training and do not require learning dynamies model. However, model-free algorithms learn only on the states in the offline dataset, which can lead to overly conservative algorithms. In contrast, model- based algorithms [26, 67] learn a pessimistic dynamics model, which in turn induces a conservative estimate the value function. By generating and training on additional synthetic data, model-based algorithms have the potential for
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

broader generalization and solving new tasks using the offline dataset [67]. However, these methods rely on some sort of strong assumption about uncertainty estimation, typically assuming access to a model error oracle that can estimate upper bounds on model error for any state-action tuple. In practice, such methods use more heuristic uncertainty estimation methods, which can be difficult or unreliable for complex datasets or deep network models. It then remains an open question as to whether we can formulate principled model-based offline RL algorithms with concrete theoretical guarantees performance without assuming access to uncertainty model error oracle. In this work, we propose precisely such a method, by eschewing direct uncertainty estimation, which we argue is not necessary for offline RL.
Our main contribution is the development of conservative offline model-based policy opti- mization (COMBO), new model-based algo- rithm for offline RI. COMBO learns dynamics model using the offline dataset. Subsequently, employs actor-critic method where the value function is learned using both the offline dataset as well as synthetically generated data from the model, similar to Dyna [57] and a number of recent methods [20, 67, 7, 48]. However, in con- trast to Dyna, COMBO learns a conservative critic function by penalizing the value function in state-action tuples that are not in the support of the offline dataset, obtained by simulating the learned model. We theoretically show that for any policy, the Q-function learned by COMBO is a lower-bound on the true Q-function. While the approach of optimizing performance lower-bound is similar in spirit to prior model-based algorithms [26, 67]. COMBO crucially does not assume access to a model error or uncertainty oracle. In addition, we show theoretically that the Q-function learned by COMBO is less conservative than model-free counterparts such as CQL [29], and quantify conditions under which the this lower bound is tighter than the one derived in CQL. This is illustrated through an example in Figure 1. Following prior works [31], we show that COMBO enjoys a safe policy improvement guarantee. By interpolating model-free and model-based components, this guarantee can utilize the best of both guarantees in certain cases. Finally, in our experiments, we find that COMBO achieves the best performance on tasks that require --i---idiiittittioon generalization and outperforms previous latent-space offline model-based RL methods image-based robotic manipulation benchmarks. We also test COMBO on commonly studied benchmarks for offline RL and find that COMBO generally performs well on the benchmarks, achieving the highest score in 9 out of 12 MuJoCo domains from the D4RL [12] benchmark suite.
COMDO volae ForESie
walle REORY
Value funceion Visa
Modei perarees Dars Real Data
IIMHI EHAHE HWHI|
Figure I: COMBO learns a conservative value function by utilizing both the offline dataset as well as simu- lated data from the model. Crucially, COMBO does not require uncertainty quantification, and the value fune- tion learned by COMBO is less conservative on the transitions seen in the dataset than CQL. This enables COMBO to steer the agent towards higher value states compared to CQL, which may steer towards more opti- mal states, illustrated in the figure.
States
2 Preliminaries
Markov Decision Processes and Offtine RL. We study RL in the framework of Markov decision pro- cesses (MDPs) specified by the tuple M = (S, A, T, r. Po, 7). S, A denote the state and action spaces. T(s'(s. a) and r(s. a) E R-aaxx: Rmax] represent the dynamics and reward function respectively. Ho(s) denotes the initial state distribution, and E (0, 1) denotes the discount factor. We denote the discounted state visitation distribution of a policy π using d' (s) == (1 - 7) Ei p s/T). where P(s, s/r) ) is the probability of reaching state at time by rolling out in M. Sim- ilarly, we denote the state-action visitation distribution with (s, a) = d% (s)a(s)) The goal of RL. is to learn policy that maximizes the return, or long term cumulative rewards: max, (M, T) == - (r(s, a)].
Offline RL is the setting where we have access only to a fixed dataset D = {(s, a, s')),, which consists of transition tuples from trajectories collected using behavior policy Tp. In other words, the dataset D is sampled from d= (s. a) = a (s)T We define M the empirical MDP induced by the dataset D and d(s, a) as sampled campled-based version of d a). In the offline setting, the goal is to find the best possible policy using the fixed offline dataset.
Model-Free Offline RL. Algorithms. One class of approaches for solving MDPs involves the use of dynamic programming and actor-critic schemes [56, 5], which do not explicitly require the learning
2

---

of a dynamics model. To capture the long term behavior of a policy without a model, we define the action value function as Q* a) == مي== 2 r(s,, Sp = S, ag a]. where future actions are sampled from T--ss) and state transitions happen according to the MDP dynamics. Consider the following Bellman operator: 8" (s.((s a) == r(s, a) yIB (. a' [Q(s', a')], and its
sample based counterpart: QT. a) := r(s,a) QQ(s', a'), associated with single transition (s, a, s') and a' ~ π (-Iss). The action-value function satisfics the Bellman consistency criterion given by B STQ a) (T a) V(s, a). When given an offline dataset D. standard approximate dynamic programming (ADP) and actor-critic methods use this criterion to alternate between policy evaluation [40] and policy improvement. A number of prior works have observed that such direct extension of ADP and actor-critic schemes to offline RL. leads to poor results due to distribution shift over the course of learning and over-estimation bias in the Q function [14, 28, 63]. To address these drawbacks, prior works have proposed a number of modifications aimed towards regularizing the policy or value function (see Section 6). In this work, we primarily focus on CQL [29], which alternates between:
Policy Evaluation: The Q function associated with the current policy π is approximated conserva- tively by repeating the following optimization:
2 arg min (E [Q(s, a)] -i-..a-a-l(- a)))
1 E...s' (aaaar-=[((t==)) ((((s - (s. (1)
where uu-ls) is a wide sampling distribution such as the uniform distribution over action bounds. CQL effectively penalizes the Q function at states in the dataset for actions not observed in the dataset. This enables a conservative estimation of the value function for any policy [29], mitigating the challenges of over-estimation bias and distribution shift.
Policy Improvement: After approximating the Q function as Q", the policy is improved as T f arg maxx Ea (* (s,, a) Actor-critic methods with parameterized policies and Q functions approximate arg max and arg min in above equations with few gradient descent steps.
Model-Based Offline RL Algorithms. the learning of the dynamics function, and using the learned model to aid policy search. Using the given dataset D, a dynamics model T is typically trained using maximum likelihood estimation as: ming E ) [ogg ((s's, a) A reward model f(s, a) can also be learned similarly if it is un-
second class of algorithms for solving MDPs involve
known. Once a model has been learned, we can construct the learned MDP M (S, A. T. f, Po: 7). which has the same state and action spaces, but uses the learned dynamics and reward function. Subsequently, any policy learning or planning algorithm can used to recover the optimal policy in the model as म = arg max, r).
This straightforward approach is known to fail in the offline RL setting, both in theory and practice, due to distribution shift and model-bias [51, 26]. In order to overcome these challenges, offline model-based algorithms like MOReL [26] and MOPO [67] use uncertainty quantification to construct a lower bound for policy performance and optimize this lower bound by assuming a model error oracle u(s, a). By using uncertainty estimation algorithm like bootstrap ensembles [43, 4, 37], we can estimate u(s, a). By constructing and optimizing such a lower bound, offline model-based RL algorithms avoid the aforementioned pitfalls like model-bias and distribution shift. While any RL or planning algorithm can be used learn the optimal policy for M, we focus specifically on MBPO [20, 57] which was used in MOPO. MBPO follows the standard structure of actor-critic algorithms, but in each iteration uses an augmented dataset D U Dmmctel for policy evaluation. Here, D is the offline dataset and D moddll is dataset obtained by simulating the current policy using the learned dynamics model. Specifically, at each iteration, MBPO performs k-step rollouts using î starting from state s E 'D with a particular rollout policy ((a)s), adds the model-ggeneatee data to Dmodet. and optimizes the policy with batch of data sampled from D U moddel where each datapoint in the batch is drawn from D with probability f E [0, 1] and Dmodel with probability 1 - f.
3 Conservative Offline Model-Based Policy Optimization
The principal limitation of prior offline model-based algorithms (discussed in Section 2) is the assumption of having access to model error oracle for uncertainty estimation and strong reliance on heuristics of quantifying the uncertainty. In practice, such heuristics could be challenging for complex datasets or deep neural network models [44]. We argue that uncertainty estimation is not
3

---

Algorithm 1 COMBO: Conservative Model Based Offline Policy Optimization
Require: Offline dataset D, rollout distribution su(-(s), learned dynamics model To, initialized policy and critic
Te and Qv-
1: Train the probabilistic dynamics model Ta r/s, a) = .V (H a), (s) on D.
2: Initialize the replay buffer Dindel f 23.
3: for i 1,2,3, 2,.33..... do
4: Collect model rollouts by sampling from l and To starting from states in D. Add model rollouts to moodel-
5: Conservatively evaluate o by repeatedly solving eq. 2 to obtain Q using samples from 'D U D' -modl--
6: Improve policy under state marginal of dy by solving 3 to obtain x
7: end for
imperative for offline model- based RL and empirically show that uncertainty estimation could be inaccurate in offline RL problems especially when generalization unknown behaviors is required in Section 5.1.1. Our goal is to develop model-basee offline RI algorithm that enables optimizing lower bound on the policy performance, but without requiring uncertainty quantification. We achieve this by extending conservative Q-learning [29], which does not require explicit uncertainty quantification, into the model-based setting. Our algorithm COMBO, summarized in Algorithm 1, alternates between conservative policy evaluation step and policy improvement step, which we outline below.
Conservative Policy Evaluation: Given a policy an offline dataset D, and learned model of the MDP M, the goal in this step is to obtain conservative estimate of Q", To achieve this, we penalize the Q-values evaluated on data drawn from a particular state-action distribution that is more likely to be out-of-support while pushing up the Q-values on state-action pairs that are trustworthy, which is implemented by repeating the following recursion:
@++1 targ min (Fa.a--ots.)] a)] FE.a (Q a))) Q
1 F T (((s,.)- (* a))) (1. (2)
Here, p(s, a) and dy are sampling distributions that we can choose. Model-based algorithms allow ample flexibility for these choices while providing the ability to control the bias introduced by these choices. For a),, we make the following choice: p(s, a) = (s) s). where d' d=,(s)) is the
discounted marginal state distribution when executing in the learned model M. Samples from d (s) can be obtained by rolling out in M. Similarly, dj is an interrpollttio between the offline dataset and synthetic rollouts from the model: (s. a) == f d(s, a) + (1 - f) d (s, where f E [0, 1] is the ratio of the datapoints drawn from the offline dataset defined in Section and y(1 ) is the rollout distribution used with the model, which can be modeled as π or a uniform distribution. To avoid notation clutter, we also denote dj := d'.
Under such choices of and df, we push down (or conservatively estimate) Q-values on state-action tuples from model rollouts and push up Q- Q-values on the real state-action pairs from the offline dataset. When updating Q-values with the Bellman backup, we use a mixture of both the model- -eneratted data and the real data, similar to Dyna [57]. Note that in comparison to CQL and other model-free algorithms, COMBO learns the Q-function over richer set of states beyond the states in the offline dataset. This made possible by performing rollouts under the learned dynamics model, denoted by d" (s, a). We will show in Section 4 that the Q function Iearned by repeating the recursion in Eq. provides a lower bound on the true function, without the need for explicit uncertainty estimation. Furthermore, we will theoretically study the advantages of using synthetic data from the learned model, and characterize the impacts of model bias.
Policy Improvement Using Conservative Critic: After learning conservative critic Q improve the policy as:
n' arg max (xrppan-t((s) (s.
(3)
where p(s) the state marginal of p(s, a). When policies are parameterized with neural networks, we approximate the arg max with a few steps of gradient descent. In addition, entropy regularization can also be used to prevent the policy from becoming degenerate if required [17]. In Section 4.2, we show that the resulting policy is guaranteed to improve over the behavior policy.

---

Practical Implementation Details. Our practical implementation largely follows MOPO, with the key exception that we perform conservative policy evaluation outlined in this section, rather than using incertainty-based reward penalties. Following MOPO, we represent the prohabilistic dynamics model using a neural network, with parameters 0, that produces a Gaussian distribution over the next
state and reward: To(+++1 r|s, a) = N (s at), (s at)). The model is trained via maximum likelihood. For conservative policy evaluation (eq. 2) and policy improvement (eq. 3), augment with states sampled from the offline dataset, which shows more stable improvement in practice. It is relatively common in prior work on model-based offline RL to select various hyperparameters using online policy rollouts [67, 26, 3, 33]. However, we would like to avoid this with our method, since requiring online rollouts tune hyperparameters contradicts the main aim of offline RI., which is to learn entirely from offline data. Therefore, we do not use online rollouts for tuming COMBO, and instead devise an automated rule for tuning important hyperparameters such as 3 and in a fully offline manner. We search over small discrete set of hyperparameters for each task, and use the value of the regularization term Ex.n-p(s.a)) a)]-Eg, aFs..--DQit s)] (shown in Eq. 2) to pick hyperparameters in an entirely offline fashion. We select the hyperparameter setting that achieves the lowest regularization objective, which indicates that the Q-values on unseen preeec-pe state-action tuples are not overestimated. Additional details about the practical implementation and the hyperparameter selection rule are provided in Appendix B. and Appendix B.2 respectively.
4 Theoretical Analysis of COMBO
In this section, we theoretically analyze our method and show that it optimizes a lower-bound on the expected return of the learned policy. This lower bound is close to the actual policy performance (modulo sampling error) when the policy's is state-action marginal distribution is in support of the state-action marginal of the behavior policy and conservatively estimates the performance of a policy otherwise. By optimizing the policy against this lower bound, COMBO guarantees policy improvement beyond the behavior policy. Furthermore, we use these insights to discuss cases when COMBO is less conservative compared to model-free counterparts.
4.1 COMBO Optimizes Lower Bound
We first show that training the Q-function using Eq. 2 produces Q-function such that the expected off-policy policy improvement objective [8] computed using this learned Q-function lower-bounds its actual value. We will reuse notation for d, and d from Sections 2 and 3. Assuming that the Q-function is tabular, the Q-function found by approximate dynamic programming in iteration k. can be obtained by differentiating Eq. 2 with respect to Q* (see App. A for details):
Q (s.a) a) (B* Q5 - 0(s, a) d(s,a)
(4)
(s.aa
Eq. effectively applies a penalty that depends on the three distributions appearing in the COMBO critic training objective (Eq. 2). of which p and dy are free variables that we choose in practice as discussed in Section 3. For given iteration I of Eq. 4, we further define the expected penalty under p(s, a) as:
p(s. a) - d(s,a)
(p, = Eg ((s.a)))
(5)
(s.
Next, we will show that the Q-function learned by COMBO lower-bounds the actual Q-function under the initial state distribution μl and any policy т. We also show that the asymptotic Q-function learned by COMBO lower-bounds the actual Q-function of any policy with high probability for large enough / 2 0, which we include in Appendix A.2. Let M represent the empirical MDP which uses the empirical transition model based on raw data counts. The Bellman backups over the dataset distribution d in Eq. 2 that we analyze is an f-interpolatioo of the backup operator in the empirical
MDP (denoted by 13t,)) and the backup operator under the learned model M (denoted by xx) The empirical backup operator suffers from sampling error, but is unbiased in expectation, whereas the model backup operator induces bias but no sampling error. We assume that all of these backups enjoy concentration properties with concentration coefficient Cr.T,s, dependent on the desired confidence value 8 (details in Appendix A.2). This is standard assumption in literature [31]. Now, we state main results below.
Propesition 4.1. For large enough 3, we have B ( a)] S Ex-po 1Q a)), where (o(s) is the initial state distribution. Furthermore, when e, is small, such in the large
5

---

sample regime, or when the model bias cm is small, a small B is sufficient to guarantee this condition along with an appropriate choice of f.
The proof for Proposition 4.1 can be found in Appendix A.2. Finally, while Kumar et al. [29] also analyze how regularized value function training can provide lower bounds on the value function at each state in the dataset [29] (Proposition 3.1-3.2), our result shows that COMBO is less conservative in that it does not underestimate the value function at every state in the dataset like CQL (Remark 1) and might even overestimate these values. Instead COMBÓ penalizes Q-values at states generated via model rollouts from p(s, a). Note that in general, the required value of 3 may be quite large similar to prior works, which typically utilize a large constant 3, which may be in the form of penalty on regularizer [36, 29] or as constants in theoretically optimal algorithms [23, 49]. While it is challenging to argue that that either COMBO or CQL attains the tightest possible lower-bound on return, in our final result of this section, we discuss a sufficient condition for the COMBO lower-bound to be tighter than CQL.
Proposition 4.2. Assuming previous notation, let CoMBO :=
(()))) (0 a) and
ToL == Ba (g a) denote the average values on the dataset under the Q- functions learned by COMBÓ and COL respectively. Then, ACOMBO z Sou if:
क (aas))
T(als) (aass
Eg (apaaa()
Eg Ls (n0)
I 0.
(+)
Ts(a|s)
Proposition 4.2 indicates that COMBO will be less conservative than CQL when the action prob- abilities under learned policy T(als) and the probabilities under the behavior policy T are closer together on state-action tuples drawn from p(s, a) (i.e., sampled from the model using the policy π (a)s)), than they are on states from the dataset and actions from the policy, (s) (als). COMBO's s objective (Eq. 2) only penalizes Q-values under p(s, a). which, in practice, are expected to primarily consist of out-of-distribution states generated from model rollouts, and does not penalize the Q-value at states drawn from d (s). As a result, the expression (*) is likely to be negative, making COMBO less conservative than CQL.
4.2 Safe Policy Improvement Guarantees
Now that we have shown various aspects of the lower-bound on the Q-function induced by COMBO, we provide policy improvement guarantees for the COMBO algorithm. Formally, Proposition 4.3 discuss safe improvement guarantees over the behavior policy. building on prior work [46, 31, 29].
Proposition 4.3 (S-safe policy improvement), Let oww (als) be the policy obtained by COMBO. Then, if ß is sufficiently large and v(p", - v(p² .f) 2 C for positive constant C, the policy aowr (a)s) is a (-safe policy improvement over TH in the actual MDP M, i.e. [foow, 2 J (TT C. with probability at least 1 - 5, where Ç is given by.
75 11 - 7)2
LAl [ss)
+0 711 1) (1 (1-7)3
o
B Band Roat
vV
Degt (Now, To)
Drv(M,M)
(2) (1 (1-7)
(1)
The complete statement (with constants and terms that grow smaller than quadratic in the horizon) and proof for Proposition 4.3 is provided in Appendix A.4. DeOL denotes a notion of probabilistic distance between policies [29] which we discuss further in Appendix A.4. The expression for Ç in Proposition 4.3 consists of three terms: term (1) captures the decrease in the policy performance due to limited data, and decays as the size of 'D increases. The second term (2) captures the suboptimality induced by the bias in the learned model. Finally, as we show in Appendix A.4, the third term (3) comes from "(p", J v(p"; JJ), which is equivalent to the improvement in policy performance as a result of running COMBO in the empirical and model MDPs. Since the learned model trained on the dataset D with transitions generated from the behavior policy T3. the marginal distribution of (s, a) is expected to be closer to d(s, a) for Ta as compared to the counterpart for the learned policy, p² Thus, the assumption that v(p",, /)- r(p", J] is positive is reasonable, and in such cases, an appropriate (large) choice of B will make term (3) large enough to counteract terms (1) and (2) that reduce policy performance. We discuss this elaborately in Appendix A.4 (Remark 3).

---

Further note that in contrast to Proposition 3.6 in Kumar et al. [29], note that our result indicates the sampling error (term (1)) reduced (multiplied by a fraction f) when a near-accuratt model is used to augment data for training the Q-function, and similarity, it can avoid the bias of model-based methods by relying more on the model-free component. This allows COMBO to attain the best-of-both model-free and bodel--bsed methods, via a suitable choice of the fraction f.
To summarize, through an appropriate choice of f. Proposition 4.3 guarantees safe improvement over the behavior policy without requiring access to an oracle uncertainty estimation algorithm.
Experiments
In our experiments, we aim to answer the follow questions: (1) Can COMBO generalize better than previous offline model-free and model-based approaches in a setting that requires generalization to tasks that are different from what the behavior policy solves? (2) How does COMBO compare with prior work in tasks with high- nigh-dimensional image observations? (3) How does COMBO compare to prior offline model-free and model- methods in standard offline RL benchmarks?
To answer those questions, we compare COMBO several prior methods. In the domains with compact state spaces, we compare with recent model-free algorithms like BEAR [28], BRAC [63], and CQL [29]; as well MOPO [67] and MOReL [26] which are two recent model-based algorithms. In addition, we also compare with an offline version of SAC [17] (denoted as SAC-off), and behavioral cloning (BC). In high-dimensional image-based domains, which we use to answer question (3), we compare to LOMPO [48], which is latent space offline model-based RL method that handles image inputs, latent space MBPO (denoted LMBPO), similar to Janner et al. [20] which uses the model to generate additional synthetic data, the fully offline version of SLAC [32] (denoted SLAC-off), which only uses a variational model for state representation purposes, and CQL from image inputs. To our knowledge, CQL, MOPO, and LOMPO are representative of sfate-of-theeart model-free and model-based offline RL methods. Hence we choose them as comparisons to COMBO. To highlight the distinction between COMBO and naïve combination of CQL and MBPO, we perform such comparison in Table 8 in Appendix C. For more details of our experimental set-up, comparisons, and hyperparameters, see Appendix B.
5.1 Results on tasks that require generalization
To answer ques- tion (1), we use two environments halfcheetah-jump and ant-angle con- structed in Yu et al. 167], which requires the agent to solve
Enviromment
Batch Mean
Batch Max
COMBO
MOPO
MOReL
CQL
(Ours)
hallcheetal--jmmp
-1022.6
1808.6 2311.9
5308.7-1:75.5
4016.6
3228.7
741.1
angle
866.7
243.6
2530.9
2660.3
2473.4
sawyar-door-close
5%
100%
98.0%
65.8%
42.9%
36.7%
Table 1:
and ant-angle and average success rate of savyer-door doo---oorllee close that require out-s--distribution generalization. All results are averaged over 6 random seeds. We include the mcan and max return success rate of episodes in the batch data (under Batch Mean and Batch Max, respectively) for comparison. We also include the 95%-confidence interval for COMBO.
Average returns of hawfcheetah--aa
task that is different from what the behav- ior policy solved. In both environments,
the offline dataset is collected by policies trained with original reward functions of half cheetah and ant, which reward the robots to run as fast as possible. The behavior policies are trained with SAC with IM steps and we take the full replay buffer as the offline dataset. Following Yu et al. [67], we relabel rewards in the offline datasets to reward the halfcheetah to jump as high as possible and the ant to run to the top corner with a 30 degree angle as fast as possible. Following the same manner, we construct a third task sawyer- wwweerrdoooo-liioo
based on the environment in Yu et al. [66], Rafailov et al. [48]. In this task, we collect the offline data with SAC policies trained with sparse reward function that only gives reward of 1 when the door is opened by the sawyer robot and O otherwise. The offline dataset similar to the "medium-expert" dataset in the D4RL benchmark since we mix equal amounts of data collected by a fully-trained SAC policy and a partially-trained SAC policy. We relabel the reward such that it is when the door is closed and 0 otherwise. Therefore, these datasets, the offline RL. methods must generalize beyond behaviors in the offline data in order to learn the intended behaviors. We visualize the sawyer-door-clon environment in the right image in Figure 3 in Appendix B.4.
7

---

We present the results on the three tasks in Table 1. COMBO significantly outperforms MOPO, MOReL and CQL, two representative model-based methods and one representative model-free methods respectively, in the half jump and saxyer-door-cloo an approximately 8%, 4% and 12% improvement over MOPO, MOReL and CQL respectively on the ant-angle task. These results validate that COMBO achieves better generalization results in practice by behaving less conservatively than prior model-free offline methods (compare to CQL, which doesn't It improve much), and does so more robustly than prior model-based offline methods (compare to MOReL and MOPO).
tasks, and achieves
5.1.1 Empirical analysis on uncertainty estimation in offline model- RL
To further understand why
Max Var UQ v.s. Model Error on haltcheetah-jump Fitted line
Max Var UQ v.s. Model Error on ant-angle
COMBO outperforms prior model-based methods in tasks that require generalization, we argue that one of the main reasons could be that uncertainty
Filted line
Modrl Enror
Node EFTO
estimation is hard in these tasks where the agent is required to go further away from the data distribution. To test this intuition, we perform empirical evaluations to study whether uncertainty quantification with deep neural networks, especially in the setting of dynamics model learning. is challenging and could cause problems with uncertainty-based model-based offline RI. methods such as MOReL [26] and MOPO [67]. In our evaluations, we consider maximum learned variance over the ensemble (denoted as Max Var) max;=1.....N .N (s. a)IlF (used in MOPO).
Var
Max: Vie
Figure 2:
We visualize the fitted linear regression line between the model error and two uncertainty quantification methods maximum learned vari- ance over the ensemble (denoted as Max Var) on two tasks that test the generalization abilities of offfine RL algorithms (half cheethhh jump and ant- angleg,le) We show that Max Var struggles to predict the true model error. Such visualizations indicates that uncortainty quantification is challenging with deep neural networks and could lead to poor perfor- mance in model-based offline RL in settings where out-of-distribution generalization is needed. In the meantime, COMBO addresses this issue by removing the burden of performing uncertainty quantification.
We consider two tasks half caefchhhhhhuuup and ant ant-angle. We normalize both the model error and the uncertainty estimates to be within scale J0. 1] and performs linear regression that learns the mapping between the uncertainty estimates and the true model error. As shown in Figure 2, on both tasks, Max Var is unable to accurately predict the true model error, suggesting that uncertainty estimation used by offline model-based methods is not accurate and might be the major factor that results in its poor performance. Meanwhile, COMBO circumvents challenging uncertainty quantification problem and achieves better performances on those tasks, indicating the effectiveness and the robustness of the method.
5.2 Results on image-based tasks To answer question (2), we evaluate COMBO on two image-based environments: the stan- dard walker (walker-walk) task from the the DeepMind Control suite [61] and a visual
door opening environment with Sawyer robotic arm (sawyer- as used in Section 5.1. For the walker task we construct 4 datasets: medium-replay (M-R), medium (M), medium-expert (M- E), and expert, similar to Fu et al. [12], each consisting of 200 tra- jectories. For sawyer-door task we use only the medium-expee and the expert datasets, due the sparse reward the agent is rewarded only when it success- fully opens the door. Both en- vironments are visulized Fig- ure 3 in Appendix B.4. To extend COMBO to the image-based setting, we follow Rafailov et al. [48] and train a recurrent variational model using the offline data and use train COMBO in the latent space of this model. We present
Dataset
Environment
COMBO
LOMPO
LMBPO
SLAC -ofr
CQL
(Ours)
M-R M M-E
walker_ walk
69.2
66.9
59.8
45.1
15.6
walker walk
57.7
60.2
61.7
41.5
38.9
walker_wald walk
76.4
78.9
47.3
34.9
36.3
expert
walker walk
61.1
55.6
13.2
12.6
43.3
M-E expert
sawyer-door
100.0%
100.0%
0.0%
0.0%
0.0%
sawyer-door
96.7%
0.0%
0.0%
0.0%
0.0%
Table 2: Results for vision experiments. For the Walker task each number the normalized score proposed in [12] of the policy the last iteration of training, averaged over 3 random soeds. For the Sawyer task, we report success rates over the last 100 evaluation runs of training. For the dataset, M refers to medium, M-R refers to medium-replay, and M-E refers medium expert.
8

---

Dataset type
Enviroument
BC
COMBO
MОPO
MOReL
CQL
SAC-off
BEAR
BRAC-P
BRAC-Y
(Ours)
random
halfcheetah
2.1
388++3.7
35.4
25.6
35.4
30.5
25.1
24.1
31.2
random
hopper
1.6
17.9+14
11.7
53.6
10.8
11.3
11.4
11.0
12.2
random
walker2d
9.8
7.0:33.6
13.6
37,3
7.0
4.1
7.3
-0.2
1.9
modium
halfchoctah
36
54.2+1.5
42.3
42.1
44.4
-4.3
41.7
43.8
46,3
medium
hopper
29.0
97.2+12.22
28.0
95.4
86.6
0.8
52.1
32.7
31.1
medium
walker2d
6.6
81.9.12.88
17.8
77.8
74.5
0.9
59.1
77.5
81.1
reeplayrepplay
halfcheetah
38.4
55.1+:1.0
53.1
40.2
46.2
-2.4
38.6
45.4
47.7
medium-roplay
hopper
11.8
89.5+1..8
67.5
93.6
48.6
3,5
33.7
0.6
0.6
medium-roplay
walker2d
11.3
56.01186
39.0
-0.3
med-expert
halfchectah
35.8
90.0+5.6
63.3
53.3
62.4
1.8
53.4
44.2
41.9
med-expert hopper
111.1129
23.7
med expert
walker2d
6.4
103.3
44.6
95.6
98.7
-0.1
40.1
76.9
81.6
Table 3: Results for D4RL datasets. Each number is the normalized score proposed in [12] of the policy at the last iteration of training, averaged over 6 random seeds. We take results of MOPO, MOReL and CQL from their original papers and results of other model-free methods from [12]. We include the performance behavior cloning (BC) for comparison. We include the 95%--confidenc interval for COMBO. across all methods.
bold the highest score
results in Table 2. On the walker-walk task, COMBO performs in line with LOMPO and previous methods. On the more challenging Sawyer task, COMBO matches LOMPO and achieves 100% success rate on the medium dataset, and substantially outperforms all other methods on the narrow expert dataset, achieving average success rate of 96.7%, when all other model-bases and model-free methods fail.
5.3 Results on the D4RL tasks
Finally, to answer the question (3), we evaluate COMBO on the OpenAl Gym [6] domains in the D4RL benchmark [12]. which contains three environments (halfchectah, hopper, and walker2d) and four dataset types (random, medium, medium-replay, and medium-cxperr, We include the results in Table 3. The numbers of BC, SAC-off, BEAR, BRAC-P and BRAC-v taken from the D4RL paper, while the results for MOPO, MOReL and CQL are based on their respective papers [67, 29]. COMBO achieves the best performance in out of 12 settings and comparable result in out of the remaining settings (hopper medium-replay). As noted by Yu et al. [67] and Rafailov et al. [48], model-based offline methods are generally more performant on datasets that are collected by wide range of policies and have diverse state-action distributions (random, medium-reppay datasets) while model-free approaches do better on datasets with narrow distributions (medium, medium-expert datasets). However, in these results, COMBO generally performs well across dataset types compared existing model-frec and model-basee approaches, suggesting that COMBO is robust to different dataset types.
Related Work
Offline RL [10. 50, 30, 34] is the task of learning policies from static dataset of past interactions with the environment. has found applications domains including robotic manipulation [25, 38, 48, 54], NLP [21, 22] and healthcare [52, 62]. Similar to interactive RL, both model-free and model-base algorithms have been studied for offline RL, with explicit or implicit regularization of the learning algorithm playing a major role.
Model-free offline RL. Prior model-free offline RL algorithms have been designed to regularize the learned policy to be "close" to the behavioral policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41J, offline actor-critic methods [53, 45, 27, 16, 64), applying uncertainty quantification to the predictions of the Q-values [2, 28, 63, 34], and learning conservative Q-values [29, 55] or explicitly measured by direct state or action constraints [14, 36), KL divergence [21, 63, 69], Wasserstein distance, MMD [28] and auxiliary imitation loss [13]. Different from these works, COMBO uses both the offline dataset as well as model gonee--geerated data.
Model-based offline RL. Model-based offline RL methods [11, 9, 24, 26, 67, 39, 3, 60, 48, 33, 68] provide an alternative approach to policy learning that involves the learning of dynamics model using techniques from supervised learning and generative modeling. Such methods however rely either on uncertainty quantification of the learned dynamics model which can be difficult for deep network models [44], or on directly constraining the policy towards the behavioral policy similar to model-free algorithms [39]. contrast, COMBO conservatively estimates the value function by penalizing it in out-or- states generated through model rollouts. This allows COMBO to

---

retain all benefits of model model-based algorithms such as broad generalization, without the constraints of explicit policy regularization or uncertainty quantification.
7 Conclusion
In the paper, we present conservative offline model-based policy optimization (COMBO), a model- based offline RL algorithm that penalizes the Q-values evaluated on out-of-support state-action pairs. In particular, COMBO removes the need of uncertainty quantification as widely used in previous model-based offline RL works [26, 67], which can be challenging and unreliable with deep neural networks [44]. Theoretically, we show that COMBO achieves less conservative Q values compared to prior model-free offline RL methods [29] and guarantees safe policy improvement. In our empirical study, COMBO achieves the best generalization performances in 3 tasks that require adaptation to unseen behaviors. Moreover, COMBO is able scale to vision-based tasks and outperforms or obtain comparable results vision-based locomotion and robotic manipulation tasks. Finlly, on standard D4RL benchmark, COMBO generally performs well across dataset types compared to prior methods Despite the advantages of COMBO, there are few challenges left such as the lack of an offline hyperparameter selection scheme that yield uniform hyperparameter across different datasets and an automatically selected f conditioned on the model error. We leave them for future work.
A Acknowledgments and Disclosure of Funding
We thank members of RAIL and IRIS for their support and feedback. This work was supported in part by ONR grants N00014-20-1-266 and N00014-214 1-001--1-11-685 as well as Intel Corporation. AK and SL are supported by the DARPA Assured Autonomy program. AR was supported by the J.P. Morgan PhD Fellowship in Al.
References
[1] Alckh Agarwal, Nan Jiang, and Sham M Kakade. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep. 2019.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[3] Arthur Argenson and Gabriel Dulac-Arold. Model-based offline planning. arXiv preprint arXiv:2008.05556, 2020.
[4] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep nenttworks. In ITA, pages 1-9. IEEE, 2018
[5] Dimitri P. Bertsekas and John N. Tsitsiklis. Denro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0155 2016.
[7] Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-augmentt actor-critic: Backpropagating through paths. arXiv preprint arXiv:2005. 08068, 2020.
[8] Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic, arXiv preprint arXiv:1205. 4839, 2012.
[9] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv: 081..881..00.... 2018.
[10] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[11] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEFE International Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
10

---

[12] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[13] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprini urXiv:2106.06860 2021.
[14] Scott Fujimoto, David Meger, and Doina Precup. Off- Off-policy deep reinforcement learning without w exploration. arXiv preprint arXiv: /812.02900 2018.
[15] Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error actor-critic methods. arXiv preprint arXiv: 09300009977, 2018.
[16] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
[18] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. International conference on machine learning. In International Conference on Machine Learning, 2019.
[19] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 1 11(4), 2010.
[20] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pages 12498-12509 2019.
[21] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rXiv:1907.00455 2019.
[22] Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.0566 2020.
[23] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rI? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
[24] Gregory Kahn, Adam Villaflor, Pieter Abbeel, and Sergey Levine. Composable action- conditioned predictors: Flexible off-policy learning for robot navigation. In Conference on Robot Learning, pages 806-816. PMLR, 2018
[25] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et Scalable deep reinforcement learning for vision- based robotic manipulation. In Conference on Robot Learning.
pages 651-673. PMLR, 2018
[26] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.0595/ 2020.
[27] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-57775 PMLR, 2021.
[28] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019
[29] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, arXiv preprint arXiv:2006.004777 2020.

---

[30] Sascha Lange. Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforcement Learning, volume 12. Springer, 2012.
[31] Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference Machine Leurning, pages 3652-3661. PMLR, 2019.
[32] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor- acttr-eritic: Deep reinforcement learning with latent variable model. In Advances in Neural Information Processing Systems, 2020.
[33] Byung-Jun Lee, Jongmin Lee, and Kee-Eung Kim. Representation balancing offline model- based reinforcement learning. In International Conference on Learning Representations, 2021. URL
_Ri2Y.
[34] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[35] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. CoRR, abs/1904.00$ 1904444444,7, 2019.
[36] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforcement learning without great exploration. arXiv preprint arXiv:2007.08202 2020.
[37] Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. In International Conference on Learning Representations (ICLR), 2019.
[38] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit rorement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[39] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efffccient reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.003677 2020.
[40] Rémi Munos and Csaba Szepesvari. Finite-time bounds for fitted value iteration. J. Mach. Learn. Res., 9:815-857, 2008
[41] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: 2019.
[42] Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International Conference Machine Learning, pages 2701-2710. PMLR, 2017.
[43] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep rein- forcement learning. CoRR, 0es/1806.03335, 2018.
[44] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncer- tainty? evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv: 1e0..99666.200 02530, 2019.
[45] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
[46] Marek Petrik, Yinlam Chow, and Mohammad Ghavamzadeh. Safe policy improvement by minimizing robust baseline regret. arXiv preprint / 167::2160..03842, 2016.
[47] Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-difference learning with function approximation. In ICML, pages 417-424, 2001.
12

---

[48] Rafael Rafailov, Tianhe Yu, A. Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. ArXiv, abs/2012.11547, 2020.
[49] Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. arXiv preprint arXiv:2103. 12021, 2021.
[50] Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural reinforcement learning method. In European Conference on Machine Learning, pages 317-328. Springer, 2005.
[51] Stephane Ross and Drew Bagnell. Agnostic system identification for model-based reinforcement learning. In ICMI., 2012.
[52] Susan M Shortreed, Eric Laber, Daniel Lizotte, T Scott Stroup, Joelle Pineau, and Susan A Murphy. Informing sequential clinical decision-making through reinforcement learning: an empirical study. Machine learning, 84(1-2):109-1) 2011.
[53] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmalcki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.0833 2020.
[54] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint 1XXv:20000.145000 2020.
[55] Samarth Sinha and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning. arXiv preprint arXiv:2103.06326 2021.
[56] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
[57] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
[58] Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off- off-policy temporal -iffrrre-iffeeeeee learning. The Journal of Machine Learning Research, (7(1):2603-261,,222 2016.
[59] Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. J. Mach. Learn. Res, 16:1711-1755, 2015.
[60] Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. arXiv preprint arXiv:2008.05533. 2020.
[61] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: /801.00600 2018
[62] L. Wang, Wei Zhang, Xiaofeng He, and H. Zha. Supervised reinforeement learning with recurrent neural network for dynamic treatment recommendation. Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, 2018
[63] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:1911.11611, 2019.
[64] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.081 00000.08440 40, 2021.
[65] Yu, H. Chen, X. Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. 2020 IFFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2633-2642, 2020.
13

---

[66] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1 1000. PMLR, 2020.
[67] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offine policy optimization. arXiv preprint arXIv:200S.13239, 2020.
[68] Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. arXiv preprini arXiv:2105.0735/, 2021.
[69] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning, arXiv preprint arXiv:2011.07213, 2020.
14


Reviews:
[{"review_id": "xv-UDihhJp", "reviewer": "Reviewer_4HWY", "summary": "This work investigates model-based reinforcement learning with offline dataset. In particular, most of the work on mbrl with offline datasets requires uncertainty estimation to determine the out-of-distribution states and actions. This work eliminates the need to have uncertainty estimation by using the existing Conservative Q-Learning(CQL) algorithm for states and actions from the dataset as well as the rollouts generated from the dynamics using the policy being learned. They show that inclusion of new dataset from the generated rollouts and by being pessimistic about their values, they tend to be better than existing Model-based offline Reinforcement Learning which requires uncertainty estimation", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality and Significance:  Uncertainty estimation for model based offline reinforcement learning has been required for determining out of distribution state and actions. This is done either using an oracle estimator or by learning an ensemble of models(dynamics) and measuring the divergence between the states generated by these ensembles. This work eliminates the need to have this uncertainty metric by simply minimizing the value estimates for all states and actions out of the given dataset; as done in the CQL algorithm. They supplement CQL by addition of new data generated using the model(dynamics) and they treat the  states and actions in the generated rollout as out of distribution. \n\nQuality:  They support their approach via a theoretical analysis to indicate the learned critic estimates would be lower-bounded than the actual values given by the bellman operator with the oracle transition model. Irrespective of inaccuracies in the learned model which can give overestimated reward to states generated in rollouts, they simply depend on the \\beta in eq. 2 for suppression of Q-values. I am wondering, how do they choose this value in practice?  And, did this choice affected their results for random-hopper? And medium-reply hopper in table 3.  \n\nThey experimented with both image-based environments as well as the d4rl dataset. It was not clear to me if they were using latent dynamics or observation-based dynamics for the image-based experiments.\n\nClarity: Yes, the ideas are conveyed in a clear manner. And, the given equations are enough for implementation; given the proposed method would only require a minimal change in existing open-source implementations of the baseline algorithms.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors talk about limitations of their work in terms of potential miss-reward specification which could be harmful for real-world. This especially arises due to learning of parametric dynamics models", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "vILxVW5otkG", "reviewer": "Reviewer_CcvH", "summary": "This paper proposed a new model-based offline RL algorithm, COMBO. The algorithm applies model-based learning to be able to generate random rollouts, and the generated data is combined with the given fixed dataset for value function training. The paper shows by using the generated rollouts as data for Conservative Q-learning provides better constraint and better empirical performance compared to the orignal CQL algorithm. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed algorithm COMBO, which is a combination of model-based RL and offline RL algorithm CQL. Instead of using random generation to explore (s, a) space for Q value constraining, the algorithm used model-based learning to learn the environment dynamics and then applied auxiliary policy to generate random rollouts for the use of constraining Q value. Theoretical proof is provided to show that the new Q update is conservative and the policy improvement is guaranteed. There are no new ideas proposed, but combining these two and making the algorithm work in most scenarios are precious work. Extensvie experiments are done to show COMBO performs well in generalization tasks, image-based tasks and OpenAI Gym tasks. \n\nThe papar is well written and well structured. After reading, I have the following questions for the authors: \n1. Do you have a ablation study of using which kind of rollout policy \\mu(a|s) to generate addtional data points is the best for improving the performance? \n\n2. Experiment details: for D4RL experiments, do you use the datasets in version 'v0' or 'v2' for performance comparison? \n\n3. It seems that the hyperparameter tuning is based on each single task, which is not a standard way for performance comparison in RL? Are the table of results shown in the paper based on the same set of hyperparameters or several set of hyperparameters? There is no validation dataset for RL task. If you are tunning hyperparameters on real rollouts, that is a really dangerous thing in practice, for example in medical and financial areas. \n\nMinor:\n\nIn proposition 4.2, the \\delta_{COMBO} has the same (s, a) input distribution as \\delta_{CQL}, which I think it should be different based on your notations in the paper. \n\nIn proposition 4.1, Equation (*), I think the subscript \\pi(a|s) is not needed in the second term. \n\nLine 68: under which the this?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes! ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "oGxKOhRasrc", "reviewer": "Reviewer_nBvp", "summary": "The paper presents a new approach for model-based offline RL. Here, the model is not used to estimate the return directly using roll-outs, but to generate synthetic data that is then used to train the Q-function. In a way, the method is a combination of CQL and MOPO. What is new is the idea of using this synthetic data to, penalize extrapolation instead of using uncertainty estimation.\nThis modification performs much better on the selected benchmarks.\nAs benchmarks a selection of D4RL benchmarks is used (3 benchmarks) and two vision based benchmarks.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\\\nVery good.\nOn the one hand, it is a combination of existing methods (CQL and MOPO), but on the other hand, the idea of using the generated synthetic data to penalize the extrapolation seems to me to be a significant idea. Simple, elegant, but seemingly functional.\n\nQuality:\\\nHigh quality.\n\nClarity:\\\nVery clear.\n\nSignificance:\\\nI think the idea to use the synthetic data as means for behavior regularization might be very influential.\n\nFurther remarks:\\\nThe results would be even more convincing if the experiments had been performed with more than just three trials.\n\n**Update after feedback:** \\\nBased on the feedback, discussion, and very competent responses and explanations from the authors, I have raised my score to 8.\n\nThe present results show statistically significantly that COMBO performs best in the experiments conducted.\nA simple t-test (student-t with 5 degrees of freedom) or a rank test clearly show this.\n\nFurthermore, I think it is clear that the algorithm is not plagiarism, but a new development, even if it is essentially a combination of two known methods.\nMy opinion is that it is absolutely common to create something new by combining what is known. The fact that this combined algorithm works recognizably better is an interesting finding worth presenting to the specialized public.\n\nRegarding the treatment of hyperparameters, I want to state my conviction from many years of practice that on the one hand the best comparison between different methods is the one where each method is near 1) its respective optimum in the hyperparameter space (otherwise one compares the suitability of the hyperparameters rather than the methods) and that in the applications of offline RL the data efficiency and performance were always in the foreground and not the effort for finding suitable hyperparameters. In short, if it is POSSIBLE, it will be done. On this topic, I find the authors' response remarkably thoughtful and coherent.\n\n1\\) by spending the same effort to come near the respective optimum in hyper-parameter space for each method, if the difficulty of finding the optimum differs significantly between the different methods\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "J8K4K2PAIxj", "reviewer": "Reviewer_GPHd", "summary": "This paper introduces a Conservative Offline Model-Based policy Optimization RL algorithm, called COMBO, that learns a pessimistic model by enabling a lower bound optimization on the policy performance without requiring uncertainty quantification. COMBO employs an actor-critic method to learn the value function on both offline and synthetic datasets. The idea is to penalize Q-values learned from state-action pairs that are out-of-support of the offline dataset. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf {Overview}$:\n\nThis paper proposes a new technique for conservative offline model-based policy optimization. Theoretical guarantees are provided to demonstrate that: 1) COMBO achieves less conservative Q-values compared to prior model-free offline RL methods, 2) COMBO's learned Q-function is the lower bound of the true Q-function that eliminates the need to access a model error or uncertainty oracle in model-based RL, and 3) it guarantees safe policy improvement. Empirical results show the outstanding generalization performance of COMBO and its adaptability to unseen behaviors. Moreover, it can be applied in vision-based tasks and achieve comparable results. \n\n$\\textbf {Major concerns}$:\n\n- Despite the outstanding performance of the COMBO in most of the standard benchmarks compared to the state-of-the-art model-based and model-free algorithms, and the theoretical guarantees discussed in the paper, COMBO is still dependent on the sub-optimality of the data (offline + synthetic). This issue can also be inferred from the occasional inferiority of COMBO comparing to model-based methods when dealing with more diverse state-action distributions. \n\n- To evaluate how well COMBO generalizes to the new tasks, experimental results are averaged over only three random seeds. This can introduce some bias in the final results due to the low number of runs. Lack of insight into the standard deviation values and the inadequate number of independent runs damage the reliability of comparing different methods.\n\n\n$\\textbf {Minor concern}$:\n\n- Regarding the results demonstrated in D4RL tasks, it is not adequately explained how they have normalized return values of COMBO and other methods neither in the paper nor in the appendix. I would like the authors to explain how they have chosen the expert scores for every three environments? How did they fine-tune the trained policy with behavioral cloning [1]?   \n\n- Since source code availability is not mentioned in the paper (but included in supplementary), I would invite the authors to release their code.\n\nOverall, I found the paper is well written, the proposed technique interesting, and the motivation clear. To the best of my knowledge, what is proposed in this article is novel. \n\n[1] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and future directions in the conclusion of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed that this work is promising. The authors also provided important clarifications in the response, including increasing the number of runs and clarifying that hyperparameter selection was actually done with an approach that they developed for this paper suitable for the offline setting. This improvements undoubtedly make the paper stronger. However, the original omission of the strategy to select hyperparameters offline is a large omission, as it is a critical part of the algorithm. This algorithm should be discussed and contrasted to other approaches. 

The theory was not extensively discussed (unfortunately), and relies heavily on the CQL theory. The CQL theory already has some constants that make for relatively loose bounds. Here, there are additional issues with the introduction of beta. The first main result relies on having a potentially very large beta, especially if you look at the proof and see the terms that beta*(small positive constant) is overcoming. The second result, on policy improvement, relies again a potentially large beta and on a term being positive that is not guaranteed to be positive (there is simply a discussion in the paper on why it reasonably could be positive in settings of interest). Theory for offline RL is hard, and so progress should be acknowledged. But, at the same time, given the complexity of these results, it is important to more clearly explain the limitations and what this theory can truly guarantee. 

As a note, the concern about per-environment hyperparameter tuning was resolved in the discussion. It is reasonable to pick hyperparameters per environment, since you have an automated algorithm to do so. This is very different from optimizing (tuning) hyperparameters with sweeps, which is not an algorithm to be used in practice but rather an approach to evaluate methods. 

As it stands, this work is borderline, and would highly benefit from another round to incorporate these changes and be re-reviewed. Some benefit of the doubt can be given that these changes may be done for the final paper, and so if there is space in the program, this paper could be accepted.
Decision: Accept (Poster)