Fine-Tuning Offline Reinforcement Learning with Model-Based Policy Optimization
Authors: Adam Villaflor
John Dolan
Jeff Schneider
Venue:  (2021)
Keywords: Offline Reinforcement Learning
Model-Based Reinforcement Learning
Off-policy Reinforcement Learning
uncertainty estimation

Abstract: In offline reinforcement learning (RL), we attempt to learn a control policy from a fixed dataset of environment interactions. This setting has the potential benefit of allowing us to learn effective policies without needing to collect additional interactive data, which can be expensive or dangerous in real-world systems. However, traditional off-policy RL methods tend to perform poorly in this setting due to the distributional shift between the fixed data set and the learned policy. In particular, they tend to extrapolate optimistically and overestimate the action-values outside of the dataset distribution. Recently, two major avenues have been explored to address this issue. First, behavior-regularized methods that penalize actions that deviate from the demonstrated action distribution. Second, uncertainty-aware model-based (MB) methods that discourage state-actions where the dynamics are uncertain. In this work, we propose an algorithmic framework that consists of two stages. In the first stage, we train a policy using behavior-regularized model-free RL on the offline dataset. Then, a second stage where we fine-tune the policy using our novel Model-Based Behavior-Regularized Policy Optimization (MB2PO) algorithm. We demonstrate that for certain tasks and dataset distributions our conservative model-based fine-tuning can greatly increase performance and allow the agent to generalize and outperform the demonstrated behavior. We evaluate our method on a variety of the Gym-MuJoCo tasks in the D4RL benchmark and demonstrate that our method is competitive and in some cases superior to the state of the art for most of the evaluated tasks.

Under review as a conference paper at ICLR 2021
FINE-TUNING OFFLINE REINFORCEMENT LEARNING WITH MODEL-BASED POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
In offline reinforcement learning (RL), we attempt to lear control policy from a fixed dataset of environment interactions. This setting has the potential benefit of allowing to learn effective policies without needing to collect additional inter- active data, which can be expensive or dangerous real-world systems. However, traditional off-policy RL methods tend to perform poorly in this setting due to the particu-
distributional shift between the fixed data set and the learned policy. lar, they tend to extrapolate optimistically and overestimate the action-values out- side the dataset distribution. Recently, two major avenues have been explored address this issue. First, eehavior-regularize methods that penalize actions that deviate from the demonstrated action distribution. Second, uncertainty-awarc model-based (MB) methods that discourage state-actions where the dynamics are uncertain. In this work, we propose an algorithmic framework that consists of two stages. In the first stage, we train a policy using aahavior-regularizze model-free RL on the offline dataset. Then, a second stage where we fine-tune the policy us- ing our novel Model-Basee algorithm. We demonstrate that for certain tasks and dataset distributions our conservative model-based fine-tuning can greatly increase performance and allow the agent to generalize and outperform the demonstrated behavior. We evaluate our method on a variety of the Gym-MuJodh tasks in the D4RL benchmark and demonstrate that our method is competitive and in some cases superior to the state of the art for most of the evaluated tasks.
Policy Optimization (MB2PO)
INTRODUCTION
Deep reinforcement learning has recently been able to achieve impressive results a variety of video games (Badia et al. 2020) and board games (Schrittwieser et al. 2020). However, it has had limited success in complicated real-world tasks. In contrast, deep supervised learning algorithms have been achieving extraordinary success in scaling
difficult real-worls datasets and tasks, especially in computer vision (Deng et al., 2009) and NLP (Rajpurkar et al.] 2016). The success of supervised learning algorithms can be attributed to the combination of deep neural networks and methods that can effectively scale with large corpora of varied data. The previous successes of deep RL (Levine) 2016 Schrittwieser et 2020) seem to indicate that reinforcement learning can potentially scale with large active data exploration to solve specific tasks. However, the ability to collect such large datasets online seems infeasible in many real-world applications such as automated driving or robot- assisted surgery, due to the difficulty and inherent risks in collecting online exploratory data with an imperfect agent.
Existing off-policy RI. algorithms can potentially leverage large, previously collected datasets, but they often struggle to learn effective policies without collecting their own online exploratory data (ggawaal et al. 2020). These failures are often attributed to the Q-function poorly extrapolating to cuttof-distribution actions, which leads to overly optimistic agents that largely over-estimate the values of unseen actions. Because train Q-functions using bootstrapping, these errors will often compound and lead to divergent Q-functions and unstable policy learning (Kumar et al. 2019).
Recently, there have been variety of offline RL approaches that have attempted to address these issues. Broadly, we group these approaches into two main categories based on how they address the extrapolation issue.

---

Under review as a conference paper at ICLR 2021
The first set of approaches (Wu et al. 2019; Kumar et al. 2019 rely Dehavior-regularizzn to limit the learned policy's 's divergence from the perceived behavioral policy that collected the data. These approaches discourage the agent from considering out-of-distrion -itto---istriutioon actions in order to avoid erroneous extrapolation. While these methods can often be effective when given some amount of expert demonstrations, they often seem too conservative and rarely outperform the best demonstrated behavior.
The second of approaches (Yu et al. 2020: Kidambi al. 2020) leverage uncertainty-aware MB RL to learn a policy that is discouraged from taking state-action transitions where the learned model has low confidence. Thus, these methods allow a certain degree of extrapolation where the models are confident. Because these methods tend to be less restrictive, they can generalize better than Sahavior-regularization this flexibility also seems to make it harder for these methods to recover the expert policy when it is present in the dataset, and reduce their effectiveness when trained with a narrow distribution.
methods and sometimes outperform the behavioral dataset. However,
In this work, we develop algorithmic framework that combines ideas from behavior-regularizztii and uncertainty-awarr model-based learning. Specifically, we first train a policy using behavior- regularized model-free RL. Then, we fine-tune our results with our novel algorithm Model-Based Setavior-Regularizee Policy Optimization (MB2PO). We find that our approach is able to combine the upside of these approaches and achieve competitive superior results most of the Gym- MuloCo (Todorov et al. 2012) tasks in the D4RL (Fu et al. 2020) benchmark.
2 RELATED WORK
While there exist many off-policy RL methods that can learn solve large variety of complex control tasks and can scale with large amounts of online data collection, these methods often perform quite poorly when run completely offline without any online data collection. Recently, there have been several methods that made progress in improving the capabilities of offline RL. For general overview of the field of offline RI., we refer the reader Levie al. (2020). Here we will discuss some recent works that are particularly relevant to our approach.
2.1 IMPROVING OFF-POLICY Q-LEARNING
Many of the recent advances in both discrete and continuous action off-policy deep RL can be attributed to improvements in stabilizing off-policy Q-learning and reducing overestimation due erroneous extrapolation. Some notable methods include target networks (Mnih al., 2013).
double Q-learning (DDQN) (van Hasselt et al., 2015), distributional RL (Bellemare et al., 2017 Dabney al.] (017), and variance reduction through invertible transforms (Pohlen al. 2018 8). In learning for continuous control, Fujimoto et al. (2018) introduced a conservative method that uses the minimum estimate of an ensemble of Q-networks as the target, which is often referred to as clipped double- double-Q-learning Q-learning AAaawal et al.) (2020) demonstrated that Quantile Regression DDQN (Dabney et al. 2017) and other ensemble methods can be effective in certain discrete action offline RL problems. However, Agaawal et al. (2020) showed that when used naively, these methods do not perform well on complex continuous control tasks. In our work, we incorporate the mentioned advances in off off-policy Q-learning into our approach to stabilize performance and prevent potential divergence.
Additionally, the offline RL algorithm Conservative Q-learning (CQL) (Kumar et al. 2020) has attempted to address Q-learning's overestimation issue on offline data directly by including a con- straint term that discourages the agent from valuing an out-of-distribution action more than the demonstrated actions. In our method, instead of using a constraint on the Q-values, we use a combi- nation of behavior-regularizzed model-free RL and uncertainty-awarc model-based RL discourage erroneous extrapolation.
2.2 BEHAVIOR-REGULARIEE REGULARIZED MODEL-FREE RL
A variety of recent offline RL approaches have incorporated constraints penalties the learned policy's 's divergence from the empirical behavioral policy. In particular, recent works have used both KI. Divergence (Wu al. 2019) and mean measure of divergence (MMD) (Kumar et al.) 2019
2

---

Under review as a conference paper at ICLR 2021
MMD is sometimes used over KL Divergence because MMD approximately constrains the learned policy to be in the support of the behavioral policy, which is less restricting than KL Divergence. However, most cehavior-regularizatoon
volicy-constraie methods require the behavioral policy to be represented explicitly in order to estimate these divergences or to enforce their policy constraint [aarcche et al. 2019). In contrast, AWAC (Nair et al.) (020) or CRR (Wang al., 2020) is able to incorporate a KL divergence constraint without explicitly representing the behavioral policy. They do this by reformulating the policy-constraie
RL optimization equations into a form that resembles behavioral cloning re- re-weighted by the exponential of the advantage. Waan et al. (2020) demonstrates that this method can effectively learn complex control tasks purely from offline data, and Nair et al.) (020) demonstrate that performance can even be improved with further online data collection. In this work, we demonstrate that these properties make AWAC work exceptionally well when used for initialization as well as when used for fine-tuning with Model-Based Policy Optimization (MBPO) Janner et al. 2019).
2.3
ANCERTAINTY-AWARE
MODEL-BASER RL
MB RL algorithms have several natural advantages for offline RL compared to model-free RL algo- rithms. First, MB RL algorithms rely on supervised learning, which provide more robust gradient signals compared to bootstrapped learning and policy gradients. Second, learning a dynamics model often provides strong lask-independent supervision, which allows MB RL algorithms to learn from sub-optimal trajectories. These benefits make generalization casier, and can allow MB RL algo- rithms to surpass the performance of the demonstrated data. In fact, in many environments, MB RL methods have already been effective in learning with offline or randomly collected datasets. Ad- ditionally, there is a rich history of prior works that have explored robust solutions to MDPs with uncertain transition dynamics (Nilim & El Ghaoui 2005 [yengar (005). However, it can be dif- ficult to scale these type of methods to high-dimensional continuous control tasks, especially when using deep neural networks. Recently, incorporating uncertainty estimation techniques from super- vised learning in MB RL has demonstrated further improvement in both online (Chua et al. 2018) and offline deep RL. In particular, two recent works, Model-Based Offlin Policy Optimization (MOPO) Yuu et al., (020)) and Model-Based Offline Reinforcement Learning (MoREL) (Kidambi cet al. 2020), have demonstrated impressive results by incorporating uncertainty-aware MB RL with the Dyna (Sutton 1991) style algorithm MBPO (Janner et al.) 2019). Both methods use these mod- els to create conservative MDPs that have a lower potential expected sum of rewards compared to the true MDP. By performing policy optimization in the conservative MDP through MBPO they are able learn conservative policy that can outperform the demonstrated trajectories. However, these methods can often fail to recover the expert policy even though it was demonstrated in the dataset. We believe that this is largely due to a lack of effective methods for estimating epistemic uncertainty for neural network regression.
3 PRELIMINARIES
In RL, we assume our agent operates within a standard Markov decision process (MDP) M = (S, A, T.r.Po, the probabilistic transition dynamics, r is the reward function, Po is the initial state distribution, and 7 (0, 1) is the discount factor. The objective in RL is to learn policy x (als) that optimizes the expected discounted sum of rewards RTT E,,T,po 1E 7"r(st,;;))
Po.?)) 7), where S denotes the state space, A denotes the action space, T(s'ls, a) represents
In offline RI, we assume that during training we only have access to a fixed dataset D3 containing a set of tuples (s, a, r) of environment transitions and associated rewards. We assume that the data was collected by policy Ty. which we call the behavioral policy. Typically, when training with data not collected by your current policy т,, we either use off-policy model-free algorithms or model-based algorithms. The most common off off-policy model-free algorithms are actor-critic algorithms that use policy iteration. Policy iteration involves alternating between policy evaluation and policy improvement order to learn an effective policy. In policy evaluation, these methods train a parametric Q fufunction by iteratively minimizing the temporal difference equation
Q = arg min Exasst [((((8, a) + 7B (1)) (8 - Q (s)
(1)
Q
3

---

Under review as a conference paper at ICLR 2021
In policy improvement, update parametric policy to maximize current Q-function Th++1 = arg max (a ( (s,a))
In MB RL, we attempt to learn a model 7e of the transition dynamics and a model f of the reward function. With this learned model of the dynamics and reward function we can create a model MDP M = (S, A, f, f, 70) to estimate the true underlying MDP M. These methods tend to use either trajectory optimization or policy optimization in the model MDP to produce their policy.
4 MODEL-BASED BEHAVIOR-REGULARTEE POLICY OPTIMIZATION FOR OFFLINE FINE-TUNING
For many offline datasets, it could be much harder to learn effective model of the MDP than to learn a reasonable policy. This especially the case when there is low variability or insufficient coverage of the state and action space in the collected dataset, or in environments with complex observations, like images, or long horizons. To overcome these issues, recent works (Yu et al. 2020 Kidambi et al. 2020) have leveraged uncertainty estimation methods in order to construct conser- vative MDPs that use soft penalties or hard thresholds on model uncertainty to discourage deviating from the confident regions. However, these methods rely on the efficacy of ensemble-based neural network uncertainty estimation methods which currently are not particularly effective at estimat- ing epistemic uncertainty in regression settings. Therefore, we propose Model-Based Behavior- Regularized Policy Optimization (MB2PO). In MB2PO, we likewise use uncertainty-awarc mod- els to perform offline MBPO, but use the jehavior-regularize, model-free algorithm AWAC (also known as CRR-exp) instead of SAC (Haarnoja et al. 2018) for policy optimization.
4.1 CONSERVATIVE MBPO
In this work, we use MOPO (Yu et al. plicity and prior effective results on the D4RL benchmarks. In MOPO, they construct a conservative MDP by augmenting the reward function as follows
2020) as basis for our conservative MBPO, due its sim-
F(s,a) a) = f(s, a) Au(x,a)
where f the learned estimate of the reward and u is the estimated uncertainty for the model transition. Note, that this general formulation for conservative MDP has also been explored in other prior work such as (Ghavamzadeh et al. 2016). Still, we specifically follow MOPO in using the maximum standard deviation across ensemble of probabilistic dynamics models of uncertainty. Therefore, we can decompose our Q-function in this conservative MDP as
measure
Q" = Q (s, XQT (s,
(4)
where QJ represents estimate of the expected discounted sum of rewards in the real MDP and Q represents our expected discounted sum of uncertainty penalties. Now at convergence, if our policy æ deviates from the behavioral policy næ that collected the data, then expect for all states in the conservative MDP that
E[Q"] 2 E[QT*)
Thus, by plugging in our decomposition we get
(6) While in theory, with well-calibrated uncertainty estimates and proper tuning of X, this should lead to only safe policy improvements over the behavioral policy, in practice it seems that MOPO often unable to recover expert-level performance when it provided in the offline dataset. This is unsurprising given that it is hard to generate well-calibrated epistemic uncertainty estimates in regression settings, and there will inevitably be model errors that will lead to overestimated Q- values.
E(Q] (sx, a)) 2 EIQ?? (s, + XE(QT(s,
To address these issues, we use policy constrained model-free RL in MB2PO. policy constrained model-free RL, we attempt to optimize the following policy objective
π = arg max (e
s.t. ( <c
4

---

Under review as a conference paper at ICLR 2021
If we estimate both and Th be roughly univariate Gaussians with similar variances, then the KL constraint becomes an &g constraint on the policy mean. Because we expect our models to be locally accurate around the data, this constraint can help ensure that stay in the effective region of the estimated MDP even if we have poorly calibrated uncertainty estimation. Additionally, ,Janner et al. (2019) demonstrated that the difference between the true expected returns (1z) and the expected returns (x) of MDP induced by an approximate model can be bounded by
(m) 2 J(x) 2 (Em + 2Ex) + 1-7
4rmax
(1-
where mmax is the maximum reward, y is the discount factor, Gn is a bound on the total variation distance (TVD) between the learned model and the true model, and én is a bound on the TVD between æ and Fh on the demonstrated states. By Pinker's inequality, bounding the KL. divergence also bounds the TVD. Therefore, by leveraging policy constraints in the policy optimization in MBPO, we can reduce the gap in expected returns and improve the algorithm's 's robustness to model errors.
4.2 BEHAVIOR- BEHAVIOR-REGULARIZER MODEL-FREE RL WITH AWAC
For performing behavior--egullrizzed policy optimization, we use AWAC (Nair et al. 2020). also known as CRR-exp (Wang et al. 2020) due its impressive results in offline RL and its ability to be fine-tuned with additional online data.
By enforcing the KKT conditions (Peng et al. 2019: Peters Schaal 2007 Gómez et al. 2014
we can derive an analytic solution to Equation 7 where the Lagrangian is
((T, E (-))))))))) 1Q" a)] + ade
DKL. (T
We can substitute A3 (s, for Q" (s,, a) because it does not affect the optimum and get the closed- form solution
1 TT(ass) exp
(A a
(als) Z(s)
where 7(s) is the normalizing partition function. In order to project this solution into our policy space, we update our parameters by minimizing DKL (x..(T"") (TT))))) This leads to the following iterative update
1 0x+1 = arg min E. Ea log (a(a))) 7(s)
A (s, a)
exp
We follow Wang al. 2020 and Peng et al. 2019) and avoid estimating Z()) and instead clamp the exponential term to be at most 20. Additionally, one could adaptively learn a using dual gradient descent, but this would require us to explicitly model the behavioral policy иy. Instead, we use fixed a for all of our results. Additionally, the Q-function is updated off-policy using the Bellman equations as described in Equation 2 and the improvements from section 2.1.
One of the major benefits of using AWAC with fixed is that we can leverage behavior regular- ization in a principled manner without needing to explicitly represent the behavioral policy. This is particularly important in 3 major cases: 1. when there are not enough data to learn the behavioral policy; when the data was collected by a variety of different policies or sources; 3. when the data was collected by a policy outside of your policy class, such as human expert or controller that leverages hidden state information.
Additionally, we can view AWAC as reweighted behavioral cloning algorithm. Unlike SAC (Haarnoja al., 2018) and DDPG (Lillicrap et al., 2015). it does not rely on the reparametriza- tion trick gradients of your learned Q-function to perform policy updates. This allows to use a wider ranger of policy classes, which in this work we take advantage of by using a tanh squashed GMM with 5 components. We suspect that there are also some additional benefits to not depending on the gradients of the learned Q-function, which might be particularly bad in offline settings, but leave further investigation to future work.
An important thing to note with AWAC is that we can influence the implicit behavioral penalty by controlling the source of the data train with. This holds, for example, if we perform series
5

---

Under review as a conference paper at ICLR 2021
of policy updates only using data collected by the previous policy iterate. Then, are implicitly performing a trust-region policy update like TRPO (Schulman et al. 2015) and PPO (Schulman
et al. 2017) of the form
Th+1 = arg max Baw 1QTH (s,
(10)
s.1. DKL (<)s(( S
In fact, if we train on data collected by the last policy iterates, then we are approximately con- straining our policy a weighted sum of the previous policies m 1 E Tk-i and damping our learning process in the policy space.
In our work, train with w [] 1] portion of the data from offline data collected by Th and (1-w) w) portion of the data collected online from the last policy iterates in the conservative MDP defined by our learned models. Therefore, we are approximately optimizing the following objective
(a- (s) (WDr) + -1- (T=
(11)
Therefore, by using AWAC as the policy optimization algorithm in MB2PO, we can easily perform behavior-regulariied policy optimization with soft damped trust region updates in the conservative MDP to reduce the effects of model errors and poor uncertainty estimation.
4.3 MODEL- -BOSED BEHAVIOR-REGULARIZEE REGULARIZED POLICY OPTIMIZATION
Train ma. Qo with AWAC with samples from D3 Train ensemble of N probabilistic dynamics
(St4s) a2) (ugg(s az), 2i (st, a on the data in Dg
for epoch k= 1, 2, do Initialize empty replay buffer D
for 1, 2.., batchsize do Sample state 81 from Dg
for = 1, 2. ..., do
0j ~s))
Uniformly sample 7 from { j
$3-1. ~ Tss a,) 7; : A max |E' ag) ||
Add sample (sg, aj. Fj. 8j=1) D
end end
Draw w portion of the samples from D3 and the rest uniformly from {D to train me and Qo with AWAC
end
We first initialize policy by training with AWAC solely the offline data.
Next for fine-tuning with MB2PO, we train an ensemble of probabilistic dynamics models rep- resented by neural networks that output diagonal Gaussian distribution over the next state and reward: (s r|s,, a,) (Hg 0.), (i, ) We construct conservative MDP that at every time step uses a randomly drawn dynamics model from ( to determine the next state transition. Additionally, incorporate penalty the largest predicted standard devia- tion among the dynamics models as a practical means penalizing both epistemic and aleatoric uncertainty.
Then, we alternate between collecting data with our current policy in the conservative MDP and updating our policy and Q-network using Equation 9 and Equation ( respectively. When collecting data in the conservative MDP, we collect h-length truncated trajectories starting from states in the original offline dataset. By collecting data this way,
able to collect variety of imagined

---

Under review as a conference paper at ICLR 2021
data without relying long model rollouts, which would inevitably lead to compounding errors. When performing training updates, we sample w E [0, [0 of the data from the original dataset and the remaining - uniformly from the last 100 policy iterates. Our full algorithm is outlined in Algorithm
EXPERIMENTS
Task and Dataset
AWAC
AWAC
MOPO
BEAR
BRAC-
CQL
MB2PO(Ours)
(H)
hallcheetah-ramdom
18.8 11.5
25.5
31.9 2.8
25.5
28.1
35.4
hopper-random
11.2 10.1
11.4 10.1
13.3 1 1.6
9.5
12.0
10.8
walker2d-random
1.4 1:000
0,2 1223
13.0 1:
6.7
0.5
7.0
halfchectah- medium
40.9 +0.3
40.7 #0.2
40.2
38.6
45.5
44.4
hopper-medium
35.0 4:3.9
55.7 1146
26.5 + 3.7
47.6
32.3
58.0
walker2d-medium
74.3 41.6
80.4 +0.9
14.0 +
10.1
33.2
81.3
79.2
halfcheetah expert
106.7 #0.6
105.1 #13
108.2
-1.1
104.8
hopper-expen
108.1 4:3.8
105.5 110.5
110.3
109.9
walker2d-expert
100.5 +12.3
107.4 +1.1
106.1
0.0
153.9
halfcheetah- medium expert
104.7 #1.6
104.8 #11
57,9 24.8
51.7
45.3
62.4
hopper-
hopper-medium medium expprtt
75.1 115.9
79.1 13.5
$1.7 = 42.9
111.0
walker2d mediummexxpert
81.8 +18.9
86.2 +33.5
19.1
26.0
66.6
halfcheetah mixed hopper-mixed walker2d-mixed
42.3 10.3
55.6 #0.6
54.0 12.6
36.2
45.9
46.2
30.1 10.9
72.6 225.
92.5 16.3
25.3
0.8
48.6
16.9 +1.7
61.9 =8.2
42.7 +8.3
10.8
0.9
26.7
Table 1: Here compare AWAC (averaged over 4 seeds) and AWAC MB2PO (averaged over seeds) to recent offline model-free and model-bases RL algorithms. We report the normalized score where 100 is the performance of fully trained SAC policy and 0 is the performance of uniform random policy. For the other methods, we report the results from their own papers or the original D4RL paper. "-expert" results for MOPO are omitted here. We include the stand deviation for our results and for previous results if reported. We bold the highest mean.
included in the original paper and thus
In our experiments, aim to address two questions: (1) Is AWAC an effective initialization algo- rithm? (2) Can we further improve performance by fine-tuning with MB2PO?
We evaluate (1) by comparing AWAC to other saate-of-the-arr model-free offline RI. algorithms. In particular, we compare our results to BRAC-v (Wu et al. 2019), BEAR (Kumar et al. 2019), and CQL Kumaa al. 2020) on the Gym-MuJoCo tasks in the D4RL benchmark.
We evaluate (2) by fine-tuning the policy and Q-function, after running AWAC for 500000 gradient steps, with MB2PO. In addition to the model-free offline RL algoirthm above, we also compare these results to MOPO, which to the best our knowledge the state-of-the-art MB offline RL algorithm the Gym-MuJoCo tasks in the D4RI. benchmark. All of hyperparameters for AWAC and MB2PO are given in the appendix.
The Gym-MuJoCC tasks are standard in evaluating modern deep RL algorithms. The goal in these tasks is to learn to travel as far forward as possible within a set horizon on variety of dif- ferent robots. The D4RL benchmark contains a variety of precollected datasets for the halfchee- tah, walker2d, and hopper tasks. For each robot task, there are different provided datasets. The "-random" datasets contain million samples collected from randomly initialized policy. The "-medium" datasets contain 1 million samples collected from a RL policy partially trained to per- formance of approximately 33. The "-expert" datasets contain million samples collected from fully trained RL policy that reaches approximately 100. The "-mixed" datasets contain all the data in the replay buffer from the partially trained "medium" policy. Finally, the "-medium-exxper" datasets combination of the "-medium" and "-expert" datasets. An important thing to note is that all datasets besides the " "-mixed" datasets were collected with only or policies, and thus probably only cover narrow part of the state-action distribution. On the other hand, the "-mixed" dataset contains the data collected by all of the policy iterates during an incomplete RL training run, and thus represents much wider part of the state-action distribution.

---

Under review as a conference paper at ICLR 2021
Results in Table I demonstrate that AWAC its get reasonable results all the datasets and can approach tfae--of-the-ar results on "-expert" and " "-medium "eeitutteeprt""""" datasets. Unlike the other rehavior-regularize
model-free methods, AWAC and CQL are able to get near or fully recover expert-level performance when trained on the "medium-expert- datasets. This indicates that AWAC and CQL are more robust there is less of drop in performance compared other methods when incorporating additional sub-optimal trajectories.
Next, we fine-tune the trained AWAC policies with MB2PO. For each task and dataset, we pretrain an ensemble of 5 probabilistic dynamics models for 100000 gradient steps on the behavioral dataset. We then perform MB2PO for 500 iterations. Each iteration consists of collecting 1000000 steps from h-length truncated trajectories in the conservative MDP, which should run in a few seconds on modern GPU hardware, followed by 1000 gradient steps.
Results in Table D demonstrate that our method is effective in improving the performance over AWAC in 11 of the 15 tasks. particular, we find that MB2PO significantly improves the per- formance on all of the "-mixed" datasets and even achieves state-of-the-an on "walker2d-mixed" by a large margin. These strong results the "-mixed" datasets demonstrate that our model-based fine-tuning method can be especially beneficial when there is sufficient variation in the behavioral dataset. Additionally, the noticeable improvement in some the "-medium" " and " "-medium exppertt"""""" datasets demonstrate that our fine-tuning can be effective even when the data was collected by one or two policies. In the 4 cases where MB2PO fine-tuning degrades performance, is always negligible and never over 3 points.
Our method also outperforms MOPO, the most direct comparison, in out of the 12 comparable tasks. These results demonstrate the benefits of combining behavior-regullarized model-free RL with uncertainty-aware MB RL as we are able to get the best of both worlds. We are able to recover high-level performance when available the dataset like AWAC, and we can still generalize and learn to outperform the best observed trajectory like MOPO.
Finally, our method is quite competitive with CQL as we beat it for 7 of the 15 tasks, and generally our results are quite comparable. However, our method significantly outperforms CQL in all of the "-mixed" datasets. These results indicate that our method might be superior in situations where the dataset was collected by a variety of different policies.
CONCLUSION
We proposed an algorithmic framework that leverages the benefits of both behavior-regularizee model-free methods and uncertainty- model-based methods. We do this by first training an ini- tial policy with the offline model-free AWAC algorithm. Then, we fine-tune with our novel MB2PO algorithm. We perform this by learning uncertainty-aware models that are used to create conser- vative MDP. Then, we continue to use AWAC to further update our policy and Q-function in this conservative MDP. By using AWAC, we are able perform policy optimization while implicitly constraining the learned policy's KI. divergence to the behavioral policy. We demonstrate that this two-stage process allows us to get the best of both worlds between behavior-regularize model-free methods and uncertainty-awer lows us to often recover the best-perforiing behavior in the dataset, and, when possible, MB2PO fine-tuning allow us generalize and outperform the demonstrated behavior.
model-based methods. Specifically, the initial AWAC training al-
We see four important directions of future work in order to extend the effectiveness and applicability of MB2PO: 1. developing a rigorous means of determining for what datasets MB2PO fine- fine-tuning can be effective: 2. improving MB RL and neural network uncertain estimation to increase the number of datasets where MB2PO can be effective; 3. better leveraging behavior-regulariiatoo in the policy optimization or the conservative MDP to improve MB2PO's 's ability to recover expert behavior when available; 4. improving off-policy evaluation (Thomas et al. 2015) for neural network policies in order to facilitate offline hyperparameter tuning.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, 2020.
8

---

Under review as a conference paper at ICLR 2021
Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark, 2020.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. CoRR, abs/ 2017. URL http:
arxiv. http://wxvvvvv..r..oo aba/ wwwpaaawaa...////////////// 00lpb///770008777777
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement
learning in a handful of trials using probabilistic dynamics models. CoRR, abs/ albs/1805.1211 14, 2018.
URL pest:://arxy
org/ abs/1805. 12114
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. CoRR, abs/1710.100444 2017. (RLLapt:///axii orrq/
abs/17100 0044
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. CoRR, abs/1882.009477. 2018. UKL.http.//aarr...or///////////
09477
Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow. Safe policy improvement by mini- mizing robust baseline regret. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 29, pp. 2298-2306. Cur- ran Associates, Inc., 2016. URL httpttttt///poceeetlteee www.ggsneurrp.... cttpsrpp...cccpppp//2200000 cctppppre//1////////// Paperr.
fi
Vicenç Gómez, Hilbert J Kappen, Jan Peters, and Gerhard Neumann. Policy search for path inte- gral control. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 497. Springer, 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, abs/ 111118110222 (1.011990) 290, 2018. URL http:///wwii...oaaa///////////// /1tpg///aaa//111111111
Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005. doi: 10. 10.1287/moor,
1040. 0129
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519- 12530, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel Model- based offline reinforcement learning, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-po q-learning via bootstrapping error reduction. Advances in Neural Information Processing Sys- tems, pp. 11784-10 114--11799 1174, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research pp. 3652-3661, 3661, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://proceediins 1 prwss / roche 9a hhtml

---

Under review as a conference paper at ICLR 2021
Sergey Levine. Deep learning for robots: Learning from large-scale interaction. Blog, Março, 2016.
Google Research
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems, 2020.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2015.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. http::/arxxi..cooo www.xii.wrrg/bbs/131222 5602
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets, 2020.
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):880-75 -988 2005. doi: 10. 10.1287/0pe::105000166 URL
1050. 0216
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, arXiv preprint 11X1v:1910000177, 2019.
Jan Peters and Stefan Schaal, Reinforcement learning by ceward-weightee regression for operational space control. In Proceedings of the 24th international conference Machine learning, pp. 745- 750, 2007.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Vecerík, Matteo Hessel, Rémi Munos, and Olivier Pietquin. Observe and look further: Achieving consistent performance atari. CoRR, abs/1805. 1559,, 2018. URL http::/axxiv..o/// wttp:/iwww.gga///////baaa///////
11593
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint htlv::6666.2220 2016.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model, 2020.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Picter Abbeel. Trust region policy optimization. CoRR, abs/1502.05577, 2015. URL http::t//:// arxiv. wwwpaaiwww.....s////a/.aa
05477
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, sas/1707006447 2017. URL http:://rxxv.orgg///ee
7707.06344
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 2(4):160-163, 1991.
Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off- policy evaluation. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. 2012 IEFF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double 4- learning. CoRR, abs//550900646 2015. URL /tt/at/tpp///xxvvoo////////////0
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg. Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcchre, Nicolas Heess, and Nando de Freitas. Critic regularized regression, 2020.
10

---

Under review as a conference paper at ICLR 2021
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, 2020.
7
APPENDIX
All methods were trained with the Adam optimizer Kingma & Ba 2014)
7.1 AWAC
Our Q-networks and policies were represented with [256, 256, 256, 256] fully connected networks with relu hidden-activatiins.. The policy putted 100 quantiles and was trained with the following Q-learning improvement: Quantile Regres- sion DQN (Dabney al. 2017). Clipped DQN(Fujmmoto et al.] 2018), Double DQN Hasselt et al. 2015), and Invertible Transforms (Pohlen et al. 2018). The advantage was estimated with 10 samples. We ran AWAC for 500000 gradient steps. The rest of the parameters were taken from the original AWAC paper (Nair et al. 2020): a = 1.. batch size 1000, policy weight decay 1.e - 4, policy and Q learning rate 3.e 4. soft target network update 5.e 3.e-3.
5-head tanh squashed GMM. The Q-newwork out-
7.2
MB RL
Our probabilistic dynamics models were represented with [200, 200, 200, 200] fully connected networks with swish hidden-activations. The models outputted mean and variance for every state variable and the reward. We trained the models for 100000 gradient steps. The rest of the parameters are: batch size 256, weight decay 1.c 4, learning rate 1.e 3.
7.3 MB2PO
MB2PO used the same AWAC parameters as before, but changed according the table below. We ran MB2PO for 500 iterations where each iteration consisted of collecting 1000000 samples from truncated h-length trajectories with the current GMM policy, then training for 1000 steps with of the data from real dataset and 1 of the data from imagined model rollouts from the last 100 iterations. We picked the better of a = 1, w (0.01, 0.05) for the mixed datasets, and the better of {1,2}. 0.8 for the other datasets. We used h for several of the halfcheetah datasets and h = for the rest because any larger tended to cause the Q-values to explode.
Task and Dataset
alpha
omega
lambda
halfcheetab-random hopper-randont walker2d-random
1.
0.8
5
1.
0.8
halfcheetah-mmdiimm
I.
0.8
5
hopper-medium
1.
0.8
walker2d-medium
0.8
halfcheetah-expert
hopper-expert
0.8
walker2d-expert
0.8
sefffcheetah-meddinn expert
0.8
hopper-medium medium medium-expert expert
walker2d-medium
0.8
halfchectah-minn
0.01
5
hopper mixed
0.01
walker2d-mixed
0.01
11


Reviews:
[{"review_id": "kY_l2uOooiH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper describes a novel method to combine two different school of thoughts for improving offline reinforcement learning. The model free part relies on keeping policy changes local and model based part uses model uncertainties to ‘fine tune’ to obtain final offline RL policy.\n\n\n### Strong Points\n\n1. Paper is well written. This is impressive as both the methods on which the paper relies on are recent additions to the literature and as such I wasnot fully aware of the techniques described therein. However, despite this the paper was easy to follow and a pleasant read.\n2. I would also like to thank authors for honestly describing the scenarios in which fine tuning led to degradation in the performance of the proposed method\n\n\n### What can be improved\n\n1. The paper relies on AWAC method which is itself novel and not peer reviewed.\n   With benefit of doubt to authors I assume that the AWAC  represents excellent\n   scholarship. This though raises another question as to how much contribution\n   the results from have on their own. There maybe excellent points the authors\n   use to improve upon AWC results, however with the time budget allocated to\n   this review it is impossible to read and understand AWAC and then objectively\n   judge the improvements bought about by this scholarship. \n2. Again, I would like to commend authors for the future work directions\n   mentioned in the conclusion. Unfortunately, these are the very same questions\n   I would have hoped this paper addresses. For example, one of the classic use\n   cases for offline RL would be that we are not aware of the situation, whether\n   the data is ‘expert’ policy or ‘naive’ policy, so it is very challenging to\n   decide whether or not to fine tune the results one obtains from AWAC for\n   example. \n3. One Page 6 penultimate paragraph: Authors say “ ... , and we are confident\n   that we can learn an effective model with the available data then we do\n   additional fine-tuning using MB2PO”. I am not sure how we are establish this\n   ? This is a genuine question and not a rhetorical remark!\n4. The details of the models used for fine tuning are absent, I did see which NN\n   models are used to build dynamics models. I am sure authors would agree that\n   a “neural networks that outputs a Gaussian distribution” would cover large\n   literature at conference like ICLR. Moreover, what network is used would\n   significantly impact what can be learned by effective referring to the\n   comment above. \n\n### Typos/ No impact on score\n\n* Is there a ‘tilde’ missing in the algorithm 4.3, just above add sample ? if\n  not do we need to learn reward model ?\n\n* On page 6 last paragraph: The iterations mentioned are on model based samples\n  (innermost For Loop in the algorithm ?)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Trd5ruDLsuE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Hi,\n\nFirst I wanted to thanks authors to put this manuscript together, I enjoyed reading it. \n\n*Summary* Authors suggest Model-Based Behavior-Regularized Policy Optimization (MB2PO) for fine tuning policies trained with behavioral regularized model free RL.\n\nI enjoyed reading the paper, I think it's a neat idea, and authors did a good job explaining them. \nAway from the details, looking at the results I have an important concern about the usefulness of the method. So authors claim that if AWAC has not achieved expert performance. But looking at the result, the method have not delivered the promise, and in many cases the performance actually degraded. So if we cannot be (at least to a good extent) confident that fine tuning will make the performance better, what is the main benefit? Specially if we are in mostly off-policy setting, and cannot really get the performance of the model by running the policy multiple times on the environment.\n\nI am happy to increase my score, if authors clarify my concern (/mis-understanding?).\n\nThanks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FhS5g2QUXeQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Final recommendation**\nI do not recommend accepting the paper. The results have been greatly improved. They now look decent and I have improved my score as a result. However I think the contributions are still not clearly highlighted. I however encourage the authors to improve their paper. \n\n**Summary**\nThis paper proposes to combine two approaches for offline reinforcement learning: behavior-regularized methods and uncertainty-aware model-based methods. The proposed approach works in two steps. First, a conservative MDP is constructed by estimating a transition functin and substracting an uncertainty penalty from the reward for each state and action pair, similarly to MOPO. (Yu et al., 2020).  AWAC (Nair et al., 2020) is used to learn a policy $\\pi$ using offline data and the penalized reward, where $\\pi$ is constrained to be closed to the behavior policy in terms of KL divergence. In a second, optional step called MB2PO, the paper proposes to iteratively refine $\\pi$ by sampling trajectories from the conservative MDP and improving $\\pi$ using AWAC. The proposed approach is compared to existing methods (MOPO, BEAR, BRAC, CQL) on the D4RL benchmark. The best result of the proposed approach, i.e. with or without the second step, is shown to be competitive with the state of the art. \n\n---\n**Strong points**\n\n1) The paper offers a possible theoretical reason about why the proposed approach may improve result: it improves the bound on the difference between the return of the MDP and the approximate MDP.\n\n2) The experiments look good to me and show the performance of the method.\n\n3) Results can be competitive with the state of the art.\n\n**Weak points**\n\n1) My key concern about the paper is that, at the moment, it does not provide a (non empirical) method to decide whether to used the second step or not. In the experiments the best out of the two results is used. This makes it hard to assess the interest of the method.\n\n2) Reading the paper was rather easy, but identifying the difference between the contributions of the paper and previous work was not so easy for me. \n\n3) I am not sure I could reproduce the experiments. For example the architecture of the models used is not described.\n\n---\n**Recommendation**\n\nI vote for rejecting the paper. The idea looks interesting and should be investigated further. However, currently, the proposed idea is in my opinion not fully developped (weak point 1) and the results, while fine, are not improving the state of the art enough to overlook this point. \n\n---\n**Details**\n\nI like the theoretical intuition. While not the foundation of the idea, it is good to have a theoretical sanity check.\n\nThe second step of the method sometimes severely deteriorates the policy. For me, evaluating both policies and picking the best one is not a very satisfactory method for an off-policy algorithm.   \n\nRegarding the contributions of the paper, I am under the impression that in the section presenting the method (section 4), the novel contributions are Equation 7 (and associated text) in section 4.1, Equation 11 (and associated text) and section 4.3. It is however nice to have information about existing methods leveraged in this work. I am not sure what the best solution is but I would like to suggest using \"they\" rather than \"we\" when the paper describes previous work.\n\nAround Equation 6, the paper states that a problem with MOPO is that estimating uncertainty is difficult and *there will inevitably be model errors that will lead to overestimated Qvalues* and that the proposed approach address this issue. I think it would be nice to have some experimental results highlighting that the proposed approach indeed fixes this issue, perhaps by showing the uncertainty estimates and different resulting actions for different methods in problematic state.\n\n\n---\n**Questions**\n\nCould you please correct me if I misunderstood the contributions of the paper?\n\n---\n**Minor details**\nSection 2.1: I think *AWAC* is used before being introduced.\nAlgorithm 1 has no caption", "extra_scores": {}, "extra_text": {}}, {"review_id": "EiXORgsKHPC", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\n\nThis paper proposes an improved offline RL (batch RL) algorithm combining the state-of-the-art behavior-regularization actor-critic method (Nair et al., 2020) with a model-based RL technique. N-trained probabilistic dynamics models generate fictitious trajectories with uncertainty-penalized rewards after pretraining the policy with the behavior-regularization solely on the offline data. Both these generated data and the original offline data are used for the further behavior-regularized actor-critic training. Numerical results showed that the proposed method outperformed recent offline model-free and model-based RL algorithms.\n\n\n\n2. Pros\n- This paper efficiently combines the state-of-the-art method with a model-based RL technique.\n- Challenges in offline RL and the related works are well-described.\n- Ablation study investigated five different kinds of data types for in-depth investigation.\n\n\n\n3. Cons\n\n(1) Experimental Support\n\n- The ablation study is unfair. The results of 'ours' in Table 1 are the maximum combination of AWAC and AWAC+MB2PO, which includes five best results from AWAC out of nine best results from 'Ours.' Although related explanations are written at the end of Section 5, AWAC+MB2PO should be considered as 'ours.' This is the reason why I had no choice but to give a harsh rating.\n- Besides, model-based learning and generating fictitious samples suffer from compound errors, so the learning can be inaccurate, especially when the length of the trajectories is long. However, the proposed algorithm uses 95% of fictitious samples in the model-based fine-tuning phase, causing degradation of overall performance due to the compound error issues.\n\n(2) Novelty\n\nThis work is not novel since this paper's algorithm heavily depends on state-of-the-art AWAC (Nair et al., 2020), and the uncertainty-penalized reward generalization is from other previous work (Yu et al., 2020).\n\n(3) Reproducibility\n\nThe paper's experiments do not guarantee reproducibility since the authors did not provide the implementation code.\n\n\n\n4. Minor concerns\n- The notion of the behavioral policy should be unified, either pi_b or pi_beta.\n-  The notion of the expectation should be unified either E or mathbb{E}.\n\n\n\n=== Post Rebuttal Responses ===\n\nI thank the authors for their replies and the corresponding new revision. After reading them and the other reviews carefully, I changed the rating accordingly. However, I still lean to reject this paper for the following reasons.\n\n1. When saying \"our method outperforms something,\" the standard deviation and the mean should be considered (such as confidence interval or statistical test). However, the authors seem only to consider the mean in the paper and the other replies. In this sense, the proposed method does not outperform CQL. Besides, the variances of CQL in ‘halfcheetah-mixed’ and ‘hopper-mixed’ are required to conclude that the proposed method outperforms CQL in the ‘mixed’ setting (e.g., the variance of the proposed method in 'hopper-mixed' is high).\n2. I agree with the other reviewers’ common concerns on the novelty / main benefit of this paper.\n3. The supplementary code can be shared with the reviewers using an anonymous link, as explained in ICLR Author Guide. It would have been better if the authors used this functionality of ‘official comments’ during the review process for better reproducibility.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1JDVGBbykH1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors take an attempt at offline RL thanks to a mix between behavioural policy regularization and model based policy optimization. They basically combine two algorithms: AWAX and, depending on the level of safety given an epistemic uncertainty evaluation, MOPO may be additionally used to fine tune the policy.\n\nUnfortunately the work suffers from several severe weaknesses:\n- the writing is not good. See the typo and minor comments section.\n- the positioning is biased and missing to many accounts. Out of 31 citations, 12 are from the same author. Even more problematic, most, if not all, the references on offline RL are from this author, and therefore lacks diversity. In particular Model-based offline RL [Iyengar2005,Nilim2005], and model-free offline RL [Thomas2015b] have a rich history. More specifically, Equation (3) is identical to that of the Reward-Adjusted MDP found in [Petrik2016]. The Safe Policy Improvement objective has been considered for instance in [Thomas2015b,Petrik2016]. Equation (7) proposes to optimize the policy under a constraint on the policy search (identical to online TRPO, which is evoked later) that is very similar to [Laroche2019], except that the constraint is not state based and therefore probably less efficient.\n- \"If our initial policy does not achieve expert level performance, and we are confident that we can learn an effective model with the available data, then ...\" => it is unclear how these decisions are sorted out. Performing those safety tests are an area of research in themselves [Thomas2015a].\n- even if we assume that the algorithmic novelty is proven, it seems pretty incremental, since it amounts to perform a test to decide between two algorithms.\n- Finally, the experimental results do not savethe day. We observe that \"Ours\" is always the max of AWAC and AWAC+MB2PO, which is a little suspicious, since we have no information on how the decision is made. In comparison with CQL, it is not better (but it is a strong baseline). So, it's not improving the state of the art. It would have been informative to show the behavioural performance in each setting.\n\nTypo and minor comments:\n- AWAC is used without citation or explanation first (2.1)\n- \"The most common off-policy model-free algorithms are actor-critic algorithms that alternate between policy evaluation and policy improvement in order to learn an effective policy.\" => this is not actor-critic but policy iteration.\n- \"Otherwise, we use the fully trained AWAC policy. These results are reported in the column Ours in Table 1.\" => otherwise what?\n- Sec. 4.2: effect => affect\n- Sec. 4.3: degredation => degradation\n\n[Iyengar2005] Iyengar, G. N. Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280, 2005.\n[Laroche2019] Laroche, R., Trichelair, P., & Tachet des Combes, R. T. (2019, May). Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning (pp. 3652-3661).\n[Nilim2005] Nilim, A. and El Ghaoui, L. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005.\n[Petrik2016] Petrik, M., Ghavamzadeh, M., & Chow, Y. (2016). Safe policy improvement by minimizing robust baseline regret. In Advances in Neural Information Processing Systems (pp. 2298-2306).\n[Thomas2015a] Thomas, P. S., Theocharous, G., & Ghavamzadeh, M. (2015, February). High-confidence off-policy evaluation. In Twenty-Ninth AAAI Conference on Artificial Intelligence.\n[Thomas2015b] Thomas, P., Theocharous, G., & Ghavamzadeh, M. (2015, June). High confidence policy improvement. In International Conference on Machine Learning (pp. 2380-2388).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for offline reinforcement learning methods with model-based policy optimization where they first learn a model of the environment to learn the transition dynamics, a critic and the policy in an offline manner. They basically learn the model by training an ensemble of probabilistic dynamics models represented by neural networks that output a diagonal Gaussian distribution over the next state and reward. Then they use the covariance of the probabilistic dynamics model to get an uncertainty measure that they incorporate into the  reward when training it with the AWAC.

There were two main concerns raised by the reviewers:

1) Experiments: As pointed out by the reviewers, the experimental gains don't look very convincing. In particular, the performance of AWAC looks bad, and MB2PO doesn't give much gains on top of it. It is not clear how much better the proposed method is doing on the tasks that it does well, without any confidence intervals or variance measures provided.

2) Novelty:  This is almost a trivial combination of two existing ideas: model based policy optimization and AWAC. It is not clear how useful this particular combination is in practice, and it seems like there is not much insights gained from it. 

I think better motivations, further ablations and more empirical analysis to understand the proposed model better. For example, analyzing the types of behaviors learned or how calibrated the uncertainty estimates that is incorporated into the reward is or some hyperparameter sensitivity analysis would make the paper more interesting.

As it stands right now, I am suggesting to reject this paper. I hope the authors will improve the paper for the future...


Decision: Reject