Pareto Policy Pool for Model-based Offline Reinforcement Learning
Authors: Yijun Yang
Jing Jiang
Tianyi Zhou
Jie Ma
Yuhui Shi
Venue: ICLR 2022 Poster (2022)
Keywords: model-based offline RL
Pareto front
multi-objective optimization
policy pool
model return-uncertainty trade-off

Abstract: Online reinforcement learning (RL) can suffer from poor exploration, sparse reward, insufficient data, and overhead caused by inefficient interactions between an immature policy and a complicated environment. Model-based offline RL instead trains an environment model using a dataset of pre-collected experiences so online RL methods can learn in an offline manner by solely interacting with the model. However, the uncertainty and accuracy of the environment model can drastically vary across different state-action pairs so the RL agent may achieve high model return but perform poorly in the true environment. Unlike previous works that need to carefully tune the trade-off between the model return and uncertainty in a single objective, we study a bi-objective formulation for model-based offline RL that aims at producing a pool of diverse policies on the Pareto front performing different levels of trade-offs, which provides the flexibility to select the best policy for each realistic environment from the pool. Our method, ''Pareto policy pool (P3)'', does not need to tune the trade-off weight but can produce policies allocated at different regions of the Pareto front. For this purpose, we develop an efficient algorithm that solves multiple bi-objective optimization problems with distinct constraints defined by reference vectors targeting diverse regions of the Pareto front. We theoretically prove that our algorithm can converge to the targeted regions. In order to obtain more Pareto optimal policies without linearly increasing the cost, we leverage the achieved policies as initialization to find more Pareto optimal policies in their neighborhoods. On the D4RL benchmark for offline RL, P3 substantially outperforms several recent baseline methods over multiple tasks, especially when the quality of pre-collected experiences is low.

Published as a conference paper at ICLR 2022
PARETO POLICY POOL FOR MODEL-BASED OFFLINE REINFORCEMENT LEARNING
Yijun Yang' 14, Jing Jiang', Tianyi Zhou? Jie Ma', Yuhui Shi
1 Australian Artificial Intelligence Institute. University of Technology Sydney
University of Washington, Seattle, University of Maryland, College Park
*Department of Computer Science and Engineering, Southern University of Science and Technology
(yjjun.yang-I. jie.ma-5] @ student. sondentt.ts.edde uts..du.auu jing. @ eu, tianyizh @ww.edu, shiyh @ sustech.edu.cn
ABSTRACT
Online reinforcement learning (RL) can suffer from poor exploration, sparse re- ward, insufficient data, and overhead caused by inefficient interactions between an immature policy and a complicated environment. Model-based offline RL instead trains an environment model using dataset of pre-collected experiences so online RL methods can learn in an offline manner by solely interacting with the model. However, the uncertainty and accuracy of the environment model can drastically vary across different state-action pairs, so the RI agent may achieve a high model return but perform poorly in the true environment. Unlike previous works that need to carefully tune the trade-off between the model return and uncertainty in single objective, we study a bi-objective formulation for model-based offline RL that aims producing pool of diverse policies the Pareto front performing different levels of trade-offs, which provides the flexibility to select the best policy for each realistic environment from the pool. Our method, "Pareto policy pool (P3)", does not need to tune the trade-off weight but can produce policies allocated at different regions of the Pareto front. For this purpose, we develop an efficient algorithm that solves multiple bi-objective optimization problems with distinct constraints defined by reference vectors targeting diverse regions of the Pareto front. We theoretically prove that our algorithm can converge the targeted regions. In order to obtain more Pareto optimal policies without linearly increasing the cost, we leverage the achieved policies as initialization to find more Pareto optimal policies in their neighborhoods. On the D4RL benchmark for offline RL, P3 substantially outperforms several recent baseline methods over multiple tasks, especially when the quality of pre-collected experiences is low.
INTRODUCTION
Offline reinforcement learning (offline RL) (Levine al., 2020) batch RL (Lange et al., 2012) can train an agent without interacting with the environment by instead using pre-collected experiences on other agents/policies. Recently, offline RL has been attracting growing interest due to the availability of large-scale datasets of diverse experiences in many RI. applications, e.g., autonomous driving (Shin & Kim, 2019), healthcare (Yu al., 2019), robot control (Schulman al., 2016), etc. However, RL algorithms developed for the online/interactive setting usually perform poorly in the offline setting (Fujimoto al., 2019; Janner al., 2019) due the data distribution shift caused by (1) the difference between the -atty-in-traiiinn and the behavior policies used to collect the data; and (2) the difference between the realistic environment in which we will deploy the policy and the environments used to collect the data. These differences can result in function approximation error and biased policy learning (Levine et al., 2020). To address these challenges, model-bbaed RI. approaches (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021; Yu et al., 202.) firstly learn an environment model from a dataset of logged experiences using supervised learning and then conduct online RL interacting with the model. The learned environment model fully exploits the pre-collected experiences and can avoid/reduce the costly interactions with the realistic environment required by RL, hence improving the sample efficiency.
That being said, due to the large size of state-action space, model-bused offline RL approaches can still suffer from "model exploitation" (Levine et al., 2020): when the dataset does not contain

---

Published as a conference paper at ICLR 2022
hullcheatah-raddaa
Ours:
4000
0.3
Return th MOP moste
.000
0.5
t-0
Return: 333.7
[-200
-2000
Farato Policy Fool
Pareit Pylley
1.0
t=0
Return: 4986.5
t= 1000
Fareto Policy
3.0
1000
MORcL MCPI
t=D
Return: 4121.9
t=1000
1500
1000
-500
uncertainty the HDP
t-0
t= =010
Return: 3379,4
MOPO:
MOReL:
t-0
Return (05): 236.5
t=130
t-0
Return (5.0): 2415.6
L-1000
t=0
Return (0.3): 135.1
L=0
Return (3.0): 2171.0
t- 1000
T=0
Return (0.1):664 66.4
t- 100
Return (1.0): 3285.1
1080
Figure 1: Model return-uncertainty trade-off of P3 (ours), MOPO (Yu et al., 2020), and MOReL (Kidambi al., 2020) on an offline RL dataset sallicheetah-randdooe from the D4RL benchmark (Fu et al., 2020). P3 achieves policies with different levels of model -eturn-uncertaii tradc-off. MOPO and MOReL are run three times with regularization weights {0.1, 0.5) and (1.0,3.0. 5.0}, respectively. Detailed discussion in Sec. 1.
sufficient samples for some state-action pairs, the epistemic uncertainty of the model on these "out- of of-distribution" pairs can lead to poor policy in offline RL. For example, an RL agent may easily exploit the model by repeatedly visiting the out-of-distribution states where the model erroneously issues higher rewards than the true environment dynamics, and thus the RL objective is biased by overly optimistic evaluation of the agent's performance. A widely-studied strategy to tackle this problem is applying regularization to the reward function, which keeps the model uncertainty small when maximizing the reward issued by the model (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021). By carefully tuning the regularization weight, it achieve preferable trade-off between the model reward and model uncertainty. However, it is usually challenging to tune the weight without access the realistic environment that the learned policy will be deployed to. Moreover, even with access to the realistic environment, hyperparameter tuning methods such as Bayesian optimization (Frazier, 2018) need run multiple instances of offline RL and can be computationally expensive. In addition, the offline-trained policy may not be directly generalized to different environments. In this paper, we address challenging problem, i.e., how balance between the model reward and model uncertainty during offline RL so it can produce diverse policies that can be adapted to different realistic environments?
Instead of formulating the problem a single-objective optimization with regularization as in previous works, a primary contribution of this paper to treat the model reward/return and model uncertainty two separate objectives and develop an efficient bi-objective optimization method producing pool of diverse policies on the Pareto front, which correspond to different trade-offs between the two objectives. Therefore, when deployed to new realistic environment, we can accordingly choose the best policy from the pool. Our method is called "Pareto policy pool (P3)," and example of the policies achieved by P3 is shown in Fig. 1. When deployed to the realistic environment, Pareto policy is overly optimal on the model return, so it runs fast at the beginning but quickly falls to the ground due the aforementioned "model exploitation". On the contrary, Pareto policy 4 favoring small model uncertainty is overly conservative and keeps standing still for 1000 time-steps because it avoids taking exploratory actions that potentially increase the uncertainty, As expected, Pareto policy 2&3 with the more balanced trade-off between the model return and uncertainty perform better and achieve higher returns in the test environment. These results imply that model-based offline RL's performance significantly relies on the trade-off between model return and uncertainty in the optimization objectives. However, is usually challenging or intractable to determine the optimal trade-off before deployment and control it during training. Moreover, for existing methods adopting a regularized single-objective, i.e.. scalarization (Boyd & Vandenberghe, 2004, Chapter 4.7), even trying all possible regularization weights cannot fully recover the Pareto front and thus cannot guarantee to find the optimal trade-off. For example, in Fig. 1, by running multiple instances with different regularization weights, MOPO (Yu et al., 2020) and MOReL. (Kidambi et al.,
2

---

Published as a conference paper at ICLR 2022
2020) can only generate few separated solutions, and it is difficult to find one with advantageous trade-off among them. In contrast, our method aims at efficiently generating rich pool of diverse and representative policies covering the entire Pareto front without tuning the trade-off during training. Thereby, when deployed to a new realistic environment, the best policy for the new environment can be efficiently selected from the pool. Hence, P3 provides a simple and principal approach that addresses the two major challenges in model-based offline RL, i.e., "model exploitation" and generalization to different unseen states in order achieve high returns.
Due to the complicated shape of the Pareto front that is unknown during training, finding a pool of diverse policies covering the whole Pareto front raises several non- non-trivial algorithmic challenges: (1) How to find policies located at different regions of the Pareto front associated with different levels of model return-uncertan uncurraitct trade-off? (2) How to avoid training each policy from scratch and reduce the computational cost linearly increasing with the number of policies? Inspired by recent works in multi-objective optimization (Cheng al., 2016; Ma et al., 2020; Xu et al., 2020), we explore different regions of the Pareto front by generating multiple diverse reference vectors in the bi-objective space, each defining a constrained bi-objective optimization whose solution resides in a local region of the Pareto front. By solving these constrained bi-objective optimization problems, can obtain a diverse set of policies covering the whole Pareto front. For solving each problem, we extend MGDA algorithm (Désidéri, 2012) to be a two-stage gradient-based method that provably converges to the Pareto front region targeted by the reference vector. In order to achieve more policies on the Pareto front without linearly increasing the cost, we start from the previously obtained policies for initialization and explore their neighborhoods on the Pareto front by perturbing their corresponding reference vectors, resulting
dense set of Pareto policies in each local region.
In experiments, we evaluate P3 and compare it with several state-of-the-art model-based/free offline RL methods on the standard D4RL Gym benchmark (Fu et al., 2020). P3 achieves the highest average-score over all datasets and significantly outperforms the baseline methods in 5 out of the 9 caineddiuu-qualliyy datasets, showing the advantages of P3 on learning from non-expert experiences. We also present a thorough ablation study to identify the most important components in P3.
RELATED WORK
Due to lack of space, we focus our discussion here on directly related works and present a more detailed overview of related work in Appendix A.2. To address the aforementioned "model exploita- tion", recent works rely on applying uncertainty regularization to the model return (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021) which can be difficult and costly tune the weight of regularization. In contrast, our work reframes the policy learning under the environment model as a bi-objective optimization problem and produces a diverse set of policies on the Pareto front performing different levels of model return-uncee return-uncertainty trade-off, which provides flexibility to select the best policy in the inference stage.
3 PRELIMINARIES
po). where S is the state space, .A is the space of actions, P is the transition probability: S x A A(S) where A(S) is a probability distribution over S, r S x A -> IR. is the reward function so r(sh, an) ) is the immediate reward for taking action Oe at state 8h. H the horizon of the process, and po is the distribution of the initial state sp. RL aims at learning a policy : S + A maximizing the expected return in Eq. (I), where π in this paper takes a deterministic action ag based on the state 5h.
We consider an episodic Markov Decision Process (MDP) M -=
H
max Rp, (Tg, M) Esor Xe
Σ
r (Sh an)
In model-based offline RL, the agent instead interacts with an environment model rather than the realistic one. We train an environment model M = A,, H. using pre-collected dataset D A {(sh, ah, 11+1, rx)l m of experiences by behavior policies, hand-designed controllers, or human demonstrators. By interacting with the model, online RL methods can learn in an offline manner. However, when D does not contain sufficient samples for some state-action pairs, the epistemic uncertainty of the model on these policy. For example, an RL agent may easily exploit the model by repeatedly visiting the OOD states where the model erroneously issues A higher than the true environment reward r and thus the RL objective is biased by overly optimistic evaluation of the agent's performance. A widely-studied
(OOD)" pairs can result in a poor
3

---

Published as a conference paper at ICLR 2022
strategy to tackle this problem is applying regularization to f, which keeps the model uncertainty small while maximizing the model reward (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021). A practical implementation of the regularized reward function is developed by (Yu et al., 2020): in = fu- Xu(Sn, an). where u( (s) denotes the estimation of the model uncertainty at the state-action pair (&n, as) and À controls the trade-off between f and u, which has to be carefully tuned in practice. Based on the regularized reward function, Yu et al. (2020) proposed a modified policy optimization objective:
max R& ( M) = Beg Poom
Σ (5p) an) du (3%, an)) h=
(2)
Despite being intuitive, this method's performance is sensitive to the regularization weight À (Ki- dambi et al., 2020; Yu et al., 2021), and tuning À is usually challenging without access to the realistic environment that the learned policy will be deployed to. Moreover, even granted the access, hy- perparameter tuning methods such as Bayesian optimization (Frazier, 2018) require running many instances of offline RL, which can be computationally prohibitive.
Instead of optimizing single regularized objective, our "Pareto policy pool (P3)" proposed later treats f and u as two separate objectives and develop an efficient bi-objective optimization method that does not need to tune the trade-off deliberately but produces pool of diverse policies, which are learned under different trade-offs between the two objectives. Thereby, when deployed to a realistic environment, the best policy can be chosen from the pool.
4 PARETO POLICY POOL FOR MODEL-BASED OFFLINE RL
4.1 PROBLEM FORMULATION
In order to estimate the model uncertainty accurately and alleviate the model exploitation problem, we follow previous works (Janner et al., 2019; Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021; Yu al., 2021) and construct bootstrap ensemble of K environment models (M*) j==1- Each model M = A, Pt, f², H. po} is a two-head feed-forward neural network that takes a state-action pair (Sn, on) as input and outputs the mean l4 and standard deviation a, of 1$ fx). i.e., the next state concatenated with the reward. More details about our model are given in Appendix A.5. As demonstrated in Yu et al. (2020), this ensemble is effective in estimating the model uncertainty as the maximal standard deviation over all models, i.e., u(S,, an) = max,e[K] (8o,, an)) 122) Moreover, by randomly selecting a model in each step to provide the reward f($), a2) = År. we can effectively mit- igate the model exploitation problem. Unlike previous works combining f and u as single objective, we treat them as two separate objectives and aims at solving the bi-objective optimization below.
max J As (TO.M)
M) = max( Jh (To, M). Ju (Ta, M))
(3)
B
H
(Jh (TT, M), Ju (To M)) = Begg Pio
(F (Bn,,n), exp (-u (Sn, an) /K))
(4)
where x a temperature applied to u ($n, an). For simplicity, in the rest of this paper, we remove Po and M. In Eq. (3), S aims to maximize the expected model return, and J" is designed to minimize the expected cumulative model uncertainty. However, the bi-objective optimization naturally has mul- tiple (up to infinite) optimal solutions instead of one single policy and each optimal policy performs a different level of trade-off between the two objectives. For example, a policy favoring small model un- certainty maybe overly conservative and avoids taking exploratory actions so its model return can be low. In contrast, a policy pursuing high model return might fail realistic environments at a state that the model is highly uncertain about. Examples of these policies are given in Fig. 1. Formally, for any two policies ,, and k.. ne dominates π, if and only if J (T.) 2 JTT,) and J ((T,) + JTT)). ). A policy nt is Pareto optimal if no any policy dominates π in Rd i.c., no objective can be improved without detriment to another objective at x². All Pareto optimal policies constitute the Pareto set, and the Pareto front is the image of the Pareto set in the space spanned by all the objectives. Unfortunately, it is almost infeasible in practice to find the whole Pareto set and the Pareto front's shape is also unknown.
As discussed in Sec. 3, it is difficult to determine the optimal trade-off between J and JJ because we cannot access the realistic environment during training. It is also challenging and costly control
4

---

Published as a conference paper at ICLR 2022
the trade-off. Hence, straightforward strategy is find diverse set of policies the Pareto front performing different levels of trade-off and select the best one when deployed to a realistic environment. However, how to find these Pareto optimal policies is still open challenge. In addition, it is expensive to train each Pareto policy from scratch. After allocating a few diverse policies the Pareto front, can start from them to find more Pareto policies we can avoid linearly increasing the computational costs? To overcome these challenges, we develop "Pareto policy pool (P3)", which can efficiently and precisely find diverse set of policies on the Pareto front of return- uncur-unnertaantyy bi-objective optimization.
4.2 PARETO POLICY POOL
Fig. 2 illustrates the main idea of "Pareto policy pool (P3)", whose detailed procedure are given in Alg. 1. In order to find diverse Pareto policies and precisely control the trade-off of each policy, P3 generates multiple reference vectors {v,} in the objective space, each forming constraint to the bi-objective optimization in Eq. (3) and targeting different re- gion on the Pareto front (Sec. 1). Thereby, the Pareto policies { achieved by solving the constrained bi-objective optimizations are diverse in terms of the objective trade-off. For solving each of them, we develop an efficient two-stage optimization algorithm, i.e., Alg. 2, which starts from an initial point and moves to a region fulfilling the constraint (correction stage), and then apply a multi-objective optimization algorithm to find a Pareto policy in the targeted region. To avoid training each Pareto policy from scratch, which leads to linearly increasing costs for n policies, develop local extension method on the Pareto front in Sec. 4.2.2. This "Pareto Extension" firstly generates more reference vectors by perturbing each vector in {u;) ) and then uses their associated Pareto policies {m² )2 warm start the bi-objective optimization constrained by the new vectors (Line 9-13 in Alg. 1). These locally extended policies, together with the n policies, compose pool of diverse policies on the Pareto front, from which we can select the best policy when deploying them to realistic environments.
1.00
14
Optimization Path
Reference Vector
Perturbed Vector
0.75
Pareto Front
Ug
Objective 2
Starl Point
0.50
TTS
122
U2
0.25
0.00
71
1.00
Objective
Figure 2: Illustration of by solving benehmark problem from (Lin et al., 2019).
Algorithm 1 Pareto policy pool (P3) for model-based offline RL
1: input: dataset D, constraint & < 0, step size num. reference vectors n, T, > Ti
2: initialize: environment models, Pareto policy pool P = Ø, 0 Ta < Tb < for Eq. (5),
O <<C < Ta for Eq. (8), number of updates: T = nT, + 2Ti)
3: Train the model on using supervised learning;
4: Generate reference vectors (v)...., )n by Eq. (5);
5: for E (1.....,) n) (in parallel) do
> Diverse Pareto Policies
6: Initialize policy
7: for j = 0, l.......7
do 8: Update the parameters of n, by Alg. 2 with og
9: Generate (v. 0, to os by Eq. (8);
Local Pareto Extension
for v' (o. 17 do
11:
for j' = 0. = PU }
do
D Store Pareto policies into the pool
13:
Update the parameters of m. by Alg. 2 with v';
14:
output: P;
4.2.1 DIVERSE PARETO POLICIES BY REFERENCE VECTORS
Inspired by recent works in multi-objective optimization (Cheng et al., 2016; Lin et al., 2019; Ma et al., 2020; Xu et al., 2020), we explore the Pareto front by generating diverse set of reference vectors defining multiple constrained bi-objective optimization problems. As shown in Fig. 2, generate
5

---

Published as a conference paper at ICLR 2022
uniformly distributed reference vectors {0;};- in 0-1 normalized objective space by Eq. (5), i.e,
8. - (To T)Tc: Fa (i 1)r,), Тb
Ta < To 1, E 21,
(5)
where Ta and т control the range covered by reference vectors. Each v; (o) vz) defines constrained bi-objective optimization problem based on Eq. (3) whose solution resides in the targeted region of the Pareto front:
J _ max(J" (To)) J")
s.t. (TO, v/) - -
V;
(T
2 th
[
where V (TT. v;)) defines a similarity metric between the reference vector d; and the objective vector (Tp) > O. When & is large, ((Te) constrained to be close to vs, and the targeted region on the Pareto front is small. Hence, solving the constrained bi-objective optimization for the diverse set of reference vectors produces a diverse set of Pareto policies associated with different trade-offs between J and J In the following, we develop efficient two-stage algorithm for solving this optimization problem in Eq. (6).
To solve the bi-objective optimization with inequality constraint in Eq. (6), we propose two-stage gradient- based method in Alg. 2. It first finds a solution meeting the constraint by alternating between optimizing the two objec- tives (correction stage) and then treats the con- straint as the third objective and applies an ex- isting multi-objective optimization algorithm to find Pareto policy in the region targeted by the constraint (ascending stage). In the first
Algorithm 2 A two-stage method for solving con- strained bi-objective optimization
1: input: Te,, 1s v
2: if ( v²) ub then
Correction stage
3:
if J (/to)//e (/J < <u /ug then
4:
Compute VaJ (
5:
0 0, + 17 V J.J (To.): else
6:
7:
Compute a (T..) Oc+1 = 0, + 17 V VaJ _
stage, the algorithm checks how the constraint is violated (line 3), e.g., whether J or J is too small, and then accordingly chooses one to apply gradient ascent towards meeting the constraint. Once it finds a feasible solution fulfilling the constraint, it switches to the sec- ond stage, which reframes the problem as a tri- objective optimization by turning the constraint 14: output: no.
8:
9:
else
Ascending stage
10:
Compute (oF(TT,); Find a to Eq. (7);
11:
12:
0c+1 = 0g na; V VoF(To)))
+ ++1
to be the third objective, i.e., maxę F(mO) A
(J" (mo), , (To). (TT. vs)) and applies MGDA (Désidéri, 2012) to find a Pareto solution for this problem. Each step of MGDA aims at finding convex combination a₁ V (TT,)) of all ob- jectives' gradients V& (T.))= a (VoJ) (Te.), Vo.'"(TT,) ). V vz)) such that no objective is decreasing on the combined direction. This is equal to solving the following min-none problem.
min ox ((F,) )ll,, s.t. |/|| 1, 0r 2 0,
ac:
which can be efficiently solved by Frank-Wolfe'ss algorithm (Jaggi, 2013). Then MGDA takes one step along the direction to update the policy 0,, i.e., 0,+1 0: + n/az F ). Since MGDA always improves each objective, the constraint will not be violated, and the algorithm will finally find a Pareto policy in the targeted region. In practice, we use OpenAl's ES (Salimans et al., 2017) to estimate the gradient of each objective (see Appendix A.3) for its efficiency and stability. Alg. 2 provably converges to the Pareto front region targeted by the reference vector with approximate gradients.
Assumption 1. Suppose m objectives {fa 1 of mullti-obeective function are differentiable and that their gradients are Lipschitz continuous with constant L; > O.
Assumption 2. For the ES gradient V (x.) we have ( (x)) f.(z)) Suppose Var( fi, (2x)) I o².
Lemma 1. mutually independent objectives fo 0. which satisfies Assumption I 2, has (e, f((x)) < -(1 L Lu22
)lde + Loj 0² where d, = Em 04, fi(x)).
1|d.112 3 the ascending stage leads to a monotonically non- 1d+1 T.p
Lemma implies that when < increasing sequence of the objectives.

---

Published as a conference paper at ICLR 2022
Theorem 1 (Non-convex convergence rate). Let Assumption 1 & 2 hold, A f((zo) - f((x*), Lon" For an arbitrary objective fü. given any <> 0, after - 0( yo2
Loi" B 1 2 and = 2
iterations of ascending stage, we have T ETDLE [|]
Complete proofs are provided in Appendix A.1. Theorem 1 provides the convergence rate of the ascending stage for non-convex objectives. For the correction stage that performs single-objective ES gradient ascent, Nesterov & Spokoiny (2017) have proved its convergence rate to a stationary point of the non-convex objective.
4.2.2 LOCAL EXTENSION OF PARETO FRONT
Although solving Eq. 6 for a diverse set of reference vectors using the algorithm in Sec. 4.2.1 can produce diverse set of Pareto policies, the computational cost linearly increases with the number of policies. In practice, a few Pareto policies cannot cover all possible trade-offs and thus may lead to sub-optimal choices of policy in deployment. In order to efficiently obtain more policies with different fine-grained levels of model return-uncertanng trade-off, we propose a "Pareto Extension" that starts from the diverse set of Pareto policies and locally searches for more policies near them on the Pareto front. This warm-start strategy avoids training more policies from scratch and practically saves a significant amount of computation. As illustrated in Fig. 2 and Alg. I, we perturb each reference vector v; for more reference vectors v." and U. i.e.,
(8) These new vectors create more constrained bi-objective optimization problems in the same form as Eq. (6). Instead of using a random initialization, we start from and fine-tune { } for a few iterations to achieve the Pareto policies of these new optimization problems. These policies and (T,)%_1 together constitute a pool of Pareto policies. When deployed to realistic environment, we first evaluate these policies for few steps and then select the one achieving the highest return to deploy. More details about our selection strategy are given in Appendix A.4. As long as we include sufficient diverse policies in the pool, we can find a promising policy that significantly outperforms the policies trained by other model-based offline RL methods.
a 0; + u v; 0, c, (c,-c),0<<<5
5 EXPERIMENTS
This section aims to answer the following questions by evaluating P3 with other offline RL methods on the datasets from the D4RL Gym benchmark (Fu et al., 2020). (1) Comparison to prior work: Does P3 outperform other state-of-the-art offline RL methods? Moreover, when the dataset does not contain high-quality samples for some state-action pairs, can the policies generated by P3 generalize to unseen states in order to achieve high returns? (2) Ablation study: In the experiments, we apply several techniques used in previous work (see Appendix A.6 for more details). How do they affect performance? (3) Effectiveness of P3: Why is it challenging to find the optimal trade-off between the model return and its uncertainty? We empirically explain how P3 efficiently alleviate this problem by generating a rich pool of diverse and representative policies.
To answer question (1), we compare P3's performance with the state-of-the-arr offline RL algo- rithms, including BCQ (Fujimoto et al., 2019), BEAR (Kumar et al., 2019), CQL (Kumar et al., 2020), UWAC (Wu et al., 2021), TD3+BC (Fujimoto & Gu, 2021). MOPO (Yu et al., 2020), MOReL (Ki- dambi et al., 2020), and COMBO (Yu et al., 2021). For faimess of comparison, we re-run these algorithms using author- provided implementations' and train each algorithm for 1000 epochs. We also carefully tune the hyperparameters of baselines such as BCQ, CQL. TD3+BC, MOPO, and MOReL. by grid search and choose the best ones for each benchmark dataset. Most of them achieve higher scores than those previous versions. For other basclines, such as UWAC and COMBO, we adopt the hyperparameters eters they can find. More details on the experimental setting can be found in Appendix A.6. Our serrge--ccoe
their original papers, assuming they have chosen the best hyperparam-
experiment results are provided in Table 1. It is obvious that P3 achieves the highest across all datasets, and significantly outperforms the baseline methods in 5 out of the 9 low/medium- quality datasets. Moreover, we find that the low/mediun-que quaalityy datasets in the D4RL benchmark are "imbalanced"'. As illustrated in Fig. 8, there are large number of bad samples, some mediocre
As noted by https: / htt//gtttttee thi.. con/ a aravi ndd 33/mj 1/ / ssues /3, we remark that the imple- mentation provided by MOReL's author achieves lower results than their reported ones.
7

---

Published as a conference paper at ICLR 2022
COL
UWAC*
TD3-BC
MOPO
MOPO*
MORAL
COMBO"
PIYRQE
P3
=35
[0064-3
Hopper
81:105
39:123
8...114
22.4 113.1
R6
16.2 113.3
104
MAR 148
17.
13.8
103
Wilker2d
4.6200.7
12.5-1022
0,5-13
155-1177
1.5:14
422657
136-=2.6
17.3 #83
7.0
19.7=00
22.
HoltCheriah
454.2177
48.346.3
665835
478-00
11.
8138
54.
61.4-200
4:1
Hopper
53.9-3.7
51.8
233
69.1 =55
2500=124
#32.1
105.9
105. (+00
Welker2d
012 5001
57.5 2:.3
575-278
$1.5 -5.0
41.2 4:30
17.8 119
10.5 285
75.5
71.1 2:355
81.3+220
Malion
HollCheetah
609:51.1
16.34:33 6:31
47.3 :0.4
46.8 :0.5
00:7::11
53.1 17.00
31.9 :6.1
55.1
43.4 5:11
46.7 1-06
409:21677
39A
17.8
#32.
59.5 89.5
WalkerM
270:6633
13.7
69.5
Moon
348-
23.2
35,6-22.2
3922772
44.8 =3.5
42.4
34.3-23
307-1133
52.5
58.0
02.1
E5p0R Eeport
HillCheeta
92.7322
92.7 1..55
90.5 21.
126
96.
81.3:1215
22:1544
$1.4
65.3 1.04
1053-481
516-23111
11500-14.1
10955
62.5 6275
262:1119
1106-112
113.3 40.5
Wolkerhd
1103:04
40.3
100
HoltCheenal
95.92112
643
70/6-13.6
427
88.9 -53
70.3
63.3 =8800
35.4 :19.2
90
57.1 =:600
699 #:005
Hopper
106 10..6-259
506-9225
111/0::122
136.7--7122
102.0-011
16.1
00.6 1:311
357 $6.0
521 4:27.7
111.1
109 1:133
1103 1:03
Whlkerhh
159 7 10..
22.1 1:4.5
109.7 10.3
99.7 112.2
[1105) 0.5
1213
46.6 12.9
39 128
96.1
9013 144
98.8 1344
135.4
17.
Meun
622 64.1
30.0
29.6
68.0
53.3
=11.0
#11.1
71.5
763
Table I: Results D4RL Gym experiments. Normalized score (mean over the final 10 evaluations and 5 seeds. * marks previously reported results. Dataset quality gradually improves from Random to Medium- expert...
MOPO
MOR
TDS+BC
PS (OURS)
6000
hndkchertah-rnnddr
4000
hopper- peree--eduunnggeee unneopay
1250
walkerdd random
walker2d medium
Aveege Rekim
4000
Assngea Paltan
3000
2000
2000
600
1000
253
1000
200
400
900
500
1000
0
200
400
600
800
1000
200
400
900
800
1000
200
400
000
800
1000
Tretning Epodhs
Treiring Epocits
Tranning Speedha
Triining Epochss
Figure 3: Learning curves on low-quality datasets. Returns are averaged over 10 evaluations and 5 seeds. The shaded area depicts the standard deviation over 5 seeds. P3 outperforms two recent model-based offline RL methods (i.e., MОPO and MOReL) and the SoTA model-free method (i.e, TD3+BC), A full results of all datasets are in Fig. 7 of Appendix.
samples, and a few good samples, causing problems with learning accurate behavior policies or generalizable environment models (Buckman et al., 2021; Zhang et al., 2021). Therefore, many offline RL algorithms,, especially model-free algorithms that heavily rely on the accurate recovery of behavior policy (Fujimoto al., 2019; Kumar et al., 2020; Fujimoto & Gu, 2021), perform poorly on these datasets. According to the results in Table 1, P3 achieves SoTA performance and outperforms other baselines by large margin on random, medium, and medium-replay datasets, indicating the advantages of P3 on learning from low- low-quality experiences. We use online policy evaluation to select the best policy during the test phase of P3, which can be computationally intensive or cause over- heads when deployed to realistic environments. To overcome this drawback, we replace the online evaluation with Fitted Q Evaluation (FQE) (Le et al., 2019), an offline policy evaluation method, which (approximately) evaluates policies using the offline data only. The implementation details and experimental results are reported in Appendix A.4 and Table 1. respectvelly We surprisingly find that FP3+FQE (offline policy evaluation)" only slightly degrades from "P3+online policy evaluation" on the performance, but still outperforms all baselines on the low/medium datasets, suggesting that FQE enables more efficient inference for P3 it has exactly the same inference cost as other baselines.
To answer question (2), we conduct a thorough ablation study toward five variants of P3, each removing/changing one component used in P3. Table 2 reports their results on the D4RL Gym benchmark with 9 representative environment-datass. combinations. In Fig. 9 in Appendix, we visualize the Pareto policies obtained by P3 to highlight the effectiveness and superiority of our method. Among the five variants of P3, "scalarization" " replaces our porposed Alg. 2 with the scalarization method (Boyd & Vandenberghe, 2004, Chapter 4.7); "no StateNorm" removes the state normalization (Mania et al., 2018; Fujimoto & Gu, 2021); "no RankShaping" removes the rank-based scale shaping (Wierstra et al., 2014); "no ParetoExtension" removes the Pareto extension proposed in Sec. 4.2.2; "no BehaviorCloning" removes the behavior cloning initialization (Kumar et al., 2020; Kidambi et al., 2020). More details on these variants are provided in Appendix A.9. According to the results in Table 2 and Fig. 9, we give the following conclusions: (1) except "scalarization" and "no ParetoExtension", other variants perform comparably to our P3 while outperforming the previous results achieved by model-based (MOPO) and model-free (TD3+BC) RL algorithms on
8

---

Published as a conference paper at ICLR 2022
Data Quality
Rundom
Mediun-replay
Medium-exper
Environment
HalfCheetah
Hogper
Walker2d
HalfChestah
Hopper
Walker2d
HallCheenhh
Hogper
Walker2d
scalarization
32.3 11.5
15.0
214
88.5
49,9
52.117.3
$4.7 18.5
PS: no StateNorm
35.3 +2.5
34.9 +0.2
218
41.7=04 =04
82.3=129
61.6 194
47.1 +0.3
99.9 1:60
90.3 4222
F5: no RankShaping
37.6 +4.4
33.6 +0.3
27.3 46.2
44,3 =0.
95.6=17
64.7 +3.9
66.3 +1.9
108. +112
97.0 +2.6
p3: ParetoEsstession
31.2
5.2 10.4
10.2
13.2
88.2 111.4
P3: BehaviorCkiing
38.2.114
35.5
24.11:111
454=18
97.1-21
26.1
52.2 13.5
89.8 116.6
69,1 1:9.1
P3: our version
40.6++.7
35.4 +0.8
22.9 +0.6
48.2 -06
94.-11
64.0 +8.2
60.9 +10.5
110.8+0.5
98,9 +33
MOPO
4.2.5.7
69.2-111
32.7 =9.
73.7 49.4
70.3 4:2.9
60.6 #:2.5
77.4 427.99
TD3+BC
10.6 41.7
8.6
4.04
41.4
46.8
$7,8
81.9
$8.9
102,0
110.1
110.5 1.0.3
Table 2: Ablation study. Normalized score (mean-tstd) of P3 variants over the final 10 evaluations and 5 seeds when applied to three representative D4RL, datasets, i.e., random, medium-replay. and medium-expert, corresponding to low, medium and high-quality data, respectively.
1000
The qualttyylyy por www.ooleeteddddeeeoooo rapecianions
High
Figure 4: Model based offiine RL's performance in the deployed environment (heatmap) under different trade- offs between the model return (y-axis) and uncertainty (x-axis). Each circle is Pareto policy from the pool generated by P3. Zoom for more details. More results are shown in Fig. 10 of Appendix.
the low/mttiu datasets, reflecting that these widely-used techniques can improve P3's performance but are not crucial to our appealing results. (2) "scalarization" shows noticeable degradation in performance and cannot obtain dense set of diverse policies, as shown in Fig. 9. The results can be explained as follows: the scalarization method only finds a few separated policies, and it is difficult to find one with advantageous trade-off among them. In addition, we remark that the computational cost of multiple training with different weight assignments is similar to the cost of running P3. (3) "no ParetoExtension" degrades P3's performance on all 9 environment-datee dataset combinations, corroborating that dense set of policies on the Pareto front is essential to our results.
To answer question (3), in Fig. 4, we study how the P3 policies with different uncertainty-return trade-off perform in the deployed environment, For Now/ediuu-qquligg datasets (the left three plots in Fig. 4), the optimal policies with high realistic returns (bright areas in the heatmap) spread across almost the whole Pareto front. Therefore, to find the best policy, it is essential to explore the whole Pareto front and select one from a diverse set of Pareto optimal/stationary policies as P3 does. This explains why P3 performs the best on all the low/medium datasets. On the contrary, for high-quality datasets (the right two plots in Fig. 4), the optimal policies with high realistic returns gather within small region of the Pareto front and associate one trade-off level. Therefore, by carefully tuning the trade-off weight, previous methods can still find the optimal policy without visiting the whole Pareto front. Hence. we observe less advantages of P3 on the high-quality datasets. The reason behind is that the MDP models are very confident on high-return (realistic) state-action pairs if most samples in the training data are with high high-return (high-quality), while they can be uncertain about many high-return pairs if the training data only cover a few high-return samples (low/medium quality). It is worth noting that collecting high-quality datasets is usually expensive or infeasible in practice and many applications lack sufficient high-quality data. In these imperfect but practical scenarios, P3 performs significantly better and more stably than existing model-based offline RL methods that only learns one single policy.
CONCLUSION
In this paper, we find that model-based offline RL's performance significantly relies on the trade-off between model return and its uncertainty. while determining the optimal trade-off is challenging with- out access to the realistic environment. To address the problem, we study a bi-objective formulation for model-based offline RL and develop an efficient method that produces a pool of diverse policies on the Pareto front performing different levels trade-offs, which provides flexibility to select the best policy in the inference stage. We extensively validate the efficacy of our method on the D4RL benchmark, where ours largely outperforms several recent baselines and exhibits promising results on low-quality datasets.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Yijun Yang and Yuhui Shi are supported in part by the Shenzhen Fundamental Research Program under Grant No. JCYI20000109141225555 61761 136008, the Shenzhen Peacock Plan under Grant No. KQTD20161 12514355531, the Program for Guangdong Introducing Innovative and Entrepreneurial Teams under Grant No. 20172T07X386.
the National Science Foundation of China under Grant No.
REPRODUCIBILIITY STATEMENTS
Code is available at https: / /ttp//gittttate com/ OverEuro/P3, We provide full description of all our experiments in Section and Appendix A.6.
REFERENCES
Abbas Abdolmaleki, Sandy H. Huang, Giulia Vezzani, Bobak Shahriari, Jost Tobias Springenberg, Shruti Mishra, Dhruva TB, Arunkumar Byravan, Konstantinos Bousmalis, András György, Csaba Szepesvári, Raia Hadsell, Nicolas Heess, and Martin A. Riedmiller. On multi-objective policy optimization as a tool for reinforcement learning. CoRR, 2021.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In NeurIPS, 2017.
Rinu Boney, Juho Kannala, and Alexander Ilin. Regularizing model-based planning with energy-based models. In CoRL, 2019.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Jacob Buckman, Carles Gelada, and Mare G. Bellemare. The importance of pessimism in fixed-dataset policy optimization. In ICLR, 2021.
Ran Cheng, Yaochu Jin, Markus Olhofer, and Bernhard Sendhoff. A reference vector guided evolutionary algorithm for many-objective optimization. IEEE Trans. Evol. Comput., 2016
Krzysztof Choromanski, Aldo Pacchiano, Jack Parker-Holded Yunhao Tang, Deepali Jain, Yuxiang Yang, Atil Iscen, Jasmine Hsu, and Vikas Sindhwani. Provably robust blackbox optimization for reinforcement learning. In CoRI., 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamies models. In NeurlPS, 2018
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In CoRL, 2018.
Kalyanmoy Deb. Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-IL. IEEE Trans. Evol. Comput., 2002.
Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization.
Comptes Rendus Mathematique, 2012.
Peter I. Frazier. A tutorial on bayesian optimization. CoRR, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. In 00X1;, 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off-policy evaluation. In ICLR, 2021.
Scott Fujimoto and Shixiang Gu. A minimalist approach offline reinforcement learning. In NeurlPS, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
10

---

Published as a conference paper at ICLR 2022
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NewrlPS, 2018.
Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In ICML, 2013.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. In NeurIPS, 2019.
Michael Janner, Igor Mordatch, and Sergey Levine. y-models: Generative temporal difference learning for infinite-horizon prediction. In NeurlPS, 2020.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In NeurIPS, 2020.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurIPS, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurIPS, 2020.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning. 2012.
Hoang Minh Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In ICML, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. In CoRR, 2020.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto multi-task learning, In NeurIPS, 2019.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In NeurIPS, 2020.
Pingchuan Ma, Tao Du, and Wojciech Matusik. Efficient continuous pareto exploration in multi-task learning. ICMI., 2020.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In NeurIPS, 2018.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In ICIR, 2021.
Nikola Milojkovic, Diego Antogmini, Giancarlo Bergamin, Boi Faltings, and Claudiu Musat. Multi- gradient descent for multi-objective recommender systems. In AAAI, 2020.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Comput. Math, 2017.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Çaglar Gülçehre, Konrad Zolna, Alexander Novikov,
Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. CoRR, 2020.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In L4DC, 2021.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. In ICML, 2020.
Martin A. Riedmiller. Neural fitted Q iteration first experiences with data efficient neural reinforcement learning method. In João Gama, Rui Camacho, Pavel Brazdil, Alípio Jorge, and Luís Torgo (eds.), ECML, 2005.
11

---

Published as a conference paper at ICLR 2022
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies scalable alternative to reinforcement learning. In arXiv, 1703.007003 2017.
Julian Schrittwieser, Thomas K Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. In NeurlPS, 2021.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. ICLR, 2016.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. NeurIPS, 2018.
JaungJae Shin and Joongheon Kim. Randomized adversarial imitation learning for autonomous driving. In IJCAI, 2019.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 1991.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference Intelligent Robots and Systems, 2012.
Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning, CoRR, 2019.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. J. Mach. Learn. Res., 2014.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In ICML, 2021.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In ICML, 2020.
C. Yu, G. Ren, and J. Liu. Deep inverse reinforcement learning for sepsis treatment. In ICHI, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. NeurIPS, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model-based policy optimization. CoRR, 2021.
Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, and Xiangyang Ji. Reducing conservativeness oriented offline reinforcement learning. CoRR, 2021.
Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decomposition. IEEE Trans. Evol. Comput., 2007.
Siyuan Zhang and Nan Jiang. Towards hyperparameter-free policy selection for offline reinforcement learning. CoRR, 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held, PLAS; latent action space for offline reinforce- ment learning. In CoRL, 2020.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 CONVERGENCE ANALYSIS OF ALG. 2
We provide proof for Lemma 1.
Proof. For a obtained by solving mina. Di ai
S. S. Viv
1, 2 0, let
I++1 I2 nd;
m Jz 1
ai (1))(
(9)
According to assumption 1:
Li
[(( S fi((2) + V (zz) (x141 xz) +
F141 xz|1
2
Li
- (zx)) de +
!nddel?
(10)
2
Take expectation both sides:
Ez, (x(e411)) S f.(a()))
(.. de + Ex,
[e. 13
- Og V((zz)
2
f((x)) (x)) dz Lan²
[dill + Var(d:)
(((()))) - 7/l 12 + Li7 (I|del + Var(dr))
(DDésidéri, 2012, Lemma 1))
= -(())) n/|d.1 12 Lin² (I|de1
m
a Var Var(V/r..
L
S f(())) 7/1d// 12 +
(1|de)
+
Σ
ai (a.
2
1=1
$((()) (1 L,n?
Lan² o²
||1da112
(11)
2
Hence, we have E., [f([e411 fo(xz) S -(1
L,0²
||dd) 12 +
Lun² 0².
2 2
Let the right term:
Lani
Lom 0² <0
-(m
)||dg1 122 +
(12)
2
2
We have:
| 1
2
1)
(13)
| 2 + 02 Li
When 7/. [[ 2 L.
2 -
(n
L.n
|| 2 Lini 0² < 0, then Ex, [fi -(())) < 0. Hence,
the ascending stage leads to monotonically non-increase sequence, convergence proved.
We provide proof for Theorem 1.
Proof. According Lemma have:
f1(1)) - Ez, (x+(1))) > 2 702
(14)
Take expectation with respect to the solution v, and do telescoping:
Ez, [f((a)] Ezo.en [fr())))] 2 BE,,1 ||| |?] 702
Ex, [f((x]] - [f((12) 2 AE, |||d| | - 700
Eyy (f(((1 ExT-1 [[()) 2 PE |1| dr. 11 -00²
(15)
13

---

Published as a conference paper at ICLR 2022
Note that Ezr,er Ee (fr++)) = Err+ [ff (ZT+ ),, hence
fi( f1(x*)
2 fl(20)
Exr (xT))
(a dominates FT)
T-1
2 T(3
Ex, [lldeell 1111111000 702)
(16)
According to Eq. 16, we can obtain
T
r0?) 1
Ez
|| 112) S T
+
B
(17)
1=0
Let
A
1
70
Se
(18)
T
Rearranging terms, we have
(19)
Bc - 3
By the condition that B - 702 > 0. we have the following:
Lin
Lit²
c(1
0² > 0
2
2
Ti
Li
> 0
2
2
2
17
(20)
+ 0² Li
2 Let < e400 T.p
given e > 0, after T = 0 Be-qo= iterations, 7 Et= Ex, Il|de 3|
A.2 DETAILED RELATED WORK
Model-based Offfine RL. Model-based RL methods are promising candidates for sequential decision- making problems due to their high sample-efficieny and compact modeling of a dynamic environ- ment. Time-dependent linear models and Gaussian processes provide excellent performance in the low-data and low-dimensional scenario. High-capacity models, e.g., deep neural networks, are more successful because they benefit from powerful supervised learning techniques that allow the leverage of large-scale datasets, even in domains with high-dimensional image observations. Although the convenience of working with large-scale datasets, these methods still suffer from the effects of distribution shift and error accumulation of the model predictions (Levine et al., 2020), especially in the offline setting. Existing works explore several classical methods to solve the problem. Dyna-style algorithms (Sutton, 1991) utilize technique named branched rollout to collect short-horizon model rollouts, which alleviates accumulated extrapolation errors. Another method introduces ensemble learning (Chua et al., 2018), i.e., the leverage of multiple dynamics models, against model errors. Other works also incorporate novel concepts into model-based RL, c.g., an energy-based model regularization (Boney et al., 2019). game- theoretc framework for model-based RL (Rajeswaran et al., 2020), meta-learning (Clavera et al., 2018), policy regularization (Berkenkamp et al., 2017), and generative temporal difference learning (Janner et al., 2020). In the offline setting, since the learned model cannot be corrected with additional data collection and training, it is crucial to prevent the policy from visiting -out-of-distribution" states. Recent two works propose incertainty-regulaiii policy optimization algorithms for this purpose (Yu et al., 2020; Kidambi et al., 2020). One adds a soft uncertainty penalty associated with user-chosen weight the reward issued by the model, and the other constructs pessimistic MDP model associated with a hard threshold for terminating the interaction between the policy and the model when the model prediction is inaccurate. (Matsushima et al., 2021) presents a deployment-efficient RL framework to avoid the offline distribution shift issue. (Rafailov et al., 2021) extends an existing model-based offline RL method to image-based tasks. More recently, conservative offline model-based policy optimization (COMBO) (Yu et al., 2021) regularizes the value function on out-of distriiiittt states yielded via interacting with the learned model, which leads conservative estimate of value function for these state-action pairs, without requiring explicit uncertainty penalty. Moreover, a recent method named MuZero Unplugged focuses on using the model-based planning directly for policy improvement (Schritwieser et al., 2021).
14

---

Published as a conference paper at ICLR 2022
Multi-objective Optimization. Multi-objective optimization (MOO), also known as Pareto optimiza- tion, aims to optimize more than one objective function. For a nontrivial MOO problem, no single solution exists that optimizes all objectives simultaneously. In this case, these objective functions are said to he conflicting, and we can find a set of Pareto optimal solutions given different trade-offs among objectives. Existing methods in MOO can be divided into two categories: gradient- based methods and heuristic methods. Multiple gradient descent algorithm (MGDA) (Désidéri, 2012) is an extension of the classical gradient descent algorithm to multiple objectives. It uses an adaptive weighting aggregation of sub-objective gradients to compute the descent direction for all objectives. Because of the ability of fast convergence with theoretical guarantee, MGDA-base been widely used to solve multi-task learning (Sener & Koltun, 2018; 8; Lin et al., 2019) and multi- objective recommendation (Milojkovic et al., 2020). However, MGDA-based methods generate only one solution or finite set of sparse solutions with different trade-offs. In contrast, heuristic methods usually find a Pareto set by various evolutionary algorithms such as the genetic algorithm (NSGA-II) (Deb et al., 2002) and the decomposition boaee evolution algorithm (MOEA/D) (Zhang & Li, 2007). They can find a dense set of Pareto near-optimal solutions by only one time running. But in practice, these black-box optimization methods usually take unacceptable run time when the parameter space is extremely large, e.g., the number of parameters for a deep neural network, or when the function evaluation is computationally expensive, e.g.. popplation-basee -baed training for RL agents. In this paper, our approach borrows the ideas from the heuristic methods to achieve dense set of policies. Moreover. we replace inefficient evolutionary operators with a gradient- based method, reducing the training cost significantly.
methods have
Model-free Offline RL. Model-frre RL directly learn a policy without requiring the learning of an environment model. A direct extension of model-fre RL methods to offline setting (Fujimoto et al., 2019) usually perform poorly due to the data distribution shift. To address this issue, prior model-free free offline RL methods regularize the learned policy to be "close" to the behavior policy either implicitly by conservative Q learning (Riedmiller, 2005; Fujimoto et al., 2019; Kumar et al., 2020; Zhang et al., 2021; Fujimoto & Gu, 2021), or explicitly by direct constraints in state action spaces (Kumar ct al., 2019; Liu et al., 2020; Zhou et al., 2020). Compared with model-based offline RL methods, these model-free methods behave more conservatively and lack broader generalization when the distribution of pre-collected experiences is narrow. Instead of formulating as a single-objective optimization with regularization, a concurrent work (DiME) by Abdolmaleki et al. (2021) takes multi-objeccive perspective for model-free offline RL. It adopts a modified scalarization method to achieve multiple policies on the Pareto front. However, scalarization cannot find any solution on the non-concave parts of Pareto front and thus may lead to sub-optimal performance.
A.3 EVOLUTION STRATEGY
For black-box or noisy objective functions such as RL's policy optimization, computing accurate the gradient is usually challenging. Hence, as a derivative-free optimization method, evolution strategy (ES) has seen recent revival in the RL community (Salimans et al., 2017; Mania et al., 2018). Instead of solving the complicated objective function F() directly, ES solves Gaussian smoothing version: (0) (,,,((o,)) [0 + vE)]] where > 0 denotes smoothing parameter. When is small, the smoothed version F, (B) is close to the original objective (Nesterov & Spokoiny, 2017), and its gradient with respect to policy parameters 0 is defined by (gF,(0) (2x) 4/2 Jge F(B -
vE)e elllledde. Although the gradient is intractable, can be estimated by a standard Monte 1 Σx_ F (0 + IEE)) E, The Monte Carlo estimation has no bias but high variance. Many following algorithms were proposed to reduce the variance while keeping the bias unchanged. In this paper, we adopt an antithetic estimator (Mania et al., 2018), as shown in Eq. (21), which uses the symmetric difference between perturbation E; ~ NNO, I) and its antithetic counterpart -E.
Carlo method: (0),(() = kv
VOF 2kv
Σ 1=1
18 F (I ve:)]
(21)
F(0) = (J'(O)) (00 o:)
(22)
where and (0 v.) results achieved by the policy no one trajectory generated from the environment model. Despite the simplicity, ES achieves competitive performance compared to policy gradient methods (Choromanski et al., 2020).
15

---

Published as a conference paper at ICLR 2022
A.4 SELECTION OF PARETO POLICIES BY FITTED Q EVALUATION (FQE)
Given a pool of Pareto policies, how do we determine the best one for a realistic environment? A straightforward method online policy selection that evaluates the performance of every Pareto policy in the real environment, and then selects the one with the highest return. Most existing offline RL methods (Kidambi et al., 2020; Kumar et al., 2020; Matsushima et al., 2021; Zhang et al., 2021) adopt this method, which however requires many steps of online interactions with the realistic environment, especially when the number of policies to be evaluated is large. We also follow these works and use online policy evaluation to achieve the results in Table 1.
Probably a more efficient alternative is offline policy selection (Voloshin et al., 2019), which is proposed very recently, and aims at choosing the best policy from set of policies, given only offline data. However, offline policy selection/evaluatic is an open problem (Levine et al., 2020), and a number of recent works in RL (Paine et al., 2020; Zhang & Jiang, 2021; Fu et al., 2021) are still exploring new strategies for while combining our method with offline policy evaluation methods can potentially improve the efficiency of P3. To evaluate this strategy, we use FQE (Le et al., 2019) to estimate the performance of each policy. Although there exist various offline policy evaluation methods, we choose FQE for its simplicity and stability in practice. By contrast, other methods have to solve complex estimation problems, such as learning a transition model from visual inputs or estimating the importance weights in a continuous action space (Fu et al., 2021). In our experiments, both P3 and FQE have access to the same offline data.
On an offline dataset D, FQE algorithm trains a critic Qo. for each Pareto policy n; generated by P3, and then uses the critic to estimate the expected value of Tं w.r.t. the initial state so, i.e., VN = E>g--DQQ (so, (so)]]]] which reasonably quantifies the performance of when deployed to a realistic environment. The pseudo-code for FQE algorithm can be found in Alg. 3. FQE can efficiently find near-optimal Pareto policy after only hundreds of epochs, which is demonstrated by the results in Appendix A.7.
Algorithm Fitted Q evaluation (FQE) for Pareto policy selection
1: input: Pareto policy pool P. dataset D, 7 = 0.99, B = 0.995
2: for n. E P (in parallel) do
3: Initialize a critic Qo, and the corresponding target critic O' + Qoi
4: for k ...,..., .....,Keooct Kepoch do
5: Sample a batch {sy, Ty, s
1111=1 Jetthssize from D;
6: Update Qo, by minimizing MSSELos(Qe..l (s, a), 1 (s', (((ss)));
7: d/ B0/ + (1 -)db
> Update target critic Q'
8: V=. EOOD Q (80, (+)))))
9: output: tbest arg max, V..
A.5 ENVIRONMENT MODEL
We train an ensemble of models and pick the best K models based on their prediction error on a hold-out set. In the training phase, each model is optimized independently using the maximum likelihood estimation with mini-batch stochastic gradient descent. In the inference phase, we randomly select one of K models and draw a state-reward concatenation from the resulting distribution, allowing for different transitions along a single episode to be sampled from different dynamics models. The method makes our model more uncertain and noisier than the realistic environment, resulting in more challenging MDP problem. Prior work (Ha & Schmidhuber, 2018; Janner et al., 2019; Yu et al., 2020) demonstrated that the method effectively alleviates the model exploitation issue (Levine ct al., 2020), especially in the offline setting. As shown in Table 3, we list the hyperparameters of environment model for D4RL Gym experiments.
A.6 D4RL GYM EXPERIMENTS
D4RL Gym Datasets. D4RL is widely- benchmark for evaluating offline RL algorithms. It provides variety of environments, tasks, and corresponding datasets containing samples of multiple trajectories generated via behavior policies. hand-designed controllers, or human demonstrators (Fu al., 2020). In the continuous control domain based on the MuloCo simulator (Todorov et al., 2012),
16

---

Published as a conference paper at ICLR 2022
apply subset of datasets, including three environments (halfcheetah, hopper, and walker2d) and five dataset types (random, medium, medium repplaa, expert, and medium-expert), to yield total of 15 benchmark problems, in which random contains 1M samples from random policy, medium contains 1M samples from a policy trained to approximately 1/3 of the performance of the expert, expert contains 1M samples from policy trained the performance of the expert, medium-replay contains the whole replay buffer of policy trained up to the performance of the medium agent, and medium-expert contains 50-50 split of medium and expert dataset (2M samples).
Practical Modifications. We list the techniques adopted by P3 for D4RL Gym experiments.
(1) P3 applies simple behavior cloning approach (Kidambi al., 2020; Matsushima al., 2021; Kumar et al., 2020) to estimate the initial policy тę. The technique effectively alleviates the data distribution shift caused by the difference between the policy-in- -n--trainnng and the behavior policies used to collect the data.
(2) We notice that the states of the high-dimensione complex tasks take the values in broad range, causing the policies only pay attention to particular features of these states. Therefore, we normalize the received state before feeding to policy network: ds = T ( (t,)) where μl, and as are the mean and standard deviation of states, respectively. In the offline setting,
can easily obtain Ns and og by computing the mean and standard deviation of the states in the dataset, similar to the feature normalization in supervised learning. Previous work (Mania et al., 2018; Fujimoto & Gu, 2021) demonstrated that the technique makes the policies more robust to multiple-scale state inputs.
(3) P3 applies the rank-based scale shaping (Wierstra et al., 2014) to compute the ES gradients, which transforms the objective function value via rank-based score function, suppressing the influence of outlier perturbations.
Hyperparameters. We outline the hyperparameters used in the experiments.
Table Hyperparameters of environment model for D4RL. Gym experiments.
Hyperparameter
Value
Number of models/elites
7/5
Structure of hidden layers
MI.P(200, 200) x
Nonlincarity function
Swish
Batch size Optimizer
256
Adam
Learning rate
10
Weight decay
10-5
Holdout ratio
0.1
Table 4: Hyperparameters of P3 for D4RL Gym experiments.
Hyperparameter
Value
Policy network
MLP(32, 32)
Nonlincarity function
Tanh
Step size
HalfCheetah: 2 x 10
Hopper: 2 10-2
Walker2d: 1.5 x 10
Number of perturbations & (ES)
HalfChectah: 30
Hopper: 30
Walker2d: 40
Std. of perturbations (ES)
HalfCheetah: 3 10
Hopper: 1
Walker2d: 2.5 x 10-2
Horizon length H
1000
Number referenge vectors
Number updates T = nTT,, + 27:)
1000 x (150 + 25)
Range (Ta, 76) of reference vectors
(0.1,0.9)
Temperature x
1.5
Constraint threshold
-10 3
Local perturbation
5 x 10-2
17

---

Published as a conference paper at ICLR 2022
A.7 RANK CORRELATION BETWEEN OFFLINE AND ONLINE POLICY EVALUATION P3 D4RL GYM EXPERIMENTS
In Section A.4, in order remove the extra computation required by online policy evaluation during the test, we instead use Fitted Q Evaluation (Alg. 3) to provide metrics for policy selection. Here we take closer look the correlation between the policy ranking by the online evaluation and the policy ranking by FQE.
First, we rank all the policies the P3 pool by the two metries and show their generated rank orders (in an ascending order) for every policy in Fig. 5. On three datasets, the two rank orders show strong correlation, which indicates that the FQE based offline policy evaluation provides an accurate approximation to the realistic return of P3 policies, and thus can be used to improve the test efficiency of P3. In addition, we report the Spearman's 's rank correlation ("Spearman" 's p²) and Kendall rank correlation ("Kendall'ss r") to quantitatively measure the correlation between the two rank orders. We report the results in Fig. 6, which demonstrates a strong correlation coefficient close to 1.0 for all datasets. Therefore, replacing the online evaluation with FQE is an efficient solution preserving the original rank orders of P3 policies.
hulkdhedat--raeee (Rank Don-
holchectat- rnd www.riodddrrrrooooooo crprypay (hakk Calw
hepper-meduu mesella Mark Car 500
Haakk clpoiious accoringla FOR aduues
10% ocdeee aoolding
mam www.
Figure 5: Scatter plots of the ordinal rankings of the FQE value estimates vs. the true values.
Park con elation between the ondinal rankings of the FOE value estimates and the the values
EN2.021
P-6.004
2-0-300 537
p-12002
For
Win
bils oyn Bindrran Dussnsts
rankings
the Due values
EEL
P-1.00
2.64
(e DUB
Wer
HG/H
W
Wim
D99 4p www.heaaaaaccooooooo
Figure 6: Rank correlation of our FQE algorithm for D4RL Gym benchmark datasets. HC HalfChectah, Hop Hopper, Walker, random, medium, medium replay. expert, me medium medium-expert.
18

---

Published as a conference paper at ICLR 2022
A.8
RESULTS ON THE D4RL GYM EXPERIMENTS
MOPO
MORcL
TD3+BC
P3 (OURS)
tafthhedss--raccco
hogpar-anddom
walkar2d-randdam
8003
2003
1250
1200
Average Rotum
1503
Average Rottmm
4003
Average
1000
2002
500
820
500
1000
200
400
500
500
1000
400
B00
500
1000
Traniing Epocha
Truning Epoccha
Traning Eppccha
Wel 20-mmeiimmm
10000
Amenge Raturn
Avergga Return
0003
4000
1003
1000
2000
1000
400
1000
Traming Epoche
Touning Epocches
Truning Epocct
hopper-madiu www.ddd----ppppe eapay
momppay
10000
4003
4200
Amarge Ratunn
8000
2000
Nvsrage Return
2000
6000
Average
2005
2000
1000
820
500
1000
200
400
500
600
1000
Traming Epochs
Eppcces
Traning Epoche
has twwfichhetsshheee
hoopae-nsppart
warbar2d--epper
10000
Amergga Raiten
Averaga Return
2000
2000
1000
600
500
1000
200
400
1000
500
1000
Traming Epochs
Traming Epoches
Traning Epoches
happer-medkumm expertt
Vis wwrkerdd-meduu--appee
4500
8000
10000
Anstage Rahan
2000
Aveeage Betum
4200
Average
2000
5000
1000
500
1000
200
-100
1000
Training Epocho
Training Epoche
Figure 7: Learning curves for the D4RL Gym experiments. Curves are averaged over the 10 evaluations and S seeds, and the shaded area represents the standard deviation across seeds. P3 performs better compared to two model-based offline RL methods while exhibiting similar performance the statc-of-the-an model-free method (TD3+BC), Note that we sweep the hyperparameters of MOPO and MOReL. and choose the best combination for each task.
19

---

Published as a conference paper at ICLR 2022
Mamber trajettryy
Qumber
2000
200
200
hofftheeth-meddam
hopper-edium
walker2d medium
Nomter (ouaec
Nomber of trueec
100
Datumm tnapettrry
Patiumal trajectryy
(aetum
haaChoothh mepia
Teoppr mozium
naltie mediam
Manbe tratetoon
Rancee toapect
100
Retum off ti
Redum eralectoryy
Retur of trojectoryy
tetopet sogpert
lan wocpert
Bumber trapetors
bumber trapetory
750
300
250
2000
5033
7500
10000
2008
3002
2030
3303
Rotum of oppcoyy
Rotum of trajectoryy
Roturn of trajecoory
-ofcheesah-moluu urt expert
hoppenmedlum expert
wolker2d neplum-expeit
1000
200
750
20
Numttar
Number
130
2000
10303
1600
2002
300
5000
1000
Enjpetory
Retum Erajectryy
trajecbory"
Figure 8: The histogram returns from low-quality to high-quality datasets. Low/medium- datasets in the D4RL Gym benchmark are "imbalanced" and degrade the training of cnvironment models. P3 (ours) exhibits more advantages on these datasets.
20

---

Published as a conference paper at ICLR 2022
A.9 ABLATION STUDY
For clarity in ablation study, we provide implementation details on the scalarization method (Boyd & Vandenberghe, 2004, Chapter 4.7). The method maximizes weighted max (T)) where w E [0, I| denotes the weight assignment. By solving multiple single-objective optimization with different weight assignments, we can obtain multiple policies with different trade- offss as shown in Fig. 9. For the experimental results in Table 2 and Fig. 9, we train 5 policies from scratch using the scalarization method with random weights and report the best score among them.
of objectives:
(OURS)
Hogpan--andon
Wükker2--unddn
4000
2000
800
Bonum dhe MDP mocial
the
200
2000
100
890
1000
-2303
Negutive uncoraai of Ihe MDP noddd
Nagalioa undurtardty the MDP model
Nogalino uncertanng the MDP
hodlcheedebvraad..cmm
Hoppe--rddaam
mocel
Rutum the MDF mossal
Toatamn the MDD mecel
1000
8000
4200
-1500
1220
-00
-150
-125
-00
-25
50
1800
18000
11400
1200
Negaliow uncerbenty of
Negalion uncerterdy
maddel
Nagetine uncurlaary
the modsl
Ratunn the MDP model
2000
1000
1500
500
MHBP model
Negathve
MGP
unceraainy MDP
Figure 9: Ablation study. Pareto front comparison between the scalarization method (red circles) and our (blue circles) on low/medium--qulity datasets. We train policies from scratch using scalarization with random weights. P3 also adopts reference vectors. Hence, we remark that P3 takes the roughly same run time as scalarization but achieves dense set of policies and better Pareto front approximation.
21

---

Published as a conference paper at ICLR 2022
A.10
EFFECTIVENESS PARETO POLICY POOL
Hopperrandom
3333
4000
Ruturn in the RID mude
1S0
2008
1000
402
240
1208
1200
893
300
722
1100
-2000
1100
-1400
-390
Negalive
Negative unteratity
sofehhetallmmeiuu
Hospes-modium
Waleer2-meddun morriim
2000
2000
4200
9000
2409
NDP
4238
NEP
1000
1030
Nour 11000
603
1200
1200
200
Ronur
3190
1MM
-1583
-1083
-500
400
150
-125
-75
-2099
-2505
-2000
-1500
Nogative uicoraiity ta of the MOP model
Megacive uncertor rit the HDP model
Negative uncortaiity of the MDP model
iesticeetthh-ee med
Wellbardd
4000
ta5.m the VOP mode
5000
1000
2209
Budum the MQT 22
900
2003
2400
the
5000
1399
2003
3030
300
2003
3003
Negutive by MDP
Negattive
the MDP
Negative uncertalnty the MDP model
Halcheetah expert
Hopo
Wokker2d.xpertt
Natum 000 model
$0.0
1,00
Iutur the RDP modkl
2399
1200
DANU the MCP
1539
2000
400
2000
2005
1000
200
279
2500
2005
3333
1000
Nogative uncoralitt
Negacive
Negetive uncersaiity
3000
3599
Intre MDP roadd
2000
2700
2030
Bocuat 1011 MDP roodd
1200
330
2538
4803
1000
3100
tethe
2000
2238
3000
05.0
2000
2087
-3000
-100
-200
-2.00
-2000
Nagebbvv UnIEMTARrty on of Uhe MLF modill
MMrgativ unicartal Fiyy of MDM mocil
Nagaltae uniirtaainty of Uhe S9LF model
Figure 10: Contour plots on how model-basee offline RL's performance (color bar) varies with different trade-off's between the model return (x-axis) and uncertainty (x-axis). Mcanwhile, red circles denote the pool of policies generated by P3.
22


Reviews:
[{"review_id": "g5awlopLCxh", "reviewer": "Reviewer_xWzd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-motivated, and the idea to approximate the Pareto front between model-estimated performance and uncertainty is novel to my knowledge. The algorithmic choices (e.g. local extension to reduce computational cost of training from scratch) also make sense. Algorithm 2 is intuitive, although not particularly novel as it is essentially just applying MGDA with gradients estimated by ES, plus some small modifications to account for the constraint.\n\nThe performance of P3 on the D4RL tasks is good. UWAC is clearly better for cases where there is expert data, but for the lower-quality datasets, P3 handily outperforms the baselines. I also appreciate that the authors have taken efforts to tune the MOPO baseline, in some cases even finding hyperparameters that work better than the original paper’s results.\n\nMy main concern is regarding the fairness of the empirical evaluation, where P3 arrives with a large set of policies to be evaluated, unlike the other methods which produce only one policy (or one for each hyperparameter setting). The authors explain that the computational cost of running P3 is similar to the cost of tuning other methods, which I can believe, but even if you don’t tune P3’s hyperparameters, you still have to draw samples to evaluate each policy, and in the offline setting we do not want to be drawing so many additional samples. Thus, while the authors frame the development of many diverse policies as a strength of their method, I feel that it is only a strength when paired with a strategy to choose from this set of policies (i.e. off-policy evaluation), which is not presented in this paper. At the very least, I feel that the authors should control for the number of samples used at model selection time when comparing against prior work, as this is arguably more of a bottleneck than runtime in offline RL. Currently P3 has an advantage at evaluation time because you get to try many policies and take the max.", "summary_of_the_paper": "In model-based offline reinforcement learning, a dynamics model is trained from the dataset and subsequently used to produce a decision-making policy which attains high reward (according to the model). However, one must also take care not to allow the policy to exploit the model’s inaccuracies, so we also want it to visit states and take actions which have low model uncertainty. Prior work has managed this tradeoff by linearly combining the two objectives.\n\nThis work proposes instead to find many diverse policies along the Pareto front of these two objectives. The algorithm, Pareto policy pool (P3), proceeds as follows:\n1. Generate reference vectors which quantify tradeoffs in the Pareto front.\n2. In parallel, find a Pareto-optimal policies for each reference vector using “Algorithm 2” to solve a constrained bi-objective optimization problem, where the constraint ties the solution to the reference vector. Algorithm 2 begins by finding a feasible solution and then applies MGDA to improve the objectives.\n3. “Local extension”: Each reference vector is perturbed in two opposing directions, and then each perturbed vector is further optimized via Algorithm 2, with intermediate policies being added to the policy pool. (This is just an optimization to reduce the computational cost.)\n4. At test time, each policy in the pool is evaluated, and the best one is selected.\n\nThe paper includes some theoretical analysis of Algorithm 2, showing convergence to an approximate stationary point. Empirical evaluation is performed on the D4RL benchmark, where P3 achieves good results, particularly on the datasets generated by lower-performing policies.", "summary_of_the_review": "I think the high-level strategy pursued by the paper is reasonable, and the results are promising. Thus, the paper is a useful contribution overall. However, I have reservations about the setting of the empirical evaluation, and think more care could be taken to ensure a fair comparison.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RifCHyVIPCF", "reviewer": "Reviewer_jeW5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths** \n\nI think the paper has the following strengths:\n- The paper outlines a novel method, with a thorough and detailed description, for offline model-based reinforcement learning that reformulates a significant problem in a bi-objective manner. \n- The paper provides a thorough explanation of the various components of their method, including algorithms, gradients and relevant parameters and hyper parameters. \n- The method shows general outperformance compared to other methods in lower data quality settings in the D4RL benchmark.\n\n**Weaknesses** \n\nI think the paper could be strengthened by providing more detail on the following:\n- Clarifying how the policies for the score in Table 1 were chosen. Given that you have a Pareto front, did you choose the one with the highest score on the given benchmark? If so, can you provide more detail on how many policies end up in your Pareto front. I am potentially concerned that creating many additional could add significant computation overhead when measuring performance if all policies are tested, so further details could help.\n- You mention that you use ES for gradient estimation for the MGDA based optimization. Did you consider other methods and how do you think gradient estimation will impact the quality of solutions along the Pareto frontier that P3 can find?\n- Can you provide more detail on how you chose the number of reference vectors for your bi-objective optimization (\"n\" in Line 5 Algorithm 1). Figure 5 has many (red-dot) Pareto optimal points, so it appears that many reference vectors were chosen. Related to the first question, I am trying to get a sense of the scale of the optimization problem.\n- Related question to Figure 5: Based on my assessment it appears that not all red dots representing Pareto policies are non-dominated along the two objective dimensions. Could you clarify if this is the case? \n-  Could you comment on why you think UWAC outperforms P3 in the medium-expert and expert cases?\n\n**Additional Remarks**  \n\n- Reproducibility: While hyperparameters and settings are provided in the appendix, there is no mention that I can find of code release. Could you provide more detail on whether you plan to release any source code?\n\n", "summary_of_the_paper": "The paper proposes a new model-based offline reinforcement learning, where a pool of Pareto optimal policies is trained on a model of the environment provided by offline RL data. First, a model of the environment is trained using supervised learning from the dataset of logged experiences and subsequently a set of Pareto optimal policies is trained along the dimension of return in the MDP model and uncertainty in the MDP model. The authors argue that providing a Pareto optimal set of policies is superior to prior methods that relied on regularization of the two objectives into a combined metric, as having a Pareto optimal set of policies enables one too explore a range of behavior that is inherent with trade-off between model return vs model uncertainty in offline model based RL.\n\nThe authors then describe their method, Pareto Policy Pool (P3), which addresses the bi-objective optimization problem two stage method. Initially, the authors initialize a set of reference vectors along the Pareto frontiers to ensure a diverse set of Pareto policies can be found. Subsequently, the authors apply \"Local Pareto Extension\" where first a given policy is corrected to a desired range along the objective dimensions provided by the reference vectors. Once the policies are within the desired range, the method applies MGDA based optimization to further improve the performance of the policy in a Pareto optimal way (ascension stage). In order to alleviate computational cost of training multiple policies, the authors apply a \"Pareto extension\" method to initialize new policies along the Pareto frontier. \n\nThe authors then perform experiments on D4RL based benchmarks and compare their method to other literature methods and claim outperformance, particularly in lower quality dataset settings. The authors also perform an ablation study pertaining to the various components of their method, as well as an analysis related to why it is challenging to obtain optimal trade-offs in model based offline RL.", "summary_of_the_review": "Overall, I think that the paper provides a novel and valuable method with a satisfactory set of experiments to support the overall claims. I have outlined some of my questions and concerns above and think that further clarification will give me a better sense to re-assess my recommendation during the discussion period. \n\n----- Discussion Period Comments -----\n\nI am adjusting my score based on the authors' response to my questions, their modifications in the paper draft and response to the feedback of other reviewers. I will follow up with specific technical questions in the individual response. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MVk7dYolUE", "reviewer": "Reviewer_EL3w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In terms of strengths, this paper presents a novel perspective on navigating the trade-off between model-based return versus uncertainty, for offline RL. The experiments are thorough in comparing against a range of baselines, including both model-based and non-model-based offline RL algorithms. Figures 1 and 2 do a nice job of visualizing the approach and the results.\n\nIn terms of weaknesses, a main weakness is that P3 does not outperform existing approaches for good-quality offline RL datasets: in Table 1, for the medium-expert and expert datasets, it consistently performs substantially worse then the UWAC baseline. Why is this the case? I would expect P3 to perform at least as well, if it is actually recovering the true Pareto front. Another weakness is that the experiments are on a relatively small number of domains—only on three MuJoCo domains, out of the many domains available in D4RL.\n\nAlso, in practice, trying out every single (fully-trained) policy in the real environment is still expensive. This should be acknowledged in the paper. If I understand correctly, the results in Table 1 for P3 show the best performance across all policies found. This is significantly more computationally expensive than prior approaches (because at least several of the Pareto policies require separate training from scratch), and also requires more runs on the real environment. So it is not entirely a fair comparison. With that said, I appreciate the ablation studies that make a more fair comparison against ablations of P3.\n\nFinally, the algorithmic design decisions should be motivated more clearly. In particular, when computing the similarity metric between the reference vector and the objective vector in Equation (6), why is KL-divergence used instead of a dot product? The dot product is the typical way of measuring similarity between vectors (as is the case here), whereas KL-divergence is meant to be used for distributions. Also, in Equation (4), does the uncertainty need to be exponentiated and tempered?\n\nAdditional comments:\n- The Related Work and Conclusion should be in the main paper, not in the Appendix.\n- In the Related Work, it would be useful to also discuss non-model-based approaches for offline RL. It is also worth mentioning DiME [1], a recent approach that, similar to P3, also takes a multi-objective perspective for offline RL (although in a non-model-based setting), and obtains state-of-the-art results by doing so.\n- There are several typos and grammatical errors. e.g. \"computational forbidden\" at the bottom of page 3\n- In the last paragraph of Section 3.1, the writing is overly verbose and unclear. For instance, it mentions twice that it is inefficient to train multiple policies from scratch to obtain the Pareto front.\n- Section 4.2 claims that scalarization does not perform well because it only finds policies on the convex part of the Pareto front, but the Pareto fronts found by P3 (in Figure 8) don’t look concave. Instead, I think the different scales of the objectives is the issue, because only five weight settings for linear scalarization are tried.\n- In A.8, for walker2d-random and walker2d-medium, why do these not look like Pareto fronts? In other words, why is there not a conflict between optimizing for model-based return and minimizing model uncertainty for this task?\n\n[1] Abdolmaleki et al. On Multi-objective Policy Optimization as a Tool for Reinforcement Learning. 2021.", "summary_of_the_paper": "This paper proposes a model-based approach for offline RL that is inspired by multi-objective RL. The approach, called P3, finds a Pareto front of policies that trade off between obtaining high return (with respect to the model trained on the offline dataset) and minimizing the model's uncertainty. To obtain this Pareto front, the approach has two stages: 1) find a few spread-out solutions on the Pareto front using a gradient-based method, and 2) initialize with these solutions to find optimal solutions for the neighboring regions of the Pareto front. The empirical evaluation shows that P3 outperforms existing approaches when the offline dataset is of low or medium quality.", "summary_of_the_review": "This is an interesting and relevant direction to study, with promising empirical results for tasks with low- and medium-quality datasets. However, the algorithmic design decisions and empirical results (for good-quality datasets) require more explanation, and the presentation requires polishing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AgyDRZ-6hzS", "reviewer": "Reviewer_ngyt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written aside from a few typos here and there. The figures are well made and clearly convey relevant information about the problem, method, and experimental results.\n- The problem identified by the authors is highly relevant to the RL and offline RL community. While this problem (and similar ones) has been identified before, there is relatively little prior work that has addressed it despite it being an important problem.\n- The proposed method seems technically sound and the techniques for multi-objective optimization in RL seem to be of interest to the community. The theory is not particularly new, but it does validate certain parts of the algorithm, which is nice.\n- The experiments are extensive and compare against a large pool of recently proposed methods. P3 appears to very clearly out-perform the other methods when low-medium quality trajectories are given in the dataset. The ablation study and heatmaps are particularly informative to understand exactly what the algorithm is achieving.\n\n\n\nWeaknesses:\n- The algorithm is purported to apply to the case of offline RL, yet there is a critical concession in practice: one must have online access to the simulator/domain to actually select the best policy from the frontier -- the algorithm is unable to determine this with offline data alone. This is concerning for two reasons. (1) In my opinion, this trivializes the model/policy/hyperparameter selection problem since we can simply try many different hyperparameters for a given algorithm and see which one is best directly, just as one can in standard supervised learning e.g. by cross-validation or hold-out sets. (2) I do not see an easy way to remedy this problem (I am certainly open to and curious about any suggestions) because Figure 5 shows that a large portion of frontier policies still have low reward so random sampling probably won’t work reliably and something like OPE requires hyperparameters of its own.\n- Given that some online access is permitted to do the final selection for P3, I think a key experiment that is missing is a comparison with hyperparameter sweeps of the benchmark algorithms like MOReL, MOPO, UWAC, etc. in each domain. That is, if one grids for example the regularization weight $\\lambda$ mentioned in Section 2 and picks the best for each domain from online evaluation, how does this compare with P3? Otherwise it seems that P3 has an unfair advantage. I think if such a comparison is made, then the previous issue is not as significant.\n\nMinor comments and questions:\n- Equation (4): What is the motivation for the exponentiated uncertainty? How do you set kappa?\n- Can you define “non-convex regions” on the Pareto frontier?\n- Fig 5 is informative to some extent, but I am confused about how to interpret the full frontier. Clearly there are some points in high reward regions, but also many in seemingly arbitrary areas. Can additional discussion be provided to aid interpretation? Why are some of the points very dense while others are not?\n- Some important sections like conclusion and related work seem to be in the appendix.\n- For the experiments, how many policies are in the set $\\mathcal{P}$ in the end?\n", "summary_of_the_paper": "This paper proposes a new algorithm for model-based offline reinforcement learning in which the learner attempts to optimally balance maximizing the reward under the learned model and minimizing the uncertainty of the model. The algorithm is motivated by the fact that offline RL methods suffer from distribution shift that often results in policies/predictions that are overly optimistic about highly uncertain areas of the state-action space. The authors’ proposed method attempts to address this problem by identifying a relevant collection of policies lying on the Pareto frontier that trades off these two criteria. The underlying supposition is that a well-performing policy should be among this collection of “non-dominated” policies on the frontier. Specialized subroutines are proposed in order to make sure the collection is diverse. Some theory is presented to validate certain aspects. Extensive experimental results support that the method performs well in relevant benchmarks compared to a number of state-of-the-art methods.\n", "summary_of_the_review": "In summary, I believe there are a lot of positives about this paper. It identifies and studies an important problem in the literature. The technical contributions and main ideas are interesting and appear to be new to this problem setting. The experiments are reasonably thorough. Unfortunately, I feel that the mentioned weaknesses (requiring online access, lack of comparison against hyperparameter sweeping other algorithms) are significant and leave open questions about the practical applicability of the proposed method and comparison to prior algorithms.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Previous approaches to model-based offline RL require carefully tuning the trade-off between model return and uncertainty. The authors propose an approach that produces a diverse pool of policies on the Pareto front of this tradeoff. On the D4RL offline RL benchmark, P3 outperforms competing approaches when the experience is collected with low or medium return policies.

Before the rebuttal, reviewers identified the following primary concerns:
* The experimental evaluation of P3 uses many policy evaluations to select the policy, which results in an unfair comparison with existing methods.
* P3 underperforms existing methods on some datasets. Why?

Overall, reviewers were satisfied by the response and raised their scores accordingly. The authors responded by including a modification of P3 that uses FQE to select the policy for evaluation, resolving the first concern. The authors explain that P3 underperforms on high return datasets because it splits its updates across the pool of policies. The authors state, "We believe (and in theory it does hold) that P3 can achieve the same performance of UWAC on high-quality datasets, if provided with more computational budget." I suggest that the authors conduct at least one experiment to verify this claim.

The proposed idea is interesting and the revisions the authors have made resolved the primary concerns from reviewers, so I recommend acceptance. The reviewer/author discussion has many substantial points that I recommend the authors integrate into the revision.
Decision: Accept (Poster)