PARETO POLICY POOL FOR MODEL-BASED OFFLINE REINFORCEMENT LEARNING
Year: 2022

Abstract:
Online reinforcement learning (RL) can suffer from poor exploration, sparse reward, insufficient data, and overhead caused by inefficient interactions between an immature policy and a complicated environment. Model-based offline RL instead trains an environment model using a dataset of pre-collected experiences so online RL methods can learn in an offline manner by solely interacting with the model. However, the uncertainty and accuracy of the environment model can drastically vary across different state-action pairs, so the RL agent may achieve a high model return but perform poorly in the true environment. Unlike previous works that need to carefully tune the trade-off between the model return and uncertainty in a single objective, we study a bi-objective formulation for model-based offline RL that aims at producing a pool of diverse policies on the Pareto front performing different levels of trade-offs, which provides the flexibility to select the best policy for each realistic environment from the pool. Our method, “Pareto policy pool (P3)”, does not need to tune the trade-off weight but can produce policies allocated at different regions of the Pareto front. For this purpose, we develop an efficient algorithm that solves multiple bi-objective optimization problems with distinct constraints defined by reference vectors targeting diverse regions of the Pareto front. We theoretically prove that our algorithm can converge to the targeted regions. In order to obtain more Pareto optimal policies without linearly increasing the cost, we leverage the achieved policies as initialization to find more Pareto optimal policies in their neighborhoods. On the D4RL benchmark for offline RL, P3 substantially outperforms several recent baseline methods over multiple tasks, especially when the quality of pre-collected experiences is low.

heading: 1 INTRODUCTION
text: Offline reinforcement learning (offline RL) (Levine et al., 2020) or batch RL (Lange et al., 2012) can train an agent without interacting with the environment by instead using pre-collected experiences on other agents/policies. Recently, offline RL has been attracting growing interest due to the availability of large-scale datasets of diverse experiences in many RL applications, e.g., autonomous driving (Shin & Kim, 2019), healthcare (Yu et al., 2019), robot control (Schulman et al., 2016), etc. However, RL algorithms developed for the online/interactive setting usually perform poorly in the offline setting (Fujimoto et al., 2019; Janner et al., 2019) due to the data distribution shift caused by (1) the difference between the policy-in-training and the behavior policies used to collect the data; and (2) the difference between the realistic environment in which we will deploy the policy and the environments used to collect the data. These differences can result in function approximation error and biased policy learning (Levine et al., 2020). To address these challenges, model-based RL approaches (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021; Yu et al., 2021) firstly learn an environment model from a dataset of logged experiences using supervised learning and then conduct online RL interacting with the model. The learned environment model fully exploits the pre-collected experiences and can avoid/reduce the costly interactions with the realistic environment required by RL, hence improving the sample efficiency.
That being said, due to the large size of state-action space, model-based offline RL approaches can still suffer from “model exploitation” (Levine et al., 2020): when the dataset does not contain
sufficient samples for some state-action pairs, the epistemic uncertainty of the model on these “outof-distribution” pairs can lead to a poor policy in offline RL. For example, an RL agent may easily exploit the model by repeatedly visiting the out-of-distribution states where the model erroneously issues higher rewards than the true environment dynamics, and thus the RL objective is biased by overly optimistic evaluation of the agent’s performance. A widely-studied strategy to tackle this problem is applying regularization to the reward function, which keeps the model uncertainty small when maximizing the reward issued by the model (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021). By carefully tuning the regularization weight, it can achieve preferable trade-off between the model reward and model uncertainty. However, it is usually challenging to tune the weight without access to the realistic environment that the learned policy will be deployed to. Moreover, even with access to the realistic environment, hyperparameter tuning methods such as Bayesian optimization (Frazier, 2018) need to run multiple instances of offline RL and can be computationally expensive. In addition, the offline-trained policy may not be directly generalized to different environments. In this paper, we address a challenging problem, i.e., how to balance between the model reward and model uncertainty during offline RL so it can produce diverse policies that can be adapted to different realistic environments?
Instead of formulating the problem as a single-objective optimization with regularization as in previous works, a primary contribution of this paper is to treat the model reward/return and model uncertainty as two separate objectives and develop an efficient bi-objective optimization method producing a pool of diverse policies on the Pareto front, which correspond to different trade-offs between the two objectives. Therefore, when deployed to a new realistic environment, we can accordingly choose the best policy from the pool. Our method is called “Pareto policy pool (P3),” and an example of the policies achieved by P3 is shown in Fig. 1. When deployed to the realistic environment, Pareto policy 1 is overly optimal on the model return, so it runs fast at the beginning but quickly falls to the ground due to the aforementioned “model exploitation”. On the contrary, Pareto policy 4 favoring small model uncertainty is overly conservative and keeps standing still for 1000 time-steps because it avoids taking exploratory actions that potentially increase the uncertainty. As expected, Pareto policy 2&3 with the more balanced trade-off between the model return and uncertainty perform better and achieve higher returns in the test environment. These results imply that model-based offline RL’s performance significantly relies on the trade-off between model return and uncertainty in the optimization objectives. However, it is usually challenging or intractable to determine the optimal trade-off before deployment and control it during training. Moreover, for existing methods adopting a regularized single-objective, i.e., scalarization (Boyd & Vandenberghe, 2004, Chapter 4.7), even trying all possible regularization weights cannot fully recover the Pareto front and thus cannot guarantee to find the optimal trade-off. For example, in Fig. 1, by running multiple instances with different regularization weights, MOPO (Yu et al., 2020) and MOReL (Kidambi et al.,
2020) can only generate a few separated solutions, and it is difficult to find one with advantageous trade-off among them. In contrast, our method aims at efficiently generating a rich pool of diverse and representative policies covering the entire Pareto front without tuning the trade-off during training. Thereby, when deployed to a new realistic environment, the best policy for the new environment can be efficiently selected from the pool. Hence, P3 provides a simple and principal approach that addresses the two major challenges in model-based offline RL, i.e., “model exploitation” and generalization to different unseen states in order to achieve high returns.
Due to the complicated shape of the Pareto front that is unknown during training, finding a pool of diverse policies covering the whole Pareto front raises several non-trivial algorithmic challenges: (1) How to find policies located at different regions of the Pareto front associated with different levels of model return-uncertainty trade-off? (2) How to avoid training each policy from scratch and reduce the computational cost linearly increasing with the number of policies? Inspired by recent works in multi-objective optimization (Cheng et al., 2016; Ma et al., 2020; Xu et al., 2020), we explore different regions of the Pareto front by generating multiple diverse reference vectors in the bi-objective space, each defining a constrained bi-objective optimization whose solution resides in a local region of the Pareto front. By solving these constrained bi-objective optimization problems, we can obtain a diverse set of policies covering the whole Pareto front. For solving each problem, we extend MGDA algorithm (Désidéri, 2012) to be a two-stage gradient-based method that provably converges to the Pareto front region targeted by the reference vector. In order to achieve more policies on the Pareto front without linearly increasing the cost, we start from the previously obtained policies for initialization and explore their neighborhoods on the Pareto front by perturbing their corresponding reference vectors, resulting in a dense set of Pareto policies in each local region.
In experiments, we evaluate P3 and compare it with several state-of-the-art model-based/free offline RL methods on the standard D4RL Gym benchmark (Fu et al., 2020). P3 achieves the highest average-score over all datasets and significantly outperforms the baseline methods in 5 out of the 9 low/medium-quality datasets, showing the advantages of P3 on learning from non-expert experiences. We also present a thorough ablation study to identify the most important components in P3.

heading: 2 RELATED WORK
text: Due to lack of space, we focus our discussion here on directly related works and present a more detailed overview of related work in Appendix A.2. To address the aforementioned “model exploitation”, recent works rely on applying uncertainty regularization to the model return (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021) which can be difficult and costly to tune the weight of regularization. In contrast, our work reframes the policy learning under the environment model as a bi-objective optimization problem and produces a diverse set of policies on the Pareto front performing different levels of model return-uncertainty trade-off, which provides flexibility to select the best policy in the inference stage.

heading: 3 PRELIMINARIES
text: We consider an episodic Markov Decision Process (MDP)M = {S,A, P, r,H, ρ0}, where S is the state space, A is the space of actions, P is the transition probability: S ×A → ∆(S) where ∆(S) is a probability distribution over S , r : S ×A → R is the reward function so r(sh, ah) is the immediate reward for taking action ah at state sh, H is the horizon of the process, and ρ0 is the distribution of the initial state s0. RL aims at learning a policy π : S → A maximizing the expected return in Eq. (1), where π in this paper takes a deterministic action ah based on the state sh.
max πθ Rρ0(πθ,M) = Es0∼ρ0,πθ [ H−1∑ h=0 r (sh, ah) ] . (1)
In model-based offline RL, the agent instead interacts with an environment model rather than the realistic one. We train an environment model M̂ = {Ŝ,A, P̂ , r̂, H, ρ̂0} using a pre-collected dataset D , {(sh, ah, sh+1, rh)|πb} of experiences by behavior policies, hand-designed controllers, or human demonstrators. By interacting with the model, online RL methods can learn in an offline manner. However, when D does not contain sufficient samples for some state-action pairs, the epistemic uncertainty of the model on these “out-of-distribution (OOD)” pairs can result in a poor policy. For example, an RL agent may easily exploit the model by repeatedly visiting the OOD states where the model erroneously issues r̂ higher than the true environment reward r and thus the RL objective is biased by overly optimistic evaluation of the agent’s performance. A widely-studied
strategy to tackle this problem is applying regularization to r̂, which keeps the model uncertainty small while maximizing the model reward (Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021). A practical implementation of the regularized reward function is developed by (Yu et al., 2020): r̃h = r̂h − λu(ŝh, ah), where u(ŝh, ah) denotes the estimation of the model uncertainty at the state-action pair (ŝh, ah) and λ controls the trade-off between r̂ and u, which has to be carefully tuned in practice. Based on the regularized reward function, Yu et al. (2020) proposed a modified policy optimization objective:
max π R̃ρ̂0(π,M̂) = Es0∼ρ̂0,π [ H−1∑ h=0 (r̂ (ŝh, ah)− λu (ŝh, ah)) ] . (2)
Despite being intuitive, this method’s performance is sensitive to the regularization weight λ (Kidambi et al., 2020; Yu et al., 2021), and tuning λ is usually challenging without access to the realistic environment that the learned policy will be deployed to. Moreover, even granted the access, hyperparameter tuning methods such as Bayesian optimization (Frazier, 2018) require running many instances of offline RL, which can be computationally prohibitive.
Instead of optimizing a single regularized objective, our “Pareto policy pool (P3)” proposed later treats r̂ and u as two separate objectives and develop an efficient bi-objective optimization method that does not need to tune the trade-off deliberately but produces a pool of diverse policies, which are learned under different trade-offs between the two objectives. Thereby, when deployed to a realistic environment, the best policy can be chosen from the pool.

heading: 4 PARETO POLICY POOL FOR MODEL-BASED OFFLINE RL

heading: 4.1 PROBLEM FORMULATION
text: In order to estimate the model uncertainty accurately and alleviate the model exploitation problem, we follow previous works (Janner et al., 2019; Yu et al., 2020; Kidambi et al., 2020; Rafailov et al., 2021; Yu et al., 2021) and construct a bootstrap ensemble of K environment models {M̂i}Ki=1. Each model M̂i = {Ŝ,A, P̂ i, r̂i, H, ρ̂0} is a two-head feed-forward neural network that takes a state-action pair (ŝh, ah) as input and outputs the mean µi and standard deviation σi of [ŝh+1, r̂h], i.e., the next state concatenated with the reward. More details about our model are given in Appendix A.5. As demonstrated in Yu et al. (2020), this ensemble is effective in estimating the model uncertainty as the maximal standard deviation over all models, i.e., u(ŝh, ah) = maxi∈[K] ‖σi(ŝh, ah)‖2. Moreover, by randomly selecting a model in each step to provide the reward r̂(ŝh, ah) = r̂h, we can effectively mitigate the model exploitation problem. Unlike previous works combining r̂ and u as a single objective, we treat them as two separate objectives and aims at solving the bi-objective optimization below.
max θ Jρ̂0(πθ,M̂) = max θ (J r̂ρ̂0(πθ,M̂), J u ρ̂0(πθ,M̂)) T, (3)
(J r̂ρ̂0(πθ,M̂), J u ρ̂0(πθ,M̂)) T = Es0∼ρ̂0,π [ H−1∑ h=0 (r̂ (ŝh, ah) , exp (−u (ŝh, ah) /κ))T ] , (4)
where κ is a temperature applied to u (ŝh, ah). For simplicity, in the rest of this paper, we remove ρ̂0 and M̂. In Eq. (3), J r̂ aims to maximize the expected model return, and Ju is designed to minimize the expected cumulative model uncertainty. However, the bi-objective optimization naturally has multiple (up to infinite) optimal solutions instead of one single policy and each optimal policy performs a different level of trade-off between the two objectives. For example, a policy favoring small model uncertainty maybe overly conservative and avoids taking exploratory actions so its model return can be low. In contrast, a policy pursuing high model return might fail in realistic environments at a state that the model is highly uncertain about. Examples of these policies are given in Fig. 1. Formally, for any two policies πi and πj , πi dominates πj if and only if J(πi) ≥ J(πj) and J(πi) 6= J(πj). A policy π∗ is Pareto optimal if no any policy dominates π∗ in Rd, i.e., no objective can be improved without detriment to another objective at π∗. All Pareto optimal policies constitute the Pareto set, and the Pareto front is the image of the Pareto set in the space spanned by all the objectives. Unfortunately, it is almost infeasible in practice to find the whole Pareto set and the Pareto front’s shape is also unknown.
As discussed in Sec. 3, it is difficult to determine the optimal trade-off between J r̂ and Ju because we cannot access the realistic environment during training. It is also challenging and costly to control
the trade-off. Hence, a straightforward strategy is to find a diverse set of policies on the Pareto front performing different levels of trade-off and select the best one when deployed to a realistic environment. However, how to find these Pareto optimal policies is still an open challenge. In addition, it is expensive to train each Pareto policy from scratch. After allocating a few diverse policies on the Pareto front, can we start from them to find more Pareto policies so we can avoid linearly increasing the computational costs? To overcome these challenges, we develop “Pareto policy pool (P3)”, which can efficiently and precisely find a diverse set of policies on the Pareto front of return-uncertainty bi-objective optimization.
4.2 PARETO POLICY POOL
Fig. 2 illustrates the main idea of “Pareto policy pool (P3)”, whose detailed procedure are given in Alg. 1. In order to find diverse Pareto policies and precisely control the trade-off of each policy, P3 generates multiple reference vectors {vi}ni=1 in the objective space, each forming a constraint to the bi-objective optimization in Eq. (3) and targeting a different region on the Pareto front (Sec. 4.2.1). Thereby, the Pareto policies {πi}ni=1 achieved by solving the n constrained bi-objective optimizations are diverse in terms of the objective trade-off. For solving each of them, we develop an efficient two-stage optimization algorithm, i.e., Alg. 2, which starts from an initial point and moves to a region fulfilling the constraint (correction stage), and then apply a multi-objective optimization algorithm to find a Pareto policy in the
targeted region. To avoid training each Pareto policy from scratch, which leads to linearly increasing costs for ≥ n policies, we develop a local extension method on the Pareto front in Sec. 4.2.2. This “Pareto Extension” firstly generates more reference vectors by perturbing each vector in {vi}ni=1 and then uses their associated Pareto policies {πi}ni=1 to warm start the bi-objective optimization constrained by the new vectors (Line 9-13 in Alg. 1). These locally extended policies, together with the n policies, compose a pool of diverse policies on the Pareto front, from which we can select the best policy when deploying them to realistic environments.
Algorithm 1 Pareto policy pool (P3) for model-based offline RL 1: input: dataset D, constraint ψ < 0, step size η, num. reference vectors n, Tg Tl 2: initialize: environment models, Pareto policy pool P = ∅, 0 < τa < τb < 1 for Eq. (5),
0 < < τa for Eq. (8), number of updates: T = n(Tg + 2Tl) 3: Train the model on D using supervised learning; 4: Generate n reference vectors {v1, . . . ,vn} by Eq. (5); 5: for i ∈ {1, . . . , n} (in parallel) do . Diverse Pareto Policies 6: Initialize a policy πi 7: for j = 0, 1, . . . , Tg − 1 do 8: Update the parameters of πi by Alg. 2 with vi; 9: Generate {v+i ,v − i } to vi by Eq. (8); . Local Pareto Extension
10: for v′ ∈ {v+i ,v − i } do 11: for j′ = 0, 1, . . . , Tl − 1 do 12: P = P ∪ {πi}; . Store Pareto policies into the pool 13: Update the parameters of πi by Alg. 2 with v′; 14: output: P;

heading: 4.2.1 DIVERSE PARETO POLICIES BY REFERENCE VECTORS
text: Inspired by recent works in multi-objective optimization (Cheng et al., 2016; Lin et al., 2019; Ma et al., 2020; Xu et al., 2020), we explore the Pareto front by generating a diverse set of reference vectors defining multiple constrained bi-objective optimization problems. As shown in Fig. 2, we generate n
uniformly distributed reference vectors {vi}ni=1 in a 0-1 normalized objective space by Eq. (5), i.e.,
vi , (τb − (i− 1)τc, τa + (i− 1)τc), τc = τb − τa n− 1 , 0 < τa < τb < 1, i ∈ {1, 2, . . . , n}, (5) where τa and τb control the range covered by reference vectors. Each vi = (v1i , v 2 i ) defines a constrained bi-objective optimization problem based on Eq. (3) whose solution resides in the targeted region of the Pareto front:
max θ J(πθ) , max θ
(J r̂(πθ), J u(πθ)) T, s.t. Ψ(πθ,vi) , −DKL (
vi ‖vi‖1 ∥∥ J(πθ) ‖J(πθ)‖1 ) ≥ ψ, (6)
where Ψ(πθ,vi) defines a similarity metric between the reference vector vi and the objective vector J(πθ) > 0. When ψ is large, J(πθ) is constrained to be close to vi, and the targeted region on the Pareto front is small. Hence, solving the constrained bi-objective optimization for the diverse set of reference vectors produces a diverse set of Pareto policies associated with different trade-offs between J r̂ and Ju. In the following, we develop an efficient two-stage algorithm for solving this optimization problem in Eq. (6).
Algorithm 2 A two-stage method for solving constrained bi-objective optimization
1: input: πθt , vi, ψ 2: if Ψ(πθt ,vi) < ψ then . Correction stage 3: if J r̂(πθt)/Ju(πθt) < v1i /v2i then 4: Compute ∇θJ r̂(πθt); 5: θt+1 = θt + η∇θJ r̂(πθt); 6: else 7: Compute ∇θJu(πθt); 8: θt+1 = θt + η∇θJu(πθt); 9: else . Ascending stage
10: Compute ∇θF(πθt); 11: Find α∗t to Eq. (7); 12: θt+1 = θt + ηα∗t∇θF(πθt); 13: t← t+ 1 14: output: πθt To solve the bi-objective optimization with an inequality constraint in Eq. (6), we propose a two-stage gradient-based method in Alg. 2. It first finds a solution meeting the constraint by alternating between optimizing the two objectives (correction stage) and then treats the constraint as the third objective and applies an existing multi-objective optimization algorithm to find a Pareto policy in the region targeted by the constraint (ascending stage). In the first stage, the algorithm checks how the constraint is violated (line 3), e.g., whether J r̂ or Ju is too small, and then accordingly chooses one to apply gradient ascent towards meeting the constraint. Once it finds a feasible solution fulfilling the constraint, it switches to the second stage, which reframes the problem as a triobjective optimization by turning the constraint to be the third objective, i.e., maxθ F(πθ) , (J r̂(πθ), J u(πθ),Ψ(πθ,vi)) T, and applies MGDA (Désidéri, 2012) to find a Pareto solution for this problem. Each step of MGDA aims at finding a convex combination αt∇θF(πθt) of all objectives’ gradients∇θF(πθt) , (∇θJ r̂(πθt),∇θJu(πθt),∇θΨ(πθt ,vi))T such that no objective is decreasing on the combined direction. This is equal to solving the following min-norm problem.
min αt ‖αt∇θF(πθt)‖2, s.t.‖α‖1 = 1,αt ≥ 0, (7)
which can be efficiently solved by Frank-Wolfe’s algorithm (Jaggi, 2013). Then MGDA takes one step along the direction to update the policy θt, i.e., θt+1 = θt+ηαt∇θF(πθt). Since MGDA always improves each objective, the constraint will not be violated, and the algorithm will finally find a Pareto policy in the targeted region. In practice, we use OpenAI’s ES (Salimans et al., 2017) to estimate the gradient of each objective (see Appendix A.3) for its efficiency and stability. Alg. 2 provably converges to the Pareto front region targeted by the reference vector with approximate gradients. Assumption 1. Suppose m objectives {fi}mi=1 of a multi-objective function are differentiable and that their gradients are Lipschitz continuous with constant Li > 0. Assumption 2. For the ES gradient∇fi,ν(xt), we have Eεt∼N (0,I)[∇fi,ν(xt)] = ∇fi(xt). Suppose Var(∇fi,ν(xt)) ≤ σ2. Lemma 1. ∀ mutually independent objectives fi ≥ 0, which satisfies Assumption 1 & 2, has Eεt [fi(xt+1)]− fi(xt) ≤ −(η − Liη 2 2 )‖d̄t‖ 2 + Liη 2 2 σ 2 where d̄t = ∑m i=1 αi,t∇fi(xt).
Lemma 1 implies that when η < ‖d̄t‖ 2
‖d̄t‖2+σ2 2 Li , the ascending stage leads to a monotonically nonincreasing sequence of the objectives.
Theorem 1 (Non-convex convergence rate). Let Assumption 1 & 2 hold, ∆ = fi(x0) − fi(x∗), β = η − Liη 2
2 , and γ = Liη
2
2 . For an arbitrary objective fi, given any > 0, after T = O( ∆ β −γσ2 ) iterations of ascending stage, we have 1T ∑T−1 t=0 Ext [‖d̄t‖2] ≤ .
Complete proofs are provided in Appendix A.1. Theorem 1 provides the convergence rate of the ascending stage for non-convex objectives. For the correction stage that performs a single-objective ES gradient ascent, Nesterov & Spokoiny (2017) have proved its convergence rate to a stationary point of the non-convex objective.

heading: 4.2.2 LOCAL EXTENSION OF PARETO FRONT
text: Although solving Eq. 6 for a diverse set of reference vectors using the algorithm in Sec. 4.2.1 can produce a diverse set of Pareto policies, the computational cost linearly increases with the number of policies. In practice, a few Pareto policies cannot cover all possible trade-offs and thus may lead to sub-optimal choices of policy in deployment. In order to efficiently obtain more policies with different fine-grained levels of model return-uncertainty trade-off, we propose a “Pareto Extension” that starts from the diverse set of Pareto policies and locally searches for more policies near them on the Pareto front. This warm-start strategy avoids training more policies from scratch and practically saves a significant amount of computation. As illustrated in Fig. 2 and Alg. 1, we perturb each reference vector vi for more reference vectors v+i and v − i , i.e.,
v+i = vi + ,v − i = vi − , , ( ,− ), 0 < < τa. (8)
These new vectors create more constrained bi-objective optimization problems in the same form as Eq. (6). Instead of using a random initialization, we start from and fine-tune {πi}ni=1 for a few iterations to achieve the Pareto policies of these new optimization problems. These policies and {πi}ni=1 together constitute a pool of Pareto policies. When deployed to a realistic environment, we first evaluate these policies for a few steps and then select the one achieving the highest return to deploy. More details about our selection strategy are given in Appendix A.4. As long as we include sufficient diverse policies in the pool, we can find a promising policy that significantly outperforms the policies trained by other model-based offline RL methods.

heading: 5 EXPERIMENTS
text: This section aims to answer the following questions by evaluating P3 with other offline RL methods on the datasets from the D4RL Gym benchmark (Fu et al., 2020). (1) Comparison to prior work: Does P3 outperform other state-of-the-art offline RL methods? Moreover, when the dataset does not contain high-quality samples for some state-action pairs, can the policies generated by P3 generalize to unseen states in order to achieve high returns? (2) Ablation study: In the experiments, we apply several techniques used in previous work (see Appendix A.6 for more details). How do they affect performance? (3) Effectiveness of P3: Why is it challenging to find the optimal trade-off between the model return and its uncertainty? We empirically explain how P3 efficiently alleviate this problem by generating a rich pool of diverse and representative policies.
To answer question (1), we compare P3’s performance with the state-of-the-art offline RL algorithms, including BCQ (Fujimoto et al., 2019), BEAR (Kumar et al., 2019), CQL (Kumar et al., 2020), UWAC (Wu et al., 2021), TD3+BC (Fujimoto & Gu, 2021), MOPO (Yu et al., 2020), MOReL (Kidambi et al., 2020), and COMBO (Yu et al., 2021). For fairness of comparison, we re-run these algorithms using author-provided implementations1 and train each algorithm for 1000 epochs. We also carefully tune the hyperparameters of baselines such as BCQ, CQL, TD3+BC, MOPO, and MOReL by grid search and choose the best ones for each benchmark dataset. Most of them achieve higher scores than those previous versions. For other baselines, such as UWAC and COMBO, we adopt the hyperparameters in their original papers, assuming they have chosen the best hyperparameters they can find. More details on the experimental setting can be found in Appendix A.6. Our experiment results are provided in Table 1. It is obvious that P3 achieves the highest average-score across all datasets, and significantly outperforms the baseline methods in 5 out of the 9 low/mediumquality datasets. Moreover, we find that the low/medium-quality datasets in the D4RL benchmark are “imbalanced”. As illustrated in Fig. 8, there are a large number of bad samples, some mediocre
1As noted by https://github.com/aravindr93/mjrl/issues/35, we remark that the implementation provided by MOReL’s author achieves lower results than their reported ones.
samples, and a few good samples, causing problems with learning accurate behavior policies or generalizable environment models (Buckman et al., 2021; Zhang et al., 2021). Therefore, many offline RL algorithms, especially model-free algorithms that heavily rely on the accurate recovery of behavior policy (Fujimoto et al., 2019; Kumar et al., 2020; Fujimoto & Gu, 2021), perform poorly on these datasets. According to the results in Table 1, P3 achieves SoTA performance and outperforms other baselines by a large margin on random, medium, and medium-replay datasets, indicating the advantages of P3 on learning from low-quality experiences. We use online policy evaluation to select the best policy during the test phase of P3, which can be computationally intensive or cause overheads when deployed to realistic environments. To overcome this drawback, we replace the online evaluation with Fitted Q Evaluation (FQE) (Le et al., 2019), an offline policy evaluation method, which (approximately) evaluates policies using the offline data only. The implementation details and experimental results are reported in Appendix A.4 and Table 1, respectively. We surprisingly find that “P3+FQE (offline policy evaluation)” only slightly degrades from “P3+online policy evaluation” on the performance, but still outperforms all baselines on the low/medium datasets, suggesting that FQE enables more efficient inference for P3 so it has exactly the same inference cost as other baselines.
To answer question (2), we conduct a thorough ablation study toward five variants of P3, each removing/changing one component used in P3. Table 2 reports their results on the D4RL Gym benchmark with 9 representative environment-dataset combinations. In Fig. 9 in Appendix, we visualize the Pareto policies obtained by P3 to highlight the effectiveness and superiority of our method. Among the five variants of P3, “scalarization” replaces our porposed Alg. 2 with the scalarization method (Boyd & Vandenberghe, 2004, Chapter 4.7); “no StateNorm” removes the state normalization (Mania et al., 2018; Fujimoto & Gu, 2021); “no RankShaping” removes the rank-based scale shaping (Wierstra et al., 2014); “no ParetoExtension” removes the Pareto extension proposed in Sec. 4.2.2; “no BehaviorCloning” removes the behavior cloning initialization (Kumar et al., 2020; Kidambi et al., 2020). More details on these variants are provided in Appendix A.9. According to the results in Table 2 and Fig. 9, we give the following conclusions: (1) except “scalarization” and “no ParetoExtension”, other variants perform comparably to our P3 while outperforming the previous results achieved by model-based (MOPO) and model-free (TD3+BC) RL algorithms on
the low/medium-quality datasets, reflecting that these widely-used techniques can improve P3’s performance but are not crucial to our appealing results. (2) “scalarization” shows noticeable degradation in performance and cannot obtain a dense set of diverse policies, as shown in Fig. 9. The results can be explained as follows: the scalarization method only finds a few separated policies, and it is difficult to find one with advantageous trade-off among them. In addition, we remark that the computational cost of multiple training with different weight assignments is similar to the cost of running P3. (3) “no ParetoExtension” degrades P3’s performance on all 9 environment-dataset combinations, corroborating that a dense set of policies on the Pareto front is essential to our results.
To answer question (3), in Fig. 4, we study how the P3 policies with different uncertainty-return trade-off perform in the deployed environment. For low/medium-quality datasets (the left three plots in Fig. 4), the optimal policies with high realistic returns (bright areas in the heatmap) spread across almost the whole Pareto front. Therefore, to find the best policy, it is essential to explore the whole Pareto front and select one from a diverse set of Pareto optimal/stationary policies as P3 does. This explains why P3 performs the best on all the low/medium datasets. On the contrary, for high-quality datasets (the right two plots in Fig. 4), the optimal policies with high realistic returns gather within a small region of the Pareto front and associate to one trade-off level. Therefore, by carefully tuning the trade-off weight, previous methods can still find the optimal policy without visiting the whole Pareto front. Hence, we observe less advantages of P3 on the high-quality datasets. The reason behind is that the MDP models are very confident on high-return (realistic) state-action pairs if most samples in the training data are with high-return (high-quality), while they can be uncertain about many high-return pairs if the training data only cover a few high-return samples (low/medium quality). It is worth noting that collecting high-quality datasets is usually expensive or infeasible in practice and many applications lack sufficient high-quality data. In these imperfect but practical scenarios, P3 performs significantly better and more stably than existing model-based offline RL methods that only learns one single policy.

heading: 6 CONCLUSION
text: In this paper, we find that model-based offline RL’s performance significantly relies on the trade-off between model return and its uncertainty, while determining the optimal trade-off is challenging without access to the realistic environment. To address the problem, we study a bi-objective formulation for model-based offline RL and develop an efficient method that produces a pool of diverse policies on the Pareto front performing different levels of trade-offs, which provides flexibility to select the best policy in the inference stage. We extensively validate the efficacy of our method on the D4RL benchmark, where ours largely outperforms several recent baselines and exhibits promising results on low-quality datasets.

heading: ACKNOWLEDGMENTS
text: Yijun Yang and Yuhui Shi are supported in part by the Shenzhen Fundamental Research Program under Grant No. JCYJ20200109141235597, the National Science Foundation of China under Grant No. 61761136008, the Shenzhen Peacock Plan under Grant No. KQTD2016112514355531, the Program for Guangdong Introducing Innovative and Entrepreneurial Teams under Grant No. 2017ZT07X386.

heading: REPRODUCIBILITY STATEMENTS
text: Code is available at https://github.com/OverEuro/P3. We provide a full description of all our experiments in Section 5 and Appendix A.6.

heading: A APPENDIX

heading: A.1 CONVERGENCE ANALYSIS OF ALG. 2
text: We provide a proof for Lemma 1. Proof. For αt obtained by solving minαt ‖ ∑m i=1 αi,t∇fi,ν(xt)‖2, s.t. ∑ αit = 1, α i t ≥ 0, let
xt+1 = xt − ηdt
= xt − η m∑ i=1 αi,t∇fi,ν(xt) (9)
According to assumption 1:
fi(xt+1) ≤ fi(xt) +∇fi(xt)T(xt+1 − xt) + Li 2 ‖xt+1 − xt‖2
= fi(xt)− η∇fi(xt)Tdt + Li 2 ‖ηdt‖2 (10)
Take expectation on both sides:
Eεt [fi(xt+1)] ≤ fi(xt)− η∇fi(xt)Td̄t + Liη
2
2 Eεt [‖dt‖2] (d̄t =
∑m i=1 αi,t∇fi(xt))
= fi(xt)− η∇fi(xt)Td̄t + Liη
2
2 (‖Eεt [dt]‖2 + Var(dt))
≤ fi(xt)− η‖d̄t‖2 + Liη
2
2 (‖d̄t‖2 + Var(dt)) ((Désidéri, 2012, Lemma 1))
= fi(xt)− η‖d̄t‖2 + Liη
2
2 (‖d̄t‖2 + m∑ i=1 α2i,tVar(∇fi,ν(xt)))
≤ fi(xt)− η‖d̄t‖2 + Liη
2
2 (‖d̄t‖2 + m∑ i=1 αi,tVar(∇fi,ν(xt)))
≤ fi(xt)− (η − Liη
2
2 )‖d̄t‖2 +
Liη 2
2 σ2 (11)
Hence, we have Eεt [fi(xt+1)]− fi(xt) ≤ −(η − Liη
2
2 )‖d̄t‖ 2 + Liη
2
2 σ 2.
Let the right term:
−(ηt − Liη
2 t
2 )‖d̄t‖2 +
Liη 2 t
2 σ2 < 0 (12)
We have:
ηt < ‖d̄t‖2 ‖d̄t‖2 + σ2 2 Li (13)
When ηt < ‖d̄t‖2 ‖d̄t‖2+σ2 2 Li ,−(ηt− Liη 2 t 2 )‖d̄t‖ 2 + Liη 2 t 2 σ 2 < 0, then Eεt [fi(xt+1)]−fi(xt) ≤ 0. Hence, the ascending stage leads to a monotonically non-increase sequence, convergence proved.
We provide a proof for Theorem 1.
Proof. According to Lemma 1, we have: fi(xt)− Eεt [fi(xt+1)] ≥ β‖d̄t‖2 − γσ2 (14) Take expectation with respect to the solution xt and do telescoping: Ex0 [fi(x0)]− Ex0,ε0 [fi(x1)] ≥ βEx0 [‖d̄0‖2]− γσ2
Ex1 [fi(x1)]− Ex1,ε1 [fi(x2)] ≥ βEx1 [‖d̄1‖2]− γσ2
. . .
ExT−1 [fi(xT−1)]− ExT−1,εT−1 [fi(xT )] ≥ βExT−1 [‖d̄T−1‖2]− γσ2 (15)
Note that ExT ,εT [fi(xT+1)] = ExT+1 [fi(xT+1)], hence ∆ = fi(x0)− fi(x∗) ≥ fi(x0)− ExT [fi(xT )] (x∗ dominates xT )
≥ T (β 1 T T−1∑ t=0 Ext [‖d̄t‖2]− γσ2) (16)
According to Eq. 16, we can obtain
1
T T−1∑ t=0 Ext [‖d̄t‖2] ≤ ( ∆ T + γσ2) 1 β (17)
Let
( ∆
T + γσ2)
1 β ≤ (18)
Rearranging terms, we have
T ≥ ∆ β − γσ2
(19)
By the condition that β − γσ2 > 0, we have the following:
(η − Liη 2 2 )− Liη 2 2 σ2 > 0
− Liη 2 − Liη 2 σ2 > 0
+ σ2 2 Li > η (20)
Let η < +σ2 2 Li , given > 0, after T = O( ∆β −γσ2 ) iterations, 1 T ∑T−1 t=0 Ext [‖d̄t‖2] ≤ .

heading: A.2 DETAILED RELATED WORK
text: Model-based Offline RL. Model-based RL methods are promising candidates for sequential decisionmaking problems due to their high sample-efficiency and compact modeling of a dynamic environment. Time-dependent linear models and Gaussian processes provide excellent performance in the low-data and low-dimensional scenario. High-capacity models, e.g., deep neural networks, are more successful because they benefit from powerful supervised learning techniques that allow the leverage of large-scale datasets, even in domains with high-dimensional image observations. Although the convenience of working with large-scale datasets, these methods still suffer from the effects of distribution shift and error accumulation of the model predictions (Levine et al., 2020), especially in the offline setting. Existing works explore several classical methods to solve the problem. Dyna-style algorithms (Sutton, 1991) utilize a technique named branched rollout to collect short-horizon model rollouts, which alleviates accumulated extrapolation errors. Another method introduces ensemble learning (Chua et al., 2018), i.e., the leverage of multiple dynamics models, against model errors. Other works also incorporate novel concepts into model-based RL, e.g., an energy-based model regularization (Boney et al., 2019), a game-theoretic framework for model-based RL (Rajeswaran et al., 2020), meta-learning (Clavera et al., 2018), policy regularization (Berkenkamp et al., 2017), and generative temporal difference learning (Janner et al., 2020). In the offline setting, since the learned model cannot be corrected with additional data collection and training, it is crucial to prevent the policy from visiting “out-of-distribution” states. Recent two works propose uncertainty-regularized policy optimization algorithms for this purpose (Yu et al., 2020; Kidambi et al., 2020). One adds a soft uncertainty penalty associated with a user-chosen weight to the reward issued by the model, and the other constructs a pessimistic MDP model associated with a hard threshold for terminating the interaction between the policy and the model when the model prediction is inaccurate. (Matsushima et al., 2021) presents a deployment-efficient RL framework to avoid the offline distribution shift issue. (Rafailov et al., 2021) extends an existing model-based offline RL method to image-based tasks. More recently, conservative offline model-based policy optimization (COMBO) (Yu et al., 2021) regularizes the value function on out-of-distribution states yielded via interacting with the learned model, which leads to a conservative estimate of value function for these state-action pairs, without requiring explicit uncertainty penalty. Moreover, a recent method named MuZero Unplugged focuses on using the model-based planning directly for policy improvement (Schrittwieser et al., 2021).
Multi-objective Optimization. Multi-objective optimization (MOO), also known as Pareto optimization, aims to optimize more than one objective function. For a nontrivial MOO problem, no single solution exists that optimizes all objectives simultaneously. In this case, these objective functions are said to be conflicting, and we can find a set of Pareto optimal solutions given different trade-offs among objectives. Existing methods in MOO can be divided into two categories: gradient-based methods and heuristic methods. Multiple gradient descent algorithm (MGDA) (Désidéri, 2012) is an extension of the classical gradient descent algorithm to multiple objectives. It uses an adaptive weighting aggregation of sub-objective gradients to compute the descent direction for all objectives. Because of the ability of fast convergence with theoretical guarantee, MGDA-based methods have been widely used to solve multi-task learning (Sener & Koltun, 2018; Lin et al., 2019) and multiobjective recommendation (Milojkovic et al., 2020). However, MGDA-based methods generate only one solution or a finite set of sparse solutions with different trade-offs. In contrast, heuristic methods usually find a Pareto set by various evolutionary algorithms such as the genetic algorithm (NSGA-II) (Deb et al., 2002) and the decomposition-base evolution algorithm (MOEA/D) (Zhang & Li, 2007). They can find a dense set of Pareto near-optimal solutions by only one time running. But in practice, these black-box optimization methods usually take unacceptable run time when the parameter space is extremely large, e.g., the number of parameters for a deep neural network, or when the function evaluation is computationally expensive, e.g., population-based training for RL agents. In this paper, our approach borrows the ideas from the heuristic methods to achieve a dense set of policies. Moreover, we replace inefficient evolutionary operators with a gradient-based method, reducing the training cost significantly.
Model-free Offline RL. Model-free RL directly learn a policy without requiring the learning of an environment model. A direct extension of model-free RL methods to offline setting (Fujimoto et al., 2019) usually perform poorly due to the data distribution shift. To address this issue, prior model-free offline RL methods regularize the learned policy to be “close” to the behavior policy either implicitly by conservative Q learning (Riedmiller, 2005; Fujimoto et al., 2019; Kumar et al., 2020; Zhang et al., 2021; Fujimoto & Gu, 2021), or explicitly by direct constraints in state or action spaces (Kumar et al., 2019; Liu et al., 2020; Zhou et al., 2020). Compared with model-based offline RL methods, these model-free methods behave more conservatively and lack broader generalization when the distribution of pre-collected experiences is narrow. Instead of formulating as a single-objective optimization with regularization, a concurrent work (DiME) by Abdolmaleki et al. (2021) takes a multi-objective perspective for model-free offline RL. It adopts a modified scalarization method to achieve multiple policies on the Pareto front. However, scalarization cannot find any solution on the non-concave parts of Pareto front and thus may lead to sub-optimal performance.

heading: A.3 EVOLUTION STRATEGY
text: For black-box or noisy objective functions such as RL’s policy optimization, computing accurate the gradient is usually challenging. Hence, as a derivative-free optimization method, evolution strategy (ES) has seen a recent revival in the RL community (Salimans et al., 2017; Mania et al., 2018). Instead of solving the complicated objective function F(θ) directly, ES solves its Gaussian smoothing version: Fν(θ) = Eε∼N (0,I)[F(θ + νε)], where ν > 0 denotes a smoothing parameter. When ν is small, the smoothed version Fν(θ) is close to the original objective (Nesterov & Spokoiny, 2017), and its gradient with respect to policy parameters θ is defined by ∇θFν(θ) = (2π)−d/2 ∫ Rd F(θ + νε)e− 1 2‖ε‖ 2 2εdε. Although the gradient is intractable, it can be estimated by a standard Monte
Carlo method: ∇θFν(θ) = 1kν ∑k i=1 F (θ + νεi) εi. The Monte Carlo estimation has no bias but high variance. Many following algorithms were proposed to reduce the variance while keeping the bias unchanged. In this paper, we adopt an antithetic estimator (Mania et al., 2018), as shown in Eq. (21), which uses the symmetric difference between a perturbation εi ∼ N (0, I) and its antithetic counterpart −εi.
∇θFν(θ) = 1
2kν k∑ i=1 [F (θ + νεi)− F (x− νεi)] εi, (21)
F(θ) = ( J r̂(θ), Ju(θ),Ψ(θ,vi) )T . (22)
where J r̂(θ), Ju(θ) and Ψ(θ,vi) are results achieved by the policy πθ on one trajectory generated from the environment model. Despite the simplicity, ES achieves competitive performance compared to policy gradient methods (Choromanski et al., 2020).

heading: A.4 SELECTION OF PARETO POLICIES BY FITTED Q EVALUATION (FQE)
text: Given a pool of Pareto policies, how do we determine the best one for a realistic environment? A straightforward method is online policy selection that evaluates the performance of every Pareto policy in the real environment, and then selects the one with the highest return. Most existing offline RL methods (Kidambi et al., 2020; Kumar et al., 2020; Matsushima et al., 2021; Zhang et al., 2021) adopt this method, which however requires many steps of online interactions with the realistic environment, especially when the number of policies to be evaluated is large. We also follow these works and use online policy evaluation to achieve the results in Table 1.
Probably a more efficient alternative is offline policy selection (Voloshin et al., 2019), which is proposed very recently, and aims at choosing the best policy from a set of policies, given only offline data. However, offline policy selection/evaluation is an open problem (Levine et al., 2020), and a number of recent works in RL (Paine et al., 2020; Zhang & Jiang, 2021; Fu et al., 2021) are still exploring new strategies for it while combining our method with offline policy evaluation methods can potentially improve the efficiency of P3. To evaluate this strategy, we use FQE (Le et al., 2019) to estimate the performance of each policy. Although there exist various offline policy evaluation methods, we choose FQE for its simplicity and stability in practice. By contrast, other methods have to solve complex estimation problems, such as learning a transition model from visual inputs or estimating the importance weights in a continuous action space (Fu et al., 2021). In our experiments, both P3 and FQE have access to the same offline data.
On an offline dataset D, FQE algorithm trains a critic Qφi for each Pareto policy πi generated by P3, and then uses the critic to estimate the expected value of πi w.r.t. the initial state s0, i.e., V̂πi = Es0∼D[Qφi(s0, πi(s0))], which reasonably quantifies the performance of πi when deployed to a realistic environment. The pseudo-code for FQE algorithm can be found in Alg. 3. FQE can efficiently find a near-optimal Pareto policy after only hundreds of epochs, which is demonstrated by the results in Appendix A.7.
Algorithm 3 Fitted Q evaluation (FQE) for Pareto policy selection 1: input: Pareto policy pool P , dataset D, γ = 0.99, β = 0.995 2: for πi ∈ P (in parallel) do 3: Initialize a critic Qφi and the corresponding target critic Q
′ φ′i ← Qφi ;
4: for k = 1, 2, . . . ,Kepoch do 5: Sample a batch {sj , aj , rj , s′j}batchsizej=1 from D; 6: Update Qφi by minimizing MSELoss(Qφi(s, a), r + γQ
′ φ′i (s′, πi(s ′)));
7: φ′i = βφ ′ i + (1− β)φi; . Update target critic Q′φ′i 8: V̂πi = Es0∼D[Qφi(s0, πi(s0))]; 9: output: ibest = arg maxi V̂πi

heading: A.5 ENVIRONMENT MODEL
text: We train an ensemble of N models and pick the best K models based on their prediction error on a hold-out set. In the training phase, each model is optimized independently using the maximum likelihood estimation with mini-batch stochastic gradient descent. In the inference phase, we randomly select one ofK models and draw a state-reward concatenation from the resulting distribution, allowing for different transitions along a single episode to be sampled from different dynamics models. The method makes our model more uncertain and noisier than the realistic environment, resulting in a more challenging MDP problem. Prior work (Ha & Schmidhuber, 2018; Janner et al., 2019; Yu et al., 2020) demonstrated that the method effectively alleviates the model exploitation issue (Levine et al., 2020), especially in the offline setting. As shown in Table 3, we list the hyperparameters of environment model for D4RL Gym experiments.

heading: A.6 D4RL GYM EXPERIMENTS
text: D4RL Gym Datasets. D4RL is a widely-used benchmark for evaluating offline RL algorithms. It provides a variety of environments, tasks, and corresponding datasets containing samples of multiple trajectories generated via behavior policies, hand-designed controllers, or human demonstrators (Fu et al., 2020). In the continuous control domain based on the MuJoCo simulator (Todorov et al., 2012),
we apply a subset of datasets, including three environments (halfcheetah, hopper, and walker2d) and five dataset types (random, medium, medium-replay, expert, and medium-expert), to yield a total of 15 benchmark problems, in which random contains 1M samples from a random policy, medium contains 1M samples from a policy trained to approximately 1/3 of the performance of the expert, expert contains 1M samples from a policy trained to the performance of the expert, medium-replay contains the whole replay buffer of a policy trained up to the performance of the medium agent, and medium-expert contains a 50-50 split of medium and expert dataset (2M samples).
Practical Modifications. We list the techniques adopted by P3 for D4RL Gym experiments.
(1) P3 applies a simple behavior cloning approach (Kidambi et al., 2020; Matsushima et al., 2021; Kumar et al., 2020) to estimate the initial policy πi. The technique effectively alleviates the data distribution shift caused by the difference between the policy-in-training and the behavior policies used to collect the data.
(2) We notice that the states of the high-dimensional complex tasks take the values in a broad range, causing the policies only pay attention to particular features of these states. Therefore, we normalize the received state before feeding it to the policy network: ah = π(σ−1s (ŝh − µs)) where µs and σs are the mean and standard deviation of states, respectively. In the offline setting, we can easily obtain µs and σs by computing the mean and standard deviation of all the states in the dataset, similar to the feature normalization in supervised learning. Previous work (Mania et al., 2018; Fujimoto & Gu, 2021) demonstrated that the technique makes the policies more robust to multiple-scale state inputs.
(3) P3 applies the rank-based scale shaping (Wierstra et al., 2014) to compute the ES gradients, which transforms the objective function value via a rank-based score function, suppressing the influence of outlier perturbations.
Hyperparameters. We outline the hyperparameters used in the experiments.

heading: A.7 RANK CORRELATION BETWEEN OFFLINE AND ONLINE POLICY EVALUATION IN P3 ON D4RL GYM EXPERIMENTS
text: In Section A.4, in order to remove the extra computation required by online policy evaluation during the test, we instead use Fitted Q Evaluation (Alg. 3) to provide metrics for policy selection. Here we take a closer look at the correlation between the policy ranking by the online evaluation and the policy ranking by FQE.
First, we rank all the policies in the P3 pool by the two metrics and show their generated rank orders (in an ascending order) for every policy in Fig. 5. On three datasets, the two rank orders show a strong correlation, which indicates that the FQE based offline policy evaluation provides an accurate approximation to the realistic return of P3 policies, and thus can be used to improve the test efficiency of P3. In addition, we report the Spearman’s rank correlation (“Spearman’s ρ”) and Kendall rank correlation (“Kendall’s τ”) to quantitatively measure the correlation between the two rank orders. We report the results in Fig. 6, which demonstrates a strong correlation coefficient close to 1.0 for all datasets. Therefore, replacing the online evaluation with FQE is an efficient solution preserving the original rank orders of P3 policies.

heading: A.8 RESULTS ON THE D4RL GYM EXPERIMENTS

heading: A.9 ABLATION STUDY
text: For clarity in ablation study, we provide implementation details on the scalarization method (Boyd & Vandenberghe, 2004, Chapter 4.7). The method maximizes a weighted sum of objectives: maxωJ(πθ) where ω ∈ [0, 1] denotes the weight assignment. By solving multiple single-objective optimization with different weight assignments, we can obtain multiple policies with different tradeoffs, as shown in Fig. 9. For the experimental results in Table 2 and Fig. 9, we train 5 policies from scratch using the scalarization method with random weights and report the best score among them.

heading: A.10 EFFECTIVENESS OF PARETO POLICY POOL

Review summaries:
In model-based offline reinforcement learning, a dynamics model is trained from the dataset and subsequently used to produce a decision-making policy which attains high reward (according to the model). However, one must also take care not to allow the policy to exploit the model’s inaccuracies, so we also want it to visit states and take actions which have low model uncertainty. Prior work has managed this tradeoff by linearly combining the two objectives.  This work proposes instead to find many diverse policies along the Pareto front of these two objectives. The algorithm, Pareto policy pool (P3), proceeds as follows: 1. Generate reference vectors which quantify tradeoffs in the Pareto front. 2. In parallel, find a Pareto-optimal policies for each reference vector using “Algorithm 2” to solve a constrained bi-objective optimization problem, where the constraint ties the solution to the reference vector. Algorithm 2 begins by finding a feasible solution and then applies MGDA to improve the objectives. 3. “Local extension”: Each reference vector is perturbed in two opposing directions, and then each perturbed vector is further optimized via Algorithm 2, with intermediate policies being added to the policy pool. (This is just an optimization to reduce the computational cost.) 4. At test time, each policy in the pool is evaluated, and the best one is selected.  The paper includes some theoretical analysis of Algorithm 2, showing convergence to an approximate stationary point. Empirical evaluation is performed on the D4RL benchmark, where P3 achieves good results, particularly on the datasets generated by lower-performing policies.

This paper proposes a new algorithm for model-based offline reinforcement learning in which the learner attempts to optimally balance maximizing the reward under the learned model and minimizing the uncertainty of the model. The algorithm is motivated by the fact that offline RL methods suffer from distribution shift that often results in policies/predictions that are overly optimistic about highly uncertain areas of the state-action space. The authors’ proposed method attempts to address this problem by identifying a relevant collection of policies lying on the Pareto frontier that trades off these two criteria. The underlying supposition is that a well-performing policy should be among this collection of “non-dominated” policies on the frontier. Specialized subroutines are proposed in order to make sure the collection is diverse. Some theory is presented to validate certain aspects. Extensive experimental results support that the method performs well in relevant benchmarks compared to a number of state-of-the-art methods. 

The paper proposes a new model-based offline reinforcement learning, where a pool of Pareto optimal policies is trained on a model of the environment provided by offline RL data. First, a model of the environment is trained using supervised learning from the dataset of logged experiences and subsequently a set of Pareto optimal policies is trained along the dimension of return in the MDP model and uncertainty in the MDP model. The authors argue that providing a Pareto optimal set of policies is superior to prior methods that relied on regularization of the two objectives into a combined metric, as having a Pareto optimal set of policies enables one too explore a range of behavior that is inherent with trade-off between model return vs model uncertainty in offline model based RL.  The authors then describe their method, Pareto Policy Pool (P3), which addresses the bi-objective optimization problem two stage method. Initially, the authors initialize a set of reference vectors along the Pareto frontiers to ensure a diverse set of Pareto policies can be found. Subsequently, the authors apply "Local Pareto Extension" where first a given policy is corrected to a desired range along the objective dimensions provided by the reference vectors. Once the policies are within the desired range, the method applies MGDA based optimization to further improve the performance of the policy in a Pareto optimal way (ascension stage). In order to alleviate computational cost of training multiple policies, the authors apply a "Pareto extension" method to initialize new policies along the Pareto frontier.   The authors then perform experiments on D4RL based benchmarks and compare their method to other literature methods and claim outperformance, particularly in lower quality dataset settings. The authors also perform an ablation study pertaining to the various components of their method, as well as an analysis related to why it is challenging to obtain optimal trade-offs in model based offline RL.

This paper proposes a model-based approach for offline RL that is inspired by multi-objective RL. The approach, called P3, finds a Pareto front of policies that trade off between obtaining high return (with respect to the model trained on the offline dataset) and minimizing the model's uncertainty. To obtain this Pareto front, the approach has two stages: 1) find a few spread-out solutions on the Pareto front using a gradient-based method, and 2) initialize with these solutions to find optimal solutions for the neighboring regions of the Pareto front. The empirical evaluation shows that P3 outperforms existing approaches when the offline dataset is of low or medium quality.