Scaling Pareto-Efficient Decision Making via Offline Multi-Objective RL
Authors: Baiting Zhu
Meihua Dang
Aditya Grover
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Offline Reinforcement Learning
Multi-Objective Reinforcement Learning
Decision Transformer
Sequential Decision Making

Abstract: The goal of multi-objective reinforcement learning (MORL) is to learn policies that simultaneously optimize multiple competing objectives. In practice, an agent's preferences over the objectives may not be known apriori, and hence, we require policies that can generalize to arbitrary preferences at test time. In this work, we propose a new data-driven setup for offline MORL, where we wish to learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences. The key contributions of this work are two-fold. First, we introduce D4MORL, (D)atasets for MORL that are specifically designed for offline settings. It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for randomly sampled preferences on 6 MuJoCo environments with 2-3 objectives each. Second, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that builds and extends Decision Transformers via a novel preference-and-return-conditioned policy. Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides an excellent approximation of the Pareto-front with appropriate conditioning, as measured by the hypervolume and sparsity metrics. 

Published as a conference paper at ICLR 2023
SCALING PARETO-EFFICIEN DECISION MAKING VIA OFFLINE MULTI-OBJECTIVE RL
Baiting Zhu, Meihua Dang, Aditya Grover University of California, Los Angeles, CA, USA ba it iigggztt.g..... .elu, mhdang@cs ucla edu, adityagecs uel edu
ABSTRACT
The goal of multi-objective reinforcement learning (MORL) is to learn poli- cies that simultaneously optimize multiple competing objectives. In practice, an agent's preferences over the objectives may not be known apriori, and hence, we require policies that generalize arbitrary preferences at test time. In this work, we propose a new data-driven setup for offfine MORL, where we wish learn a preference-agnostic policy agent using only a finite dataset of offline demonstrations of other agents and their preferences. The key contributions this work are two-fold. First, we introduce D4MORL, (D)datasets for MORL that are specifically designed for offline settings. It contains 1.8 million annotated demonstrations obtained by rolling out reference policies that optimize for ran- domly sampled preferences on 6 MuJoCo environments with 2-3 objectives each. Second, we propose Aaffci-Efficient Decision Agents (PEDA), family of offline MORL algorithms that builds and extends eturn-conditionee offline methods in- cluding Decision Transformers (Chen et al. 202T) and RvS (Emmons et al.) (00TT) via novel wreference-and-retuue conditioned policy. Empirically, we show that PEDA closely approximates the behavioral policy on the D4MORL benchmark and provides excellent approximation of the Pareto-front with appropriate con- ditioning, as measured by the hypervolume and sparsity metrics.
INTRODUCTION
We are interested in learning agents for multi-ohjective reinforcement learning (MORL) that opti- mize for one or more competing objectives. This setting is commonly observed in many real-world scenarios. For instance, an autonomous driving car might trade off high speed and energy savings depending on the user's preferences. If the user has a relatively high preference for speed, the agent will move fast regardless of power usage; on the other hand, if the user tries to save energy, the agent will keep more steady speed. One key challenge with MORL is that different users might have dif- ferent preferences on the objectives and systematically exploring policies for each preference might be expensive, even impossible. In the online setting, prior work considers several approximations based on scalarizing the vector-valued rewards of different objectives based on single preference [Lin 2005). learning an ensemble of policies based on enumerating preferences (Mossalam al., 2016
Xu et al. 2020), or extensions of single-objective algorithms such as Q-learning to vectorized value functions Yang al. 2019).
We introduce the setting of offline multi-objective reinforcement learning for high- sigh-dimensional state and action spaces, where our goal is to train an MORL policy agent using an offline dataset of demonstrations from multiple agents with known preferences. Similar to the single-task setting, offline MORI. can utilize auxiliary logged datasets to minimize interactions, thus improving data efficiency and minimizing interactions when deploying agents in high-risk settings. In addition to its practical utility, offline RL (Levine et al. 2020) has enjoyed major successes in the last few years (Kumar et al. 2020 Kostrikov al.) 2021 Chen et al. 2021) challenging tigh-dimensiooal environments for continuous control and game pel--ppllynnnn.. Our contributions in this work are two-fold in introducing benchmarking datasets and new family of MORL, described below.
We introduce Datasets for Multi Mujti-Objective Reinforcement Learning (D4MORL), a collection of 1.8 million trajectories 6 multi-objective MuJoCo environments (Xu al. 2020). Here, 5 environ-

---

Published as a conference paper at ICLR 2023
ments consist of 2 objectives and environment consists of 3 objectives. For each environment in D4MORL, we collect demonstrations from 2 pretrained behavioral agents: expert and amateur, where the relative expertise is defined in terms of the Pareto-efficiency of the agents and measured empirically via their hypervolumes. Furthermore, we also include 3 kinds of preference distribu- tions with varying entropies to expose additional data-centric aspects for downstream benchmark- ing. Lack of MORL datasets and large-scale benchmarking has been a major challenge for basic research (Hayes et al., 2022), and we hope that D4MORL can aid future research in the field.
Next, we propose Pareto-Efficient Decision Agents (PEDA), a family of offline MORL algorithms that extends conut-condd methods including Decision Transformer (DT) (Chen et al., 2 and RvS (Emmons et al. (021)) to the multi-objective setting. These methods learn return- conditioned policy via supervised loss on the predicted actions. In recent work, these methods have successfully scaled to agents that demonstrate broad capabilities in multi-task settings (Lee) al., Reed Reed et al., 2022). For MORL, we introduce a novel preference and return conditioned policy network and train it via a supervised learning loss. At test time, naively conditioning on the default preferences and maximum possible returns leads to out-of-distribution behavior for the model, neither has it seen maximum returns for all objectives in the training data nor is it possible to simultaneously maximize all objectives under competition. We address this issue by learning to map preferences appropriate returns and hence, enabling predictable generalization at test-time.
Empirically, we find PEDA performs exceedingly well on D4MORL and closely approximates the reference Pareto-frontier of the behavioral policy used for data generation. In the multi-objective HalfCheetah environment, compared with an average upper bound on the hypervolume of 5.79 x 106 achieved by the behavioral policy, PEDA achieves an average hypervolume of 5.77 10 on the Expert and 5.76 x 10 on the Amateur datasets.
2 RELATED WORK
Multi-Objective Reinforcement Learning Predominant works in MORL focus on the online setting where the goal is to train agents that can generalize to arbitrary preferences. This can be achieved by training single preference-conditionn
policy (Yang et al. 2019 Parisi et al. 2016). or an ensemble of single-objective policies for finite set of preferences (Mossalam et [al.]2216 Xu iet al. 2020; Zhang& & Li 2007). Many of these algorithms consider vectorized variants of standard algorithms such as Q-learning (Mossalam et al. 2016 Yang et al. 2019), often augmented with strategies to guide the policy ensemble towards the Pareto front using evolutionary or incremen- tally updated algorithms (Xu et al. 2020 Zhang & Li 2007| Mossalam et al., 2016 Roijers et al.) 2014 Huang et al. 2022). Other approaches have also been studied, such as framing MORL as a meta-learning problem (Chen et al. 2019), learning the action distribution for each objective (Ab- dolmaleki et al. 22020), and learning the relationship between objectives (Zhan & Caol (2199) among others. In contrast to these online MORL works, our focus is on learning a single policy that works for all preferences using only offline datasets.
There are also a few works that study decision-making with multiple objectives in the offline setting and sidestep any interaction with the environments. [wu et al.) [022] propose a provably efficient offline MORL algorithm for tabular MDPs based on dual gradient ascent. [Thomas et al. [22 study learning of safe policies by extending the approach of Laroche et al.] 2019 to the offline MORI. setting. Their proposed algorithm assumes knowledge of the behavioral policy used to collect the offline data and is demonstrated primarily tabular MDPs with finite state and action spaces. In contrast, we are interested in developing dataset benchmarks and algorithms for scalable offline policy optimization in high-dimensional MDPs with continuous states and actions.
Multi- Task Reinforcement Learning MORL is also closely related to multi-task reinforcement learning, where every task can be interpreted as a distinct objective. There is an extensive body of work in learning multi-task policies both in the online and offline setups (Wilson et al. 2007 Lazaric & Ghavamzadeh, 2010 Teh et al. 2017) inter alia. However, the key difference is that typical MTRL benchmarks and algorithms do not consider solving multiple tasks that involve inherent trade-offs. Consequently, there is no notion of Pareto efficiency and an agent can simultancously excel in all the tasks without accounting for user preferences.
2

---

Published as a conference paper at ICLR 2023
Reinforcement Learning Via Supervised Learning A body of recent works have formulated of- fline reinforcement learning as an autoregressive sequence modeling problem using Decision Trans- formers (DT) Trajectory Transformers (Chen et al.) 2021 [ Janner et al.) 2021) The key idea in DT is to learn a transformer-base policy that conditions on the past history and a dynamic esti- mate of the returns (a. returns-to-go). Follow-up works consider online learning (Zheng al., 2022) as well as simpler variants that rely only on multi-layer perceptrons (Emmons et al. 2021) Such agents are generally more stable and robust to optimize due the simplicity of loss function and easier to scale to more complex settings such as environments with high-dimensional actions or states, as shown in recent works in multi-tast RL (Lee al. 2022
Reed et al. 2022).
PRELIMINARIES
Setup and Notation. We operate in the general framework of multi-objective Markov decision process (MOMDP) with linear preferences (Wakuta) 1995). An MOMDP is represented by the tuple (S, A, P. R, Q J) At each timestep 6, the agent with current state 8, E S takes an action a, E A to transition into a new state 8 with probability P(* 88t, a.) and observes a reward vector T. R(s; a²) RR. Here, n is the number of objectives. The vector-valued return R E R" of an agent is given by the discounted sum of reward vectors over a time horizon, R = E; 7'r,- We also assume that there exists a linear utility function and space of preferences 5 that can map the reward vector ry and a preference vector w Q to a scalar reward rt. ie., rt = f(r,, w) wr. The expected vector return of policy is given an G [G G G T where the expected return of the 8t objective is given as G, = Pa4410a((ee. ">, R(s,, a.). for some predefined time horizon and preference vector w. The goal to train a multi-objective policy ा T(aas, ((a)sww) w) such that the expected scalarized return T G BF[w Σ, R(s;, at)] is maximized.
Pareto Optimality. In MORL, one cannot optimize all objectives simultaneously. so poli- cies are evaluated based on the Pareto set of their vector-valued expected returns. Consider preference-condiiionee policy π Tia 8, w) that is evaluated for m distinct preferences Wi. Wms and let the resulting policy set be represented and FO is the corresponding unweighted expected return. We say the solution G is dom- inated by G when there is no objective for which Ty is worse than Ty. i... G for Vi E [1,2,...,
(Tplp=-,....m where Ty x 8,
wp).
G If a solution is not dominated, it is part of the Pareto set denoted as P. The curve traced by the solutions in a Pareto set is also known as the Pareto front. In MORL, our goal is to define a policy such that its empirical Pareto set is good approxima- tion of the true Pareto front. While we do not know the true Pareto front for many problems, we can define metrics for relative comparisons between different algorithms. Specifically, we evaluate a Pareto set P based on two metrics, hypervolume and sparsity that we describe next.
Definition 1 (Hypervolume). Hypervolume H(P) measures the space or volume enclosed by the solu- tions in the Pareto set P:
Sparsity
Hypervolume
Pareto Front
Pareto Set Solution
[ [((P(() dz,
H(P)
RRm
Objective 2
Dominated
where H(P) - (2 7 1 & i P P()). P(1) the 7th solution in P. ss the dominance relation operator, and (z) equals 1 if (P)) and otherwise. Higher hypervolumes are better. Definition 2 (Sparsity). Sparsity S(P) measures the density of the Pareto front covered by Pareto set P:
IPI-1 Σ
S(P)
(P:(k) -..k ( 1))3,
Objective 1
Figure 1: Illustration of the Hypervolume and Sparsity Metrics. Only undominated solutions (i.e., the Pareto set) are used for calculating the evaluation metrics.
where P. represents list sorted as per the values of the th objective in P and (.( the t.th value in the
sorted list. Lower sparsity is better.
See Figure for illustration and Appendix F for discussion other possible metrics.
3

---

Published as a conference paper at ICLR 2023
4 D4MORL: DATASETS FOR OFFLINE MULTI-OBJECTIVE REINFORCEMENT LEARNING
In offline RL, the goal of an RL agent is to learn the optimal policy using fixed dataset without any interactions with the environment (Levine et al. 2020). This perspective brings RL closer to supervised learning, where the presence of large-scale datasets has been foundational for further progress in the field. Many such data benchmarks exist for offline RL as well; a notable one is the D4RL (Fu et al., 2020) benchmark for continuous control which has led to the development of several state-of-the-an offline RL algorithms (Kostrikov et al. 2021 Kumar et al., 2020) Chen et al.) 2021) that can scale favorably even in high dimensions. To the best of our knowledge, there no such existing benchmarks for offline MORL. Even for the online setting, most works in MORL conduct evaluations control, e.g.. Chen et al. (219
toy MDPs (e.g., gridworld) with few exceptions that include continuous : Xu et al. (2020). This calls for a much-needed push towards more challenging benchmarks for reliable evaluation of MORL, especially in the offline setting.
We introduce Datasets for Multi-Objective Reinforcement Learning (D4MORL), a large-scale benchmark for offline MORL.. Our benchmark consists of offline trajectories from 6 multi- objective MuloCo enviromments including 5 environments with objectives each (MO-Ant, MO- HalfCheetah, MO-Hopper, MO-Swimmer, MO-Walker2d), and one environment with three objec- tives (MO-Hopper-3obj). The objectives are conflicting for each environment; for instance, the two objectives MO-Hopper correspond to jumping and running; MO-HalfCheetah, MO-Swimmer, and K0-Walker2d, they correspond to the speed and energy savings of the agent. See Appendix [A] for more details on the semantics of the target objectives for each environment. These environments were first introduced in Xu et al. (2020) for online MORL, and as such, we use their pretrained en- semble policies as building blocks for defining new behavioral policies for dataset collection, which discuss next.
4.1 TRAJECTORY SAMPLING
The quality of the behavioral policy used for sampling trajectories in the offline dataset is key factor for benchmarking downstream offline RL algorithms. In existing benchmarks for single-objective RL such D4RL (Fu et al.) 2020), the quality of behavioral policy can be ascertained and varied based on its closeness to single expert policy, MOMDP, we do not have the notion of a scalar return and hence, a reference expert policy (or set of policies) should reflect the optimal returns for all possible preferences in the preference space.
measured by its scalar-valued returns. For
We use Prediction-Guidee Multi- Objjet-beeeeivee Reinforcement Learning (PGMORL), MORL algorithm for defining reference expert policies. PGMORL (Xu et al. 2020) uses evolution- ary algorithms to train an ensemble of policies to approximate the Pareto set. Each reference policy in the ensemble is associated with unique preference; as for any new preference, it is mapped to the closest preference in the reference set. The number of policies in the ensemble can vary sig- nificantly; for instance, we have roughly 70 reference policies for MO-Antand 2445 policies for harder environments such as HO-Hopper-3obj Given a desired preference, we define two sets of behavioral policies:
satte-of-the-ae
1. Expert Dataset: We find the best reference policy in the policy ensemble, and always follow the action taken by the selected reference policy.
2. Amateur Dataset:
As before, first find the best reference policy in the policy ensem- ble. With a fixed probability p. we randomly perturb the actions of the reference policies. Otherwise, with probability p. we take the same action as the reference policy. In D4MORL, we set 0.65.
Further details are described in Appendix C In Figure 2 we show the returns of the trajectories rolled out from the expert and amateur policies for the 2 objective environments evaluated for uniform sampling of preferences. We can see that the expert trajectories typically dominate the amateur trajectories, as desired. For the amateur trajectories, returns for both objectives under consideration. The return patterns for the amateur trajectories vary across different environments providing diverse suite of datasets in benchmark.
see more diversity in the empirical
4

---

Published as a conference paper at ICLR 2023
Vertical Spoed
Energy
neiight
Energy
Energy
amneteur
Horizontal Speed
Speed
Speed
Speed
Speed
(a) MO-Ant
(b) MO-HalfChectah
(c) MO-Hopper
(d) MO-Swimmer
(c) MO-Walker2d
Figure 2: Empirical returns for expert and amateur trajectory datasets for the two-objective environ-
ments in D4MORL. For each environment and dataset, we randomly plot returns for 300 trajectories.
pref 1.0
1.0
1.0
full pref space
0.5
0.0
0.0
0.0
1.0
1.0
1.0
0.0
0.5
0.5
0.0
0.5
0.5
0.5
1.00.0
1.00.0
1.00.0
(a) High Entropy, = 0.2
(b) Medium Entropy, H 0.35
(c) Low Entropy, = 1.54
Figure 3: Illustration of the preference distributions for 3 objectives. Entropy is estimated 50K
preference samples using the Vasicek estimator in Scipy [Vasicek 1976 Virtanen et al. 2020
4.2 PREFERENCE SAMPLING
The coverage of any offline dataset is important factor in dictating the performance of downstream offline RL algorithms (Levine al., (020). For MORL, the coverage depends on both the behavioral MORL policy as well as the distribution preferences over which this policy is evaluated. We use the following protocols for sampling from the preference space Ω. First, we restrict our samples to lie within a physically plausible preference space Ω S Ω covered by the behavioral policy πд- For instance, MO-Hopper has two objectives: jumping and rumning. Since the agent can gain running rewards without leaving the floor. Thus, the preference of 100% running and 0% jumping not achievable and excluded from our preference sampling distribution.
Second, we are primarily interested in offline trajectories that emphasize competition between mul- tiple objectives rather than focusing a singular objective. To enforce this criterion, define 3 sampling distributions concentrated around the centroid of the preference simplex. The largest spread distribution samples uniformly from S2* and denoted High-Entropy (High-HH), Next, we have a Medium- Medium-Entropy (Med-H) distribution specified via samples of Dirichlet distributions with large values of their concentration hyperparameters (aka a.). Finally, we have Low-Entropy (Low-H) distribution that is again specified via samples of Dirichlet distributions but with low values of their concentration hyperparameters. We illustrate the samples for each of the preference distribu- tions along with their empirical entropies in Figure 3 Further details the sampling distributions are deferred to Appendix B By ensuring different levels of coverage, we can test the generalizability of MORL policy to preferences unseen during training. In general, expect Low-H to be the hardest of the three distributions due to its restricted coverage, followed by Med-H and High-
Overall Data Generation Pipeline. The pseudocode for generating the dataset is described in Algorithm [ Given a preference distribution, we first sample preference w and query the closest behavioral policy in either the amateur/expert ensemble matching We roll out this policy for T time steps (or until the end of an episode if sooner) and record the state, action, and reward information. Each trajectory in dataset is represented
w. 81, a1 r1,...; : AT,
5

---

Published as a conference paper at ICLR 2023
For every environment in D4MORL, collect 50K trajectories length = 500 for both expert and amateur trajectory distributions under each of the 3 preference distributions. Overall, this results in total of 1.8M trajectories over all 6 environ- ments, which corresponds to roughly 867M time steps. We refer the reader Table { Appendix B for additional statistics on the dataset.
Algorithm Data Collection in D4MORL
procedure COLLECT(prefDist, nTraj, env, pretrainedAAents, T)
agents pretrainedAgents prefs prefDist(nTrai)
all trajs = []
for in prefs do
agent = closestAggen((geent,, w)
8 = env.reset()
done False
5 PARETO-EFFICIENT
т = [w]
DECISION AGENTS (PEDA)
t 0
while (NOT done) AND (t < T) do
s', = agent.get action(s)
In this section, we propose Pareto-Efficient De- cision Agents (PEDA), multi-objectivv RL agents. PEDA aims to achieve Pareto-efficiency by extending Decision Trans- formers (Chen et al.] 202T) into multi-objeecivee setting. We first introduce the architecture of Deci- sion Transformers (DT) and its variant, Reinforce- ment Learning Via Supervised Learning (RvS), followed by our modifications extending them to the multi-objecctive setting.
done, r env.tep((a)
family of offline
append 8, a, s', to
8 ==8
t= append to all trajs return all trajs
DT casts offline RL as a conditional sequence modeling problem that predicts the next action by conditioning a transformer on past states, actions, and desired returns. The desired returns are defined as returns-to-ge (RTG) 9t = Ei= TV', the future returns that this action is intended to achieve. Therefore, the trajectory is represented by =< $1, a1 P1..... 8T, ar.9r In practice, we use a causally masked transformer architecture such as GPT (Radford et al. 2019) to process this sequence and predict the actions by observing the past K timesteps consisting of 3K tokens. DT and its variants have been shown to be more stable and robust to optimize due to the simplicity of loss function; easier scale to more complex settings such as environments with high-dimensional actions or states, and agents with broad capabilities such as multitask settings (Lee et al. (2022). Hence, we adopt Decision Transformers (Chen et al. 2021) the representative base algorithm on which we build our work.
In follow-up work, Emmons et al. (202T) extend DT and shows that even multi-layer perceptrons conditioned on the average returns-to-go can achieve similar performance without the use of trans- formers. They call their model Reinforcement Learning Via Supervised Learning (RvS). However, RyS is generally not very stable when conditioned on very large returns, unlike DT.
5.1 MULTTI-BJECTIVE REINFORCEMENT LEARNING VIA SUPERVISED LEARNING
In PEDA, our goal is to train a single preference-conditionel agent for offline MORL. By includ- ing preference conditioning, trajectories collected from behavioral policies that are associated with alternate preferences. To pa- rameterize our policy agents, we extend the DT and RvS architectures include preference tokens and pector-valued returns. We refer to such preference-condition as MODT(P) and MORvS(P) respectively, which we describe next.
enable the policy be trained on arbitrary offline data, including
extensions of these architectures
Preference Conditioning. Naively, we can easily incorporate the preference w into DT by adding this token for each timestep and feeding it a separate embedding layer. However, empirically we find that such a model design tends to ignore w and the correlation between the preferences and predicted actions is weak. Therefore, we propose to concatenate to other tokens before any layers in MODT(P). Concretely, we define s" ① w, a* = a θ w, and g* = g D w where + denotes the concatenation operator. Hence, triples of s* a* g" form the new trajectory. As for MORYS(P), we concatenate the preference with the states and the average RTGs by default and the network interprets everything one single input.

---

Published as a conference paper at ICLR 2023
Multi-Objective Returns-to-Go. Similar to RTG for the single objective case, can define vector-valued RTG 9: Er ry Given a preference vector w,
scalarize the total returns-to-go Øt w gr. In principle, the scalarized RTG Dr can be recovered given the pref- erence vector w and the vector-valued RTG St- However, empirically we find that directly feeding MODT/MORVS with the preference-wee weighted RTG vector g: O is slightly preferable for stable training. where @ denotes the elementwise product operator.
Another unique challenge in the MORL setting concerns the scale of different objectives. Since different objectives can signify different physical quantities (e.g., energy and speed), the choice of scaling can influence policy optimization. We adopt a simple normalization scheme, where the returns for each objective are normalized by subtracting the minimum observed value for that ob- jective and dividing by the range values (max-min). Note that the maximum and minimum are computed based on the offline dataset and hence, they are not necessarily the true min/max objective values. Post this normalization, the values for every objective in the trajectory are on the same scale between 0 and 1. For evaluating the hypervolume and sparsity, we use the unnormalized values so that we can make comparisons across different datasets that may have different min/max boundaries.
Training. We follow simple supervised training procedure where train the policies domly sampled mini-batches with MSE loss (for continuous actions). In MODT and MODT(P), the input states, actions, and returns-to-ge (with concatenated preferences) are treated tokens and embedded through one layer of MIP. We apply layer of MI. and Tanh on the last hidden state of GPT-2 transformer predict next action. In MORVS and MORVS(P), from the current timestep and MLP layers to predict the next action.
only information
EXPERIMENTS
In this section, we evaluate the performance of PEDA on D4MORL benchmark. First, we investi- gate the benefits of preference conditioning by evaluating on decision transformers (DT) and RvS (MORVS) where no preference information available and we scalarize multi-objective vector returns into weighted sums. We denote our methods with preference conditioning MODT(P) and MORvS(P). Second, we compare our methods with classic imitation learning and temporal difference learning algorithms with preference conditioning.
Imitation learning. Imitation learning simply uses supervised loss to train mapping from states (w/ w/o concatenating preferences) to actions. We use behavioral cloning (BC) here and train multi-layer MLPs as models named BC (w/o preference) and BC(P) (w/ preference).
Temporal difference learning. Conservative Q-Learning (CQL) (Kumar et al. 2020 is the state- of-the-art standard offline RL method, which lears a conservative Q-function f x A R through neural networks. We modify the network architecture such that it also takes preference vectors as inputs to learn a preference-condiiionee Q-function f* A x X2 IR. We denote this method CQL(P).
6.1 MULTI-OBJECTIVE OFFLINE BENCHMARK
Table 1: Hypervolume performance H High- HigheH-Expeet. dataset. PEDA variants MODT(P) and MORYS(P) always approach the expert behavioral policy. (B: Behavioral policy)
Environmentts
B
MODT(P) MORYS(P) BC(P)
CQL(P)
MODT
MORVS
BC
CQL
MO-Amt (10")
6.32
6.21#:01
6.412:01
4.88-.17
5.76#:10
5.52+.16
5.52+
0.842.60
3.52 45
MO-HallCheetaah (10")
5.79
5.731.00
5.78:.00
5.54=005
5.63
5.591.03
4.192.74
1.532.09
3.78::46
MO-Hogper (10")
2.09
2.001.02
2.02:.02
1.23_110
0.33
1.68:1.33
1.731.07
0.28:1211
0.02:1.02
MO Ho--Hogeer (10"")
3,73
3.38 1.05
3.42
2.29-07
0.78.1.24
1.05..43
2.53
0.06.102
0.00-1.00
M0-Swimmer (10%)
3.25
3.144.02 02
3.24+.00
3:21-000
3.224.08
2.49+t 19
3.19
1.68
2.08-t. .8
MO-Walker2d (10%)
5.21
4.89
1.99-.05
5.14-.01
3174-111
3.212.32
0.65+
5.10
0.07+.02
0.82.4.0
We compare hypervolume of our methods with all baselines on expert datasets in Table T as well amateur dataset in Table 2 For the two-objective environments, evaluate the
Hypervolume.

---

Published as a conference paper at ICLR 2023
Table Hypervolume performance on High-H- -ag---hmattt f--mmttuur dataset. PEDA variants still approach or even exceed the behavioral policy even when considerable portion of data is suboptimal. MODT(P) and MORYS(P) still present to be the strongest models and outperform other base- lines. (B: Behavioral policy)
Environments
MODT(P) MORVS(P) BC(P)
CQI((P)
MODT
MORVS
CQL.
MO-Ani (10
5.61
5,921.04
6.071.02
4.37 06
5.621.23
4,88
4.37.1.56
2.341.15
2.80 16.6
MO- HallCheetah (10")
5.68
5.691.01
5.771.00
5.46_002
5.54
5.51
4.661.05
2.921.38
4.41.1088
MO-Hopper (10')
1.761.03
1.35=005
1.61:01
1.57..01
0.011011
MO N0-Ilogper-3ok Sobyj (10*))
3,09
16
2.771.24
2142_118
0.591142
1.641.23
1.301.22
0.031.01
0.101.16
MO-Swimmer (1)
2,11
1.67
2.79
2.82- 04
1.69 93
0.96+
2.93=003
0.46
0.74
MO-Walker2d (10*)
4.99
3.10+.34
4.98+.01
344-
1.78
4.32-
0.91
0.76+.81
dataset. MODT(P) and MORVS(P) have a lower density. PC(P) also has a competitive sparsity in smaller environments such as Swimmer.
Table 3: Sparsity (1) performance on
Enviromennts
(ODT(P)
MORYS(P)
BC(P)
CQL(P)
MO-Ant (x10*)
8.26
6,50+0.81
46.2116.4
0.58:10.10
MO-HallCheetth 1
1.2410233
0.6710.05
1.7810039
0.10
MO MO-Hogper (x 10")
16.31 10.6
10.36
52.5.14.88
2.84
MO-Hogper-soby (x 1x10))
1.4020.4
2.72:1993
0.72:10.09
2.6013.11
MO-Swinmer (x
15.0=749
4.39+0.07
4.50+0.39
13.6+5.31
MO-Walker2d 10
0.99=0044
3.22+0.73
75.6+52.3
6.23 10.7
models on 501 equally spaced preference points in the range [0, 1]; on the three-objective environ- ment MO- (00-Hoper--3ob) models are evaluated on 325 equally spaced points. Each point is evaluated 5 times with random environment re-initialization. and the median value is recorded. Finally, all the results are based on 3 random seeds and we report the mean performance along with the standard error. In Table I and Table [ we can see that MODT(P) and MORVS(P) outperform other base- lines and has relatively very low standard error. Also, PEDA variants including MODT(P) and MORVS(P) approaches the behavioral policy upper-bound.
Sparsity. We also evaluate sparsity performance. Since sparsity comparison is only meaningful between models that are sensitive to preference and have a relatively similar hypervolume perfor- mance, we only show results for models that concatenate preference. Overall, MORvS(P) has the lowest sparsity in most environments, while at the same time featuring an outstanding hypervolume.
6.2 ABLATION STUDY
Pareto front approximation. We ablate how well the MODT(P) and MORVS(P) can approxi- mate the Pareto front through conditioning on different preference points. We show the results in Figure where we can see that the models can approximate the Pareto front while having some dominated points colored pink mostly in the MO-Hopper and MO-Walker2d environments. The results are based on the average of 3 seeds, and the full plot can be found in Appendix G
Table 4: Sparsity (1) performance High-H-Amateur dataset. We
that all models still have a similar or stronger sparsity performance when trained on amateur datasets. Furthermore, MORvS(P) still presents the strongest performance. While BC(P) has strong performance in MO- Hopper-3obj and MO-Swimmer, it also fails to give a dense solution in other environments and has higher standard error.
Envirenments
MODT(P)
MORYS(P)
BC(P)
CQL(P)
MO-Ant 10")
8.721.777
5.241.52
25.9116.4
1.06::.88
MO Ho-lHal/hhhethh (x 10")
1.161.42
0.57 1.09
2.22
0..55 1.27
MO- Mo-Herper (x 10")
1.6
3.50
2.42.1..08
3.30-1.5.25
MO-Hopper (x 10
10.23+2.78
1.03--111
1.00++1.22
MO-Swimmer (x1)
2.87+1332
1.03
5.05
8.87+6.24
Mo-Walke2d )
164.2=13.5
1.94::006
53.1
1.33
8

---

Published as a conference paper at ICLR 2023
MO-Ant
MO Mo-HaifCheetah
Mo-Hepper
MO-Swimmer
MO-Wa lieeeddd
MO-Hoppee--otk
MODTIT expert
2000
amateur
1300
PUSUIPT
expertt
1000
0000
7103
2000
2210
amateur
20.0
4.00
200
7103
Figure 4: We show that MPDT(P) and MORvS(P) can be good approximator to the Pareto front. There relatively more dominated points in MO-Hopper and MO-Walker2d colored in red.
oracle
dataset min/max
undominated
dominated
MO-Ant
lalfCheetah
MO-llopper
MO- So-Swimmer
MO-Wallcrr2d
achieved objo
target objo
target abja
target abja
target abjO
target abjo
achieved obj1
target objl
target objl
target obj1
target obj1
target obj1
Figure 5: We show that MORvS(P) Model can follow the given target reward that is within the dataset's minimum and maximum record. The plots are for all the two-objective environments. In addition, MO-Hoppet and MO-W Ko-Walker2d present to be the most challenging environments for PEDA variants, featuring more dominated solutions than other environments.
Return distribution. We ablate how well MODT(P) and MORvS(P) follow their given target return, based normalized and weighted value. We present the results in Figure MORVS(P) under Exxperr datasets and refer to Appendix H for full settings. Here, we see that the models follow the oracle line nicely when conditioned the target within the dataset distribution, and generalize to targets outside of the dataset distribution as well.
CONCLUSION
We proposed new problem setup for offline Multi-Objective Reinforcement Learning scale Pareto-Efficient decision-making using offline datasets. To characterize progress, we introduced D4MORL, dataset benchmark consisting of offline datasets generated from behavioral policies of different fidelities (expert/amateur) and rolled out under preference distributions with varying entropies (bigh/medium//lo). Then, we propose PEDA, a family of offline MORL policy optimiza- tion algorithms based on decision transformers. To our knowledge, the PEDA variants are the first offline MORL policies that support continuous action and preference spaces. We showed that by concatenating and embedding preferences together with other inputs, our policies effectively approximate the Pareto front of the underlying behavioral policy as measured by the hypervolume and sparsity metrics. Our proposed family includes MLP and transformer- based variants, viz. the MORYS(P) and MODT(P), with MORVS(P) performing the best overall. In some scenarios, the learned policies also generalize to higher target rewards that exceed the data distribution.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our code is available at:
ACKNOWLEDGEMENTS
AG's research supported by a Meta Research Award and Cisco grant.
REFERENCES
Abbas Abdolmalcki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song. Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi nujtti-bbeective policy optimization. In International Conference on Machine Learning, pp. 11-22. PMLR, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learnin via sequence modeling. Advances in neural information processing systems, 34:15084 50844-1097 2021.
Xi Chen, Ali Ghadirzadeh, Mårten Björkman, and Patric Jensfelt. Meta-learning for multi-objective reinforcement learning. In 2019 IEEE/RS.J International Conference on Intelligent Robots and Systems (IROS), pp. 977-983. IEEE, 2019.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.107511 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Conor Hayes, Roxana Rädulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
Sandy Huang, Abbas Abdolmaleki, Giulia Vezzani, Philemon Brakel, Daniel J Mankowitz, Michael Neunert, Steven Bohez, Yuval Tassa, Nicolas Heess, Martin Riedmiller, et al. A constrained multi-objective reinforcement learning framework. In Conference on Robot Learning, pp. 883- 893. PMLR, 2022.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
lyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179 19:1199-119,, 2020.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. International conference on machine learning, pp. 3652-3661. PMLR, 2019.
Alessandro Lazaric and Mohammad Ghavamzadeh. Bayesian multi-task reinforcement learning. In ICML-27th international conference on machine learning, pp. 599-606. Omnipress, 2010.
Kuang-Husl Huei Lec, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transform- ers. arXiv preprint 2205 15241, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643, 2020.
10

---

Published as a conference paper at ICLR 2023
JiGuan G Lin. On min- and min-max max methods of multi-objectivve optimization. Mathematical programming, 103(1):1-33, 2005.
Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arXiv preprint
2016.
Simone Parisi, Matteo Pirotta, and Marcello Restelli. Multi-objective reinforcement learning
through continuous pareto manifold approximation. Journal of Artificial Intelligence Research, 57:187-227 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimencz, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXivv2205.001/75 2022.
Diederik M Roijers, Shimon Whiteson, Frans Oliehoek, et al. Linear support for multi-objective coordination graphs. In AAMAS' 14: PROCEEDINGS OF THE 2014 INTERNATIONAL CON- FERENCE ON AUTONOMOUS AGENTS & MULTTIAGENT SYSTEMS, pp. 1297-1304, 2014.
Yee Teh, Vietor Bapst, Wojciech M Czameeki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
Philip S Thomas, Joelle Pineau, Romain Laroche, et al. Multi-objective spibb: Seldonian offline policy improvement with safety constraints finite mdps. Advances Neural Information Pro- 2021.
cessing Systems,
Oldrich Vasicek. A test for normality based on sample entropy. Journal of the Royal Statistical Society: Series B (Methodological), 38(1):54-59, 1976.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: funda- mental algorithms for scientific computing in python. Nature methods, 17(3):261-272, 2020.
Kazuyoshi Wakuta. Vector-valued markov decision processes and the systems of linear inequalities. Stochastic processes and their applications, (6(1):159911)) 169, 1995.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference Machine learning, pp. 1015-1022, 2007.
Runzhe Wu, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Offline constrained multi-objective reinforcement learning via pessimistic dual value iteration. In Advances in Neural Information Processing Systems, volume 34, pp. 25439-25451, 2021.
Jie Xu, Yunsheng Tian, Pingchuan Ma. Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guiie guided multi-objeccive reinforcement learning for continuous robot control. In Pro- ceedings of the 37th International Conference on Machine Learning, 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objectivy reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems 32. 2019.
Huixin Zhan and Yongcan Cao. Relationship explainable multi-objective optimization via vector value function based reinforcement learning. arXiv preprint 2011
Qingfu Zhang and Hui Li. Moea/d: A multiobjective evolutionary algorithm based on decomposi- tion. IEEE Transactions on evolutionary computation, 1166):12-731, 2007.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In International Conference Machine Learning, pp. 27042-27059. PMLR, 2022.

---

Published as a conference paper at ICLR 2023
A ENVIRONMENT DESCRIPTION
All environments are the same as in Xu et al. each parameter is uniformly sampled from the Except we always reset height as 1.25 for MO-Hopper and MO-Hoppe--Sobjsincee this parameter directly relates the reward function. All environments have max episode length of 500 steps per trajectory, but the agent may also die before reaching the maximum length.
2020
except for when resetting the environment,
10 ,x 10-3] with being the default value.
A.1
MO-ANT
The two objectives in MO-Ant are achieved distance and y axes respectively, denoted [rt "(', [T]
Consider the position of the agent is represented as (Ir, yt) at time and takes the action a, The agent has fixed survival reward y. 1.0, dl 0.05, and an action cost of r 1 Ex rewards are calculated as:
2 The
t (xt T2-1) / dt. + 1* rda rt" = (yt - 11-1) / dt + 7d pd
A.2 MO-HALFCHEETAH
The two objectives in MO-HaliCheetah are running speed, and energy saving, denoted as r [rT,
Consider the position of the agent is represented as (Ie, 31) at time and takes the action at. The agent has a fixed survival reward y* 1.0, fixed dt = 0.05, and an action cost of r" = Σσ o The rewards are calculated as:
r min(4.0, (0+- - x(-1) / dt) + r
(2)
rf 4.0 7d + +r*
A.3 MO-HOPPER
The two objectives in MO-Hopper are running and jumping, denoted as r r"].
Consider the position of the agent is represented as (æz, he) time and takes the action a, The
agent has a fixed survival reward r = 1.0, a fixed initial height as himit 1.25, a fixed di = 0.01,
and action cost of a 2 10-4 Σn az. The rewards calculated as:
re 1.5 (xt xt-1) dt + r" a rf = 12 (he hina) / dt + 8 - p
A.4 MO-HOPPER-3OBI
The physical dynamics are the same in MO-Hopper and MO-Hopper-3oth while this environment has 3 objectives: running, jumping, and energy saving. The rewards are denoted as [r". r TT
Consider the position of the agent is represented as (Zz, fu)) at time t and takes the action a,. The agent has a fixed survival reward ,* 1.0, fixed initial height as hinit 1.25, fixed dt 0.01. and an action cost of r Σ. az. The rewards are calculated as:
r = 1.5 x (xt - R4-1)/ / dt + 8
r 12 x (h: hinis) / dt 8
(4)
r = 4.0 - pa + p
12

---

Published as a conference paper at ICLR 2023
A.5 NO-SWIMMER
The two objectives in MO-Swimmer are speed and energy saving, denoted as = [r", r Consider the position of the agent is represented as (t) time and takes the action dz- The agent has a fixed dt 0.05, and an action cost of rd Ls ax. The rewards are calculated as:
rf (2+ Fz-1)/ dt
(5)
r = 0.3 0.15 x r
A.6 MO-WALKER2D
The objectives in MO-Walker2d are speed and energy saving, denoted as r = [r") r"]
Consider the position of the agent is represented as (Iz, yt) time and takes the action dz. The agent has a fixed survival reward r 1.0, a fixed 0.008, and an action cost of r Σ. oz. The rewards are calculated
r = (xq - 21-1) / dt r*
(6)
rf 4.0 p" + 7d
To uniformly sample the High-H data from the entire preference space, the problem is equivalent to sampling from a n-dimennional simplex, where n is the number objectives. The resulting sampling
high [| '. A = 1)ll2
We take the 1-norm following the exponential distribution to make sure each preference add up to When Q * 52, we perform rejection sampling to restrict the range.
To sample the Med-H it and Low-H data, we first sample from a non-negative uniform distribution, then sample the corresponding Dirichlet preference. Here, we sample different alpha to make sure the center of the Dirichlet changes and thus allows more variation.
med
f (a: where Unif(0,, 10%)
(8)
low fDirihlee (x) where Unif(1/3 x 10 2/3 108)
For sampling from behavioral policy consists of group of single-objective policies Th {......, π8} with B being the total number of candidate policies, we recommend first find the expected unweighted raw rewards G ,,.., G Then, find the estimated 6 ".",,..,.0 6Fn by let- G
ting T.Th ΣT= G which represents the estimated preference 7th objective of 6th candidate
policy, For sampled preference lance d(w, ").. Empirically, this means picking the candidate policy that has the expected reward ratio closest to w.
f2 the policy that provides the smallest euclidean dis-
B DATASET DETAILS
To uniformly sample the Hi High- data from the entire preference space, we can equivalently sample from a n-dimensional simplex, where is the number of objectives. The resulting sampling scheme is:
high || ' 1)]la
The 1-norm following the exponential distribution makes sure each preference vector have entries add up to When ② Ω, we perform rejection sampling restrict the range.
13

---

Published as a conference paper at ICLR 2023
Table A comprehensive view of the dataset. All datasets have 500 maximum step per trajectory,
and 50K trajectories are collected under each setting. As indicated by average step per trajectory, we can see Amateur trajectories are always shorter same as Expert, thus leading lower return.
max step
expert avg-
amateur avg.
trajectories
per traj
step traj
step traj
per datasct
MO-Ant
500
500
500
S0K
500
499.91
482.11
50K
MO-Hooppr
500
499.94
387.72
50K
MO Hopperper-
500
499.99
442.87
50K
-O--Simmer
500
500
500
50K
MO- Mo--alker2d
500
500
466.18
50K
To sample the H and Low-H data, we first sample a from a non-negative uniform distribution, then sample the corresponding Dirichlet preference. Here, we sample a different alpha make sure the mode of the Dirichlet changes and thus allows more variation.
w med
DDirichle (a): where Unif(0, 10%)
(10)
Jow
Dirichter (x: where Unif(1/3 10 2/3 109)
Since our behavioral policy consists of a group of single-objective policies Te (....... TB) with B being the total number of candidate policies, we first find the expected unweighted raw rewards G ,t..,,.,CC B Then, we find the estimated preferences w",,.... We by letting 4, 6 2y- G on 1th objective of 1th candidate policy. For each sampled preference w S following
or (TOp. we sample complete trajectory using the single-objective behavioral policy that provides the smallest euclidean distance min d(w, 6..). Empirically, this means picking the candidate policy that has the expected reward ratio closest w.
C EXPERT & AMATEUR DATASETS
In Expert collection, we sample trajectories using the fully-trained behavioral policy ти. In this paper, we PGMORL by Xu et al. 2020 as behavioral policy πı
expert a2 ++
TT(a|s = 82, = w)
(11)
In the Amateur collection, the policies has 35% chance being stochastic top of the expert collection. Actions has a chance being stochastic, during which it is scaled from the expert action, following:
aa
expert
35%
a E++1
amatcur
a1+1 x Unif(0.33, 1.65)
expert
65%
(12)
In the MO-Swimmer environment only, we let actions has 35% chance to be uniform random sample from the entire action space rather than being the same as expert to increase variance and achieve performance similar to amateur. The resulting strategy for MO-Swimmer is:
Unif((A) A))
35%
a ++1
amaaieur
expert x Unif((0.35, 1.65)
65%
(13)
+++1
D FINDING APPROPRIATE MULTI-OBIECTIVE RTG
In Decision Transformer Chen et al. 1021 and [Emmons et al. (202T). I). RIG denotes the future de- sired reward. In MORI,, however, designing appropriate mulli-objectiiv RTG is necessary. On top
14

---

Published as a conference paper at ICLR 2023
of discounting each objective's desired reward separately, empirically find that since some ob- jectives are inherently conflicting, setting RTG high for one objective means we should accordingly lower RTG for other objectives (i.c. we shouldn't t use maximum RTG for both). In this way, our test-time RTG can follow closer to the training distribution.
In this paper, to find corresponding RTG conditioned on the given preference. Figure [6] demonstrates the weighted RTG
linear regression G f(w)
running: weighted return-to
of the "running" objective as a function of its prefer- ence in MO-Hopper where the conflicting objectives are "running" and "jumping". It is clear that RTG closely correlates with the conditioned preference for running and we should adjust the initial RTG during test-time accordingly.
180
140
100
Ir-fitted target
dacaset
Finally, we only use the learned linear regression model from Expert dataset. This is because regres- sion models fitted on sub-optimal data can easily pro- duce an RTG lower than optimal. In practice, we can easily achieve similar result by training the regres- sion model only on the best-performing trajectories for respective preferences. Other regression or clustering methods to find appropriate RTG can also work, and leave it as future works especially when assum- ing linearly weighted objectives.
min max target
60
0.4
0.5
running: pref
Figure 6: RTG fitted through linear regres- sion (Ir) models stay closer to the dataset distribution. Furthermore, linear regres- sion models can also efficiently generalize unseen preferences during test time.
E TRAINING DETAILS
In this section, we list our aaper-parameters and model details. In specific, we use the same hyper- parameters for all algorithms, except for the learning rate scheduler and warm-up steps. In MODT family, inputs are first embedded by a I-layer fully-connected network, and n layer represents the number of transformer blocks; BC family, n layer represents the number of MLP layers to embed each input: in MORVS and MORvS(P), we leverage the same embedding strategy in Emmmon et al. (202T). Additionally, we consider MORvS and MORVS(P) both have context length of because they only use the current state to predict the next action, whereas MODT and BC use the past 20.
E.1 PARAMETERS
Hyperparameter
MODT
MORYS
BC
Context Length- K
20
20
Batch Size
64
Hidden Size
512
Learning Rate
Ic-4
Weight Decay
Ie-3
Dropout
0.1
nllayer
3
Optimizer
AdamW
Loss Function
MSE
LR Scheduler
lambda
None
lambda
Warm-up Steps
10000
N/A
4000
Activation
ReLU
15

---

Published as a conference paper at ICLR 2023
E.2
TRAINING STEPS
Dataset Name
MODT Steps
RvS/BC Steps
MO-Ant
20K
200K
MO-HallChhettah
80K
200K
MO-Hopper
400K
200K
MO-Hopper-3ob)
400K
200K
260K
200K
MO-Walker2d
360K
200K
E.3 OTHER ATTEMPTED ARCHITECTURES FOR MODT AND MODT(P)
We tried the following MODT architectures our preliminary experiments. We picked Case eventually as it gave the best performance in experiments.
1. Consider as an independent token of the transformer.
2. Train separate embedding for concatenate the embeddings to get fa. + Sow (w), foa (a) + fow (w), and fea fou(w (w) then pass into the transformer.
3. Add another MLP layer on top of Case after concatenation, then pass output into the transformer.
4. Concatenate to other tokens before any layers. This means we have s*
a* * = a + w, and g* = g θ W.
F OTHER EVALUATION METRICS
Among variety of metrics for MORL, we use Hypervolume (HV) and Sparsity (SP) to benchmark models for several reasons. First, many metrics such the e-metric require prior knowledge of the true Pareto Fronts, which are not available for our MuJoCo Environments. Second, we only assume linear reward function and cannot collect real-time user feedback, thus utility- metrics such as expected utility metric (EUM) not applicable. Finally, using the same metric in the original behavioral policy paper facilitate algorithm comparisons.
G PARETO SET VISUALIZATIONS
We present the Pareto Set visualizations for all of our models trained under each High-H dataset Figure 7 Each point in each subplot is based on the average result of 3 seeds. 2 objective environments, we evaluate the model using 501 equally spaced preference points. In 3 objective environments, we use 351 equally spaced preference points instead. Since the environments are stochastically initialized, we evaluate 5 times at each preference point and take the mean value. This makes each point the average value 15 runs. We here allow a small tolerance for coloring the dominated points.
a preference point is within the achievable preference S2* but the solution is dominated, we color it in red. Since models conditioned continuous preference points and environments initialized stochastically, we give a small tolerance (3%-8%) for points to he colored in blue. The hypervolume and sparsity metrics, the other hand, based strictly undominated solutions without tolerance.
H MEDIUM & Low ENTROPY DATASET TRAINING
We train on the Medium-Entropy and Low-Entropy datasets for the MO-HalfCheetah environment. Overall, models have a similar performance under Med-H and High-H datasets but suffer when only trained on Low-H. We present the results in Table in which illustrate that the Low-H dataset has a worse expert and amateur performance due to reduced variability on preference. How- ever, MODT(P) and MORvS(P) still able get close exceed in hypervolume all
16

---

Published as a conference paper at ICLR 2023
MO-Ant
MO-HalfCheetah
MO-Hopper
MO-Swimmer
MO- WelkwWldd
MO Hop-or----bb
amateur MODT(P)
expers
100
12:0
22.00
1200
7300
350
2008
RuS(P)
expert
2000
1000
1300
120
srmateur
4200
2000
2000
100
1000
120
1300
BCIP)
expers
2000
amalee
2000
2008
MOD
expert
1230
13c
300
aratteu
4200
1300
emateuur expert
1230
2000
1230
10C
3000
Figure 7: Pareto Visualization for all PEDA variants and baselines on High-B H datasets. Notice that MO-Hopper and MO-Walker2d are more challenging and have significantly more dominated points for all variants. In other environments, however, the PEDA variants produce better results while other baselines fail at higher chances. Since BCwithout preference is completely single objective, we don't t show the result here.
17

---

Published as a conference paper at ICLR 2023
Table 6: We ablate how well PEDA variants perform and generalize under a different preference distribution in the Mo-HalfCheetah environment. We can see that PEDA can still perform well when trained under the partially-clustreed Med-H dataset. However, performance drops when trained under the entirely clustered Low- H dataset. (B: Behavioral Policy)
Dataset (x 10")
MODT(P)
MORVS(P)
BC(P)
MODT
MORVS
BC
Boed-B-Amate e'22
5.69
5.681.01
1.771.00
1.441.26
5.61 1 01
4.67.1.03
3.21 1:11:300
Tow-F-Rmat BUY
4.754.04
4.72-2005 05
4.394.04
Bigh AmaGeur
5.68
5.69+.01
5.77+.00
5,46+.02
5.54= 02
4.66+t05
2.92
Table 7: We ablate the importance of using multi-dimensional rtgs instead of a one-dimensional rigs by taking the weighted sum of objectives the MO-HalfCheetah environment. We multi- objective rtgs provide variance reduction for MORvS(P) and hypervolume performance improve- ment in other models. (B: Behavioral Policy)
Setting
Dataset (x 10%)
MODT(P)
MORvS(P)
MODT
MORVS
High-B-uxpert
5.79
5.731.00
5.781100
5.5911.3
4.1911.74
1-dim ng
nigh 0 Expert
5.79
5.70
5.78:t.00
4.43:009
2.894.06
datasets, which showcases the effectiveness of PEDA as an efficient MORL. policy. Results are based on an average of seeds with the standard error given.
I TRAINING WITH 1-DIM RTG
We attempted train MODT and RvS with I-dim return-to-go rather than a separate rtg for each objective. According to results on MO-HalfCheetah and the High-H H datasets in 7 using multi- dimensional RTG enhances the performance of MODT(P), and are about the same for MORvS(P) when preference is concatenated to states. However, it reduces standard eтror significantly in both MODT(P) and MORvS(P). the naive models when preferences are not concatenated states, using a multi-dimensional RTG helps achieve much more competitive hypervolume. We thus believe multi-dimensional RTG conveys important preference information when the model doesn't directly take preference input. Results are based on average of 3 seeds with the standard error given.
18


Reviews:
[{"review_id": "xT1N4AJR_Oa", "reviewer": "Reviewer_oRVV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript did two things. One is to propose a new dataset that collects agent trajectories from multiple independent agents with different preferences on the objectives. This dataset contains trajectories from both well-trained agents and semi-trained agents. HalfCheetah is a typical example where actions that are large in absolute value will consume more energy while the agent wants the cheetah to run as fast as possible. Two is to propose a decision transformer-like algorithm that handles an offline multi-objective dataset (which is possibly the one proposed). This algorithm is quite intuitive and the main idea is to condition its output on the preference of the newly tested task. Several experiments are presented.\n\n", "summary_of_the_review": "I think it's quite an interesting problem and the work is solid (intuitive, works well, and is presented well). I believe it's a borderline paper and I lean slightly towards acceptance.", "strength_and_weaknesses": "Pro:\n\n1. This paper introduces a new dataset that targets the multi-objective RL with offline data.\n\n2. This paper introduces a decision transformer-like algorithm which is intuitive and works in experiments.\n\nCons:\n\nThe baselines in the experiments seem lacking as only two methods (BC/CQL) are compared with. Is any offline RL and inverse RL approach relevant?\n\nQuestions\n\n1. One question is how the two contributions - namely the dataset and the algorithm - are entangled to each other. With the presentation of this paper one would expect the contributions to be independent, i.e. we expect the algorithm to work on other datasets as well. Is there any way to validate this? I understand a similar dataset might not be present in the community for now.\n\n2. Following the last question, is the dataset general enough to be an independent contribution? From the generation of this dataset I could many variants in the process. Should the community use this dataset in the future or it's good only for this work. \n\n3. What is the difference of this work and multi-objective inverse RL? I'm not seeing much difference despite the work follows the vein of offline RL mostly. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "vaTucca-sP", "reviewer": "Reviewer_gXRB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies offline multi-objective reinforcement learning (offline MORL). In the first half of the paper, the authors introduce a new benchmarking dataset. For 6 tasks it contains trajectories with different preferences, where for each preference we have data from an expert policy (with similar preference profile) and a noisy amateur policy. In the second half of the paper, the authors introduce a new approach for offline MORL, based on return-conditioned sequence modeling. Experiments show that their method outperforms baseline approaches on their own benchmark. ", "summary_of_the_review": "This paper has two contributions: 1) a new benchmarking dataset for offline multi-objective RL, and 2) a new algorithm class for offline MORL based on return-conditioned sequence modeling. I think the first aspect is a clear contribution, which will be of merit to the community. The second part is also interesting although slightly incremental. However, together I think the paper is a good candidate for acceptance at ICLR. ", "strength_and_weaknesses": "Strength:\n* The paper is well written and clear. \n* The benchmarking dataset is a useful contribution for the community. \n* Related work is well covered. \n* The paper contains very useful illustrations, such as Fig 1 to illustrate Sec. 3, and Figures 2 and 3 to illustrate the data generation process for the benchmark.\n* Extensive experiments. The generalization performance in Fig. 5 is very impressive. \n\nWeaknesses: \n* For data generation of the amateur policy, you set the exploration parameter p to 65%. This seems rather high to me: in most problems you will not get very far with such high exploration. Couldn’t you vary the amount of exploration, from low to high, for different episodes? Or gradually increase the noise during the episode, to make sure that your amateur policy sometimes gets a bit further in the domain? I think you see this effect in Fig 2: in MO-Swimmer for example you see that the amateur policies cover a quite demarcated area of the return space, indicating that your exploration/noise scheme covers a too small region of the overall return space. \n* You extensively mention Decision Transformers in your abstract and introduction, but actually your best performing models are the MORvS(P) ones, that do not use a transformer. I think you would need to phrase this differently. \n* On the algorithmic side, the innovation is mostly in the application of the (return-conditioned) sequence modeling approach to the offline RL setting. There are some details about how to feed the preferences and returns into the model, with some claims about what worked and did not work (without results though). It is a useful insight to use sequence modeling for offline RL though, since you typically want to stay close to the original data. \n* Top of page 7: You say that a scalarized \\hat{g} and the preference vector omega can recover the vector valued g. This is not true right? Imagine omega = [0.5,0.5], and \\hat{g] = 0.5, then both g = [1,0] and g=[0,1] would work (or any g for which sum(g) = 1.0). I think this explains why you need the elementwise product between g and omega (but why not feed them in completely separately?). \n\nMinor:\n* Sec 2: previous work is “primarily” demonstrated on tabular tasks → But not only right? Try to be precise here, i.e., what is your extension? \n* Sec 3: I miss an explanation why “sparsity” is a relevant measure? \n* What type of noise distribution do you inject in generating the amateur policy? These are continuous tasks right, so do you use Gaussian noise, or simply a uniform distribution within the bounds of the action space? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JqtuEyXZzyp", "reviewer": "Reviewer_wtvp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a reference benchmark for multi-objective offline learning data set which is built on a formerly proposed set of benchmark environments for multi-objective RL consisting of 6 MuCojo environments.\nIn addition, the authors adapt decision transformers and the RL via Supervised learning method to the multi-objective setting. This is achieved by concatenating the preference vector to state, action and return. Furthermore, the return to go is modified to represent the scalarized return.\nIn the experiments, the authors compare to behavioural cloning and conservative Q-learning which is modified to receive the input as preference as well. Results indicate that the supervised learning-based offline-RL approaches outperform compared methods.", "summary_of_the_review": "The paper is a solid study of offline multi-objective RL on the MuCojo settings. However, the paper lacks novelty and technical depth. All solutions are kind of relatively straightforward and as compared methods do not fit the problem results are not too convincing. The sampling scheme to generate the offline data seems solid but sampling the data did not require overcoming any technical problems.\nTo conclude, the paper yields interesting results but does not come up with the technical depth and novelty of most accepted ICLR papers.", "strength_and_weaknesses": "Strength:\n* The paper is easy to follow.\n* The paper examines the questions of how well decision transformers (and RLvSL) can be adapted to the multi-objective setting\n* Multi-objective offline RL was not examined on Mucojo environments before.\n\nWeaknesses:\n* The proposed extension of the offline RL methods is relatively straightforward.\n* Results are not too surprising.\n* The novel data set is helpful, but given that the online benchmark was already available it is not too difficult to sample an offline data set.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7BmewdEMxO", "reviewer": "Reviewer_6PPS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors propose a flexible multi-objective RL framework that can handle different preferences over objectives, that are not known in training time. They achieve this with an offline RL approach. First, they present a novel dataset of demonstrations designed for offline multi-objective settings. Second, they propose a Pareto-efficient decision mechanism that builds on Decision Transformers with a novel approach for conditioning the policies on preferences and returns. The authors continue to show that their proposed framework closely matches the behavior policies in the demonstrations, meaning it effectively approximates the Pareto front.  ", "summary_of_the_review": "Overall an interesting paper with some caveats that need addressing.", "strength_and_weaknesses": "strengths\n-------------\n- An interesting approach to an interesting (and nontrivial) problem.\n\n- The idea of a sample-efficient way for approximating the Pareto front at execution time seems quite promising. \n\nWeaknesses\n---------------\n- The dataset itself doesn't seem like that much of a contribution, compared to how it is framed in the paper.\n\n- Adding random perturbations to expert demonstrations is not a good way of approximating amateur demonstrations.\n\n- It's unclear how this translates to a more organic multi-objective setting (consider a navigation task where vehicles need to maximize transport throughput while minimizing risk and travel time). \n\n- Not clear how well this would generalize to remote parts of the trajectory space (and preference space).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster