On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning
Authors: Abbas Abdolmaleki
Sandy Huang
Giulia Vezzani
Bobak Shahriari
Jost Tobias Springenberg
Shruti Mishra
Dhruva Tirumala
Arunkumar Byravan
Konstantinos Bousmalis
András György
Csaba Szepesvari
raia hadsell
Nicolas Heess
Martin Riedmiller
Venue: ICLR 2022 Submitted (2022)
Keywords: offline RL
learning from experts
finetuning
multi-objective RL
deep RL
continuous control

Abstract: Many advances that have improved the robustness and efficiency of deep reinforcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives or constraints in the policy optimization step. This includes ideas as far ranging as exploration bonuses, entropy regularization, and regularization toward teachers or data priors. Often, the task reward and auxiliary objectives are in conflict, and in this paper we argue that this makes it natural to treat these cases as instances of multi-objective (MO) optimization problems. We demonstrate how this perspective allows us to develop novel and more effective RL algorithms. In particular, we focus on offline RL and finetuning as case studies, and show that existing approaches can be understood as MO algorithms relying on linear scalarization. We hypothesize that replacing linear scalarization with a better algorithm can improve performance. We introduce Distillation of a Mixture of Experts (DiME), a new MORL algorithm that outperforms linear scalarization and can be applied to these non-standard MO problems. We demonstrate that for offline RL, DiME leads to a simple new algorithm that outperforms state-of-the-art. For finetuning, we derive new algorithms that learn to outperform the teacher policy.

Under review as a conference paper at ICLR 2022
ON MULTI-OBIECTIVE POLICY OPTIMIZATION AS A TOOL FOR REINFORCEMENT LEARNING: CASE STUDIES IN OFFLINE RL AND FINETUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Many advances that have improved the robustness and efficiency of deep rein- forcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives or constraints in the policy optimization step. This includes ideas as far ranging as exploration bonuses, entropy regularization, and regularization toward teachers or data priors. Often, the task reward and auxiliary objectives are in conflict, and in this paper we argue that this makes it natural to treat these cases as instances of multi-objective (MO) optimization problems. We demonstrate how this perspective allows us to develop novel and more effective RL algorithms. particular, we focus on offline RL and finetuning as case studies, and show that existing approaches can be understood MO algorithms relying linear scalarization. We hypothesize that replacing linear scalarization with a better algorithm can improve performance. We introduce Distillation of Mixture of Experts (DiME), a new MORL algorithm that outperforms linear scalarization and can be applied to these non-standard MO problems. We demonstrate that for offline RL, DiME leads to a simple new algorithm that outperforms state-of-the-art For finetuning, we derive new algorithms that learn to outperform the teacher policy.
INTRODUCTION
Deep reinforcement learning (RI.) algorithms have solved number of challenging problems, includ- ing in games (Mnih et al.) 2015 Silver et al.] 2016). simulated continuous control (Heess et al. [017]] Peng et al. 2018), and roboties (OpenAl et al. 2018). The standard RL setting appeals through its simplicity: agent acts in the environment and can discover complex solutions simply by maximiz- ing cumulative discounted reward. In practice, however, the situation is often more complicated. For instance, without carefully crafted reward function sophisticated exploration strategy, learning may require hundreds of millions of environment interactions, or may not be possible at all.
number of strategies have been developed mitigate the shortcomings of the pure RL paradigm. These include strategies that regularize the final solution, for instance by maximizing auxiliary
rewards (Jaderberg al. 2017) the entropy of the policy (Mnih et al.
2016
Haarnoja et al. 2018).
Others facilitate exploration, for instance via demonstrations (Brys et al.
2015), shaping rewards (Ng
al. 1999), exploration bonuses (Bellemare al. 2016 Pathak al.
2017). or guidance from
pre- tre-trained teacher policy, known as policy finetuning (Xie et al.) 202
Strategies for reusing old environment interactions can in the extreme case enable agents learn solely from a fixed dataset,
known as offline RI. (Fu et al. 2020 Gulcehre et al. 2020). In this case regularization is needed to ensure that the learned policy is well supported by the data distribution.
These approaches may seem quite different at first glance, but they can all be understood as introducing additional objectives, or constraints, in the policy optimization step. Since reward maximization and the additional objective(s) are usually in conflict, these algorithmic approaches can also be understood as solving a multi-objective RL (MORL) problem. In practice, the solution to this MORL problem is usually implemented via a form of linear scalarization (LS), i.c., by taking weighted sum of the objectives. However, LS suffers from a number of limitations: is sensitive to the scale of the objectives (Abdolmaleki al. 2020 and is limited in the solutions can find (Das & Dennis. 1997).

---

Under review as a conference paper at ICLR 2022
Our key insight is that an interpretation of algorithmic approaches, such as the above, in terms of MORL provides us with novel tools to solve fundamental challenges in RL. To demonstrate this, we focus offline RL and finetuning as case studies. We show recent offline RL algorithms (e.g.. CRR (Wang et al. 2020), AWAC (Nair et al. 2020)) can be understood as using LS to solve the MO problem. We hypothesize that by replacing LS with more appropriate approach, we can obtain better- performing algorithms. We thus propose a new MORL approach, called Distillation of a Mixture of Experts (DiME), that can replace LS in this setting. We derive novel algorithms for finetuning and offline RL, based on DiME. In offline RL, this new algorithm outperforms state-of of-the-art (Sec. 6
Our main contributions are as follows:
We propose that MORL can be used as a tool for tackling fundamental challenges in RL. We show that existing algorithms (c.g., for offline RL) rely on LS solve the MO problem.
Based on this insight, we explore replacing LS with a new, improved MO algorithm.
By replacing LS, we derive novel offline RL RL. algorithm from the multi-objective perspective
(Sec. 5.2 that is simple, intuitive and outperforms the state-of-the-arr (Sec. 6..2
In similar manner, we also derive new finetuning algorithms (Sec. [.1] and show that
these algorithms can train policies that exceed the teacher's s performance (Sec. 6.3)
To replace LS, we introduce new MORL approach. DIME. We support DiME by theoretical derivations (Sec. 4.1 and show it outperforms LS on standard MORL tasks (Sec. 6.1
RELATED WORK
Offline RL and Finetuning. Our case studies in this work will involve two related learning problems: offline (or batch) RL (Levine et al.] 2020 Lange et al.] 2012) and policy finetuning (Xie) et al. 2021). Both leverage existing information, either in the form of a dataset or a teacher policy. In this work, we use proximity the dataset or policy as an additional objective in a MORL problem.
In offline RI., the goal is to learn policy from a given, fixed dataset of transitions. While classical off-policy RL algorithms can be used, they perform worse than bespoke algorithms (Fujimoto et al.) 2019). This is likely due to the extrapolation errors incurred when learning a D-function--navve policy optimization could optimistically exploit such errors. Therefore, many existing approaches attempt to constrain the policy optimization to stay close to the samples present in the available dataset (Peng et al., 2019 Nair et al., 2020; Siegel et al.] 2020 Wang et al. 2020). Finetuning seeks to improve upon an expert (or teacher) policy, which can be specified either nonparametrically via a dataset of transitions or as a parametric model. contrast to offline RI., finetuning happens in an online setting, where the agent is able to interact with the environment. Existing approaches for finetuning minimize the cross-entropy between the teacher and agent policies (Schmitt et al., 2018). based on distillation mimicking (Rusu et al.
2015 Parisotto al. 2015).
Multi-Objective RL. MORL algorithms train policies for environments with multiple sources of reward. There are single-policy and multi-policy approaches. The former finds policy that is optimal for a single preference trade-off across rewards. One common technique is to use the trade-off to combine the rewards into single scalar reward, and then use standard RL optimize this (Roijers) let al. 2013). Typically linear scalarization is used, but it is sensitive to reward scales and cannot always find optimal solutions (Das & Dennis 1997). Non-linear scalarizations exist (Van Moffaert ct al. 2013 Golovin & Zhang 2020), but are hard to combine with value-based RL. Instead of scalarization, recent work on MO-MPO (Abdolmaleki et al. 2020) relies on constrained optimization.
Multi-policy approaches seek to find a set of policies that cover the Pareto front. Various techniques exist, for instance repeatedly calling a single-policy approach with strategically- ccosen trade-off settings (Roijers et al. [0144] Mossalam et al. [2016] Zuuluaaa et al. 2016), simultaneously learning set of policies by using a multi-ohjective variant of Q-learning (Moffaert & Nowé, 2014| Reymond & Nowe, 2019 Yang et al.) 2019), learning manifold of policies in parameter space (Parisi et al. 2017), or combining single-policy approaches with an overarching objective (Xu et al. 2020).
In contrast, here we consider objectives that are not only environment rewards, but also learning- or regularizationfoontr such as staying close to a behavioral prior. State-of-the art MORL approaches, including MO-MPO, cannot be readily applied. We thus introduce new algorithm, DIME, which is on-par with state-of-the-an and can handle these non-standard objectives (Sec. 4 We use DIME as a vehicle to demonstrate the benefits of taking a multi-objective perspective toward challenges in RL.
2

---

Under review as a conference paper at ICLR 2022
Algorithm
Step Improvement
Step 2: Projection
LS
((als) #x(a
E myy. (x.w)
arg ming Ex-p
DKLL (a(-1s) ([(+
MO-MPO
9 (als) x (a]s) exp
ming Espe
Σx (a
My (eyg)
DIME
(a(s)) xc (ale) exp
arg ming Ex-o
Ex (gt (1)))))))))))))
%
Table 1: Comparison of policy improvement step for existing approaches versus DiME, Adjustments to the single-objective setting to accommodate multiple objectives are highlighted red.
3 BACKGROUND
We consider the RL problem defined by Markov Decision Process (MDP). An MDP consists of states s E S, actions E A, an initial state distribution p(so), transition probabilities p($++ 1|s,, reward function r(s, a) E R, and discount 7 [0,1), A policy ne (alss) is a state-conditional distribution over actions, parametrized by 0. Together with the transition probabilities, this gives rise to state visitation distribution uus)). The aclion-value function is the expected return from choosing action a in state s and then following policy π:: Q (s. a) = Eg -==o7"r(ss
[x=0 7'r(St, as)[sp = S, Go =0
This can also be written as Q" (s, a) E (r(saa) 7V* (8'), where V = Ra (8,
In this paper, both the LS and DiME approaches to multi-objective RL follow a policy iteration scheme, repeatedly alternating between policy evaluation and policy improvement. In the policy evaluation step, at iteration the current policy Te, is evaluated by training a separate value function Q for every objective k. This requires bootstrapping" according Q-decomposition (Russell & Zimdars 2003). Any policy evaluation algorithm can be used to learn these Q-functions. In the policy improvement step, given the previous policy n; (short for TT, when clear from context) and associated Q-functions {Q k= [Q the aim is to improve the policy for given state visitation distribution μu
3.1 POLICY IMPROVEMENT: SINGLE OBJECTIVE
DiME introduces a new way to perform policy improvement with respect to multiple objectives, so we will now describe the policy improvement step, first for the single-objective setting. Policy improvement can itself be decomposed into the following two alternating steps (Ghosh et al. 2020).
Improvement.
This step finds (nonparametric) policy g that improves with respect to the Q- function, while staying close to the current iterate n. leading to the Lagrangian objective function
F(q: 0,) =
[(((((()) a1] En (()
(I)
a~4g
with the known analytic solution q(ass) x (a(as exp(Q(s, a)/n) (Peters et al. 2010).
Projection. This step projects the improved policy to the space of parametric policies by maximizing the following (which simplifies to policy gradient if one uses parametric q in the improvement step):
(()) Eggp DKL (q(-1s) mo(ss)))
3.2 POLICY IMPROVEMENT: MULTIPLE OBJECTIVES
Suppose the policy must improve with respect to K objectives instead. In multi-objective problems, a solution is Pareto-optimal if there is no other solution with better performance for one objective without worse performance for another. The set of all Pareto-optimal solutions is the Pareto front. To find a specific Pareto-optimal policy, let a = fau K k=1 define preference trade-off across objectives, such that all on 2 O. We can further enforce Σe O& 1. A larger og means the k-tth objective is more important, and should lead to increased influence of objective & on the policy optimization.
Linear Scalarization. Here the trade-off is used to produce a single scalar reward via weighted sum of the objectives. This leads to the nonparametric solution for g in Table while the projection step the same as for single objective. drawback of LS is that not only the trade-off', but also the scales of the Qr. influence the contribution of each objective.
Not all objectives are necessarily evaluated via bootstrapping, but we denote all as Qs to lighten notation. We will drop the superscript on Q always assuming the latest iterate of the policy evaluation algorithm.
3

---

Under review as a conference paper at ICLR 2022
MO-MPO. In contrast, Abdolmaleki et al. (2020) optimizes separately for each objective, to obtain K improved policies 19kl2=1 The trade-offs } specify the constraint threshold on the KL-divergence between 9s and Te. As seen in Table T this leads to nonparametric solutions with temperatures T),, that are each a solution to a convex optimization problem involving Og.
A main advantage of MO-MPO is that it combines objectives in scale-invariant way unlike LSS. However, because the trade-offs in MO-MPO define KL-constraints, the improvement operator used for qu must be able to meet this constraint exactly. This means MO-MPO cannot be easily applied to non-standard settings like offline RL, as we will discuss later in Sec. 5.2 Thus we propose DiME, which can be applied non non-standard settings and maintains the benefit of ccale-invariance
DISTILLED MIXTURE OF EXPERTS
Like MO-MPO. DiME computes separate ya per objective. As seen in Table the nonparametric solution is almost identical, with the crucial exception that the temperature 7/ no longer depends on ag. Unlike LS and MO-MPO, DiME only incorporates the trade-offs in the projection step objective
JIME (0) --E ar DKL (ac(-ls) mo(-1s)).
(3)
so any improvement operator can used to obtain qs- This allows us to apply DiME to offline RL and other non-standard settings. In addition, the trade-offs in DiME more interpretable than those in MO-MPO, since they are weights on local experts gk. rather than KL-divergence constraints. Appendix B describes DiME in more detail, including how the temperature mi is computed.
However, we may not a priori know what the preference trade-off across objectives should be. In this extend DiME leam conditional policy, thereby distilling the entire Pareto front of optimal policies into a single ( a). We train this policy by augmenting states with normalized trade-offs a sampled from some distribution v. and then optimizing the following objective:
JiME (0) --
- srup
Ci DRT. (((()180)) To (1)))))))
(4)
where (als, a:) x (a(as, a) exp(Qu (s. a, a)/mk) Note that in order to evaluate this policy, we must also learn an a-conditioned value function Qu(s, a, a). Appendix A contains more details.
After training an a-conditioned policy, one can use (a) Ez~- J T a a))x (s, a) da fully offline to pick the best trade-off a with respect to objective kt using any hyperparameter optimization (HPO) algorithm, including simple search. If the learned Q; are inaccurate, which is often true in offline RL, we can instead deploy the policy to search for the best trade-off. This is cheap; in our case studies is between 0 and 1. In this paper we aim to learn high-quality a-conditioned policies, which we will show is possible using DIME. Investigating other ways of picking a is out of this scope.
4.1 MULTI-OBIECTIVE RL AS PROBABILISTIC INFERENCE
We will now derive LS and DiME from the RL inference perspective for MORL, showing that both optimize for the same high-level problem, but differ in their choice of variational distribution. Consider a binary random variable Rt indicating policy improvement event: R indicates the policy has improved, while R = O indicates that it has not. Policy optimization then corresponds to maximizing the marginal likelihood E,, log pe (R)) with respect to 9. Employing the expectation- maximization (EM) algorithm and assuming conditional probability p( RRs, a) x (s,(((s,a)/1 one can recover many algorithms in the literature (Levine 2018).
Let us instead define a separate indicator Rs for each objective, with similarly defined conditional probabilities, and consider the following weighted ligeeiiel objective:
K
K
F(0) = E, log II po Rxls)
Σ 1=1
Bn log po (Rxls),
(5)
if the improvement step is scale-invariant, which is true for the known nonparametric solution if we optimize the temperature 7 meet constraint on the Kl-divergence, as is done in e.g. MPO (Ahdolmaleki et al. 2018
4

---

Under review as a conference paper at ICLR 2022
where E is a shorthand for Egwa and a E R specify preference trade-off across the K objectives. This generalization has a few nice properties. First, when all objectives are equally preferred, the ag are all equal and it reduces to a straightforward probabilistic factorization of independent events: p([ (())) = II pl Re). Second, any vanishing Okg leads to an objective being ignored.
Linear Scalarization. To apply EM, could introduce variational distribution q{a]s) and use it to decompose and lower-bound ( This choice of variational distribution results in LS: the E-step corresponds to I with Q(s, a) replaced by Σx on QM (s, a), and the M-step is identical 2
DiME. We could instead introduce a separate variational distribution qJ for each log Re s), which results in DiME. The E-step now corresponds to K independent copies of the improvement step in each with its own Qu a). The M-step is the projection step introduced in (3).
This reveals that both standard LS and DiME optimize the same information teformatioo-thhcoreiic objective (see Appendix [B for complete derivation). Both employ EM, but differ in their choice of variational decomposition. We hypothesize that fitting separate variational distribution per objective allows DiME to find a tighter variational lower bound, which may lead to better optimization.
5 CASE STUDIES
Our primary aim in this work is to showcase MORL as a tool for tackling fundamental challenges in deep RL. We therefore demonstrate its benefits in two active areas of research: finetuning and offline RL. Finetuning is concerned with improving upon an expert (or teacher) policy, whether it is available in the form of parametric model or dataset of transitions. Meanwhile in offline RI., one must learn a policy from a given fixed dataset of transitions. In contrast to offline RI., finetuning happens in an online fashion so the agent interacts with the environment to collect additional experience.
These two settings share a fundamental similarity-they both use behavioral priors to make learning more efficient. Existing algorithms in both settings constrain the policy optimization, either explicitly or implicitly, to stay close to the given priors (Levine et al.) 2020 Monier et al. 2020; Schmitt et al., 2018). Instead, we propose explicitly formulate these the MORL framework with two objectives:
1. one for the task return via the task Q-function Q(s, a) learned via bootstrapping, 2. and one for staying close the behavioral prior via the log-density ratio log Ta(a s)
(a]a)
where n, is the current policy and mi is the behavioral prior. When we have direct access to the density Tg. we can evaluate the leg-density ratio pointwise. However, in many real-word world applications, we only have access to samples from Te via dataset of historical interactions. In the following subsections, we dive into each of these cases, deriving the corresponding policy losses for both LS and DiME. We use a scalar a E (0, 1] to specify the preference trade-off between the two objectives.
5.1 FINETUNING
Equipped with the MORL perspective, we introduce two novel finetuning algorithms by substituting the objectives above in the LS and DiME rows of Table yielding the policy optimization functions
1
exp (1 a1-a) (11-))((,0 ((s.a)
log (a(s)))
(t) = E smyje
log To (als)
and
(6)
Zo(s)
(a(s))
log (als) (alss)
JDiME(0)
= smuze arakis
(1 (1-a) a)
(s) exp
(Q(s,a) a)
+
a
exp
log (a(as)
(7)
1
Zz(s)
All functions Z, (s) normalize the product of n; and the corresponding tempered and exponentiated objective; though analytically intractable, they can be estimated from sampled actions for each state. Note that both of the equations above are equivalent at the extreme values of a = 0 and 1.
Relation to existing approaches. We recover existing algorithms with slight modifications to these policy objectives. In particular, setting 7/2 1 in the DiME row of Table T yields 92 = T3 and
'The log-density ratio is typically used as reward for distribution matching Arenz Neumann, 2020 'eer, and in our finetuning experiments, we assume the prior is parametric model. If the prior instead dataset of transitions, then the update rules identical the ones derived later for offline RI.
5

---

Under review as a conference paper at ICLR 2022
the following policy objective function:
1 (((saa) log To (
(1 ax E goup
(1(x) exp
ar Es~a DKL (T (T))
(8)
1f
which is similar to the finetuning approach in (Schmitt et al. 2018). Similarly, taking instead the LS approach to MORL and employing the policy gradient (which uses parametric q in Sec. 3.1 as the underlying policy optimization, we get:
(a(als)
E snipe
(1 Q)) a) + a log my (a|s)
(1 a)) 4~~ amay
Q(s,a) - D (Toll (9)
which similar to (Galashov et al. 2019). Both simply add a weighted KI. regularization to a standard RL objective, but they differ the direction of the KL.
5.2 OFFLINE RL
Recall that in offline RL, we only have access to the behavioral prior тT via a dataset Dr, of sampled transitions (s, r, s'); we cannot evaluate its density and thus cannot directly compute the objective function for staying close to the prior. Instead, we must simplify both JLS and J/IIE from the finetuning subsection so that they only require samples from TT and μu, for which we can rely on D Pr..
Linear Scalarization. By tying the preference and scale parameters so that 17. we can change the expectation over actions (6) analytically to be with respect to the behavior policy T
Q(s, a)
( ED.,
exp
log Zo log To (als)
(10)
where 3 - a/ a). There are close similarities between this LS- based policy objective and other policy losses used in recent offline RL algorithms. For example, 3 corresponds to the temperature in CRR (Wang et al. 2020) and AWAC (Nair et al. (2000), and we can recover them exactly by replacing the task return objective with the advantage function. As an other example, BEAR (Kumar al. 2019) and BCQ (Fujimoto al. 2019) can both be formulated using LS, but with the policy gradient as their underlying policy optimizer as in (9). However, unlike the objective function in (T0). which only requires samples from T6, these methods have the limitation that they require modeling
T6 to evaluate the second term in (9) (Levine et al., 2020 Monier et al. 2020
DiME. Setting 7/2 and renaming 7j 11 in 77, we obtain
(((saa) log To(als) + a ED., log (als)...
(D) = (1 - a E
amp
(11)
(s)
Notice that the first term the loss used by policy optimization methods such as REPS (Peters et al. 2010) and MPO (Abdolmaleki al. 2018), ) and the second term is exactly a behavioral cloning loss. We call this new algorithm DiME (BC). This is a simple way of adapting online algorithms for offline RL, while being theoretically grounded in multi-objective perspective. Intuitively, since both terms are independent of the scale of the Q-values, optimizing the weighted sum is a reasonable thing to do. This is in contrast recent concurrent work that adds behavioral cloning loss to policy gradient loss, which depends on the scale of the Q-values (Fujimoto & Gu 2021).
Remark on drawbacks of LS approaches. First, the preference a and scale 7 can longer be set independently; they are coupled together in the temperature 33. This means we cannot efficiently train mndde-offcoondtiooee policy using the LS-based loss in (IO), because the normalization constant Zo J(e.a)=-D., exp(Q(s, a)/3) depends on the choice of the trade-off- would
require estimating (T)) via samples separately per trade-off. Second, since JLS only involves an expectation with respect to the expert behavior policy, the policy cannot evaluate and exploit the Q Q-value estimates for actions beyond those taken by the expert. One could argue that this is by design, in order to avoid fitting extrapolation emrors in the Q-function. However, with JDIME this is choice we can make by setting the preference parameter independently (and perhaps adaptively).
Remark on using other MORL approaches. Although there exist many MORL algorithms that outperform LS on standard MORL tasks, they typically require that the rewards for all objectives given. Thus it is not straightforward to apply these algorithms for offline RL, where we cannot directly compute the reward for staying close to the prior. We tried getting around this in order to use MO-MPO for offline RL, but it was very complicated implement (for details see Appendix B.4).

---

Under review as a conference paper at ICLR 2022
Schallie
Famecaa Flenning
walk
run
aporaago took rrward
1000
Ilypervolume
walk
5000
LS
1.26x 10
DIME
1.35x10
-1.0
-1.00
-0.50
0.00
run
(1
r2(4)
average negative action
-1000
-2000
LS
DIME
2.58x106
1.75x106
DIME LS
DiME LS
(c)
(a)
Figure I: Each point is a solution found for a different tradcoff a. (a) DiME can find solutions on a concave Pareto front (right) whereas lincar scalarization (LS) cannot. (b,c) For humanoid, DIME finds better solutions and obtains higher hypervolume. Above (i.e., higher task reward) and to the right (i.c., lower cost) is better.
EXPERIMENTS
We first investigate how DiME performs on standard MORL tasks, compared to LS (Sec. 6.1 Since our focus is on applying MORL for general challenges in RL, we do not consider MORL algorithms that cannot be applied to offiin RI. Then we investigate the application of DiME in two case studies, offline RL (Sec. (.2) and finetuning (Sec. (6.3). where our aim is to evaluate whether replacing L.S with DiME leads to better-performing algorithms. Implementation details are in Appendix C
6.1 MULTI-OBJECTIVE RL
Toy Domain. Our bandit task has action a E R. and reward r(a) E R², with the reward defined by either the Schaffer or Fonseca-Fleming function (Okabe et al. 2004). The Pareto front is the set of all points of the function. By varying the trade-off, DiME finds solutions along the entire Pareto front for both functions. However, no matter what the trade-off is, LS only finds solutions at the extremes of Fonseca- Fonseca-Fleming, because this Pareto front is concave (Fig. Tap. This is a fundamental limitation of LS it cannot find solutions the concave portions of Pareto front (Das & Dennis 1997).
Humanoid. We also evaluate on two humanoid tasks from the DeepMind Control Suite TTasa et al. 2018). For each, the original task reward objective is augmented with "energy" eexendituue penalty on actions: -Ila]|2. In both tasks, the Pareto front found by DiME is superior to that found by LS, both qualitatively (Fig. [b) and in terms of hypervolume, which is standard metric for capturing the overall quality of a Pareto front (Table Icp.
Additional Experiments. We also find that DiME ocale-invariant-iit obtains similar performance when the energy cost is scaled up by ten times, whereas LS no longer can learn the task. DiME also performs on-par with MO-MPO, which state-of-the-art. Please refer Appendix for details. Recall that DiME can be applied to non- standaa settings like offline RL, whereas MO-MPO cannot.
6.2 OFFLINE RL
Baselines. For offline RL. our main baseline uses LS trade off between the task return and staying close to the behavioral prior, with the advantage function as the task return objective. This optimizes for IF (e.a)>DDr
a) log (a]s)) This the same policy loss as CRR-exp (Wang al. 2020) and AWAC (Nair et al., 2020), two state-of-the-ar offline RL. algorithms. Our behavioral cloning (BC) baseline optimizes for ((,,a)DD., log To (a(ss)) which is LS or DiME with a = 1.
Variations of DIME. For offline RL, our HAME-based algorithm optimizes for (II.. where the second term behavior cloning loss; we call this DiME (BC). For this second term, we could instead use an advantage-weighhtg is exactly the CRR policy loss with a temperature of 1; we call this DIME (AWBC). Since training a separate policy per trade-off is computationally expensive, we also experiment with training single conde-off-condition
behavior cloning loss, Emy(als) (exppl [exp(AA A(s, a)) log To (a]s)). which
policy To (a]s, a),, that can optimize for all trade-offs in the range [0, 1]. We refer to this as DiME multi, and combine it with both BC and AWBC. We choose train a single policy that can optimize for all trade-offs. rather than learning the best trade-off during training, because it is well-known problem that policy evaluation is inaccurate in offline RI.
7

---

Under review as a conference paper at ICLR 2022
LS
DIME
DIME (AWBC)
DiME
DIME (AWRC)
Task
BRAC
CRRi
MZU
BC
(CRR)
(BC)
LS (CRR)
DIME (BC)
DIME (AWBC)
(BC)
o=.3
o=.45
o=.5
multi
multi
caripole swingup
869
664
343
380
865
832
882
881
1 878 TT
cheetah rum
539
377
799
376
628
705
759
796
733
759
808 7 802 1
finger turn hard
227
714
405
132
360
741
8.37
515
885
892
899 + 903 +
fish swim
222
517
585
493
566
652
666
613
652
669
721 =4 701 +:
humanoid run
10
586
633
390
507
596
582
635
638
660
636 3 661
manip insert ball
56
625
557
554
578
652
626
+ 709
manip insert peg
50
387
433
261
409
304
281
410
386
435
432 + 7 436 1
walker stand
829
797
760
286
672
783
861
706
955
951
964
= 1 967
+
walker walk
786
901
902
361
865
842
796
868
955
957
961
1 962 11
mean score
399
641
602
333
584
671
688
665
749
759
778
780
L. (CRR)
DIME
DIME
Task
BRAC
BEAR
AWR
BCQ
CQL
BC
LS (CRR)
(BC)
(AWBC)
Q=.63
multi
multi
antmaaze mean score
0.23
0.23
0.22
0.25
0.48
0.15
0.78
0.89
0.78
0.88
kitchen mean score
0.0
0.80
0.33
0.47
1.90
2.09
2.16
2.49
2.55
2.13
Table 2: Offline RL results for sfatc-of-the art approaches (left columns), and our baselines and DiME with either a single trade-off across all tasks (middle) or the best trade-off per task (right). Tasks are from RL Unplugged (above) and D4RL (below). For DIME multi we include standard error bounds across ten random seeds. Numbers within 5% of the best score in bold. DiME performs par better than LS across all tasks. Even when the trade-off is fixed across tasks (for a more fair comparison with prior work), DiME still outperforms state-of-the-aat on the RI. Unplugged tasks. For RI. Unplugged, the results for BRAC are from Gulcehre et al. (2020): CRR is from Wang et al. (2020). referred to as "CRR exp" in their paper: and MZU (MuZero Unplugged) is from [Schritwieserr et al. (2021). For D4RL. the results for the italicized algorithms from Fu et al. (2020). tNote that CRR results are for the best checkpoint from training, whereas all other results denote performance after a fixed number of learning steps. Per-task results for D4RL are in Appendix [
Evaluation. We evaluate on tasks from two offline RL benchmarks, RL Unplugged (Gulcehre et al. 2020) and D4RL (Fu et al. (020), with results in Table For LS, DiME (BC), and DIME (AWBC), train one policy per trade-off a, linearly spaced from 0.05 to 1. For DiME (BC) multi and DiME (AWBC) multi, we train ten policies with different random seeds. We train all policies for one million learner steps. After training, we evaluate the policy in the test-time environment once per trade-off.
From RL Unplugged, we evaluate on the nine Control Suite tasks. For both LS and DiME, the setting of a has a significant impact on performance, and the optimal setting is different per task (Fig. ( Comparing the best performance obtained for each task (across all trade-offs). all DiME variants outperform LS. In addition, training a single raaeofff-ondditooee policy (with DiME multi) is not only more efficient, but also leads to better performance. This may be because conditioning on regularizes learning. With LS though, we cannot efficiently train a-conditioned policies because a controls both the trade-off and scaling. To make more fair comparison against prior work, we also find the best fixed trade-off across all tasks for LS and DIME (BC/AWBC), and report these results in Both DiME variants still outperform existing approaches.
the middle columns of Table 2
From D4RL., we evaluate on the six Ant Maze tasks and three Franka Kitchen tasks. We only evaluate the DiME multi variants, with one random seed, since they perform best the Control Suite tasks and have low variance. LS and DiME perform on par, and both outperform state-of-theaa approaches.
6.3 FINETUNING
Finally, we will evaluate whether our new finetuning algorithms, derived from the multi-objective perspective, can improve upon the performance of a teacher policy. We evaluate both the L.- and DiME-based algorithm with either a fixed or learned tradeoff. To learn the tradeoff, we update based on the loss J(a)
(Es-g,arn, Q(s, a) c). This is analogous to the update for the Lagrange multiplier in Lagrangian relaxation for constrained optimization. The resulting strategy is intuitive: stay close to the behavioral prior while the expected return is below the threshold and otherwise optimize for the bootstrapped Q function. We pick e based on the expected return from fully imitating the prior for the given task (i.e., 1); these values given in Appendix C
8

---

Under review as a conference paper at ICLR 2022
Ennger tuen hard
humasoid run
manip insert bal)
Bager turn hard
humnanoid run
misuip lisert
1000
1000
ang taok retam
wan task mntumn
500
A
300
Le
1.0
0.5 tradooff
1.0
0.5 tradbodf
0.5 trudcoof
1.0
tradeodf
trudcoE
trudkodd
DEME (BC)
DIME (BC) malti
LS
DIME (AWBC)
DiME (AWBC) muiti
LS
Figure 2:
Per-tradeoff performance for a representative subset of tasks; Appendix D contains plots for all fifteen tasks. On some tasks, the best solution found by DiME (any variation) obtains significantly higher task performance than the best found by LS. In addition, DIME multi can train single policy for range of trade-off'. that performs comparably learning a separate policy each tradeoff, after the same number of learning steps.
hummnned stand
Inmanoid
manip insert bal.
lumoenoid stand
kumancid
manip insert ball
1000
task rosurd
500
20
nation stope (xc10")
actor (xC1U)
0.0
0.5
learned
Finetuning learning curves, for DIME (left three) and LS (right three). (Appendix D contains plots for all five tasks.) For both DIME and LS, learning the tradeoff' (orange) converges to better performance than fully imitating the teacher (dashed purple), while learming quickly. Error bars show standard deviation for ten seeds.
Figure 3:
We evaluate on the humanoid and manipulator tasks from the DeepMind Control Suite. For the three humanoid tasks, we start with a suboptimal humanoid stand policy as the behavioral prior. For the two manipulator tasks, the behavioral prior policy trained for that task.
The choice of fixed tradeoff' impacts the result significantly, and the optimal choice depends on the task, so either searching over or learning the tradeoff is necessary. Across all five tasks, for both LS and DiME, the learned tradeoff converges to better performance than fully imitating the prior (a = 1) and learns much more quickly than learning from scratch (a = 0). This has the advantage training only single policy, as opposed to training separate policy per fixed tradeoff. Adapting the trade-off helps because finetuning, the agent is able to gather online interactions. An adaptive scheduling for the trade-off allows the agent initially copy the teacher and then eventually deviate from the teacher to achieve better performance based on its new experiences. In contrast, in the offline RL setting, needed fixed trade-offs because such online interactions not allowed.
7 CONCLUSION AND FUTURE WORK
In this work, we propose using multi-objective optimization as a tool for tackling challenges in RL. We explored the benefits of this in offline RL and finetuning, both of which involve staying close to a behavioral prior. Our derivations revealed that existing approaches optimize for a linear scalarization of two objectives, task return and a log-density ratio. We proposed a new MORL algorithm, DiME, as an alternative to LS. Applying DiME to offline RL leads to state-of-the-art performance.
A key takeaway is the impact of the trade-off on performance, and the need to either search over or optimize for As first solution we explored either training onde-offfconddtiooned policies or adapting the trade-off during training; it is worth exploring other approaches in future work.
In offline RL, a limitation of optimizing for the trade-off is that requires being able to accurately evaluate the policy for multiple trade-offs. We chose the simplest option in experiments: deploy the fulle-trained policy for each trade-off in the test-time environment, and pick the best-performing trade-off. But this may not be feasible in real- world applications, for instance if it is expensive or dangerous to deploy a policy. One option is to use recent offline policy evaluation algorithms to accurately estimate policy's 's test-time performance (for all trade-offs) without requiring deployment.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
In Appendix C we provide full implementation details for DiME, our novel algorithms for finetuning and offline RL, and the baselines. We report hyperparameter settings and network architectures in Tables 3 and 4 also in Appendix C Our code is based the open-sourced implementations of MO-MPO and MPO in Acme (Hoffman et al., 2020), which an open-source RL framework. We in the process of open-soureing the code.
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. In Proceedings of the Sixth International Conference on Learning Representations (ICLR), 2018.
Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo F. Martins, Nicolas Heess, Raia Hadsell, and Martin Reidmiller. A distributional view on multi-objective policy optimization. In Pruceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods, 2020.
Gabriel
Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. In International Conference Learning Representations (ICLR), 2018.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
Marc G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. CoRR, 71111777706887,, 2017. URL htt.:::/arxii...t/ wwwp:////////..oo..//./oo /bbp://0700008777
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E. Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In Proceedings of the 24th International Conference on Artificial Intelligence (ICCAI), 2015.
I. Das and E. Dennis. A closer look drawbacks minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Structural Optimization, 14:63-69, 69, 1997.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv22004.02199, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. arXiv preprint arXiv:2106.0668 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learn- ing Research, pp. 2052 2062. PMLR, 09-15 Jun 2019. URL httt:::/prrceeeinnsssse.mo/.ppee
2xess/v97/fujime
9a. html
Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry KL-regularized RL. Proceedings of the 7th Interna- lional Conference on Learning Representationss (ICLR), 2019.
Dibya Ghosh, Marlos C. Machado, and Nicolas Le Roux. An operator view of policy gradient methods. CoRR, abs/2006.1 1266,, 2020. htt.://ee//aa...oooo/aa///////// 11266
10

---

Under review as a conference paper at ICLR 2022
Daniel Golovin and Qiuyi Zhang. Random hypervolume scalarizations for provable multi-objective black box optimization. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arold, Jerry Li, Mohammad Norouzi, Matthew Hoffman, Nicolas Heess, and Nando de Freitas. RI unplugged: A suite of benchmarks for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 1707 022000 2017.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.09979 2020.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations (ICLR), 2017. 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the Third International Conference on Learning Representations (ICLR), 2015.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy
q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems, 2019.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement leannin. In Reinforcement learning, pp. 45-73. Springer, 2012.
Sergey Levine. Reinforeement learning and control probabilistic inference: Tutorial and review. arXiv preprint arXiv: 000018000000000 2018
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, 2016.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of Pareto dominating policies. Journal of Machine Learning Research, 15:3663-3692, 2014.
Louis Monier, Jakub Kmec, Alexandre Laterre, Thomas Pierrot, Valentin Courgeau, Olivier Sigaud, and Karim Beguir. Offline reinforcement learning hands-on, 2020.
Hossam Mossalam, Yannis M. Assacl, Diederik M. Roijers, and Shimon Whiteson. Multi-obeective deep reinforcement learning. arXiv preprint 110.07161. 2016.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
11

---

Under review as a conference paper at ICLR 2022
Andrew Y. Ng. Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. Proceedings of the Sixteenth International Conference Machine Learning (ICML), 1999.
Tatsuya Okabe, Yaochu Jin, Markus Olhofer, and Bernhard Sendhoff. On test functions for evo- lutionary multi-objective optimization. In Parallel Problem Solving from Nature PPSN VIII. 2004.
OpenAI, Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. arXiv preprint orXiv:/808.00/1 2018.
Simone Parisi, Matteo Pirotta, and Marcello Restelli. Multi-objective reinforcement learning through continuous pareto manifold approximation. Journal of Artificial Intelligence Research, 57:187-227, 2016.
Simone Parisi, Matteo Pirotta, and Jan Peters. Manifold hansed multi-objeccive policy search with sample reuse. Neurocomputing, 263:3-13 14: 2017.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning, arXiv preprint arXiv:1511.03342 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics, 37 (4), 2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighteed regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00/77 2019.
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI' 10, pp. 1607-1612. AAAI Press, 2010.
Mathieu Reymond and Ann Nowé. Pareto-DQN: Approximating the Pareto front in complex multi- objective decision problems. In Proceedings of the Adaptive and Learning Agents Workshop al the 18th International Conference on Autonomous Agents and MultiAgent Systems AAMAS, 2019.
Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision-making. Journal of Artificial Intelligence Research, 48(1 (8)6)36711 2013.
Diederik M. Roijers, Shimon Whiteson, and Frans A. Oliehoek. Linear support for multi-objective coordination graphs. Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 1297-1304, 2014.
Stuart Russell and Andrew L. Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the Twentieth International Conference on International Conference Machine Learning (ICML), pp. 656-663, 2003.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint 15 2015.
Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M. Czarnecki. Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and M. Ali Eslami. Kickstarting deep reinforcement learning. arXiv preprint arXiv: 1803 03835, 2018.
12

---

Under review as a conference paper at ICLR 2022
Julian Schrittwieser. Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, loannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. arXiv preprint arXiv:2104.06294, 2021.
Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin A. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. CoRR, abs/2002.08336. 2020. URL https: / www.iiivv............. org abs/ 2000000022222 08396
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, Kalchbrenner, Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, $29:484-503, 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint
2018.
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 191-199, IEEE, 2013.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. arXiv preprint arXiv:2 2021.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guiie guided multi-objective reinforcement learning for continuous robot control. In Proceed- ings of the 37th International Conference Machine Learning (ICML), 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS), pp. [4610-14621) 2019.
Marcela Zuluaga, Andreas Krause, and Markus Püschel. e e-pal: An active learning approach the multi-objective optimization problem. Journal of Machine Learning Research, 17(1):36193366 2016.
13

---

Under review as a conference paper at ICLR 2022
A TRAINING TRADE-OFF-CONDITION POLICIES
In this section we provide details on training a trade-off-conditionn policy To(a]s (, a) conditioned on preference parameters v(a). (Note that here a corresponds exactly from the main body of this manuscript; we had to distinguish it from the single scalar that appears in our case studies, but we drop the underline here to lighten notation.) Conditioning the policy on the trade-off a requires that the Q- Q-functions are also trade-off-come cili-off--oonitionedd We also use hindsight relabeling of trade-offs to perform off- off-po learning.
policy evaluation. We evaluate the current policy т; (a]ss, a) via off-policy learning of per-objective Q-functions Qi (s, a, ax.. When training the Q-network, every sampled transition (s, a, {rx (= s ) augmented such that the states include a sampled trade-off parameter V, resulting in transitions (Is, a). a, [= al). From here, any critic learning algorithm can be used; we use distributional Q-learning (Bellemare et al. 2017 Barth-Maron et al. 2018). See Section for more details.
Preference-conditiooee policy improvement. The policy improvement step produces a trade-off- conditioned policy π (a|s,, a:) that improves upon its predecessor ne
Improvement: To find per-objective improved action distributions ma(als, a), we optimize the follow- ing policy optimization problem for each objective:
max Ok
Eswz (L (s(s,a,
(12)
s.t.
Eany ()
S Ek.
(13)
Similarly, from here we can follow steps analogous to those in Section B.1 but with trade-off- conditioned policies and Q-functions. This means that this set of trade -o-niiiiond problems have solutions that are analogous to (38) for each objective.
Projection: After obtaining per-objective improved policies qe. we can use supervised learning to distill these distributions into new parameterized policy:
mAx 6
Σ on DKL. (1((111,a)) a)) no(-18,0)) k=1
(14)
S.t.
Eawy
DKL (1s a))] < 8
subject to a trust region with bound use Lagrangian relaxation as described in Abdolmaleki et al. (2018 2020).
for more stable learning. To solve this optimization, we
In practice. To approximate the expectations over the state and trade-off distributions, we draw states from a dataset (or replay buffer in online RL) and augment each with sampled trade-off ~ v. To approximate the integrals over actions a, for each (s, a) pair we sample N actions from the current policy a.).
B ALGORITHMIC DETAILS AND DERIVATIONS
B. OULTI-OBIECTIVE RL PROBABILISTIC INFERENCE
In this section we present further details of our derivation, which unifies both linear scalarization and DIME, relying on the RI. as inference framework. We begin by recalling some of the motivation behind the objective.
Consider a set of binary random variables Rg indicating a policy improvement event with respect to objective k; Rs 1 indicates that our policy has improved for objective k while R2 0 indicates that it has not. We seek a policy that improves with respect to all objectives given some preferences over objectives. Concretely, we seek policy parameters 0 that maximize the following marginal likelihood, given preference trade-off coefficients {ax
max E,, log.o popp = Ho 0
(15)
14

---

Under review as a conference paper at ICLR 2022
Assuming independence between improvement events R3 the following model to enforce trade-offs between objectives, i.e,
K (0)) y log II po (R
Σ By log po (Rx
(16)
This generalization has few nice properties. First, when all objectives are equally preferred, the oi are all equal and reduces to a straightforward probabilistic factorization of independent events: p( ) II (R)).. Second, any vanishing ag leads to objective being ignored.
Consider the intractable marginal likelihood L that we wish to maximize, then introduce variational distributions ds and expand as follows
K
F(0) En Pe (R)ss)
(17)
k=
Σ
O E.g. log po Rs(s)
(18)
K
po Rk, a ss) (a) Rxs s) ga (a s) pa [Rx, als) pa(a Rs s) qu (a)s)
ag Eg
log
(19)
k=1
K
04 Eg log
(20)
Bn [DKI. (gu(a)s) ( s)) (I
(21)
Σ E. [Dx (l 8)) (eu.aalals)l((Rsssss))o
(22)
k=
Let us denote the second term as F, defined as follows
K
L(O; {gxh) == Σ
on DgL (als) | (Rx ( a))ss))))
(23)
k=1
Then one can rearrange terms in the expansion of the marginal likelihood F to yield
F(0) DKL (a (a,R, s)) L(0; {gx1).
(24)
Since the KI, divergence is a non-negative quantity and the 0g are all positive, this last equation shows that T. lower-bounds the quantity wish to maximize, namely F. Whether call it sxpectation-maximizztioo or improvement-projection the steps are as follows:
1. tighten the gap between the lower bound I. and F at the current iterate 0,, by maximizing T. with respect to qu while keeping = 0; fixed, then
2. maximize the lower bound L with respect to 8, holding 9u fixed to the solution of step 1.
Formally, these two steps result in the following optimization problems
K
max L(0;; (gx1) min {xr)
a,L, (aa
(25)
{
K
max L(0;; (gx]) = min
Dkt (a|s) (a)sss)) ([
(26)
8
where in the second step removed terms that do not depend on 0. These steps are respectively referred to as the E- and M-step in the statistics literature, and improvement and projection policy decoupled from the original goal of
optimization. While these two optimization problems maximizing F, the connection with the EM algorithm allows us to benefit from the guarantee that iterates 0g obtained in this way lead to improvements in F.
15

---

Under review as a conference paper at ICLR 2022
B.2 IMPROVEMENT OR E-STEP
In the next sections we detail the two steps in turn, beginning with the improvement E-step, where first notice that the objective weighted of K independent terms that can all be minimized separately
K
arg min {ge]
B (gx II a)),ss))
(27)
arg min By (pa((s)))) || R& (a)ss))) =k=1... 1...,..., K.
(28)
q
These separate problems can be solved analytically by making the KL vanish with the following solution
p (( qh (als) = J (<(sss)p(( a) da
(29)
This nonparametric distribution reweights the actions based on the improvement likelihood p(R& (8, a). This likelihood is free for us to model as follows and standard the RL as inference literature
Qx(5.a)
pl [klss,, a) x exp
(30)
77
where V/ is a objective-dependet temperature parameter that controls how greedy the solution qo (a())) is with respect to its associated objective Q& a). Substituting this choice of p [ a)
into the independent problems (28) we obtain the following objective
DRI.(gn(ols) |l Rg (alss) Eqg log (a(ss)
(31)
p(Rg a.)))
Eyg log ya (als)
[a(a) logp(Rxls.o)
(32)
-DKL (a|s) ma(ss)) - Ey, log (Rlls,
(33)
Qu(s,a)
(gL (a|s) (((as)) Eng
(34)
After multiplying this objective by -Tk, we obtain the following equivalent maximization problem max Ey Qx(s,) - DKL
(a(a)s)),
(35)
where recognize KL regularization term that materializes naturally from this derivation. Indeed this last optimization problem can be seen as Lagrangian, imposing a soft constraint on the KL between the improved policy @ and the current iterate 0, If we instead impose hard such constraint, we obtain the following problem:
max E. Qu(s,a) (x(e(a( || (a(s))) S Ek-
(36)
s.t.
This problem's Lagrangian form is as follows, and is nearly identical (35) albeit
albeit with different
dual variable instead of 7ı; nevertheless we reuse the notation 77K. instead of introducing a new dual variable:
C(gk. mR) = E.g (s) 77s [E DKL (a)ss) - -ck]-
(37)
Putting everything together. Since the original problem is equivalent to the Lagrangian relaxation of this one, we can simply substitute our improvement likelihood (30) into the solution to the original problem 22 to obtain the solution:
Qx(s,a)
(a(aas) Zx(s) (als)) exp
where
(38)
Zx(s) = / (a(ss) exp
Qu(s, a) da.
(39)
7
16

---

Under review as a conference paper at ICLR 2022
Adaptively varying 7ik- Finally, this solution can in turn be substituted back into 37 following convex dual function:
to obtain the
1 Ze(s)
T. (als) exp (x,a,))
C (ma) = E, Qx(s,a) + n
Ck: Bg log
(40)
T=(aas)
1 exp
Qu(s, a
= E,, Qx(s,a) + MnEn
Cx Egg log
(41)
(s)s)
EN + log [x(s) Eng (s(s.a)
= E, (s a) + T
(42)
E. Qs(s, a) + By
(cx + log [s)] E,, (s(),))
(43)
= Ths [xx + E, log (s)]
(44)
Q&(5,a)
C² = T] 6 P. log B,, exp
(45)
7]
where in the final step we substitute (39 to reveal the hidden dual variable. In practice, this function optimized alongside of in order adapt the dual variable 7e and mitigate overfitting Qk-
B.3 PROJECTION OR M-STEP
Let us now look at the projection step (M-step in EM terminology), which produces the next iterate 0(+1 for our parametric policy. Recall the optimization problem involved in the M-step is the following
min L(0: min
ag DeL (a(a)s)),
(46)
k=1
K
min
Oa F.p F.g. log no (als),
(47)
where is fixed to the solution from the improvement step, namely (38). This problem corresponds finding the policy सy that acts barycentre between the nonparametric policies qk, each weighted by its specified preference trade-off coefficient at
Other than this important difference,, the multi-objective follows from the single-objective case in straightforward way. Substituting the solution Qı from 38 into the expression for L(0; qk) yields
K
JDIME (0) L(0: qu) E,,
ag log (a(ss)
(48)
K
Qx
E
ak Zx(s)
T((ass) exp
log (a(ss)
(49)
K
Qu(s,a) a)
E,,
En, exp
log To (als)
(50)
(s(s)
K
Qu(s, a)
E,
O
exp
log (a(as))
(51)
(s(s)
7
This is the objective function that is optimized to obtain the next iterate in practice. this stage we can further impose a trust-region or soft KI. constraint on this optimization problem.
In practice. We compute this policy objective via Monte Carlo, sampling states s from a replay buffer, then sampling actions for each state from previous iterate m; We use these samples to compute weights cxp(Qu(s, (s, a)/ my) and normalize them across states, which corresponds to using sample-based approximation to Zx(s) well. Finally, we take the convex combination of these weights, according to the preference trade-offs, which yields the ultimate weights- seights-seen seen brackets in (51)
17

---

Under review as a conference paper at ICLR 2022
B.4 ON HOW DIME IS BETTER SUITED THAN MO-MPO FOR OFFLINE RL.
In this section we expand on precisely why MO-MPO in its current published form is not readily suitable for offline RL and how our proposed method improves it in this aspect. Let us begin by revisiting the MO-MPO objective function (Eq. 1; Abdolmalleki et al. 2020) and substituting in the offline RL objectives discussed in the paper.
JMO-MPO (0) = -IE
DILLTTE Qu/va [TTO]
(52)
K
= ybn,
Σ
log T Bs, exx/me
+C
(53)
k=
log IA
Ep
log Re Ex, BRre
exp
log T
+ +
(54)
7/2
where we deliberatly omit the normalization constants to lighten the notation. Recall that MO-MPO enforces the preference trade-off across objectives by setting the KL-constraint thresholds e between the improved policies and the current policy iterate as in (45). Therefore the preference trade-offs implicit in the varying temperatures 7ık, which are continuously adapted throughout training to ensure the satisfaction of the corresponding constraint eg. As result, when applying the same simplifying trick as used for DiME setting 722 = 1 and renaming ? n-we completely. Concretely, the second term above reduces to an expectation with respect to the behavior dataset Di,, as follows:
we forfeit control of the trade-off off
= E,, e0/n log Th + ED., log he C.
(55)
where we recover the now common form of an RL objective combined with an additional BC regularization term. Notice that not only is there trade-off a but the temperature 72, which implicitly controls it, has also disappeared. Let us now compare this expression to that of DiME:
JDIME x E,, 20 log Th + 1- a ED., log Te.
(56)
where we highlight the key difference between the two losses, namely that the preference parameter a allows us to explicitly control the relative importance of the task's Q-value objective and the objective that keeps the policy close to the one that generated the offline dataset. In contrast, MO-MPO can no longer satisfy the preference setting specified by the choice of cg. because of the simplifying assumption that 7/2 = 1.
An alternative approach would be to the dataset Dr,, to learn distilled behavior policy स. to model Th. effectively performing behavioral cloning as a preliminary subroutine. This would allow us compute the log-density ratio directly, to use as an objective in the original expression (54). This strategy introduces approximation errors in the form of assumptions made on the form of ne and generalization states that are not present in Dr,. While we tried both this and the MO-MPO objective in (55) in preliminary experiments, neither proved to be competitive with the LS/ CRR baseline and were thus omitted from further investigations.
C IMPLEMENTATION DETAILS
This section describes how we implemented our algorithm, DIME, and the linear scalarization (LS) and behavioral cloning (BC) baselines. To ensure a fair comparison, we use the same hyperparameters and network architectures for DiME and the baselines, wherever possible. All algorithms are implemented in Acme (Hoffman al. 2020), open-source framework for distributed RL.
Training Setup. For the multi-objective RL and finetuning settings, we use an asynchronous actor-learner setup, with multiple actors. In this setup, actors fetch policy parameters from the learner and act in the environment, storing those transitions in the replay buffer. The learner samples batches of transitions from the replay buffer and uses these to update the policy and Q-function networks. For the offline RL setting, the dataset of transitions is given and fixed (i.c., there are no actors) and the learner samples batches of transitions from that dataset.
18

---

Under review as a conference paper at ICLR 2022
To stabilize learning,
the common technique of maintaining target network for each trained network. These target networks are used for computing gradients. Every fixed number of steps, the target network's 's weights are updated to match the online network's weights. For optimization, use Adam (Kingma & Ba 2015). For the offline RI. experiments, we use Adam with weight decay to stabilize learning.
Gathering Data. The actors gather data and store in the replay buffer. When the policy is conditioned on trade-offs, the trade-off is fixed for each episode. In other words, at the start of each episode, the actor samples a trade-off a' from the distribution v, and acts based on T [a|s, a') for the remainder of the episode. At the start of the next episode, the actor samples a different trade-off, and repeats.
Hyperparameters and Architecture. The policy and Q-function networks are feed-forward networks, with an ELU activation after each hidden layer. For both, after the first hidden layer, we add layer normalization followed by hyperbolic tangent (tanh); we found that this improves stability of learning. The policy outputs Gaussian distribution with diagonal covariance matrix.
The default hyperparameters we use in our experiments are reported in Table setting-specific hyperparameteers are reported in Table 4
Distributional Q-learning. For our policy evaluation we use distributional Q-network, C51 (Belle- mare et al., 2017). It has proven to be very effective critic in actor-critic methods for continuous control Barth-Maron et al. 2018 Hoffman et al., 2020), We use the exact same training procedure as published and in open-source implementations (Barth-Maron et al.) 2018: Hoffman et al.) 2020), i.e. n-step return bootstrap targets and projected cross-entropy loss. See Table under Q-learning for our chosen hyperparameters.
Evaluation. We evaluate a deterministic policy, by using the mean of the output Gaussian distribution rather than sampling from it. When we evaluate wrade-off-conditione) policy, condition it trade-offs linearly spaced from 0.05 to 1.0. For each trade-off a', we perform several rollouts where we execute the mean action from 7 (als,, o²). Recall that this trade-off specifies convex combination for two objectives.
For the multi-objective RL experiments, all policies are evaluated after 500 million actor steps. For offline RL, all policies are evaluated after million learner steps.
Learned Trade-off for Finetuning. our finetuning experiments, the learned trade-off is passed through the sigmoid function (i.e., 1/(1 + exp(-x))) to ensure it is bounded between and 1.
Compute Resources. To train single policy for the multi-objective RL experiments (humanoid walk and run), we used an NVIDIA v v100 GPU for about 24 hours. To train single policy for finetuning, we used a NVIDIA V100 GPU for 9-12 hours. To train a single policy for offline RL, we used a v2-32 TPU for about 6 hours for the RL Unplugged tasks, and for about 3 hours for the D4RL tasks. These computational costs were the same for all algorithms we ran: LS and the DiME variants.
D EXPERIMENTS: DETAILS AND ADDITIONAL RESULTS
This section provides details regarding our experiment domains, as well additional experiments and plots.
D.1 MULTI-ORIRCTIVE RI.
D.1.1 TOY DOMAIN
Our toy domain is a bandit with a continuous action a E RR. The reward r(a) E IR2 is specified by either the Schaffer function or the Conseca-Fleming function, f IR + R² Each point along this function is optimal for particular tradeoff between the two reward objectives, so the function can be seen as a Pareto front.
The Schaffer function corresponds to convex Pareto front, and is defined as
fi(a) a² and fz(a) (a 2)²
19

---

Under review as a conference paper at ICLR 2022
Category
Hyperparameter
Default
training setup
batch size
512 4
number of actors
replay buffer size
3 x 10-?
target network update period
100
Adam learning
10-4
policy & Q-function networks layer sizes
(1024, 1024, 1024, 1024, 1024, 512)
Q-lcarning
support
[- 150, 150]
number of atoms
101
n-step returns discount
5
0.99
policy loss
actions sampled per state
30
KL-constraint qu(als).
0.1
KL-constraint on policy mean, 3,
0.0025
KL-constraint on policy covariance, $g
10-6
initial temperature 1j
10
Adam learning rate (for dual variables)
10-2
Table 3: Default hyperparameters for all approaches, with decoupled KI.-constraint mean and covariance of the policy M-step.
Multi-Objective RL
training setup
number actors replay buffer size target network update period
10
200
experiment setup
trade-off' (DiME)
Ctrsk 1
Openalty E linspacee 1.5)
trade-offs (MO-MPO) humanoid walk humanoid run
Crak
Epenalty E liuspace (0, 0.3)
Cpenalty E linspace (0, 0.15)
trade-offs (LS)
Ctuek Openaalt
Oppenalty E linspace 0, 0.15)
policy loss
actions sampled per state KL-constraint on policy mean, 8 KI.-constraint on policy covariance, f8
20
0.001
10-7
Offline RL
training setup
Adam weight decay (RL Unplugged)
0,999999
Adam weight decay (D4RL)
0.99999
experiment setup
trade-offs
a E linspace(0.005, 1)
trade-off distribution, V
4400005,, 1)
policy loss
KL-constraint e (LS)
0.01
KL-coostraint e (RL Unplugged)
0.5
Kl.-constrintt (Kitchen)
0.001
KL-constraint (Maze, DiME multi)
0.5 (BC), 0.01 (AWBC)
Finetuning
training setup
batch size
1024
experiment setup
initial trade-off (for learned trade-off's)
0.5
Table 4: Hyperparameters that are either setting-specific or differ from the defaults Table
20

---

Under review as a conference paper at ICLR 2022
walk
Fun
walk
Average took revard
1000
1000
fack
- 10000
-2000
1000
-2000
negattive action
averapy action
DiME LS
DIME
MO-MPO
(A)
(b)
Figure 4: (a) This plot shows the per-episode task reward and action cost, rather than the average episodes, Note that the linear scalarization (LS) cannot solutions along Pareto front walk, because is likely concave. In contrast, DiME can find solutions that compromise between the two objectives. (b) DiME finds similar Pareto fronts as MO-MPO, state-of of wf-the-an art multi muuli-objective RL algorithm.
The Conseca-Flemingg function corresponds to concave Pareto front, and is defined as
(a) - exp( - (a 1)?) and 12(a) 1 exp(e + 1)?).
These are standard test functions in multi-objective optimization, where the aim is to minimize them (Okabe et al. 2004). Since in RL we want to maximize reward, we define the reward be the negative of these functions.
D.1.2 HUMANOID
We also evaluate on the humanoid walk and run tasks from the ooen-sourced DeepMind Control Suite (Tassa et al. 2018), with the two objectives proposed Abdolmaleki et al. (2020). The first objective is the original task reward, which is shaped reward that is given for moving at target speed (1 meters per second for walking and m/s for running), any direction. The second objective is the negative 22-norm of the actions, -||a||2, which can be seen as encouraging the agent to be more energy-efficient.
The observations are 67-dimensional, consisting of joint angles, joint velocities, center-of-mass velocity, head height, torso orientation, and hand and feet positions. The actions are 21-dimensional and in |1,, 1]; these correspond to setting joint accelerations. Each episode has 1000 timesteps. At the start of every episode, the configuration of the humanoid is randomly initialized.
Additional Results. When plot the per-episode performance (rather than average across episodes), we see that linear scalarization only finds solutions at the two extremes for the humanoid walk task, i.c. policies that achieve high task reward but with high action norm cost, or incur zero cost but fail to obtain any reward (Fig. 4a). We hypothesize that this may be because the humanoid walk task has concave Pareto front of solutions; recall that linear scalarization fundamentally unable to find solutions on concave Pareto front (Das & Dennis 1997). In contrast, DiME is able to find solutions that trade off between the two objectives.
On both tasks, DiME finds similar Pareto front as MO-MPO, which is a state-of-the-arr approach for multi-objective RL (Fig. 4b).
DiME requires training a separate policy per trade-off, which can be computationally expensive. In contrast, our trade-off- trade-off-cooditionn conditioned version of DIME, which we call DiME multi, trains a single policy for range of trade-offs. This obtains a similar Pareto front compared to that found by DiME (Fig. 5). after the same number of actor steps (500 million).
We also ran experiment to test the ocale-invariane, of DiME. We modified the humanoid run task such that the action norm cost now multiplied by ten. We used DiME and linear scalarization to train policies for three random seeds, for each of two trade-off settings. For both the original task and the task with scaled-up action norm costs, DiME trains policies with similar performance (Fig. 6 right). This is the case for both trade-off settings note the two clusters of tiffereentlycollorg points in the plot. In contrast, linear scalarization is not ccale-invariant: task performance suffers when the cost is scaled up (Fig. 6, left). With trade-off of 0.05, although this obtains better performance than
21

---

Under review as a conference paper at ICLR 2022
TAME
1000
Cloeet 1.5
1000
155
tack rooord
0000
600
tock
0.3
1000
1000
9000 action BOTT cost
-000 norm
actior Form cost
action BOTT
(8)
(b)
Figure 5: DiME multi trains a single oadd-off-conditioned policy, that finds a similar Pareto front as DIME does for humanoid (a) walk and (b) run. Each plot for DiME multi shows five trained policies, for different random seeds; these seeds vary somewhat in their performance because the policy leams different strategies (e-g., spinning. running sideways, etc.). Each policy is conditioned on trade-off's linearly spaced between O and The color denotes the trade-off. For DiME, Crask
1; DiME multi uses a convex combination (i.e..
Cheack (kcost).
LS
DIME
ateragee task resped
1000
300
0-
-4000
2000
0
-4000
-0000
average negatioo action
0.01
(original)
0.3 (original)
0.01 (10x penalty)
0.3 (10x penalty)
0.05 (original)
0.45 (original)
0.05 (10x penalty)
0.45 (10x penalty)
Figure 6: With original objective scales for humanoid run, both linear scalarization (left plot) and DiME (right) obtain similar performance. When the action norm cost is scaled up hy ten times, DiME's performance unaffected because its tradc-off setting is invariant to reward scales-nott note the two clusters of diffcrently-colorg points. The color denotes the tradcoff and the marker symbol denotes whether the action norm cost is scaled by ten times. Note that in the plots, the y-axis is the unscaled value of action
cost.
Opocun
0.08
Onoem = 0.105
Cgurm 0.115
1000
mmw
LS
2500
taok retarm
aotion nomm
Oporm
Choor
0.62
Oporm
1000
DIME
2500
01
400M
400M
400M
actor steps
Figure 7: Per-objective learning curves for humanoid run. During training, DiME (below) is able improve on both objectives simultaneously. In contrast, LS (above) can typically only improve one objective time. This helps DiME find better solutions-in each column, the LS and DiME policies converge similar action norm cost, but the DIME policy performs substantially better on the task.
the trade-off of 0.01 with the original objectives (i.e., gets higher reward with lower cost), with the scaled-up action norm cost, policies do not obtain any task reward all with trade-off of 0.05.
Finally, we delve into why DiME outperforms LS, Our hypothesis that this because DiME computes an improved policy for each objective (before projecting them), and is thus more likely to ensure improvement with respect to all objectives. In contrast, since LS combines rewards directly, it cannot guarantee improvement with respect to all objectives. To test out this theory, plotted how policies trained by LS and DiME perform with respect to each objective over the course of training, for humanoid run. We found that policies trained with DiME indeed exhibit improvement
22

---

Under review as a conference paper at ICLR 2022
cacspole ewingup
cbeetal, run
Bnger turn hard
cartpole swingup
cheetah run
Rager bursi hard
soturre
1000
1000
a0g
fish humanoid
manip
swim
hurnunoid
munipp insert
took veham
1000
1000
n0g task
manip instt Day.
walker
manip insert peg
walkor stand
walk
sask retan
1000
A9g refim
1000
300
0.5 tradeoff
1.0
0.5 teadooff
1.0
0.5 teadooff
1.0
0.5 tradkoff
1.0
0.5 tradeoff
1.0
0.5 tradooof
1.0
DIME (BC)
DIME (BC) multi
LS
DIME (AWBO)
DIME (AWBB) multi
LS
Figure 8: Per-tradeoff performance for nine Control Suite tasks from RL Unplugged. Across all nine tasks, the best solution found by DiME (any variation) obtains either higher or on-par task performance than the best found by linear scalarization (LS). In addition, DiME can train single policy for a range of tradcoff's (DiME multi, dark orange and dark bluc), that performs comparably to lcarning separate policy for each tradcoff (orange and blue), after the number of learning steps.
umaze
tmaaze diverse
medlum play medium diverse
large play
large dlivesse
1.0
0.5-
avg
0
tradeoff
tradeoff
tradeoff
tradeoff
tradeoff
tradeoff
DIME (BC) multi
DIME (AWBC) multi
LS
kischen complete
kitchen partial
kitchen mixed
ang taek return
tradeofff
teadeeoff
tradeoll
Figure Across nine tasks from D4RL, DIME and LS (CRR) perform on-par.
with respect to both objectives simultaneously over the course of training, whereas policies trained with LS typically suffer from worsening performance for one objective while improving for the other.
D.2 OFFLINE RL
In our offline RL experiments, we evaluate on two open-sourced benchmarks for offline RL: RL Unplugged (Gulcehre al. (200)) and D4RL (Fu et al. 2020). We use the nine Control Suite tasks from RL Unplugged, the six Ant Maze tasks from D4RL, and the three Franka Kitchen tasks from D4RL. For each task, we train policies on the dataset transitions provided by the benchmark. These are continuous control tasks, that include particularly challenging ones that of-te-of-th--al art approaches perform poorly on (c.g., Ant Maze large play and large diverse). For a comprehensive description of the environments, please refer to Gulcehre al. 2020) and Fu al. (2020).
23

---

Under review as a conference paper at ICLR 2022
DIME
DIME
L.
Task
BRAC
BEAR
AWR
BCQ
COL
BC
(CRR)
(BC)
(AWBC)
(CRR)
.63
multi
multi
antimaze umaaz
0.7
0.7
0,42
0.97
0.99
0.95
antiaaze umaze diverse
0.7
0.6
0.7
0.6
0.8
0.45
0.87
0.88
0.83
0.86
antmaae medium play
0
0
0
0.6
0.02
0.66
0.87
0.86
0.86
antmaze medium diverse
0.1
0.5
0.01
0.88
0.95
0.90
0.91
antinaze large play
0.1
0.2
0.76
0.87
0.62
0.87
antmaaze large diverse
0.1
0.51
kitchen complete
0
0.3
1.8
2.18
2.70
3.09
2.99
2.63
kitchen partial
0
0.5
0.6
0.8
1.9
1.77
1.89
1.93
2.08
1.67
kitchen mixed
0
1.9
0.4
0.3
2.0
2.32
1.90
2.46
2.58
2.08
mean score
0.16
0.42
0.26
0.32
0.96
0.80
1.24
1.42
1.37
1.30
Table 5: Results for LS and DiME (average cumulative reward) obtained for the best tradeoff per algorithm, tasks from D4RL. The results for the italicized algorithms taken from Pu et al. (2020); these the best-performing algorithms from their comprehensive evaluation. Numbers within 5% the best score are bold.
Additional Results. Here show plots for per-tradeoff performance for all nine Control Suite tasks from RL Unplugged (Fig. 8) and for the Ant Maze and Franka Kitchen tasks from D4RL (Fig. 9). We also provide the full table of results for the D4RL tasks (Table 5 Across all 18 tasks, the best tradeoff found by DiME obtains either higher or on-par performance compared to the best tradeoff found by LS. Recall that LS is equivalent to CRR, tfate-of-the-ar approach for offline RL, in terms of the policy loss it optimizes for.
D.3 FINETUNING
For finetuning, we evaluate on three humanoid tasks (stand, walk, and run) and two manipulator tasks (insert ball and insert peg) from the DeepMind Control Suite. There two main differences between finetuning and offline RL: 1) in finetuning the policy is able to interact with the environment, whereas in offline RL it cannot, and 2) in finetuning the behavioral prior may be in the form of teacher policy that we can query to obtain (a(s),, whereas in offline RL the behavior prior is in the form of dataset of transitions.
For the humanoid tasks, we trained a policy for humanoid stand with standard deep RL (using MPO (Abdolmaleki al. 2018) and stopped it midway through training, when reached an average task reward of about 400. We then used this teacher policy for finetuning in all three humanoid tasks. However, for the manipulator tasks, training policy from scratch highly sample-inefficient, as can be see by the learning curves for learning from scratch (i.e., a 0) in Fig. TO, So for manipulator insert ball and insert peg, we took the best policy trained using DiME in our offline RL. experiments, and used that the teacher policy for finetuning in that task.
Fig. 10 shows learning curves for all five tasks. When tradeoff is learned (in orange), DiME and linear scalarization perform on-par. However, when the tradeoff is fixed, for linear scalarization it is more difficult to pick an appropriate tradeoff, because the appropriate tradeoff depends on the relative scales of the rewards. In fact, for the humanoid tasks, none of the fixed (non-zero) tradeoffs for LS perform better than learning from scratch.
24

---

Under review as a conference paper at ICLR 2022
humanaid stand
humannid walk
humanoid run
manip insert bxll
manip peg
1000
rewsard
500
0
20
20
20
20
20
actor stepps (x10%)
0.0
0.25
0.5
1.0
Jearned
humanold stand
humanoid walk
humanoid run
manip insert 'xlll
manip insert peg
1000
500
20
20
20
20
20
actor steps (x10*)
Figure 10: Learning curves for finetuning, for DiME (top row) and lincar scalarization (LSS, bottom row). a corresponds to fully imitating the behavioral prior, while O corresponds to learning from scratch. The optimal fixed tradcoff depends on the task. For both DiME and LS, learning the tradcoff (orange) converges better performance than fully imitating the behavioral prior (dashed purple), while learning as quickly. The error bars show standard deviation across ten seeds.
25


Reviews:
[{"review_id": "zwtqTroIjYL", "reviewer": "Reviewer_gFyL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies a foundamental problem in RL optimization with an interesting view of multi-objective policy optimization, trying to tackle challenges in existing RL approaches. \nThe main contributions of this work are in two-fold. First, it provides a novel view that uses MORL as a tool to address fundamental challenges in RL, and in the paper, as case studies, the authors studies the use of MORL in finetuning and offline RL. Second, it proposes a novel MORL approach DiME that outperforms tranditional LS on standard MORL tasks.\n\nThe paper did a good derivation and analysis for the proposed DiME approach. Table 1 is nice to show the comparison between prior works. However, it is not theoretically shown that why DiME can outperform LS, though in the paper the authors hypothesize that a separate variational distribution allows DIME to find a tighter variational lower bound. In the experiments, the paper shows that DiME outperforms LS on standard MORL tasks empirically. However, it did not show comparison to MO-MPO --- this is wierd because MO-MPO seems to serve as a better baseline than LS.\n\nIn section 5, the paper did case studies of MORL for finetuning and offline RL, and showed how to formulate the algorithms of finetuning and offline RL as MORL problem. The authors provide a general formulation and shows that small modifications to the general objective can recover to existing algorithms. Extensive experiments show that DiME allows better optimization for finetuning algorithms and offline RL algorithms.\n", "summary_of_the_paper": "This paper proposes using multi-objective optimization as a tool for tackling challenges in RL. The motivation is that the different additional contraints (or objectives) in the policy optimization step are always in conflict, which is natural to view the existing RL as an instance of MORL problem. Typical MORL applies linear scalarization (LS) by taking a weight sum of the objectives. In this paper, first, the authors identifies the disadvantages of prior work (including LS and MO-MPO) and proposes a new objective called DiME. The proposed DiME fits a separate variational distribution per objective and thus can find a tighter variational lower bound. Second, the authors show case studies of DiME for finetuning and offline RL. Experiments show that DiME outperforms LS on standard MORL problems as well as two case studies (finetuning and offline RL).\n", "summary_of_the_review": "Pros:\n1. Provide an interesting perspective that regards RL as MORL problems.\n2. Propose a better MORL algorithm DiME.\n3. Experimental results are interesting and convincing.\n\nCons:\n1. Lack of theoretical results.\n2. Did not compare with all baselines, such as MO-MPO.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "wvgSdzZSSf8", "reviewer": "Reviewer_gxGj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe experimental study in this paper is quite extensive, with results on two important applications (offline RL, finetuning) and various task domains. From Table 1 and Table 2, it is quite convincing that DiME outperforms the baseline method of LS (and other existing non multi-objective based algorithms) on offline RL.  \n\nThe presentation of this paper is quite clear.\n\n**Weaknesses**\n\nMy central concern about this paper is about how DiME compares with the very related existing method of MO-MPO. I have two main questions on this end.\n1. Experimentally, MO-MPO is not tested in the main experimental results (Table 2, Figures 2-3). Given that MO-MPO is presented as one of the two main existing algorithms in Table 1 (alongside LS), and LS is tested quite extensively in the above tables/figures, it is somewhat surprising that MO-MPO is not tested in the above results. Only Section 6.1 (multi-objective RL) mentions MO-MPO with results in appendix D, showing that DiME only performs on par with MO-MPO.  \nSection 5.2 contains a short discussion on why MO-MPO is not suitable for offline RL. However I find the explanations quite vague and not convincing why it is not implementable or necessarily performs badly in offline RL.\n\n2. Algorithmically, I feel like DiME may actually be not different at all from MO-MPO, especially after taking into account an important implementation trick hidden in the appendix.  \nComparing DiME vs. MO-MPO in Table 1 and Section 3.1, DiME involves choosing both the stepsize $\\eta_k$ as well as the weights $\\alpha_k$ for the $K$ objectives, whereas for MO-MPO the step-size $\\eta_k$ is determined from $\\alpha_k$. However, when taking into account the actual implementation details, $\\eta_k$ in DiME are actually optimized too, in a fashion very similar to MO-MPO indeed. \nFor example, the DiME objective (11) for offline RL has $K=2$, and it chooses $\\eta_2=1$ but still leaves $\\eta_1=\\eta$ a hyperparameter. The main paper and the implementation detail section in Appendix C only provides the initial value of $\\eta$, but does not further tell how it is chosen or tuned.  \nGoing through the appendix, the way to choose $\\eta_k$ seems to be hidden in Section B.1, Page 17 (which ought to be a section for the math intuition about variational inference, not for implementation details). From there it seems like $\\eta_k$ needs to be optimized with gradient-based optimization on the dual objective (45)---This makes it exactly the same to Eq (4) of the MO-MPO paper (Abdolmaleki et al. 2020). Thus I am afraid that DiME is in essence almost the same as MO-MPO.\n\nRegarding these two questions, I’m curious whether the authors could either provide some more details on how the actual implementation of DiME differs from MO-MPO, and/or present some experiments on the MO-MPO algorithm in conjunction with DiME and LS.\n", "summary_of_the_paper": "This paper proposes a new algorithm DiME for multi-objective policy optimization. The main target application is in the problems of offline RL and policy finetuning, where some closeness to the behavior policy is often desired, in conjunction with optimizing the value function. Experimentally, the paper shows that DiME outperforms the baseline method of Linear Scalarization (LS) on the above applications, and presents theoretical insights on the comparisons between the proposed DiME and several existing algorithms.", "summary_of_the_review": "Overall, this is a well-executed paper that presents a new algorithm DiME for multi-objective policy optimization. However, currently, I am afraid it is missing comparisons with the very related prior method MO-MPO, and hiding important implementation details with which the present algorithm becomes almost equivalent to that prior method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wQZZk68MdhA", "reviewer": "Reviewer_JB6Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n- The idea can be naturally applied to the offline rl and finetuning setting, since in these two problems, we can view the original objectives as two different objectives when learning the policy.\n- The experiments are conducted thoroughly and demonstrate the effectiveness of the current approach. \n\nWeakness\n\n- The computational costs. Since the new proposed method requires to sample alpha to learn, it may introduce additional computational costs to learn the multiobjective, especially when we need to learn the whole pareto. Would the authors discuss the computational costs than the single linear combination baseline methods?\n\n- I wonder if it is possible for the authors to compare some other multi objective baselines instead of linear scalarization, such as multiple gradient descent or pcgrad? since I think linear scalarization is the most native baseline for multi objective optimization. \n\n- I am always curious how the authors implemented the multiobjective optimization part, ever since your first paper. But I never see an opensource code and it is really hard to get the intuition and computation efficiency of the proposed method just from what you described in the appendix. \n\n", "summary_of_the_paper": "The authors in this paper propose to utilize multi-objective optimization techniques to improve offline reinforcement learning and finetunning. \nBased on the extensive empirical experiments, the proposed method can outperform other baselines which do not view the problem as a multi-objective optimization problem. \n", "summary_of_the_review": "Overall I think it is interesting to introduce multiobjective to improve offline rl and policy finetuning tasks, although the computational costs might increase a lot compare with the native approach. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z1-hlgf-Cc", "reviewer": "Reviewer_kKtt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Between weighing the Q-values directly in the policy update vs. weighing KL distances, as DiME does, the paper repeatedly points to the fact that the former is sensitive to the magnitude of rewards. Is there a more fundamental distinction in the tasks one can perform better on vs. the other, especially for a fixed scalar reward (the abstract seems to make a case for this; but the justification for this is not followed up on the paper)? Possibly in terms of some qualitative or quantitative features one might ascribe to the task? \n2. The fact that the proposed approach can discover the pareto frontier  for parameterized rewards is impressive. But, as I understand, both LS (which may have similarities to existing approaches) & DiME are both algorithms proposed in this paper. Is a more direct comparsion possible to an exisiting, published result? MORL is a quite a well established problem in RL. Is there a reason why considering MO-MPO's (for instance) experimental setup beyond humanoid walk-run (for which results are fairly similar) for MORL are prohibitive here?", "summary_of_the_paper": "Modern day policy (and value) function optimization procedures routinely include terms beyond the typical scalar reward, such as to induce regularization, encourage exploration and perform discriminative feature learning. How must one go about balancing these objectives? The typical scheme is to take a weighted sum; this paper recasts that into a multi-objective optimization question, and proposes an objective that combines policies from various objectives in a weighted KL sense.", "summary_of_the_review": "Futher comparisons on existing benchmarks along with characterization (empirically is more than good enough) of environment where the proposal works would really add value.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers in general did not seem to be strongly impressed by the contribution of the paper. As the authors noted, some reviewers seemed to misinterpret the claims of the paper --- the paper is not to design new MORL algorithms that are significantly better on standard MORL benchmarks but is to apply MORL on offline RL and fine-tuning. On the other hand, the AC suspects that the paper's exposition could be more centered around the applications, e.g., arguing why offline RL can be benefited from better MO training, and why the challenge of offline RL is to balance some given notions of risk and return computationally (instead of, e.g., developing the right notion/formula for quantifying the risk and return.) Moreover, I think the paper would be stronger if the evaluation for offline RL setting can be made stronger, e.g., including more tasks and algorithms on the D4RL dataset. If the paper's claim is that MORL is a great tool for offline RL, perhaps it's useful to demonstrate that MORL can achieve SOTA reliably when used on top of existing offline RL algorithms (which almost always have two parts in the objective). In summary, in the AC's opinion, the paper has a valuable contribution to the community but is somewhat boardline for ICLR in the current form, and the AC encourages the authors to resubmit to a top venue conference after addressing some of the reviewers' comments.
Decision: Reject