Year: 2021

Abstract:
Many advances that have improved the robustness and efficiency of deep reinforcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives or constraints in the policy optimization step. This includes ideas as far ranging as exploration bonuses, entropy regularization, and regularization toward teachers or data priors. Often, the task reward and auxiliary objectives are in conflict, and in this paper we argue that this makes it natural to treat these cases as instances of multi-objective (MO) optimization problems. We demonstrate how this perspective allows us to develop novel and more effective RL algorithms. In particular, we focus on offline RL and finetuning as case studies, and show that existing approaches can be understood as MO algorithms relying on linear scalarization. We hypothesize that replacing linear scalarization with a better algorithm can improve performance. We introduce Distillation of a Mixture of Experts (DiME), a new MORL algorithm that outperforms linear scalarization and can be applied to these non-standard MO problems. We demonstrate that for offline RL, DiME leads to a simple new algorithm that outperforms state-of-the-art. For finetuning, we derive new algorithms that learn to outperform the teacher policy.

text: Many advances that have improved the robustness and efficiency of deep reinforcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives or constraints in the policy optimization step. This includes ideas as far ranging as exploration bonuses, entropy regularization, and regularization toward teachers or data priors. Often, the task reward and auxiliary objectives are in conflict, and in this paper we argue that this makes it natural to treat these cases as instances of multi-objective (MO) optimization problems. We demonstrate how this perspective allows us to develop novel and more effective RL algorithms. In particular, we focus on offline RL and finetuning as case studies, and show that existing approaches can be understood as MO algorithms relying on linear scalarization. We hypothesize that replacing linear scalarization with a better algorithm can improve performance. We introduce Distillation of a Mixture of Experts (DiME), a new MORL algorithm that outperforms linear scalarization and can be applied to these non-standard MO problems. We demonstrate that for offline RL, DiME leads to a simple new algorithm that outperforms state-of-the-art. For finetuning, we derive new algorithms that learn to outperform the teacher policy.

heading: 1 INTRODUCTION
text: Deep reinforcement learning (RL) algorithms have solved a number of challenging problems, including in games (Mnih et al., 2015; Silver et al., 2016), simulated continuous control (Heess et al., 2017; Peng et al., 2018), and robotics (OpenAI et al., 2018). The standard RL setting appeals through its simplicity: an agent acts in the environment and can discover complex solutions simply by maximizing cumulative discounted reward. In practice, however, the situation is often more complicated. For instance, without a carefully crafted reward function or sophisticated exploration strategy, learning may require hundreds of millions of environment interactions, or may not be possible at all.
A number of strategies have been developed to mitigate the shortcomings of the pure RL paradigm. These include strategies that regularize the final solution, for instance by maximizing auxiliary rewards (Jaderberg et al., 2017) or the entropy of the policy (Mnih et al., 2016; Haarnoja et al., 2018). Others facilitate exploration, for instance via demonstrations (Brys et al., 2015), shaping rewards (Ng et al., 1999), exploration bonuses (Bellemare et al., 2016; Pathak et al., 2017), or guidance from a pre-trained teacher policy, known as policy finetuning (Xie et al., 2021). Strategies for reusing old environment interactions can in the extreme case enable agents to learn solely from a fixed dataset, known as offline RL (Fu et al., 2020; Gulcehre et al., 2020). In this case regularization is needed to ensure that the learned policy is well supported by the data distribution.
These approaches may seem quite different at first glance, but they can all be understood as introducing additional objectives, or constraints, in the policy optimization step. Since reward maximization and the additional objective(s) are usually in conflict, these algorithmic approaches can also be understood as solving a multi-objective RL (MORL) problem. In practice, the solution to this MORL problem is usually implemented via a form of linear scalarization (LS), i.e., by taking a weighted sum of the objectives. However, LS suffers from a number of limitations: it is sensitive to the scale of the objectives (Abdolmaleki et al., 2020) and is limited in the solutions it can find (Das & Dennis, 1997).
Our key insight is that an interpretation of algorithmic approaches, such as the above, in terms of MORL provides us with novel tools to solve fundamental challenges in RL. To demonstrate this, we focus on offline RL and finetuning as case studies. We show recent offline RL algorithms (e.g., CRR (Wang et al., 2020), AWAC (Nair et al., 2020)) can be understood as using LS to solve the MO problem. We hypothesize that by replacing LS with a more appropriate approach, we can obtain betterperforming algorithms. We thus propose a new MORL approach, called Distillation of a Mixture of Experts (DiME), that can replace LS in this setting. We derive novel algorithms for finetuning and offline RL, based on DiME. In offline RL, this new algorithm outperforms state-of-the-art (Sec. 6).
Our main contributions are as follows:
• We propose that MORL can be used as a tool for tackling fundamental challenges in RL. We show that existing algorithms (e.g., for offline RL) rely on LS to solve the MO problem. Based on this insight, we explore replacing LS with a new, improved MO algorithm. • By replacing LS, we derive a novel offline RL algorithm from the multi-objective perspective
(Sec. 5.2), that is simple, intuitive and outperforms the state-of-the-art (Sec. 6.2). • In a similar manner, we also derive new finetuning algorithms (Sec. 5.1), and show that
these algorithms can train policies that exceed the teacher’s performance (Sec. 6.3). • To replace LS, we introduce a new MORL approach, DiME. We support DiME by theoretical
derivations (Sec. 4.1) and show it outperforms LS on standard MORL tasks (Sec. 6.1).

heading: 2 RELATED WORK
text: Offline RL and Finetuning. Our case studies in this work will involve two related learning problems: offline (or batch) RL (Levine et al., 2020; Lange et al., 2012) and policy finetuning (Xie et al., 2021). Both leverage existing information, either in the form of a dataset or a teacher policy. In this work, we use proximity to the dataset or policy as an additional objective in a MORL problem.
In offline RL, the goal is to learn a policy from a given, fixed dataset of transitions. While classical off-policy RL algorithms can be used, they perform worse than bespoke algorithms (Fujimoto et al., 2019). This is likely due to the extrapolation errors incurred when learning a Q-function—naïve policy optimization could optimistically exploit such errors. Therefore, many existing approaches attempt to constrain the policy optimization to stay close to the samples present in the available dataset (Peng et al., 2019; Nair et al., 2020; Siegel et al., 2020; Wang et al., 2020). Finetuning seeks to improve upon an expert (or teacher) policy, which can be specified either nonparametrically via a dataset of transitions or as a parametric model. In contrast to offline RL, finetuning happens in an online setting, where the agent is able to interact with the environment. Existing approaches for finetuning minimize the cross-entropy between the teacher and agent policies (Schmitt et al., 2018), based on distillation or mimicking (Rusu et al., 2015; Parisotto et al., 2015).
Multi-Objective RL. MORL algorithms train policies for environments with multiple sources of reward. There are single-policy and multi-policy approaches. The former finds a policy that is optimal for a single preference trade-off across rewards. One common technique is to use the trade-off to combine the rewards into a single scalar reward, and then use standard RL to optimize this (Roijers et al., 2013). Typically linear scalarization is used, but it is sensitive to reward scales and cannot always find optimal solutions (Das & Dennis, 1997). Non-linear scalarizations exist (Van Moffaert et al., 2013; Golovin & Zhang, 2020), but are hard to combine with value-based RL. Instead of scalarization, recent work on MO-MPO (Abdolmaleki et al., 2020) relies on constrained optimization.
Multi-policy approaches seek to find a set of policies that cover the Pareto front. Various techniques exist, for instance repeatedly calling a single-policy approach with strategically-chosen trade-off settings (Roijers et al., 2014; Mossalam et al., 2016; Zuluaga et al., 2016), simultaneously learning a set of policies by using a multi-objective variant of Q-learning (Moffaert & Nowé, 2014; Reymond & Nowé, 2019; Yang et al., 2019), learning a manifold of policies in parameter space (Parisi et al., 2016; 2017), or combining single-policy approaches with an overarching objective (Xu et al., 2020).
In contrast, here we consider objectives that are not only environment rewards, but also learning- or regularization-focused such as staying close to a behavioral prior. State-of-the-art MORL approaches, including MO-MPO, cannot be readily applied. We thus introduce a new algorithm, DiME, which is on-par with state-of-the-art and can handle these non-standard objectives (Sec. 4). We use DiME as a vehicle to demonstrate the benefits of taking a multi-objective perspective toward challenges in RL.

heading: 3 BACKGROUND
text: We consider the RL problem defined by a Markov Decision Process (MDP). An MDP consists of states s ∈ S , actions a ∈ A, an initial state distribution p(s0), transition probabilities p(st+1|st, at), a reward function r(s, a) ∈ R, and a discount γ ∈ [0, 1). A policy πθ(a|s) is a state-conditional distribution over actions, parametrized by θ. Together with the transition probabilities, this gives rise to a state visitation distribution µ(s). The action-value function is the expected return from choosing action a in state s and then following policy π: Qπ(s, a) = Eπ[ ∑∞ t=0 γ
tr(st, at)|s0 = s, a0 = a]. This can also be written as Qπ(s, a) = Es′∼p[r(s, a) + γV π(s′)], where V π(s) = Ea∼π[Qπ(s, a)].
In this paper, both the LS and DiME approaches to multi-objective RL follow a policy iteration scheme, repeatedly alternating between policy evaluation and policy improvement. In the policy evaluation step, at iteration i the current policy πθi is evaluated by training a separate value function Qik for every objective k. This requires bootstrapping
1 according to Q-decomposition (Russell & Zimdars, 2003). Any policy evaluation algorithm can be used to learn these Q-functions. In the policy improvement step, given the previous policy πi (short for πθi when clear from context) and associated Q-functions {Qk}Kk=1, the aim is to improve the policy for a given state visitation distribution µ.2

heading: 3.1 POLICY IMPROVEMENT: SINGLE OBJECTIVE
text: DiME introduces a new way to perform policy improvement with respect to multiple objectives, so we will now describe the policy improvement step, first for the single-objective setting. Policy improvement can itself be decomposed into the following two alternating steps (Ghosh et al., 2020).
Improvement. This step finds a (nonparametric) policy q that improves with respect to the Qfunction, while staying close to the current iterate πi, leading to the Lagrangian objective function
F (q; θi) = Es∼µ a∼q
[ Q(s, a) ] − η Es∼µ [ DKL ( q(·|s)‖πi(·|s) )] , (1)
with the known analytic solution q(a|s) ∝ πi(a|s) exp(Q(s, a)/η) (Peters et al., 2010). Projection. This step projects the improved policy to the space of parametric policies by maximizing the following (which simplifies to policy gradient if one uses a parametric q in the improvement step):
J(θ) = −Es∼µ DKL ( q(·|s) ‖πθ(·|s) ) . (2)

heading: 3.2 POLICY IMPROVEMENT: MULTIPLE OBJECTIVES
text: Suppose the policy must improve with respect to K objectives instead. In multi-objective problems, a solution is Pareto-optimal if there is no other solution with better performance for one objective without worse performance for another. The set of all Pareto-optimal solutions is the Pareto front. To find a specific Pareto-optimal policy, let α = {αk}Kk=1 define a preference trade-off across objectives, such that all αk ≥ 0. We can further enforce ∑ k αk = 1. A larger αk means the k-th objective is more important, and should lead to increased influence of objective k on the policy optimization.
Linear Scalarization. Here the trade-off is used to produce a single scalar reward via a weighted sum of the objectives. This leads to the nonparametric solution for q in Table 1, while the projection step is the same as for a single objective. A drawback of LS is that not only the trade-offs, but also the scales of the Qk, influence the contribution of each objective.
1Not all objectives are necessarily evaluated via bootstrapping, but we denote all as Qk to lighten notation. 2We will drop the superscript i on Qik, always assuming the latest iterate of the policy evaluation algorithm.
MO-MPO. In contrast, Abdolmaleki et al. (2020) optimizes (1) separately for each objective, to obtain K improved policies {qk}Kk=1. The trade-offs {αk} specify the constraint threshold on the KL-divergence between qk and πi. As seen in Table 1, this leads to nonparametric solutions with temperatures ηk, that are each a solution to a convex optimization problem involving αk.
A main advantage of MO-MPO is that it combines objectives in a scale-invariant way3, unlike LS. However, because the trade-offs in MO-MPO define KL-constraints, the improvement operator used for qk must be able to meet this constraint exactly. This means MO-MPO cannot be easily applied to non-standard settings like offline RL, as we will discuss later in Sec. 5.2. Thus we propose DiME, which can be applied to non-standard settings and maintains the benefit of scale-invariance.

heading: 4 DISTILLED MIXTURE OF EXPERTS
text: Like MO-MPO, DiME computes a separate qk per objective. As seen in Table 1, the nonparametric solution is almost identical, with the crucial exception that the temperature ηk no longer depends on αk. Unlike LS and MO-MPO, DiME only incorporates the trade-offs in the projection step objective
JDiME(θ) = −Es∼µ K∑ k=1 αkDKL ( qk(·|s) ‖πθ(·|s) ) , (3)
so any improvement operator can be used to obtain qk. This allows us to apply DiME to offline RL and other non-standard settings. In addition, the trade-offs in DiME are more interpretable than those in MO-MPO, since they are weights on local experts qk, rather than KL-divergence constraints. Appendix B describes DiME in more detail, including how the temperature ηk is computed.
However, we may not a priori know what the preference trade-off across objectives should be. In this case we can extend DiME to learn a conditional policy, thereby distilling the entire Pareto front of optimal policies into a single πθ(·|s, α). We train this policy by augmenting states with normalized trade-offs α sampled from some distribution ν, and then optimizing the following objective:
JαDiME(θ) = −Es∼µα∼ν K∑ k=1 αkDKL ( qk(·|s, α) ‖πθ(·|s, α) ) , (4)
where qk(a|s, α) ∝ πi(a|s, α) exp(Qk(s, a, α)/ηk). Note that in order to evaluate this policy, we must also learn an α-conditioned value function Qk(s, a, α). Appendix A contains more details.
After training an α-conditioned policy, one can use Qk(α) = Es∼µ ∫ π(a|s, α)Qk(s, a, α) da fully offline to pick the best trade-off α with respect to objective k, using any hyperparameter optimization (HPO) algorithm, including a simple search. If the learned Qk are inaccurate, which is often true in offline RL, we can instead deploy the policy to search for the best trade-off. This is cheap; in our case studies α is between 0 and 1. In this paper we aim to learn high-quality α-conditioned policies, which we will show is possible using DiME. Investigating other ways of picking α is out of this scope.

heading: 4.1 MULTI-OBJECTIVE RL AS PROBABILISTIC INFERENCE
text: We will now derive LS and DiME from the RL as inference perspective for MORL, showing that both optimize for the same high-level problem, but differ in their choice of variational distribution. Consider a binary random variable R indicating a policy improvement event: R = 1 indicates the policy has improved, while R = 0 indicates that it has not. Policy optimization then corresponds to maximizing the marginal likelihood Eµ log pθ(R|s) with respect to θ. Employing the expectationmaximization (EM) algorithm and assuming a conditional probability p(R|s, a) ∝ exp(Q(s, a)/η), one can recover many algorithms in the literature (Levine, 2018).
Let us instead define a separate indicator Rk for each objective, with similarly defined conditional probabilities, and consider the following weighted log-likelihood objective:
F (θ) = Eµ log K∏ k=1 pθ(Rk|s)αk = K∑ k=1 αkEµ log pθ(Rk|s) , (5)
3if the improvement step is scale-invariant, which is true for the known nonparametric solution if we optimize the temperature η to meet a constraint on the KL-divergence, as is done in e.g. MPO (Abdolmaleki et al., 2018).
where Eµ is a shorthand for Es∼µ and α ∈ RK+ specify a preference trade-off across the K objectives. This generalization has a few nice properties. First, when all objectives are equally preferred, the αk are all equal and it reduces to a straightforward probabilistic factorization of independent events: p({Rk}Kk=1) = ∏ k p(Rk). Second, any vanishing αk leads to an objective being ignored.
Linear Scalarization. To apply EM, we could introduce a variational distribution q(a|s) and use it to decompose and lower-bound (5). This choice of variational distribution results in LS: the E-step corresponds to (1) with Q(s, a) replaced by ∑ k αkQk(s, a), and the M-step is identical to (2).
DiME. We could instead introduce a separate variational distribution qk for each log pθ(Rk|s), which results in DiME. The E-step now corresponds to K independent copies of the improvement step in (1), each with its own Qk(s, a). The M-step is the projection step introduced in (3).
This reveals that both standard LS and DiME optimize the same information-theoretic objective (see Appendix B.1 for a complete derivation). Both employ EM, but differ in their choice of variational decomposition. We hypothesize that fitting a separate variational distribution per objective allows DiME to find a tighter variational lower bound, which may lead to better optimization.

heading: 5 CASE STUDIES
text: Our primary aim in this work is to showcase MORL as a tool for tackling fundamental challenges in deep RL. We therefore demonstrate its benefits in two active areas of research: finetuning and offline RL. Finetuning is concerned with improving upon an expert (or teacher) policy, whether it is available in the form of a parametric model or a dataset of transitions. Meanwhile in offline RL, one must learn a policy from a given fixed dataset of transitions. In contrast to offline RL, finetuning happens in an online fashion so the agent interacts with the environment to collect additional experience.
These two settings share a fundamental similarity—they both use behavioral priors to make learning more efficient. Existing algorithms in both settings constrain the policy optimization, either explicitly or implicitly, to stay close to the given priors (Levine et al., 2020; Monier et al., 2020; Schmitt et al., 2018). Instead, we propose to explicitly formulate these in the MORL framework with two objectives:
1. one for the task return via the task Q-function Q(s, a) learned via bootstrapping, 2. and one for staying close to the behavioral prior via the log-density4 ratio log πb(a|s)πi(a|s) ,
where πi is the current policy and πb is the behavioral prior. When we have direct access to the density πb, we can evaluate the log-density ratio pointwise. However, in many real-world applications, we only have access to samples from πb via a dataset of historical interactions. In the following subsections, we dive into each of these cases, deriving the corresponding policy losses for both LS and DiME. We use a scalar α ∈ [0, 1] to specify the preference trade-off between the two objectives.

heading: 5.1 FINETUNING
text: Equipped with the MORL perspective, we introduce two novel finetuning algorithms by substituting the objectives above in the LS and DiME rows of Table 1, yielding the policy optimization functions:5
JLS(θ) = E s∼µ a∼πi
[ 1
Z0(s) exp
( (1− α)Q(s, a)
η + α η log πb(a|s) πi(a|s)
) log πθ(a|s) ] and (6)
JDiME(θ) = E s∼µ a∼πi  (1− α) Z1(s) exp ( Q(s, a) η1 ) + α Z2(s) exp ( log πb(a|s)πi(a|s) η2 ) log πθ(a|s)  . (7)
All functions Z∗(s) normalize the product of πi and the corresponding tempered and exponentiated objective; though analytically intractable, they can be estimated from sampled actions for each state. Note that both of the equations above are equivalent at the extreme values of α = 0 and 1.
Relation to existing approaches. We can recover existing algorithms with slight modifications to these policy objectives. In particular, setting η2 = 1 in the DiME row of Table 1 yields q2 = πb and
4The log-density ratio is typically used as reward for distribution matching (Arenz & Neumann, 2020). 5Here, and in our finetuning experiments, we assume the prior is a parametric model. If the prior is instead a
dataset of transitions, then the update rules are identical to the ones derived later for offline RL.
the following policy objective function:
(1− α)E s∼µ a∼πi
[ 1
Z1(s) exp
( Q(s, a)
η
) log πθ(a|s) ] − αEs∼µDKL ( πb‖πθ ) , (8)
which is similar to the finetuning approach in (Schmitt et al., 2018). Similarly, taking instead the LS approach to MORL and employing the policy gradient (which uses a parametric q in Sec. 3.1) as the underlying policy optimization, we get:
E s∼µ a∼πθ
[ (1− α)Q(s, a) + α log πb(a|s)
πθ(a|s)
] = (1− α)E s∼µ
a∼πθ Q(s, a)− αEs∼µDKL
( πθ‖πb ) , (9)
which is similar to (Galashov et al., 2019). Both simply add a weighted KL regularization to a standard RL objective, but they differ in the direction of the KL.

heading: 5.2 OFFLINE RL
text: Recall that in offline RL, we only have access to the behavioral prior πb via a dataset Dπb of sampled transitions (s, a, r, s′); we cannot evaluate its density and thus cannot directly compute the objective function for staying close to the prior. Instead, we must simplify both JLS and JDiME from the finetuning subsection so that they only require samples from πb and µ, for which we can rely on Dπb . Linear Scalarization. By tying the preference and scale parameters so that α = η, we can change the expectation over actions in (6) analytically to be with respect to the behavior policy πb
JLS(θ) = EDπb
[ exp ( Q(s, a)
β − logZ0
) log πθ(a|s) ] , (10)
where β = α/(1− α). There are close similarities between this LS-based policy objective and other policy losses used in recent offline RL algorithms. For example, β corresponds to the temperature in CRR (Wang et al., 2020) and AWAC (Nair et al., 2020), and we can recover them exactly by replacing the task return objective with the advantage function. As an other example, BEAR (Kumar et al., 2019) and BCQ (Fujimoto et al., 2019) can both be formulated as using LS, but with the policy gradient as their underlying policy optimizer as in (9). However, unlike the objective function in (10), which only requires samples from πb, these methods have the limitation that they require modeling πb to evaluate the second term in (9) (Levine et al., 2020; Monier et al., 2020).
DiME. Setting η2 = 1 and renaming η1 = η in (7), we obtain
JDiME(θ) = (1− α)E s∼µ a∼πi
[ 1
Z1(s) exp
( Q(s, a)
η
) log πθ(a|s) ] + αEDπb log πθ(a|s). (11)
Notice that the first term is the loss used by policy optimization methods such as REPS (Peters et al., 2010) and MPO (Abdolmaleki et al., 2018), and the second term is exactly a behavioral cloning loss. We call this new algorithm DiME (BC). This is a simple way of adapting online algorithms for offline RL, while being theoretically grounded in a multi-objective perspective. Intuitively, since both terms are independent of the scale of the Q-values, optimizing the weighted sum is a reasonable thing to do. This is in contrast to recent concurrent work that adds a behavioral cloning loss to a policy gradient loss, which depends on the scale of the Q-values (Fujimoto & Gu, 2021).
Remark on drawbacks of LS approaches. First, the preference α and scale η can no longer be set independently; they are coupled together in the temperature β. This means we cannot efficiently train a trade-off-conditioned policy using the LS-based loss in (10), because the normalization constant Z0 = ∫ (s,a)∼Dπb
exp(Q(s, a)/β) depends on the choice of the trade-off—this would require estimating (10) via samples separately per trade-off. Second, since JLS only involves an expectation with respect to the expert behavior policy, the policy cannot evaluate and exploit the Q-value estimates for actions beyond those taken by the expert. One could argue that this is by design, in order to avoid fitting extrapolation errors in the Q-function. However, with JDiME this is a choice we can make by setting the preference parameter independently (and perhaps adaptively).
Remark on using other MORL approaches. Although there exist many MORL algorithms that outperform LS on standard MORL tasks, they typically require that the rewards for all objectives are given. Thus it is not straightforward to apply these algorithms for offline RL, where we cannot directly compute the reward for staying close to the prior. We tried getting around this in order to use MO-MPO for offline RL, but it was very complicated to implement (for details see Appendix B.4).

heading: 6 EXPERIMENTS
text: We first investigate how DiME performs on standard MORL tasks, compared to LS (Sec. 6.1). Since our focus is on applying MORL for general challenges in RL, we do not consider MORL algorithms that cannot be applied to offline RL. Then we investigate the application of DiME in two case studies, offline RL (Sec. 6.2) and finetuning (Sec. 6.3), where our aim is to evaluate whether replacing LS with DiME leads to better-performing algorithms. Implementation details are in Appendix C.

heading: 6.1 MULTI-OBJECTIVE RL
text: Toy Domain. Our bandit task has action a ∈ R and reward r(a) ∈ R2, with the reward defined by either the Schaffer or Fonseca-Fleming function (Okabe et al., 2004). The Pareto front is the set of all points of the function. By varying the trade-off, DiME finds solutions along the entire Pareto front for both functions. However, no matter what the trade-off is, LS only finds solutions at the extremes of Fonseca-Fleming, because this Pareto front is concave (Fig. 1a). This is a fundamental limitation of LS—it cannot find solutions on the concave portions of a Pareto front (Das & Dennis, 1997).
Humanoid. We also evaluate on two humanoid tasks from the DeepMind Control Suite (Tassa et al., 2018). For each, the original task reward objective is augmented with an “energy”-expenditure penalty on actions: −‖a‖2. In both tasks, the Pareto front found by DiME is superior to that found by LS, both qualitatively (Fig. 1b) and in terms of hypervolume, which is a standard metric for capturing the overall quality of a Pareto front (Table 1c).
Additional Experiments. We also find that DiME is scale-invariant—it obtains similar performance when the energy cost is scaled up by ten times, whereas LS no longer can learn the task. DiME also performs on-par with MO-MPO, which is state-of-the-art. Please refer to Appendix D for details. Recall that DiME can be applied to non-standard settings like offline RL, whereas MO-MPO cannot.

heading: 6.2 OFFLINE RL
text: Baselines. For offline RL, our main baseline uses LS to trade off between the task return and staying close to the behavioral prior, with the advantage function as the task return objective. This optimizes for E(s,a)∼Dπb exp((1−α)/αA(s, a)) log πθ(a|s). This is the same policy loss as CRR-exp (Wang et al., 2020) and AWAC (Nair et al., 2020), two state-of-the-art offline RL algorithms. Our behavioral cloning (BC) baseline optimizes for E(s,a)∼Dπb log πθ(a|s), which is LS or DiME with α = 1. Variations of DiME. For offline RL, our DiME-based algorithm optimizes for (11), where the second term is a behavior cloning loss; we call this DiME (BC). For this second term, we could instead use an advantage-weighted behavior cloning loss, Eπb(a|s)[exp(A(s, a)) log πθ(a|s)], which is exactly the CRR policy loss with a temperature of 1; we call this DiME (AWBC). Since training a separate policy per trade-off is computationally expensive, we also experiment with training a single trade-off-conditioned policy πθ(a|s, α), that can optimize for all trade-offs in the range [0, 1]. We refer to this as DiME multi, and combine it with both BC and AWBC. We choose to train a single policy that can optimize for all trade-offs, rather than learning the best trade-off during training, because it is a well-known problem that policy evaluation is inaccurate in offline RL.
Evaluation. We evaluate on tasks from two offline RL benchmarks, RL Unplugged (Gulcehre et al., 2020) and D4RL (Fu et al., 2020), with results in Table 2. For LS, DiME (BC), and DiME (AWBC), we train one policy per trade-off α, linearly spaced from 0.05 to 1. For DiME (BC) multi and DiME (AWBC) multi, we train ten policies with different random seeds. We train all policies for one million learner steps. After training, we evaluate the policy in the test-time environment once per trade-off.
From RL Unplugged, we evaluate on the nine Control Suite tasks. For both LS and DiME, the setting of α has a significant impact on performance, and the optimal setting is different per task (Fig. 2). Comparing the best performance obtained for each task (across all trade-offs), all DiME variants outperform LS. In addition, training a single tradeoff-conditioned policy (with DiME multi) is not only more efficient, but also leads to better performance. This may be because conditioning on α regularizes learning. With LS though, we cannot efficiently train α-conditioned policies because α controls both the trade-off and scaling. To make a more fair comparison against prior work, we also find the best fixed trade-off across all tasks for LS and DiME (BC/AWBC), and report these results in the middle columns of Table 2. Both DiME variants still outperform existing approaches.
From D4RL, we evaluate on the six Ant Maze tasks and three Franka Kitchen tasks. We only evaluate the DiME multi variants, with one random seed, since they perform best on the Control Suite tasks and have low variance. LS and DiME perform on par, and both outperform state-of-the-art approaches.

heading: 6.3 FINETUNING
text: Finally, we will evaluate whether our new finetuning algorithms, derived from the multi-objective perspective, can improve upon the performance of a teacher policy. We evaluate both the LS-based and DiME-based algorithm with either a fixed or learned tradeoff. To learn the tradeoff, we update α based on the loss J(α) = α (Es∼µ,a∼πiQ(s, a) − c). This is analogous to the update for the Lagrange multiplier in Lagrangian relaxation for constrained optimization. The resulting strategy is intuitive: stay close to the behavioral prior while the expected return is below the threshold c, and otherwise optimize for the bootstrapped Q-function. We pick c based on the expected return from fully imitating the prior for the given task (i.e., α = 1); these values are given in Appendix C.
We evaluate on the humanoid and manipulator tasks from the DeepMind Control Suite. For the three humanoid tasks, we start with a suboptimal humanoid stand policy as the behavioral prior. For the two manipulator tasks, the behavioral prior is a policy trained for that task.
The choice of fixed tradeoff impacts the result significantly, and the optimal choice depends on the task, so either searching over or learning the tradeoff is necessary. Across all five tasks, for both LS and DiME, the learned tradeoff converges to better performance than fully imitating the prior (α = 1) and learns much more quickly than learning from scratch (α = 0). This has the advantage of training only a single policy, as opposed to training a separate policy per fixed tradeoff. Adapting the trade-off helps because in finetuning, the agent is able to gather online interactions. An adaptive scheduling for the trade-off allows the agent to initially copy the teacher and then eventually deviate from the teacher to achieve better performance based on its new experiences. In contrast, in the offline RL setting, we needed to use fixed trade-offs because such online interactions are not allowed.

heading: 7 CONCLUSION AND FUTURE WORK
text: In this work, we propose using multi-objective optimization as a tool for tackling challenges in RL. We explored the benefits of this in offline RL and finetuning, both of which involve staying close to a behavioral prior. Our derivations revealed that existing approaches optimize for a linear scalarization of two objectives, task return and a log-density ratio. We proposed a new MORL algorithm, DiME, as an alternative to LS. Applying DiME to offline RL leads to state-of-the-art performance.
A key takeaway is the impact of the trade-off α on performance, and the need to either search over or optimize for it. As a first solution we explored either training trade-off-conditioned policies or adapting the trade-off during training; it is worth exploring other approaches in future work.
In offline RL, a limitation of optimizing for the trade-off is that it requires being able to accurately evaluate the policy for multiple trade-offs. We chose the simplest option in our experiments: deploy the fully-trained policy for each trade-off in the test-time environment, and pick the best-performing trade-off. But this may not be feasible in real-world applications, for instance if it is expensive or dangerous to deploy a policy. One option is to use recent offline policy evaluation algorithms to accurately estimate a policy’s test-time performance (for all trade-offs) without requiring deployment.

heading: REPRODUCIBILITY STATEMENT
text: In Appendix C, we provide full implementation details for DiME, our novel algorithms for finetuning and offline RL, and the baselines. We report hyperparameter settings and network architectures in Tables 3 and 4, also in Appendix C. Our code is based on the open-sourced implementations of MO-MPO and MPO in Acme (Hoffman et al., 2020), which is an open-source RL framework. We are in the process of open-sourcing the code.

heading: A TRAINING TRADE-OFF-CONDITIONED POLICIES
text: In this section we provide details on training a trade-off-conditioned policy πθ(a|s, α) conditioned on preference parameters α ∼ ν(α). (Note that here α corresponds exactly to α from the main body of this manuscript; we had to distinguish it from the single scalar that appears in our case studies, but we drop the underline here to lighten notation.) Conditioning the policy on the trade-off α requires that the Q-functions are also trade-off-conditioned. We also use hindsight relabeling of trade-offs to perform off-policy learning.
Preference-conditioned policy evaluation. We evaluate the current policy πi(a|s, α) via off-policy learning of per-objective Q-functions Qik(s, a, α). When training the Q-network, every sampled transition (s, a, {rk}Kk=1, s′) is augmented such that the states include a sampled trade-off parameter α ∼ ν, resulting in transitions ([s, α], a, {rk}Kk=1, [s′, α]). From here, any critic learning algorithm can be used; we use distributional Q-learning (Bellemare et al., 2017; Barth-Maron et al., 2018). See Section C for more details.
Preference-conditioned policy improvement. The policy improvement step produces a trade-offconditioned policy πi+1(a|s, α) that improves upon its predecessor πi. Improvement: To find per-objective improved action distributions qk(a|s, α), we optimize the following policy optimization problem for each objective:
max qk Es∼µ α∼ν
[ Ea∼qk(·|s,α)Q i k(s, a, α) ] (12)
s.t. Es∼µ α∼ν DKL(qk(a|s, α)‖πi(a|s, α)) ≤ k. (13)
Similarly, from here we can follow steps analogous to those in Section B.1, but with trade-offconditioned policies and Q-functions. This means that this set of trade-off-conditioned problems have solutions that are analogous to (38) for each objective.
Projection: After obtaining per-objective improved policies qk, we can use supervised learning to distill these distributions into a new parameterized policy:
max θ Es∼µ α∼ν K∑ k=1 αkDKL ( qk(·|s, α) ‖πθ(·|s, α) ) (14)
s.t. Es∼µ α∼ν
[ DKL(πi(·|s, α)‖πθ(·|s, α)) ] ≤ β,
subject to a trust region with bound β > 0 for more stable learning. To solve this optimization, we use Lagrangian relaxation as described in Abdolmaleki et al. (2018; 2020).
In practice. To approximate the expectations over the state and trade-off distributions, we draw states from a dataset (or replay buffer in online RL) and augment each with a sampled trade-off α ∼ ν. To approximate the integrals over actions a, for each (s, α) pair we sample N actions from the current policy πi(a|s, α).

heading: B ALGORITHMIC DETAILS AND DERIVATIONS

heading: B.1 MULTI-OBJECTIVE RL AS PROBABILISTIC INFERENCE
text: In this section we present further details of our derivation, which unifies both linear scalarization and DiME, relying on the RL as inference framework. We begin by recalling some of the motivation behind the objective.
Consider a set of binary random variables Rk indicating a policy improvement event with respect to objective k; Rk = 1 indicates that our policy has improved for objective k, while Rk = 0 indicates that it has not. We seek a policy that improves with respect to all objectives given some preferences over objectives. Concretely, we seek policy parameters θ that maximize the following marginal likelihood, given preference trade-off coefficients {αk}:
max θ
Eµ log pθ({Rk = 1}Kk=1|{αk}Kk=1). (15)
Assuming independence between improvement events Rk we use the following model to enforce trade-offs between objectives, i.e,
F (θ) = Eµ log K∏ k=1 pθ(Rk|s)αk = K∑ k=1 αkEµ log pθ(Rk|s). (16)
This generalization has a few nice properties. First, when all objectives are equally preferred, the αk are all equal and it reduces to a straightforward probabilistic factorization of independent events: p({Rk}Kk=1) = ∏ k p(Rk). Second, any vanishing αk leads to an objective being ignored.
Consider the intractable marginal likelihood L that we wish to maximize, then introduce variational distributions qk and expand as follows
F (θ) = K∑ k=1 αkEµ log pθ(Rk|s) (17)
= K∑ k=1 αkEµEqk log pθ(Rk|s) (18)
= K∑ k=1 αkEµEqk log pθ(Rk, a|s) pθ(a|Rk, s)
(19)
= K∑ k=1 αkEµEqk log qk(a|s) pθ(Rk, a|s) pθ(a|Rk, s) qk(a|s)
(20)
= K∑ k=1 αkEµ [DKL(qk(a|s) ‖ pθ(a|Rk, s))−DKL(qk(a|s) ‖ pθ(Rk, a|s))] (21)
= K∑ k=1 αkEµ [DKL(qk(a|s) ‖ pθ(a|Rk, s))−DKL(qk(a|s) ‖ p(Rk|s, a)πθ(a|s))] (22)
Let us denote the second term as F , defined as follows
L(θ; {qk}) := − K∑ k=1 αkEµDKL(qk(a|s) ‖ p(Rk|s, a)πθ(a|s)) (23)
Then one can rearrange terms in the expansion of the marginal likelihood F to yield
F (θ)− K∑ k=1 αkEµDKL(qk(a|s) ‖ pθ(a|Rk, s)) = L(θ; {qk}). (24)
Since the KL divergence is a non-negative quantity and the αk are all positive, this last equation shows that L lower-bounds the quantity we wish to maximize, namely F . Whether we call it expectation-maximization or improvement-projection, the steps are as follows:
1. tighten the gap between the lower bound L and F at the current iterate θi, by maximizing L with respect to qk while keeping θ = θi fixed, then
2. maximize the lower bound L with respect to θ, holding qk fixed to the solution of step 1.
Formally, these two steps result in the following optimization problems
max {qk} L(θi; {qk}) = min {qk} K∑ k=1 αkEµDKL(qk(a|s) ‖ p(Rk|s, a)πi(a|s)) (25)
max θ L(θ; {qk}) = min θ K∑ k=1 αkEµDKL(qk(a|s) ‖πθ(a|s)), (26)
where in the second step we removed terms that do not depend on θ. These steps are respectively referred to as the E- and M-step in the statistics literature, and improvement and projection in policy optimization. While these two optimization problems seem decoupled from the original goal of maximizing F , the connection with the EM algorithm allows us to benefit from the guarantee that iterates θi obtained in this way lead to improvements in F .
B.2 IMPROVEMENT OR E-STEP
In the next sections we detail the two steps in turn, beginning with the improvement E-step, where we first notice that the objective is a weighted sum of K independent terms that can all be minimized separately
argmin {qk} K∑ k=1 αkEµDKL(qk(a|s) ‖ p(Rk|s, a)πi(a|s)) (27)
⇔ argmin qk EµDKL(qk(a|s) ‖ p(Rk|s, a)πi(a|s)), ∀k = 1, . . . ,K. (28)
These separate problems can be solved analytically by making the KL vanish with the following solution
qk(a|s) = p(Rk|s, a)πi(a|s)∫ πi(a|s)p(Rk|s, a) da
(29)
This nonparametric distribution reweights the actions based on the improvement likelihood p(Rk|s, a). This likelihood is free for us to model as follows and as is standard in the RL as inference literature
p(Rk|s, a) ∝ exp Qk(s, a)
ηk , (30)
where ηk is a objective-dependent temperature parameter that controls how greedy the solution qk(a|s) is with respect to its associated objective Qk(s, a). Substituting this choice of p(Rk|s, a) into the independent problems (28) we obtain the following objective
DKL(qk(a|s) ‖ p(Rk|s, a)πi(a|s)) = Eqk log qk(a|s)
p(Rk|s, a)πi(a|s) (31) = Eqk [ log
qk(a|s) πi(a|s)
− log p(Rk|s, a) ]
(32)
= DKL(qk(a|s) ‖πi(a|s))− Eqk log p(Rk|s, a), (33)
= DKL(qk(a|s) ‖πi(a|s))− Eqk Qk(s, a)
ηk . (34)
After multiplying this objective by −ηk, we obtain the following equivalent maximization problem max qk EµEqkQk(s, a)− ηkEµDKL(qk(a|s) ‖πi(a|s)), (35)
where we recognize a KL regularization term that materializes naturally from this derivation. Indeed this last optimization problem can be seen as a Lagrangian, imposing a soft constraint on the KL between the improved policy qk and the current iterate θi. If we instead impose a hard such constraint, we obtain the following problem:
max qk EµEqkQk(s, a) (36)
s.t. EµDKL(qk(a|s) ‖πi(a|s)) ≤ k. This problem’s Lagrangian form is as follows, and is nearly identical to (35)—albeit with a different dual variable instead of ηk; nevertheless we reuse the notation ηk instead of introducing a new dual variable:
L(qk, ηk) = EµEqkQk(s, a)− ηk [EµDKL(qk(a|s) ‖πi(a|s))− k] . (37)
Putting everything together. Since the original problem is equivalent to the Lagrangian relaxation of this one, we can simply substitute our improvement likelihood (30) into the solution to the original problem (29) to obtain the solution:
qk(a|s) = 1
Zk(s) πi(a|s) exp
Qk(s, a)
ηk , where (38)
Zk(s) = ∫ πi(a|s) exp Qk(s, a)
ηk da. (39)
Adaptively varying ηk. Finally, this solution can in turn be substituted back into (37) to obtain the following convex dual function:
L∗(ηk) = EµEqkQk(s, a) + ηkEµ  k − Eqk log 1Zk(s)πi(a|s) exp Qk(s,a)ηkπi(a|s)  (40)
= EµEqkQk(s, a) + ηkEµ [ k − Eqk log ( 1
Zk(s) exp
Qk(s, a)
ηk
)] (41)
= EµEqkQk(s, a) + ηkEµ [ k + logZk(s)− Eqk Qk(s, a)
ηk
] (42)
= EµEqkQk(s, a) + ηkEµ [ k + logZk(s)]− EµEqkQk(s, a) (43) = ηk [ k + Eµ logZk(s)] (44)
L∗(ηk) = ηk [ k + Eµ logEπi exp Qk(s, a)
ηk
] , (45)
where in the final step we substitute (39) to reveal the hidden dual variable. In practice, this function is optimized alongside of θ in order to adapt the dual variable ηk and mitigate overfitting to Qk.

heading: B.3 PROJECTION OR M-STEP
text: Let us now look at the projection step (M-step in EM terminology), which produces the next iterate θi+1 for our parametric policy. Recall the optimization problem involved in the M-step is the following
min θ L(θ; qk) = min θ K∑ k=1 αkEµDKL(qk(a|s) ‖πθ(a|s)), (46)
= min θ K∑ k=1 αkEµEqk log πθ(a|s), (47)
where qk is fixed to the solution from the improvement step, namely (38). This problem corresponds to finding the policy πθ that acts as a barycentre between the nonparametric policies qk, each weighted by its specified preference trade-off coefficient αk.
Other than this important difference, the multi-objective follows from the single-objective case in a straightforward way. Substituting the solution qk from (38) into the expression for L(θ; qk) yields
JDiME(θ) := L(θ; qk) = Eµ [ K∑ k=1 αkEqk log πθ(a|s) ]
(48)
= Eµ [ K∑ k=1 αk ∫ 1 Zk(s) πi(a|s) exp Qk(s, a) ηk log πθ(a|s) da ] (49)
= Eµ [ K∑ k=1 αk Zk(s) Eπi exp Qk(s, a) ηk log πθ(a|s) ] (50)
= EµEπi [ K∑ k=1 αk Zk(s) exp Qk(s, a) ηk ] log πθ(a|s). (51)
This is the objective function that is optimized to obtain the next iterate in practice. At this stage we can further impose a trust-region or soft KL constraint on this optimization problem.
In practice. We compute this policy objective via Monte Carlo, sampling states s from a replay buffer, then sampling actions for each state from our previous iterate πi. We use these samples to compute weights exp(Qk(s, a)/ηk) and normalize them across states, which corresponds to using a sample-based approximation to Zk(s) as well. Finally, we take the convex combination of these weights, according to the preference trade-offs, which yields the ultimate weights—seen in brackets in (51).

heading: B.4 ON HOW DIME IS BETTER SUITED THAN MO-MPO FOR OFFLINE RL.
text: In this section we expand on precisely why MO-MPO in its current published form is not readily suitable for offline RL and how our proposed method improves on it in this aspect. Let us begin by revisiting the MO-MPO objective function (Eq. 1; Abdolmaleki et al., 2020) and substituting in the offline RL objectives discussed in the paper.
JMO−MPO(θ) = −Eµ [ K∑ k=1 DKL[πie Qk/ηk‖πθ] ] (52)
= Eµ [ K∑ k=1 EπieQk/ηk log πθ ] + C (53)
= Eµ [ EπieQ/η1 log πθ + Eπi ( exp
log πbπi η2
) log πθ ] + C. (54)
where we deliberatly omit the normalization constants to lighten the notation. Recall that MO-MPO enforces the preference trade-off across objectives by setting the KL-constraint thresholds k between the improved policies and the current policy iterate as in (45). Therefore the preference trade-offs are implicit in the varying temperatures ηk, which are continuously adapted throughout training to ensure the satisfaction of the corresponding constraint k. As a result, when applying the same simplifying trick as used for DiME—setting η2 = 1 and renaming η1 = η—we forfeit control of the trade-off completely. Concretely, the second term above reduces to an expectation with respect to the behavior dataset Dπb as follows:
= EµEπieQ/η log πθ + EDπb log πθ + C, (55)
where we recover the now common form of an RL objective combined with an additional BC regularization term. Notice that not only is there no trade-off α but the temperature η2, which implicitly controls it, has also disappeared. Let us now compare this expression to that of DiME:
JDiME ∝ EµEπieQ/η log πθ + α
1− αEDπb log πθ, (56)
where we highlight the key difference between the two losses, namely that the preference parameter α allows us to explicitly control the relative importance of the task’s Q-value objective and the objective that keeps the policy close to the one that generated the offline dataset. In contrast, MO-MPO can no longer satisfy the preference setting specified by the choice of k, because of the simplifying assumption that η2 = 1.
An alternative approach would be to use the dataset Dπb to learn a distilled behavior policy π̂b to model πb, effectively performing behavioral cloning as a preliminary subroutine. This would allow us to compute the log-density ratio directly, to use it as an objective in the original expression (54). This strategy introduces approximation errors in the form of assumptions made on the form of πb and generalization to states that are not present in Dπb . While we tried both this and the MO-MPO objective in (55) in preliminary experiments, neither proved to be competitive with the LS / CRR baseline and were thus omitted from further investigations.
C IMPLEMENTATION DETAILS
This section describes how we implemented our algorithm, DiME, and the linear scalarization (LS) and behavioral cloning (BC) baselines. To ensure a fair comparison, we use the same hyperparameters and network architectures for DiME and the baselines, wherever possible. All algorithms are implemented in Acme (Hoffman et al., 2020), an open-source framework for distributed RL.
Training Setup. For the multi-objective RL and finetuning settings, we use an asynchronous actor-learner setup, with multiple actors. In this setup, actors fetch policy parameters from the learner and act in the environment, storing those transitions in the replay buffer. The learner samples batches of transitions from the replay buffer and uses these to update the policy and Q-function networks. For the offline RL setting, the dataset of transitions is given and fixed (i.e., there are no actors) and the learner samples batches of transitions from that dataset.
To stabilize learning, we use the common technique of maintaining a target network for each trained network. These target networks are used for computing gradients. Every fixed number of steps, the target network’s weights are updated to match the online network’s weights. For optimization, we use Adam (Kingma & Ba, 2015). For the offline RL experiments, we use Adam with weight decay to stabilize learning.
Gathering Data. The actors gather data and store it in the replay buffer. When the policy is conditioned on trade-offs, the trade-off is fixed for each episode. In other words, at the start of each episode, the actor samples a trade-off α′ from the distribution ν, and acts based on π(a|s, α′) for the remainder of the episode. At the start of the next episode, the actor samples a different trade-off, and repeats.
Hyperparameters and Architecture. The policy and Q-function networks are feed-forward networks, with an ELU activation after each hidden layer. For both, after the first hidden layer, we add layer normalization followed by a hyperbolic tangent (tanh); we found that this improves stability of learning. The policy outputs a Gaussian distribution with a diagonal covariance matrix.
The default hyperparameters we use in our experiments are reported in Table 3; setting-specific hyperparameters are reported in Table 4.
Distributional Q-learning. For our policy evaluation we use a distributional Q-network, C51 (Bellemare et al., 2017). It has proven to be a very effective critic in actor-critic methods for continuous control (Barth-Maron et al., 2018; Hoffman et al., 2020). We use the exact same training procedure as published and in open-source implementations (Barth-Maron et al., 2018; Hoffman et al., 2020), i.e., n-step return bootstrap targets and a projected cross-entropy loss. See Table 3 under Q-learning for our chosen hyperparameters.
Evaluation. We evaluate a deterministic policy, by using the mean of the output Gaussian distribution rather than sampling from it. When we evaluate a trade-off-conditioned policy, we condition it on trade-offs linearly spaced from 0.05 to 1.0. For each trade-off α′, we perform several rollouts where we execute the mean action from π(a|s, α′). Recall that this trade-off specifies a convex combination for two objectives.
For the multi-objective RL experiments, all policies are evaluated after 500 million actor steps. For offline RL, all policies are evaluated after 1 million learner steps.
Learned Trade-off for Finetuning. In our finetuning experiments, the learned trade-off is passed through the sigmoid function (i.e., 1/(1 + exp(−x))) to ensure it is bounded between 0 and 1. Compute Resources. To train a single policy for the multi-objective RL experiments (humanoid walk and run), we used an NVIDIA v100 GPU for about 24 hours. To train a single policy for finetuning, we used a NVIDIA V100 GPU for 9-12 hours. To train a single policy for offline RL, we used a v2-32 TPU for about 6 hours for the RL Unplugged tasks, and for about 3 hours for the D4RL tasks. These computational costs were the same for all algorithms we ran: LS and the DiME variants.

heading: D EXPERIMENTS: DETAILS AND ADDITIONAL RESULTS
text: This section provides details regarding our experiment domains, as well as additional experiments and plots.

heading: D.1 MULTI-OBJECTIVE RL

heading: D.1.1 TOY DOMAIN
text: Our toy domain is a bandit with a continuous action a ∈ R. The reward r(a) ∈ R2 is specified by either the Schaffer function or the Fonseca-Fleming function, f : R 7→ R2. Each point along this function is optimal for a particular tradeoff between the two reward objectives, so the function can be seen as a Pareto front.
The Schaffer function corresponds to a convex Pareto front, and is defined as
f1(a) = a 2 and f2(a) = (a− 2)2 .

heading: Finetuning

heading: Multi-Objective RL
text: The Fonseca-Fleming function corresponds to a concave Pareto front, and is defined as
f1(a) = 1− exp(−(a− 1)2) and f2(a) = 1− exp(−(a+ 1)2) .
These are standard test functions in multi-objective optimization, where the aim is to minimize them (Okabe et al., 2004). Since in RL we want to maximize reward, we define the reward to be the negative of these functions.

heading: D.1.2 HUMANOID
text: We also evaluate on the humanoid walk and run tasks from the open-sourced DeepMind Control Suite (Tassa et al., 2018), with the two objectives proposed in Abdolmaleki et al. (2020). The first objective is the original task reward, which is a shaped reward that is given for moving at a target speed (1 meters per second for walking and 10 m/s for running), in any direction. The second objective is the negative `2-norm of the actions, −‖a‖2, which can be seen as encouraging the agent to be more energy-efficient.
The observations are 67-dimensional, consisting of joint angles, joint velocities, center-of-mass velocity, head height, torso orientation, and hand and feet positions. The actions are 21-dimensional and in [−1, 1]; these correspond to setting joint accelerations. Each episode has 1000 timesteps. At the start of every episode, the configuration of the humanoid is randomly initialized.
Additional Results. When we plot the per-episode performance (rather than an average across episodes), we see that linear scalarization only finds solutions at the two extremes for the humanoid walk task, i.e. policies that achieve high task reward but with high action norm cost, or incur zero cost but fail to obtain any reward (Fig. 4a). We hypothesize that this may be because the humanoid walk task has a concave Pareto front of solutions; recall that linear scalarization is fundamentally unable to find solutions on a concave Pareto front (Das & Dennis, 1997). In contrast, DiME is able to find solutions that trade off between the two objectives.
On both tasks, DiME finds a similar Pareto front as MO-MPO, which is a state-of-the-art approach for multi-objective RL (Fig. 4b).
DiME requires training a separate policy per trade-off, which can be computationally expensive. In contrast, our trade-off-conditioned version of DiME, which we call DiME multi, trains a single policy for a range of trade-offs. This obtains a similar Pareto front compared to that found by DiME (Fig. 5), after the same number of actor steps (500 million).
We also ran an experiment to test the scale-invariance of DiME. We modified the humanoid run task such that the action norm cost is now multiplied by ten. We used DiME and linear scalarization to train policies for three random seeds, for each of two trade-off settings. For both the original task and the task with scaled-up action norm costs, DiME trains policies with similar performance (Fig. 6, right). This is the case for both trade-off settings—note the two clusters of differently-colored points in the plot. In contrast, linear scalarization is not scale-invariant: task performance suffers when the cost is scaled up (Fig. 6, left). With a trade-off of 0.05, although this obtains better performance than
the trade-off of 0.01 with the original objectives (i.e., gets higher reward with lower cost), with the scaled-up action norm cost, policies do not obtain any task reward at all with a trade-off of 0.05.
Finally, we delve into why DiME outperforms LS. Our hypothesis is that this is because DiME computes an improved policy for each objective (before projecting them), and is thus more likely to ensure improvement with respect to all objectives. In contrast, since LS combines rewards directly, it cannot guarantee improvement with respect to all objectives. To test out this theory, we plotted how policies trained by LS and DiME perform with respect to each objective over the course of training, for humanoid run. We found that policies trained with DiME indeed exhibit improvement
with respect to both objectives simultaneously over the course of training, whereas policies trained with LS typically suffer from worsening performance for one objective while improving for the other.

heading: D.2 OFFLINE RL
text: In our offline RL experiments, we evaluate on two open-sourced benchmarks for offline RL: RL Unplugged (Gulcehre et al., 2020) and D4RL (Fu et al., 2020). We use the nine Control Suite tasks from RL Unplugged, the six Ant Maze tasks from D4RL, and the three Franka Kitchen tasks from D4RL. For each task, we train policies on the dataset of transitions provided by the benchmark. These are continuous control tasks, that include particularly challenging ones that state-of-the-art approaches perform poorly on (e.g., Ant Maze large play and large diverse). For a comprehensive description of the environments, please refer to Gulcehre et al. (2020) and Fu et al. (2020).
Additional Results. Here we show plots for per-tradeoff performance for all nine Control Suite tasks from RL Unplugged (Fig. 8) and for the Ant Maze and Franka Kitchen tasks from D4RL (Fig. 9). We also provide the full table of results for the D4RL tasks (Table 5). Across all 18 tasks, the best tradeoff found by DiME obtains either higher or on-par performance compared to the best tradeoff found by LS. Recall that LS is equivalent to CRR, a state-of-the-art approach for offline RL, in terms of the policy loss it optimizes for.

heading: D.3 FINETUNING
text: For finetuning, we evaluate on three humanoid tasks (stand, walk, and run) and two manipulator tasks (insert ball and insert peg) from the DeepMind Control Suite. There are two main differences between finetuning and offline RL: 1) in finetuning the policy is able to interact with the environment, whereas in offline RL it cannot, and 2) in finetuning the behavioral prior may be in the form of a teacher policy that we can query to obtain πb(a|s), whereas in offline RL the behavior prior is in the form of a dataset of transitions.
For the humanoid tasks, we trained a policy for humanoid stand with standard deep RL (using MPO (Abdolmaleki et al., 2018)) and stopped it midway through training, when it reached an average task reward of about 400. We then used this teacher policy for finetuning in all three humanoid tasks. However, for the manipulator tasks, training a policy from scratch is highly sample-inefficient, as can be see by the learning curves for learning from scratch (i.e., α = 0) in Fig. 10. So for manipulator insert ball and insert peg, we took the best policy trained using DiME in our offline RL experiments, and used that as the teacher policy for finetuning in that task.
Fig. 10 shows learning curves for all five tasks. When the tradeoff is learned (in orange), DiME and linear scalarization perform on-par. However, when the tradeoff is fixed, for linear scalarization it is more difficult to pick an appropriate tradeoff, because the appropriate tradeoff depends on the relative scales of the rewards. In fact, for the humanoid tasks, none of the fixed (non-zero) tradeoffs for LS perform better than learning from scratch.

Review summaries:
Modern day policy (and value) function optimization procedures routinely include terms beyond the typical scalar reward, such as to induce regularization, encourage exploration and perform discriminative feature learning. How must one go about balancing these objectives? The typical scheme is to take a weighted sum; this paper recasts that into a multi-objective optimization question, and proposes an objective that combines policies from various objectives in a weighted KL sense.

This paper proposes using multi-objective optimization as a tool for tackling challenges in RL. The motivation is that the different additional contraints (or objectives) in the policy optimization step are always in conflict, which is natural to view the existing RL as an instance of MORL problem. Typical MORL applies linear scalarization (LS) by taking a weight sum of the objectives. In this paper, first, the authors identifies the disadvantages of prior work (including LS and MO-MPO) and proposes a new objective called DiME. The proposed DiME fits a separate variational distribution per objective and thus can find a tighter variational lower bound. Second, the authors show case studies of DiME for finetuning and offline RL. Experiments show that DiME outperforms LS on standard MORL problems as well as two case studies (finetuning and offline RL). 

This paper proposes a new algorithm DiME for multi-objective policy optimization. The main target application is in the problems of offline RL and policy finetuning, where some closeness to the behavior policy is often desired, in conjunction with optimizing the value function. Experimentally, the paper shows that DiME outperforms the baseline method of Linear Scalarization (LS) on the above applications, and presents theoretical insights on the comparisons between the proposed DiME and several existing algorithms.