Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learning
Authors: Finn Rietz
Erik Schaffernicht
Stefan Heinrich
Johannes A. Stork
Venue: ICLR 2024 poster (2024)
Keywords: Multi-Objective Reinforcement Learning
Lexicographic Task Priorities
Constrained RL
Transfer RL
Primary area: reinforcement learning
TL;DR: We propose a decomposed algorithm to solve lexicographic multi-objective reinforcement learning problems with continuous action spaces that satisfies constraints by construction.

Abstract: Reinforcement learning (RL) for complex tasks remains a challenge, primarily due to the difficulties of engineering scalar reward functions and the inherent inefficiency of training models from scratch. Instead, it would be better to specify complex tasks in terms of elementary subtasks and to reuse subtask solutions whenever possible. In this work, we address continuous space lexicographic multi-objective RL problems, consisting of prioritized subtasks, which are notoriously difficult to solve. We show that these can be scalarized with a subtask transformation and then solved incrementally using value decomposition. Exploiting this insight, we propose prioritized soft Q-decomposition (PSQD), a novel algorithm for learning and adapting subtask solutions under lexicographic priorities in continuous state-action spaces. PSQD offers the ability to reuse previously learned subtask solutions in a zero-shot composition, followed by an adaptation step. Its ability to use retained subtask training data for offline learning eliminates the need for new environment interaction during adaptation. We demonstrate the efficacy of our approach by presenting successful learning, reuse, and adaptation results for both low- and high-dimensional simulated robot control tasks, as well as offline learning results. In contrast to baseline approaches, PSQD does not trade off between conflicting subtasks or priority constraints and satisfies subtask priorities during learning. PSQD provides an intuitive framework for tackling complex RL problems, offering insights into the inner workings of the subtask composition.

Published as a conference paper at ICLR 2024
PRIORITIZED SOFT Q-DECOMPOSITION FOR LEXICOGRAPHIC REINFORCEMENT LEARNING
Finn Rietz*
Erik Schaffernicht Örebro University Sweden
Stefan Heinrich IT University of Copenhagen Denmark
Johannes A. Stork
Örebro University Sweden
Örebro University Sweden
ABSTRACT
Reinforcement learning (RL) for complex tasks remains challenge, primarily due to the difficulties of engineering scalar reward functions and the inherent ineffi ciency of training models from scratch. Instead, it would be better to specifyy com- plex tasks terms of elementary subtasks and to reuse subtask solutions whenever possible. In this work, we address continuous space lexicographic multi-objective RL problems, consisting of prioritized subtasks, which are notoriously difficult to solve. We show that these be scalarized with subtask transformation and then solved incrementally using value decomposition. Exploiting this insight, we propose prioritized soft Q-decomposition (PSQD), novel algorithm for learn- ing and adapting subtask solutions under lexicographic priorities in continuous state-action spaces. PSQD offers the ability to reuse previously learned subtask solutions in a zero-shot composition, followed by an adaptation step. Its ability use retained subtask training data for offline learning eliminates the need for new environment interaction during adaptation. We demonstrate the efficacy of our approach by presenting successful learning, reuse, and adaptation results for both low- and nigh-dimensional simulated robot control tasks, as well as offline learning results. In contrast to baseline approaches, PSQD does not trade off be- tween conflicting subtasks or priority constraints and satisfies subtask priorities during learning. PSQD provides an intuitive framework for tackling complex RL. problems, offering insights into the inner workings of the subtask composition.
INTRODUCTION
Reinforeement learning (RL) for complex tasks is challenging because as practitioners we must design scalar- valued reward function that induces the desired behavior (Ha et al.) 2021: Tessler al.) 2019: Irpan 2018) and we must expensively learn each task from seratch. Instead, it might be more suitable to model complex tasks as several simpler subtasks (Gábor et al. 1998: Roijers al. 2013) Nguyen et al., 2020. Haaes et al., 2022), and more efficient to reuse or adapt subtask
solutions that were learned previously (Arnekvist et al. 2019 Hausman et al. 2018 Finn et al. 2017). To prevent potentially conflicting subtasks from deteriorating the overal solution, we can impose lexicographic priorities the subtasks (Skalse et al.) 2022, Gábor et al. 1998 Zhang al.; 2022]. This means ordering all subtasks by priority and forbidding lower-priority subtasks from worsening the performance of higher-priority subtasks, which is similar null-space projection approaches for complex multi-task control problems (Dietrich et al., 2015). In RL, this setting is referred to as a lexicographic multi-objective reinforcement learning (MORL) problem.
However, state-of- the-offfthh----- art RL algorithms for continuous state and action spaces like Soft Q-Learning (SQL) (Haarnoja et 2017). Soft Actor-Critic (SAC) (Haarnoja al. (0188), Proximal Policy Optimization (PPO) (Schulman et al., (017) do not support lexicographic MORL problems out of the box and obvious extensions still trade off conflicting subtasks, leading suboptimal solutions, as we demonstrate in our experiments. Current lexicographic MORL algorithms like (Li & Czar-) necki 20199] Zhang et al., 2022) enumerate actions and are therefore limited to discrete problems or rely on constrained optimization methods that trade-off between optimizing subtask and prior- ity objectives (Skalse et al. 2022). All these lexicographic algorithms learn a monolithic policy,
"Corresponding author: finn. rietz0oru.

---

Published as a conference paper at ICLR 2024
precluding any interpretation reuse of previously learned subtask solutions. Furthermore, it is un- known what overall reward function these approaches actually optimize, which hampers principled understanding of lexicographic MORL algorithms.
In this paper, we are the first to show that such complex, lexicographic MORL problems can be scalarized (Roijers al., 2013) and propose prioritized soft Q-decomposition (PSQD), learn- ing algorithm that solves lexicographic MORL tasks in a decomposed fashion (Russell & Zimdars) 2003). For this, we proposed subtask transformation that recovers the Q-function of the lexico- graphic MORL problem as the sum of Q-functions of the transformed subtasks and so provides the overall scalar reward function. PSQD computes this sum incrementally by learning and transforming subtask Q-functions, beginning at the highest- highest-priority subtask and step-by-step learning and adding Q-functions for lower-priority subtasks. In every step, an arbiter ensures that the lower-priority sub- tasks do not worsen already solved higher-priority subtasks and restricts the wwwer-priority subtask to solutions within the so-called action indifference space of the vides safe learning and exploration framework, which is crucial for many real-world scenarios, and allows reuse and adaptation of separately learned subtask solutions to more complex, lexicographic MORL task, which saves data and computation costs. The decomposable Q-function benefits inter- pretability of the learned behavior by allowing analysis and inspection of subtask solutions and their action indifference spaces.
subtasks. PSQD pro-
The contributions of this paper are: (1) We show that lexicographic MORL problems can be scalar- ized using our proposed subtask transformation. (2) We show how reuse and adapt separately learned subtask solutions to more complex, lexicographic MORL problems, starting from a priority- respecting zero-shot composition solution. (3) We provide PSQD, an algorithm for lexicographic MORL problems with continuous state-action spaces, that reuses both subtask solutions and retained offline data.
2 PROBLEM DEFINITION AND PRELIMINARIES
We formalize the complex lexicographic MORL problem (kalle et al. 2022; Gábor et al., 1998) as a Markov decision process (MDP) M = (S, r, Ps. 7, >) consisting of state space S, action space A, reward function r: S x A + R", and discount factor 7 E [0. 1]. The subtasks are numbered i = 1. n and ranked with the transitive priority relation > 2 ...n with task having the highest and task n having the lowest priority. Each dimension reward corresponds to one scalar subtask reward function (r(s. a)]; = r: a). The discrete-time dynamics are given by a conditional distribution P. (S S,, a,) with S,, S E S and a, E A. We use (sr, a.) ~ Px to denote the state-action marginal induced by policy T(a, [s:).
the velter-valued
Lexicographic subtask priorities can be defined by constraining the policy in each subtask to stay close optimal (i.e., e-optimal) for the next higher-priority task i (Skalse et al., 2022 Zhang let al., (2022). Formally, such an e-lexicographic MORL problem implies a set of allowed policies II; for each subtask i,
II, { E I1-1
max J=-((=) -z- Ji-I(T)= S $8-1)
x'Ell:
where J are performance criteria like value-functions, and 0 < E; E R are thresholds for each subtask. However, it is not practical to compute the sets II, explicitly when solving e-lexicographic MORL problems.
Current approaches solve e-lexicographic MORL problems with constrained RL or incremental ac- tion selection in discrete domains (Skalse al. 2022 Zhang et al. 2022) and it is unclear what equivalent scalar reward function these methods actually optimize (Skalse et al. 2022 Gábor et al.) 1998 Vamplew al. 2022). In contrast, we transform the individual subtasks and their Q-functions to implement the priority constraints in equation I This leads to an equivalent scalar reward func- tion for the complex e-lexicographic MORL problem. Our definition of the performance criteria J, in Sec. 3.1 is key to our incremental and decomposed learning approach with this reward function.
2.1
O-DECOMPOSITION
Our goal is to reuse and adapt subtask solutions that were learned previously to new and more complex e-lexicographic MORL problems. For this, we require decomposed learning algorithm
2

---

Published as a conference paper at ICLR 2024
that models individual subtask solutions and combines them to the solution of the complex E- lexicographic MORL problem.
Q-decomposition (Russell & Zimdars complex task's reward function is the sum of subtask rewards, Ty a,) ) = Σm= [r(s;, at]le. In- stead of learning the sum Q-function, Qy, the algorithm learns Q-function Q; for each sub- task and submits them to a central arbiter. The arbiter combines the Q-functions, Qx(s, a) = Di Qi a), and selects the maximizing action for its policy xy. Q-decomposition uses on-policy SARSA TAASA-styy -sylle (Sutton & Barto 2018: Rummery & Niranjan 1994) updates with the arbiter policy Ty to avoid illusion of control Russell& & Zimdars 2003) or the aftractor phenomenon Laroche al.
2003 is such algorithm for MORL problems where the
2017 van Seijen et al. (2017) in subtask learning.
Summing and scaling subtask rewards alone does not result in lexicographic priorities because low-priority subtasks with sufficiently large reward scales can still dictate over high-priority sub- tasks. Therefore, we cannot directly use the Q-decomposition algorithm for lexicographic MORL. Our algorithm builds on Q-decomposition but transform subtasks to ensure that reward- and Q-functions can be summed up whils! implementing lexicographic priorities. To emphasize this difference, we denote transformed subtask Q-functions by Q- which estimate the long-term value of transformed subtask rewards Тーーs such that the Q-function of the lexicographic MORL prob- lem corrssoonss to Q1>2>... (s. a) "i=1 (s, a). or Q~ for short. Qi and r, refer to the untransformed subtask Q-functions and rewards in the lexicographic MORL problem.
2.2
MAXIMUM ENTROPY REINFORCEMENT LEARNING
We aim solve e-lexicographic MORL problems with continuous states and actions, but instead of explicitly computing equation 1 we focus on identifying as many near optimal actions as possible, such that nower-prioriy subtasks have more alternative actions choose from. For this reason, we use maximum entropy reinforcement learning (MaxEnt RL) (Ziebart et al., 2008: Haarnoja et al.) 2018b 2017) for learning subtask policies n; and subtask Q-functions Qi. MaxEnt RL maximizes the reward signal augmented by the Shannon entropy of the policy's 's action distribution, H((a, s:) = Ex, (a, log (Ta ss)],, i.c.
MaxIin arg max
> B (s)pp ( a.) + aHL(a |s)))]
(2)
where the scaler can be used to determine the relative importance of the entropy term. This improves exploration in multimodal problems but also prevents overly deterministic policies, which key for identifying many near-optimal actions. Following Haarnoja et al. (2017). we drop because it can be replaced with reward scaling. Importantly, the entropy term in the objective leads to the following energy- based Boltzmann distribution the optimal MaxEnt policy,
Max (a. s:) exp (Qion (St, ax) (s:)))
where Qian (St, ar) acts as the negative energy and Vinn (s:) is the log- oog-partiiion function. The saft Q-function Qioh and soft value function V in equation Jare given by
(r(S4 act1) + H [same))
Qinn ax) = r(se, ax) + +E (ss4t1... >upa
(4)
Vion (St) log f exp(Q (st, a') da',
(5)
where the entropy in equation is based on the optimal MaxEnt policy Mantin For sampling actions, we can disregard the nrg-tttitionn function in equation 3 and directly exploit the proportion- ality relationship.
(st, MaxFmi (a, ss)) exp (Q (st, a.)).
between the policy and the soft Q-function, e.g., with Monte Carlo methods (Metropolis et al. 1953 Hastings 1970; Duane et al. 1987). We drop the MaxEn: and soft subscripts for the remainder of the paper to avoid visual clutter. The soft Q-learning (Haarnoja et al. (017) algorithm implements a soft Bellman contraction operator and in practice learns model for sampling the complex Boltzmann distribution equation 6 Our algorithm, PSQD, builds on soft Q-learning and combines it with e-lexicographic priorities as well as Q-decomposition from above.
3

---

Published as a conference paper at ICLR 2024
3 PRIORITIZED SOFT Q-DECOMPOSITION
In this section, explain approach for solving e-lexicographic MORL problems. In Sec. 3.1 we explain our subtask transformation for modeling e-lexicographic priorities and show how it scalarizes s-lexicographic MORL problems. In Sec. 3.2] we derive our decomposed learning algo- rithm, PSQD, for e-lexicographic MORL problems and describe a practical version of this algorithm for continuous spaces in Sec. 3.3
3.1 SUBTASK TRANSFORMATION FOR LEXICOGRAPIIC PRIORITIES IN MAXENT RL
Our goal is to learn MaxEnt (arbiter) policy (a s) exp(Q= (s. a)) for the e-lexicographic
MORL problem that fulfills the constraint in equation| [ However, instead of explicitly represent- ing the sets HI, we focus the action space and define state-based version of the e-optimality constraint on the policy Ty using the on-arbiter subtask Q-functions Qi
max Qi(s, a Qi(s, a) S Ei, Va Th, Vs E S, (L...., n- a'EAA
This restricts Ty in each state to actions that are close to the optimal for each subtask i < n. As discussed in Sec. [21]] summing up subtask Q-functions and using the proportionality relationship in equation
6 to directly define an arbiter policy Ty will in general not satisfy the s-lexicographic priority constraints in equation 7 Therefore, we propose a transformation of the subtasks and their Q-functions which provides that the constraints are satisfied when the transformed subtask Q-functions are summed up to Q-. Our transformation relies on constraint indicator function c; S x A + {0, 1f that shows whether action in state s is permitted by the priority constraint for subtasks i, (, a)
[1. 0.
e. otherwise
Qi(s, S
(8)
We refer to the set of permitted actions by subtask i in state 8, As:(s) = {a E A (s, = 1}. as
the action indifference space of subtask i. The intuition is that subtask i is indifferent to which e- optimal action in (s)(s) is selected because they are all close enough to optimal. Analogously, the intersection A (s) = n (s)(s) is the global action indifference space and A} (s) = A A (s) the set of forbidden actions. As mentioned above, keeping Asi(s) large is crucial because provides more actions for optimizing the arbiter policy.
Using Qi and & as defined above, we can now write down the Q-function of the e-lexicographic MORL problem, which shows that cunstrint-vvolatiin actions are forbidden by their Q-values.
(s, = Σ In(c(ss (s.
(9)
where we use In(1) = 0 and lim In(xr) -c. Here, the In(c;) terms are the transformed Q- functions for subtasks < n the view of Q-decomposition. Moreover, Q~ is equal to Q, for all E As (s) and Q~ (s. a) = -o otherwise. Our MaxEnt arbiter policy क (a l s) x exp(Q= a)) has a product structure,
Th (a (a| 8) x exp Σ In(c((s))) + ( = III 11-1
II (() -I<(()))) exp (s, (10)
confirming that マュ has likelihood 0 for actions outside the global indifference space whilst selecting actions inside the global indifference spaces proportional to exp Q..
close inspection of equation shows that the corresponding transformed subtask reward functions further revealing ry Σ r>i the scalar re- ward function of the s-lexicographic MORL problem. For detailed derivation, see supplementary material in Sec. A The Q-decomposition view suggests decomposed learning algorithm with the transformed rewards т>І- However, because the transformed rewards ry=; are defined as the trans- formed Q- p-functions Q- our decomposed algorithm learns Q2 and obtains Q by first computing
are (s. a) = as) and Pyn
c; for equation | Furthermore, Q; are learned on on-poicc for the arbiter. to avoid illusion of control in the subtasks (Russell & Zimdars, 2003 Laroche et al. 2017 van Seijen et al. 2017).
4

---

Published as a conference paper at ICLR 2024
3.2 INCREMENTAL AND DECOMPOSED LEARNING WITH LEXICOGRAPHIE PRIORITIES
We want to exploit the structure of the Q-function Q- and policy TTy of the s-lexicographic MORL problem as described in Sec. 3.1 to define an incremental and decomposed learning approach that can reuse and adapt subtask solutions. Our algorithm incrementally includes more subtasks ordered by their priority: In the first step, we learn subtask (with highest priority) without any restrictions. In the following steps, we learn subtask < i S restricted to the action indifference spaces of subtasks with higher priority than i and reuse the results of for the components of Q² Subtasks with priority lower than do not affect the learning of subtask i. In this and the following sections, we describe the process generically for the last subtask = n because it aligns with the notation in equation 9
subtasks, i.e. Qi to Qi-1
In the Q-decomposition view, we are optimizing the arbiter policy T with Q-function Q₂ for the MORL problem with (transformed) rewards "" 1 L we are given on-arbiter-policy es- timates Q for subtasks 1 I < n. As seen in equation 9 our means for improving the arbiter policy тh maximizing the term Qn for subtask n. For this, we perform policy evaluation step under the arbiter policy to get new estimate of the subtask Q-function Q% whiich also changes our estimate of the global arbiter Q-function Q- call this arbiter policy evaluation. Next, we per- form policy improvement step by softly maximizing Q which again also changes Q₂ we call this arbiter policy improvement. This algorithm closely resembles the Q-decomposition algorithm in Sec. 2. because Qn is updated on-policy with the global arbiter policy πT We theoretically analyze a subtask view of this algorithm below and provide an additional arbiter view analysis and interpretation of this algorithm in supplementary material Sec.
Subtask view theory. In the subtask view, the algorithm optimizes subtask with soft Q-learning in an MDP Myn that is like M in Sec. uses scalar reward r, and has the action space restricted to the global action indifference space, A~ (s). in every state. Therefore, the action space of MM satisfies the lexicographic priority constraints by construction. This allows us to perform off- soft Q-learning in Myn to directly obtain Q'n,. while still respecting all priority constraints. We do this with an off-policy, soft Bellman backup operator T
TQ(s. a) e r(s. a) + BB log / (x))
exp (Q)ss(aa))
(11)
V(8")
which is the update step from soft Q-learning (Haarnoja et al.,
2017 and where a E As)..
Theorem 3.1 (Prioritized Soft Q-learning). Consider the soft Bellman backup operator T, and an initial mapping Q" S x Ax - R with |Ay1 < oo and define Q++1 =TQ', then the sequence of Q' converges to Qi, the soft Q-value of the optimal arbiter policy के. as - oc.
Proof. Proof and detailed derivation in supplementary material Sec. F44
While the described backup operator T yields the optimal global solution in tabular settings, it is intractable in large and continuous state and action spaces. In the next section, we convert T and Theorem P.moorem..Tintte into a stochastic optimization problem that can be solved approximately via parametric function approximation.
3.3 PRACTICAL. LEARNING AND ADAPTATION ALGORITHM FOR CONTINUOUS SPACES
For PSQD, we implement the subtask view described in Sec. 3.2 building on the soft Q-learning (SQL) algorithm (Haarnoja et al., 2017). Learning subtask n the MDP Myn requires an effi cient way for sampling in the global action indifference space A_ (s), which becomes increasingly difficult higher dimensions due to the curse of dimensionality. However, we can exploit the incre- mentally learned subtask policies T; (a s) x exp (a(as)) s) to efficiently sample in subtask action indifference spaces because the high- sigh-rrbbbbility regions of subtask policies are the subtask indif- ference spaces. For rejecting action samples that outside of the global action indifference space, a & A (s), we evaluate the constraint indicator functions e; in equation The remaining action samples are used in importance sampling proportionally to exp Qn (a, s).
5

---

Published as a conference paper at ICLR 2024
Technically, we learn oy and Qn for the subtask n by integrating the sampling process from above in SQL, which corresponds to performing soft Q-learning in My-n We model the subtask Q-function loss
by DNN Qa with parameters 0 and minimizes the temporal-differee
(0) Es, air 2 (o ar) (s,, ar) + 71s [V (s+
(12)
where 0 is a target parameter obtained exponentially moving average of 0 and V. is the empirical approximation of equation | The subtask policy is represented by a state-conditioned, stochastic DNN f. with parameters o, such that a, -f (Gt, where Çe represents noise from arbi- trary distribution. The parameter & is updated by minimizing the (empirical approximation of the) Kullback-Leible divergence between f and the optimal MaxEnt arbiter policy in equation 3
(+) = DKI. (1% (-1se Go)|| exp (Q" (st. V
(13)
The objectives Jx (Ø) and (0) can be computed offline, given a replay buffer of saved transitions. Thus, PSQD can optimize the n-th subtask offline, for example reusing the transitions collected dur- ing independent and isolated training for subtask n in the subsequent adaptation step for subtask n as part of the e-lexicgrraphic MORL problem, detailed in supplementary material[]] pictographic overview of our method as well as pseudocode can be found in supplementary material D
EXPERIMENTS
We evaluate PSQD in instructive 2D navigation environment and high-dimensional control problem (details in Sec. G in the supplementary material). In Sec. 4.1 and Sec. 4.2 we qualitatively analyze zero-shot and offline adaptation results, respectively. In Sec. 4.3 empirically compare our method with a number of baseline methods. Finally, in Sec. 4.4 we demonstrate PSQD's effi- cacy in a high-dimensional, continuous control environment. For all experiments, we first learn the (untransformed) subtasks individually such that we have access to DNN approximations of Q, and π. for reuse and adaptation. We use the hat symbol, e.g., â., denote DNN approximations.
4.1 ZERO-SHOT REUSE OF PREVIOUSLY LEARNED SUBTASKS SOLUTIONS
In this experiment, we demonstrate that our approach provides solutions that satisfy lexicographic priorities even in the zero-shot setting without interacting with the new environment. We design a 2D navigation environment (see Fig. Iap. where the agent initialized random positions and has to reach the goal area at the top while avoiding a n-shaped obstacle. For the zero-shot solution, we simply apply our transformation (Sec. 3.1) to the already learned subtask Q-functions Qi Q2 to obtain the Q-function Q1~2 In(ci) + Q2- The modular structure of Q1>2 allows us to inspect the components of the zero-shot solution: Fig- lb] reveals that Qi learns the obstacle avoidance subtask with large negative values inside the obstacle and inspecting the In(c)) component shows our transformation and how implements the priority constraint. Fig. Ic we plot in(c; ((s, a)), Va A. with the agent placed at the red dot in Fig. la to show the action indifference space at that state. Our transformation excludes actions that would lead to collisions, allowing us to visually infer the agent's 's behavior. This is in stark contrast to prior zero-shot methods, e.g. (Haarnoja et al. [011 8a), where the behavior the zero-shot solution is unpredictable and potentially arbitrary sub-optimal, as noted by Hunn et al. (2019). In Fig. ld we visualize the policy xi>- ε Q1x2 with sample trajectories and project (n(c;) onto the image to illustrate the global action indifference space. Our zero-shot solution never collides with the obstacle due to the priority constraint without any training on the lexicographic MORL problem. However, our zero-shot is suboptimal for the lexicographic MORL problem, because it sometimes gets trapped and fails to reach the goal for starting positions in the middle. In Sec. G (supplementary material), we show more detailed zero-shot results.
4.2 OFFLINE ADAPTATION AND DATA REUSE
In this experiment, we demonstrate that PSQD can adapt previously learned subtask solutions to the lexicographic MORL problem using only the training data from the previous, isolated learning process without environment interaction. After adapting Q3 with the offline data, Q longer

---

Published as a conference paper at ICLR 2024
Top Goal
Forbidden actions
Obstacle
A+
Ax1
up
A₂1
Permitted actions right
(a) 2D environment.
(b) Sa Q (s, a)da.
(c) In(c2) in action space. (d) Zero-shot trajectories.
Figure I: Zero-shot experiment in the 2D navigation environment. Qi in 1b (brighter hues indicate higher value) and its transformed version Ic (evaluated at red dot) forbid actions that lead to obstacle collisions. Sample traces in [d (larger version in Fig. ( show navigation towards the goal at the top, sometimes getting stuck but without colliding with the obstacle. The background in is
colored according discretized angles of the policy NF1.1 11 100 20. 1) +1. 20. 0. 4j.
Moan Return
100
-200
O-shot
Ofline
-30
Online
RRT
-400
(a) Sa (s, a)da.
(b) f2 (s, a)da.
(c) Adapted trajectories.
(d) Return comparison.
Figure 2: Offfine adaptation experiment in 2D navigation environment. Our learning algorithm
adapts the pre-trained Q; in 2a to Q in 2b (brighter hues indicate higher value), reflecting the long-term value of 72 under the arbiter policy. The adapted agent has learned to drive out of and around the obstacle, as shown
(larger version in Fig. (). The background in [2 is colored the same way Id Both online and offline adaptation improve upon the zero-shot agent considerably,
as shown in 2d
reflects the greedy policy Tे that gets the zero-shot solution stuck, but instead reflects the arbiter agent T² and accounts for the long-term outcome. Comparing Qù in Fig. 2a with Q in Fig. 2b shows that the adaptation process leads lower values inside the O-shape such that our solution escapes and avoids that region. This optimal behavior is illustrated with sample traces in Fig. 2c Fig. 2d we compare the mean return of the zero-shot, offline, and online adapted agents and an RRT (La Valle& & Kuffner Jr. 2001]] Karaman & Frazzoli (2100) oracle. This shows that the zero-shot agent, which gets stuck, performs considerably worse than the adapted agents, which instead are on par with the RRT oracle. We continue this analysis for the more complex tasks 71>22-3 and r1>33-2 in supplementary material Sec, G
4.3 EMPIRICAL COMPARISON AGAINST BASELINES
of five baseline methods which all try imple- ment lexicographic priorities in MORL problems. As, to the best of our knowledge, PSQD is the first method for solving lexicographic MORL problems with continuous action spaces, we compare against obvious extensions of established RL algorithms and ablations of PSQD: Naive subtask- priority implementation for SAC (Haarnoja al.) 2018b) and PPO (Schulman al. 2017), a mod- ified version of Q-Deecmposition (Russell & Zimdars 2003) for continuous action spaces that we call soft Q-decomposition (SQD), as well as the SQL composition method proposed by |Haarnoja al. (2)18a). For the ablation of PSQD, instead of transforming subtasks, we introduce additional negative reward for priority constraint violations based on the constraint indicator functions c;. For SAC and PPO naively implement subtask- subtask-priorities by augmenting the learning objective to ad- ditionally include the KL-divergence between the current and the previously learned higher-prione policy. SQD concurrently learns the on-arbiter-policy Q-functions for all subtasks. The SQL compo-
In this experiment, compare PSQD

---

Published as a conference paper at ICLR 2024
SAC 0.1 SAC 0.9
PPO 0.1
SQL-Comp
PSQD ablation PSQD
PPO
SQD
Relurn f
Return
-200
-50
-400
-100
0
100
200
300
400
500
0
100
200
300
400
500
Episodes
Episodes
Figure 3: Baseline comparison in the 2D navigation environment. Left: Cost of the high-priority obstacle avoidance subtask during learning of the lower priority task. Right: Lower-priority navi- gation cost. For SAC and PPO, the scalar in the legend refers to the weight 3 of the convex policy objective, where
places all weight on the KL term.
80T also learns solutions for all subtasks concurrently, however off-policy and thereby with illusion of control. See Sec. G.3 for more details on the baseline methods.
sition (Harrnoj et al.
For SAC, PPO, and PSQD (inel. ablation), we first train all algorithms on the obstacle avoidance subtask 1, where all methods converge
approximately optimal solution. Subsequently, we reuse the obtained Qı and train each method on subtask 2, recording the returns for both subtasks separately in Fig. 3 PSQD is the only method that maintains zero return for subtask 1 (i.e. zero collisions) while learning for subtask 2, outperforming all other baselines. As seen in the right panel of Fig. 3 PSQD obtains less reward for subtask than other baselines. However, this is because the priority constraint prevents PSQD from navigating through the obstacle, while some baselines accept obstacle collisions reach the goal quicker. In contrast PSQD, none of the baseline methods converge to a solution that satisfies the lexicographic priorities and maintains the optimal performance in the top-priority collision avoidance subtask that they initially achieved.
4.4 HIGH--DIEENIONAL CONTROL
In this experiment, we demonstrate that PSQD scales well to high-dimensional action spaces while maintaining subtask priorities, unlike traditionally composed multi-objective agents. We simulated an Franka Emika Panda joint-control task with a 9-dimensional action space (Fig left), where the high-priority subtasks wants to avoid the red cube while the low-priority subtask want to reach the green sphere with the end-effector. We start with the zero-shot Q-function Q 1> In(c;) ) + Q2 and sample from πTy as described in Sec. 3.3 We subsequently adapt the zero-shot solution to obtain
Qi In(cr) - We also include the SQL-Comp. method (Haarnoja et al. 2018a)
baseline, by composing Q1+2 = Qi Q;. Representative trajectories from our adapted solution and the baseline are shown in Fig. (left.. Starting from 0 in the leftmost panel, our adapted contrast, the baseline violates the priority and moves through
agent moves around the obstacle. the forbidden part of the workspace. More quantitatively, we show mean returns for the different agents in Fig. (right). Our solution avoids the volume even in the zero-shot setting and is improved considerably through the adaptation process, while the baseline induces high costs in Y1 because it moves through the forbidden part of the workspace. We note that Haarnoja et al. (2018a) do not claim that their method solves lexicographic MORL problems, we use it to illustrate that priorities are generally difficult to express in traditional multi-objective tasks.
5 RELATED WORK
Task priorities for continuous action-spaces. To the best of our knowledge, PSQD is the first method to solve general lexicographic MORL tasks with continuous action-spaces. The following prior works consider task priorities in RL: Yang et al. (202I) combine null-space control and RL by constraining policy search to the null-space of higher priority constraints. However, Yang et al. (2021)ss 's method a special case of PSQD since it requires access to the fask-space Jacobian, while PSQD exploits given O-functions. Skalse et al. (2022) Zhang al. (2022); Li & Czarneck (2019) also model lexicographic task priorities, however only in discrete settings and without revealing the reward function of the lexicographic task. PSQD generalizes these methods to continuous action-
8

---

Published as a conference paper at ICLR 2024
-200
Monn
1=0
t-5 1=11 1-20
constraint
100
violation
Babot
Adapted
Buseline
Figure 4: High-dimensional joint control using a simulated Franka Emika Panda robot. Top left: Our prioritized agent respects constraints even in the zero-shot setting and is improved by our learn- ing algorithm. Bottom left: multi-objective baseline HHarrooja et al. (2018a) does not respect lexicographic priorities and generates high costs in rı (collision) favor of fewer costs in 72 (tra- jectory length), moving through the forbidden part of the workspace.
spaces and maximizes a known reward function. Furthermore, none of these works emphasize composition and reuse of subtask solutions, which is central to our method.
Composition for knowledge transfer. Composition for transfer between tasks in the MaxEnt framework has initially been demonstrated by Haarroja et al. (2018a), however, only in a zero-shot setting without providing a learning algorithm to improve the composed agent and without priorities. Hunt et al. (2019) build upon Haarnoja et al. (201 8a) and propose a learning algorithm that learns the divergence between the subtask Q-functions and obtains the optimal solution convex task compositions. Thus, similar to our method, these works facilitate knowledge transfer between tasks via composition, however, they offer no way for modeling strict task priorities.
Task priorities for safety. Task priorities can be used to implement safety. Prior works mostly implement (safety-) constrained RI. via constrained MDPs (CMDPs) (Altman 1999), where policy search is constrained by an unordered set of cost functions. CMDPs can be solved by optimizing Lagrangian objectives and dual gradient descent techniques Boyd al. (2004). This way, (Ha al.) 2021 Tessler et al. 2019 Chow et al.] 2017) focus on CMDPs with one handcrafted cost function, while (Achiam et al.) 2017 Liang al. 2018) provide methods for solving CMPDs with multiple constraints. While these methods yield constraint constraint-satisfying agents, unlike our method, they do not facilitate reuse and adaptation via composition, since they optimize monolithic policies from scratch. An exception w.r.f reuse is safety critic (Srinivasan et al. 2020). which pre-trains transferable estimator of expected future safety incidents. However, safety-critic can only incorporate one such critic and thus only transfer knowledge from pre-training task, while our method can exploit any number pre-trained Q-functions.
LIMITATIONS
Our method depends on the manual selection of E 5n-1 thresholds. These scalars are on the scale of subtask Q-functions Qi, which might be hard to estimate when function approximators employed and depend on domain-specific reward functions, user preferences, and task semantics. There are, however, number of informed ways for finding adequate E; scalars. practice, one can analyze the Q-function in key-states and select such that undesired actions are excluded. This works even when the action space of high dimensionality, since approximate Q-function mean, min, max, and percentiles can be computed via sampling.
7 CONCLUSION
The main contribution of this paper is principled framework for solving lexicographic MORL problems with continuous action spaces in a decomposed fashion. In this framework, we can zero- shot interpretable agents that respect task priority constraints in low- and high-dimensional settings. Our learning algorithm, PSQD, facilitates reuse subtask solutions by adapting them to solve the lexicographic MORL problem optimally.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Pro- gram (WASP) funded by the Knut and Alice Wallenberg Foundation.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC press, 1999.
Isac Amekvist, Danica Kragic, and Johannes A Stork. Vpe: Variational policy embedding for transfer reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 36-42. IEEE, 2019.
Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge uni- versity press, 2004.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained re- inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6070-6120, 2017.
Rodrigo de Lazcano, Kallinteris Andreas. Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2023. URL /tttp://githu> cwm/ Saarama-Foundat.o Gymnas i um- Rinb-Robotics.
Alexander Dietrich, Christian Ott, and Alin Albu-Schäffer. An overview of null space projections for redundant, torque-controlled robots. The International Journal of Robotics Research, 34(1 1385-1400, 2015.
Simon Duane, A.D. Kennedy, Brian J. Pendleton, and Duncan Roweth. Hybrid monte carlo. Physies Letters B, (95(2):21-2222 1987. ISSN 0370-2693. doi: https:/doi.org/ 10. 101101600370--229347797991911999
BRLhttps:: / http://ww.iiieeee/. sciencediiecct......cc....c. www./ooo/co.co...oo com/science/ articlee/p1100370263377777
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1..111. 1126-1135. PMLR, 2017. URL http:///rroceedings www.mmrrr....ppm.mmmmmmm http://///iinnaaaaaam/
Zoltán Gâbor, Zsolt Kalmár, and Csaba Szepesvári. Multi-criteria reinforcement learning. In Jude W.
Shavlik (ed.), Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998, pp. 995-205. Morgan Kaufmann, 1998.
Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. Conference on Robot Learning, pp. 1110-1120. PMLR, 2021.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep based policies. In International conference machine learning, pp. 1352-122 1361. PMLR, 2017.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE international conference on roboties and automation (ICRA), pp. 6244 6251. IEEE, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018b.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: (8te:099990 2018c.
10

---

Published as a conference paper at ICLR 2024
W. K. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97-109, 04 1970. ISSN 0006-3444. doi: 10.1095/biomet/57..197 URL https: http:///o.............o////eeeee
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin A. Riedmiller. Learning an embedding space for transferable robot skills. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck P Proceedings. OpenReview.net, 2018. URL htttpp::/ppeereevee.... net./
forum? Carum?id=rken dd=nk(22x77R
Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Aueti-Agent Systems, 36(1):26, 2022.
Jonathan Hunt, Andre Barreto, Timothy Lillicrap. and Nicolas Heess. Composing entropic policies using divergence correction. In International Conference on Machine Learning, pp. 291 19200. PMLR, 2019.
Alex Irpan. Deep reinforcement learning doesn't work yet. http:://wwwwallxxiee... com/ 2018/02/144 hatp:/4//11aaaa..... html 2018.
Sertac Karaman and Emilio Frazzoli. Incremental sampling- algorithms for optimal motion planning. Robotics Science and Systems VI, 104(2), 2010.
Romain Laroche, Mehdi Fatemi, Joshua Romoff, and Harm van Seijen. Melti learning. CoRR, Pess/100..0777 2017. URL httpppp://aaxii.ooooaaa/////////
Steven M Valle and James J Kuffner Jr. Randomized kinodynamic planning. The international journal of robotics research, 20(5):378-400, 2001.
Changjian Li and Krzysztof Czarnecki. Urban driving with multi-ohjective deep reinforcement learning. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor (eds.), Pro- ceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019, pp. 359-367. International Founda- tion for Autonomous Agents and Multiagent Systems, 2019. URL http: //tp:///ll acm. org/ citation. wtt.:iioo.cmmmm=3311113
Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv: 2018.
Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 1186):1087-109,, 1953.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 8(7540) 2015.
Thanh Thi Nguyen, Ngoc Duy Nguyen, Peter Vamplew, Saeid Nahavandi, Richard Dazeley, and Chee Peng Lim. multi-objective deep reinforcement learning framework. Engineering Appli- cations of Artificial Intelligence, 96:1039,5, 2020.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley, A survey of multi- objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
Gavin Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems, vol- ume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994.
Stuart J Russell and Andrew Zimdars. Oddcoomposition for reinforcement learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 656-663, 2003.
11

---

Published as a conference paper at ICLR 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707706647, 2017. URL http:// http://aaxvv..o.. org./v.orr//. abs/
1707.0634
Joar Skalse, Lewis Hammond, Charlie Griffin, and Alessandro Abate. Lexicographic multi-objective reinforcement learning. In Luc De Raedt (ed.), Proceedings of the Thirty-Firy International Joint Conference on Artificial Intelligence. IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 3430- 3436. orgai.org, 2022. doi: 00244963/jjcai12222/766 URL ijcai. 2022/476
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint arXiv:2010.14600 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
net/ forum?ics
SkfrvsA9FX
Peter Vamplew, Benjamin J Smith, Johan Källström, Gabriel Ramos, Roxana Rádulescu, Diederik M Roijers, Conor Hayes, Fredrik Heintz, Patrick Mannion, Pieter JK Libin, et al. Scalar reward is not enough: A response to silver, singh, precup and sutton (2021). Autonomous Agents and Multi-Agent Systems, 36(2):41, 2022.
Harm van Seijen, Mehdi Fatemi, Romain Laroche, Joshua Romoff, Tavian Barnes, and Jeffrey Tsang. Hybrid reward architecture for reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5392-5402, 2017. URL 1264a05id@222edae1155 57460 http://22eeae1111100000022500000 Abstroct..bbtt.t..... htm
neeris www.iiiss.ccpaeeee /tcppppa////1//ast//
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292 1992.
Quantao Yang, Johannes A Stork, and Todor Stoyanov. Null space based efficient reinforcement learning with hierarchical safety constraints. In 2021 European Conference on Mobile Robots (ECMR), pp. 1-6. IEEE, 2021.
Hengrui Zhang, Youfang Lin, Sheng Han, and Kai Lv. Lexicographic actor-critic deep reinforcement learning for urban autonomous driving. IEEE Transactions Vehicular Technology, 2022.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
12

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
A LEXICOGRAPHIC MULTI-OBJECTIVE TASKS
Here, we describe in detail how we model lexicographic task-priority constraints as scalarizable multi-objective RI tasks. To recap, we have multiple reward functions ".,..., -n. and we want to find a global arbiter policy T that maximizes Tn. while behaving almost optimal w.r.t. all higher priority tasks r,,
1n--- This is expressed with the constraint
max (i(s, - (i(s.a) S Eis va n, Vs E a'EA
(14)
on the performance measure (TT, which in this case is the Q-value. We note again the indicator functions
(s a) = [1, max Qi(s, - S otherwise
(15)
and that [[.-1 (s, 1 when an action a is allowed in state s according to each of the higher priority constraints (and thus the global action indifference space A-(s)), The key idea now is to transform the higher-priorine Q-functions Q, into new reward functions r>i by applying the natural logarithm to the constraint indicator functions, such that (s, In(e, (s. a)). Based on these transformed Q-functions and new rewards, we now define a multi-objective, vector-valued reward function that reflects strict task priorities
Tx (s, a) ())
(s, (n)
ry a))
:
(16)
7>n-1(ssa0) nn (5, a)
In ( Fn (s a)
We define the global reward function that the arbiter maximizes sum of transformed rewards
r (s, a) = Σ (r(s, a));
Σ (s, (s,
(17)
such that we are in the Q-decomposition setting Russell & Zimdars 2003). This allows to decom- pose the learning problem into n subtask Q-functions Q1...., Qn and one global policy k._. The global Q-function Q₂ for the prioritized task т> corresponds to the sum of subtask Q-functions, Q- E.= Q- Qn. where the first 1 criiherrrrioitty O-functions rewards, while the n-th Q-function is for the ordinary reward function rn- Because of MaxEnt RL the global arbiter policy Ty satisfies the proportional relationship क. (als) exp (s, a)), as described in Sec. 2.2
for the transformed
The global reward function in equation allows us to infer certain properties of Q₂ and mT Firstly, by noting that In(1) = 0 we see that the constraint indication summation in equation 17 is zero when action a is allowed according all higher priority tasks and in the global action indifference space A (s). This means that r> (s. (s a). Vs E S, Va E A (s), i.c. maximizing the global task corresponds to maximizing the lowest priority tasks in the global indifference space, which is property we exploit extensively.
Next, by defining lim In(x) -0, we see that ry (s, a) -00 when any number of higher priority constraints are violated by a in s. Furthermore, by noting that -oo = R - and -00 R200 = -00, we see that Q~ (s, a) -0,, Vs E S, va (s),), i.c. the value of any constraint-violating action is -os under the global task ry. Importantly, due to the proportional relationship тュ т (a|s) x exp(Q~(s. a)), follows that such constraint-violatti probability under our arbiter policies. This also means that the expected future constraint-violatioo value arbiter policies is zero, since the arbiter policy can not select constraint-violatin actions in the future. With this in mind, for the global Q-function of arbiter policies, we that
actions have zero
13

---

Published as a conference paper at ICLR 2024
(x = Σ ())
Σ
Qsi a) (s,a)
Σ (s, a) + (r. + +1 BB, 1 Q- (8t4,,1, - ((al))-n((nn (s
Σ (s, a) + (r,(s, + BB p. [n (5c+1,
(s, an) (s) (n)
(18)
This shows that we only have to learn the n-th subtask Q-function for ry (s,a) during the adaptation step, because the transformed Q-values of all n higher-priority subtask Q-functions are know by construction.
Lastly, since we are using the MaxEnt framework we have the proportional relationship
TTy (a 8) exp (Q> as)
n-1
(s(c((s))) a)) + (s)
exp
(19)
II ca(s, a)
exp expQn(s.a
and can see that
J. (a s) x exp a)) if II= (s, a)
(20)
otherwise.
This shows that `Ty- softly maximizes rn a) in the global action indifference space (s), the policy is proportional to exp(Qn.s, a)) and has zero probability for constraint-violating actions.
B ARBITER VIEW THEORY
Here we provide an analysis of our algorithm from the perspective of the arbiter agent for the lexico- graphic MORL task. In this view, the algorithm optimizes the arbiter policy with policy evaluation and policy improvement steps in transformed MORL problem using on- on-policy updates for subtask n, like Q-decomposition. Now we show that this learning scheme converges by considering fixed arbiter policy πy with the on-policy soft Bellman backup operator T defined as
T (s, - (8t, + 7 Fxx V (s++1)).
(21)
with
V (st) = EaINT, (st - log( (ar ss))]]
(22)
where V (st) is the soft, on-policy value function (Harmoja et al.
2b18b
Theorem B.1 (Arbiter policy evaluation). Consider the soft Bellman backup operator Th and an
initial mapping Qh S A + R with [A] < 3o and define Q++1 = = Qr The sequence of Q'
converges to Q Th the soft Q-value of n as + Do.
Proof. See supplementary material Sec. R1
14

---

Published as a conference paper at ICLR 2024
Theorem B.2 (Arbiter policy improvement). Given arbiter policy Tz,, define arbiter policy as TTy_ (-|s) & exp(Q" (s. .), Vs S. Then Q (+.s) 2 Q (:,8), Vs.
Proof. See supplementary material Sec. F.2
Corollary B.3 (Arbiter policy iteration). Let xy be the initial arbiter policy and define TH s) exp(Q= (s, )). Assume |A] oo and that Tn is bounded. Then Q improves monotonically and xy converges to T
Proof. See supplementary material Sec.
F.3
C OFFLINE ADAPTATION
Our algorithm, PSQD, relies on the off-policy soft Bellman backup operator T in Eq. equation and the objectives (0) in Eq. equation 12 and J=(0) in Eq. equation 13 which can be computed without online policy rollouts. We can therefore adapt previously learned subtask solutions Qi, T offline with retained training data for subtask i, (St, ar, 8t+1) D, However, since æ, was unconstrained during pre-training D, likely contains constraint-violatin
transitions that are im- possible in My To account for this, when sampling a transition (s;, Ar, adaptation of the pre-trained Qi. Ti, we can check whether a, E A (s:) and discard all constraint violating transitions. This is like sampling from new dataset D that contains only transitions from Myi. Depending on how well Dr; covers Myis we can learn the optimal solution for the global task entirely offline, without requiring additional environment interaction. This makes our approach maximally sample-efficient for learning complex, lexicographic tasks by re-using data retained from subtask pre-training.
D, during
15

---

Published as a conference paper at ICLR 2024
Sub-task enviroments
Sub-lask agents
Afbiter agent
Multi odjective envirnmtet with tranffrmed
Pistroneett
Fi
Qu.
a.
Equail
Policy sampling
to po
Exiroment
Ok-a
Ecrasence
Fremeneet:
Q
Taskt transforaaionn
Rx.
Sub-lask pre pe-leerring Concurent, online
Acbiler adgitationoop Transfer, offline
Sub-tack adaptation loop Transfer, offling, le agent
[r.
Figure 5: high-level overview of our method. Starting on the left, n agents individually learn to solve each subtask, we refer to this + as the subtask pre-training step. In the middle box, the subtask agents are combined into lexicographic arbiter agent. The subtask adaptation loop that
implement in practice, as described in Sec. 3.2 and 3.3 is denoted by -,, while the arbiter
learning perspective, described in App. is denoted by
D PSQD ALGORITHM DETAILS
method. A high-level overview of framework and learning algorithm given in Fig. 5) As can be seen, we begin by pre-training agents on the un- transformed, original subtasks rn. During the pre-training step, the agents learning in isolation and greedily optimize their respective reward signals. Different learning algorithms could be used for pre-training the subtask agents, but in our case, the pre-training step corresponds to running instance of Soft Q Q-aammnng Haarnoja et al. (2017). The pseudo-code for pre-training is given Alg.
Here, provide additional details
Pn-
Algorithm Subtask pre-training with SQL
Require: Subtask reward functions Ti,...,,
Samplers + 0
Q-functions+ 0
Buffers 0
for i E (1....,,)) do Initialize di, 0, 0, D; for é (l....,,). T do Online interaction Se ~ P
Iterate (concurrently) over subtasks {1,.... n)
Initialize subtask network parameters and empty buffer > T total steps
> Random noise for ASVGD sampling network > Unconstrained action selection using sampling network
f (G)
8:41 ~ O(5:+1 az)
DUD + {st, ar, Ti az), S
Store transition and subtask reward
Update networks
Update 8,
SQL Q-function update SQL sampling network update Polyak target network update
Update o;
0, + 70;
-)0
end for
Samplers +- Samplers Ufd+}
Q-functions + Q-functions U(0,) end for return Subtask sampling networks du)) Q-functions {0, (.....0)). and Buffers (Di.... )
Buffers Buffers U(D.)
16

---

Published as a conference paper at ICLR 2024
After the SQL pre-training step in Alg 1 have access to converged subtask Q-functions, sam- pling networks for those O-functions, and populated replay buffers for each subtask TÜ. Next, we perform adaption step where finetune the pre-trained Q-functions and sampling networks der the consideration of the lexicographic constraint on higher-priority subtask performance. This corresponds to the subtask adaptation loop in Figure [ Importantly, adapting the i-th subtask Q- function and sampling network requires that all higher-priority subtasks components have already been adapted to the lexicographic setting because otherwise, their indifference spaces might overlap (see also App. E). We therefore employ an iterative adaptation procedure where we start with the second highest task (the highest-priorit task is unconstrained) and adapt it under consid- eration of the lexicographic constraint for the highest-priority task. Then, the third-highest-prioriy task can be adapted under consideration of the lexicographic constraint for the highest- and second- highest-priority tasks, and so on.
Algorithm Incremental PSQD subtask adaptation
Require: Subtask reward functions "i,...,T'
Require: Threshold scalars E1,...,,n--
Require: Pre-trained subtask sampling network parameters .1...., On
Require: Pre-trained subtask Q-function parameters .I....,6, 8,
Require: Populated subtask replay buffers Di....,,, Dn
for i [2. do D D -s,, a,. r; (st, at). S441 & A,
& Iterate over subtasks, starting with second-highest priority task Discard constraint-violating transitions T total steps
for / , T} do
Optional online interaction
// As described in Sec. Sec.3.3 we perform importance sampling to sample from πーşs the // lexicographically constrained policy for the current subtask. The already adapted I1 Φ1,-. Φi-1 and the current Φ; sampling networks are used as proposal distributions, // with the unnormalized target density being given by Eq. TO
ag Tzi S p(5++1 St, ar) DUD - {s,, ar (S A,),s1411
Store transition and subtask reward
Update networks
Update 0g using Eq.
Update 6, using Eq. 13
+ T0, + (1 ( 0
b Polyak target network update
end for end for return Adapted sampling network parameters di and Q-funetion parameters 8
Notice that the online adaptation step in Alg. 2 is optional, it is also possible to compute the network updates entirely offline, using the filtered versions of the subtask replay buffers from pre-training. The filtered versions of the replay buffer are obtained described in App. C where ensure that no constrain-violatin data should be collected from the environment, this be done by sampling actions from the constrained policy for the current subtask.
transitions are used during offline adaptation. If additional online interaction
17

---

Published as a conference paper at ICLR 2024
E DISCUSSION ON SEMANTICALLY INCOMPATIBLE SUBTASKS
Here, we discuss an interesting edge case, namely, situations where subtasks are semantically in- compatible. Let us consider an example where we wish to jointly optimize three subtasks: Subtask T1 rewards moving to the left, 12 = -r1 rewards moving to the right, and ry rewards moving to the top. Clearly, T1 and 12 are semantically incompatible, since r2 is the inverse of r1. We should first notice that such MORL tasks with semantically incompatible subtasks are inherently ill-posed, since agent can, at the same time, behave optimally for both ri and 72- Intuitively, either move to the left, or to the right, but it can not tend to both directions at the same time. We emphasize the semantic incompatibility, since, from the MORL algorithm's 's perspective, tasks can not be "incompatible". Even when '2 = -7,, maximizing the summed reward rı + 12 is still valid, even though the optimal behavior for this task might not be aligned with the RL practitioner hoped to obtain.
agent can
In the general case, given semantically incompatible subtasks, non-lexicographic MORL agent that maximizes + 12 13 will likely fail to progress on either of the incompatible subtasks, as shown by (Russell & Zimdars 2003). The behavior depends on subtask reward scale, the discount factor, and function approximation errors, which can be considerable since deep regression for Bellman optimality tends to overestimate state-action values et al. (2015). Therefore, MORL algorithms without subtask priorities are likely to produce unexpected and undesired behavior when subtasks are semantically incompatible.
Importantly. if we instead optimize the lexicographic task r1 > 72 T's, most of the above issues alleviated: The lexicographic problem definition states clearly how the agent ought to behave, since, matter how different the subtasks might be, near-optimal performance of high-priority tasks is required by definition. For the same reason, lexicographic tasks are also insensitive with respect to subtask reward scale and discount factors, since they are not optimizing a weighted combination of subtask rewards. In fact, lexicographic RL was introduced and motivated by (Gábor et al. 1998) to precisely resolve situations with semantically incompatible subtasks, e.g. "Buridan's 's Ass" dilemma, which is a philosophical paradox that translates directly to conflicting subtasks in MORL.
While lexicographic RL inherently resolves subtask incompatibility via its problem definition, we now clarify how our learning algorithm resolves these situations in practice. First, we notice that greedy subtask solutions obtained through SQL pre-training in Alg. Dean indeed be "incompatiblee. Pre-training independently on our exemplary tasks 1 and 12 = -rı would result in Q-functions Qi and Q2 that assign high-value to inverse parts of the action space (moving left and moving right). Thus, the intersection of the indifference spaces given by Qi and Q2 can be empty, which would imply an empty action space for lower- tewer-priorit subtasks and an ioo- learning problem.
Our definition of the lexicographic constraint in Eq. however, formulates the lexicographic con- straints on the on-policy O-functions for the constraint- trespecitt- arbiter agent. Our learning and adaption procedure in Alg. acccunts for this since it sequentially adapts subtask solutions, starting with higher-priority subtasks. This means that in the first iteration, we adapt the solution of the sub- task with second-highest priority the lexicographic optimality constraint on the highest-priority subtasks. This changes the greedy subtask agent (i.e. its Q-function) Q2 ~- Q to an agent that is no longer greedy with respect to its original subtask the adapted agent instead solves r22 as well as possible, while respecting the lexicographic constraint. As described in A Q assigns -oo to all actions that violate the lexicographic constraint, therefore, the high- region and indifference space of Q has to be inside the higu-value region in Qi, i.e. inside the indifference space of the higher-priority task r1- Thus, although Qi and Q2 might be "incompatible", the adapted Q is indeed compatible with Qi, even when r2 -71- Since in our adaption procedure, lower-priority subtasks always use the already adapted subtask solutions for tigher--riority tasks, the intersection of all higher-pin ppiooityy indifference spaces can never be empty.
In summary, lexicographic MORL tasks resolve incompatible subtasks by definition, since lower- priority subtasks are constrained to solutions that also near-optimal for all higher-priority sub- tasks. Our adaption procedure accounts for this by adapting subtasks sequentially, which ensures that the intersection of all higher-priority subtasks is never empty.
18

---

Published as a conference paper at ICLR 2024
F PROOFS
In Sec. F.T to Sec. we prove the "on-arbiter-poliey" view of our algorithm. We prove the subtask view of our algorithm in Sec. [.44 These proofs are based on and make use of the following obser- vation. Due the incremental nature our learning algorithm, the higher-priority tasks have already been learned in an on-arbiter- ----arteer--licyy fashion and are fixed during learning of the n subtask Q-function. Thus we have access to the transformed subtask Q-functions Q~ for all higher prior- ity tasks ,,..., 1, which are stationary during learning of the n-th subtask Q-function. As consequence, the arbiter policy ky, which we want to improve and for which we want to learn n-th subtask Q-function in an on-policy fashion, is already constrained to the global indifference space and can never select monstraint-villatin actions. Thus, we can rely on existing proofs for MaxEnt RL, since our arbiter policy is just particular MaxEnt policy.
F.1 ARBITER POLICY EVALUATION
Lemma F.1. Arbiter policy evaluation. Consider the soft Bellman backup operator T and an initial mapping Q S x A I with |A] oc and define Q = TTQ' The sequence of Q' will converge to Q% the soft Q-value of æ, as I - 30.
The proof straightforward based on the observation that the n-th subtask Q-function is simply soft Q-function for the task Tu. learned under the expectation of the global arbiter policy (Russell) Zimdars 2003), which has zero-probability for constraint-violating actions. Thus, the proof is analogous to the one by Haarnoja et al. (2018b) except in our case π is constraint-respecting arbiter policy. We repeat the proof here only for completeness:
Proof. We define the on-policy, entropy augmented reward signal for soft Q-functions as (st, ar) Tn (St, ar) + Ex, (=>pp=/ (18)+11) and rewrite 7
07 a.) f r,, (St, + 1=
(23)
The standard convergence result for policy evaluation (Suttonn & Barto 2018) thus holds. The as- sumption |A|] < is required guarantee that the entropy augmented reward is bounded.
L
F.2 ARBITER POLICY IMPROVEMENT
We want to show that we can obtain an arbiter policy Ty that is better or as least as good as any other arbiter policy by softly maximizing the soft global Q-function. Since the arbiter policy maximizes
the sum of all (transformed) subtask Q-functions we set = Σj= rrs + In and Q
O ΣØ Qr Qn for the remainder of this section.
Since we are concerned with finding improved arbiter policy and since all arbiter policies have zero probability for selecting constraint-violatie actions, the T>j and Q~ terms evaluate to zero and we can only improve Q by increasing Tu- Q still correctly assigns value of ooo to tions that violate constraints, however, this value is known through the constraint on already optimal higher-priorit. Q-functions and not backed up while learning task Tn. In practice, since arbiter poli- cies can never select sesttraint-volating actions, the soft policy improvement theorem by Haarnoja al. (2017) also holds for us since we are softly maximizing the ordinary task rn with particular soft policy. Arbiter policy improvement for the global tasks thus degrades to maximizing the n-th task while respecting priority constraints. In the following, we use the short-hand r, r(st, a,) for a reward at time for more compact notation.
Theorem F22 Arbiter policy improvement. Given an arbiter policy n, define a new arbiter policy as n (-Is) x exp(Q (s, )),, v's E S. Then r'(+ss) 2 T(-ss), Vs. a.
Proof. The soft policy maximization objective
1T J(TT) 2 Σ Ree.. ( (St, a.) + H(T [s.))]
(24)
can be improved either by increasing the soft Q-value or the entropy term: (sp (s, a.)] + H(T(((ss)) (e 10 a:)] H(T (-(ss)))
(25)
19

---

Published as a conference paper at ICLR 2024
Recalling the definition of the soft Q-function
(st ar) - ro(se, a.) + E (s Jopa
7 (r(s:41, ac+1) + H(a sstii)))
(26)
the proof for the (soft) policy improvement theorem now simply expands the right side of equation 25 with one-step look-aheads until one obtains Q
Quon (so: an) = ro Ls
((H((T(-|s))) Ea ~ (si
$ ru Es, (((((=(((ss))) Ea, ~ (Q (S1. ]1] ((H((=(((s))
rn + Es,
+ 2 Re H((x(1152)) + Enz~r (82, a2)]
S ro + Bay ((H((=((1|s))
+ + 2
H((T'( ((=x((11))) (-112) + T Ra ( (82, a2))
(H((((((s))) + +n)] + + (-152) + (s2, a2)
ro + ((ss (2)
2
Σ
drew + H(T (((1))))441))) S(t1)))
ro + 1. supp
Q (So, an)
(27)
F.3 ARBITER POLICY ITERATION
Corollary F.3 (Arbiter policy iteration). Let x be the initial arbiter policy and define a (-1s) x Assume |A| and that rn is bounded. Then Q improves monotonically and x² converges to n..
exp(Q= (0000~(ss))))))))) (s.
Proof. Let be the arbiter policy at iteration I. According to Theorem B.2 the sequence Q im- proves monotonically. Since arbiter policies only select actions inside the global action indifference space, the In(c;) components of Q evaluate to zero and since r, is assumed to be bounded and since the entropy term is bounded by the |A| < oc assumption, the sequences converges to some
is optimal, since according Theorem we must have Q O Vuz # m..
n T
F.4 PRIORITIZED SOFT Q-LEARNING
We now prove the local view on our algorithm, by showing that we can define an off-policy backup operator that accurately reflects the global arbiter policy, thus connecting the global on-policy and local off- ooflpo views on our algorithm. For this, we first note the soft Bellman optimality equations that have already been proven by HHaanoja al. (2017)
Qin (St, ax) r(st, ar) B Vin (S
(28)
with
V Kon (S:) = log / exp
1 Qun d'),
(29)
softmax
where the softmax operator represents a policy Toolmmx that softly maximizes Q" Thus, we can also write
(so, a.) r(st, ar) 7 F pmp. Tantina (8
(30)
20

---

Published as a conference paper at ICLR 2024
which shows that soft Q-learning is equivalent to Q-learning with softmax instead of greedy action selection.
We now show that off-policy, soft Q-learning fashion. In principal, this simply replaces the softmax policy Tolimmx with the the arbiter policy лh in equation [0] However, the arbiter policy does not softly maximize the n-th subtask Q-fucction like the locally greedy softmax policy in equation 29 would do. Instead, nー Q>j + (s, a). To it softly maximizes the sum of subtask Q-functions (s, a) = j==
learn the optimal 72-hh subtask Q-function under the arbiter policy in an
show that we can approximate the global arbiter oplicy in an off-policy fashion, we split the action space of the MDP into two disjoints sub-spaces A ( (s). The first sub-space is the action indifference space A~ (s) that contains all constraint respecting actions in each state, while the second sub-space A (s) contains all constraint violating actions. Thus, we can split the integral
in equation 29
f
V (s:) = log
exp (Qion (st, az da + log
exp (Q (s,, a))
(31)
A- (s.)
(s
We now insert our optimal global Q-function and obtain
exp (Q (St, (((s,,a)))))) d) + log J/ (8)
(s:) = log
exp (Qr (sc, a))
(32)
A. (s)
which immediately simplifies
(S+)s) = log /
exp (Q* (St: a> day
As (x)
(33)
log
exp (Q (s:, a, da
(s)
where the second integral over A disappears because actions in have zero probability under the arbiter policy while Q becomes Q; because the n-1 subtask Q-function all evaluate to zero. Thus, optimal value state under the arbiter policy corresponds to the softmax of the untransformed, n-th subtask f-fuuction, in the global indifference space. Thus for the optimal state-action value function the arbiter policy we have
Q (St, ar) = r(St, ar) +
(34)
which is equivalent to
O (s,, a,) r(s,, a.) 2 10 (S a
(35)
We can learn Q with the off-policy, soft Bellman backup operator
TQ(s, r(s, + Eo log /
JA_ exp (Q(s), a a-)
(36)
(=()')
Theorem F.4 (Prioritized soft Q-learning). Consider the soft Bellman backup operator T, and an initial mapping Q" :S Ar R. with |Ay co and define Q++1 =TQ' then the sequence of Q' converges Q2, the soft Q-value of the optimal arbiter policy nं as
As we have shown, V is the value function of the global arbiter policy, while Q = Q: is an ordinary soft Q-function (because we are indifference space). Thus, the original convergence proof by Haarnoja et al. (2017) directly applies to T, we repeat it here for completeness with some annotations:
Proof. We want to show that T is a contraction, thus we note that the definition contraction on some metric space M with norm is
((((),,()) kal( v).
(37)
21

---

Published as a conference paper at ICLR 2024
with 0 < k 1. Next define the supremum norm for soft Q-functions 1|Q: Q2ll - max3,a [Q1 a) Q2(s, a) and set e = 1|Q1 - Q2l1. It follows that
log / eep((i (s' a' d'a''' s log / exp((2(s', a') + e)da'
log
cxp(c) exp(Q2(s',a) aa)) )da')
(38)
/
=c+ log
exp(Q2(s' a') d')
From the last row and using the subtraction property of inequalities, we get
log / (xp((1(s',a')))) - log / (xp((2(s',a'))) a')) Se
(39)
and immediately see that the soft multi-objective Bellman operator is indeed contraction:
IyEs'rp nup log / exp (Qi(s', a')) da' log / exp (Q2(s', a')) S yC >l1Q1 Q2ll (40)
Here, in the left side of the inequality, 1| T=Q22112 22, the r(s, a) terms cancel out and we collapse the two expectations Egp into one. The same is true on the right side of the inequality. Thus T is contraction with optimal Q-function as fixed point.
22

---

Published as a conference paper at ICLR 2024
(a) Zero-shot agent for r1 The agent does not collide with the obstacle but not globally optimal, since it greedily navigates to the top and gets stuck in front of the obstacle.
(b) Adapted agent for ri>2. The agent has adapted the task priority straint and navigates around out of the obstacle (yellow area in the mid- dle).
[[- 1] 00, 1] (1, 1] -[1 0] (1, 0] < <|-1, 1 100, -1] [1 -1]
Figure 6: Larger copies of Fig. ld and 2c of the policy, using the colormap above.
The background is colored according to discretized angles
23

---

Published as a conference paper at ICLR 2024
G EXPERIMENT DETAILS
G.1 2D NAVIGATION ENVIRONMENT
The action space for this environment is in R and corresponds to positional changes in the plane, while the observation space is the current (x,y) coordinate. The agent's position is clipped to the range [-10, 10j for both dimensions. We normalize actions to unit length to bound the action space and penalize non-straight actions. The high-priority task r1 corresponds to obstacle avoidance and yiclds negative rewards in close proximity to the M-shaped obstacle (see Fig. la
- 0² exp( d - 0² exp( (- 2
if dd>0 0 otherwise,
(8)
2
(41)
where d is obstacle distance (inferred from s), = 1 and = parameterize a squared exponential kernel, and B 10 is a an additional punishment for colliding with the obstacle. The auxiliary rewards 72 and respectively yield negative rewards everywhere except in small areas at the top and at the right side of the environment
if > 7 otherwise,
r2(s) =
(42)
0 sf >
rs(s)
(43)
otherwise,"
where we use 5 in all our experiments. Thus, the lexicographically prioritized task r1>2 corresponds to reaching the top without colliding with the obstacle. Due to the low dimensionality of the environment we directly exploit the proportionality relationship in equation I0 and rely on importance sampling instead of learning policy network for the 2D navigation experiments.
G.2 FRANKA EMIKA PANDA ENVIRONMENT
This environment features a simulated Franka Emika Panda arm, shown in Fig. 4 and based on the Gymnasium Roboties package (de Lazcano et al. 2023 The action space A IR corresponds to joint positions while the state space S E RT contains all joint positions and velocities. In this environment, the high-priority task r1 corresponds
avoiding a fixed volume (red block) in the robots workspace and returns - 10 when any number of robot joints are inside the volume. The low-priority task 72 corresponds to reaching a fixed end-effector position (green sphere) and yields rewards proportional to the negative distance between the end-effector and the target plus a bonus of 100 for reaching the target. The prioritized task ri>- thus corresponds to reaching the target end-effector location while keeping all joints outside of the avoidance volume.
G.3 BASELINE ALGORITHMS
Here we provide additional details on the algorithms used in the baseline comparison in Sec. 4.3 Since, to the best of our knowledge, PSQD is the first method that solves lexicographic MORL problems with continuous action spaces, we can not rely on existing lexicographic RL algorithms for discrete problems as baselines. Instead, we implement lexicographic task priority constraints by simplistic means in the following, state-of-the-are DRL algorithms that support continuous action spaces.
Prioritized Soft Actor-Critic Soft Actor-Critic (SAC) (Haarnoja al.. 2018b learns uni- variate Gaussian actor by minimizing the Kullback-Leibler divergence between the policy and the normalized, soft Q-function. This policy improvement step given by
arg minDKL (T' exp ( (s
nnw
(44)
wen
where II is set of tractable policies (e.g. parameterized Gaussians), and 7 is the normal- izing constant for the unnormalized density given by (s;, To make for lexicographic, i.. task priority-constrained version of SAC, simple approach is to add regularization term to
24

---

Published as a conference paper at ICLR 2024
equation 44 to penalize lower-priorige subtask policies from diverging from higher- ceaher-priorty subtask policies. Based on this intuition, for our prioritized SAC baseline, we augment the objective in equa- tion 44 by adding another KL term that measures the divergence between the current subtask policy and the higher-priority subtask policy
Tnew arg minn1 DKL st) exp (2 (1 T'
+ 3DKL (O (e) (Tre (45)
(Se)
where 8 is weight for the convex combination of the two KL terms and Tpe is the pre-trained, higher higher-priority policy. In practice, the parameters of the univariate Gaussian actor for our prioritized SAC are given by DNN parameterized by Φ, that uses the reparametrization trick to minimize the
loss
(x) = Ea [Ba, ~x [11 (og 8e)) ((s ac))
(46)
+B (a ss)) log (a. ss]]]]
which is the original SAC objective with the additional KL regularization.
Prioritized Proximal Policy Optimization To implement a prioritized version of Proximal Policy Optimization (PPO) Schulman et (2017). we apply the same intuition as for the prioritized SAC version, i.e. we regularize the policy to be similar to the pre-trained, higher-priority policy. PPO maximizes a clipped version of the surrogate objective
o 5t) Ar].
(Ø) Eg
(47)
Ι dhus (ar
88.)
where A, is an estimator of the advantage function for task return at timestep L and data is the parameter vector of the policy before doing an update. To encourage learning a policy that respects task priority constraints, change the maximization to
(a 5i) Tha
(Ø) E,
(1- 3) 3KL ( (a' (:)))) (ppe((:ss)))
(48)
aa ss))
such that the update now increases the probability of actions that have positive advantage and similar probability under the current and pre-trained, higher-priority policy.
PSQD Ablation For the ablation to PSQD, also pre-train the higher-priority Q-function for the obstacle avoidance task and rely on a the ci (s, a) priority constraint indicator functions. However, instead of using them project the policy into the indifference space, as PSQD does, we use them as an indicator for punishment of -100 that is subtracted from the lower-priority task reward whenever the agent selects a mnnttraint-voolating action.
Soft Q-Decomposition With "Soft Q-Decomposition" refer a continuous version of Russell & Zimdars (2003''s is discrete Q-Decomposition algorithm that we describe in Sec. 2. This algo- rithm concurrently learns the subtask Q-functions Q4 for a MORL task with vectorized reward func- tion, such that the overall Q-function, Qz, can be recovered from the sum of subtask Q-functions. A key property of this algorithm is that the constituent Q-functions are learned on-policy for a central arbiter agent that maximizes the sum of rewards. To adapt the discrete Q O-Decomposition algorithm to continuous action spaces, we combine it with the Soft Q-Learning (SQL) (Haarnoja et al.) (017) algorithm. However, SQL is soft version of Q-learning (Watkins & Dayan 1992) and thereby off-policy, thus we can not directly use SQL to learn the constituent Q-functions. This is because the SQL update to the Q-function
(o) = Ex, D
2 (s), -
-rr (s, a.) + +BBuur [V (3
(49)
V = log / (sp(Q%(ss. a')) da',
(50)
calculates the value of the next state by integrating over the entire action space, thereby finding the softmax of the next state value. In practice, this integral is not computed exactly, but instead approximated with actions sampled from some proposal distribution Ga'
4 n2 (sx) = log Eng exp 1 Q (st. ag)
(51)
qa (a')
25

---

Published as a conference paper at ICLR 2024
which typically is the current policy. Thus, to fix the illusion of control in SQL update to the subtask Q-functions, suffices to use the arbiter policy instead of the greedy subtask policy for fa' This way, the approximate soft value function in equation [1] assigns state values that correspond to the real softmax ("ILogSumExp" expression) of Q-values for actions sampled from the arbiter policy.
H ADDITIONAL RESULTS
Here we provide additional qualitative results from the 2D navigation environment that aim to make for additional intuition on method and how it solves lexicographic MORL problems.
H.1 OBSTACLE INDIFFERENCE SPACE
To provide further intuition for how our composed agent implements constraints, we provide addi- tional examples for Q- In Fig. we plot Q₂ In(c1) ) + 2% at multiple locations in the environ- ment. The black areas indicate Az,, the areas forbidden by the higher-priority obstacle avoidance task, which remain fixed during subsequent adaptation steps. The non-black areas correspond to O- in Ar, where the lower-priority tasks learn and express its preferences.
H.2 MULTIPLE CONSTRAINTS
We illustrate tasks with multiple lexicographic priority constraints in the 2D environment. For this, we consider the prioritized tasks r1=22-3 and 3>>-3>2- Both of these lexicographic MORL tasks assign the highest priority to the obstacle avoidance subtask v1, but prioritize r2 (moving to the top) and r3. (moving to the right) differently. The prioritized task T1>2-3 assigns higher priority to reaching the top than reaching the side, while r1-3>2 assigns higher priority to reaching the side than reaching the top. This is a concrete example of how task priorities can be used to model different, complex tasks. The composed, adapted agents corresponding to these lexicographic tasks are visualized in Fig. Sa &b
8b Each white arrow corresponds to one action taken by the agent.
As can be seen, the behavior of the agents for the different prioritized tasks differs noticeably. which is expected for different task prioritizations. The learned agent for Ti>-2>3 2>-3 first drives to the top, then to the side, while the agent for ri>3>2 first drives the side, then to the top. We can again explain this behavior by inspecting the components of these agents in Fig. [c and 8bb For this, we place the agent marker 5 in Fig. 7e and plot the constraint indicator functions for the higher priority tasks as well as the Q-function for the low priority task, for the entire action space. Both agents are constrained by In(e)) which induces AA1 and forbids actions that bring the agent into close proximity of the obstacle. In(c2) in Fig. induuuss Asz, which constraints π. to actions that move up. In(eg) Fig. Sd on the other hand, restricts the Tr to actions that move to the side.
26

---

Published as a conference paper at ICLR 2024
(a) Marker O.
(b) Marker 1.
(c) Marker 2.
(d) Marker 3.
(e) Environment marked positions.
(f) Marker
(g) Marker 5.
(h) Marker 6.
(i) Marker
Figure 7: Analysis of the global Q-function in action space: Q- = In(ci) + Q at different posi- tions in the 2D environment. The x and y axes of the images correspond the and y components of the action space. Brighter colored (yellow) hues indicate high probability, while darker colored (blue) hues indicate low probability under kソ. The black mask shows A.. As can be seen, the task priority constraint forbids actions that lead to obstacle collisions, since those actions are not within E value range of the optimal action for the high-priority task 1.
27

---

Published as a conference paper at ICLR 2024
<[-1, 1] 100, 1] 11, 1] -[-1, 0] 11, 0]
<[-1, 11 1[0, -1]
>[1, -1]
20
(a) Trajectories and policy for r1x-2-3-
(b) Trajectories and policy r1>3322
Ax
A
A
n
A
A
0 -
(c) Indifference space and global policy for P15-2>3- (d) Indifference space and global policy for P1>a=>2
Figure 8: Trajectories, policy, action indifference space, and value function for global agents. For the bottom row images, the agent was placed at marker 5 in Fig. Fig-77e(i (i.e., in the bottom left, close to the starting points of the trajectories). While both tasks are constrained by A_1, which prevents the agent from getting too close to the obstacle, the differing prioritization of 72 and induces different global indifference spaces through Ay and Ay3. The respective lower-priority task expresses its preference in the global indifference space (colored area). As can be seen, the policy in (c) constrained to actions that move up. due to the optimality constraint on Q2. while in (d), the optimality constraint on Q3 constraints the policy to actions that move right.
28

---

Published as a conference paper at ICLR 2024
0.0
6]
0.25
Ei 1.58
El
10
0.1
5]
0.63
E1
3.98
E1 100
Relurn f
Return Pg
-200-
-50-
-100
-100
0
100
200
300
400
500
100
200
300
400
500
Episodes
Episodes
Figure 9: Ablation in the 2D navigation environment. Left: Episode return of the high-priority
obstacle-avoidance task during learning of the low-priority goal-reach task. Right: Episode returns of the low-priority task. High-priorio subtask performance decreases and low-priority subtask per- formance increases with larger values.
H.3 é ABLATION
To illustrate how e thresholds affect the learned behavior and performance, we perform an ablation study using different values for 51. We again use the 2D navigation environment from App. G and assign high priority to the obstacle avoidance tasks and low priority to reaching the top goal. We generate values for E that are roughly log-spaced between 0 and 100. For this experiment, pre-train only the high-priority obstacle avoidance task and learn the low-priority goal-navigation agent from scratch. We report the same metrics for our baseline comparison in Sec. 4.3 meaning during learning of the low- now-priority tasks r2, we log the episode return of both subtasks, to show the trade-off between the two objectives, depending on the threshold value.
The average results from five different random seeds are shown in Fig. 9 with high- task returns shown on the left, and low- low-priority task returns shown on the right. Performance on the low-priority subtask roughly falls into three groups, depending on E c1-threshold. With very small values for E1, i.e. ε = 0.0, 0.1, or 0.25, the algorithm can not improve the low-priority subtasks. This is expected because the small thresholds induce such small indifference spaces that none of the available actions allow the agent to improve the performance of the low-priority subtask. At the same time, these small E thresholds result in optimal performance for the high-priority task, since they allow almost no divergence from the optimal behavior. Independently of the value for E as can be seen in the left panel of Fig. 9 the episode returns for ri roughly constant, since Q does not change during learning of r2.
On the other end of the range, i.e. with large E values of 3.98, 10, 100, the algorithm achieves optimal performance for the low-priority task. This is because the large values induce indiffer- ences space that are not restrictive enough prevent the agent from hitting and moving through the obstacle, which is reflected by the high costs that these agents obtain under the high-priority task.
Most interesting are the remaining E values, namely 1 0.63 and E 1.58. With these values, the agent achieves the same near-optimal performance with respect to the high-priority obstacle avoidance task while also managing to improve fow-priority subtask performance considerably. This is because the resulting indifference spaces prevent obstacle collisions (as can be seen in Fig. 7 with Ei 1) but still allow for many actions that can be used to optimize the -r-irrioity task.
In summary, the E; scalars have a strong effect on the behavior and lower-priority subtask perfor- mance. Too large thresholds are ineffective at forbidding undesired actions, while too small thresh- olds are too restrictive and prevent the agent from improving subtask performance at all. However, as described in Sec. 6 it is straightforward to infer these adequate threshold values by analysis of the higher-priority Q-function, even when the action space is of high dimensionality.
29

---

Published as a conference paper at ICLR 2024
1 REPRODUCIBILITY
A GitHub repository with the implementation of the algorithm, experiment setup with hyperparam- eters, and documentation is available here: https: / //githa.... com/ (uleet288psqccl The repository provides the complete PSQD implementation and can be used to reproduce the results in this paper.
30


Reviews:
[{"review_id": "OoC6vt98UA", "reviewer": "Reviewer_tsYf", "summary": "This paper presents a new methods to learn Reinforcement Learning policies that obey lexicographic subtask constraints. To make this efficient, the presented method creates zero shot Q-functions and strategies for the priority constrained task by composing transformed versions of the individual subtasks through their limitation to the indifference space of actions for the higher priority tasks and the transformation of the reward and Q value function to infinitely penalize such action choices. The resulting one shot version of each task value function and policy can then be adapted offline or online to achieve potentially near-optimal performing policies.\nThe main contributions in the paper are in the novel methodology and corresponding learning algorithm to form RL strategies by composing subtask functions such as to obey priority constraints expressed in the value function of the higher level tasks.", "questions": "The main questions arise out of the weaknesses stated above:\nHow does the proposed approach deal with incompatible subtasks ? Does it simply eliminate all tasks with empty indifference action sets for higher priority tasks and then operate in the same way as presented in the paper ?\n\nHow sensitive is the approach to the specific choice of \\epsilon thresholds ? Is there a way that an ablation could be performed that would investigate the sensitivity of the top priority task's threshold in the navigation experiments ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper tries to address two very important problems in Reinforcement Learning for control tasks in: i) providing an effective means of composing overall behavior from learned subtasks without significant need for new data collection and re-learning, and ii) to enforce strict priority constraints during composition as well as subsequent learning to optimize from the initial one-shot policy. These abilities are very important in the area of robot control in order to be able to enforce learned safety and performance constraints when new task compositions have to be learned.\nThe approach seems overall sound (although the description is lacking in a few places) and the results demonstrate that the method can form both single-shot strategies and more optimized policies using additional off-line (or on-line training) training that maintains the highest priority constraint.", "weaknesses": "The main weaknesses of the paper are in a lack of discussion of the full range of situations in the presentation of the underpinnings of the framework and in the incomplete description of the experiments presented in the paper.\nThe former here seems the biggest weakness and mainly relates to the complete absence of a discussion and consideration of cases where subtasks might not be compatible and thus the indifference space of actions might become empty. It would be very important for the authors to discuss this situation and how the algorithm would react under those circumstances. This is even more important as in the description of the decomposed learning task only the highest and the lowest priority subtasks are discussed  (where in the case of contradictory tasks, the lowest priority task would no longer have a usable actions space and could therefore not have a policy). \n\nIn terms of the experiment presentation, it would be very useful if the main paper could contain at least a basic description of the environment, the action space, and in particular the subtasks and corresponding reward functions. For the latter (the reward functions), even the Appendices do not seem to contain more than an rough description of the principles of the reward function of the obstacle subtask. It would be very important for reproducability but also for a better understanding of the reader if the authors were to include the exact reward function for each subtask as well as the \\epsilon thresholds that were used for the experiments presented in the paper (and these should be in the main paper).\n\nAnother slight weakness is that while the paper indicates that the pick of thresholds \\epsilon is difficult, it does not provide any analysis of this. A brief ablation in term of \\epsilon for the obstacle task in the 2D navigation experiment would have been very useful, as would be a brief discussion how such thresholds might be picked and what the tradeoffs of different picks are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3AwBrnuLq1", "reviewer": "Reviewer_SKG7", "summary": "This paper proposes an algorithm called prioritized soft Q-decomposition (PSQD) to solve complex lexicographic multi-objective reinforcement learning (MORL) problems. In the setting, n subtasks are prioritized, and the available policy set is reduced as each subtask is optimized in the predefined order. Instead of explicitly representing the available policy sets, the authors consider restricting action space to satisfy the lexicographic order. For implementation, Soft Q-learning (SQL) is adopted to deal with continuous action space. Numerical results show that PSQD performs better than the other baselines in the considered environments, where there are two subtasks.", "questions": "Please check the weakness part. Additional questions are as follows.\n\n7. In number 2 in Weakness, if one of the candidates is PSQD, do the authors think that the other one is also a valid algorithm? (It looks like candidate 1 does not use order information of 1 > ... > n-1). \n\n8. If SQL is used, PSQD can be extended to discrete action space since the integral is changed to summation in equation 5. Then we may compare PSQD with the previous work of Skalse et al (2022). \nAlso in that discrete action case, do authors think that action set refinement is still valid?\n\n9. Confusing notation in eq (1). J_0, epsilon_0, Pi_0 is not explicitly defined.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors provide mathematical formulations on the main paper to support the soundness of the algorithm.\n- This paper contains a dense appendix for detailed explanations.\n- The authors provide extensive study on previous works.", "weaknesses": "1. Presentation needs to be improved. While the considered setting - lexicographic MORL - is clear, the flow of the algorithm is hard to understand. High-level pictograms can help the readers understand the content.\n\n2. There is no pseudocode, so I am confused about the implementation of the algorithm. As far as I understand, the proposed algorithm is one of the following:\n\nCandidate 1) For subtask 1 to n-1, run parallel SQL in equation (12). Then restrict action space satisfying epsilon-optimalty of subtasks 1 to n-1. In the restricted action set, run SQL for subtask n in equation (12) and recover optimal policy using (13).\n\nCandidate 2) For subtask 1, run SQL in equation (12). Acquire restricted action space A_1.  Run (12) on A_1 and acquire A_2. ... After acquiring A_{n-1}, run (12) and (13) for subtask n.\n\nWhich one is right? Please provide a pseudocode.\n\n3. For clarity of the setting, I want to raise several fundamental discussions regarding the lexicographic setting.\n- When lexicographic MORL setting is considered in practice?\n- What is the clear difference between constrained (MO)RL?\n- Who decides the priority order? Is it valid to assume that we always know the priority order?\n- How do orders of subtasks affect the final performance in experiments? If the order is crucial, discussion on setting order is important.\n- What if there are tasks that are not prioritized (e.g., \"equally\" important, or \"we do not know\")?\n\n4. Experiments deal with only two subtasks. Can the authors show another environment containing more than two subtasks?\n\n5. For reproducibility, it would be better for the authors to provide anonymous source code.\n\n6. There is confusion in eq. (7). Does Q_i in eq. (7) mean the optimally trained one (i.e., Q_i^*)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NuHQbvzf9r", "reviewer": "Reviewer_rq81", "summary": "The paper addresses the challenges of lexicographic MORL problems by introducing a novel approach called prioritized soft Q-decomposition. This technique leverages the value function of previously learned subtasks to constrain the action space of subsequent subtasks. The experimental results conducted on both simple and complex scenarios substantiate the efficacy of this method.", "questions": "1. The paper employs equation 7 to approximate ε-optimality based on equation 1, but the relationship between the two equations is not very clear. Including more theoretical insights could enhance the paper's rigor.\n2. It's worth pondering whether εi should be state-dependent. Is it possible to find a constant εi that precisely represents the task requirements? If not, it might indicate that the action space is overly restricted in some states, hindering exploration in subsequent tasks, or, conversely, that undesired actions cannot be excluded in certain states.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well organized and easy to read.\nThe proposed method, in its simplicity, manages to be effective in tackling the problem.", "weaknesses": "The proposed method appears to be sensitive to the parameter ε, and the manual selection of this parameter is non-trivial. \nAdditionally, the paper falls short in providing a comparative analysis with existing lexicographic MORL algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)