PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm
Authors: Toygun Basaklar
Suat Gumussoy
Umit Ogras
Venue: ICLR 2023 poster (2023)
Keywords: multi-objective reinforcement learning
MORL
DDQN
TD3
HER
continuous control
robotics application

Abstract: Multi-objective reinforcement learning (MORL) approaches have emerged to tackle many real-world problems with multiple conflicting objectives by maximizing a joint objective function weighted by a preference vector. These approaches find fixed customized policies corresponding to preference vectors specified during training. However, the design constraints and objectives typically change dynamically in real-life scenarios. Furthermore, storing a policy for each potential preference is not scalable. Hence, obtaining a set of Pareto front solutions for the entire preference space in a given domain with a single training is critical. To this end, we propose a novel MORL algorithm that trains a single universal network to cover the entire preference space scalable to continuous robotic tasks. The proposed approach, Preference-Driven MORL (PD-MORL), utilizes the preferences as guidance to update the network parameters. It also employs a novel parallelization approach to increase sample efficiency. We show that PD-MORL achieves up to 25% larger hypervolume for challenging continuous control tasks and uses an order of magnitude fewer trainable parameters compared to prior approaches.

Published as a conference paper at ICLR 2023
PD-MORL: PREFERENCE-DRIVEN MULTI-OBIECTIVE REINFORCEMENT LEARNING ALGORITHM
Toygun Basaklar UW-Madison
Suat Gumussoy
Umit Y. Ogras UW-Madison
Siemens Corporate Technology
Madison, WI 53706 basakla @ sc edu
Princeton, NJ 08540 suat. gumussoy@ e emens. .com
Madison, WI 53706
uogras@wisc: edu
ABSTRACT
Multi-objective reinforcement learning (MORL) approaches have emerged tackle many real-world problems with multiple conflicting objectives by maximizing a joint objective function weighted by a preference vector. These approaches find fixed customized policies corresponding to preference vectors specified during training. However, the design constraints and objectives typically change dy- namically in real-life scenarios. Furthermore, storing policy for each potential preference is not scalable. Hence, obtaining a set of Pareto front solutions for the entire preference space in given domain with single training is critical. To this end, we propose a novel MORL algorithm that trains a single universal network to cover the entire preference space scalable continuous robotic tasks. The pro- posed approach, Preference-Driven MORL (PD-MORL), utilizes the preferences as guidance update the network parameters. also employs novel parallelization approach to increase sample efficiency. We show that PD-MORL achieves up to 25% larger hypervolume for challenging continuous control tasks and uses an order of magnitude fewer trainable parameters compared to prior approaches.
INTRODUCTION
Reinforcement learning (RL) has emerged as a promising approach solve various challenging problems including board/video games (Silver et al. 2016 Mnih et al. 2015 Shao et al.) (019). robotics (Nguyen & La! 2019), smart systems (Gupta et 2020; Yu al.) 2021). and chip de- sign/placement (Zheng & Louri 2019 Mirhoseini et al.) (020). The main objective in standard RI. setting is to obtain a policy that maximizes single cumulative reward by interacting with the envi- ronment. However, many real-world problems involve multiple, possibly conflicting, objectives. For example, robotics tasks should maximize speed while minimizing energy consumption. In contrast to single-objective environments, performance is measured using multiple objectives. Consequently, there are multiple Pareto-optimal solutions as a function the preference between objectives (Navon et al. 2020). Multi-objective reinforcement learning (MORL) approaches (Hayes et al. 2022) ) have emerged to tackle these problems by maximizing a vector of rewards depending on the preferences.
Existing approaches for multi-objective optimization generally transform the multidimensional objec- tive space into a single dimension by statically assigning weights (preferences) to each objective (Liu al. 2014). Then, they use standard RL algorithms to obtain a policy optimized for the given preferences. These approaches suffer when the objectives have widely varying magnitudes since setting the preference weights requires application domain expertise. More importantly, they can find only single solution for given set of goals and constraints. Thus, they need to repeat the training progress when the constraints or goals change. However, repetitive retraining is impractical since the constraints and design can change frequently depending on the application domain. Therefore, obtaining a set of Pareto front solutions that covers the entire preference space with single training is critical (Xu et al. 2020 Yang al. 2019 Abdolmaleki et al. 2020).
This paper presents a novel multi-objective reinforcement learning algorithm using a single policy network that covers the entire preference space scalable to continuous robotic tasks. At its core, it uses a multi objetivo version Q-Learning, where we approximate the Q-values with a neural network. This network takes the states and preferences inputs during training. Making the preferences input parameters allows the trained model to produce the optimal policy for any user-specified preference at run-time. Since the user-specified preferences effectively drive the policy decisions, it is called

---

Published as a conference paper at ICLR 2023
preference-driven (PD) MORL. For each episode during training, we randomly sample preference vector (w Ω: Σ Wi = 1) from uniform distribution. Since the number of collected transitions by interacting with the environment for some preferences may be underrepresented, we utilize hindsight experience replay buffer (HER) [Andrychowicz et al. 2017). As a key insight, we observe that the preference vectors have similar directional angles to the corresponding vectorized Q-values for a given state. Using the insight, we utilize the cosine similarity between the preference vector and vectorized Q-values in the Bellman's optimality operator to guide the training. However, not every Pareto front perfectly aligned with the preference vectors. To mitigate this adverse effect, we fit multi-dimensional interpolator to project the original preference vectors (w E Q2) to normalized solution space to align preferences with the multi-objectio solutions. The projected preference vectors are used in our novel preference-drive Additionally, to increase the sample efficiency of the algorithm, we divide the preference space into sub-spaces and assign child process to these sub-spaces. Each child process is responsible for its own preference sub-space to collect transitions. This parallelization provides efficient exploration during training, assuring that there is no bias towards any preference sub-space.
optimality operator to obtain the target Q-values.
PD-MORL. can be employed in any off-policy RI. algorithm. We develop a multi-objective version of the double deep Q-network algorithm with hindsight experience replay buffer (MO-DDQN- HER) (Schaul et al. 2013) for problems with discrete action spaces and evaluate PD- MOR''s'
performance on two commonly used MORL benchmarks: Deep Sea Treasure (Hayes et al. 2022 and Fruit Tree Navigation Task (Yang et al.] 2019). We specifically choose these two benchmarks to make a fair comparison with the prior approach (Yang et 2019) that also aims to achieve a unified policy network. Additionally, we develop a multi-objective version of the Twin Delayed Deep Deterministic policy gradient algorithm with hindsight experience replay buffer MO-TD3-HER) (Fujimoto et al., 2018a) for problems with continuous action spaces. Using MO-TD3-HER, we evaluate PD-MORL on the multi-objective continuous control tasks such as MO- 2o--Waler2d-v2, MO- 10-HarCCheetah-v22 MO-Ant-v2, MO-Swimmer-v2 MO-Hopper-v2 that are presented by Xu et al. (020).. With the combination of the use of the cosine similarity term, the HER, and the parallel exploration, PD-MORL achieves up to 78% larger hypervolume for simple benchmarks and 25% larger hypervolume for continuous control tasks and uses an order of magnitude fewer trainable parameters compared to prior approaches while achieving broad and dense Pareto front solutions. We emphasize that it achieves these results with single policy network.
RELATED WORK
Existing MORL approaches can be classified as single-policy, multi-policy, and meta-policy ap- proaches. The main difference among them is the number of policies learned during training. Single-policy approaches transform a multi-objective problem into a single-objective problem by combining the rewards into single scalar reward using scalarization function. Then, they use stan- dard RL approaches to maximize the scalar reward (Roijers et al. 2013). Most of the previous studies find the optimal policy for given preference between the objectives using scalarization (Van Moffaert ct al. 2013). Additionally, recent work takes an orthogonal approach and encodes preferences as constraints instead of scalarization (Abdolmaleki et al. 2020). These approaches have two primary drawbacks: they require domain-specific knowledge, and preferences must be set beforehand.
Multi-policy approaches aim obtain a set of policies that approximates the Pareto front of optimal solutions. The most widely used approach is to repeatedly perform a single-policy algorithm over various preferences (Roijers et al. 2014]] Mossalam et al.) 2016 Zuluaga et al. 2016). However, this approach suffers from a large number of objectives and dense Pareto solutions for complex control problems. In contrast, Pirotta et al. (2015) suggest which assumes policies to be sampled from manifold. This manifold is defined as a parametric distribution over the policy parameter space. Their approach updates the manifold according to an indicator function such that sample policies yield an improved Pareto front. Parisi et al. (2017) extend this method with hypervolume and non-dominance count indicator functions using importance sampling to increase the sample efficiency of the algorithm. However, the number of parameters to model the manifold grows quadratically (Chen et al.) 2019) as the number of policy parameters increases. Therefore, these approaches not scalable for complex problems such as continuous control robotics tasks to achieve a dense Pareto front where deep neural networks with at least thousands of parameters needed.
munifold-based policy search MORL approach
2

---

Published as a conference paper at ICLR 2023
Abels et al. (2019) and Yang et al. (2019) use multi-objective Q-learning approach that simultane- ously learns a set of policies over multiple preferences. These studies use a single network that takes preferences inputs and uses vectorized value function updates in contrast standard scalarized value function updates. An orthogonal approach by by(Chen et al. (2019) is the meta-policy approach that frames MORL as a meta-learning problem using task distribution given by distribution over the preference space. The authors first train meta-policy to approximate the Pareto front implicitly, Then, they obtain the Pareto optimal solution of given preference by only fine-tuning the meta- policy with few gradient updates. A more recent study proposes an efficient evolutionary learning algorithm update population of policies simultaneously in each run to improve the approximation of the Pareto optimal solutions (Xu et al. 2020). All non-dominated policies for each generation of policies are stored as an external Pareto front archive during the training. Finally, outputs this Pareto front archive approximated Pareto front solutions. However, this approach obtains a policy network for each solution on the Pareto front. Moreover, there is no specific knowledge of correspondence between preference vectors and solutions in this Pareto front.
Distinct from the prior work, PD-MORL utilizes the relation between the Q-values and the preferences and proposes novel update rule, including the cosine similarity between the Q-values and preferences. Additionally, PD-MORL increases its sample efficiency by using a novel parallelization approach with HER and provides efficient exploration. The combination of the cosine similarity term, the HER, and the parallel exploration achieve up to 25% larger hypervolume for challenging continuous control tasks compared prior approaches using an order of magnitude fewer trainable parameters.
BACKGROUND
MORL requires learning several tasks with different rewards simultaneously. Each objective has an associated reward signal, transforming the reward from a scalar to a vector, r = [r1, 72, .2., rr]", where I, the number of objectives. A multi-objective problem can be formulated as a multi-objective Markov decision process (MOMDP) defined by the tuple (S, P. Q, fw), where S, A. P(s'|s,, a), r, and S2 represents state space, action space, transition distribution, reward vector, and preference space, respectively. The function (r) wt r yields scalarized reward using preference of w E Ω. If ws is taken as fixed vector, the MOMDP boils down to a standard MDP, which can be solved using standard RL techniques. However, if we consider all possible returns from MOMDP and all possible preferences in Ω, a set of non-dominated policies called the Pareto front can be obtained. A policy æ is Pareto optimal if there is other policy n' that improves its expected return for an objective without degrading the expected return of any other objective. For complex problems, such as continuous control tasks, obtaining an optimal Pareto front using an RL setting is an NP-hard problem (Xu et al. 2020). Hence, the main goal of the MORL algorithms is to obtain an approximation of the Pareto front. The quality of the approximated Pareto front typically measured by two metrics: (i) hypervolume and (ii) sparsity (Hayes et al. 2022).
Definition (Hypervolume Indicator). Let P be a Pareto front approximation in an L- dimensional objective space and contains N so- lutions. Let ro TR. L be the reference point. Then, the hypervolume indicator is defined as:
Pureto front
Reference Point
do
Objective
Objectve 2
dg
(P) = ((H(P,
(1)
ds
where H(P,ro) = {z E R 31 i
P.) with P. being the 7th solution in P and < is the relation operator of multi-objective dominance. A(-) denotes the Lebesgue measure with ((H(P, ro)) = Jzc 1 (z)dz and 1 (IP()))) being the char- acteristic function of (P,)ro).
N ru
Objective
Objective
Figure 1: (a) The hypervolume between a refer- ence point and solutions in Pareto front is shown with the shaded area. (b) The sparsity is the aver- age square distance (d,, i = (0....,3) 3)) between consecutive solutions in the Pareto front.
Definition 2 (Sparsity). For the same P, sparsity is defined as:
N-1
Sp(P) N
1
ΣΣ
P.
P2+2,)²
(2)
j=1
s=1
where P; is the ith solution in P and P.j is the sorted set for the jth objective.
3

---

Published as a conference paper at ICLR 2023
Improvement in any objective manifests increase in the hypervolume, illustrated in Figure However, the hypervolume alone is insufficient to assess the quality of the Pareto front. For instance. hypervolume may increase due to increase in only one of the objectives which indicates that the algorithm does not improve other objectives. The sparsity metric is introduced to assess whether the Pareto front solutions are dense sparse. A larger hypervolume indicates that front approximation is achieved. Lower sparsity values imply that a dense set of solutions is achieved.
desired Pareto
4 PD-MORL: PREFERENCE DRIVEN MORL ALGORITHM
This section introduces the proposed preference-driven MORL algorithm. We first provide theoreti- cal analysis of PD-MORL based on the multi-objective version of Q-learning (Watkins & Dayan 1992) by following the framework provided by Yang et al. (2019). The proofs for this theoretical analysis are available in Appendix A We then introduce multi-objective double deep Q-networks with preference-driven optimality operator for problems with discrete action space. Finally, we extend the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm (Fujimoto et al.) (0188) to a multi-objective version utilizing our proposed approach for problems with continuous action space.
4.1 THEORETICAL ANALYSIS OF PD-MORL
In standard Q-learning, the value space is defined as Q E RSXA, containing all bounded functions Q(s, a) which are the estimates of the total expected rewards when the agent at state s, taking action a. We extend it to a multi-objective value space by defining the value space as Q E IR T containing all bounded functions Q(s, a, w) which are the estimates of expected total rewards under preference w E 22y0 El Wi = We then define a metric in this value space as:
sup d(Q, Q') aEWEEE
I (F((QQ(s,, a,) - Q'(s, a, w))l
(3)
This metric gives the distance between Q and Q' as the supremum norm of the scalarized distance between these two vectors where the identity of indiscernables (d( Q, Q') 0 <> Q = Q') does not hold and thus, makes metric a pseudo-metric. Further details for the axioms of this pseudo-metric are given in Appendix A In the following Theorem, this metric is complete since the limit of the sequence of operators is required to be in this space.
show that the metric space equipped with
Theorem 1 (Multi-objective Metric Space (Q. d) is Complete). The metric space (Q, d) is complete and every Cauchy sequence (s,s, a, w) is convergent in metric space (Q, V E S,A,
Given a policy x and sampled transition т, we can define multi-objective Bellman's evaluation operator Tx using the metric space (Q, d) as:
(s, a, w) = r(s,a) + -BTT Q(s', ,',w)
where (8', a', w) denotes the next state-action-preference pair and E (0, 1) is the discount factor.
We define preference-driven optimality operator 'I by adding a cosine similarity term between preference vectors and state-action values to the multi-objective Bellman's 's optimality operator as:
(s, = r(s, + +BBoNP(().)) Q(s,, sup (Sa(w, Q(s',d', w)) (w Q(s', a', w w))), () a'EA
(5) where Sw( Q(s', ,)) denotes the cosine similarity between preference vector and Q-value. This term enables our optimality operator to choose actions that align the preferences with the Q-values and maximize the target value, as elaborated in Section 4.2 under the preference alignment subtitle.
Theorem 2 (Multi-objective Bellman's 's Evaluation Operator is Contraction). Let ( (Q, d) be complete metric space (as in Theorem []. Let Q and Q' be any two multi-objective Q-value functions in this space. The multi-objective Bellman's 's evaluation operator is a contraction and d( Q, T=Q') S yd Q, Q') holds for the Lipschitz constant (the discount factor).
Theorem (Preference-drree driven Multi-objective Bellman's 's Optimality Operator is Contraction). Let (Q, d) be a complete metric space. Let Q and Q' be any two multi-objectivee Q-value functions in this space. The preference- driven mbjeivobeective Bellman's 's optimality operator is a contraction and
d(TQ, TQ')= & 7d(QQ Q') holds for the Lipschitz constant 7 (the discount factor).
4

---

Published as a conference paper at ICLR 2023
Theorem 2 and Theorem state that multi-objeccivee evaluation and optimality operators are contrac- tions. They ensure that we can apply our optimality operator in Equation [5 Diteratively to obtain the optimal multi nultti-bbeective value function given by Theorem 4 and Theorem S below, respectively.
Theorem 4 (Preference-driven Multi-objective Optimality Operator Converges to a Fixed-Point). Let (Q, d) be complete metric space which is proved above and let T Q + Q be contraction on Q with modulus ?. Then, TT has a unique fixed point Q* Q such that T(Q) = Q"
Theorem 5 (Optimal Fixed Point of Optimality Operator). Let Q E Q be the optimal multi-objecciive value function, such that it takes multi-objecttiv Q-value corresponding the supremum of expected discounted rewards under a policy π then Q = TQ
These two theorems state that applying our multi-objective optimality operator T iteratively on any multi-objective Q-value function Q converges the optimal Q' under metric d and the Bellman equation holds at the same fixed-point. Hence, we use this optimality operator to obtain target values when optimizing the loss function in Algorithm
4.2 PREFERENCE DRIVEN MO-DDQN-HES AND MO-TD3-HER
The main objective of this work is obtain a single policy network to approximate the Pareto front that covers the entire preference space. For this purpose, we extend double deep Q-network (DDQN) (Van Hasselt al. 2016) to a multi-objective version (MO-DDQN) with the preference- driven optimality operator to obtain a single parameterized function representing the Q E R with parameters 0. This network takes w input and outputs |A] x L Q-values, as described in Algorithm To pull Q towards T(Q), MO-DDQN minimizes the following loss function at each step k:
(0) = -Eaaan ((x-))--_((_a asssa,:w:0))) 21
(6)
where 'D is the experience replay buffer that stores transitions (8, r, 8 w) for every time step, y y= - + (upp/Ee Q(s', a', w)) -wo" Q(s', ''',ad a', w))); 0*) denotes the preference-driven target value which is obtained using target Q-network's 's parameters 0'. Here, Se(w, Q(s', a' w)) denotes the cosine similarity between the preference vector and the Q-values. Without the cosine similarity term, the supremum operator yields the action that only maximizes w Q(''''', wd) This may pull the target Q-values in the wrong direction, especially when the scales of the objectives are in different orders of magnitude. For instance, let us assume a multi-objectivve problem with percent energy efficiency (E [0, 1) being the first objective and the latency (E [1, 10) ) in milliseconds being the second objective. Let us also assume that the preference vector (w {0.9, 0.13) favors the energy efficiency objective and there are two separate actions (a1, a2) that yields Q-values of Q = {0.9, 1}, Q2 = {0.1, 10) respectively. The supremum operator chooses action, 02 since " TTQ2- = 1.09 is higher than w Q2 0.911. Since the scales of these two objectives are in different orders of magnitudes, sup,'e e A (w Q(s', ('s',a,,w) always chooses the action that favors the latency objective. This behavior negatively affects the training since the target Q-values are pulled in the wrong direction. On the contrary, our novel cosine similarity term enables the optimality operator to choose actions that align the preferences with the Q-values and maximize the target value the same time. The supremum operator now chooses action, a1 since S. wn = 0.75 x 0.91 = 0.68 0.12 x 1.09 0.13. With this addition. the algorithm disregards the large wn "Ts(''s a', w) where the preference vector and the Q-values are not aligned since the Se(w,, Q(s', a', ,w)) takes small values.
higher than S. wTQQ²
For each episode during training, we randomly sample a preference vector (w E Ω: El Wi = 1) from a uniform distribution. For example, assume the randomly sampled preference vector (w) favors energy efficiency rather than speed in multi-objective continuous control tasks, Increasing energy effi- ciency (i.e., lower speed) will increase the number of transitions before reaching terminal condition since it decreases the risk of reaching a terminal state (e.g., fall). Hence, transitions that favor speed instead of energy efficiency may be underrepresented in the experience replay buffer in this example. This behavior may create a bias towards overrepresented preferences, and the network cannot learn to cover the entire preference space. To overcome this issue, we employ hindsight experience replay buffer (HER) (Andrychowicz et al. 2017), where every transition is also stored with N., randomly sampled preferences (wo E 8. E
2/=0 w/ 1) different than the original preference of the transition.
5

---

Published as a conference paper at ICLR 2023
To increase the sample efficiency of
Algorithm 1: Preference Driven MO-DDQN -O--DOON-HER
our algorithm, we also divide the pref-
erence space into Cp sub-spaces ($2), where Cp denotes the number of child processes. Each child process is re- sponsible for its own preference sub- space. The agent, hence the network, is shared among child processes and the main process. In each child process, for each episode, we randomly sample a preference vector from child's prefer- ence sub-space (w E Ω Σl=o Wi 1). These Cp child processes run in par- allel to collect transitions. These tran- sitions are stored inside the HER in the main process. Network architectures and implementation details are avail- able Appendix B This parallelization provides efficient exploration during training.
Input: Minibatch size Nm. Number of time steps Discount factor ", Target network update coefficient
Multi-dimensional interpolator [(w). Initialize: Replay buller DD Current Qa and larget network Qr Qu with parameters and D'
for
0: N do
// Child Process
Initialize f = U and done False.
Reset the environment to randomly initialized stare ep. Sample preference vector from the subspace while done = False do
Observe state and select action at e-grendily:
W.P.
maxuue a ( R, 0), 2.
Observe s', and done.
Transfer (8c, Fi, w. done) to main process.
// Main Process Sample Nu preferences w'
for = 1: N..
L Store transition (#1, az, re. wej, done)
Sample N.. transitions from D.
up f I(w)
r QQs' "llo',uupu/ca (So Q(s', w))
(w Q('''',', w));0")
Preference aligmment: A solution in the Pareto front may not align perfectly with its preference vector. This may result in a bias in our updating scheme due to the cosine similarity term we introduced. To mitigate this adverse effect, fit a multi- dimensional interpolator to project the original preference vectors (w E Q2) to normalized solution space to align preferences with the multi-objective solutions. We identify the key preferences and obtain solutions for each of these preferences. We set key preference points as w, = = i, W, =
(x Eu w [(v Q(e, a, w: 8)
Update O2 by applying SGD to L3 (0).
Upuate target network purameters 0g < 70, + (1 +)0g
0: j * i i,j E (0..... [.} where hth element corresponds to the objective we try to maximize. A preference vector of w = ...... )) for a two-dimensional objective space, the key preference set becomes {[1, 0j. [0.5, 0.5), [0, 1]}- We first obtain solutions for each preference in this key preference set by training the agent with a fixed preference vector and without using HER (see Figure 2(a)). We then use these solutions to obtain normalized solution space, as shown in Figure b). Here, normalization is the process of obtaining unit vectors for these solutions. For example, the normalized vector for solution f is described as f = - Then, we use the normalized solution space and key preference point to fit a multi-dimensionn normalized solution space. As a result, we obtain projected preference vectors (wp) as illustrated in Figure c). These projected preference vectors are incorporated in the cosine similarity term of the preference-driven optimality operator. This practical modification extends to theoretical results as the proof for Theorem 3 is also valid for wp. The interpolator is also updated during training as PD-MORL may find new non-dominated solutions for identified key preferences.
is also added to this key preference set. For example,
interpolator I(w) to project the original preference vectors (w Q)) to the
Extension to continuous action space: Finally, we extend the TD3 algorithm to a multi-objectio version using PD-MORL for problems with continuous action space. In this case, the target values are no longer computed using the optimality operator. Instead, the actions for the target Q-value are
Energy Efficiency
(0. 1)
140
120
*
5..5.0.5
0.2 0.4 0.6 0.8
0.2 04 0.6 0.8
100
80
Normalized Solutions
Normalized Solutions
20
[1.0)
Preference Vectors
Preference Vectors
50
100
150
200
250
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Forward Speed
Figure 2: Overview of the interpolation scheme for the MO-Swimmer-v2 problem. (a) Obtained solutions for key preference points. (b) Normalized solution space and corresponding preference vectors. (c) Interpolated preference space.

---

Published as a conference paper at ICLR 2023
determined by the target actor network. The actor network is updated to obtain policy that maximizes the Q-values generated by the critic network. This update rule eventually boils down to obtaining a policy that maximizes the expected discounted rewards since, by definition, this expectation is the Q-value itself. To incorporate the relation between preference vectors and Q-values for the multi- objective version of TD3, include directional angle term g(w, Q(8, 0. w; to both actor's and
critic's loss function. This directional angle term g(w, Q(s, a, w; 0)) = cos- ~ ((s.a.w:)))
||
denotes the angle between the preference vector (w) and multi-objective Q-value Q( a w; and provides alignment between preferences and the Q-values. Algorithm, network architectures, and implementation details are available in Appendix 1.1.5 and B.3
EXPERIMENTS
This section extensively evaluates the proposed PD-MORL technique using commonly used MORL benchmarks with discrete state-action spaces (Section [5.T) and complex MORL environments with continuous state-action spaces (Section 5.2). Detailed descriptions of these benchmarks are provided in Appendix [B.2] Our results are compared to state-of-the -ttthee-rttt techniques in terms of hypervolume,
sparsity, and the Pareto front plots.
5.1 MORL BENCHMARKS WITH DISCRETE STATE-ACTION SPACES
This section illustrates the efficacy of PD-MORL in obtaining a single universal network that covers the entire preference space in discrete state-action environments. We the following commonly used benchmarks to (i) make a fair comparison with prior work (Yang et al. 2019), and (ii) show that PD-MORL scales more than two objectives:
Deep Sea Treasure (DST) has discrete state (S C N2) and action spaces (A4 C N It has two competing objectives: time penalty and treasure value. The treasure values increase as their distance from the starting point 8o = (0, 0) increases. Agent gets a -1 time penalty for every time step.
Fruit Tree Navigation (FTN) has discrete state (S C N2) and action spaces (A N²) with six objectives: different nutrition facts of the fruits on the tree: (Protein, Carbs, Fats, Vitamins, Minerals, Water]. The goal of the agent to find a path on the tree collect the fruit that maximizes the nutrition facts for given preference.
Figure iillstrats the policies achieved by PD-MORL for three preference vectors capture corner cases. Although our policy is not specific to any preference, the submarine achieves optimal treasure values for the given preference. For example, finds the best trade-off within the shortest time one step) when we only value the time penalty (w = {0. 1)). As the importance of the treasure value increases (w = =0.5,0055) w (J). it spends optimal amount time (8 steps and 19 steps, respectively) to find a deeper treasure. Figure Arovides further insights into the progression of the policies found by PD-MORL. Let the first objective be the treasure value and the second objective be the time penalty in this figure. At the beginning of the training, each child process starts collecting transitions within their own preference sub-space Ω. Since the agent acts e-greedily at early phases and cannot reach the higher treasure values due to terminal condition (time limit), we observe that most of the solutions are stuck in the first-subspace (Ω₂) The transitions collected from child processes are stored in the hindsight experience replay buffer with different preferences other than their original preferences. This buffer enables extra exploration for the agent and leads the agent to different sub-spaces, illustrated in the middle figure. As the training progresses, HER and novel optimality operator with the angle term guide the search to expand the Pareto front and cover the entire preference space with a single network.
Among existing algorithms that learns a unified policy YYang et al. 2019 Abels et al. 2019), the Envelope algorithm (Yang et al., 2019) the superior and a more recent ap- proach. Hence, we compare and evaluate PD-MORL and the Envelope algorithm on DST and FTN by obtaining set of preference vectors that covers the entire preference space.
5,0.5)
(1.0)
11.5
15.3
124
PAY
Figure 3: Optimal policies obtained by the trained agent for three corner preferences.
7

---

Published as a conference paper at ICLR 2023
At the beginning
Early phuses
Comsequent progression
PD-MORL achieves both 6.1% larger hypervolume and 56% lower sparsity than the Envelope algorithm for DST problem, as shown in Table Simi- larly, PD-MORL generates up to 78% larger hypervolume than the Envelope algorithm for FTN. Since FTN is a bi- nary tree, the distance between Pareto solutions is the same, and hence, the sparsity metric is the same between the two approaches. Our experiments show that the improvement of PD-MORL compared to the Envelope algorithm increases with the problem complexity (i.e., increasing tree depth), as shown in Table [] PD-MORL provides efficient and robust exploration using HER and our novel optimality operator together. Further comparisons against the Envelope algorithm (Yang al. 2 provided in Appendix B.4
f
Objective 2
Objechin O
A2
Objective 2
HER
Bg
Angle
Term
Objective 1
Objective 1
Objective
Figure 4: Policy progression during training.
Table 1: Comparison of our approach with prior work Yang et al. 2019) using two simple MORL benchmarks in terms of hypervolume and sparsity metrics. Reference point for hypervolume calcula- tion is set to (0,-19) and (0,0) for DST and FTN, respectively.
Deep Sea Treasure
Fruit Tree Navigation (d=6)
Fruit Tree Navigation (d=7)
Hypervolume
Sparsity
Hypervolume
Sparsity
Hypervolume
Sparsity
Envelope Yang et al. 2019
227.89
2.62
8427.51 9299.15
N/A
6395.27
N/A
PD-MORL.
241.73
1.14
N/A
11419.58
N/A
5.2 MORL BENCHMARKS WITH CONTINUOUS CONTROL TASKS
This section evaluates PD-MORL popular multi- objective continuous control tasks based on MuJoCo physics engine (Xu et al. 2020 Todorov al., 2012) The environment details of these benchmarks are given in Table 2 The action and state spaces take continuous values and consist of multiple dimensions, making these benchmarks challenging to solve. The goal is tune the amount of torque applied on the hinges/rotors for given preference w while satisfying multiple objectives (e.g., forward speed vs. energy efficiency, forward speed vs. jumping height, etc.). We compare the performance of PD-MORL against two sfate-of-the- art approaches (Xu et al. 2020, Chen et al., 2019) that use these continuous control benchmarks. We emphasize that these approaches must learn different policy network for every solution in the Pareto front. In contrast, PD-MORL learns a single universal network that covers the entire preference space. The Envelope algorithm (Yang et al. 2019) is not included in these comparisons since it was not evaluated with continuous action spaces, and PD-MORL already outperforms it on simpler problems. Training details are reported in Appendix B.3
Table 2: Environment details of continu-
ous control benchmarks.
State Space
Action Space
M0-Walker2dwv2
R17
A C
g
h2
S
R17
A C R
MO-Ant-v2
8
R27
A C R
MO-Swimmer-v
S
C R
A C X²
MO-Hopper-12
R
A C x²
We first compare PD-MORL to prior work using hypervolume and sparsity metrics. Since META (Chen et al. 2019) and PG-MORI. (Xu al. 2020 report the average of six runs, also ran each benchmark six times with PD-MORL. The average metrics are reported in Table 3 while the standard deviations and results of individual runs are given in Appendix AppendixBB.4Tablle Table
The desired Pareto front approximation should have high hypervolume and low sparsity metrics. PD-MORL outperforms the current state-of-the-art techniques both in terms of hypervolume and sparsity on every benchmark except MO-Hopper-v2, as summarized in Table 3 We emphasize that our PD-MORL technique trains only a single network, while the other methods use customized policies network for each Pareto point. For example, we achieve 12% higher hypervolume and 25% better sparsity than the most competitive prior work PG-MORI (Xu et af.) 2020) on the Walker environment. They reported 263 Pareto front solutions for this environment. This corresponds to, in total, 2.8 x 106 trainable parameters since they have different policy network for each solution. In contrast, PD-MOR achieves better comparable result using only 3.4 x 10a trainable parameters. This may enable the deployment of PD-MORL in a real-life scenario where there are dynamic changes in the design constraints and goals. We also note that PG-MORL achieves better sparsity metric for H.FfhhhhChhttrr-12 and MO-Ant-v2 environments. However, PG-MORL also achieves a smaller hypervolume compared to PD-MORI. This suggests that determine the quality of Pareto front,
8

---

Published as a conference paper at ICLR 2023
Table 3: Performance comparison of the proposed approach and state-of-the-art algorithms on the continuous control benchmarks in terms of hypervolume and sparsity metries. Reference point for hypervolume calculation is set to (0,0) point. HV* Hypervolume
M0-Walker2--2
MO-HalIC hewbsh-22
MO-Ani-v2
MO-Swiimme--2
MO-Hopper-2
HV
Sparsity
HV
Sparsity
HY
Sparsity
HV
Sparsily
HV
Spansity
PG-MORL Nuret al 5020
4.62 x 10
0.54 x 10t
0.77 x 106
0.44 x 10
6.36 x 30F
0.57 x 10
2,57 x 101
9.
2.02 x 10
0.5 101
META Chereral ar 20ny
2.10 x 10a
2,10
x 10
5.18
x KP
2.13x
10"
2.40 x 10
1.50
10
1.23 x 10
264
122 x 10
4.81 10
PD (Ours)
5.41x 10"
0.03 x 10
5.89 x 10"
0.49 102
7.48 x 10"
0.78 10*
3.31 x 10
5.7
1 x 10
0.3 10
these metrics rather limited and should be further investigated by the existing literature. Therefore,
we also plot Pareto front plots for all algorithms in Figure 5] (c) to have a better understanding of the quality of the Pareto front. Figure F shows that the proposed PD-MORL technique achieves a broader and denser Pareto front than the state-of-the-ar approaches despite using a single unified network. PD-MORL mostly relies on the interpolation procedure before the actual training. The key solutions obtained during this period determine the initial convergence of the algorithm. However, for MO-Hopper--2 the obtained key solutions for preference vectors {1, 0}, {0.5, 0.5), (0. 1 are {1778, 4971 J. 1227,, 1310], (3533, 3165] respectively. These key solutions are not a good repre- sentative of the Pareto front shown in Figure ((e). To have representative key solutions is a limitation of PD-MORL: however, it can be solved with hyperparameter tuning.
Finally, Figure 51)) plots the progression of the hypervolume of the Pareto front for the environ- ments with similar scales. The plots show that PD-MORL effectively pushes the hypervolume by discovering new Pareto solutions with the help of efficient and robust exploration. We ensure this by using HER, our novel directional angle term, and dividing the preference space into sub-spaces to collect transitions in parallel. The progression of the Pareto front, sparsity, and hypervolume of all benchmarks are provided in Appendix B.4
3000
Enery Elfictency
MO-Wallke2d-vz
Energy Elficiency
MO-HallChectah--z
MO-Ant-v2
Y-axis Speed
2000
2000
2000
PG-MORL [11]
META [24]
1000
1000
PD-MORL
0
1000
2000
3000
1000
2000
3000
1000
2000
3000
Forward Spred
Forward Speed
X-axis Speed
x10*
MO-Swimmer-v2
M0-Hupper.v2
Energy Efficiency
Jumping Height
Hypervolume
#000
100
2000
MO-Aut--2 MO-HaltCheetahss2
MO-Walker2dw2
0
10
2000
4000
Forward Speed
Forward Speed
Time Steps
Figure 5: Pareto front comparison for continuous control tasks (a)-(e). Results for META and PG-MORL are obtained from (Xu et al. 2020) and code base of PG-MORL. (f) Progression of hypervolume of the Pareto front. Results for environments that have similar scales are given.
6 CONCLUSIONS AND LIMITATIONS
Many real-life applications have multiple conflicting objectives that can change dynamically. Existing MORL techniques either learn different policy for each preference or fail to extend to complex continuous state-action benchmarks. This paper presented novel MORL technique that addresses both of these limitations. It trains a universal network to cover the entire preference space using the preferences as guidance to update the network, hence the name preference-driven MORL (PD- MORL). We evaluated PD-MORL using both discrete and continuous benchmarks and demonstrated that PD-MORL outperforms the -tate-of-the-art approaches.
PD-MORL aims to obtain generalized network that covers the entire preference space. It updates its network parameters by sampling transitions from HER. Since these transitions can represent anywhere in the preference space, the updates should be conservative. Therefore, it may fail to obtain a dense Pareto front on problems with massive action, state, and preference spaces which is the primary concern of any MORL. algorithm.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by NSF CAREER award CNS-1651624, and DARPA Young Faculty Award (YFA) Grant D14AP00068.
ETHICS STATEMENT
This work introduces novel multi-objective reinforcement learning approach scalable continuous control tasks. While reinforcement learning has several ethical consequences, especially in settings where the automated decision affects humans directly, our algorithmic approach does not address these concerns specifically. Therefore, we do not expect any ethical issues raised by our work. Instead, helps to alleviate these concerns since considering multiple objectives in societal problems enables various trade-offs.
REPRODUCIBILITY STATEMENT
Proofs of all our theoretical results given in Appendix A Appendix B provides details of the implementation of our approach and additional experimental results. The source code attached with the rest of the supplementary material, providing complete description of the multi-objective RL environments and instructions on reproducing our experiments.
REFERENCES
Abbas Abdolmaleki, Sandy Huang. Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In International Conference Machine Learning, pp. 11 11-22. PMLR, 2020.
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights multi-objective deep reinforcement learning. In International Conference on Machine Learning, pp. 11-20.
PMLR, 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Xi Chen, Ghadirzadeh, Märten Björkman, and Patric Jensfelt. Meta-learning for multi-ohjecctive reinforce- ment lcarning. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp- 977-983. IEBE, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation eror in actor-critic methods. In International Conference on Machine Learning, pp. 1587-155 1596. PMLR, 2018a.
Scntt Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error actor-critic methods. In International conference on machine learning, pp- 1587-1596. PMIRR 2018h. 8h.
Saurabh Gupta, Siddhant Bhambri, Karan Dhingra, Arun Balaji Buduru, and Ponnurangam Kumaraguru. Multi-objective reinforcement learning based approach for user-centric power optimization in smart home environments. 2020 IEEE International Conference on Smart Data Services (SMDS), pp. 89-96 96. IEEE, 2020.
Conor F Hayes, Roxana Rádulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond. Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems. 36(1): 1-59, 2022.
Chunming Liu, Xu, and Dewen Hu. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385-395, 2014.
Azalia Mithoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang. Ebrahim Songhori, Shen Wang. Young-Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bac, et al. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004. 10746, 2020.
10

---

Published as a conference paper at ICLR 2023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 51 518(7540):529- 533, 2015.
Hossam Mossalam, Yannis Assael, Diederik Rojjers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arXiv preprint /oI.:::6600020000000 2016.
Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypemetworks. arXiv preprint wXXi:2010.04104. 2020.
Nguyen and Hung La. Review deep reinforcement learning robot manipulation. 2019 Third IEEE International Conference on Robotic Computing (IRC), pp. 590-595. IEEE, 2019.
Simone Parisi, Matteo Pirotta, and Jan Peters. Manifold-based multi-objective policy search with sample reuse. Neurocomputing, 263:3-11 14,-14422.. 2017.
Matteo Pirotta, Simone Parisi, and Marcello Restelli. Multi-objective reinforcement learning with continuous pareto frontier approximation. In Twenty-ninth AAAI conference on artificial intelligence, 2015.
Diederik M Roijers, Peter Vamplew. Shimon Whiteson, and Richard Dazeley. survey multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
Diederik Roijers, Shimon Whiteson, Frans Oliehoek, et al. Linear support multi-objective coordination graphs. In AAMAS'144 14: Proceedings of the 2014 International Conference on Autonomous Agents & MMultiaeet Systems, pp. 1297-1304. 2014.
Tom Schaul, John Quan. Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 1.Xiv:5511.05955 2015.
Kun Shao, Zhentao Tang, Yuanheng Zhu, Nannan Li, and Dongbin Zhao. A survey of deep reinforcement learning in video games. arXiv preprint a91i: 10944, 2019.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). pp. 191 199, IEEE, 2013.
Pauli Virtanen al. SeiPy 1.0: Fundamental Algorithms for Scientific Computing in Python, 2020. https::/docs: http://wwww...aoeo//////////a//a
wttp:eee/eecpp....eppoo....ooo interpolate. accessed 13 November
RBF htt.://www......ooooooo.....oooa htm
2022.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279-294) 1992.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In International Conference on Machine Learning, pp. 10607-10616 PMLR, 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objectivv reinforce- ment learning and policy adaptation. Advances in Neural Information Processing Systems. 32:44766-14647, 2019.
Liang Yu, Shuqi Qin, Meng Zhang, Chao Shen, Tao Jiang, and Xiaohong Guan. A review of deep reinforcement learning for smart building energy management. IEEE Internet of Things Journal, 2021.
Hao Zheng and Ahmed Louri. An energy-efficient network-on-chip design using reinforcement learning. In Proceedings of the 56th Annual Design Automation Conference 2019, pp. 1-6, 2019.
Marcela Zuluaga, Andreas Krause, and Markus Püschel. e-pal: an active learning approach to the multi-objective optimization problem. The Journal Machine Learning Research. 7(1):3619-365) 2016.
11

---

Published as a conference paper at ICLR 2023
Supplementary Material: PD-MORL: Siferrence-rivenn Multi-Objective Reinforcement Learning Algorithm
A THEORETICAL ANALYSIS OF THE DIRECTIONAL ANGLE GUIDED MORL ALGORITHM
This section introduces a theoretical analysis of the proposed PD-MORL algorithm. We follow the theoretical framework for value-based MORL algorithms proposed by Yang et al. (2019). which is based Banach's 's Fixed-Point Theorem (also known Contraction Mapping Theorem). This theorem states that every contraction on a complete metric space has a unique exed-point. Considering this, we define (i) contraction operators and (ii) a metric space design our value-based MORL algorithm.
A.1 METRIC SPACE
In standard Q-Jearning, the value space is defined as R.SAA, containing all bounded functions Q(s, a) which are the estimates of the total expected rewards when the agent is at state s, taking action We extend it a multi-objective value space by defining the value space as IR ==x containing all bounded functions Q(s, a, w) which are the estimates of expected total rewards under preference E R5
E = 1. We then define metric in this value space as:
sup d (1Q Q') WEW
w T (Q(s, - Q'(s,a,
This metric gives the distance between Q and Q' as the supremum norm of the scalarized distance between these two vectors. Notice that should satisfy the following axioms be considered metric:
Non-negativity: d(Q, Q') 2 0. This axiom holds for metric d. The definition of supremum norm is the largest value of a set of absolute values, and thus, is greater than or equal to zero.
Symmetry: ((Q,Q)) d(Q's Q). Similarly to above axiom, this axiom also holds for d since absolute values are considered in supremum norm.
Triangle Inequality: d(Q, Q') < d(Q, Q") + d(Q", Q"). This axiom holds for metric as shown by the following proof:
I (Q(ssa,ww) - Q"(s)
sup sES.aEA welt
sup
sES.aEd weft
2 w (Q(s,a,,
Q" a, w))l + (Q" a'(s,a,w) w))
2 (Q(s,a,,
Q'(8, ww))
supseS | aw ("(ss,aw)) + supsES (" Went Q'(s,a,w)) the upper bound the set w wT(QQ(s Q'(s,a,w))), Hence,
wen
12

---

Published as a conference paper at ICLR 2023
by definition of supremum norm (least upper bound);
(Q'(((( a, Q')s,,
w I ()(s,a,))- Q" Qs,
sup
sup
*E..EEAA
sE8,nE.A
WES?
WEB
w ("" Q's(s,,,)))))
sup #ES,eEA
WES
Identity of indiscernibles: d(Q. Q') = 0 + Q Q'. This axiom does not hold for metric
d since the dot product of different Q(8,, a, w) functions may result in same value for a specific preference vector
In summary, the metric d is pseudo-metric since the identity of indiscernibles does not hold for it.
In the following Theorem, we showed that the metric space equipped with this metric complete since the limit point of the sequence of operators is required to be in this space.
Theorem 1 (Multi-objective Metric Space (Q. d) is Complete). The metric space (Q, d) is complete and every Cauchy sequence (n, a, w) convergent metric space (Q.d) s, a, E S,A.
Proof. Let Qn (5, a, w) be Cauchy sequence in Q Given e > O, there exist n., m N such that ae ww" (Qn(s, w) - Qm (8, a, w)) < Hence,
|w ()) (m,
<e, Ve,.EEE sup w" (Qn (,,,,,,)) w) <e
WEl
This implies that for each s. w S, S, the sequence of L-dimensional real numbers Qn(saaww) a Cauchy sequence. Since E² is complete. Qn, a, w) is convergent. Let Q(s,a,w) lim,- Qn w). Cauchy sequence in normed space must be bounded. Let there be an M > 0 A,
such that |ww" (Qn (s. a, a,w))l S WEEE w (Q, a, w))l S M, V E Taking n + Do, we find (QQ((ss a.w)) = lim w (Qn a, w)) < M. V E S,
A,w A E Ω. This shows that Q(s, a, w) bounded function and thus Q O For all n N. l
(), lim [= (Q, (s, a, (m, a, w))l S e, Vs, Nsa, A,
m=
and hence V n 2 N, Nu AwEE W (Qn a w) ((s,aww)) < This implies that Q(s, a, w) is the limit of Qu (s, a, w) and proves that Qn a, w) is convergent in Q.
This completes our proof that the metric space (2, d) is complete.
A.2 MULTI-OBJECTIVE BELLMAN'S 'S EVALUATION AND PREFERENCE-DRIVER OPTIMALITY OPERATORS
Given a policy x and sampled transition т, we can define multi-objective Bellman's evaluation operator T- using the metric space (Q, d) as:
(T=Q) a, w) == r(s,a) a) + +TBT Q(5 a',
where (s', a', w) denotes the next state-action -rrtfirn--peeeecceee pair and y E (0, 1) the discount factor.
Theorem 2 (Multi-objective Bellman's Evaluation Operator Contraction). Let (Q, d) be complete metric space (as in Theorem [ Let Q and Q' be any two multi-objectio Q-value functions in this space. The multi-objective Bellman's 's evaluation operator is a contraction and d( Q T Q') S ydd Q, Q') holds for the Lipschitz constant 7 (the discount factor).
Proof. We start by expanding the expression (TaQ,T=Q'):
(IT.QTT.')) = sup
w (wT((TQQ(ss)) (T=Q(s, (s,
sES.aEA went
= sup hw
T ET~ Q(s','"' w)) E(s.
WEB
13

---

Published as a conference paper at ICLR 2023
= =y sup w TeWWWTET-- - wEn
Q'(s',
S y sup wEnl
R TT 1( Q((''''',w) ([[[ & E||- I])
y sup By (P. wesl
- Q'(s',''
S y sup sup (Q('s Q'(s', w()) (Ell || 1 sup |-|)
well deS
d'EA
WEDl
= -Y sup I (Q(s', Q'(s', ,'' w) d(Q Q')
s'ES
a' EAA
WE
d(T, Q, S d Q Q')
To avoid confusion, the last step (Step 7) has only one supremum norm since the outside supremum does not see any variable. The definition of the variables ends with the inside supremum. This completes our proof that multi-objective Bellman's evaluation operator, Tx, is contraction.
We define a preference-driver optimality operator T by adding a cosine similarity term between preference vectors and dtte-action values to the multi-objective Bellman's 'ss optimality operator
(T,) = rs, (B Q(s', sup (Sa(w, Q(5', w)) (wo Q(' a' w) d'EA
where Selw, Q(s', a', w)) denotes the cosine similarity between preference vector and Q-value. sup/ee (S. Q(5', a', w)) (w" Q a', w))) yields the action a' that maximizes the multiplica- tion inside the supremum. This term enables our optimality operator to choose actions that align the preferences with the Q-values and maximize the target value, as elaborated in Section 4.2 under the preference alignment subtitle.
Theorem (Preference-driven Multi-objective Bellman's 's Optimality Operator is Contraction). Let (Q, d) be a complete metric space. Let and Q' be any two mulii-objective Q-value functions in this space. The preference-driver multi-objective Bellman's 's optimality operator is contraction and d(T Q, TQ') S 7d(Q, Q') holds for the Lipschitz constant 7 (the factor..
Proof. We use Se(w, Q(s', a', w)) Se and Sc(w, Q'(s', (() St in the proof for notational simplicity. We start by expanding the expression (TQ,TQQ))
d(TQ, = sup
WT(TE
- -Q''s)))T
aES WEB
hw T EgrP(() Q(s sup (S.) (wT d' w)),
sup SEA WEl
d'EA
T -nw" EE Exi Q'(s', sup (S)) (w Q' (s', d", w)) d'EA
E. Ex-D(iima) W [ (Q(s', sup (S.) (w (w (w]s ((sssdd,w)) w)
=Y. sup
.ES.eEA wElt
d'eA
Q'ss', sup (S)) (w"'Q'ss,d'" w)), w))l d'Ed
E. (P ( sup (S.) (w QQsss ((s,',d, d),w)),
S sup EE..EAA wEl
d'EA
Q'('', sup (S)) (w (WT''( d", ()), ()) d'EA
[B]] < E1 !])
W (Q(6'/ sup (S.) (TQQss, a' d'w w)
57. sup eES. a' d
sup
d'EA
WEll
14

---

Published as a conference paper at ICLR 2023
Q'(s', sup d"Ed
(S") (s", ,)),, (BIl [] sup I-D
w ( (S.) Qw'' d' w)),
y sup d'ESWER
a'ed
Q''s', sup (S')
(w Q'sss, d" w)), (Rearrange supremums)
d'Ed
Let a' be the action that maximizes (Sa) (wr Q(s', , ,', w)) for state s' and preference w, then we have
d(TQ, s
sup SEE
w (Q((s, ('''',d'.w) Q'(s', sup (S)) "w'''s'''"" d". (s))
a"EA
W.l.o.g we assume T Q(''s(sss d', w) w Q'(s', supa
Q'''Q''', w)))
(w)))(w) w) 2 Proof
similar for the other inequality case. Since w Q'''''s'' wd)" we have:
I wi Q'('ss suu,a" E.A (S
((TQ,TQ)
sup
3" (Q(S' Q'ss', = -
SEEA
WEnt
(ITQ.QQ) s Q Q')
This completes our proof that preference-driven multi-objective Bellman's optimality operator, T. is contraction.
Theorem 2 and Theorem 3 state that multi-objective evaluation and optimality operators are contrac- tions. They ensure that we can apply our optimality operator in Equation iteratively to obtain the optimal multi multi-objecttiv value function given by Theorem 4 and Theorem [ below, respectively.
Theorem 4 (Prefenence-driven Multi-objective Optimality Operator Converges to Fixed-Point). Let (2, d) be complete metric space which is proved above and let T Q - Q be contraction on Q with modulus ?). Then, T has unique fixed-point Q" Q such that T(Q) =Q'
Proof. Let Qo Q and define sequence (Qn) where Q++1 = ((Q,),, = 1, 2....
di d(Qn11, Qn) = d(T((x)) T(Q
< yd(Q Qn-1) (d(T(Qu T(Q,.-))
T(Q,=))))
S 7² Qm 1: Qn-2)
:
nd Q1, Qo)
Hence, for m n m,n N by the triangle inequality we have
d(Qm: Qn) S d(Qm, Qm-1) d(Qmm1; Qm-2) d(QQ+1,, Qn)
I (7 m-1 + m-2
+,")d((Q,,)) Qo)
S dl (11 Qo)
Therefore, Qn is Cauchy: for e > 0, let N be large enough that d(1 Qo) < which ensures that n, m >> N + d(Q,.. Qm) < Since (2, d) is complete, Q. converges to Q" E Q. Now, we show that Q" is indeed fixed point since Qn converges Q" and T continuous.
Q" lim Qn lim T(Qn-1) = T( lim Q T(Q")
m-000
R2400
15

---

Published as a conference paper at ICLR 2023
Finally, T cannot have more than one fixed point in (Q, d), since any pair of distinct fixed points Qi and Q would contradict the contraction of
d(T(Qi): T(Q))) d(Qi Q y Qi Q
This completes our proof that multi multi-objective optimality operator converges to Q"
Theorem 5 (Optimal Fixed Point of Optimality Operator), Ler Q E Q be the optimal multi-objective value function, such that it takes multi-ohjective Q-value corresponding the supremum of expected discounted rewards under a policy æ then Q = (QQ").
Proof. We start by defining the optimal Q-function, Q" as:
Q",s, argq sup w BP xell
7rr(s a))
1
From Theorem we observe that lim d(T Q*) = 0 for any Q E Q. This sug-
gests that w T (s,)s, a. w" Q" as,,, w) from the definition of metric d. Expanding the ""(TQQ)) (8, a, w) we have:
T(Q""))), = =)= (ss,a)) B. B. (s. sup (5.(w, Q' ('','a ,))") (w² q' w'))),)))))))
e'EA
Let a' is the action that maximizes the sup d' ( (' d' w)) Q d' w)) for state and preference W. Then, by substituting Q" into T(Q") we have:
T(Q'")(s, ((s,a))))))))))) +7 w B. ((ike))) argq sup wf Ey = momal Σ ((s
rT)ss + w argo sup WF T~(P,x) Σ (rr(
nell
sp-P(-(ssa)) t=0
Now, merge r(s, a) with the summation operation by rearranging the conditions of the expectation operator and obtain:
=w arrq sup w E ET (P) well
"" -s, as
This completes proof that optimal multi-objective value function Q" is the fixed point of the preference-driven multi-objective optimality operator T.
B IMPLEMENTATION-TR..E RESULTS
DETAILS AND ADDITIONAL EXPERIMENTAL
This section provides details the Preference- MO-DDQN-HHE and MO-TD3-HED algo- rithms. expands the explanation of the algorithms with implementation details. Then, it explains the training details for all benchmarks. Furthermore, we provide additional experimental results for discrete MORL and multi-objective continuous control benchmarks.
B.1 PREFGRENCE--DRVVER MO-DDQN-HER AND MO-TD3-HER
B. HINDSIGHT EXPERIENCE REPLAY
We randomly sample preference vector (w E Ω: Σ/= = 1) from a uniform distribution each episode during training. Depending the problem, the number of transitions observed by interacting with the environment for some preferences may be underrepresented. This behavior creates bias towards overrepresented preferences, and the network cannot learn to cover the entire preference
16

---

Published as a conference paper at ICLR 2023
space. Therefore, employ hindsight experience replay buffer (HER) (Andrychowicz et al. 2017 where every transition is also stored with N₂ randomly sampled preferences (w' f Σ w = 1) from a uniform distribution different than the original preference of the transition. Specifically. for each transition (s, a, r, s', w, done), additional No transitions (s, a, r, s w', done) are also stored in the experience replay buffer. This strategy provides efficient exploration and generalizability to the agent. Using HER, the agent learns recover from undesired states and continue to align itself with its original preference.
B. EXPLORATION IN PARALLEL
To further increase the sample efficiency of our algorithm, we equally divide the preference space into Cp sub-spaces (92), where Cp denotes the number of child processes. The number of child processes Gp is set to 10 for all benchmarks. The agent is shared among these child processes and the main process. In each child process, for each episode, we randomly sample a preference vector from child's preference sub-space (w E Ω Σ WP = 1). These Cp child processes run in parallel to collect transitions. After collecting a transition, each child process transfers it to the main process and waits for other child processes. After the main process receives transitions from every child process and stores them using HER, the child processes continue to collect transitions. Since we collect and store Cp transitions at every child-main process loop, networks are also updated Cp times. Parallel exploration and HER together provides efficient exploration.
B.1.3 PREFERENCE ALIGNMENT WITH INTERPOLATION
A solution in the Pareto front may not align perfectly with its preference vector. To mitigate this adverse effect, we fit multi-dimensional interpolator to project the original preference vectors (w (2) normalized solution space to align preferences with the multi-objective solutions. Here, normalization is the process of obtaining unit vectors for these solutions. For example, the normalized vector for a solution I is described as f = f/- We identify the key preferences and obtain solutions for each of these preferences without HER. We set key preference points as w : = i, w, = 0: j + i i.j E (o..... L}} where hth element corresponds to the objective we try to maximize. A preference vector of w = { ...., }} We first obtain solutions for each preference in this key preference set by training the agent with fixed preference vector and without using HER. We use these solutions first to obtain a normalized solution space. Then, we use the normalized solution space and key preference vectors fit multi- dimensional interpolator / (w) to project the original preference vectors (w E ?2) to the normalized solution space. We use radial basis function interpolation with linear kernel this work (Virtanen et al. 2020). As a result, we obtain projected preference vectors (w,) that are incorporated in the cosine similantty term of the pefference-driver optimality operator for the MO-DDQN-HER HER algorithm and in the directional angle term for the MO-TD3-HER algorithm. The interpolator is also updated during training as PD-MORL may find new non-dominated solutions for identified key preferences. To this end, at every episode, we evaluate our agent using the key preference set.
is also added to this key preference set.
1.1.4 DETAILS ON MO-DDQN-HER HER
The main objective of this work is to obtain a single policy network to approximate the Pareto front that covers the entire preference space. For this purpose, we extend double deep Q-network (DDQN) (Van Hasselt et al. 2016) to a multi-objective version (MO-DDQN) with the preference- driven optimality operator. Algorithm describes the training using the proposed MO-DDQN-HER.
We first initialize an empty buffer 2, Q-network and target Q-network with parameters 0 and 0'. The preference space then equally divided into Cp sub-spaces, where we initialize a child process for each sub-space. In each child process, for each episode, we randomly sample a preference vector from child's preference sub-space (w á Σ.-0 Wi environment using c-greedy policy and collect transition (s, s' w. done). This transition is then transferred to the main process. After main process receives transitions from every child process, it stores transitions inside D. For each transition, we also store N,, transitions where a different preference (wo E f Et_ w/ = 1) is sampled. Specifically, for each transition (s, a, r, s', w. done). additional No transitions (s, a, r, s' w², done) are also stored in D. We then sample minibatch of
1). The agent interacts with the
17

---

Published as a conference paper at ICLR 2023
transitions from D to update the network. To calculate the cosine similarity metric, we first project the preferences w to normalized solution space and obtain projected preferences wp. MO-DDON- -ER minimizes the following loss function at each step k:
L = E(s (aaoi(((xQ(ss Q( Q(3, w: w())),
(10)
where y = r 7Q6s, sup,'s EA (S,(w, Q( a', ) (w"Q(''s al, w));; denotes the preference- driven target value which is obtained using target Q-network's parameters 0' Instead of MSE, Smooth LI loss may also be used for this loss function. We soft update the target network at every time step k. Note that each child process runs for N time steps. Hence, in total, collect N Cp transitions.
Algorithm 2: Preference Driven MO-DDQN-HER
Input: Minibatch size No, Number of time steps N, Discount factor Target network update coefficient T, Multi-dimensional interpolator [[w).
Initialize: Replay buffer Current Qo and target network Qu <- Qo with parameters and 0'. for = 0: N do
// Child Froccss Initialize t 0 and done False.
Reset the environment randomly initialized state so-
Sample preference vector w from the subspace f
while done False do Observe state 8; and select action e-greedily:
E A
w.p.
maxec (Q(s, w; 0),
w.p. 1
Observe r, s', and done.
Transfer (8t, de, Fe: w, done) main process.
// Main Process
Store the transition (se, Fe, 8', w. done) obtained from every child process in D. Sample No preferences w'
for j 1: No do
L Store transition (8t, at, s', w'', done) in D
Sample Nm. transitions from D. WP I(w)
y e (Sc a w))) (u=)
(Sc
La (0) (E (y Q(s,a,
Update 0& by applying SGD to (0).
Update target network parameters 0% +- 70p + -1- 7)0%
B.5. DETAILS ON MO-TD3-HER
We extend the TD3 algorithm to multi-objective version using PD-MORL for problems with contin- uous action space. To incorporate the relation between preference vectors and Q-values for the multi- objective version of TD3, we include a directional angle term g(w, Q (ss, a, w; 8)) to both actor's 's and
critic's loss function. This directional angle term g(w, Q(s, w; 0)) = cos 1( (w | (0)0)))))))) denotes the angle between the preference vector (w) and multi-objective Q-value Q(s, a, w 0) and provides an alignment between preferences and the Q-values.
Q(s.a.a:))))
Similar MO-DDQN HERR we first initialize an empty buffer D, critic networks Q8, Qe, and actor network ग. with parameters 0,, 02, and é. We also initialize critic and actor target networks. The process of initialization child processes the same with MO-DDQN-HER, The agent interacts with the environment to collect transition (s, a. r' done) by selecting actions according to policy with an exploration noise term e. The utilization of HER for collected transitions also the same the MO-DDQN-HER We then sample minibatch of transitions from D. The algorithm computes actions for the next state using the target actor network plus a target smoothing noise. In the conventional TD3 algorithm, both critics use single target value to update their parameters calculated using whichever of the two critics gives a smaller Qavluu. In multi-objective version of TD3, we modified this clipped double-Q-leammen approach such that target Q-value is calculated
18

---

Published as a conference paper at ICLR 2023
using whichever of the two crities gives smaller wTQ. To calculate the directional angle term, similar to MO-DDQN-HER, we project the preferences w to normalized solution space and obtain projected preferences wp. MO-TD3-HES minimizes the following loss function to update both critic networks at each step k:
Leritica (0:) (s w)-D w)D (s Q(s a(: 8.)) ((s.aa)) plw, Q(s, als w: (11)
where y = r argg min;- 2 Q w; 02) denotes the target value which is obtained using target critic network's ' parameters. Instead of MSE, Smooth LI loss may also be used for this loss function. One of the essential tricks that TD3 has is that it updates actor and target networks less frequently than the critics. The actor network is updated every Paelay step by maximizing the w wTQ while minimizing the directional angle term using the following loss:
V Lactory (Ø) = (' W)D Va w Q(8, a, w; 01))0=))) V (8, w; x))
(12)
a a. (s,a,)DD
Va g(wp, Q a, (100))= V wi の)
where a denotes the loss coefficient to scale the directional angle term to match the w² term. Here, we also soft update the target networks every time step k. Similar to MO-DDQN-HEL HER, each child process rums for N time steps. Hence, in total, we collect N x Cp transitions during training.
Algorithm 3: Preference-Driver MO-TD3-HER
Input: Minibatch size Nm. Number of time steps N Discount factor 7; Target network update coefficient T, Multi-dimensional interpolator I(w).
Policy update delay pdetoy. Standard deviation for Gaussian exploration noise added to the policy a, Standard deviation for smoothing noise added to target policy o'.
Limit for absolute value of target policy smoothing noise
Initialize: Replay buffer D, Critic networks Q02 and actor network Te with parameters 61, 82 and Target networks Qo;
Q8 Oog f
Q02 Teg f- Re.
for = N do
// Child Process Initialize t 0 and done = False.
Reset the enviromment randomly initialized state so.
Sample preference vector w from the subspace
while done = False do
Observe state 8; and select action with exploration noise 0r ~ =
Observe reward s', and done.
Transfer (8t, Gr, Tt; 8', w, done) to main process.
// Main Process
Store the transition (Se, ae, s", done) obtained from every child process in D
Sample No, preferences w'
for j 1: N_ do
L Store transition (8t, at, 8', w'y, done) in D Sample Nm. transitions from D.
à ( ww) +e: clip(N o'), -c,
7-targg mini;=1,2 w Q a, Wp I(w)
g(wp, Q(,, a, w.), & cos ( w ((saaa))))))))
Lariticn (0.) E( wiwD -0, Q(s,a, 9]
ElG. folw,, Q(s, a(s,a,
Update Bi, by applying SGD to Lorition (9:). if v. mod partay then
V Toactor, (Ø) E V T Q(s. w; (1)) Vor(s,
a E(e.n,,)) Va Vag(w, g(w,, Q(s, a, wi 01 (1))l==mm VoT(
Update target critics parameters 0% + Tb,, + +1--))0/ 7)
Update target actor parameters oi <- ThE + (1 T)%
19

---

Published as a conference paper at ICLR 2023
B.2 BENCHMARKS
We first evaluate PD-MORL's performance on two commonly used discrete MORL benchmarks: Deep Sea Treasure and Fruit Tree Navigation. For these benchmarks, we make use of the codebase provided by Yang et al. 2019). We further evaluate PD-MORL on multi-objectivv continuous control tasks such as MO- Walker2d-v2, MO-HalfCheetah-vv2. MO-Ant-v2, MO-Swimmer-v2, and MO-Hopper-v2. These benchmarks are presented by Xu et al. (2020) and are licensed under the terms of the MIT license. The details for all benchmarks are provided below:
Deep Sea Treasure (DST): well-studied MORL benchmark (Hayes et al. 2022 Liu al. 2014) where an agent is a submarine trying to collect treasures in 10 x 11 grid-world The treasure values increase as their distance from the starting point 8n (0, 0) increase. The submarine has two objectives: the time penalty and the treasure value. The actions are navigation in four directions and are discrete. The reward a two-element vector. The first element shows the treasure value, and the second element shows the time penalty.
Fruit Tree Navigation (FTN): A recent MORL benchmark presented by Yang et al. (2019). It is binary tree of depth d with randomly assigned reward r R66 on the leaf nodes. These rewards show the amounts of six different nutrition facts of the fruits on the tree: (Protein, Carbs, Fats, Vitamins, Minerals, Water). The goal of the agent is to find a path on the tree to collect the fruit that maximizes the nutrition facts for a given preference.
MO- H0-Walker2d-v2. The state space and action space is defined as S C RR",4 A R. The agent is a two-dimensional two-legged figure. It has two objectives to consider: forward speed and energy efficiency. The goal is tune the amount of torque applied on the hinges for a given preference w while moving in the forward direction.
MO- HO-HHafCCheetah-e v2: The state space and action space is defined as S C R A C R The agent is a two-dimensional robot that resembles a cheetah. It has two objectives to consider: forward speed and energy efficiency. The goal is to tune the amount of torque applied on the joints for given preference w while running in the forward direction.
The state space and action space defined as S S R27,, A R The agent is a 3D robot that resembles an ant. It has two objectives to consider: x-axis speed and y-axis speed. The goal is to tune the amount of torque applied on the hinges connecting the legs and the torso for a given preference w.
MO-Ant-v2:
The state space and action space defined as S S R A R. The agent is a two-dimensional robot. It has two objectives to consider: forward speed and energy efficiency. The goal is to tune the amount of torque applied on the rotors for a given preference w while swimming in the forward direction inside a two-dimensional pool.
MO-Swimmer--2:
MO- Mo--ooppe--v2: The state space and action space is defined as S C R A S RR. The agent is a woo-dimensional one-legged figure. It has two objectives to consider: forward speed and jumping height. The goal is to tume the amount of torque applied on the hinges for a given preference w while moving in the forward direction by making hops
B.3 TRAINING DETAILS
We run all our experiments on local server including Intel Xeon Gold 6242R. We do not use any GPU in our implementation. For Deep Sea Treasure and Fruit Tree Navigation benchmarks, we employ the proposed MO-DDQN-HER algorithm. The network here takes state and preference w as inputs and outputs IA] x I. Q-values. The number of hidden layers and hidden neurons among other hyperparameters for MO-DDQN-HER given in Table 4
For continuous control benchmarks, we employ the proposed MO-TD3-HER algorithm. The critic network takes state 8, preference and action input and outputs L Q-values for each objective. The actor network takes state s and preference w as inputs and outputs |A] actions. The number of hidden layers and hidden neurons among other hyperparameters for MO-TD3-HED for each continuous benchmark are given in Table The hyperparameters are the same for all benchmarks except the policy update delay for MO-Hopper-v2, which is set to 20.
20

---

Published as a conference paper at ICLR 2023
As it is elaborated in Section B.1.3
first obtain key solutions for each of the preferences in the key preference set to fit a multi-dimensional interpolator. Since we already know the Pareto front solutions for discrete benchmarks, continuous control benchmarks, we train an agent utilizing the multi-objective version of TD3 with a fixed key preference vector and without using HER. The number of hidden layers and hidden neurons among other hyperparameters for this approach on each continuous benchmark are given in Table 6
directly use key solutions in the Pareto front to fit this interpolator. For
Table 4: Hyperparameters for MO-DDON-HER
Deep Sea Treasure
Fruit Tree Navigation
Total number steps (N)
10
x 10
Minibatch size (Nvr))
32
32
Discount factor (?)
0.99
0.99
Soft update coefficient (r)
0.005
0.005
Buller size
1 10*
10
Number of child processes (Cp)
10
10
Number of preferences sampled for HER (N.)
3
Learning rate
3 10
x 10
Number of hidden layers
Number of hidden neurons
256
512
Table 5: Hyperparameters for MO-TD3-HER
MO-Walker2dwv2 MO-HallChhetahbw2
MO-Ant-v2
MO-Swimmer-r2
MO-Hopper--2
Total number of steps
x 10a
10"
10"
10"
10
Minibateh size Discount
256
256
256
256
256
0.995
0.995
0.995
0.995
Soft updale cvefficient Buffer size
0,005
0.005
0.005
0.005
0.005
x 10a
10"
x 10"
10
Number of child processes
10
10
10
10
10
Number of preferences
sampled for HER
3
3
Learning rate-Critic
10
10-4
10
10
10
Number of hidden layers Critic
1
1
1
1
Number of hidden
Critic
400
400
400
Learning rate-Actor
10
3 10-1
x 10-1
x 10
10-1
Number of hidden layers Actor
Number of hidden neurons- Actor
400
400
400
400
400
Policy update delay
Exploration noise std.
0.1
0.1
0.1
0.1
0.1
Target policy smoothing noise std.
0.2
0.2
Noise elipping limit
0.5
0.5
0.5
0.5
0.5
Loss coefficient
10.
10
10
Table 6: Hyperparameters for MO-TD3 algorithm to obtain key solutions
MO-Walker2d-v2 MO-HalCCheeah--2
MO-Ant-v2 MO-Swimme-x2
MO-Hopper-r2
Total number of steps
2 x 10
2 x 10f
10
x 10
x
Minibateh size Discount factor
100
100
100
100
100
0.99
0.99
0.99
0.99
0.99
Soft update coefficient
0,005
0.005
0.005
0.005
0.005
Buffer size
Gx 10
x 10P
5 10"
x 10
x 10
Learning rafe-Critic
x 10
x 10 1
3 10
x 10 1
x 10
Number of hidden layers Critie
1
Number of hidden neurons- Critic
400
400
400
400
400
Learning rafe-Actor
10
3 10
x 10
10
10
Number of hidden layers Actor
1
1
1
Number of hidden neurons- Actor
400
400
400
400
400
Policy update delay
2
2
5
10
Exploration noise
0.1
0.1
0.1
0.1
0.1
Target policy smoothing moise std.
0.2
0.2
0.2
0.2
0.2
Noise clipping limit
0.5
0.5
B.4 ADDITIONAL EXPERIMENTAL RESULTS
Since the main objective of this work to obtain a single universal network that covers the entire preference space, first obtain a representative set of preference vectors. For Deep Sea Treasure benchmark, we obtain a preference vector set with a step size of 0.01 ((0, 1), (0.01, 0.99).......... (0.99, 0.01), ) {1,0) 0}) Similarly, step size of 0.1 and 0.001 for Fruit Tree Navigation and continuous control benchmarks, respectively.
obtain preference vector sets with
Since the Envelope algorithm (Yang al. 2019 is the superior and more recent approach that learss unified policy, we first compare PD-MORL with this algorithm using discrete MORL benchmarks. For the Envelope algorithm, we the authors' codebase with their default hyperparameters mentioned
21

---

Published as a conference paper at ICLR 2023
in the paper since we believe that they are already optimized for these two benchmarks. In addition to hypervolume and sparsity metries that we provide in Section [5.]] we also compare PD-MORL with the Envelope algorithm using the coverage ration F1 (CRF1) metrie. This metric, which is coined by Yang et al. 2019), evaluates the agent's 's ability to recover optimal solutions in the Pareto front. It assumed that we have prior knowledge of the optimal solutions for these benchmarks. Let B be the set of solutions obtained by the agent for various preferences, and let P be the Pareto front set of solutions. The intersection between B and P with a tolerance of e is defined as B PnPP P b E B E P: ||b p Ilpll S The precision in this context is defined as Precision
[BniI [B] [P] Then CRF1 is computed CRF1 = 2 Precision x Recall
and recall is defined as Recall |BnPI
Precision + Recall
Table summarizes the comparison of PD-MORL with the Envelope algorithm in terms of CRFI, hypervolume, and sparsity metrics on discrete MORL benchmarks. We emphasize that our evaluation process is more extensive than the work proposed by Yang et al. (2019). In their implementation, they report an average of 2000 evaluation episodes where a random preference is sampled uniformly in each episode. In contrast, evaluation of PD-MORL sweeps the entire preference space. The hypervolume and sparsity values for both approaches are obtained through PD-MORL's evaluation process and are discussed in Section 5.T In this table, we provide the CRFI values reported in (Yang et al. 2019) for the Envelope algorithm. PD-MORL achieves larger or the same CRF1 values compared to the Envelope algorithm for all benchmarks. Specifically, it achieves up to 12% higher CRFI for the Fruit Tree Navigation task with depth of d 7. Additionally, comparison with (Abels et al.) 2019) is given in Table 7 which shows the superiority of both the Envelope algorithm and PD-MORL over the proposed algorithm by Abels et al. (2019) which also learns a unified policy.
Table Comparison of our approach with prior works (Yang et al. 2019 Abels et al. 2019) using discrete MORL benchmarks in terms CRF1, hypervolume, and sparsity metrics." are the reported
values in YYagg et al. 2019
Deep Sea Treasure
Fruit Teer Navigttonn (d::5)
Fruit Tre Novigation (dtr))
Fruit Yeee Navigation (d=:)
CHF1
Hypersotome
Spanity
CRF1
Hypersatume
Sparsity
CRFF
Hyperrotome
Spanity
CRF1
Hypervotame
Spandiy
Envolupe
227.89
0.985
6427.51
N/A
CN+DER
0.589"
0.9258*
NIA
0,6719
Ours
241.73 1.14
6920.58
NIA
9299.15
N/A
0.92
11419,58
NIA
For the continuous control benchmarks, we provide additional plots on the Pareto front and hypervol- ume progression. Figure (ggure(a)-(c) (())(c) shows the progression of the Pareto front solutions during training. We plot the Pareto front solutions at early stages, mid stages, and late stages of the training process. As the training progresses, the Pareto front expands to a broader and denser curve. This behavior is supported by Figure 7 (e). which shows the progression of the hypervolume for all benchmarks. This figure shows that PD-MORL effectively pushes the hypervolume by discovering new Pareto solutions with the help of efficient and robust exploration. For the MO-HalfCheetah-v2 problem, a broad and dense Pareto front is already achieved at the early stages of the training. Therefore, do not observe the progression of the Pareto front and hypervolume for this specific benchmark.
3000
3000
3000
2000
2000
2000
Early-Stage Mid-Stage Late-Stage
1000
1000
MO-HallCheetah--2
M0-Ant-v2
0
1000
2000
3000
0
1000
2000
3000
1000
2000
3000
200
6000
d)
e)
4000
100
2000
MO-Swimmeer-w
NO-Hooppe--22
0
0
100
200
2000
4000
Figure 6: Pareto front progression for (a) MO- MO-Walker-v2, (b) MO-HalfCheetah-v2..G (c) MO-Ant-v2,
(d) MO-Swimmer-v2. (e) MO-Hopper-v..
22

---

Published as a conference paper at ICLR 2023
8
x x10
x 100
XIO
a)
b)
c)
Hypervolume
6
4
M0-Walkerrdd-22
MO-HalfCheetabb-2
MO-Ant-v2
0
10
10
Time Steps(x109)
Time S119s(x109)
Time Steps(x109)
x10*
x107
d)
Hypervolume
1.5
2
MO--Siimme--22
MO-Hopper-vv
Time Steps(x10*)
Time Steps(x109)
Figure 7: Hypervolume progression for (a) MO-Walker-v2, (b) MO-HalfCheetah-v2,, (c) MO-Ant-v2, (d) M0-Swimmer-v2, (e) MO-Hopper-v2.
also report standard deviations for all benchmarks. As explained the main manuscript, since META (Chen et al. 2019) and PG-MORL (Xu et al. 2020) report the average of six runs, we also ran each benchmark six times with PD-MORL. Table reports average metrics as well as standard deviations obtained from these six runs. The standard deviations obtained from six runs not reported in the prior work. This table shows that the individual differences of different runs are mostly three orders of magnitudes less than the average of these runs. This also suggests that PD-MORL achieves a generalizable network independent of starting state of the agent. We also provide more visual results for preferences that captures corner cases for these benchmarks in the supplementary video.
Additionally, we conduct an ablation study to show the effects of the proposed angle term in the loss function and the proposed parallel exploration in our algorithm on continuous control benchmarks. Table reports average metrics and standard deviations obtained from six runs. Since we fit mimti-dimensional interpolator using solutions with key preferences, the interpolator may introduce bias the beginning of the training depending on how representative these key solutions are. Hence, this bias may have negative effect the training. For tasks with broad and more convex Pareto front, the probability of this bias increases as we choose key preferences from corner cases.
Table 8: Performance comparison of the proposed approach and state-of ofht---rt- algorithms on the continuous control benchmarks in terms of hypervolume and sparsity metrics. Reference point for hypervolume calculation is set to (0,0) point. HV*: Hypervolume
PG-MORL [Xu al., 2020
META (Chen al. 2019
PD-MORL (Ours)
HV*
4.82 x 10
2.10 x 10
5.41 + 0.004 x 10"
MO-Walker2d-v2
Sparsity
0.04 10a
2.10 x 10*
0.03 +: 0.005 10a
HV*
5.77 106
5.18 109
5.89 0.002 10
MO-HalfCheetah-v2
Sparsity
0.44 10a
2.13 10a
0.49 0.041 10a
HV*
6.35 x 108
2.40 x 10a
7.48 + 0.019 106
MO-Ant-v2
Sparsity
0.37 1
1.56 x 10a
0.78 4 0.2 x 10
HV"
2.57 10*
1.23 10* 24.4
3.21 0.001 10a
MO-Swimmer-v2
Sparsity
9.9
5.7 0.7
HV*
2.02 x 107
1.25 x 107
1.88 1 0.005 x 107
MO-Hopper-v2
Sparsity
0.5 x 10
4.84 x 10a
0.3 + 0.09 x 10a
23

---

Published as a conference paper at ICLR 2023
Table 9: Ablation study of the proposed approach with and without proposed terms and improvements on continuous control benchmarks in terms of hypervolume and sparsity metrics. Reference point for hypervolume calculation is set (0,0) point. HV* Hypervolume
PD-MORL w/o
PD-MORL w/o Parallel Exploration
PG-MORI et
PD-MORL
Angle Term
MO-Walker2d-v2
Sparsity
НY*
0.04 10*
1.82 10"
5.51 0.001 x 10
0.02 0.004 10
2.81 = 0.009 100
0.22 = 0,026
10"
5.41 + 0.001 x 10"
0.03 0.005 x 10*
MO-HallCheettb-x22
HV*
5.77 10"
5.61 + 0.003 x 10"
5.84 + 0.001 x 10"
5.89 - 0.002 10
Sparsity
0.44 10
49.73 - 3.917 10"
1,218 + 0.172 x 10*
0.49 0.041 X 10*
MO-Ant-v2
HY"
6.35 10"
9.12 0.017 100
4.98
100
7.48
10F
Sparsity
0.37 10
0.58 = 0.09
10*
0.91 + 0.25 10"
0.78 = 10"
MO-Swimmer-v2
HV
2.57 10 9.9
2.78 + 0.001 x 10'
3.21 = 0.001 x 10"
3.21 = 0.000 x 104 0.7
Sparsity
6.3
9.1
MO-Hopperww2
IIV
2,02 107
1,92 0.023 10?
0.63 0.005 100
1.88 1 0.005 10"
Sparsity
0.5 x 10*
6.1 + 2.30 10"
20.53 + 33.20 x 10*
0.3 0.09 x 10*
Therefore, by removing the angle term in the loss function and thus, removing the bias, the perfor- mance of our algorithm slightly degrades due to the convex and broad nature of the Pareto front in 1O-Walker2d-v2 and MO-Ant-v2 problems. However, removing the angle term has a negative effect for HO-HalfCheetah-v22 and MO-Hopper-v2, where the Pareto front is dense. Figure [ shows the obtained Pareto front with and without the directional angle term in the loss function. 'he solutions obtained without the directional angle term are sparse. Specifically, for HO-HaafCheetah-v22 Table 9 shows that both the hypervolume and the sparsity metrics are negatively affected. For MO-Hopper-v2. although the hypervolume metric seems be increased, the change in the sparsity metric suggests that the algorithm cannot find dense Pareto front. This, in fact, supports that the hypervolume alone insufficient assess the quality of the Pareto front. We further investigate the effects of using parallel exploration. To this end, we collect transitions using single preference space instead of dividing them into sub-spaces. Parallel exploration guarantees that obtained transitions are not biased towards a preference sub-space. Hence, it increases the sample efficiency of the algorithm. Table 2 shows that removing the parallel exploration significantly reduces the performance of the algorithm. For all tasks, the hypervolume metric decreases, and the sparsity metric increases. We also observe that both PG-MORL and PD-MORL with proposed novel aspects are superior to PD-MORL without the novel aspects. This observation is crucial since the prior work uses PPO, whereas PD-MORL uses TD3 in their algorithm pipeline. Using this observation, we conclude that the superiority of PD-MORL is due the proposed novel aspects.
PD-MORL PD-MORL w/o Angle Term
3000
2000
4000
1000
2000
MO-HalfCheetah-v2
MO-Hopper-v2
0
0
1000
2000
3000
0
2000
4000
Figure 8: Pareto front with and without the directional angle term in the loss function for MO- HalfCheetah-v2 and MO-Hopper-v2 problems. In the latter case, the solutions are sparse and represented by few points.
24


Reviews:
[{"review_id": "b_tuNORuj1", "reviewer": "Reviewer_sFJy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an algorithm for solving multi-objective reinforcement learning (MORL) problems with a single network, termed preference-driven MORL (PD-MORL). The main insight to do so (instead of computing a policy for each potential preference)", "summary_of_the_review": "This work provides a strong contribution, back up with theoretical results as well as empirical experiments.", "strength_and_weaknesses": "The main strengths of this paper:\n- Novel approach to solve MORL with a single network\n- Extensive experiments both in discrete and continuous state-action spaces\n- Theoretical support of optimality\n\nThe main weakness of this paper:\n- Some details are lacking in Section 4.2, specifically concerning the division of the preference space, the preference alignment using normalization and more importantly the extension of the algorithm to continuous action space. While it is true that these details are provided in the supplementary material, it would be helpful to have some hints in the main text. Space could be saved by condensing the introduction and related work section, which are rather detailed, thus leaving more space to describe the main novel components of this work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N2pL2d8q-Ll", "reviewer": "Reviewer_Mx6C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes PD-MORL, which extends Envelope (Yang et al., 2019). The major difference is illustrated in Equation 5 in the paper: an extra cosine similarity term is added, and the supremum is only taken over actions, not preferences (compared to Envelope). The authors evaluated their method with Envelope on Deep Sea Treasure and Fruit Tree Navigation, and with PG-MORL and META on some MuJoCo continuous control tasks.", "summary_of_the_review": "This paper proposes a new method of MORL based on Envelope, where they include a cosine similarity term in their optimality operator. They provided a theoretical analysis of their method, as well as performance evaluations with several baselines on several different tasks. My major concern in this work is the effectiveness of the cosine similarity term, and I believe an ablation study can demonstrate it. ", "strength_and_weaknesses": "- Multi-objective reinforcement learning is an important task, and learning a single network for multiple preferences is an efficient approach to it.\n- Theoretical analysis is provided. The authors proved (in the appendix) that their optimality operator is a contraction and the optimal value function is a fixed point for the optimality operator.\n- The effectiveness of the cosine similarity term in equation 5. In equation 5, we have cos(w, Q)•w^T•Q in the supremum operator. However, w^T•Q=cos(w, Q)•|w|•|Q|, which already includes the cosine similarity term. Is there a specific reason we need an extra cosine similarity term? A good way to verify its effectiveness is to perform an ablation study. Note that simply removing the cosine similarity term is not equivalent to Envelope as the supremum in the Envelope algorithm takes over different preferences.\n\n----Rebuttal----\nDear authors,\n\nThank you for your detailed response. Unfortunately, I'm afraid I disagree with your explanation of the cosine similarity term. In your explanation, if we simply change the range of latency to [1, 100] and set Q2 to {0.1, 100}, even with the cosine similarity term, we still will prefer action 2 (you can verify as 0.68 < 1.12). Therefore, if we have to think of w as a preference vector, the cosine similarity term is an inappropriate approach to fix the issue you mentioned. Instead, we can rescale and normalize the magnitude of all the objectives, which easily fixes the issue you mentioned.\n\nAs I can't agree with your explanation of the cosine similarity term, I have to lower my rating of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MAtcORzpyKd", "reviewer": "Reviewer_scaX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents PD-MORL as a fascinating new method to learn a single policy that can integrate task preferences and cover more of the Pareto front than any baseline algorithm. This differentiates it from previous methods that must learn a new policy for any particular preference. It demonstrates marginally better results in existing multi-objective baseline problems, with excellent analysis to explain the improvement.", "summary_of_the_review": "This is a good paper that takes a clever, novel, and not overly complex approach to multi-objective reinforcement learning. While the empirical results aren't outstanding and the description of the algorithm itself is a little unclear, the rest of the analysis is solid and overall everything seems sufficiently suitable and solid for this conference. I think this is sure to inspire good follow-up work, further justifying why it should be shared with the wider community.\n\nUpdate: The author responses have addressed all of my earlier concerns and the few new concerns are very minor. I think the changes address most of my fellow reviewers concerns and improved the paper overall, so I have decided to maintain my score of 8. ", "strength_and_weaknesses": "The paper excels at explaining the problem, related works, algorithm, and results. The clarity of most sections is very good and the design of the algorithm, choice of experiments, metrics, figures, etc are all well justified and effective. It also very effectively highlights the differences of PD-MORL from previous works, and justifies why these differences make this approach a promising direction for multi-objective reinforcement learning. Figures 4 and 5 are also excellent at visualizing the algorithm and key ingredients to its success.\n\nThere are a few weaknesses in the paper too, however. First, MORL benchmarks used are not very well explained and the visualizations for the tasks and policies are not very clear. I have minimal understanding of what the challenges in each benchmark are and why baselines failed in these tasks -- this paper does not need an exhaustive description of these benchmarks but could certainly explain them and the learned policies better through some textual revisions and more figures (e.g., fig 3 is not very well explained/useful in its current state). Next, the experimental results are not sufficiently explored. For example, the paper does not address why the contributed PD-MORL performs significantly worse than PG-MORL in the hopper experiment, or why sparsity is sometimes worse than PG-MORL. Including these results is essential, but so is some analysis insight into the weaknesses of PD-MORL relative to PG-MORL in these test cases, and perhaps also some ideas on how these issues in PD-MORL might be overcome. It also seems worth including some discussion on the limitations of these metrics. On a related note, the results in Table 1 should have error values representing. Thirdly, the usage of the cosine similarity in eq. 5 is not well enough explained -- the inner product of omega and Q should already reflect the cosine similarity of omega and Q(s,a, omega), so why is the scalar Sc(omega, Q) also multiplied by the inner product of omega and Q? I also don't fully understand the preference alignment procedure, including how the normalized solutions in Fig 2b are formed and what purpose there is to interpolating them, e.g. rather than interpolating in a different space. Finally, the related works and experiments sections are rather monolithic and would benefit greatly from splitting the large paragraphs into individual ideas or techniques, possibly using other methods (e.g. figures/tables and subsections) to better organize and discuss the ideas in them. \n\nA few more small notes: the vector nature of Q is not well explained (why is the dimensionality of Q(s, a, w) equal to L and what does it represent before/after taking the inner product with omega?), the walker results in fig 5 need y-axis labels, the paper should justify why fewer parameters is a big deal, the limitations/future work discussion isn't very interesting/compelling and should be more specific and try to offer some deeper insights. Also, the use of L2 distance in sparsity is possibly worth discussing if there's space -- it seems like a rather arbitrary choice of metric so I am curious if there is a reason for it.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8F56OJncCjQ", "reviewer": "Reviewer_QTgN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new multi-objective RL algorithm named Preference-Driven MORL (PD-MORL). PD-MORL uses the preference to align the Q function updates. Theoretical analysis shows that the new preference-driven optimality operator ensures convergence to the optimal Q function. Experiments on classical benchmark environments and multi-objective variants of MuJoCo environments are provided.", "summary_of_the_review": "Because of the important baseline results missing, I am not convinced that the paper in its current form is ready to be published.", "strength_and_weaknesses": "Strength: \nThe idea to align the preference vector better with the Q function is interesting and relevant to the multi-objective RL community. \n\nThe writing is mostly clear and easy to follow.\n\nWeakness:\nThere are important baselines missing in the empirical evaluations. The envelope Q-learning algorithm by Yang et al. is only compared in DST and FTN but missing from MuJoCo environments. \n\nAnother important baseline is the conditional network with diverse experience replay algorithm in Abels et al. (Dynamic Weights in Multi-Objective Deep Reinforcement Learning). I believe adding both baselines would make the comparisons more informative. \n\nThere seems to be a mismatch between the practical version of the algorithm and the theoretical version. Algorithm 1 uses the interpolated preference rather than the raw preference. Can the authors comment on this choice and how the theoretical result extends to this different update rule?\n\nThere are some details missing in the description of the algorithm. The preference interpolation is an important part of the algorithm but its description is lacking. I suggest having a more detailed exposition on this, including how the key preference points are selected, how the normalization procedure is done, and a mathematical expression of the interpolation function.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster