Demonstration-Guided Multi-Objective Reinforcement Learning
Authors: Junlin Lu
Patrick Mannion
Karl Mason
Venue: Accepted by TMLR ()

Abstract: Multi-objective reinforcement learning (MORL) closely mirrors real-world conditions and has consequently gained attention. However, training a MORL policy from scratch is more challenging as it needs to balance multiple objectives according to differing preferences during policy optimization. Demonstrations often embody a wealth of domain knowledge that can improve MORL training efficiency without specific design. We propose an algorithm i.e. demonstration-guided multi-objective reinforcement learning (DG-MORL), which is the first MORL algorithm that can use prior demonstrations to enhance training efficiency seamlessly. Our novel algorithm aligns prior demonstrations with latent preferences via corner weight support. We also propose a \textit{self-evolving mechanism} to gradually refine the demonstration set and avoid sub-optimal demonstration from hindering the training. DG-MORL offers a universal framework that can be utilized for any MORL algorithm. Our empirical studies demonstrate DG-MORL's superiority over state-of-the-art MORL algorithms, establishing its robustness and efficacy. We also provide the sample complexity lower bound and the upper bound of Pareto regret of the algorithm.

Published in Transactions on Machine Learning Research (11/2024)
Demonstration-Guidee e Reinforcement Learn-
ing
Junlin Lu
Lu5 wwnnveersstyyyoggnknnn... se matrick.mannion monnion Suniversihyofgaaungg ie
Patrick Mannion
patrick.
Karl Mason
karl. maslm Swniverssyyoggguguuuyr
School of Computer Science
University of Galway
Galway, Ireland, 1911 TK33
Reviewed on OpenReview: https: // optp///eee.rvveee..em net/ forum? id=FQAgFgkaud
Abstract
Multi-objective reinforcement learning (MORL) closely mirrors real-world conditions and has consequently gained attention. However, training MORL policy from scratch is more challenging as it needs to balance multiple objectives according to differing preferences during policy optimization. Demonstrations often embody a wealth of domain knowledge that can improve MORL training elficiency without specific design. We propose algorithm i.e. demonstration-guide guided multi-objective reinforcement learning (DG-MORL), which is the first MORL algorithm that can use prior demonstrations to enhance training efficiency seamlessly. Our novel algorithm aligns prior demonstrations with latent preferences via corner weight support. We also propose self-evolving mechanism to gradually refine the demonstration and avoid sub-optimal demonstration from hindering the training. DG-MORL offers universal framework that can be utilized for any MORL algorithm. Our empirical studies demonstrate DG-MORL's superiority over stale-of-the-arl MORL algorithms,, establishing its robustness and efficacy. We also provide the sample complexity lower bound and the upper bound of Pareto regret of the algorithm.
1 Introduction
Single-objectiv reinforcement learning (SORL) has exhibited promise in various lasks, such as Atari Games (Mnih al. 2013), robotics (Schulman al. (017), and smart home energy management (Lu et al. 2022a). However, most real-world challenges often encompass multiple objectives. For example, there is always need balance between safety and time efficiency in driving or between heating-energy usage and cost-saving for home heating. These real-world scenarios extend SORL into multi-objective reinforcement learning (MORL) (Hayes et al. 2022). In MORL, the target is to learn an optimal policy set that maximizes the cunulative discounted reward vector. Users can set preferences to scalarize the reward vector to pick a proper candidate from the optimal policy set.
However, MORL introduces additional complexities and significantly amplifies the challenges already present in SORL. The challenges include: (i) Sparse reward: In scenarios where rewards sparse, agents struggle to acquire sufficient information for policy improvement. This challenge, widely recognized in reinforeement lcarning (RL) (Oh et al. 2018 Ecoffet et al. 2019 2021 Wang et al. 2023). can slow down the training process, heighten the risk of reward misalignment, and may result. in suboptimal policies or even training failure; (ii) Hard beginning: At the onset of training, policies often face substantial challenges improving Li 2022 Uchendu et al. 2023). This stems from the agent's lack of prior knowledge about the task. Though exploration strategy like e-greedy can be used to gradually mitigate this typically inefficient in the early stages of training; (iii) Derailment: The agent may steer off the path returning to promising areas of state

---

Published in Transactions on Machine Learning Research (11/2024)
space due to environmental stochasticity, policy uncertainty, and the randomness inherent in exploration strategies (Ecoffet et al. 2019). This derailment can impede the agent's 's ability to revisit these ad advantageous areas, thereby restricting policy improvement.
These challenges are all different reflections of low exploration efficiency. An intuitive solution is to leverage prior demonstrations as guidance to integrate expert knowledge. Unfortunately, existing techniques (Fooffet ett al.) 2019 2021 Uchendu et al. 2023) are exclusively designed for SORL. There is currently no established method for using prior demonstrations in MORL. Morcover, the direct application of demonstration in MORL within the existing paradigm is however unrealistie. Because: (i) Cemmonstration-Prefeeeee
Misalignment: In MORL, demonstration may be optimal under certain preferences, but detrimental under others. For an agent to effectively learn from a demonstration, it needs to know the preference that renders the demonstration optimal, as a guide of training direction. In most cases, however, even the demonstrators themselves cannot identify the specific numeric preference under which the provided demonstration is the optimal (Lu et a 2023). Consequently, when an agent is provided with a set of demonstrations without associated preference information, it struggles to correctly match demonstration with the corresponding preference factor. This can lead the agent in misguided direction, and potentially fail the training; (ii) Demonstration Deadlock: In SORL, the demonstration guidance method typically relies on at least one demonstration to assist in learning the optimal policy. However, in MORL, it becomes more complex due to the presence of several, let's say n, Pareto non-dominated policies, each optimal under different preferences. Tf we were to directly apply the existing demonstration-oasee SORL approach to MORL, it would imply the need for at least n demonstrations to cover these various policies. The challenge is that the exact number n remains unknown until all Pareto non-dominated policies are identified. This creates a paradox: the requirement to have the number of demonstrations is contingent upon the outcome of the training process itself. It can lead to lack of necessary prior data, and polentially impair the overall performance. We refer to this paradox as the demonstration deadlock; (iii) Sub-optimal Demonstration: Though demonstrations that are slightly sub-optimal can still effectively guide the training process (Uchendu et al. 2023), it is uncertain whether sub-optimal demonstrations can similarly work in MORL. Sub-optimal demonstrations might obstruct the training, potentially leading to the learning of a sub-optimal policy, or in more severe cases, failure of the training.
In this paper, we propose novel MORL algorithm, i.c. demonstration-guude multt-objective reinforcement learning (DG-MORT.) to seamlessly improve MORL training efficiency with demonstrations. DG-MORT, is the first MORL algorithm that can use demonstration as guidance. Though a demonstration generally does not correspond to a policy, it can be considered a fixed, hard-coded policy that merely executes actions regardless of the state observation. It akin to the methods described in (Fcoffet et al. 2019 2021), ) where only sequences of actions were utilized to guide the training process. In this study, demonstrations are employed as a guide policy. In the following sections demonstrations and guide policies are interchangeable. The demonstrations can take any form i.e. trajectory from human experts, a pre-trained policy, or even hand-crafted rules. We assess various types of demonstrations and experimentally show their effectiveness. DG-MORL surpasses the stato-of-the-art MORL algorithms in complex tasks. The key contributions of our work include:
(i) We propose the first MORL algorithm (DG-MORL) that can leverage demonstrations to improve training performance.
(iiWee introduce self-evolving mechanism to adapt and improve the quality of guiding data by gradually transferring from prior generated demonstrations to the agent's 's self-generated demonstrations, This mechanism safeguards the agent against sub-optimal demonstrations.
We outline the related work in Section 2 Tn Section 3 we present the preliminary of this paper. In Section
4 we formally introduce the DG-MORL algorithm and provide theoretical analysis for our algorithm. We
illustaate the baseline algorithms, benchmark environments, experiment setting, metrics, and demonstrate and discuss the experiment results in Section 5 We discuss the limitations of our method in Section 0 The paper is concluded in Section 7
2

---

Published in Transactions on Machine Learning Research (11/2024)
2 Related work
2.1 Multi-objective reinforcement learning
MORL methods are categorized as single-policy methods and multi-policy methods (Yang et al. 2019 distinguished by the number of policics. In single-policy methods, MORL reduccs to SORL. The agent uses the SORL learning method to maximize the cumulative reward vector scalarized by a given preference (Mannor & Shimkin 2001; Tesauro et al. 2007 Van Moffaert et al. 2013). However, when the preference is not given, Single-policy algorithms cannot work properly.
The multi-policy methods aim at learning policy set to approximate the Pareto front A common method is to train single policy multiple times upon the different preferences (Roijers et al. 2014 Mossalam et al. 2016). Pirotta et al. (2015) introduced a policy-based algorithm to learn a manifold-based optimal policy. These methods suffer from the curse of dimensionality when the preference space grows and their scalability is limited in complex tasks. To overcome this, policy conditioned on preference to approximate a general optimal policy were proposed, where Abels al. (2019), and Yang al. (2019) use vectorized value function updates. Distinguished from the other two methods, Källström & Heintz (019 use a scalarized value function approximation. By directing the policy training with updating the gradient according to the alignment of value evaluation and preference, these methods are improved. Basaklar et al. (2023) used a cosine similarity between Q- vavauu and preference to achieve an efficient update, while Alegre et al. (2023) used generalized policy improvement (GPI) (Puterman, 2014) to guide the policy to update along with the preference that, can bring the largest improvement to the policy.
2.2 Prior demonstration utilization approaches
Methods using prior demonstrations for RL usually are combination of imitation learning (IL) and RL. One direct way to initialize an RL agent with demonstration is to use behavior cloning to copy the behavior pattern of the prior policy (Rajeswaran et al.] 2017 Lu et al. 2022a). This method cannot work well with value-based RL frameworks (Uchendu et al. 2023). Furthermore, behavior cloning cannot work with multi-objective settings if the preference not known beforehand.
Vecerik et al. (2017): Nair et al. (2018); Kumar et al. (2020) have proposed methods for integrating prior data into replay memory. These techniques have been advanced through the use of offline datasets for pre-training, followed by policy fine-tuning (Nair et al., 2020 Lu et al. 2022b). [Ball et al. 20233 proposed simple way to use offline data. However, these methods lypically require an ollline dataset. In contrast, our approach operates under less stringent assumptions, relying only on a few selected demoustrations.
A Bayesian methodology offers another avenue for employing demonstration data. Ramachandran & Amir (2007) proposed the Bayesian Inverse Reinforcement Learning (BIRL) method, modeling the Inverse Reinforcement Learning (IRL) problem as a Bayesian inference process. By combining prior knowledge with evidence from expert behaviors, they can effectively infer the postcrior distribution of the reward function, improving the efficiency and accuracy of policy learning. Choi & Kim (2013) proposed Bayesian non-parametric approach to construct reward function features in Inverse Reinforcement Learning (IRI.). Their method uses the Indian Buffet Process (IBP). It integrates domain expert prior knowledge by having experts provide atomic features representing key factors influencing rewards and enabling more accurate reward reconstruction. Choi & Kim (2012) proposed a non-parametric Bayesian IRI. method using the Dirichlet Process Mixture Model (DPMM) to infer an unknown number of reward functions from unlabelled behavior data. This approach can automatically determine the appropriate set of reward functions based the data, without predefining the mumber. It is particularly effective in handling expert demonstrations with different goals. They also developed an efficient Metropolis- Hastiings sampling algorithm leveraging posterior gradients to estimate the reward functions. However, these works are more from IRL and are not specifically designed for MORL. This is different. from our method which uses demonstrations to guide the policy training but not to infer the reward function in a MORL setting.
Furthermore, the absence of extension and empirical validation of these mothods in MORL settings hinders direct comparisons to our method. Additionally, multi-objective imitation learning remains under-explored,
3

---

Published in Transactions on Machine Learning Research (11/2024)
presenting another limitation for comparative analysis. Therefore, our algorithm is compared with other existing MORL algorithms.
2.3 Effective exploration approaches
Several approaches have been introduced to improve the exploration efficioncy, including reward shaping Ng
et al. 1999: Mannion 2017), curiosity-driven exploration (Barto et al. 2004), and model-based reinforcement learning (MBRL) (Polydoros & Nalpantidis 2017; Alegre et al. 2023).
However, they still have notable shortcomings. For example, the reliance domain expertise for reward The inherent vulnerability trapped in local optina in curiosity-driven exploration (Fooffet et al., 2021 ;; The computation overload and inaccuracy of enviromment model construction
shaping Devidze et al.,
2022);
for MBRL. It is necessary to effectively explore the environment so that the model is closer to the ground truth of the environment dynamics. However, this has an additional computational cost when the environment is hard explore, the environment. model may not be accurate enough and this will impact the training adversely.
Using demonstration has the following advantages to mitigate these drawbacks. It allows for the integration of domain knowledge without explicit reward-shaping function. This can alleviate the reliance on domain expertise; Guided by a demonstration, the agent can return to promising arcas that it might have struggled to revisit. before the curiosity intrinsic signal is used up; It does not require an environment model and therefore frees from computational demands and model uncertainties.
3 Preliminaries
We formalise a mellfi-objective Markov decision process (MOMDP) as M: (S,A,T,7,,4 R) (Hayes al., 2022), where S and A are state and action space; T S A - [0, 1] is a probabilistic transition function: E (0,1) is discount factor; μ So - [0, 1] is the initial state probability distribution; R S x A x S + Rd is a vectorized reward function, providing the reward signal for the multiple (d 2 2) objectives.
The MORL state-action volue function under policy π is defined as: q" a) := E, -= ls a]. where qtt (s. a) is a d-dimensional vector denoting the expected vectorized return by following policy n, Tr is the reward vector on step.
The multi-ohjectiive value vector of policy with the initial state distribution z is: o² == Eso (I (so, n (50)], where ot is the value vector of policy t, and the i-th element of vx is the returned value of the i-th objective by following policy æ. The Pareto front (PF) consists of a set of nondominated value vectors. The Pareto dominance relation (>p) is: 8x >p dx =
(Vi of 2 of n)(i:: : of > u We say that on nondominated when all element in n are not worse than any o² that at least one element of oz is greater than the other v= The Pareto front is defined as PF = {iT s.t. v² >p v"
To select an optimal policy from the PF, the critoria are usually given by the user and therefore termed as user-defined preference. ulility function is used (Hayes el al. 2022) to map the value vector to a scalar. One of the most frequently used utility functions the linear utility function (Hayes et al. 2022) i.e.
--) =" w. where w is on a simplex W E; w. = 1, w, 0.
The Conver Coverage Set (CCS) a finite convex subset of the PF (Hayes et al. 2022). CCS == (0" E BF s.t. Ya" E -PF oz w 2 v" ww. When a lincar utility function is used, the CCS comprises all non-dominated values, making the PF and CCS equivalent.
In the cutting-edge MORL methods Yang et al. 2019 Alegre et al. 2023 π is typically trained through learning a vectorized Q function that can adapt to any preference weight vector. These methods still suffer from poor sample efficiency as they search for the CCS by starting from the original point in solution space. See Figure [)), traditional MORL method starts from the red point, and searches to expand to the orange CCS. Idcally, by giving demonstrations, the training process can be transformed into Figure Πb). The policy is informed by domain knowledge, allowing it to start from the red points (demonstrations) This shift significantly reduces the distance between the "known" solution and the CCS, thereby reducing sample
4

---

Published in Transactions on Machine Learning Research (11/2024)
complexity of the training process. Another limitation is the demonstrations' sub-optimality. This introduces the risk of the policy being confined within the bounds of sub-optimal demonstrations 1 (c), where the green dot represents the policy approximated by a Q network). The gap between the red starting points and the CCS can be challenging to bridge the policy solely satisfics about meeting the initial demonstration standard. the policy may chance upon belter Irajectories, when the policy turns to this better guidance, it will exceed the initial demonstrations. Figure d illustrates this, where the dot alternating between red and green represents demonstration identified by the policy. Importantly, these new demonstrations are discovered by the agent itself. The initial demonstrations serve merely catalyst. This concept is formalized as the self-evolving mechanism.
b.
d.
Objective 2
Objective
Objective
Objective
Objective
Objective 1
Objective
Objective 1
Start point
Solution by Q network
New point by
Non-optimal solution after training Iteration
Figure 1: (a) Training process of traditional MORL; (b) Training process start from demonstration; (c)
DG-MORL without self-evolving; (d) DG-MORL with self-evolving.
4 Demonstration-guided multi-objective reinforcement learning
In this section, we present the DG-MORL algorithm. We begin by addressing the demoosstratioon-preeeeeene misalignment, followed by introducing the self- eelfiing mechanism and then we address demonstration deadlock and the sub-optimal demonstration problem. Finally, we provide an in-depth description of the DG-MORL algorithm, accompanied by theoretical analysis.
4.1 Demonstrations corner weight computation
When addressing the issue of Demonstration- Preeeernnce Misaligmment, it is noteworthy that single demon- stration can be optimal under an infinite number of preference weights as the weight simplex is continuous. Sampling and iterating through the entire simplex of preferences is computationally expensive and impractical. To mitigate this challenge, we utilize the concept corner weights (Roijers) 2016) i.e. denoted as Weorners to discretize the contimuous simplex into several critical points where the policy undergoes essential changes.
We now give clear explanation of the corner weights. When using different weights scalarize the return get different scalar values. If we plot these scalar values against the weights, get
of the objectives, piecewise linear convex curve or surface. Corner weights are specific combinations of weights where the slope of this curve changes. In simpler terms, they are the "corners" or "edges" on the graph of the scalarized value function. As shown in Figure 2 the points depicted by the blue lines are the corner weights according to the curve, the y-axis is the return value while the x axis is the weight objective (assuming this is a 2-objective problem). These points are significant bocause they represent weights where the optimal strategy or policy change. According to Roijers (2016), that the maximum possible improvement the set of known returns occurs at one of the corner weights. This means that if we're going to find better policy, it's most likely to happen when we look at these specific weights. See more rigorous definition of the corner weights in Definition |
Definition 1. (Definition 19 of Roijers 2016))
5

---

Published in Transactions on Machine Learning Research (11/2024)
Given set of value vectors v, we con define a polyhedron P by:
P= f= A ( - 0, Σ w. w. 2 vi}
(1)
where V+ represents a matris in which the elements of the set serve as the row vectors augmented by -1 as the column vector. The element x=(w1 Wa ow) in consists of the element of the weight vector and its scalar return value. Corner weights refer to the weights located at the vertices of P.
By iterating conduct the demonstrations, denote the convex coverage set of the demonstration results as CCSa. The corner weights effectively partition the solution space and classify the policies. The choice of the opti- mal policy maximizing utility for a given weight varies when traversing across corner weights. By computing Weorner on CCSs, we can assign corresponding weights to the demonstrations. Employing corner weights efficiently reduces the computational burden by calculating Weorner rather than Additionally, the significance of using corner weights in training is emphasized by the theorem:
get set of vectorized returns Va. We
finite set infinite weight simplex W.
15
14
Theorem 1. (Theorem of Rotjers (Rojjers There is a corner weight w that marimizes:
2016
A(w) CCS max 2
(2)
V 13
TEll
where ((w) is the difference between the utility of optimal policy given weight w and the utility ob- II, where denotes the current. agent's policy set. Theorem [ suggests that training along corner weight allows for faster learning of the optimal policy (sct), as it offers the potential for achieving maximum improvement. Assuming the prior demonstration is the known optimal solution, by replacing the CCS in equa- tion with CCS2, Equation 2 can be rewritten as: (d(w) Vew CCS max W arg max A(w)) wEW.ormer It further exploits the thcorem to select the corner weight from Woorner that denotes the largest gap between the performance of the current policy and the demonstration to increase the training efficiency.
12
tained by a policy
11
0.0
0.2
0.4
0.6
0.8
1.0
w1
Figure 2: Corner Weights Showcase: The points de- picted by the blue lines are the corner weights according to the curve, the y-axis is the return value while the axis is the weight of objective (assuming this is 2-objective problem).
xEll
We have elucidated the calculation the corner
weights set and the selection of an appropriate candi- date corner weight for aligning the demonstration. With this approach, the issue of demonsttrtionnppeefrrnee misalignment is addressed.
4.2 Self-evolving mechanism
An inaccurate assumption exists that "the use of demonstrations should naturally lead to performance improvements". It is too optimistic sub-optimal demonstrations easily impact the training process and cause low performance. We propose self-cvolving mechanism to tackle the problem of sub-optimal initial demonstrations. It is designed to improve upon given demonstrations for better guidance. This is another main contribution of our work. In the following sections, the demonstrations are referred to as guide policies. Although demonstration generally does not correspond to a policy, it can be considered fixed, hard-codod policy that merely exccutes actions regardless of the state observation. The policies developed by the agent are called exploration policies.

---

Published in Transactions on Machine Learning Research (11/2024)
We use the notation I1 to represent the set of guide policies. Since the exploration policy set is approximated by a single neural network approximator, we denote it as T,2 { Vvu, W). It is conditioned on the input preference weight vector and serves as the policy to be improved via learning, The agent interacts with the environment during training with a mix of guide policy and exploration policy, we denote this mixed policy as и.
The self-evolving mechanism is designed continuously update and improve the guide policy set Hg When the agent utilizes the mixed policy я, it may discover better demonstrations. When new, superior demonstrations are found, they are dynamically incorporated into the repository and treated new curriculum guide to the agent. This enables more effective learning from these superior demonstrations rather than persisting with initial data. The newly discovered demonstrations are closer to the optimal policy. Even when starting with sub-optimal demonstrations, as long as they guide the agent to valuable regions within the state space, the agent can begin exploring from there and identify better solutions. This progression makes the guide policy gradually move closer to the optimal policy, thereby mitigating the impact of sub-optimal data. Moreover, the agent might uncover demonstrations that were not initially provided. These new demonstrations can introduce a fresh set of corner weights, enriching the guiding curriculum and addressing the demonstrution deadlock. As learning progresses, less effective demonstrations are phased out, allowing the demonstration set to evolve. We symbolize this process as IIg + II' This process, while it involves updating the learning policy based on new demonstrations, does not strictly align with the traditional definitions of on-policy or off- off-policy training.
4.3 Multi-stage curriculum
To make full use of the demonstration, the mixed policy æ is mostly controlled by the guide policy Tg at the beginning of training. This can navigale the agent to a more promising state than randomly exploring. The control of the guide policy is diluted during training and taken over by the exploration policy. However, swapping the guide policy with the erploration policy introduces the challenge of the policy shift. A multi-stage curriculum pattern, suggested by Uchendu et al. (2023), is implemented. This strategy involves gradually increasing the number of steps governed by the exploration policy, facilitating a smoother transition and mitigating the policy shift issue.
4.4 Algorithm
Compared with JSRL UChendu et al. 22223 DG-MORL introduces the integration of the corner weight support to align the provided demonstration with proper preferences; The self-evoloing mechanism of DG- MORL substantially reduces the need for numerous demonstrations to attain strong performance]] Another departure is that DG-MORL entails a controlling swap between the set of guide policies to one exploration policy. Given a preference weight vector w, a guide policy upon which the exploration may gain the largest improvement is selected as shown in Equation
Kg = arg maxA (w)) arg max " - uH xElI,
(3)
llll,
We provide detailed explanation of the DG-MORL in Algorithm I To initiate the process, a set of initial demonstrations is evaluated, and the return values are added to CCSa. Dominated values are subsequently discarded, and assume that these retained values represent "optimal" solutions. The corner weights are then computed based on these value
Next, suitable guide policy is selected by using Equation The episode horizon is denoted as H. The guide policy control scale is denoted as h, ie. for the first h steps, the guide policy controls the agent, while for the next H - h steps, the exploration policy is performed. The erploration policy's 's performance is periodically evaluated. When improved performance is demonstrated,, the parameter is reduced to grant the exploration policy more controllable steps. The superior demonstration is then added to the demonstration repository,
JJSRL necossitates large number of c.g. 2000 demonstrations. We utilize the implementation from Alegre al. (2023) s which relies on the pyeddlib library http://gtthb...oommmmmnttteee/// identify the vertices polyhedron P.
7

---

Published in Transactions on Machine Learning Research (11/2024)
enabling the demonstration set to evolve autonomously (self-cvolving). The CCS is periodically updated to eliminate dominated solutions. A new set of corner weights is computed based on the updated CCS.
In certain instances, the guide policy may be so advanced that it becomes challenging for the exploration policy to exceed its performance. This scenario can restrict the rollback steps and therefore limit the exploration policy's is touching scope of states, consequently impeding the learning efficiency. For example, the agent could find itself unable to progress beyond the imitial stages of learning a solution due to the high difficulty level of the guide policy. To address this issue, introduce the concept of passing percentage, denoted B E [0, 1]. This strategy involves setting comparatively lower performance threshold initially, making it more achievable for the agent to learn. This threshold is incrementally raised, thereby progressively enhancing the agent's 's performance.
Algorithm DG-MORL Algorithm
1: Input: prior demonstration Ng. pass threshold ,3 2: Evaluate the Nl. add demonstration results to CCSa
3:
CCS CCS,
while max training step not reached do Get corner weight set Weorner based Equation 1 with CCS Given Weorner; gg get candidate weight we based on Equation With w. and H, sample the guide policy Rg based on Equation Calculate the utility threshold: ug = Un, We
8:
9:
The guide policy Fy controllable scale = H, ie. can control the whole episode. while h 2 0 do
10:
[0:h]
[++1:9]
11:
Set mix policy æ = Ti + me
12:
Roll out n, gather the experience
13:
Train explore policy 0r
14:
Evaluate if u > Ug then
15:
16:
Add the better value vector vn to CCS h + Ah, Ah is the rollback span end if
17:
18:
19: end while
20: Remove dominated solutions, update CCS and Nl
21: Update the curriculum pass threshold 3
22:
end while
& Te can trained with any MORL algorithm, we use GPI-LS this work.
4.5 Theoretical analysis
We provide a theoretical analysis the lower bound and upper bound on the sample complexity of the algorithm
4.5.1 Lower bound discussion
When the exploration policy (-initialized (all Q-values are zero initially), a non-oplimism-based method, e-g. e-greddy, suffers from exponential sample complexity when the horizon expands. We extend the combination lock (Koenig & Simmons 1993 Uchendu al.) 2023) under the multi-objective setting (2 objectives), and visualize example in Figure
In the multi-objective combination lock example, the agent encounters three distinct secnarios where it can earn & non-zero reward vector, each scenario is determined by a specific preference weight vector. The three preference weight vectors are [1, 0], [0.5, 0.5], and [0, 1]. These vectors correspond to the agent's varying preferences towards two objectives: When the agent demonstrates an extreme preference for objective 1, it may receive a reward vector of [1, 0]. Conversely, if the agent shows an extreme preference for objective it could be awarded reward vector of [0. 1].
8

---

Published in Transactions on Machine Learning Research (11/2024)
h 1
h
=2
h =3
H-1
h=
ag
d1
dct
d
/ bulance
S1
So1
S1
S
balance
S02
So2
So2
S
So2
0,2
a22
ag
Mutation Point
Figure 3: Lower bound example: MO combination lock
In the scenario where the agent's preference is evenly split between the two objectives, as indicated by a preference vector of [0.5,0.3 0.5]. there's a possibility for it to obtain a reward vector of [0.5, This outcome reflects a balanced interest both objectives. for the agent vary significantly based on its preference weights and corresponding actions. When the preference weights are set to [1,0], the agent's goal is to reach the red state 6.1t To achieve this, it must consistently perform the action 4a1 from the beginning of the episode, which enables it to remain in the red stale and subsequently receive reward feedback of [1, OJ. Similarly, with preference weights of [o. 1], the agent needs to execute the action 422 right from the start to eventually reach a reward vector of [0, 1]. For balanced
this multi-objectivy combination lock scenario, the outcomes
preference of 0.5]. the agent is expected to take a specific action, batance at step H 1 to transition to the state balancesh and receive a balanced reward of 0.5). This time point is referred point. If the agent dcrails from these prescribed trajectorics, only receives a reward of [0, 0]. Morcover, any improper action leads the agent to blue state, from which cannot. return to the correct trajectory (derailment).
as the mutation
As the agent is O-initialized, it needs to select an action from uniform distribution. The preference given is also sampled from a uniform distribution. Then if the agent wants to know the optimal result for any of the three preferences, it needs at least 3d-1 samples. If it thoroughly knows the optimal solution one of the three preferences, it can achieve an expected utility of 0.3. There exist several (K) paths in this MOMDP instance delivering good rewards. To find all good policies, the agent needs least see K.3 3 samples. We formalize this in Theorem 2
The scenario where preferences precisely match the three specific weights of [1,0], [0.5, 0.5], and [0, 1] is considerably rare, as preferences are typically drawn from an infinite weight simplex. In practical training situations, encountering a variety of preferences can lead to conflicting scenarios for the agent. This conflict. arises because different preferences may necessitate different actions or strategies, potentially causing the agent's policy to experience "forgetfulness" an inability consistently apply learned behaviors across varying preferences. Such situation complicates the training process and increases sample complexity.
Theorem [Koenig & Simmons 1993: Uchendu al., 2023) When using 0-initialized exploration policy, there exists MOMDP instance where the sample complerity is exponential to the horizon to see partially optimal solution and multiplicative exponential complezity to see optimal solution.
4.5.2 Upper bound of DG-MORL
We now give the upper bound of DG-MORL's regret. We start by making assumptions about the quality of the guide policy, i.c., it can cover some of the features that can be visited under the optimal policy, and the coverage rate is restricted by concentrate ability coefficient. Then we give Lemma to show that
9

---

Published in Transactions on Machine Learning Research (11/2024)
in sequential decision making setting, the difference between the optimal policy and the guide policy is bounded and this difference will not increase under the sell-evolution mechanism; but may decrease. Next, we provide several key definitions including Pareto suboptimality gap and sequence Pareto regret. Then we give a performance guarantec.
Assumption 1. [Uchendu et ol. 2023) Assume there exists feature mapping function Φ: + Rd, that for any policy n, Q" (s, , a) and T()) depends on 8 only through Φ(s). The guide policy " is assumed to cover the stotes visited by the optimal policy corresponding to the preference weight vector w.
(o(s) 8, 8,h, (6(8)) SC
sup
(4)
Tf (6())) and dr (d(s)) represents the probability distribution of feature o(s) being visited at time step under the preference weight w, while following the optimal policy x² or guide policy Xg respectively. C' is the concentratability coefficient (Rashidinejad et al., 2021 Guo et al. 2023). quantifies the extent which the guide policy 7g covers the optimal policy x² in the feature space and restricts the probability of the optimal policy visiting certain feature (bs) to not exceed C times the probability of the guide policy visiting that feature.
Nonethcless, Assumption is formulated within contextual bandit which does not sufficiently capture the complexities of sequential decision-making processes. We extend Assumption to the context of sequential decision- making and incorporates a self- evolving mechanism, thereby establishing Lemma As a result of the self-evolving mechanism, the guide policy "g progrossively enhances its performance over time.
Lemma 3. For a MOMDP, the sequence concentration coefficient (t) under preference weight w defined as the marimum value of the concentration coefficient. across all time steps while with the self-evolving mechanism:
(o(s))
max he (1.2...... } d, (6(s)) S Cseg(1, w) Csog(lo,
(5)
Furthermore, give the Poreto concentration coefficient Crareto So have:
max (t, w) S CPareto(to) NEW
max di. (d(s)) (1.2...... H. d, (d()) < CParrctott) S CPareto
(6)
We now give a few key definitions to calculate the upper bound of regret. One critical concept is the Pareto suboplimalily gap. This term refers to the measure of distance between the current policy's 's performance on a specific arm of the multi-objective contextual bandit (MOCB) and the true Pareto optimal solution for that arm.
Definition 2. (Pareto suboptimality gap [Lu et al. 2019) Let x be an arm in X, i.e. the arm set of the MOCB. The Pureto suboplimalily gap Ar is defined as the minimal scalar 2 so that x becomes Parelo optimal by adding all entries of its expected reward.
We now extend the Definition 2 to MOMDP to fit in sequence decision making process:
Definition 3. (Sequence Pareto Regret) In a MOMDP, given trajectory T (8), (1,a1, 82, a2, 8H), the expectation of sequence Pareto regret A seg for a finite horizon is:
A seg A EEA,, [1 + AE 82, 02 +... E (H-1,
(7)
where De, is the Pareto suboptimilly gap in state sh, defined as follows:
A min fo Va A, Bo' A, Q 2 (* + Q ()
(8)
Given that the policy Tg he interpreted as mixed palicy tailored specific scenarios, it retains its generality even when Hl is considered singular policy.
10

---

Published in Transactions on Machine Learning Research (11/2024)
In Equation Ageg is the expected sequence Pareto regret of policy It quantifies the cumulative performance loss of policy и relative to the Pareto optimal policy throughout the decision-making process. A5, is the Pareto regret at the initial state $1. E| 5h-1,0h-1 is the expected Pareto regret at state Sa which is conditioned on the previous state and action. By summing the initial Pareto regret and the conditional expected regrets at each subsequent state, we obtain the total expected sequence Pareto regret.
Equation 8 clarifies the conditions for Pareto dominance. U represents the multi-obeective Q-value of the optimal policy nT" for action in state 82. Q= a) is the multi-objective Q-value of policy for action in state Sh. indicates the minimum amount that needs to be added to all objectives for the current policy æ to reach the level of the optimal policy across all actions while is full ones vector whose dimensionality is the same as the reward space. For each action a E A of strategy there does not exist an action a' E A such that Q is no worse than (sh, a) + o across all objectives, and strictly better on at least one objective. We accurately measure the Pareto suboptimality when following π relative the optimal policy at state 8h-
Theorem 4. The expectation of the sequence Pareto regret Sng is the sum of the expectation of the Pareto regret for each slep.
EE
x seg
Σ
EIA
Equalion 9 indicntes the additivily of regrel, ie.. in MOMDP, the expected Parelo regret of a policy over the entire decision sequence can be decomposed into the sum of the expected Pareto regret at each time step.
We put the proof of Theorem 4 below.
Proof. Take expectations on both side:
1[ E[ E| [1) + B [22 + ... E|F[AT 9
According to the Law of total expectation:
E[A B|A E E +... E T sq
Σ B[D,5]
We can now give the upper bound of the sum of the sequence Pareto regret during the training for T rounds. Theorem 5. (Performance Guarantee) Assuming that the encironment satisfies the Markov property, and there crists a feature mapping function & such that for any policy æ both Q= a) and rTs) depend only on d(s). The DG-MORL algoritlmm guarantees that the guiding policy Tg progressively approaches the optimal policy n² through self-evolution mechanisms. f(t is an abstract regret function, depends on the specific algorithm we used to troin ne. During each training round E [T], the algoritim executes policy Te, and the sum of sequence Pareto regret P PRzzg(T) is bounded by:
T
PR (T) EA A < STHR RTHR C
(10)
t=1
where C= CPareto(to) + f(a), (Pareto and (to) are the Pareto concentration coefficient and regret function at the first round of training. This is to get a conservative upper bound as (Pareto(to) 2 CPareto because of the self-evolving mechanism and f(t) > f(t) because of the policy improvement. Rmax this 1-norm to get relative conservative and general upper bound
max,,0 [|r(s, a)ll,,
WWiil acknowledge that using preference weight might add specificity, we prefer the 1-norm relaxes the upper bound. This approach ensures that theorem remains valid
worst-case scenario.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Proof. of Theorem 5
Decomposing the regret into contributions from the guide policy Ty and the exploration policy Te. In the t-th round of training, the Pareto regret of the sequence of the mixed strategies næ can be divided into two parts:
(1). Regret from gg.t
hg
Tg. A
Tg.
séq
h=1
where hg the last time step controlled by Wg.t (2). Regret from Ne,t
Nest seg
E hmhy +1
A 5n
Re.t
Thus, the total regret is:
sog
F + A sog
Felt
With the help of the self-evoluing mechanism, Ty can improve overtime. According Lemma h timestep, the expected regret of g.t is:
Es [A S Rmax CCaarto(t)
In the worst case, the reward difference Rmax, and the difference in state distribution is controlled by CPoreto(t). In the worst case, each time step can cause the maximized regret. The total expected regret for gg.t is:
A S h Rumax Cpareto(t)
Assuming we use e-greedy strategy in the exploration policy Te. For the exploration policy Tes at each time step h, the expected regret of Te, is:
Ex, 1A.11 S f(() Rmax
This is because a random action is selected with probability which may lead to a reward loss of up to Rmax in the worst case. Therefore, the total expected regret for the exploration policy is:
A A (H - htf(t) Rmax
For T training rounds, the total regret is:
PRang (T)
(he Rmaxx Cparcto + (H - -h,) f(t) Rmmx))
where h2 is the number of time steps controlled by the guided policy during the t-th training round. During the training process, the number of time steps controlled by the guided policy decreases over time, i.c., hy decreases with I. Since (rareto and f(I) decreases with 4, we approximate it by CParcto(In) and f(lo), the value at the final round. We approximate that the steps controlled by Tg and all the maximized number of steps, i.c. H, the upper bound on the total regret is:
PR (T) S TTH Rmax C
where C CPrreto (to) f(to)
5 Experiments
In this section, we introduce the basclines, benchmark environments and metrics. We then illustrate and discuss the results. We have also provided the sensitivity analysis about the how different qualities and quantities of the initial demonstrations can influence the training process in the Appendix. Please note that, our DG-MORL does not require specific preferences be covered by demonstrations. In MORL, as some preferences may share the same optimal policy, the true PF is not known, it impossible cover all important preferences by demonstrations.
12

---

Published in Transactions on Machine Learning Research (11/2024)
5.1 Baseline algorithms
We use two state-of-the-art MORL algorithms as baselines: GPI linear support (GPI-LS) and GPI prioritized Dyna (GPI-PD) (Alegre et al. 2023). GPI-PD) algorithm is model-based and GPI-L. is model-free.
The GPI-LS and GPI-PD are based on generalized policy improvement (GPI) Puterman 2014 while the GPI-PD algorithm is model- based and GPI-LS model-free. The GPI method combines a set of policies as an assembled policy where an overall improvement is implemented (Barreto et al. It was introduced to the MORL domain by Alegre al. (2022). The GPI policy in MORL defined (gpr() E arg max maxo (s, a). The GPI policy is at least well as any policy in II (policy set). The aEAA =elf
2017 2020
GPI agent can identify the most potential preference to follow at each moment rather than randomly pick a preference as in Envelope MOQ-learning. This facilitates faster policy improvement in MORL. Furthermore, the model- mosel-based GPI-PD can find out which piece of experience is the most relevant to the particular candidate preference to achieve even faster learning. The two baseline algorithms GPI-LS and GPL-PD cannot. utilize demonstrations. There is not yet a MORL algorithm that can use demonstrations in training and DG-MORL is the first. one.
As indicated in the literature (Alegre et al. 2023), the Envelope SFOLS (Alegr et al., 2022), and PGMORL (Wurman et al. 2022) are consistently outperformed by the GPI-PD L.S. Therefore, if our algorithm can surpass GPI-PD/ LS, it can be inferred that it also exceeds the performance of these algorithms. We also use a no-self-evolving version of DG-MORL in the ablation study as basclinc.
algorithm (Yang et al. 2019
5.1.1 Implementation detail
The code of this work is available on http:/iiiaaoooooo0////////////// neural network architecture as the GPI-PD implementation in all benchmarks, i.c. 4 layers of 256 neurons in DST and Minecart, 2 layers with 256 neurons for both the critic and actor in MO-Hopper, MO-Ant and MO-Humanoid. We use Adam optimizer, the learning rate is 3 10-4 and the batch size is 128 in all implementations. As for the exploration, we adopted the same annealing pattern e-greedy strategy. DST, the e anneals from 0 during the first 50000 time steps. In Minecart, the e is annealed from 0.05 in the first 50000 time steps. For the TD3 algorithm doing MO-Hopper, MO-Ant and MO-Humanoid, we take zero-mean Gaussian noise with standard deviation of 0.02 to actions from the actor-network. All hyper-parameters are consistent with literature (Alegre el al. 2023
We use the same
The initial demonstration data are from different sources to validate the fact that any form of demonstration can be used as it can be converted to action sequences. The demonstration for DST is from hard-coded action sequences which are sub-optimal. Minecart uses the demonstrations manually played by the author, i.e. us, while the MO-Hopper, MO-Ant and MO-Humanoid, use the demonstration from a underdeveloped policy from the TD3 algorithm.
5.2 Benchmark environments and metrics
We conduct the evaluation on MORL tasks with escalating complexity: from an instance featuring discrete
states and actions, ie. Deep Sea Treasure (DST) (Yang et al. 2019
Alegre et al. 2023) to tasks with
continuous states and discrete actions, i.c. Minecart (Abels al. 2019:
Alegre ct al.] 2023) We also test it
in control tasks with both continuous states and actions, i.e. MO- Hopper (Basaklar et. al. 2023 Alegre et al.,
2023 MO-Ant, and MO HO--Hnnnoid. We assess the candidate methods using variety of metrics sourced
from Hayes et al. (2022.
5.2.1
Benchmark environment setting
The deep sea treasure (DST) environment is a commonly used benchmark for the MORL setting (Vamplew ct al. [2011] Abels et al. 2019 Yang et al., 2019 Alegre et al., 2023). see Figure 4 a). The agent needs make a trade-off between the treasure value collected and the time penalty. is featured by a discrete state space and action space. The state is the coordinate of the agent, i.e. [x,y]- The agent can move either left,
13

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 4: (a) Deep sea treasure; (b) Minecart; (c) MO-Hopper; (d) MO-Ant; (e) MO-Humanoid
right, up, down in the environment. The reward vector is defined as r = [Itreasures Fstepp: where Ttreasure is
the value of the treasure collected, and Tstep -1 is time penalty per step. The treasure values are adapted The discounted factor 7 = 0.99.
to keep aligned with recent literature Yang et al. 2019
Alegre et al. 2023
Minecart. The Minecart environment (Abels et al. 2019
Alegre al. 2023 see Figure 4b), is featured
with continuous state space and discrete action space. It provides relatively sparse reward signal. The agent needs to balance among three objectives, i.e. 1) to collect ore 1; 2) lo collect ore 2; 3) to save fuel. The minecart agent kicks off from the left upper corner, i.e. the base, and goes to any of the five mines consisting of two different kinds of ores and does collection, then it returns to the base to sell it and gets delayed rewards according to the volume of the two kinds ores.
This environment has a 7-dimension continuous state space, including the minecart's coordinates, the speed, its angle cosine value and sine value, and the current collection of the two kinds of ores (normalized by the capacity of the minecart). The agent can select actions from the 6-dimension set, i.e. mine, turn left, turn right, accelerate, brake, idle. The reward signal is a 3-dimensional vector consisting of the amount of the two kinds of ore sold and the amount of fuel it has used during the episode. The reward for selling ores is sparse and delayed while the reward signal of fuel consumption is a constant penalty plus extra consumption when doing acceleration or mining.
MO-Hopper, MO-Ant and MO-Humanoid are extensions of the Mujoco continuous control tasks from OpenAI Gym (Borsa et al. 2019). They are used in literature to evaluate MORL algorithm performance in robot control (Bsakllar et al. 2023 Alegre et al. 2023
MO-Hopper. The agent controls the three torque of a single-legged robot to learn to hop. This environment involves two objectives, i.e. maximizing the forward speed and maximizing the jumping height. Tt has a continuous Il-dimensional state space denoting the positional status the hopper's body components. This environment has a 3-dimensional contimuus action space, i.c. the different torques applied on the three joints. We the same reward function and discounted factor 7 0.99 (Alegre et. al. 2023
Pvelocity = Ux C
(11)
Theight 10(h himit) + C'
(12)
where Up is the forward speed, h and hinit are the hoppers relative jumping height, C= E, az is the a leedback of the agent healthiness.
MO-Ant. The ant robot is a 3D model comprising a freely rotating torso with four legs attachod. Each leg consists of two segments, making total of eight body parts for the legs. The objective to synchronize the movement of all four legs to propel the robot. forward (toward the right.) by exerting torques hinges that connect each leg's two segments to the torso. This configuration results in nine body parts and eight hinges in total that need to be coordinated for movement. This environment has an 8-dimensional continuous action space and 27-dimensional continuous state space (Schulman et al. 2015 discounted factor y = 0.99.
the eight
We use the
Ts = U2 +
(13)
ry by C
(14)
14

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Benchmark Environments
Environment
DST
Minecart
MO-Hopper
MO-Ant
MO-Humanoid
State space S
discrete
contimuous
contimuous
continuous
continuous
dim(S)
7
11
27
376
Action space A
discrete
discrete
continuous
continuous
continuous
(Am(
4
6
3
8
17
dim(R)*
2
3
2
2
2
Episode max horizon
100
30
500
500
500
Discounted factor
0.99
0.98
0.99
0.99
0.99
Passing Percentage
1
0.8-+0.99
0.6-+0.99
0.6-+0.99
Rollback Span
2
100
100
100
The number of objectives / the dimension of reward vectors.
Consistent with baselines from Alegre et al. (2023).
3 The number of steps rolled back when the found trajectory surpasses the given demonstration.
where Us and by is the x-velocity and the y-velocity, and hinst are the hoppers relative jumping height, is the a feedback of the agent healthiness.
-0O-Hmaana The 3D bipedal robot is engineered to mimic human movement, featuring torso (abdomen) equipped with two legs and two arms. Each leg is composed of three segments, simulating the upper leg, lower leg, and foot, while each arm consists of two segments, analogous to the upper and lower arms. The primary objective this simulation environment to enable the robot to walk forward as swiftly as possible while maintaining balance and avoiding falls. This enviroument has a 17-dimensional continuous action space and 376-dimensional continuous state space Tassa et al., 2012 We use the discounted factor y 0.99.
fforward U2 +
(15)
Teontrol 10 Σ control
(16)
where U2 is the forward spood, Σ control is the sum control cost of the joints is the a foedback of the agent healthiness.
To more effectively compare the exploration efficiency of our algorithm with baseline methods, we impose constraints episode lengths across different environments. Specifically, we set the DST environment to terminate after 100 steps, the Minecart environment to conclude after 30 steps, and the MO-Hopper, MO-Ant and MO-Humanoid to end after 500 steps. This limitation intensifies the difficulty of the tasks, compelling the agent to extract sufficient information from a restricted number of interactions. This constraint ensures more rigorous assessment of each algorithm's 's ability to efficiently navigate and learn within these more challenging conditions. The summary of benchmark settings is illustrated in Table I
5.2.2 Metrics
To thoroughly compare our algorithms with baselines, we use multiple metrics (Hayes et al. 2022).
Expected utilily (EU) (Zintgral al., 2015): Compared many other metrics, such as the hypervolume metric, EU is better suited for comparing different algorithms because they are specifically designed to evaluate agent's 's ability to maximize user utility, which is consistently our primary objective. It is calculated as (II) = wwww[max,ee w. The expectation in this work is the average over 100 evenly distributed weights from W.
Hyperoolume ZZtzzerr & Thiele 1999): The Hypervolume metric quantifics the volume in the value-space that is Pareto-dominated by set of policies within an approximate coverage set. We use reference point [0,0] in all benchmark environments except [-1,-1,-2] in Minecart.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Sparsity: To enhance the evaluation of Pareto front approximations, Xu al. (2020) introduced method that integrates the hypervolume metric with sparsity metric. This approach aims to assess solution sets not only for their expansive coverage of the Pareto front but also for their distribution and separation, providing users with diverse and well-distributed options across the value space. The sparsity is defined as: Sporsity(S)) ()= = Dز objectives, (i) is the i-th element of the sorted solutions on j-th objectives in S.
(5 1))², where S is the PF approximation, m is the number of
Hypervolume SSparsity: As Pareto front approximations with higher hypervolume metric while maintaining a lower sparsity metric are deemed better (Hayes et. al. 2022). we assess the ratio of Hypervolume and Sparsity as another metric.
PF Approzimation: We also present a visualization of the PF approximation to offer clear understanding of the policy set that has been learned.
5.3 Result
In this section, we show the results of the evaluations of EU, PF, Hypervolume, Sparsity, and the ratio of Hypervolume and Sparsity. We conduct 5 independent runs with randomly picked seeds. For the initial number of demonstrations, we used 10 for DST, for Minecart, and 5 for MO- JO-Hopper, MO-Ant, and MO-Hundnoid. We have conducted a scnsitivity study of different initial demonstration quantitics is in Appendix A.I The sensitivity study of qualities of the guidance policy is in Appendix A.2 The experiments are run machine with 12th Gen Intel(R) Core(TM) i9-12900 CPU, NVIDIA T1000 graphic card and 32 GB memory. For the number of initial demonstrations, for Minecart, and 5 demonstrations for MO-Hopper, MO-Ant and MO- KO--Hmmanod.
use 10 demonstrations for DST, 7 demonstrations
5.3.1 Main experiment result
The evaluation of the EU of the training process is shown in Figure 3 (the evaluation frequency is shown in the title of each diagram). Demonstrations from different sources are used to train DG-MORL, i.e. hard-coded action sequences in DST and Minecart, and the underdeveloped policy of a TD3 agent in MO-Hopper, MO-Aut, and MO-Humanoid. This showcases our hypothesis that DG-MORL can work with demonstrations from any source. All evaluations of DG-MORL are solely with its ezploration policy. To statistically and reliably evaluate the results of EU, we use stratified bootstrap confidence intervals (with 50000 runs) and interquartile mean (IQM) (Agarwal et al. 2021 i.e. 1QMs are depicted with solid lines, while 95% confidence intervals are illustrated by color-matched shaded areas.
In Figure 5 the result shows that DG-MORT. has learned a better or at least competitive policy than the baselines in the first evaluation in most benchmarks. An exception occurs in MO-Ant, where DG-MORL starts with an EU approaching 0. This may be due to the more conflicting objectives in MO-Ant, where initial demonstrations display contradictory behaviors, e.g. exclusively moving along the x-axis y-axis. Starting from a demonstration committed to single direction can impede progress in improving the policy going in the alternative direction. Conversely, DG-MORL in MO-Humanoid exhibits a notably strong start, attributed to its objectives of forward movement and control cost reduction, which approximately align as a single goal of advancing with minimal control energy (similar to MO-Hopper). This alignment results in less contradictory demonstrations than MO-Ant, as further evidenced by the sparse points on the PF illustrated in Figure This phenomenon indicates that. the degree of conflict among objectives can be potential factor influencing the training process of DG-MORL. However, after training DG-MORL achieves better performance than all of the baselines, surpassing the initial demonstrations and reaching the standard of the best achievable performancd 3 In comparison, GPI-PD also attains a commendable level of performance in the early stages, but during the whole training process, it shows larger variations and inferior
Foor DST and Minecart, where known PF exists, the black dashed line indicating the best performance represents the results from the actual PP. As MO-Hopper, MO-Aut, and MO-Humanoid do not have known PFs, we trained singlo-objecctve TD3 (SO-TD3) agents with preferences [1, 0), [0.75, 0.25), [0.5, 0.5), [0.25, 0.75), [0.1]. Each agent is trained for 1.5M steps, therefore the training budget 7.5M stops. We the result the best performance (Note that these "bost" performances can be exceeded they only from policy trained with significantly more steps). Our algorithm trained by 0.225M steps, which takes only 3% training budget compared SO-TD3.
16

---

Published in Transactions on Machine Learning Research (11/2024)
final performance than DG-MORL. The GPI-LS algorithm exhibits significantly greater variances in all environments than DG-MORT, suggesting reduced robustness. It is also noted that GPI-PD's superiority over GPI-LS diminishes and is eventually surpassed by GPI-LS in the MO-Humanoid environment, highlighting the challenges model-based MORL faces in complex tasks.
DST (eval. per 4k steps)
b. Minecart (eval. per 10k steps)
MO-Hopper (eval. per 15k steps)
Expected Utility
0.2
200
5.0
150
4.5
100
4.0
5
10
3
5
9
1101110 13
8 10 1101122 13
Timesteps (4k steps)
Timesteps (10k steps)
Timesteps (15k steps)
MO-Ant (eval. per 15k steps)
e. MO-Humanoid (eval. per 15k steps)
Expected Utility
300
400
DG-MORL (ours)
200
300
DG-MORL (no self-evolution)
100
GPI-PD
GPI-LS
100
Initial Demonstration
2
10 15
10
Timesteps (15k steps)
Timesteps steps)
Best Performance
Figure 5: EU evaluation (a) DST; (b) Minecart; (c) MO-Hopper; (d) MO-Ant; (e) MO-Humanoid. The numbers in brackets on the x-axis represent the interval of timesteps between each evaluation. The result shows that DG-MORL has learned a better or at least competitive policy in most environments.
In the DST, DG-MORL successfully reconstructs the ground truth PF. We therefore did not conduct the PF visualization and Hypervolume, Sparsity evaluation for DST. For Minecart, where the PF is 3- dimensional and challenging to interpret visually and the environment is stochastically initialized, we have confined representation to only EU metric. Figure G illustrates the PF approximation in the other three environments. In MO-Hopper and MO-Ant, DG-MORL's PF dominates both baseline approaches. Although in MO-Humanoid, DG-MORL's PF does not dominate that of GPI-PD, it does dominate GPI-LS. The PF approximation results for MO-Humanoid reveal an interesting observation: the solutions for each method tend to cluster, suggesting that despite being multi-objective task, MO-Humanoid behaves similarly to single-objective task. This observation underscores the need for more thoughtfully designed MORL benchmark environments that can more effectively differentiate the performance of various algorithms. Furthermore, as indicated by Figure 5 and Table 2 DG-MORL demonstrates superior EU, higher Hypervolume, lower Sparsity, and improved Hypervolume /Sparsity ratio compared to GPI-PD, suggesting a significantly enhanced quality of the solution set. The PFs reveal that DG-MORL surpasscs the initial demonstrations in all three environments, while one instance of the best. demonstrations in MO-Ant out perforrmm some solutions of DG-MORL.
Table 2 shows the results for Hypervolume and Sparsity + DG-MORL records the highest Hypervolume across ratios do not consistently
most environments. Although DG-MORL's Sparsity and Hypervolume/Spanne achieve top marks, they still maintain competitive levels. In instances where DG-MORL's Sparsity and Hypervolume//pparitty are exceeded, GPI-LS achieves the best results. Notably, DG-MORL (no self-evolving) sees a Sparsity of 0 in Mumannd,,,,, attributable the single point on its PF, which limits interpretative value. Nonetheless, while GPI-LS has higher Sparsity, this comes at the cost of overall performance in terms of EU, Hypervolume, and PF coverage.
6The t/4 means the higher/lower the value, the better the performance.
17

---

Published in Transactions on Machine Learning Research (11/2024)
a. MO-Hopper
MO-Ant
c. MO-Humanoid
300
350
jump on 2-axis
control reward
DG (ours)
200
Avelciity
400
DG-MORL (no selt-evaliving)
150
100
200
250
350
GPI-PD
GPI-LS
50
150
300
Initial Demo
100
Best Performance
50
250
200
250
300
350
400
480
500
520
540
forward on x-axis
x-velocity
run forward
Figure PF approx. (a) MO-Hopper; (b) MO-Ant; (c) MO-Humanoid; In MO-Hopper and MO-Ant, DG-MORL's PF dominates both baseline approaches. Although in MO-Humanoid, DG-MORL's PF does not dominate that of GPI-PD, it does dominate GPI-LS.
Table Evaluations on Hypervolume and Sparsity
Method
Hypervolumef
Sparsity
Hypervohume/Sparsityy
MO-Hopper
DG-MORL
68343.44
36.95
1849.49
GPI-PD
52710.74
57.92
910.03
GPI-LS
51545.64
24.87
2072.96
DG-MORL (no self-evolving)
58508.60
163.55
357.74
Initial Demo
35915.02
29010.59
1.24
Best Performance
53766.39
4603.11
11.68
MO-Ant
DG-MORL
136021.51
607.24
224.0
GPI-PD
103513.67
1292.10
80.11
GPI-LS
110327.95
762.32
144.73
DG-MORL, (no self-evolving
128635.06
1073.22
119.86
Initial Demo
42682.51
25469.52
1.68
Best Performance
133711.36
101590.39
1.32
MO-Humanoid
DG-MORL
236740.17
0.43
545903.71
GPI-PD
139359.20
2.80
49784.78
GPI-LS
125137.57
0.28
451889.30
DG-MORL (no self-evolving)
200264.31
0.00
N/A
Initial Demo
145364.09
149.35
973.28
Best Performance
146949.70
523.19
280.87
5.3.2 Ablation study
We conduct ablation study on DG-MORL by excluding the self-evolving mechanism. The agent relies solely the initial demonstrations, opposed to progressively enhancing the guidance demonstration with self-generated data. The result. is shown in yellow in Figure Figure 6 and DG-MORL (no self-evolving) in Table 2 The absence of this mechanism makes DG-MORL fail to reach the performance levels of the fully equipped DG-MORL setup (except DST), however, it still outperform the baselines and the initial demonstrations in most benchmarks (excopt the Mumanod,,,,, the instance without self-cvolving saw a drop in the middle.). The findings from the ablation study confirm the following: The self- evolving mechonism is effective. 2. Generally, using demonstrations to guide MORL. training leads to improved performance.
18

---

Published in Transactions on Machine Learning Research (11/2024)
6 Limitation
DG-MORL has pioncered the use of demonstrations to inform MORL training. However, our method has some limitations. One limitation of this work is that DG-MORL is only applicable to linear preference weights. It needs further improvement to work with non-linear preferences. Another limitation is that DG-MORL, has not been evaluated in a non-stationary environment. is worthwhile to explore how a demonstration can guide the training process in non-stationary MORL scttings. We will leave these to the future. Additionally, since DG-MORL in the MO-Ant scenario does not achieve jump-start learning, exploring how to effectively use demonstrations to aid training in environments with relatively contradictory objectives is worth further investigation.
7 Conclusion
We proposed the pioneering DG-MORL algorithı that uses demonstrations to enhance MORL training efficiency. Our algorithm has overcome the challenge of sparse reward, the hard beginning of training, and the possible derailment from a complex environment. We address the challenges of demonstration perettrrrtttoo--eeeeeneee preference misalignment and demonstration deadlock by implementing corner weight computation of the demonstrations. We introduced the self-euolving mechanism to handle sub-opt demonstrations or the case with limited number of demonstrations. Empirical evidence confirms DG-MORL's superiority over state-of-the-art across various environments. Additionally, experiments highlight DG-MORL's robustness by excluding self- evolving and under other unfavoured conditions. We also provided theoretical insights into the lower bound and upper bound of DG-MORL's sample efficioncy.
Acknowledgments
This work was financially supported by the Irish Research Council under Project ID: COIPG/2022/21 The authors have applied CC BY 4.0 public copyright license to the Author Accepted Manuscript version arising from this submission to ensure Open Access. We would also like to express sincere appreciation to the reviewers for their insightful and constructive feedback, which has played a crucial role in refining our research and strengthening the final manuscript.
References
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International conference on machine learning, pp. 11-20. PMLR, 2019.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-2983 2021.
Lucas N Alegre, Ana Bazzan, and Bruno C Da Silva. Optimistic linear support and successor features basis for optimal policy transfer. In International Conference on Machine Learning, pp. 394-413. PMLR, 2022.
Lueas N Alegre, Ana LC Bazzan, Diederik M Roijers, Ann Nowé, and Bruno C da Silva. Sample-effeiint multi-objective learning via generalized policy improvement prioritization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 2003-201 2023.
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Proceedings of the 40th International Conference Machine Learning, pp. 1577-1594, 2023.
Andró Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
19

---

Published in Transactions on Machine Learning Research (11/2024)
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Acodemy of Sciences, 117(48):30077-3000 -3008,, 2020.
Andrew G Barto, Satinder Singh, Nuttapong Chentanez, al. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning, volume 112, pp. 19. Citescer, 2004.
Toygun Basaklar, Suat Gumussoy, and Umit Ogras. Pd-morl: Preference-driven multi-objective reinforce- ment learning algorithm. 2023.
Diana Borsa, Andre Barreto, John Quan, Danicl J Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul. Universal successor features approximalors. In International Conference on Learning Representations, 2019.
Jaedeug Choi and Kee- Eung Kim. Nonparametric bayesian inverse reinforcement learning for multiple reward functions. Advances in neural information processing systems, 25, 2012.
Jaedeug Choi and Kee-Eung Kim. Bayesian nonparametric feature construction for inverse reinforcement learning. In Twenty- thintyy-Third International Joint Conference Artificial Intelligence. Citescer, 2013.
Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guided reward shaping for reinforcement learning under sparse rewards. Advances in Newral Information Processing Systems, 35: 5829-5842, 2022.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Chune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv: 1atp:9901.10000 10995, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590((847):5800-66) 2021.
Siyuan Guo, Lixin Zou, Hechang Chen, Bohao Qu, Haotian Chi, S Yu Philip, and Yi Chang. Sample efficient. oflline-o-online reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 2023.
Conor F Hayes, Roxana Rádulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstracten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Aulonomous Agents and Mulli- Agent. Systems, 36 (1):26, 2022.
Johan Källström and Fredrik Heintz. Tunable dynamics in based simulation using multi-objective reinforcement lcarning. In Adaptive and Learning Agents Workshop (ALA-19) at AAMAS, Montreal, Canada, May 13-14, 2019, pp. 7, 2019.
Sven Koenig and Reid G Simmons. Complexity analysis of real-time reinforcement learning. In AAAI, volume 93, pp. 99-105, 1993.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for oflline reinforce- ment learning. Advances in Neural Information Processing Systems, 19:1179-119)) 2020.
Yuxi Li. Reinforcement learning in practice: Opportumities and challenges. arXiv preprint arXiv:2202.11290 2022.
Junlin Lu, Patrick Mannion, and Karl Mason. A oulti-objective multi-agent deep reinforcement learning approach to residential appliance scheduling. IET Smart Grid, 5(4):260-28) 280, 2022a.
Junlin Lu, Patrick Mannion, and Karl Mason. Inferring preferences from demonstrations in multi-objective reinforcement learning; A dynamic weight-based approach. Adaptive Learning Agent Workshop @ AAMAS 2023 London, 2023.
Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Multi-objective generalized linear bandits. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3086, 2019.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Trpan, Mohi Khansari, Dmitry Kalashnikov, et al. Aw-opt: Learning robotic skills with imitation andreinforcement at scale. In Conference on Robot Learning, pp. 1078 1088. PMLR, 2022b.
Patrick Mannion. Knowledge-based multi- nulti-obeective malti-agen reinforcement learning. PhD thesis, University of Galway, 2017.
Shie Mannor and Nahum Shimkin. The steering approach for multi-criteria reinforcement learning. Advances in Neural Information Processing Systems, 14, 2001.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. ar preprint arXic:1312.5602, 2013.
Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arviv preprint ar 2Xii::6100027000 0270002777 2016.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robolics and aulomation (ICRA), pp. 6292 6299.. TEFF, 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint 2Xii:2006.09330 2020.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transfrrmations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278-287. Citeseer, 1999.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self- delf-imitation learning. In International Conference on Machine Learning, pp. 1878-3887. PMLR, 2018.
Matteo Pirotta, Simone Parisi, and Marcello Restelli. Multi-objective reinforcement learning with continuous parcto frontier approximation. In Proceedings of the AAAI conference artificial intelligence, volume 29, 2015.
Athanasios S Polydoros and Lazaros Nalpantidis. Survey of model-based reinforcement learning: Applications robotics. Journal of Intelligent & Robotic Systems, 86(2):158-173, 2017.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic progromming. John Wiley & Sons, 2014.
Aravind Rajeswaran, Vikash Kumar, Abhishck Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 70977791100 10087, 2017.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAT, volume 7, pp- 2586 2591, 2007.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:11702 11716, 2021.
Diederik M Roijers.
Multi-objective dccision-thcoretic planning. AI Matters, 2(4):11-12, 2016.
Dioderik M. Roijers, Shimon Whiteson, and Frans A Olichock. Linear support for multi-objective coordination graphs. In Proceedings of the 2014 International Conference Autonomous Agents and Multiagent Systems, pp. 1297-1304, 2014.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. or Xiu preprint ar 177.637777 2017.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IBEF/RSJ International Conference on Intelligent Robols and Systems, pp. 4906 4913. IEEE, 2012.
Gerald Tesauro, Rajarshi Das, Hoi Chan, Jeffrey Kephart, David Levine, Freeman Rawson, and Charles Lefurgy. Managing power consumption and performance of computing systems using reinforcement learning. Advances in neural informalion processing systems, 20, 2007.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In Internalional Conference Machine Learning, pp- 34556-34583. PMLR, 2023.
Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjoctive reinforcement lcarning algorithms. Machine learning, 84:51-80, 2011.
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 191-199. IEEE, 2013.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pielquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.0881, 2017.
Guojian Wang, Faguo WI, Xiao Zhang, Jianxiang Liu, al. Learning diverse policies with soft self-generated guidance. International Journal of Intelligent Systems, 2023, 2023.
Peter Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):22 228, 2022.
Jie Xu, Yunsheng Tian, Pingehuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guide multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, pp. 10607-10616. PMLR, 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective re- inforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
Luisa M Zintgraf, Timon V Kanters, Diederik M Roijers, Frans Oliehoek, and Philipp Beau. Quality assessment of morl algorithms: A utility-based approach. In Benelearn 2015: proceedings of the 24th annual machine learning conference of Belgium and the Netherlands, 2015.
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Compulation, 3(4):257 271, 1999.
22

---

Published in Transactions on Machine Learning Research (11/2024)
A Sensitivity study
It is widely acknowledged in machine learning that both the quantity and quality of data play crucial roles in enhancing the outcomes. To understand the extent and manner in which each of these factors impacts the learning process, we have carried out a scries of scnsitivity studies. We continue to utilize the same five benchmark environments and the EU metric. However, the focus shifts to a comparison among DG-MORL agents rather than among other baseline algorithms.
In the first part, we explore the sensitivity of the DG-MORL algorithm to the quantity of initial demonstrations. The second part of this section examines how the quality of data impacts the algorithm's 's performance. For better comparison, the GPI-PD result is included for reference depicted by blue dashed line.
A.1 Sensitivity to initial demonstration quantity
and shows DG-MORL algorithm's sensitivity to the quantity of initial demonstrations in the three benchmark environments separately. The figures presented depict the average results of 5 independent runs for the sake of clarity. It should be noted that scenarios without any initial demonstrations are not taken into account, the presence of such demonstrations is a fundamental prerequisite for DG-MORL. We ensure that the process commences with a minimum number of demonstrations equivalent to the number of objectives, e.g. for DST, as there are 2 objectives, the least amount of demonnstratin provided should be 2 as well.
Figure 7
Though over time, there are more demonstrations added by the self-evolving mechanism, the outcomes of our research reveal clear relationship between the quantity of initial demonstrations and the performance of the DG-MORL algorithm, particularly evident in the DST environment. Specifically, there is a positive correlation observed: as the number of initial demoustrations increases, there is a notable enhancement in training efficiency and a corresponding improvement in policy performance.
This trend is consistent in the Minecart enviromment as well. Here too, the quantity of initial demonstrations plays a sigmificant role in influencing the performance the DG-MORL algorithm. The increased number of initial demonstrations provides more comprehensive guidance and information, which in turn facilitates more effective learning and decision-making by the algorithm. This consistency across different environments underscores the importance of initial demonstration quantity as key factor in the effectiveness of the DG-MORL algorithm.
An intriguing observation from the Minecart experiment was that an increase in the number of initial demonstrations actually led to lower performance at the beginning of training. This phenomenon was not evident in the DST experiment, suggesting a unique interaction in the Minecart environment. We hypothesize that this initial decrease in performance is attributable to the complexity of the Minecart environment when paired with large set of demonstrations. In such a scenario, the agent may struggle to simultancously focus multiple tasks or objectives presented by these demonstrations, leading to temporary dip in performance at the early stages of training. This is indicative of an initial overload or confusion state for the agent, it tries to assimilate and prioritize the extensive information provided by the numerous demonstrations. However, as training progresses and the agent becomes more familiar with the environment and its various tasks, it starts to overcome this initial challenge. With time, the agent effectively integrates the insights from the demonstrations, leading to improved performance. This dynamic suggests that while a wealth of demonstrations can initially seem overwhelming in complex environments, they eventually contribute to the agent's 's deeper understanding and better performance, underscoring the importance of considering both the quantity and nature of demonstrations in relation to the environment's complexity.
The results from the MO-Hopper, MO-Ant further support the notion that a greater number of initial demonstrations tends to yield better performance. However, in MO-Hunanoid, the quantity of initial demonstrations have no significant influence on the learning process. This may be because that MO- Humanoid environment from the reward side is more like a single objective problem and any reasonable demonstration can provide adequate guidance.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Expacted Ubllty DST (Demoo Quantity Evalustion)
Expected Utility of Ninecart (Demo Quantety Evoluation)
5.50
3.55
0.90
423
denoretran
-0.55
Time
Time Stap
Figure 7: a. DST
Figure b. Minecart
Quaniity
200
200
expeced 1ofo)
350
220
340
DG-MOIL detroraluur D01
DC MORL, dettoratur O0
DO MODR 2 demorotral OYD DO PRORL demTorGDraa O QF%) PD- FO--OTLLS
DC PORL, demTrrGDG O0S OF FD TD-OTILL
300
Time 12000
1ime Stop 13000
Figure 9: c. MO-Hopper
Figure 10: d. MO-Ant
Expected Utilty MO Humanoid (Demo Quanaity Evsluction)
Dosected LUKO
GR-RONGRILL
Figure 11: e. MO-Hamooodd
However, an interesting deviation was obscrved MO-Hopper and MO-Humanoid where the performance with fewer initial demonstrations was marginally better than with more. This outcome implies that, beyond certain point, additional demonstrations might not always contribute positively to the learning process. Potential reasons for this include:
Conflicting Action Selection: More demonstrations could introduce complexity in terms of conflicting action choices in similar states under different preferences. This conflict might hinder the agent's ability to learn a coherent and effective policy. This further emphasized by the result of the MO-Ant, as the nature of the objectives is very conflicting. The learning process therefore falling behind the GPI-PD curve at the start, though the with more demonstrations catches up the performance of GPI-PD at last.
24

---

Published in Transactions on Machine Learning Research (11/2024)
Complexity and Long Horizon: The intrinsic complexity of the MO-Hopper, MO-Ant and MO-Hunoooid, along with their extended horizon, might complicate the learning process. When the agent learns an effective policy that surpasses some of the guide policies, it might inadvertently "forget" how to surpass others, particularly those aligned with different preferences.
These observations indicate that while having a sufficient number of demonstrations is beneficial, there is a muanced balance to be struck. Too many demonstrations, especially in complex environments, can introduce new challenges that potentially offset the benefits of additional information.
Remarkably, the DG-MORI. algorithm demonstrates its robust learning capabilities by outperforming the GPI-PD baseline (except in MO-Ant, it needs more training steps to surpass GPI-PD, see Section 5.3.1 However, it can catch up the performance of GPL-PD at last.), even when provided with a limited number of initial demonstrations. This aspect of DG-MORL underscores its efficacy in leveraging available resources to achieve superior learning outcomes, highlighting its potential for applications where abundant demonstration data may not be readily available.
A.2 Sensitivity to initial demonstration quality
The quality of initial demonstrations emerges as another factor influencing the training process. While the self- evolving mechanism can to some degree mitigate this impact, it is still presumed that initial demonstrations of higher quality are associated with more effective learning outcomes because they may lead the agent to more potential areas in the state space.
and [6 collectively illustrate the DG-MORL algorithm's responsiveness to the quality of initial demonstrations across three different environments. These figures compare the overall performance of the DG-MORL agent when initialized with varying levels of demonstration quality. Additionally, they present. the FU for each set of demonstrations, indicated by dashed lines in colors corresponding to each demonstration standard.
Figures 12 13
In the DST environment, for instance, the learning outcomes for all three categories of initial demonstrations (high, medium, and low quality) eventually reach the environment's upper-performance limit. However, noteworthy differences are evident in the initial phases of training. The DG-MORL agent initialized with the highest quality demonstrations achieves significantly better performance right from the first evaluation round. The performance curve of the agent with medium-quality demonstrations shows slightly quicker ascent compared to the one with lower-quality demonstrations. This pattern underscores the impact of initial demonstration quality on both the speed and efficiency of learning, especially in the early stages of training.
In the Minecart, MO-Hopper, MO-Ant and MO (O-HHumnoid environments, the performance outcomes for the three sets of initial demonstrations reveal a notable observation: the quality of the demonstrations has a relatively minor influence on the final learning results. This outcome contrasts with the more pronounced effect seen in the DST environment. In Minecart, MO-Hopper, MO-Ant and MO-Humanoid, despite variations in the quality of the initial demonstrations (high, medium, and low), the differences in the final learning outcomes are less significant (but still higher). This suggests that, in these particular environments, the DG-MORL, algorithm is capable of achieving comparable levels of performance regardless of the initial quality of demonstrations.
This could be attributed to the inherent characteristics of the environments, where the algorithm might have more flexibility or capability to compensate for the initial quality of demonstrations during the learning process. It highlights the adaptive nature of the DG-MORL algorithm and suggests that its performance is not solely dependent on the quality of initial demonstrations, especially in certain types of euvironments.
Similar to the observations made in the experiment evaluating the impact of demonstration quantity, the DG-MORL algorithm consistently outperformed the GPI-PD method in all scenarios (except in MO-Ant, where DG-MORL only caught up with the performance of GPI-PD at last. But it does have outperformed GPI-PD in longer training steps, see Section 5.3.1 therefore does not harm the superiority of DG-MORL) tested for demonstration quality. This consistent superiority across different conditions and environments empirically validates the efficacy and robustness of the DG-MORL algorithm. It demonstrates that DG-MORL can effectively utilize the information provided in the demonstrations, her of high lower quality, to
25

---

Published in Transactions on Machine Learning Research (11/2024)
Expected Utiley of DST (Danto Quality Evaluation)
Expected Utilty of Minecart (Deme Qualtt Evalustion)
575
5.50
4.50
good
(geod)
Demonctracin
3/5
3
30 21 12
Time
Time
Figure 12: a. DST
Figure 13: Minecart
Liblity Hopper Evaluation)
Juilyy Qualtyy
200
expeced (ofo)
350
200
220
DC COC. good deroo
DC 9OO good Jeme
In tii Comanatraiion Igood)
hai Cemanitttion (good)
340
BC S/CIL medimm cume
DC SOOII, medium como
In lai Cemmomiiation Insduni
In Demamiiaaiinn Insduni
DC S/on, bad dume
DC PrOnL bud dame
Dal Demonscrater (bod)
tal Eemonstraben (bat) on PD ion
OTI-O CM L1
200
Time 12000
1im2 Stop 13000
Figure 14: c. MO-Hopper
Figure 15: d. MO-Ant
Expocted Ublity of MO Humencid (Demo Quality Eveluation)
Dovected L0Kf
CO-MON Demonstrab
202
100
Demonerration bas) GRI-1D+GPP-L
Figure 16: MO-Humanoid
enhance its learning process and achieve superior outcomes. This finding is significant as it not only attests to the performance capabilities of MG-MORL but also reinforces its potential applicability in wide range of real-world scenarios where the quality of available demonstrations may vary.
26


Reviews:
[{"review_id": "YmnyI9QHmc", "reviewer": "Reviewer_6uAU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are signs of better policy improvement compared to GPI baselines. The heavy use of acronyms makes it hard to keep track of what the paper is doing - IMO the clarity would be better if metrics like Hypervolume, Sparsity, etc were spelled out rather than abbreviated so readily. \n\nEven after reading the paper more closely, it is still not clear to me how the guide policy is implemented. Is it a learned behavior cloning model, or simply a rollout of the existing demonstration? If it's the latter, the method seems very similar to Go Explore (which is cited in related work).\n\nI overall think the paper has issues on clarity, but believe the results are reasonable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors tackle the problem of multi-objective RL. In this problem setting, we have multiple potentially-competing objectives. Rewards are vectors of multiple objectives rather than scalars, and our goal is to learn a Pareto frontier of non-dominated policies. The typical formulation for this is to have some preference weight vector $w$ as input to the policy.\n\nTaking inspiration from methods like jump-start RL (Uchendu 2023), the goal is to speed up policy learning by using demonstrations to guide the policy learning. In JSRL, given some guide policy $\\pi_g$, we run $\\pi_g$ for $h$ steps, then run an exploratory policy for the remaining steps. As the exploratory policy's performance improves, $h$ is reduced to yield more control to exploratory policy.\n\nThe primary modification needed in this work is to support a multi-objective RL setup. The guide policies are a set of demonstrations, which (I believe) are simply replayed in the environment. Given a new candidate weight, we find the demonstration that best maximizes that candidate weight, and roll it out as the guide policy. As RL progresses, if new episodes exceed some fraction (passing percentage) of the return from the selected demonstration, that new episode is added as a new demonstration. This passing threshold starts low and increases over time, with the set of demonstrations updated over time to remove any dominated rollouts.\n\nThis method is compared to various generalized policy improvement (GPI) baselines, a standard technique for multi-objective RL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend include more text in figure captions (what is the important conclusion you are arguing from the figure?), and use fewer abbreviations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gm3LkUcXKi", "reviewer": "Reviewer_SfsV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The problem the paper studies is quite interesting, practically relevant and understudied. To the best of my knowledge this is one of the first approach for MORL which can incorporate prior demonstrations with unknown preferences.\n* The overall approach is relatively simple and straightforward, mostly extending the JSRL work to a multi-objective setting, which I view as a positive. \n* While there are some issues with the experiments (discussed below), the results demonstrate the performance gains over the GPI baselines.\n\n**Weaknesses:**\nI think there are several places where the paper needs some major revisions.\n\n* A primary area of concern for me is the section on the theoretical analysis. Specifically, the upper bound analysis. There are several issues in the analysis. First, there are several quantities used without being defined. A lot of the content is taken directly from Uchenda et al., (2023), but not explained in sufficient detail. The proof for the theorem is missing (even if it is taken from a prior work and adapted to the setting it would be good to have the proof). More importantly, the authors simply analyze the multi-objective contextual bandit setting and use that as an upper bound for their approach with the justification \"our algorithm works based on\nthe Q-learning and TD3 paradigm [...] it\nis supposed to be less likely to violate the assumption\". This claim is incorrect and as such the analysis in the section, in my view is not meaningful. Some more issues are detailed in the requested changes section. \n\n* Further, the authors only use a single baseline GPI-PD. The authors explain that this is because GPI outperforms other methods like Envelope-MOQ, PGMORL. While I can understand limitations due to compute for running every method, but I believe the baseline comparisons could be considered a bit more deeply. For example, while GPI-PD may perform better, an approach like Envelope-MOQ might be a better suited baseline since it is also an off-policy approach and in-principle a similar approach of computing corner weights can be used to have a stronger baseline.\n\n* Additionally, the empirical validation seems somewhat limited. The authors test DG-MORL on four different tasks, however, they all appear to be relatively simple and small in scale (except the Humanoid task). I think other environments with high-dimensional observations (instead of the state itself), for example, the SuperMario task considered in Envelope-MOQ would strengthen the results.\n\n\n* Overall, I think the writing quality is quite poor and the paper could have benefitted from a thorough proof-reading. There are several places where quantities are undefined, not introduced, details missing, and use of incorrect notation. (See some instances in the requested changes section). \n\n\nUchendu, Ikechukwu, et al. \"Jump-start reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Broadly, the paper studies the problem of using prior demonstrations for bootstrapping policies in multi-objective reinforcement learning (MORL). This problem of leveraging prior demonstrations is challenging in multi-objective RL, since the preference weights over the objectives under the which the demonstration was drawn are typically unknown. The authors build upon prior work on MORL and the Jump Start RL approach (Uchenda et al., 2023). Specifically, under the assumption of a linear utility (which would capture only the convex hull of the Pareto front), the authors propose an approach, called DG-MORL, which consists of maintaining a set of demonstrations as guide policies, inferring the corner weights from the vectorized returns of the demonstrations, mixing the guide policy with an exploratory policy to collect better demonstrations, following a cirriculum controlling the mixing fraction. The authors present some theoretical analysis (more on this below) on the sample complexity of the proposed approach. Finally the authors evaluate DG-MORL on four tasks and observe that empirically the approach outperforms the baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The theoretical results section needs a complete revision based on the Weaknesses section and the comments below. \n* Experiments on one more domain with an unstructured observation space would make the results much more reliable.\n* The authors should consider designing stronger baselines based on off-policy MORL algorithms (e.g. in the Weaknesses section above)\n* Additionally, the authors claim one of the key contributions to be, what they call \"self-evolving mechanism\". I find it confusing, since what it really seems to be is just shifting to on-policy training as more experience is collected. I am not sure if it is helpful to introduce new terminology for something which can be described with the standard terminology.\n* There are a lot of writing issues some of which I have listed below. Overall, a thorough proof read would improve the paper considerably.\n\n\n\nSpecific changes:\n* Throughout the paper, the equations are not typeset correctly. Please use TeX commands when typesetting equations. e.g. in the entire paper the authors have used `max` instead of `\\max` which makes the equations much harder to read. \n* Throughout the paper, the wrong quotations are used. Please use ``...'' instead of \"\". \n* Page 1, paragraph 2, L1: \"magnifies\" -> \"amplifies\" / \"exacerbates\"\n* Page 2, paragraph 1, last line: \"development of a sub-optimal policy\" -> \"learning of a sub-optimal policy\".\n* Page 3, paragraph 1, last line: what does \"most potential preference\" mean? \n* Page 4, paragraph 5: CCS is used before being defined. (I assume it is convex coverage set?)\n* Figure 1: The color scheme in the figure could be improved. Currently it is hard to distinguish between the orange and red circles.\n* Page 5, above equation (1):  \"proposed by (Roijers, 2016)\" should be using `\\citet` instead. \n* Page 5, below Theorem 1, last sentence: \"most proper corner weight\". What does most proper mean here? \n* Page 6, Section 4.4, L1: \"DG-MORL pioneers\" -> \"DG-MORL introduces\"\n* Page 6, Section 4.4, L3: \"commendable outcomes\" -> \"strong performance\"\n* Page 6, Section 4.4, paragraph 3, L5: The sentence \"the better demonstration is popped into the demonstration repository to achieve self-evolving\" is poorly phrased.\n* Algorithm 1:\n    * L7-L9: What does it mean for a variable e.g. $\\mathbf{w}_c$ to be \"=\" to an equation? Something like \"Set $\\mathbf{w}_c$ based on Equation 2\" is more appropriate.\n    * L9: why is $h$ set equal to $H$? Shouldn't it be a hyperparameter? If they are set equal then there is no mixing. \n    * L13: How is $\\pi_e$ trained? There is no discussion of this anywhere in the paper.\n* Page 7, Section 4.5 L1: \"We provide a theoretical analysis of the algorithm sample efficiency’s lower bound and upper bound\" -> \"We provide a theoretical analysis of the lower bound and upper bound on the sample complexity of the algorithm\"\n* Page 9, Assumption 1: What do $d^{\\pi^*}$ and $d^{\\pi_g}$ mean? What does the constant $C$ denote?\n* Page 9, Corollary 3: The authors write \"We have provided the upper bound of the guide policy under a self-evolving mechanism\". Upper bound on what quantity? \n* Page 9, definition 2: What is $\\mathcal X$? The reader should not have to open another paper to understand the notation for a definition in the the paper.", "broader_impact_concerns": "The authors do not discuss any broader impact, since their work is about general algorithmic improvements with no direct applications with broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ylXbmNryc7", "reviewer": "Reviewer_kgw3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- DG-MORL is an interesting approach - integrating information from prior demonstrations seems to be a fairly natural idea and it is good to have it analyzed in a rigorous setting\n- Paper is well structure and clearly outlines the different problems in MORL that are addressed\n- Theoretical analysis gives a good mathematical grounding to the method\n\nWeaknesses:\n- Some sections are unclear and are hard to follow - particularly section 4\n- I would have expected a more thorough literature review, e.g. encompassing Bayesian methods which can be an naive way of integrating the information contained in the demonstrations\n- Very little details on the construction of demonstrations and the case of suboptimal demonstrations is not treated at all in the experiments - while proving the value of adding demonstrations to MORL is certainly good, I believe that a deeper analysis of the added value of suboptimal demonstrations would significantly increase the novelty of the paper's contribution", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces DG-MORL, a multi-objective reinforcement learning method leveraging prior demonstrations. It outlines and addresses several common issues with existing MORL methods and also proposes a self-evolving mechanism to refine the demonstration set, preventing sub-optimal examples from hindering progress. It also provides some theoretical results on sample complexity and benchmarks its performances against known other MORL algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses section. A few more details:\n- Section 4.1: intuitive explanation of corner weights not very clear\n- Section 4.2: How do you derive the demonstration policies in the case where the demonstrations do not cover all states?\n- In general section 4 is very unclear and some practical questions (such as the examples above) are not addressed enough\n- I would suggest adding much more explanation on the construction of DG-MORL and the self-evolving mechanism to make the derivation of the full model more intuitive", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the initial reviews of the paper, the authors have made several changes and corrections which seem to address all the concerns raised by the reviewers. There is a consensus that the paper now provides sufficient evidence to support its claims, and that this work synthesizes several important results relevant to TMLR's audience. I therefore recommend acceptance of the paper.

I note two minor points in the manuscript that the authors should address in the final version:

 - p.1 - (i) We propose the first MORL algorithm (DG-MORL) can use demonstrations to enhance the training. -> There is a typo

 - p.9 Fig. 3 - The text font in this figure is larger that the text in the manuscript. I recommend down-scaling that figure a little, and expand the figure's caption to be more explicit and self-contained (as recommended by reviewer 6uAU)
Decision: Accept (poster)