On Generalization Across Environments In Multi-Objective Reinforcement Learning
Authors: Jayden Teoh
Pradeep Varakantham
Peter Vamplew
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Multi-Objective Reinforcement Learning
Generalization
Primary area: reinforcement learning
TL;DR: This paper formalizes generalization in Multi-Objective Reinforcement Learning (MORL) and paves the way for future research on developing more adaptable multi-objective agents.

Abstract: Real-world sequential decision-making tasks often require balancing trade-offs between multiple conflicting objectives, making Multi-Objective Reinforcement Learning (MORL) an increasingly prominent field of research. Despite recent advances, existing MORL literature has narrowly focused on performance within static environments, neglecting the importance of generalizing across diverse settings. Conversely, existing research on generalization in RL has always assumed scalar rewards, overlooking the inherent multi-objectivity of real-world problems. Generalization in the multi-objective context is fundamentally more challenging, as it requires learning a Pareto set of policies addressing varying preferences across multiple objectives. In this paper, we formalize the concept of generalization in MORL and how it can be evaluated. We then contribute a novel benchmark featuring diverse multi-objective domains with parameterized environment configurations to facilitate future studies in this area. Our baseline evaluations of state-of-the-art MORL algorithms on this benchmark reveals limited generalization capabilities, suggesting significant room for improvement. Our empirical findings also expose limitations in the expressivity of scalar rewards, emphasizing the need for multi-objective specifications to achieve effective generalization. We further analyzed the algorithmic complexities within current MORL approaches that could impede the transfer in performance from the single- to multiple-environment settings. This work fills a critical gap and lays the groundwork for future research that brings together two key areas in reinforcement learning: solving multi-objective decision-making problems and generalizing across diverse environments. We make our code available at [https://github.com/JaydenTeoh/MORL-Generalization](https://github.com/JaydenTeoh/MORL-Generalization).

Published as a conference paper at ICLR 2025
ON GENERALIZATION ACROSS ENVIRONMENTS IN MULTI-OBJECTIVE REINFORCEMENT LEARNING
Jayden Teoh Pradeep Varakantham Peter Vamplew2 Singapore Management University FFederation University Australia
(jxtech. 2023, pradeepv) @ edu. sg "g.vampleweffederatiinn edu.
ABSTRACT
world sequential decision-making tasks often require balancing tradc-off's between multiple conflicting objectives, making Multi-Objective Reinforcement Learning (MORL) an increasingly prominent field of research. Despite recent ad- vances, existing MORL literature has narrowly focused on performance within static environments, neglecting the importance of generalizing across diverse settings. Conversely, existing research on generalization in RL has always as- sumed scalar rewards, overlooking the inherent multi-objectivity of real-world problems. Generalization in the multi-objectio context is fundamentally more challenging, it requires learning Pareto set of policies addressing varying preferences across multiple objectives. In this paper, we formalize the con- cept of generalization MORL and how can be evaluated. We then con- tribute novel benchmark featuring diverse multi-objective domains with pa- rameterized environment configurations to facilitate future studies in this area. Our baseline evaluations of state-of-the-art MORL algorithms this bench- mark reveals limited generalization capabilities. suggesting significant room for improvement. Our empirical findings also expose limitations in the expressiv- ity of scalar rewards, emphasizing the need for multi-objective specifications to achieve effective generalization. We further analyzed the algorithmic complex- ities within current MORL approaches that could impede the transfer in perfor- mance from the single- to muliple-environment settings. This work fills critical gap and lays the groundwork for future research that brings together two key ar- eas in reinforcement learning: solving multi-objective and generalizing across diverse environments. We make our code available at httpps://gtccabb.cmm//aaaaaeerre GenpeeehhhhooonG-Greaaaaaaaaaa
problems
INTRODUCTION
Developing agents capable of generalizing across diverse environments central challenge in reinforcement learning (RL) research. While significant progress has been made in studying the generalizability of RL algorithms, these efforts predominantly focus on optimizing single scalar reward signal (Zhang et al. 2018 Cobbe et al.) 2019 Irpan & Song 2019 Packer al. 2019 Kirk et al. 2023). Single-objective RL (SORL) overlooks the complexity of real-world problems, which often necessitate trade-offs be made between multiple conflicting objectives. Reducing these mul- tifaceted considerations to single scalar reward (objective) obscures critical interactions between objectives and limits the agent's utility (Vamplew et al.) 2022]. The field of Multi-Objective Reinforcement Learning (MORL) has sought to address the inherent multi-objective nature of quential decision decision-making tasks (Roijers et al., 20133 3 Hayes et al. 2022). However, the existing body of MORL research has concentrated optimizing agent performance within static environments, neglecting the dimension of generalization across varying situations. Consequently. there exists a significant gap in the RL literature: the intersection of generalization and MORL.
Generalising over multiple scenarios and objectives simultaneously is routinely demanded in many real-world applications, such healthcare management, autonomous driving, and recommendation
"Corresponding author.

---

Published as a conference paper at ICLR 2025
systems. Consider an autonomous vehicle, which must not only generalize across varied environ- mental conditions- conditions-differeett weather patterns, lighting, and road surfaces but also learn optimal trade-offss between competing objectives such as fuel consumption, travel time, passenger's 's comfort, and safety. Failure to effectively generalize across these environments and objectives would lead to inefficient operation or even catastrophic outcomes. The real world's dynamic nature extends be- yond just environmental variability, but also includes evolving goals and utility preferences. An agent optimizing a single scalar reward may exhibit some level of generalization, such as adapting to state variations, but it will struggle to generalize when faced with new goals or reward structures. This is because the agent has only observed its current reward signal, and lacks the basis for adapt- ing its behaviour should the reward signal change. In contrast, a MORL agent learns to consider all dimensions of vector reward, even those that are not immediately relevant to current goals. This holistic approach to learning allows the agent to adapt swiftly when its utility landscape evolves or when stakeholders' prioritisation over the different objectives shifts. For example, in autonomous driving, generally capable MORL agent can satisfy unique preferences over objectives for different passengers without the need for retraining. Therefore, developing generally capable multi-objective agents enables not only generalization across diverse environments, but also across dynamic goals and utility functions-an overlooked aspect in current single-objective RL generalization literature, yet one that is arguably essential for real-world applicability.
The contributions of this paper are as follows: (1) we present formal frameworks for discussing and evaluating generalization in MORL, (2) we introduce the MORL-Generalization benchmark, a collection of multi-objective domains with rich environmental variations, (3) we conduct extensive evaluations demonstrating that state-of-the-art (SOTA) algorithms fall short on our benchmark, and (4) we provide post-hoc analyses of possible failure modes in these methods and offer directions for future research. Notably, we open-source our software and release comprehensive dataset derived from over 1,700 cumulative days of baseline evaluations across multiple SOTA algorithms. These contributions lay the foundation for advancing MORL generalization research, ultimately pushing the boundaries of what RL agents can achieve in complex, real-world environments.
2 BACKGROUND
In this section, we introduce MORL and establish the formal notations referenced throughout this paper. A multi-objective sequential decision-making problem can be modeled by Multi- Oujji-Objecttve Markov Decision Process (MOMDP; White (1982)) represented by the tuple: (S, AS,A,T T, R.R.,, 7) with state space S, action space A. transition function T : S x A x S + (0, 1]. initial state distribution μ., and discount factor E [0, 1). The key distinction between MOMDPs and standard MDPs lies in the vector-valued reward function R S A x S + IR, where k is the number of objectives. The goal of a standard RL agent is to maximize its expected long-term discounted sum of rewards, i.e. value function. For a stationary policy w : x A + [0, 1]. the multi-ohjectiive state value function at state E S is given by
V (s) : E
/r44 80
where r1+1 = R(st, a,, St+1) is the k- d-dimensional reward vector received at timestep t + 1. The expected value vector of π under the initial state distribution j is defined as v E V (V)() Since (*)(s) is vector-valued function, it can only specify partial ordering over the policy space for a given state. Given two policies and x². it is possible that v > v.¹ for objective i, but v < v for objective j. Since there may be no single policy that is optimal across all objectives, MORL requires the agent to learn a solution set of policies, each reflecting different trade-offss across objectives. There are two primary approaches to deriving an optimal solution set: the axiomatic approach and the utility-based approach (Roijers et al. 2013).
The axiomatic approach operates on the axiom that the optimal solution set is the Pareto set. This leads us to the concept of Pareto dominance. We say policy Pareto dominates (denoted by pp) another policy n' if its expected value vector is higher or equal across all objectives, that is: v >p y= + (Vi : o. 2 of (3i: : of
> of ). The Pareto Set consists of all nondominated
(Pareto optimal) policies:
PS { E II 3K' E II V' vP
2

---

Published as a conference paper at ICLR 2025
where II is the set of all possible policies. The image of the Pareto set under the expected value function mapping is known as the Pareto front.
On the other hand, the more prevalent stility-basck approach assumes each user's preferences over the objectives can be modeled by a utility function that projects the multi-objective value vectors to scalar utility. In the multi-policy setting, utility-based approaches assume user preferences can be represented by a space of weight vectors w E W that parameterize the utility function, i.e. IR* W + R. These utility functions are usually assumed to be monotonically increasing in more
every objective. This is a natural assumption in accordance with notions of reward for an objective should not decrease a user's utility as long as does not result in a decrease in reward for another. Linear utility functions, u(v, w) = wTv, with weights satisfying Σ, w, = and w; 2 0, are commonly employed. During training, utility-based agents aim to learn a coverage set, CS C IT, that maximizes the scalar utility for all weights. Formally, this means:
Vw E W, 3m E CS
s.t.
V
max V T'Ell
where v denotes the scalar utility of policy under the utility function u(: Thus, every weight vector w. CS contains at least one optimal policy that achieves the maximal scalar utility. There optimization criteria for which the calculation of v are different: expected scalarized return (ESR), V. = Be,,o~p (r 7'r,, w) [sn]. and scalarized expected return (SER), v. = u(E Ei= | w). When the utility function u is non-linear, these two formulations generally yield different optimal solution sets.
Axiomatic approaches seek to learn the optimal Pareto set and present it to users for selection a posteriori. While the Pareto set always contains the policy that maximizes the scalarized value for any monotonically increasing utility function, it can become prohibitively large to learn and to retrieve. In contrast, utility-based approaches narrows the solution set by only focussing on policies that maximize the user's utility. It also simplifies policy selection, as the user's preference, represented by a weight vector w. directly guides the choice of policy during inference. However, this approach requires the users' utility function to be specifiable in closed form and known priori.
3 MULTI-ORIJETIVV CONTEXTUAL MARKOV DECISION PROCESS
To formalize the notion of generalization in the context of MORL, need to start with way to reason about a collection of multi-objectinn objective environments. In single-objective RL (SORL), this is often done using the Contextual MDP (CMDP; Hallak et al. (2015)) framework. As such, we for- mally define a Multi-OOjeective Contextual Markov Decision Process (MOC-MDP)-al adaptation of the CMDP framework the multi-objective setting.
Definition (Multi-Objective Contextual MDP). A MOC-MDP is defined by the tuple
(C, S,A, M)
where C is the context space and S and A are the shared state and action spaces across environments respectively. This is similar to a CMDP except M a function mapping any E a MOMDP, i.c. M(c) A, A. R x),7).
The context space, C defines a set of parameters, each representing a different MOMDP. Intuitively, the context can be viewed discrete continuous parameter specifying the multi-objective en- vironment configuration, such as seed or vector controlling the environment dynamics. Each configuration varies in its initial state distribution, transition functions and multi-objective rewards, but share enough common structure across which the MORI. agent can generalize. MOC-MDP describes a model where for each context there potentially distinct optimal Pareto front. The context remains constant within a single episode. Throughout this paper, will also refer to a par- ticular MOC-MDP as a "domain", and its associated "contexts" as "environments" interchangeably.
We begin by formalizing the generalization objective for axiomatic MORL approaches. The main objective the axiomatic approach lies in identifying all nondominated vectors across the Pareto front. In the case of a MOC-MDP, since there are different Pareto fronts for each context, to attain optimality in any scalarization function for any context, it would involve a union of policies from Pareto sets across contexts. Collectively, these policies form global Pareto set.
3

---

Published as a conference paper at ICLR 2025
Definition 2 (Axiomatic Generalization). Given MOC-MDP with policy space II, the generaliza- tion problem for axiomatic approaches is to learn a global Pareto set PSc given by
PSc { Be C II. Va P vP.)
where v is the expected value vector in a context Thus, the global Pareto set comprises policies that are nondominated in at least one context, ensuring that all necessary policies for constructing the Pareto fronts every context are captured.
Recall that utility-based based approaches, each user's preference is modeled by weight vector, w W. which parameterizes a utility function u. Thus, utility-based approaches seek to learn optimal policies for each w. However, in MOC-MDP, the optimal policies for given w may vary across contexts. Therefore, the generalization objective in utility-based approaches is to find policies that maximize the expected utility over the context distribution for each w.
Definition 3 (Utility-baseed Generalization). Given MOC-MDP, for each weight w E W, the generalization objective for utility-based approaches is to learn generalized coverage set CSc given by
r'ell Vw W, CSc S.L. Bewple) IV E.wl max (esple) (ve.w:
where p(c) is the context distribution and v.w denotes the scalar utility of for context under the utility function u(., w), computed using either the ESR or SER criterion. In this framework, each policy must generalize across contexts for a specific w. This structure retains the key advantage of sallity-based methods- Sathhods-allowing users to easily select their preferred policy by indicating at inference time, independent the context.
When the context is fully observable, one can simply augment the state-space with the sampled context itself, e.g. s' concat(s, c). This effectively reduces the MOC-MDP to a "universal" MOMDP that unifies the context-dependent components, i.e. T R ,ra, into a single model. In this setting, the global Pareto set PSe and the generalized coverage set CSe correspond directly to the Pareto set PS and coverage set CS the "universal" MOMDP, respectively.
In settings where the context is hidden or only partially observable at test time, both axiomatic and utility-based generalization methods must infer the underlying context to achieve optimality. That is, the agent must learn posterior the context space, i.e. p(c|H), where H represents observable information in the test environment such as state-action history and rewards. This process analogous to solving a partially-observe
MOMDP, in which the agent forms and updates belief over the current context to determine the optimal policy. Note here that we do not impose restrictions on the nature of the policies learned PSe and CSe. The policies may be either Markovian non-Markovian, depending whether context inference is decoupled from the policy.
EMPIRICAL EVALUATION OF MORL GENERALIZATION PERFORMANCE
Let Ceval = (C1, C2. ,.....,,., represent a set of independent evaluation contexts. Measuring an agent's generalization performance in SORL is straightforward: the larger the reward value across Coval. the better. In MORL, however, agents produce an approximate Pareto front comprising multiple value vectors for each Cevat and translating the quality of this Pareto front into a scalar metric that captures generalization performance is non-trivial.
In what follows, we propose an axiomatic-bart based evaluation metric called the Normalized Hyper- volume Generalization Ratio (NHGR) to enable fairer assessments of generalization performance MORL. Our discussions and evaluations focus primarily on this axiomatic- based metric because specifying meaningful user utility function a priori can often be infeasible in reinforcement learn- ing benchmark tasks, including those introduced in Section 5 For completeness, we also introduce a utility-based evaluation metric, the Expected Utility Generalization Ratio (EUGR) in the appendix. The EUGR is useful in scenarios where the MORL agent must generalize across multiple environ- ments with well-defined user utility functions. Both metrics assume that optimal Pareto front is available for each evaluation context. In practice, when the true front is unavailable, it can be ap- proximated by aggregating nondominated value vectors from specialist agents trained independently
4

---

Published as a conference paper at ICLR 2025
in each context. For extended discussions the benefits of NHGR, evaluations using EUGR and other metrics, we refer motivated readers to Appendix D
The Normalized Hypervolume Gencralization Ratio (NHGR) is extension of the widely used Hypervolume (HV) indicator (Zizzlrr & Thiele 1998). The HV is a standard metric for evaluating the quality of approximate Pareto fronts in single-context MORL, measuring the volume in objec- tive space covered by a Pareto front relative to a reference point. However, the HV metric is not scale-invariant and tends bias evaluations toward objectives with larger magnitudes. While nor- malization has been explored in multi-objective optimization (MOO) (Deb & Kalyanmoy (001), has been largely overlooked MORL literature. Therefore, before we introduce the NHGR, we briefly introduce the Normalized Hypervolume nom..
Consider a k-objective domain and let F2 C. R denote the optimal Pareto front for an evaluation context Ceval- We define the element-wise minimum and maximum boundary value vectors of Fo as v-inn and v. respectively. We refer to MORL agent trained generalize across multiple multi-objective contexts the generalist. Our goal is to evaluate the performance of this generalist. Let F. C IR denote the approximate Pareto front produced by the generalist for context The Normalized Hypervolume is defined as
min V vmaxx ymin
HV norm (Fe) AL U V
where À is the k-dimensional Lebesgue measure (ebessue 1902). Since the objec- tives are normalized, we can use the origin 0 as the reference point. The use of nommm also enhances interpretability as it is bounded within O and the hypervolume of the unit hypercube (which is I). Using HV HVnomm- we can then introduce the NHGR metric:
Definition 4 (Normalized Hypervolume Generalization Ratio). The NHGR for the generalist in context is de- fined as:
V2
14
NHGR
HV norm (Fc)
NHGR (F. Fo.
HV nomm (F:)
NHGR measures the ratio of normalized hypervolume between the generalist's approximate Pareto front and the optimal Pareto front. NHGR draws similarities to the Hyperarea Ratio (Veldhuizen & Allen 1999) met- ric in MOO literature but additionally employs HV norm Fig. illustrates the NHGR metric for biobjective domain. Intuitively, truly gen- cralizable MORL agent should achieve optimal perfor- mance in every context, which corresponds to attaining an NHGR across all contexts.
Normalized F. (generalist's) Normalized F (optimal)
to ensure
Figure 1: NHGR visualized as the ratio between the hypervolume of the normal- ized generalist's 'sss and the optimal Pareto fronts (dashed and shaded areas).
MORL GENERALIZATION BENCHMARK
We now introduce the MORL-Generrlizaain benchmark, a diverse collection of multi-objective do- mains with rich environmental variations facilitate future research on generalization in MORL.. We adapted existing domains from MO-Gymnasium (Felten et al. (023). a multi-objective extension of the Gymnasium library [Towers et al. 2024 Brockman al. 2016). and introduced new ones, each with expressive parameters controlling environmental variations. Kirk et al.) (2023) identified four key types of domain variations for studying generalization: 1) state-space variation (S), which alters the initial state distribution, 2) dynamics variation (D), which alters the transition function,
3) visual variation (O), which impacts the observation function, and 4) reward function variation (R). This benchmark primarily focuses state-space and dynamics variations. Observation varia- tions do not alter the underlying MOMDP structure (Du et al. 2019). Hence, they provide limited insights into the multi-objective decision-makin capabilities of the agent since the optimal Pareto front across variations remain isomorphic. Reward variations are often introduced through multiple goals. Multiple goals naturally be modeled multiple objectives by treating each goal
5

---

Published as a conference paper at ICLR 2025
an g
Fectie
J0181
Bind SPPOT
Turrsslence Pownr
1110)
933
Figure 2: Domains in the MORL Generalization benchmark. Top from left to right: 1) MO- LunarLander, 2) MO-Hopper. 3) MO-Cheetah, 4) MO-Humanoid. Middle row: MO-LavaGrid (8 handcrafted evaluation environments). Bottom row: MO-SuperMariooros (8 out of 32 stages).
conflicting objective (Sener Koltun 2018), which means MORL inherently involves learning to adapt to reward function variations. Nevertheless, we provided a novel maze domain that explicitly segregates the goals and multiple objectives, for completeness. Fig. 2 visualizes the domains pro- vided in the NORLL-Genralliztioon benchmark with annotations for their environment parameters, where applicable. We provide detailed descriptions each benchmark domain in Appendix F.T
When designing generalization benchmark for MORL, it is important to keep in mind of The Principle of Unchanged Optimality (Irpan & Song 2019). This principle asserts that, for a domain to support generalization, it should provide all necessary information such that policy optimal in every context can exist. We discuss how we uphold this important principle within Appendix B
6 EXPERIMENTS
In this section, we evaluate state-of-the-art MORL algorithms on the newly-developed benchmark to establish baseline expectations for their generalization capabilities. The implementations of these algorithms are adapted from Felten et al. (2023). Specifically, the algorithms evaluated are CAPQL (Lu et al. 2023), Envelope (Yang et al. 2019). GPI-LS (Alegre al. 2023). PCN (Rey- mond et al.) 2022 PGMORL (Xu et al. 2020), and MORL/D SB (Felten et al. 2024). We also include the model-based extension of GPI-LS, i.e. GPI-PD, and the weight adaptation variant of MORL/D SB, i.c. MORL/D SB+PSA. Additionally, we include the SAC (Haarnoja et al. (2018) al- gorithm trained with a single objective/ttility function our evaluations to verify that the objectives are not so highly correlated that single-objective agent could also achieve high performance across multiple objectives. In total, we evaluate MORL algorithms across domains using seeds each. These established baseline performances are open-sourced via Weights and Biases (Biewald 2020). facilitating future research and saving computational resources.
Note that Envelope is restricted to discrete-action domains, while CAPQL and PGMORL apply only to continuous-action domains. MORL/D SB and MORL/D SB+PSA were excluded from MO-SuperMariiorros evaluations due to the high memory demands of convolutional neural net- works, which are incompatible with their evolutionary approach. GPI-PD was omitted from MO- HalfCheetah, MO-Hunnnoid, MO- SuperrarioBBrss, and MO-LavaGrid, as the large state spaces in these domains cause its dynamics model to degrade rapidly, resulting in zero simulated transitions and effectively reducing it to its model-free counterpart, GPI-LS.
Domain Randomization (DR) is an efficient method to expose the agent to a wide range environ- ments during training by uniformly sampling from the environment parameter space. It has found

---

Published as a conference paper at ICLR 2025
success in deep RL even for complex visual domains and real-world robotic control (Tobin al. 2017 Peng et al. 2018). We utilise DR for all our experiments by randomizing the environment parameters after every training episode. This also enables to standardise the presentation of environments across algorithms via the RNG seed, and evaluate the algorithms solely for their gen- eralization capabilities. At each evaluation time step, each algorithm is assessed over 100 episodes] across set of environment configurations. Whenever possible, these configurations are chosen using the boundary values of environment parameter ranges ensure diverse evaluation environ- ments and behaviors, For MORL algorithms using linear scalarization, a weight vector is uniformly sampled from the unit simplex for each evaluation episode. We aggregate the NHGR performance across all evaluation environments for each domain and report results in terms of inter-quartile mean (IQM) and optimality gap using the rl iabll library (Agarwal et al. 2021), which helps account for statistical uncertainty prevalent in deep RI.. IQM focuses on the middle 50% of combined runs, discarding the bottom and top 25%. Optimality gap captures the amount by which the algorithm fails to meet desirable target, i.c. when NHGR=1. The evaluation environment configurations and other experiment setups are detailed in Section of the appendix for reproducibility.
6.1 MORL GENERALIZATION RESULTS
The baseline results using NHGR reveal significant performance gaps in the generalist agents across various domains, highlighting the benchmark's ' potential to serve as a foundational benchmark for future research in MORL generalization.
Optimality
IQM
Optimality Gap
SAC
SAC
PCN
Envelope
PGMORI
GPI-LS
GPI GPI-PD
GP-PD
MORL-D(SB)
MORL-D(SB)
NORL-D(SB-4 D(SB+PAA)
MORL-D(SB PSA)
0.45 0.60
0.75
0.15 0.30 0.45
0.60
0.75
Normalized Hypervolume Genaralization Ratio
Normalized Hypervolume Generalization Ratio
(a) MO-LunarLandes
(b) MO-Hooper
IQM
Optimality Gap
Optimality Gap
SAC
PCN
PCN
CAPOL
CAPQL
PGMORI
PGMORL
GPI LS
GPH-LS
MORL-DISB)
MORL-D(SB)
NORL-D(SB+P5A)
MORL-D(SB
0.60
0.90
Normalized Hypervolume Generalization Ratio
Normalibed Hypervolume Generalization Ratio
(c) MO-HalfCheetahh
(d) MO-Humanoid
Optimality Gap
IQM
Optimality Gap
SAC
SAC
PCN
PCN
Envelspe
Envelope GPI-LS
GPH-L
MORL-D(SB) MORL-D(3B-PPAN
0.03
0.06 0.09
0.88
0.92
0.96
0.00
0.35
0.30
Normalized Hypervolume Generalization Ratio
Normalized Hypervolume Generalization Ratio
(e) MO-SuperMarioBros
(I) MO-LavaGrid
Figure 3: Aggregate NHGR performance in domains of the benchmark. Each algorithm is eval- uated across 5 independent seeds and several evaluation environment configurations. Higher IQM and lower optimality gap scores are better. The best algorithm for each domain is bolded.
MO-LunarLander MO-LunarLander is one of the simplest domains in the benchmark, featuring a low-dimensional observation space and discrete action space. The best Dest-performing algorithm, GPI-LS, achieved an IQM NHGR score 0.66 with an optimality gap of 33.9 33.9% (see Fig. Ba). Given its simplicity, we recommend using MO-LunarlLandee algorithms. Additionally, our software includes a continuous-actee variant of this domain, which may reveal larger NHGR optimality gaps and provide greater opportunities for improvement.
starting point for evaluating new
In Mossiner 32 evaluation episodes are used keep runtime under 72 hours.

---

Published as a conference paper at ICLR 2025
MuJoCC--bssed Domains The challenge of generalization in MORL becomes pronounced in the MuJoCo-based (Todorov et al. 2012) domains. Across the 3 domains, a wider spread in performances and noticeably lower performance ceilings observed. In the MO-Hopper domain, the leading algorithm, MORL/D SB+PSA A, managed to reach an IQM NHGR score of only 0.56 and optimality gap of 46.4% (see Fig. 3b) This gap widens in higher-dimensional domains like MO- HalfCheetah (Fig. 3c) and MO-Humanoid (Fig. 3d), where the top algorithms, MORL/D SB+PSA and MORL/D SB, achieve IQM NHGR scores of just 0.28 and 0.41, respectively. These wide optimality gaps, combined with the strong relevance to real-world robotic control tasks, suggest that these domains may serve as enduring benchmarks for studying generalization in MORL.
Lava-Time 10000 Lava-Time: 0.5-0.5 Lave-Time 0.0-10
Spacc Choocialie
0:20
0:5
NAGA
6:00
Suge 23
Stuge 3-2
Stuge 5-3
Stape 3-3 (Zero Shob)
SAC
PCN
Envelope
GPH-5
(a) NO Mu-SurrrMarroBrros Performances
(b) Hcatmap of visited tiles
Figure 4: (a) Muo- performances on 4 stages. Stage 3-3 in the rightmost column shares visual similarities with the other stages so it is excluded from training evaluate for zero- shot generalization. (b) Heatmap of visited tiles for specialist and generalist in the MO-LavaGrid "Room" enviromment. Each column's title shows the conditioned linear weights for the lava and time penalty objectives.
MO-SuperMarioBros Fig. 3e presents the NHGR performance of 3 discrete MORL algorithms and SAC on MO-SuperMarioorro. The leading algorithm, GPI-LS, achieved an IQM NHGR score of only 0.07 and optimality gap of 87.7%. We also conducted zero-shot generalization experiment by excluding stage 3-3 from the training distribution. This stage shares a combination of visual features with stages 3-2, 4-3, and 5-3, allowing to test if agent has learned generalizable behaviors over the pixel space or merely memorized stage-specific sequences. The results in Fig, [4 negligible NHGR performances in the zero-shot environment (stage 3-3), suggesting the latter.
MO-LavaGrid Fig. 3f shows the performance of five discrete MORL algorithms on MO-LavaGrid, with MORL/D SB achieving the highest IQM NHGR score of 0.37, though still far from optimal. To analyze this, we recorded trajectories for generalist (MORL/D SB) and specialist (GPI-LS) in the "Room" environment, both using linear scalarization. Fig. presennss heatmaps of tile visit counts when conditioned on three different weightings of lava and time penalties. While the specialist consistently follows optimal routes, the generalist exhibits random walks overlapping with the three goals. This likely explains MORL/D SB's nonzero NHGR performance across environments but significant optimality gap, as it incurs high penaltics from inefficient navigation.
In summary, the generalization performance of the current MORL algorithms leaves much to be desired. This outcome not surprising, as these experiments were aimed to provide a baseline un- derstanding of existing methods without any tailored interventions enhance generalization yet. Despite not attaining the top performance in every domain, MORL/D SB and MORL/D SB+PSA, demonstrated the most consistent results overall. Future research aiming improve MORL gener- alizability can consider building upon these algorithms for more reliable testing.
6.2 SCALAR REWARD IS NOT ENOUGH FOR RL GENERALIZATION
Real-world problems are often multi-objective. In fact, many popular SORL benchmarks are natu- rally multi-objective but are simplified using hidden scalarization functions. For instance, the origi- nal Hopper domain combines forward velocity (vz), control cost (c), and bonus for not falling (h) into a scalar reward: 1.5u,, + 0.001c + h. In contrast, MORL approaches treat these as independent objectives and may introduce additional ones, such torso height. One might argue that if
8

---

Published as a conference paper at ICLR 2025
stakeholder's sole goal is to maximize the agent's forward movement, MORL approaches that seek to learn tradeoffs across multiple objectives would be redundant in RL generalization. However, our empirical results challenge this assumption.
Hoppeprrrroouttt
Hopporlugnt
MO HopserS1psery
HogperLowSsmptnn
MO HapperHard
2500
3400
1400
1200
2000
3000
933
1997
800
1000
620
430
M00
660
1997
500
233
500
250
200
Staps
Stags
Staps
Steps
Steps
Steps
MORL-D PSA)
MCRL-O/SB)
CATQL
Figure 5: Single-objective return on 6 MO-Hopper testing environments during training. Each curve is measured across 5 seeds (mean and standard error).
Let SSORI. denote the fixed scalar reward function that SORL agents trained to optimise during generalization. Throughout the generalization training horizon of the MORI. algorithms in Sec- tion [.T] sampled solution vectors across their Pareto fronts, scalarized them using SSORL, and recorded the highest scalar reward in each evaluation environment. For the SORL agent, which al- ready specializes on fSORL. we allow agents, and took the best result. Our results reveal that when trained on the same generalization procedure, MORL algorithms can outperform the SORL agent on its own specialized reward func- tion SSORI. Fig. S shows several MORL algorithms surpassing the SAC agent by large margins in JSORI. return across six distinct test environments during generalization training the MO-Hopper domain. Note that CAPQL is multi-objective variant of SAC, while MORL/D SB and MORL/D SB+PSA is population- based approach of SAC. All baC-based implementations are from the same library, CleanRL (Huang al. 2022), making these results fair. Similar findings observed in other domains (see Section D.3 3 in appendix), where leading MORL algorithms could outperform or achieved parity with SAC on JSoRL performance.
many runs solution vectors sampled from the MORL
Fig. 6 presents snapshots from the highest JSORI. episodes of the MORL/D SB+PSA agent on three MO-Hopper environments. Since MORL/D SB+PSA a linear utility-based ap- proach, the table in Fig. [6] provides the linear weight vectors which the agent conditioned on. In the Default environment, the agent placed higher weight on forward velocity, causing to lean forward and cover more distance. In Slippery environment, where low friction makes leaning forward dangerous, the agent balances forward velocity with torso height, maintaining upright posture prevent slip- ping. In the Hard environment, which features slippery floor, unbalanced body masses, and low joint damping, the agent maximized fSORL. by emphasizing torso height and minimizing control cost, which help maintain stability and avoid abrupt movements. In contrast, the single-objective SAC agent learned only a single behavior to generalise across all environments, causing it to fail in dire conditions.
(a) Default
(b) Slippery
(c) Hard
Default
Slippery
Hard
forward velocity
0.84
0.52
0.3
torso height control cost
0.09
0.4
0.23
0.07
0.08
0.47
Screenshots of MORL/D SB+PSA agent's 's behavior in different MO-Hopper environ- ments and the corresponding linear weights
Figure 6:
The single-ohjective approach to RL. generalization heavily reliant on reward engineering, i.e. finding an optimal scalar reward signal through trial-and-error search of scalarization functions (Sutton & Barto (2018), Chapter 17.4). However, our observations suggest that there may no universal scalarization function capable of optimizing performance across all environments. Each environment demands distinct behaviors from the agent, even for a fixed goal like moving for- ward. Consequently, priori scalarization in SORL limits the agent's 's adaptability to environmental changes. In contrast, generalization with MORL approaches circumvents the reward engineering problem by considering all dimensions of vector reward independently, those not immedi-

---

Published as a conference paper at ICLR 2025
ately relevant to current goals. This flexibility allows agents to learn diverse behaviors that address different trade-offs among objectives. Stakeholders can then select policies from the agent's s solution set that best match their utility preferences for any given environment, enhancing the adaptability of MORL agents in generalization tasks. These observations align with recent studies that chal- lenge the expressivity of scalar rewards and advocate for the adoption of multi-objective reward formulations [Vamplew et al. 2022 Skalse & Abate 2024 Subramani et al. 2024).
7 RELATED WORK
Multi-Objective Contextual Multi-Armed Bandits: Multi-Objectio Contextual Multi-Armed Bandits (MOC-MAB: Tekinn & Turgay| (2017): Turgay et al. (2018)) are a context-dependent, multi- objective extension of the classic Mulli-Arm Bandil (MAB) problem. In MOC-MAB, at each deci- sion point, the agent observes a context and selects an action (arm) to maximize a vector of immedi- ate rewards corresponding to different objectives. While MOC-MAB provides valuable insights into handling contexts and balancing multiple objectives simultancously, it fundamentally differs from the MOC-MDP framework. Specifically, MOC-MAB does not address the state-transitions and quential decision- making inherent in MORL. Our work extends beyond the MOC-MAB setting by focusing on the generalization of RL agents in context-dependent, multi-objecctive environment problem that, to our knowledge, has not been previously explored in the literature. However, bandit analysis often forms the foundations of progress in RL, we implore future work to look into the MOC-MAB framework for inspiration on improving generalization in MORL.
Multi-Task Learning: Multi- Task Learning (MTL: Caruana (1998)) and Multi-Task Reinforcement Learning (MTRL: (anaaa & Yamamural ) aim to improve learning efficiency and performance by leveraging shared representations across multiple tasks. Reinforcement learning based on Con- textual MDP (CMDP) is closely related to MTRL but involves parameterized variable, termed the context, which allows for more unified modeling of tasks within a single framework. However, both MTRL and CMDP have predominantly been studied in the single-objective setting, focusing on maximizing scalar reward function. [Sener & Koltun (2018) 8) framed MTL as MOO problem by treating different tasks confficting objectives. While this perspective introduces multi-objectivite into MTL, it primarily addresses trade-offs between tasks rather than scenarios where each task in- volves multiple objectives. In the optimization domain, the Multi-Ohjectivv Multifactorial Optimiza- tion paradigm (Gupta et al. (017) considers multitasking across multiple multi-objective problems by leveraging shared evolutionary operators to solve them simultaneously.
Despite these advancements, there is a notable gap in the literature regarding the simultaneous con- sideration of multi-objectivity and generalization across contexts (i.e. environments tasks) in reinforcement learning. To the best of our knowledge, this is the first study to systematically explore generalization in MORL and highlight unique difficulties within this combined setting.
DISCUSSION AND CONCLUSION
Developing reinforcement learning agents for real-world tasks necessitates not only generalization across diverse environments, but also across multiple objectives. By formally introducing frame- work for discussing and evaluating generalization MORL, we bridge a crucial gap between RL generalization and multi-objective decision-making To measure progress in this area, we con- tributed a novel benchmark to facilitate rigorous investigations into MORL generalization. The extensive baseline evaluations of state-of-thee art MORL algorithms on the benchmark also highlight significant room for future research to improve upon. We encourage readers to look at Section of the appendix, where analyzed algorithmic failure modes in current MORL approaches, offering insights into their poor generalization performance. Extended discussions of MORL generalization and future research directions are also provided in the appendix. Moreover, we have open-sourced our software, alongside the raw dataset derived from our baseline evaluations. These contribu- tions would streamline future investigations into MORL generalization. We hope this paper spurs greater recognition of the importance of objti-obeecttve reward structures for RL generalization. Ultimately, by unifying these two fields, this paper lays the groundwork for advancing RL agents capable of tackling the complexities of real-world, multi-objective scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research/project is supported by the National Research Foundation Singapore and DSO Na- tional Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2000-017),
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 34:29304-29933 29320, 2021.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization, 2019. URL http::::/aaxiv..or//bb/////////
11214
I Lucas N. Alegre, Ana I.. C. Bazzan, Diederik M. Roijers, Ann Nowé, and Bruno C. da Silva. Sample-efficient multi-ohjective learning via generalized policy improvement prioritization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Sys- tems, AAMAS '23, pp. 2003-2012, Richland, SC, 2023. International Foundation for Au- tonomous Agents and Multiagent Systems. ISBN 9781450394321.
Lukas Biewald. Experiment tracking with weights and biases, 2020. wandb. com/ Software available from wandb.com.
URL https: //ww.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 17X6::1606.0/544: 2016.
Rich Caruana. Multitask Learning, pp. 95-133. Springer US, Boston, MA, 1998. ISBN 978- doi: 00.1007/978-1-4615-555 978-1-4615-5529-22
1-4615-5529-2.
5. URL httpts://p..tttooo
Maxime Chevalier-Boisee Bolun Dai, Mark Towers, Rodrigo Vicez-V Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Mod- ular customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman, Quantifying generaliza- tion in reinforcement learning. 2019. [RRllatts:///axkt org. abs 1812. 02
Kalyanmoy Deb and Deb Kalyanmoy. Multi-Obeective Optimization Using Evolutionary Algo- rithms. John Wiley & Sons, Inc., USA, 2001. ISBN 047187339X.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficcent RL with rich observations via latent state decoding. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1665-1674. PMLR, 09-15 Jun 2019. htl.hhttss///proceeeee
www.sss..........mm...m......... www.r.rres//v//.
9b. ht.ll
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Robust predictable control, 2021.
URL https: wttp:bb////.......////o 03214
Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. toolkit for reliable benchmarking and research in multi-objective re- inforcement learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
Florian Felten, El-Ghazali Talbi, and Grégoire Danoy. Multi-objective reinforcement learning based on decomposition: A taxonomy and framework. J. Artif. Int Res., 79, feb 2024. ISSN 1076-9757. doi: /t16://aarr...577000 URL http::ttt////tt //oii. /10. 613 1.1.///......1000...
Abhishek Gupta, Yew-Soon Ong, Liang Feng. and Kay Chen Tan. Multiobjective multifactorial optimization in evolutionary multitasking. IEEE Transactions on Cybernetics, 4777):155--1655, 2017. doi: 10. 10.1100 0D9/TCYB.2216..55444

---

Published as a conference paper at ICLR 2025
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policcy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes, 2015. URL
httpg//bs/15022/ 02259
Conor F. Hayes, Roxana Rádulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. A practical guide to multi-objective rein- forcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36( (1), apr 2022. doi: 100.1007/11458-022--9035 5 0104458-00 02--8-002--95522
ISSN 1387-2532.
URL https: /tttps://00..ogg110.1000
Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning, 2018. URL /tttts://aaxiiiiie org/ abw/aa///..7//// 00547755777.075
Shengyi Huang. Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-fil implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022. URL http::::/mm...oro///eeeee
1341.12224 342.html
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Neural Information Processing Systems, 2019.
Alex Irpan and Xingyou Song. The principle of unchanged optimality in reinforcement learning generalization, 2019. URL httttp:://axxiv.orggg wttp:/wwwww..r//////...... 00336
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay, 2021. URL
03934
Yiding Jiang, Zico Kolter, and Roberta Raileanu. On the importance exploration for general- ization in reinforcement learning, 2023. URL https::/arxxi.....t org/ ottp://bbb//..0550//33 05483
2018. Christian Kauten. Super Mario Bros for OpenAl Gym. GitHub, 2018. URL httpp:://t/tthhb
gym super-maris www.i-brriis-asee
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey zero-shot gener- alisation in deep reinforcement learning. J. Artif. Int. Res., 76, May 2023, ISSN 1076-9757. doi: 10.1613/jair. 1.11774 URL hollhttp: /doi. org 10. 01000..6111.. /aaoe//aar.... 1.
Henri Lebesgue. Integrale, Longueur, Aire. PhD thesis, PhD Thesis. Universite de Paris, 1902.
Xiang Li, Wenhao Yang, and Zhihua Zhang. regularized approach to sparse optimal policy in reinforcement learning, 2019.
Haoye Lu, Daniel Herman, and Yaoliang Yu. Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540))5529-555 533, February 2015. ISSN 1476-4687. doi: 10. 1038/nature 14236.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: framework and survey, 2020. URL
https::/erxivv http:///wwxivvvoooo./o/////o//// 04960
12

---

Published as a conference paper at ICLR 2025
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl. Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning, 2019. URL https:::/axxx..orrtttt
abs/1810. 18280.22222.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Picter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3803-3810, 2018. doi: 10.1109/1CA 2.8.440052...
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement httpo//ggaa///0.....aaaaa
learning, 2021.
Mathieu Reymond, Eugenio Bargiacchi, and Ann Nowé. Pareto conditioned networks. In Pro- ceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, pp. 1110-1118, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781 177192339.
Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision-making. J. Artif. Int. Res., 48(1):67-113, oct 2013. ISSN 1076- 9757.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 525-536 Red Hook, NY, USA, 2018. Curran Associates Inc.
Joar Skalse and Alessandro Abate. On the limitations of markovian rewards to express multi- objective, risk-sensitive, and modal tasks, 2024. URL
14811
Rohan Subramani, Marcus Williams, Max Heitmann, Halfdan Holm, Charlie Griffin, and Joar Skalse. On the expressivity of objective-speeccfccatioo formalisms in reinforcement learning, 2024.
URL
wbtp://///.//////// 2140 11840
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
F. Tanaka and M. Yamamura. Multitask reinforcement learning on the distribution of mdps. In Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium (Cat. No.03EX694), volume 3, pp. 1108-1113 vol.3, 2003. doi: 10. 10.1109/CIRA..20 2003. 1222152.
Yujin Tang, Duong Nguyen, and David Ha. Neuroevolution of self-interpretable agents. In Proceed- ings of the 2020 Genetic and Evolutionary Computation Conference, GECCO '20. ACM, June 2020. doi: 10. 0091149//377999.3388947, URL htt.::tt////e/ doi htt.://.....ggg/.0111///////9990 3389847
Cem Tekin and Eralp Turgay. Multi-objective contextual bandits with dominant objective. In 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6, 2017. doi: 10.1109/MLSP22017....
Jayden Teoh, Wenjun Li, and Pradeep Varakantham. Improving environment novelty quan- tification for effective unsupervised environment design. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 135299 135333. Curran Associates, Inc., 2024. URL httpps://proceedd.nnn nwur cc httpc/papr_fflessseaeeeee wtt.fff/ppeee//////// Paper Conference.pdf
Gabriele Tiboni, Pascal Klink, Jan Peters, Tatiana Tommasi, Carlo D'Eramo, and Georgia Chal- vatzaki. Domain randomization via entropy maximization. In The Tweifth International Confer- ence on Learning Representations, 2024. URL httpp::///////o //ttp::///penzeveww wwt.:nrrueetiiiettet. forummid
GGtmuiVrOM
13

---

Published as a conference paper at ICLR 2025
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 23-30, 2017. doi: 10. (009/IRSS.001..82000
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10. 2009//RRS..012.666666
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint 1MXXv:2407..10333 2024.
Eralp Turgay, Doruk Oner, and Cem Tekin. Multi-objective contextual bandit problem with sim-
ilarity information. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings of the Twenty-Firs, International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 1673-1683 PMLR, 09-11 Apr 2018. URL pttps::/proceecinng wrw.spppee//eeee v84 wtw.g/tttaayyaaeeeaeeoooagaaaaaa htmmmmllml.
Peter Vamplew, John Yearwood, Richard Dazeley, and Adam Berry. On the limitations of scalarisa- tion for multi-objective reinforcement learning of pareto fronts. In Wayne Wobcke and Mengjie Zhang (eds.). A/ 2008: Advances in Artificial Intelligence, pp. 3p233723788 Berlin, Heidelberg, 2008
Springer Berlin Heidelberg. ISBN 978-3-540-89378-3.
Peter Vamplew, Richard Dazeley, and Cameron Foale. Softmax exploration strategies for mul- tiobjective reinforcement learning. Neurocomputingg 263:74-86, 2017. ISSN 0925-2312. doi: https::/doi.org//00011///// w0tp:///....0000011001 htt.:///o..eccomm010.......... URL http::://www sciencedirect.. Multiobjective Reinforcement
Learning: Theory and Applications.
Peter Vamplew, Benjamin Smith, Johan Källström, Gabriel Ramos, Roxana Rădulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick Mannion, Pieter J. K. Libin, Richard Dazeley, and Cameron Foale. Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021). Autonomous Agents and Multi-Agent Systems, 36(2):41, July 2022. ISSN 1573-7454. doi: 1.10077/104588-22--09555
Peter Vamplew, Cameron Foale, and Richard Dazeley. Value function interference and greedy action selection in value-based multi-objective reinforcement learning. arXiv preprint arXiv:2402.06266, 2024.
Van Veldhuizen and David Allen. Multiobjective evolutionary algorithms: classifications, analyses, and new innovations. PhD thesis, Department of Electrical and Computer Engineering. Graduate School of Engineering. Air Force Institute of Technology, 1999.
DJJ White. Multi-objective infinite-horizon discounted markov decision processes. Journal of
Mathematical Analysis and Applications, 89(2):639-6477 1982. ISSN 0022-247X. doi: https: http:///...111000022020200000000
URL. https:: www. sci encedi rect com/ sci sct.ennc///rrrccl///2///22222222
Jie Xu, Yunsheng Tian, Pingchuan Ma. Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Ma- chine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 10607-10616. PMLR, 13-18 Jul 2020.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. generalized algorithm for multi-objective reinforcement learning and policy adaptation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 16610-1462.. Curran Associates, Inc., 2019.
Wenhao Yu. Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification, 2017. URL https: / axxiv. wttp://wwwgg/////////oaa/ 02453
14

---

Published as a conference paper at ICLR 2025
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning, 2018. htLtttaaa//xxiiii wttpgb////...0000001/ 0ttp//8806.999933
abs
Luisa Zintgraf, Timon Kanters, Diederik Roijers, Frans Oliehoek, and Philipp Beau. Quality as- sessment of MORL algorithms: utility-based approach. In Benelearn 2015, Proceedings of the 24th Annual Machine Learning Conference of Belgium and the Netherlands, 2015.
Eckart Zitzler and Lothar Thiele. Multiobjective optimization using evolutionary algorithms comparative case study. Agoston E. Eiben, Thomas Bäck, Mare Schoenauer, and Hans- Paul Schwefel (eds.), Parallel Problem Solving from Nature PPSN V. pp. 292-301, Berlin, Heidelberg, 1998. Springer Berlin Heidelberg. ISBN 978-3-540-49672-4.
15

---

Published as a conference paper at ICLR 2025
A ANALYSIS OF FAILURE MODES IN MORL APPROACHESS
In this section, we seek deeper understanding failure modes within the current MORL algo- rithms that can hinder generalization. We caution readers looking to further MORL generalization to be wary of them and encourage exploration to solve these failure modes. Note that we will only discuss challenges that are unique to generalization within MORI, and problems pertaining to the broader RL generalization literature excluded.
Pareto Archival Methods MORL methods often maintain a Pareto archive-a set of nondom- inated policies discovered during training. This archive is constantly updated by comparing the value vector of new policies with old ones, and discarding the dominated ones. This archive can then aid the agent's search process within the objective space, or be used as solutions during test time. This technique is commonly used in multi-objective evolutionary algorithms like PGMORL and MORL/D. Similarly, GPI-LS and GPI-PD track finite convex subset of the Pareto front where dominance defined only for linear utility functions. However, when extending these methods to a MOC-MDP where each context has its own optimal Pareto front- current archiving mechanisms can lead to suboptimal outcomes. Most MORL literature assumes a static environment, so existing Pareto archival mechanisms are not designed to handle context variability in MOC-MDPs. As result, the archive overrepresents policies that perform well in a narrow set of contexts with higher reward scales or lower difficulty, while discarding those optimal for more challenging or less re- warding contexts. This has severe implications as it will cause the agent to converge to a maximax strategy, adopting policies that are only optimized yield the best of the best possible outcomes during test time, and results in poor generalization across the entire range of contexts in the MOC- MDP.
Reliance on Linear Scalarization The convexity of the induced value functions' range deter- mines if MORL algorithms relying on linear scalarization (LS), are capable of finding all policies corresponding to the optimal Pareto front (Vamplew et al., 2008 Roijers et al., 2013). Lu et al.) (2023) showed that in the static-enviromment setting, the induced value functions' range of stochas- tic stationary policies in a MOMDP is convex, which means LS is not a bottleneck for approximating the Pareto front. If the learning objective is maximize the expected (average) multi-objective value function across contexts in a context-agnosti
manner, the results from Lu et al. (2023) can be di- rectly applied to show that the range of expected value vectors in MOC-MDP remains convex. This follows trivially from the linearity of the expectation operator, which preserves convexity. However, our maximization objective is the recovery of the optimal Pareto Front or coverage set across all contexts, the policies that the agent learn may need to be non-stationary and/or ManMarkoviann (fur- ther discussed in Section B). Prima facie, in cases where the policies exhibits nonlinear dependence on state-action history, methods relying solely on LS would be insufficient to identify all globally Pareto-optimal policies in a MOC-MDP. We encourage future research to further investigate the viability of LS in MORL generalization and to explore alternative approaches capable of approxi- mating the Pareto front without convexity assumptions, such as nonlinear scalarization methods or evolutionary algorithms.
Value Function Interference Within state-of-the-art MORL, many approaches extend value- based scalar RL. algorithms such as Q-learning or Deep Q-Networks to handle vector rewards. If the utility function allows actions with widely differing vector outcomes to map to the same utility value, then the vector value function learned for earlier states may be inconsistent with the actual optimal policy (Vamplew et al. 2024). This problem particularly likely to arise in environments which We note that for MOC-MDPs, the dynamics and rewards ob-
are stochastic or vartially-obervabbl served by the agent may appear to be stochastic even if the underlying MDPs are deterministic, due to the influence of the hidden context variables. Therefore, value function interference may pose particular problem when naively applying value-based MORL algorithms to MOC-MDPs. We note that if the utility function is linear then value interference does not impact on selecting the optimal action, hence there is an implicit tension between this failure mode, and the issues of reliance on linear scalarisation raised in the previous paragraph.
16

---

Published as a conference paper at ICLR 2025
B PRINCIPLE OF UNCHANGED PARETO OPTIMALITY
When constructing benchmark in reinforcement learning, it is essential that the domain adheres to The Principle of Unchanged Optimality (Irpan & Song, 2019) fundamental yet underappreciated principle. This principle states that, for a domain to support true generalization, it must provide all necessary information such that an optimal policy exists in every context. In the MOC-MDP framework, The Principle of Unchanged (Pareto) Optimality implies the existence of globally Pareto optimal policies, '², such that:
Ve P'.
where IT is the set of feasible policies and (II denotes the Pareto set containing nondomi- nated policies for a given c E C. This principle has significant theoretical implications. When the unchanged optimality principle is disregarded,, the benchmark can become a proxy measure of the memorization capability (Zhang et al. (2188) of the MORL agents, instead of generalization. More- over, if the principle is violated, generalist agents would never achieve an aggregated NHGR score of 1, since they would fail to recover Pareto optimality across all contexts.
This section examines how our MORL-Generulization benchmark upholds The Principle of Un- changed Pareto Optimality, thereby validating our baseline evaluations. Note that each context in our benchmark varies in initial state distribution, transition dynamics, and multi-objective reward function. When the context is fully observable, the agent can simply include the context in its state representation, enabling it to learn "universal" policies that adapt across contexts. In scenarios where the context is hidden, the agent must infer it from its observations. Therefore, to respect The Principle of Unchanged Pareto Optimality, sary information about the context can be recovered from the agent's observations in the proposed domains.
benchmark is designed to ensure that neces-
In MO-SuperrMarioBrss, despite visual similarities across levels, each observation provides suffi- cient information determine the optimal action at every time step. For example, the locations of coins, enemies, and bricks are clearly visible. Moreover, since there are only a finite number of stages (32), the agent can deduce its current stage directly from its observations with enough training. Similarly, in MO-LavaGrid, the complete layout of lava and goals, along with the agent's s position and orientation, fully observable at each time step. Furthermore, as described later in Section F we concatenate the reward weights for each goal with the agent's 's observation, ensuring that the current reward function is explicitly provided.
For the continuous control domains like MO-Hopper, MO-HalfCheetah, and MO-Humanoid, con- text variations arise from changes in dynamics (c.g., gravity, friction), yet the agent's observations typically include only joint positions and velocities. Consequently, optimal actions cannot be in- ferred from a single time step. A similar limitation exists in the discrete domain MO-LunarLandde where the observations are typically restricted to orientation and velocity. The environment dynam- ics is, however, inferable when the agent considers its state-action history. Prior work has shown that history-based policies are effective in domains with changing dynamics (Yu et al.) 2017 Peng eet al., 22018] Tiiooi et al., 2024).. Therefore, we adopt the standard approach of augmenting the state with fixed-length history of past state-action pairs. In our main experiments for MO-Hopper, MO-HalfCheetah, MO-Humanoid, and MO-LunarLander, we use a history length of 2 so that the observed state at time is a vector of the form: (St-2, 0z-2, 8t- 0 0t-1, 8t). For time steps before 2, we repeat the initial state and pad missing actions with zeros.
We recommend that future researchers verify that The Principle of Unchanged Pareto Optimality is upheld when using our software to study MORL generalization. This is crucial for establish- ing benchmarks that accurately assess an algorithm's capacity to generalize across diverse multi- objective domains.
C FUTURE WORK AND LIMITATIONS
Our extensive evaluations of current MORL algorithms on the introduced benchmark, have high- lighted a significant gap in generalization capabilities. The suboptimal performance observed un- derscores the necessity for innovative approaches to enhance MORL generalization.
17

---

Published as a conference paper at ICLR 2025
A promising direction for future research involves adapting established methods from single- objective RL generalization to the multi-objective context. Techniques such as regularization tech- niques (Cobbe al., 2019; Ahmed et al., 2019; Li et al.
2019 [gl et al.] [2099] Eysenbach et al. 2021 incorporating inductive biases (Tang et al. 2020) Raileanu & Fergus] 2022 Higgins et al.
2018 and curriculum learning methods (Narvekar et al. 1020:Jiang al.)
Teoh et al. 2024) have demonstrated efficacy in single-objective settings and could be tailored to address the complex- ities inherent in MORL.
Beyond adapting existing methods, there is a need to develop specialized techniques targetted to- wards MORL. As highlighted in Section A] many current methods rely heavily on linear scalar- ization, potentially limiting the generalization potential of MORL agents. Exploring alternative approaches that move beyond this constraint can be promising direction. Recently, evolution- ary methods such as those proposed by Xu et al. (2020) and [Fetten et al.] (2024) have been in- troduced, but they remain underexplored in the MORL literature and warrant further investigation to enhance generalization. Effective exploration is critical for generalization in RL (Jiang et al.) 2023). Thus, approaches like Vamplew et al. (2017), which incorporate exploration techniques from single-objective RL into the MORL framework could be of interest to future research. In many real-world scenarios, agents operate under partially observable contexts, such as the dynamics of the environment, just described in Section B Developing MORL algorithms capable of adapting to partially-observable contexts is crucial for generalization. Therefore, future work can take inspi- ration from the POMDP or model-basee RL literature to develop methods capable of inferring the hidden context and recovering Pareto optimality.
Limitations Our MORI-Gennraaizzafi benchmark is built using Gymnasium's API, which has become the standard interface for reinforcement learning due to its accessibility. However, Gymna- sium relies on CPU-based policy rollouts, leading to significant computational bottlenecks caused by frequent GPU-CPU data transfers. This limitation is particularly problematic for MORL algo- rithms, which generally require longer training times. Most of our experiments complete within 1-2 days, with all runs kept under five days on a single NVIDIA RTX A5000 GPU and a 48-core AMD EPYC 7643 CPU. Given these computational constraints, exhaustive hyperparameter tuning is impractical. Instead, we tune hyperparameters within the neighborhood of those already validated by the MORL-Baselines (Felten al. 2023) library in single-environment settings, from which we also derive the implementations of the baseline algorithms. Future work on extending the MORL- Generalization domains to run directly on GPU, enabling fully end-to-end MORL training, would significantly benefit the community.
D EXTENDED METRIC DISCUSSIONS AND RESULTS
In this section, we extend our discussions on the benefits of using the NHGR metric. We also pro- vided utied evaluation metric for measuring generalization performance. However, as a pioneering effort, our aim is not to prescribe single definitive metric for evaluating generalization in future research, but rather to establish well-justified and flexible options. The choice of perfor- mance metric has long been debated in multi-objective optimization, as assessing the quality of Pareto fronts inherently more complex than single-objective evaluations single metric can fully capture all aspects of performance. To provide a comprehensive assessment, we report results using multiple metrics, including hypervolume, expected utility metric (EUM), NHGR, and EUGR, as shown in Table Additionally, while not explicitly included in our results, our software supports other evaluation metrics such as cardinality and sparsity for further analysis.
D.1 EXTENDED DISCUSSIONS ON BENEFITS OF NHGR
In Section we introduced the Normalized Hypervolume (HV to ensure equal weighting across objectives in hypervolume calculations. This is achieved by normalizing reward scales using the minimum and maximum values derived from the optimal Pareto front (or approximation thereof) before computing hypervolume, Ensuring equal weightage across objectives when measuring gener- alization performance is important. This is because, unlike in single-objective RL, MORL requires agents to generalize across both environments and objectives, making equal weighting crucial to ensure that they learn all trade-offss across objectives effectively. Additionally, HV nern removes the
18

---

Published as a conference paper at ICLR 2025
dependence arbitrary reference point often used in hypervolume calculations. Instead, the origin vector can be used, as all objective ranges are normalized to the [0, 1] interval.
While HV norm addresses scale bias and reference point dependence, it introduces does not account for variations in the maximally achievable HV norm across contexts. Intuitively, contexts with more convex Pareto fronts allow for higher maximum HV values compared to those with concave fronts. Naively aggregating HV nom scores across contexts without accounting for these differences skews evaluations toward contexts with higher achievable hypervolume, un- fairly penalizing agents that balance learning across all enviromentss the goal of generalization.
challenge:
To address this, we proposed the NHGR metric, which adjusts for these discrepancies by comparing the generalist agent's performance against the maximum achievable HV norm- derived from the opti- mal Pareto front, in each context. By evaluating performance as ratio of this maximum, NHGR ensures that contexts with inherently lower achievable hypervolumes are weighted equally against those with higher achievable hypervolumes, enabling fair and unbiased generalization assessments across all contexts.
Table T presents the mean performance of the GPI- LS (Alegre et af.) (023) algorithm across three envi- ronments in the MO- the raw HV metric exhibits performance differences on the order of 10s between environments. This disparity arises because small variations in reward ranges com- pound in hypervolume calculations, partieularly as the number of objectives increases, reducing interpretabil- ity. While nom mitigates this issue by normalizing objective ranges, it unfairly penalizes the generalist in the Hard and Slippery environment, where the optimal Pareto front has a significantly lower hyper- volume compared to other contexts. In contrast, the NHGR metric offers a more balanced assess- ment by evaluating the generalist's normalized hypervolume relative to the maximum achievable value. This ensures fair comparisons across all contexts. Additionally, from generalization per- spective, NHGR more interpretable, as directly quantifies how close an agent is to achieving the optimal performance in each evaluation context.
Default
Hard
Slippery
domain. As shown,
HV
1.4e
5.
9.3e
HV nomm
0.20
0.065
0.094
NHGR
0.26
0.10
0.14
Table I: Illustration of different metrics MO-HalfCheetah environments.
Domains
Metries
PCN
CAPQL
PUMORL
Envelope
GPL-
LS
GPI-
PD
SD
MORL/D
MORL/D SB+PSA
(10')
6.09 1.2.8
N/A
NIA
4.14 0.65
9:08 0.55
7.02 0.47
8.52
8,55 0.48
MO-Lumar
(UM()00)
1.83
0:25
NHGR
0.06
0.06
0.07
0.07
0.05
0:003
0.00 0.00
0.50
0.14
- 0.10
0.27 + 0.13
0.25 0.10
HV (10F)
0.44 0:006
1.40 0.20
0 = 1118
NA
1.32 2: 0.25
118 0.24
1.45 :: 0.26
1.57 :: 0.27
MD--
EUM (10F)
0.59 0.07
1.45 0.14
LOIT = 0.14
NA
1.3 1: 0.14
126
1 0.17
1.45 1 0.15
1.50 1: 0.15
Ilopper
NHGR
0.00 +.003
0.50 0.10
0.20 0.00
NA
0.46 + 0.15
0.37 0.11
0.46 0.14
0.54 + 0.15
EUGR
0:31 1.006
0.77
0.57
NAX
0.74
0.10 0.66
0.77
0.80 0.12
(10")
(10')
0.005
-2.86 5:2.4
N/
0.66 6.64
NIA
3,28
4.27
NHGR
0.03
0.07 0.02
N/A
NIA
0.21
0.29
DUGR
0.00 1.000
0.00 0.00
0.00 0.01
N/A
0.11
0.12
NIA
0.16
1.122
0.20 0.13
IEV (10')
3.50 0.27
5.14 0.79
0.15 0.32
NA
1.33 0.09
NIA
4.86 1. 1.08
4.78 1.15
MO-
(10')
0.96 0.0.8
0.18 1
0.55 0.25
NIA
0.21 0.000
NIA
1.27
1.24
Humanoid
NHGR
0.07
0.44
0.20
0.56
0.55
0.18
(10")
NIA
NAA
MCI-Super
CUM (10*)
1.22 1.881
N/A
NIA
1.41 1.7.0
1,30 0.52
NIA
NIA
N/A
MarooDros
NHGR
0.04 0.006
N/A
NIA
0.08
= 0.08
0.12 + 0.14
NIA
NIA
N/A
FUGR
0:40 0.16
N/A
NIA
0.47
2 0.14
D.4T 2: 0.19
NIA
NIA
N/A
HV (10")
1.12::00.7
N/A
N/A
1.10 0.57
1.12+
NIA
3.54 2: 1.88
2.76
MO-
-2.0
0.55
LavaGrid
NHGR
0.02
0.006
0.27 0.28
0.04
Table 2: Performance of various MORL algorithms across different domains and evaluation metrics. Higher values indicate better performance for all metrics. Each entry indicates the mean and stan- dard deviation computed over independent runs. Bolded values fall within one standard deviation of the best mean.
19

---

Published as a conference paper at ICLR 2025
D.2 EXPECTED UTILITY GENERALIZATION RATIO
When good prior over possible user utility functions is known, The Expected Utility Metric (EUM) proposed by ZZintgraf al. (2015) can be used for evaluating MORL algorithms. This metric calculates the expected utility of the agent's approximate Pareto front under the prior distribution of user utility functions, parameterized by a weight space W. A higher EUM indicates that the policies yields better expected utility across the user utility functions. Under the SER criterion, the expected utility of the approximate Pareto front produced by a MORL agent is given by:
FUM = Www- max "(v", w)
Lv
where w is the user's utility function, and v is the expected value vector of policy a within the Pareto set.
There are several scenarions in which the EUM can be applied. In practical applications, the utility function of the stakeholders might be known due to domain knowledge. Using EUM would therefore allow for more direct evaluations on how the solutions generated by the MORL agent corresponds improving the utility of the stakeholders. Not every point on the Pareto front would contribute to an increase in the EUM for given utility function. In addition, the hypervolume metric is known for its computational challenge especially in higher dimensions, although various approximation algorithms and heuristics have been developed to estimate the hypervolume more efficiently. The EUM, on the other hand, depends only on the number of solutions on the approximate Pareto front and the number of weights sampled from the weight space.
As mentioned Section of the main body, when aggregating performances across multiple con- texts for measuring generalization, must ensure that each context is equally attributed. Specif- ically, we can calculate a variant on the NHGR metric we call the Expected Utility Generalization Ratio (EUGR).
Definition 5 (Expected Utility Generalization Ratio). Let Fe and F be the approximate Pareto front obtained by generalist MORL agent and the optimal Pareto front for context c respectively. The EUGR for the agent in is defined as:
EUM(F.) EUM(
EUGR Fz)
Unlike in NHGR, the Pareto front is not normalized here. This is because the utility functions used in the EUM should already inherently reflect the stakeholders' preferences over objectives, including their relative importance.
Since there is no clear utility function distribution that is suited for toy domains like those in our benchmark, we followed standard conventions and evaluated EUM and EUGR in Table using linear utility functions with weights summing to unity:
u( ulv". w) = wTw Tv where
Σw;
Σ ww = w;
2 0, i 1,...,k k
where & the number of objectives. Specifically, we scalarized the value vectors the approximate Pareto front obtained by each algorithm at the end of each run using 100 weight vectors sampled uniformly from the unit weight simplex to compute the expected utility.
D.3 SINGLE OBJECTIVE UTILITY FUNCTION RESULTS
As discussed in Section most classic SORL domains in Gymnasium inherently perform an implicit scalarization of multiple objectives when defining the scalar reward function. As such, for every domain, we track and plot the single-objective scalarization function fsORL returns for all MORL algorithms, well as for the SAC algorithm, across all evaluation episodes throughout training. Across many domains in our benchmark, we observe that the leading MORL algorithms could outperform achieve comparable performance to the single-objective SAC algorithm in terms of maximum fSORi. return. Below, we present the fSORl. equations for each environment along with their corresponding plots.
20

---

Published as a conference paper at ICLR 2025
D.3.1 MO-HOPPER
The default single objective utility function of the MO-Hopper domain is same as the one used in Gymnasium's Hopper, which is
fSORL 1.5u 0.001c h
where Ug is the forward speed, e is the control cost and h is the reward for staying alive.
MD-HepperDeauutt
MO-HHpperLugg
Hopperieary
HD-HHperrSpppryy
MO-Hepperiard
5438
2300
2000
3300
2983
1200
2393
1000
1300
920
1993
2000
630
600
400
Single
239
500
Staps
Stess
Stops
Steps
Stops
Staps
NONL-D5B-1P5A
MORL-D138)
GR-PD
GR--5
PONON
CAPOL
PCN
SAC
Figure 7: oingle-b return on HO-Hopper testing environments during training. Each curve measured across 5 seeds (mean and standard error).
D.4 MO-HALFCHEETAI
The default single objective utility function of the MO-HalfCheetahh domain is same as the one used in Gymnasium's HalfCheetah, which is
fSORL = 1.Ov + 0.1c
where es the forward reward and is the control cost. The HalfCheetah is always alive so has no alive reward.
NO. No-HHrCChestaOsfouit
MO-HafChheetaniigt
MO Wo-HaatChettttileayy
MO Ho.-HatChestat@pppry
M/ MC-HHrtChestah-trd
3405
490
1263-
000
2000
200)
2000
KCC
2000
200
1000
5000000
5333232
5000000
5030333
5E00000
Steps
Steps
Steps
Steps
Figure& 8: obngle-obeective return on 5 MO- HO-HalfCheetah testing environments during training. Each curve is measured across 5 seeds (mean and standard error).
D.4.1 MO-HUMANOID
The single objective utility function of the MO-Humanoid domain is
SSORL. 1.25v, 0.001 + 2.0h
where is the forward speed, is the control cost and h is the reward for staying alive. The original Gymansium's Humanoid domain uses a 5.0 coefficient for the alive reward but we tuned it down to because it dominating all the other objectives in terms of magnitude. We have also verified that the convergence the SAC agent on the original single-objective Humanoid environment remains unchanged with this lower alive reward.
21

---

Published as a conference paper at ICLR 2025
MC-Hlumano dDetaut
MC-.I.mansd..qe
NO ho--fumandddheary
MD www.tuaaaaddiiioooo www.arrddeeoOmpiinn
NO Iumannidfaan
2566
1790
2000
2040
1000
255
3550
2000
3000
352
500
10000000
19933230
TECEC000
10933333
Steps
Steps
MORL D158
MORL
G5)
PGMORL
PCN
SAC
Figure 9: objeleee return on 5 MO-Humanoid testing environments during training. Each curve is measured across 5 seeds (mean and standard error).
D.4.2 MO-LUNARLANDER
The default single objective utility function of the MO-LunarLander domain is same the one used Gymnasium's LunarLander, which
fSORL I + + 0.3mc + 0.03sc
where is a 1000+ 100 one-time reward the lander lands successfully or crashes, s the shaping reward, me is the main engine cost and sc is the side engine cost.
Single-ohjective return on 8 MO-Lunarl.ande testing environments during training. Each curve is measured across 5 seeds (mean and standard error).
Figure 10:
D.4.3 MO-SUPERMARIOBROS
The default single objective utility function of the MO-SuperMarioBros domain is same as the one used in Gym Super Mario Bros (Kauten 2018), which is
fSORL = +
where f is forward reward, / is the time penalty, d is the death penalty.
Figure 11: Single-ohjective return on MO-SuperMarioBros (abbreviated as MO-SMB) testing environments during training. Each curve is measured across 5 seeds (mean and standard error).
E TRAINING DETAILS
Table shows shared training hyperparameters across algorithms for each domain in the MORL generalization benchmark. The scripts to reproduce the results this paper are provided in the codebase, alongside with specific hyperparameters for the different algorithms. To have fair evaluations, we utilize the same architectures for the policy and value functions across all algorithms for each domain. Specifically, for MO-LavaGrid, ho-LLunarlaader MO-Hopper, Mo-HalfCheetah,
22

---

Published as a conference paper at ICLR 2025
and HO-Humanoid, the policy and value functions multi-layer perceptrons (MLPs) with four hidden layers of 256 units each. For MO-SSuerMarioBros which has image observations, the policy and value functions consist of a NatureCNN (Mnih al. 2015] followed by MLP with two hidden layers of 512 units each. For off-policy algorithms that depend on experience replay, we ensure the same replay buffer size is used. We direct researchers to codebase for detailed scripts and hyperparameter settings used in training each bascline algorithm for the evaluations in Section 6
MO-
MO-Lunar Lander
MO-Super MarioBros
MO- Hopper
MO-Half Cheetah
MO-
Parameter
LavaGrid
Humanoid
Discount 7
0.995
0.99
0.99
0.99
0.99
0.99
Adam lcarning rate
3e
3c
3e
3e
3e
3e"
Adam
1e
1e
1e
le
1e
le
Batch Size
128
128
64
256
256
256
Replay buffer size
1e
1ec
1e
1e
1e
1e
Max episode steps
256
1000
2000
1000
1000
1000
Env Steps
5e
3e
3e
3e
5e
1e
Table 3: Hyperparameters used for training on MORL generalization benchmark.
F MORL-GENERALIZATION BENCHMARK DETAILS
In this section, we begin by providing overview of the domains in MORL-Generalization benchmark, highlighting the distinet context variations each domain introduces. Next, for each domain, we detail the environment parameters used to create their evaluation environments. The code commands to initialize these environments using Gymnasium are also included within our codebase.
F.1 BENCHMARK DOMAIN DETAILS
Kirk et al. (2023) identified four key types of domain variations for studying generalization: 1) state- space variation (S), which alters the initial state distribution, dynamics variation (D), which alters the transition function, 3) visual variation (O), which impacts the observation function, and 4) reward
function variation (R). We provide detailed descriptions of the benchmark domains introduced in Section beloo:
MO-Lunare HO-LunarLander (D+S) This is multi-objective adaptation of Gymnasium's LunarLander do- main where the agent has to balance between successfully landing on the moon surface, the stability of the spacecraft, the fuel cost of the main engine, and the fuel cost of the side engine. The agent spaces. We introduce 7-dimensional pa-
operates over snlcreteeaction and continuous-observalte rameter that varies the environment's gravity, wind power, turbulence, and the lander's main engine power, side engine power, and initial coordinates.
MO-Hopper (D) This is multi-objective adaptation of Gymnasium's Hopper domain. The one- legged agent must balance optimizing for its forward velocity, torso height, and energy cost. The agent operates over continuous action and observation spaces. We introduce an 8-dimensional pa- rameter that varies the hopper's body masses (4D). joint damping (3D), and the floor's friction.
Mo-HalfCheetah (D) This is multi-objer multi-obeecctiv adaptation of Gymnasium's 's HalfCheetah domain. The 2-dimensional cheetah robot must balance optimizing for its forward velocity and energy cost. The agent operates over continuous action and observation spaces. We introduce an 8-dimensional parameter that varies the cheetah's 's body masses (7D) and the floor's friction.
23

---

Published as a conference paper at ICLR 2025
MO-Humanoid (D) This is multi-objective adaptation of Gymnasium's Humanoid domain. The humanoid robot must balance between optimizing for its forward velocity and its energy cost. The agent operates continuous action and observation spaces. We introduce 30 30-dimensional envi- ronment parameter that varies the humanoid's body masses (13D) and joint damping (17D).
multi-objective adaptation of the Gym Super Mario Bros (Kauten 2018) domain based on the popular Super Mario Bros video game. The agent has balance between moving forward, collecting coins, and increasing the game score (by stomping enemies, breaking bricks, etc.). The agent operates over discrete-action and discrete-observation (pixel images) spaces. We introduce a 2-dimensional parameter that controls which stage of the Super Mario Bros game to place the agent in. There are total of 32 possible stages.
MO-SuperMarioBros (S) This
MO-LavaGrid (S+R) This is novel multi-objective domain based on MiniGrid (Chevalier- Boisvert et al. 2023). The agent (red triangle) has to navigate a 11 11 grid, incurring penalty each time it touches lava and another for every step takes to collect 3 goals (blue, green, and yellow blocks), after which the episode terminates. The placements of the agent, goals, and lava blocks are fully configurable, providing diverse evaluation contexts. Additionally, we introduce a 3- dimensional parameter controlling the reward weight of each goal. These weights are concatenated with the state space, ensuring the agent has the necessary information about the reward function to plan its trajectory while balancing goal collection and lava avoidance. The agent operates over discrete-action and mixed continuous-discrete (because of the reward weights) observation spaces.
F.2 MO-LAVAGRID
The environment parameters for the MO-LavaGrid domain are represented using bit maps, which are unable directly translate into this paper. Instead, the evaluation environments visually shown in Fig. 12 Also, as mentioned in [ the MO-LavaGrid environment has a 3-dimensional parameter controlling the reward weightages of each goal square (green, blue, yellow). The reward weights for each goal are concatenated to the state space of the agent, and the weights sum to unity. The reward weightages for each goal in each evaluation environment are shown in Table
During training using domain randomization, after each episode concludes, the agent's 's start position and orientation, the number of lava blocks, the placement of the goals and lava blocks, and the reward weightages of the goals are all randomly set. When an agent has collected/visited goal, the weightage of the goal the state space is set to 0, indicate that the reward corresponding that goal has already been awarded.
11
88
88
889
86
Figure 12: MO-LavaGrid Evaluation Environments. Top (left to right): MO-LavaGridSnake. MO-LavaGridRoom, MO-LavaGridSmileey MO-LavaGridMaze. Bottom row (left to right): MO- LavaGridCCheckrrBant MO-LavaGridCCoridon MO-LavaGriddllands MO-LavaGridLabyrint
24

---

Published as a conference paper at ICLR 2025
Environment
Green
Yellow
Blue
MO-LavaCriuSnake
0.20
0.30
0.50
MO-LavaGrridRomm
0.50
0.30
0.20
MO- Lo-LavaGridSmiley
0.40
0.40
0.20
MO-LavaGridMaze
0.05
0.05
0,90
M0-LavaGriuCheckerBaae
0.30
0.10
0.60
MO-LavaGriiCCoridorr
0.60
0.10
0.30
MO-LavaGridlsaands
0.33
0.33
0.33
MO-LavaGridLabyrinnt
0.50
0.05
0.45
Table 4: Reward weightages for MO-LavaGrid evaluation environments.
F.3 MO-SUPERMARIOBROS
In SuSuurMarriBrrs,, each environment configuration is instantiated via 2-dimensional param- eter. The first dimension has discrete values {,, 3, 4,5, 6, 7,8). and indicates the SuperMarioBros world. The second dimension has discrete values (1, 3, 4), and indicates the level within the cho- world. Together, the parameters sworld><<level defines the stage (configuration) of the environment.
During training using domain randomization, environment is randomly selected from the 32 possible stages, except Stage 3-3 which is reserved for zero-shot generalization evaluation. During evaluation, the agents evaluated only 8/32 stages keep the runtime within reasonable limits. The evaluation stages are visually shown Fig. [3] The evaluation stages are carefully selected encompass wide range of environment dynamics and visual renditions. Additionally, they are chosen to ensure that each stage offers non-zero rewards across all objective dimensions. This crucial to prevent hypervolume evaluations from collapsing zero, which would occur if any dimension of the objective space had zero achievable range.
MORIDD
MERILDD
TINE
MORLO
TANG
MERILLO
MOPLLD
Figure 13: Mo-SSuperMarioBros Evaluation Environments. Top row (left to right): Stage 1-2, Stage 3-2, Stage 3-3 (zero shot), Stage 4-3. Bottom row (left to right): Stage 5-2, Stage 5-3, Stage 7-3, Stage 8-1
F.4 MO-LUNARLANDER
In MO-LunarLander each environment configuration is instantiated via 7-dimensional parameter. The dimensions of the environment parameter corresponds the gravity coefficient, wind power, turbulence, the lander's main engine power, the lander's side engine power, the lander's initial x- coordinate, and the lander's initial y-coordinate.
25

---

Published as a conference paper at ICLR 2025
During evaluation, assess the agents performances predefined set of environment figuration:: Default, High Gravity, Windy, Turbulent, Low Main Engine, Low Side Engine, Start Right, and Hard. Table displays the environment parameter values used for each environment configuration.
High Gravity
Low Main Engine
Low Side
Start Right
Parameters
Default
Windy
Turbulent
Hard
Engine
Gravity
-10.0
-13.0
-10.0
-10.0
-10.0
-10.0
-10.0
-12.0
Wind
15.0
20.0
15.0
15.0
15.0
15.0
17.0
Power
Turbulence
1.5
1.5
1.5
3.5
1.5
1.5
1.5
2.5
Power
Main Engine Power
13.0
13.0
13.0
13.0
10.0
13.0
13.0
12.0
Side Engine Power
0.6
0.3
0.6
Initial Coeff
0.5
0.5
0.5
0.5
0.5
0.4
0.75
0.4
Initial Y Coeff
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
Table 5: Environment parameters for MO-LunarLander
F,5 MO-HOPPER
In MO-Hopper, each environment configuration are instantiated via a 8-dimensional parameter that varies the hopper's 's body masses (4D). joint damping (3D), and the floor's friction (1D).
During evaluation, we assess the agents performances on predefined set of 6 environment configu- rations: Default, Light, Heavy, Slippery, Low Damping, and Hard. displays the environment parameter values used for each environment configuration.
Parameters
Default
Light
Heavy
Slippery
Low Damping
Hard
Torso Mass
3.7
0.5
9.0
3.7
3.7
0.1
Thigh Mass
4.0
0.5
9.0
4.0
4.0
9.0
Leg Mass
2.8
0.3
8.5
2.8
2.8
9.0
Foot Mass
5.3
0.7
10.0
5.3
5.3
0.1
Damping 0
1.0
1.0
1.0
1.0
0.1
0.1
Damping
1.0
0.1
0.1
Friction
1.0
1.0
1.0
0.1
1.0
0.1
Table Environment parameters for MO-Hopper
F.6 MO-HALFCHEETAH
In MO- HalfCheetah, each environment configuration is instantiated via 8-dimensional parameter that varies the cheetah's body masses (7D) and the floor's friction (1D).
26

---

Published as a conference paper at ICLR 2025
During evaluation, assess the agents performances predefined set of environment config- urations: Default, Light, Heavy, Slippery, and Hard. Table 7 displays the environment parameter values used for each environment configuration.
Parameters
Default
Light
Heavy
Slippery
Hard
Torso Mass
6.25
0.5
10.0
6.25
6.25
Back Thigh Mass
1.538
0.1
9.5
1.54
9.5
Back Shin Mass
1.441
0.1
9.5
1.59
9.5
Back Foot Mass
0.891
0.1
9.5
1.10
9.5
Front Thigh Mass
1.434
0.1
9.5
1.44
0.1
Front Shin Mass
1.198
0.1
9.5
1.20
Front Foot Mass
0.869
0.1
9.5
0.88
0.1
Friction
0.4
0.4
0.4
0.1
Table 7: Environment parameters for MO-HalfCheetah
F.7 MO-HUMANOID
In MO-Humanoid, each environment configuration is instantiated via a 30-dimensional parameter that varies the humanoid's body masses (13D) and joint damping (17D).
During evaluation, we assess the agents performances on a predefined set of environment con- figurations: Default, Light, Heavy, Low Damping, and Hard. Table displays the environment parameter values used for each environment configuration.
27

---

Published as a conference paper at ICLR 2025
Parameters
Default
Light
Heavy
Low Damping
Hard
Mass
8.91
1.7
10.0
8.91
8.91
Mass 2
2.26
0.5
7.0
2.26
2.26
Mass 3
6.62
1.3
9.0
6.62
6.62
Mass 4
4.75
0.7
4.75
0.7
Mass 5
2.76
0.6
7.0
2.76
0.6
Mass 6
1.77
0.5
6.0
1.77
0.5
Mass 7
4.75
0.7
8.0
4.75
8.0
Mass 8
2.76
0.5
7.0
2.76
7.0
Mass 9
1.77
0.3
1.77
6.0
Mass 10
1.66
0.3
6.0
1.66
0.1
Mass 11
1.23
0.1
5.5
1.23
0.1
Mass 12
1.66
0.3
6.0
1.66
5.0
Mass 13
0.1
5.5
1.23
5.0
Damp 1
1.0
5.0
5.0
1.0
1.0
Damp 2
1.0
5.0
5.0
1.0
1.0
Damp 3
1.0
5.0
5.0
1.0
1.0
Damp 4
1.0
5.0
5.0
1.0
1.0
Damp 5
1.0
1.0
Damp 6
1.0
5.0
5.0
1.0
1.0
Damp 7
0.2
10
1.0
0.2
0.2
Damp 8
1.0
5.0
5.0
1.0
1.0
Damp
1.0
1.0
Damp 10
1.0
5.0
1.0
1.0
Damp 11
0.2
1.0
1.0
0.2
0.2
Damp 12
0.2
1.0
1.0
0.2
0.2
Damp 13
0.2
1.0
1.0
0.2
0.2
Damp 14
0.2
0,2
Damp 15
0.2
1.0
1.0
0.2
0.2
Damp 16
0.2
1.0
1.0
0.2
0,2
Damp 17
0.2
1.0
1.0
0.2
0.2
Table Environment parameters for MO-Humanoid
28


Reviews:
[{"review_id": "qiuwKZ5GPt", "reviewer": "Reviewer_8Wih", "summary": "This paper studies generalization in the context of multi-objective reinforcement learning (MORL), which extends standard RL to the case in which the reward function is a vector containing multiple conflicting objectives. In particular, the paper defines the problem as a contextual multi-objective MDP, in which a context variable defines changes in the MDP’s state-transition and reward functions. The goal of an agent is to learn a set of Pareto optimal policies that generalize over different contexts. Next, the paper introduces a set of environments that extends existing benchmark environments in MO-Gymnasium to the case in which the dynamics and the state space can be modified. The paper evaluates several state-of-the-art MORL algorithms in the introduced benchmark and shows that existing algorithms often do not generalize well to different contexts.", "questions": "Below, I have questions and constructive feedback to the authors:\n\n1) Regarding Definition 3, what does it mean for a policy to be “generalized across contexts”? There are two options here: (i) learn a single policy that maximizes its expected value over contexts, possibly being suboptimal because a policy that is optimal to every context simultaneously likely does not exist; or (ii) learn a policy that is conditioned on the context, and then can adapt its behavior depending on the context. I suggest discussing and clarifying this aspect of the problem defined in Section 3. In case (ii), the environments introduced should contain the context as part of the state space.\n\n2) In Definition 4, I suggest defining how the Pareto front is normalized and mentioning the range of the normalization, e.g., [0,1].\n\n3) Given that, in Definition 3, the authors are restricting to linear utility problems (which induce convex Pareto Fronts), it is worth noting that hypervolume is a metric that considers Pareto-optimal points in concave regions of the Pareto front, which are not useful for increasing the expected utility. For example, if we have an optimal convex Pareto front and we add a point in a concave region, this point would increase the hypervolume but would not increase the expected utility. For this reason, I suggest focusing on expected utility metrics (following the utility-based approach) instead of using hypervolume, which is an axiomatic metric. Currently, it is contradictory that the paper advocates for utility-based approaches, but mainly uses an axiomatic metric for evaluation. I suggest the authors either justify their use of hypervolume in this context or adopt metrics more consistent with their utility-based approach (e.g., normalized expected utility).\n\n4) It is unclear whether (in both the problem formulation and in the proposed benchmark) the context is part of the observation, i.e., whether agents know which context they are in. If the agents do not have access to the context, it is expected that generalist agents do not perform well. In general, a different policy is required to act optimally with respect to each context, and learning a policy that optimizes for the average context will be probably suboptimal to every context (unless contexts are too similar). Hence, the NHGR metric in Definition 5 would never be equal to 1, even assuming access to a perfect algorithm that learns the optimal policy in Definition 3. I suggest the authors clarify whether perfect generalization (NHGR=1) is theoretically possible in these environments.\n\n5) Regarding the MO-SuperMarioBros, the paper mentions that “There are a total of 32 possible stages”. Is that correct? Based on the Appendix, there are 8 stages.\n\n6) In Section 6.1, what was the training budget in terms of environment interactions for the specialist and generalist agents? Since generalist agents have to learn to solve multiple contexts instead of only one, it would be fair that they are given sufficient time to learn them. Otherwise, would it be possible to explain the results solely on the fact that the generalist agents did not have enough training time?\n\n7) I also suggest discussing the hyperparameters used for the algorithms in the experiments. For instance, would it be necessary to use larger neural networks or larger replay buffers when we train an agent to optimize for many contexts simultaneously? It is not clear whether the algorithms do not perform well due to a lack of hyperparameter turning.\n\nMinor: “long-term discounted reward” - > long-term discounted sum of rewards.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This is the first work, to the best of my knowledge, that studies MORL under environments with changes in its dynamics/state-transition function. \n- The introduced benchmarks are potentially useful to the community for designing novel MORL algorithms that can tackle problems in which the MOMDP's dynamics can change.", "weaknesses": "- Since the main contribution of the paper is the testbed containing new extensions of MORL benchmarks and an extensive evaluation of existing algorithms, it is possible that this paper would be more suitable to the Dataset & Benchmarks track than the main track of the conference.\n- The theoretical contributions are limited to the definition of a contextual MOMDP, which follows directly from previously studied formalisms of MOMDP and contextual MDPs, and the definition of a normalized evaluation metric for evaluating agents in this formalism.\n- The paper requires clarifications regarding the mathematical formulation of the problem, and its implications on the evaluation metrics used and the experimental results (see below).\n- In particular, it is not clear if the existing MORL algorithms are performing poorly due to their limitations or if the proposed benchmark is actually impossible to solve with optimality (i.e., obtaining NHGR=1). For instance, in the Mujoco environments, it is impossible for an agent to act optimally w.r.t. all possible friction coefficients without having any form of access to the current coefficients.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zqmyURZdfK", "reviewer": "Reviewer_yE1f", "summary": "This paper investigates a very important problem of generalization in multi-objective reinforcement learning (MORL), an area that has received limited attention compared to single-objective RL. The main contribution of the paper includes a novel testbed featuring diverse MORL domains with different contexts, which provides a systematic evaluation framework for generalization in MORL. The paper also presents a comprehensive analysis of state-of-the-art MORL methods (Envelope, GPI, CAPQL, PGMORL, MORL/D) using normalized hypervolume generalization ratio. Overall, the paper provides a foundation for understanding how MORL algorithms generalize across different environmental contexts and parameters, addressing a crucial gap in the current literature.", "questions": "1. How does the proposed framework handle cases where utility functions are unknown, non-linear, or non-monotonic? \n2. What specific scalarization functions were used in the MORL algorithms, and how were weight vectors initialized in methods like the Envelope algorithm? \n3. Why were other MORL metrics such as sparsity, cardinality, and expected utility not considered during evaluation? \n4. How do MORL methods generalization affected by varying numbers of objectives in underlying environments? \n5. Can the authors explain the discrepancy in CleanRL SAC performance in default hopper environment which achieved 3000 episodic return vs the one reported in Figure 5 (left most plot)? Could this because of the hyperparameter? If yes, then a well-tuned SAC may outperform the MORL methods.\n\n\nMinor comments:\n\nIn Definition 1, should the reward be represented as a vector in bold notation for consistency with Section 2? \n\nThe related work should be placed in the main paper. This can be done by moving the detailed environment descriptions to the appendix for better space utilization.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper contributes a novel testbed featuring various multi-objective RL environments with parameterized environment configurations, facilitating valuable research in MORL generalization. \n* The proposed framework for evaluating generalization in MORL using normalized hypervolume generalization ratio is sound and promising. \n* The paper provides reasonable coverage of relevant literature and effectively justifies its claims with thorough experimental evaluations.\n* The paper is generally well-written and easy to follow.", "weaknesses": "* The proposed testbed appears to be a straightforward extension of MORL environments to generalization with limited variations, lacking clear evidence of full interpolation/extrapolation properties across context changes. \n* The evaluation lacks analysis of important MORL metrics such as sparsity, expected utility, and cardinality, which limits the comprehensive understanding of algorithm performance. For instance, while hypervolume measures Pareto coverage, it does not address how spread the solutions are within the Pareto front, which is an important concept in utility-based approaches. Similarly, the expected utility metric is essential as it directly reflects an agent’s ability to maximize total utility, which is the ultimate goal in utility-based approaches. \n* The paper's focus on utility-based approaches with limited discussion of different utility or scalarization functions restricts the broader applicability. For instance, for linear scalarization functions, the resulting Pareto front is usually convex while for non-linear scalarization functions, the resulting Pareto front may have concave regions. It is unclear which class of scalarization method is used and how different classes affect generalization in MORL.\n* The empirical evaluation is limited to environments with a small number of objectives (maximum 4) which raise questions about scalability and generalizability. Testing in environments with higher objective counts, such as the Fruit Tree Navigation environment (5-7 objectives) by Yang et al. (2019) or the MO-highway environment from the mo-gymnasium API, which can be configured for more objectives, would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KAkwJDzEQV", "reviewer": "Reviewer_JmYT", "summary": "This paper, based on the MuJoCo platform, presents a comprehensive set of parameter variations for different MORL (Multi-Objective Reinforcement Learning) testing scenarios. Additionally, it introduces a novel metric designed to standardize the evaluation of the Pareto front and hypervolume measurements. By establishing a wide range of parameterized benchmarks, this work evaluates recent state-of-the-art MORL algorithms across the proposed environment variations, highlighting the performance differences under diverse parameter settings. This serves as a valuable resource for researchers aiming to study the generalization capabilities of MORL algorithms, providing a flexible and well-defined platform for experimentation.", "questions": "1. **Selection of Mujoco Environments:**\n   Why were only certain environments from the Mujoco,i.e., only some of the two-ojective problems,  are selected as baselines for demonstration, while other important environments were omitted (problems of 3 or more objectives)? Including a broader range of environments would strengthen the validity and generalizability of the motivation.\n\n2. **Modifications to the Traditional Mujoco Environment:**\n   Is there any basis or reference for the numerical changes made to the standard Mujoco environment?\n\n3.**Normalization range (NR):**\n   Under what circumstances are NRs conducted, and why are they remain reliable despite changes in training or neural network parameters?\n\n4.**Mujoco modifications:**\n Can you provide more details on any specific challenges encountered or innovations that has been made in implementing these modifications.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a substantial number of parameterized scenarios, significantly modifying standard MuJoCo test environments to induce pronounced changes in specific features. This extensive and detailed effort reflects a thorough approach in expanding the range of testing conditions.\n\n2. It challenges the widely-used hypervolume metric by proposing a new metric based on hyperarea coverage, aimed at standardizing the quality assessment of the Pareto front.\n\n3. Although there are some flaws in the writing, the overall readability is smooth, and the paper maintains a clear focus on its key contributions throughout the narrative.", "weaknesses": "1. **On the Use of Lebesgue Measure and Hypervolume:**\n   The Lebesgue measure in two-dimensional space corresponds to calculating area. If we assume the solution set's Pareto front is a closed, well-defined curve, the area enclosed by the origin and the Pareto front can indeed be computed using the Lebesgue measure. In this specific two-dimensional case, where all solution sets are located in the first quadrant (i.e., all objective values are positive) and the origin is used as the reference point, the calculated hypervolume effectively becomes equivalent to the Lebesgue measure.\n\n   It is important to note that the magnitude of the Lebesgue value is smaller and does not alter the comparative outcomes of optimization results. A smaller hypervolume (HV) will still lead to smaller two-point Lebesgue values. Although $HV_{norm} $ and $NHGR$ exhibit slightly higher penalty effects compared to the default in challenging scenarios, they do not provide additional benefits in terms of interpretability. Therefore, the proposed normalization does not seem to address the issue of result surges caused by outliers, as mentioned by the authors. Consequently, the introduction of $NHGR$ based on the hyper-area ratio appears to lack sufficient motivation.\n\n2. **Regarding Metrics in Figure 3:**\n   The two metrics presented in Figure 3 are actually different measurement dimensions of the same indicator. The mirrored images illustrate the overlap between these two metrics, which suggests they cannot be considered as introducing two new metrics.\n\n3. **Multi-Objective Reward and Weight Vector Scalarization:**\n   Multi-objective reward and weight vector scalarization is a well-established method. A substantial body of work has demonstrated the inefficiency of single scalar rewards in MORL optimization, such as in PGMORL and dynamic-weight MORL. Incorporating this concept as part of the experimental results does not significantly contribute to your experimental results, as it reflects existing knowledge in the field.\n\n4. **Modifications to the Traditional Mujoco Environment:**\n  In the original Mujoco environment, altering these values only requires adding instructions in the `step()` function, which is facilitated by the Mujoco simulation environment's API. The paper does not clearly indicate or demonstrate any additional code development or engineering improvements in this aspect.\n\n5. **Efficiency of Benchmarking:**\n   The benchmarking approach appears inefficient, as no new algorithm is proposed to compare against the state-of-the-art (SOTA).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7hth5Gi3R", "reviewer": "Reviewer_nVsL", "summary": "This paper introduces a framework and testbed for evaluating generalization in multi-objective reinforcement learning (MORL). **The authors claim that generalization in MORL is a crucial research direction, but existing MORL algorithms are not sufficient for MORL generalization.** To support the claim, the authors propose the concept of a multi-objective contextual MDP and introduce the normalized hypervolume generalization ratio (NHGR) as a metric for assessing generalization in MORL. To empirically test this, they modify existing MORL environments by adjusting key environment parameters and applying domain randomization techniques during training. The paper reports NHGR performance across different MORL algorithms during the evaluation phase and concludes by highlighting the need for further research on MORL generalization.", "questions": "Please see the weaknesses part above.\n\n**Post Rebuttal Comment**\n\n\nAfter careful consideration, I have revised my final rating. \n\nAs the authors acknowledged, the proposed metric has inherent limitations and these can be mitigated by also providing standard metrics. Each standard metric in MORL - hypervolume, expected utility metric, sparsity, and so on - has its strengths and weaknesses; therefore, the best approach can be to report multiple metrics to complement one another. This principle can be applied to the paper's setting as well. Introducing a new evaluation metric for MORL is a promising research direction, and it may be harsh to penalize the effort solely because the metric is not perfect.\n\nI hope the authors include a thorough discussion on the metric's limitations and potential mitigations in a future version, if possible.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Considering generalization in MORL is a sound direction in the RL community.\n* The necessity of MORL generalization is empirically demonstrated in Section 6.2, partly supporting the authors' claim.", "weaknesses": "1. **The overall contribution and novelty of this paper remain incomplete.** While the paper highlights a promising research direction, it lacks a concrete solution for addressing the problem of MORL generalization. To enhance the paper’s completeness, the authors should propose and integrate their own method to tackle MORL generalization effectively. \n\n2. **The authors' claim is not fully supported due to the unclear soundness of the proposed NHGR metric.**\n* 2-1. For clarity, does the combined specialist Pareto front consist of the *filtered* nondominated value vectors from the *union* of nondominated value vectors across the 8 MORL algorithms, each trained individually on a specific context? \n\n* 2-2. The computation of NHGR is highly demanding, especially as the number of MORL algorithms and contexts increases during evaluation. If performance can be directly computed for any given context, there would be little need for a generalized algorithm, as the \"combined specialist\" would suffice. This raises concerns about NHGR's practical applicability in scenarios with large computational requirements.\n\n* 2-3. A key weakness of NHGR is that its denominator relies on the performance of existing MORL algorithms, which may not perform well in certain contexts or in environments with many objectives. In such cases, where the \"combined specialist\" performs poorly, NHGR becomes an unreliable metric.\n\n* 2-4. The necessity for normalization in NHGR is questionable, especially since true upper and lower bounds for each objective are not typically known a priori. If these bounds are not tightly set, the reliability of NHGR results is compromised. Moreover, determining accurate bounds (e.g., $v^c_{min}$ and $v^c_{max}$) requires running multiple MORL algorithms, increasing computational complexity. Establishing these bounds is itself a challenging task, particularly during online training.\n\n\n* 2-5. A potentially better approach would be to use the expected standard hypervolume across evaluation contexts, supplemented by the expected standard sparsity to address the limitations of relying solely on hypervolume. \n\n\n* 2-6. Figure 1 does not align with the definition of normalization bounds $v^c_{min}$ and $v^c_{max}$. The reference point is determined by the x-coordinate of the upper-leftmost point and the y-coordinate of the lower-rightmost point.\n\n\n* 2-7. For clarity, is the number of the episodes evaluated for each context is set at 100/(number of evaluation contexts) (excluding Mario)?\n\n\n3. **The paper’s organization needs improvement.**\n- 3-1.  It would improve the flow of the paper if Section 6.2 were moved closer to the background section to better emphasize the motivation behind the work.\n- 3-2. Including the related work section in the main body of the paper would enhance readability, as it is critical for contextual understanding.\n- 3-3. A more detailed explanation of how each MORL algorithm operates, such as the weight adaptation variant of MORL/D SB, would be beneficial (this could be placed in the Appendix for conciseness).\n\n4. **Several technical clarifications are needed.**\n- 4-1. Is SAC trained with a standard scalar reward function (described in Appendix D) for each context vector independently, or does it follow domain randomization (i.e., sampling random environment parameters for each episode)? I assume the authors intended the latter.\n- 4-2. In Line 479, the phrase \"...scalarized them using $f_{SORL}$​...\" is unclear. Additionally, how many solution vectors are sampled for each algorithm?\n- 4-3. How is the optimality gap calculated?\n- 4-4. For the SAC implementation in Mario and LavaGrid, is it \"SAC-discrete\"? If so, how was SAC-discrete implemented, and how does its performance compare to DQN?\n- 4-5. In Section 6.2, what is meant by the \"upward speed\" in the scalar reward formulation (Line 461 and Figure 6)?\n\n5. Minor Suggestions:\n- I recommend dividing paragraphs to improve readability.\n- Line 302: \"We introduce an 8-dimensional parameter...\" should be corrected to \"7-dimensional.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)