Reward Dimension Reduction for Scalable Multi-Objective Reinforcement Learning
Authors: Giseung Park
Youngchul Sung
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Objective Reinforcement Learning
Reinforcement Learning
Dimension Reduction
Primary area: reinforcement learning
TL;DR: We propose an online reward dimension reduction method that enables MORL algorithms to scale to environments with many objectives.

Abstract: In this paper, we introduce a simple yet effective reward dimension reduction method to tackle the scalability challenges of multi-objective reinforcement learning algorithms. While most existing approaches focus on optimizing two to four objectives, their abilities to scale to environments with more objectives remain uncertain. Our method uses a dimension reduction approach to enhance learning efficiency and policy performance in multi-objective settings. While most traditional dimension reduction methods are designed for static datasets, our approach is tailored for online learning and preserves Pareto-optimality after transformation. We propose a new training and evaluation framework for reward dimension reduction in multi-objective reinforcement learning and demonstrate the superiority of our method in environments including one with sixteen objectives, significantly outperforming existing online dimension reduction methods.

Published as a conference paper at ICLR 2025
REWARD DIMENSION REDUCTION FOR SCALABLE MULTI- REINFORCEMENT LEARNING
Giseung Park, Youngchul Sung
School of Electrical Engineering
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon 34141, Republic of Korea
(gs.paxe ycsung) @ kaist. aa.iistt..kk. ac.kr
ABSTRACT
In this paper, we introduce a simple yet effective reward dimension reduction method to tackle the scalability challenges of multi-objective reinforcement learn- ing algorithms. While most existing approaches focus on optimizing two to four objectives, their abilities scale to environments with more objectives remain uncertain. Our method uses a dimension reduction approach to enhance learning efficiency and policy performance in multi-objective settings. While most tradi- tional dimension reduction methods are designed for static datasets, our approach is tailored for online learning and preserves Pareto-optimality after transformation. We propose new training and evaluation framework for reward dimension reduc- tion in multi-objective reinforcement learning and demonstrate the superiority our method in environments including one with sixteen objectives, significantly outperforming existing online dimension reduction methods.
INTRODUCTION
Reinforcement Learning (RL) is a powerful machine learning paradigm focused on training agents to make sequential decisions by interacting with their environment. Through trial and error, RL algorithms allow agents to iteratively improve their decision-making policies, with the ultimate goal of maximizing cumulative rewards. In recent years, the field of Multi-Objective Reinforcement Learning (MORL) has gained considerable attention due to its relevance in solving real-world con- trol problems involving multiple, often conflicting, objectives. These problems span across domains such advanced autonomous control (Weber al., (023), power system management, and logistics optimization (Hayes et al. 2022), where balancing trade-offs among competing objectives is crucial (Roijers et al. 2213)).
MORL extends the traditional RL framework by enabling agents to handle multiple objectives si- multancously. This requires methods capable of identifying and managing trade-offss among these objectives. MORL specifically focuses on learning a set of policies that approximate the Pareto frontier. representing solutions where no objective can be improved without compromising others. Most of the current approaches scalarize vector rewards into scalar objectives to generate diverse set of policies (Abels et al., 2019 Yang et al. 2019 Xu et al. 22200: Basaklar et al. 2023 Lu al., 2023). thereby avoiding the need for retraining during the test phase.
Although these methods have proven effective in standard MORL benchmarks (Felten et al. 2023 most benchmarks involve only two to four objectives, leaving open the question of whether existing MORL algorithms can scale effectively to environments with more objectives (Hayes et al. 2022 Indeed, various practical applications demand optimizing many objectives simultaneously (Li al. 2013). For example, Flleming et al. (2005) introduced an example of optimizing a complex jet engine control system that requires balancing eight physical objectives. In military contexts, commander should manage dozens of objectives that directly influence decision-making (Dagistanli & Üstün 2023), including the positions of allies and enemies, casualty rates, combat capabilities of allies and enemies, and time estimations for achieving strategic goals. When planning for multiple potential
Youngchul Sung the corresponding author.

---

Published as a conference paper at ICLR 2025
battle scenarios, exploring such nigh-dimensional objective space in its raw form is inefficient and very challenging due to the complexity of the original space (Wang & Sebag 2013).
An advantageous feature of many real-world MORL applications is that objectives often exhibit correlations, leading to inherent conflicts or trade-offs. For example, an autonomous vehicle must balance safety and speed, where optimizing one compromise the other. Similarly, traffic light control must manage multiple interrelated objectives to ensure smooth traffic flow (Hayes et al.) 2022). These correlations suggest that reducing the dimensionality of the reward space while pre- serving its essential features could be viable strategy to make current MORL algorithms scalable to many objective spaces.
Dimension reduction techniques (Roweis & Saul 2000; Tenenbaum et al. 2000 Zass & Shashua 2006 Lee al., 2007; Cardot & Degras 2018: 8 McInnes Healy 2018 Bank et al.) (020). widely used in other machine learning domains, capture the most significant features of high-dimensional data while filtering out irrelevant noise. However, typical approaches operate on static datasets, whereas RL necessitates continuous data collection during online training. This introduces unique challenge: applying dimension reduction in MORL while retaining the essential structure of the original reward space. To our knowledge, few studies have addressed this challenge within the MORL context.
In this paper, address these challenges by introducing a simple yet effective reward dimension reduction method that scales MORL algorithms to higher-dimensional reward spaces. We propose new training and evaluation framework tailored for the online reward dimension reduction setting. Our approach ensures that Pareto-optimality is preserved after transformation, allowing the agent to learn and execute policies that remain effective in the original reward space.
Our contributions are as follows. First, we propose a new training and evaluation framework for online reward dimension reduction in MORL. We also derive conditions and introduce learning techniques to ensure that online training and Pareto-optimality are maintained, providing a stable and efficient approach for scalable MORL. Lastly, our method demonstrates superior performance compared to existing online dimension reduction methods in MORL environments including one with sixteen objectives.
BACKGROUND
A oojti-objeectve Markov decision process (MOMDP) is defined by the tuple (S, A, P. po r, Here, S represents the set of states, A the set of actions, P the state transition probabilities, μi the initial state distribution, r the reward function, and 7 E [0, 1) the discount factor. Unlike the traditional single-objective MDP, the reward function r S A R in MOMDP is vector- valued, where K 2 is the number of objectives. This vector-valued nature of the reward function allows the agent receive multiple rewards for each state-action pair, each corresponding different objective.
In the context of MORL, the performance of policy æ evaluated by its expected cumulative reward, denoted as (Tx) = (J)( , == E, -i= r'r,]] E RRK To compare vector- valued rewards, we use the notion of Pareto-dominance (Roijers et al. 2013), denoted >p. For two vector returns, J(r) and [Ir',, we have:
00
J(T) > (1, K,,,,KK,JJ J,,)((T)) 2 J((T)) and (3j E (l,..., K}, (T))
This means that (TT') Pareto-dominate J(T) if it is at least good as (TT) in all objectives and strictly better at least one.
The goal of MORL is to identify a policy whose J(x) lies on the Pareto frontier (or boundary) F of all achievable return tuples J [[J (T).... Jx (T)) π E IT), where denotes the set of all possible policies. The formal definition of the Pareto frontier is follows (Roijers et al. 2013
'Strictly speaking. the Pareto frontier can also be defined as set non-dominated policies, {- E 3n' s.t. J(x') >p } (Hayes et al. 2022), rather than the set of non-donin vector returns as shown in equation 2. In this case, multiple policies may achieve the same vector return (Hayes et al, 2022). To avoid this redundancy, in this paper, we define the Pareto frontier and the convex coverage set presented in equation and equation respectively.
2

---

Published as a conference paper at ICLR 2025
Yang et al. 2019)
F=( Pn' s.t. J(T') >p J(z).
(2)
In other words, single policy achieving F improve one objective without sacrificing least one other objective. Finding policy achieving the Pareto frontier ensures optimal balance among the competing objectives with the best possible trade-offs.
Researchers are also interested in obtaining policies that cover the convex coverage set (CCS) of given MOMDP defined as follows YYang et al. 2
CCS (J(z) Bw A s.t. w J(x) 2 w JJ''), Vn's II with J(T)) E
(3)
where A is the ( and a E A represents a preference vector that specifies the relative importance
J2
A= CCS
of each objective (i.e., Σ W% = w, 2 0, Vk).
B
A,,,D,
Figure T dillustrates the relationship between Pareto fron- tier and CCS. In Figure D. we assume that the achievable points B, D, F form the Pareto frontier. Then, for the preference vector w in Figure | the inner product between w and the return vector at the point than the inner product between w and the return vector at the point D. The inner product between w and the return vector at the point C is smaller than that between w and any other point in the Pareto frontier. Hence, the point C is not inc included in the CCS. Note that the CCS represents the set of achievable returns that are optimal for some lin- combination of objectives, and it is a subset of the Pareto frontier F by definition. Since the weighted sum is widely used in real-world applications to express spe- cific preferences over multiple objectives (Hayes et al.) 2022). CCS proper refinement of the Pareto frontier.
w
D
smaller
E
OC' <OD'
J
Figure 1: frontier and the CCS for K = 2, where C' and D' represent the projec- tions of points C' and 1) onto the pref- erence vector w, respectively. Yellow dashed line represents the outer convex boundary of F.
Comparison of the Pareto
In the context of multi-policy MORL. (Roijers et al.
2013), the goal is find multiple policies that cover (an approximation of) either the Pareto fron- tier or the CCS so that during test phases, we perform well across various scenarios without having retrain from scratch. Specifically, we aim achieve Pareto-optimal points that maximize the hypervolume while minimizing the sparsity (Hayes et al. 2022)
.P of (A, = 0.5(|A |BD|2)
J2
J2
SP of (A,C,D) O. 0,5(|AC + |CD|2)
Green dots:
Pareto frontier
D
0
Jr
Reference point
Figure 2: Evaluation metrics in multi-policy MORL: hypervolume and sparsity. (Left) Hypervolume represented by the pink area the figure. (Right) The sparsity of the solution set (A, B, D} is lower than that of {AA, C, D} when points C and D are close, indicating that {A, B, D} offers more diverse set of solutions than {A, D).
As seen in the left figure of Figure 2. the hypervolume measures the volume in the objective space dominated by the set of current Pareto frontier points and bounded by a reference point. In the figure, the hypervolume corresponds to the area of the pink region. This metric pro- vides a scalar value quantifying how well the policies cover the objective space. Formally, let X (#1,".. n C T be set of N Pareto frontier points and xo E R be reference point,
3

---

Published as a conference paper at ICLR 2025
where 2; = (I.1....., ..... Tik), i i= =0...... NN. Then, the hypervolume metric HV zu) is defined by the K X-dimensional volume of the union of hybercubes U c [ZoR, where C [ Tael is the hypercube of which side at the k-th dimension is given by the line segment (Rak:
Another metric is sparsity, which assesses the distribution of policies within the objective space. As seen in the right figure of Figure 2 set of Pareto frontier points with low sparsity ensures that the solutions are well-distributed, offering diverse range of trade-offs among the objectives. If there is set of N Pareto frontier points X sparsity is defined as:
[T1,"" ,NN C IR with (Ea..... Tik) (i .,....N. N),
K N
SP(X) = N Σ Σ
(S - S. 1))
(4)
where S₁ = Sort( 1 S is S } in descending order in the k-tth objective, I k < K. Given a dimension 1: and its two endpoints, [1(1] and Sx[N], the Cauchy- Cauchy-Schwarz Schwarz inequality implies that all i N S2 - 1])2 is minimized when the differences Sx Sx + 1] are constant for Therefore, sparsity acts an indicator of how well-distributed a of return vectors is. Reducing sparsity while maintaining a high hypervolume helps avoid situations where only few objectives perform well. Therefore, considering both low sparsity and high hypervolume offers more comprehensive evaluation criterion than relying solely on hypervolume.
1
3 RELATED WORK
There are mainly two branches in MORL. The first branch is single-policy MORL, where the goal is to obtain an optimal policy x² = arg max, (x)) where h IR. > R is a fixed decceaceg utility function, mostly for non-linear one (Siddique et al., 2020 Park et al. 2024). The other branch the multi-policy MORL, where we aim to acquire nultiple policies that cover an approximation of the Pareto frontier or CCS. Beyond several classical methods such as iterative single-polig ap- proaches (Roijers et al., 2014), current approaches in the multi-policy MORL either train a set of multiple policies (Xu et al. (020) train single network to cover multiple policies (Abels et al. 2019 Yang et al. 2019: Basaklar et al. 2023 Lu et al. (023). For completeness, these approaches should be followed by a preference elicitation method for the test phase given that additional inter-
active approaches are allowed (Hayes et al. (022) (e.g.. /Zintrraf et al. (2018) inferred unknown user preference using queries of pairwise comparison on the Pareto frontier). Nonetheless, the area of elicitation has received far less attention than the learning methods themselves. Researchers usually focus solely on the learning algorithms during the training phase assuming that test preferences will be explicitly given.
Xu al. (2020) trains a set of multiple policies in parallel using the concept of evolutionary learn- ing, and the best policy in the policy set is used for evaluation during the test phase. Other works construct a single policy network parameterized by w E A K to cover the CCS, which easier than direct parameterization over the set of non-decreasing functions to cover the Pareto frontier. [Abels] jet al. (2019) and Yang et al. (2019) constructed single-policy networks to exploit the advantages of CCS. Specifically, Yang et al. (201)) defined the optimal multi-objective action-value function [E 7 E R where the optimal pol- for all w E A Q* as = (1::)) oc
icy n² is given by T (-1:w) arg sup, Ep,- D> - 7 7-re]. Based on a new definition of the multi-objective optimality operator, the authors proposed an algorithm for training a neural network Qo(s, a, w) to approximate (' a, w.. Basaklar et al. (2023) modified the multi-objective optimal- ity operator match each direction of the learned action- value function and preference vector, and Lu et al. (2023) tackled learning stability issue of multi-policy MORI. by providing theoretical analysis on linear scalarization.
While these methods have demonstrated promising performance in MORL benchmarks with two to four objectives, it remains an open question whether current algorithms can effectively scale to environments with more objectives (Hayes et al. 2022). The challenge lies in effectively covering all possible preferences during training. In most previous MORL algorithms, agents sample ran- dom preferences in each episode to collect diverse behaviors. However, performing this sampling naively in high-din high-dimensional spaces becomes computationally expensive because the coverage (or hypervolume) grows exponentially with the number of objectives (Wang & Sebag 2013).
4

---

Published as a conference paper at ICLR 2025
In this paper, we address the scalability issue by proposing reward dimension reduction technique with a suitable training and evaluation framework to narrow down the search space while preserving the most relevant information. Our approach is motivated by the observation that objectives
correlated in many real-world cases. While a variety of dimension reduction techniques exist in machine learning (Roweis Saul] 2000) [Tenenbaum al.} 2000) [ee et al.) 2007] McInnes & Healy, 2018), most are designed for static (batch-based) datasets. Only few methods are suitable for online settings, and some cases, no online version exists all (McInnes & Healy 2018).
Developing online variants of batch-based dimension reduction techniques is itself an active area of research. Currently, incremental principal component analysis (PCA) and online autoencoders are commonly used for online dimension reduction (Cardot & Degras 2018)) Bank et al. 2020).
However, we will demonstrate that they fail to preserve Pareto-optimality after transformation the context of multi-policy MORI..
To our knowledge, few studies have explored reward dimension reduction MORL. For instance, Giuliani et al. (2014) applied non-negative principal component analysis (NPCA) to a fixed set of return vectors collected from several pre-defined scenarios, identifying the principal components. However, they did not perform any further online interactions, but multi multi-policy MORL algorithms require online learning. In this paper, we propose simple yet stable method for online dimension reduction that preserves Pareto-optimality after transformation, described in the following section.
METHOD
4.1 TRAINING AND EVALUATION FRAMEWORK
M <SA,,,, R: K-dim
M' (.),, f(R): m-dim
J3
f: RK Rm
J2
(K m)
J2
m
J
Figure 3: Our proposed reward dimension reduction framework. We design mapping function f: BR n from the original reward space to the reduced reward space.
As seen in Figure reduction, where > m > 2. f transforms the original MOMDP M = (S,A, P. Ao. r,7) into another MOMDP M' = (S.A, P. no. fo (r),,)), reducing the dimensionality of the reward space while preserving essential features. We assume that standard multi-polics policy MORL approaches, such as Yang et al. (2019), perform adequately in the reduced-dimensionall reward space of M'. Then for any preference vector Wm E A the (m 1)-simplex, the optimal multi-objective action-value function and the optimal policy are defined as:
aim to design mapping function : RK TM for reward dimension
DC E ER
Qi (5, a, Wm) E E. (.118om 11g 8000
Σ ((rr)
where
(5)
T (-l-;.m))) = arg sup w TT E
Σ f(r))
arg sup Ep,
Σ ==
(6)
t=0
Th is also expressed as T (als,wm) otherwise. Note that the key aspect of multi-policy MORL is that we learn the action- value function Qt (s. a, Wm) not just for particular linear combination weight Wm but for all possible weights {wm Am] in the training phase so that we can choose the optimal policy for any arbitrary combi- nation weight depending the agent's preference in the test evaluation phase.
1 if
arg max, WT Q* (s, a', wm), - (al Wm
5

---

Published as a conference paper at ICLR 2025
Our goal is to design a dimension reduction function, f. such that the policies learned in the reduced- dimensional space achieve high performance in the original reward space while satisfying two key requirements: (i) online updates for dimension reduction and (ii) the preservation of Pareto- optimality the sense that Vulm E A
E CCCS = Reg (-I-wo))
Σ 1-=
7'r EF
where CCS represents the convex coverage set in the reduced reward space and F C RRK repre- sents the Pareto frontier in the original reward space.
To the best of our knowledge, research on reward dimension reduction in MORL is limited, and there is no well-established evaluation protocol for this task. In this section, we propose new training and evaluation framework tailored to the reward dimension reduction problem, along with the algorithm itself (outlined in Section 4.2)). This framework facilitates fair comparison of online dimension reduction techniques within the context of the original MOMDP.
During the training phase, we aim learn the optimal multi-objective action-value function @i (s, a, Wm) while we simultaneously update the dimension reduction function f online. For the action-value function update, we sample data (s, a, r, 8") from a replay buffer but utilize the reduced-dimensiioall rewards f(rr instead of the original rewards Our goal is to ensure that E,r, (-1wam))))
> (rrr) E CCS, after the training phase ends.
In the evaluation phase, the learned policy t (+),, (m) is tested on a set of N₂ preferences Pm,.N. C A" with N. = |Smm,N. .N. where the points are evenly distributed on the (m 1)-simplex. For each Win E Smm.N.. we compute the expected cumulative reward (-),a.) [ 7'rx] E R² in the original reward space, as the MOMDP provides the high-dimensional vector reward each timestep. Our goal is for the Pareto frontier points of {E.r,, (clam)) 1 r E IR (4m Nm,N.. to maximize hypervolume while minimizing sparsity.
4.2 DESIGN OF DIMENSION REDUCTION FUNCTION
To preserve the Pareto-optimality shown in equation impose two minimal conditions on the dimension reduction function f:
Theorem 1. If is affine and each element of the matrix is positive, then equation is satisfied.
First, if f affine, then f(r)= = Ar R", where A E Rmnk By linearity, Vum Am Egg, (():amm) 7 ((rr + +b] = E (.1,,)) (-1-wmm)
Proof.
E ( + --be CCC...
Next, if each element of A is positive, we claim that Ex (.1.wm)) Eeco xre E F so that the Pareto-optimality in equation is preserved.
This is proved by contradiction. Given win E A "., suppose 3r' E II s.t. E, IΣ r >p
(=:.--)-amm) DE rrr] in the original reward space. By the definition of P in equation ! each dimension of Ex oc 7're] - Em (-l-.mm) Ee r're] E R non-negative and at least one dimension positive.
For each 1 S j S m, let a. E RIxk be the j-th row vector of A and by be the j-th ele- Xrre] - -rr(()hw))) ('rlo > O. In other words, have Iim ('re] in the reduced-dimensiona. space. By linearity, I 7bb] = to both sides gives
ment of b. Then a, (E,,, (ET EE
AE 1> 7'rc] >p ALE (+1m0mm))
adding E,, IEe bb = Ex (1)))))
oc (Ar; + +b) Σ
E
>p Bx (.hmam))
Σn'(Ar, b)
(8)
t=0
Since CCS,, is by definition subset of the Pareto frontier in the reduced-dimenssooall space, CCSp consists of vector returns in the Pareto frontier. Therefore, equation gives contradiction since Ex, (+haam)))) 7 b)] E CCSm is Pareto dominated by E,, E2 (Are b)].

---

The next question "How should we design the affine transform A to preserve the information of the original vector reward function r?" To address this question, propose constructing reconstruction neural network Jø. where the input is the aeduced-dimension reward f(r). The Published as a conference paper at ICLR 2025
In short, the condition of f(r) Ar + b with A RT** guarantees the preservation of Pareto-
optimality in equation 7
From equation 6
8
Tyn (11,, Wm) sup E
Σ 7 (w
+ w b=
sup E
Σ
MaTTAA
(9)
In discounted reward settings, the bias term b does not affect the determination of x we set
b = 0 for simplicity.
In addition, we impose another condition that A is row-stochastic: Then
Σx= ASt = 1, S m.
K Σ (AT Wm)k =
K m
m
m
Σ
(wm)s
AAk k=1
(Wm)) = 1.
(10)
k=1j=1
i=1
j=1
In other words, Vum A we have the corresponding preference vector A Wm E A in the original reward space. Let A [a1,-.. am] E RK" where aj A S S m. Then
Al A Wm E A and the set {A "im [W Wmm E Am) C A is the convex combination of aj E AK 1,1 < j m. Conceptually, the role of the matrix to narrow down the preference search space from K to proper subset of A
network Is is trained to minimize the reconstruction loss:
min A>0, row siochusiciiic..
E. |||[(ssss) a) - go(((r(s,
(11)
where A > 0 denotes that each element of A is positive. This approach, combining compression with reconstruction, is widely employed to capture the essential features of input data while dis- carding irrelevant information (Baldi, 2012) Kingma & Welling 2014: Berahmand et al. (202)). However, solving the optimization problem equation [1 more challenging than conventional autoencoder-style learning, where the encoder is general neural network trained without con- straints. In contrast, our method must ensure that the matrix A satisfies both the positivity constraint A > 0 and row-stochasticity during online training.
To overcome this challenge, we introduce a novel approach by parameterizing using softmax parameterization, ensuring both positivity and row-stochasticity constraints satisfied through- out training. Our implementation in PyTorch (Paszke et al.) 2019) effectively applies this param- eterization, and we solve the optimization in equation 11 using stochastic gradient descent in online setting. The reconstruction loss is minimized alongside the training of the parameterized multi-objective action-value function Qo(s, a, Wm), which approximates Q", (s, Wm) defined in cquation 55] Additionally, we found that applying dropout (Srivastava et al. (014 to Jø during training equation II further enhances overall performance.
EXPERIMENTS
5.1 ENVIRONMENT AND BASELINES
While various practical applications require addressing many objectives (Fleming et al. 2005 Li
al., 2015: Hayes et al., 2022). there currently exist few MORL simulation environments with reward dimensions exceeding four (Hayes et al. 2022 Felten et al. (023). To address this issue, we considered the following two MORL. environments: LunarLanders (Hung et al.) (023) and our
modified implementation of an existing traffic light control environment (Alegre 2019) to create
sixteen-dimension. reward setting.
LunarLander-5D is challenging MORL environment with five-dimensional reward function where the agent aims to land lunar module on the moon's 's surface successfully. Each reward dimension represents: (i) a sparse binary indicator for successful landing (+ for success, for crash), (ii) a combined measure of the module's 's position, velocity, and orientation, (iii the fuel cost of the main engine, (iv) the fuel cost of the side engines, and (v) time penalty. This environment
7

---

Published as a conference paper at ICLR 2025
presents significant challenges because failing balance these objectives effectively can easily lead to unsuccessful landings (Felten et al. 2023 Hung et al. 2023).
Traffic light control is practical example of prob- lem that can be formulated as MORL, where efficiently balancing many correlated objectives is crucial (Hayes et al. 2022). As shown in Figure the traffic intersec- tion features four road directions (North, South, East, and West), each with four inbound and four outbound lanes, At each time step. the agent receives a state represent- ing traffic flow information. The traffic light controller selects a phase as its action, and the reward a sixteen- dimensional vector where each dimension corresponds to a measure proportional the negative total waiting time of cars on the respective inbound lanes.
1535
2205
B1G.
88
133
155
7:5
1001
In our experiments, we used the MORL algorithm from Yang et al. (2019) the base algorithm for the original reward space (Base). We incorporated several online di- mension reduction methods to the vector rewards in the base algorithm, including online autoencoder (AE) (Bank et al. 2020 incremental PCA (Cardot & Degras 2018), our online implementation of conventional batch-based NPCA (Zass & Shashua 2006), and our proposed approach. We followed the training and evaluation framework outlined in Section setting m 16 when evaluating the base algorithm alone.
Figure 4: A snapshot of our considered environment: traffic light control.
For incremental PCA, we update the sample mean vector la E RR and the sample covariance matrix Ce CERK* k at each timestep using vector reward rs. We periodically perform eigendecomposition on C' and select the top eigenvectors V/...... Um E I corresponding to the largest eigenvalues, maximizing E1=1 u/ C'u. We construct the matrix U [u....., um IR Kg.K x so that U (r
4)) E 1.th represents the reduced vector for following the PCA assumption that the transformed vectors are centered (Cardot & Degras 2018).
For the online implementation of original NPCA (Zass & Shashua 2006), we directly parameterize ju,..... [um]] E IR xm with a non-negativity constraint for efficient training (which we denote as U > 0). This direct parameterization removes extra hyperparameter tuning for the constraint U 0 and gives a fair implementation compared with our method that also uses direct parameteri- zation. We optimize the objective maxu20 Li=1 in Cuu 3 U ---A 12 using gradient descent, with a hyperparameter 3 >> O. Balancing the PCA loss with the orthonormality constraint creates trade-off between capturing principal component information and maintaining orthonormal basis vectors. Both PCA and NPCA do not use reconstructor 96-
In the traffic environment, set m for all online dimension reduction methods. We set 3 for Lunarl Lndeer55. To enhance the statistical reliability of our experimental results, we applied 12.5% trimmed mean by excluding the seeds with maximum and minimum hypervolume values over eight random seeds and reporting the averages of the metrics over the remaining six random seeds. This offers better robustness against outliers than the standard average (Maronna al. 2019) (Additional implementation details can be found in Appendix E)
5.2 RESULTS
Base
PCA
AE
NPCA
Ours
HV(x 1V',10''7 1)
3.1 4.7
3.2 + 4.2
0
1.7 3.1
25.6 + 6.9
SP(x 102.4)
31.2 + 25.3
188.6 + 180.7 31.3 + 30.6 53.0 + 47.7
1.1 1.2
Table 1: Performance comparison in LunarLander-5D environment, with the reference point for hypervolume evaluation set to (0, -100, - -100, - 100, -000) E R HV: hypervolume, SP: sparsity.
Table I demonstrates that in the LunarLander-5D environment, algorithm outperforms the base- line methods in both hypervolume and sparsity metrics. Specifically, our approach improves the base algorithm's 's hypervolume by factor of 8.3. It also reduces sparsity the ratio of 28.3 resulting in
8

---

Published as a conference paper at ICLR 2025
more diverse and better pertorrnr solutions. Note that the hypervolume values reflect successful landing episodes, so our dimension reduction is more efficient for achieving successful landing and balancing remaining objectives simultaneously than the baselines.
Base
PCA
AE
NPCA
Ours 166.9 + 48.1
HV(x10 1)
4.4 6.8
0
0.007 + 0.018
19.4 + 15.3
SP(x10".
1842 + 1290 3837 + 2164
7834 E 3323
34.2 + 52.3
2.3 = 1.0
Table 2: Performance comparison in our traffic experiment where we set reference point for hyper- HV: hypervolume, SP: sparsity.
volume evaluation to (-10², -10 .........--- 100) R
Next, Table demonstratts that our algorithm consistently outperforms the baseline methods in the traffic environment with sixteen-iiennsonaal reward. Our algorithm improves the base algorithm's hypervolume by factor of 37.9 while significantly reducing sparsity, indicating that reward dimen- sion reduction effectively scales the base algorithm dimension reduction is an affine transformation, but because the matrix does not meet the positivity condition in Theorem [ it fails to guarantee Pareto-optimality as outlined in equation 7 Simi- larly, the AE method uses a nonlinear transformation that fails to satisfy the linearity requirement in producing worse hypervolume and sparsity than the base case.
dimensonime spaces. The PCA-based
Theorem
NPCA
NPCA-ortho
Ours
10 1)
19.4 + 15.3
0.3 + 0.5
166.9
48.1
SP(x x SP(110", ())
34.2 + 52.3
203.7 24.1
2.3 + 1.0
Rank
4
Table 3: Performance comparison in the traffic experiment with NPCA and NPCA-ortho where "Rank" refers to the rank of the matrix in each method.
In Table 2 our method outperforms NPCA by significantly increasing hypervolume and reducing sparsity, with improvements of 8.6x in hypervolume. Although NPCA employs an affine transfor- mation with a nonnegative matrix, its online variant encounters instability due to the conflicting objectives of optimizing the principal component loss while maintaining the orthonormality con- straint. As shown in Table 3 the best-performing NPCA models, in terms of hypervolume and sparsity, had matrices of rank 1. The learning process prioritized maximizing the PCA loss, the expense of enforcing the orthonormality constraint, producing completely overlapping basis vectors.
To address this, we tuned hyperparameters to emphasize the orthonormality constraint, denoting this variant as NPCA-ortho. However, Table shows that this adjustment led to performance de- cline compared to NPCA. The reason is that assigning more weight to the orthonormality constraint weakened the PCA update, significantly reducing its ability to capture relevant information from the original reward space. Additionally, we found that balancing the two losses was highly sensitive and difficult fine-tune. In contrast, our method avoids these issues, offering more stable and effective solution for reward dimension reduction without the trade-off's inherent in NPCA's design.
To better illustrate results in high-dimensional space, visualized the Pareto frontier points ob- tained from the traffic environment using t-SNE (Van der Maaten & Hinton 2008), as detailed in Appendix B. We also present hypervolume values for different reference points and additional metric called Expected Utility Metric (EUM) (Zintgraf et al. 2015 Hayes et al. 2022 in Appendix B.2 and B.3 respectively.
5.3
ABLATION STUDY
In this section, investigate the impact of key components in our proposed dimension reduction approach. First, we analyze the effect of the constraints imposed on the dimension reduction function f. Specifically,
examine three aspects: bias, row-stochastieity. and positivity to assess their influence on preserving Pareto-optimality. Next, we evaluate the effect of applying dropout to the reconstructor 9g in equation II We conducted our ablation study in the traffic environment.
Table presents the results of our first ablation study. Adding a bias term to f results in a slight de- crease in hypervolume and increase in sparsity compared to method. However, the impact is

---

Published as a conference paper at ICLR 2025
Ours
+bias
-rowst
-positivity
-rowst, -positivity
HV(x10'1, .1) 166.9
132.9
46.8
SP(x10' .)
2.3
2.7
38.8
4066.6
5310.7
Table 4: Ablation study examining the effect of different conditions on the dimension reduction function f. "+bias" adds bias term in f: "-rowst" removes the row-stochasticity constraint while retaining the positivity condition; "-positivity" removes the positivity condition.
less severe than the other modifications. While, in theory, the bias term b does not affect the determi- nation of the optimal policy under discounted reward settings (as shown in equation 99, in practice, introducing a bias term offers minimal benefit, so purely linear transformation is sufficient.
We next observe performance drop when the row-stochasticity condition is removed. Notably, sparsity increased sharply by a factor of 16.9, highlighting the detrimental impact of this removal. Note that the direction of each preference vector um, not the magniude, matters for the deter- mination of optimal policy T equation 6 By confining the search space to the simplex, learning process focus finding the correct direction to extract essential reward information, rather than expending unnecessary effort on adjusting magnitudes. Consequently, enforcing the row-stochasticity constraint enhances learning efficiency, leading more diverse solutions.
If we remove the positivity condition while maintaining the row-stochasticity constraint, the algo- rithm produces zero hypervolume. This is due the lack of the positivity condition required by Theorem I Finally, further removing the row-stochasticity gives [(r) = Ar with a generic linear matrix A that also fails to preserve Pareto-optimality in equation 7
In summary, the positivity condition is essential for maintaining Pareto-optimality, while the row- stochasticity constraint improves the efficiency of online learning under the positivity condition.
Ours
Without dropout
HV(x10 ,1)
166.9
127.1
SP(x10$.3)
2.3
9.5
Table 5: Ablation study on the effect of applying dropout to the reconstructor Je in equation
Next, we evaluated the effect of applying dropout to ge in equation IT As shown in Table omitting dropout resulted in a slight decrease in hypervolume, while sparsity increased by factor of 4.1. Dropout can be interpreted after normalizing the input vectors (Wager et al. (013). L2 regularization helps prevent certain weights neural network from becoming excessively large.
form of regularization, approximately equivalent L2 regularization
In our context, each output feature of the reconstructor Io corresponds to objective in MORL, and there are implicit correlations among these K objectives. By applying dropout, we mitigate the risk of the model prematurely focusing on subset of the K objectives by preventing the weights of 9o from growing excessively large. This explains why sparsity increases when dropout is not used: the model may lose the opportunity generate diverse solutions along the Pareto frontier, as learning tends to focus prematurely on a subset the K objectives without weight regularization. (We also provide ablation study the effect of the reduced dimensionality in Appendix Cl)
CONCLUSION AND FUTURE WORK
In this paper, we proposed simple yet effective reward dimension reduction technique address the scalability challenges of multi-policy MORL algorithms. Our dimension reduction method effi- ciently captures the key features of the reward space, enhancing both learning efficiency and policy performance while preserving Pareto-optimality during online learning. We also introduced new training and evaluation framework tailored to reward dimension reduction, demonstrating superior performance compared to existing methods. Our future work includes developing more benchmarks and informative metrics for high-dimensional MORL scenarios to gain deeper insight and further advance the field.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We provide detailed descriptions of each algorithm in Section 5.1 and Appendix D including the techniques, fine-tuned hyperparameters, and infrastructures used in our experiments. The evaluation Furthermore, Theorem T
protocol for performance comparison is described in Section contained, so it is easy to verify the theoretical results. The link to our code is htshttpss///giiii Reeeeeeeductttt deduuction-MORI
is self-
ACKNOWLEDGMENTS
This work was supported in part by Institute of Information & Communications Technology Plan- ning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2024-00457882,
AI Research Hub Project) and in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercializa- tion of Autonomous Drones).
REFERENCES
Axel Abels, Diederik M. Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 11-20. PMLR, 2019. URL http::/prooeeeanngs.noit http::/eeeeiinn..mm...eee.e.e. v97 v97 1 9a. nt.
Lucas N. Alegre. SUMO-RL. http:://ggthabb.cc///aaaaeeeeeee
r1 2019.
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Isabelle Guyon, Gideon Dror, Vincent Lemaire, Graham W. Taylor, and Daniel L. Silver (eds.), Unsupervised and Transfer Learning Workshop held at ICML 2011, Bellevue, Washington, USA, July 2, 2011, volume of JMLR Proceedings, pp. 37-50. JMLR.org, 2012. UR//ttp::/prroeedngg mlr. www.ppress/veee wtlpes//////baaai1/// html
Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders. CoRR, abs/2003//000 2020. URL
https://arxiv.orxg h 05991
Toygun Basaklar, Suat Gumussoy, and Ümit Y. Ogras. PD-MORI:: preference-driven multi- objective reinforcement learning algorithm. In The Eleventh International Conference on Learn- ing Representatiinss, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://cpenrewiew net /ttf//dddddddeeeeee 259s1yyas2
Kamal Berahmand, Fatemeh Daneshfar, Elaheh Sadat Salehi, Yuefeng Li, and Yue Xu. Au- toencoders and their applications in machine learning: survey. Artif. Intell. Rev, 57 (2):28, 2024. doi: 00.1007/11042--023- 106622--6. URL Ihttps://don http::////o.oo/11100000.0000 1007/ 610462-023-10662-
Hervé Cardot and David Degras. Online principal component analysis in high dimension: Which algorithm to choose? International Statistical Review, 86(1):29-50, 2018.
Hakan Ayhan Dagistanli and Özden Üstün. An integrated multi-criteria decision making and multi- choice conic goal programming approach for customer evaluation and manager assignment. De- cision Analytics Journal, 8:100270, 2023.
Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, Ghazali Talbi, Grégoire Danoy, and Bruno Castro da Silva. A toolkit for reliable benchmarking and research in multi- multi-objective reinforcement learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
11

---

Published as a conference paper at ICLR 2025
Peter J. Fleming, Robin C. Purshouse, and Robert J. Lygoe. Many-objective optimization: An engineering design perspective. In Carlos A. Coello Coello, Arturo Heránndez Aguirre, and Eckart Zitzler (eds.), Evolutionary Multi-Criteriin Criterion Optimization, Third International Conference, EMO 2005, Guanajuato, Mexico, March 9-11, 2005, Proceedings, volume 3410 Lecture Notes in Computer Science, pp. 14 32. Springer, 2005. doi: 10.10077978-3-540--22 https:://oi.org// 100701700777888 540- 31880-488 2
22 URL
Matteo Giuliani, Stefano Galelli, and Rodolfo Soncini-Sessa. dimensionality reduction approach for many-objective markov decision processes: Application to water reservoir operation prob- lem. Environ. Model. Softw. 57:101-114, 2014. doi: 10. 1..101//..E.SSO... ENVS000.22000.000200111111 URL
https://doi. org.orggg/oo. 1016 www.ooe.evoo..ooo 2014. 02. 011
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. arXiv preprint arXiv:2006. 13900, 2020.
Conor F. Hayes, Roxana Radulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel de Oliveira Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. A practical guide multi- objective reinforcement learning and planning. Auton. Agents Multi Agent Syst. 36(1): 26, 2022. doi: 10. s10458 022 09552-y
URL https: httt://dci..ro /ttp:/iiiooogo.o..
Wei Hung, Bo-Kai Huang, Ping-Chun Hsieh, and Xi Liu. Q-pensieve: Boosting sample efficiency of multi-objective RL through memory sharing of q-snapshots. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. httl:ttts:///ppnom
net
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http:
//arxiv. wwwxivvorg//bae abs 1412.11444 6980
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Bansf. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
2bss13322.61 6114
John A Lee, Michel Verleysen, et. al. Nonlinear dimensionality reduction, volume 1. Springer, 2007.
Bingdong Li, Jinlong Li, Ke Tang, and Xin Yao. Many-objective exonebbi algorithms: survey. ACM Comput. Surv., 48:13313:1-13335, 2015. doi: 10.1145/2792984, URL https://doi.
org 100 1wl.1112/22999....
Haoye Lu, Daniel Herman, and Yaoliang Yu. Multi-objective reinforcement learning: Convex- ity, stationarity and pareto optimality. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL .nt wtwp:eeedddddddiiiiiie.eeee
Ricardo A Maronna, R Douglas Martin, Victor J Yohai, and Matías Salibian-Barrera. Robust statis- fics: theory and methods (with R). John Wiley & Sons, 2019.
Leland McInnes and John Healy. UMAP: uniform manifold approximation and projection for di- mension reduction. CoRR, abs/1 hss/1802.03426 2018. URL
03426
Giseung Park, Woohyeon Byeon, Seongmin Kim, Elad Havakuk, Amir Leshem, and Youngchul Sung. The max-min formulation of multi-objective reinforcement learning: From theory to a model-free algorithm. In Forty-first International Conference on Machine Learning, (CM22024 Vienna, Austria, July 21-27. 2024. OpenReview.net, 2024. URL https: / //tp///evp//....////
net//orum?iddccee
12

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp- 8024 8035, 2019. URL /ttppp////rooooedagg.....ee. bobca288ffee
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foer- ster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th In- lernational Conference on Machine Learning, ICML 2018, Stockholmsmüssann Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4292 4301. PMLR, 2018. URL http:tttppccpeeiiinigeee a html
mir press v80 rashidlaahhaaaanaa
Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision making. J. Artif. Intell. Res., 183, 2013. doi: 10.1613/1AIR. 3987. URL/sttps://abii org/ 10. 1613/ jair...i............r... 3987
makini.
Diederik M. Roijers, Shimon Whiteson, and Frans A. Oliehoek. Linear support for multi-objective coordination graphs. In Ana L. C. Bazzan, Michael N. Huhns, Alessio Lomuscio, and Paul Scerri (eds.), International conference on Autonomous Agents and Multi-Agent Systems, AAMAS '14 Paris, France, May 5-9, 2014, pp. 1297-1304. IFAAMAS/ACM, 2014. URI. http://dl. acm. wwwporggggttiin.ooooo./. cf wtt.:en.cm.mmmm211555m511
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. science, 290(5500):2323-2555 2000.
Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In Proceedings of the 37th Inter- national Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 9905-8915. PMIR, 2020. URL html
http::/proceedding mllr. mt
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1929-1958, 2014.
doi: 10.555522224435.670033 URL BRLhhtts:://dl acm. ocm.org/do//1..
5555/2627432 26703126677333
Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in neural information processing systems, 35:37719-37334, 2022.
Joshua Tenenbaum, Vin de Silva, and John Langford. global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2119-2322 2000.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008
Stefan Wager, Sida Wang, and Percy Liang. Dropout training as adaptive regularization. In Christo- pher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q Weinberger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 351-355 2013. URL
ngs neur seerrips.. ips.ccs../cc...............c. cc/ .html
Weijia Wang and Michele Sebag. Hypervolume indicator and dominance reward based multi- objective monte-carlo tree search. Machine learning, 92:403-429 429, 2013.
13

---

Published as a conference paper at ICLR 2025
Mare Weber, Phillip Swazinna, Daniel Hein, Steffen Udluft, and Volkmar Sterzing. Learning control policies for variable objectives from offline data. In 2023 IEEE Symposium Series on Computa- tional Intelligence (SSCI), pp. 1674-1681. IEEE, 2023.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojeiech Matusik. Prediction-guiied multi-objective reinforcement learning for continuous robot control. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10607- 10616. PMIR, 2020. URL
ml press/viil 9/ xu20h. html
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc. Emily B. Fox, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- fion Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 14610-14621, 2019. URL http:://procedinng newrips. cc/paper/2019/ Abst .hmll
hash/4a wttee44fooco11111ee222aa0...f...
Ron 7ass and Amnon Shashua. Nonnegative sparse pea. Advances in neural information processing systems, 19, 2006.
Luisa M Zintgraf, Timon V Kanters, Diederik M Roijers, Frans Olichoek, and Philipp Beau. Quality assessment of morl algorithms: A basei approach. In Benelearn 2015: proceedings the 24th annual machine learning conference of Belgium and the Netherlands, 2015.
Luisa M. Zintgraf, Diederik M. Roijers, Sjoerd Linders, Catholijn M. Jonker, and Ann Nowé. Ordered preference elicitation strategies for supporting multi-objective decision making. In Elisabeth André, Sven Koenig, Mchdi Dastani. and Gita Sukthankar (eds.). Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, pp. 1477-1485. International Foundation for Au- tonomous Agents and Multiagent Systems Richland, SC, USA ACM, 2018.
URL http:
7/dl. acm.
14

---

Published as a conference paper at ICLR 2025
A LIMITATION AND FUTURE WORK
While our approach offers a promising solution for scaling MORL algorithms, several avenues re- main for future research. First, as discussed in Section the lack of benchmarks for environments with more than ten objectives limits the comprehensive validation of our method. Developing robust benchmarks for high-dimensional MORL scenarios erucial direction for our future research.
Second, recent research in MORL has focused on developing additional metrics to better evaluate performance, recognizing that the data behavior in MORL more complex than in standard RL. High-dimensional scenarios are difficult to visualize, and data behavior often deviates from intuitive expectations (Lee et al. 2007). Therefore, designing informative metrics beyond standard measures like hypervolume and sparsity is essential for gaining deeper insights and advancing the field.
Third, although we provided mathematical conditions for preserving Pareto-optimality Theorem 1 these are only sufficient conditions. We investigated the effect of each condition in Section 5.3 However, our method's theoretical guarantees will be more solid if we establish the necessary con- ditions that pinpoint when Pareto-optimality fails. We provide detailed discussion in Appendix
Fourth, while our approach enables effective training for scalable MORL, for practical use, test preference vectors in their original high-dimen nigh-dimensional form must reduced to the lower-dimensional space learned by model. Developing methods for preference vector reduction, and potentially integrating preference elicitation mentioned in Section 3 will be essential for making our approach more practical and complete.
Lastly, our method can be extended in various directions. For example, constrained MORL rep- resents a promising direction, especially for safety-critical tasks where additional constraints must be considered. This extension could open up new applications where optimality and safety are paramount. Also, combining reward dimension reduction with reward canonicalization (Gleave et al. 2020) and extending reward linear shifting (Sun et al., 2022) to high-dimensional offline MORL represent promising avenues for extending our work, both theoretically and experimentally.
15

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS
B.I VISUALIZATION OF THE PARETO FRONTIERS
L-SNE Visualization
MPCA
Base
AE
20
E-NN Dirranion
-20
-40
-20
20
60
+:SNE Dimenson
Figure 5: t-SNE visualization of the acquired Pareto frontier points
In Figure 5 we visualized the Pareto frontier obtained in the traffic environment for each algorithm using t-SNE (Van der Maaten & Hinton 2008). We emphasize that our primary objective is cover broad region of the Pareto frontier, not merely cover wide region of high- dimensional reward space itself. Although AE solutions may appear widely distributed, this does not necessarily imply extensive coverage of the Pareto fronticr because the Pareto frontier a subset of the original space. Given that AE yields low hypervolume, it is less likely to represent wide range of the Pareto frontier.
According to the overlap analysis, smaller overlaps with the Base method suggest different local structure, for example, either in a way of our method (with high hypervolume) or a way of PCA (with very low hypervolume). This qualitative difference suggests that method and PCA distinct in their approach to exploring the solution space. Also, NPCA overlaps with more points from the Base and AE methods than ours, demonstrating the insufficiency of NPCA in covering larger region of the Pareto frontier than the Base method.
B.2 HYPERVOLUMES WITH DIFFERENT REFERENCE POINTS
In our traffic environment, we selected the reference point (-10+, -10 ................................ -10.. 16 based on observations that, after the initial exploration phase, most points in the current Pareto fronts fell within this defined region (except for PCA). However, some points may deviate from this region. To account for these outliers, the reference points can be adjusted accordingly. We evaluated the hypervolume using different reference points in both the traffic environment and LunarLander-SDD As shown in Tables 6 and 7 algorithm consistently outperforms the baseline methods.
Base
PCA
AE
NPCA
Ours
HV1 x 107, 1) HV2 x 10 1)
3.1 + 4.7
3.2 4.2
1.7 3.1
25.6 + 6.9 37.1 + 6.7
7.6 + 9.0 8.8
4.9 6.8
Table 6:
Performance comparison in the LunarLander-5D environment. points for hypervolume evaluation are 10,-100, -000, -000, 100) (0, 150, 150,- -150, - 1550 RR5 for HV2. HV: hypervolume.
The reference
g for HV1 and
16

---

Published as a conference paper at ICLR 2025
Base
PCA
AE
NPCA
Ours
HV1(x 10 f)
4.4 6.8
0
0.007 + 0.018
19.4 15.3
166.9 + 48.1
HV2(x 10 1)
5.0 6.7
0
0.3 + 0.7
11.5 + 4.8
29.3 + 3.4
HV3(X10'3
1.6 + 0.8
0.2 + 0.3
1.9
2.9 + 0.2
Table 7: Performance comparison the traffic experiment. The reference points for hypervolume evaluation are set to (-10*, -104, -100% E IR for HVI, (-2 x 104, -2 10 10 , x 10*) E 16 for HV2, and (-4 x 10', -4 10 -4x E R16 for HV3. HV: hypervolume.
B.3 DISCUSSION ON EUM METRIC
To ensure that the current Pareto frontier JF captures diverse set of return vectors in the original reward space, we introduce an additional metric: the Expected Utility Metric (EUM) (Zintgraf et al.
2015 Hayes al. 2022). The EUM defined as follows:
FU, QK,N.) == EEER [max Sa(w,, r)]-
(12)
rEF
Here, 8g,8, C A represents a set of N. preferences in the original reward space and fs is scalarization function. We consider EUM because it effectively evaluates agent's performance across a broad range of preferences (Hayes et al. 2022), aiming for a higher EUM to prevent the Pareto frontier from covering only narrow region within the original reward space. To compute EUM, we define QK,N, as the set of Ne normalized points evenly distributed on the (K - 1)- simplex AR We also set the scalarization function by (w, | [r]]l, representing the projected length of the vector r onto w IK.N. We set Ne to 126 and 15,504 for LunarLander and the traffic environment, respectively.
LuderLanderr-5D
Base
PCA 20.2 21.5
AE 76.2 1 48.6
NPCA
Ours
EUM(1)
25.8 1 24.3
28.4 13.9
11.5 1 5.4
Table 8:
Performance comparison in LunarLander-SD experiment. EUM: expected utility metric.
Traffic
Base
PCA
AE
NPCA 44.4 1.2
Ours
BUM(x102.1)
-3.4 2.9
33.1 + 15.2 - -16.1 + 8.8
220 + 1.0
Table 9:
Performance comparison in our traffic experiment. EUM: expected utility metric.
method outperforms the baseline approaches in terms of the EUM. important to note that during training. the Base method uses equidistant points in the original reward space, which naturally leads to high EUM values, especially when the reward space has a high dimensionality. However, our algorithm demonstrates superior performance compared other reward dimension reduction methods, particularly in higher-dimensional environments like the traffic scenario.
As demonstrated in Tables and Tad
C EFFECT OF THE REDUCED DIMENSIONALITY
m
4
6
8
10
HV(x 60 1)
1.4
1.7
1.7
1.9
1.1
SP(x10',
1.5
2.3
30
20
81
Table 10: Ablation study the effect of the reduced dimensionality of f.
Table 10 presents the effect of varying the reduced dimensionality in the traffic environment. As m increases from to 6, the sparsity increases sigmificantly while the hypervolume remains unchanged, resulting in scenarios where only few objectives perform well. When m increases further from to 10, both sparsity and hypervolume decrease, leading to a quwaitquality set of returns in the original reward space.
17

---

Published as a conference paper at ICLR 2025
m is hyperparameter for algorithm, and selecting an appropriate value in practice is achievable. This is because we can estimate the effective rank the sample covariance matrix recursively (Car- dot & Degras 2018) during the early exploration phase of the RL algorithm, rather than throughout the entire training process. We found that this straightforward approach performs effectively in our experiments.
D DISCUSSION ON THE NECESSARY CONDITION OF THEOREM
We acknowledge that theoretically analyzing the opposite direction of Theorem is challeng- ing. To find conditions for a counterexample of f beyond f(r) Ar with
Rmi
may follow similar flow in the proof of the Theorem [ Given Wm E A" suppose in the original reward space. We a monotonically strictly increasing function satisfying (A(A) (B) for all -j (Hayes al. 22022). Then we have f(Ez, IEr= rrr))) >p f(Brg, in (.1..)))) IEr 7're)) in the vaduced-dimensions space. If f further
3x E II s.t. Eg E==o0"ra >p Br 1E []
first impose that f [f,,... fm]
A >p B
satisfies
E Σ M(rr))
>p Bun (-hram)))
Σ t=
(13)
this gives contradiction and f becomes target counterexample. This is directly satisfied when ff is affine. However, it is difficult to find such an example other than affine functions, primarily due to the inequality in >p.
Alternatively, we conducted an empirical analysis by relaxing the condition equation 13 and directly optimizing f under the sole constraint of strictly monotonically increasing function. As part of our ablation study, we parameterized / as a strictly monotonically increasing function using neural network (similar approaches like Rashid al. (2018) but maintaining strict monotonicity) and trained it within the traffic environment, denoted by "Monotone."
Ours
Monotone
HV(x 10 1)
166.9
0
SP(x 10
2.3
5353.0
Table 11: Ablation study the effect of imposing strict monotonicity on
Table 11 shows that this approach resulted in hypervolume of zero, similar to the "-positivity" and "- r-owsst, pposiivitty cases in Table 4 This suggests that merely imposing strictly monotonically increasing function condition is insufficient to construct a meaningful counterexample in practice. Importantly, nonzero hypervolume was only achieved when both the affine and positivity conditions were satisfied, as demonstrated in the "-rowst" case from Table 4 These results underscore the empirical effectiveness of algorithm based on Theorem T
E
IMPLEMENTATION DETAILS
E SOURCE CODE AND ENVIRONMENT
For our implementation, we adapted morl-baselines (Felten et al. 2023) and integrated with sumo- rl (Alegre) 2019), toolkit designed for traffic light control simulations, discussed in Section For LunarLander-S1I we used bas-liass (Felten et al. 2023) with the reward function provided by the source code of Hung et al. (2023).
The traffic light system offers four distinct phases: (i) Straight and right turns for North-South traffic, (ii) Left turns for North-South traffic, (ii) Straight and right turns for East-Wes traffic, and (iv) Left turns for East- West traffic. At each time step, the agent receives 37-dimensional state, which includes one-hot encoded vector representing the current traffic light phase, the number of vehicles
18

---

Published as a conference paper at ICLR 2025
in each incoming lane, and the number of vehicles traveling at less than 0.1 meters per second for each lane. The simulation starts with a one-hot vector where the first element is set to one. Based on this state, the controller chooses the next traffic light phase. The time between phase changes is 20 seconds, with each episode spanning 4000 seconds, or 200 timesteps. When transitioning to a different phase, the last 2 seconds of the interval display yellow light minimize vehicle collisions. The reward, represented by a sixteen- siicen-dimeniionll vector, is calculated as the negative total waiting time for vehicles on each inbound lane. The simulation runs for 52,000 timesteps in total. For AunarLander-511 the simulation runs for 2M timesteps.
E.2 BASELINES
For our proposed method and the baselines, we set the discount factor y = 0.99 and use a buffer size of 52,000 and IM for traffic and LunarLander, respectively. In Base algorithm (Yang et al. 2019), we utilize a multi-objective action-value network Qo with an input size of observation dimension units each, and ReLU activations after each hidden layer. The output layer has a size of |A| x K. For the dimension reduction methods, the Qo network has an input size of input size of observation dimension plus m, two hidden layers of
plus K, two hidden layers of (B$(LLmmrLanddr/2256(rraaaaa
units with ReLU activations, and an output layer of size |A|] x m.
We train Qe using the Adam optimizer (Kingma & Ba, 2015), applying the loss function after the first 200 timesteps, with a learning rate of 0.0003 and minibatch size of 32. Exploration follows an c-greedy strategy, with c linearly decaying from 1.0 0.05 over the first 10% of the total timesteps. The target network is updated every 500 timesteps. We update @ using the gradient + (0), where I. (0) is the primary loss and L (0) the auxiliary loss in Yang et al. (2119) The weight À is linearly scheduled from 0 to 1 over the first 75% and 25% percent of the total timesteps in traffic and Lunarl andar, respectively. Sampling preference vectors Wm E n during training and execution follows the uniform Dirichlet distribution.
(o L())= = (1 - A) main
For the three online dimension reduction methods (our approach, the autoencoder, and our imple- mentation of online NPCA), we utilize the Adam optimizer for updates. In our method, the matrix A is initialized with each entry set to 1/K. The neural network ge has an input dimension of m, two hidden layers of 32 units each, and ReLU activations after each hidden layer. The output layer has a size of K. We use a dropout rate of 0.75 and 0.25 in in traffic and Lunarl andar, respectively (with 0 meaning no dropout). Equation 11 is optimized with learning rate of 0.0003 and update interval of 5 timesteps.
For the autoencoder, the encoder network has an input size of K, two hidden layers with 32 units each, and ReLU activations after each hidden layer. The output layer has a size of m. The de- coder follows the same architecture as Jo. but without dropout. The reward reconstruction loss is optimized with a learning rate of 0,0001 and an update interval of 20 timesteps.
For the online NPCA. we use ReLU parameterization for efficient learning (also implemented in PyTorch (Paszke et al. 2019)) to meet the constraint on matrix U. The matrix U is initialized similarly with each entry set to 1/K. NPCA is optimized with a learning rate of 0.0001, an update interval of ( reduced vector representation of r is (T - (4) E Imm following the PCA assumption that the transformed vectors are centered (Zass & Shashua 2006 Cardot & Degras 2018). For NPCA-ortho in traffic, increasing the value of 3 did not yield better orthonomality, so we set the update interval to 5 timesteps, keeping the same / value.
timesteps, and B 50000 10r00000nnnaaaneeeeee
The
For incremental PCA, we recursively update the sample mean vector of rewards as Mu+1 = 441/4 141 P++1 K and the sample covariance matrix as C++1 : C. + (141)= (re+1 po)(re+1 - T E R for each timestep t (Cardot & Degras 2018). Every 20 timesteps, we eigen-decompose the covariance matrix, selecting the top m eigenvectors ........ Um E R corre- sponding to the largest eigenvalues, and update U = [ui,..., "m] E k The reduced vector representation of is (r μ1) E RT, assuming the vectors are centered (Cardot & Degras 2018). U is initialized as a matrix with each entry set to 1/K. For evaluation, we generated fifteen and thirty five equidistant points on the simplex for LunarLander and the traffic environment, respectively. For evenly distributed sampling and calculating hypervolume and sparsity, we use the implementation provided in Felten et al. (023). We use infrastructures of Intel Xeon Gold 6238R CPU @ 2.20GHz and Intel Core 1090XX CPU @ 3.70GHz.
t
19


Reviews:
[{"review_id": "OnTO73RZpD", "reviewer": "Reviewer_Pwfy", "summary": "This paper proposes a reward dimension reduction technique to improve the scalability of multi-objective RL (MORL). Specifically, it adopts an affine transformation and theoretically establishes conditions to guarantee the Pareto-optimality of learned policies in the original reward space and to build a valid mapping between the reduced and original preference spaces. Additionally, a reconstruction neural network helps preserve the information of the original rewards. The paper evaluates the proposed method through experiments with sixteen objectives, demonstrating superior performance compared to existing reduction and non-reduction methods.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated. The scalability of MORL is crucial for real-world applications but is rarely studied. This work thoroughly considers the nature of multi-objective settings and develops a simple yet effective dimension reduction solution.\n- The paper is well-written and well-organized, with clear theoretical explanations and implementation details that make the methodology easy to understand.\n- Empirical evaluations and ablation studies are convincing, demonstrating the effectiveness of the proposed approach.", "weaknesses": "- The number of test preferences $N_e $, seems slightly insufficient, even for a 4-dimensional reduced reward space. I’m a bit concerned about whether this is enough to represent the performance across the whole Pareto front in the original 16-dim reward space.\n- I wonder if it would be helpful to evaluate the dimension reduction technique on existing MORL benchmarks, even those with fewer objectives. Doing so could help assess the influence of the reduction method on performance and visualize the Pareto front in both the reduced and original reward spaces, helping us understand the relationship of Pareto in two reward spaces. \n- In certain cases, learned policies may cover a broad Pareto front in the reduced reward space but occupy only a narrow area in the original reward space and lead to low sparsity. How could this situation be avoided or detected during evaluation? Without visualizing the high-dimensional Pareto front, I am curious whether metrics like hypervolume and sparsity are sufficient to detect such issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FCzJwxkmwJ", "reviewer": "Reviewer_uQAS", "summary": "The authors propose a novel method for reward dimensionality reduction as well as a new evaluation protocol & benchmark for high dimensional multi objective reinforcement learning. The developed approach reduces the dimensionality of the rewards with an affine transformation that is additionally constrained to be row stochastic & feature only positive elements in its matrix. The authors show in an empirical evaluation that their approach beats relevant baselines in a high-dimensional multi-objective environment.", "questions": "I am not quite sure whether due to the affine nature of your approach, the resulting transformation function will deterministically always be the same given the same data (?), but I presume that the online data collection as well as the base RL algorithm used will introduce some stochasticity into the results, so could you please add corresponding uncertainty estimates in your tables (it seems you should have the data already anyways since you mention 8 random seeds)?\n\nI think the following paper could enrich your related work section, would you care to review and potentially add it?\nWeber, M., Swazinna, P., Hein, D., Udluft, S., & Sterzing, V. Learning Control Policies for Variable Objectives from Offline Data. In 2023 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 1674-1681). IEEE.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "To the best of my knowledge, the approach is novel. It is furthermore simple & elegant and leads to demonstrated improvements over baselines on a high-dimensional multi-objective environment. In addition, the paper proposes a new evaluation setup for their work. Finally, it is well written & understandable also for readers outside of the multi-objective sphere.", "weaknesses": "The empirical evaluation is limited to a single environment. This is partially due to the fact that the authors address a problem that has so far been only explored to a limited degree before, however it still means the demonstrated superiority of the approach has to be taken with a grain of salt. Also, the evaluation doesn't feature uncertainty estimates. I would be glad to improve my score if those two points are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GyabcAyvLw", "reviewer": "Reviewer_iWoJ", "summary": "This paper presents a reward reduction approach for multi-objective reinforcement learning (MORL) that transforms a high-dimensional reward vector into a lower-dimensional one. Unlike previous methods that primarily rely on static datasets, this approach supports online learning while ensuring that Pareto optimality is preserved after the transformation. Additionally, the authors introduce a comprehensive training and evaluation framework for reward dimension reduction in MORL.", "questions": "1. In Tables 1 and 2, it seems the scale of the reported metrics is really large. Could the author illustrate more about metrics?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on the important problem of multi-objective reinforcement learning in the presence of high-dimensional rewards.", "weaknesses": "1. This paper proposes applying an affine transformation to reduce the original high-dimensional reward vector to a lower-dimensional representation. However, the authors do not provide a theoretical justification demonstrating why reducing the reward dimension would improve multi-objective training, particularly when compression may result in information loss.\n2. One major limitation is the experimental scope. The experiments are conducted solely on a traffic light control environment, without exploring the impact of varying the dimensionality mmm, which undermines the generalizability and robustness of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "od2SR8YbHD", "reviewer": "Reviewer_ywhQ", "summary": "This work studies multi objective RL (MORL), considering the Pareto frontier and measures hypervolume and sparsity of solutions to judge viability in downstream tasks. They propose a new dimensionality reduction technique to tackle existing challenges in high-dimensional problems. This can help to collapse similar objectives to reduce the effective reward space. The authors give some theoretical and empirical demonstration validating their method.", "questions": "Beyond those in the previous section, here are a few more questions for the authors. If the authors are able to sufficiently abate my concerns, I can raise the score.\n\n- In your theorem, do you think it would be possible to generalize to PSD matrices? I'm not seeing the hurdle, but it would be enlightening (for me at least (:  )  to discuss either way\n- Can you discuss any potential use of work like [3] to reduce reward space by canonicalizing objectives (ensuring they truly are distinct)? Would that be applicable here?\n- RE Table 4, Can you also give an ablation of \"-positivity\"? (with the rowst constraint)\n- L500 is unclear. Why does finite ep length introduce more overhead?\n- Relatedly, what is the complexity of the constrained optimization problem?\n- Table 1: why did you choose this reference point? Maybe you can normalize the values wrt ${10^{4*16}}=10^64$ then?\n- Can you provide the learning curves or more dynamic information about your algorithm? How about other statistics (cf https://github.com/google-research/rliable)\n- L325, A brief discussion on the connection to [1] would be interesting, if possible\n- How does this work (or MORL more generally) relate to the successor features framework, e.g. [2]?\n\nminor:\n- Fig 2, can you change to e.g. $|AB|^2$ if that is meant. Also the quality of figures is a bit low, maybe just increase DPI a bit\n- \"works\" -> \"work\"\n\n[1]: https://proceedings.neurips.cc/paper_files/paper/2022/file/f600d1a3f6a63f782680031f3ce241a7-Paper-Conference.pdf\n[2]: https://arxiv.org/abs/1606.05312\n[3]: http://aima.cs.berkeley.edu/~russell/papers/iclr21-epic.pdf\n\n\nI'm genuinely curious to understand the effect of dropout. I think the explanation starting at L475 is a bit confusing (for me). Can you offer any additional explanations/re-wordings?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Simplicity of idea and execution\n- Providing well-founded theory to motivate their method, and proof given is easy to follow\n- Modular (could be applied with other MORL methods)\n- Provides potential new SOTA for the sample experiment\n- nice explanation of MORL problem for non-experts\n- I was hoping to see an ablation on dropout as soon as it was mentioned, and it was indeed given (Maybe mention Table 3 in L363)", "weaknesses": "Overall, I can't quite tell if I should think of this as a theory or experiment paper. For the former, I believe the theory given is not strong enough. But for the latter, there is only one environment, so I think it is not fair to claim total SOTA performance without a more systematic set of experiments. Below I'll give some points of weakness mixed in with related questions.\n\n- Somewhat unclear empirics (the authors only provide scalar statistics on a single environment)\n- Theory is somewhat weak. Specifically, they provide a sufficient, but not necessary condition. However, the surrounding text (Line 320) makes it sound like a necessary condition \"we require...\". Just be careful to not over-sell the result.\n- I'm unsure how limiting the linearity in Theorem 1 is. Although it is beneficial for its simplicity from a computational perspective, is something being lost here? It seems to only capture linear relationships among various objectives. Can a counterexample be provided to better understand the breakdown of the (<=) other direction of the theorem? \n- Some pre-assumed knowledge: As mentioned in L412, the final rank=4 was assumed to be known. But this would come at the end of training. How big an impact is there in different choices of rank? Can the rank be changed adaptively?\n- L449: It's a bit unclear to me if NPCA was tuned appropriately for a fair comparison. Can better choices of the regularization strength be chosen? (Also, where's the reference to NPCA, I think I missed it)\n\nI'm not totally aware of the MORL literature, but are there any other baselines with which to compare?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)