COLA: Towards Efficient Multi-Objective Reinforcement Learning with Conflict Objective Regularization in Latent Space
Authors: Pengyi Li
Hongyao Tang
Yifu Yuan
Jianye HAO
Zibin Dong
YAN ZHENG
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Objective Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Many real-world control problems require continual policy adjustments to balance multiple objectives, which requires the acquisition of high-quality policies to cover diverse preferences. Multi-Objective Reinforcement Learning (MORL) provides a general framework to solve such problems. However, current MORL methods suffer from high sample complexity, primarily due to the neglect of efficient knowledge sharing and conflicts in optimization with different preferences.
To this end, this paper introduces a novel framework, Conflict
Objective Regularization in Latent Space (**COLA**).
To enable efficient knowledge sharing, COLA establishes a shared latent representation space for common knowledge, which can avoid redundant learning under different preferences. Besides, COLA introduces a regularization term for the value function to mitigate the negative effects of conflicting preferences on the value function approximation, thereby improving the accuracy of value estimation. The experimental results across  various multi-objective continuous control tasks demonstrate the significant superiority of COLA over the state-of-the-art MORL baselines. Code is available at https://github.com/yeshenpy/COLA.

COLA: Towards Efficient Mujti-Objective Reinforcement Learning with Conflict Objective Regularization in Latent Space
Pengyi Lis, Hongyao Tangł, Yifu Yuan, Jianye Hao*, Zibin Dong, Yan Zheng College of Intelligence and Computing, Tianjin University
Abstract
Many real-world control problems require continual policy adjustments to bal- ance multiple objectives, which requires the acquisition of high-quality policies cover diverse preferences. Multi-Objectivy Reinforcement Learning (MORL) provides a general framework to solve such problems. However, current MORL methods suffer from high sample complexity, primarily due to the neglect of effi- cient knowledge sharing and conflicts in optimization with different preferences. To this end, this paper introduces novel framework, Conflict Objective Regular- ization in Latent Space (COLA). To enable efficient knowledge sharing, COLA establishes shared latent representation space for common knowledge, which can avoid redundant learning under different preferences. Besides, COLA in- troduces regularization term for the value function to mitigate the negative ef- fects of conflicting preferences on the value function approximation, thereby im- proving the accuracy of value estimation. The experimental results across vari- ous multi-objective continuous control tasks demonstrate the significant superi- ority of COLA over the ofateoff-the-ar MORL baselines. Code available https: / gi thub. Seem//eshhnny/CL..
Introduction
Real-world problems often involve multiple performance metrics, where conflicts among these met- ries make infeasible maximize all metrics simultaneously with single policy. This necessitates a trade-off among different metrics, requiring the learning of multiple high-quality policies to meet diverse preferences [1-3]. For instance, in robot control problems, performance metries may include both forward speed and energy consumption. In such cases, two reward signals are returned, and multiple optimal policies exist based on different trade-offs between the two metries [4]. Learning these policies enables us to control the robot more flexibly [5]. For example, initially, one might prefer a policy that prioritizes speed without considering energy consumption. Still, as energy loss accumulates, preference may shift towards an energy-efficient policy to extend running time.
popular approach addressing the multi-objective control problems involves leveraging Multi- Objective Reinforcement Learning (MORL) to acquire policies that meet human preferences [6, 7]. Current MORL algorithms can be roughly categorized into three categories: Single-policy methods, Multi-policy methods, and General policy methods. Single-policy methods directly transform multi- objective problems into lungle-objectiv problems and solve them using standard RL [6, 8, 9]. These methods typically require expert knowledge to predefine preferences, and the obtained policies are difficult to generalize to unknown preferences. Multi-policy methods seek a set of policies to cover optimal solutions under diverse preferences, which often require substantial amount of sample
"Contact me at lipengyi@ www..du.uuunnnnn.m Cooresponding authors: Hongyao Tang iatphoonggao@tu..dduuuuu Jianye Hao (jianye.hae hao @ @tw...duuu.nnn)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

cost to construct high-qualinn qualiit policies [10-12]. General policy methods aim to construct a single network with preferences as the condition to cover all policies. which have higher sample efficiency compared to the above two categories due to the implicit knowledge sharing [13-15].
However, the general policy methods still require a large number of samples to learn high-quality policies. We summarize two challenges. 1) Inefficiency of Knowledge Sharing: Although using single network allows implicit knowledge sharing, policies under different preferences need to redundantly learn common knowledge, resulting in inefficient learning. 2) Conffict of Optimization Directions: Different preferences may conflict in optimization directions. When using a shared network to approximate the value of a specific preference, it may compromise the approximation of the values of other preferences, resulting in tug-of-war during optimization. These conflicts impede the accurate approximation of the value function to distinct preferences, thereby resulting in suboptimal policies.
To address these challenges, we propose a novel framework named Conflict Objective Regular- ization in Latent Space (COLA). The core insight of COLA is to seek commonalities while ac- commodating differences, i.e., extracting and sharing the common knowledge required for learning under various preferences, while mitigating interference between different preferences when their optimization directions conflict. COLA comprises two main components: I) To solve the prob- lem of Inefficiency of Knowledge Sharing. we introduce the Objective-agnostic Latent Dynamics Model (OADM), which utilizes underlying dynamics transitions in the environment to extract a universal knowledge representation through temporal latent consistency. The knowledge extracted by OADM provides more compact general information of dynamics and is essential for learning policies under all preferences. Subsequent multi-objective optimization based on the latent repre- sentation constructed by OADM can thus be more efficient. 2) To address the problem of Conftict of Optimization Directions, COLA introduces Conflict Objective Regularization (COR). When the optimization direction of the current preference conflicts with the optimization direction of other preferences, COR constrains the value estimates for other preferences from deviating away from the original value estimates. This minimizes negative impacts (or interference) between optimizations for different preferences, allowing the value function to simultancously maintain value estimates for multiple preferences. The experiments in range of multi-objective control tasks show that COLA MORL algorithms across various general metrics,
significantly outperforms current achieving higher sample efficiency and ultimate performance.
2 Background
2.1 Multi-Objective Markov Decision Process
Consider a Multi-Objective Markov Decision Process (MOMDP) [16], defined by tuple (S, A. P, r, 72, f2) with state space S, action space A, transition function P(s,a), the vector- valued reward function with d objectives, the discount factor y E (0, the preference space Q and the scalarization function fo which produces a scalar utility using preference w e Ω. In our work, we use linear preference function, i.e., fw( a)) = rrss, a), which is commonly adopted in the MORL works [13, 14, 17]. When the number of objectives is d = 1, the MOMDP simplifies to standard MDP. If multiple objectives (d 1) and all possible returns are considered, a set of Pareto optimal solutions called Pareto Frontier (PF) can be obtained. A policy π is Pareto Optimal when no other policy n' can increase its expected return in any objective without decreas- ing the expected return in any other objective. In continuous control tasks, obtaining the PF is an NP-hard problem [4]. Therefore, the goal of MORL is obtain an approximation of the PF
In this paper. we focus on the general policy methods. Therefore, our objective is to construct a general policy T[als, w) [18, 19] that, given a state s S, can generalize across the entire space of preferences by adjusting the preference w E J2
2.2 Envelope Soft Actor Critic
We introduce foundational MORL algorithm, which integrates Envelope Q-Learning with SAC. We refer this algorithm as Envelope SAC [14, 20], which serves as the foundational algorithm for our work. Specifically, the multi-objective Bellman equation for Envelope SAC can be formalized

---

as follows:
Lq ED wnl? TWT (Qb(s,,,w) - (r V (s', w)) (s',
=
V) E [Qu (s, (o(a)s, - ar log Te (als, w) 1d]
where (s, a, r, ) sampled from the replay buffer D. Φ C (or () are the parameters for corre- sponding networks (or the target network), and 1 denotes a d-dimensional vector of all ones. To ensure that the samples contribute to the optimization of all preferences, w is randomly sampled from the preference space Ω rather than using the preferences applied during the interaction. The general policy for different preferences optimized through the following equation:
L (w) = E sup fa log (To(a)ss w)) - w TTQ (s, a, w() w'
(2)
where a is sampled from Te. We follow the structures of SAC, maintaining two Q-functions to alleviate overestimation issues and adjusting in learning manner.
3 COLA Framework
This section introduces our framework Conflict Objective Regularization in Latent Space (COLA). We begin by introducing how to construct a latent space by abjestive-agnostcc Latent Dynamics Model for efficient knowledge sharing. Subsequently, we elaborate on how mitigate the impact of conflicting preferences on value function approximation through Conflict Objective Regularization. Finally, we provide an overview of the COLA framework.
3.1 Objective-agnostic Latent Dynamics Model
In complex multi-objective continuous control tasks, optimizing a policy or set of policies to cover the entire preference space typically requires a significant sample cost. To enhance sample effi- ciency, general policy methods employ shared networks and integrate preferences as conditional inputs to facilitate implicit knowledge sharing across various preferences. However, the lack of effi- cient extraction and utilization of general knowledge limits the efficiency of this learning approach.
To solve the problem, we propose the Objective-agnostcc Latent Dynamics Model (OADM) to ex- tract and share the representations of common knowledge in the underlying environment dynamics transitions. Our key insight is that the dynamic transition information underlies robot motion and must be mastered, regardless the preference policies for robots. By explicitly extracting and shar- ing the representations, the policies avoid redundant learning of the common knowledge for different preferences, and a more compact and beneficial latent space is constructed, making multi-objective optimization more efficient.
The architectural diagram of OADM is de- picted in Figure 1. Specifically, the OADM consists of two components: a state encoder E. and a transition model D We formalize the components as follows:
a2
241
MSE
OCt
MSE
Encoder: 21 = Egg (s:) Transition : Cc+1 = Dy (Zt, (c)
Sz
5t+1
Figure 1: The conceptual illustration of Objective- agnostic Latent Dynamics Model (OADM). We construct consistent state representations 2 and state-action representations +.a in latent space for extracting common dynamics information.
In OADM, the encoder maps state 8, to the latent representation zt. The transition model takes the latent representation 2t and a, as in- puts and output Ct+11 To extract the consistent representations in dynamic transitions, OADM leverages the temporal consistency loss to make C++1 predict the latent representation 21+1 of the next state 8141- The loss is defined as follows:
CoxDn(e; = Ep [E Dv(2,, 1
(3)
where (8r, 42 8141) ) is sampled from the replay buffer D.
3

---

The temporal consistency loss is used in several previous methods [21-26], which are predominantly tailored for single-objective optimization tasks. In addition to capturing dynamics, these methods inject task-specific information by maximizing the value function or predicting rewards. However, when applied to multi-objective optimization, this approach may introduce potential conflicts, par- ticularly in the optimization of conflicting objectives. Therefore, in COLA, we remove the influence of different preference values and rewards on OADM. This intentional separation allows us to con- centrate specifically providing objettive-agnostic state transition information.
For the multi-objectto optimization process, the main distinction lies in the optimization be- ing performed the more compact latent space constructed by OADM, rather than in the orig- inal state space. The latent policy optimiza- tion process is depicted in Figure 2, which con- sists of two parts: multi-ohjective value func- tion approximation and multi-objective policy optimization. Specifically, to optimize more ef- ficiently, the value function takes the latent state representation 24, the latent state-action repre- sentation c++1, the action at, and the preference w as inputs. The original state information 8t can be optional. However, in the experiments, we observe that discarding the original infor- mation 8t sometimes results in unstable policy updates, possibly due to real-time updates of OADM. Therefore, during the learning process, we also take the stable input s, to mitigate the problem. The formulation of the optimization loss Lo can be rewritten follows:
St
Z
I+1
(( E.1. we)
at
Wt
Optional
W
Figure 2: Multi-objective policy optimization in latent space.
LQ = ED W (Qo(se, Ze, C441, (,,1,,,,,,) w)²
(4)
where y = T(2,, az) + VE ($141, 2(:1, The policy x makes decisions by taking St, 21 and w as inputs. The loss Lx of policy is formulated as follows:
{a log (at 8t, 21, w)) TTQ (st, Ze, D(z,, at, w²) }]]
(w) = EDD -ED
E Rage
sup w'eR
(5)
With efficient representation and sharing of common knowledge, multi-objective value function and policy optimization can become more efficient.
3.2 Conflict Objective Regularization
The multi-objective value network is crucial for providing policy gradient guidance under various preferences. In current general policy methods, a shared architecture is employed for the value net- work. Leveraging the shared architecture facilitates task-related knowledge sharing among similar preferences, leading to more efficient learning. However, in multi-objective optimization, numerous conflicting preferences exist. Approximating the value for one preference can have negative impacts on the value estimates for other potentially conflicting preferences. In essence, this is a manifesta- tion of Neural Network Churn [27, 28) in the context of multi-objective optimization. To this end, naturally we aim to study: How can we mitigate the negative impact of conflicting preferences?
To solve the problem, propose the Conflict Objective Regularization Loss (COR) to mitigate the negative effects of optimization direction conflicts among preferences. When the value function approximates the values of preference w;. COR aims to minimize the value estimate variation of random preference Wi if the optimization directions the value function under preferences Wi and wy conflict. This enhances the stability of value function learning, preventing drastic changes in the estimated values of conflicting preferences, thus preserving effective policy gradient guidance under different preferences.
To implement COR, we first need to establish metric for the degree of conflict. Concretely, we measure the stiffness (W;, wy) [29-31] of the value network between preferences wy and wj, which

---

Algorithm COLA Algorithm
1: Initialize: a replay buffer D, the objective-agnoste Latent Dynamics Model with encoder Eg and transi- tion model Dv, the MO policy nØ, the MO value function Qo. Ve.
2: repeat
3: Interaction for experiences
Sample preference wa from
5: The MO policy interacts with the environment based on wi
6: Store the transitions (s, 0, s',, done) to D
7:
# Optimize the OADM
8:
Optimize OADM with experiences in D (sec Eq.
9:
MO Learning the latent space
10:
Optimize the Critic:
11:
Sample experiences to calculate the COR loss with wa and random sampled wy [see Eq. 71
12:
Optimize
approximate the values under wa [see Eq. 8]
13:
Optimize the Policy: Optimize under wi (see Eq. 5)
14:
until reaching maximum training steps
is defined by:
VoLo(d:w:)) (0) ():
p pwss w)) (o: CQQ (d: wj)ll2
(6)
Low stiffiness indicates that updating the value network parameters toward minimizing Lo for pref- erence Wy will have a negative effect on the minimization of Lę for preference wj. Meanwhile, a higher stiffness indicates more consistent optimization direction for the value network parameters between Wi and wj. We set a conflict boundary. denoted as c. When the stiffness (w), wj) is greater than or equal e, consider that the optimization directions of the value functions under preferences Wy and wj are relatively consistent, mutually reinforcing each other. However, when p (wa, wj) is less than and wy. Additionally, a larger value of c-p (W,, wy) indicates more severe conflict.
assume that there is conflict in the optimization directions between w,
If (wi, wj)
O, introduce the COR loss to the value function under the preference wy to ensure that the optimization under w; does not significantly compromise the value estimation under wy. The COR loss can be formalized as follows:
CoOr ED max(c- - p wy), ()) ((w)))²
A(wy)
where the term À represents the degree of conflict, with a larger value indicating greater conflict, consequently, the regularization term's 's constraining effect will be stronger. The term A A(wy) rep- resents the magnitude of the change in Q. Q' the value function of Q optimized based on the preference Wi. i.c., Q وi Q'. Minimizing A(wy) helps mitigate the impact on the value estima- tion under Wj. In the practical implementation, obtaining the optimized value function Q' for the next step to optimize the current parameters challenging. This can be addressed by leveraging the MAML method [32]. However, it introduces significant additional computational overhead. To simplify this process, we employ a surrogate approach by maintaining an old Q network from the previous step, denoted as Qoid. and using the squared difference between the current ((wy) and the previous step's 's (wi) as replacement for AWwy). Finally, the value function loss COLA is defined as follows:
(8) where 3 is the hyperparameter to balance the impact of COR loss. Through CCOLA. we can reduce the negative impact on the value functions of other preferences when optimizing any preference wi, thereby enhancing the ability of the value function to maintain values for different preferences.
LOOLa = LQ + 3 Leor,
3.3 COLA Algorithm
We provide the pseudo-code of COLA in Algorithm 1. In each iteration, the algorithm proceeds across three phases (denoted by blue). First, we randomly sample a preference w; from the pref- erence space Ω and interact with the environment based on and w;, the interaction experiences
5

---

Table 1: Comparison of COLA with other MORL algorithms across ten tasks in terms of the three metrics. All algorithms are trained for 3 million environment steps. The reference point is set to the origin for HV calculation. COLA achieves remarkable superiority other algorithms in all metrics with an equivalent number of samples.
Tasks
Metric
Envelope SAC
PGMORL
Q-Pensieve
CAPQL
COLA (Ours)
HV (x10%)
5.54 = 0.47
5.13 + 0.06
5.66 = 0.16
5.84 + 0.14
7.91 + 0.06
HalfCheetah-22
UT (x10")
2.19 = 0.11
2.14 0.03
2.22 = 0.06
2.24 0.03
2.90 +.07 0.07
ED
0.40 0.04
0.34 0.00
0.37 0.00
0.50 0.00
0.00
HV (x10")
2,00 + 0.10
1.90
2.06 = 0.08
1.65 0.07
2.11 0.03
Hopper-2d
UT (x10))
4.15 0.09
4.12
4.21 + 0.07
3.95 0.09
4.28 + 0.02
ED
0,61 = 0.00
0.23
0.69 = 0.01
0.04 0.00
0.91 + 0.00
HV (x10%)
6.90 = 0.06
+
6.93 0.16
7.07 0.98
11.33 + 2.76
Ant-2d
UT (x
2.38 1 0.01
2.12
2.39 0.03
2,43 0.15
3.04 0.40
U.44 0.05
0.00 + 0.00
0.62 0.01 0.01
0.34 0.09
0.90 0.02
HV (x10%)
4.47 + 0.08
4.50 0.16
4.37 0.19
3.35 0.26
4.87 0.05
Walker-2d
UT (x10))
1.95
1.94
1.91
1.78 0.05
2.02 102:0.00
ED
0.52 0.00
0.51
0.32
0.26 0.01
0.90 0.00
(x (x10
3.29 + 0.27
3.35 0.07
3.29 + 0.20
1.92 + 0.23
3.41 + 0.04
Hopper-3d
UT (T
3.21 + 0.04
3.21
3:20
2.70 0.11
3.22 + 0.06
ED
0.61 + 0.03
0.55 0.000
0.45 + 0.03
0.03 + 0.00
0.85 + 0.01
IIV (x10%)
5.76 0.09
7,80 0.552
7.32 2.97
9.88 1 0.89
18.58 + 0.76
Ant-3d
UT (x
1.60
+
1.82 0.24
2.02 0.06
2.68 + 0.04
ED
0.13
0.39 0.01
0.31 + 0.04
0.72 + 0.00
0.95 0.00
HV (x10'*)
4.30 = 0.75
3.15 0.11
3.90 = 1.23
5.03 = 0.12
7.23 + 2.84
Ant-4d
UT (x10")
4.53 = 0.19
4.19 + 0.07
4.36 = 0.34
4.71 0.02
5.17 + 0.45
ED
0.51 1 0.01
0.15 0.00
0.16 1 0.02
0.70 1 0.00
0.97 + 0.00
(x10
5.24 2.00
0.55
5.03 2.77
7.56 2.59
9.15 + 1.58
HalfCheetah-5d
UT (x10")
2.56 0.32
2.64
2.60 0.26
2.65 0.35
3.11 + 0.12
ED
0.39 0.02
0.56
0.46 = 0.03
0.48 0.02
0.61 + 0.00
HV (x10"%)
7.83 = 0.95
9.80 0.31
8.68 0.89
3.37 3
10.41 + 0.46
Hopper-5d
UT (x10")
2.59 1 0.05
2.71100011
2.63
1.68 0.05
2.71 + 0.01
ED (x10"9
U.44 0.01
0.67 0.00
0.39 0.39
0.12 0.00
0.68 0.00
3.25
3.02
0.00
3.53 0.08
3.56
Ant-Sd
UT (x10%)
2.02 0.03
1.002
000003
2.04 0.02
405
ED
0.34 0.02
0.50
0.50
0.39 0.03
0.57 0.00
are then stored into the replay buffer D. Subsequently, we optimize the Objective-agnoste La- tent Dynamics Model (OADM) based on Eq. 3, extracting dynamic transition representations and constructing a more compact latent space. Within this latent space, we randomly sample new pref- erence wi to optimize the value function and policy. To calculate COR loss, an additional preference wy is sampled, and the value function optimized based on our new loss with Eq. 8. Finally, policy improvement is conducted on the policy based Eq. 5.
4 Experiments
This section provides a comprehensive evaluation of COLA through experiments. We first present the experimental setup, followed by performance comparisons and experimental analysis.
4.1 Experimental Setting
Our experiments are primarily conducted on multi-objective continuous control tasks proposed by PGMORL [4]. Additionally, we construct more complex multi-objective optimization tasks, such as Ant, Hopper, and HalfCheetah, each with more than two objectives. Detailed information can be found in Appendix B. We compare COLA with several state-of-the-arr MORL algorithms, including Envelope [14], PGMORL [4], Q-Pensieve [20], and CAPQL [33]. For fair comparison, we use the official implementation, fine-tune the hyperparameters and present the best results obtained in each

---

240
MO-Aun-M
5.5
MO-WalleeSt
Mo-llaECheea-44
No--liepeed
1.50
4.
1.22
2.5
100
075
2,000
Eandgg
022
53
10
13
55
0.0
13
20
23
30
00
35
50
13
23
30
0.0
10
13
53
30
Mo-Higppedd
1000
305
186
1800
166
240
170
126
200
140
1800
Za loateae Suprenic)
Za Sevorte Sepretie
Ia boloom Deprotic)
Figure 3: Hypervolume and Expected Utility comparison for continuous control tasks.
2d
MO-Ant-2d
M0-Hopper-24
6000
Eaccgyy Efticleney
Y Forward Spoed
2000
Reward
2000
2000
1900-
COLA (Ound)
Earoolope SAC
2000
2000
PGMORL
500-
CAPQU.
Pensieve
1000
3000
1000
2000 Speed
30d0
Speed
Speed Forward
MO-Walker-2d
MO-Hopper-3d
MO-Ant-3d
Eleegy Enticlency
2000
Rag Sgecc
Foreed Speal Plasad
Speed
Figure 4: Pareto front comparison for continuous control tasks.
task. It's worth noting that, as the original Envelope is tailored for discrete tasks, we provide its SAC version for comparison in continuous tasks. All algorithms are trained for 3 million environment steps. All statistics are obtained from 5 independent runs and the mean and standard deviation are reported, which is consistent with the setting in previous MORL works. More implementation details can be found in Appendix C.
Performance Metries: We follow the performance evaluation metrics proposed in previous meth- ods [20]. which focus on the following metrics:
Hypervolume (HV) is metric designed to quantify the convergence speed of Pareto fron- tier and the diversity policies within it. HV is defined HV:= SHOR) H[z E HR))ddz,
where R a set of return vectors attained, To Rd is reference point, H(R)
E Rd 3r R, < < r) and T is the indicator function. Expected Utility (UT) serves as a metric to evaluate the overall performance of policies under linear scalarization and is defined as [UT Ewen [maxrer (w T r)]. where R a of return vectors attained.
Episodic Dominance (ED) is defined as ED1.2 = Ewen [I EEE w WT! El r
where r/ r are return vectors, and Ti. Tr are the episode lengths. ED is metric designed for pairwise comparison of algorithms, particularly suitable for problems where there significant difference in returns for different preferences. In such cases, HV and UT may be dominated by a minority of preferences. We calculate ED between algorithms pairwise and report the average ED.
7

---

Table 2: Comparison between Envelope and COLA on DMC.
Tasks
Metric
Envelope
COLA
Cheetah Run-2d
HV (x10*)
8.61 + 0.02
12.71 = 0.02
Cheetah Run-2d
UT
318.80 = 29.25
415,98 = 30.51
Hopper Hop-2d
HV (x10²)
3,47 + 1.60
31.59 = 0.14
Hlopper_Hop-dd
UT
122.06 = 1.93
128.53 0.78
Walker_walk-
HV (x109)
5.89 + 3.98
1 3.14
Walker_walk-24
UT
238.42 + 84.43
311.69 = 59.90
Table Comparison between Envelope and COLA on the DeepSea Treasure task.
DeepSea Treasure
Envelope
COLA
HV
799.9 + 97.00
1028.93 + 3.37
UT
6.323 + 0.64
7.94 + 0.08
4.2 Performance Evaluation
We evaluate COLA and related strong MORL baselines across ten continuous control tasks. The experimental results shown in Table 1 indicate significant performance advantage of COLA over other methods. Compared to the second- best approach in different tasks, COLA demonstrates an average improvement of 26.8% in HV and 12% in UT. In pairwise comparisons with other algo- rithms, COLA consistently achieves significantly higher ED than other methods. Additionally, we also observe that the number of objectives increases, such as in HalfCheetah-5d and Hopper-5d, COLA exhibits more substantial performance improvements compared to other general policy meth- ods. This indicates that COLA exhibits better adaptability tasks with more objectives. Besides, we present the training process of COLA in comparison to other general policy methods in Figure 3. Notably, COLA demonstrates significant superiority over other algorithms in both HV and UT met- rics with an equivalent number of samples, showcasing its higher sample efficiency. Furthermore, we compare the Pareto fronts constructed by different algorithms, which offers a more intuitive rep- resentation of the quality of the discovered Pareto fronts. As depicted in Figure 4, COLA constructs a broader and denser Pareto front compared other algorithms. In the Ant-2d task, COLA's Pareto front is significantly superior to other algorithms. Moreover, in Hopper-3d, COLA almost covers other a algorithms. These visualizations consistently highlight the superiority of COLA.
In addition to the main experiments described above, we further evaluate COLA on three additional benchmarks to examine its generality and effectiveness. (1) First, we consider the DeepMind Con- trol Suite (DMC) [34] and extend it to the multi multi-objective setting, which involves image-based tasks. The objectives of these tasks include energy consumption and the original task goals, such as run- ning speed, walking speed, and jumping height. In this setting, we combined COLA with RAD and compared it against Envelope-RAN RAD with 800,000 environment steps. The results shown Table indicate that COLA consistently provides significant performance improvements across different tasks. (2) The second benchmark is the continuous-control DeepSea Treasure task proposed in Q- Pensieve [20]. Here, COLA is integrated with SAC and compared against Envelope. As shown in Table 3, COLA achieves better performance on both HV and UT metrics. (3) The third experi- ment involves two classic tasks from MO-Gym [3], namely Mo-lunar-lander and Mo-mountain-cam As reported in Table 4, we compared against GPI/LS, GPI/PD [35], and MORL/D [36]. The re- sults demonstrate that COLA exhibits notable performance advantage on Mo-lunar-lander while on Mo-mountain-car. it shows a slight disadvantage in UT compared to GPI/PD, Overall, COLA demonstrates strong and consistent performance across diverse tasks and algorithmic combinations.
4.3 Superiority of Components & Parameter Analysis
In this section, we delve into the impact COR loss on value function approximation. Our analysis is conducted on MO-Hopper-3d, encompassing three objectives: forward speed, jump height, and energy efficiency. We gather 10,000 states from the interaction between policies and the environment under diverse preferences, calculate the corresponding Monte Carlo ground truth, and determine

---

Table 4: Experiment on the two tasks from MO- My-Gym with 10,000 environment steps.
Task
Metric
GPI/LS
GPI/PD
MORL/D
COLA
Mo-lunar-landee
HV (x10*)
4.37
4.66
3.84
6.42
UT
13.52
15.12
3.71
30.18
Mo-mountain-crm
HV (x10*)
1.79
3.46
1.10
3.31
UT
-48.57
-42.47
-50.23
-50.42
(3J1,88,0)
(80)680
All preferences
290
220
200
150
150
Doge
Sepe
Dxpe
Figure 5: The comparison of ground truth error. The first three plots illustrate the error in three extreme preference settings, while the last plot showcases the average true error across multiple preferences sampled uniformly from the entire preference space.
the errors between the estimated values based on these states and the ground truth. The above follows the evaluation method for value approximation accuracy in TD3 [37]. The experimental results illustrated in Figure 5 demonstrate that using COR results in significantly smaller erross compared to not using it for extreme preferences (1.0, 0.0, 0.0) and (0.0, 1.0, 0.0), while the errors uniformly sample all possible preferences with
comparable for (0.0, 0.0, 1.0). Furthermore, a step size of 0.25 to calculate the mean errors. The results show that using COR can lead to smaller mean errors in the value function, indicating that COR is more conducive to maintaining value functions with multiple preferences.
Next, we conduct ablation experiments on two components of COLA, i.e. OADM and COR. We conduct ablation experiments on the Ant task with three objectives and the Hopper task with five objectives. We gradually remove two components. The results in Figure 6 demonstrate that remov- ing COR significantly decreases both UT and HV. Further removal of OADM on top of that leads to further decrease in the performance of both metrics. This indicates the crucial role of the two components in COLA.
Subsequently, we conduct the hyperparameters analysis introduced by COLA, i.c., e and B, where c used control the conflict boundary, and 3 is used control the weight of COR. For hyperpa- rameters we primarily chose values from {0.0, 0.25]. We conduct the experiments on the Ant-3d task. The results in Figure show that 0.25 outperforms c 0.0 in terms of both metrics. In ex- periments, we set to 0.25 for all tasks. Regarding B, we select values from {1.0, 0.1, 0.01, 0.001). We observe that 3 = 1.0 is more efficient, allowing COR to have greater impact. Thus we set 3 to 1.0 in most multi-objective tasks. For more details, please refer to Appendix B.2. In addition to the experiments above, we provide further analyses in Appendix D. including studies on the number of sampled preferences and the ablations and 2, and more detailed analysis of the parameter
5 Related Works
The MORL algorithms can be categorized into three categories. Single-policy methods transform multi-objective problems into single-objective problems by using scalarization function [6. 8, 9]. These methods typically require predefined preferences and cannot be generalized to other prefer- ences. Multi-policy methods approximate the Pareto front of the optimal solution by constructing a set of policies [10-12, 38 38-40, 4, 35, 3]. These methods commonly face the challenge of high sample complexity, as they explicitly learn each individual policy. PGMORL [4] employs model to predict the improvement along with preferences to guide the learning process. General policy methods take preferences as input and aim to cover policies through a single policy network. CN [13] extends single-objective DQN to multi-objective for approximating Q values under differ- ent preferences. Besides, CN introduces Diverse Experience Replay to improve sample efficiency. Envelope Q-Learning [14] introduces envelope Q-functions, enabling efficient knowledge sharing

---

2.5
Mo-Ast-Id
10
NOO HeatCCeeddah-5
2750
MO.Ast-34
340 3.0-EiotCChethh-FF
2.30
22.0
275
.40
175
1753
130
Zae ceccoaed Sopeodid
Iivironco Sope
Figure 6: Ablation study on the components of COLA.
M0-Aatt-3
MO-Awi-M
2.0
MO-Anb-M Ad
MD-Aub-3d
10:00
2-0061
p-13
190
Unvtvement Octsls
Unvtrmer begs (4tch
(a) control conflict boundary
(b) 3 to control regular terms
Figure 7: Analysis of main hyperparameters.
when learning different preference objectives. PD-MORL [15] guides multi-objective training based on the correlation between preference and Q values. In addition, PD-MORL further enhances per- formance through parallel sampling and Hindsight Experience Replay [41]. Q-PENSIEVE [20] maintains multiple Q values generated during the learning process to efficiently leverage existing knowledge. CAPQL [33] addresses the failure of linear scalarization methods in multi-objective training by adding a concave term to the immediate reward. Moreover, some works have been devoted to developing benchmark toolkits for MORL [3]. However, existing policy methods face challenges associated with Inefficiency of Knowledge Sharing and Conflict of Optimization Direc- tions, limiting sample efficiency and ultimate performance. Thus propose OADM and COR loss to solve the problems.
Conclusion
In this paper, we propose novel framework, COLA, to more efficiently cover the entire preference space through a single network. We attribute the primary challenges of current general policy meth- ods to the Inefficiency of Knowledge Sharing and Conflict of Optimization Directions. To address these challenges, we first propose the Objective-agnostic Latent Dynamics Model. This model ex- plicitly extracts and shares consistent representations in dynamic transitions, establishing latent space to accelerate multi-objective learning. Additionally, we propose Conflict Objective Regu- larization to alleviate negative influences among different preferences in the value approximation process, constructing more accurate value function. our experiments, we demonstrate the sig- nificant superiority of COLA compared with various strong baselines.
Acknowledgments
This work is supported by the National Key Research and Development Program of China (Grant No. 2024YF the National Natural Science Foundation of China (Grant Nos. 624B2101, 62422605, 92370132), and Xiaomi Young Talents Program of Xiaomi Foundation. We would like to thank all the anonymous reviewers for their valuable comments and constructive suggestions, which have greatly improved the quality of this paper.
References
[1] C. Liu, X. Xu, and Hu. Multiobjective reinforcement learning: A comprehensive overview. TSMC: Systems, 2014.
10

---

[2] C. F. Hayes, R. Rádulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Ver- straeten, L. M. Zintgraf. R. Dazeley, F. Heintz, et al. A practical guide multi-objective reinforcement learning and planning. AAMAS, 2022.
[3] Felten, L. N. Alegre, A. Nowe, A. LC Bazzan, E. Talbi, G. Danoy, and B. C da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In NeurIPS, 2023.
[4] J. Xu, Y. Tian, P. Ma, D. Rus, S. Sueda, and W. Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In JCML, 2020.
[5] D. M. Roijers and S. Whiteson. Multi-Objective Decision Making. 2017.
[6] D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley. A survey of multi-objective sequen- tial decision-makinn making. CoRR, 2014.
[7] C. F. Hayes, R. Radulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Ver- straeten, L. M. Zintgraf, R. Dazeley, Heintz, E. Howley, A. A. Irissappane, Mannion, A. Nowé, G. de Oliveira Ramos, M. Restelli, P. Vamplew, and D. M. Roijers. A practical guide to multi-objective reinforcement learning and planning. Auton. Agents Multi Agent Syst., 2022.
[8] K. V. Moffaert, M. M. Drugan, and A. Nowé. Scalarized multi-objective reinforcement learn- ing: Novel design techniques. In ADPRL. IEEE, 2013.
[9] A. Abdolmaleki, S. H. Huang, L. Hasenclever, M. Neunert, H. F. Song, M. Zambelli, M. F. Martins, Heess, R. Hadsell, and M. A. Riedmiller. A distributional view on multi-objective policy optimization. In ICML, 2020.
[10] D. M. Roijers, S. Whiteson, and F. A. Oliehoek. Linear support for multi-objective coordina- tion graphs. In AAMAS, 2014.
[11] H. Mossalam, Y. M. Assael, D. M. Roijers, and S. Whiteson. Multi-objective deep reinforce- ment learning. CoRR, 2016.
[12] M. Zuluaga, A. Krause, and M. Püschel. e-pal: An active learning approach to the multi- objective optimization problem. 2016.
[13] A. Abels, D. M. Roijers, T. Lenaerts, A. Nowé, and D. Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In ICML, 2019.
[14] R. Yang, X. Sun, and K. Narasimhan, A generalized algorithm for multi-objective reinforce- ment learning and policy adaptation. In NeurIPS, 2019.
[15] T. Basaklar, S. Gumussoy, and Ü. Y. Ogras. PD-MORL: preference-driven multi-objective reinforcement learning algorithm. ICLR, 2023.
[16] K. Chatterjee, R. Majumdar, and T.A. Henzinger. Markov decision processes with multiple objectives. STACS, 2006.
[17] P. Kyriakis, J. Deshmukh, and P. Bogdan. Pareto policy adaptation. In ICLR, 2022.
[18] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approx- imators. In ICML., 2015.
[19] H. Tang, Z. Meng. J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Y. Yang, W. Tao, and I.. Wang. What about inputting policy in value function: Policy representation and policy- extended value function approximator. In AAAI, 2022.
[20] W. Hung, B. Huang, P. Hsich, and X. Liu. Q-pensieve: Boosting sample efficiency of multi- objective RL through memory sharing of q-snapshots. In ICLR. OpenReview.net, 2023.
[21] M. Schwarzer, A. Anand, R. Goel, R. Devon Hjelm, A. C. Courville, and P. Bachman. Data- efficient reinforcement learning with self pelf-predietive representations. In ICLR, 2021.

---

[22] N. Hansen, H. Su, and X. Wang. Temporal difference learning for model predictive control. In ICML, 2022.
[23] McInroe, Schäfer, and S. V. Albrecht. Learning temporally-consiseee representations for data-efficient reinforcement learning. CoRR, 2021.
[24] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint, 2023.
[25] Y. Yuan, J. Hao, F. Ni, Y. Mu, Zheng, Y. Hu, Liu, Y. Chen, and C. Fan. Euclid: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model. ICLR, 2023.
[26] Y. Zhao, W. Zhao, R. Boney, J. Kannala, and J. Pajarinen. Simplified temporal consistency reinforcement learning. In ICML, 2023.
[27] Schaul, A. Barreto, Quan, and G. Ostrovski. The phenomenon of policy churn. arXiv preprint, arXiv:2206.00736 2022.
[28) Hongyao Tang and Glen Berseth. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. In NeurlPS, 2024.
[29] S. Fort, K. Nowak, and S. Narayanan. Stiffness: A new perspective on generalization in neural networks. CoRR, 2019.
[30] S. Moon, J. Lee, and H. Oh Song. Rethinking value function learning for generalization in reinforcement learning. In NeurlPS, 2022.
[31] E. Bengio, J. Pineau, and D. Precup. Interference and generalization in temporal difference learning. In ICMI., 2020.
[32] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
[33] H. Lu, D. Herman, and Y. Yu. Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality. In ICLR, 2023.
[34] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq. T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint arXiv: 1801,00690 2018.
[35] L.N. Alegre, A. L. Bazzan, D. M. Roijers, A. Nowé, and B. C. da Silva. Sample-efficient multi-objective learning via generalized policy improvement prioritization. AAMAS, 2023.
[36] F. Felten, E. Talbi, and G. Danoy. Multi-objective reinforcement learning based on decompo- sition: taxonomy and framework. JAIR, 2024.
[37] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In JCML, 2018.
[38] S. Natarajan and P. Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In ICML, 2005.
[39] M. Pirotta, S. Parisi, and M. Restelli. Multi-objective reinforcement learning with continuous pareto frontier approximation. In AAAI, 2015.
[40] S. Parisi, M. Pirotta, and J. Peters. Manifold-based multi-objective policy search with sample reuse. Neurocomputing. 2017.
[41] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, Welinder, B. McGrew, J. Tobin,
P. Abbeel, and W. Zaremba. Hindsight experience replay. In NewrIPS, 2017.
[42] S. Khadka and K. Tumer. Evolution-guided policy gradient in reinforcement learning. In NeurIPS, 2018.
[43] A. Pourchot and Sigaud. CEM-RL: combining evolutionary and gradient- based methods for policy search. In ICLR, 2019.
12

---

[44] J. Hao, P. Li, H. Tang, Y. Zheng, X. Fu, and Z. Meng. Erl-re²: Efficient evolutionary reinforce- learning with shared state representation and individual policy representation. In ICLR, 2023.
[45] P. J. Hao, H. Tang, Y. Zheng, and F. Barez. Value--rvolttinaar--baeedd reinforcement learn- ing. In ICML, 2024.
[46] Li, Y. Zheng. H. Tang, X. Fu, and J. Hao. Evorainbow: Combining improvements in evolu- tionary reinforcement learning for policy search. In ICML, 2024.
[47] S. Majumdar, Khadka, S. Miret, S. McAleer, and K. Tumer. Evolutionary reinforcement learning for sample-efficient multiagent coordination. In ICML, 2020.
[48] P. Li, J. Hao, H. Tang, Y. Zheng, and X. Fu. Race: Improve multi-agent reinforcement learning with representation asymmetry and collaborative evolution. ICMI,, 2023.
[49] Y. Zheng, X. Xie, T. Su, L. Ma, J. Hao, Z. Meng, Y. Liu, R. Shen, Chen, and C. Fan. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In ASE. IEEE, 2019.
[50] Li, J. Hao, H. Tang, X. Fu, Zheng, and K. Tang. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey. IEEE TEVC, 2024.
13

---

A Limitations & Future Work
For limitations and future work, firstly our work is empirical proof of the effectiveness of the COLA idea and we provide no theory on optimality, convergence, and complexity. Second, one critical point is the expressivity of the representation method, which is not optimally addressed by this work. Thirdly, our work currently focuses solely on linear utility functions. Therefore, the theoretical foundation, improved representation methods, and the incorporation of nonlinear utility functions are all worthy of further investigation and exploration, which we leave for future work.
In addition, in previous analysis, we observed certain degree of policy degradation and forget- ting, where behaviors learned in the early stages could not be retained. However, such phenomena have not been observed in population- paalattoo--baed multi-objective algorithms. Recently the integration of evolutionary algorithms (EAs) and RL has been proven effective in single-objective optimiza- tion tasks [42-49], but its framework for multi-objective optimization remains underexplored. We believe that exploring this direction, which emphasizes collaborative optimization and the mutual enhancement between RL and EAs [50], can better address the current challenges faced by MORL. We regard this as key direction for future research in MORL and plan to further explore it based on the COLA framework.
B Environment Setup
We conduct performance tests in multi-objective MUJOCO tasks. Below are the specific configura- tion details for each task.
MO-HaffCheetah-2dl Observation and action space dimensionality: S E IR! E RS and the enviromment runs for 500 steps. The first objective is forward speed: R1 = U2 +C. The second objective is control cost: R2 4 Σ, of + C. C = 1 is the alive bonus, U/2 is the speed in x direction, a; is the action of each actuator
MO-Hopper-2d: Observation and action space dimensionality: S E I A 3 and the environment runs for 500 steps. The first objective is forward speed: R₁ = 1.5v + C. The second objective is jumping height: R₂ 12 x (h himst) + C. C is composed of alive bonus and energy efficiency, U2 is the speed in x direction, h is the current height, Aint is the initial height, a, is the action of each actuator.
- 0.0002 Σ, o?
MO-Walker-24 Observation and action space dimensionality: S 1 R and the environment runs for 500 steps. The first objective is forward speed: R₁ = Up +. The second objective is energy efficiency: R2 is the speed in z direction, O; is the action of each actuator.
Σ, af C. C' i the alive bonus, U/
MO-Ant-2d: Observation and action space dimensionality: S S R22 A E R and the environment runs for 500 steps. The first objective is x-axis speed: R1
C. The second objective is y-axis speed: R2 = uy + C. C = - 0.5 x Σ, at is composed of alive bonus and energy efficiency, Ur is x-axis speed, By is y-axis speed, a; is the action of each actuator.
MO-Hopper-3dd Observation and action space dimensionality: S E B E R and the environment runs for 500 steps. The first objective is forward speed: = 1 + The second objective is jumping height: R2 = 12 x (h - himit C.C. The third objective is energy efficiency: R3 = 4 Σ, at + C. C
is the alive bonus, V. is the speed in z direction, is the current height, himit is the initial height, o, is the action of cach actuator.
MO-Ant-3d: Observation and action space dimensionality; S E R27 A E R. and the environment runs for 500 steps. The first objective x-axis speed: R1 = Ug. The second objective is y-axis speed: Rs = uy- The third objective is energy efficiency: R3 = 4 0.5 x Σ, a. C = 1- 0.5 x Σ, of is composed of alive bonus and energy efficiency, ez is x x-axis speed, By y-axis speed. a; is the action of each actuator.
MO-Ant-4d: Observation and action space dimensionality: S IR22, E R& and the environment runs for 500 steps. Each objective is energy efficiency of one leg and x- (af + af.) ))=)) + Up. agn and a; are the action of each actuator the ith leg. The objective of this task is to enable the agent to maximize velocity
axis speed: R(1,23314 4.0
14

---

while simultaneously controlling the energy cost of different legs, aiming to achieve precise control of each leg.
Observation and action space dimensionality: S E R A R. and the environment runs for 500 steps. The first objective is forward speed: R1 0.5 x
MO-HallCheetah-sd:
Ve + C. The second objective is jumping height: R₂ = 20 (h himit) + C. The third
objective is the control cost of the front leg: R3 = 4 1.5 x (a f. + 47,1 1, 1)++0 C. The fourth
objective is the control cost of the torso: R4 4 1.5 (a%,0 1, + (
C. The fifth
objective is the control cost of the hind leg: Rs 4 1.5 x (4 + ( C. C = 1
is the alive bonus, V2 is the speed in x direction, af.o. af.i. ar,0 1,0,00 and 0h,1 are the actions of different actuators.
MO-Hopper-5d: Observation and action space dimensionality: S E R .A R3, and the environment runs for 500 steps. The first objective is forward speed: RS = 1.5 x Eg + C. The second objective is jumping height: R2 = 12 x (4 - -/inst + C. The third objective
is the energy efficiency of the first actuator: Rg 4- 3 ag + C. The fourth objective is
the energy efficiency of the second actuator: R4 = 4- 3 x af + C. The fifth objective is
the energy efficiency of the third actuator: Rg 4-3 az + C. C = 1 is the alive bonus,
is the speed in x direction, h is the current height, himit is the initial height, a, is the action of each actuator.
Observation and action space dimensionality: S R22 . RR, and the environment runs for 500 steps. The first objective is x-axis speed: Ri = Ug. The second the fifth objectives are the energy efficiency of different legs: R12,3,433
MO-Ant-5d:
4.0 2 x (a + ( + Uz. ago0 and 0,,1 are the actions of each actuator on the ith leg. This task is similar to MO-Ant-4d, but the key difference lies in the desire to achieve precise energy consumption control for different legs under varying speeds.
B.1 Experiment Details
During training, we predefined a preference set to record HV and UT every 50000 environment steps. These preferences are sampled regular intervals the preference space, covering all possible preferences. For two-objective tasks, the preference interval is set to 0.005. For three-objective tasks, the preference interval is set to 0.05. For four-ohjective tasks, the preference interval is set to 0.1. For five-objective tasks, the preference interval is set to 0.2. Then we use these preferences as conditions for the general policy methods, collecting return vectors for each preference. Based on these vectors, we calculate HV and UT. For algorithms like PGMORL which are multi-policy methods, we select the best policy for each preference to calculate HV and UT. To visualize the Pareto frontier, PGMORL directly plots the discovered set of policies. In contrast, for general policy methods, we set small intervals to explore their potential as much as possible. For two-objective tasks, preference interval of 0.0001 is used, and for threc-objective tasks, a preference interval of 0.002 is employed. All experiments are run on an Intel Xeon E5-2680 v4 @ 2.40GHz machine with an NVIDIA GTX 2080 Ti. Except for the DMC experiments, all other experiments conducted using only the CPU.
B.2 Experiments Compute Resources
All experiments are carried on CPU E5-2680 v4 @ 2.40GHz with 256GB memory
C Implementation Details
We introduce the implementation details of COLA in this section:
The baselines we compared include PGMORL', Envelope SAC, CAPQL', and Q- Pensieve?. We reproduce the results based on their official implementations, fine-tuning the hyperparameters involved in each algorithm to achieve the best performance results.
hhups://githubb. hhtnps//gihuh..om/haaay/////// http://www..coaaaaaccc/CCAA.... btups//gittub.co//XXYUUU Banddtts--abbli Laaa//-Pensieeee
15

---

Table Details of the hyperparameters of COLA that are varied across tasks.
Env name
3
MO-HaifCheetah-2d
0.01
MO-Hopper-2d
1.0
MO-Walker-2d
1.0
MO-Ant-2d
0.001
MO-Hopper-3d
1.0
MO-Ant-3d
1.0
MO-Ant-4d
1.0
MO-H HO-HHalCCheeta--55
1.0
MO- M0-HHoppe--5d
0.01
MO-Ant-5d
We built our code following official implementation of Q-Pensieve, with the excep- tion that the policy network has layer sizes of {128, 128) while keeping the rest of the architecture and hyperparameters consistent.
For the Objective-agnostic Latent Dynamics Model, a neural network with two layers, each having 256 units, is constructed. It utilizes the ReLU activation function, and the output dimension is set to 50.
Wet set = 0.25 for all tasks. Besides, we list the hyperparameters specific to COLA which varied across tasks in Table 5. you don't want tune hyperparameters, competitive performance can be obtained when keeping 0.25 and B = 1.0 in most tasks.
D Additional Experiments
Table Comparison between COLA with and without state variable under different environment steps.
Env steps
500k
1000k
1500k
2000k
COLA w/ s
15986288 COLA w/o 13874916 3685
3764
16765694
3822
18089111
3996
18933218
4062
16473696 I 1791
16564312 3870
14904042 3535
Why do we need to retain as an input? This is primarily based on empirical observations. We find that omitting causes performance fluctuations in some tasks, such as Hopper-2d. The HV and UT results are shown in Table 6: We observe that excluding leads to performance drops in learning process. Moreover, using as input is more efficient.
Table 7: Comparison between COLA and COLA with multi-prefferncce (w) under 2M environment steps.
2M env steps
Walker-2d
Hopper-3d
Ant-2d
HallCheetah-2d
COLA
4806728 2006
29541776029 3063
8722546
2696
7390226 2766
COLA w/ Multi w
4529242 1942 29957011493 I 3083
7927880 2545
7221848 2798
Is sampling single preference at each iteration sufficient, should multiple preferences be sampled instead? To answer this question, we follow CAPQL and sample a batch of preferences at each step. The results shown in Table 7. Overall, we find that sampling single preference yields better performance in most tasks. This aligns with recent findings suggesting that multiple training iterations the same batch lead improved outcomes. Fully learning under preference before switching to another appears more beneficial for training.
Ablation study on We conduct ablation study using only e while removing The experi- mental results are shown in Table We observe that incorporating z leads better performance compared to the variant without This improvement is primarily due to the fact that the encoder
16

---

Table 8: Comparison between COLA with and without latent variable 2 under 2M environment steps.
2M env steps
Walker-2d
Hopper-3d
Ant-2d
HalfCheetah-2d
COLA w/
4806728
2006
29541776029
3063
8722546
2696
7390226 2766
COLA w/o
4682616 1992
29431977661 3083
7974809 2543
7036496 2656
extracts higher-level latent information from the original state 8, which helps accelerate the approx- imation the critic function and thereby improves learning efficiency.
Table 9: Parameter analysis on the coefficient c. The best result for each task and metric high-
lighted in bold.
Task
Metric
= 0.0
c = 0.25
C == 0.5
= 1.0
Ant-3d
HV (1e10)
1.33
1.59
1.56
1.37
UT
2285.39
2483.19
2558
2403
HalfCheetah-5d
HV (le16)
6.78
8.59
7.20 + 0.81
5.94
UT
2962
2949
2902
2939
Analysis of the parameter that the COR loss has less influence, whereas e = 1.0 means that COR is always active, regardless of whether conflicts are present not. We initially conduct preliminary comparison between = 0.0 and e = 0.25 on a single case and find that = 0.25 yields the best performance. Based on this observation, adopt e 0.25 consistently across all experiments. We also conduct ablation study on the effect of c, as shown below in Table 9. The results indicate that c = 0.25 generally delivers strong performance across different tasks.
In all experiments, is set 0.25. A smaller value of indicates
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper focuses on further improving the performance MORL from two perspectives: knowledge sharing and value function approximation, and provides extensive experimental evidence to demonstrate the effectiveness of the proposed method. The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide a discussion of the limitations Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
This study focuses on empirical evaluation and does not provide theoretical proofs formal analysis. We leave the development of theoretical guarantees and formal analysis as future work.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide detailed reproduction details in the appendix and will release the code upon public release.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way to accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
19

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We commit to releasing the code upon the public availability of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips www.cccpppcccgcccccccccccc /tgiiiegg////////eee/eee/ieieeee CooeeSubieeeeeeeeeeeee ssi siooncicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide the detailed experimental settings in both the main experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Following the experimental settings of prior work, conduct each experi- ment with 5 independent runs and report error bars to reflect statistically significant vari- ability.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report 1-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https /neuripu..../// cttes.cc///// ctcpppbbii//thhcccciieeeeeeee
Answer: [Yes]
Justification: Yes, the research conducted in this paper fully conforms with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
21

---

Justification:
This paper presents work whose goal is to advance the field of MORL. There are many potential societal consequences of our work, none which feel must be specif- ically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all creators and original owners of the assets used in the paper are prop- erly credited and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
22

---

The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to NeurlPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: /ttps://neerree /ttp/reerspp.//// cc/Connfreeccs/2225/11 for what should or should not be described.
24


Reviews:
[{"review_id": "ZD1WbBD8PS", "reviewer": "Reviewer_G6fw", "summary": "This paper proposes COLA, a novel framework for Multi-Objective Reinforcement Learning (MORL) that combines two key components: an Objective-Agnostic Latent Dynamics Model (OADM) to enable efficient knowledge sharing across objectives, and Conflict Objective Regularization (COR) to mitigate optimization conflicts in the latent space. Experiments on 10 MuJoCo tasks show notable gains over state-of-the-art baselines.", "questions": "1. What is the per-iteration training overhead introduced by COR compared to baselines?\n2. Is there a theoretical motivation for choosing c = 0.25? Could it be task-specific? Would an adaptive or learned threshold perform better?\n3. How does the computational complexity of COR scale with the number of objectives or preference dimensions?\n4. The paper attributes instability from discarding raw states s_t to “real-time updates of OADM” (p.4). Could you provide a deeper analysis or diagnostics to clarify this issue?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method tackles two core challenges in MORL—inefficient knowledge sharing and conflicting gradients—through a thoughtful combination of latent dynamics modeling (OADM) and regularization (COR). The guiding idea of “seeking commonalities while accommodating differences” is intuitive and well-motivated.\n2. The paper is well-written and easy to understand.\n\nWeaknesses:\n1. The threshold choice for detecting conflicting preferences is empirically set without theoretical justification. Although the sensitivity analysis (Fig. 7) suggests this value works better than c = 0, its generality across tasks is unclear.\n2. The computational cost of COR is not analyzed. Training time and resource usage are not compared to baselines, leaving performance-efficiency trade-offs ambiguous.", "was_revised": true, "final_justification": "All my concerns have been satisfactorily addressed.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper.\n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:\n\n**1. [Re: What is the per-iteration training overhead introduced by COR compared to baselines?]**\n    \nWe thank the reviewer for the valuable suggestion. To address the concern, we have conducted the runtime experiments. The results show that COR incurs an additional time overhead of approximately 12.9% with one single CPU.\n    \n**2. [Re: Is there a theoretical motivation for choosing c = 0.25? Could it be task-specific? Would an adaptive or learned threshold perform better? ]**\n\nIn all experiments, *c* is set to 0.25. A smaller value of *c* indicates that the COR loss has less influence, whereas *c* = 1.0 means that COR is always active, regardless of whether conflicts are present or not. We initially conduct a preliminary comparison between *c* = 0.0 and *c* = 0.25 on a single case and find that *c* = 0.25 yields the best performance. Based on this observation, we adopt *c* = 0.25 consistently across all experiments.\n    \nWe also conducted an ablation study on the effect of *c*, as shown below. The results indicate that *c* = 0.25 generally delivers strong performance across different tasks.\n    \n|  | Metric | c = 0.0 | c = 0.25 | c = 0.5 | c = 1.0 |\n| --- | --- | --- | --- | --- | --- |\n| Ant-3d | HV (1e10) | 1.33 | 1.59 | 1.56 | 1.37 |\n|  | UT | 2285.39 | 2483.19 | 2558 | 2403 |\n| HalfCheetah-5d | HV (1e16) | 6.78 | 8.59 | 7.20 ± 0.81 | 5.94 |\n| HalfCheetah-5 | UT | 2962 | 2949 | 2902 | 2939 |\n    \nWe have not yet identified a robust solution for the automatic adjustment of *c*. One potential approach is to introduce heuristic methods, such as evolutionary algorithms, to explore and iteratively refine multiple *c* values. However, we have observed that *c* = 0.25 consistently serves as an effective choice across a wide range of tasks. For example, the following two domains, COLA also uses *c* = 0.25, further demonstrating the generality of this parameter choice.\n    \n- **experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)**\n    \n| Tasks | Metric | Envelope | COLA |\n| --- | --- | --- | --- |\n| Cheetah_Run-2d | HV (1e4) | 8.61 ± 0.02 | 12.71 ± 0.02 |\n| Cheetah_Run-2d | UT | 318.8 ± 29.25 | 415.98 ± 30.51 |\n| Hopper_Hop-2d | HV (1e2) | 3.47 ± 1.60 | 31.59 ± 0.14 |\n| Hopper_Hop-2d | UT | 122.06 ± 1.93 | 128.53 ± 0.78 |\n| Walker_walk-2d | HV (1e5) | 5.89 ± 3.98 | 7.73 ± 3.14 |\n| Walker_walk-2d | UT | 238.42 ± 84.43 | 311.69 ± 59.90 |\n\n- **Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds**\n    \n| DeepSea Treasure | Envelope | COLA |\n| --- | --- | --- |\n| HV | 799.9 ± 96.995 | 1028.933 ± 3.365 |\n| UT | 6.323 ± 0.6367 | 7.942 ± 0.07538 |\n\n- **Experiment on the two tasks from MO-Gym with 10.000 environment steps and 5 seeds**\n    \n|  | GPI/LS | GPI/PD | MORL/D | COLA |\n| --- | --- | --- | --- | --- |\n| Mo-lunar-lander HV(10^8) | 4.37 | 4.66 | 3.84 | 6.42 |\n| UT | 13.52 | 15.12 | 3.71 | 30.18 |\n| Mo-mountain-car HV(10^3) | 1.79 | 3.46 | 1.10 | 3.31 |\n| UT | -48.57 | -42.47 | -50.23 | -50.42 |\n\n**3. [Re: How does the computational complexity of COR scale with the number of objectives or preference dimensions?]**\nThe complexity of COR remains unchanged as the number of optimization objectives increases. This is because, regardless of how many objectives are involved, the optimization procedure still samples a pair of preference vectors (w_i, w_j) from the preference space for each batch of data and computes the COR loss based on Equation (7). Therefore, the computational complexity does not grow with the number of objectives.\n\n**4. [Re: The paper attributes instability from discarding raw states s_t to “real-time updates of OADM” (p.4). Could you provide a deeper analysis or diagnostics to clarify this issue?]**\n    \nThis is primarily based on empirical observations. We find that omitting s causes performance fluctuations in some tasks, such as Hopper-2d. The HV and UT results are as follows:\n    \n| Env steps | 500k | 1000k | 1500k | 2000k |\n| --- | --- | --- | --- | --- |\n| COLA w/ s | 15986288 \\| 3764 | 16765694 \\| 3822 | 18089111 \\| 3996 | 18933218 \\| 4062 |\n| COLA w/o s | 13874916 \\| 3685 | 16473696 \\| 3791 | 16564312 \\| 3870 | 14904042 \\| 3535 |\n    \nWe observe that excluding s leads to performance drops in learning process. Moreover, using s as input is more efficient.\n    \n\n---\n\nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work** and we are looking forward to more inspiring discussions."}, {"review_id": "fuk3Jh8ow0", "reviewer": "Reviewer_7qCH", "summary": "This paper proposes a multi-objective reinforcement learning framework called COLA. It introduces a shared base module to model states and state transitions in latent space, facilitating policy learning across different objectives. Additionally, COLA employs a regularization term on the Q-function to manage conflicts between objectives. Experimental results on multiple multi-objective tasks demonstrate COLA's superior performance compared to baseline methods.", "questions": "No more questions.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is clearly written, effectively presenting most methodological details.\n2. The methodology is natural and concise, facilitating reproduction and enabling further comparative research.\n3. Experimental results on several tasks are promising, clearly showing COLA’s superior performance compared to multiple baselines.\n\nWeaknesses:\n\n1. The rationale for incorporating the latent state-action representation into the inputs of the value function is unclear. From Eq. (3), it appears that the function $D(a, z)$ estimates the expectation of latent representation of the subsequent state resulting from action $a$. However, the benefit and necessity of including this representation in the Q-function inputs remain unclear and intuitively unconvincing for me.\n2. The design of the regularization term seems overly simplistic. Regularizing by sampling a single random preference within each iteration may be inefficient, potentially leading to instability or convergence issues.\n3. Methodological novelty is somewhat limited. Both latent space modeling of states and state-action transitions, as well as regularization approaches for managing conflicting objectives, are common designs in ML literature.", "was_revised": true, "final_justification": "The authors’ rebuttal primarily addressed the concerns from an empirical perspective. However, I still find the explanation insufficient regarding the specific methodological design and the novelty of the work. Therefore, I intend to maintain my current rating. Nevertheless, after reading the reviews from other reviewers, I’m fully comfortable with the final acceptance of this paper.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper. \n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:\n\n**1. [Re: The rationale for incorporating the latent state-action representation into the inputs of the value function is unclear. From Eq. (3), it appears that the function estimates the expectation of latent representation of the subsequent state resulting from action. However, the benefit and necessity of including this representation in the Q-function inputs remain unclear and intuitively unconvincing for me. ]**\n    \nThrough OADM, the latent state-action representation *c* captures an important piece of information—dynamic information. In locomotion tasks such as those in MuJoCo, it is difficult to accurately estimate the next state based solely on the current state and action. However, this missing information can often be implicitly captured and supplemented by the latent representation *c*.\n    \nBased on the state-action representation *c*, the value network effectively gains access to an implicit prediction of the next state. This can be seen as a one-step model rollout, helping the network better understand how a given state-action pair influences long-term returns under the current preference, and thus produce more accurate value estimates.\n    \nTo address the reviewer's concern, we conduct ablation studies on c across four tasks mentioned above (5 runs, 2M steps): Envelope (UT 1.0, HV 1.0), COLA (UT 1.29, HV 2.14), and COLA w/o c (UT 1.15, HV 1.37). We observe that removing c reduces performance due to absent temporal information, which hinders future state predictions and value approximation. \n    \n**2. [Re: Regularizing by sampling a single random preference within each iteration may be inefficient, potentially leading to instability or convergence issues.]**\n    \nTo address the reviewer’s concern, we follow CAPQL and sample a batch preferences each step. The results are shown below.\n    \n| 2M env steps | Walker-2d | Hopper-3d | Ant-2d | Half-2d |\n| --- | --- | --- | --- | --- |\n| COLA | 4806728 \\| 2006 | 29541776029 \\| 3063 | 8722546 \\| 2696 | 7390226 \\| 2766 |\n| COLA w/ Multi w | 4529242 \\|1942 | 29957011493 \\| 3083 | 7927880 \\| 2545 | 7221848 \\| 2798 |\n    \nOverall, we find that sampling a single preference yields better performance in most tasks. This aligns with recent findings suggesting that multiple training iterations on the same batch lead to improved outcomes. Fully learning under one preference before switching to another appears more beneficial for training.\n    \n**3. [Re: Methodological novelty is somewhat limited. Both latent space modeling of states and state-action transitions, as well as regularization approaches for managing conflicting objectives, are common designs in ML literature.]**\n\nCOLA targets two key challenges in MORL: the difficulty of knowledge sharing and the presence of conflicting objectives. To address these issues, we propose **OADM** and **COR**. OADM is designed to extract temporal dynamic information, while COR serves to constrain conflicting objectives and approximate the value function. **Our goal is to provide a simple yet effective MORL algorithm, supported by extensive experimental evidence**. In addition to the experiments presented in the main text, we further validate the effectiveness of our approach across several additional domains.\n- **experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)**\n    \n| Tasks | Metric | Envelope | COLA |\n| --- | --- | --- | --- |\n| Cheetah_Run-2d | HV (1e4) | 8.61 ± 0.02 | 12.71 ± 0.02 |\n| Cheetah_Run-2d | UT | 318.8 ± 29.25 | 415.98 ± 30.51 |\n| Hopper_Hop-2d | HV (1e2) | 3.47 ± 1.60 | 31.59 ± 0.14 |\n| Hopper_Hop-2d | UT | 122.06 ± 1.93 | 128.53 ± 0.78 |\n| Walker_walk-2d | HV (1e5) | 5.89 ± 3.98 | 7.73 ± 3.14 |\n| Walker_walk-2d | UT | 238.42 ± 84.43 | 311.69 ± 59.90 |\n- **Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds**\n    \n| DeepSea Treasure | Envelope | COLA |\n| --- | --- | --- |\n| HV | 799.9 ± 96.995 | 1028.933 ± 3.365 |\n| UT | 6.323 ± 0.6367 | 7.942 ± 0.07538 |\n- **Experiment on the two tasks from MO-Gym with 10.000 environment steps and 5 seeds**\n        \n        \n|  | GPI/LS | GPI/PD | MORL/D | COLA |\n| --- | --- | --- | --- | --- |\n| Mo-lunar-lander HV(10^8) | 4.37 | 4.66 | 3.84 | 6.42 |\n| UT | 13.52 | 15.12 | 3.71 | 30.18 |\n| Mo-mountain-car HV(10^3) | 1.79 | 3.46 | 1.10 | 3.31 |\n| UT | -48.57 | -42.47 | -50.23 | -50.42 |\n\n---\n\nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work** and we are looking forward to more inspiring discussions."}, {"review_id": "gbDXnb4h95", "reviewer": "Reviewer_okx7", "summary": "The paper proposes an empirical method to perform general policy multiobjective reinforcement learning, that is, the authors train a network that is able to output a policy for any arbitrary linear set of weights. The main novelties introduced by this paper focus on the fact that general policy methods are trained by sampling random sets of weights and sequentially training using them, which often results in conficts between the learning objectives. The authors propose to (1) learn the state transition function separately from the reward function, since the transition function is independent from the reward preferences this transfer smoothly across different training iterations; and (2) introduce a regularization term taking into account how much attrition there is with the previous set of weights.", "questions": "- What are the assumptions behind the schedule for the set of weights using in the training so that the regularization proposed works as expexted?\n\n- where is PGMORL's results in Figure 3?\n\n- It's not clear how the pareto front results in Figure 4 are built. Are those results for the fully-trained policies queried for different sets of preferences only after the training is done, or are those results for policies during training? same question for the numeric results provided in Table 1.\n\n- For halfcheetah it seems like CAPQL has discovered some policies that way more efficient for speed forward values of around 2000. This could result in significantly superior policies depending on the scenario (if the application mandates power efficiency should be >=2000, for example, CAPQL's policy would be better. This goes against all other resuts presented, I would expect some explanation of in which conditions this behavior could arise, because it could affect where the new proposed method is better or not.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "------\nStrenghts\n\n- The contributions from the paper are simple to implement and very logical\n- There is prospect of relatively high impact in a subarea of RL, and the proposal is widely applicable across pretty much any multiobjective problem.\n- Paper is very clear and well-written.\n- The proposal is a clear winner in performance in the experimental evaluation performed (which is limited in some ways, see weaknesses section)\n-----------\nWeaknesses\n\n- For some specific parts of the paper I have the feeling there wasn't much thought devoted to make the approach very general and applicable across other scenarios beyond the evaluated. IN special, I have the feeling the regularization proposed (Eq 7) might not make sense depending on the sequence of set of weights that are presented during training. The regularization only takes the current and the previous set of weights, so if the previous set of weights is very similar to the current one but they are extremely different from the previous 10 sets, there will probably be the \"tug of war\" effect the authors were trying to avoid and would very likely result in a reduction of performace. There is no comment regarding this in the paper nor an ablation experiment exploring how different regimes for line 4 of the algorithm (when the random set of weights is chosen) would affect performance (Fig 5 shows only uniform sampling and perhaps not even shown in the best way).\n\n- Also related, all the evaluations presented are done in Mujoco tasks, which effectively means there is just a single evaluation domain, it would be good to have some other different domains included to make sure the gain of performance is not only due artifacts of this domain.", "was_revised": true, "final_justification": "Strengths outweight the weaknesses in special for the work being widely applicable. I recommend the authors add some comments to the manuscript regarding how the choice of preference sampling strategy might affect the performance.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper. \n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:\n\n**1. [Re: There is no comment regarding this in the paper nor an ablation experiment exploring how different regimes for line 4 of the algorithm (when the random set of weights is chosen) would affect performance (Fig 5 shows only uniform sampling and perhaps not even shown in the best way). What are the assumptions behind the schedule for the set of weights using in the training so that the regularization proposed works as expected?]**\n    \nWe thank the reviewer for the helpful suggestion. In our current implementation, we sample a single preference w_i at each step and perform training based on it. This strategy follows the prior work Q-Pensieve, and we did not modify it in order to ensure a fair comparison.\n    \nThat said, we agree that more efficient preference sampling strategies may exist. To explore this, we conducted additional experiments using an alternative approach: instead of sampling only one preference vector during training, we sample multiple preferences instead of one preference. \n    \n| 2M env steps | Walker-2d | Hopper-3d | Ant-2d | Half-2d |\n| --- | --- | --- | --- | --- |\n| COLA | 4806728 \\| 2006 | 29541776029 \\| 3063 | 8722546 \\| 2696 | 7390226 \\| 2766 |\n| COLA w/ Multi w | 4529242 \\|1942 | 29957011493 \\| 3083 | 7927880 \\| 2545 | 7221848 \\| 2798 |\n\nOverall, we find that sampling a single preference yields better performance in most tasks. This aligns with recent findings suggesting that multiple training iterations on the same batch lead to improved outcomes. Fully learning under one preference before switching to another appears more beneficial for training.\n    \n**2.  [Re: Also related, all the evaluations presented are done in Mujoco tasks, which effectively means there is just a single evaluation domain, it would be good to have some other different domains included to make sure the gain of performance is not only due artifacts of this domain.]**\n- **Further experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)**\n    \nThe objectives for these tasks include energy consumption and the original task objectives, such as running speed, walking speed, and jumping height. In this case, we combine COLA with RAD and compared it against Envelope-RAD with 800.000 environment steps. The experimental results are shown as follows:\n    \n| Tasks | Metric | Envelope | COLA |\n| --- | --- | --- | --- |\n| Cheetah_Run-2d | HV (1e4) | 8.61 ± 0.02 | 12.71 ± 0.02 |\n| Cheetah_Run-2d | UT | 318.8 ± 29.25 | 415.98 ± 30.51 |\n| Hopper_Hop-2d | HV (1e2) | 3.47 ± 1.60 | 31.59 ± 0.14 |\n| Hopper_Hop-2d | UT | 122.06 ± 1.93 | 128.53 ± 0.78 |\n| Walker_walk-2d | HV (1e5) | 5.89 ± 3.98 | 7.73 ± 3.14 |\n| Walker_walk-2d | UT | 238.42 ± 84.43 | 311.69 ± 59.90 |\n- **Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds**\n    \n| DeepSea Treasure | Envelope | COLA |\n| --- | --- | --- |\n| HV | 799.9 ± 96.995 | 1028.933 ± 3.365 |\n| UT | 6.323 ± 0.6367 | 7.942 ± 0.07538 |\n- **Experiment on the two tasks from MO-Gym with 10.000 environment steps and 5 seeds**\n    \n|  | GPI/LS | GPI/PD | MORL/D | COLA |\n| --- | --- | --- | --- | --- |\n| Mo-lunar-lander HV(10^8) | 4.37 | 4.66 | 3.84 | 6.42 |\n| UT | 13.52 | 15.12 | 3.71 | 30.18 |\n| Mo-mountain-car HV(10^3) | 1.79 | 3.46 | 1.10 | 3.31 |\n| UT | -48.57 | -42.47 | -50.23 | -50.42 |\n    \nThe experiments above demonstrate that COLA can further improve the performance across both HV and UT, whether in image input tasks or other types of tasks.\n    \n**3. [Re: where is PGMORL's results in Figure 3?]**\n    \nWe sincerely apologize for overlooking this issue. The comparison table is provided below, and we will update the curves in a revised version.\n    \n| Ant-3d (env steps) | 1.6 M | 2 M | 2.5 M | 3 M |\n| --- | --- | --- | --- | --- |\n| PGMORL HV (1e9) | 3.10 | 4.63 | 7.22 | 7.80 |\n| UT (1e3) | 1.49 | 1.63 | 1.83 | 1.90 |\n| COLA HV (1e9) | 12.92 | 15.90 | 18.08 | 18.58 |\n| UT (1e3) | 2.39 | 2.48 | 2.61 | 2.68 |\n    \n| Walker2d (env steps) | 1 M | 2 M | 2.5 M | 3 M |\n| --- | --- | --- | --- | --- |\n| PGMORL HV (1e6) | 3.41 | 4.31 | 4.45 | 4.50 |\n| UT (1e3) | 1.73 | 1.89 | 1.92 | 1.94 |\n| COLA HV (1e6) | 4.53 | 4.81 | 4.83 | 4.87 |\n| UT (1e3) | 1.96 | 2.01 | 2.02 | 2.02 |\n    \n| Half-5d (env steps) | 1 M | 2 M | 2.5 M | 3 M |\n| --- | --- | --- | --- | --- |\n| PGMORL HV (1e16) | 4.31 | 6.54 | 7.08 | 7.52 |\n| UT (1e3) | 2.36 | 2.59 | 2.63 | 2.64 |\n| COLA HV (1e16) | 7.33 | 8.59 | 8.95 | 9.15 |\n| UT (1e3) | 2.73 | 2.95 | 3.07 | 3.11 |\n    \n| Hopper-5d (env steps) | 1.6 M | 2 M | 2.5 M | 3 M |\n| --- | --- | --- | --- | --- |\n| PGMORL HV (1e16) | 8.95 | 8.99 | 9.60 | 9.80 |\n| UT (1e3) | 2.69 | 2.70 | 2.71 | 2.71 |\n| COLA HV (1e16) | 9.65 | 9.93 | 10.02 | 10.41 |\n| UT (1e3) | 2.69 | 2.69 | 2.70 | 2.71 |\n**4. [Re: It's not clear how the pareto front results in Figure 4 are built. Are those results for the fully-trained policies queried for different sets of preferences only after the training is done, or are those results for policies during training? same question for the numeric results provided in Table 1.]**\n    \nFor all evaluation results, including Pareto front evaluations, we assess the performance **after training is complete**, using the trained policy. The full details of the evaluation procedure are provided in **Appendix B.1**.\n    \n**5. [Re: For halfcheetah it seems like CAPQL has discovered some policies that way more efficient for speed forward values of around 2000. This could result in significantly superior policies depending on the scenario (if the application mandates power efficiency should be >=2000, for example, CAPQL's policy would be better. This goes against all other results presented, I would expect some explanation of in which conditions this behavior could arise, because it could affect where the new proposed method is better or not. ]**\n    \nIn our experiments, we observed that in the HalfCheetah task, COLA initially achieves better performance in speed. However, COLA must maintain a significantly larger Pareto front compared to other methods. During training, this requirement can lead to a phenomenon where performance under certain preferences improves as a result of maximizing cumulative returns, while performance under other preferences degrades over time. We believe this may be attributed to a form of catastrophic forgetting, preventing the algorithm from consistently maintaining the best-found performance across all objectives. This issue may require additional constraint mechanisms to mitigate.\n    \nTherefore, if the goal is to satisfy a specific requirement—e.g., power efficiency ≥ 2000—this constraint should be explicitly incorporated into the optimization objective, such as by modifying the reward function accordingly. Doing so typically leads to better and more targeted results.\n    \n\n---\n\nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work** and we are looking forward to more inspiring discussions."}, {"review_id": "D9YAcoXRiO", "reviewer": "Reviewer_4KPM", "summary": "This paper introduces a novel MORL algorithm which provides two big improvements: knowledge sharing and resolution of conflicting gradients. The first improvement is done by sharing knowledge for multiple preferences under a single (conditioned) network and learning a latent model for the dynamics of the environment—this latent code is used as an additional input for both actor and critic. The second improvement is done by adding a loss term (regularization) which prevents the network from forgetting other preferences when updating for a given preference. Some experiments are run on MuJoCo-based environments, tested against some MORL algorithms, and showing the proposed method does well in terms of hypervolume, expected utility, and episodic dominance. There is also an ablation study showing that the two proposed improvements are actually helpful in terms of performance.", "questions": "- Why limitting the analysis only to Mujocos? Why not using the other MO-Gymnasium problems?\n- Why not comparing the SOTA algorithms: GPI, MORL/D or CMORL?\n- How was the hyperparameter search done?\n- Did you run experiments where you condition only with `c` (not `z` ) to see if z is adding anything at all? I suspect it should not, since it is essentially an encoded `s`?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Quality \nThe paper seems technically sound. I have a few comments that I believe would improve the overall quality of the paper.\n1. I like the idea of the gradient resolution. However, I’m missing a bit more intuition on why we need the latent encodings at all, as in the end the raw states are used as input to the models—is the benefit of the latent encoder only coming from `c`, or `z` is playing a role too?\n2. This is probably the biggest point: although the paper claims it is comparing against the current SOTA MORL, I believe GPI-LS [2] and MORL/D [3] or even C-MORL [4]—which are the current SOTA—are missing in this. \n3. I think the paper could benefit from a bit more experiments: on different environments than Mujoco—MO-Gymnasium provides some. Of course, a few more seeds would help make sure the results are more robust (currently 5) [1].\n4. Could the author give insights on how much more compute is needed when adding the improvements? A plot of hypervolume vs. walltime comparing several methods would help understanding how computationally heavy the proposed method is vs. the SOTA.\n\n\n## Clarity\nOverall, the paper is well written and well organized. I just have a few remarks here.\n\n1. For the COR loss (line 184): how is this different from Envelope in the end? It might be nice to have the two losses side by side in the appendix to clearly be able to see where they differ.\n2. How did you conduct the hyperparameter search? What was the budget? What was the search space? Optimizing which metric?\n3. To what kind of latent size are you compressing the states (z)? And what is the size of c?\n4. Fig 2: I think `s_t` is missing as input in the bottom model.\n5. Line 125: There is a typo or missing word before “predominantly”.\n6. Lines 154-155 add nothing. I’d remove it.\n7. Fig 3. The y-axis below should be “expected utility.”\n8. Please consider adding the values of the hyperparameters in the appendix for reproducibility.\n9. I like that the authors clearly identify the problems they want to tackle: knowledge sharing (note that [3] also emphasizes on this), and gradient conflicts. \n\n## Significance\nThe proposed improvements seem promising. I would appreciate seeing a comparison vs. GPI and MORL/D, and on different environments than the Mujocos, so we have a better view and understanding of the algorithm’s limits.\n\n## Originality\nI like the idea of solving the gradient conflict and think this is opening up a new way of thinking in MORL. There was Envelope doing this before, but it was not identified as a central problem in the community.\n\n\n\n## Bibliography\n[1] A. Patterson, S. Neumann, M. White, and A. White, “Empirical Design in Reinforcement Learning,” *Journal of Machine Learning Research*, vol. 25, no. 318, pp. 1–63, 2024, [Online]. Available: http://jmlr.org/papers/v25/23-0183.html\n\n[2] L. N. Alegre, A. L. C. Bazzan, D. M. Roijers, and A. Nowé, “Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization,” in *Proc. of the 22nd International Conference on Autonomous Agents and Multiagent Systems*, 2023.\n\n[3] F. Felten, E.-G. Talbi, and G. Danoy, “Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework,” *Journal of Artificial Intelligence Research*, vol. 79, pp. 679–723, Feb. 2024, doi: [10.1613/jair.1.15702](https://doi.org/10.1613/jair.1.15702).\n\n[4] Liu, Ruohong and Pan, Yuxin and Xu, Linjie and Song, Lei and You, Pengcheng and Chen, Yize and Bian, Jiang, “Efficient Discovery of Pareto Front for Multi-Objective Reinforcement Learning”, The Thirteenth International Conference on Learning Representations.", "was_revised": true, "final_justification": "My main concern was the lack of experiments and confidence intervals. The authors provided more results and I am now more convinced that the claims are correct.", "rebuttal": "We appreciate the reviewer's valuable review and constructive comments, and we would like you to know that your questions provide considerably helpful guidance to improve the quality of our paper. \n\nWe will try our best to address each of the concerns and questions raised by the reviewer below:\n\n**1. [Re: How did you conduct the hyperparameter search? What was the budget? What was the search space? Optimizing which metric?]**\n    \nI primarily focus on two core metrics: HV and UT. The evaluation budget is typically set to 3 million steps. The search space consists of several hyperparameters:\n    \n - c ∈ {0.25, 0.0}\n - β ∈ {0.001, 0.01, 0.1, 1.0}\n    \nNotably, c = 0.25 is kept consistent across all experiments.\n    \n**2. [Re: To what kind of latent size are you compressing the states (z)? And what is the size of c?]**\n    \nWe set latent size to 50 for both z and c.\n    \n**3. [Re: Fig 2: I think s_t is missing as input in the bottom model.]**\n    \nThanks to the reviewer for pointing out this oversight. We will address it in the revised version.\n    \n**4. [Re: Please consider adding the values of the hyperparameters in the appendix for reproducibility.]**\n    \nWe provide the detailed hyperparameter settings in the appendix; see Appendix C and Table 2 for more information. **We will also release the code to ensure reproducibility.**\n    \n**5. [Re: I would appreciate seeing a comparison vs. GPI and MORL/D, and on different environments than the Mujocos, so we have a better view and understanding of the algorithm’s limits.]**\n    \nWe thank the reviewer for the valuable suggestion. We conduct evaluations on two continuous action space tasks from MO-Gym. Both sets of experiments are carried out using 100k environment steps.\n    \n|  | GPI/LS | GPI/PD | MORL/D | COLA |\n| --- | --- | --- | --- | --- |\n| Mo-lunar-lander HV(10^8) | 4.37 | 4.66 | 3.84 | 6.42 |\n| UT | 13.52 | 15.12 | 3.71 | 30.18 |\n| Mo-mountain-car HV(10^3) | 1.79 | 3.46 | 1.10 | 3.31 |\n| UT  | -48.57 | -42.47 | -50.23 | -50.42 |\n    \nThese results indicate that COLA exhibits strong or competitive performance on such tasks.\n    \n**6. [Re: Did you run experiments where you condition only with c (not z ) to see if z is adding anything at all? I suspect it should not, since it is essentially an encoded s?]**\n    \nTo address the reviewer's concern, we conduct an ablation study using only c while removing z.\n\nThe experimental results are shown in the table below:\n    \n\n| 2M env steps | Walker-2d | Hopper-3d | Ant-2d | Half-2d |\n| --- | --- | --- | --- | --- |\n| COLA w z | 4806728 \\| 2006 | 29541776029 \\| 3063 | 8722546 \\| 2696 | 7390226 \\| 2766 |\n| COLA w/o z | 4682616 \\|1992 | 29431977661 \\| 3083 | 7974809 \\| 2543 | 7036496 \\| 2656 |\n\nWe observe that incorporating z leads to better performance compared to the variant without *z*. This improvement is primarily due to the fact that the encoder extracts higher-level latent information from the original state *s*, which helps accelerate the approximation of the critic function and thereby improves learning efficiency.\n\n**7. [Re: For the COR loss (line 184): how is this different from Envelope in the end? It might be nice to have the two losses side by side in the appendix to clearly be able to see where they differ.]**\n    \nThe COR loss is not necessarily tied to the Envelope framework and can be applied to any MORL algorithm. It can be directly added to the original value function loss as an auxiliary term.\n    \n**8. [Re: I think the paper could benefit from a bit more experiments: on different environments than Mujoco.]**\n    \nIn addition to the experiments mentioned above, we also conduct further evaluations on two other domains:\n    \n- **Further experiments on the DeepMind Control Suite, using image-based inputs (800.000 environment steps and 5 seeds)**\n        \nThe objectives for these tasks include energy consumption and the original task objectives, such as running speed, walking speed, and jumping height. In this case, we combine COLA with RAD and compared it against Envelope-RAD with 800.000 environment steps. The experimental results are shown as follows:\n        \n| Tasks | Metric | Envelope | COLA |\n| --- | --- | --- | --- |\n| Cheetah_Run-2d | HV (1e4) | 8.61 ± 0.02 | 12.71 ± 0.02 |\n| Cheetah_Run-2d | UT | 318.8 ± 29.25 | 415.98 ± 30.51 |\n| Hopper_Hop-2d | HV (1e2) | 3.47 ± 1.60 | 31.59 ± 0.14 |\n| Hopper_Hop-2d | UT | 122.06 ± 1.93 | 128.53 ± 0.78 |\n| Walker_walk-2d | HV (1e5) | 5.89 ± 3.98 | 7.73 ± 3.14 |\n| Walker_walk-2d | UT | 238.42 ± 84.43 | 311.69 ± 59.90 |\n- **Experiment on the DeepSea Treasure task from Q-Pensieve with 150.000 environment steps and 5 seeds**\n        \n        \n| DeepSea Treasure | Envelope | COLA |\n| --- | --- | --- |\n| HV | 799.9 ± 96.995 | 1028.933 ± 3.365 |\n| UT | 6.323 ± 0.6367 | 7.942 ± 0.07538 |\n        \nThe experiments above demonstrate that COLA can further improve the performance across both HV and UT, whether in image input tasks or other types of tasks.\n        \n\n---\n\nWe hope our replies have addressed the concerns the reviewer posed and shown the improved quality of the paper. **We are always willing to answer any of the reviewer's concerns about our work** and we are looking forward to more inspiring discussions."}]

Author rebuttal:

Meta-review: This work tackles two core challenges in MORL, inefficient knowledge sharing and conflicting gradients. The approach is through a combination of latent dynamics modeling and regularization.

This is the third time I have reviewed this manuscript. I am happy to see the improvement over the submissions to this extent that all reviewers are positive on the submission. I recommend acceptance.
Decision: Accept (poster)