Title: COLA: Towards Efficient Multi-Objective Reinforcement Learning with Conflict Objective Regularization in Latent Space
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Objective Reinforcement Learning
Abstract: Many real-world control problems require continual policy adjustments to balance multiple objectives, which requires the acquisition of high-quality policies to cover diverse preferences. Multi-Objective Reinforcement Learning (MORL) provides a general framework to solve such problems. However, current MORL methods suffer from high sample complexity, primarily due to the neglect of efficient knowledge sharing and conflicts in optimization with different preferences.
To this end, this paper introduces a novel framework, Conflict
Objective Regularization in Latent Space (**COLA**).
To enable efficient knowledge sharing, COLA establishes a shared latent representation space for common knowledge, which can avoid redundant learning under different preferences. Besides, COLA introduces a regularization term for the value function to mitigate the negative effects of conflicting preferences on the value function approximation, thereby improving the accuracy of value estimation. The experimental results across  various multi-objective continuous control tasks demonstrate the significant superiority of COLA over the state-of-the-art MORL baselines. Code is available at https://github.com/yeshenpy/COLA.

COLA: Towards Efficient Multi-Objective Reinforcement Learning with Conflict Objective Regularization in Latent Space

Abstract

Many real-world control problems require continual policy adjustments to balance multiple objectives, which requires the acquisition of high-quality policies to cover diverse preferences. Multi-Objective Reinforcement Learning (MORL) provides a general framework to solve such problems. However, current MORL methods suffer from high sample complexity, primarily due to the neglect of efficient knowledge sharing and conflicts in optimization with different preferences. To this end, this paper introduces a novel framework, Conflict Objective Regularization in Latent Space ( COLA ). To enable efficient knowledge sharing, COLA establishes a shared latent representation space for common knowledge, which can avoid redundant learning under different preferences. Besides, COLA introduces a regularization term for the value function to mitigate the negative effects of conflicting preferences on the value function approximation, thereby improving the accuracy of value estimation. The experimental results across various multi-objective continuous control tasks demonstrate the significant superiority of COLA over the state-of-the-art MORL baselines. Code is available at https://github.com/yeshenpy/COLA .

1 Introduction

Real-world problems often involve multiple performance metrics, where conflicts among these metrics make it infeasible to maximize all metrics simultaneously with a single policy. This necessitates a trade-off among different metrics, requiring the learning of multiple high-quality policies to meet diverse preferences [1-3]. For instance, in robot control problems, performance metrics may include both forward speed and energy consumption. In such cases, two reward signals are returned, and multiple optimal policies exist based on different trade-offs between the two metrics [4]. Learning these policies enables us to control the robot more flexibly [5]. For example, initially, one might prefer a policy that prioritizes speed without considering energy consumption. Still, as energy loss accumulates, a preference may shift towards an energy-efficient policy to extend running time.

A popular approach to addressing the multi-objective control problems involves leveraging MultiObjective Reinforcement Learning (MORL) to acquire policies that meet human preferences [6, 7]. Current MORL algorithms can be roughly categorized into three categories: Single-policy methods, Multi-policy methods, and General policy methods. Single-policy methods directly transform multiobjective problems into single-objective problems and solve them using standard RL [6, 8, 9]. These methods typically require expert knowledge to predefine preferences, and the obtained policies are difficult to generalize to unknown preferences. Multi-policy methods seek a set of policies to cover optimal solutions under diverse preferences, which often require a substantial amount of sample

cost to construct high-quality policies [10-12]. General policy methods aim to construct a single network with preferences as the condition to cover all policies, which have higher sample efficiency compared to the above two categories due to the implicit knowledge sharing [13-15].

However, the general policy methods still require a large number of samples to learn high-quality policies. We summarize two challenges. 1) Inefficiency of Knowledge Sharing : Although using a single network allows implicit knowledge sharing, policies under different preferences need to redundantly learn common knowledge, resulting in inefficient learning. 2) Conflict of Optimization Directions : Different preferences may conflict in optimization directions. When using a shared network to approximate the value of a specific preference, it may compromise the approximation of the values of other preferences, resulting in a tug-of-war during optimization. These conflicts impede the accurate approximation of the value function to distinct preferences, thereby resulting in suboptimal policies.

To address these challenges, we propose a novel framework named Conflict Objective Regularization in Latent Space (COLA). The core insight of COLA is to seek commonalities while accommodating differences , i.e., extracting and sharing the common knowledge required for learning under various preferences, while mitigating interference between different preferences when their optimization directions conflict. COLA comprises two main components: 1) To solve the problem of Inefficiency of Knowledge Sharing , we introduce the Objective-agnostic Latent Dynamics Model (OADM), which utilizes underlying dynamics transitions in the environment to extract a universal knowledge representation through temporal latent consistency. The knowledge extracted by OADM provides more compact general information of dynamics and is essential for learning policies under all preferences. Subsequent multi-objective optimization based on the latent representation constructed by OADM can thus be more efficient. 2) To address the problem of Conflict of Optimization Directions , COLA introduces Conflict Objective Regularization (COR). When the optimization direction of the current preference conflicts with the optimization direction of other preferences, COR constrains the value estimates for other preferences from deviating away from the original value estimates. This minimizes negative impacts (or interference) between optimizations for different preferences, allowing the value function to simultaneously maintain value estimates for multiple preferences. The experiments in a range of multi-objective control tasks show that COLA significantly outperforms current state-of-the-art MORL algorithms across various general metrics, achieving higher sample efficiency and ultimate performance.

2 Background

2.1 Multi-Objective Markov Decision Process

Consider a Multi-Objective Markov Decision Process (MOMDP) [16], defined by a tuple ⟨S , A , P , r , γ, Ω , f Ω ⟩ with state space S , action space A , transition function P ( s, a ) , the vectorvalued reward function r with d objectives, the discount factor γ ∈ (0 , 1) , the preference space Ω and the scalarization function f Ω which produces a scalar utility using preference ω ∈ Ω . In our work, we use a linear preference function, i.e., f ω ( r ( s, a )) = ω ⊤ r ( s, a ) , which is commonly adopted in the MORL works [13, 14, 17]. When the number of objectives is d = 1 , the MOMDP simplifies to a standard MDP. If multiple objectives ( d > 1 ) and all possible returns are considered, a set of Pareto optimal solutions called Pareto Frontier (PF) can be obtained. A policy π is Pareto Optimal when no other policy π ′ can increase its expected return in any objective without decreasing the expected return in any other objective. In continuous control tasks, obtaining the PF is an NP-hard problem [4]. Therefore, the goal of MORL is to obtain an approximation of the PF.

In this paper, we focus on the general policy methods. Therefore, our objective is to construct a general policy π ( a | s, ω ) [18, 19] that, given a state s ∈ S , can generalize across the entire space of preferences by adjusting the preference ω ∈ Ω .

2.2 Envelope Soft Actor Critic

We introduce a foundational MORL algorithm, which integrates Envelope Q-Learning with SAC. We refer to this algorithm as Envelope SAC [14, 20], which serves as the foundational algorithm for our work. Specifically, the multi-objective Bellman equation for Envelope SAC can be formalized

as follows:

where ( s, a, r , s ′ ) is sampled from the replay buffer D , ϕ, θ, ξ (or ¯ ξ ) are the parameters for corresponding networks (or the target network), and 1 d denotes a d -dimensional vector of all ones. To ensure that the samples contribute to the optimization of all preferences, ω is randomly sampled from the preference space Ω rather than using the preferences applied during the interaction. The general policy for different preferences is optimized through the following equation:

where a is sampled from π θ . We follow the structures of SAC, maintaining two Q-functions to alleviate overestimation issues and adjusting α in a learning manner.

3 COLA Framework

This section introduces our framework Conflict Objective Regularization in Latent Space (COLA). We begin by introducing how to construct a latent space by Objective-agnostic Latent Dynamics Model for efficient knowledge sharing. Subsequently, we elaborate on how to mitigate the impact of conflicting preferences on value function approximation through Conflict Objective Regularization. Finally, we provide an overview of the COLA framework.

3.1 Objective-agnostic Latent Dynamics Model

In complex multi-objective continuous control tasks, optimizing a policy or set of policies to cover the entire preference space typically requires a significant sample cost. To enhance sample efficiency, general policy methods employ shared networks and integrate preferences as conditional inputs to facilitate implicit knowledge sharing across various preferences. However, the lack of efficient extraction and utilization of general knowledge limits the efficiency of this learning approach.

To solve the problem, we propose the Objective-agnostic Latent Dynamics Model (OADM) to extract and share the representations of common knowledge in the underlying environment dynamics transitions. Our key insight is that the dynamic transition information underlies robot motion and must be mastered, regardless of the preference policies for robots. By explicitly extracting and sharing the representations, the policies avoid redundant learning of the common knowledge for different preferences, and a more compact and beneficial latent space is constructed, making multi-objective optimization more efficient.

The architectural diagram of OADM is depicted in Figure 1. Specifically, the OADM consists of two components: a state encoder E ψ and a transition model D ψ . We formalize the components as follows:

Encoder :

Transition

:

c t +1 = D ψ ( z t , a t )

In OADM, the encoder maps state s t to the latent representation z t . The transition model takes the latent representation z t and a t as inputs and output c t +1 . To extract the consistent representations in dynamic transitions, OADM leverages the temporal consistency loss to make

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)+GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)+GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

MSE

GLYPH(cmap:d835)

GLYPH(cmap:d835)+GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)+GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

GLYPH(cmap:d835)+GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

MSE

GLYPH(cmap:d835)

GLYPH(cmap:d835)+GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(optional) c t +1 predict the latent representation z t +1 of the next state s t +1 . The loss is defined as follows:

where ( s t , a t , s t +1 ) is sampled from the replay buffer D .

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Objective-a

La

𝑎𝑡

𝒔,𝒂

The temporal consistency loss is used in several previous methods [21-26], which are predominantly tailored for single-objective optimization tasks. In addition to capturing dynamics, these methods inject task-specific information by maximizing the value function or predicting rewards. However, when applied to multi-objective optimization, this approach may introduce potential conflicts, particularly in the optimization of conflicting objectives. Therefore, in COLA, we remove the influence of different preference values and rewards on OADM. This intentional separation allows us to concentrate specifically on providing objective-agnostic state transition information. 𝒔𝒕 𝒛𝒕 𝒔 𝒛 𝒕+𝟏 𝒔 𝒄 𝒕+𝟏 MSE 𝒔𝒕+𝟏 𝒛 𝒕+𝟐 𝒔 𝒄 𝒕+𝟐 MSE 𝒔𝒕+𝟐

For the multi-objective optimization process, the main distinction lies in the optimization being performed in the more compact latent space constructed by OADM, rather than in the original state space. The latent policy optimization process is depicted in Figure 2, which consists of two parts: multi-objective value function approximation and multi-objective policy optimization. Specifically, to optimize more efficiently, the value function takes the latent state representation z t , the latent state-action representation c t +1 , the action a t , and the preference ω as inputs. The original state information s t can be optional. However, in the experiments, we observe that discarding the original information s t sometimes results in unstable policy updates, possibly due to real-time updates of OADM.Therefore, during the learning process,

𝑎𝑡+1

𝒔,𝒂

Objective-agnostic Latent Dynamics Model

𝒔𝒕

𝒛𝒕

𝒔

𝒂𝒕

𝒄 𝒕+𝟏

𝒔,𝒂

𝒘𝒕

𝑸(𝒔𝒕, 𝒛

𝒕

𝒔

, 𝒄

𝒕+𝟏

𝒔,𝒂

, 𝒂

𝐭

, 𝒘

𝐭

)

𝒛𝒕

𝒔

𝒘𝒕

𝝅(𝒔𝒕, 𝒛

𝒕

𝒔

, 𝒘

𝐭

)

Optional

we also take the stable input s t to mitigate the problem. The formulation of the optimization loss L Q can be rewritten as follows:

where y = r ( s t , a t ) + γ V ¯ ξ ( s t +1 , z t +1 , ω ) . The policy π makes decisions by taking s t , z t and ω as inputs. The loss L π of policy is formulated as follows:

With efficient representation and sharing of common knowledge, multi-objective value function and policy optimization can become more efficient.

3.2 Conflict Objective Regularization

The multi-objective value network is crucial for providing policy gradient guidance under various preferences. In current general policy methods, a shared architecture is employed for the value network. Leveraging the shared architecture facilitates task-related knowledge sharing among similar preferences, leading to more efficient learning. However, in multi-objective optimization, numerous conflicting preferences exist. Approximating the value for one preference can have negative impacts on the value estimates for other potentially conflicting preferences. In essence, this is a manifestation of Neural Network Churn [27, 28] in the context of multi-objective optimization. To this end, naturally we aim to study: How can we mitigate the negative impact of conflicting preferences?

To solve the problem, we propose the Conflict Objective Regularization Loss (COR) to mitigate the negative effects of optimization direction conflicts among preferences. When the value function approximates the values of preference ω i , COR aims to minimize the value estimate variation of random preference ω j if the optimization directions of the value function under preferences ω i and ω j conflict. This enhances the stability of value function learning, preventing drastic changes in the estimated values of conflicting preferences, thus preserving effective policy gradient guidance under different preferences.

To implement COR, we first need to establish a metric for the degree of conflict. Concretely, we measure the stiffness ρ ( ω i , ω j ) [29-31] of the value network between preferences ω i and ω j , which

Latent Policy Optimi

Algorithm 1 COLA Algorithm

1: Initialize: a replay buffer D , the Objective-agnostic Latent Dynamics Model with encoder E ψ and transition model D ψ , the MO policy π θ , the MO value function Q ϕ , V ξ .

2: repeat

3: # Interaction for experiences

4: Sample a preference ω i from Ω

5: The MO policy interacts with the environment based on ω i

6: Store the transitions { s, a, s ′ , r, done } to D

7: # Optimize the OADM

8: Optimize OADM with experiences in D {see Eq. 3}

9: # MO Learning in the latent space

10: Optimize the Critic:

11: Sample experiences to calculate the COR loss with ω i and a random sampled ω j {see Eq. 7}

12: Optimize Q to approximate the values under ω i {see Eq. 8}

13: Optimize the Policy: Optimize π under ω i {see Eq. 5}

14: until reaching maximum training steps

is defined by:

Low stiffness indicates that updating the value network parameters toward minimizing L Q for preference ω i will have a negative effect on the minimization of L Q for preference ω j . Meanwhile, a higher stiffness indicates a more consistent optimization direction for the value network parameters between ω i and ω j . We set a conflict boundary, denoted as c . When the stiffness ρ ( ω i , ω j ) is greater than or equal to c , we consider that the optimization directions of the value functions under preferences ω i and ω j are relatively consistent, mutually reinforcing each other. However, when ρ ( ω i , ω j ) is less than c , we assume that there is a conflict in the optimization directions between ω i and ω j . Additionally, a larger value of cρ ( ω i , ω j ) indicates a more severe conflict.

If ρ ( ω i , ω j ) -c < 0 , we introduce the COR loss to the value function under the preference ω j to ensure that the optimization under ω i does not significantly compromise the value estimation under ω j . The COR loss can be formalized as follows:

where the term λ represents the degree of conflict, with a larger value indicating a greater conflict, consequently, the regularization term's constraining effect will be stronger. The term ∆( ω j ) represents the magnitude of the change in Q . Q ′ is the value function of Q optimized based on the preference ω i , i.e., Q ω i -→ Q ′ . Minimizing ∆( ω j ) helps mitigate the impact on the value estimation under ω j . In the practical implementation, obtaining the optimized value function Q ′ for the next step to optimize the current parameters is challenging. This can be addressed by leveraging the MAML method [32]. However, it introduces significant additional computational overhead. To simplify this process, we employ a surrogate approach by maintaining an old Q network from the previous step, denoted as Q old , and using the squared difference between the current Q ( ω j ) and the previous step's Q old ( ω j ) as a replacement for ∆( ω j ) . Finally, the value function loss of COLA is defined as follows:

where β is the hyperparameter to balance the impact of COR loss. Through L COLA , we can reduce the negative impact on the value functions of other preferences when optimizing any preference ω i , thereby enhancing the ability of the value function to maintain values for different preferences.

3.3 COLA Algorithm

We provide the pseudo-code of COLA in Algorithm 1. In each iteration, the algorithm proceeds across three phases (denoted by blue). First, we randomly sample a preference ω i from the preference space Ω and interact with the environment based on π and ω i , the interaction experiences

are then stored into the replay buffer D . Subsequently, we optimize the Objective-agnostic Latent Dynamics Model (OADM) based on Eq. 3, extracting dynamic transition representations and constructing a more compact latent space. Within this latent space, we randomly sample a new preference ω i to optimize the value function and policy. To calculate COR loss, an additional preference ω j is sampled, and the value function is optimized based on our new loss with Eq. 8. Finally, policy improvement is conducted on the policy based on Eq. 5.

4 Experiments

This section provides a comprehensive evaluation of COLA through experiments. We first present the experimental setup, followed by performance comparisons and experimental analysis.

4.1 Experimental Setting

Our experiments are primarily conducted on multi-objective continuous control tasks proposed by PGMORL [4]. Additionally, we construct more complex multi-objective optimization tasks, such as Ant, Hopper, and HalfCheetah, each with more than two objectives. Detailed information can be found in Appendix B. We compare COLA with several state-of-the-art MORL algorithms, including Envelope [14], PGMORL [4], Q-Pensieve [20], and CAPQL [33]. For a fair comparison, we use the official implementation, fine-tune the hyperparameters and present the best results obtained in each

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Hypervolume (×1e

10

)

MO-Ant-3d

COLA

CAPQL

Envelope SAC

Q-Pensieve

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Hypervolume (×1e

6

)

MO-Walker-2d

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

0.2

0.4

0.6

0.8

1.0

Hypervolume (×1e

17

)

MO-HalfCheetah-5d

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

0.0

0.2

0.4

0.6

0.8

1.0

Hypervolume (×1e

17

)

MO-Hopper-5d

0.0

0.5

1.0

1.5

2.0

2.5

Environment Steps (×1e

6

)

1000

1250

1500

1750

2000

2250

2500

2750

Utility

MO-Ant-3d

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

1400

1500

1600

1700

1800

1900

2000

Utility

MO-Walker-2d

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

1600

1800

2000

2200

2400

2600

2800

3000

3200

Utility

MO-HalfCheetah-5d

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

1600

1800

2000

2200

2400

2600

Utility

MO-Hopper-5d

0

1000

2000

3000

4000

Speed Forward

0

500

1000

1500

2000

2500

Energy Efficiency

MO-HalfCheetah-2d

COLA (Ours)

Envelope SAC

PGMORL

CAPQL

Q-Pensieve

0

1000

2000

3000

4000

X Forward Speed

0

1000

2000

3000

4000

Y Forward Speed

MO-Ant-2d

2000

3000

4000

Speed Forward

3000

4000

5000

6000

MO-Hopper-2d

500

1000

1500

2000

1500

2000

2500

Energy Efficiency

MO-Walker-2d

Forward Speed

0

1000

2000

3000

4000

0

1500

3000

4500

Energy Efficiency

500

1000

1500

2000

2500

MO-Hopper-3d

X Forward Speed

0

1000

2000

3000

4000

Y Forward Speed

0

1000

2000

3000

4000

Energy Efficiency

800

1000

1200

1400

1600

1800

2000

MO-Ant-3d

Speed Forward

task. It's worth noting that, as the original Envelope is tailored for discrete tasks, we provide its SAC version for comparison in continuous tasks. All algorithms are trained for 3 million environment steps. All statistics are obtained from 5 independent runs and the mean and standard deviation are reported, which is consistent with the setting in previous MORL works. More implementation details can be found in Appendix C.

Performance Metrics: We follow the performance evaluation metrics proposed in previous methods [20], which focus on the following metrics:

Hypervolume (HV) is a metric designed to quantify the convergence speed of a Pareto frontier and the diversity of policies within it. HV is defined HV := ∫ H ( R ) I { z ∈ H ( R ) } dz , where R is a set of return vectors attained, r 0 ∈ R d is a reference point, H ( R ) := { z ∈ R d : ∃ r ∈ R , r 0 ≺ z ≺ r } and I is the indicator function.

Expected Utility (UT) serves as a metric to evaluate the overall performance of policies under linear scalarization and is defined as [UT := E ω ∈ Ω [ max r ∈R ( ω ⊤ r ) ] , where R is a set of return vectors attained.

Episodic Dominance (ED) is defined as ED 1 , 2 := E ω ∈ Ω [ I { ∑ T 1 t =0 ω ⊤ r 1 t > ∑ T 2 t =0 ω ⊤ r 2 t }] , where r 1 t , r 2 t are return vectors, and T 1 , T 2 are the episode lengths. ED is a metric designed for pairwise comparison of algorithms, particularly suitable for problems where there is a significant difference in returns for different preferences. In such cases, HV and UT may be dominated by a minority of preferences. We calculate ED between algorithms pairwise and report the average ED.

4.2 Performance Evaluation

We evaluate COLA and related strong MORL baselines across ten continuous control tasks. The experimental results shown in Table 1 indicate a significant performance advantage of COLA over other methods. Compared to the second-best approach in different tasks, COLA demonstrates an average improvement of 26 . 8% in HV and 12% in UT. In pairwise comparisons with other algorithms, COLA consistently achieves significantly higher ED than other methods. Additionally, we also observe that as the number of objectives increases, such as in HalfCheetah-5d and Hopper-5d, COLAexhibits more substantial performance improvements compared to other general policy methods. This indicates that COLA exhibits better adaptability to tasks with more objectives. Besides, we present the training process of COLA in comparison to other general policy methods in Figure 3. Notably, COLA demonstrates significant superiority over other algorithms in both HV and UT metrics with an equivalent number of samples, showcasing its higher sample efficiency. Furthermore, we compare the Pareto fronts constructed by different algorithms, which offers a more intuitive representation of the quality of the discovered Pareto fronts. As depicted in Figure 4, COLA constructs a broader and denser Pareto front compared to other algorithms. In the Ant-2d task, COLA's Pareto front is significantly superior to other algorithms. Moreover, in Hopper-3d, COLA almost covers other algorithms. These visualizations consistently highlight the superiority of COLA.

In addition to the main experiments described above, we further evaluate COLA on three additional benchmarks to examine its generality and effectiveness. (1) First, we consider the DeepMind Control Suite (DMC) [34] and extend it to the multi-objective setting, which involves image-based tasks. The objectives of these tasks include energy consumption and the original task goals, such as running speed, walking speed, and jumping height. In this setting, we combined COLA with RAD and compared it against Envelope-RAD with 800,000 environment steps. The results shown in Table 2 indicate that COLA consistently provides significant performance improvements across different tasks. (2) The second benchmark is the continuous-control DeepSea Treasure task proposed in QPensieve [20]. Here, COLA is integrated with SAC and compared against Envelope. As shown in Table 3, COLA achieves better performance on both HV and UT metrics. (3) The third experiment involves two classic tasks from MO-Gym [3], namely Mo-lunar-lander and Mo-mountain-car. As reported in Table 4, we compared against GPI/LS, GPI/PD [35], and MORL/D [36]. The results demonstrate that COLA exhibits a notable performance advantage on Mo-lunar-lander, while on Mo-mountain-car, it shows a slight disadvantage in UT compared to GPI/PD. Overall, COLA demonstrates strong and consistent performance across diverse tasks and algorithmic combinations.

4.3 Superiority of Components & Parameter Analysis

In this section, we delve into the impact of COR loss on value function approximation. Our analysis is conducted on MO-Hopper-3d, encompassing three objectives: forward speed, jump height, and energy efficiency. We gather 10,000 states from the interaction between policies and the environment under diverse preferences, calculate the corresponding Monte Carlo ground truth, and determine

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

50

100

150

200

250

True Error

(1.0, 0.0, 0.0)

COLA w/ COR

COLA w/o COR

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

100

200

300

400

True Error

(0.0, 1.0, 0.0)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

80

100

120

140

160

180

200

220

True Error

(0.0, 0.0, 1.0)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

50

75

100

125

150

175

200

True Error

All preferences

the errors between the estimated values based on these states and the ground truth. The above follows the evaluation method for value approximation accuracy in TD3 [37]. The experimental results illustrated in Figure 5 demonstrate that using COR results in significantly smaller errors compared to not using it for extreme preferences (1 . 0 , 0 . 0 , 0 . 0) and (0 . 0 , 1 . 0 , 0 . 0) , while the errors are comparable for (0 . 0 , 0 . 0 , 1 . 0) . Furthermore, we uniformly sample all possible preferences with a step size of 0 . 25 to calculate the mean errors. The results show that using COR can lead to smaller mean errors in the value function, indicating that COR is more conducive to maintaining value functions with multiple preferences.

Next, we conduct ablation experiments on two components of COLA, i.e., OADM and COR. We conduct ablation experiments on the Ant task with three objectives and the Hopper task with five objectives. We gradually remove two components. The results in Figure 6 demonstrate that removing COR significantly decreases both UT and HV. Further removal of OADM on top of that leads to a further decrease in the performance of both metrics. This indicates the crucial role of the two components in COLA.

Subsequently, we conduct the hyperparameters analysis introduced by COLA, i.e., c and β , where c is used to control the conflict boundary, and β is used to control the weight of COR. For hyperparameters c , we primarily chose values from { 0 . 0 , 0 . 25 } . We conduct the experiments on the Ant-3d task. The results in Figure 7 show that c = 0 . 25 outperforms c = 0 . 0 in terms of both metrics. In experiments, we set c to 0.25 for all tasks. Regarding β , we select values from { 1 . 0 , 0 . 1 , 0 . 01 , 0 . 001 } . We observe that β = 1 . 0 is more efficient, allowing COR to have a greater impact. Thus we set β to 1.0 in most multi-objective tasks. For more details, please refer to Appendix B.2. In addition to the experiments above, we provide further analyses in Appendix D, including studies on the number of sampled preferences and the ablations on s and z , and a more detailed analysis of the parameter c .

5 Related Works

The MORL algorithms can be categorized into three categories. Single-policy methods transform multi-objective problems into single-objective problems by using a scalarization function [6, 8, 9]. These methods typically require predefined preferences and cannot be generalized to other preferences. Multi-policy methods approximate the Pareto front of the optimal solution by constructing a set of policies [10-12, 38-40, 4, 35, 3]. These methods commonly face the challenge of high sample complexity, as they explicitly learn each individual policy. PGMORL [4] employs a model to predict the improvement along with preferences to guide the learning process. General policy methods take preferences as input and aim to cover all policies through a single policy network. CN [13] extends single-objective DQN to multi-objective for approximating Q values under different preferences. Besides, CN introduces Diverse Experience Replay to improve sample efficiency. Envelope Q-Learning [14] introduces envelope Q-functions, enabling efficient knowledge sharing

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

0.5

1.0

1.5

2.0

2.5

Hypervolume (×1e

10

)

MO-Ant-3d

COLA

COLA w\o COR

COLA w\o OADM

COLA w\o COR and OADM

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

0.0

0.2

0.4

0.6

0.8

1.0

Hypervolume (×1e

17

)

MO-HalfCheetah-5d

0.5

1.0

1.5

2.0

2.5

Environment Steps (×1e

6

)

1000

1250

1500

1750

2000

2250

2500

2750

Utility

MO-Ant-3d

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

1500

1750

2000

2250

2500

2750

3000

Utility

MO-HalfCheetah-5d

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Hypervolume (×1e

10

)

MO-Ant-3d

c=0.0

c=0.25

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

1000

1250

1500

1750

2000

2250

2500

2750

Utility

MO-Ant-3d

(a)

c

to control conflict boundary

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Hypervolume (×1e

10

)

MO-Ant-3d

=0.001

=0.01

=0.1

=1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Environment Steps (×1e

6

)

1000

1500

2000

2500

Utility

MO-Ant-3d

(b)

β

to control regular terms

when learning different preference objectives. PD-MORL [15] guides multi-objective training based on the correlation between preference and Q values. In addition, PD-MORL further enhances performance through parallel sampling and Hindsight Experience Replay [41]. Q-PENSIEVE [20] maintains multiple Q values generated during the learning process to efficiently leverage existing knowledge. CAPQL [33] addresses the failure of linear scalarization methods in multi-objective training by adding a concave term to the immediate reward. Moreover, some works have been devoted to developing benchmark toolkits for MORL [3]. However, existing policy methods face challenges associated with Inefficiency of Knowledge Sharing and Conflict of Optimization Directions , limiting sample efficiency and ultimate performance. Thus we propose OADM and COR loss to solve the problems.

6 Conclusion

In this paper, we propose a novel framework, COLA, to more efficiently cover the entire preference space through a single network. We attribute the primary challenges of current general policy methods to the Inefficiency of Knowledge Sharing and Conflict of Optimization Directions . To address these challenges, we first propose the Objective-agnostic Latent Dynamics Model. This model explicitly extracts and shares consistent representations in dynamic transitions, establishing a latent space to accelerate multi-objective learning. Additionally, we propose Conflict Objective Regularization to alleviate negative influences among different preferences in the value approximation process, constructing a more accurate value function. In our experiments, we demonstrate the significant superiority of COLA compared with various strong baselines.

Acknowledgments

This work is supported by the National Key Research and Development Program of China (Grant No. 2024YFE0210900), the National Natural Science Foundation of China (Grant Nos. 624B2101, 62422605, 92370132), and Xiaomi Young Talents Program of Xiaomi Foundation. We would like to thank all the anonymous reviewers for their valuable comments and constructive suggestions, which have greatly improved the quality of this paper.

References

C. Liu, X. Xu, and D. Hu. Multiobjective reinforcement learning: A comprehensive overview. TSMC: Systems , 2014.

C. F. Hayes, R. R˘ adulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. AAMAS , 2022.

F. Felten, L. N. Alegre, A. Nowe, A. LC Bazzan, E. Talbi, G. Danoy, and B. C da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In NeurIPS , 2023.

J. Xu, Y. Tian, P. Ma, D. Rus, S. Sueda, and W. Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In ICML , 2020.

D. M. Roijers and S. Whiteson. Multi-Objective Decision Making . 2017.

D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley. A survey of multi-objective sequential decision-making. CoRR , 2014.

C. F. Hayes, R. Radulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, P. Mannion, A. Nowé, G. de Oliveira Ramos, M. Restelli, P. Vamplew, and D. M. Roijers. A practical guide to multi-objective reinforcement learning and planning. Auton. Agents Multi Agent Syst. , 2022.

K. V. Moffaert, M. M. Drugan, and A. Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In ADPRL . IEEE, 2013.

A. Abdolmaleki, S. H. Huang, L. Hasenclever, M. Neunert, H. F. Song, M. Zambelli, M. F. Martins, N. Heess, R. Hadsell, and M. A. Riedmiller. A distributional view on multi-objective policy optimization. In ICML , 2020.

D. M. Roijers, S. Whiteson, and F. A. Oliehoek. Linear support for multi-objective coordination graphs. In AAMAS , 2014.

H. Mossalam, Y. M. Assael, D. M. Roijers, and S. Whiteson. Multi-objective deep reinforcement learning. CoRR , 2016.

M. Zuluaga, A. Krause, and M. Püschel. e-pal: An active learning approach to the multiobjective optimization problem. 2016.

A. Abels, D. M. Roijers, T. Lenaerts, A. Nowé, and D. Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In ICML , 2019.

R. Yang, X. Sun, and K. Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In NeurIPS , 2019.

T. Basaklar, S. Gumussoy, and Ü. Y. Ogras. PD-MORL: preference-driven multi-objective reinforcement learning algorithm. In ICLR , 2023.

K. Chatterjee, R. Majumdar, and T.A. Henzinger. Markov decision processes with multiple objectives. In STACS , 2006.

P. Kyriakis, J. Deshmukh, and P. Bogdan. Pareto policy adaptation. In ICLR , 2022.

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In ICML , 2015.

H. Tang, Z. Meng, J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Y. Yang, W. Tao, and L. Wang. What about inputting policy in value function: Policy representation and policyextended value function approximator. In AAAI , 2022.

W. Hung, B. Huang, P. Hsieh, and X. Liu. Q-pensieve: Boosting sample efficiency of multiobjective RL through memory sharing of q-snapshots. In ICLR . OpenReview.net, 2023.

M. Schwarzer, A. Anand, R. Goel, R. Devon Hjelm, A. C. Courville, and P. Bachman. Dataefficient reinforcement learning with self-predictive representations. In ICLR , 2021.

N. Hansen, H. Su, and X. Wang. Temporal difference learning for model predictive control. In ICML , 2022.

T. McInroe, L. Schäfer, and S. V. Albrecht. Learning temporally-consistent representations for data-efficient reinforcement learning. CoRR , 2021.

N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint , 2023.

Y. Yuan, J. Hao, F. Ni, Y. Mu, Y. Zheng, Y. Hu, J. Liu, Y. Chen, and C. Fan. Euclid: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model. ICLR , 2023.

Y. Zhao, W. Zhao, R. Boney, J. Kannala, and J. Pajarinen. Simplified temporal consistency reinforcement learning. In ICML , 2023.

T. Schaul, A. Barreto, J. Quan, and G. Ostrovski. The phenomenon of policy churn. arXiv preprint , arXiv:2206.00730, 2022.

Hongyao Tang and Glen Berseth. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. In NeurIPS , 2024.

S. Fort, P. K. Nowak, and S. Narayanan. Stiffness: A new perspective on generalization in neural networks. CoRR , 2019.

S. Moon, J. Lee, and H. Oh Song. Rethinking value function learning for generalization in reinforcement learning. In NeurIPS , 2022.

E. Bengio, J. Pineau, and D. Precup. Interference and generalization in temporal difference learning. In ICML , 2020.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

H. Lu, D. Herman, and Y. Yu. Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality. In ICLR , 2023.

Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

L.N. Alegre, A. L. Bazzan, D. M. Roijers, A. Nowé, and B. C. da Silva. Sample-efficient multi-objective learning via generalized policy improvement prioritization. AAMAS , 2023.

F. Felten, E. Talbi, and G. Danoy. Multi-objective reinforcement learning based on decomposition: A taxonomy and framework. JAIR , 2024.

S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML , 2018.

S. Natarajan and P. Tadepalli. Dynamic preferences in multi-criteria reinforcement learning. In ICML , 2005.

M. Pirotta, S. Parisi, and M. Restelli. Multi-objective reinforcement learning with continuous pareto frontier approximation. In AAAI , 2015.

S. Parisi, M. Pirotta, and J. Peters. Manifold-based multi-objective policy search with sample reuse. Neurocomputing , 2017.

M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In NeurIPS , 2017.

S. Khadka and K. Tumer. Evolution-guided policy gradient in reinforcement learning. In NeurIPS , 2018.

A. Pourchot and O. Sigaud. CEM-RL: combining evolutionary and gradient-based methods for policy search. In ICLR , 2019.

J. Hao, P. Li, H. Tang, Y . Zheng, X. Fu, and Z. Meng. Erl-re 2 : Efficient evolutionary reinforcement learning with shared state representation and individual policy representation. In ICLR , 2023.

P. Li, J. Hao, H. Tang, Y. Zheng, and F. Barez. Value-evolutionary-based reinforcement learning. In ICML , 2024.

P. Li, Y . Zheng, H. Tang, X. Fu, and J. Hao. Evorainbow: Combining improvements in evolutionary reinforcement learning for policy search. In ICML , 2024.

S. Majumdar, S. Khadka, S. Miret, S. McAleer, and K. Tumer. Evolutionary reinforcement learning for sample-efficient multiagent coordination. In ICML , 2020.

P. Li, J. Hao, H. Tang, Y . Zheng, and X. Fu. Race: Improve multi-agent reinforcement learning with representation asymmetry and collaborative evolution. In ICML , 2023.

Y. Zheng, X. Xie, T. Su, L. Ma, J. Hao, Z. Meng, Y. Liu, R. Shen, Y. Chen, and C. Fan. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In ASE . IEEE, 2019.

P. Li, J. Hao, H. Tang, X. Fu, Y. Zheng, and K. Tang. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey. IEEE TEVC , 2024.

A Limitations & Future Work

For limitations and future work, firstly our work is empirical proof of the effectiveness of the COLA idea and we provide no theory on optimality, convergence, and complexity. Second, one critical point is the expressivity of the representation method, which is not optimally addressed by this work. Thirdly, our work currently focuses solely on linear utility functions. Therefore, the theoretical foundation, improved representation methods, and the incorporation of nonlinear utility functions are all worthy of further investigation and exploration, which we leave for future work.

In addition, in our previous analysis, we observed a certain degree of policy degradation and forgetting, where behaviors learned in the early stages could not be retained. However, such phenomena have not been observed in population-based multi-objective algorithms. Recently, the integration of evolutionary algorithms (EAs) and RL has been proven effective in single-objective optimization tasks [42-49], but its framework for multi-objective optimization remains underexplored. We believe that exploring this direction, which emphasizes collaborative optimization and the mutual enhancement between RL and EAs [50], can better address the current challenges faced by MORL. We regard this as a key direction for future research in MORL and plan to further explore it based on the COLA framework.

B Environment Setup

We conduct performance tests in multi-objective MUJOCO tasks. Below are the specific configuration details for each task.

MO-HalfCheetah-2d: Observation and action space dimensionality: S ∈ R 17 , A ∈ R 6 , and the environment runs for 500 steps. The first objective is forward speed: R 1 = v x + C . The second objective is control cost: R 2 = 4 -∑ i a 2 i + C . C = 1 is the alive bonus, v x is the speed in x direction, a i is the action of each actuator

MO-Hopper-2d: Observation and action space dimensionality: S ∈ R 11 , A ∈ R 3 , and the environment runs for 500 steps. The first objective is forward speed: R 1 = 1 . 5 v x + C . The second objective is jumping height: R 2 = 12 × ( h -h init ) + C . C = 1 -0 . 0002 × ∑ i a 2 i is composed of alive bonus and energy efficiency, v x is the speed in x direction, h is the current height, h init is the initial height, a i is the action of each actuator.

MO-Walker-2d: Observation and action space dimensionality: S ∈ R 17 , A ∈ R 6 , and the environment runs for 500 steps. The first objective is forward speed: R 1 = v x + C . The second objective is energy efficiency: R 2 = 4 -∑ i a 2 i + C . C = 1 is the alive bonus, v x is the speed in x direction, a i is the action of each actuator.

MO-Ant-2d: Observation and action space dimensionality: S ∈ R 27 , A ∈ R 8 , and the environment runs for 500 steps. The first objective is x -axis speed: R 1 = v x + C . The second objective is y-axis speed: R 2 = v y + C . C = 1 -0 . 5 × ∑ i a 2 i is composed of alive bonus and energy efficiency, v x is x-axis speed, v y is y-axis speed, a i is the action of each actuator.

MO-Hopper-3d: Observation and action space dimensionality: S ∈ R 11 , A ∈ R 3 , and the environment runs for 500 steps. The first objective is forward speed: R 1 = 1 . 5 v x + C . The second objective is jumping height: R 2 = 12 × ( h -h init ) + C . The third objective is energy efficiency: R 3 = 4 -∑ i a 2 i + C . C = 1 is the alive bonus, v x is the speed in x direction, h is the current height, h init is the initial height, a i is the action of each actuator.

MO-Ant-3d: Observation and action space dimensionality: S ∈ R 27 , A ∈ R 8 , and the environment runs for 500 steps. The first objective is x -axis speed: R 1 = v x . The second objective is y-axis speed: R 2 = v y . The third objective is energy efficiency: R 3 = 4 -0 . 5 × ∑ i a 2 i . C = 1 -0 . 5 × ∑ i a 2 i is composed of alive bonus and energy efficiency, v x is x-axis speed, v y is y-axis speed, a i is the action of each actuator.

MO-Ant-4d: Observation and action space dimensionality: S ∈ R 27 , A ∈ R 8 , and the environment runs for 500 steps. Each objective is energy efficiency of one leg and x -axis speed: R { 1 , 2 , 3 , 4 } = 4 . 0 -2 × ( a 2 i, 0 + a 2 i, 1 ) + v x . a i, 0 and a i, 1 are the action of each actuator on the ith leg. The objective of this task is to enable the agent to maximize velocity

while simultaneously controlling the energy cost of different legs, aiming to achieve precise control of each leg.

MO-HalfCheetah-5d: Observation and action space dimensionality: S ∈ R 17 , A ∈ R 6 , and the environment runs for 500 steps. The first objective is forward speed: R 1 = 0 . 5 × v x + C . The second objective is jumping height: R 2 = 20 × ( h -h init ) + C . The third objective is the control cost of the front leg: R 3 = 4 -1 . 5 × ( a 2 f, 0 + a 2 f, 1 ) + C . The fourth objective is the control cost of the torso: R 4 = 4 -1 . 5 × ( a 2 t, 0 + a 2 t, 1 ) + C . The fifth objective is the control cost of the hind leg: R 5 = 4 -1 . 5 × ( a 2 h, 0 + a 2 h, 1 ) + C . C = 1 is the alive bonus, v x is the speed in x direction, a f, 0 , a f, 1 , a t, 0 , a t, 1 , a h, 0 and a h, 1 are the actions of different actuators.

MO-Hopper-5d: Observation and action space dimensionality: S ∈ R 11 , A ∈ R 3 , and the environment runs for 500 steps. The first objective is forward speed: R 1 = 1 . 5 × v x + C . The second objective is jumping height: R 2 = 12 × ( h -h init ) + C . The third objective is the energy efficiency of the first actuator: R 3 = 4 -3 × a 2 0 + C . The fourth objective is the energy efficiency of the second actuator: R 4 = 4 -3 × a 2 1 + C . The fifth objective is the energy efficiency of the third actuator: R 5 = 4 -3 × a 2 2 + C . C = 1 is the alive bonus, v x is the speed in x direction, h is the current height, h init is the initial height, a i is the action of each actuator.

MO-Ant-5d: Observation and action space dimensionality: S ∈ R 27 , A ∈ R 8 , and the environment runs for 500 steps. The first objective is x -axis speed: R 1 = v x . The second to the fifth objectives are the energy efficiency of different legs: R { 2 , 3 , 4 , 5 } = 4 . 0 -2 × ( a 2 i, 0 + a 2 i, 1 ) + v x . a i, 0 and a i, 1 are the actions of each actuator on the ith leg. This task is similar to MO-Ant-4d, but the key difference lies in the desire to achieve precise energy consumption control for different legs under varying speeds.

B.1 Experiment Details

During training, we predefined a preference set to record HV and UT every 50000 environment steps. These preferences are sampled at regular intervals in the preference space, covering all possible preferences. For two-objective tasks, the preference interval is set to 0.005. For three-objective tasks, the preference interval is set to 0.05. For four-objective tasks, the preference interval is set to 0.1. For five-objective tasks, the preference interval is set to 0.2. Then we use these preferences as conditions for the general policy methods, collecting return vectors for each preference. Based on these vectors, we calculate HV and UT. For algorithms like PGMORL which are multi-policy methods, we select the best policy for each preference to calculate HV and UT. To visualize the Pareto frontier, PGMORL directly plots the discovered set of policies. In contrast, for general policy methods, we set small intervals to explore their potential as much as possible. For two-objective tasks, a preference interval of 0.0001 is used, and for three-objective tasks, a preference interval of 0.002 is employed. All experiments are run on an Intel Xeon E5-2680 v4 @ 2.40GHz machine with an NVIDIA GTX 2080 Ti. Except for the DMC experiments, all other experiments are conducted using only the CPU.

B.2 Experiments Compute Resources

All experiments are carried out on CPU E5-2680 v4 @ 2.40GHz with 256GB memory

C Implementation Details

We introduce the implementation details of COLA in this section:

The baselines we compared include PGMORL 1 , Envelope SAC, CAPQL 2 , and QPensieve 3 . We reproduce the results based on their official implementations, fine-tuning the hyperparameters involved in each algorithm to achieve the best performance results.

We built our code following the official implementation of Q-Pensieve, with the exception that the policy network has layer sizes of { 128 , 128 } while keeping the rest of the architecture and hyperparameters consistent.

For the Objective-agnostic Latent Dynamics Model, a neural network with two layers, each having 256 units, is constructed. It utilizes the ReLU activation function, and the output dimension is set to 50.

We set c = 0 . 25 for all tasks. Besides, we list the hyperparameters specific to COLA which varied across tasks in Table 5. if you don't want to tune hyperparameters , competitive performance can be obtained when keeping c = 0 . 25 and β = 1 . 0 in most tasks.

D Additional Experiments

Why do we need to retain s as an input? This is primarily based on empirical observations. We find that omitting s causes performance fluctuations in some tasks, such as Hopper-2d. The HV and UTresults are shown in Table 6: We observe that excluding s leads to performance drops in learning process. Moreover, using s as input is more efficient.

Is sampling a single preference at each iteration sufficient, or should multiple preferences be sampled instead? To answer this question, we follow CAPQL and sample a batch of preferences at each step. The results are shown in Table 7. Overall, we find that sampling a single preference yields better performance in most tasks. This aligns with recent findings suggesting that multiple training iterations on the same batch lead to improved outcomes. Fully learning under one preference before switching to another appears more beneficial for training.

Ablation study on z . We conduct an ablation study using only c while removing z . The experimental results are shown in Table 8. We observe that incorporating z leads to better performance compared to the variant without z . This improvement is primarily due to the fact that the encoder

extracts higher-level latent information from the original state s , which helps accelerate the approximation of the critic function and thereby improves learning efficiency.

Analysis of the parameter on c . In all experiments, c is set to 0.25. A smaller value of c indicates that the COR loss has less influence, whereas c = 1 . 0 means that COR is always active, regardless of whether conflicts are present or not. We initially conduct a preliminary comparison between c = 0 . 0 and c = 0 . 25 on a single case and find that c = 0 . 25 yields the best performance. Based on this observation, we adopt c = 0 . 25 consistently across all experiments. We also conduct an ablation study on the effect of c , as shown below in Table 9. The results indicate that c = 0 . 25 generally delivers strong performance across different tasks.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This paper focuses on further improving the performance of MORL from two perspectives: knowledge sharing and value function approximation, and provides extensive experimental evidence to demonstrate the effectiveness of the proposed method. The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a discussion of the limitations in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This study focuses on empirical evaluation and does not provide theoretical proofs or formal analysis. We leave the development of theoretical guarantees and formal analysis as future work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed reproduction details in the appendix and will release the code upon public release.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We commit to releasing the code upon the public availability of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the detailed experimental settings in both the main experiment section and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Following the experimental settings of prior work, we conduct each experiment with 5 independent runs and report error bars to reflect statistically significant variability.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: These details are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research conducted in this paper fully conforms with the NeurIPS Code of Ethics in all respects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of MORL. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, all creators and original owners of the assets used in the paper are properly credited and fully respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel MORL algorithm which provides two big improvements: knowledge sharing and resolution of conflicting gradients. The first improvement is done by sharing knowledge for multiple preferences under a single (conditioned) network and learning a latent model for the dynamics of the environment—this latent code is used as an additional input for both actor and critic. The second improvement is done by adding a loss term (regularization) which prevents the network from forgetting other preferences when updating for a given preference. Some experiments are run on MuJoCo-based environments, tested against some MORL algorithms, and showing the proposed method does well in terms of hypervolume, expected utility, and episodic dominance. There is also an ablation study showing that the two proposed improvements are actually helpful in terms of performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Quality \nThe paper seems technically sound. I have a few comments that I believe would improve the overall quality of the paper.\n1. I like the idea of the gradient resolution. However, I’m missing a bit more intuition on why we need the latent encodings at all, as in the end the raw states are used as input to the models—is the benefit of the latent encoder only coming from `c`, or `z` is playing a role too?\n2. This is probably the biggest point: although the paper claims it is comparing against the current SOTA MORL, I believe GPI-LS [2] and MORL/D [3] or even C-MORL [4]—which are the current SOTA—are missing in this. \n3. I think the paper could benefit from a bit more experiments: on different environments than Mujoco—MO-Gymnasium provides some. Of course, a few more seeds would help make sure the results are more robust (currently 5) [1].\n4. Could the author give insights on how much more compute is needed when adding the improvements? A plot of hypervolume vs. walltime comparing several methods would help understanding how computationally heavy the proposed method is vs. the SOTA.\n\n\n## Clarity\nOverall, the paper is well written and well organized. I just have a few remarks here.\n\n1. For the COR loss (line 184): how is this different from Envelope in the end? It might be nice to have the two losses side by side in the appendix to clearly be able to see where they differ.\n2. How did you conduct the hyperparameter search? What was the budget? What was the search space? Optimizing which metric?\n3. To what kind of latent size are you compressing the states (z)? And what is the size of c?\n4. Fig 2: I think `s_t` is missing as input in the bottom model.\n5. Line 125: There is a typo or missing word before “predominantly”.\n6. Lines 154-155 add nothing. I’d remove it.\n7. Fig 3. The y-axis below should be “expected utility.”\n8. Please consider adding the values of the hyperparameters in the appendix for reproducibility.\n9. I like that the authors clearly identify the problems they want to tackle: knowledge sharing (note that [3] also emphasizes on this), and gradient conflicts. \n\n## Significance\nThe proposed improvements seem promising. I would appreciate seeing a comparison vs. GPI and MORL/D, and on different environments than the Mujocos, so we have a better view and understanding of the algorithm’s limits.\n\n## Originality\nI like the idea of solving the gradient conflict and think this is opening up a new way of thinking in MORL. There was Envelope doing this before, but it was not identified as a central problem in the community.\n\n\n\n## Bibliography\n[1] A. Patterson, S. Neumann, M. White, and A. White, “Empirical Design in Reinforcement Learning,” *Journal of Machine Learning Research*, vol. 25, no. 318, pp. 1–63, 2024, [Online]. Available: http://jmlr.org/papers/v25/23-0183.html\n\n[2] L. N. Alegre, A. L. C. Bazzan, D. M. Roijers, and A. Nowé, “Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization,” in *Proc. of the 22nd International Conference on Autonomous Agents and Multiagent Systems*, 2023.\n\n[3] F. Felten, E.-G. Talbi, and G. Danoy, “Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework,” *Journal of Artificial Intelligence Research*, vol. 79, pp. 679–723, Feb. 2024, doi: [10.1613/jair.1.15702](https://doi.org/10.1613/jair.1.15702).\n\n[4] Liu, Ruohong and Pan, Yuxin and Xu, Linjie and Song, Lei and You, Pengcheng and Chen, Yize and Bian, Jiang, “Efficient Discovery of Pareto Front for Multi-Objective Reinforcement Learning”, The Thirteenth International Conference on Learning Representations.\n\nQUESTIONS:\n- Why limitting the analysis only to Mujocos? Why not using the other MO-Gymnasium problems?\n- Why not comparing the SOTA algorithms: GPI, MORL/D or CMORL?\n- How was the hyperparameter search done?\n- Did you run experiments where you condition only with `c` (not `z` ) to see if z is adding anything at all? I suspect it should not, since it is essentially an encoded `s`?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an empirical method to perform general policy multiobjective reinforcement learning, that is, the authors train a network that is able to output a policy for any arbitrary linear set of weights. The main novelties introduced by this paper focus on the fact that general policy methods are trained by sampling random sets of weights and sequentially training using them, which often results in conficts between the learning objectives. The authors propose to (1) learn the state transition function separately from the reward function, since the transition function is independent from the reward preferences this transfer smoothly across different training iterations; and (2) introduce a regularization term taking into account how much attrition there is with the previous set of weights.\n\nSTRENGTHS AND WEAKNESSES:\n------\nStrenghts\n\n- The contributions from the paper are simple to implement and very logical\n- There is prospect of relatively high impact in a subarea of RL, and the proposal is widely applicable across pretty much any multiobjective problem.\n- Paper is very clear and well-written.\n- The proposal is a clear winner in performance in the experimental evaluation performed (which is limited in some ways, see weaknesses section)\n-----------\nWeaknesses\n\n- For some specific parts of the paper I have the feeling there wasn't much thought devoted to make the approach very general and applicable across other scenarios beyond the evaluated. IN special, I have the feeling the regularization proposed (Eq 7) might not make sense depending on the sequence of set of weights that are presented during training. The regularization only takes the current and the previous set of weights, so if the previous set of weights is very similar to the current one but they are extremely different from the previous 10 sets, there will probably be the \"tug of war\" effect the authors were trying to avoid and would very likely result in a reduction of performace. There is no comment regarding this in the paper nor an ablation experiment exploring how different regimes for line 4 of the algorithm (when the random set of weights is chosen) would affect performance (Fig 5 shows only uniform sampling and perhaps not even shown in the best way).\n\n- Also related, all the evaluations presented are done in Mujoco tasks, which effectively means there is just a single evaluation domain, it would be good to have some other different domains included to make sure the gain of performance is not only due artifacts of this domain.\n\nQUESTIONS:\n- What are the assumptions behind the schedule for the set of weights using in the training so that the regularization proposed works as expexted?\n\n- where is PGMORL's results in Figure 3?\n\n- It's not clear how the pareto front results in Figure 4 are built. Are those results for the fully-trained policies queried for different sets of preferences only after the training is done, or are those results for policies during training? same question for the numeric results provided in Table 1.\n\n- For halfcheetah it seems like CAPQL has discovered some policies that way more efficient for speed forward values of around 2000. This could result in significantly superior policies depending on the scenario (if the application mandates power efficiency should be >=2000, for example, CAPQL's policy would be better. This goes against all other resuts presented, I would expect some explanation of in which conditions this behavior could arise, because it could affect where the new proposed method is better or not.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper proposes a multi-objective reinforcement learning framework called COLA. It introduces a shared base module to model states and state transitions in latent space, facilitating policy learning across different objectives. Additionally, COLA employs a regularization term on the Q-function to manage conflicts between objectives. Experimental results on multiple multi-objective tasks demonstrate COLA's superior performance compared to baseline methods.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper is clearly written, effectively presenting most methodological details.
2. The methodology is natural and concise, facilitating reproduction and enabling further comparative research.
3. Experimental results on several tasks are promising, clearly showing COLA’s superior performance compared to multiple baselines.

Weaknesses:

1. The rationale for incorporating the latent state-action representation into the inputs of the value function is unclear. From Eq. (3), it appears that the function $D(a, z)$ estimates the expectation of latent representation of the subsequent state resulting from action $a$. However, the benefit and necessity of including this representation in the Q-function inputs remain unclear and intuitively unconvincing for me.
2. The design of the regularization term seems overly simplistic. Regularizing by sampling a single random preference within each iteration may be inefficient, potentially leading to instability or convergence issues.
3. Methodological novelty is somewhat limited. Both latent space modeling of states and state-action transitions, as well as regularization approaches for managing conflicting objectives, are common designs in ML literature.

QUESTIONS:
No more questions.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes COLA, a novel framework for Multi-Objective Reinforcement Learning (MORL) that combines two key components: an Objective-Agnostic Latent Dynamics Model (OADM) to enable efficient knowledge sharing across objectives, and Conflict Objective Regularization (COR) to mitigate optimization conflicts in the latent space. Experiments on 10 MuJoCo tasks show notable gains over state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method tackles two core challenges in MORL—inefficient knowledge sharing and conflicting gradients—through a thoughtful combination of latent dynamics modeling (OADM) and regularization (COR). The guiding idea of “seeking commonalities while accommodating differences” is intuitive and well-motivated.\n2. The paper is well-written and easy to understand.\n\nWeaknesses:\n1. The threshold choice for detecting conflicting preferences is empirically set without theoretical justification. Although the sensitivity analysis (Fig. 7) suggests this value works better than c = 0, its generality across tasks is unclear.\n2. The computational cost of COR is not analyzed. Training time and resource usage are not compared to baselines, leaving performance-efficiency trade-offs ambiguous.\n\nQUESTIONS:\n1. What is the per-iteration training overhead introduced by COR compared to baselines?\n2. Is there a theoretical motivation for choosing c = 0.25? Could it be task-specific? Would an adaptive or learned threshold perform better?\n3. How does the computational complexity of COR scale with the number of objectives or preference dimensions?\n4. The paper attributes instability from discarding raw states s_t to “real-time updates of OADM” (p.4). Could you provide a deeper analysis or diagnostics to clarify this issue?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}