Customizing Reinforcement Learning Agent with Multi-Objective Preference Control

Authors: Yucheng Yang, Tianyi Zhou, Meng Fang, Mykola Pechenizkiy
Primary area: reinforcement learning
Keywords: reinforcement learning, multi-objective optimization, deep reinforcement learning

Abstract:
Practical reinforcement learning (RL) usually requires agents to be optimized for multiple potentially conflicting criteria, e.g. speed vs. safety. 
Although Multi-Objective RL (MORL) algorithms have been studied in previous works, their trained agents often lack precise controllability of the delicate trade-off among multiple objectives. Hence, the resulting agent is not versatile in aligning with customized requests from different users. 
To bridge the gap, we develop ``Preference control (PC) RL'', which aims to train a meta-policy that takes user preference as input controlling the generation of a trajectory on the Pareto frontier adhering to the preference. To this end, we train a preference-conditioned meta-policy by our proposed preference-regularized MORL algorithm. The achieved meta-policy performs as a multi-objective optimizer that can produce user-desired solutions on the Pareto frontier. The proposed algorithm is analyzed and its convergence and controllability are theoretically justified. 
Experiments from discrete toy examples to higher-dimension robotic control tasks and experiments with more than two objectives are conducted to show its performance.  In these experiments, PCRL-trained policies show significantly better controllability than existing approaches and can generate Pareto optimal solutions with better diversity and utilities.

Decision: reject
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729724828832

thread_type:
official_review

<thread_object>
creator: reviewer_tib3

<content>
summary:
This paper studies multi-objective reinforcement learning (MORL) algorithms that can be controlled to produce policies with different trade-offs between conflicting objectives. In particular, the paper tackles the case in which the policy is conditioned on a preference vector $p$ that controls the desired trade-offs. The paper introduces a novel method, named Preference Control RL (PreCo), which learns a policy that encourages the preference vector $p$ to have high similarity with the value vector of the corresponding policy. Experiments were performed in discrete toy domains as well as in multi-objective versions of the Mujoco benchmark to assess the properties of the method proposed.

strengths:
- The problem of designing MORL algorithms with better controllability is of high relevance in the RL community, and the idea of using a similarity function to achieve that is novel to the best of my knowledge.
- The experiments consider many different MORL problems with diverse characteristics and complexities.

weaknesses:
- The paper has a significant problem of clarity, to the point of being very difficult to follow what is being proposed in terms of theory, mathematical notation, and algorithms. I discuss this in my Questions below.
- The experiments do not provide confidence intervals or information regarding the number of random seeds used to compute the metrics. Hence, it is not possible to assess whether the results are not due by chance.

questions:
Below, I have some questions and constructive feedback to the authors:

1) “... Alegre et al. (2023) require learning multiple models to identify the Pareto front” 
This is incorrect. Alegre et al. (2023) introduce a version of their method that uses a single policy conditioned on the preferences $w$, $\pi(s|w)$. This is also done in many state-of-the-art MORL algorithms. 

2) The authors claim to learn a “meta-policy”. However, I do not think “meta-policy” is the appropriate term since the policy learned outputs actions as defined in the original action space of the MDP. In RL, meta-policies are policies that control a standard RL policy by learning in the space of meta-actions that are different than the regular actions.

3) In Figure 1, the PFs are “convex” instead of “concave”. That is, linear scalarization is only able to reach points in the convex part of the PF. The paper is confusing both terms.

4) The idea of training a policy conditioned on agent preferences is well-studied and applied in the MORL literature. The authors are proposing a different representation of the preferences. However, the paper does not discuss that previous MORL algorithms also can control MORL policies via a vector representation.

5) “we sample $p\in\mathcal{P}$ uniformly”. What is the domain of $p$? How do you sample it uniformly from this space?

6) How is the value of $\lambda$ selected in Equations 6 and 7?

7) What does it mean to solve the min-norm problem in the third step at “policy level” vs. “parameter level”? This is not clear.

8) In Definition 4.1, the preferences $p$ are vectors such that their elements sum to 1. However, $v$ does not have this constraint and can have very different magnitudes in its elements. Hence, if the maximum element of $v$ has a value of $10$, this value could always be selected in the max in Eq. 8, and the preference vector would be pushed towards $v$. Is this correct?

9) “$\Pi_{W}$ means the projection to the set of convex coefficients.” It is not clear what does this mean. Please provide a more detailed explanation. What are the coefficients? How is this projection computed?

10) Algorithm 1 and its explanation are very difficult to follow. What is the data collected in line 230? What are the variables $\zeta$? 

11) Equations 9 and 10 require some intuitive explanation. It is not clear how $G$ is computed during the RL learning agent training. For instance, the expected value in Eq. 9 is w.r.t what distribution?

12) Regarding Remark 4.1, how is the value of $\lambda$ increased?

13) In Table 1, how many random seeds were used to compute these metrics? The authors should also provide confidence intervals or dispersions metrics. It is not possible to infer whether the results are due to chance.

14) In Figure 4, why did linear scalarization only achieve a single point? This is very strange since previous MORL algorithms based on linear scalarization have been employed in this problem and have been able to identify many points in the PF.

15) The claim that LS agents are uncontrollable by $p$ is not true given that many previous works have proposed MORL agents that generate different Pareto-optimal solutions conditioned on a preference vector $w$.

16) Based on the results from Figures 5 and 6, and given that no confidence intervals were provided, it is not possible to infer that PreCo results in better Hypervolume than the competitors.

17) The paper has a considerable number of grammar issues. I suggest the authors to carefully review the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730145009947

thread_type:
official_review

<thread_object>
creator: reviewer_nupd

<content>
summary:
The paper provides a method for optimizing multi-objective RL problems such that the obtained policy lies on the Pareto frontier. Instead of relying on scalarization of the objective, like standard MORL methods do, the proposed method instead optimized for a similarity metric between the preference vector and the value function. The paper provides a theoretical convergence analysis and compares with a number of adequate baselines on multiple environments.

strengths:
The paper provides a novel and reasonable approach for finding pareto-optimal policies. This is non-trivial, most existing methods rely on scalarization and fall short, or require costly and complex constrained optimization methods. This paper instead provides a simpler updating scheme applicable in both discrete and continuous action-space MDPs. Good, adequate baseline comparison.

weaknesses:
Clarity: Method section is hard to follow. In order to understand fully why the method converges to pareto-optimal policies, I'd appreciate if the authors could provide additional intuition (one or two sentences) for step 3 in section 3.2.

Experiments & reproducibility: While the paper provides a number of experiments, it does not seem like results were averaged across multiple random seeds, maybe I missed it? This is crucial though, because RL methods depend strongly on the seed. I strongly feel like the authors should report statistics across multiple seeds, otherwise it is very hard to say if the results are reproducible and significant.

Meta-RL: It is not clear to me why the authors learn a preference conditioned meta policy. Is this a requirement of the learning algorithm or is this solely done to have access to multiple policies at inference time? Does the method rely on training with uniformly sampled preference vectors or can it also optimize and find the pareto-optimal policy for a single preference, without uniformly sampling from the preference space during training at random?

questions:
See the points under weaknesses. I think this is a good approach but also believe that the current version of the manuscript has some limitations that need to accounted for (or the clarity needs to improved) to justify a higher score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730442219171

thread_type:
official_review

<thread_object>
creator: reviewer_nkse

<content>
summary:
This paper presents Multi-objective Preference Control RL, an approach for multi-objective RL, which trains a meta-policy that takes user preference as input controlling the generated trajectories within the preference region on the Pareto frontier. They show that their meta-policy performs as
a multi-objective optimizer that can directly generate user-desired Pareto solutions. They theoretically analyze the convergence and controllability of the MORL objectives, and perform experiments on challenging robotics tasks.

strengths:
The approach for employing preference control for MORL, by controlling generates trajectories is interesting and novel. 

The paper performs detailed theoretical analysis and discusses convergence and controllability of the learnt policies, which is impressive.

The authors perform experiments and show impressive performance on challenging robotics tasks.

weaknesses:
The paper method would benefit by adding a detailed algorithm explaining the method. The paper could benefit from the addition of pseudocode for the key steps of preference-conditioned policy training, and additional details on how the preference regularization is implemented algorithmically. 

Extensive experimental and implementation details are missing, which would make the results hard to reproduce. Addition of information on hyperparameters, network architectures, training procedures, or environment specifications needed to reproduce the results will greatly help increase paper reproducibility.

I am currently recommending a weak reject, but will accordingly update the score based on the discussion and ratings by other reviewers.

questions:
None.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498345943

thread_type:
official_review

<thread_object>
creator: reviewer_cabw

<content>
summary:
This paper proposes a new multi-objective algorithm, PreCo, which aligns with preferences. Additionally, this work introduces a new similarity function as a regularizer for policy updates. Empirically, it demonstrates improved hypervolume (HV) scores across multiple multi-objective environments, showing the effectiveness of preference alignment.

strengths:
+  This work provides comprehensive theoretical analysis for the convergence of Pareto stationary points.

+  Extensive experiments are conducted in both discrete and continuous state-action environments.

weaknesses:
- There is a lack of a formal definition for the policy-level gradient, $\nabla_{\pi_p}v^{\pi_p}$, especially in continuous state-action spaces. Additionally, a major concern is the efficiency of computing the policy-level gradient in continuous state-action spaces. Recent RL works often use more expressive models as the policy network, such as diffusion models [1, 2] or normalizing flows [3]. It remains unclear how to compute the policy-level gradient for such generative models.

- The training time is not reported, which makes the claim of computational efficiency for solving the min-norm problem (Equation 6) in Section 3 unconvincing..

- The evaluation metrics are limited. The experiments mainly focus on hypervolume (HV) and cosine similarity between preferences and value functions. The similarity metric is designed to demonstrate the effectiveness of the proposed similarity function $\mathcal{\Psi}(\cdot, \cdot)$. Consequently, HV remains the sole evaluation metric for assessing the quality of the Pareto front. However, HV may increase due to improvement in only one of the objectives. In Figures 5(d) and 6(d), the authors provide visualizations of the Pareto front, which show only marginal improvements over EPO. Several other evaluation metrics could be used to assess the quality of results: Overall Non-dominated Vector Generation Ratio [4], Error Ratio [4], and Sparsity [5].

- Comparisons could be made more extensive. There exists a state-of-the-art preference-driven multi-objective RL algorithm [5].

- The authors may also wish to compare their approach with another baseline that incorporates preferences as input [6].

- Additionally, this work references SDMGrad [7] and aims to address a similar min-norm problem (Equation 6 in this work, Equation 8 in SDMGrad), yet experimental results from SDMGrad are missing. It would be valuable to observe the effectiveness of the similarity function based on a comparison between this work and SDMGrad.

[1] Michael Janner, Yilun Du, Joshua B. Tenenbaum, & Sergey Levine (2022). Planning with Diffusion for Flexible Behavior Synthesis. In International Conference on Machine Learning.
[2] Zhendong Wang, Jonathan J Hunt, & Mingyuan Zhou (2023). Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. In The Eleventh International Conference on Learning Representations.
[3] Brahmanage, Janaka, Jiajing Ling, and Akshat Kumar. "FlowPG: action-constrained policy gradient with normalizing flows." Advances in Neural Information Processing Systems 36 (2024).
[4] Van Veldhuizen, David Allen. Multiobjective evolutionary algorithms: classifications, analyses, and new innovations. Air Force Institute of Technology, 1999.
[5] Toygun Basaklar, Suat Gumussoy, & Umit Ogras (2023). PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm. In The Eleventh International Conference on Learning Representations.
[6] Abels, Axel, et al. "Dynamic weights in multi-objective deep reinforcement learning." International conference on machine learning. PMLR, 2019.
[7] Xiao, Peiyao, Hao Ban, and Kaiyi Ji. "Direction-oriented multi-objective learning: Simple and provable stochastic algorithms." Advances in Neural Information Processing Systems 36 (2024).

questions:
- Are there other similarity functions that can be used in Equation (6), or what properties should these similarity functions have?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732285032379

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of revision

comment:
We sincerely thank all reviewers [R1 (caBw), R2 (NkSe), R3 (nUpd), R4 (tiB3)] for their thoughtful feedback. We are glad that the reviewers recognized the novelty and importance of the problem we address [R2, R3, R4], appreciated the depth and comprehensiveness of our theoretical analysis [R1,R2], and found our choice of evaluation environments extensive and appropriate [R1,R2,R3,R4].

Based on the feedback, we have revised the manuscript with the following adjustments to enhance clarity and reproducibility:

* Added a more detailed pseudocode for improved clarity(Algo.2).
* Included an illustrative diagram to enhance intuition of the proposed method(Fig.9).
* Provided code and figures for a [toy example](https://anonymous.4open.science/r/ICLR255131) to improve understanding(App.F).
* Included implementation details and [easy-to-run code](https://anonymous.4open.science/r/PCRL-6E05) to ensure reproducibility(App.C).
* Reran experiments with multiple seeds and updated results with statistical dispersion metrics(Sec.5, App.C).
* Added additional comparisons with related works(App.A).
* Refined the text for better clarity, improved intuition, and consistent terminology(Sec.2-5).

Our contributions include three key aspects:
* **Conceptual**: Introduced similarity-based preference control to overcome limitations of linear scalarization, and incorporated gradient manipulation from optimization literature for MORL.
* **Theoretical:** Proved that our proposed method has the desired properties for MORL, specifically that it converges to preference-aligned Pareto-optimal policies under stochastic gradients.
* **Algorithimic:** Developed a practical algorithm and conducted experiments that demonstrate its effectiveness.

We hope the revision clarifies our theoretical contributions and further justifies the effectiveness of our algorithm.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732634959094

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final comment and addtional SOTA baseline

comment:
We sincerely thank all reviewers for their time and valuable feedback. We appreciate your careful examination of our rebuttal and the subsequent score adjustments. Here, we provide a final comment with additional results in our 2nd revision to further strengthen your confidence in the evaluation.
***
Regarding the comparisons raised by reviewers tiB3 ([1][2][4]) and reviewer caBw ([3]): 

- **[1][2][3]:** These are methods limited to off-policy RL algorithms with Q-functions (e.g., HER[5], GPI[6]). They are incompatible with our general setting, which also applies to A3C/PPO, methods that do not require Q-functions.

- **[1][2]:** Exclusively optimizing the Linear Scalarization (LS) objective, their solutions are restricted to the Convex Coverage Set (CCS)[7], only a subset of the Pareto front, whereas our proposed PreCo and EPO are similarity-based methods that are not restricted by CCS solutions.

- **CAPQL[4]**, Though implemented for SAC, it is the only SOTA method overcoming LS limitations and is potentially generalizable to policy-based methods without Q-functions. However, as noted in their Remark 5, its reward augmentation can cause information loss in the original problem, with excessive augmentation potentially leading to performance degradation. In contrast, our proposed method has the theoretical advantage of preserving the original problem for robust performance.

For a convincing demonstration of our method's superiority over SOTA methods in a fair comparison, we adapted CAPQL for PPO and tested it independently of HER-like techniques, highlighting the advantages of our approach.
***

We have implemented a new SOTA baseline incorporating the reward augmentation term from CAPQL[1]. CAPQL was originally designed for SAC, as the entropy maximization in SAC serves as the reward augmentation focus[1]. To ensure a fair comparison independent of settings, code-level implementations, and algorithmic techniques (such as HER[2]), we modified our original LS with PPO into a "maximum entropy PPO", where the critic values $v^\pi(s,p)$ learn the weighted sums of returns and entropies of policy distribution $\pi(\cdot|s,p)$. The advantages are then calculated by weighted sums of returns and entropies minus values $v^\pi(s,p)$. Details are in the Appendix G of 2nd revision of our manuscript.

The results for the 6D Fruit Tree environment are shown below:
|$\alpha$|Hyper volume(1e3)|Cosine Similarity|
|-|-|-|
|$0 $ (LS) |$5.74±0.88$|$0.718±0.040$|
|$0.01$|$5.95±1.12$|$0.722±0.006$|
|$0.05$|$5.18±0.36$|$0.718±0.040$|
|$0.1$|$1.75±1.35$|$0.633±0.141$|
|EPO|$14.97±2.29$|$0.77±0.03$|
|PreCo(ours)|$15.61±0.75$|$0.78±0.03$|

The Fruit-Tree environment is simple for RL due to small discrete state and action spaces but challenging for MORL with 6 objectives and a non-strictly convex Pareto front. This comparison isolates MORL performance from lower-level RL factors, directly highlighting our method's strengths. The results show when the augmentation strength $\alpha=0.01$, the performance of CAPQL-modified PPO is marginally better than the original LS ($\alpha=0$), but still significantly worse than similarity-based methods (EPO, PreCo(ours)). Larger $\alpha$ values lead to performance drops. This result aligns with Remark 5 and Figure 9 in [1], which highlights that such augmentation can cause information loss in the original problem, and excessive augmentation results in performance degradation. In contrast, our method has the theoretical advantage of overcoming the LS limitation without any reward augmentation, thus avoiding information loss from the original problem.

The results in this environment, with a high objective number of 6 and a non-strictly convex Pareto front, effectively demonstrate the advantages of our proposed methods over CAPQL. We have included this result and analysis in the Appendix G of the 2nd revision of our manuscript.

***
[1] Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[2] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[3] Basaklar et al.(2023). "PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm.

[4] Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"

[5] Andrychowicz et al.(2018) "Hindsight Experience Replay. "

[6] Barreto et al.(2020) "Fast reinforcement learning with generalized policy updates"

[7] Roijers et al.(2013) "A Survey of Multi-Objective Sequential Decision-Making"
</content>
</thread_object>

</thread>
<thread>
creation date:
1732790309333

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Evidence showing our original implementation of Linear Scalarization (LS) is fair and adequate

comment:
Thanks again for your attention! This is further evidence that our LS baseline has been properly implemented. If this resolves your concerns, we would be truly grateful if you could consider increasing your ratings or confidence.

Using the [link](https://wandb.ai/openrlbenchmark/MORL-Baselines/runs/wjclz5r4?nw=nwuseraraffin) provided by reviewer tiB3, we retrieved results for the state-of-the-art method GPI-LS [2]. We extracted the objective data and calculated the hypervolume using the origin as the reference point, finding it to be $\mathbf{5.63e3}$. This value is very close to the $\mathbf{5.74e3}$ achieved by our LS baseline, confirming that our implementation is properly executed and achieves comparable performance to SOTA methods.
 
A complete comparison of the discussed SOTA methods[1][2][3][4] is presented in the table below:

**Fruit-tree 6D**
***
|Method|HV(1e3)|
|-|-|
|GPI-LS[2]|$\mathbf{5.63\pm 0.1}$|
|Envelope[1]| $8.43$|
|PD-MORL[3]|$9.30$|
|CAPQL[4]|$5.95\pm 1.12$|
|LS (our baseline)|$\mathbf{5.74\pm 0.88}$|
|EPO (our proposed, used as baseline)|$14.97\pm 2.29$|
|PreCo (our proposed, used as main method)|$15.61\pm 0.75$|

The fruit-tree environment features lower-dimensional discrete state and action spaces, combined with the challenge of 6 objectives and a non-strictly convex Pareto front. **This design isolates MORL performance from lower-level RL factors, enabling a fair comparison independent of such modifications**. As shown, our LS baseline achieves performance comparable to the SOTA method GPI-LS [2], confirming it is not "poorly" implemented. Additionally, the results further validate our claimed advantages of similarity-based methods (EPO/PreCo), as they significantly outperform the SOTA methods, even without the benefits of a specific training curriculum or HER.

*Note: GPI-LS results are calculated using the "front" data from [link](https://wandb.ai/openrlbenchmark/MORL-Baselines/runs/wjclz5r4?nw=nwuseraraffin) provided by reviewer tiB3. Enelope and PD-MORL are reported in [3], and CAPQL represents our implementation of PPO with maximum entropy explained in Appendix G. All results are calculated from objective returns, using the origin as the reference point.*

***
[1] Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[2] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[3] Basaklar et al.(2023). "PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm.

[4] Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"
</content>
</thread_object>

</thread>
<thread>
creation date:
1733162510268

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Discussion

comment:
## Summarised contribution
We propose a novel general framework, **Preference Control (PC) RL**, for Multi-Objective Reinforcement Learning (MORL) with preference control. Our main contribution can be summarized as:

* PCRL is the first preference conditioned MORL framework to integrate recent advances in Multi-Objective Optimization (MOO) [1][2][3].
* It supports both policy-based and value-based RL algorithms.

Our proposed algoirthm PREference COntrol (**PreCo**) unifies the following key advantages of several recent MOO algorithms and is specifically designed for RL. 

1. Exact preference alignment [1].
2. Conflict resolution for multi-objective gradients [2].
3. Robustness to gradient stochasticity [3].

Rigorous theoretical analysis, illustrative toy example and extensive MORL experiments are conducted to justify the advantages of PreCo

***
## Before and after discusion
We provide a summary of the reviewer-author discussion: 

**Main concerns of reviewers before discusion:**
1. Clarity of definitions, math notations and algorithm details.
2. Reproducibility and seed-sensitivity of experiments.
3. Evaluation metrics can be more extensive.
4. Not sure if our baselines represent SOTA performance of MORL methods

**During the discussion:**
1. We improved the manuscript with additional diagrams, pseudocode, source code, and illustrative toy examples. 
2. We reported the variance and confidence interval as requested. For reproducibility, we provided source code, seeds, and implementation details (hyperparameters, network architectures, training procedures, and environment specifications).
3. We reported additional evaluation metrics, such as sparsity and overall non-dominated vector generation ratio.
4. We provided evidence confirming that our linear scalarization (LS) baseline's performance is SOTA among LS-typed methods by comparing it with a SOTA method of the type [4]. We also compared our proposed PreCo against all discussed SOTA methods in a challenging MORL environment, demonstrating its superiority over [4][5][6][7], even though [4][5][6] rely on unfair lower-level modifications such as HER [8] and specific training curriculums.

**After the discussion:**
**All reviewers increased their scores**, acknowledging that we adequately addressed their concerns. While all other reviewers provided positive scores, reviewer tiB3 only raised their score to 5. We reached out for clarification on their remaining concerns, but unfortunately, we have not receive a response so far.


***
[1] Mahapatra et al.2020 "Multi-Task Learning with User Preferences:..."

[2] Liu et al 2021 "Conflict-averse gradient descent for multi-task learning."

[3] Xiao et al 2023 "Direction-oriented multi-objective learning:..."

[4] Alegre et al. (2023) "Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization"

[5] Yang et al. (2019) "A generalized algorithm for multi-objective reinforcement learning and policy adaptation"

[6] Basaklar et al.(2023). "PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm.

[7] Lu et al., (2023) "Multi-objective reinforcement learning: Convexity, stationarity and pareto optimality"

[8] Andrychowicz et al.(2018) "Hindsight Experience Replay. "
</content>
</thread_object>

</thread>
<thread>
creation date:
1733224493379

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Hour Reminder: Your feedbacks are invaluable

comment:
Dear Reviewers

Thank you once again for all your invaluable feedback! As we enter the final hour of discussion, we kindly ask you to consider providing greater support for our work by increasing your scores and confidence levels.

We believe this is justified for the following reasons:

1. **Solid Approach:** We have introduced a systematic and general methodology to address the limitations of most MORL methods based on Linear Scalarization (LS). This is supported by both rigorous theoretical analysis and extensive experimental evidence.
1. **Novelty:** To our knowledge, we were among the first to design a MORL algorithm inspired by recent advancements in multi-objective optimization. Before our work, existing MORL methods mentioned during the discussion had not explicitly addressed the challenges of gradient conflicts and common improving directions.
2. **Advancing the Field:** Our work highlights overlooked limitations of LS, which even the most recent MORL publications fail to address[1], claiming full Pareto front discovery of LS method. Recognizing our contributions could help clarify this misconception and guide future research in the MORL community.
3. **Addressing Concerns:** We have thoroughly addressed the initial concerns through improved clarity, additional experimental results, and by providing source code and implementation details.
4. **Resolved Misunderstandings:** Many critiques stemmed from misunderstandings, which we have since clarified. After these clarifications, we believe no doubts remain regarding the validity of our contributions.

Thank you for your consideration and support!


***
[1] Liu et al. (2024 Oct.3rd)"C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front"

https://arxiv.org/abs/2410.02236
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors kindly provided a summary of contributions and reviewers' concerns. As strongly suggested by the authors, the AC reconsidered the evaluation of a specific reviewer.

This paper investigate the multi-objective reinforcement learning (MORL) problem with user preferences. It "takes user preference as input
controlling the generated trajectories within the preference region on the Pareto frontier." The authors provided theoretical analysis and simple empirical evaluations. There is a lack of reference on Pareto optimization and preference-based RL. On the analytical side, the theoretical results are as expected. It would be better to have sufficient empirical validation across a broader range of environments.

It is a nice incremental work for MORL, but hasn't reached the bar of ICLR 2025.