Preference Controllable Reinforcement Learning with Advanced Multi-Objective Optimization
Authors: Yucheng Yang
Tianyi Zhou
Mykola Pechenizkiy
Meng Fang
Venue: ICML 2025 poster (2025)
Keywords: Multi-Objective Optimization
Multi-Objective Reinforcement Learning
Primary area: reinforcement_learning->everything_else
TL;DR: Novel Multi-Objective Reinforcement Learning (MORL) method that discovers more Pareto optimal solutions  than most previous MORL

Abstract: Practical reinforcement learning (RL) usually requires agents to be optimized for multiple potentially conflicting criteria, e.g. speed vs. safety. 
Although Multi-Objective RL (MORL) algorithms have been studied in previous works, their trained agents often cover limited Pareto optimal solutions and they lack precise controllability of the delicate trade-off among multiple objectives. Hence, the resulting agent is not versatile in aligning with customized requests from different users. 
To bridge the gap, we develop the ``Preference controllable (PC) RL'' framework, which trains a preference-conditioned meta-policy that takes user preference as input controlling the generated trajectories within the preference region on the Pareto frontier. The PCRL framework is compatible with advanced Multi-Objective Optimization~(MOO) algorithms that are rarely seen in previous MORL approaches. 

We also proposed a novel preference-regularized MOO algorithm specifically for PCRL. We provide a comprehensive theoretical analysis to justify its convergence and preference controllability.
We evaluate PCRL with different MOO algorithms against state-of-the-art MORL baselines in various challenging environments with up to six objectives. In these experiments, our proposed method exhibits significantly better controllability than existing approaches and can generate Pareto solutions with better diversity and utilities.

Preference Controllable Reinforcement Learning with Advanced Multi-Objective Optimization
Yucheng Yang'
Tianyi Zhou? 2
Mykola Pechenizkiy'
Meng Fang" 31
Abstract
1. Introduction
Multi-Objective Reinforcement Learning (MORL) has attracted growing interests in applications of training sequential decision-making agents that satisfy multiple objectives. In practice, optimizing for potentially multiple criteria often involves managing trade-offs between them. For example, speed vs. safety or distance vs. energy for robotic control tasks. Many previous MORL methods (Yang et al., 2019; Abels al., 2019; Xu al., 2020; Lu al., 2023; Alegre et al., 2023) tried to address the trade-off issue by optimizing a linearly scalarized objective, which sums up multiple objectives with preference weights. However, the Linear Scalarization (LS) approach's solutions are limited to subset of the Pareto optimal solutions, as shown in Fig. a result, L. solutions have limited optimality and are often not well aligned with the trade-off preferences. However, the Linear Scalarization (LS) approach restricts solutions to subset of the Pareto-optimal solutions, as illustrated in Fig. 1. Consequently, LS solutions often exhibit limited optimality and may not align well with trade-off preferences. More recently, Lu et al. (2023) sought to enhance LS through reward augmentation, but this can lead to information loss from the original problem. Additionally, Basaklar et al. (2023) proposed optimizing cosine similarity for better preference alignment; however, their approach lacks mechanism to resolve conflicts between similarity gradients and objective gradients.
Practical reinforcement learning (RL) usually requires agents be optimized for multiple potentially conflicting criteria, e.g. speed safety. Although Multi-Objective RL (MORL) algorithms have been studied in previous works, their trained agents often cover limited Pareto optimal solutions and they lack precise control- lability the delicate trade-off among multiple objectives. Hence, the resulting agent is versatile in aligning with customized requests from different users. To bridge the gap, we develop the "Preference controllable (PC) RL." framework, which trains a preference-conndiiooee meta-policy that takes user preference as input controlling the generated trajectories within the preference region on the Pareto frontier. PCRL is compatible with advanced Multi-Objective Optimization (MOO) algorithms that are rarely seen in previous MORL approaches. We also proposed a novel preference-regularized MOO algorithm specifically for PCRI.. We provide a comprehensive theoretical analysis to justify its convergence and preference controllability. We evaluate PCRL with different MOO algorithms against state-of-the-art MORL baselines in various challenging environments with up to six objectives. In these experiments, our proposed method exhibits significantly better controllability than existing approaches and can generate Pareto solutions with better diversity and utilities.
This motivates us to develop a novel MORL framework, Preference Control (PC) RL, to train a preference control- lable agent for user's preference trade-offs with advanced Multi-Ohjective Optimization (MOO) algorithms. By lever- aging well-established MOO methods (Lin al., 2019; Mahapatra & Rajan, 2020), our approach overcomes the limitations of LS and enables the discovery of preference- specific solutions along the Pareto front. Moreover, inspired by how certain MOO methods (Désidéri, 2009; Liu et al., 2021: Xiao et al., 2023) deal with conflicting gradients and stochastic gradients, we propose a novel MORL-specific algorithm PreCo. We also conduct comprehensive theoret- ical analysis, demonstrating that in MORL's noisy gradient setting, PCRL with PreCo can achieve preference--ontrroled Pareto-stationary solutions.
'Deaartment of Mathematics and Computer Science, Eind- hoven University Technology, Eindhoven, The Netherlands DDepartment of Computer Science, University of Maryland, Col- lege Park, Maryland, The United States Department of Computer Science, University of Liverpool, Liverpool, The United Kingdom. Correspondence Yucheng Yang <yny @ @tt,,.nl, Meng Fang <Meng. Fang @ liverpool.ac.uk.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Preference Controllable RL with Advanced Multi-Objective Optimization
We conducted experiments in environments with conflicting objectives (Felten et al., 2023) to empirically demonstrate that (I) our PCRL scheme is compatible with various MOO methods; and (2) PCRL with PreCo consistently achieves superior performance across multiple MORI. environments. In particular, our method excels in cases with a large number of objectives or conflicting objectives.
on the Pareto front and closest to the intersection of between the Pareto front and the ray from the origin to the direction of p.
2.3
25
25
2.0
20
2.0
8 1.3
8 115
8 215
1.8
10
10
2. Preliminaries
05
05
0.
06
0.7
1.0 15 20
25
00
0.5
10
1. 2.0
2.5
0.0
0.5
10
15
2.0
2.5
Multiple Objective Reinforcement Learning In the multi-objective RL (MORL) setting, agent needs to optimize possibly conflicting objectíves with their separate reward function. MORI. setting can be modeled as Multi-Objectiin Markov Decision Process (MOMDP), Unlike the scalar re- ward function in conventional MDP, the reward function in MOMDP is vector-valued. A MOMDP is defined as M AS, Po. with state space S and action space A, dynamies P(s++1) St, (t), initial state distribution Ps)) and discount factor Y E [0,1) 1). The vector-valued function S x A + Ih is a multi-objective reward function with m objectives. A policy x: S + A is a func- tion mapping states to actions. The multi-objective value functions for a policy π are:
023
(8/2
(a) Non-convex
(b) Non-strict convex
(c) Striet convex
The plots show results and limitations of optimizing the two objectives using LS objective max p" v²² The blue solid curve is the Pareto front, the colored dotted rays are the preference directions of p, and the same colored points are the resulted values u"P The Pareto front in the left is non-convex, which could happen with deterministic policies, the middle is convex but not strictly convex which often happens with discrete action space, and the right is strictly convex. We observe an obvious gap between the preferences and the achieved values in all situations. Linear scalarization can not always discover all Pareto optimal solutions and the discovered solutions are not in the intersection between the preference rays and the Pareto front. This explains why optimizing similarity (p, v") necessary for preference control.
Figure I:
q" (s, = E,T
Σ
rr(S
= =s, A, A, a
Previous works (Yang et al., 2019; Xu et al., 2020; Ale- gre et al., 2023) have focused on maximizing a linear scalarization of objectives p" v"=". However, the solution to max, p" ort or max, pT Th is confined to the convex region of the Pareto front (Chapter 4.7, Boyd & Vanden- berghe (2004)), excluding the non-convex regions. Lu et al. (2023) demonstrated that for stochastic policies, the Pareto front can be treated as convex. Even so, as shown in Fig. 1b, the solution of linear scalarization (LS) is re- stricted to a Convex Coverage Set (CCS) (Roijers et al., 2013), which is only subset of the full Pareto front. Fur- thermore, even when the Pareto front is strictly convex, IS is not guaranteed to align closely with the direction of p. as illustrated in Fig. Ic. As
tn (s) = E,
y AT(S =
Let Rx E Th to be the multi-objective value vector of π under the initial state distribution Po:
x= = -sopp iq" (So, π (So))
(3)
Each entry of o²th a value for an objective. The Pareto Front a set of nondominated multi-objective value functions JF = {e- DT' s.t. of > v" where > is the relation of Pareto dominance such that 8" > R means (Vi, u7" > oT)) A (Fj, wy > v.)) Intuitively, if " is dominated by u² then there is no objective where πı performs better so T2 is always a better choice than v,- An optimal MORL agent should have its value vector on the Pareto front
v1
result, LS often fails dis- cover all optimal solutions and is not well-suited for prefer- ence alignment.
Cb2
Hypervoluma
Cojt
Instead of learning a policy Tp for each possible p E P. our goal is to learn an agent with conditional policy ( p) that achieves Pareto optimal values (=((1-p) E arg max, (p, o²') for any p E P. For conciseness, we denote (-1-))))) as "Tr in the following text. There are
Figure 2: Illustration of hypervolume of three value vectors U2, U1.V2,0s for a two objective optimiza- tion. Their hypervolume the volume of the union set of their dominated regions (the green shaded area), re- flecting their diversity and coverage.
Preference Control Preference quantifies the trade-off among the multiple objectives. We define the set of pref- erences P {p E R p = 1,p > 0}. The desired policy for preference should have the value o" opti- mizing a similarity metric (p, )"), which can be cosine similarity or what we define in Definition 4.2. The optimal o" should be on the Pareto Front with a maximal similarity to p. In other words, the ideal xh for preference p should be
2

---

Preference Controllable RL with Advanced Multi-Objective Optimization
TWO requirements for the agent. One explore the Pareto front much as possible, and the other is to have performance trade-off close to the input preference. These two requirements can be evaluated for two metrics: HY))rvolume(HV) for exploration of Pareto front and Similarity (p, v") for controllability.
propose MOO approach specifically for PCRL while these existing MOO methods and LS will be tested as baselines. In the following section, we first introduce how to estimate the aー values then explain our proposed update method. We provide theoretical guarantee of our proposed update method in the next section.
Multi-Objective Optimization Methods Previous Multi- Objective Optimization (MOO) methods deal with how to manipulate gradients from multiple objectives so that up- dating with the manipulated gradient can reach Pareto op- timality. A typical method MGDA (Désidéri, 2009) can guarantee to update in a common ascending direction and stops when the Pareto stationary points are reached. Meth- ods such as CAgrad (Liu et al., 2021) and SDMGrad (Xiao et al., 2023) can provide Pareto optimal solutions by linear scalarization with preference weights. However, men- tioned above, optimizing linearly scalarized objective with weight p can not guarantee a large similarity V (p, v²).
3.1. Objective Estimation
Preference control aims to achieve the desired trade-off on conflicting objectives. the previous RL experiments of MOO methods like (Yu et al., 2020; Liu et al., 2021; Xiao et al., 2023), the loss of the value function used as the objective for MOO, and equal weight is given to all value losses to balance the multi-objectives. While this may be appropriate for RL tasks with minimal conflict, in our setting for preference control, it is essential to align the objective with the preference, so the objective to be aligned with the preference should UFr itself rather than its approximation loss. Here, we show how estimate " for mainstream RL algorithms.
Methods such as PMTL (Lin et al., 2019) and EPO (Mahap- atra & Rajan, 2020) apply similarity constraints to reach the Pareto front with the desired preference, so they can be used for preference control purpose. In the next section, we show how these methods can be used for learning ralss p) and they will be used as baselines for our proposed new MOO algorithm.
When learning Tp with value-based methods like
DDPG (Lillicrap et al., 2016). TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018), we can estimate T.R by
TP Ep (go TT p)]
(4)
where qe is multi-objective critic network that outputs vector of Q-values, it is also conditioned on the prefernece p, because p controls the policy thus controlling the value qº.
3. Learning preference controllable agent
Preferred
Trade-ofts
Update
Agent
Environment
For policy-based methods such as A3C (Mnih et al., 2016), PPO (Schulman et al., 2017). they update with a whole episode so " can be estimated by episodic returns.
Estimated
Objectives
Evaluation
Figure 3: PCRL updates the agent based on its performance and the user preference of objectives.
Rr E Eg rupo
E2'r(S
(5)
We propose "Preference control (PC) RL" scheme to incor- porate MOO algorithms handle the trade-offs between multiple conflicting objectives. We train single agent that can be conditioned on different performance preferences. Conditional preference p controls the agent's emphasis on different objectives and corresponds to a desired point on the Pareto front. We denote the policy conditioned on a pref- erence (+1 p) as Tp. During training, v is estimated for uniform sampled p P. Then we can evaluate similarity V/ (p, r and obtain update direction for Tp. In PCRL scheme, the update direction can be obtained using any methods that can incorporate preference on the objectives, including LS (optimizing max,, p" TTT), or other MOO methods with extra optimization or regularization of the similarity (detailed implementations Appendix B). We
As result, our scheme is applicable to both discrete action space and continuous action space. With the estimated value vector 0"P, we can evaluate the similarity V (p, 6*)).
3.2. Updating Procedure
After estimating objective values and similarity for pref- erence control, we need to manipulate the gradients from different objectives and update the agent using the manip- ulated gradient. Our scheme has the following updating procedure:
1. Get the Jacobian matrix Vn, n 6m
Each row of the Jacobian matrix Va, Th is gradient for one objective. The gradient can be obtained by
3

---

Preference Controllable RL with Advanced Multi-Objective Optimization
conventional RL methods, such as the policy gradient and the deterministic policy gradient. An illustrative diagram (Fig. 9 in Appendix E) provides intuition and i for different RL algo-
Algorithm PreCo in the theoretical analysis setting
1: Initialize: Preference p. preference-conditione policy
Tp. and weights wo
shows how to estimate V up
2: for t = 0, 1,.., 1 do
rithms.
3: Rollout and estimate the value to get data E,
4: wi = [1
2.
Get similarity gradient n.,, (pp 6"r):
G. (G( + (Tp: E')])
V should effectively measure the similarity between the preference and the estimated value p'P. While cosine similarity could be reasonable choice for quan- tifying the closeness between the value vector and the preference, it may conflict with the underlying objec- tives. To address this, we propose a novel design for the similarity function, combined with a gradient manip- ulation approach, that is capable of both reaching the Pareto front and enabling effective preference control.
xp.41 = Tp.t
04 (G(Tp,ti ()w,,1 + degs(TTp.ti)) ())
6: end for
4. Theoretical Analysis
In this section, we provide the formal definition our pro- posed similarity function --.-) and the theoretical analy- sis for the PreCo update. We will prove that it converges
3. Manipulate the gradients and find the optimal update direction d' by solving:
to Pareto stationary points, and the resulting similarity (p, v"r) will also converge to stationary points. Definition 4.1.
" E arg min [dd]; 10
We define our similarity function as
d A V T 6 + AV o VP, (*)
V; P -v11
(pp, v) = - 2 P:
mmx
(8)
d' v B p + AV,, 4 6*)
(7)
This is a min-norm problem similar to MGDA and SMGrad, but it adds a similarity gradient to every objective gradient, making the update not only ascent in a common improving direction but also closing the value R' to the preference p. We call this update PREference COntrol(PreCo) update. Intuitively, updating with the solution d' converges to where ||d* || 0, indicating direction possible for common improvment of all objectives thus satisfying Pareto stationary. We will prove the convergence of this gradient under our proposed similarity function.
Intuitively, the similarity gradient V, v/ (p. v) encourages to focus on the less optimal objectives reach the preference p. A visualization for (pp.)) can be found in Appendix F.
Deep reinforcement learning is inherently stochastic and sen- sitive to sample complexity. Therefore, we analyze the con- vergence rate of the proposed PreCo update in the stochastic gradient setting. The PreCo algorithm that we analyze in this case is Algorithm 1, where ur is the coefficient defined in Equation (6) and 1l means the projection to the set W == {w E Rm w 1, different noise samples when estimating ((Tp) and go (Tp) gradients, which are defined as:
0}. Data & are
This is a general update procedure that can employ any RL algorithm for the calculation of the objective gradients 6=, In the third step, the gradient manipulation can also be performed by not only PreCo but also existing MOO algorithms. In the experiment, with PreCo against the baselines with existing MOOs such as EPO (Mahapatra & Rajan, 2020) CAGrad (Liu et al., 2021). Computationally, the min-norm problem in the third step is solved at the policy level with V., h0 instead of the parameter level with Vr The size for a sample of V v., ih is only m B for batch of B transitions, while V00"> of size m x M could have a parameter size M m. M can even be billions for large models. For those cases, solving the min-norm problem at the parameter level could be -inttorriiefffiienttt and computationally intractable. A pseudo-code for the PCRL scheme with PreCo update and more details on the definitions of policy level gradients and parameter level gradients can be found in Appendix E.
G(TT) = B[G(Tpii)] = VT VT 0J
E 6"²), - Tp
(9)
(.(pT) = Blg.(TT::)) E)]
examine our scheme
(10)
= VG (*))
where the expectation is taken w.r.t the noise &, the ith column of G(πTi is the gradient of ith objective and gs (xp) is the similarity gradient.
Algorithm is only for theoretical analysis; In practice, the weight w does not need to be updated only once every itera- tion but can be fully optimized for the min-norm problem (6) and a more practical Algorithm 2 is provided in Appendix E.
4.1. Convergence Analysis
First, we define what Pareto stationary is:

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Definition 4.2. We define a is an e-accurate Pareto station- ary policy if miin [[G(m))wll) w S where w is convex coefficient.
we have + ET=o El (T((T,)) O To achieve 6-accurate Pareto stationary point, requires T = O updates.
We assume the continuity and smoothness of the objectives.
Theorem 4.1 shows PreCo converges to Pareto stationary points when À is a constant. This theorem applies to our proposed similarity function (p..).
Assumption 4.1. For every objective i E [m]] (x(Tp) is L--Lipschitz continuous and (Tp( is 11,1 -Lipschit con- tinuous for any preference conditioned policy Tp.
Theorem 4.2. Under the Assumptions 4.1-4.3, setting Om 1omm-1-p 2), Be O(m O(m-'TT3), with Lipshitz smooth similarity function with g (Tp.) being convex com- bination of 9. (TT))) all t, there can be an increasing X = Olog T) and we have T Et= Blmmnno [[G((T,))
This assumption is quite common in RL setting. By the "branched returns bound" in (Janner et al., 2019).
[m: - -0: (T2)) S mi,, ( 7)2
7/Ex
1- (11)
O 4 log T).
where ( max aa ri (s(8 a) and 6, can be any scalar satisfying En 2 max, Drv (-|s), (2 (-]s)) Because
Theorem 4.2 consider a case requiring similarity gradient to be a convex combination of objective gradients, of which its design is discussed in Appendix F.2. In this case À can increase without an upper limit and eventually Is will dominate the min-norm solution of (6). Proofs are in Ap- pendix 1.2.
1 T T2
max (rx - (TI --
2
(12)
we can derive
1
Remark 4.1. In practice, Theorem 4.1 still applies to cases where X increases but with an upper limit. Because after X gets close to the limit, it can be considered constant. This offers theoretical justification for implementing PreCo with V (p.) )) and an increasing X.
[02(T1) -v:))) S Timx,i (1 71- 1 m 22
+
(13)
and L; can be Tmax, (I-T)) 1 Therefore, the Lips- chitz continuity of objectives is naturally satisfied for con- ventional RL settings, and we only need to assume the
The Pareto front is convex (but not necessar- ily strictly convex) for MORL (Lu et al., 2023), so local Pareto stationarity (Definition 4.2) is equivalent to Pareto optimality.
Remark 4.2.
gradients are also Lipschitz continuous.
Next, we make an assumption on the bias and variance of the stochastic gradient 9: (T)
Assumption 4.2. For every objective i E [m]) the gradients ge (Tp: {) is unbiased estimate of (T ), and the variances is bounded by (g(l(g((Tp:EE) - 9: (T || S o²
4.2. Controllability Analysis
Controllability in our setting is the similarity between the desired preference p and the value 80 of the preference- conditioned policy Tp. It measured by V/ (p, or)). We provide the following results show how Fh will converge to the point close to the p direction.
We also assume bounded gradient.
Assumption 4.3. There exists a constant Og such that [|G(Tp)]| Cg.
Lemma 4.1. The similarity function (pp.,) is (1 + max, (p:)) -Lipschitz smooth and 9-(-) is Lipschitz continu- ous under Assumption 4.1 and Assumption 4.3.
Theorem 4.3. Under the Assumptions 4.1-4.3, setting at O(m Omm-3T-3 -T- ), Br = O(m-11 3), with a constant X and Lipschitz smooth similarity function like Vp +), we have 2c² x Om.T-)).
+ ErO (lllss
This lemma shows that our proposed similarity function is Lipschitz smooth. The detailed proof is in Appendix I.1.
Theorem 4.3 provides an intuitive result, that with constant X. the norms of the similarity gradient + ET (T))) will converge and be bounded. The 20 larger X, the lower the bound x², and the closer the solution will reach the stationary points for maximizing similairty.
PreCo and SDMgrad (Xiao et al., 2023) both belong to MGDA-variant methods that solve a min-norm problem for gradient manipulation. Leveraging the fact that g, (Tp) is positive linear combination of G(TT) and the Lipschitz smoothness property, we therefore build upon their results to prove that PreCo converges to Pareto stationary points.
Theorem 4.4. Under the Assumptions 4.1-4.3, setting B, = 0(m-'T-3 with constant X and Lipschitz smooth similarity function like (p )). there be increasing À @ and we have + IT= (l(1gg(TT))))) O(m log T).
a O(m - 1imm-BT-3
Theorem 4.1. Under the Assumptions 4.1-4.3, setting at = ((m 0mm-1T 3. = 9mm-'T-1) with a con- stant X and Lipschitz smooth similarity function Vpp.)) )),
5

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Theorem 4.4 shows PreCO with increasing À will converge to the stationary points for the similarity objective. The proof' of the theorems are in Appendix I.3.
objectives are highly conflicting.
Evaluation metrics We evaluated the results using two metries: hyperVolume(HIV) for Pareto front exploration and Cosine Similarity for controllability evaluation. They are measured in test time with preference samples unseen in training (Appendix D). We report the mean and standard deviation results of 5 seeds.
Remark 4.3. Similar to Theorem 4.1, Theorem 4.3 applies to practical implementations where increases but with an upper limit.
Remark 4.4. The converged stationary points do not guar- antee to have always high similarity metrics. For ex- ample, when using (p..), our results show (e(mp) = V(TT)V., v)>) converges to O. However, the value ahe coincide with the preference p only when IV (p, u"))|
Baselines We compare PreCo with existing MOO gradient manipulation methods in the PCRL scheme and existing MORL algorithms.
[Tp(l can also be O when IV. uT))ll > 0, with G(Tp) and V. (p v=x) be- ing orthogonal or G(Tp)= 0. These situations means the points desired by the preference might not exist on the Pareto front. We discuss in practice how to deal with unreachable regions of Pareto front in Appendix H.
0.
Linear Scalarization (LS): It optimizes max,, p p" 0" Ex- isting MORL methods such as Yang et al. (2019); Xu et al. (2020); Alegre et al. (2023) all optimize the LS objective with implementation--evel modifications.
MOO algorithms: Such as EPO (Mahapatra & Rajan, 2020), CAGrad (Liu et al., 2021), SDMgrad (Xiao et al., 2023). They can find a common ascending direction to handle conflicting gradients.
The theoretical results show that PreCo can discover not only Pareto stationary solutions but also preference-specific solutions. Using a 2-objective MOO example with noisy gradients in Appendix G, we highlight PreCo's ability to find preference-specific solutions in the context of stochastic and conflicting multi-objective problems.
SOTA MORL methods: Oft (SOTA) MORL methods such as CAPQL (CAP) (Lu et al., 2023) and PDMORL (PDM) (Basaklar et al., 2023). They try to ad- dress the limitations of the LS-based MORI..
5. Experiments
More details about the baselines and their implementations can be found in Appendix B and Table 3.
Beyond the toy example MOO problem in Appendix G, which illustrates the theoretical advantages of PreCo, we conduct experiments in MORL environments to empirically demonstrate the practical effectiveness of PCRL with PreCo.
5.1. Fruit Tree
We evaluate our method against MOO baselines in set- tings with reward dimensions ranging from 3 6. The HV and CS results, presented in Table 1, demonstrate our proposed PreCo outperforms the baselines, particularly in higher-diment dimees-aimensional reward scenarios. The results in Table 2 highlight the significant advantage of our PCRL scheme with EPO/PreCo (ours) over SOTA MORL methods.
Benchmarking Environments Common continuous con- trol environments like MO-Hopper and MO-Ant (Felten et al., 2023) feature higher-dimennionall spaces but symmet- ric objectives (e.g., moving north and east), resulting in strictly convex Pareto fronts. In such cases, low-level RL implementations (c.g., HER (Andrychowicz et al., 2017), curriculum design (Alegre et al., 2023)) that improve sample efficiency are more critical than algorithm design. To isolate the effects of different multi-objectivg methods from imple- mentation details, we focus on empirical analyses in environ- ments with discrete action spaces, more objectives, and non- strictly convex Pareto fronts, such as Fruit-Tre and MO- Reacher. Results for MO-Hopper and MO-Ant are in Ap- pendix C, where we also demonstrate that PreCo is compat- ible with these low-level implementation improvements.
This environment exemplifies a scenario where LS meth- ods learn only limited solutions for non-strictly convex Pareto front. As shown in Fig. 4, in the 3-D reward setting, PCRL with PreCo successfully discovers the blue points representing all Pareto solutions. In contrast, the LS agent learns only a single constant F. at the red point (a CCS solution), regardless of the preference input. This occurs because the red point optimizes the LS objective pz 6t, for most p (shown as red rays). The singular learned value reveals that the LS agent is uncontrollable by p. despite being conditioned on it.
Fruit-Tree: A discrete environment with up to 6- dimensional reward, presenting significant challenges due to its six objectives and a non-strictly convex Pareto front. LS-based methods can only find limited Pareto solutions.
In the 6-D setting, our implemented LS baseline demon- strates performance identical to another SOTA MORL al- gorithm, GPI-LS/PD (Alegre et al., 2023). This observa- tion highlights that based methods inherently face an
MeaRherh A robotic control environment with a con- tinuous state space and a discrete action space. The four
6

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Method
3D
5D
0.12 4: 0.01
0.78 0.03
0.33 d: 0.13
0.76
0.05
1.59 4: 0.29
0.74
0.01
5.74 0.88
0.72
0.01
SDMgrad
0.14
0.78 = 0.03
0.66 + 0.02
0.72
0.00
2.74 0.09
0.00 = 0.01
13.30 0.15
[0.72 0.01
EPO
0.15 0.01
0.84
0.02
1.04 4: 0.05
0.89 +: 0.02
3.98 1 0.48
0.86
0.03
14.97
2.29
00.7 0.03
CAGrad
0.14 + 0.02
0.78
+ 0.02
0.30 + 0.06
0.87
= 0.01
1.23 1 0.14
0.69
= 0.01
4.93 + 0.81
0.60 0.09
PreCo(Ours)
0.15 0.01
0.84 0.02
0.02
0.91 0.01
0.21
0.87 0.011
15.61 1 0.75
0.78 0.03
Table 1: "HV (higher is better for both) in fruit-tree environment with HV in the scale of 10 Our method consistently achieves the best optimality (HV) and controllability (CS) from 3-6 objectives.
Method
HV
CS
Daj 18z for tralt Buu
LS/GP--LS
5.74 + 0.88
0.72
CAPQL (entropy coef = 0.01)
5.95 = 1.12
0.72 0.02
PDMORI (with HER)
9.30 + 0.08
0.89 : 0.05
EPO
14.97 2.29
0.77 0.03
PreCo (Ours)
15.61 = 0.75
0.78 = 0.03
Table 2: PCRL with advanced MOO algorithms such as EPO and our proposed PreCo outperforms SOTA MORL methods in Fruit-tree environment.
Protein
(a) Protein-Carbs-
(b) Protein-Carbs
ward one target typically increases the distance from others. Therefore, the LS gradient, V o"rp, associated with dif- ferent preferences p. can exhibit significant conflicts. More- over, local optimum for the LS objective maxx, pO*' may simply involve the agent staying near the origin position that is "equidistant" from all four goals but fails to fully achieve any of them. As shown in Fig. 6a, unsurprisingly, LS and SDMgrad that optimize max p" learned only the 'equidistant" local optimum. Moreover, when prefer- ences p are sampled uniformly, the gradients can exhibit significant variance. This explains why CAGrad also fails to learn meaningful results, as demonstrated in Appendix G and discussed in (Xiao et al., 2023), where it is shown that CAGrad struggles with stochastic gradients. Consequently, these methods learn only limited solutions that cannot be effectively controlled by the preference parameter p.
Ocl 163 for fruit Tree
Ob) 263 for Frui Tre
TTp
Potast
Photenn
(c) Protein
(d) Carbs Fats
Figure 4: (a) shows the 3-D values Th achieved under difference preference input p. Blue points are T of PreCo while LS only learns the red point for some runs. The red point is the optimal LS solution for the preference directions represented by the red rays. (b) shows the Protein-Carf Carbss view, (c) shows the Protein-Fats view, and (d) shows the Protein-fats view.
As for existing MORL methods, while both CAPQL and PDMORL address some limitations of LS, they have no- table shortcomings. CAPQL relies on reward augmentation, which may lead to suboptimal behavior due to the informa- tion loss of the original objectives. Like PCRL with EPO or PreCo, PDMORL optimizes for similarity but directly adds cosine similarity to the original objectives. However, the gradients of cosine similarity and the original objec- tives can conflict, as it does not leverage conffict-avoidancee techniques from MOO algorithms, thereby failing to deliver competitive results in this environment.
upper limit when applied to non-strictly convex problems, regardless of lower-level modifications or enhancements. PDMORL explicitly optimizes for cosine similarity, which may introduce conflicts with other objectives. This could explain why PDMORL achieves high CS but fails deliver competitive HV results..
5.2. MO-Reacher
The MO-Reacher presents a challenging environment with 4-dimensional rewards, requiring the agent to reach four targets illustrated in Fig. 5a. Fig. 5 shows the quantitative results of HV and CS and Fig. 6 shows the state coverage of robotic arm tip positions.
Fig. 6b shows the state coverage of PreCo and EPO con- trolled by 4 different p. From left to right they are [0, 1, 0, 0). [0, 0.67, 0.33, 0J. [0, 0.33, 0.67, 0j. (0, 0, 1,
The preference [o. 1,0,0) 0] means full focus on closing to the top target, while [o, 0, 1, Oj full focus on the left target. The first row is EPO and the second is PreCo. Their state
The primary difficulty of this environment lies in the fact that the four objectives are often highly conflictng-mmoiiie to-
7

---

Preference Controllable RL with Advanced Multi-Ohjective Optimization
with the preference directions. However, Xu et al. (2020) reported the TS aproach has suboptimal performance in practice, due to the oscillation and stagnation issues, as noted by Mahapatra & Rajan (2021). In particular, for MORI,, which is sensitive to stochasticity and conflicting gradients, the convergence of 'S can be problematic. Our baseline implementation of EPO with a small constraint threshold can be considered as a version of TS tailored to the MORL setting, designed to mitigate oscillation when near preference direction. PDMORL (Basaklar et al., 2023) incorporates cosine similarity for preference alignment but it is designed specifically for off- value-based RL and its performance relies heavily on the HER (Andrychow- icz et al., 2017) technique since it often fails without HER as explained by "underrepresentaaion" in (Basaklar et al.. 2023). Another potential reason could be gradient conflicts between the cosine similarity optimization and the original objectives. CAPQL (Lu et al., 2023) addressed LS limita- tions by adding a concave augmentation term to the reward, transforming the original Pareto front into a strictly convex one. However, this introduces information loss, making the approach sensitive to the augmentation term's 's magnitude.
Sorrrt
(a) MO Reacher
(b) Hypervolumee
(c) Cosine Similarity
Figure 5: HV and CS of MO-Reacher. The dotted line is the
performance of randomly initialized agent as a reference.
Fandom
15
Schigndd
CRGre
48
(a) State coverage of Random, LS, SDMgrad, and CAGrad (Ieft to right). Each state coverage hcatmap is the same for diffcrent preference p.
p=[0.0.1,0)
(=000.3330066.,00
p=10,0.67,0.33
p=(0,1,0,0]
EPO
k41
To our best knowledge, PCRL is the first preference- conditioned MORL framework to integrate recent advances in Multi-Objective Optimization (MOO). Among the MOO algorithms, PMTL and EPO (Lin et al., 2019; Mahapatra & Rajan, 2020) can be viewed as enhanced versions of TS for preference-specific optimization. CAGrad and SDMGrad (Liu al., 2021; Xiao et al., 2023) originally optimize only for the average objective, but they can handle confficting gradients. MGDA and SDMgrad (Désidéri, 2009; Xiao et al., 2023) also address a min-norm problem like (6) for gradient manipulation. However, a significant advantage of our PreCo is that it not only identifies a common ascending direction but also discovers preference- specific solutions. Our theoretical analysis of PreCo builds on some lemmas from Xiao al. (2023), but incorporating the similarity gradient from V is novel contribution, which complicates the proof. A more detailed discussion of a broader range of related works can be found in Appendix A.
PreCo
60 (b) State coverage heatmaps of EPO (top row) and PreCo (bottom row) under four different preferences p (columns). It shows that they produce preference-specific policies given different
Figure 6: State coverage heatmaps for the positional states of the tip of the robotic arm. Red for higher density of coverage and blue for lower. EPO and PreCo exhibit different state coverage con- trolled by different p. while random, LS, SDMgrad, and CAGrad show the same state coverage for different preferences.
coverage can be smoothly controlled from more density to the top to more density to the left. This accords with their higher results in CS. More experiments in higher di- mensional continuous control environments can be found in Appendix C. Our PCRL scheme with advanced MOO al- gorithms is shown to have competitive performance against existing MORL methods. Especially when incorporated with our proposed PreCo algorithm, it consistently outper- forms existing methods across all environments.
7. Conclusion
6. Related Works
We propose PCRL for preference control in multi-objective trade-off', integrating recent MOO algorithms into MORL. We also introduce PreCo, novel MOO approach, with convergence analysis supporting its ability to learn preference-specific Pareto-optimal solutions and handle stochastic gradients. Experiments across multiple RL environments show that PCRL with PreCo consistently outperforms baselines, demonstrating its effectiveness in learning preference-controllablee agents. Future work could enhance the efficiency through curriculum learning.
Existing MORL methods that learn a similar preference- conditioned policy include (Abels et al., 2019; Chen al., 2019; Lu et al., 2023; Basaklar et al., 2023), of which (Abels et al., 2019; Lu et al., 2023) are LS methods and they care more about discovering all Pareto optimal policies rather than the similarity between the weight input and the resulted value. Chen et al. (2019) employs a setting most similar to our PCRL since they optimize a Tehebycheff Scalar- ized (TS) (Ehrgott, 2005) objective for solutions aligned
8

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Acknowledgements
Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference
This work was supported by the TKI Smart2 project, a collaboration between KPN and Eindhoven University of Technology (TU/e).
Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Impact Statement
Chen, X., Ghadirzadeh, A., Björkmman, M., and Jensfelt, P. Meta-learning for multi-objective reinforcement learning. In 2019 IEEE/RSJ International Conference on Intelli- gent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pp. 977-983, IEEE, 2019, doi: 10. URL httpp://ddo 1 1109/RR054089..22998888888888889 201 9. 9.8998880000 8968092.
This work addresses key limitations of existing multi- objective reinforcement learning methods by leveraging recent advances multi-objective optimization. We propose general and flexible approach applicable across diverse domains, including gaming agents, robotics, and fine-tuning large language models. While the method itself poses no direct risks, its impact depends on the context and manner in which it is applied.
Désidéri, J.-A. Multiple-Gradient Descent Algo- rithm (MGDA). Research Report RR-6953, June 2009. URL https://inriss hal. sellsscienn science/ inria-00389811 In this report, the problem of min- imizing simultaneously n smooth and unconstrained cri- teria is considered. A descent direction common to all the criteria identified, knowing all the gradients. An algorithm is defined in which the optimization process is carried out two phases one that is cooperative yielding to the Pareto front, and the other optional and competitive.
References
Abels, A., Roijers, D. M., Lenaerts, T., Nowé, A., and Steck- elmacher, D. Dynamic weights in multi-objective deep reinforcement learning. In Chaudhuri, K. and Salakhut- dinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 11- 20. PMLR, 2019. URL http: htttp::/pprcceeinggg mllr. press/v91 / abe 1 s 9a. htmll
Ehrgott, M. Multicriteria optimization. 2005.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019. URL https: //arxiv. wwwp////////..o../////// 10wp://10..0000 06070.
Alegre, L. N., Bazzan, A. L. C.,, Roijers, D. M., Nowé, A., and da Silva, B. Sample-efficient multi- oojeitvb learn- ing via generalized policy improvement prioritization. In Proceedings of the 22nd International Conference Au- tonomous Agents and Multiagent Systems (AAMAS), pp. 2003-2013 International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2023. URI https: / aaaii.iiivv///////aaaaa/aaaaavva org/ abs 210. 07784.
Eysenbach, B., Salakhutdinov, R., and Levine, S. The in- formation geometry of unsupervised reinforcement learn- ing. In Proceedings of the 10th International Confer- ence on Learning Representations (ICLR), 2022. URL /ttEps:/arxiv..oom http://axxxv...g/abaaa wttp:///ggaaaaaaa.aaaaaaaas 02719.
Andrychowiez, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017. URL https: http://apprrs...... cc/ httpcc//ppppee /aper/ 1090-hinds'o gitt wwweeeeeeeeeeeeeeeee replaceeeerrrrr...eeeeee
Felten, F., Alegre. L. N., Nowé, A., Bazzan, A. L. C.. Talbi, E. G., Danoy, G., and Silva, B. C. da. A toolkit for reliable benchmarking and research in multi-objectivg reinforce- ment learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
Basaklar, T., Gumussoy, S., and Ogras, U. Y. Pd-morl: Preference-driven oojectiv reinforcement learning algorithm. In Proceedings of the 11th International Con- ference Learning Representations (ICLR), 2023. URL https: / /aaxivvvvvv. org/ abs/ 22t./220...2 /11.//222........
Fujimoto, S., van Hoof, H., and Meger, D. Address- ing function approximation error in actor-critic meth- ods. In Dy, G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learn- ing, ICML 2018, Stockholmsmisssun, Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018. URL http: /pppcceeeeiggg..pp//ee/eeee...ae httbttppp/ppoeeeeiiiaa v mujimotol Ba html.
Boyd, S. and Vandenberghe, L. Convex optimization. Cam- bridge university press, 2004.
Brahmanage, J. C., Ling, J., and Kumar, A. Flowpg: Action- constrained policy gradient with normalizing flows. In
mlr ppesss

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Confer- ence on Machine Learning, ICMI. 2018, Stockholmsmäs- san, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856- 1865. PMLR, 2018. URL http: http::/proceeeeeeeeeee http://poooceeeineeeeee mlr. press.rsss..om / /80/ /tt//aaarnsae hwwaaaaaaaoaaaaaaaaaaaaaaa 8b. html.
Lin, X., Zhen, H., Li, Z.,, Zhang, Q.. and Kwong, S. Pareto multi-task learning- In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12037-12047 2019.
Lin, X., Zhang. X., Yang. Z., Liu, F., Wang, Z., and Zhang, Q. Smooth tchebycheff scalarization for multi-objective optimization. In Proceedings of the 41st International Conference Machine Learning (ICML), 2024. URL org/ abs 2402 19078.
Hansen, S., Dabney, W. Barreto, A., Warde-Farley, D., de Wiele, T. V., and Mnih, V. Fast task inference with variational intrinsic successor features. In 8th Interna- tional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL hwtpt:ht///pott Forum?1d=BJeAHat htt.tpss///peeeviie...oo net/
Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34, 2021.
Liu, M., Zhu, M., and Zhang, W. Goal-conditioned reinforeement learning: Problems and solutions. In Raedt, L. D. (ed.), Proceedings of the Thirty-First In- ternational Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 5502-5511, ijeai.org, 2022. doi: 10.24933/JCAA12022 770. URL htt.tt:tt////......t http://otooooooo//o////oo .rgg///ggg00.0 24 / jiaai.. 2022/770.
Hung, W., Huang, B.-K., Hsieh, P.-.., and Liu, X. Q- pensieve: Boosting sample efficiency of multi-objective RI. through memory sharing of q-snapshots. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https: / /pwp//ppeeeeweeeooeeee net/ forum? farummiid-ane aid
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché- Buc, F,, Fox, F. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp.
Lu, H., Herman, D., and Yu, Y. Multi-objective reinforce- ment learning: Convexity, stationarity and pareto opti- mality. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nett 2023. URL https: //openreview net / forum? d=T JEZIsEES0
2019.
Mahapatra, D. and Rajan, V. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICMI. 2020, 13- 18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 6597-6607. PMLR, 2020. URL.
Kingma, D. P. and Ba, Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Lee, Y., Sun, S., Somasundaram, S., Hu, E. S., and Lim, J. Composing complex skills by learning transition policies. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //openreviews wwt. iid
ngs ml press/ v119 htt//aaaapatraa22 .tt.m.00.....m...m
Mahapatra, D. and Rajan, V. Exact pareto optimal search for multi-task learning: Touring the pareto front. CoRR, 2021. URL httpt:////eett /arxiv..rggg////// abs 21088. 00597.
Li, Z., Li, T., Smith, V., Bilmes, J., and Zhou, T. Many- objective multi-solution transport. In The Thirteenth Inter- national Conference on Learning Representations (ICLR), Boston, MA, USA, 2025.
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. Large language models: A survey. arXiv preprint arXiv:2402.06116, 2024.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In Bengio, Y. and Le- Cun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:/ /ttp:///axxvv.. http://aadvv.oo/aas///////////// 02971.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lilli- crap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learn ing. In Balcan, M. and Weinberger, K. Q. (eds.), Pro- ceedings of the 33nd International Conference on Ma- chine Learning, ICML. 2016, New York City, NY, USA,
10

---

Preference Controllable RL with Advanced Multi-Objective Optimization
June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1928-1937. oog. 2016. URL http::/ http::://ppcceeinngs.... mwr.m.prree press/ v48 / mmmmaahll.. 6.mmll.
Vinyals, O., Babuschkin, I., Czamnecki W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-35) 2019.
Nam, T., Sun, S., Pertsch, K., Hwang, S. J., and Lim, J. Skill-based meta-reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRe- n
Wang, Z.., Hunt, J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning, In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023. URL https: / //arxiv. org/ abs 2208 8 061 93.
view.net, 2022. URL https: forum? je-
Xiao, P. Ban, H., and Ji, K. Direction-oriente multi- objective learning: Simple and provable stochastic algo- rithms. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.). Advances in Neu- ral Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Tay- lor, M. F., and Stone, P. Curriculum learning for reinforcement learning domains: A framework and survey. J. Mach. Learn. Res., 21:181:1-181:50, 2020. URL https:/ https::/jmlr.rrggtt http://lll.rororprrr//o/// 20-212 html.
Nguyen, D., Chen, J., and Zhou, T. Multi-objective linguistic control of large language models. In Find- ings of the Association for Computational Linguis- tics: ACL 2024, pp. 4336-4347, Bangkok, Thai- land, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024 finding.fiiinn--aa acl. 257. URL https://aaat findings-acl ha/aaaaaaahoooogyy.. org/2024. 2024.
Xu, J., Tian, Y., Ma, P., Rus, D., Sueda, S., and Matusik, W. Prediction-guide multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, pp. 10607-10616. PMLR, 2020.
Yang, R., Sun, X., and Narasimhan, K. A generalized algorithm for multi-objective reinforcement learning and policy adaptation.
Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 14610-1462) 2019.
Portelas, R., Colas, C., Weng, L., Hofmann, K., and Oudeyer, P. Automatic curriculum learning for deep RL: A short survey. In Bessiere, C. (ed.), Proceedings of the Twenty-inth International Joint Conference on Arti- ficial Intelligence, IJCAI 2020, pp. 4819 4825. ijcai.org, 2020. doi: 10.24966/ICCA122200661.. 1. URL ht tts::s: // //doi. . org/ 10. 9 2020/671.
Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., Li, X., Han, L., and Zhang, C. Rethinking goal-conditioned supervised learning and its connection to offline rl. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022. URL https: /larxiv.org// abs /202 04478.
Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R. A survey of multi-objective sequential decision-making. J. Artif. Intell. Res., 48:67-113, 2013. doi: 10.I613/JAIR. 3987. URL https: doi. owt.:/..rrg11..... 3987.
jair.
Yang, R., Pan, X., Luo, F., Qiu, S., Zhong, H., Yu, D., and Chen, J. Rewards-in-contett Multi-objective align- ment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024a. URL https://azzee org. 22..
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707:07777.6347, 2017.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning explore via self-supervised world models. In International Conference on Machine Learning, pp. 8583-8592. PMLR, 2020.
Yang, Y., Zhou, T., He, Q. Han, L., Pechenizkiy, M., and Fang. M. Task adaptation from skills: Information geome- try, disentanglement, and new objectives for unsupervised reinforcement learning. In The Twelfth International Con- ference on Learning Representations, ICLR 2024, spot- light, Vienna, Austria, May 7-11, 2024. OpenReview.net 2024b. URL ht /ttttps:///ppeeeveiiee net forum? i id=zSxpnkhlys
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Grae- pel, T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Sci- ence, (4466666999111111 40 44, 2018

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learn- ing. In Advances in Neural Information Processing Sys- tems, volume 33, pp. 5836, 2020. URL https: //arxiv. abs/ 2001 06782.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
12

---

Preference Controllable RL with Advanced Multi-Objective Optimization
A. Related Works
A.1. Meta-policies
Existing MORL methods that learn a similar meta-policy conditioned on a weight or preference include (Abels et al., 2019; Chen et al., 2019; Lu et al., 2023), of which (Abels et al., 2019; Lu et al., 2023) are LS methods and they care more about discovering all Pareto optimal policies rather than the similarity between the weight input and the resulted value. (Chen al., 2019) employs setting most similar objective for solutions aligned with the preference directions. However, (Xu et al., 2020) reported that (Chen et al., 2019) has suboptimal performance in practice. This might be due to the oscillation and stagnation issue inherent in the TS approach, as noted by (Mahapatra& & Rajan, 2021). In particular, for MORI, which is sensitive to stochasticity and conflicinng gradients, the convergence of TS can be problematic. Our baseline implementation of EPO with a small constraint threshold can be considered as a version of TS tailored to the MORL setting, designed to mitigate oscillation when near preference direction. recent MORL algorithm (Basaklar et al., 2023) incorporates cosine similarity for preference aligmment but it is designed specifically for off off-polley value- based RL and its performance relies heavily on the HER (Andrychowicz et al., 2017) technique. In contrast, our PCRL with PreCo is a broader MORL framework compatible with both on-policy and off-policy RI., capable of learning quality policies without HER. Nonetheless, HER can still be integrated into PCRL with PreCo in off-policy settings to enhance sample efficiency. While not targeting exact preference alignment, (Lu et al., 2023) addressed LS limitations by adding a concave augmentation term to the reward, transforming the original Pareto front into a strictly convex one. However, this introduces information loss, making the approach sensitive to the augmentation term's magnitude. Their implementation is limited to SAC, using policy entropy as the augmentation term. Compared (Lu et al., 2023), our method possesses the theoretical advantages of no information loss and exact preference alignment, which can be empirically demonstrated by our additional experiment in Appendix B.3. Because of the memory memory-efficient design of our PCRL with PreCo, they are also suitable for multi-objective fintuning of Large language models, like Nguyen et al. (2024); Yang et al. (2024a).
PCRL since they optimize Tchebycheff Scalarized (TS) (Ehrgott, 2005)
Other RL paradigms employing meta-policies include Goal-Conditioned RL (GCRL) (Sekar et al., 2020; Yang et al., 2022; Liu et al., 2022) and skill- RL (SBRL) (Nam et al., 2022; Lee et al., 2019). GCRL is controlled by an additional input of target state that aims to reach. SBRL is conditioned by a skill latent z that often has lower dimension than the state for specific primitive skill. Similar to SBRL, the skill learning methods of Unsupervised Reinforcement Learning (Eysenbach et al., 2019; Hansen et al., 2020) learns skills without external task rewards by optimizing Mutual Information Skill Learning (MISL) (Eysenbach et al., 2022; Yang et al., 2024b) objective [(s; 2) = (s)) H (s|z). Maximizing I(s;2) encourages the state space coverage to be high and the state distribution to be certain when controlled by skill z. The concept of Preference Control (PC) has a resemblance to optimizing =s;;))= = H()) - (s) [s]z). The purpose of PCRL.
can also be interpreted as optimizing (lu;p) = (())- - H (vlp) to encourage diverse values on the Pareto front and the distribution of the values needs to be controlled by preference p.
A.2. Multi-objective optimization
We have already introduced PMTL and EPO (Lin et al., 2019; Mahapatra & Rajan, 2020) that could find preference-specific solutions and CAGrad, SDMGrad (Liu et al., 2021; Xiao et al., 2023) that optimize for the average objective but can deal with conflicting gradients. (Désidéri, 2009; Xiao al., 2023) has the most similarity to our proposed PreCo because they all solve a min norm problem like 6 for gradient manipulation. The advantage of our PreCo is not only like SDMGrad, which can provably deal with stochastic gradients, but also can follow preference like EPO. Our theoretical analysis of PreCo is based on some results from (Xiao et al., 2023), but incorporating the similarity gradient from Yi makes it more complicated and novel. Li et al. (2025) is a method large objective numbers, which could also inspire future MORI. design for simultaneously achieving a large number of tasks.
A.3. Training schemes
We uniformly sample p for every episode during training. Techniques from curriculum reinforcement learning (Narvekar et al., 2020; Portelas et al., 2020) can also potentially improve the training of PCRL by using a progressing p preference distribution instead of uniform p E P.
13

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Baselines
obj
Similarity obj
Conflitt-vvidance
Characteristic
<
Yang et (2019);
(2020); Alegre et (2023) are implementation level modifications It is used as an ablation for PreCo with LS regularization instead of similarity regularization.
SDMgrad
CAPQI.
Its reward augmentation can cause information loss of the original problem directly adds cosine similarity in the objectives without considering potential gradient conflict
PDMORL
EPO
Similar to Tehebychoff Scalarization (TS), but it addresses the oscillation issuc of TS.
CAGrad
Only convergence guarantee with average gradient regularization. Our progosed algorithm specifically for preference controllable MORL with similarit
implemented
similarity
PreCo
Table 3: PCRL with advanced MOO algorithms outperforms SOTA MORL methods in Fruit-tree environment. PDMORL
results are means reported in Basaklar et al. (2023).
B. Implementation of baseline methods
We modify existing MOO algorithms EPO (Mahapatra & Rajan, 2020), CAGrad (Liu et al., 2021), and SDMGrad (Xiao al., 2023) for our proposed PCRL scheme and use them baselines for our proposed PreCo algorithm. In this section, explain how these existing baseline MOO algorithms work. Appendix E offers more details for pratical implementation.
B.1. Linear scalarization preference control
Linear Scalarization (LS) For preference p conditioned policy Tp- it is updated by d V Equivalently, it can be d= p=pp"ve Vo"r, which means to linearly combine the objective gradients with coefficient equal to the preference p.
SDMGrad
Similar to PreCo that needs to solve the min-norm problem (6) for update direction, Our implementation of
SDMGrad solves:
min V 0 PP VT FF AV "g tTD
Xp"VV,
V.,, 0=1
(14)
d
(15)
where w" is the solution for problem (14). The update direction is for Tp is d. We can see that this SDMGrad implementation and our proposed PreCo differ only the term multiplied by X. SDMGrad uses LS gradient for preference alignment while PreCo uses gradient of our proposed similarity function V. This why SDMGrad can be used as a case for ablation study of our method.
B.2. Similarity preference control
Exact Pareto Optimal (EPO) MOO methods such as PMTL (Lin al., 2019) and EPO (Mahapatra Rajan, 2020) apply similarity constraints and have two modes for situations of low and high similarity. Based on this idea, we implement the EPO baseline as: When similarity is low, only similarity gradients V., V (ww 6*r) will be used for update. When similarity is high enough, a common ascent direction calculated by MGDA (Désidéri, 2009) is used for update.
d V xp V (w, 6))),
if T (w, 6TT) C, if (w, )TT) S E,
(16)
d = vJ J arg min V o
@rrw|l,
(17)
where is threshold of similarity and y can be cosine similarity or our proposed V. Equation (17) is the min-norm update from (Désidéri, 2009), which is equivalent to finding a common ascent direction that maximizes the least improvement among the objectives:
d = arg max mix V pd
(18)
Because most of the time during training, similarity is not high enough and only similarity gradients are applied in updates, this implementation of EPO can also be seen as an implementation of a relaxed Tchebycheff Scalarization, which avoids gradient oscillation as claimed in (Mahapatra & Rajan, 2020).
Conffict-Averse Gradient (CAGrad) CAGrad (Liu et al., 2021) tries to find a common ascent direction that is not too far from the average gradient. In our setting, we modify be a common ascent direction not is not too far from the similarity
14

---

Preference Controllable RL with Advanced Multi-Objective Optimization
gradient.
arg max min VT rd Sp
s.t. ||d TT V(u, 0=2) S cV (we d*x),
(19)
where c E E implementation might not apply to the convergence analysis (Liu al., 2021). However, as shown by the empirical results, works in practice for our PCRL scheme.
E R O <
13 is a constraint constant to keep d close to the similarity gradient V (w, 8"r). This
B.3. Implementation of SOTA MORL methods
Implementation of CAPQL (Lu et al., 2023) has tried to address LS limitations by adding a concave augmentation term to the reward, transforming the original Pareto front into a strictly convex one. Then for this "more convex" new problem, LS can find more optimal solutions. Their implementation only included SAC (Haarnoja et al., 2018), as the entropy maximization in SAC serves the reward augmentation. To ensure fair comparison independent of settings, code-level implementations, and algorithmic techniques (such as HER (Andrychowicz et al., 2017)), we modified our original LS with PPO into "maximum entropy PPO". The modified multi-objective advantages are:
( a) = R + E 6(s,p)
(20)
where R represents the vector of multi-objectiive episodic returns, E denotes the of future policy entropies in the sampled episode, and ö(s, p) is the multi-objective vector value conditioned on preference p, approximating both expected returns and entropies:
min R - E ((s.p)
(21)
With these modifications, our modified "maximum entropy multi-objective PPO" with Linear Scalarization(LS) is optimizing the concave augumented objective in (q.( from Lu et al. (2023).
To showcase the advantage of our PCRL (Ours) framework with similarity the 'simple but hard' fruit-tree environment. is simple for RL due to small discrete state and action spaces but challenging for MORL with 6 objectives and a non-strictly convex Pareto front. This comparison isolates MORL performance from lower-level RL factors, directly highlighting method's strengths.
methods EPO and PreCo (ours), we test in
a/Method
Hyper volume(1e3)
Cosine Similarity
0 (LS)
5.74 + 0.88
0.718 + 0.040
0.01
5.95 + 1.12
0.722 + 0.006
0.05
5.18 + 0.36
0.718 + 0.040
0.10
1.75 + 1.35
0.633 + 0.141
EPO
14.97 2.29
0.77 + 0.03
PreCo(ours)
15.61 + 0.75
0.78 0.03
Table 4: HV and CS performance in 6D Fruit-tree environment, the HV value has unit of 1e The comparison is between PCRL (Ours) framework with similarity- -based methods such as EPO and PreCo (Ours) and LS with different strength of concave augmentation from (Lu et al., 2023)
The results in Table 4 show when the augmentation strength a 0.01, the performance of CAPQL-modified PPO is marginally better than the original LS (a 0), but still significantly worse than similarity- -ased methods (EPO, PreCo(ours)). Larger values lead to performance drops. This result aligns with Remark 5 and Figure in (Lu et al., 2023), which highlights that such augmentation can cause information loss in the original problem, and excessive augmentation results in performance degradation. In contrast, our method has the theoretical advantage of overcoming the LS limitation without any reward augmentation, thus avoiding information loss from the original problem.
Implementation of PDMORL (Basaklar et al., 2023) We implement PDMORL using its official GitHub codebase.
Unfortunately, we encountered several issues with their implementation. It is computationally inefficient, requiring 14 hours to complete training in the Fruit- environment and over 70 hours for MO-Hopper, even when using an NVIDIA A100
15

---

Preference Controllable RL with Advanced Multi-Objective Optimization
GPU and 72 CPU cores. In contrast, PCRL implementation completes training in just 20 minutes for Fruit-Tree and hours for MO-Hopper.
Additionally, while we successfully reproduced the Fruit-Tree results reported in their paper, we were unable to reproduce the MO-Hopper results. The training appears to fail due to the "underrepresentation" problem mentioned in their paper. This aligns with the intuition that directly adding cosine similarity to the original objectives, as done PDMORL, can introduce gradient confficts, especially since lacks conflict-avoidance mechanisms commonly used in MOO algorithms.
Due to these computational inefficiencies and reproducibility issues, we are unable to compare PDMORL for MO-Hopper and MO-Ant.
C. Additional Experimental results
In this section, we present experimental results in high-dimensional continuous environments and demonstrate the compati- bility our method with low-level sample efficiency modifications.
C.1. Continuous Environments
As discussed Section 5, typical robotic control MORL environments, such as MO-Hopper and MO-Ant, present challenges for sample-efficien RL but often feature simple and strictly convex Pareto fronts. As a result, these environments fail to fully reveal the limitations of common I.S-based MORL methods. Consequently, the advantages of our PCRL and PreCo less pronounced in these settings.
To provide more comprehensive evaluation, introduce additional metric Overall Non-dominated Ratio (ONR)-u to demonstrate that, while our method may not show clear advantage in terms of HV and CS, it still outperforms existing approaches in ONR.
In our setting, the ONR of one method is defined as:
ON R the number of this method's s samples that are non-dominated by all samples of all methods total sample number of this method
(22)
We attempted to implement PDMORL using its official GitHub codebase but encountered reproducibility issues and computational inefficiencies mentioned in Appendix B.3. As result, were unable to compare PDMORL in the continuous control environments.
1.1.1. MO-HOPPER
The MO-Hopper is a classic continuous robotic control environment, with one objective rewards for going forward in the x-axis, and the other rewards for jumping high in the z-axis as shown in Fig. 7a. The two objectives are less symmetric than MO-Ant and there a clear trade-off in directions. The HV and CS results are shown in Fig. 7. LS and CAPQL have
PrcCe
S00gp0 CAPEL 6P0 0NU *orrah
50gg
MO-Hopper
(b) Pareto Front
(c) Hypervolume
Cosine Similarity
(c) Overall Non-dominated Ratio
Figure 7: Optimality (HV) and Controllability (CS) of MO-Hopper.
relatively much better performance for MO-Hopper than Fruit-Tree and MO-Reacher. This is because the Pareto front of MO-Hopper a simple 2-D front that is already strictly convex, adding a concave term will not cause an unacceptable information loss of the original problem. Our method demonstrated superior performance in HV, while its CS was only slightly lower than that of CAPQL. CAPQL adds a concave term the reward to make the augmented Pareto more strictly
16

---

Preference Controllable RL with Advanced Multi-Objective Optimization
convex, resulting in better CS but the augmentation caused information loss of the original problem, thus causing suboptimal HV and ONR. Unlike EPO, which employs hard constraints similarity, our proposed PreCo utilizes soft constraints. This could be the reason why PreCo can sacrifice small degree of controllability for significant enhancement in optimality. The hopper has to be able to jump before jumping forward, this is why objective 2 is higher than objective for most methods. As a result, the asymmetric objectives make the discovered Pareto front not as symmetric as that of MO-Ant, We have a calibration approach to further improve controllability in Appendix H.
C.1.2. MO-ANT
The MO-Ant higher dimensional continuous control environment challenging for RL but not necessarily hard for MORL. The reward is 2-dimensional, with for x-velocity and one for y-velocity. Although the robotic agent has more complex dynamics, the objectives appear be very similar since both involve the movement of the agent, making it relatively easy for preference control. As shown in Fig. 8, the Pareto front has an intuitive convex shape. The similarity approaches such as PreCo and EPO have high CS metrics over 0.98. This indicates that our proposed PCRL scheme with similarity optimization scalable for higher dimensional environments and has better preference controllability than LS methods such as LS and SDMgrad. They also have much better ONRs.
Yondec by
330
1125
RICE
Chjactie Kuplecity
CNG
KC
THI 200 1920 1050
40Mgal CAME B/C
DVBB
D602 K0x
11
10Pg DPC
NO
Other
(a) MO-Ant
(b) Pareto Front
(c) Hypervolume
(d) Cosine Similarity
(e) Overall Non dominated Ratio
Figure 8: Optimality (HV) and Controllability (CS) of MO-Ant. PreCo (ours) achieves the best CS. Though being the second best on HV, it achieves the widest spread the Pareto front in (d). In (b)-(c), methods of blue bars are based on linear scalarization, while methods of orange bars optimize the similarity.
The relative performance of CAPQL drops from lower dimensional Hopper to higher dimensional Ant. The potential reason could be that it requires learning an actor outputting not only the means but also the log standard deviations of the action distributions, which is equivalent to learning with an action space of double the original size. As a result, it could be less sample efficient than other methods that do not need to learn variance or standard deviation, especially for the high dimensional Ant environment. In contrast, LS has competitive HV and CS this environment. One reason that the symmetric and strictly convex Pareto front does not expose the limitations of LS, and another reason could be that its simplicity does not complicate the learning of lower-level robotic control.
C.2. Compatiblity with lower-level modifications
Mainstream MORL methods have focused on optimizing the LS objective, often incorporating different low-level modifica- tions to improve sample efficiency. As summarized in Table Q-envelope (Yang al., 2019) uses HER (Andrychowicz al., 2017), Q-pensive (Hung al., 2023) combines Q-buffer and HER.
Table 5: Summary of methods and their low-level modifications
Method
Modifications
Q-envelope
HER
Q-pensive
Q-buffer, HER
Intuitively, HER allows reusing experiences from one preference for others, while Q-buffer maintains Q-value snapshots that optimistically estimate values to leverage this stored information. Since Q-pensieve essentially combines Q-envelope with Q-buffer, we compare the performance of our PreCo with HER and Q-buffer against Q-pensieve. The results are shown Table 6.
17

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Table 6: Comparison of Q-Pensieve and PreCo (ours) on Fruit-Tree and MO-Reacher environments.
Environment
Metric
Q-Pensieve
PreCo (Ours)
HV
6.86 + 0.01
15.61 0.75
Fruit-Tree
CS
0.72 0.01
0.78 0.03
HV
47.36 + 0.73
49.55 + 0.74
MO-Reacher
CS
0.89 + 0.02
0.91 0.02
Our PreCo with HER and Q-buffer consistently outperforms Q-Pensieve, the state-of-the-art LS method, across environments and metrics, achieving higher HV and CS scores.
In non-strictly convex environments such Fruit- Tree, the advantage is particularly pronounced, Q-Pensieve's reliance on the LS objective limits its ability to discover broad set of optimal policies. In contrast, in strictly convex settings like Reacher, the performance gap narrows, as expected.
These results further underscore PreCo's superiority over -S-based methods and demonstrate its compatibility with low-level sample efficiency enhancements.
D. Experimental details
The preferences are p P with resolution of 0.1 for each dimension.
For instance, in 3-D cases, these preferences include
[0, 0, 1), [0, 0.1, ..... [0, 1, 0), (011, 0,99) [0.9,011,
[1,0,0], (1,00,
with quantity of 66. There are 286 test preferences for 4-D, 1001 for 5-D, and 3003 for 6-D.
During training, the preferences were sampled uniformly from the convex coefficient P, making the probability of sampling an exact preference nearly zero. Therefore, high CS metric in test time means the ability generalize unseen preferences.
We run 5 seeds for each environment setting, and for each run, we select the best-performing agent as a candidate for testing.
The results are presented as the mean and standard deviation of the 5 candidates.
D.1. MO-Ant
The exact data for bar charts in Fig. 8 is shown in Table 7.
LS
SDMgrad
CAPQL
EPO
CAGrad
PreCO
HV(#1e9)
6.81 + 0.24
3.67 1.76
1.53 0.22
6.75 0.08
6.79 + 0.20
6.85 0.21
CS
0.988 + 0.012
0.937 0.014
0.976 0.008
0.989 + 0.004
0.988 +'t 0.0004
0.990 + 0.004
Table 7: HV and CS performance in MO-Ant environment, the HV value has unit of 1
For MO-Ant, both SDMgrad and PreCo have À that increase linearly with each update from 1 to 5. EPO has a constraint threshold of 3e 4 for cosine similarity V., which is very small, making it comparable to Tchebycheff Scalarization and also similar to PreCo with large constant X; = >> 1. CAgrad has constraint c 0.2. The definitions of e and can be found in Appendix B.
D.2. MO-Hopper
The exact data for bar charts in Fig. 7 is shown in Table 9.
For MO-Hopper, both SDMgrad and PreCo have À increasing linearly with every update from 3 to 11. EPO has a constraint threshold of 3e for cosine similarity V. CAgrad has constraint c 0.1.
18

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Table 8: Hyper-parameters settings MO-Ant.
Hyper-parameter
Value
Discount (7)
0.99
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate for networks
3 x 10-4
Number of hidden layers for all networks
3
Number of hidden units per layer Activation function
256
ReLU
Batch size
256
Buffer Size
x 106
Starting timesteps
2.5 x 103
Gradient clipping
False Noise (0, 0.12)
Exploration method
Noise distribution
Noise clipping limit
0.5
Policy frequency (delay)
2
Target network update rate (T)
5 x 10-3
Maximum episode timesteps
500
Peference sampling
every new episode untill total steps is reached
Evaluation episodes for each test preference
10
LS
SDMgrad
CAPQL
EPO
CAGrad
PreCO
HV((1e") CS
2.63 + 0.12
2.24 = 0.13
2.55 + 0.09
2.53 + 0.28
0.94 0.932 + 0.024
2.67 + 0.26
0.933 + 0.007
0.922 + 0.018
0.978 + 0.005
0.963 + 0.023
0.957 + 0.011
Table 9: HV and CS performance in MO-hopper environment, the HV value has a unit of 1e
D.3. MO-Reacher
The exact data for Fig. 5 is shown in Table 11.
For MO-Reacher, both SDMgrad and PreCo have X increasing linearly with every update from 10 20. EPO has constraint threshold of = 3e 4 for cosine similarity V which is very small, making it comparable to Tchebycheff Scalarization and also similar to PreCo with a large constant X₂ X >>> 1. CAgrad employs a constraint constant of c= 0.1.
D.4. Fruit Tree
For 3-D reward, most runs of LS only learn a very limited number of values like [4.71, 5.39, 5.40) and the values SDMGrad for most test preferences lie at [4.01, 7.17, 1.47J. They, as the LS approach, discover much fewer Pareto optimal policies than methods of the similarity approach, which have one value for each test preference. This shows the limitation of linear scalarization methods. In theory, I.S methods have the potential discover Pareto optimal policies for MORL (Lu et al., 2023). However, in practice, this often not the case. Possible reasons could be the numerical instability inherent in deep RL, limitations of model capacity, and the fact that the value space is usually not strictly convex.
E. Practical implementation of the update procedure
F.1. Algorithm framework
Our goal is to train a single agent that can be conditioned on different performance preferences and 0-shot adapt to user preference at test time. During training, we need to uniformly sample preferences from P. and let the agent learn to find Pareto optimal policies with values aligned to p. The procedure is shown Algorithm 2.
19

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Table 10: Hyper paree--rrameterss settings MO-Hopper.
Hyper-parameter
Value
Discount (7)
0.99
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate for networks
3 x 10-4
Number of hidden layers for all networks
3
Number of hidden units per layer Activation function
256 ReLU
Batch size
256
Buffer Size
1 x 106
Starting timesteps
2.5 x 10a
Gradient clipping
False Noise
Exploration method
Noise distribution
(00 0.13)
Noise clipping limit
0.5
Policy frequency (dclay)
2
Target network update rate (r)
5 10-3
Maximum episode timesteps
500
Peference sampling
every new episode untill max total steps is reached
Evaluation episodes for each test preference
10
Algorithm 2 PCRL with PreCo update
1: Initialize:
Buffer.
N: Number of training samples for p.
E: Number of training episodes for every p sample,
Ne: Preference-condittonee actor model.
Qs for DDPG/TD3 or V for A3C/PPO: Preference--onndtiioned critic model with m-dimensional output, where m is the objective number.
2: for n = 0, 1,....... 1 do
3: Sample preference p E P do
4: 0,
5: Rollout with policy Th (-(-1-p))
6: Store transitions (s. a, in B
7: end for
8: Update Qp or V by minimizing TD error for every objective.
9:
Estimate policy-level gradient V ; by 24/26 for TD3 or Eq.30/37 for PPO.
10:
V thH V (p, 6*T) Get policy-level update direction d' by solving E with VT Tp Th and x, T (p, )*v)
Estimate similarity gradient V (' TT'
11:
12:
Update 0 by solving 25.25 for TD3 or Eq.32/34 for PPO with d
end for
20

---

Preference Controllable RL with Advanced Multi-Objective Optimization
random
LS
SDMgrad
CAPQL
PDMORL
EPO
CAGrad
PreCO
IV CS
13.47
15.66 1. 8.03
20.37 1. 0.37
20.56 0.47
22.69 1. 0.15
9.87 1. 3.11
13.46 1 9.71
33.11 1. 6.29
0.758
0.652 0.030
0.761
0.001
0.760 + 0.001
0.754 0.002
0.845 + 0.078
0.760 + 0.002
0.906 0.002
Table 11: HV and CS performance in MO-reacher environment, the HV value has a unit of le
Table 12: Hyper-parameters settings MO-Reacher.
Hyper-parameter
Value
Discount (7)
0.99
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate for networks
3 x 10-4
Number of hidden layers for all networks
3
Number of hidden units per layer
256
Activation function
ReLU 250
Batch size
Gradient clipping
False
Exploration method
Policy Entropy
Entropy Coefficient epsilon-clip for PPO Epochs per PPO update Timesteps every update
0.001
0.001
3
100
Maximum episode timesteps
250
Number of episodes per preference sample
40
Number of preference samples (for 4D reward)
600
Evaluation episode for each test preference
10
E.2. Policy-level gradient
Solving the min-norm problem (6) with parameter-level gradients V at every gradient update can be memory and computationally expensive when (8) is large. Video game playing agents like AlphaZero (Silver al., 2018) and AlphaS- tar (Vinyals al., 2019) can have millions of model parameters. Besides, Large models with billions of model parameters have become very common with recent developments in Large Language Models (LLMs) (Zhao et al., 2023; Minaee et al., 2024). To circumvent this issue, we suggest solving the min-norm problem (6) before the gradient propagates to the model parameter 0. Therefore, ideally, we want to solve the min-norm problem with gradients at the policy-level V vx, 0", which has only a size of batch size B of hundreds at each update: In practice for deep RL implementations, as shown by Fig. 9, V R can also be replaced by the gradients of the value " with respect to the policy model outputs, such as V for logits of categorical distribution policies and V Vpp.op, hT for means and standard deviations of diagonal Gaussian distribution policies.
1. CONTINUOUS ACTION SPACE
Value-based methods like TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018) are often used for continuous action spaces. To avoid computing min-norm with parameter gradient
VorT VoQ a, a) (ppp))
(23)
We look at their policy formulations. Their policy x())s,p) is often Gaussian or squashed Gaussian distribution with parameters mean (s) and log standard deviation log (s(s). We denote a distribution parameter vector Pp with (s) = (s), ((s,, log (s)]T and we can get
T E(V PD a(s,a,)))
(24)
For each update, the size of each objective gradient V GT is 2|A| x B, where B is the batch size and p has a size of
21

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Table 13: Hyper-parameters settings Fruit-tree.
Hyper-parameter
Value
Discount (7)
0.99
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate for networks
3 x 10 10-4
Number of hidden layers for all networks
3
Number of hidden units per layer
256
Activation function
ReLU
Batch size
100
Gradient clipping
False
Exploration method
Policy Entropy
Entropy Coefficient epsilon-clip for PPO Epochs per PPO update Timesteps every update Maximum episode timesteps
0.001
0.001
3
100
100
Number of episodes per preference sample
20
Number of preference samples (for 4D reward)
3000
Evaluation episode for each test preference
10
Policy level
Parameter level
Policy distribution probability density Eq.29 Eq.31
Actor Network parameters
Policy distribution parameters Eq.24 Eq.35
Objectives Eq.4 / Eq.5
Eq,23/ 28.28/ Eq.30
Action sample Eq
Size from thousands to billions
Size of O(B). B is batch size
Figure 9: Backward path of policy update. we can see that the gradient from objectives to the parameters of the actor network first backpropagate through the probability density of action distribution (for policy-based methods such as A3C/PPO) action sample (for value-based methods such as DDPG/TD3), then propagate through the distribution parameters of policies such as the logits for categorical distribution p. Σ for Gaussian distributions. We consider these the pelicy-level gradients. They often have size of O (BB), where B is the batch size. Since B is often limited to few hundreds, the size of a policy-level gradient would be much smaller than the size of the neural network parameter.
22

---

Preference Controllable RL with Advanced Multi-Objective Optimization
2|. This means that p (s p) could have much lower dimension than Veth(s, p), thus increasing the memory and computational efficiency.
After getting the update direction d for Pp by solving the min norm problem (6) with V. Pp h", we update model parameter θ by solving
max { VoPp s.t. lp - Potal12 2 < 6),
0
which is trust region formulation that updates p in the direction of d while keeping in a local region where is valid. simple and practical implementation for parameter update can be as follows:
max [(0 = Es.a (clip(pp(s, a), Pp(s, a) - e, Pp(s. a) + e)d(s,
(25)
0
The update of every entry of x clipped to e, so || Tell2 12 S VB * x2 where B is the batch size.
For more expressive models such diffusion models (Wang et al., 2023) or normalizing flows (Brahmanage et al., 2023), the mean and covariance gradients would not be adequate. We can instead use the gradient of action samples policy-level gradients and we get update direction
a(s,a) = V. as)
(26)
for every (s, a) sample in the batch. Then we can perform min- min-normm with and update the more expressive policy networks by reparameterization techniques.
E.2.2. DISCRETE ACTION SPACE
Policy-based methods like A3C (Mnih et al., 2016) and PPO (Schulman et al., 2017) are often used for discrete action spaces. We can approximate the multi-objective value function (**(s) by function ú(s,p) p) that takes and as inputs, sample the episodic returns as vector R. and calculate the multi-objective advantage function as
A a) o(s.p)
(27)
Then, the policy gradient in the model parameter space is
Vot(als,p) A A(s.a.p) x(als,p)
V VOO =
(28)
And for gradient at the policy space d n. (s, p), we have
d(s,a) 1 as..
(29)
=(als,p)
for very a) sample. When using policy optimization methods like PPO/TRPO they are
v00T = Von A(s,a,p)
(30)
Tota (a 8, p))
d(s, a) = Jota (als,p) as,...))
(31)
In practical situations, every update, the size of each objective gradient 6s, P) is the batch size B, and the min norm problem (6) be performed with gradients of batch size B, which could be much smaller than the parameter size of deep neural networks, especially when implemented for large language models.
After getting the update direction d = V n. u(s, p) for Tp, we optimize model parameters by
max (d Voup s.l. ||TT Tp.olal2 ll2 < <6}.
(32)
which is a trust region formulation that updates Tp in the direction of d while keeping in a local region where d is valid. This can be practically implemented by an objective as follows:
[(0) Es.a (clip(To (a)s, p). vow (als, (s.p) p) e, now (als. + c)d(s) (s,
(33)
23

---

Preference Controllable RL with Advanced Multi-Objective Optimization
The update of every entry of is clipped to e, || rella S v e² 6, where B is the batch size.
Trust region formulation with KL-divergence could be more suitable for categorical distribution Tp. another formulation of parameter update could be
max {d VoTp 8.1. p.op < <6}
(34)
Whether a KL divergence trust region is theoretically compatible with the solution d of the min norm problem (6) will be further researched in our future work.
One potential issue of V =. (ss ()) is that (Tp+aVe ((8, p)) may not be in the probability simplex. As a result, projecting it back onto the probability simplex could cause to deviate from the intended update direction. Since policies for discrete action spaces are often categorical distributions, one way to avoid this issue to consider the gradient of lp. which denotes the logits for policy <<-|;, p) conditioned preference p, and (s)) are the logits for x (-188, p). The logits do not have the constraint to be in the probability simplex.
1i Tp(s)
(35)
( p) (((alls,p), x x(a2)s,)) pa)ss - 1, 1,..,, [(1))
where VI, (p)) the s-th entry of the jacobian Vip Tp. TT, and
() (als. - -.,...?
a vector of action space size [A].
Then, get
VIp (s)
n faass, 1, p) (ala(a s.
(36)
A(s, a, ()[(ss p), (aglss p)..., <alls, p) - 1,.., (a
where VI, (*)), of size m x A, is the index [3, s for Vi, tr tensor, of size m |S| |AA|.
In every update, the size of the objective gradient Vi, ( s) for the objective has size of x |A]. For large language models, the action space could be the vocabulary size of tens of thousands, so B |A| could be in millions, but is still much smaller than the parameter size that is often in billions. Moreover, for large action spaces, (alssp) T[alls, p) could be much smaller than x[a|s, p). so Vi, ( as can be approximated by
Talss p) π (a(s,
V (5
A(s, a) A(s,a,p).
(37)
This approximation of Vi, is what implemented to replace V 6r in the min norm problem (6) to avoid solving min norm with large parameter gradient Vo0ūP, The model parameters are updated by solving (32) using Vi, Rhr d.
F. More details about the similarity objective
F.I. About proposed similarity function
For two objective cases, when p = [0.5, 0.5]. (p, v) is shown in Fig. 10. The x-axis is the first element of and y-axis is the second element of v. The z-axis the value of L (p, v).
We that the similarity is maximized 0 only when un
"1 It is also smooth proved by Lemma 4.1.
po
F.2. Similarity objective design for better theoretical properties
Theorem 4.2 requires the similarity gradient to be both Lipschitz continuous and convex combinations of the objective gradients. Which formally means that for an similarity objective V V((,.))
(p, E W,
24

---

Preference Controllable RL with Advanced Multi-Objectio Optimization
Figure 10: (p, -) when p = [0.5, 0.5]*
which can satisfied by V, v) max; =.P because V, V (p, v) will be when p and are perfectly aligned. Moreover, we can not directly normalize V. (p, v) by dividing || V V.. v) because this will make the normalized gradient not Lipschitz continuous (the gradient changes drastically when passes the direction of p).
Our hints design such similarity function V (p. follows:
Its similarity gradient V. V (p, (p, v) could get close to when has a high similarity to the preference p:
(p v) should be close to the normalized gradient v.V(p.) when the similarity is low.
One possible design is:
1-CS(p..)
V. V.p(p. (p,
V. (p, = exp
p + (1 exxp
(38)
where CS is the cosine similarity. In this design, V, Vp, v) is always convex combination between p and so itself is always convex coefficient.
V. ((..))
V...p))
We have for u, v' # O:
V. V' (p,
V of V (p,
o')l
12' - 1/
exp
1-C8(p.'')
exp
1-CS(p.))
p+ 1 11-ea
exp
exp
1-05(par))
V... (1-exp exp
V., V(p.v')
1-CS(p.
V. VP.rjll
V (p,0)
10' - 21
1-C8(p.v' ? - -exp
1-(3(p.v)
1-C8(p.e')
1-C5(p.e)
exp
7
p
+
exp
- -exp
1/
m|
1-OSpp 9
1-08(pe)
(1 +
exp
exp
11
(39)
25

---

2xx(22/)) Preference Controllable RL with Advanced Multi-Objective Optimization
v-
which is equivalent to proving exp "
is Lipschitz continuous.
Let + 0 be fixed. Consider:
(0, p) 0 |
CS(u,p)
* 0.
Compute the gradient:
{0. {u.p) p) | 10
V.CS(u,p) = 1p |||||/|3 [10||pp
Bounding its norm:
/V,CS(U,))II 2 || (1lv)
|el| + || [pl)
(40)
|
Thus, for ||a|| 8> O, (VC p)ll 7. showing CS is Lipschitz continuous with constant 8-
Then for exp c.
let
1 - CS(U,P)
S(0) =exp
exp
7)
and we have
1
1 - CS(U,))
V.f
exp
(.
1
"
By Equation (40), we get
2
(2)
exp
(41)
1/||01
Then for || 2 8:
| 2exp(2/n)
170
Hence, ((v) Lipschitz continuous on ||w||| 2 with constant
1/0
Therefore, V', v) defined in Equation (38) is both Lipschitz continuous and is a convex coefficient when ||/||| 2 6, which is very common in practice with non-zero values.
G. MOO Toy Example
This is toy example used in SDMGrad (Xiao et al., 2023) to show that in MOO, our proposed PreCo can achieve better or comparable performance under stochastic settings. Besides, PreCo can find the Pareto optimal point optimizing the similarity function // (p..).
The two objectives L (x) and (z( shown in Fig. are defined on æ = (2), e R?,
(z) = (x)))1 (z and Lz(x) (=))g((=) (z()))2(z)
where the functions given by
fi fx) = max (tanh
fa(x) max (tanh((0..5x2).))
91 (x) log
(max(10.5(-))))) 7) tanh( a.n0(-0)),,
92(2:) log
(max(]0.5(=z] 3) tanh((z2) + 2). 0.000005)
hy (x) = ((-x)+
+ + 8)²) 20
h2(x)) = - - 8)2)/10 /10- 20.
26

---

Preference Controllable RL with Advanced Multi-Objective Optimization
-15
-10
-10
-10
-3
10
10
(a) Mean objective
(b) Objective 1
(c) Objective 2
(d) MGDA
(c) GD
(f) CAGrad
(g) SDMGrad
(h) PreCo [0.4, 0.6)
(i PreCo [0.9, 0.1]
Figure 11: A two-objective toy example.
-5-
(a) p [0.4, 0.6]
(b) p 009,
Figure 12: Plots showing the Pareto front: The x-axis L1 and the y-axis is L2. The blue line is the direction of the preference, for (a) it is p [0.4, 0.6). and for (b) is p [0.9, 0.1]. All three initial points converged to the Pareto optimal point that intersects with the line of p direction.
Initializations points from {(-8.5, 7.5). (-8.5, The optimization trajectories visualized in Fig. 11. The starting point of every trajectory in Fig. 11d-Fig. 11g is given by the symbol, and the color of every trajectory changes gradually from red to yellow. The gray horizontal line illustrates the Pareto front, and the symbol denotes the global optimum for the mean objective Lo = 0.5L, + 0. The setting is the same as in (Xiao et al., 2023) and all other methods except PreCo optimize for Lg- Zero-mean Gaussian noise is added to the gradient of each objective for all the methods except MGDA. Adam optimizer is adopted with learning rate of 0.002 and 70000 iterations for each run. can see that GD and CAGrad can fail to converge to the Pareto front certain circumstances. Only SDMGrad and our proposed PCGrad converge to the Pareto front in all cases. Notice that, the preference PreCo Fig. 1 is p [0.4, 0.6], shown in Fig. 12, point optimizing V (p, [L,, L2)"). In addition, Figs. li and 12b show for the case where
that it converges [0.9. 0. PreCo also updates to the preference specific Pareto optimal point.
Below is figure comparing PreCo with existing preference following MOO algorithms such as Tchebycheff scalariza- tion (Lin et al., (TS) (Ehrgott, 2005) and smooth Tchebycheff scalarization
In Fig. 13, PreCo first converges to the Pareto front, then goes up, it converges the preference desired solution.
27

---

Preference Controllable RL with Advanced Multi-Objective Optimization
TS
smoothTS
ProCo
-5
-5
-5
-10
-15
-20
-20
-20
-20-15-10
-20-13-10 -5 0
-20
-20-15--1
-5
0
10
10
10
10
-30
-10
-10
Figure 13: Plots showing the results of TS, SmoothTS and PreCo for preference p 0.91. Our proposed PreCo converges the preference-aligned Pareto front for all initialization.
H. Calibration
The reachable Pareto front for PCRL is often not the entire R" value space, and there are often gaps between the desired preferences and the values reached. To calibrate the possible misalignment between the input preference and the reached value a sample-efficient way, we employ a Gaussian process (GP) based method to model the relationship between the input of the desired preference and the actual values reached by the agent.
1.0
Mean
0.9
Sample
GP model
Estimated Objectives
0.8
Sample
Sample
0.7
Evaluation
Desired Preference
Input condition
Agent
Environment
Calibration
0.0
0.2
0.4
1.0
Figure 14: After training, due to general errors in deep learning or unreachable regions of the Pareto front, there could still be gaps between the actually reached objectives ratios and the desired preference. The calibration procedure obtains the reachable reachable Pareto front and modifies the desired preference into an input that results performance more aligned with the desired preference.
Figure 15: Example of GP regression for the (p. u sumples, this has objectives and shows the relation between the first element the value and the first element the preferenes. Each red
(po. " where po. PP the first elements p.
this
After training, there might still be an input p' with better (p, w=r') than S0 We want to find p' that solves maxp' E[ (p, vrrr) }] for any p. first uniformly sample the values whr reached by giving the agent preference in- put from {p Tm p 1}. then perform GP regression for the vT)) samples. As shown in Fig. 15, some samples provide Gaussian distribution of the mapping ((p) from p to Tr, Based the distribution of é, for desired preference p, we can find good input p' to solve
max E (pp)(()))
(42)
This procedure learns the reachable regions the agent and calibrates the desired preference into the best input for reaching the preference. Also, it is general and can be applied to any preference control approach. Here an empirical example for calibration:
28

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Figure 16: Pareto front before and after calibration.
3020
3000
2300
2300
2002
2000
1500
3500
503
2000
2200
2000
2500
3000
500
1030
1900
2000
2500
300
Discovered Pareto Front before cali-
(b) Resampled Pareto Front after calibra- tion
bration
The colored rays are preferences p. The points with the same color as the preference vector p are value u0 of preference conditioned policy Tp- The left plot is before calibration, p is directly used the input for тр. The right plot is after calibration, we know which regions can be reached, so preferences p are not all directions but reachable directions. Also, the input for Tp is p' by solving (42), resulting in higher similairty and the CS metric improved from 0.991 to 0.997.
I. Theoretical Proofs
L.I. Proof for Lemma 4.1
Lemma 4.1. The similarity function (p. is (1 max; under Assumption 4.1 and Assumption 4.3.
[pl p.
-Lipschitz smooth and ga is Lipschitz continuous
Proof. By definition:
V:
(p, = - max
(43)
2
p:
and
U,
V. V. == max - p = d(v.p)p-
(44)
p:
where d(v.p)
denotes max; m. p,
we have
V))( V.. = ||d(u, p)p - v - d(v', p)p + v'll ((((u,))p d(v', p)p (r r')l S dd(u.p)p d(v', p) || + || o'll
(45)
Without loss of generality, we first consider the case where d(v, p) d(v'' p) 2 0. denote i, = arg max
[d(u, p)p - -(w'', p)pll + || - 0'll = (d(r p) d(v', p) | |/0 ='1l
1%
v/_ )|p|| + ||0
Pi.
Pio
|p]|
+ |
l'll
(46)
Pi.
max
|
+
|/
o'
lp:
|p|||
(1
max
)|
p
29

---

Preference Controllable RL with Advanced Multi-Objective Optimization
The first inequality is because o is optimal for but not necessarily for v'. The where d(u, p) d( .p) can be
proved by the same procedure by denote " = arg max; p.
vi
then
pdp - d(v', p + 11e - w'll = (d(v', p) c(u,p))
p + |/ -''l
2/2
Vg
)|pp|
||0 o'll
Pig
Ptc
|pl|
||00
|| + ||0 1'll
(47)
Pr
|pl|
max
|
l' + 100 - 0'1l
P:
|pl|
$ (1 +
max
)| - r'll.
P
Therefore, have proven
|ll| pal
| (P. V. (pp w'll s (1+ max
|| - 0'll
(48)
and the similarity function for preference p is 1 + max;
| Ip:
Lipschitz smooth.
Next, we prove that 9. (TTp) = V. (p, )T is Lipschitz continuous.
We have:
= -1)"))),
= [I(G(z) - G(v)) V. + G(y) (V) V.V(p,
(49)
- v.
where the incquality is by Cauchy-Schward Since under Assumption 4.1 and Assumption 4.3, |[G S C, and [l9((x) - -)((t)l 4,11 yll. and
|pl| p.
|pl ppl
V V. V. uz)ll (1 max
I|
b8ll (1 max
|||4||| vll.
(50)
where = 12,,2,,.. lma" is the vector of Lipschitz constants of all objectives. Denoting Lm = (1 + max;
||l|| [p:]
||4], we have
(19.(x)- (x)ll - (V..
Σ ln + Cy 1|x- vll
(51)
By definition in Equation (44):
v.2 w²1 S max max
V V7 (p, x²] =
max
pl.
(52)
p;
p,
where the inequality because the values of should be no larger than the maximum values for the objectives. Denoting
Lp max max
pl.
(53)
p,
we have
m
(l.. - gx(ll S Lp M ha + CyLm
|| - 1yll
(54)
and we define L, = ELp 4.1 + CyLm). We have proven (-) is to be L,-Lsshittz continuous. Therefore, both claims of this lemma have been proven.
30

---

Preference Controllable RL with Advanced Multi-Objective Optimization
1.2. Proof for Theorem 4.1 and 4.2
Before proving Theorem 4.2, we prove Lemma I.1 for the requirements to use the proof idea in (Xiao al., 2023) for their theorem.3 and obtain Lemma 1.2. To be consistent with their proof, consider minimizing the negative value and similarity with gradient descent.
Lemma 1.1. Under the Assumptions 4.1-4.3, we have
(T - Lp E (Tpc.))).)))E) (T(TT.)) mg
(55)
ElIG(TT.:: + (T.
[ ()) 8(mo² + 02)2 8 + C2)2
(56)
C
(T):) + (Tp( s 4mo + 40g 4x L + 122
(57)
C2
B[I(G(Tpp))w + (xpp)())))) G(Tp.))lll - W1+1ll <
2(1 + LpA) + Cg)²
(58)
Proof. Under the Assumptions 4.1.,.3, by (52) and (53), we have
11., (TTx)) < V v=Al)] v S IpDC.
(59)
and
9o (Tp. E - S (1 vm..)) =
s m6
(60)
The first two claims in (55) are proven.
Next, we have
E GlI (C(T)))))))) + (Tp.)
G(Tp)() G(Tp:))))) +2x² ()) 9a (. 11)
(61)
N₂
where (i) by the Young's inequality, Next, we provide bounds for E[N]] and [[ separately:
B < [B]|(G(Tp (:) G(Tpp)) + + (((pp)) + G(Tp.)))11
G(Tp.) [[I 8)Th (((pp:))))())) G(Tp:)))
(1 (G(mp. 5)
((Tp)))))G(( G((p.))
+ (Gtp::=))) ((t
G( G(Tp,)))
||G(TT))) " |11 - G( 12 [((pp.)))) [((Tp:)) |=
(8i)
< 4m 8mo C2 + 4C* = 4(mo² + cg)2
(62)
where (i) follows from Cauchy-Schwarz inequality and w, is a convex coeffiecient, (ii) follows from Young's 's inequality and (iii) follows from Assumption 4.2 and Assumption 4.3. For another term,
E [2) -B]I(G(Tp.()))) G(Tp.)) G(Tp.)) 1 (as(pppt:E)) (x(pp.)) + (x(T)))))
(9) "
< [[ - G(TT,)))))))))))))))( ()) (Tp, go (T IKG 5)" ( (Tp.))1
+ [[G(Tp)) R" (g.(TTp.: - go (T | (Tp.))
(11)
4Izmio0 813mo C 4 41, + cg),,
(63)
31

---

Preference Controllable RL with Advanced Multi-Objective Optimization
where (i) follows from Young's inequality, (ti) follows from (59) and (60). Then substituting (62) and (63) into (61),
obtain,
(i (G(Trti + Aga(T 8(mo² + +gg)2 881288p + +z2)² =Ci.
We have proved (56). Then, we look at (57):
(T) (g [)1
[ u/: G(Tp,)) (TTp.) (g.(Tp,t () Aga (Tp,)) + Ago (Tp.
4E G () - G(Tp,))) 4B p 41 - Do (T(
2 g9 (T..)1)))1
(1)
4mo² 2 + 40 mo 122 p
(64)
where (1) follows from Young's inequality, and (ii) follows from 59 and 60.
Finally,
((] + (Tp.)()TT)) " G(Tp
w
= w + d (Tmp))))))
+ (T:,
) (0m (G(T))))))) +) ('II]
=B, + (Tp.(TT.))) G(Tpa))l(V [ Ni /N2)l
33,2(1 Cg + C₂)² 3,Cs,
(65)
Lemma 1.2. Under the Assumptions 4.1.4.3, setting o, = O(m =)-b),, Br O(m (m-m) the updates by our method satisfy
1 (Tppt)
-1((ppet1) 1 Elllw; w. w012 || wn|²]
El) + Aeg. (Tp.))1
a2
23,
(66)
+ 8
1/, (2(2))++
C1(1)
+
2
2
where is a fixed convex coefficient, and
1' (TTT) = w v(Tx)()))))))) - X (p. xp.),
(67)
1 = max 41,1 Ales
(68)
C 8(mo² + C2)² + (mo² + Cz)2
(69)
C2 4mo² + 4C2 4x I In-" +4x213
(70)
C3 = 2(1 + C2 (ma+ + C2)
(71)
where Lg is the Lipschitz constant for 9.).), defined in (54).
Under Assumptions. 14.13),,) previous results show Lemma 4.1 and Lemma 1.1 hold. Therefore, we replace go in their analysis with and apply their (33) in our case and it becomes Equation (66). Stochastic gradient samples like G(Tp..
have been taken expectations and become G(Tp.)) or a.
This is an intuitive result of convergence analysis for smooth non-convex objective functions using conventional techniques. Next we prove our main theoretical contributions based on Lemma I.2.
32

---

Preference Controllable RL with Advanced Multi-Ohjective Optimization
Theorem 4.1. Under the Assumptions 4. 4.1., setting 04 - Br 17-)), with constant X and Lipschitz smooth similarity function 4(p,,) -), we have ] - Blmino, [C((Tp:)W:ll = O(mLI=)) To achieve an e-accurate Pureto stationary point, it requires T O(m updates.
Proof. By definition,
V. (p) max
v >
(72)
p:
So (x(pp) = V. d²P) can be considered as positive linear combination of objective gradients. We have
Wl)G(T)))wL. + (TT(TT.))) = (lI((G(Tp.)) + AG.
(73)
E'min ((Tpp))w.11)
For every time step t. by Equation (66) from Lemma I.2 and constant X,
E Elmin [[[[T,)))
1| Ell (T) ( [[T Blll w|12 1|10++11 - w0l3]
at
23,
(74)
8 C(())
1e0 C2(A) +
We take a₂ = a and Bg 3 as constants and telescope (74),
1 aT
E min (2
E[l' (Tp.0) - (((xp.T)) +
E1l 200 -wl2 2 - 1|
w|l
wi
2,T
+
Σ
T-1
Ci()) + 7 ΣΣ
T-1
1a
C2(
+ +
T-1
Σ
3C2(A)
(75)
2
1=0
t=
1=0
1
1
< 0
3m3 + am)
aT
3T
By setting a = O(m -1m-) 3 0(m (1m-)1-)) 1), we can get
T-
M
E.
mii [|G([I O(m 4).
ue
To achieve an e-accurate Pareto stationary point, requires T O updates.
After proving for cases with constant X, need to prove further for cases with increasing À O(T!).
-0 Theorem 4.2. Under the Assumptions 4.1.,,3, setting 0mm 4), 8r IT with a Lipshitz smooth similarity function with g' (Tp.) being convex combination of ge( for all there can be an increasing X = Olog T) and we have Ei=u L E[min [[G(TT,)))) O(mT log T).
T-1
Proof. Because the similarity gradients g' (Tp.t) are convex combinations of G(āp..), let g. (Tp,t) = G( ) where wn convex coefficient, then
E| B[Ip + (TT''(TT.)) =[]]][((TTpp))u + A,G
(76)
2 E[(1 + Xx)² min [G.(T))):
holds because (urr + Xaox)) is also a convex coefficient which can not be more optimal than arg min [[G((pp)) w| For
33

---

Preference Controllable RL with Advanced Multi-Objective Optimization
every time step I, by Equation (66) from Lemma I.2,
E[( ^x)² min [G(().))
Elu' (Tp.)) [Tpp,,++1 + 2/g
E w. w|l [1+11 w0l²
αt
31 (1t)) G00 C2(Ac) + (,C)
E mim [G((Tp)) 11 5
- Blu' (Tp,)) - -1((Tpet11) +
Bllle - -wll2 - -11w1+1 2011
04
2/
3.
Be
C2(Ae)
C2(AA) + (1+ Xx)² C2(A,)
2(1 + xx)²
2(1 + xx)
1
(77)
E[(' (Tp,)) -I((Tp,t+1] +
B Blllw -wl ||11+1 w0l3
23g
Be (1t))) +
1i
Be
C2(Ac) +
(1 )x)² Cs(Ae)
2X
2X²
E[1' (Tp,)) -T(Tp,,++ +
B wwy -wl12 11/8-|[0+++ - w/12
α2
23
B1 (1t))) + (max, 4,1 + L Ls) C2((e) - (1+ 1o)2 C3(2e)
Br
+ 2X
2X
where (i) is by the definition of Lj in (68). In the proofs of Theroem and of (Xiao et al., 2023), V was considered as constant. However, for more rigor (it increases with \), we upper bound it here with Olog T). We take ar and Be as constants and telescope (77), and by d O(log t),, we have
1
iimin (00,) 1i
S
Bll' Bl1'((TT (Tp,0) - ('((T +
Bllwoo
E
wo
w012
1
w1l
T
aT
23T
+
Σ
6 (Arr))
+
(max; 4,1 + -xrLL
C2(Ae)
3
Ca(Ae)
(78)
2X?
T
2X;
T
+ Xz)²
-
1=U
1=
1
1
3m2
0
Bm3² amm ammlog? T + am)
aT
ST
log
T
By setting O(m (
0 b), B Om 1)), we can get
}),
E min || [[G(Tp)))0119]]
= Om log T),
T 1m
0
and proof is done.
Remark L.I. The convergence rate for Theorem 4.2 seems slower than results (Xiao et al., 2023) because we rigorously considered the changes the Lipschitz constant of the (G(T,.))w + >gg((Tp))) caused by increasing X
I.3. Proof for Theorem 4.3 and 4.4
To be consistent with previous results in MOO literature, we consider minimizing the negative objectives and similarity with gradient descent.
Theorem 4.3. Under the Assumptions 4. 4.1,,,3, setting at 1(m- b). B, 10m-10mm))))--
- 2). with a constant X and Lipschitz smooth similarity function like T (p. :), we have + E Ei= [1 - x² =0m
34

---

Preference Controllable RL with Advanced Multi-Objective Optimization
Proof. By Equation (66) from Lemma 1.2 and constant X, have
2 - [[][((Tp.)),, + )g)(T +
2 (Tpp)
E| [[I9g(TTp))11]
X2
2
S a²a,
EI' (Tp.t) -T(
[illw, wl
|| - -01²
(79)
123,
B + 1, C2(A) X
23 C1(
20 A
+ x
CI()
C3
By the definition of i1 in (68), it a constant when À constant. Take at and 3r = B as constants telescope (79), we get
2C
2
Σ
9 ( (pp.))
Ell Ell'(Tp.0) ('(TTTT))] +
B[|lw0 w0 - | w [7]
T t=
X2
Xaa
X²3T
1
T
1
3
(1)
+ T t=
Σ
T-1
1i C2(2)
Σ
T-1
23 C2(A)
+
2C
(80)
T
X
X2
X²
A2
t=
t=
S 0
+
+ Bm? + am)
aT
BT
By setting a = 10-mmm7-1 3 = 0(m- }), we can get
20 g A2
Elmin [G((T,
= =mmmT 1).
T
we
To achieve an e-accurate stationary point, it requires T = O(m²E updates.
Theorem 4.4. Under the Assumptions 4.1.4.3, setting a 0(m -m b), Br Omm-'ITT--- )), with constant À and Lips-
chitz smooth similarity function like V (p.-). )). there can be an increasing O(Tb) and we have 4 ETO (l O(m''= l).
Proof. Suppose for all time steps t > to. A 2 I, by Equation (79) we have
2 d,
Ellgg (Tp, 11 < BlIG((Tp:)urx \ego((Tp:))1 + [lI]G(TTp:
s _ 2 (Tpp( - (TTT.p+11]] +
Billw, - w/l |10++1
w1|2
(81)
3.
B C1)))) + Ga x²
2 Cs(Ne) +
202
+
C2(le)
g
X?
A?
X?
Take a₁ = a and Be = 33 as constants telescope (81) and by the definition of 11 in (68) we get
T
Blllg (Tp.))
(1(((( - (((TT.T)]
E 20 - w0l2
1| - w/|
t0
C1())) + Σ
C2(1))
+
23
Cs(Ae)
+
T-1 20
Σ
X
X
X?
t=ta
t=
tmto
(82)
(Tp.o) - 1' (T(pp.))]
Σ
wop
- w012
1| - 2012
t=to
(max; 1,11 + Lr Cz(A,))
T-1 28
T-1 20
Ci()) +
Σ
Σ
Cs (Ac)
+ Σ
X?
X
X;
t=
t=ta
t=to
t=
35

---

Preference Controllable RL with Advanced Multi-Objective Optimization
then by X (log 1) have:
T
T-1 2
(Tp.))11
Σ
Bll (T)p0) (T(ppT)) + T
R
B1ll20 20122
1
201
T
Σ
B X
ci())
+
T
(max; 4,1 + Lo C2(Ax) + T Σ t
X2
T-1 28 C2(A,)
X
+
Σ
T-1 20
X
(83)
t=to
to
toto
3m2
00
+
+ Bm3² + am aamlooo? am +
of
BT
log
(log T)2
Adding the terms before to, have
1
10
[Tp
(Tp,))1
T
T
T
(84)
1
1
3m?
< 0
+
+ Bm3 + am lomlogo T + am + (log 772
T
aT
3T
log
By setting a O(m - 0(m 1 -
}), we can get
T-1
Bllla( (P= Om log T),
(85)
and the proof done.
36


Reviews:
[{"review_id": "GIOpCzduHF", "reviewer": "Reviewer_MJ3t", "summary": "This paper addresses the limited controllability and coverage of Pareto-optimal solutions in Multi-Objective Reinforcement Learning (MORL), where existing methods based on linear scalarization might struggle to align with user-defined trade-offs and fail to explore the full Pareto front. To overcome these limitations, the authors propose Preference Controllable Reinforcement Learning (PCRL), a framework that trains a preference-conditioned meta-policy to generate solutions based on user-specified preferences. They further introduce PREference COntrol (PreCo), a novel algorithm tailored for PCRL, providing theoretical guarantees on convergence and preference-controlled Pareto-stationary solutions. By integrating advanced Multi-Objective Optimization (MOO) methods, PCRL enhances controllability in MORL and proves particularly effective in environments with highly conflicting objectives, improving the diversity and utility of Pareto-optimal solutions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nWe sincerely thank you for your time and detailed review. We hope the responses below address your concerns.\n\n---\n\n#  Answers To Your Questions\n\n---\n\n**A1:**  In practical deep RL/MORL, first-order gradient-based algorithms are the most widely-used and stationarity is the strongest guarantee [1] they can achieve in practice. Hence, prominent MOO algorithms [2,3,4] establish only Pareto stationarity. Compared to existing works, ours is one of the few having provable convergence under noisy gradients.\n\n---\n\n**A2:** We only need to train a single policy to handle different preferences as input conditions. During training, preferences are sampled uniformly without using prior knowledge. These are explained in lines 143–145 of Section 3 and Algorithm 2 in Appendix E. This standard approach in conditional MORL [5,6] ensures shared parameters and knowledge across all preferences.\n\n---\n\n**A3:** As noted in A2, preferences are uniformly sampled during training in a meta-learning fashion, applied consistently across all methods.\n\n\nOur high-level objective design makes it a generally applicable framework to all RL backbones (on-policy & off-policy). Its contribution is orthogonal to lower-level improvement of sample efficiency. As discussed in Appendix A.1, when adapted to off-policy methods, it can integrate techniques like Hindsight Experience Replay(HER) to enhance sample efficiency.\n\n---\n\n**A4:** In the MuJoCo environments (Ant, Hopper, Reacher), all methods use 3e6 environment steps.\n\nFor the FruitTree environment, all methods except PDMORL use 3.6e5 steps, while PDMORL follows its original implementation with 1e6 steps.\n\n---\n\n**A5:** PreCo's theoretical properties require the similarity function to be:\n\n1. **lipschitz smooth**\n2. Its gradient $g_s$ must be **positive linear combination** of objective gradients $g_{1:m}$\n\nThe similarity in Definition 4.1 meets these criteria, and its magnitude does not affect PreCo's properties. Intuitively, it encourages certain objectives to improve more, so the achieved trade-offs get closer to the pre-defined preference. See Appendix F.1 and Figure 10 for further insight.\n\n---\n\n # Concerns About Methods\nTo address your concerns, we summarize a few key points:\n\n **Generalizability:** We focus on a MORL's framework design that is generally applicable to all on & off-policy RL. In contrast, techniques like HER that exploit inherent RL properties are limited to off-policy methods.\n\n **Novelty & Contributions:**\n\n1. **Conceptual:** We formulate PCRL for any-preference alignment, overcoming the limitations of prior MORL methods using Linear Scalarization (LS) that has no alignment guarantee.\n\n2. **Methodological:** We integrate modern MOO algorithms into MORL to handle conflicting and stochastic gradients—key aspects previously overlooked. Specifically for PCRL, we design PreCo to inherit these strengths while promoting preference alignment.\n\n3. **Technical:** PreCo’s similarity design requires independent proofs (App. I.1) and significant adaptations (App. I.2-3), as no prior method incorporates such similarity. Moreover, as noted in lines 1670-1672, our analysis is more rigorous than previous literature. \n\n **Practicality:** Our assumptions align with standard RL/MORL settings: most RL methods (e.g. policy gradients, DQN loss) are **unbiased** (Ass.2), and their values typically change with a certain level of smoothness (Ass.1).\n\n---\n# About Theoretical Correctness\nAs noted in Assumption 2, the expectation here is over the gradient noise $\\xi$ and Assumptions 1-3 ensures unbiased, bounded gradients for all $\\pi_{p,t}$ samples. Thus, the bounds in (51)-(54), (56)-(61), and (68)-(70) apply to all $\\pi_{p,t}$ samples and remain valid under expectation over $\\pi_{p,t}$. \n\nWe appreciate the detailed feedback, but treating (51)–(54) as conditional expectations does not impact the proof's correctness.\n\n---\n # About Experiments\n**A2-A4** have answered questions about handling different preferences and environmental steps. Here, we address the concerns about the baselines.\n\nEnvelope Q-learning (EQL), Conditioned Network (CN), and Q-Pensieve (QP) all optimize a linear scalarization (LS) objective and inherit its limitations. The best-performing QP achieves a **6859.94** hypervolume for 6D FruitTree, identical to the best cases of our implemented general LS baseline. This is expected, as QP is simply a more sample-efficient version of LS.\n\nSince we already include an LS baseline and CAPQL, which achieves a higher upper bound than LS methods, comparing against EQL, CN, or QP is unnecessary.\n\nPCN is a heuristic method relying on model generalizability. Its comparison to other methods is limited. It only accepts a unique input condition, making it less relevant to our study.\n\n---\nPlease let us know if you have further questions.\n\n[1] Kushner 1978 Stochastic\n\n[2] Sener 2018 MGDA\n\n[3] Liu 2021 CAGrad\n\n[4] Xiao 2023 SDMGrad\n\n[5] Liu 2023 CAPQL\n\n[6] Yang 2019 Envelope Q", "claims_and_evidence": "Regarding the claims made in the paper, below are the positive and the negative parts:\n\n**Strengths**\n- This paper addresses the inherent controllability issue in many existing MORL methods that are meant to only find a Pareto-stationary point for some arbitrary / uncontrollable preference. By using PreCo, one can choose to find the corresponding Pareto-stationary point for a specific preference of interest.\n- The proposed approach can be viewed as a generalized version of linear scalarization (i.e., linear scalarization is a special case with $\\Psi(p,v)=p^\\top v$). This is an interesting and reasonable extension.\n- Motivated by the MOO literature, this paper extends the min-norm problem in the classic MGDA to the case of similarity function for preference control.\n\n\n**Weaknesses**\n- One main concern is that the proposed algorithm does not really solve the target PCRL problem considered by this paper. Specifically, on page 2, right column, it is mentioned that the goal is to learn a preference-conditional policy to achieve a Pareto-optimal value that maximizes the similarity for any preference $p$. However, based on Theorems 4.1-4.4, it appears that PreCo can only achieve Pareto stationarity (Theorems 4.1-4.2) and near-stationary points of the similarity function (Theorem 4.3-4.4). Hence, the theoretical results do not fully match the claims in the introduction.\n\n- Another concern is that I do not see why the conditional policy is needed in PreCo. In Algorithm 1, the (projected) gradient updates (Lines 4-5) are done only for one given preference $p$ (Line 1), and there seems nothing to do with varying preferences. My guess is that there shall be some meta-level method that determines how to switch between different preferences during training, like many other single-policy-network MORL methods. Otherwise, one would need to have a full run of PreCo for **each** individual preference, and this can be extremely sample-inefficient (this issue is also relevant to the issues of the experiments described below). However, this part appears not described at all in the paper.", "methods_and_evaluation_criteria": "- Regarding the methods, one major concern is that technically this paper is more like an MOO paper instead of an MORL work. Based on the formulation and the PreCo algorithm, it appears that the vector-valued objective function $\\mathbb{v}$ can be replaced by any objective function and does not necessarily need to be the total expected reward. Indeed, with the unbiasedness assumption (Assumption 4.2) and regularity conditions (Assumptions 4.1 and 4.3), the analysis in this paper completely gets rid of the inherent properties of MORL and thereby can simply follow the standard analysis of the MOO literature. With that said, in my view, this makes the analysis in this paper not that interesting. Moreover, it can even degrade the performance as the inherent properties of MORL are completely ignored.   \n\n- The choice of the similarity function (Definition 4.1) needs to be further justified. I can understand that $\\Psi(p,v)$ is maximized when $p$ and $v$ are parallel. However, it is not clear how the similarity function would contribute to the overall policy update when $p$ and $v$ are far from being parallel, especially if the number of objectives is large.", "theoretical_claims": "I have checked the proof in Appendix I. I can see that the proofs can go through based on the existing MOO literature, there are some issues to be fixed. Specifically, all the expectations in (51)-(54) need to be conditional. Otherwise, $\\pi_{p,t}$ is a random variable and then (51)-(54) would not hold. Similarly, the authors need to check the correctness of Eqs. (56)-(61) and (68)-(70).", "experimental_designs_or_analyses": "- Evaluation domains: The domains used in this paper appear quite standard in the MORL literature. Both discrete and continuous control tasks in the MO-Gymnasium are considered in the evaluation.\n- Metrics: Hypervolume (HV) is a fairly standard MORL metric. As this paper focuses on the similarity, the authors also take the cosine similarity (CS) into account.\n- Performance: PreCo appears to achieve the best HV and CS in both Fruit Tree and the MO MuJoCo tasks.\n\nThat being said, I do have several concerns:\n- About the sample efficiency: As mentioned in the “Claims And Evidence,” it remains unclear to me how PreCo handles different preferences during training (either each preference is treated separately or there is indeed some knowledge sharing across preferences in the implementation). Therefore, it is not clear why PreCo can have much better performance than the MORL benchmark methods (like PDMORL and CAPQL).\n\n- About the number of environment steps used by each algorithm: Based on the above, one possibility is that different algorithms actually use different numbers of environment steps. If this is the case, then the comparison is actually not fair. With that said, I checked the experimental details in the appendix, but I did not find anything specifically mentioned about the environment steps. Please correct me if I missed anything. \n\n- About the missing baselines: There are several missing recent baselines that are known to be strong in finding the Pareto front in the context of MORL, such as Envelope Q-learning (and Envelope DQN)  [1], Conditioned Network [2], Q-Pensieve [3], and PCN [4].\n\n[1] Yang et al., “A generalized algorithm for multi-objective reinforcement learning and policy adaptation,” NeurIPS 2019.\n\n[2] Abels et al., “Dynamic weights in multi-objective deep reinforcement learning,” ICML 2019.\n\n[3] Hung et al., “Q-Pensieve: Boosting sample efficiency of multi-objective RL through memory sharing of Q-snapshots,” ICLR 2023.\n\n[4] Reymond et al., “Pareto Conditioned Networks,” AAMAS 2022.", "relation_to_broader_scientific_literature": "The paper contributes to MORL by addressing the challenge of preference-controllable policy learning, which is not well studies in the existing MORL literature. Prior works, such as LS-based MORL methods, primarily optimize a scalarized objective but fail to capture a diverse set of Pareto-optimal solutions, limiting their ability to align with user preferences. PreCo incorporates preference-awareness into the learning process by conditioning policy updates on user-specified trade-offs. The lack of controllability in existing methods limits their practical applicability, making the research direction of PCRL necessary.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n- The main strength is that the paper points out the inherent controllability issue in many existing MORL methods that are meant to only find a Pareto-stationary point for some arbitrary / uncontrollable preference. By using the proposed PreCo, one can choose to find the corresponding Pareto-stationary point for a specific preference of interest. Moreover, PreCo can be viewed as a generalized version of linear scalarization (i.e., linear scalarization is a special case with $\\Psi(p,v)=p^\\top v$), and this is a nice extension of the LS-based methods.\n\n**Weaknesses**\n- As mentioned above, the proposed algorithm does not seem to fully solve the PCRL problem. Notably, the goal is to learn a preference-conditional policy to achieve a Pareto-optimal value that maximizes the similarity for any preference $p$. However, based on Theorems 4.1-4.4, it appears that PreCo can only achieve Pareto stationarity (Theorems 4.1-4.2) and near-stationary points of the similarity function (Theorem 4.3-4.4). Hence, the theoretical results do not fully match the claims in the introduction.\n- The clarity of the proposed method can be improved. Specifically, the use of the conditional policy in PreCo, how the different preferences are handled, and the chosen similarity function need to be further explained and justified.", "other_comments_or_suggestions": "- Eq. (53)-(54): The notation $C_2$ is overloaded.\n- Eq. (56):  The right bracket at the end shall be moved to the left.", "questions_for_authors": "1. Based on the theoretical results in Theorems 4.1-4.4, it appears that PreCo can only achieve Pareto stationarity and near-stationary points of the similarity function. Accordingly, the results do not fully match the claims in the introduction. Can the authors comment on this?\n\n2. How is the conditional policy utilized in PreCo (Algorithm 1)? Is there any knowledge sharing across preferences in PreCo? How the different preferences are handled during training?\n\n3. Built on 1., if PreCo needs to handle each preference separately, then how PreCo handles the critical sample efficiency issue in MORL? \n\n4. How many environment steps are used by each algorithm in the experiment?\n\n5. How is the similarity function chosen in PreCo? While I can understand that it is maximized when $p$ and $v$ are fully aligned, it would be good to explain the nice properties of the chosen function in more detail.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7FNtMfnrRJ", "reviewer": "Reviewer_wiWW", "summary": "This paper proposes PCRL for preference control of multi-objective trade-offs, incorporating recent MOO algorithms into MORL. Convergence analysis is provided to show the approach can learn preference-specific Pareto optimal solutions and handles tochastic gradients as well.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nWe sincerely thank you for your time and valuable review.\n\n---\n# Response to your questions\nBelow are our answers to your questions\n\n> Q1. About Theorem 4.2's assumption\n\n**A1:** Theorem 4.2 considers the case where $\\lambda$ increases indefinitely. **In practice, Theorem 4.1 already guarantees PreCo’s convergence as long as $\\lambda$ has an upper bound**, as noted in Remark 4.1.\n\nIn Appendix F.2, We provided some hints for constructing such a similarity function $\\Psi'(p,\\cdot)$ as follows: \n* When $v$ gets close to preference $p$: Its similarity gradient $\\nabla_v\\Psi'(p,v)$ should approach the convex preference weight $p$; \n* When $v$ is NOT close to preference $p$: $\\nabla_v\\Psi'(p,v)$ should be close to the convex coefficient $\\nabla_v\\Psi(p,v)/\\|\\nabla_v\\Psi(p,v)\\|_1$, where $\\Psi$ is our original similarity defined in Definition 4.1.\n\n> Q2. About MOO and choices other than the MOO\n\n**A2:** MOO refers to the algorithms specificly designed for deep learning with multiple objectives, such as MGDA[1], CAGrad[2], PCGrad[3], SDMGrad[4]. They are known to find a conflict avoidance direction for common improvements across all objectives.\n\nAdditionally, evolutionary algorithms[5] have been developed for multi-objective problems, but they tend to be less scalable and less sample-efficient compared to gradient-based MOO approaches.\n\n> Q3. Evidence for gradient conflict\n\n**A3:** For a two-objective case, when the values are $[v_1,v_2]=[10.0,1.5]$ and the preference is $p=[0.9,0.1]$, and let $g_1,g_2$ denote the gradients of objective 1 and 2, respectively. \n\nThe gradient of cosine similarity is $0.1483 g_1 -0.9889 g_2$, which is nearly opposite to $g_2$, creating a direct conflict with objective 2’s gradient.\n\nTo resolve this, we need a conflict-avoidance mechanism that optimizes similarity without negatively impacting objective 2. Intuitively, the goal is to find a direction that improves both $v_1,v_2$ while prioritizing $v_1$ to get close to the preference $p$. This can be precisely acheived by our proposed PreCo algorithm.\n\n>Q4. Computation complexity of implementing Jacobian matrix and similarity gradient\n\n**A4:** Solving the min-norm problem (6) introduces additional computation to determine a common ascent direction, improving sample efficiency at the cost of extra computation.\n\nHowever, **memory consumption remains comparable** to plain policy gradient algorithms:\n\n* When solving the problem (6), objective gradients $\\nabla_\\pi \\mathbf{v}^{\\pi}$ and similarity gradient $\\nabla_\\pi \\Psi(\\mathbf{p},\\mathbf{v}^{\\pi})$ are stored, but their size are at most batch size $\\times$ # objectives, which typically **much smaller than the parameter size**. Once (6) is solved, we obtain a batch-size update direction $d^*$ for $\\pi$, allowing $\\nabla_\\pi \\mathbf{v}^{\\pi},\\nabla_\\pi \\Psi(\\mathbf{p},\\mathbf{v}^{\\pi})$ to be released. \n\n* After solving (6), we compute $\\nabla_\\theta^T\\pi\\ d^*$ to update the model parameter. **The jacobian $\\nabla_\\theta^T\\pi$ is not explicitly computed and stored**. Instead, we take the gradient for ${d^*}^T \\pi$ to obtain it. In a formulation similar to the policy gradient, it corresponds to $E_{s,a}[\\nabla_\\theta \\log\\pi(a|s)\\ d^*(s,a)\\pi_{old}(a|s)]$, as detailed in Appendix E. \n\nTherefore, memory usage at any given time does not exceed that of linear scalarization methods that computes the gradient $\\nabla_\\theta^T\\pi \\nabla_\\pi \\mathbf{p}^T\\mathbf{v}^{\\pi}$. In the policy gradient form, it is expressed as $E_{s,a}[\\nabla_\\theta \\log\\pi(a|s)\\ \\mathbf{p}^T\\mathbf{q}^{\\pi}(s,a)]$.\n\n---\n# Addressing concerns of weeknesses\n> However, it is unclear about the computation complexity... large volume of gradient information.\n\nAs mentioned in A4, keeping large volume of gradient information is unnecessary and the memory consumption remains comparable to traditional methods.\n\n> It is unclear ... to transfer to new preference vector, as the algorithm needs to re-implement the whole optimization again.\n\nAs mentioned in lines 143–144 of Section 3 and Algorithm 2 in Appendix E, preferences for training are uniformly sampled, and the stepsize for each preference sample remains constant. This enables a single agent to learn in a \"meta-learning\" fashion. This enables the agent to generalize to unseen preferences, as demonstrated in our experiments (Section 5 and Appendix D), where we test the agent on preferences it has not encountered during training.\n\n---\nWe hope these clarifications address your concerns. Please let us know if you have any further questions.\n\n# Reference\n\n[1] Désidéri,  MGDA, 2009\n\n[2] Liu et al. Conflict-averse gradient descent for multi-task learning, 2021\n\n[3] Yu et al. Gradient Surgery for Multi-Task Learning, 2020\n\n[4] Xiao et al. Direction-oriented multi-objective learning: Simple and provable stochastic algorithms, 2023 \n\n[5] Tan et al. Evolutionary Algorithms for Solving Multi-Objective Problems, 2007", "claims_and_evidence": "Yes, the paper is showing evidence in both theoretical proofs and simulation results.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, the Theorems in main paper are making sense, though I have not checked the full proof.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This paper proposes a new technique to scale up reinforcement learning algorithms to multi-objective setting.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper proposes a novel approach based on multi-objective optimization. The method is clearly described, and the simulation results especially on higher number of objectives and conflicting objectives validate the model design. \n\nHowever, it is unclear about the computation complexity. And the method seems to consume more iterations and need to keep large volume of gradient information. It is unclear how the stepsize choice for each preference would affect the performance. Moreover, the methods seems to be hard to transfer to new preference vector, as the algorithm needs to re-implement the whole optimization again. In addition, more explanations about MGDA are required, as it is not explained in this paper.", "other_comments_or_suggestions": "Please see below for questions.", "questions_for_authors": "The Theorem 4.2's assumption is that g' is a convex combination of g_i. Is this practical for practical MORL setting?\n\nWhat does Multi-Objective Optimization(MOO) algorithm exactly refers to? Are there any other choices other than the MOO mentioned in this work? \n\n\"However, the gradients of cosine similarity and the original objectives can conflict, as it does not leverage conflict-avoidance techniques from MOO algorithms.\" Are there any evidence showing this claim, rather than just looking at the final performance?\n\nCan the authors comment on the computation complexity of implementing Jacobian matrix and similarity gradient, especially compared to plain policy gradient algorithms?\n\n### Update after rebuttal\nThanks for the authors' efforts in rebuttal. My evaluation has been updated.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SgFM8uTFHK", "reviewer": "Reviewer_2f4Z", "summary": "The paper proposes a novel approach to learning the ϵ-Pareto efficient frontier in multi-objective optimization using standard reinforcement learning algorithms, specifically TD3 and PPO. The authors introduce a method where preferences are sampled uniformly, and a similarity function between the preference and value function is used as feedback to guide the RL algorithm. Theoretical guarantees are provided regarding convergence to ϵ-optimality. Empirical results are presented across multiple environments to demonstrate the efficacy of the proposed approach. The paper also discusses the potential application of the method in real-world scenarios, particularly in multi-agent reinforcement learning settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nWe sincerely thank you for your valuable review and constructive feedback.\n\nWe address your concerns and aswer your questions below:\n\n---\n>**C1: About Figure 4b and 4d:** The empirical results do not consistently ... in Figure 4b and 4d.\n\n**R1:** Figures 4b and 4d are **2D projections of the 3D results** presented in Figure 4a. While some points may appear dominated in the 2D projections, all points in the figures are actually non-dominated when all three objectives are taken into account.\n\nWe hope this clarification helps address your concern regarding the empirical results.\n\n---\n>**C2:** The delta from previous RL algorithms is not clearly ...  the novelty of the method.\n\n**R2:** TD3 and PPO are representative single-objective RL algorithms used as lower-level backbones in our approach. More details on how they are integrated can be found in Appendix E.\n\nThe key novelties of our work are as follows:\n\n1. **Conceptual:** We propose a preference control framework (PCRL) that overcomes the limitations of traditional linear scalarization methods, which can only discover a limited set of Pareto-efficient solutions.\n\n2. **Methodological:** We integrate modern MOO algorithms into MORL, enabling better handling of conflicting and stochastic gradients—challenges that previous MORL methods largely overlooked. Additionally, we design a novel PreCo algorithm to leverage these strengths while ensuring preference alignment.\n\n3. **Technical:** Beyond theoretical analyses, we propose a memory-efficient approach for policy-level computation (see Figure 9, Algorithm 2, and Appendix E), making our method scalable for large models used in practical applications.\n\n---\n>Q1: Justification for Similarity Function (Eq. 8): ... choice of similarity function?\n\n**A1:** PreCo's theoretical properties require the similarity function to:\n\n1. Be **lipschitz smooth**.\n2. have a gradient $g_s$ must be **positive linear combination** of objective gradients $g_{1:m}$ ($m$ is the number of objectives).\n\nThe similarity defined in Definition 4.1 satisfies these conditions. Using a different similarity function could disrupt the convergence guarantees of our algorithm. In practice, conventional cosine similarity could work in some cases, but it lacks theoretical guarantees.\n\n---\n>Q2: Non-Uniform Preference Sampling:  ... handle such cases?\n\n**A2:** Non-uniform preference sampling may cause certain preferences to be learned better than others or lead to overfitting. The impact depends on the model's generalizability, so training with a uniform distribution is preferable without prior knowledge of test preferences.\n\nHowever, as noted in Appendix A.3, a progressive curriculum for preference distributions could accelerate learning. For instance, the agent could start with a small, diverse set of preferences and gradually expand to neighboring ones, progressively uncovering the full Pareto front. Further exploration of this is an interesting direction for future work.\n\n---\n>Q3: Non-Transient Preferences: ... convergence?\n\n**A3:** If we understand correctly, non-transient preferences refer to preferences that are consistently sampled during training. While this could lead to overfitting to specific preferences, a well-designed curriculum that gradually introduces new preference distributions can mitigate this issue and promote exploration of the full Pareto front (as discussed in Appendix A2).\n\nIn fact, the training scheme in Algorithm 2 (Appendix E.1) can be viewed as a form of meta-learning, where the outer loop samples different preferences and the inner loop optimizes for a specific sampled preference. While convergence is always guaranteed within the inner loop, non-stationary preference distributions may affect the overall meta-learning process, potentially leading to overfitting. However, a carefully designed curriculum with progressively shifting preference distributions, even if non-stationary, can accelerate learning and improve the agent’s performance.\n\nTheoretical analysis of such a curriculum would be an interesting direction for future work.\n\n---\n>Q4: Economic Benchmarks: ... the method's applicability.\n\n**A4:** Economic benchmarks are indeed an interesting direction for future work.  We will add more discussion on economic applications in the related work section.\n\nA key technical contribution of our method is to solve the min-norm problem (6) at the policy level (details in Figure 9 and Appendix E), removing the need to store multiple parameter gradients for different objectives. This results in minimal **additional memory cost**, making it highly suitable for training large-scale practical models. In fact, we are already applying our method to multi-objective preference learning for large language models with promising results, suggesting its potential applicability to practical economic settings as well.\n\n---\nThanks again, and please let us know if there are further questions.", "claims_and_evidence": "The claims made in the paper are generally supported by empirical evidence and theoretical analysis. The authors provide convergence guarantees for their method, which are backed by theoretical proofs. The empirical results are extensive and demonstrate the effectiveness of the proposed approach across various environments. However, some claims could benefit from further clarification or additional evidence:\n\nThe claim that the method achieves ϵ-Pareto efficiency is supported by theoretical analysis, but the empirical results (e.g., Figure 4b and 4d) do not clearly show that the solutions lie on the Pareto front. This discrepancy should be addressed.\n\nThe justification for the choice of similarity function in Eq. (8) is not thoroughly explained. A more detailed discussion or comparison with alternative similarity functions would strengthen this claim.", "methods_and_evaluation_criteria": "The proposed methods are appropriate for the problem of learning the ϵ-Pareto efficient frontier in multi-objective optimization. The use of TD3 and PPO as base algorithms is reasonable, given their widespread success in RL. The evaluation criteria, such as hypervolume optimization, are standard in multi-objective optimization literature and align well with the goals of the paper. However, the following points could be improved:\n\nThe paper could benefit from a more detailed discussion of why hypervolume was chosen as the primary metric and how it relates to the ϵ-Pareto efficiency goal.\n\nThe uniform sampling of preferences is a simplifying assumption. The authors should discuss how non-uniform preference sampling might affect the results and whether the method can be adapted to handle such cases.", "theoretical_claims": "The theoretical claims regarding convergence to ϵ-optimality are a key contribution of the paper. The proofs appear to be correct, but the theoretical analysis could be deepened. For instance:\n\nThe learning rates for the RL algorithms are mentioned, but a more detailed discussion of their impact on convergence would be valuable.\n\nThe paper could explore the theoretical implications of non-transient preferences or non-linear preference transformations, as these scenarios are common in real-world applications.", "experimental_designs_or_analyses": "The experimental design is sound, with results presented across multiple environments to demonstrate the robustness of the proposed method. However, there are a few areas for improvement:\n\nIn Figure 4b and 4d, the solutions do not clearly lie on the Pareto front, which raises questions about the empirical validity of the ϵ-Pareto efficiency claim. The authors should address this discrepancy.\n\nThe experiments focus on synthetic environments. Including real-world benchmarks, such as economic settings (e.g., profit vs. customer satisfaction), would strengthen the practical relevance of the paper.", "relation_to_broader_scientific_literature": "The paper builds on prior work in multi-objective optimization and reinforcement learning. The key contribution—learning the ϵ-Pareto efficient frontier using RL—is novel and addresses an important gap in the literature. However, the paper could better situate itself within the broader context by discussing how the proposed method compares to other approaches for Pareto front learning, such as evolutionary algorithms or gradient-based methods.", "essential_references_not_discussed": "The paper could benefit from citing and discussing the following works:\n\nRecent advances in non-linear preference transformations for multi-objective optimization.\n\nMethods for handling non-uniform preference sampling in RL.\n\nApplications of Pareto optimization in economic settings, which would provide a more nuanced benchmark for the proposed method.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and addresses an important problem in multi-agent reinforcement learning.\n\nThe empirical results are extensive and demonstrate the potential of the proposed method in applied settings.\n\nThe theoretical guarantees provide a solid foundation for the proposed approach.\n\nWeaknesses:\n\nThe delta from previous RL algorithms is not clearly articulated. The reliance on TD3 and PPO without significant modification raises questions about the novelty of the method.\n\nThe empirical results do not consistently demonstrate ϵ-Pareto efficiency, particularly in Figure 4b and 4d.\n\nThe paper could benefit from a more thorough discussion of the broader implications and limitations of the proposed method.", "other_comments_or_suggestions": "Move the discussion of related works to the introduction for better flow.\n\nClarify whether the objectives are truly \"conflicting\" or simply involve trade-offs (Line 138).\n\nMove Algorithm 2 to the main paper, as it represents the actual algorithm used, while Algorithm 1 is more illustrative.", "questions_for_authors": "Justification for Similarity Function (Eq. 8): What justifies the use of this specific similarity function over alternatives? How sensitive are the results to the choice of similarity function?\n\nNon-Uniform Preference Sampling: How would non-uniform sampling of preferences affect the learned policy? Can the method be adapted to handle such cases?\n\nNon-Transient Preferences: Can the method extend to non-transient preferences? Is consistency in preferences a requirement for convergence?\n\nEconomic Benchmarks: Have the authors considered benchmarking the method on economic settings, such as maximizing profit versus customer satisfaction? This would provide a more nuanced evaluation of the method's applicability.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KD8jLjnle", "reviewer": "Reviewer_AukZ", "summary": "The paper introduces a novel framework, Preference Controllable Reinforcement Learning (PCRL), which trains a single, preference-conditioned policy capable of generating Pareto optimal solutions according to user-specified trade-offs. The approach leverages advanced multi-objective optimization (MOO) techniques and proposes a new update method, PreCo, that incorporates a similarity function to align the policy’s performance with a given preference. The paper supports its claims with comprehensive theoretical analysis—providing convergence guarantees—and demonstrates the method’s effectiveness through experiments in environments such as Fruit-Tree and MO-Reacher, showing improved hypervolume and cosine similarity metrics compared to traditional linear scalarization (LS) and several existing MOO methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nWe sincerely thank you for your valuable review and constructive feedback.\n\nWe address your concerns and aswer your questions below:\n\n---\n> Q1. Can the authors provide ablation studies on the similarity weight $\\lambda$ to illustrate its impact on both convergence speed and final control performance?\n\nA1: The table below demonstrates the robustness of our method to different values of $\\lambda$:\n\n|$\\lambda$ |5|10|50|100|300|550|\n|-|-|-|-|-|-|-|\n|Hyper Volume (*1e3)|13.83$\\pm$1.74|13.98$\\pm$2.11|14.74$\\pm$1.36|15.02$\\pm$1.7|15.21$\\pm$0.79|15.61$\\pm$0.75|\n|Cosine Similarity|0.76$\\pm$0.03|0.76$\\pm$0.04|0.77$\\pm$0.02|0.77$\\pm$0.03|0.79$\\pm$0.02|0.78$\\pm$0.03|\n\n\nThis table illustrates how different upper bounds of $\\lambda$ affect PreCo's performance in the FruitTree environment.  While smaller values of $\\lambda$ may slightly change the performance, PreCo remains consistently superior to existing MORL methods. Notably, the best-performing baseline, PDMORL, achieves a hypervolume of only **9.3$\\pm$0.08*1e3**, demonstrating PreCo's promising advantage.\n\n---\n> Q2. For continuous action spaces, does the PreCo update method require any modifications? Could the authors elaborate on potential challenges and necessary adjustments for continuous control environments?\n\nA2. This has been discussed in Appendix E, with experiments in continuous action space presented in Appendix C. \n\nIn continuous action spaces, the \"policy-level\" gradient corresponds to the gradients of (1) action samples (Eq. 26) or (2) the parameters of the Gaussian action distribution (Eq. 24), rather than the gradients of action probabilities. As illustrated in Figure 9 and Algorithm 2, our approach first determines a common descent direction for all objectives by solving the min-norm problem (6) using their gradients (Eq. 24 or 26). The policy model is then updated to fit the new actions along this direction.\n\nThe experiments in Appendix C focus on high-dimensional continuous control tasks with simple, strictly convex objectives. While the advantage on performance is smaller compared to tasks with non-strictly convex objectives and many objectives, our method still outperforms previous MORL approaches.\n\n---\n> Q3. Although the paper suggests that the method could be applied to real robotic tasks, are there any plans for experimental validation on actual robotic platforms? What practical issues might arise during such a transition?\n\nA3: Real-world applications, such as robotics, are indeed an exciting direction. In fact, we have already been applying our method to large language models and acheived promising results. One of our future plans is to leverage preference-controllable language models for high-level robotic planning and human-robot interaction.\n\nFor low-level robotic control, several challenges include common sim-to-real issues such as sample efficiency, dynamics mismatch, and safety concerns. One critical study is to ensure that the robot does not exhibit unpredictable behavior when conditioned on previously unseen preferences.\n\n---\n> Q4. The current experiments involve up to six objectives. How does the proposed method scale when the number of objectives increases further, and are there anticipated bottlenecks in terms of computational complexity or performance?\n\nThe MOO algorithms with conflict-avoidance ability, including our proposed PreCo, require solving the min-norm problem (6) to determine a weight vector $w$, whose dimension corresponds to the number of objectives. When the number of objectives becomes very large, the computational cost of solving (6) can increase significantly.\n\nTo mitigate this, instead of explicitly solving (6) at every training step (Algorithm 2), we can update $w$ using the gradient of (6) once per training step (Algorithm 1). Our theoretical results justify the convergence of this novel approach, making it a more scalable alternative in practice.\n\n---\nWe hope these clarifications address your concerns. Let us know if you have any further questions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The theoretical analysis, while rigorous, makes some idealized assumptions that may not fully capture the noise and uncertainties present in real-world scenarios, thereby leaving questions about the practical applicability of the method.", "experimental_designs_or_analyses": "There is a lack of ablation studies or sensitivity analysis regarding key hyperparameters—such as the similarity weight $\\lambda$ in the PreCo update—which raises concerns about the robustness and practical tuning of the approach.", "relation_to_broader_scientific_literature": "The proposed method represents an important contribution to the field of multi-objective reinforcement learning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The PCRL framework addresses the well-known limitation of LS methods by enabling more extensive exploration of the Pareto front and precise control over policy outputs in non-strictly convex settings.\n2. The proposed PreCo update method is innovative; by integrating a similarity gradient to balance conflicting multi-objective gradients, it provides a clear theoretical advancement, as evidenced by detailed convergence proofs (e.g., Theorems 4.1–4.4).\n3. The experimental evaluation is extensive, covering both discrete action environments (Fruit-Tree) and robotic control tasks (MO-Reacher), with state coverage visualizations that effectively illustrate how the method produces diverse, preference-specific policies.\n4. The authors also consider computational efficiency by solving the min-norm problem at the policy level rather than at the parameter level, which is beneficial for scalability in large models.\n\nWeaknesses:\n\n1. The set of baseline methods used in the experiments is somewhat limited; some of the compared methods are relatively dated, which narrows the scope of the comparative analysis.\n\n2. Experiments are primarily conducted in discrete action spaces. Although the authors mention potential applications to continuous control and real robots, there is insufficient experimental evidence to support the method’s effectiveness in these more challenging settings.", "other_comments_or_suggestions": "No", "questions_for_authors": "1.  Can the authors provide ablation studies on the similarity weight $\\lambda$ to illustrate its impact on both convergence speed and final control performance?\n\n2. For continuous action spaces, does the PreCo update method require any modifications? Could the authors elaborate on potential challenges and necessary adjustments for continuous control environments?\n\n3. Although the paper suggests that the method could be applied to real robotic tasks, are there any plans for experimental validation on actual robotic platforms? What practical issues might arise during such a transition?\n\n4. The current experiments involve up to six objectives. How does the proposed method scale when the number of objectives increases further, and are there anticipated bottlenecks in terms of computational complexity or performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel framework called Preference Controllable Reinforcement Learning (PCRL), designed to train a single, preference-conditioned policy capable of generating Pareto-optimal solutions according to user-specified trade-offs. In particular, it uses standard reinforcement learning algorithms (TD3 and PPO) to identify an $\epsilon$-Pareto efficient frontier in multi-objective optimization (MOO). The paper also introduces PREference COntrol (PreCo), an algorithm that provides theoretical guarantees on convergence to preference-controlled Pareto-stationary solutions. The theoretical results also support the authors' claim that their method can identify such solutions while being robust to stochastic gradients. Empirical results were reported across multiple environments.

All reviewers agree that this paper offers a valuable contribution to the MOO and MORL communities. They noted that the key claims made in this work are supported by both empirical evidence and theoretical analyses, and highlighted that some of the theoretical guarantees introduced by the authors—particularly those relating to convergence to $\epsilon$-optimality—are a key contribution of this paper. All reviewers agreed that the empirical results are extensive, properly designed, and convincing.

A few concerns were raised. One reviewer questioned whether the proposed method truly solves the PCRL problem as defined, and whether the intended contributions were aimed at the MOO community or at multi-objective RL. The authors provided a detailed response, and the reviewer confirmed that most of their concerns had been adequately addressed. Another reviewer raised questions about the computational complexity of the method, along with several clarification questions. The authors addressed those points in their rebuttal, and the reviewer confirmed that the rebuttal had properly addressed their questions. A third reviewer had two main concerns: first, that some of the empirical results (particularly Figures 4b and 4d) appeared inconsistent with the authors’ claim of achieving $\epsilon$-Pareto efficiency. The authors explained that the figures were 2D projections of 3D results and clarified that while some points may appear dominated, they are in fact non-dominated when all objectives are considered. Secondly, the same reviewer felt that the paper did not clearly articulate the method’s technical contributions and improvements over existing approaches. The authors prepared a detailed response, and the reviewer confirmed that their main concerns had been addressed. Lastly, a fourth reviewer argued that additional ablation studies and sensitivity analyses were needed to support some of the authors' main claims. In response, the authors ran a new set of experiments and included the results, which confirmed and supported their initial findings and claims. The reviewer confirmed that their concerns had been resolved.

Overall, reviewers agree that this paper offers valuable insights that will benefit the ICML community. They noted that the concerns raised during the discussion phase were either addressed or were not critical and could be resolved in a revised version. The reviewers encouraged the authors to incorporate the feedback from the reviews and discussion and argued that doing so would strengthen the overall quality and impact of the work.
Decision: Accept (poster)