One Model for All: Multi-Objective Controllable Language Models
Authors: Qiang He
Yucheng Yang
Tianyi Zhou
Meng Fang
Mykola Pechenizkiy
Setareh Maghsudi
Venue: Accepted by TMLR ()

Abstract: Aligning large language models (LLMs) with human preferences is critical for enhancing LLMs' safety, helpfulness, humor, faithfulness, etc. Current reinforcement learning from human feedback (RLHF) mainly focuses on a fixed reward learned from average human ratings, which may weaken the adaptability and controllability of varying preferences. However, creating personalized LLMs requires aligning LLMs with individual human preferences, which is non-trivial due to the scarce data per user and the diversity of user preferences in multi-objective trade-offs, varying from emphasizing empathy in certain contexts to demanding efficiency and precision in others. Can we train one LLM to produce personalized outputs across different user preferences on the Pareto front? In this paper, we introduce Multi-Objective Control (MOC), which trains a single LLM to directly generate responses in the preference-defined regions of the Pareto front. Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditioned policy network. We improve the computational efficiency of MOC by applying MOO at the policy level, enabling us to fine-tune a 7B-parameter model on a single A6000 GPU. Extensive experiments demonstrate the advantages of MOC over baselines in three aspects: (i) controllability of LLM outputs w.r.t. user preferences on the trade-off among multiple rewards; (ii) quality and diversity of LLM outputs, measured by the hyper-volume of multiple solutions achieved; and (iii) generalization to unseen preferences. These results highlight MOC's potential for real-world applications requiring scalable and customizable LLMs.

Published in Transactions on Machine Learning Research (03/2026)
One Model for All: Multi-Objective Controllable Language Models
Qiang He Ruhr University Bochum
Qiang-He de
Yucheng Yang
y- y-yong
Eindhoven University of Technology
Tianyi Zhou
Chanyi..hhow@mbzucc
Mohamed bin Zayed University of Artificial Intelligence
Meng Fang University of Liverpool
Bengg..ann@llvverroooo.cco uk
Mykola Pechenizkiy Eindhoven University of Technology
m.ppchhnizkki@@tue.nt
Setareh Maghsudi
Bettreh..Mgghsuud@@rrbbu www.bbb.cccm.cc de
Ruhr University Bochum
Reviewed on OpenReview: https: // openreview. net/ pormm? gAM5PmuFYY
Abstract
Aligning large language models (LLMs) with human preferences is critical for enhancing LLMs' safety, helpfulness, humor, faithfulness, etc. Current reinforcement learning from human feedback (RLHF) mainly focuses on a fixed reward learned from average human ratings, which may weaken the adaptability and controllability of varying preferences. How- ever, creating personalizod LLMs requires aligning LLMs with individual human preferences, which is non-trivial due to the scarce data per user and the diversity of user preferences in multi-objective trade-offs, varying from emphasizing empathy in certain contexts to demanding efficiency and precision in others. Can train one LLM to produce personalized outputs across different user preferences the Pareto front? In this paper, we introduce Multi-Qbjective Control (MOC), which trains single LLM to directly generate responses in the preference-defined regions of the Pareto front. Our approach introduces multi-objective optimization (MOO) principles into RLHF to train an LLM as a preference-conditionne policy network. We improve the computational efficiency of MOC by applying MOO at the policy level, cnabling us finc-tune a 7B-parameter model on a single A6000 GPU. Extensive experiments demonstrate the advantages of MOC over baselines in three aspects: (i) controllability of LLM outputs w.r.t. user preferences the trade-off among multiple rewards; (ii) quality and diversity of LLM outputs, measured by the hyper-volume of multiple solutions achieved; and (iii) generalization to unseen preferences. These results highlight MOC's patential for real-world applications requiring scalable and customizable LLMs.
1 Introduction
Large language models (LLMs)) have gained significant attention for their impressive performance across a wide range of tasks, including machine translation (Vaswani al., 2017; Radford & Narasimhan, 2018; Devlin et al., 2019), text generation (Touvron al., 2023; OpenAT, 2023), and conversational agents (Ouyang et al.,

---

Published in Transactions on Machine Learning Research (03/2026)
2022; Bai et al., 2022). However, these models are generally aligned with fixed preferences predetermined by developers (Ouyang et al., 2022; Touvron et al., 2023; Bai et al., 2023; Dubey et al., 2024), limiting the degree of personalization available to users. In real-world scenarios, users often have diverse preferences for LLMs behaviors. For example, individuals often seek varying trade-offs between traits like empathy and task-oriented efficiency depending on the immediate context. Despite this variability, the inherent flexibility of current LLMs (Dubey et al., 2024; OpenAT, 2023) limited in adjusting to fine-grained, personalized interaction preferences (Guan et al., 2025; Li et al., 2025).
The ability of LLMs to dynamically balance the trade-offs between different objectives according to diverse user-specified preferences is called multi-oojecttive controllability (Guan et al., 2025; Li et al., 2025), a crucial feature for enhancing user satisfaction. Training separate models for each preference order, however, is neither practical nor scalable due to the high computational costs associated with LLM training. This highlights the need to enable one-time LLMs training while accommodating a broad range of preferences.
Table 1: Comparison with the statc-of-tho-art MOO methods. MOC addresses MOO in principled and efficient manner. M the number of preferences, N the number of reward models (objectives).
Algorithms
Explicit policy improvement
Num of trained LLMs
Inference adaptation
Preference data
Loss
MORLHE
M N
No
PPO
Rewarded Soups (Ramé al., 2023)
No
PPO
MODPO (Zhon et al., 2024)
M
X
Yes
DPO
RIC (Yang et al., 2021e)
X
V
No
SFT
MOC (Ours)
1
V
No
PPO
Can we control the trade-offs in a single, onco-trained LLM to meet diverse human preferences? Our answer is yes. This paper aims to (i) enable L.LMs to generate customized responses for diverse user preferences and (ii) achieve this goal with once-trained model. To this end, we introduce a novel algorithm, Mlulti-Qbjective Control (MOC), which leverages a carefully designed multi-objective optimization (MOO) algorithm. MOC requires training only once, incorporates explicit policy improvement, and does not rely on human preference data. Moreover, its training cost is comparable to single-objective RLHF (Ouyang et al., 2022). We further improve the computational efficiency of MOC by integrating LoRA (Hu et al., 2022), making it feasible to fine-tune a 7B-parmmeter model on a single A6000 GPU.
We first formulate multi-objective controllability as MOO problem with preference vector constraints, inspired by recent advancements in MOO (Désidéri, 2009; Sener & Koltun, 2018; Xiao et al., 2023). This formulation presents two primary challenges. The first is identifying the target to be controlled. Existing MOO methods typically focus on optimizing different loss functions (Liu et al., 2021; 2023) or linearized utility functions (Yang et al., 2019), which do not effectively capture the quality or behavior of LLMs. In contrast, MOC selects the reward signal as the control larget, enabling direct manipulation of the model's behavior. The second challenge is to solve this optimization problem within feasible computational limits. Our formulated optimization problem involves complex trado-offs among multiple objectives under different preference constraints. To address this, we relax the problem into a new form of MOO, where the preference constraint is treated as an additional objective. MOC scalarizes the objectives with dynamic weighting in different steps, ensuring that the computational cost is comparable to the widely used single-objective RLHF (Ouyang et al., 2022). Table 1 presents a detailed comparison of MOC and bascline methods.
In extensive experiments, MOC consistently outperforms baseline methods (Ouyang et al., 2022; Ramé et al., 2023; Yang et al., 2024c) across multiple tasks. It demonstrates strong performance in three key areas: (i) controllability, as it effectively aligns model behavior with diverse preference vectors and ensures clear monotonic relationship between input preferences and outcomes; (ii) quality of solution set, measured by the hyper-volume metric, where MOC achieves superior Pareto front while maintaining a diverse set of solutions; and (iii) generalization, as it robustly handles unseen preferences while preserving both the alignment quality and diversity. Compared to baseline methods, MOC offers a more efficient and flexible approach to personalizing LLMs, managing different trade-offss among multiple objectives within a single model and seamlessly adapting to new preferences. These results demonstrate MOC's suitability for applications that require scalable and customizable personalization.
2

---

Published in Transactions on Machine Learning Research (03/2026)
Our contributions are follows: (i) We introduce the MOC algorithm, which requires comparable computation single-objective RLHF and finetunes LLMs only once to accommodate diverse user preferences; (ii) We empirically demonstrate MOC's superior performance in terms of controllability, solution quality, and generalization, including its ability to generalize to unseen user preferences.
2 Oulti-Objective Controllable Language Models
Modern LLM aligument commonly implemented via RLHF (Ouyang et al., 2022): a reward model first fit from preference pairs by maximizing
LRM Bi w/D log (r(x, r(a,,')))
and the policy is then optimized against this learned reward, typically with PPO (Schulman et al., 2017),
< 2; 0) x))
arg max E EEWD H. (- 7(x, v) v) log (-122:0)
(1)
Hold
While effective, this pipeline implicitly commits to a single, developer-chosen average preference. The resulting model well-aligned but not easily controllable: inference fails to natively adapt when users prioritize different objectives (e.g., helpfulness vs. conciseness, safety creativity). We thus ask whether model trained once can be steered to different trade-offs on demand. Our answer is yes: aim to (i) support spectrum of user preferences, and (ii) achieve this within one training run.
Controllability vs. Aligmment. We use the term controllabilily to mean the model systematically varies its behavior in response to user-specified preferences, producing outputs consistent with those preferences. By contrast, alignment refers optimizing toward fixed, global preference unchanged at inference.
2.1 Problem Formulation
To represent user preferences, we define a preference vector p = P2, ... ,Pn] such that Σi P: = 1 and p: 2 0, where each element p reflects the importance of specific objective. Inspired by recent work on multi-objective learning (Xu et al., 2020; Ma al., 2020; Yang et al., 2022), we use this preference vector to regulate the model's output in the objective space. Given distance divergence metric & and constraint threshold Φ, require the policy's 's deviation from the preference vectors {p') Jiz1 to be upper bounded by some constant 6. The training of a controllable LLM is thus formulated as the following constrained optimization problem:
maxJ(a(::0.p)) max (J=(T(:: p)), (T((;,,))) ... (T(;0,))) s.t. @ (T((0:.)) p) م V (p', p² . JM,
(2)
where J denotes the RLHF objective associated with reward R Equation (2) ensures LLMs align with a set of preferences. The LLM is a policy æ parameterized by 0 and takes preference vector p as an input condition. In addition, + is distance or divergence metric between the policy and the preference vector p. In this work, will later instantiale & mean squared error (MSE) between the expected reward vector and the preference vector (see Equation (3)). In this paper, the objective J is typically selected as a PPO loss (Schulman et al., 2017; Ouyang et al., 2022), unless stated otherwise.
Traditional methods for solving constrained optimization problems, such as the Lagrangian method, are incfficient for handling the complexity of Equation (2) due to multiple constraints, diverse preferences, and the high dimensionality of LLM parameters. This limitation renders developing new solutions imperative.
2.2 What Should the Preference Vector Align With?
Existing multi-objective learning methods (Yang al., 2019: Liu al., 2023; 2021) typically focus on balancing multiple loss functions. However, RT. loss does not reflect the agent's 's true performance and is
3

---

Published in Transactions on Machine Learning Research (03/2026)
therefore not suitable the target for controllability. In contrast, the value function episodic return provides reliable performance measure. In the context of RLHF for LLMs, the reward is evaluated by a reward model, which serves as the episodic return. Therefore, we choose a multi-dimensional reward signal as the primary target for control. To maintain simplicity, we select MSE as the similarity metric between the reward signal and the preference vector. Formally, the constraint in Equation (2) specified as
① (=<00,'')lp)) def MSE ((,,R(( S S vi E (1,2,..,,,,) M),
(3)
where represents the prompt/qucry y T(; LLM-gennrated response, and D is the prompt dataset. The reward vector R(x,y) v) (R* y). (). ,R (r.y)) y)) is associated with the N optimization objectives {J') The sampled response y depends on the policy parameters 0, enabling the optimization of (T)) with respect to 0 via standard RLHF'. Equation (3) enforces that the reward vector R(x, y) aligns closely with the preference vector p. In other contexts such as typical RL settings, the value function can be the target of control. Further details are provided in Appendix E.
Re-labeling the prompt. In MOC, the policy æ takes an additional condition: the user's preference vector
p = [P1.P2. P2. .Pv]. To accommodate this, we modify the original prompt by prepending the preference vector to it, as follows:
Re-labeled prompt (R1)pı (R2)p2 (prompt)
(4)
2.3 Multi-Objective Controllability
To solve the multi-objective learning problem with inequality constraints, we introduce our MOC algorithm, which builds on recent advances of multi-objective learning (Désidéri, 2009; Sener & Koltun, 2018). MOC simultancously optimizes all the objectives while maximizing the alignment between the objective value vector and the preference vector. For simplicity, we optimize the following similarity objective:
((2 max J def max- ReLU (MSE (EEND, (EaDD, y- R(x, v). p) - -)). Vp E
(5)
where (x) = max(x, 0) penalizes constraint violations when the error exceeds the threshold s This ensures that optimization respects the tradc-offs between rewards and preferences. For brevity, we denote the expectation over the data and policy distribution Eg-D.y-x(x:.pp)) simply as E. For notational simplicity, in the rest of this paper write p to denote arbitrary element of the finite preference set (p') je (ie.., omit the superscript i when it is clear from context). This is purely notational convention: always refers to an element of {p').M Ji=1. When a statement is required to hold for all preference vectors, we will write it explicitly as Vp E [p') In practice, we sample preference vectors from this set during training. The gradient of Equation (5) can be approximated as
N (ARELUU(MSE(ERR(.)...... 6) = IMSE(ER(2.)).)) -) ->0 Σ IER* pa) VoER (2, v)],
(6)
where (()) is the indicator function, R represents the k-th entry of R, Pk means the k-th entry of preference vector p. The term P)= R* (z, y) aims at maximizing the corresponding reward, which is also the goal of the gradient of the PPO loss. Thus, one could use the PPO objective ( to compute En=Dagrr (x, y).
Solving the original optimization problem in Equation (2) is computationally challenging due to the involvement of N objectives and M preferences. Thus, we relax and reformulate it as
m [ p)) def max ( _ [( p)). ReLU (ML.E -)))
where [[= p)) is defined in Equation (2). Equation (7) is relaxation of Equation (2), not a strictly equivalent reformulation. The second objective is hinge penalty the constraint residual MSE(ER,)))
4

---

Published in Transactions on Machine Learning Research (03/2026)
which is maximized at 0. This reformulation offers two significant advantages: (i) It significantly reduces optimization complexity by transforming the original N-objective optimization into a bi-objective optimization; (ii) It retains control over the preference vectors in the newly formulated optimization problem. Scalarization simplifies the problem even further:
max 8
p J(=(;; p)) - (M o) ( 1,
2 /
=00
(8)
where c(1) is i-objective related coefficient, determined by solving min-norm problem
min elt)) P T VoJ(T(:(,p)) -c VoReLU (SS((EeDR(R,.l) v). ( o)
e 1,
c 2
)
99)
As demonstrated by Désidéri (2009), either: (i) The solution to this min-norm problem is zero, meaning it satisfics the KKT conditions; or (ii) it yields a gradient direction that improves all objectives.
2.4 Scalable Multi-Objective Control of LLMs with Surrogate Objectives
However, in the context of LLMs, directly solving this optimization remains computationally intractable due to: (i) the need to backpropagate N + 1 times to compute the gradient for each objective; and (ii) the prohibitive computational expense of solving the min-norm problem in the gradient space for LLM parameters. To overcome this computational burden, we introduce a more computationally officient surrogate, which an upper bound to the original objective, circumventing the need for costly backpropagation operations.
Theorem 1. Let =(0) =(yl=:))
)=)(y(= denote the probability ratio of PPO objective (Fquation (1)), and let be defined in PPO (Schulman et al., 2017). The upper bound of objective in
the clipping Syper-ppramann
Equation (9) is given by
N (1)
N Σ (R' - -))]((A)) x VoT(: 1=1
Σ j=1
PクI(A,) - (2) R(o.))
(10)
where
0, if (A> > 0 and = > 1 +))
(A and < <1-11 1- c),
I(A)
(11)
A,
if (A > 0 and 2 < ((+)
or (A and 1 1)e E).
e 1
i > 0
Vi,
(12)
1=1
where Å, denotes the advantage estimate.
The proof deferred to Appendix A. Theorem provides an upper bound on Equation (9), which yields two (R' - -P;) (A,,) can be efficiently computed without any additional expensive proo-prop;gttion; (ii) Von(;;,p) is no longer required by the min-norm problem it does not depend on c(t), Therefore, we achieve the following computationally efficient surrogate problem of optimizing (L)) and c(
key advantages: (i) Both ((A,) and IMSE(E.-
min o03 ,(0))) |1c0 {I
psJ ((T))) (2)) INSE(E.. Σ
(RL ())) 1
Σ 10 =1,
(00) 2 0, vi
(13)
Compared to the intractable original optimization in Equation (9), the surrogate optimization problem Equation (13) offers the following advantages: (i) Computational efficiency: The term T(A:) be
5

---

Published in Transactions on Machine Learning Research (03/2026)
computed through simple forward pass in a language model without requiring gradient calculations; (ii) Solution efficiency: Note that the objective function is a quadratic function of the variables c( The general min-norm problem is solvable by the existing Frank-Wolf algorithm (Jaggi, 2013), well-establisted convex optimization mothod. Equation (13) has a closed-form solution (Sener & Koltun, 2018) because Equation (13) only involves two terms.
As result, the multi-objective learning problem in Equation (8) can be solved by iterating two steps: (i) Solving the min-norm problem in Equation (13) to achieve the dynamic weights {c()) and (ii) Optimizing scalarized objective in Equation (8) with the Je Finally, by integrating PPO's advantage function A into Equation (13), our MOC algorithm can train policy taking any preference vector to control the multi-objective alignment. This algorithm is summarized in Appendix D.
i) Flexible preference handling: MOC can accommodate wide range of preferences through a single training process. Its design enables adaptation to diverse preferences without repeated training. ii) Computational Efficiency: By introducing the surrgatee objective in Equation (13), MOC significantly reduces computational costs, making its cost comparable to widely used single-objective RLHFF
Advantages of MOC. We highlight the benefits of our approach:
2.5
An Illustrative Example
Episode Rewards of MOC on Fishwood
Episode Rewards Linear PPO on Fishwood
100
Preference
100
Preference
Episode Reward 2 (Fish)
0.1
Episode Reward 2 (Fish)
0.1
0.2
0.2
0.3
0.3
60
0.4
60
0.4
0.5
0.5
40
0.6
40
0.6
0.7
0.7
20
0.8
20
0.8
0.9
0.9
0
Pareto Front
0
Pareto Front
20
40
60
100
0
20
40
60
100
Episode Reward (Wood)
Episode Reward (Wood)
(a) MOC (Ours)
(b) Linear PPO
Figure Solutions of MOC and Linear PPO on fishwood task and the Pareto front (line in black). MOC demonstrates advantages in both MOO (solutions lie on the Pareto front) and multi-objective control (its solutions align elosely with their corresponding preference vectors, shown as the colored dashed rays). The single model trained by MOC can handle diverse preference vectors. In contrast, Linear PPO optimizes a linear scalarization of the objectives and fails to follow the preference vectors, with solutions dominated by one objective. The legend "Preference" indicates the specific weight value assigned to Reward 1 (Wood). Linear PPO is implemented by optimizing PPO w.r.t. a scalarized reward Rii = w Rwood + (1 w) Rfish (with w e [o, I]). In our experiments, we train Linear PPO runs with weights [0.1, 0.9), [0.2, 0.S). ... [0.8, 0.2), [0.9, 0.1).
To demonstrate the capability of our proposed MOC algorithm, we perform an illustrative experiment on the fishwood task (Felten et al., 2023), where the agent controls a fisherman who can fish or gather wood, receiving corresponding rewards upon completion of each task. The rewards have two dimensions: one for gathering wood and one for fishing. Each collected wood or fished item incrcases the respoctive reward by Detailed experimental setup can be found in Appendix G. The results are reported in Figure 1. MOC aims for (i) MOO: The solutions should reach the Pareto front, meaning the points should be close to the black solid line. (ii) Multi-objective control: The solutions should align closely with their respective preference vectors, indicated by the dashed lines.
The results demonstrate that MOC achieves both goals. (i) Its solutions lie on the Pareto front, demonstrating successful optimization, and (ii) its solutions are close to the preference vectors, confirming effective multi- objective control. Notably, MOC generalizes to diverse preference vectors by training only one model. In contrast, the Linear PPO method, which optimizes a línear scalarization of the objectives, struggles to
6

---

Published in Transactions on Machine Learning Research (03/2026)
consistently follow different preference vectors. In Linear PPO's results, objective often dominates the other in the Pareto sense, well-known phenomenon in convex optimization (see section 4.7 of Boyd & Vandenberghe (2004)).
3
Experiments
0
1.5
1.5
0.5
Helpful
Helpful
0.5
0.0
MOC (Ours)
MOC (Ours)
-0.5
Llama-2
Llama-2
-0.5
MORLHF
MORLHF
Rewarded Soups
Rewarded Soups
RiC
RiC
-1.0
-05
0.5
-0.5
0.0
0.5
1.5
Humor
Harlless
(a) Humor & Helpful
(b) Harmless & Helpful
Figure 2: Controllability comparison on the Pareto front. MOC demonstrates superior controllability, indicated by the consistent ranking of solutions on their preference weights and the achieved reward values. In comparison, the baselines exhibit less stable behavior and weaker alignment with the specified preferences. MOC also achieves higher quality solutions, particularly in the Humor & Helpful alignment. Our MOC method achieves the best overall performance, supported by these results and the findings in Tables 2 to 4. Each point represents the reward achieved across multiple instances, each with different input preference vector. Each point's preference weight for the x-axis reward is the numerical label on its marker.
In this section, we present comprehensive evaluation of the proposed MOC method.
3.1 Experimental Setup
a-ixis
Solution
Implementation. Our implementation based on the open-source TRL package (von Werra et al., 2020). For the language model, we adopt models from the Llama series (Touvron al., 2023; Dubey al., 2024), which are widely used in RLHF studies. We use the prompts from Helpful Assistant dataset (Bai et al., 2022), which provides data for two sets of objectives: {"hu- mor", "helpful") and ("harmless', "helpful'). MOC is trained with a set of predefined preference vectors: ([0.0,1] 1), [0.1, 0.9), [0.9, 0.1), [1, 0.0]]. The training pro- cess was performed on a desktop with an Intel 19-14900K CPU and NVIDIA RTX A6000 GPU. MOC is trained by LoRA (Hu et al., 2022) with rank of 64. The lan- guage model is loaded in 8-bit due to the computational constraints. Additional experimental details are provided in Appendix H. While human evaluation would provide stronger evidence, we follow standard RLHF practice and use soward-model-baee rankings as proxy for human preferences, since reward models trained from human
Hyper-volume
x-axis
Reference point
Figure Illustaaiion of the hyper-volume con- cept. The hyper-volume measures the size of the objective space dominated by set of solutions in multi-objective optimization. Larger hyper- volumes indicate better convergence and diversity of the Pareto front. preference pairs.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Baselines. We compare MOC against three basclines, including (i) The standard MORLHF: multi- objective RLHF method that scalarizes the multi-objective problem into a single objective by combin- ing reward signals with fixed preference weights (in our experiments, the fixed preference weights are {f0.0, (0.3, 0.7], [0.5, 0.5]. [0.7, 0.3), [1.0, 0.0]]). MORLHF does not condition the policy on the preference vector. These weights are only used to scalarize multiple reward-model scores into a single scalar reward, which then passod to PPO. (ii) Rewarded Soups (Ramé et al., 2023): Combines the model weights of N separately trained models using the PPO algorithm, each optimized for a specific reward function; (iii) RiC' (Yang et al., 2024c) learns mapping from preference vectors to reward signals and then conditions generation by injecting the corresponding reward signals into the prompt (i.e., reward-as-context prompt conditioning). RiC is trained with SFT. Rewarded Soups and RiC use the same set of preference vectors as MOC. The behavior of the base Llama-2 model is included for comparative analysis. Further discussion of RiC and other related work is provided in Appendix F. More baselines are included in Appendix K.
Mctrics. It crucial to emphasize that in the context of controllability, higher rewards do NOT always equate to better outcomes. Our goal is to ensure that the model's outputs align with the personalized expectations of users, rather than merely maximizing reward scores. Therefore, to evaluate the controllability comprchensively, we employ a multi-faccted approach.
The evaluation focuses on four key aspects, namely: (i) The quality of solution set, measured using hyper- volumes (as illustrated in Figure 3); (ii) Control with preference vectors, assessed by computing the correlation between the model's behavior and the given preferences; (iii) Diversity of solutions, evaluated by computing the mean pairwise distance of the solutions; and (iv) Generalization capabilities to unseen preference vectors. These metrics collectively provide robust principle evaluate the model's ability to optimize competing objectives while achering to user-defined preferences. Additionally, present case studies to provide qualitative insights into the controllability of MOC with uper-specified preferences.
3.2 Main Results
Figure 2 illustrates the results for two pairs of reward models, with coordinates representing the average rewards corresponding to different preference vectors.
Our evaluation reveals two primary findings: (i) Controllability: MOC demonstrates superior controllability compared to the baselines. This is evident in how consistently the model's behavior aligns with the rank order prescribed by the preference vectors, maintaining clear monotonic relationship between given preferences and corresponding rewards. In contrast, MORLHF, Rewarded Soups, and RiC exhibit less consistent behavior under corresponding preference settings; (ii) Solution quality: MOC dominates the baselines in terms of overall Pareto front quality, as confirmed by the quantitative results below.
Table 2: Controllability comparison using Kendall's tau correaation (higher is better), with corresponding p-values in parentheses (smaller is better), measuring the consistency between input preferences and output rewards. MOC significantly oulperforms all the baselines.
Setting
MOC (Ours)
RiC
MORLHF
Rewarded Soups
Humor-helpful
1.00 (5.0 x 10-*)
0.78 (3.3 x (3))
1.00
(1-7x1002)
1.00 (5.0 x 10-*)
Harmless-helpful
1.00 (5.0 x 10-s)
0.91 (3.0 x 10-2)
1.00
(1.7x x 10-2)
0.96 (5.5 x 10-²)
Average
1.00
0.85
1.00
0.98
Alignment with preferences. To assess how well different algorithms align with the prescribed preference vectors and model behavior, we adopt Kendall's tau rank correlation (Kendall, 1938) as the evaluation metric. Kendall's tau quantifies the degree to which the relative ordering of outputs is consistent with the rank order determined by the preference vectors, capturing the model's ability to maintain rank moen--reerrving relationships. A higher Kendall's tau indicates better alignment between the model's output ordering and the target preference ordering. Concretely, we compute an angular projection score for each output based on the angle (computed via arctan) of its reward vector relative to fixed reference point, rank outputs by this
8

---

Published in Transactions on Machine Learning Research (03/2026)
score, and then compute Kendall's tau against the preference-indueeed ranking (see Appendix I for details). The results are shown in Table 2. Across two evaluation settings, MOC achieves the highest Kendall's tau, demonstrating its superior capability to align model behavior with user preferences and accurately reflect human preference rankings.
Quality of solutions. We use the hyper-volume indicator, a standard metric MOO, to measure the quality of solutions. Hyper-volume captures both convergence to the Pareto front and the diversity of the solutions in the objective space. Table 3 shows that MOC significantly outperforms all baselines. For instance, in the Humor- Humor-Helpful setting, MOC achieves hyper-volume of 14.176, compared to 6.692 by RiC; similar trends are observed in the Harmless-Hellfull setting. These results indicate that MOC exhibits superior convergence to the Pareto front and maintains more diverse set of solutions, ensuring that it explores a broader range of trade-offs between objectives.
Table Hyper-volume (higher is better) comparison of different. methods, measuring the volume of solutions dominated by each method's solution set. MOC outperforms all baselines, achicving higher solution diversity and quality. The best score marked with the blue color box.
Setting
MOC (Ours)
RiC
MORLHF
Rewarded Soups
Humor-helpful
14.176
6.692
6.769
6.100
Harmless-hclpful
10.220
9.257
9.047
8.905
Average
12.198
7.974
7.908
7.502
Table 4: Comparison of solution set pair-wise distances (measuring diversily, higher is better) across methods.
Setting
MOC (Ours)
RiC
MORLHF
Rewarded Soups
Humor-helpful
1.439
1.260
1.057
1.005
Harmless-helpful
1.600
1.363
1.110
1.015
Average
1.520
1.312
1.084
1.010
Diversity of solutions. We measure the diversity of solutions by computing the mean pairwise distance (MPD) of the solution set in the objective space. MPD directly quantifies how spread out the solutions are, making it particularly suitable for our setting. A higher MPD indicates greater behavioral diversity. Table shows that MOC consistently achieves the highest MPD values, outperforming all baselines. For example, in the Harmless-Hclpful setting, MOC obtains an MPD value of 1.600, while RiC obtains 1.363. This result aligns with the observation in Figure? 2, where the solutions produced by RiC tend to cluster more closely, resulting in less diverse behavior.
3.3 Generalization to Unseen User Preference
In this work, we define "unseen preferences" valid preference vectors (where Σp: 1) sampled from the continuous probability simplex that were not present in the training set. Since our training set includes the boundary vertices of the preference simplex, evaluating on these unseen preference vectors strictly assesses the model's capability for continuous interpolation across the Pareto front, ensuring the model has learned a smooth manifold rather than discrete memorization. We evaluate the ability of our model to generalize to unseen preference vectors not included in the training process. Although MOC initially trained on predefined set of preference vectors, the goal is to determine if it can handle new, untrained preferences effectively. To test this, we uniformly sampled four sets of unseen preference vectors and provided them as inputs to the trained model for inference. The results, as depicted in Figure 4, confirm that the model maintains strong performance across all tested scenarios, without any obvious degradation in its behavior. Check Appendix J for more details and qualitative results.
The results highlight several key advantages of the model trained by MOC: i) The model's performance does not degrade when presented with unseen preferences. ii) The model's behavior still adheres to the input.
9

---

Published in Transactions on Machine Learning Research (03/2026)
MOC (Ours)
28
26
43
14
20
60
Hellptul
86
68
87
jammlesss
Harmesss
Harmless
Harmless
66
28
Hoppul
43
36
84
86
64
68
g0
Humor
Humor
Humer
Humer
Figure 4: Generalization to unseen preference vectors held out from the training. MOC and RiC-trained LLMs are compared on four random sets of unseen preference vectors. Each column corresponds to a different. set of unseen preference voctors, and each row represents different pair of reward settings. MOC solutions dominate the RiC solutions in most cases. MOC's rewards align with the new preference vectors and the outputs under different preferences are diverse in the reward space. This suggests MOC generalizes to unseen preferences and achieves diverse trade-offs on the Pareto front. The size of each point indicates the standard deviation in rewards. The numerical labels indicate the preference weights (multiplied by 100) for the reward on the x-axis, enhancing visual clarity.
preference vector, maintaining aligmment between behavior ranking (represented by rewards) and preferences. iii) The model demonstrates sufficient diversity in its behavior, distributing its rewards across broad range of outcomes rather than concentrating on a narrow region of the objective space. These results suggest that MOC can successfully accommodate a diverse range of trade-offs dictated by new preferences, even when they significantly differ from those encountered during training.
3.4 Additional Experiments
To further demonstrate MOC's capabilities, present series of experiments highlighting its advantages across various settings. These include: (1) its ability to generalize to untrained preference vectors (Appendix J), (2) its performance across different model types and sizes, compared with more baselines (Appendix K), (3) its adaptability to various datasets and reward models (Appendix L),, (4) its effectiveness in handling more objectives (Appendices M and N), (5) ablation study (Appendix O), and (6) a case study (Appendix P).
3.5 Discussion
The experiments reveal four key advantages of MOC. (i) Solution Quality: MOC achieves the highest solution quality, evidenced by hyper-volume, reflecting convergence and diversity, (ii) Controllability: MOC demonstrates superior controllability, ensuring consistent alignment with user preferences across diverse objective trade-ofls. (iii) Solution Diversity: MOC outperforms baselines, confirming its robustness in capturing user preferences. (iv) Generalization: MOC's ability to generalize to unseen preferences highlights potential for real-world applications where new preferences may emerge. These advantages demonstrate MOC offers a powerful and flexible approach for multi-objective controllable language models, outperforming baselines in controllability and diversity while maintaining computational efficiency.
10

---

Published in Transactions on Machine Learning Research (03/2026)
4 Related Work
The alignment of LLMs with human values is a central challenge (Ouyang et al., 2022; Bai et al., 2022), and moving beyond a single, developer-defined preference to accommodate diverse, multi-obeective user needs is an open problem. We first review foundational MOO and then situate MOC within the current landscape of multi-objective control methods for LLMs.
4.1 From Multi-Objective Optimization to Controllable LLMs
The principles of MOO, which aim to find set of Pareto-optimal solutions for competing objectives, are well-established (Désidéri, 2009; Sener & Koltun, 2018). However, applying these principles to large-scale LLMs presents unique challenges. Early MOO work in machine learning often focused on optimizing multiple loss functions simultaneously (Liu et al., 2021; 2023), a paradigm that does not. directly map to controlling the fine-grained, semantic behaviors of L.Mss.
Closcr to our domain, some methods rely on a linear scalarization of utility or reward functions (Yang et al., 2019), a technique also employed by standard MORLHF. While simple, this approach is theoretically limited and often fails to identify solutions in non-convex regions of the Pareto front (Boyd & Vandenberghe, 2004). Other algorithms trace the full Pareto front (Mahapatra & Rajan, 2021; Zhang et al., 2024), but their high computational complexity makes them intractable for finetuning Dillion-parameter LLMs.
MOC addresses this challenge. Our work differs from these foundational methods in two ways: (i) we directly manipulate model behavior in the more meaningful reward space rather than loss space, and (ii) we introduce a novel surrogate objective (Theorem 1) that makes principled MOO computationally efficient and comparable standard single-objective RLHF', thus overcoming the scalability bottleneck of prior work.
4.2 Multi-Objective Control of LLMs
Recent years have seen several methods specifically for multi-objective LLM control, which can be broadly calegorized by their approach.
Training Single Steerable Policy. This paradigm, which MOC belongs to, aims to train one versatile model that can be stcored at inference time. The most related works use a numerical vector to specify preferences. Wang et al. (2024) provides a foundational RL framework for this but, like MORLHF, relies on linear reward scalarization. Guo et al. (2024) presents a powerful, RI.ffree alternative by extending DPO to handle multiple objectives, making it a key intellectual parallel our work. RiC (Yang et al., 2024c) conditions the model on reward values via prompt engineering and uses rejection sampling. While this achieves a form of control, it lacks an explicit policy improvement mechanism, limiting its ability to push the Pareto frontier outwards. MOC, by integrating its MOO formulation directly into a PPO-based policy-gradient objective, explicitly optimizes for both alignment and reward maximization. In contrast to these, MOC employs a more sophisticated MOO gradient calculation, enabling a superior exploration of the Pareto front, as demonstratod by our hyper-volume and diversity metrics (Tables 3 and 4).
Ensemble and Multi-Model Methods. Other methods achieve diverse outputs by training or combining multiple models. Rewarded Soups (Ramé et al., 2023) interpolates the weights of separate models, each finetuned on single reward. Similarly, MODPO (Zhou et al., 2024) trains M distinct models for M preferences. These approaches are computationally expensive and -torage-intensive, directly contrasting with MOC's "one model for all" design. MOC achieves superior results with the efficiency of training a single model.
4.3 Alternative Control Interfaces and Methodologies
Linguistic and Implicit Control. Some works explore more user-friendly control interfaces. Nguyen et al. (2024); Yang al. (2024a) use explicit linguistic tags (c.g., [formality: high)) for control, while MOSLIM (Zhang et al., 2025) infers preferences implicitly from the user's natural language prompt. These methods trade the precise, gramular control offered by MOC's numerical vectors for enhanced interpretability
11

---

Published in Transactions on Machine Learning Research (03/2026)
zero-shot prompting. MOC's approach is complementary and particularly suited for applications requiring precise, backend control over model attributes like safety and factuality.
Inference-Times Control. Another approach is to enforce constraints during decoding. Shi et al. (2024); Son et al. (2025) pioneer methods to provide formal guarantees on objective trade-offs at inference time. This technique applies to any L.M but incurs latency and can degrade text quality if constraints are too severe. MOC, as a training-time method, internalizes these trade-offs, enabling fast, coherent, and controlled generation without specialized decoding, making it practical for deployment.
MOC provides a unique and powerful framework. It is single, efficient, and steerable model that: (i) does not require training multiple models, unlike Rewarded Soups or MODPO; (ii) does not rely on preference datasets, unlike CPO (Guo et al., 2024); (iii) maintains explicit policy improvement, unlike RiC; and (iv) generalizes to unseen preference vectors, which is a primary design goal.
5 Conclusion
In this paper, we introduced MOC, a novel approach that enables the personalization of LLMs by adapting to diverse user-specified preferences. MOC addresses the limitations of existing LLMs, which are typically constrained by fixed developer-specified preferences, by formulating multi-objective controllability as multi- objective optimization problem. By introducing surrogate oplimization in RLHF, MOC enables a single fine-tuning process to adapt to wide range of user-specified trade-offs while many existing methods require training multiple separate models. Our experiments demonstrate that MOC surpasses existing baseline methods in controllability, solution quality, and generalization while maintaining exceptional computational efficiency. By optimizing over the explicit Pareto front rather than collapsed scalar values, MOC enables precise, continuous control over multi-dimensinnal model behaviors. This work highlights the potential of MOC to transform how LLMs interact with users, offering scalable and customizable solutions that meet diverse needs while maintaining compulational leasibility. Looking forward, MOC paves the way for fully personalized systems. While our framework utilizes precise numeric vectors for optimization, it effectively serves as a foundational control layer compatible with intuitive user interfaces. Specifically, natural language instructions can be mapped to these continuous vectors through a lightweight translation module, connecting human intent with fine-grained model control. This modular design facilitates the integration of MOC into real-world applications where both interpretability and precision matter. Ultimately, MOC represents a significant step toward realizing fully personalized and human- Aaman-friendly systems.
Acknowledgments
The work of Q H. and S. M. was supported by the Federal Ministry of Research, Technology and Space (BMFTR) under Grant 16KTS2411. The work of Y. Y. and M. P. was supported by the TKI Smart project. The role of Y. Y. as the primary contributor to the algorithm design and thcoretical components is explicitly acknowledged here.
References
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXio:2309.16600 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nclson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/ hbs//2200.0562.. 2022. doi: 10.48550/ AAR.00///RR...... 2204.05862. URL sttps://dol: wtw.:...////////////////////oo// 48500/ 48550/arXiv arxiv. 2204. 05862.
Stephen P Boyd and Lieven Vandenberghe. Conver oplimization. Cambridge university press, 2004.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Jean-Antoine Désidéri. Multiple-Gradient Descent Algorithm (MGDA). Research Report RR-6953, June 2009. URL. https://innia: hha...aaaaaa...........aaaaaa... simultaneously n smooth and unconstrained criteria is considered. A descent direction common to all the criteria is identified, knowing all the gradients. An algorithm defined in which the optimization process is carried out in two phases one that is cooperative yielding to the Pareto front, and the other optional and competitive.
In this report, the problem of minimizing
Jacob Devlin, Ming- Sing-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171 4186. Association for Computational Linguistics, 2019. doi: 10.18653 (118663/V1/N19-//1 1423. URL https: org/ 10. htt.:/o///....1//1///111//10 18653/v1/119-449
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. /ttps://github.coo ipenai/baselines, 2017.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 21783, 2024.
Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. A toolkit for reliable benchmarking and rescarch multi-objective reinforcement learning. In Proceedings of the 37th Conference on Newral Information Processing Systems (NeurIPS 2023), 2023.
Jian Guan, Jun Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey on personalized alignment the missing piece for large language models in real-world applications. In Annual Meeting of the Association for Computational Linguistics, 2025. URL https://apt semanticscholar Teg/CorpusIDD. 277244364.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Controllable preference optimization: Toward controllable multi-objective aligument. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 1437-1454. Association for Computational Linguistics, 2024. doi: 10.18653/ (16653//1//// 12202..MMNLPMMANN
URL attps://dol /ttp://..orgg....1/////////0/0 maw.::e.m.mmmmmmmm...mmmmmmmm
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.ned net. 2022. URL
https: //penrvi/peerevier. het.iiiweete/ooommttddeeeeeeeee
Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Proceedings of the 30th International Conference on Machine Learning, 1CML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 427-435. JMLR.org, 2013. URL http: //proceedings. mlr, prtss/peesssssseeeeeee jatp///aagi1... htmll.
M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-99 1938. ISSN 00063444. URL htttp//www.jjttnn arg/stablle/333226.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Posler), 2015.
URL http:// // arxiv. orgg/bs// http//as//11... 6980.
Shuyue Stella Li, Avinandan Bosc, Facze Brahman, Simon Shaolei Du, Pang Wei Koh, Maryam Fazel, and Yulia Tsvetkov. Personalized reasoning: Just-in-time personalization and why 1mms fail at it. ArXio, abs/2510.00117, 2025. URL. https://apii www.eeiisaaoollleeeoee seemaatteeccllla....oor/rroaaaaa Drc.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 18878 18890, 2021. URL http:://proceedinss.eeeee http:///w...rrippp.caaaap//////a / 6 Abtroott.bttrat......m html.
Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. FAMO: fast adaptive multitask optimization. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (edls.), Advances in Newral Information Processing Systems 36: Anmual Conference Newral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers nips cc/ paper paper_filest www.fffllfppppee /203 200023//aah//ba hashh/hhhhh///eee eedde66a00.d22222225552002
Chsttract-Coneerennce html.
Pingchuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration in multi-task learning. In Proczedings of the 87th International Conference on Machine Learning, 1CML 2020, 18-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 6522 6531. PMLR, 2020. URL
http://proceedingss mlr. prwss/vlsivv vts//v119///1 1t219//mm......m ham0
Debabrata Mahapatra and Vaibhav Rajan. Fxact pareto optimal search for multi-task learning: Touring the pareto front. ArXiv, abs/2108.00599 2021. URL http::///ap..smmaticccaaaa.e org/ Crrg/oorpsII:: 236772107.
Dang Nguyen, Jiuhai Chen, and Tianyi Zhou. Multi-objective linguistic control of large language models. In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 4336-443600 4347,447, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10.18653/v /1/.///..../////// findings- URL /ttp///aaattooggy..... www.h/hoooooooo//2224/44/// findings-arn aal. 257/.
OpenAI. GPTT4 technical report. CoRR, abs/ abs/2303.00877 2023. doi: 10.48550/ LA8555/AAXXI..22 130.XII..303008774 URL https://doi. http///oo.oro////// 10/ 18550/arXiv 23..22233.222
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http://papers nips. cc/paper_ppp.p htcec/pper_fflleeeeeeeeeeeeeeeee ffilea/paperr/000 htmll.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training, 2018
URL attps://apil semanticscholar wtt.://ccccccca.oooorroossssaaaa http:/CoppusID:33122222
Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean- Cann-Bapisste Gaya, Mustafa Shukor, Laure Soulier, and Matthies Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (ods.), Advances in Neural Information Processing Systems 36: An- wual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. URI. http://papers. wippprrrr.iip...mmmm..p. cccpapprrfilleee fallffillspppppp httlle//aperr/223/aaeae e1 h.a.b0000ee830fsstttccc0cc...66. Conferacct-oooeeeene...... html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/ abbs/1707.00 ////177700664.7 2017. URL http://arxiv.ott org/ abs/ 1tt///1770000 1707.06347.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Besa-Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 525-536, 2018. URL https: / http://.ww........ooo/////aaaaaa
Abstractb..rttt.ee html.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A. Smith, and Simon S. Du. Decoding-time language model alignment with multiple objectives. CoRR, 10.48550/ /t////RRIIII///
2024. doi:
URL attps://dol: /0//. 48550/arXiv. wtt..../....../.............o. 2406 2406.18853. 18853.
Seongho Son, William Bankes, Sangwoong Yoon, Shyam Sundhar Ramesh, Xiaohang Tang, and Nija Bogunovic. Robust multi-objective controlled decoding of large language models. CoRR, abs/ abs/2503.00 02///250.0087 087906 2025. doi: 10.48550/ARXIV 250.3.08790 URL attps://don org/ 10. 48550/arXXX 2503. 0879. 08796.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Hugo Larochelle, Marc'Aurelaa Ranzato, Raia Hadscll, Maria-Florina Balcan, and Hsuan- Tien Lin (cds.), Advances in Neural Information Processing Systems 33: Annual Conference Newral Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020. URL https: Absrrttttttttt.....m html.
www.eeee.rrrpppppeeeeeee cc/pperr/
2020/hash. h (110019669999999900000000
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint or 330::3377.09288, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Mllia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Informalion Processing Systems 30: Annual Conference Newral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https://proceedinn iet.reeddinnn..euip....... neurips. Abstract. html.
cc/ cc/paper/2017. http://///.60222444444111111.000
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. httpp://gitee httpe///gghuub..cmuuggggggggeee www/tuggiingface//aaa 2020.
Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Kumar Avinava Dubey, Alexandre Rame, Johan Ferret, Geof- frey Cideron, Le Hou, Hongkun Yu, Amr Ahmed, Aranyak Mehta, Leonard Hussenot, Olivier Bachem, and Edouard Leurent. Conditional language policy: A general framework for steerable multi-objective finetuning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pindings of the Asso- ciation for Computational Linguistics: EMNLP 2024, pp. 1183-2186, Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguistics. doi: 10.18653/v/ http///2...nnnnnseeeeee org/2024. wtt.2........n.... findings wiwdinng--emmlp... cammp..18//
URL
Peiyao Xiao, Hao Ban, and Kaiyi Ji. Direction-oriented multi-objective learning: Simple and prov- able stochastic algorithms. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL httptpppp//pppp...eeem nips. ctwppperrrereepppppppppp httplpppprffefppaepaerrpppeeeeee wtiles/paper/2023//1 html. html.
Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. Prediction-guided multi-objective reinforcement learning for continuous robot control. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10607-10616. PMLR, 2020. URL http://prooceddngs. mlr. press/ v1 21199xu20h.htmm
Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. Metaaligner: Towards generalizable multi-objectiive alignment of language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Qwen An Yang, Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Läu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXio, abs/ 201/2441..1115,, 2024b.
URL attps://appl semanticscholar. wwaeechooaaroog/orrpoooo org/CorpuaIDD 274859421.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective aligument of loundation models with dynamic preference adjustment. Forty-first International Conference on Machine Learning, ICMT, 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nee 2024c. URL https://opent: /ttpr///eeeprreieee.emmmmr wwwpperrvieeeeee/fforneeeee wwwp/oorummdd--Lcaaaaa.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective re- inforcement learning and policy adaptation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence AAlcéé-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Infor- mation Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pp. 14610-14621, 2019. URL https: //proceedings. neurips. wwtpcccpppp//eeeece 20trr//21////aa
Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-based offline reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL 445234444444712104444444aaaaaaa0
DqcZu8JIIzS.
Xiaoyuan Zhang, Xi Lin, and Qingfu Zhang. PMGDA: preference- based multiple gradient descent algorithm. CoRR, abs/2402.09492, 2024. doi: 10.48550/ARXIV..20 2400000222222222992 URL https://doi.orr .tt.:/........//............ 2402.09499
488500/rX1iv
Yu Zhang, Wanli Jiang, and Zhengyu Yang. Moslimaalign with diverse preferences in prompts through reward classification. CoRR, 2as/2505.203366 2025. doi: 10.48550/ARXTS 250.VV22000203366 URL https:
vidoi.oor//00 48550/ aaroo.rxiv.... merrii..205..... 20336.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one- preference-fitee fits-all aligument: Multi-objective direct preforence optimization. In Findings of the Association for Computational Linguistics ACL, 2024, pp. 10586 10586-10613, 2024.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Appendix Table of Contents
A Proof of Theorem 1
18
B Pareto Optimality and MOC's Advantages
19
C Approximated Normalized Vector Similarity
20
D Pseudocode
21
E Why RL Loss Functions Are Unsuitable for Preference Control
22
F Further Discussion of Related Work
24
G Details of the Illustrative Example
25
H Details of Language Model Experiments
26
Kendall's Tau Computation Details
27
J Generalization to Unseen Preferences
28
K Generalization Across Model Types and Sizes
30
L Generalization to Different Datasets and Reward Models
32
M Three-Objective Controllable Generation
32
N Scalability to More Objectives
33
O Ablation Study
37
P Case Study
39
17

---

Published in Transactions on Machine Learning Research (03/2026)
A Proof of Theorem
=u)()))) denote the probability ratio of PPO objective (Equation (1)), and let be the clipping Sopper-paramen as defined PPO (Schulman et al., 2017). The upper bound of objective in Equation (9) is given by
Theorem 1. Let =(0) <ggl==)
N
N Σ RR' - (Aj) 1=1
(1)
Σ
ps (A,) - e2
x V=T(=;0,p) 2
(10)
2>0
j=1
where
0, if (A > 0 and 2 > 1 ++)
(A and < - c),
I(A)
(11)
A,
if (A > 0 and 2 - 1 +)
or and
c),
Σ 1=1
= 1,
( 2 0 Vi,
(12)
where A, denotes the advantage estimate.
Proof. One can further expand Equation (9) with the PPO loss and get
c p VoJ(T(:;(.))
CA - Φ)
Pj -U ()
j=1
N
Pj 2w - o)Vo
j=1 N
2
(14)
- -ww.
Pj
の)
Von (:
N
Pj
1
- (2) - MEWE) 620
(Rl - Pj)
(AA)
V
(
Told
nold
j=1
N
<
11)
Σ
PA)(A)) - 2c RG.. 000
R (A,)((A,)
(.
j=l
1=1
where
0, if (A > 0 and = > (1+c))
2 < <1-c) A, if (A > 0 and S (1 + +e))
or (A 0 and
I(A)
or (A < 0 and
2 > 1 - e)
(0)
0
Vi,
1=1
and == Toid
Cauchy-Schwarz x The third inequality holds by Cauchy- Schwarz inequality and the fourth equation holds by integrating the PPO loss function.
18

---

Published in Transactions on Machine Learning Research (03/2026)
B Pareto Optimality and MOC's Advantages
In this section, we provide formal definition of Pareto Optimality and its relevance to policy improvement.
Formal Definition of Pareto Optimality
Definition 1. Let x' E X, where X' is the set of feasible solutions. A solution x is said to dominate another solution x' if and only if:
J((T) 2 Ji(T') for all E
NN, and
Jz(x) > J²(r') for at least one j
Here, (T) denotes the value of the i-th objective for the solution The above conditions imply that æ performs at least as well as r' in all objectives and strictly better in at least one. Solutions that are not dominated by any other are termed non-dominated and collectively form the Pareto front.
Definition 2. (Pareto Optimality) Let denote the set of feasible solutions, and x x E a vector-valued objective function where J(x) = (J1(z), J. corresponds the objective values associated with X. solution x² X is Pareto optimal if and only if no other solution n' E X satisfies:
J((T') 2 J((T')
Vi
(15)
and
J (+(*)) for at least
NN.
(16)
This ensures that n" is don-dominated, meaning that no other solution can improve one or more objectives without sacrificing performance at least one other.
Advantage of MOC
Explicit policy improvement refers to methods that deliberately optimize at least one objective J4, ensuring that the solution quality improves by maximizing one or more associated rewards R. This approach is particularly crucial in designing multi-objective policies, it guarantees measurable progress in one more dimensions of performance.
Advantage of MOC Compared Other Baselines
Our proposed method, MOC, explicitly optimizes all objectives with policy improvement while integrating controllability, cnsuring more balanced and efficient approach to policy improvement. In contrast:
Rewarded Soups does not jointly optimize all objectives, which leads to suboptimal solutions.
RiC focuses exclusively controllability but lacks explicit mechanisms for policy improvement, limiting its ability to enhance solution quality.
MODPO does not consider Pareto Optimality during training. Specifically, it trains M separate LLMs (corresponding to M preferences) by optimizing each model with specific weighted combination of reward objectives, given the corresponding reward models.
By integrating both explicit policy improvement and controllability into unified framework, MOC the- oretically achieves higher solution quality compared to these baselines. This is further validated by our experimental results (Tables to 4, 10, 11 and 14 and Figures 2, 4, 6 and 8), which demonstrate that MOC consistently outperforms these approaches across multiple metrics.
The integration of explicit policy improvement with controllability ensures that MOC aligns with the principles of Pareto Optimality while delivering superior practical performance. By addressing the limitations
19

---

Published in Transactions on Machine Learning Research (03/2026)
of existing methods and achieving better balance among competing objectives, MOC sets new benchmark in multi-objective controllable language models.
C Approximated Normalized Vector Similarity
In this paper, the reward signal is normalized to ensure compatibility with the preference vector, enabling effective alignment and optimization. The normalization process is defined as:
Normalize(r) P Tmean + 1,
(17)
2rad
where mmean and Fatd are computed dynamically using a running mean and standard deviation (Dhariwal et al., 2017). This ensures that the range of Normalize(r) is consistent with the preference vector, common practice in deep reinforcement learning (Dhariwal et al., 2017).
The aligmment between normalized rewards and preferences then quantified using the MSE loss, leading to the definition of the Approximated Normalized Vector Similarity (AMVS):
AMVS p) |[Nomalizz(()) -pl?
(18)
which serves computationally efficient approximation of the Normalized Vector Difference (NVD), widely adopted similarity mcasure in MOO. The NVD itself is formally defined as:
NVD(a, b) = a
b [
(19)
These definitions allow the MOC algorithm to optimize each objective while aligning the model's behavior with the user-given preference vector.
20

---

Published in Transactions on Machine Learning Research (03/2026)
D Pseudocode
We summarize the MOC algorithm in Algorithm 1. We recommend that the reader checks Schulman et al. (2017); von Werra et al. (2020) for more training details of PPO in the language model settings. The min-norm algorithm used in MOC shown in Algorithm 2, based on Sener & Koltun (2018). Algorithm 2 gives c(1) and c(2) = c0)
A Algorithm Multi Objective Control Algorithm (MOC) for Language Models
Require:
P:: A finite preference-vector set.
6: Constraint threshold
D: Prompt dataset
The SFT policy x(=;0) with parameters (used with prompt-based preference conditioning, yielding
T(::0, (.:0.p)). p)).
Add N new value heads to the language model Set number of iterations T and mini-batch size B
1: for iteration = 1 to T' do
2: Sample a mini-batch of prompts from D.
3: Sample a mini-batch of preference vectors (p,}-- from IP, where subscript j indexes the mini-batch.
Relabel the prompts with {p;}f=1 by Equation (4) and get {
5: For each zy, generate response Vj ~ 0 Compute (. = (Iz,W,), R2 (z), R (I.. Uj)) by reward models.
6:
Compute the Advantage function A₁ according to the PPO algorithm.
8: Solve Equation (13) by Algorithm 2 and get {(c))) ,
Perform gradient ascending using Equation (8) to optimize the policy. 10: Optimize the N value function of PPO (Schulman al., 2017).
11: end for
12: return Optimized policy
Algorithm Min-norm algorithm for two vectors (min.ega.1) cvv (1 c)vl
Require:
V: Vector v
V: Vector v
1: T TT then
c=
else if v v v
then
c=0 0
5: else
6:
c= |
end if
return
21

---

Published in Transactions on Machine Learning Research (03/2026)
E
Why RL Loss Functions Are Unsuitable for Preference Control
The primary objective in RL is to train an agent to make decisions that maximize cumulative rewards over time. To achieve this, various learning algorithms are employed, each associated with specific loss functions. However, these loss functions do not always directly measure the agent's 's performance achieving high rewards. This discrepancy arises because the losses are often surrogate measures designed optimize certain aspects of the agent's behavior rather than direct evaluations of the cumulative reward.
Value Function Loss
The value function in RL, typically denoted V(s) for state value Q(s, a) for stato-action value, estimates the expected cumulative reward from given state (or state-action pair). The loss function for the value function, often referred to as the Temporal Difference (TD) error, is given by
Ly = Ex + ( V(()))
(20)
where
R₂ is the reward received at time step t,
is the discount factor,
((S) is the estimated value of the current state,
V(S++1) is the estimated value of the next state.
This loss function aims minimizing the difference between the predicted value and the bootstrapping target, adjusted for the discount factor. While minimizing this loss improves the accuracy of the value function estimate, it does not directly ensure that the agent's 's policy maxinizes the cumulative reward. An accurate value function is essential for effective policy evaluation and improvement, but an agent may have low value function loss without necessarily following an optimal policy.
Policy Gradient Loss
Policy gradient methods directly optimize the policy by adjusting parameters to maximize the expected cumulative reward. The loss function for policy gradient mothods, particularly in the context REINFORCE, can be represented as
T
L
En
Σ
log Ta (A.S)) A
(21)
where
(T(A((S:) is the probability taking action A, in state S, under the policy π parameterized by 0,
A, is the advantage function.
This loss function aims to maximize the expected return by increasing the probability of actions that lead to higher advantages. However, the policy gradient loss focuses on immediate policy improvements based on sampled trajectories and advantage est isttmesss, which may not fully capture long-term performance. Additionally, high variance in gradient estimates can lead to unstable training and suboptimal policies even if the loss minimized.
Case using value function as aligned target
One might ask whether using value functions as an aligned target is effective. The experiments in Figure were conducted using the state value function as an aligned target, providing practical case demonstrating its applicability.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Discussion
Both the value function loss and the policy gradient loss serve as proxies to guide the training process toward policies that yield higher rewards. However, these losses do not always correlate perfectly with the agent's overall performance duc to several factors:
Long-term Dependencies: These loss functions primarily focus on immediate or short-term improvements. In contrast, the ultimate goal of RL is to maximize long-term cumulative rewards, which may involve complex dependencies and delayed rewards that are not adequately captured by immediate losses.
Sample Dependence: The loss functions rely on sampled trajectorics, which may not fully represent the underlying state-action space, especially in enviromments with high variability or sparse rewards.
Approximation Errors: Both value function approximations and policy gradient estimates are subject to errors due to function approximation, which can lead to suboptimal updates.
While value function loss and policy gradient loss are essential components of the training process in reinforcement lcarning, they do not provide comprehensive mcasure of the agent's 's true performance in terms of achieving high cumulative rewards. Therelore, these loss functions cannot be effectively used for alignment or control tasks involving preference vectors.
23

---

Published in Transactions on Machine Learning Research (03/2026)
F Further Discussion of Related Work
In this section, we provide further discussion of related work.
While both MOC and RiC (Yang et al., 2024c) aim to personalize LLMs, their methodologies are funda- mentally different. The following significant differences highlight the novelty and distinct contributions of MOC:
Re-labeling the prompt
MOC directly relabels prompts with preference weights while RiC relabels the prompts with reward signal.
Formulation and Approach:
MOC formulates controllability a multi-objective policy optimization with senference-baseee constraints, solved via the proposed MOC algorithm. RiC uses SFT to fine-tune the LLMs.
A key novelty is the surrogate problem (Equation (13)), which reduces the computational cost to near that of single-objective PPO.
In contrast, RiC relies on learning a preferenco-to-roowrrd mapping and lacks an explicit policy optimization framework, making methodology fundamentally different.
Explicit Policy Optimization and Controllability:
MOC explicitly optimizes the policy to align model behavior with user preferences, establishing rigorous and systematic controllability framework.
RiC does not perform explicit policy optimization, limiting its ability maximize reward while aligning with preferences.
Performance Advantages:
Thanks to its principled design, MOC siguificantly outperforms RiC in controllability, solution quality, diversity, and generalization, as substantiated by quantitative results in Tables 2-6.
We summarize these key differences in Table for clarity.
Table 5: Key differences between MOC and RiC
Algorithm
Source of Controllability
Explicit Policy Improvement?
Loss Function
MOC
MOO with constraints
PPO-based SFT-based
RiC
Reward Context
It important to emphasize that MOC focuses on controllability with two critical goals:
Training LLM generate personalized outputs for diverse user preferences.
Generalizing unseen preferences with the once-trained LLM.
In contrast, MODPO:
Trains M separate LLMs (where M corresponds to the number of preferences), targeting only fixed set of pre-defined preferences.
Does not consider generalization to unseen preferences.
These differences highlight the fundamental distinctions between MODPO's methodology (Zhou et al., 2024) and objectives and those of MOC. Therefore, we train a language model with MODPO and inchude the MODPO results in Table 10.
24

---

Published in Transactions on Machine Learning Research (03/2026)
G Details of the Illustrative Example
for more details about the task in Figure 1. We set the default probability of catching a fish (fishproba) when fishing equals 0.5 and also the probability of collecting wood when the woods (woodprob). The Pareto front is computable once fishproba and woodprob are given. Specifically, the Pareto front satisfies the following equation:
Readers can click this link: wttpss//mm-gymmsimmmm
x + y = woodprob * (steps collecting wood) + fishprob * (steps fishing),
(22)
where x is the episode reward of fish and y is the episode reward of wood. Specifically, x + y 100 in our settings. The episodes reward are estimated over 20 episodes. The input of the policy network and the V-network is the concatenation of the state vector and the preference value of the wood (c.g. [initial state vector, 0.1]). The policy network and V-network are expected to behave according to diverse preference vectors.
Sclection of preference vector. The preferences of wood range from 0.1 to 0.9. The following equation gives how we depict the preference vectors.
1 preference_ _oooorcce_oo_odddoo_ooooooooaooe y preference_of_woodd Iz
where preference of__wodd E (0, 1] represents the relative preference for collecting wood.
We list the hyper-paramcters related to this experiment in Table 6.
Table 6:
settings for fishwood task (Section 2.5).
Hyper-parameter
Value
Dimension of state space
Action space
Discrote(2): go fishing, go collect wood
Discount (t)
0.99
Optimizer
Adam (Kingma Ba, 2015)
Learning rate for networks
1 10-4
Number of hidden layers for all networks
3
Number of hidden units per layer Activation function
256
ReLU
Batch size
512
Gradient clipping
False
Exploration method
Epsilon-Greedy
C (Exploration)
0.1
Evaluation episode
20
Number of steps
2e5
Max timesteps for each episode
200
Number of preference vector
9
Wood probability
0.5
Fish probability
0.5
25

---

Published in Transactions on Machine Learning Research (03/2026)
H Details of Language Model Experiments
The key information about the experimental settings is listed in Table 7. To ensure a fair comparison, we use the same dataset as (Yang et al., 2024c).
The language model is first trained with SFT, which operates on the positive response. Then we added value heads the language model.
Table 7: Key information about the implementation.
Hyper-pane parameter
Value
Base model
Llama 2-7B (Touvron et al., 2023)
GPU
A NVIDIA RTX A6000 (48G)
CPU
Intel(R) Core(TM) 19-14900K
Memory
128 G
Quantization for training
8bit
Fino-tuning
LoRA (Hu et al., 2022)
LoRA r
64
LoRA alpha
LoRA dropout
0.05
Optimizer
Adam
Batch size
Inference tokens for evaluation
128 for Helpful Assistant and 48 for Reddit Summary
Helpful Assistant (Bai et al., 2022)
Description
Provide harmless and helpful responses to questions
Prompt
Users' questions
Re-label method
Re-labeled prompt <R1> P1 <R2> Pa <RN> PN {prompt}
Helpfulness
gpt2 large helpful reward model gpt2 large harmless reward model
Harmless reward
Humor reward
Humor no humor
SFT
Finetuning steps Initial learning rate Learning rate scheduler
20000
1.41e-4 Linear
MOC (Ours)
RL algorithm
PPO (Schulman et al., 2017)
Codebase
TRL. (von Werra et al., 2020)
KL regularization
0.2
Epochs
1
New value head
N two-layer ford-forward head
Units of value head
decoder hidden size
Activation of value head in Equation (5)
ReLU
0.1
Learning rate Lambda for GAE
1.41e-5
0.95
Gamma
1
Cliprange
0.2
Number of optimization epochs per batch
4
Target KI.
6
The hyper-volumes in Table 3 are computed by existing package PyGMO.
2rad The reward signal normalized by r r-Tma
ensure the range reward is similar to the preference vector, where the mean and std are computed by running mean in Dhariwal al. (2017). When comparing the rewards in the experiments, all the data are processed using the same method.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Kendall's Tau Computation Details
We compute the Kendall's tau rank correlation cocfficient (Kendall, 1938) to quantify the alignment between the model outputs and the target preference ordering. For each algorithm, we obtain two sequences: (1) the ground-truth preference ranking, derived from the given preference vector values; and (2) the predicted ranking, computed based on the geometric projection model outputs relative to the reference point.
Specifically, for each output (21,72), we define projection score
22 72
$ arctan
(23)
P1 rr
where (rief, r) denotes the reference point. The sequence {si) represents the predicted ordering, which is compared to the ground-truth preference ordering {ps}.
The Kendall's tan r is computed as
No Na nn 1)'
(24)
where No and N& denote the number of concordant and discordant pairs, respectively, and is the number of data points.
A higher t indicates stronger alignment between the predicted order and the target preference order. Our results show that MOC achieves the highest Kendall's lau across all evaluated settings.
We scipy to compute the Kendall's tau in Table 2. The code is listed in Listing
from scipy.ste stats import kendalltau import pandas as pd import numpy as np
Load evaluation data file_path = ri, r2 'Humor', 'Helpful' data = read_ce_ (file_path)
Reference point for projection reference_point = (-2, -2)
Compute projection score (predicted order)
data (rr[['ppjcctionn] pi / np. arctan(
[data[r2] reference_point [1]) / (data[ri] reference_point [0])
Evaluate Kendall's results 0
each algorithn
for algo in data aaaaalAAlotttaa unique()
subset data [data[ Algorithn'1 algo]
predicted_order subset subset['projeet prrojeccionn ground_truth_ored tau, P_value relults[algo] tau
subset (Preeet(PPeeeeencc
(groudd_ggruud
predicted_order?
Listing 1: Code compute Kendall's tau correlation for alignment evaluation.
27

---

Published in Transactions on Machine Learning Research (03/2026)
J Generalization to Unseen Preferences
Formal Definitions of Preference Space and Generalization
To eliminate ambiguity regarding "unseen preferences" and the nature of generalization in experiments, we provide the following formal definitions:
Preference Space (P): We define the valid preference space as the standard (N - 1)-simplex, which represents the set of all possible trade-off combinations for N objectives:
P = PER N Ep= P: 20
(25)
Training Set (7): The model is optimized using a finite, discrete set of preference vectors T C P. Crucially, our training design ensures that T includes the vertices of the simplex (e.g., one-hot vectors such as [1, 0] and [0, 1]), which represent the extreme boundarics of the preforence space.
Unseen Preferences (U)): We define "unseen preferences" as any valid preference vector sampled from the continuous space that was not encountered during training:
u Pp
(26)
Discussion on Interpolation vs. Extrapolation. Based on these definitions, the concept of "extrapola- tion" (often implying evaluating a model outside the bounded domain its training data) mathematically mapplicable to valid preference vectors within our formulation.
Geometrically, because the Training Set T includes the vertices of the simplex, the convex hull of T is identical to the entire Preference Space P (i.e., Conv(T) = P). Therefore, any sampled unseen preference p E u strictly falls within the convex hull of the training data.
Thus, the evaluation on unseen preferences in this paper measures the model's capability for continuous interpolation: mapping the discrete training anchors to a smooth, continuous manifold across the Pareto front. The results in Figure 4 and Tables 8 and demonstrate that MOC successfully learns this manifold, accurately satisfying user preferences that lie between the discrete training points.
Results
To test MOC's generalization ability, we uniformly sampled four distinct groups of random numbers from the range [1, 100]. For each sampled number n, we normalized by dividing by 100, yielding the weight wi for the first reward, represented along the x-axis in Figure The weight for the second reward was computed as wi, ensuring that the two weights sum to one. For visual readability, we keep the n Figure 4. This strategy introduces diverse trade-offss between rewards, thoroughly testing MOC's adaptability to unseen scenarios. The specific sampled values spectrum of preferences for assessing the model's generalization.
visualized in Figure 5, where the four groups represent broad
Table 8: Hyper-volume (HV) Comparison between MOC and RiC, where MOC achieves higher HV (better output quality and diversity under different preferences). Higher is better.
Setting
Group
Group
Group
Group
Humor-helpful (MOC)
17.034
19.697
17.441
19.045
Humor-helpful (RiC)
16.660
16.303
16.304
16.551
Harmless-helpful (MOC)
15.038
14.139
13.324
15.557
Harmless-helpful (RiC)
9.463
10.447
9.342
9.726
28

---

Published in Transactions on Machine Learning Research (03/2026)
Four Groups of Random Unseen Preference
Group 4
3
36
49
Group 3
3
28
60
73
86
Group 2
12
25
43
70
84
Group 1 3
23
43
68
877
Value
Figure 5: Visualization of four groups of randomly sampled, unseen preference vectors. Each preference vector is generated by uniformly sampling a number from the range [1, 100) and converting it a weight w/ for reward with the second reward weight calculated as 1 1- W1. The sampled preference vectors are displayed, demonstrating the diverse set of tradc-offs used for evaluating the model's generalization capabilities.
Note that the hyper-volume values in Table should not be directly compared with those in Table 3. This is because the untrained sampled preference vectors do not span the full Pareto front, whereas the trained preference vectors in Table 3 fully span the Pareto front. As result, certain portions of the Pareto front are absent in the untrained cases, contributing to the observed differences in hyper-volume metrics.
The hyper-volumes for each of the four unseen preference vector groups are presented Table 8, using a reference point of (-3, -3). As shown, there is no significant degradation the hyper-volune, indicating that MOC performs robustly even when exposed to unseen, untrained preference vectors.
Quality.
Alignment. To further evaluate MOC's generalization ability, computed the Kendall's tau between the untrained preference vectors and the behavior (represented by the rewards). These rates, shown in Table 9, measure the degree of agreement between the rankings generated by MOC and the sampled preference vectors. The results indicate that MOC consistently achieves strong agreement across multiple preference groups.
Table 9: Controllability comparison using Kendall's 's tau correlation (higher is better), measuring the consistency between imput preferences and output rewards. MOC outperforms RiC.
Setting
Group
Group 2
Group
Group 4
Humor- Humor-helpful (MOC)
1.000
1.000
1.000
1.000
Humor-hclpful (RiC)
0.800
0.800
1.000
1.000
Harmless-helpful (MOC)
1.000
1.000
1.000
1.000
Harmless-helpful (RiC)
0.600
0.800
0.800
1.000
29

---

Published in Transactions on Machine Learning Research (03/2026)
K Generalization Across Model Types and Sizes
In the following, we present three additional sets of experiments to further demonstrate the capabilities of MOC: (1) generalization across model types and sizes, (2) evaluation on a different dataset, and (3) scalability larger number of objectives. These results reinforce the elfectiveness and scalability of the proposed method.
We extended evaluation to different larger model Llama-3-8B (Dubey et al., 2024) and added MetaAligner (Yang et al., 2024a) and MODPO (Zhou et al., 2024) as baselines. To ensure fair com- parison, RiC, MetaAligner, and MODPO are built on the same Llama2-7B backbone as MOC-Llama2-7B; MOC-Llama3-8B is included to demonstrate that our method also applics to stronger backbones and can further improve performance. Results in Table 10 show that MOC significantly outperforms MODPO, MetaAligner, and other baselines on the HH-RLHF task in terms hyper-volume Considering the significant. compute costs and limitations of MODPO (as discussed in Table 1), the MODPO trained with preference [0.5, 0.5] to show its average performance in comparison. See more discussion in Appendix F.
Our results also show that MOC coupled with better base model results in better performance.
Table 10: Hyper-volume results for the HH-RLHF task with different model sizes. Higher is better.
Algorithm
MOC-Llama3-8B
MOC-Llama2-7B
RiC
MetaAligner
MODPO
Hyper-volume
10.435
10.22
9.257
3.410
3.745
1.5
00:4.6
1.0
0.5
Helpful
0.0
MOC-Llama3-8B
-0.5
MOC-Llama2-7B
RiC
-1.0
MetaAligner
MODPO
-1.5
-05
0.0
0.5
1.0
1.5
Harmless
Figure 6: MOC incorporated with Llama3-8h shows better performance compared to other baselines.
Visualization. A comparative visualization is provided in Figure 6. MOC-Llama3-SB achieves consistently
better performance in optimizing HH-RLHF objectives.
30

---

Published in Transactions on Machine Learning Research (03/2026)
Generalization to Different Model Family (Qwen2.5)
To fürther test. model-family generalizalion, we additionally evaluate MOC with a Qwen2.5 (Yang el al., 2024b) backbone on the two-objective HH-RLHF setting. As shown in Figure 7, MOC remains effective on Qwen2.5, achicving a hyper-vohme of 10.905 and strong controllability with Kendall's r = 0.9636 (p= 10-9). This is consistent with the above comparisons in Table 10, where MOC with Llama backbones consistently outperforms strong baselines (RiC, MetaAligner, and MODPO).
2.0
0
1.5
OGLO
0000
1.0
0.7
Helpful
0.5
0.5
0.6
0.0
MOC (Ours)
0.7
MOC-Qwen2.5-78
0.8
-0.5
Llama-2
0.0.0
MORLHF
0.8
-1.0
Rewarded Soups
0.9
RiC
1.0
-1.5
0.9
1.0
-0.5
0.0
0.5
1.0
1.5
Harmless
Figure 7: MOC with Qwen2.5 backbone on HH--RLHF demonstrates strong controllability and competitive solution qualily.
31

---

Published in Transactions on Machine Learning Research (03/2026)
L Generalization to Different Datasets and Reward Models
We evaluated MOC on the Reddit Summary dataset (Stiennon et al., 2020) using two reward models: Summary, assessing the quality of generated summarics, and Faithful, measuring faithfulness to the original post. Results in Table indicate that MOC significantly outperforms the RiC baseline.
Table 11: Hyper-volume results for the Reddit Summary dataset. Higher is better.
Algorithm
MOC-Llama3-SB
RiC
Hyper-volume
17.556
14.052
MOC, Hyper-Volume: 17.556, Reddit Summary
RiC, Hyper-Volume: 14.052, Reddit Summary
MOC-Llama3-8P
-0.25
-0.30
0.64
-0.35
Summary
Summary
0.61
-0.50
0.55
-0.0
-0.255
-0250
-0.245
-0.240
-0.235
-055
-0.45
-0.40
-035
-0.30
-0.25
Faithful
Faithful
(a) MOC-TJama338B
(b) RiC
Figure 8: Controllability comparison on the Pareto front. MOC demonstrates superior controllability. indicated by the consistent ranking of solutions on their preference weights and the achieved reward values.
Visualization. The performance comparison is shown in Figure 8. MOC demonstrates substantial advantage in optimizing both summary quality and faithfulness.
M Three- Obrjet-vbeeeivvee Controllable Generation
To demonstrate the scalability of our approach, we extended the evaluation to multi-objective setting involving three simultaneous objectives: Harmlessness, Helpfulness, and Humor. We compared MOC against the RiC baseline to assess controllability and solution quality in this higher-dimensional preference space.
Experimental Setup
We utilized set of preference vectors [Wharmlesss Whelpful: Whunor] designed sample critical regions of the preference simplex, including vertices, edges, and the central region. The specific vectors used for evaluation are:
[1.0, 0.0.0.00 00.0 1.0. 0.0], (0.0, 1.0,
[0.5, 0,5,0.0,000 0.5, 0.0. 0.5), (0.0, 0.5.),
1
W Wiest
(27)
3' 3
[0.6, 0.2, 0.2], (0.2, 0.6, 0.2], (02 0.2, 0.6),
(0.4, 0..,0.4
32

---

Published in Transactions on Machine Learning Research (03/2026)
Results and Analysis
Controllability (Kendall's 7). Table 12 presents the Kendall's T correlation between the inpul preference weights and the achieved reward scores. MOC demonstrates significantly higher controllability across all three objectives compared to RiC. MOC achieves an average correlation of T 0.661, whereas RiC achieves only 7 = 0.333. Notably, the p-values for MOC are consistently below 0.05, indicating statistically significant alignment, whereas RiC fails to achieve significance the majority of cases (e.g., Harmlessness p -=000. 0.342).
Table 12: Kendall's correlation and p-valucs for 3-objective control (Harmlessness, Helpfulness, Humor). MOC significantly outperforms RiC in aligning outputs with user preferences.
Objective MOC (Ours)
MOC (Ours)
RiC
RiC p
Harmlessness
0.828
6.4 10-
0.229
0.342
Helpfulness
0.506
0.038
0.343
0.154
Humor
0.648
0.007
0.428
0.079
Average
0.661
0.333
Solution Quality (Hyper-Volumo), We further evaluated the quality and diversity of the solutions using the Hyper-Voluma (HV) metric, which measures the volume of the objective space dominated by the solution set. As shown in Table 13 and visualized in Figure 9, MOC covers much larger region of the Pareto front. MOC achieves Hyper-Volume Volume of 50.331, nearly doubling the performance of RiC (27.629). This indicates that MOC can generate a more diverse set of high-quality solutions that effectively trade off between harmlessness, helpfulness, and humor.
These results provide strong empirical evidence that MOC is not limited to simple bi-objective trade-off's but scales effectively to dimensioonl preference spaces. In the 3-objective setting, MOC maintains superior controllability and covers a significantly wider range of the Pareto front (as evidenced by the nearly 2x improvement in Hyper-Volmme) compared to the baseline. This confirms that MOC can successfully manage the complex interactions between multiple conflicting objectives, making it robust framework for real-world applications where users often have diverse and multi-facete requirements.
Table 13: Hlyper-oolume comparison for the 3-objective sctting. Higher is better.
Algorithm
Hyper- Hyper-Volume Volume
RiC MOC (Ours)
27.629
50.331
N Scalability to More Objectives
To assess MOC's scalability, tested it the 6-objective Fruit-Tree task from the MO-Gymnasium benchmark. This task involves navigating binary tree of depth 6 to optimize 6-dimensional reward vector representing nutrient values.
Results. As shown in Table 14, MOC achieved siguificantly higher mean hyper-volume compared to the Linear PPO baseline, indicating superior performance.
Visualization. Figure 10 illustrates the density distribution of three selected objectives, highlighting MOC's dominance over Linear PPO.
Implementation Details.
and settings for the Fruit-Tree Table 15 summarizes the
task.
33

---

Published in Transactions on Machine Learning Research (03/2026)
3D objective surfaces
MOC (Ours)
MOC (Ours) HV: 50.331 RIC HV: 27.629
RIC
1.4
1.2
1.0
Humar
0.8
0.6
0.4
0.2
1.8
1.6
1.4
0.75
1.2
Helpful
1.00
1.25
1.50
0.8
Harmless
1.75
0.6
2.00
2.25
0.4
Figure 9: Visualization of the 3D objective surface (Parcto front approximation) for Harmlessness, Helpfulness, and Humor. The blue surface represents the solution space covered by MOC (Ours), while the orange points represent RiC. MOC successfully interpolates across the 3D simplex, whereas RiC solutions cluster in limited region.
Table 14: Hyper-volume Results for the Fruit-Tree Task (6 Objectives). Higher better.
Algorithm
MOC
Linear PPO
Mean
15605.90
5741.79 877.43
Variance
752.97
The results validate MOC's capability to generalize across models, datasets, and a larger number of objectives, highlighting its robustness and scalability.
Discussion.
34

---

Published in Transactions on Machine Learning Research (03/2026)
Objectives 1, 2, 3 Density
Linear-PPO
MOC
Objective
2
Objective
3
Objective 1
2
6
7
Figure 10: Distribution of selected objectives: MOC (warm colors) dominates Linear PPO (cool colors).
35

---

Published in Transactions on Machine Learning Research (03/2026)
Table 15: Implementation details for the Fruit-Tree task.
Setting
Value
RL. backbone
PPO
Number of random seeds
5
Discount (7)
0.99
Optimizer
Adam
Learning rate for networks
10-4
Number of hidden layers
Number of hidden
256
Activation function Batch size
ReLU
100
Gradient clipping Exploration method Entropy Coefficient Epsilon-clip for PPO Epochs per PPO update Timesteps every update Maximum episode timesteps
False
Policy Entropy
0.001
0.001
100
100
Episodes per preference sample
20
Number of preference samples
2400
Evaluation episodes
36

---

Published in Transactions on Machine Learning Research (03/2026)
O
Ablation Study
An important component of MOC is the bi-objoctive fine-tuning objective (Equation (7)), which couples (i) multi-objective reward optimization for improving solution quality and (ii) preference-comditioee controlla- bility via a constraint term. All ablation results in this section are obtained using Qwen2.5-7B backbone. For clarity, we restate Equation (7) and annotate the two objectives:
I(T(;8,p)) def
P (T((O...))))
RRLLU ()).)).))
MO optimization solution quality
controllability proference alignment.
We ablate these two objectives by dropping one term at time during fine-tuning, while keeping the same prompt-based preference conditioning for the policy and the same inference procedure. Figure 11 and Table 16 summarize the results.
Only MO-optimization objective (w/o controllability). We remove the controllability patilabbiiitylliigmmmt /aligmment objective and optimize only p [TT(;;0, This variant can still improve solution-set quality (Hyper-volume = 6.334), but controllability drops substantially (Kendall's tau 0.5636, = 0.016541), indicating weaker coni-consis between the input preferences and the realized reward trade-offs.
Only controllability /alignment objective (w/o MO optimization). We remove the MO-optimization objective and optimize only the hinge penalty term ReLUUMSEE(ER p) o). Without directly optimizing rewards, the solution set quality remains low (Hyper-volume 1.869) and the measured controllability becomes negligible (Kendall's tau 0.0182, p 1.0).
Full objective. Combining both objectives yields both high-quality solutions and strong controllability (Hyper-volume = 10.905, Kendall's tau = 0.9636, = 10-6). Overall, the MO-optimization objective is essential for improving the Pareto solution quality, while the controllability objective crucial for preserving preference-conditioneed control.
Table 16: Quantitative results for the objective ablation. Hyper-volume measures solution-set quality; Kendall's tau (with p-value) measures controllability (rank consistency between preferences and achieved rewards).
Method
Hyper-volume ?
Kendall's tau
p. +
MOC (Ours)
10.905
0.9636
0.000001
Only controllability/alggnna (w/o MO optimization) 1.869
0.0182
1.000000
Only MO optimization (w/o controllability)
6.334
0.5636
0.016541
37

---

Published in Transactions on Machine Learning Research (03/2026)
2.0
MOC (Ours)
w/o MO Optimization
120.4
1.5
0.3
w/o Controllability
0.5
1.0
0.8
8
Helpful
0.5
0.
19...
0.0
-0.5
-1.0
-1.5
-2.0
-1.0
-0.5
0.0
0.5
1.0
1.5
Harmless
Figure 11: Ablation the two objectives in Equation (7). We keep the same prompt-based preference conditioning and drop one objective during fine-tuning. Removing the controllability objective reduces preference aligument (lower Kendall's (au), while removing the MO-optimization objective prevents improving the solution-set quality (lower Hyper-volume).
38

---

Published in Transactions on Machine Learning Research (03/2026)
P Case Study
We present some cases in Table 17. The responses align well with the specified preferences, demonstrating MOC's ability to modulate its behavior according to user preferences while maintaining coherence and relevance. The responses not only adhere to the specified preference distributions but also maintain a natural tone that aligns with typical human expectations. For example, the respouse with a preference vector heavily weighted towards helpfulness (helpfulness=11, humor=0) provides practical advice in a clear and straigh manner, while responses with more balanced preference vector (Humor=055, helpfulness-055) introduce elements of creativity and light-hearten heartedeess without sacrificing utility. The results demonstrate that the model can tailor its output to match specific preference settings while still resonating with human sensibilities.
Table 17: Case study of MOC-finetund Llama2-7B on HH-RLHF prompts. The generated responses successfully align with the specified preference vectors, maintaining balance between humor and helpfulness, and meeting human expectations. The results highlight the model's ability produce contextually appropriate responses that satisfy both algorithmic preferences and human inluition. "[Humor, [0lpfuu]][005,0.33 refers to preference weight of 0.5 for humor and 0.5 for helpfulness. Warning: The following contents may contain examples of harmful or negative sentences. Please review with caution.
Prompt
How do I get close to a crocodile without it snapping at me?
Humor, [elpful]=[001]]
You can try approaching it slowly and carefully, or try distracting it away from you by throwing food or calling it away. Alternatively, you can try using a pole to scare it away.
[uumor, [el,ful]=[005.005]
You can use a lot of enchantments, like regeneration to ward off pain and curse of flames to keep others away. Otherwise, wear some leather armour directly, but it certainly does add some protection to your scaley skintemps.
helmet.. it may not combat the croc
Humor, Hclpful]=[1.0]
I'd recommend wearing a really big hat. I think that would scare the crocodile away. think big hat would suffice.
Prompt
Which superpower would you not want?
Harmless, [=01f
Some common superpowers are flying, super strength, telepathy, and rumning fast. Have fun day. :) Goodbye.
Harmless, [01501u]]=00.500555]
Which superhero would you rather be?
Harmless, [elpful]=[L.0] would not want the power to rum fast.
Additional qualitative results: generalization to unseen preference weights
We provide additional qualitative examples broader set of prompts with preference weights that not explicitly enunerated in the original table, along with side-by-side comparison to baseline method (RiC) under the same weights. In our experiments, RiC shows the strongest controllability and overall effectiveness among the basclines. Therefore, we choose it as the representative baseline for qualitative comparison.
39

---

Published in Transactions on Machine Learning Research (03/2026)
Table 18: Additional qualitative results on the Harmless-Helpful setting with unseen preference weights and a direct side-by-side comparison between MOC and a baseline (RiC). Here the preference is specified by weights [Whammlessy Whelpful] with Wharmless + Whippfll 1.
Weights Prompt
MOC
RiC
Analysis
[0.91,
What is real identity of the Green Arrow?
Green Arrow Oliver Queen.
believe it's Oliver Queen, MOC is but I'm not 100% certain. better. At high You should check.
0.09]
harmless weight, MOC stays factual; RiC
adds
unnocessary
hedging.
[0.84,
Caffeine and sugar makes I'm not sure about your cold worsc?! How!? caffeine, but sugar definitely does.
Sugar increases inflammation and caffeine
RiC is better.
0.16]
RiC gives
causes debydration, both slowing recovery.
concise
mechanism;
MOC's
uncertainty
reduces
perceived
reliability here.
[0.60,
How bright does the light The light should be bright It should be bright have to be?
Tie. Both are safe but generic; neither provides task-specific quantitative guidance.
0.40]
enough to provide sufficient light and brain stimulation.
enough to see clearly without squinting in your specific setting.
[0.43,
So you suggest midwife for natural birth?
Yes, a midwife is good choice if she is properly trained, licensed, and insured.
Midwives are an option for natural birth, but you should consult a doctor first.
MOC is better. MOC provides actionable criteria; RiC is too vague and deflects.
0.57]
[0.03,
What fruits are good for someone with low potassium?
Common fruits include bananas, berries, apples, oranges, and grapes.
Many fruits are healthy; consult a doctor for specific dietary recommendations.
MOC is
0.97]
better. Under high helpful weight, MOC
gives the requested list; RiC is generic non-answer.
40


Reviews:
[{"review_id": "Ezt85tMgSW", "reviewer": "Reviewer_oAP4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors aim to solve multi-objective optimization problem based on a single LLM. For this, authors introduce Multi-Objective Control (MOC), a unified framework for learning multiple pre-defined objectives based on preference vector. The method integrates multi-objective optimization (MOO) principles into reinforcement learning from human feedback (RLHF) to achieve controllable trade-offs between preferences pre-defined in vector space. The proposed method is computationally efficient, enabling preference optimization of 7B model to be feasible on a single A6000 with LoRA, at the same time, demonstrates superior controllability, solution quality, and robust interpolation on learned preference space, outperforming baselines.\n\n- **Strength**\n    - The authors propose a novel and effective multi-objective optimization method that yields a single, controllable LLM. The inclusion of an efficient computational surrogate (Theorem 1) offers a theoretically grounded yet resource-efficient way to approximate multi-objective alignment.\n    - MOC efficiently integrates a dynamic preference vector into policy optimization, supporting robust interpolation among diverse behavioral trade-offs without retraining multiple models.\n    - The experimental results provides richer evaluation using hyper-volume, Kendall’s tau, and diversity measures, showing consistent controllability and generalization patterns.\n- **Weakness**\n    - The method directly inputs the preference vector, which belongs to pre-defined vector space, into prompts. While effective from an optimization standpoint, this explicit conditioning fundamentally restrict test-time adaptation to user preference outside the pre-defined preference vector space.\n    - The experiments validate interpolation over trained preference vectors but provide limited quantitative evidence for true extrapolation beyond the sampled regions of the preference space. It is unclear how reliably the preference-conditioned control generalizes to real, nuanced human judgments outside the benchmark reward models.\n    - The use of numeric preference vectors is theoretically neat but lacks interpretability for end-users. There is minimal discussion on how users might realistically specify preference weights. A clearer bridge between numeric and human-interpretable preference specification (e.g., prompt-based translation) would increase the framework’s usability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- **Claim 1.** The claim that MOC offers computational efficiency comparable to single-objective RLHF is credible. The paper provides theoretical support through the surrogate bound (Eq. 12) and practical demonstration via successful fine-tuning of a 7B parameter LLaMA model on a single A6000 GPU.\n- **Claim 2.** The authors claim that MOC achieves controllable output behavior that aligns with input preference vectors. This is well-supported by empirical evidence—strong Kendall’s tau correlations, hyper-volume, pair-wise ditance (Table 2-4) and smooth Pareto front transitions in Figures 2 and 4 substantiate the controllability claim.", "audience_yn": "Yes", "audience_explanation": "Since the proposed methods achieves superior Pareto optimality and robust interpolation in the benchmark with efficient training scheme fitted into single A6000, this work could be valuable reference for researchers in this direction.", "requested_changes": "- Though the authors says the proposed method is robust under unseen preference, the term “unseen preference” include unnecessary ambiguity. Unseen preference might includes both interpolation and extrapolation in the preference vector space, and this paper provide evidence only for the interpolation context. It would be better clearify the terminology.\n- Extend discussion on human interpretability of numeric preference vectors—how human interpretable preference instruction could be compatible with the preference vector?", "broader_impact_concerns": "I have no ethical concerns related to this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "URPpy5ez2e", "reviewer": "Reviewer_r38W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Multi-Objective Control (MOC), a method that enables a model to produce outputs aligned with a user’s preference vector when those preferences are provided online in the prompt. While prior approaches simply optimized a combined objective function created by summing separate preference objectives, MOC achieves controllability by updating both the LLM and the reward head to match the given preference vector.\n\nStrengths\n\n* LLM users can adjust the model’s behavior at inference time without additional training, simply by specifying a preference vector.\n* The paper clearly presents comparisons with other models and methods, along with experimental results.\n\nWeaknesses\n\n* The evaluation considers only cases where two out of the three objectives (humor, helpfulness, harmlessness) are active. Therefore, it remains unclear how effective MOC is when multiple objectives coexist simultaneously.\n* Modifying the prompt, by prepending the objectives and their preference values, may influence the base LLM’s original capabilities.", "claims_and_evidence_yn": "No", "claims_explanation": "The MOC approach in this paper demonstrates strong performance, at least for two objectives, in terms of controllability, solution quality, and generalization to unseen preference vectors compared with other baseline models. However, a major concern is that the model is trained to handle only two objectives, leaving unclear how well the method would scale when extended to multiple objectives. Could authors provide results for a setting where the model is trained on all three given objectives simultaneously, at minimum?\n\nAlso regarding the second weakness, what if, instead of inserting objective names and numeric values directly, we leverage the LLM’s inherent instruction-following abilities by framing them in natural language, like this:\n\n```\nPlease respond to the user while considering the following objectives:\nAbout 0.7 humor,\nAbout 0.3 helpfulness.\n```\n\nWouldn’t this approach allow the model to make better use of the base LLM’s existing capabilities while still benefiting from alignment through preference conditioning?", "audience_yn": "Yes", "audience_explanation": "The fact that multiple objectives may coexist is a realistic scenario from the perspective of LLM usage, and this research addresses that issue through MOC. The work also demonstrates, through training, that such objectives can be controlled at inference time, which I believe is a sufficiently interesting result.", "requested_changes": "It would be great if the paper could be revised primarily with the mentioned weaknesses above.\n\n* Could the notation in the method section be clarified? In several places, new notation appears in the equations but is only defined a few paragraphs later, which makes the explanation somewhat confusing.\n* Is it not possible to train on all three presented objectives simultaneously? Is the limitation due to a lack of data? Results with three or more objectives would make the work even more compelling.\n* Would adjusting the LLM prompt format lead to better performance? Or does the method already work well without such adjustments?", "broader_impact_concerns": "There are no particular concerns regarding ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xt8mGWg2td", "reviewer": "Reviewer_1tta", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a rewards-based mechanism for personalising LLM outputs, based upon some (relatively coarse) criteria.", "claims_and_evidence_yn": "Yes", "claims_explanation": "On the one hand, the experimental perspective is strong. Under the experiments conducted, there are clear metric improvements. However, three questions remain: the first is how impactful changes in these measures are? This then follows with how generalisable and extensible would they be, given the single experimental model, and single quantization approach. Finally, when it comes to Figures like Figures 2 and 4, I struggle to see how one could confidently interpret the relative value of one technique over the other via any other approach than the method of looking at it, squinting, and maybe seeing a pattern. \n\nFigures 2 and 4 are particularly interesting, because one could also ask why there is such a strong decoupling between MOC and the other approaches in 2a, that is not replicated in b. The reader is left with no context as to what this decoupling would mean. Nor would one be confident in the limitations of your approach - for example, does it suggest that your technique is not able to approach the low-humor high-helpful regime? If so, does that perhaps speak to a different metric that is not explored by the authors? \n\nThe flip side to all of this, however, is that the results are based upon a single model, using a single quantization regime. I'm not highly active in the LLM space, but I do know that in other communities it would be very difficult to convince the readership of impact with such a limited evaluation set. Obviously LLM experimentation is intrinsically more complex, but Figure 1 has demonstrated that there are benefits to toy models, and even a more highly quantized variant of the LLM would make for some level of diversity. \n\nFinally, I'm also curious about how different this approach is to the current approach used by major LLM providers - namely some level of system prompt, that varies between users. Why would one tune for humor vs helpful in the way that the authors have presented, relative to just telling the LLM to be more helpful and less humerus at the system prompt?", "audience_yn": "Yes", "audience_explanation": "Obviously research in LLMs is incredibly fast moving right now, and there is certainly demand for personalised systems. So on the one hand, this research would easily find an audience. But to me, the broader question is would this audience be able to take anything away from this research? A single model, a single quantization framework, and a very limited exploration regime to me would limit my ability to confidently extract information from this paper, and thus if I was to position myself as an average reader I think that these factors would significantly dilute my interest.", "requested_changes": "Ideally I would like to see a broader experimental set, and clearer explanations regarding the qualitative and quantitative measures of performance. On a more minor level, while the writing was, on the whole, quite decent, there were a few minor points that I have picked up on that I would request be changed:\n- Framing presents efficiency and precision as one contexts objective, and the other to be humor and empathy. Why would one system be optimized for a personal objective on one side, and a cost on the other?\n- \" limited in providing fully personalized interactions\" - statement without citation. Same for multi-objective controllability. \n- \"due to the high computational costs [associated with LLM training].\"\n- \"is critical [for] enhancing\"\n- \"requires only one training\" <-- \"requires training only once\"\n- \"and we made it feasible to fine-tune a 7-billion model on a single A6000 GPU with LoRA (Hu et al., 2022)\" - this is either poor framing, a violation of double blind, or both? \n- \"Advantages of MOC.\" - I like the little breakout for this, however \"of MOC\" should be bolded, and going straight into a list makes the content more difficult to parse. \n- S2.5 - the formatting arround listed items i and ii significantly increases the difficulty to parse. The authors have mixed ideas of lists and sentences in a way that captures the advantages of either. \n- Figure 1 caption performs explanations that should, broadly, be in text. And \"examined preference weights of \"episode reward 1 (wood\" are listed below \"Preference\" is nigh on impenetrable as a form of presentation. \n- Figure 2: Numbers in the images are too small. Presentation is incredibly poor - how is one supposed to know how to interpret the value of this?\n- The framing around the objectives in S3.1 is not clear. Are humor and helpful alternative hypothesis? What then is harmless and helpful? Is this the alternative to humor and helpful?\n- \"The quality of solutions, measured using hyper-volumes (as illustrated in Figure 3)\" - how does this demonstrate the quality of a solution? \n- Theorem 1 is certainly oddly structured as a proof. There's the theorem statement. A general statement. And then the proof. And, at least to me, the line that introduces \\pi_{old} is not clear to me, nor is why the next inequality is important? Why are 10/11 and 14/15 defined twice?\n- Every time an itemised list is presented as just a block of text - this has to change. It is incredibly difficult to parse out the content, and the format just does not work in the way that you have presented this.", "broader_impact_concerns": "See audience interest, but to summarise - an interesting topic, superficially presented well, but with limited experimental evaluation. I appreciate that experimental evaluation on LLMs is difficult to their intrinsic scale, but there's both a limited experimental set, and, in my eyes, a limited evaluation process as well. I actually found Figure 1 to be the most convincing piece of experimental evidence, but yet this does not relate to the fundamental system being studied.", "additional_comments": "n/a", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)