Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF
Authors: Nuoya Xiong
Aarti Singh
Venue: ICML 2025 poster (2025)
Keywords: multi-objective learning
RLHF theory
Primary area: theory->learning_theory
TL;DR: We provide a reward-free algorithmic framework for multi-objective and multi-group RLHF with general non-linear aggregation.

Abstract: Reinforcement Learning with Human Feedback (RLHF) is a widely used fine-tuning approach that aligns machine learning models, particularly Language Models (LMs) with human preferences. There are typically multiple objectives driving the preference, hence humans find it easier to express per-objective comparisons rather than a global preference between two choices, e.g. compare two papers on their novelty, clarity, correctness, etc. Multi-Objective RLHF aims to use per-objective preference feedback and achieve a Pareto optimal tradeoff among these objectives by aggregating them into a single unified objective for optimization. However, nearly all prior works rely on linear aggregation, which rules out policies that favor specific objectives such as the worst one. The only existing approach using non-linear aggregation  is computationally expensive due to its reward-based nature and the need for retraining whenever the aggregation parameters change. In this work, we address this limitation by transforming the non-linear aggregation maximization problem into a series of sub-problems. Each sub-problem involves only linear aggregation, making it computationally efficient to solve. We further extend our framework to handle multi-group scenarios, where each group has distinct weights for the objectives. Our method enables achieving consensus or maximizing the aggregated objective across all groups. Theoretically, we demonstrate that our algorithmic framework achieves sublinear regret and can be easily adapted to a reward-free algorithm. Empirically, leveraging our theoretical insights, we propose a nearly training-free algorithm once the optimal policies for individual objectives are obtained.

Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF
Nuoya Xiong Aarti Singh"
Abstract
1. Introduction
In recent years, there has been considerable effort to fine- tune machine learning model, particularly Large Language Model (LLM), to perform better on particular tasks. RLHF is a popular fine-tuning approach, which receives the hu- man's 's preference feedback and aligns the LLM model with human values using mne-tuning. Standard RLHF exploits hu- man preference feedback between two outputs to maximize the expectation of the implicit or explicit reward function.
Reinforcement Learning with Human Feedback (RLHF) a widely used fine-tuning approach that aligns machine learning model, particularly Language Model (LM) with human preferences. There are typically multiple objectives driving the preference, hence humans find it easier to express per-objective comparisons rather than global preference between two choices. Multi-Objective RLHF aims to use per-objective preference feed- back and achieve Pareto optimality among these objectives by aggregating them into a single uni- fied objective for optimization. However, nearly all prior works rely on linear aggregation, which rules out policies that favor specific objectives such as the worst one. The only existing approach using non-linear aggregation computationally expensive due to its reward-basce nature and the need for retraining whenever the aggregation pa- rameters change. In this work, we address this limitation by transforming the non-linear aggre- gation maximization problem into series of sub- problems. Each sub-problem involves only linear aggregation, making it computationally efficient to solve. We further extend our framework to han- dle multi-group scenarios, where each group has distinct weights for the objectives. Our method enables achieving consensus maximizing the aggregated objective across all groups. Theoreti- cally, we demonstrate that our algorithmic frame- work achieves sublinear regret and can be easily adapted to a reward-free algorithm. Empirically, leveraging our theoretical insights, we propose a nearly training-fre free algorithm once the optimal policies for individual objectives are obtained.
However, there are two main challenges for the application of RLHF in the real world. First, standard RLHF only maxi- mizes a single reward function. However, people often find it hard to evaluate choices in an overall sense as, in reality, there are often multiple objectives. For example, comparing two papers or essays overall harder than comparing them specific objectives such as novelty, clarity, correctness etc. Similarly, recommending city for vacation is harder than comparing cities on food options, nightlife, safety, etc. Each objective has its own implicit or explicit reward function, and the LLM needs to achieve a Pareto optimal trade-off' between them by, for example, maximizing an aggregation of these reward function. Second, there are multiple groups of users in the real world who may prefer different aggrega- tions of the objectives. For example, groups with different genders, political views, marital status, etc. This requires that the LLM either (a) satisfies the requirements of all the groups simultaneously, or (b) optimizes some aggregation across multiple groups.
Multi Multi-Ohjective Problem There are some works (Rame et al., 2024; Yang et al., 2024; Shi et al., 2024) that con- sider balancing the utilities of multiple objectives to get the Pareto optimal point or maximize the average expecta- tion. Some works (Zhong et al., 2024; Park et al., 2024) consider multi-party problem which each reward rep- resents a group, which can also be regarded as a multi- objective problem. We assume that we have m different objectives, and each objective has its own reward function ra v) < i < m.). Each reward corresponds to an objective of the response like safety or helpfulness of the LLM, and the expected rewards (x= [r, y)] evaluate the LLM on these objectives respectively, where p is the distribution of the prompt. Nearly all of the
'Camegic Mellon University, PA, USA. Correspondence to: Aarti Singh <aarti @ es. emu.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Projection Optimization: A General Framework for Multi-Ohjecttve and Multi-Group RLHF
previous work consider only linear aggregation, i.e., op- timizing u(*²) Σ (..Baspp.srattttt (n; w), where {or [e(m) is the weight of all objectives that is as- sumed to be known.
Moreover, previous work typically assumes that the weight for each objective is known. This assumption simplifies the problem and allows for straightforward optimization. How- ever, in real-world applications, the importance weights {as} for each objective are usually unknown. In our work, we observe that the weight of an objective reflects its impor- tance, which can be learned by how frequently the objective is reported in the human preferences. We propose a learning paradigm where the LLM learns objective weights from collected data, enabling the estimation of {ax) and incorpo- rating them into our theoretical results.
However, this kind of aggregation may not lead to an LLM that treats all objectives fairly. For example, the LLM may favor one objective significantly at the expense of another. In social choice theory, certain natural axioms such as mono- tonicity, symmetry, scale invariance, etc. which apply to multi-objective aggregation as well, lead to more general function class (Cousins, 2021)
Multi-Group Problem Classical RLHF often assumes a uinglegroup setting, ignoring the heterogeneity in human feedback and assuming that the human feedback relies on one unique reward function. However, real-world scenarios involve multiple groups with distinct preferences. Fine- tuning an LLM for each group is computationally expensive, making it essential to fine-tune the LLM to accommodate all groups' preferences simultaneously.
1/8
m
u(=)= =
ai E p. Σ
P 1
(I)
The general p-norm aggregation with p S promotes fair- ness across multiple objectives, which is particularly useful when aiming for a machine model that achieves a balanced performance among different objectives. Only one paper (Zhong et al., 2024) addresses the p-norm aggregation set- ting. In that work, the authors first learn a reward function for each objective, aggregate them into a new reward, and then attempt to optimize this new reward directly. How- ever, this reward-based approach is computationally ineffi- cient compared to the reward-free, DPO- -PO-based algorithm (Rafailov et al., 2024). Moreover, it requires retraining the entire policy whenever the aggregation method changes, which becomes even more time-consuming.
Since previous papers (Zhong et al., 2024; Park et al., 2024) working on multi-group RLHF only consider learning the reward funetion of each group under single objective and then aggregating them, we regard them as special case of the MORLHF. Hence, there is a lack of discussion about the multi-group setting where each group may have different importance for different objectives.
Formally, assume that we have N group and m objectives, and each group n [N] has their own weight (x))) E Am--1 The utility of the group for LLM is then defined by
To reduce the computational cost of the reward-based based RLHF algorithm, the paper (Shi al., 2024) shows that for p = 1, once the optimal policy Tr, for each individual objective is obtained, the optimal policy TT, for the linear averaged sum can be calculated as T, Tv zz) x II Tr, (y x)): How- ever, the derivation heavily depends on the linear structure of the aggregated reward r(x, 3/) When * 1, this ap- proach breaks and the optimal policy cannot be written as a simple elosed-form of the optimal policies of each objective. That raises the first question:
1/p(n)
(T) = Σ (m) Beeepsrtt w))p(n)
where p(n) < 1 is the parameter of the aggregation for group n. The reward function of each objective, {ri y) he]m)) remains fixed across different groups, while the weight a and the parameter p can vary. In other words, the reward of each objective is the inherent value, and the importance weight represents the subjective part of each group. Now we pose the last question:
Question 1: Can we derive a computationally efficient MORLHF algorithm with non-linear aggregation?
Question 2: Can we formulate and tackle the multi-group problem under MORLHF setting?
In our work, we propose projection- based algorithm both in offline and online preference data settings, which trans- forms the nonlinear objective maximization problem into a sequence of subproblems, each involving only a linear maximization problem. Theoretically, we provide thor- ough analysis for both offline and online setting, showing that it can converge to the optimal policy with sublinear regret. Empirically, by leveraging the fact that there is a training-fre free algorithm for linear aggregation maximization, we derive a training-free free algorithm for the generalized re- ward aggregation, which saves significant training time.
In this paper, we consider two final goals for multi-group group problem. Motivated by the poll theory, the first objective is called "consensus", in which LLM needs meet the re- quirements of all groups as good as possible simultancously. Motivated by social choice theory, the second objective is called "aggregation", in which the LLM needs to optimize general aggregation of the utilities of all groups. We will show that our formulation and algorithmic framework natu- rally solve these two final goals. In summary, we have the following contributions:
2

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
We reformulate the reward maximization in MORLHF
Table I. Comparison of previous work for MORLHE. The parame- ter means the exponent in Eq.(I), Algorithm 3 (offline setting) & (online setting) have theoretical guarantees, while Algorithm 5 is the more practical version.
as minimizing the distance between the current reward vector and target set. This reframing decomposes the aggregated reward maximization into sub uub-problems, each focusing on minimizing the distance in a specific direction. These sub-probi problems reduce to linear aggre- gation and can be efficiently solved using previous approaches. Theoretically, we provide converge guar- antees for both offline and online setting. Empirically, we provide training-free algorithm once the optimal policy and the reward function for each objective is given, making it more computationally efficient.
Reward Free
Traning Free
Multi-
Aggergation
Group
MORLHF (Wu 2003)
X
x
x
RS (Rame atal. 2024)
p = 1
/
/
x
MOD
p= 1
/
/
x
(Sti et al. 2024)
PNB (2hngg al. 2024)
S 1
x
x
x
Algorithm &
P.S 1
/
V
We tackle the multi-group problem in two ways: (1) achieving consensus by defining the target set the in- tersection of all groups' target sets, and (2) minimizing the malfare function (Cousins, 2021) which aggregates the distance between each group's expected reward vector and its target set. Our framework addresses both problems concisely with theoretical guarantees.
Algorithm 5
P.1
X
V
pose theoretical framework that can be easily adapted to a reward- weward-free algorithm, along with a training- frree empirical algorithm built on the same theoretical framework. Detailed comparisons are shown in Table 1.
We establish a learning paradigm where the LMs learn from data. We integrate weight estimation into the online setting and provide theoreti- cal guarantees.
the importance weight
Pluralistic Alignment and Preference Aggregation There is growing body of work on aligning machine learn- ing models with diverse preferences, accounting for differ- ent values and perspectives. The works (Chakraborty et al., 2024; Ramesh et al., 2024) focus on optimizing the worst- case group loss, ensuring that the model achieves reasonable performance across all groups (Park et al., 2024; Sorensen et al., 2024; Conitzer et al., 2024) explore how to aggregate preferences using social choice and voting theory, outlining a high-level roadmap for pluralistic AI alignment. (Ge et al., 2024) technically demonstrate that the BTL model fails to satisfy well-known standards in social choice theory and propose novel rule-based approach for learning reward functions. (Chen et al., 2024) further study the generaliza- tion of the BTL model and introduce an ideal point model that better accommodates diverse groups
2. Related Works
RLHF Fine-tuning LLMs with human feedback and RL is known as RLHF. The reward- based RLHF first extracts a reward model with a Bradley-Terry (BT) assumption on human preferences, and then optimizes the reward model (Ouyang al., 2022; Bai et al., 2022; Touvron al., 2023; Azar et al., 2024). On the other hand, the reward-free RLHF avoids explicit reward modeling by directly formulating the preference loss as a function of the policy and then using supervised learning ( (Wang et al., 2023; Rafailov et al., 2024), which is more stable and computation-fiit ffrinndly.
MORLHF Multi-Objective RLHF (MORLHF) aims to
3. Preliminaries and Notations
align an LLM with human preferences while optimizing for multiple objectives, such harmlessness, helpfulness, and humor. Most previous works aggregate rewards or models as the weighted sum of individual components. MORLHF (Wu et al., 2023; Bai et al., 2022) directly optimizes the aggre- gated reward using PPO, while MODPO (Zhou et al., 2023) provides a lightweight reward-free alternative, RS (Rame et al., 2024) combines individual models by averaging them. MOD (Shi et al., 2024) calculates the closed-form solution of the optimal policy for aggregated reward directly and derives maining-free algorithm. Only one work (Zhong et al., 2024) consider non-linear aggregation, and they op- timize the aggregated reward function directly. However, this approach is computationally expensive and requires retraining when the aggregation changes. Instead, we pro-
Denote the prompt space of the L.M as X' and the response space as V The distribution P E ((X) represent the distri- bution of the prompt. A policy π X' + A()) represents an LLM that generates a response distribution given prompt x. In RLHF, we assume that we can get a pre-trained LLM Trof that is usually trained on supervised data. The goal is to fine-tune the pre-trained model to align the model with the human preference on one particular task. To be more spe- cific, given prompt
p. LLM can generate two responses y1, V/ then the human gives a preference feedback on the response pairs as either y1 < y/2 or > y2. The responses V1, V/2 are labeled as yoo Vt respectively with probability P(y1 > y/ z), and are labeled as yts yu with probability P(y > y/2 2:). It is further assumed that the human
3

---

Projection Optimization: A General Framework for Multi-Ohjecttve and Multi-Group RLHF
preference is modeled by a Bradley-Terry (BT) model with the reward function r* (3r, y) X x + B]:
The goal is to find the optimal policy for the aggregated utility function. One natural approach solving multi- objective RLHF is to first learn reward model for each individual objective, and then aggregate these models to formulate a new reward. Finally, RI. methods like PPO can be applied to optimize this new reward. However, this reward-based approach is significantly more computation- ally inefficient and unstable compared to reward-fire ap- proaches, such DPO (Rafailov al., 2024). Additionally, it requires retraining the entire model for all possible reward aggregations, which becomes time-consuming when the aggregation parameters change. In this work, we first pro- vide a theoretical algorithmic framework for multi-objective RLHE, which naturally leads to the derivation of reward- free algorithm. Based on this theoretical framework, we propose a nearly training-free practical algorithm that incurs almost zero computational cost once the optimal policy for each objective is obtained.
P(yl > 3/ x) = orr (x, v1) - r" (z, (", v2)).
where (=) 1+exp(-33 and B 2 1. Given the reward function r, the optimal policy 7, maximizes the expected reward function, with an additional KI. divergence term that prevents the policy from deviating too much from Tref-
7. = arg max (T)
:= Ee-olyyrntt I (x, v) - (T Il Trer)] (2)
In this paper, we consider both offline and online RLHF. For the offline RLHF setting, the LLM has access to a pre-collected offline data DD consisting of prompts and corre- sponding winning and losing responses, and the expectation in the optimal policy is calculated on the offline data. For the online setting, at each round LLM can generate two responses y 1/2 following the policy म, and then receive the preference feedback by human for data collection.
Previous techniques cannot be easily applied to this set- ting. In fact, for the linear aggregation when p = 1, the paper (Shi et al., 2024) finds that the optimal policy म, can be written as a closed-form of the optimal policy Tr, as (r x:) x (e xr) .exp (2r(zz-)) and conduct de- coding algorithm MOD using this derivation. By the linear aggregation r(x. y) = Σm a3r;(2, y) and 1= it is easy to verify that (y x)) x II Tr. Hence, one natural reward-frre free algorithm is to first learn the optimal policy ny, for each objective using DPO, then calcu- late the optimal policy x,. It is also training- free algorithm once the optimal policy for each objective is known. How- ever, when we choose the general aggregation with S. 1, this derivation will fail due to the non-linear structure of the reward, making the problem much more complicated.
We assume there are m known representations {ø, v) E Rd [e[m) and the corresponding reward function class (ri y) = 0J (1 y) E [0, B). 1lo:ll2 S 1, ||0 S B} for each objective E [rr]]. The true reward rf for ob- jective can be written as (x, v) = (T) This assumption is purely theoretical. In practice, the reward can be parameterized as rd using a neural network, and our practical algorithm 5 also does not rely on this assumption.
0;
2:)0.
Since the preference only contains the information of rI(2, - (z, 32) for each objective i, rewards are in- variant to constant shifts in feedback. Follow (Cen et al., 2024), we can assume there is a known policy Thase and constant C, such that for each i E [m]] the reward parameter space O is defined as
To avoid this technical difficulty, we draw inspiration from RL with Blackwell apprracabpp (Yu et al., 2021), which focuses minimizing the distance between the reward vector and a specified target set. This approach makes the problem more tractable since we can incorporate the non- linear aggregation into the definition of the target set. To be more specific, a target set W C I is a convex set that is defined by
θ = (0 E Rdd Emn (0i, (1 ()) = =C.
(3)
3.1. Multi-Objective Learning
We assume that there are different objectives, and each objective has reward function r;(x, y) E [0, B) for i E [m]. As discussed in the introduction, we apply the definition of social welfare function in social choice theory to multi- objective setting and consider the weighted p-norm aggre- gation across objectives
1/p
Wa
2 E 20
Σ
2 C
where a represents the weights assigned the objectives by humans, p represents the degree of fairness, and reflects the requirement of humans. In practice, we can learn a and p from supervised and preference data, and the parameter e can be provided by humans or chosen by parameter tuning, The definition of target set implies that the group can be satisfied the aggregation of the reward function is larger than some pre-defined constant, We also define the expected
1/P u(x)= = Σ a E Ey w)Pp)
m
. 1
where a E Am-1 are weights of the objectives. Note that for positive rewards, aggregation yields Pareto optimality.
4

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
reward vector S(T)) E R" as (S(=)), (x,
3.2. Multi-Group Learning
(Dx (x][aat) which is the expected reward following the policy with regularized term of KL divergence. Now assume P, are all given, we can transfer the aggregation maximization problem to minimizing the distance between the expected reward vector (with some regularizer) and the target set W. The goal changes to minimizing the distance between (T) and p.e-
Beyond the single group setting, we also study the multi- group setting, where each group has different aggregation approach (parameterized by c,p and a). For each group we assume there is target set
win)
W(n)
R 20
(n)) a
c(m)
arg min D(T)= = d(s())) p.e))
(4)
representing the aggregation rule across objectives for them. We consider two types of goals that represent the effective- ness of alignment across diverse groups.
Note that we choose e as the maximum value that there exists policy x that satisfies (()()), Wp.e)) = O, then π is one of the optimal policies and
1/p
The first goal is called "consensus", in which we wants to minimize the distance between the expected reward vector and the intersection of all target sets from diverse groups. Formally, the goal is to choose the optimal policy that minimizes the Euclidean distance
Consensus
0g Erlir))))))) (I (Tll
arg max WEll
where every т E II satisfies that [x (Tll Tref) 2 0. This statement highlights the connection between the original maximization problem Eq. (2) and our formu- lation Eq. (4). Therefore, our formulation can be viewed as an alternative metric for measuring the performance of LLMs achieving multi-objective learning tasks.
π = arg min d ( ( nwwn)
N
((T):
(5)
==1
Malfare Function Minimization imize the aggregated malfare function, where the malfare function for each group is the square of the distance be- tween the expected reward vector and the group's target set. Formally, with group weight G. > 0 and Σx=1 Sa 1, the goal is find the optimal policy x² that
Another goal is to min-
Now demonstrate that more general aggregation meth- ods can enable L.M to accommodate a wider range of objectives by selecting different values p.
Example 3.1 (p-
Linear Aggregation). Now if we choose p= 1, and choose e 2 max, Ei a.EE, [r? (z, v)],
then the goal (TT) will become
1/4
(mx) = d(S(T), WT.))
Σ Sa (((S()),w(>)))))
)
π arg mig
.9 1922.
c c- En (r, y)l (T Tret)
VEET of
4. Algorithms for Multiple Objectives with Linear Aggregation
The last equality is because the selection of c. From this derivation, we know that it is equivalent to the previous classical MORLHF with linear aggregation.
In this section, we consider the simplest setting where the reward function is a linear aggregation, i.e. r(x, y) E drr (z, y), where E RT is called the direction. In fact, the linear aggregation can be viewed as projecting the reward vector onto specific direction d. As we will show later, this will become essential sub- sub-problem in final algorithm for non-linear aggregation.
Example 3.2 (p = -o worst-case reward). When
-00, the target set becomes
Th = {2 E 18% min 2, 2 c
which represents that the human wants to find an LLM with no obvious drawback for any of the objectives, i.e., requiring min, Er (Ir y)] 1 IKk.. (T [(ett) larger than some thresh- old. Now we establish the connection between p- = -00 and the max-min RLHF in (Chakraborty et al., 2024). The proof is provided in Appendix B.I.
Given the dataset D; = {x² (yl wae ]jejar) containing M data points for objective i, we provide offline and online al- gorithms to learn the optimal policy with respect to multiple objectives in consistent way. Now we aim to minimize the negative log- likelihooood loss of preference data
Theorem 3.3. Define the max-min value as c¹ max, [min;; Ex [r3] 30K,, (T [ve]] Then, if we choose the target set Wa - such that e is close to c², the resulting optimal policy also achieves max-min value that close to c.. To be more specific, we have
(.,) (2)
Σ
0 log(o (19(o (x. vw) r (x. v)))
for each objective i. Following (Cen et al., 2024),
min R. (+. v) DKL (x) 2 c c' c2'-(mm (vm 1)]c" ccl.
can refine our estimation of the reward by adding an
5

---

Projection Optimization: A General Framework for Multi-Ohjecttve and Multi-Group RLHF
additional exploration term max; _ d, max, E Ex= (r kK (TT [ret))], which represents the optimism/pessimiss principle of the online/offline learning process. To be more specific, for the offline and online setting, LLM learns the Boffine and onllin respectively by
where ((D;, = E (a.N...))D log a log Frot (Bmllz)
me, (2 (a)
@ log Ke, [x(2)) Kyaf (xc/a)) is the reward-free loss function, and the expectation E means (exp-
m
Oofflie arg MaX
max dd,)
Σ
(0.)
(6)
Algorithm 2 -eo--deward Free (RF) (Online Version)
1: Input: Direction d, dataset {D, (isjigjm)) (m] 7), S.
Calculate Oontine E Rm by Eq. (8) and 7
folline arg max 6 pm
max [r d. Σ (0:)
(7)
3: Given the prompt z, Generate two responses y1. V22 and get preference y = (ywr V1).
where use single parameter 0 to refer the set {6, hee[m] The difference lies in the optimism and the pessimism principle. In the offline setting, we subtract the explo- ration term to avoid over-optimization (Cen et al., 2024; Liu et al., 2024) while in the online setting, we add the exploration term to encourage the model to explore (Cen et al., 2024). Then, the LLM executes the greedy policy T" arg max, J(r", d, π) generate the response and re- ceives the human feedback (yw, W1). We called the algorithm Multi-Objectio Projection (MOP), and the pseudocode for online setting is shown in Algorithm 1. (There is no Line 4 and the output only has 9 for the offline setting.)
4: Output: Data point D = {ar, (Bws 31)} and O.
The Eq. (8) involves an optimization problem on 8, which is complicated joint optimization since 0 refers to pa- rameter 0j, 60. In Appendix E, we further study the computational cost of Eq. (8), showing that the gradient descent update rule can be easily computed once the expec- tation of the score function is available, which commonly appears in previous RL algorithms such as REINFORCE (Williams, 1992).
Algorithm 1 Reward Based (RB)
5. General Algorithm for Preference Aggregation
1: Input: Direction d, dataset {Dr [iejee 17, B.
2: Calculate foffine by Eq. (6) or Oonline by Eq. (7).
3: Execute 7² arg max, J(1,, r ... , rm d, T).
In this section, we introduce general offline and online algo- rithms that work for both linear and non-linear preference aggregation, and provide their theoretical guarantees. Both algorithms transform the non-linear aggregation into a series of linear aggregation sub-problem. using Algorithm I and 2 as their core aub-procedures.
4: Given the prompt z, Generate two responses yl, 3/2 n, and get preference y (yuu, 31).
5: Output: Data point D = {I, (Yw. v1) and A.
The computational cost of Algorithm 1 mainly lies on Line 2. In fact, it needs to learn multiple reward functions directly, and then get the estimation of the optimal policy, which requires a joint optimization subprocedure. In the following, we consider the reward-free version of Algorithm 1.
5.1. Offline Algorithm
Now we introduce our algorithm Multi Objective Projection Optimization (MOPO), which follows from the competi- tive RL with Blackwel-approcchabitt literature (Yu et al., 2021). We receive the offline data set D = (Di)ie(m)) which contains M data points D, for each objective i. The algorithm learns the reward or optimizes the policy directly from the offline data. Our algorithm contains T iterations. In each iteration t, we first project the reward vector on the direction d' E R. defined in the last itera- tion, i.e. r(z, y) = Em d'r:(2, y), and then using the sub-procedure in the previous section find the estimated parameter 0f and determine the corresponding policy mí. Finally. we derive the estimated expected reward vector V' E RT as (V'), = E,, (r?" (2, v) - DKLL ( (T']|[_er) and calculate the averaged reward vector as V E/ Vi. Finally, the direction is updated based the projection of the estimated point V onto the target set, guided by either the consensus problem or the malfare function minimization problem. The pseudocode is in Algorithm 3.
Reward- Modification We now show that Algorithm 1 can be easily adapted to reward-fres version. We mainly consider the online setting since the offline setting is similar. Denote T² = arg max, [r d. x). By the same derivation in (Cen et al., 2024), we can get
J(1 d) = C HT F.
log
π
T
(y
xx)
Tref xx)
where C and Thase are the constant and the baseline policy in Eq. (3), Tg. is the policy for objective i and x x Feet (y/ r)) Ih (Te, (y d)) is the optimal policy for linear ag- gregation. The detailed derivation above will be provided in Appendix E. By the derivation in (Rafailov al., 2024),
you can further get the reward-free version of Eq. (7) as
arg min _E hane log log (v x) - Σ ((D...)) 0.)
(8)
6

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Algorithm 3 MOPO-Offiine
Theorem 5.1 (Consensus Problem). Let 1j = 1/vM M and ( (1))))))))))))) (6(r, yve) Ó(x, u1) T be the empirical covariance matrix of the data for objective i. We consider the consensus problem that W = W(n) and calculate the direction using Eq. (9). Define ((T)) d(S(=), - W(r)). For 6 E (0, 1), with probability at least 1 8, we have
INE[N]. 1: Initial: Dataset D (D, (Em)) (W(") [W[N) n,,..
ED. M Liz.y...DDD
2: 1, 2........................ . do
3: Collect 0f by MOP-RB d', D) or (d D).
Get the corresponding policy n' =T"
4: Calculate the point V = B (xr, v)
[DKL [Fee[[Tett]] = -C Blyyysaa [log x (vlm)
+ yy (log " (2lza)) and V = - V
D(#*) D(=²)
(1yllz) 5: Calculate the direction t by Eq. (9) or Eq. (10),
m 3/2 Va
1 B,, (min Amin (a.,) + M ))
S δ
poly
and calculate d'+
d
VM
ld+ttll1"
6: end for
Bym
1. ΣT 7: Return TT = T Ei=IT.
VT
The above theorem shows that the final gap of returned policy depends on the coverage term min; Amin (Ex,) of the offline dataset and the number of iterations T. As T increases, we achieve a standard convergence rate of (L//MM,, which is standard in prior offline RL algorithms (Jin al., 2021; Liu al., 2020). We also provide the theo- retical guarantee for malfare function minimization.
The key component of our algorithm is the direction cal- culation in each iteration. Intuitively, the algorithm aims to optimize the reward to guide the expected reward vec- tor toward the target set as effectively as possible. Sup- pose the target set is W, the direction can be caleulated by
d++1 = Pr, V = Hw (V)VV
(V) For the consensus
problem, we can substitute into W = =n W()) and get
Theorem 5.2 (Malfare). With the same definitions and con- ditions in Theorem 5.1, we consider the malfare function minimization problem with an integer' exponential param- eter q E N+ and use Eq. (10) for the direction. Define (T( = 2V 1 (S(T)) W())). For ō E (0.1). with probability at least 1 - 5 we have
N O W AM
= Proj
(9)
For the malfare function minimization problem, we can first calculate the projection to each target set W(n) and then aggregate them as
Dy(57) DD("²)
SallW( - 1-120-- 212011
Proj (wrl)) F')
Nm332
p0lol poly eB' min Amin (ED,) \Ein)ED
Σ D Cn [W(n))
1 1'122")
2q
VM
(10)
( (min Sn) -1/20 (2))-1/0)) +0 B BB/mT-1/20) NE[N]
Note that if we apply MOPO with p 1, it reduces to the classical MORLHF algorithm. This is because the direction d' = Proj(V', Wy:a) = a for each t as long as is large. However, for p * 1, MOPO solves the non-linear aggrega- tion maximization problem by transforming into a series of subproblems, in which each subproblem only contains the linear aggregation and can be easily solved using any previ- ous algorithm. Thus, MOPO serves as general framework for MF with non-linear aggregation. Moreover, sup- pose we use MOP-RF for each subproblem, MOPO is also a reward free algorithm since the current reward vector can be computed as (V',- = T.. [r," (xr, v) - fT (x (1 [[r]) =
5.2. Online Algorithm
Now we provide the online version of MOPO, which is sim- ilar to the offline setting. The main difference is the adop- tion optimism principle (Eq. (7)) rather than the pessimism principle (Eq. (6)). Additionally, the dataset collected incrementally online, and we also estimate the importance weight a instead of assuming is known.
Additionally, rather than assuming the weight is known, we estimate it based on the frequency with which humans re- port the objective. This method also works offline by using the frequency of related data in the dataset. At each round t, given a prompt x' ~ P and two responses y/ and y/. each group n identifies an objective 7r((m) E [m] showing the greatest difference and provides preference feedback
" (yle) (See a (yle) + 9B [log (((aa))
C' C"-38 - [log Appendix E.3 for the derivation.)
Now we provide theoretical guarantee of Algorithm 3. The following result shows that MfP-offline can learn the opti- mal policy well if the offline dataset D has sufficient cover- age for each objective.
We focus on the integer case to simplify the proof.
7

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Algorithm 4 VPO-abjecctve--lmmiin--eeeee
Theorem 5.4 (Consensus). For the consensus problem, sup- pose the Assumption 5.3 holds and the group has param- eter pln) and c!")., then if we use Eq. (9) to calculate the direction, for å E (0, 1) and 1 = 1/VT, with probability at least - 6 we have
1: Initial: D = 0. parameter (p(m), c(() Jne[M], 8.
2: for = 1, 2, ... IT' do
3: Calculate 6 = arg ming (0) for all i E [m]]
4: Estimate (..(m) {á: (1) kmmm) for each n E [M]
MLE with D and (6) (e()) by Eq. (11)
D(# - D(T))
5: Calculate Wr((n) = Woo pln) where a²-(n) =
S > polyccp((11)) m, M,,,,,,,, N, d.e _d, log((1)))))
a(a0=1,(0) + ((tt(())) for each a E [N].
(min (()))_1)), Bi) Ö(1/T)),
6: Collect Dr by MOP-RB(d,D) or MOP-
nE[N]
RF(d', D), and update D = DT.
7: Calculate the point V = 1 (2)
where AT Et T n², and K. supa,y n.y
Thane Cy
Went (g) x) B
[DKL (Te([ = C C Bly [log x (ylr)
2 2/m - max,, c( are constants.
+ +B (log
a (((y)))) and V - Vs.
For the malfare minimization problem, we can derive online results similar to the offline setting.
*'vl==)
8: Calculate the direction d by Eq. (9) Eq. (10),
and calculate dt+T
d
Theorem 5.5 (Malfare). With the same setting in Theorem 5.4, if we consider the malfare function minimization prob- lem with an integer exponential parameter E N+ and uses Eq. (10) to compute the direction, then for E (0, 1) and least 1 we have
9: end for
10: Return T" = 1 T +
== 1/VT. with probability at
.yt (.())) on that objective. The model collects the data (at, yw (mm) 1/ (x)()) (1,(()) into D(") for all group n. Next, we model how humans select the objective index. For re- sponses you and yi, the gap objective is quantified as las, (z, yv) - (x, v.)).. with the selection following softmax distribution:
D. ( DT((T")
t polpppolyxxxll//pp m, N. N, (og(11)), B1.
(min ( ( min Cn) 11/29) (120). nE(N]
[E[N]
where AT = 1 21=1 n², and k supa,
Thnen ( (ylx))
B1
Neat (3) (2/0)
P([ aa,r,, F. Mor st) x exp(a; - (r] (zc, vw) rī (x, I))..
2 2/m(BB + max, (( are constants.
Then if we define the likelihood function as
6. Experiments
[a, D(0) ,0)
Σ
P([ Ve,
We fine-tune LLAMA2-7B model using Anthropic-HH dataset (Bai et al., 2022) with three different objectives of an LM assistant: Humor, Helpful, and Harmless. We run the offline version of MOPO, and use MOD (Shi et al., 2024) as the sub- aub-procedure to solve the linear aggregation maximization problem at each round. The pseudocode is shown in Algorithm 5.
(z.8 Mi ()ED(()
we can estimate the importance weight vector for each group by MLE as
(x) arg max maa, D("),, ,²).
(11)
a6Am-1
where we use an estimated reward parameter 0' to approx- imate 0* Before we present our results, we assume there is a gap between the reward obtained by following the op- timal policy x" and the reference policy Tref- This gap is reasonable since the expected reward should be improved after fine-tuning.
For p 0.5, we compare the RS algorithm (Rame al., 2024) and MOD algorithm (Shi et al., 2024) (both of which use linear aggregation), and a baseline AR that directly aggregates the reward using non- non-linear aggregation. The experimental results show that MOPO performs generally better. For p -o,, we compare MOPO with max-min RLHF (Chakraborty et al., 2024), showing that we achieve comparable performance. Note that MOPO is an iterate algorithm, thus the computational cost still be high due to the large number of iterations. In practice, we can mitigate this by either reducing the number of iterations or computing a single gradient update per iteration (Guo et al., 2024). In our experiments, we set the number of iterations to 7, striking balance between computational efficiency
Assumption 5.3. There exists constant 7 >> 0 such that
min E Exspaiset anrwnee |r] (x, v2)) 2 7. 2Emms
The following theorem shows that Algorithm 4 is a no-regret online algorithm that can converge to the optimal policy for the consensus problem.
8

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
and performance. To compute the expected reward vector V we calculate the expectation by taking the expectation over 100 training samples, and we believe the performance of MOPO can be improved by using more training samples to calculate the expectation.
There are many future directions worth exploring. First, one can study how to learn the parameter p in the aggregation function like (Pardeshi et al., 2024) using the preference feedback. Second, one can further study the token-level MORLHF (Zeng et al., 2024) based on our idea. Last, it is interesting to further study the multiple preference ag- gregation in Stochastic Transitivity model (Fishburn, 1973) instead of BTL model, and further discuss the relationship between them and previous distortion negative results (An- shelevich et al., 2021).
MOPO is a more general framework and can be applied to multi- multi-group group problems. More experiments and details are provided in Appendix A.
Table 2. Comparison of previous representative work for MORLHF with p 0.5, c 0.5 and objective Harmless and Helpful. The score is the distance between the reward vector and the target set. The smaller one is better.
Impact Statement
The goal of this paper is to advance the field of multi- objective RLHF, which can be applied to many applica- tions in society. Our approach aims to mitigate biases in language models and promote faimess across diverse popu- lations. However, it should be careful for implementation and evaluation to avoid unintended consequences, such as exacerbating inequalities or overlooking underrepresented groups
Ours
RS
MOD
AR
(0.1,0.9)
0.229
0.971
0.808
0.555
(0.3,0.7)
0.051
0.666
0.079
1.459
(0.5.0.5)
0.015
0.078
0.103
1.314
(0.7,0.3)
0.067
0.707
0.800
L.004
(0.9,0.1)
0.184
1.153
1.137
1.526
Table Comparison of previous representative work for MORLHF with p 0.5, c = 1.3 and the objective Harmless and Humor. The score is the distance between the evaluated re- ward vector and the target set. The smaller one better.
A Acknowledgement
This work is supported in part by NSF AI Institute for So- cietal Decision Making under award I182229881 and ONR award N000142212363.
a
Ours
MOD
(0.1.0.9)
0.335
0.362
0.337
1.767
(0.3,0.7)
0.578
0.678
0.572
2.011
References
(0.5,0.5)
0.720
0.882
0.723
1.970
(0.7,0.3)
0.630
0,860
0.722
2,411
Y. Abbasi- Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
(0.9,0.1)
0.217
0.391
0.396
2.068
Table 4. Comparison with max-min RLHF for objectives Humor and Harmless. The number pair represents the reward vector. The pair with the larger minimum value is better.
E. Anshelevich, A. Filos-Ratsikas, N. Shah, and A. A. Voudouris. Distortion in social choice problems: The first 15 years and beyond. arXiv preprint 1xXiv:2103.00911 2021.
Ours
Max-Min RLHF
(Harmless, Humor)
(1.097.1.297)
(1,530, 1.146)
(Harmless, Helpful)
(0.034,0.497)
(-0.135, 0.393)
M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 1447-4455 PMLR, 2024.
7. Conclusion
In this paper, we study efficient multi-objective and multi- group RLHF problems under non-linear aggregation. By
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. Das- Sarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforce- ment learning from human feedback. arXiv preprint WrXiv:2204.05880 2022.
transforming the non-linear aggregation maximization into a series of linear aggregation maximization sub-problems, we
find a computationally efficient algorithm that can converge to the optimal policy. Theoretically, establish a general framework with converge guarantees for both offline and online settings, and the framework is also adaptable to a reward-free version. Empirically, we present a training-free free framework given the reward functions and optimal policies for all objectives.
S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi. Fast global convergence of natural policy gradient methods with en- tropy regularization. Operations Research, 70(4):2563- 2578, 2022.

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
S. Cen, J. Mei, K. Goshvadi, H. Dai, T. Yang, S. Yang, D. Schuurmans, Y. Chi, and B. Dai. Value-incentivized preference optimization: A unified approach online and offline rlhf. arXiv preprint arXiv:2405. 19320, 2024.
K. S. Pardeshi, Shapira, A. D. Procaccia, and A. Singh. Learning social welfare functions. arXiv preprint urXiv:2405.77700 2024.
C. Park, M. Liu, D. Kong, K. Zhang, and A. E. Ozdaglar. Rthf from heterogeneous feedback via personalization and preference aggregation. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
S. Chakraborty, J. Qiu, H. Yuan, Koppel, D. Manocha, F. Huang, A. Bedi, and M. Wang. Maxmin-rlhff: Align- ment with diverse human preferences. In forst-first Inter- national Conference on Machine Learning, 2024.
D. Chen, Y. Chen, A. Rege, and R. K. Vinayak. Pal: Pluralis- tic alignment framework for learning from heterogeneous preferences. arXiv preprint arXiv:2406. 00060, 2024.
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Er- mon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
V. Conitzer, R. Freedman, Heitzig, W. H. Holliday, M. Jacobs, N. Lambert, M. Mossé, E. Pacuit, Russell, H. Schoelkopf, et al. Social choice should guide ai align- ment in dealing with diverse human feedback. arXiv preprint arXiv:2404. 10271, 2024.
A. Rame, G. Couairon, C. Dancette, J.-B. Gaya, M. Shukor, L. Soulier, and M. Cord. Rewarded soups: towards pareto- optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
C. Cousins. An axiomatic theory of provably-fair welfare- centric machine learning, Advances in Neural Informa- fion Processing Systems, 34:16610-1662) 2021.
S. S. Ramesh. Y. Hu, I. Chaimalas, V. Mehta, P. G. Sessa, H. B. Ammar, and I. Bogunovic. Group robust prefer- ence optimization in reward-free rlhf. arXiv preprint urXiv:2405. 20304, 2024.
P. C. Fishburn. Binary choice probabilities: on the vari- eties of stochastic transitivity. Journal of Mathematical psychology, 10(4):327-352, 1973.
R. Shi, Y. Chen, Y. Hu, A. Liu, N. Smith, H. Hajishirzi, and S. Du Decoding- language model alignment with multiple objectives. arXiv preprint urXiv:2406.1885. 2024.
L. Ge, D. Halpern, E. Micha, A. D. Procaccia, I. Shapira, Y. Vorobeychik, and J. Wu. Axioms for ai alignment from human feedback. arXiv preprint arXiv:2405.1 14758, 2024.
T. Sorensen, J. Moore, J. Fisher, M. L. Gordon, N. Mireshghallah, C. M. Rytting, A. Ye, L. Jiang, X. Lu, N. Dziri, et al. Position: A roadmap to pluralistic align- ment. In Forty-first International Conference on Machine Learning, 2024.
S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.00 04792, 2024.
C. Jin, Q. Liu, and S. Miryoosefi. Bellman eluder dimension:
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhos- ale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint vrXiv:2307.09288 2023.
New rich classes of problems, and sample-efficient algorithms, Advances in neural information processing systems, 34:13404 13418, 2021.
Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances neural information processing systems, 33:1264 1274, 2020.
C. Wang, Y. Jiang, C. Yang, Liu, and Chen. Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints. arXiv preprint arXiv: 16240, 2023.
7. Liu, M. Lu, S. Zhang, B. Liu, H. Guo, Y. Yang, J. Blanchet, and Z. Wang. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. 16436, 2024.
R. Williams. Simple statistical fralient- algo- rithms for connectionist reinforcement learning. Machine learning, 8:229 256, 1992.
L. Ouyang, J. WU, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2774 2022.
Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi. Fine- grained human feedback gives better rewards for lan- guage model training. Advances in Neural Information Processing Systems, 36:59008-5903 2023.
10

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
R. Yang, X. Pan, F. Luo, S. Qiu, H. Zhong, D. Yu, and J. Chen. Rewards-in-context Multi-objective alignment of foundation models with dynamic preference adjust- ment XrXiv preprint arXiv:2402.1020) 2024.
T. Yu, Y. Tian, Zhang, and Sra. Provably efficient algo- rithms for multi-objective competitive rl. In International Conference on Machine Learning, pages 12167-12176. PMLR, 2021.
Y. Zeng, G. Liu, W. Ma, N. Yang, H. Zhang, and J. Wang. Token-level direct preference optimization. arXiv preprint arXiv:2404.1199 11999, 2024.
H. Zhong, Z. Deng, W. Su, Z. S. Wu, and L. Zhang. Provable multi-party reinforcement learning with diverse human feedback. arXiv preprint
Sil::400305000 2024.
Z. Zhou, J. Liu, C. Yang. J. Shao, Y. Liu, X. Yue, W. Ouyang, and Y. Qiao. Beyond one-preference-foraall Multi- objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023.
B. Zhu, M. Jordan, and J. Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037-43067. PMLR, 2023.

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
A. Experiment Details
In this section, we provide our practical algorithm. We run the offline version MOPO, and use MOD (Shi et al., 2024) as the bab--rocedure to solve the linear aggregation maximization problem each round. The pseudocode shown in Algorithm 5.
Algorithm 5 MOPO(Practical Version)-Offline
1: Initial: d . ... m TT,, dataset Dottines W.
2: Calculate the optimal policy n; for each objective i E [m]] using offline dataset D Dottine- for 1, 2........ . do
4: Execute x'
(oz((mm)ssmsd 5: Calculate the point Vt E R.m.
6: Calculate the direction d' Proj(W, V*), and get the average direction d Ej= 1
7:
end for
Note that the algorithm average the direction instead of averaging the estimated reward vector function, which can lead to more stable result. To execute the Line following the previous paper (Shi et al., 2024), we first fine-tune the model LLAMA2-7B on the Anthropic-Hi HH dataset (Ouyang al., 2022) to get the reference policy Tr We then get the optimal policy π; for each objective (1,2,3) using PPO approach trained on three off-sheld reward model:
Harmlessness:
m
Humor: http::///hnggggggaee...c/c/oaaae
Multi-Group Problem with Multiple Objectives We perform the experiments Harmless and Humor dataset when we have N = groups. One group has the target set W.., 1.3 and the other has the target set W -oc,1" We compare our consensus algorithm with Eq. (9) and a variant of max-min RLHE. In this variant of max-min RLHF, we use minfrr,, 72, (maxfr1, 03) + a: (maxfr2, 03)0.5} as the reward. We also perform experiments on the Harmless and Helpful dataset with the target set W.... and the target set W The following tables show the experiment results. The results show that our algorithms perform relatively stable and better, while this variant of max-min RLHF performs unstable. However, note that this variant of max-min RLHF also needs retraining whenever one group changes the aggregation approach, which consuming for real-world applications.
Table 5. Comparison of MOPO and variant of Max-Min RLHIF on qulti-group setting. The objectives are Harmless and Humor. The score is the distance between evaluated reward vector the target set. The smaller one is better.
α
Ours
Max RLHF
(0.1.0.9)
0.408
0.992
(0.3.0.7)
0.577
1.171
(0.5.0.5)
0.708
0.429
(0.7,0.3)
0.619
1.342
(0.9,0.1)
0.406
0.208
Table 6. Comparison of MOPO and a variant of Max-Min RLHF multi-group setting. The objectives Harmless and Helpful. The
score is the distance between the evaluated reward vector and the target set. The smaller one is better.
Ours
Max-Min RLHF
(0.1.0.9)
0.230
1.073
(0.3.0.7)
0.052
0.123
(0.5,0.5)
0.015
0.261
(0.7.0.3)
0.067
0.204
(0.9.0.1)
0.184
0.121
12

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
B. Proof of Theorems
B.1. Proof Theorem 3.3
Proof. Then, suppose the reward vector S(T) is ($1,-., 5m) then by the definition (TT), we have
D(T)= max S/. 032 maxfc si 03²
where 87 = (S(xT))) = E. (1, v) Ml ( [ret]]. Hence we have
max min 8i, 0)2 max fc 8;, 032
max 53,
1=1
$ (c min s/
mlc c²)²
which implies that.
mins; s; vm |c c²1,
and
c" - min t
s (vm + 1)1c* cl.
Thus, if e is selected such that |e c*| is small, then also find policy such that
min E,, (r, v) Der. Dxr (=|[+orr) 2 e' (Vm 1))c" cl.
B.2. Proof Theorem 5.1
For simplicity, for the following proof, we use E. to represent Ex-p,y (-1)]]) Since we do not assume the target set W is approachable, we have the following property for the approachability:
Lemma B.1. For each 12 with |10||2 = 1, we have
0 W", m- + D(=*)
min (0,z) S E. zEW-
Σ
0, kL [[Tet) D(T") [101 J(ii,
|10
==1
0
I Vm J(r
rm.
W x²" + ((TT)
|1011
Proof. By the definition of D(T*) d(S(T*), W*), we know that there exists vector with S(T*) + E W and
|lpl|2 = D(=*). Then we can have
m
min (0.z) < (8,S(T)) p) S E.- E. (x.
0, 0L KL (ar
[ret)] + D(r*).
zEW-
The last inequality holds because of |00|| S vm.
We can first bound the regret by
D(#")= -D(TT)
=dW"" Eapr (x, - (* D(xr)
13

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
8
Id d
W. BBT (r" (x,
DKL(T (
-D(T*)
T
t=1
8
8
W Byr (r" (x, d
DeL (7 (
d
W
B 11 (2, v)]
Σ
DKL. ((to))
T
T
7 t=1
(A)
1
B
+d W" T
Ex [+ (a.
DKL (α [(reet D(=*)
(A) d ( ) (T=)).
(12)
The inequality uses the fact that
DrL [
(KL(T] S T
Σ
Recall that
D(x) = d(W", (( [r(, U)1 - 3DKL ( (I(()T(t))
and x- ming D(T*). Now, by Lemma B.I, for each 0 E R with |00| S 1, we have
m (0,z) E,, r, Σ zEW*
min
KL [[ert) D(T*) J(ri ,r W (w,""") D(x*).
Denote V R. with (V'); Ey: [f] 3TL ( and 4p¹ -E V We have
d(V" w7)² / -I (V') )112
-V
-Il (V" 1112
T -
a(V""',,W + 1
T 1V -w- -ww-(v-- (VTT
T
2(T
(
IIw- Iw-((V'-)) (V=-1)) (VT - II -Iw--V"1111
T
First, based on the definition W it is easy show that d' O. σ² is the optimal policy such that
[dd f(x, y)) Σ d'L ( ((trr) > En lid f(z, wi> 2 0,
thus (2 d:) (I. ((ooot < [d f(x, v)1 < B. Hence, given d' > 0 and ||d ll2 = 1,
kL (7t ((=o) < m i=
d S B.
we have ((V'); < B and
VV
mw- T1 S B2m.
Thus by iteration we can have
T
B'm+ + 2(t 1)(("))- -w (v-v(( -v- Tw (V*
Σ
1=1
Now, by the definition of d,, we have
(V Iw (V' -)) (V+ Iw-
(V* -Iw-(V'')
d(v'-1 _ dd. -Iw- (v* VO)
Then, we prove the following lemma.
14

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Lemma B.2. mingew (d', 2) = d' 1w (V'-)..
Proof. In fact, we only need to prove that for any I E W (V (V"-) Il (V (V" ) x- 1lw (V" s 0. Suppose there
exists x W such that (V'-1 Hw--( VV' ) mw (V }) > O. then since W is a convex set, for any À E (0, 1),
have 2λ λa + (1 - A)llw- (V W Consider the line
Iw (V'-))- -x
Hw- (V
tE RR
|[[ (V (v'-)) x|||
Also, we consider the projection of V on this line, and denote it as p. Then we can get
0 (V' Iw (V + Iw- (V (V' (p Iw "V' x- II-- (W- (V'
Hence when A 0, 2. is between p and IT (V--1). Also,
z> 1| V pl1 1 xx11 S V pll + llp Iww- (12 s I[l IIw--(V'' (V' dd113
which contradicts the selection of Iw (V
Now, by Lemma B.I and Lemma B.2, we can get
d' -.IIw (V' S J(ri. " d,xxx)x))) + D(T)).
Then, since we define d= d' /d/1dd/11, we continue the analysis by
(v--1 Nw (V' (V- Tw Tw.((V'') (V*
= d(('"- w') (Id')) .. rm. d',''' (-) +(==")(("))) d' d'.
= =d("' w') (I(d")) (J fr Pm. ",) - J(A,,.. ... , J,.
x.) ,fm;,t d(v"-1 wh) ([ld") || (n - Σ - (()) + D(x*)).
D(xx')))
1=1
Thus get
TdV" way < Bmm +
Σ T
2(1 1)
d(v"
(W*)""""".........."" (n]ld"))a Σ (.(00)--000 n/|dd11 E1(00 +
m
m
( D(7)))
Now we use induction method to show that
T
dv', w < D(T*) +
7
Σ
|d lla Σ - + 2Bm/Vtt
T
==1
1=1
When t = 1, the inequality holds by
| , W - DI s d(V' S((*)) < 2B.
Denote A, 17 ||d2111 (2 (0') - (.()))) and S. Σラ Aj. then for all t E [T 1]. suppose we have
d(vi-), w < D(TT) 1 S + 2B
m
v--T
15

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Then substitute these induction hypothesis into the recursion inequality and get
Td(V".
S B'm m +
Σ
T
2(1 -11]
1)
(((=)) + 11-1++22
+B
vt-T
V
)) (At (Ac+ + D(x*))
T
t-1
1=1
2(( 2(1-1)
1
(2 1) (Ar + D(=*))
< B2m+ +
Σ
D(TT))
-S +B
T
T
T
t=1
BB'm + 1DD xx)²
+
1
St- 1 +
Σ
S1-1
2(t-1) 1)
A1 D(at)
T
T
T
2 22mmi-1
+
2B
(Ac + D(
T
1=1
S BBm+ (T- (7-11)77 (1))))(=") +
T 7 + Σ D(=T)
(T+,-) (T+1-1A)
A1) + +2/m 122mmmBB/TI 2BVTT((")) + (2/m 2B/ AB/VT)Sr
T
1=1
S SB'm+ (T- 1)D( ) + - } + D(T*). (2ST) + ()mm)=(TT)(++++++
+ 2 2B/ /TST
ST. TBBBBBBm ((=*) ST)²
Thus we have
T
dv" w < D(T*)
|d (L.()) - -.(0)) + 2B m
T
VT
i=1
Now we derive the final regret. By inequality Eq. (12), we can get
D(#") - D(**) - (A) d(W" V") D(T*)
Σ (L,(0)) - (())))))((())))))))))))))))))))) 2
$ (A)
vT
BB)
Now we consider the error term (A), which represents the approximation error of the reward function. Now we have
(A)
Ex
(x, r3 (z,
T
T
m
Ex
[ v))( +A1 =100; 07 011,,,,
T
Similar to (Cen et al., 2024), since (r0,, =)) can be formulated as saddle point of the objective J(r, d, σ) Em (T.())))
for any direction (IR + tmm we have
V (oi)) + d, d.B (o) + +1 A v)] = 0.
Also, denote ÓMLE = arg mingee Em (0,), we have
V., L. + (2) 0.
Follow the same derivation in (Cen al., 2024), we can get
di (3 B ((Amii A) /XBB22
|00 B.MLLllEs
1
M
16

---

Projection Optimization: A General Framework for Multi-Ohjeetive and Multi-Group RLHF
(3 + )4(A)4((0)) (m(Amm (Ep, + + 2V
VM
where B' is the upper bound of norm of 0, i.e. maxøge |0|1 - B'.
Now we recall the Lemma 3.1 in (Zhu et al., 2023), which bounds the true parameter and the MLE parameter.
Lemma B.3 (Lemma 3.1 in (Zhu et al., 2023)). À > 0 is positive constant. For E (0, 1), with probability at least 1 - 8, we will have
10: - OL SO ( (3 + log(1/6)
M + V(BB3
Also, (0) is convexfiuctionn In fact,
1 (iI((0) = DD.
+ c R DD. 1 M
Hence, we get
T
(A) S
1 T
(3 (3+c5))((M)) (ED,) + 11)--//a (1/6)
E (1( V)1(( (10,,)/) +X1)+
+ A(B²)
T
VM
(ml
m(3 + B' )4(Amin (ED,) + 2 V d log(1/8)
A(BB3
VM
The notation Ö()) hides all the logarithm term like log(1/6).
Now we consider the term (B). First, based on the convexity of (0),,, we have
(0)) - (.(0).ME) < (VoL. 0 M.ME
d Rarppppr (0, y)] Ba (x, w)), 0f - -L,MLE)
d: (- Ea-ayrr (z, v)] B (x, 0f 0,,MLE)
B ( (6.(I.))) Ex- (0 110 0.MLEllEo
2d:
(Amin (DD,) + 1111 110: 0 0.,.LEllso
"1
(3 + e
B' (>miin (Ep,) + +2)
+ VABB' (Amiu (No..) + ( 2)-1)
0
VM
The last inequality uses the fact that d, S 1. Also, with probability at least we have
Le(0,,LE) I,,(0') < Ō(1)
Now sum over i E [7), we can get
B <
1/
(T:(0)) Tx((0)))
T
<
m
Ö
(3 + BB (min, Amin (2',,) + x)
A(BB)=
(min Amiin (Ev,) + +)- -
M
VM
17

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
where the last inequality the fact that v 1/vM and [d'll vm. Hence, have
D(#*) (m²)
(A) (B) 2Bm
VT (min, Amin (ED; + A) 2 / log(1/0)
(3 + e
4m
m
Bym
B' VX (min Amin (Eo,,) 1)
VM
TV M
VM
VT
(3 + e (0")(inn, Amin (ED + A) 2 v 2
+ log(1/0)
4m 4m B'
3/2
Bym
(min Amin (Eo;) + ++)+
VM
VM
VM
VT
m Ō poly (e""min min Amin (ED,) v log (1g(1)8), B)) + + Bym
VM
VT
The last step is because 17 = 1/1 M 1/M. Hence we complete the proof.
B.3. Proof of Theorem 5.2
Proof. The main proof framework is similar to Theorem 5.1. The difference lies in the approach to deal with the aggregated p-norm of the distance.
Σ Gn G W Σ V Iw (v 1120
(13)
N
Sall V
Nw (V (VT 11
(14)
n=1
T T-1 1 (V T- -Iw= (V )) + T
(VT -ITw: (V TT-1 ))
2g
Sn
(15)
T
n=1 1
For the vector In, Un R with In = (T 1)(v"-1 T-1 Iw (V (1--1))) 1).mn (VT - Ill; (v*-1)) we know || S
BTB 1l |l < 2B 2m. Hence, since > 1, we have
| + yn.112 < (1 1| + 11 + 2(Inn vnn))²
|z||| 2g + 2Z vn)) || 2g-22 + 3 . (21)- m"
We can further bound the inequality (15) as
N
24
dnd 20
V
W
<
Sm Sr + yvlll
(16)
==1
< ST 1)24
W.)) 129 724-2 239
(17)
N
2(T 1) Sa (VT-1
Iw: ( 1) ()
vv= ) 24-2 (18)
IIw
Then since ||.m~||| = -T-1)d(VV" Wt),, we can finally get
2.
S, da W:) < -T-1)*3 Σ
24
24 (v'
w 24- 2 m
(19)
N T-1
+ 2(T 29
So
Iw: (V T-1 D)(T"
---== - -Ilw: (V (1== ()dp-2((TT)) WW.))
==1
(20)
18

---

Projection Optimization: A General Framework for Mujti-Ohjective and Multi-Group RLHF
Hence by the recursion, we have
N
22
22 (V' W < 12
T N 1-20-11
+ +
G.
Nw: (V )( - Iw )) 220-2(1 W.).
==
Now the last term at the right side can be further bounded by
(t - (
1w: (V" ")((" - 11 (V" 2) w.))
N
(t 1)2 -I--1(v11 Wh)d (1ww (V' Vn)
T
N
(a(( 22 1---1 1
W.)) (lid) [ri
rims ti x") d(((t))), w.) [|dd ,m of x'))
t= n=l
1
<
Σ
(t- - 1)20-1 Σ Sn ( W
N
x²) | [d']| s ci 1
29
Σ
(nd((((())) - [Id")) pm, T. F,+t)
(21)
n=1
2 (Di(er)+ nlld'1la Σ E1((0- -
T
N 1320-1
m
Σ
Σ End wi))
(22)
The inequality Eq. (21) derives from the definition of de in Eq. (10) and Cauchy's inequality.
Let ST
V V (V T Wx),, then we can get
2(t 1)20-1 (Dy(+')+ +lldd
m
322
TS < 12 B
129-1 S
Σ
11
Σ
Define A, = (=*=)) + n/|d'|1 EE T1((0)= - Em I2(00)) then we use the induction to show that there exists a
constant Cg such that
S < 7 #=1
Σ
Σ
Ag + C2I-1/21
In fact, it holds when t 1. Now suppose holds for t 1,2, ... T 1, we have
2(t 1)29-1
22--1 1 A - T
26g 129. B2 T
T22-1
29-1
A
129. B20 TT +2 Σ
Σ
A,
-
1/24
T
T
t=1
s=1
< 129. BB2 1 2
29-1
(2q 1
Σ A
k+1
A
(C, (0,))20-1-k7)
2--1 29
T
T
1=1 0=0
122. BB29
29-1
(29 - 1)
))
(C,)
1--
24
1-k
- Σ
A
k+1
h:
k=U
1=1
19

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF Now we choose C (129. B2 &
V12B²m. m. then we have
k+1
2q 1
24 -
1) ) (Cg)20- (00))00---7 24- 1-k
1
< 0 (T)
+
24
A
k
T 1=1
==
2q-1
29
+++1 ) (Co)
1 Σ
010 (T) +
20-2--1--- 109---kk-- TT
24
Ar
t=1
24
A 1/2 1/20
which implies that
1 ST S T Σ
Ag + CTT
1/211/2
1
Σ
A. + 4B 4B 1/2
(23)
T
Hence we have
N
T
2q
(V W;) - (T**
S
||
> - + DBB/mT/1/22
T
Now we derive the final regret. We can see
Da("") Da(=")
(24)
20
(ndd)(((*")) Wz) - (=(=")
0
20
(n Ear (x,
Der. (T< ((rr))))))) 1m
T
20
So dy (Wi:: 1
Σ Eng [1 (x, v)]
8
DKC ( 1m
2x
Go (W2, F-) - -Dz(TT')
T
T
2a
B
8
Sm
d(W;, Rer [r- (z,1))
DKL (T' ((et) 1m)
d(Wa. n:
Ent [f* v)]
DKL(Z* ( - 1m)
(A)
+
| 111 -(0))
(O(())))+(BBB)))))
(25)
The last inequality uses the triangle inequality for 2q-norm. Now also note that
Ber (r" (x,
B
DEL. (5 [|]rret) 1m - d(W. x2 Σ
T
En
fr
(x,
3 T T
-
Σ
DKI ((]()))))) 1
T
T
Eat 117 - (2,,1)1b
==1
we have
N
T
2q
T
(A) <
V
EG Gm
Σ - Σ
B (r7 v) (( w)]
Σ
-
E 1r7 (x, f((a, w)l
n=
20

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Now follow the same proof as Theorem 5.1,
m(3 + BB )4 (ED)) + 2) log (1/8) + (BB1)
(A) = δ
VM
and
7 T
(B) -
||d| 12
(1:(0)) - L)((9))
T 1=1
2=1
$
Nym.nn m
(3+ e (mii, Amin (Ep,) 1)- +
v /ABB)2 (min Amin
+ 2) 2)-1+11 (Ex,) Amin(Em)
M
M
where the last inequality the fact that
N
1-120-11 E GallW(() V 1220
(.1lW(()) -V'12
N
|
111
Σ
dh
ll 22/2 S N Nmm.
2 2g
n=
Combining the Eq. (25) and the upper bounds for (A) and (B), substitute into fj 1/vM M and X = 1/M, we can complete the final proof.
B.4. Proof of Theorem 5.4
Proof. Recall that V E RT with (V'), [r (z, 3) 3DLL (T' [[trr)] and V + Et V., We also define
Wo ((0,0)) Since W' is the estimation of W at round t, we have
d(v", WT IV
1l )12
V I T )1
V
-r (V' 1|2
1112 + [[ (V- -r V 112
y12
+ 2( Uw
) HwT 1- -p (V 1)
(26)
Now by Lemma C.3, since ||V
11000 < B is bounded, we have
) S 4d(v"-1 WT Wd- (WT, WT-1) + 2dl (WT, WT-1).
11
Then we can get
Iwl (v (VT=))- 1l -V )l2 s 2V d(v" T- (V ()d (WT, WT-1) + v2d (W", WT-,).,
Then the third term on the right side can be bounded by
(W ), Hw (V
(wT--(V"-1)
s (
II -w (V" ) Tw (V" 11- II TT (V ()))
+ +1V - 11 1[ (V" -Iwr- (V" 12--111 |||
S av wT--) (d,, Nw (V -w (1- + T
1
[ w (V"-1)
Tw -1
-V -
Now denote d'
111 (V (V" b-- " then by Lemma C.4, we can get
d(v-11 WT-1) 1ldd 2'l s 4 a WT-1)dd, (WT-1, WT) + 2ds, (V/T-IWW")
21

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Then we bound the inner product term as
d(v"-1 T- WT-1) (d', 1ww (V' 1w 1
d(v wT-1) -|dd - 1 ||I (VT-(( -ww (VT-))]
+ ddv wr--) (dt II (w" -l -I ())
By the definition of d', we know that
( F.llTT (ourTVV"')
min (d' x) S (d) Uw (WWT 1))
zEW
S dp, (W?, w + (d' Twr-i (V--1)
Hence the inner product term can be further bounded by
d( WT-1) (de Hw (VV 1w (V"
AV d(v WT-1)dBs (WT-1, ,WT) + 2dg, (WT-1, w 2
+ ddv=" 1 dB, (WT, -WT-1)
- 33d WT-1) d3, (WT, WT + sdy. (WT-I,
(27)
Now continue to bound the right side in Eq. (26), we can further get that
T²a(y", woy² TVIV' Hw (VV I| + 37T"'((" .WT-)) .ds, (WT, WT-1) + 1072dh, (WT-1, Ww). (28)
Now we can further bound the Eq. (28) by expanding the first term on the right side:
T-lv"" lww . (V"-))r (T - 1)² IV lwT (V 112 + +IVT - -IlwT (VV
2(T 1) ( (---1 Tw (V T1 ( V. -I (1
T 11-1))
(29)
s (T-11) - 1)3 | -w ( xf 11
112 +(B (B+ 2
+ 2 - 1) (TT Iw (V ) ).
V- -w (V 1)
(30)
The inner product term is
111,1111r v -1l--: (V ) ),v-_ ".r.T--llpp -1 d(v"-' " [ ( (V -vT)
Note that (df' -1 (V (VT-1)) }) min (d (d_-112) 2). Because |[l (VT-1)] ||| S B1, there is a E W such
that ||= Nw- (V" ||| 5 dR, (W*, 1. Hence,
-.--,, (V T }) s min (d"
, 22) (1 S min (d'-1 2) + de, (W*, WT
=EWT-1
=EW
< J Jr* df + D(T*) dp (W*, WT-1).
The last inequality holds by Lemma B.I. Now we continue bound the inner product term. We have
-- ( ) VT_ -W -WT-> (1--))
=d -1 ( (f -w (V 1 VT)
Salle -.4pp (.-- (11d- Jrrr.... s d -I- "T,"")+++ D(x') + dB, (WTT,, - -JA,,.... P/, of (0-1aa0) x²))
22

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
I d(v'' WT-1), (nlddT-1/h-
m Σ
1 (0*)
Σ _ -00-))))))) + D(T*) + dB, (WT-1, ,Wo))
1=1
Thus the Eq. (28) can be rewritten
TV,(v", w²
$ (T- - 1) |V--1
-w ||=
+ (B B.) + 10T (W"=,,"''
2(T 1)d(V" W.--1).
Σ -0')
Σ Li (03) + D(a + do, (WT-) "oy--,ww) w + 37Tds, (W", (1-1
1=1
Then by the recursion, we can get
Td(V". 2
10f (Wt-1,Wg
< (B + Bj )m
T
T 2(t 1)
d(v' t-1 ,Wt-1). (nlld
Σ
m
L
(00)
Σ L 1 (0 (0-11
m
T
+ D(T)) + dB, (W+-1, (W* + 37dd (W', w(-1))
By this recursion formula, we can use the induction method to prove that
a(v", w²) <
(B + (B+BB)' Pm
10t
de
( +D(=") +
T Σ
7 T-1
|d
E(t( - 1((09)
/T
73/2 B1
tm1
im
(A)
1
1
de,
(*) (Wi-1 (,/
+
37td B, (W*, W(-l).
T
T
1=1
(D)
Now bound all four terms. We first prove that term (A), (C) and (D) are all at level Õ(1/VT).
Term (A): First we consider term (A). Since W = - Wol
pio), ((t)) the term dz,, (W"-1, w can be bounded by
(W0-1, <
Σ dB, (wrat pawss Wo Wo t.(m)
pin),,snss ))
N
(Wat- clo), (rahss))
B,
Since a' = --tat 1-1 + Lāa, we can know Ila' - 1lo0 I tllaall 110 S 4. Then, by Lemma C.2, we have
m3/2BB
dR, (Wat-L(a) Wat((n)
piu) oows p())) S
(31)
|n(n
Thus by Eq. (31), we know that
10N
N
T m B² B
(A) <
73/2
(pln)}z
n=l
N
10N'3 B²
(32)
(plm))z
VT
n=1
23

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Term (C): We have
B1
1
B1
(C)
|la [a'=1,(m) a01oo
T
T
|p(n)|
m3/2
B
T Σ 1=1
-7- (xx((4/)) (poly(m, eB ,d, d, log(1/)))
Vt +1
[p(n)
n=1
Nm
B1
1 exp(4/B) Ö (mo e .d, log (b())))))
(33)
min |pin]]
Term (D): First, we have
1 T (D) v
Σ 37t Σ
da (Wa(n), w(-1((0))
T
1=1
n=1
Then, by Lemma C.2,
37B1
37tm3/2B, Bi
(D)
[|a -a" 10_1,(0))
T
T
|o(n)
1=2
3/2
B
+1)
T
3/2 B
N
(1a2-(n) - a=,(n)) 1l00 + a².(0) a'-1.(m) (2) +1
T
3/m B
T
37m 3/2 B1
Σ
[((.(n) - ax)( 10 +
la (a-=1,(m)) (m))
(34)
T
(E)
(
For the term (E), by Eq. (57), we have
T
(E)
||a
.6 (poly(m, (oooy(m,e,,ex(((//)_.. loggee VT. -
Also by Eq. (57),
T
(F) S y -1. Ō (poly(m, eB, eop( log Vi
S -T- ō ( dog( VT.
By Theorem B.4, the term (B) can be bounded by
(E) 7 pplly(epp((11/) m, e. d.oog(//)))
Thus substitute these upper bound the Eq. (34), get
T
(D) < T Σ 37t Σ dB, (WL,(n), wt-1((m)
I - polylyy(x))(11) m. N,e d, log(1/5), Br. (min (m)))- 0(1/VT).
(35)
nE[N]
24

---

Projection Optimization: A General Framework for Multi-Ohjeetive and Multi-Group RLHF
Combine them: Now we combine the upper bound of (A), (C), (D), i.e. Eq. (32), (33), (35), can get
(B (B+ (B) + pplooyyxxx((//) m, _ ,d, log(1/)),, min 1 + D(T)) (B).
d(V', (²)
VT
ne(N] pinj
(36)
Now we consider the proof of Theorem 5.4.
D( -(x")
d(W", Exr (r" (x, v)]
3I..... (# (7 [Tottt ((=")
T
s <d(W", E (x,
DKL( [[et)) - D(T))
T
T
8 T
=d(W", E (x, v)l
8
DKL( [ret) d(W", 1
B [F (z, v)]
Σ
DEL (x (
T
T 1=1
T
t=1
T + d(W",
8
En (+' (x,1))
Dea. (x (ret) - D(x*)
T 1=1
T
(+) + d (w", v²) - D(TT)
(+) d(W" W d (WT, ,v') D(T').
(+o)
Term (+): First, the term (+) can be bounded by
T
Ex [f((,.)) - -f
T
Now note that
T
Σ Ree
- -
E Ey yoottowe [((F)(2,))) - (x, (1, (x,,,2)))) 12)))
(37)
T
Now since the reward contains a linear structure, by Lemma D.3 with deover (1/T) Ö(d), for any i > 0 we can derive that
deover (1/T) + O(Bd)
(+) 5
f4
Uy u Ther (ri((, - ri(,)))) (ri(a.)) - -r (z.,
4t4;
deover (1/T) + ((B))
Mi exp(( 4/13)cc
B B u w 11 r0 vo) (r(( (x,
App
(1/T) + O(Bd),
Pi xxp(4)
01.22 [1
(38)
4p
The last incquality uses the fact that
Tbase (y xx)
Tbase (y x))
Teerly x)
sup z.9
Ti xx)
< sup
2.9 Teet
x)
sup 2.
T ( 2) - exp(4/8)
where K sup,.g
Thase (vla) (eaf(g)))
(Cen al., 2024).
25

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Term (**); Now we consider the term (**). By Eq. (36), we know that
VT (B
B1)
1 (I)O(1/VT)
(**)
S
poly(exp(11/3) m, e. ,d, log(1/6), min nE[M] pin)
T
(1(50))))))) - L(()))
(39)
t=l
Now by the MLE loss, there exists constant C" such that
n (1 L((0())
T
TR
C' Σ
m
S
nl || log(1R1/6) Σ
glld"||
Σ
Bynre
[1 y)
JED.
C²
(2mm
nlld
IE
yyuns
(40)
JED
Now consider the second term in Eq. (40). We can bound it by
m
Σ
E
Be (1) v)
JED
B wu (1) IIT i)]
T
2
M1 12 N ' +}}
1=j+1
T
T
K1
Eyg m [2) v)
m
T
K1
ΣΣΣ
By Ma [1) y)
(41)
where the inequality uses the fact that inf a Fala 2.81. Ki for some constant Ki Since the distribution of index a bounded softmax distribution, we can derive that K1 2 20 1)el m
Thus we can get
T
T
ΣV
Σ
[Z, 8)
ya v2 [A
me B
JED
Hence, the Eq. (40) can be further bounded by
1
T (T(())) - I=(0 & O(2mmy/md)
E
En
(x)
(42)
Tme
C/C
T
< (2mmy/md)
E My 3e (2.
(43)
Tme
The last incquality uses the fact that [|d' [1 2 1. Now combining (*) (Eq. (38)) and O'
(**) (Eq. (39)), by choosing μ4
Bh eeV/ we can get
D(#T)= (xx))
26

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
(+)+ + d(W", WT) me exp(4/3) (rde (1/T)
(B + Pm
mm3/2d
+ ((Bd)++
+ =d(",,"'
(44)
4C'
VT
VT
Note that
N d(W" W S Σ
N
d3 (ponise plts, (ppm))) S mm 3/
112 a ax.((1))la
p(a)
n=l
m Bi
Σ 1 n= p(n) "9 " poly(expp1//)) eB ,d log((11)))
(45)
VT
m m
min NE[N
"oloy((cpp(()//)) m, e. d. lo
(46)
VT
where the inequality Eq. (45) holds by Theorem (B.4).
Hence, combining Eq. (44) and Eq. (46), we complete the proof.
B.5. Proof of Theorem 5.5
Proof. First, note that
d(V", 2T 1V
1W (V') (V") )||29
1120 Ow )1/30
-w 1,60 (V" |1= + [ (V (VT-1) -rr--1o (wr-LLo0(V 11
+ +2 11 (V H. (v [,
-1 (V (V" -
(47)
Now by Lemma C.3, since ||VT--1|| 11x0 < B is bounded, we have
W (VT-1) 1w 11 I 4d(vT-1 .wT (WI.61) (.0-1(0)) + 20% (WT,(a), w(-11(00)
(48)
Also, by Eq. (27), we can also have
WT 1.m) (V MWT. (V (V--1)- ww 1mmm) (V )
< 33d(v wI-1) da, (W2.6)), (1-)(( + sdy (10)) WT.)))).
Hence, by Eq. (47), we can get
22 2T.60)22
11 (1 11? + |1l1 (T. (1"-1)- 11 11 (V--1)1 1|2
9 1,00) (V (V--1))
+ +2(V" -I 1,00 (V II Ww (VT-1) -IWT-LLM
<220 (IV" Iw 1,60 11 112 + 37d(V" STd(V"--WT--J
.WI-1) dR, (WTi(n) WT-1((m) 10d2 (1T-), (olli))TTxx))))) wT(()))
(4) (
27

---

Projection Optimization; A General Framework for Multi-Ohjective and Multi-Group RLHF
Now, since d(WT.(), WT-1.(m)) <
m 3/2 B1. 1pinl
-Ilat(m)) a a011,(m)l -
m//2 B1, we know IpiNIT
37 37Bmm² mp 3/2
10.fm²
37d WT-1) dB, (WT(()), (T)),( + 10d (WT-1,(m) ".TTiit))TTTiin) v
+
(50)
(p(n) IT
12(2)222²
Hence, the Eq. (49) can be further bounded by
T T d(V WT.()))3
< T29 w + O(poly(By m²,
(min nE(N]
min (/m))- -9)
11 - (V ||| 20-2
(37
1- 1. (y.--1.(m) dB, (WT,(x) WT-1,(m)) + 10cB, (WT-1,(m) WT.(m)))
(l-iit + ((poly Bi m min p(m) - 2)
(V" wT-1,(x)) 1x) /s, (WT.(m), WT-
+
The last inequality is because pr poly( B1, m, (mi 001))/111111//1/11101111 (D/TT)), and
V
2 V - -1 WT I o(poly(B) m² (min ( 20 HWT (V T (|² 2|2= 20-2
NE
Now we further bound the first term Tea (.))). Using the same derivation for Eq. (20), we know that
N
24
(V
7=
$ (T 1)20
Snd (V wT-)(()) + 12972022B3 m
+(T - 1028-11
Sm WT 1.61) (V DV''- Tw 1.10 LV (V" 2)P00- wT-1,00)
))(TT
Hence, we can derive
N
24
dvd(V" WT.
n=1
(T- - 2)
Snd2 (VT (T-)(()) + (polyy BBq, m2, ( (min - 22
"E[N]
n=1
379 24
(V T- wT-1,(1) da, (WT.(1), WT-1,(m) Ln
N T-1 - (V" D(V" I1 w. (VT-) ) )0620-2(1 w.(.).
+ 2( - 1)20-1
(V
Now consider the last term in the inequation above. Similar to the Eq. (22), we have
Ga Uwr 1ew (V (V==)))(VT Hww 1.60 ()
29-1
20 (V--1 wT-))(())
Dy(1T)) D)(T*)+ 7lldd'11
Σ 1i(00) E1i(0
(4)
28

---

Projection Optimization: A General Framework for Multi-Ohjeetive and Multi-Group RLHF
then we can get
N Sa d(V", WW..(n))3g
T
I (T 1)22
Snr ( wT--((0) + ((poly(B) m (min p(ui) -2)
mE|M
n=1
Sn ( 11-111(0)) )
+ 379720
2g Bi (WT.WM) WT-1.(m)
11=
n=1
2(T - -1)20-1
Σ
2 ( w
(D)((=') + (x))
$ (T 2)
2 ( (T-1,(n)) + OpoolyBB) mm
min -(m -9)720-2 =E[N
NV
+ 2TT - 1)
Σ
37qT24
de (WT.(M) WT-1,()) + Dz(=*)) + (0)
1T-1)22-1
Hence, by the reduction and the fact that - S 2 for T 2 2,, we can further get
N
T
2a 20 (V" (T(n))
T
N
s Copooly(B) m²,
min ne
1)24-1 2(t - Σ 1=1
Σ
Sn S .WT-1,00))
37q 2
(WT.(n), WT-),(m) + DT(+")+
(WT- W (+)
Similar to the Section B.3, we can use the induction method to derive
(V WT) - Dx(xx))
T
N
< θ
poly( (Bly(B m,
min ME[M]
+ (*)
+
24
S
2q Bj
(W+-1,(m), W*)
T
N
1
37q 24t 20
Σ
Sn 24
(W+((a), w(-l,(m)).
T
B,
1=1
n=1
Now note that
N
33 B220 | (m)))
(n)
2g
s 2g
Σ
S
(51)
1p(1))22
29

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Y 11
O(poly(m, e (og((1)))) log(1/6))
(52)
min |p!")
Vt
Also, by Eq. (35), we have
T
N
T
Σ
Sn 2g B,
(W4(n), W)))(()) (Wh(m),Wt-1(m))
- T
Σ t Σ dB, (W*(()) w(-1,(m)
n=1
- y poly(expp11/1)) m. N. eB d, log(1)))), B1,
min p(m))-1) (1IVVT).
nE[N]
(53)
Hence, combining Eq. (52) and Eq. (53),
N
(V' .WT) (x(
< Ō poly(BI, m, min 2111 mE[M]
+ poloyyesp(L//))mm.mm m, N,e" d. (d.log(1///) (0/), B1. min p()) - + (B)
NEIN]
2 (r="pooy(cep(11/)),,,, eB d, lo,(o111/0) B1, (min (m)) (1))-11)-1/22 + (*).
nE(N]
Now derive the proof. First,
D2( )- - 2x(x²)
N
Σ (.
Σ
(5(7))))))
(S WT) D,(x")
24
Σ Ga Gald(S(T)).W)) d(S(#")
+
20
dnd (WT BAr (x,
0
DeL. (T [| 1m)
20
(WF, Vz)
T
- -D,(x")
d(W, w
n=1
29
+
Sa
d(WT, E.r (2)
DKL( 1 d(WT, v")
T
(x))
+
(r "oly'pyyxxp(1/)), m, e d, log(1/6), min
me[N] p(n) 22) + (+).
1
First, for the term Σm= d(W', ni W ) we can bound it by
3/2 B
Σ d(w W
-100
T.
[2())
30

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
From the Theorem B.4, can get
m 1 N e p(n) ? !- Ö (poly(m, e", ,exp(L/3), 1
Σ d( W
(54)
VT
Now by Eq. (38), note that
T
T
3
B
r (I,
Σ
DeL (x [ 1
T
T
1=1
t=1
we have
N
T
24
N
T
(**)
S
Sn
Σ
E [r] (z,, v) - f((c:ww]
Σ
Sn
B - v))
T
T
n=l
1=1
n=1
i=1
1=1
deover (1/T) ONBdd)
#4
un wa (x,
(55)
4/r,
Consider the term (B). By Eq. (42), we can get
1 m (B) T
C 112 E
m
Σ
aldd (L)
LI(05) - (2mmm /md)
B1 (1, y)
Tmm
j=1
nyC 1/24
T
o (2mm//md)
E
v1. m (A) (56)
Tme N
V² The last inequality is because, if we choose n' nEIN Sa (W(W)
12 then
Sa. [W(n)) V²1 2
Sa Wn V²
224-1
1/24
[ St.
1/24
2
ENN -
29
N
2g-1
Spp
29
[W(''')
v21
2
N
24
N
2y
2q
-
Hence, we have
N
Ca (/ v' 11122-11
22--1
minnelN C1/20
||d =
Σ
d
2 |
2q-1
N
GollW(()) V 24
N
min C2/20
Now choose 14; = 17
xg-r. 11 1/ I/VT, and use the inequality Eq. (56) for bounding (B), we finally get
me! exp(4/3)kN 24
( - Dy(TT)
d(W W + (A) + (B) + 10 (o=-pooyyesxp(1/3) m, BB d, doloo(1/)), min 1 ==1
nE(N] p(nj- B,) 11))NN/220-1111
$ (oll((xxp11/1)) M,N,N,NNN C. ,d, log(1/5)) k. B1, minn m1 (in (n) -n)0 (TT-1/2))
nE(N]
mE[M]
B.6. Estimation of a
In this subsection, we give a theoretical upper bound of the estimation error of a.
31

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Theorem B.4 (Estimation of oi). Assume that we execute the Algorithm 4 with the Assumption 5.3, then for each & [T], with probability at least 1 5 we have
1 Vi
||a¹ - a'llo
Σ 1a" a= 1 Ö (poly(m, e", exx((1/8), d, log((16)))
(57)
Proof. First, for any E [4], we estimate á with 0jt 02. .... m.
where 0f = arg ming (0) only minimizes the log- 8:
likelihood loss without optimistic exploration. Define 0f (x, y) = (r 5h by theorem D.1, with probability 8 we have
(z, b1) r (r. v2) - (r, in) - (x, a2)) then,
8h (2, v1) - D, (Sottmax (át |r,
T 8: (2))) - Softmax (ag -17 vn) r7 (1)
|
?
TV
- 2o dd (1/12))))) + 1/k:
where (Softmax(x::) | iisss m, x; I 1}, and the log of e covering number log(d>(1/k2)) Ō(m).
thus by the Cauchy's inequality, can get
2k (1g(//(1111))/ +1
E y- Softmax (a+ 1r. oz (x, - -r
ot (z, b2)l - Softmax(a] |r] (x, - (2, (1)011)))
E ErD |Softmax 1 (1, r v2)) Softmax(a) (2r, 3n) rI (2, (2,
En.yDD Softmax 1r? (s, v1) ri 0 (r, v2))) Softmax(at - (z,,)))) (x..))))
(58)
Now bound the difference of a based the difference of the softmax distribution.
0h Fixed k since the upper bound of 0 < ri (x, v) S B and S r* (xx v) S B, define X; =11 84 (x, (x, ri
(z. 12)1 < B
and X (x, y1) r (I, y2)l $ B, then
Softmax
I1r 0t (2.11) - ri 8%
(2)))))) Softmax(á
rī (2, (c..82)1)llyv
exii
ex.aa
Σ
Ej exvay
Eje
Ejes X3.0f+xx0t
Σ
(,, xxxrt) (x))))n))))
$ Σ Eies Eimee exjat xxat (6 -1) m²
where the last inequality uses the fact that E, e Xx 2 m and E, exjaf 2 m. Now since xja)+xia" - {(a(ttt <
BB and e 1 < e a for every 0 < a < B, we can have
Ejte
2B (8) 6%6
m²
2B
Σ
+ 67
m2
iti
32

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
2B
Now choose the index = arg max
X |a - a.l and WLOG, assume af = aj + E, then we can bound
(oottaa
(1) ( (=,,012)) Softmax(a) (r, (1) - rf (r,, v2)))))))
exia
exitaj
E, xxaaj
Ejetias
(x((ttt))
exi
x((a+te) Ejjt Xjay exiai + EESte
Explexio 870g
Ejfle ex33 tc) - e235+42701
( (E (())))
Now by the selection of the we can have
Xja (i +5) Xzā xfaai
hence
e xja 2 x3a xxat+xiai
Also, since at = Ei a. =1, 1, and the fact that ot = = + can further derive
Σaj = Σ
- a
j#1
j+l
then at least one j' + such that oz. 2 az. + e/m. then
(7a34xx(00-=4) - a0xx7+X770
XT +6) Eta
xx (c -1) 1)
of eXi.
Thus,
Softmax (x, ri (a,, 02)1)
Softmax(of (a. - of (z, 12)
xx
E X3at (E x(a3) -xX,
(me la/ ailli
Now define X = (X², X3,
X Rm and |a - a²1 = (la] a&t. ... laan - - E Rm We can get
||X* o laa - à < m 2 2B
Softmax(a
|n (z. v1) - r (z. 32))) Softmax(a (n. (1) - rf (2)9))
where Y denotes the Hadamard product. then take the expectation we can get
Ex.yD,,II' o laa a al m² BB ||ootmax(a (r, (x, v2)l) Softmax (a 117 -r 12)
33

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Hence, by (5.) have
2k (1/
E Bry
|Softmax(at (1, - (x, 122)l) Softmax(of (1, - (2,
En, Softmax(á 1r, (x, v1) - 8t (2)) - Softmax 117 - (7 v2)
R-1
Er.y D. 2B
|| y))a" a011
Ex /Softma((a 1r, 0 (a, v1)
Ti (2)l) - Softmax(a (r, (z) ( (2..)))))
8
R-1
2 k-1
EnyDD D. 28
|| (2, - =0llo0
Eg y D. (o.
m
Hence we finally get
E En.yD, (xx o ola" - a0 S mB (V2h (2hloo((d(11/k2)/))
2B k
m
(x v )at
m
poly(m,exp(B)) δ
(Vhmlog(1//0 log.log((()) (1/5))
E -22- 10% (Ir, 2)]aa
(59)
the last inequality holds by Azuma-Hoeffding" inequality with probability least 1 5. Now by Lemma D.2, we can get (vl=)
sup 5,2,g = I exp(4/8) and sup..y (est(gin)) [g) I exp((/3). we can get
r(k 1))la² a* 101 S (k- 1) B mam X (z, (1 o la= - lâ
cxp(8/3)
E yn Wer |X [x v) la"" - -ā"llooo
(60)
The first inequality uses the Assumption 5.3 that Eny wer [X (x,,)) 2 7. Now combining (5. and (6,), we can further get
(Kk 1))la* - 1011200 S exp(8/3) exp((m)exp(B)) Ō
Vkmlog(1/5))
21 - 18 (x. v))a
k-1
cxp(8/3) poly(m, ()))) 5
(kmlog(1/0))
E Mn wa
18 (I, (2.
(61)
Now we further derive the final result. Frist, by a' = Et āt, we can get
||a" a'llo S Σ aa @
2 r1 exp(8/3) poly(m, exp( (B)
Σ σ
m (og((1))))
k- 1
E wa (0, m\) (62)
Vk
1=1
Now we derive the final result. First, we can get
(1 10.- 0 6sla"
.vn) - 02l ,v²i
34

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
< 1 07 p 110,(10" wf) 0,(x", 112 04 1)7;;
where DM-1 {s [k: 1] L, i} and -T. E 02(2", y² (x'" ,²") is the covariance matrix. then by
Lemma 3.1 in (Zhu al., 2023), we can get |10 02llspp-11 S C(d, B, 6) poly(d, B, log(1/)) for some constant C(d, B,8). and then we can get
S C(d, B,6) |1: - 0i(x", w 1.
Now apply the same technique in (41) we can get
172
m
Bu T: (z, S me" B
En w2 _ 10x (z, y)a gz)
(0
SED
me² B
E U1. 12..gama
Σ
claot (Ir, va
S me B
To (t
yn tra T
The second line is because that, the summation is over
[0c, i) & E [t], E [m]. s E DD-11) = ((k,i,s) k E it) i E [m] 8 S k - 1, =1
= [(k, 8 E [t] k > 8, 1'}.
the last inequality uses the fact that a4, S 1. then we can use the Azuma- Hoeff-nnsss inequality to further get
m
1 10 (2.
E yn w2
y)a S me" "
8%, (x", ./)) + O(VElog(t/8)
S me
B
k
Odd, B, 1lob,,, (x",wf] (,, 12)) 1000o
ot
k= 1228
+ +O(vtlog(t/0)
(63)
with probability at least - 8. Now to present the proof in simple way, we simplify Epp- 1 as Ek- We will have
B
C,d, (0", vf) - (x (=", 1
me
S mme" B
ΣΣ
C(d, B,ō) (0", vii) (x", , :
K
s=1 #>8
me
B
C(d, B,8) vi) - (x", v5) ( k
logt
Kl Σ
C(d, B, o, (x", - (02,,22)(
(64)
Now, we can decompose {1, 2, ,t} into m different set D; (s E |4| ta i}. (x0g )-2 with ||M < B², by Cauchy's inequality, M, = |0, (2") O1" v2 1220
then, we fixed i and denote
Σ | (x", - Φr, (x", vzil (x:.0))= SED.
35

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Vv
Σ
M
sED:
S V
Σ
M,H(M, S 11f VE
Σ
M,LLM, > 1}
SED:
SED:
Vt.
min +
B Σ
1,M, > 13
SED;
SED;
S Ö(BdVt.
Then, by summing over E [m]. we can get
logt
Σ
C(d, B, 6)1lor, (o - Φr, (2", 11)))(
61
logt
C(d, B, m
K1
Omm²eB. Bd C(d,
(65)
Now combining Eq.( (63), (64) and (6.) we can finally get
1 Σ ||a" l s exp(8/3) Ō (poly(m, eB, d, log(1/6)))
- S
Vt
=タ- Ô (poly(m, do Vt
with probability at least 36. By substituting 8/3 with 8, we complete the proof.
C. Error of Estimating the Target Set
First we provide a lemma to show that the projection on W* is also bounded.
Lemma C.1. Fixed the requirement p(n), c! for all k [K]. For any importance weight (a (a(")) ]RE[K] such that ((
and [1)
||1 = for all & E for B1 = 2/ + max, c( we have
K
[f)llx) s Br, w' n
Wa
plu),cts)
holds for all ||0| < B.
Proof. Suppose we choose any E W*, then by the definition of projection, we can get
|Il (x))lo vmB < ||æ 1w (x) < || - -lw- (x) S l| - ull S VmB + 1lll,
which induces
[Iww-(x)l s mB + llyll-
Now consider y = (z,... TTT E Rm when max, c(n), for any (n)
1/p(n)
(/D(n)
m
m
Σ
a y
(n)
(n)
(n)
That means y (n) p00) and then y E W* for any & [K]. Hence have
[[- (x)1l S 2B || 2 max (1)
We complete the proof of lemma.
36

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Now we consider the estimation of the Firt, we consider the estimation error of Wa when have an estimation error of a. The following lemma tells us the estimation error of parameterized target set.
Lemma C.2 (Estimation error of parameterized target set). Suppose we have different a, o', the distance between W and W can be bounded by P.C.
m3/2 B Bla a a'1
dg(Wo (Wa W - piet
p
where
dB(S,S")= max
max (*
d(z, ((x,)ls(x))
max
d(z, (z())
wES' +|
represents the distance of two sets S and S' restricted to some bounded set.
Proof.
Suppose p E [0, 1] and x E P.c. with || S B, then we have
m
Σ
a.a c c.
First, if E1= a'=," 2 c², then x E Wo' and the distance d(z, Iw (x)) 0. Now we consider the auxillary vector y E R where ys = x for E [m]. Then Ei= agyy 2 c². By the formula of the distance between one point to a line, the distance between y and Wo p.c = {v: E i= 0gli 2 c², Vi 0} can have the following upper bound:
d(y, 11 wst (v)) c Σ divi 0}
max E (a, d()us: 0f ||a dBl (a()
V2Th t -i=16 (a²)
V (a,)
Now consider p- O we have E aga? c.. If E a/ys c then p.e and the distance d(sr, Il wa! (x)) 0.
Otherwise, note that we can rewrite Wa = {y: E ail S dd, - 0}. We have
E1 d/y c
||a a' ~|00 Σ th
||a a a'11>0 mB
d(y, D (v))
-= (a()2
(a
So in both cases, we can find
d(y, wg: po (v)) < |a - a' mB
|a a' 1100 . mB
m 3/2BBp B2 1|ax al
(a))
1/ 1//m
Now since by Langarian mean value theorem we have 100 - 1pl 2 |pBp-1 111 - yl. the distance between x can be bounded by
dly. W. (v)) S 3/22 B ||a a'l
3/2 B
la
a' a'll
d(x, ww (x)) S pB b.c
[pl B
P
The second lemma shows that the distance between the projection of one point on different convex set. Lemma C.3 (Distance of Projections). Fixed point with || B. Suppose we have two conver sets AI, Ag. then the distance of two projections can be bounded by
- - < 4d(I, AI)dn, (AI, + 2d A2)?
Proof. WLOG, we can assume d(I, < d(I, A2). First, we consider II (II (I)) E A and d(Tl A (x)), AA, (x)) < dp, (AI, A2), where B, is from the bounded assumption of the target set. Now we only need to consider d(IIA, ( (z)), (x)).. Since A2 is a convex set and IIA, (IIA, (x)) E Ag, we can have
(x- IT,, (x), AA II (TA (x)) 2 0,
37

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
then it easy get
A(L (UIA, (z)), x)² d(z, A2)² + d(Ll,a (x),, 1l (
Also, by the triangle inequality, we can derive
A (IT (z)), S d(z, A1) + d(IIA, (I, (IIA, (x)) - d(x, A1) + dB, (A1, A2).
By combining these inequality we can get
(TTA,
s 2d(2, A,)d,
(A, (A1, + dB, (A1, A2)²
Hence we can finally get
[[ (I (x) < 2d([IA; (IIA, (z)) (z)) (d((a) (IIA, (x)) IIA, (x))
S 4d(x, A dBB A2) + 2dB, (2),
Now we derive the difference between the direction.
Lemma C.4. If the angle between the direction IA (x)) and (x))x= less than n/2, then the difference between them can be bounded by
d(a.A1)
d(arA2)
AA (I) x d(x, A1)
IA (z) 4Vd(x, AI)dR, (A1, A2) 2d (A1, A2) d(x, A2)
<
max d(x, A1), d(x, A2)]
Proof. Denote the angle as A Consider the triangle (ar, II, (I:), [lg (I)) By the law of sines, can get
d(IIA, (z), d(I (z))) max [(x (z. A1), d(x, A2)]
sin A
By Lemma C.3, we can get
sin A < 2 2 Ai)da, (Ai, A2) + 2d (Ai, A2)
max (d(x, A d(x, A2)
Now since A < nT/2 and the direction be bounded by
A1 (z) - x d(x, A1) d(z, A2)
(x) -
sin
I 22 A A1)x, (A1, + 2d (A1, A2) max(dl(x, A A2))
sin
D. Auxiliary Lemmas
Lemma D.1 (MLE Lemma). We are given dataset D = ((I;,)), v.), where a; D. DD( 111-1) and ys pl-
x;)= = (2,,)) -). Now if we calculate the MLE by
arg max JEF
log f(zi, Wi),
then fixed E (0, assume |F] < oo and 1 E F, then with probability at least 1 8, we have
EgED,
[f(x,)) f* (x.,
< ( 2og((())1/)
38

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Lemma D.2. For any n, x E { n² x Treth J. we can have
Tt x) (y x)
S exp(4/3))
sup 2.
Proof. First, note that and x' are both optimal policy with respect to some reward f, then can be rewritten as
( ax) x Tzot æ) exp f)))
Thus by the Appendix A.2 in (Cen et al., 2022), then for any y and x, we have
log my x) - log mer x) < 2B/3.
Then
(y x)) 2.9
Tyr x))
sup
S exp(2B/3),
sup a.y
Ty x) S exp(2B/3).
Tet (y/ x)
Now from the two inequalities following
sup ( x)
< exp(2B/B),
a.y Tee (y) x]
Tee (y/ x)
sup
I exp((BBBB)
a,y 7'y x))
can multiply them and get
( 12) sup
2.9 (' x)) s exp(4B/B).
Lemma D.3 (Linear Structure). Suppose that we have reward sequence (r"()) with r(()) (0', <bx)))
with
||l| < 1, [[xx))] < B, then for any policy {x² herry for any μ > 0, we can have
Exans (r'(x)) Ö(Bd) deover (1/T)
Bxasi (x)) Sp.
Proof. First, denote X = Exi [d(x)), then
Bx (r'
Ex ((x))
tm/
(0* xt).
Now define E, =l + Ei=l X' then can decompose the term above
T 10'
x2 Σ (0'
("]((1X"'1,,++ 11)
00' x
1).
(A)
(B)
The term (A) can be bounded as
T Σ
(A)
10011 1x11s=-11111
||s_+ S 1}
39

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
|00 11,, min [X, 2
/1/2
1/2
e110%112 Σ 00' Xij²
min[1, 2 3112
i=1
1-1 =|10'112
]
Σ (00, (0, x113
Σ min (1, |X .
where the last inequality uses the Cauchy's inequality.
Now we recall the elliptical potential lemma (Abbasi-Yadkori et al., 2011), we can get
min [1, [1,1 d(e)
(dgoo(())))
(66)
Thus substitute it into the the inequality for (A), we can get
(A) S /
d(e) Σ
e1l00112
(0,, xxie
Now by the inequality that bb va + vb, can get
(A) ( d(E) Σ
e =110c112 Σ
000 x:1z
T
S d(E)ET
d(E) Σ
(0 x212
d(e)
Vd(E)=T +
00, X212
μ
4
d(E))T d(E)
T
+ μ
E (z)))²
44
j=1
Now we choose & 1/T, then d(E) Ö(d), and the upper bound of (A) becomes
(co0er(11T)
T t-1
(A) S v deover (1/T)
+ p
Σ
Em
(x))))
41
Now we derive the upper bound of (B).
(B)
(0' X x1 1| > 17
T
< B
H|| |5-1 > 17
T
B Σ
<
< Bdeover (
40

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
So by adding (A) and (B), we finally get
00 x < Bdeover (1/T)) v deover (1/T) + deover (1/T)
+ p
(L [r (x)))
4t
s O(Bd) (1/T)
+ μ
( (r² (x:)))
4p
E. Some Derivations in Section 4 and Section 5
E.1. Derivation of Reward-free Modification
Now we derive the equation
log x (y x))
122
J ,2, . fm Ct, )
Σ (l,(0.) C -..
I,(O.).
1)
Trae (y) x))
In fact, since
J(r 12 ... mm , T) = E (-
pm
a. 0: (x, -
Σ ar (log (y x))
log Fea (3) x²)
E Eynx"(-1e))
ar7(z,y) v) -
a (log T (y x)) log Tre x))
= Ey (-In)
[log (r,,))),
where z(r,z) = EEvEy Tyerly x) exp(r(xr, B) is normalization factor independent with (Rafailov et al., 2024).
Now, since Z(r, x)) is independent with y. we can get
J(r 0: .22 ...
Om m
a π)) =
Ey ( (+1x) [log ((r,
E Byye (122) [log Z(r, x]]
E Ey (11z)
[r(x,, 3/ B(log T (y (x)
log Trof [))]
log ~ (3) x))
C BE El
(ee x))
We complete the derivation.
E.2. Update Rule of Gradient Descent
In this section, we show that the computational cost of Eq. (8) can easily computed once the expectation of the score function can be derived.
In fact.
m
Ve, (- olog T
Ve, Σ ((D.. ( (y x1))
3 E Bynppyy-m Va, log x (y x)] V v, O))
(b)
Term (b) in the last line is the gradient of log- likelihood loss that appears in classical reward-free algorithm like DPO. For
term (a), note that if |d]]] 1, we have
2 x Trefly) | x) exp
Σ
Bd 0: (z,
II ( (y x))4

---

Projection Optimization: A General Framework for Multi-Ohjective and Multi-Group RLHF
Hence, denote =0,, E, [V log T (y x) is the expectation of the score function, we can then derive that
3d1 (s(01, Tbase) s(0 +)))
Hence, the update rule can be efficiently computed as long as the score function is available, which commonly appears in previous RL algorithms such REINFORCE.
Thus, if the learning rate is
O, the gradient descent update rule of 01 is
0f = 0p-1 - (sd](s(01), Tbase) s(01, (")) v o0
Also, for the reward-free version, we can change the term (09-1)
log (s. (118
[lo
π (Yar x3) Wrof (1/m x))
log
x (y) x))
nn (yr x:)
(L)
E.3. Derivation the reward-free equation of expected reward vector
We now prove that
T x))
T (v x))
(V/) E Ire (x, (Ik [ret)] C -BB
log
31E youn
log
Trre (y/ x))
7 xx)
Proof. We note that
(I, BB log T'(1 xx)
Eg [r (I. v) -DDL [[ = Egg
Thet x))
7(r .z) + 3 log n T x))
T'(y xr)
= E,,
log
Tree (v xx)
Tret (y x))
E., (Z xx]] + SE [ log T x.)
7' x))
Now note that 2(r xx) is independent y. hence
x]] E [r o: ,
Errac [Z(raxz)
Er 1 (m. - B(og π o (y) x²)
log Feat x)))
log o (y) x))
C Blaynmaoun
Traf (y x)
42


Reviews:
[{"review_id": "YHdF0mwhVd", "reviewer": "Reviewer_Ekmw", "summary": "The paper proposes a general framework for multi-objective multi-group RLHF. The authors creatively draw inspiration from\nRL with Blackwell-approachability to resolve handle the non-linear structure of the aggregated reward.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive response and meaningful review! We will address your question as follows. \n\n>1. If negative values occur, the paper should discuss how p-norm aggregation applies.\n\nBoth in theory and practice, our algorithm can handle both the negative reward and the positive reward. In fact, by shifting the objective from maximizing the aggregated expected reward to minimizing the distance between the target set and the expected reward vector, we can effectively handle situations where the expected reward is negative (as the expected reward vector can be negative while the target set remains in the positive range). In this case, the goal is to make the estimated reward vector (which may be negative) closely align with the target set. For the theoretical part, note that the p-norm aggregation appears in the definition of the target set, and the expected reward vector $S(\\pi)$ can be negative. In practice,  MOPO performs well in experiments where the reward is negative, whereas truncating rewards to ensure positivity and directly aggregating them leads to failure (AR in our table).\n\nMoreover, the boundedness of the reward function is a common assumption and holds for most reward models.\n\n>2. The minimax formulation in Equations (6) and (7) is a central theoretical contribution. However, solving it might be computationally challenging; the paper would benefit from a more concrete explanation of how one can efficiently compute or approximate the solution in large-scale settings.\n\nThis is an important question. We have two approaches to make it practical.  The first way is to use a reward-free DPO algorithm, as in Eq.(8) in our paper.  We can further simplify the Eq.(8) by replace $\\log \\pi_\\theta(y\\mid x)$ to $\\alpha_i \\log \\pi_{\\theta_i}(y\\mid x)$, and Eq.(8) becomes decentralized optimization, where  $\\\\theta_i = \\arg\\min_{\\theta}\\beta \\alpha_i\\mathbb{E}\\_{\\pi_{base}} [\\log \\pi_{\\theta}(y\\mid x) - \\eta \\ell(D_i, \\theta_i)]$ for each objective $i$, and we can perform VPO [1] for estimating $\\theta_i$.\n\nSecond, one can ignore the first exploration term, and Eq.(6) and Eq.(7) become traditional MLE for estimating the reward model. Also, one can use a pre-existing reward model if it is available. In our practical MOPO algorithm, we use a pre-existing reward model and use PPO to get the optimal policy $\\pi_{\\theta_i}$ for each objective. Then, Line 2 and Line 3 in Algorithm 1 can be implemented by previous linear MORLHF algorithms like MOD and RS. We will discuss it in detail in our next version.\n\n>3. While the results on anthropic HH illustrate the method’s viability, this dataset is somewhat outdated. Testing on newer datasets, such as HelpSteer2, with higher-quality multi-objective labels and more categories would better demonstrate robustness and broad applicability.\n\nWe also perform experiment results on the Summarize-from-Feedback dataset [3]. We fine-tune a different LLM ALPACA-7B for objectives 'Summary' and 'Faithful' and get two policies $\\pi_1, \\pi_2$. Then, we perform experiments on these two policies. We use $\\alpha=(1/2,1/2)$ and $p=0.5$ for the aggregation and choose $c$ adaptively by $c=u(\\pi^t) + 0.01$ where $u(\\pi^t)$ is the expected utility for $\\pi^t$ at round $t$ (See Response for Reviewer MMQx for the selection of $c$ and Response for reviewer uhy5 for the definition of $u(\\pi)$).\n\n The following table shows the aggregated expected utility, which shows that our MOPO performs better than RS and MOD. Note that we do not select the HelpSteer dataset since the variance of its reward model is too small (see Appendix F.4 in [2]), making it less suitable for demonstrating the differences between linear and non-linear aggregation.\n\n| $\\alpha$  | MOPO  | RS    | MOD   |\n|-----------|-------|-------|-------|\n| (0.3,0.7) | 0.521 | 0.515 | 0.507 |\n| (0.5,0.5) | 0.522 | 0.517 | 0.502 |\n| (0.7,0.3) | 0.549 | 0.552 | 0.530 |\n\nThanks again for your time and effort! We will be happy to answer any further questions you may have.\n\n\n[1]. Cen et al. 2024: Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF.\n\n[2]. Shi et al. 2024: Decoding-Time Language Model Alignment with Multiple Objectives.\n\n[3]. Stiennon et al. 2024: Learning to summarize from human feedback.", "claims_and_evidence": "The authors provided theoretical and empirical guarantees for the proposed offline and online algorithms. Empirical evidence on anthropic HH supports these claims, with metrics showing performance gains over simpler baselines.", "methods_and_evaluation_criteria": "The proposed method is a good addition to the literature. I have some slight concerns regarding the theoretical and empirical analysis, discussed in the two sections below.", "theoretical_claims": "- The methods developed are built on the assumption that the reward is non-negative and bounded. However,  the Bradley–Terry–derived reward may not be strictly in [0,B]. If negative values occur, the paper should discuss how p-norm aggregation (and potentially other proposed methods) might be affected or require rescaling.\n- The minimax formulation in Equations (6) and (7) is a central theoretical contribution. However, solving it might be computationally challenging; the paper would benefit from a more concrete explanation of how one can efficiently compute or approximate the solution in large-scale settings.", "experimental_designs_or_analyses": "While the results on anthropic HH illustrate the method’s viability, this dataset is somewhat outdated. Testing on newer datasets, such as HelpSteer2 with higher-quality multi-objective labels and more categories, would better demonstrate robustness and broad applicability.", "relation_to_broader_scientific_literature": "The paper extends the literature of multi-objective and multi-group RLHF and provides novel algorithms.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Please see above", "other_comments_or_suggestions": "Please see above", "questions_for_authors": "Please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldw2nTYman", "reviewer": "Reviewer_MMQx", "summary": "This paper introduces a projection-based optimization framework for Multi-Objective Reinforcement Learning with Human Feedback (MORLHF). The approach reformulates non-linear reward aggregation as a series of linear sub-problems, enabling computationally efficient Pareto-optimal solutions. The framework is extended to multi-group settings, allowing for consensus or aggregated utility optimization across diverse preference groups.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive response and time in reviewing our paper! We will address your questions as follows.\n\n\n\n1. Analysis of hyperparameter impacts and selection.\n\nSince $\\alpha$ and $p$ is assumed to be given in the experiment, the only hyperparameter specifically included in our algorithm is the parameter $c$, which represents the requirement by human. \n\nIf $c$ is too small, the target set is easily achieved, causing MOPO to lose the motivation to change direction. (In this case, the induced LLM is enough for achieving the human's requirement.) On the other hand, if $c$ is too large, the target set becomes distant, and the projection direction will tend to balance $[1/m,1/m,\\cdots,1/m]$, which is also not conducive to maximizing the weighted p-norm expected utility. Hence, a value that is slightly larger than the expected utility could be suitable for $c$, for maximizing the expected utility function. \n\nIn practice, it can be selected in different ways:\n(a) Since it represents a requirement set by the human, it can be provided directly by the human. Note that in this case, if LLM already satisfied the requirement, it will have no motivation to further enhance the expected utility. \n(b) If the goal is to maximize the expected utility function $u(\\pi)=\\sum_{i=1}^m(\\mathbb{E}\\_{x\\sim \\rho, y\\sim \\pi}[r_i(x,y)])^p,$ $c$ can be adaptively chosen to be slightly larger than the current expected utility, such as $c = u(\\pi^t) + 0.01$ after getting the policy $\\pi^t$ at each round $t$. Then, the model will tend to maximize the expected utility by approaching the target set. \n\nIn experiments in our original paper, we manually choose the parameter $c$ that is slightly larger than the previously calculated expected utility. We also provide additional experiments using the adaptive approach (b) to validate our statements. See the additional experiment for details. \n\nFor other hyperparameters like $\\beta$ and $\\eta$, these are standard hyperparameters used in other works ($\\beta$ in [1], $\\eta$ in [4]). We can just use the standard configuration.\n\n>2. The lack of analytical experiments on multiple LLMs may limit the universality of the findings.\n\nWe conduct experiments on LLAMA2-7B, which is a commonly used model.  We use three objectives to evaluate our algorithm since they are classical metrics that are used in previous works [1,2,3]. Since the primary contribution of this paper is theoretical, we provide preliminary experiments on popular benchmarks. \n\n We also perform experiment results on the Summarize-from-Feedback dataset. We fine-tune a different LLM ALPACA-7B for objectives 'Summary' and 'Faithful' and get two policies $\\pi_1, \\pi_2$. Then, we perform experiments on these two policies. We use $\\alpha=(1/2,1/2)$ and $p=0.5$ for the aggregation, and $c=u(\\pi^t) + 0.01$ for round $t$. The results are shown in the following table, which shows that our MOPO performs better than RS and MOD, with adaptively selected parameter $c$.\n\n| $\\alpha$  | MOPO  | RS    | MOD   |\n|-----------|-------|-------|-------|\n| (0.3,0.7) | 0.521 | 0.515 | 0.507 |\n| (0.5,0.5) | 0.522 | 0.517 | 0.502 |\n| (0.7,0.3) | 0.549 | 0.552 | 0.530 |\n\n>3. Computational efficiency compared to non-linear aggregation baseline.\n\nThe nonlinear aggregation baseline in our paper is Aggregated Reward (AR), which means that we directly aggregate the reward to $r(x,y) = (\\sum_{i=1}^n \\alpha_i r_i^p(x,y))^{1/p}$, and training the reward directly using PPO. It cannot be directly applied to the negative reward setting because of p-norm aggregation. To address this, we transform the rewards into non-negative values by applying $r_i(x,y) = \\max\\\\{r_i(x,y),0\\\\}$.\n\nFor computational efficiency, AR needs retraining each time we change the aggregation method or the weights, which takes more than 7 hours in our experiment and 30 minutes evaluation times per different weight. In contrast, our training-free algorithm does not require retraining. Instead, we use pre-calculated optimal policies for each objective, which only requires less than 30 minutes for evaluation to get the expected reward for each weight.\n\n\nWe will release our code later in our revised version. Thanks again for your time and effort! If you have more questions, we are happy to solve them.\n\n[1]. Shi et al. 2024: Decoding-Time Language Model Alignment with Multiple Objectives.\n\n[2]. Zhou et al. 2024: Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization.\n\n[3]. Yang et al. 2024: Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.\n\n[4]. Cen et al. 2024: Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF.", "claims_and_evidence": "The authors argue that existing MORLHF methods, which rely on linear aggregation, fail to address fairness and multi-group preferences effectively. These claims are supported by theoretical sublinear regret guarantees for both offline and online settings and empirical results demonstrating improved fairness and competitive performance against baselines like RS, MOD, and max-min RLHF.", "methods_and_evaluation_criteria": "Yes, it makes sense.", "theoretical_claims": "This paper provides a relatively comprehensive proof for its theoretical statements.", "experimental_designs_or_analyses": "The experimental design in this paper is not comprehensive enough. \n1. The lack of analytical experiments on multiple LLMs may limit the universality of the findings, and the experimental verification of the framework for solving multi-objective problems is limited to only three objectives, which raises uncertainty about its potential extension or wider application. \n2. While the experiments validate the framework’s effectiveness, they lack an analysis of hyperparameter impacts. \n3. Although the authors emphasize computational efficiency, they fail to provide a direct cost comparison with state-of-the-art LLM models or nonlinear aggregation baselines.", "relation_to_broader_scientific_literature": "The paper situates itself within RLHF and multi-objective RL literature, and proposes multi-objective and multigroup RLHF problems under non-linear aggregation. It also connects to social choice theory and Blackwell approachability.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. Efficient solution to address Multi-Objective and Multi-Group RLHF Problems Under Non-Linear Aggregation.\n2. The theoretical derivation is comprehensive.\n\nWeaknesses:\n1. While the experimental results demonstrate strong performance, the scalability of the proposed method remains uncertain, as it was solely tested on one LLM and three objectives.\n2. Lack of guidance on choosing hyperparameters for practitioners.", "other_comments_or_suggestions": "Providing a code library would assist other researchers in validating and extending the work.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BFLNDgOXln", "reviewer": "Reviewer_3eRZ", "summary": "This paper introduces a novel Multi-Objective RLHF (MORLHF) framework that leverages per-objective preference feedback to achieve Pareto optimality by aggregating multiple objectives into a single unified optimization target. Unlike existing approaches that rely on linear aggregation, this work overcomes their limitations by reformulating the non-linear aggregation maximization problem into a series of sub-problems, each involving only linear aggregation. This decomposition enhances computational efficiency. The framework is further extended to handle multi-group settings and scenarios with unknown objective weights. Experimental results demonstrate a clear advantage of the proposed method over benchmark approaches.\n\n## update after rebuttal: Thanks for the rebuttal. I will keep my original rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive response and time in reviewing our paper! We will address your questions as follows.\n\n>1. The paper assumes that the type/group of each human is known in advance.\n\nIn this paper, we assume the group information is known. However, if the group information is unknown, we can use EM or other clustering algorithms to first get an estimation of the group indexes. This approach is also used in previous work [1]. \n\n>2. Does the paper assume that human provides pairwise feedback for all objectives? \n\nYes. We assume that humans can give feedback for all objectives. However, this is not a strong assumption. \nIn fact,  it is a common assumption that we have feedback or the reward model for all objectives in previous MORLHF works [2,3,4,5]. Also, in our online setting, we do not need the human to give all objectives for one response. Instead, we only need users to give pairwise feedback on the objectives they consider most important. Specifically, we ask them to first select the most important objective and provide a comparison between them. The selection of the objective is modeled by a softmax distribution based on the weighted reward gap. $\\alpha_i \\cdot (r_i(x,y_1)-r_i(x,y_2))$ (See Line 392 in our paper). \n\n>3. The assumption that objectives are explicitly available may not always hold in practice. \n\nThat's a great question! One possible way to address this challenge is to require a \"reason\" alongside the human's response. This reason could be a property or a short explanation, such as \"helpfulness\" or \"Response 2 is too long\". When the model encounters a reason that differs from the previous ones, it can identify this reason as a potentially new factor and add it as a new objective. This is an exciting and meaningful direction for future research, which we leave for future work.\n\n4. The assumption 5.3 seems too strong. \n\nIn fact, the distribution of $y_1$ and $y_2$ will not be close as training advances: $y_1\\sim \\pi^*(y\\mid x)$, which is the optimal policy, while $y_2\\sim \\pi_{ref}(y\\mid x)$ only follows the reference policy (typically the policy after SFT). Therefore, we simply assume that the performance gap between the optimal human-aligned policy and the reference policy remains constant, which is generally a reasonable assumption in practice.\n \n \n\nThanks again for your time and effort! If you have more questions, we are happy to solve them. \n\n\n\n\n[1]. Chakraborty et al. 2024: MaxMin-RLHF: Alignment with Diverse Human Preferences\n\n[2]. Shi et al. 2024: Decoding-Time Language Model Alignment with Multiple Objectives.\n\n[3]. Yang et al. 2024: Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment.\n\n[4]. Wang et al. 2024: Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts\n\n[5]. Mukherjee et al. 2024: Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization", "claims_and_evidence": "Yes. The claim is supported by evidence in theorem and experiments.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "See my questions.", "experimental_designs_or_analyses": "Yes, the experiments are good.", "relation_to_broader_scientific_literature": "The Multi-Objective RLHF is an interesting topic and has received increasing attentions recently.", "essential_references_not_discussed": "See my questions.", "other_strengths_and_weaknesses": "See my questions.", "other_comments_or_suggestions": "See my questions.", "questions_for_authors": "1. The multi-group problem considered in this paper is closely related to reinforcement learning from human feedback (RLHF) with heterogeneous feedback, where individuals may provide varying feedback based on their expertise or background. However, the paper assumes that the type/group of each human is known in advance and that the reward of each objective remains invariant across groups. In practice, this group information may be unknown. Lee et al. (2024) address this challenge by learning a latent representation of heterogeneous human feedback. Could the authors discuss potential ways to relax this assumption?  \n\nLee et al. (2024), *Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback*, https://arxiv.org/abs/2412.19436.  \n\n2. Given that there are $m$ different objectives and $N$ distinct human groups, does the framework assume that each human provides pairwise feedback for all $m$ objectives? If so, this may be a strong assumption. In practical online interaction settings, when asking a human to compare two responses, it may be impractical to request comparisons across all specified objectives.  \n\n3. Building on the previous question, the assumption that $m$ objectives are explicitly available may not always hold in practice. For example, when soliciting human feedback on two responses to a prompt, we do not necessarily know which specific objectives a human considers when making their judgment. How does the proposed framework address this challenge?  \n\n4. Assumption 5.3 requires a constant gap between the reward functions of two provided responses, which appears to be a strong assumption. In practice, this gap may decrease over time as the learning process progresses—i.e., as training advances, both responses may improve, leading to a diminishing difference in rewards. Would it be possible to model this gap as a decaying function of the learning horizon $T$ or other parameters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wypAqwOwF", "reviewer": "Reviewer_uhy5", "summary": "This paper primarily focuses on aligning large language models (LLMs) to multiple objectives using per-objective preference data. Prior works on this topic primarily aim to achieve Pareto optimal alignment across all objectives by linearly aggregating all objectives into a single unified form and optimizing the unified objective. In this work, they aim to propose an efficient strategy to optimize for the non-linear aggregation of the multiple objectives. Moreover, they consider a multi-group setting wherein each group has a different level of importances over the objectives. They propose to use the multi-objective p-norm aggregation framework from the social choice theory that leads to general function classes (non-linear) and also account for the group preferences in this aggregation framework.\n\nInstead of training a policy that optimizes for this aggregated objective, the authors propose an alternative approach: training a policy that minimizes the distance between its expected reward vector and a target set $W$. This target set consists of all vectors whose aggregated p-norm value exceeds a certain constant $c$. Moreover, they also consider the case where each group has a different target set $W$.  Then the policy optimization aims to minimize either (i) the distance between the expected reward vector incurred by the policy and the intersection of all target sets across groups (consensus setting) or (ii) minimize a weighted average of the distances between the expected reward vector incurred by the policy and the target set of each group (malfare setting). \n\nThe proposed algorithm consists of two key steps. First, it computes the projection direction between the expected reward vector and the target set. Then, using this direction vector as weights over the objectives, it optimizes a weighted loss using two approaches: reward-free multi-objective optimization and reward-based multi-objective optimization. The algorithm iteratively alternates between computing the direction vector and updating the policy. The authors also provide convergence guarantees for both the consensus and malfare settings.\n\nFinally, they propose an online variant of the algorithm, where responses are collected using the current policy, and preferences are gathered based on the objective with the greatest difference. The importance weights over objectives are estimated online using a likelihood-based approach. For experiments, they fine-tune the Llama2-7B model on Anthropic-HH data using the offline version of the proposed approach for $p=0.5$ and $p=-\\infty$ and compare the performance against relevant baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review! We are happy to address your questions as follows.\n\n\n> 1. The discussion of the expectation over prompt\n\nThanks for the good question! Our goal is to find the model $\\pi$ that maximize the total expected reward $u(\\pi)=\\sum_{i=1}^m(\\mathbb{E}\\_{x\\sim \\rho, y\\sim \\pi}[r_i(x,y)])^p, $ where the aggregation is applied to the **expected reward $u_i(\\pi) = \\mathbb{E}\\_{x\\sim \\rho, y\\sim \\pi}[r_i(x,y)]$**, rather than the reward for each prompt. In practice, the utility function (i.e., performance) of the language model is always measured over the entire prompt distribution, and our aggregation is applied to this overall utility.\n\nWe acknowledge that our introduction to multi-objective learning (namely, Eq.(1), (2) and the beginning of Section 3.1) may have been unclear and misrepresented our motivation, potentially leading to this misunderstanding. We can clarify it as follows: For a language model $\\pi$ (and single group), the utility for the model and the objective $i$ and the group $n$ is defined as $u_i(\\pi) = \\mathbb{E}\\_{x\\sim \\rho, y\\sim \\pi}[r_i(x,y)]$, and the model aims to maximize the aggregated social welfare $(\\sum_{i=1}^m \\alpha_i (u_i(\\pi))^p)^{1/p}$. To make this problem more tractable, we try to minimize the distance between target set and current expected reward vector $S(\\pi) = (u_1(\\pi), u_2(\\pi), \\cdots, u_m(\\pi))$ (ignore the regularizer for clarity), which is a reasonable alternative. (See Example 3.1 and 3.2 in our paper).\n\n>2. The experimental evaluation and discussion\n\nWe conduct our experiments on the Anthropic-HH dataset using the objectives Helpful, Harmless, and Humor, which are popular choices in previous MORLHF papers. We provide comprehensive results across different weights and objectives in both single-group and multi-group settings. Additionally, we include further experiments in the additional experiment part. We also provide an analysis for hyperparameter selection (the selection of $c$) in response to Reviewer MMQx.\n\nThe experimental results demonstrate that our approach achieves a smaller distance between the target set and the expected reward vector, and also **achieves balance between diverse objectives**. For example, with the weight (0.7,0.3), objectives 'helpful' and 'harmless', and the aggregation parameter $p=0.5$, MOPO get the utility $u_{MOPO}=(0.123,0.412)$, while $u_{RS}=(1.203,-0.707), u_{MOD}=(1.295, -0.8)$. This shows that MOPO not only achieves a smaller distance to the target set but also better balances multiple objectives, aligning with our theoretical motivation of striking a balance between linear aggregation and max-min RLHF.\n\nSince the offline RLHF is more common, practical, and convenient (we can access offline dataset and reward-model easily), we only perform our method in offline RLHF. We believe that the online MOPO can also achieve better expected utility and better performance compared to linear aggregation, and we leave it as future work. \n\nThe AR baseline means that we directly aggregate the reward to $r(x,y) = (\\sum_{i=1}^n \\alpha_i r_i^p(x,y))^{1/p}$, and train the reward directly using PPO. Since it requires aggregating the individual reward, it cannot be applied to the negative reward setting, while our algorithm can also work for the negative reward. In the experiment, we perform $r_i(x,y) = \\max\\{r_i(x,y),0\\}$ to change the reward into non-negative.\n\n>3. The paper is overly dense in many places.\n\nThanks for the reminder. We will improve our writing and proof to make it easier to follow in our final version. \n\n>4. Additional experiments.\n\nWe also perform experiment results on the Summarize-from-Feedback dataset. We fine-tune ALPACA-7B for objectives Summary and Faithful and get two policies $\\pi_1, \\pi_2$. Then, we perform experiments on these two policies. We use $\\alpha=(1/2,1/2)$ and $p=0.5$ for the aggregation, choose $c$ adaptively by $c=u(\\pi^t) + 0.01$ where $u(\\pi^t)$ is the expected utility for $\\pi^t$ at round $t$ (See Response for Reviewer MMQx for the selection of $c$). The results are shown in the following table, which shows that our MOPO performs better than RS and MOD. \n\n| $\\alpha$  | MOPO  | RS    | MOD   |\n|-----------|-------|-------|-------|\n| (0.3,0.7) | 0.521 | 0.515 | 0.507 |\n| (0.5,0.5) | 0.522 | 0.517 | 0.502 |\n| (0.7,0.3) | 0.549 | 0.552 | 0.530 |\n\n>5. Typos and definitions\nThanks for the reminder. Eq. (3) should be $\\theta_i$ instead of $\\theta$, and $\\Pi_W(V)$ means the projection of the point $V$ on the set $W$, which can be written as $\\Pi_W(V) = \\arg \\min_{x \\in W} \\|\\|V-x\\|\\|_2^2$. \n\n>6. Missing references\n\nThank you for providing additional relevant references! We will include a detailed discussion in our next version.\n\nThanks again for your time and effort! We will be happy to answer any further questions you may have.", "claims_and_evidence": "The paper primarily builds upon the target set approach from [Yu et al., 2021] which aims to minimize the distance between the expected reward vector $ S(\\pi) $ incurred by the policy and the set of acceptable reward vector values whose aggregation exceeds a predefined constant. However, [Yu et al., 2021] was designed for a reinforcement learning (RL) setting where the initial state is fixed (see Page 4 - Interaction Protocol), resulting in a single $ S(\\pi) $ that can be directly compared with the target set. \n\nIn contrast, in this RLHF setting, optimization is typically performed over a distribution of prompts, where the objective is to maximize the expected reward across prompts. This key aspect is not addressed in the paper. Furthermore, according to the objective defined in Section 3.1, where the expectation over prompts is taken outside the aggregation, the aggregated reward for each prompt should either satisfy the target set constraint or be close to it. However, the feasibility of this approach and whether it is actually implemented in the experiments remain unaddressed. \n\nTo the best of my knowledge, this presents a fundamental issue that requires clarification from the authors.\n\n\n## update after rebuttal\nMy concerns regarding having multiple algorithms are addressed. Essentially, there are only 2, as the authors point out, and only one of them (the offline version) has been evaluated. I still believe the paper can be restructured quite a bit to improve the clarity. Kindly recommend the authors include additional experiments and more experimental discussions in the paper. Also, improve the introduction such that $ S(\\pi) $ is well defined, as pointed out.", "methods_and_evaluation_criteria": "The experimental evaluation is quite limited. Only one of the proposed algorithms has been evaluated, with minimal discussion of the results. Furthermore, the online version of the algorithm lacks any experimental evaluation.", "theoretical_claims": "In the theoretical proofs, $ S(\\pi) $ is treated as the expectation over both prompts and policy, with the bounds based on the distance between the expected reward vector over all prompts and the target set. However, optimizing the expected aggregated reward, as detailed in Section 3.1, is not equivalent to ensuring that the aggregation of the expected reward vector satisfies the constraints (or is close to the target set of vectors that satisfies them). This is equivalent to interchanging expectation and aggregation and requires proper comments from the authors discussing the validity of such a maneuver.", "experimental_designs_or_analyses": "The experimental evaluation is quite limited. Only one of the proposed algorithms has been evaluated, with minimal discussion of the results. Furthermore, the online version of the algorithm lacks any experimental evaluation, and details regarding the AR baseline are not discussed.", "relation_to_broader_scientific_literature": "This paper primarily focuses on aligning large language models (LLMs) to multiple objectives using per-objective preference data. Prior work in this area mainly seeks to achieve Pareto-optimal alignment by linearly aggregating all objectives into a single unified form and optimizing the resulting objective. In contrast, this work proposes an efficient strategy to optimize for the non-linear aggregation of multiple objectives. \n\nMoreover, the authors consider a multi-group setting where each group assigns different levels of importance to the objectives. They adopt the multi-objective $ p $-norm aggregation framework from social choice theory, which generalizes to non-linear function classes and incorporates group preferences into the aggregation process. \n\nTo the best of my knowledge, the proposed methodology of minimizing the reward vector's distance to the target set represents a novel framework in this area of research.", "essential_references_not_discussed": "The following works related to group preference and multi-objective alignment are missing in related works:\n\n1. Zhao, Siyan, John Dang, and Aditya Grover. \"Group preference optimization: Few-shot alignment of large language models.\" arXiv preprint arXiv:2310.11523 (2023).\n\n2. Ramesh, Shyam Sundhar, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. \"Group robust preference optimization in reward-free rlhf.\" Advances in Neural Information Processing Systems 37 (2024): 37100-37137.\n\n3. Guo, Yiju, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen et al. \"Controllable preference optimization: Toward controllable multi-objective alignment.\" arXiv preprint arXiv:2402.19085 (2024).\n\n4. Mukherjee, Subhojyoti, Anusha Lalitha, Sailik Sengupta, Aniket Deshmukh, and Branislav Kveton. \"Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization.\" arXiv preprint arXiv:2412.05469 (2024).", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper focuses on aligning large language models (LLMs) to multiple objectives using per-objective preference data. It proposes an efficient strategy for optimizing the non-linear aggregation of multiple objectives. Moreover, it considers a multi-group setting where each group assigns different levels of importance to the objectives. To the best of my knowledge, the proposed methodology of minimizing the reward vector's distance to the target set represents a novel framework in this area of research.  \n\n2. Multiple algorithms have been proposed for both offline and online settings, with theoretical analysis establishing their convergence guarantees.\n\nWeaknesses:\n\n1. See comments above regarding the theoretical claims and experimental evaluation/design.\n\n2. The paper is overly dense and difficult to follow in many places. It introduces numerous algorithms and concepts, but not all have been experimentally validated.", "other_comments_or_suggestions": "Should it be $\\theta_i$ instead of $\\theta$ in (3)?\n\nWhat is $\\Pi_{W}(V)$?", "questions_for_authors": "Kindly comment on the issues in the theoretical claims detailed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes MOPO, a general framework for aligning large language models (LLMs) with human feedback across multiple objectives and user groups. Unlike most existing works that use linear aggregation of objectives, MOPO enables efficient optimization with non-linear aggregation through projection-based reformulations. The framework supports both offline and online RLHF and includes theoretical convergence guarantees. Empirical results on standard datasets and multiple model/objective setups demonstrate that MOPO improves over baseline approaches. 

The paper tackles an increasingly relevant problem in LLM alignment—how to balance multiple competing objectives (e.g., helpfulness, harmlessness, humor) and diverse user preferences—using a principled and computationally efficient method. The projection-based approach enables support for non-linear utility aggregation and offers provable guarantees. The authors propose both offline and online variants and demonstrate the framework’s flexibility across single- and multi-group settings. Reviewers consistently highlighted the paper's strong theoretical depth and well-motivated methodology. The rebuttal further clarified confusion around prompt-level expectations and validated robustness via new experiments, including on the Summarize-from-Feedback dataset. Some concerns were raised about assumptions around known group identities, reward structures, and feedback availability—though these were addressed reasonably well in the rebuttal. Further refinement of the writing and presentation could improve accessibility.

Based on the common consensus, the paper is recommended for acceptance once the authors incorporate all the reviewer's feedback in the final version.
Decision: Accept (poster)