Group Robust Preference Optimization in Reward-free RLHF
Authors: Shyam Sundhar Ramesh
Yifan Hu
Iason Chaimalas
Viraj Mehta
Pier Giuseppe Sessa
Haitham Bou Ammar
Ilija Bogunovic
Venue: NeurIPS 2024 poster (2024)
Keywords: RLHF
DPO
Robust Alignment
Primary area: generative_models

Abstract: Adapting large language models (LLMs) for specific tasks usually involves fine-tuning through reinforcement learning with human feedback (RLHF) on preference data. While these data often come from diverse labelers' groups (e.g., different demographics, ethnicities, company teams, etc.), traditional RLHF approaches adopt a "one-size-fits-all" approach, i.e., they indiscriminately assume and optimize a single preference model, thus not being robust to unique characteristics and needs of the various groups. To address this limitation, we propose a novel Group Robust Preference Optimization (GRPO) method to align LLMs to individual groups' preferences robustly. Our approach builds upon reward-free direct preference optimization methods, but unlike previous approaches, it seeks a robust policy which maximizes the worst-case group performance.  To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We theoretically study the feasibility of GRPO and analyze its convergence for the log-linear policy class. By fine-tuning LLMs with GRPO using diverse group-based global opinion data, we significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies compared to non-robust baselines.

Group Robust Preference Optimization in Reward-free RLHF
Shyam Sundhar Ramesh' University College London (UCL)
Yifan Hu ETH Zurich, EPFL
lason Chaimalas
University College London (UCL)
Viraj Mehta
Pier Giuseppe Sessa
TensorZero
ETH Zurich
Haitham Bou Ammar
Iiija Bogunovic
University College London (UCL) Huawei Noah's Ark Lab
University College London (UCL)
Abstract
Adapting large language models (LLMs) for specific tasks usually involves fine- tuming through reinforcement learning with human feedback (RLHF) preference data. While these data often come from diverse labelers' groups (c.g., different demographies. ethnicities, company teams, etc.), traditional RLHF approaches adopt "one-size-fits-ei approach, i.e., they indiscriminately assume and optimize single preference model, thus not being robust unique characteristics and needs of the various groups. To address this limitation, we propose a novel Group Robust Preference Optimization (GRPO) method to align LLMs individual groups' preferences robustly. Our approach builds upon reward-free direct preference opti- mization methods, but unlike previous approaches, seeks robust policy which maximizes the worst-case group performance. To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We theoretically study the feasibility of GRPO and analyze its convergence for the log-linear policy class. By fine-tuning LLMs with GRPO using diverse group-based global opinion data, we significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies compared to non-robust baselines.
Introduction
As the usage of large language models (LLMs) has grown in recent years, the question of their alignment has come to the forefront. Their remarkable capability to address wide range of tasks (Radford et al. (36)) stems from pre-training on self-supervised objective over internet-scale text. This vast internet-scale content, however, carries higher risk of biases, inaccuracies, and controversial content than smaller, curated datasets. Thus, ensuring that the model's responses and behaviors correspond to human intentions and values crucial.
Typical approaches to alignment [ 34 37 involve gathering preference feedback from human labelers train models that reflect their desires. Such approaches often treat individual preferences as samples from broader preference distribution. However, this perspective often oversimplifies the complex reality that human societies consist of numerous distinct groups (e.g., different demographics, ethnicities, company teams, etc.), each with their own set of preferences that can significantly
Corresponding author: shyam. ramesh. 220ucl.ac...
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Alignment performence
Pre-troined
Supervised Tuming (5FT)
Pref. optimization DPO IPO
LLM
61
Promot: Should social media platforms enforce stricter
Group Prefeeencc
Adoptive group weights (loss influenced)
regulations?
62
Rt Yea, aecil molia slatfoms shoul enforee etricte regulrtione consat anrformction
Gradient updattes scaled by group weights
G1
62
R2: he, Hhyy should not enfance stricter naguitisee becsuse it rake intringng on free speech
R1
R2
R2
Alignment performence
Pre-trained
Supervised Tuning (SFT)
GRPO (group-rebust)
LLM
61
62
Figure I: Current reward-free preference optimization methods typically optimize based on average human feedback. This often aligns predominantly with the preferences of the majority group (GI, R1 > R2) the expense of minority groups (G2, R2 > R1). contrast, our GRPO algorithm introduces adaptive weighting for different user groups and prioritizes optimizing for the worst-case group performance, leading to better alignment for the most disadvantaged groups
diverge. Consequently, prevalent alignment strategies tend to adopt disproportionately favor the preferences of the majority group, often at the expense of minority groups and their preferences, as illustrated in Figure] I
model and
To improve alignment performance for even the most disadvantaged groups, we propose to robustly solve the problem of diverse group preferences by (i) including group information the context of the LLM and (ii) optimizing against the worst-case alignment performance across all groups. We develop policies that guarantee equitable performance across all groups, ensuring that no group is disproportionately disadvantaged due to inherent biases or imbalances in the training data.
Related work. The established process for alignment of LLMs using Reinforcement Learning from
Human Feedback (RLHF) is set out in 45 60 and (34). The RLHF fine-tuning process consists of learning a reward model from human comparisons between responses to a given prompt, using the Bradley-Terry model [5]. Then, one performs policy optimization using Proximal Policy Optimization (40) to learn policy that maximizes the learned reward function. For a comprehensive overview and perspective of the RLHF topic, we refer the reader to [24
23
Due the challenges of tuning PPO and the vulnerability of reward models ((50 17 49 ). alterna-
tive approaches to PPO-based RLHF have been proposed. including rejection sampling fine-tuning [14]
18 49 30) and conditional supervised fine-tuning [20] (4.9) 9). In particular, Rafailov et al. (37) intro- duce Direct Preference Optimization (DPO), which optimizes policies directly based on human prefer- ences, avoiding the need for separate reward model. This approach simplifies training and reduces re- ward overfitting. Other studies, such as [1 58 47 43 16], propose novel reward-free RLHF methods, with some bypassing preference datasets altogether (1666)) We utilize reward- free framework simi- lar to [7 however, unlike previous works that assume single preference distribution, we consider multiple preference distributions from diverse groups. Further, we aim to robustly fine-tune the LLM to ensure minimal disparity in performance across all groups. Other studies addressing robustness in preference optimization include (2I) and [26). However, these works primarily focus on different aspects of robustness, such as robustness to noise and resilience against out-of preffreceee data.
optimize performance of language models over a wide-range of topics. They consider robust training of language models based on the group Distributionally Robust Optimization (DRO) approach. concrete theoretical study of the group DRO approach was performed in [39] and applied to vision problems. These are designed by extending previous minimax algorithm for solving DRO from (31). In the RLHF setup, (3] consider weighting of loss from different topics (harmless vs helpful) for robust reward learning. Also, Chakraborty et al. (8) consider robust policy optimization by learning multiple reward functions corresponding sub-populations and learning robust policy w.r.t. the learned
Robust language modeling techniques have been studied by 33 52

---

rewards. Differing from these works, we embed group robustness directly into the reward-free tuning paradigm. We provide an concrete algorithm that adaptively weighs the loss for different groups and optimizes for policy that minimizes the weighted loss. Further, our algorithm employs a novel gradient estimator tailored to the group robust DPO problem.
In the non-robust setup, [57] also explore group- preference learning with LLMs by including group information in prompts with a transformer module that is trained optimally to choose an example sequence of prompts, LLM responses, and group preferences for in-context learning. [49 consider alignment with user preferences assuming that each user has varied importance over the distinct metrics in their multi-objective reward model. In contrast, we are not modeling multi- objectives but consider a reward-free setting. And, our methodology directly models each group's s preferences through group-dependent latent reward model where the group dependency is injected through the prompt. Further, unlike the non-robust problem setups of (57, 49), we consider the robust alignment problem optimizing for the worst group performance. We detail other related works extensively in Appendix A
Main Contributions. The following are the main contributions of this work: (i) We present GRPO, the group robust formulation of Direct Preference Optimization (DPO) [37], wherein augment the context of the LLM with the group information, and pose the problem as robust optimization problem to minimize the worst-case loss amongst the diverse groups. We also show that a naive application of group robustness to the LLM policy maximization objective does not offer robustness benefits. To the best of our knowledge, this is the first study that focuses on group robustmess in RLHF preference optimization; (ii) We analyze the theoretical aspects of GRPO by examining the convergence and the feasibility of finding optimal solutions within the log-linear policy class. (iii) We present a tailored algorithm to tackle this robust optimization challenge, providing convergence guarantees for certain loss functions; (iv) We show the versatility of our approach by demonstrating how our algorithm can be utilized with other reward-free preference optimization methods such as Identity Preference Optimization (IPO) [T. In particular, for the GR-IPO objective optimized over the log-linear policy class, we derive closed-form weighted regression update for the policy parameters rather than a gradient update. To the best of our knowledge, this is a novel contribution towards efficient fine-tuning through preferential data; (v) Our empirical evaluations across synthetic datasets, real-world data, and publicly available LLMs show that the proposed GRPO significantly improves performance for the worst- worst-performin performing groups, reduces loss imbalances across groups, and increases probability accuracies compared to non-robust baselines.
Background
We address the challenge of fine tuning a large language model (LLM) to align with user preferences. This process usually follows the Reinforcement Learning from Human Feedback (RLHF) protocol, using either an explicit reward model (3) 34 45 60)) or an implicit reward model (37)) RLHF typically comprises three key phases: (i) supervised fine-tuning of an initial (pre-trained) large- language model, (ii) reward learning. and (iii) RL fine-tuning.
In the supervised fine-tuning phase (SFT), the goal quality dataset suited for the downstream task of interest. It results in probabilistic model expressing the probability of the response y given a prompt x as T (y) )). Subsequently, in the reward learning phase, the goal is to learn a reward model from a dataset of prompts r and responses Vie Y1, with y X yw x meaning that human labellers preferred Vou over yr. It is typically assumed that preferences follow some choice models with an unknown reward (utility) r" (a, y) function. A popular model the Bradley-Terry model that assumes the preference distribution p admits the following form
fine-tune a pre-trained LLM on a specific high-
exp(r" (I, yz))
p(y1- < y2 x) exp(r" v1)) + exp(r* y2))
Based on the above model, a maximum likelihood estimate of the reward function is obtained as:
min(Cn(r: == -- -....... [log (a (r(z, Vww r(z, 31))))
(2)
where (.)) is the sigmoid function and D represents the dataset consisting of {(x,, Yur J1)
Then, in the RL fine-tuning phase the objective to train a policy и that maximizes the learned
reward function. Simultancouslyy, the policy should stay closely aligned with the reference, Tefs as
3

---

quantified by the KL divergence, leading to the following KL-regularized optimization problem:
max BewPa [Ry~a (r(x.)) - SKT, (+
Direct Preference Optimization (DPO). The recent approach proposed by [37 exploits the closed-form solution of the problem in Equation (3) and sidesteps the explicit modelling of rewards to directly optimize the policy. Specifically, under the Tradley-Terry preference model, the reward function can be expressed directly terms of the optimal policy " as follows:
T x.) x)
r(z, v) = B log
+ B Blog Z(x).
[y
for a partition function ((x) My Trer (r) exp( (z v)). Via change of variable, finding the
optimal reward function in Equation 2 is equivalent to finding the optimal policy n' utilizing the given set of preference data D. With a slight abuse notation, we use и to denote क². Denote h (z. Vve 81) == log( (y ix) Ner( (2) log( nt (x)
Then, Equation 2 translates into the DPO loss:
Tnt( 2))
LDPO(T, D) --
E MM)
[log (a h (ar, Uw: (o))))))
With a parameterized policy model Ta, minimizing the DPO loss involves calculating the gradient over 0 using backpropagation and the log- uag-pobbabilitess of each completion, You and given the prompt for both the policy my and the reference policy Tee--
Group Robust Preference Optimization
In this section, we discuss Group Robust Preference Optimization (GRPO), i.e., instead of learning reward function that maximizes the likelihood, we aim to derive (implicitly) a robust reward function and subsequently learn robust DPO policy.
Group Preferences. Suppose that preferences come from an underlying latent reward r" (2x, y: g). with g E G {1,2, }} indexing the groups. When group information is available (e.g., text), we can represent the reward as r" y). where Vig x @ denotes merging with group information (e.g.. string concatenation). We continue apply Bradley-Terre model as described in Equation @ substituting x with Tg. Moreover, we assume access to a collective dataset
TI of the prompt
D = - y=1 Dg where D,- ((=g) Yiu .2 ) with the available group information. Additionally, our dataset accommodates the exposure of different groups identical prompts, meaning that the same x can appear across various groups in our dataset, and these groups may favor different responses
(f)
Given such D, although one may obtain common reward model using Equation (2), it could result in poor generalization for particular groups, especially with significant group-wing disparities in the data (see Figure TD. Such disparities might stem from imbalanced data across groups difficulties associated with learning different groups
GRPO Objective. Consequently, we propose to measure the alignment of the reward model on the worst-case group loss:
Ln(r; 9EG
(6)
Incorporating the reward expression from (Equation 44) into (Equation (6)) establish the group robust preference optimization (GRPO) objective for a specified policy क:
LGR (T) = max (DPO(T, D,) = max g6g
ELeg, log (Bl Uor m()
Leveraging the equivalent formulation of maximizing over discrete set, the GRPO problem becomes
K
min (cn(T) =
min max a
ag
E5
log (0(3 (Zg, Uwu, 31) )
(8)
9=1
'The group information added the prompt can represent, e.g., index textual description of the group

---

where AK-1 represents the (K 1) -idmennsio simplex of probabilities The inner maximization becomes a linear programming over simplex such that a represents the weights of groups. addition, it forms two-player zero-sum game (see Section (.T), where the policy and a act as opponents with inversely related payoffs. The DPO loss (logistic log loss) in Equation (7) can be replaced with alternatives like hinge or squared loss (see (47)) We label this objective GR-DPO when using DPO
loss, and explore GRPO with squared loss in Section 4.1
Applications. In this study, we do not assume any specific distribution for groups D,. The col- lection of prompts per group. Pz,. may have varying degrees of overlap. The GRPO framework accommodates both distinct and overlapping prompt scenarios across different groups.
Apart from human groups, GRPO can be useful in scenarios where groups D3 represent distinct tasks or topics within preference datasets, like helpful/harmful, truthfuu/unannwerable instances, or domain- specific categories (e.g., math, physics, chemistry). Typically, these prompt distributions (Pz, J.V are disjoint, and GRPO seeks to optimize performance even across the most challenging categories.
GRPO is also applicable in scenarios where groups reflect diverse user preferences for shared set of prompts, with the goal achieving equitable performance across user groups. This contrasts with non-robust DPO, which aims to optimize preferences on average and might overlook minority groups.
Lastly, we acknowledge that the max-min objective of Equation (8) might be overly conservative, potentially degrading average performance. We explore more balanced approach between worst-case and standard preference optimization objective in Appendix [.44
3.1 Further Discussion and Insights
This section provides two insights regarding the GR-DPO loss in Equation
Log-linear policy class. The zero-sum game perspective allows us to explore the presence of a Nash equilibrium, serving as benchmark for convergence during the policy optimization process. Given that the domain of a is simplex Ax (in Equation (8) we further define parameterized policy
class Hl for the policy wy. We assume that the parameterized policy my of the form To x)) =
exp fa()))) Evey exp fo(a.))) where fe is linear function or neural network, and A belongs to a convex set θ.
In LLM fine-tuning, sometimes practitioners concentrate on modifying solely the final layer. It corresponds to linear function, fo(x, y) 0f o(x, y). with Φ(x,1) .y) denoting the embedding derived from the language model removing its last layer, and 0 as the parameters of the last layer. When applying this linear parameterization, in conjunction with uniform reference policy Ted the robust objective outlined Equation (8)
as follows (details in Appendix B.II:
K
Σ 0g (- ( wn) log (3((6((z, yo)) b(x 91), ()) 0))
min max BEB aEAK-1
The objective defined in Equation 99 concave with respect to a and convex with respect This structure allows the invocation of the minimax theorem for convex-concan functions (44)) to assert the existence of a Nash equilibrium.
Proposition 3.1. Under lineai parameterization of the policy class, there exists a Nash equilib- rium for the group robust direct preference optimization problem in Equation (9).
Robust policy optimization. The earlier derivation for the GR-DPO objective LGR-DPO (T) relies on incorporating robustness in the reward modeling step (in Equation (7h) while using the solution to the non-robust Kl.-regularized reward maxinization objective in Equation
Interestingly, we can obtain the identical expression for CeR-DrO(1 if incorporating robustness in the KL-regularized reward maximization objective and using the reward function learnt a non-robust way. Consider the robust KL-regularized reward maximization
( v) 3KT [KT.[T(y) xy)) mar(( =n)]].
max min Rag ww-. gEg
(10)
The following proposition characterizes such an invariant property.
From a distributionallly-obur viewpoint, this is equivalent to defining the uncertainty set as |K og Dgyyy E Ax) ). and minimizing the worst-case expected loss across the uncertainty
5

---

Algorithm 1 Mirror Descent for Group Robust Preference Optimization (GRPO)
1: Initialize: Step size Te for group weights a, step size n/ for policy т with weights 0, initial weights ((0) of the policy and weights over each group (0)) Projection operator Pe
2: Input: Dataset D with size N = (D), group size Ng for g 2 ... K), loss (
3: for = ..,;;.? do
4: a' t -a(=i))
5: g~ Categorical(N ..,..... ,KKNN), (Ig, ywr 11) ~ D3
6: a'g + d' cap
Nil(TT ((e ) // Update weights for group
No
7:
1 t a' Egg a'z'
I/ Renormalize
8: 000) + Pe [
Na VallF oi ). // Use a to update 0
N,
9: end for
10: Return: Output the robust policy ((((2))
Proposition 3.2. Substituting the closed-form solution of the robust KL-regularized policy maximization problem (Equation (10p) into the robust reward maximization objective in Equation ( leads to the same group robust DPO loss LGR-DPO in Equation (8)
The analysis leverages the fact that the optimal policy of Equation is identical to the solution of
the non-robust KL-regularized reward maximization in Equation and is derived in Appendix B.2
4 Algorithm
In this section, we discuss the policy optimization algorithm for solving the group robust DPO problem in Equation (8). In particular, we aim to design an algorithm that performs updates in the parameter- ized space C Rdd i.e. updating of the parameterized policy TT. Leveraging the perspective of the 2-player zero-sum game, we propose an alternating updating algorithm wherein one updates and 0 alternatively. We summarize the overall approach in Algorithm D
which we discuss and analyze next.
We employ the DPO loss ((TT:-) = log (6h (-))) (Equation 5) in Algorithm | however, our algorithm can support other preference optimization losses (see Section 4.1 The algorithm performs a gradient descent type update on 0 and a deterministic mirror ascent on using Bregman divergence with the distance generating function as the KI. divergence. Since the a lies in a simplex and the objective is linear, the update of becomes multiplicative weights update with renormalization to a simplex via softmax (see Nemirovski al. (32) for details). Further, the weights a are determined by the cumulative losses ( accrued by each group, ensuring that groups with higher cumulative losses get higher weights. The size of the group Ng appears as the empirical distribution D, involves Ng. We call alternating update as the updated a² is used in the update from 6t- to 6 In particular, the gradient descent type update on 0 is weighted by a in order to orient the update towards groups with higher losses. The projection operator Pe ensures the updated 6 lies within e.
What does the weighted DPO update do? In Line 9 in Algorithm [ the algorithm performs parame- ter updates based on the weighted gradients. By using the DPO loss, i.e., 1( .) = log (Bhh. (.) (see Equation
we obtain the following gradient update expression ignoring the N/N, constant
a ! V (Eg, Bw, 3t)) a Vg log ( (o(Bher (Igs Vwr v.l)
(11)
- (x 31) - roa-u1 (Igz ynm) x (Vo log Tou- (Vmr - Vo log T&c n (wl(²
The final term plays the critical role of enhancing the likelihood of the preferred response while simultaneously diminishing the likelihood of the rejected response. This adjustment is proportional to the disparity in rewards between the two responses. Moreover, the inclusion of ag is pivotal for ensuring group robustness. This coefficient scales the gradient w.r.t. 0 based on the cumulative loss previously received by all samples within specific group. Such a mechanism ensures that the model's sss focus is increasingly directed towards groups that have historically suffered higher losses. Additionally, the scaling factor N, guarantees that groups with smaller volume of data do not face disadvantage. We defer further details in obtaining the gradient update expression to Appendix B.3

---

We demonstrate the global convergence with the following proposition.
Proposition 4.1. Suppose that the loss I(: (T y, v.y') non-negative, convex, By -ipisccchtzz con- timuous, and bounded by Bı for all (Zg. v.Y') E X @ x x y and [|0]|2 S Be for all 0 E O with convex θ RRd The error of the average iterate of Algorithm i.e., TgarT + + 0', satisfies
E|CGH(TTar))
min (T) (TT-1/2).
We defer the proof of this proposition to Appendix E The analysis follows from an adaptation of the analysis in Nemirovski et al. (32) for the proposed sampling strategy in Algorithm We note that when fine-tuning only the final layer of a LLM, the output policy exists within the log-linear policy class (see Section 3.T. and the corresponding loss function satisfies the assumptions in Proposition
4.1
(see Lemma
4.1 Group Robust Identity Preference Optimization
The standard regularized reward maximization objective (Equation (3) in DPO 37 tends to overlook the KL-regularization and learn deterministic policies. This learned policy assigns preference probability one to winning responses the data which is often not realistic (see [[][Scction 4.2] and Appendix C). Recently, Azar et al. [T] show that the standard regularized reward maximization objective (Equation 3 in DPO (37] tends to overlook the KL-regularization and learn deterministic policies (see [T] Section 4.2] and Appendix C They thus propose an alternative approach called Identity Preference Optimization (IPO) that is more likely learn randomized policy which assigns appropriate probability to the preferred response and prevents overfitting. Following similar derivation as we did for group robust DPO with details given in Appendix C corresponding group robust IPO (GR-IPO):
we develop the
min LGR (T) max LIPO (T,, Dg) 9E9
max aEAK-1
Σ 0gg g=1
(F Vuu m1
2/3
For the log-linear policy class (introduced in Section
3.I.. the objective function simplifies to
Σ ag (E(s,..o...))),
[ (10( Mw) - G(Eg, v1),
23 )
-/2
min max
BEB acAx-1
To solve the GR-IPO above, suffices to use Algorithm [ with slight modifications, see Algorithm in Appendix C In particular, the update of 0 is replaced by weighted regression update:
+ arg min
Σ
(lo(zg) yno) oz v1), 8) N
ag
2
23
OE
For fixed a, we show (in Appendix C that such an update admits a closed-form solution:
= 23 5' with W = Diag
Ogu Ogg(N)
N Nga: N Nggts
where g0 the group of each sample i, Ngoo is the number of samples in group g(ó) and 1 is column vector of ones of dimension N. Here S is matrix S (o(=)) (1) (1) -(x)()))) represents the difference in feature mappings & of the preferred and less preferred response for each prompt. The group robust IPO (GR-IPO) algorithm is presented in Appendix C and its empirical
(1) . ((.))).....(((
yyu o(x
(N) (*)))
Each row
S
results are shown in Section 5
Experiments
In this section, we study the empirical performance of our proposed Algorithm 1 on synthetic and real-world dataset First, we simulate multi-group data disparities by varying the size and
We study different sampling strategies and their error rates in Appendix and present the most numerically stable one Algorithm
'Codes for synthetic and real world experiments can be found at aulppp//gtthlloommmsssya//PPaPad and http://///gthub..oo////ha/a////e
respectively.
7

---

DPO Max Validation Group Loss
IPO Max Valiciation Group Loss
Converged Max Reward Error
0.6
Valuo
DPO
Value
20
IPO
Valne
IS-IPO GR-IPO
GR-DFO
0.05
0.5
18
0
2500
Iterations
5000
7000
10000
25
Iterations
100
0.00
Methods
Figure 2: Synthetic experiments: Algorithm 1 GRR-DPO and GR-IPO) leads to a significantly lower worst-case validation loss and reward error compared to importance sampling (IS-DPO/IPO) and vanilla methods (DPO, IPO). Results refer to the scenario in which groups have different sizes and responses' distribution.
preference distributions of two synthetic groups. In the real-world setup, we study the alignment of an LLM to the real preferences of people from various countries. We examine whether GRPO aligns the LLM in a more equitable manner to reduce discrepancies alignment among various groups. Finally, demonstrate that performance is improved by explicitly addressing the grouped nature of the data during alignment.
5.1 Synthetic Experiments
We evaluate the performance of Algorithm using synthetically generated group preference data for the loss function ((TT;.)) - either DPO loss or IPO loss and denote them as GR-DPO and GR-IPO, respectively. We compare them against vanilla DPO and IPO (37 D), and the importance-sampling (IS) variants of DPO and IPO (where the loss of each datapoint is inversely weighted by its group data size).
Experimental Setup. Our experiments are designed to analyze settings where there exist multiple groups with distinct characteristics. We adapt the standard (non-group based) experimental setup proposed by [26] for the group preferences setting by incorporating group information into the reward function r X' x y x G + R. Here, X represents a two-dimensional state space [0, 1] x [0, 1]. y denotes a discrete action space {0, 1, 2, ........... and & signifies a discrete group space 1,, .... KJ. The reward function, defined by the group-dependent feature vector o(2, y, g) and parameter vector fg, is given as r(x, y. := (Ø(x, y. 1)),g)) while the feature vectors 6(I y. have a pardinate-flipped relationship and are defined in Appendix D.I
We consider the following scenarios: (i) Groups are imbalanced in terms of size but have the same distribution over responses, (ii) Groups are balanced in terms of size but have different response distributions, and (iii) Groups are imbalanced in terms of size and also have different response distributions. Note that having different response distributions leads to difference in the difficulty of learning, since groups with responses distant from each other (in terms of rewards or preference probabilities) are typically more distinguishable and easier to learn. We discuss in Appendix D.2 how we generate these three scenarios.
Implementation. Leveraging the linearity in the reward model, we utilize a log-linear policy class exp (#(2.g.9).9
parameterized by 0: To(y)<>) Ey exp (ø(a.y' 99) We run Algorithm for both DPO and IPO loss relative to the policy class detailed above with a dataset of 300 action pairs with preferences.
Evaluation Metrics. We use the following criteria to assess the performance of the algorithms:
Max Validation Loss. For each group g. with preference data denoted as (a* .y.. y/ Ny where N₂, the number of data points in the group, we compute the DPO/IPO validation loss separately for each group and identify the maximum loss among them in each run.
Max Reward Error. This metric compares the true reward of the optimal action determined by 8 with that of the action deemed optimal by estimate Å for each group, and identifies the maximum error across all groups in each run. In particular, for data in the form (a* g) N2
which includes only states and groups, we calculate reward errors for every group as follows: E (x.g)~(af fmax, (0(x, x. v. g), 03) (0(x, arg max, (Ó(x, v y, v.g), 0). ()).)). 03)]-
Results. We present the average performance of Algorithm | (error bars over 20 seeds) alongside baseline methods in Figure 2 for scenario (iii), while scenarios (i) and (ii) are Figures 4 and

---

in Appendix D.2 to space constraints. Our findings indicate that the robust methods consistently surpass both vanilla and importance-sampling approaches. Notably, the robust methods demonstrate significant superiority in uneven group scenarios, where the importance-sampling technique falls short as it exclusively deals with data imbalance.
5.2
Global Opinion Experiments
For the real-data experiments, we consider the survey dataset GlobalOpinionQA (15) and the publicly available Ge model [48). The data contains multiple choice questions answered by participants from various countries, amounting 2,554 questions covering various topics, including politics, media, technology, religion, race, and ethnicity. For each question, the dataset provides a prob- ability vector over the choices, signifying the percentage of people from particular country choosing each option. Note that this probability vector would be different for different countries, Hence, the goal is to align the LLM to the probability vector corresponding to each country in robust manner.
We consider the following five countries in the dataset: Nigeria, Egypt, India, China and Japan, with data sizes 572, 570, 376, 309, and 712, respectively. We construct our training set as follows: For the SFT training, we choose the best option (the choice with the highest probability) as the target. For both IPO and GR-IPO training, we consider the best option the winning response and another randomly chosen option as the losing response. We outline the exact prompt we use in Appendix D
We run the SFT training for one epoch over the training data on the pre-trained GB model. For both IPO/GR-IPO training we use the AdamW (27 optimizer with adaptive learning rates. For SFT/IPO/GR-IPG training. we apply the LoRA strategy to fine-tune all layers of the model. We then evaluate both the methods based on the worst group loss and accuracy. Here, the loss refers to the IPO loss for each group and the accuracy refers to the percentage of winning response and losing response pairs correctly ordered by the learned preference function (Equation 35 ). We defer further training and hyperparameter details to Appendix (
Results. We present the average performance of GR-IPO over five seeds alongside IPO Figure (top plots). Our findings indicate that GR-IPO outperforms IPO in terms of maximum group loss and minimum group reward accuracies. Moreover, GR-IPO effectively reduces the imbalance in loss values among different groups. Additionally, we observe an improvement in log- tog-probability accuracies (which measure if the probability assigned by the fine-tuned model is higher for the winning response compared to the losing response) for both IPO and GR-IPO, with GR-IPO demonstrating better alignment for the worst-performing group compared to IPO.
Insights. We further note that the worst-performing groups are Groups-2,5, as shown in Figure 3 GR-IPO improves the loss for these groups by assigning more weight to them, as illustrated in (botton middle plot). Additionally, we plot the initial log- log-probability accuracies for different groups in Figure 3 (bottom right plot), assessing how accurately the SFT model classifies the winning versus losing response for different groups. It is evident that Groups-2,5 are already underperforming. Given that SFT training converges within one epoch without any discrepancies between groups, this indicates that the base LLM inherently struggles with classifying responses for Groups-2,5. However, by employing GR-IPO, we have mitigated the imbalance in the performance of the fine-tuned model.
6 Conclusions
We formalize the problem of robustly aligning an LLM to preference distributions from diverse groups. To tackle the same, we introduced GRPO, a group robust formulation of reward-fre free RLHF, aiming to minimize worst-case loss among groups. We explored the theoretical aspects of GRPO and demon- strated its improved robust alignment performance through various experiments. We believe our ap- proach will be highly valuable for future tailored LLM fine-tuning, specifically aimed at aligning with the needs of diverse teams and user groups. In broader context, it holds promise for mitigating biases and discrepancies across various societal groups encountered in the task-specifit adaptation of LLMs.
When the dataset is balanced among groups and difficulty levels are comparable, our GRPO approach does not offer a significant advantage over standard reward-free RL.FF algorithms. In addition, minimax methods often improve the worst group's ' performance at the cost of reducing the average or best group's performance. Our proposed GRPO formulation (see Equation (8).
Limitations.
'The model and dataset are available on Hugging Face at hutp://hngiingiicccooogg//mm/mmm and http://wwgggiggcccccoddduttttttt global opinions respectively.

---

Max Group Loss (Tist)
Min Reward Accuracios (Test)
Group Loss (Test)
2000
2000
0.75
Value
Value
Value
1800
1600
GR
1600
0.70
GR-IPO
IPO
GR-IPO
IPQ
1000
Grap1
Goep
Graga Groups
Dongy
Methods
Methods
Min Log-Prob Accuracios (Tert)
Group Weights (Train)
Log-Prob Accuracios (Start)
0.50
0.70
Value
Value
GR (10
GR-IPO
0.7
0.3
2000
3003
2000
2:00
30000
1eotp
Drosp
Dyeepp Groupe
Gropp
Gropp
Herations
Iterations
Top plots: GR-IPO leads to better worst-case final test loss and reward accuracy compared to IPO. Moreover. it leads to more balanced losses best and worst-group loss (Group-1 1 vs. Group-S). Bottom plots: Log-prob, accuracy (left plot) and group weights (middle plot) during GR-IPO training. GR-IPO increases the weight on worse-performing groups (Groups-2,5) and decreases it on high-performing ones (Groups- 1,3,4) leading better worst-case accuracy. Groups-2.5 are the ones with worse log pro.p accuracy at the beginning of training (right plot with random subset of the training data). We show corresponding aud-of-training log- log-prob. accuracies for GR-IPO in Figure 13 Appendix
Figure 3: Global opinion data:
the different groups, reducing the gap between
stemming from minimax framework, will comply with the same property. Hence, in scenarios where optimizing worst-case performance is less critical, we define a trade-off parameter to balance between the worst-case performance and the average performance. This modified objective and the necessary algorithmic changes are elaborated in Appendix [B.4] The appropriate tuning of the trade-off parameter for the specific application remains subject for future investigation. Further, we focus on settings with known groups, which are common in pluralistic alignment datasets and tasks (see (57 44)). When groups are unknown, one can still undertake several approaches such as clustering, representation learning, feature analysis, expert consultations, etc., to help uncover group structures in the data.
Acknowledgments
PGS was gratefully supported by ELSA (European Lighthouse Secure and Safe AI) funded by the European Union under grant agreement No. 101070617. YH was supported as a part of NCCR Automation, National Centre of Competence (or Excellence) in Research, funded by the Swiss National Science Foundation (grant number 51 51NF40_225155) IB was supported by the EPSRC New Investigator Award EP/X039 17X/1; the Engineering and Physical Sciences Research Council CP/S021566/1 and Google Research Scholar award. SSR was supported by Department of Electronic and Electrical Engineering and the Institute of Communications and Connected Systems UCL. The authors would like to thank William Bankes, Seongho Son, Matthieu Zimmer, Afroditi Papadaki, Eduardo Pignatelli, and Nagham Osman for the useful discussion.
References
[1] Mohammad Gheshlaghi Azar et al. "A general theoretical paradigm understand learning from human preferences". In: International Conference on Artificial Intelligence and Statistics.
2024.
[2] Yuntao Bai al. "Constitutional ai: Harmlessness from ai feedback". In: arXiv preprint arXiv:22/2.0 N.Nv:22/2.08003.. 2022.
[3] Yuntao Bai al. "Training a helpful and harmless assistant with reinforcement learning from human feedback". " In: arXiv preprint arXiv:2204.0524 0.08622.......... 2022.
[4] William Bankes et al. "REDUCR: Robust Data Downsampling Using Class Priority Reweight- ing". In: Neural Information Processing Systems. 2024.
10

---

[5] Ralph Allan Bradley and Milton E Terry. "Rank analysis of incomplete block designs: I. The method of paired comparisons". In: Biometrika. JSTOR, 1952.
[6] Tianchi Cai et al. "ULMA: Unified Language Model Alignment with Demonstration and Point-wise Human Preference". In: arXiv preprint arXiv:2312.0233
2023.
[7] Stephen Casper et al. "Open problems and fundamental limitations of reinforcement learning from human feedback". In: Transactions on Machine Learning Research. 2023.
[8] Souradip Chakraborty et al. "MaxMin-RAee RLHFF: Alignment with Diverse Human Preferences".
In: International Conference on Machine Learning. 2024.
[9] Lili Chen et al. "Decision transformer: Reinforcement learning via sequence modeling". In: Neural Information Processing Systems. 2021.
[10] Zixiang Chen al. "SSef--lay fine-tuning converts weak language models to strong language
models". In: International Conference on Machine Learning. 2024.
[11] Paul F Christiano et al. "Deep reinforcement learning from human preferences". In: Neural Information Processing Systems. 2017.
[12] Josef Dai al. "Safe rlhf: Safe reinforcement learning from human feedback". In: International Conference on Learning Representations. 2024.
[13] Nirjhar Das et al. "Provably Sample Efficient RLHF via Active Preference Optimization". In: arXiv preprint arXiv:2402. 10500. 2024.
[14] Hanze Dong et al. "Raft: Reward ranked finetuning for generative foundation model alignment".
In: Transactions on Machine Learning Research. 2023.
[15] Esin Durmus al. "Towards measuring the representation of subjective global opinions in language models". In: Conference on Language Modeling. 2024.
[16] Kawin Ethayarajh et al. "Kto: Model alignment prospect theoretic optimization", In: Inter-
national Conference on Machine Learning. 2024.
[17] Leo Gao, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization".
In: International Conference on Machine Learning. 2023.
[18] Caglar Gulcehre et al. "Reinforced self-training (rest) for language modeling". In: arXiv preprint arXiv: 230.8.8898. 2023.
[19] Jiwoo Hong. Noah Lee, and James Thome "Reference- frree monolithic preference optimization with odds ratio". " In: arXiv preprint arXiv:2403.0766 2024.
[20] Jian Hu et al. "Aligning language models with offline reinforcement learning from human feedback". In: arXiv preprint arXiv:2308. 12050. 2023.
[21] Shawn Im and Yixuan Li. "Understanding the Learning Dynamics of Alignment with Human Feedback".. In: International Conference on Machine Learning. 2024.
[22] Kaixuan Ji, Jiafan He, and Quanquan Gu. "Reinforcement Learning from Human Feedback with Active Queries". In: arXiv preprint arXiv:2402.09401 2024.
[23] Timo Kaufmann et al. "A survey of reinforcement learning from human feedback", In: arXiv
preprint arXiv:2312. 14925.. 2023.
[24] Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. "Entangled preferences: The history and risks of reinforcement learning and human feedback". In: arXiv preprint arXiv:2310.1350 2023.
[25] Harrison Lee et al. "Rlaif: Scaling reinforcement learning from human feedback with ai feedback". In: arXiv preprini urXiv:2309.0026; 2023.
[26] Ziniu Li, Tian Xu, and Yang Yu. "Policy Optimization in RLHF: The Impact of Out-of- preference Data". In: arXiv preprint arXiv:2312. 10584. 2023.
[27] Ilya Loshchilov and Frank Hutter. "Decoupled weight decay regularization". In: International Conference on Learning Representations. 2019.
[28] Viraj Mehta et al. "Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration", In: 2023.
[29] Rémi Munos et al. "Nash learning from human feedback". In: International Conference on Machine Learning. 2024.
[30] Reichiro Nakano et al. "Webgpt: Browser-assisted question-answering with human feedback".
In: arXiv preprint arXiv:2 12.0332. 2021.
[31] Hongseok Namkoong and John Duchi. "Stochastic gradient methods for distributionally robust optimization with f-divergences", In: Neural Information Processing Systems. 2016.
11

---

[32] Arkadi Nemirovski et al. "Robust stochastic approximation approach to stochastic program- ming". In: Society for Industrial and Applied Mathematics. 2009.
[33] Yonatan Oren et al. "Distributionally robust language modeling". In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
[34] Long Ouyang et al. "Training language models to follow instructions with human feedback".
In: Neural Information Processing Systems. 2022.
[35] Jing-Cheng Pang al. "Language model self-improvement by reinforcement learning contem-
plation". In: International Conference on Learning Representations. 2023.
[36] Alec Radford et al. "Language models are unsupervised multitask learners.. In: OpenAl blog. Vol. 1.8. 8. 2019, p. 9.
[37] Rafael Rafailov et al. "Direct preference optimization: Your language model is secretly
reward model". In: Neural Information Processing Systems. 2024.
[38] Corby Rosset et al. "Direct nash optimization: Teaching language models to self-improve with general preferences". In: arXiv preprint arXiv:2404. 00..0037155. 2024.
[39] Shiori Sagawa al. "Distributionally robust neural networks for group shifts: On the im- portance of regularization for worst-case generalization". In: International Conference on Learning Represenlations. 2019.
[40] John Schulman et al. "Proximal policy optimization algorithms". In: arXiv preprini arXiv: 1e01177777.677..... 2017.
[41] Avi Singh et al. "Beyond human data: Scaling self-training for problem-solving with language models". In: Transactions on Machine Learning Research. 2023.
[42] Maurice Sion. "On general minimax theorems." " In: 1958.
[43] Feifan Song et al. "Preference ranking optimization for human alignment". In: AAAI Confer- ence on Artificial Intelligence. 2024.
[44] Taylor Sorensen et al. "A roadmap to pluralistic alignment". In: International Conference on
Machine Learning. 2024.
[45] Nisan Stiennon et al. "Learning to summarize with human feedback". In: Neural Information Processing Systems. 2020.
[46] Gokul Swamy et al. "A minimaximalist approach to reinforcement learning from human feedback". In: International Conference on Machine Learning. 2024.
[47] Yunhao Tang et al. "Generalized Preference Optimization: A Unified Approach to Offline Alignment". In: International Conference on Machine Learning. 2024.
[48] Gemma Team al. "Gemma: Open models based gemini research and technology". In:
arXiv preprint arXiv:2403.0829) 2024.
[49] Haoxiang Wang et al. " "Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards". In: Association for Computational Linguistics. 2024.
[50] Tianhao Wu et al. "Pairwise proximal policy optimization: Harnessing relative feedback for Ilm alignment". In: arXiv preprint trXiv:2310.002//4 2023. [51] Yue Wu et al. "Self-Play Preference Optimization for Language Model Alignment". In: arXiv preprint arXiv:2405. 0066500075.... 2024.
[52] Sang Michael Xie et al. "DoReMi: Optimizing Data Mixtures Speeds Up Language Model
Pretraining". In: Neural Information Processing Systems. 2023.
[53] Wei Xiong al. "Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint". In: International Conference on Machine Learning. 2023.
[54] Rui Yang et al. Hewward--in-Context Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment". In: International Conference on Machine Learning. 2024.
[55] Chenlu Ye et al. "A theoretical analysis of nash learning from human feedback under general kl-regularized preference". In: arXiv preprint arXiv:2402.077 220.24222.7314.. 2024.
[56] Weizhe Yuan et al. "Self-rewarding language models". In: International Conference on Ma- chine Learning. 2024.
[57] Siyan Zhao, John Dang, and Aditya Grover. "Group Preference Optimization: Few-Shot Align-
ment of Large Language Models". In: International Conference on Learning Representations.
2024.
12

---

[58] Yao Zhao et al. "Slic-hf: Sequence likelihood calibration with human feedback". In: arXiv preprint arXiv:2305. 10425. 2023.
[59]
Han Zhong et al. "DPO Meets PPO: Reinforced Token Optimization for RLHF". In: arXiv preprint arXiv:2404. 18922. 2024.
[60] Daniel M Ziegler et al. "Fine-tuning language models from human preferences". In: arXiv preprint 1rXiv:199.008593 2019.
13

---

Appendix
The supplementary section constitutes the following: An extended related work section in Appendix A proofs for the theoretical insights and the weighted DPO gradient expression in Appendix B an overview of IPO and the simplification for robust IPO in Appendix [ detailed experimental setup in Appendix D and convergence proof for Algorithm and other sampling strategies in Appendix B
A Additional Related Work
We detail a more extensive related work exposition in this section.
The Reinforcement Learning from Human Feedback (RLHF) is a concrete way to adapt a LLM towards specific tasks or objectives using human feedback. This framework for fine-tuning L.MMs inspired from [11) that uses human feedback to improve reinforcement learning policies. For LLMs, it was first established in (60) and further developed in [45] 334. It involves learning reward function us-
ing the pairwise human feedback data that provides comparisons between responses for given prompt. This reward learning is performed through the Bradley-Tere Terry model (5). This learned reward model is then used as an objective for optimizing the LLM policy using Proximal Policy Optimization [40]
However, obtaining high-quality human feedback data expensive and several works such as [35] 2
225 study the usage of Al feedback to reduce reliance on human input. Further, studies like [13] 28
22] analyze the usage of active learning strategies to minimize the amount of human feedback data
required to perform RLHF For a comprehensive overview and perspective of the RLHF topic, with
detailed discussions on various approaches and gaps, we refer the reader to 24 23
37) and susceptible nature of reward models ( (17) [4)), alternative approaches to the PPO-based RLHF have been proposed.
Due to the inherent issues of tuning the hyperparameters of a PPO (0
Foremost of them is the rejection sampling fine-tuning ([14) 18 49 which is inspired by the best-of-n inference technique (30)). The technique involves sampling n responses per prompt and fine-tuning the model based on the highest scoring ones in terms of a learned reward function. Further works such as 220 54) avoid reward learning and propose conditional Supervised Fine- Fine-Tuning (SFT) inspired by the reward-coomm -owaid-condiiooned RL (9))
alternative to PPO-based RLPHF is DPO (Rafailov et al. (37). wherein one optimizes the policy directly based on human preferences, bypassing the need for learning separate reward model. This method simplifies the training process and potentially reduces overfitting and model misalignment issues. Theoretical advancements in understanding the dynamics of DPO have been provided by [2] (3). who analyzed the convergence and stability of DPO policy learning. Some other works such as 58 47 43 16] also study a similar reward-free RLHF
Another strategy proposed
setup. In particular, (T) analyze the potential overfiting issues in DPO. Further, [16] 6 propose
strategies that bypass the need for preference datasets. [38 55 29 adopt game-theory perspective to learn Nash equilibrium policies. Typically, the reward-free RLHF approaches utilize a reference
policy trained via supervised fine-tuning, which is then further optimized. However, [19] propose Monolithic Preference Optimization without reference model, bypassing supervised tine-tuning entirely. [59] address the RLHF problem from an MDP framework, introducing a DPO-type -PO-type algorithm integrated with Proximal Policy Optimization (PPO). Additionally, works such as [5T
46 10 41 (6) investigate self-play preference optimization, where new data generated in each round
by the current policy is used to train an improved policy. Our work utilizes reward-free framework similar to [37] but differs from all the above works in the following sense. Unlike previous works that assume a single preference distribution, we consider fine-tuning a LLM given data from multiple preference distributions across diverse groups. Further, we aim to fine-tune the LLM in robust manner such that there is minimal imbalance in performance across all groups.
Robust language modeling technique in the context of language model was first studied in 33 to optimize performance over wide-range of topics. It was further improved by [52) wherein they use smaller model to learn the weights/importance that needs to be assigned to different topics and train a larger model based on these weights. Both works aim to minimize the worst group loss based on the group-DRO approach and focus on the pre-training aspects of language models. Concrete theoretical study of this group-DRe approach was performed (39) which studies the minimax formulation of this group-DRO problem based on 31] In the context of online batch selection, 4 propose a robust
14

---

data downsampling method that performs class-aware aware priority reweighting to reduce training costs while preserving worst-class performance.
In the RL.F setup, recent research has increasingly focused on enhancing the robustness of policy learning to address certain inherent gaps in the RL.F framework as detailed in (7) such reward hacking and model misspecification. In particular, (3) formulated weighted group loss for reward model learning w.r.t. a parameter À that determines the importance assigned to a group and apply it to the anthropic Harmless vs Helpful data O In a similar manner, [1] design a weighted objective for policy optimization w.r.t. parameter
trade-off the importance between task reward maximization and safety cost minimzation. Another work (8) consider alignment to different sub-populations by learning multiple reward functions corresponding to each sub- populaatuliinnn and learning robust policy (max-min) w.r.t. the reward functions. Distinct from previous works, our approach circumvents reward model learning and integrates group robustness directly into the reward-free tuning framework. This allows us to directly learn a robust policy without the need for learning reward models. To learn this robust policy, we present concrete algorithm that adaptively assigns weights to the losses of different groups and optimizes the policy minimize this weighted loss. Additionally, our algorithm features a novel gradient estimator specifically designed for the group robust DPO problem.
Other studies addressing robustness in preference optimization, such as [2 and (26), focus on different facets of robustness, including robustness to noise and resilience against -rref---reefeeece data. In the non non-robust group preference alignment setup. [57] explores learning group preferences with LLMs by incorporating group information into prompts. However, their approach does not involve fine-tuning LLM but training separate transformer module that optimally selects an example sequence of prompts, L.M responses, and group preferences for in-context learning using a LLM. Also, Wang et al. [49] consider alignment with user preferences assuming that each user/group has varied importance over the distinct metries in their multi-objective reward model. The output policy is trained to output response based both the prompt and the importance/weights over the individual metrics. In contrast, the distinctive feature of our method that we are not modeling multi-reward objectives but consider a reward-free setting. Specifically, we consider the robust alignment problem optimizing for the worst group performance. And, our methodology directly models each group's 's preferences through group-dependent latent reward model where the group dependency is injected through the prompt.
B Theoretical Insights
In this section, we detail the proofs for the theoretical insights elucidated in Section 3 and the weighted DPO gradient expression discussed in Section 4
B.1 Robust Objective for the Log-linear Policy Class
Here, we describe obtaining the robust objective for the log-linear policy class as in Equation P Starting from the robust objective for general policy class in Equation (8). LGR can be specialized as follows:
K
(CR(T)=
max
Σ
ag
E ( Mn Dr log (o(3h (zgg yous 2n)]]
acAx-1
g=1
K
(
Σ
Nw. log (o(3log) log =()w(1g)) merl (ea))
,3 log( *<( (xng)
merl (ve ( )))
max
0gg
eεAK-1
g=1
(
Beey (10g(1(8loog(( (0(3log((((mnlz))) - llg((T(u( (x)))))))))))
max
ag
aεAK
9
TThe data is publicly avaiable in Huggingface website http:///whgggiiihhooe//aw/////at
15

---

K
(i7)
max aEAx-1
0gg
Blxg (xg. Men
E
log ( (0(3log( log( Evey exp So (1)
folo,,
9=1 exp folmggs11 Eney exp fo (alorg.0)
3 log(
(1))
log (o (0 6 o(Eg.
max wEAK
ag
B
((
g=1
3 log(exp 0J o( w)))
Σ 0g (- B (r,,,,,
log (0(3(p(zg) Hwm Bw) Φx vi). 8)) 0))
max aEAK
Here, (i) follows from the definition of b (Zgg yhu 21) = log( m[y=l [zn)
) loog( =( ( ). (ii) follows from the fact that for uniform reference policy (ye (gg) = Te (xrg). ) (iii) follows from sub- exp fo( and (iv) follows from substituting fo y) = 0 o(zg.)). v).
Ter (Zml(1
stituting ()(()r))) Evey exp to M.)
B.2 Robust KL- KL-Regularized Policy Maximization
In this section, we consider the robust version of the classical KL-regularized reward maximization objective (Equation (3) detailed in Equation (10) for a reward function r(zg, v), reference policy Tet. and general class of policies
max min E,,- P 2/ (r(gg - [KKL [T( [gg1 xgg] WEll gEG
(12)
Lemma B.1. The optimal policy x² for the robust KL-regularized reward maximization objective (Equation 12 for reward function r(æg: y), reference policy Toff is
1
(r(#gi))) 11
Tt" (y(o"))) =
[ng(U) exp
(13)
Z(x.g)
B
where Z(xg) Ey tno(y(=e)) exp( (F n)) a partition function.
Proof. We recast Equation 112) based on Equation (8) and perform the following analysis similar to 37 [Appendix A.1]:
K
max min
Σ
ag
E y)ur()1zz))
[r(zg, v)] - ( [ [
(14)
Tell aEAR-1
xy
g=1
max min nell dEAx-1
Σ
ag
E
E
[r(zg.)) 4) - 3 log [ )1)
(15)
xy Pay
goot
9=1
K
=- =-3 min max aell
Σ
0g
E
E
[og (a(()lzg)))
r(ar =lsgg:2))
(16)
Ppp (-2)()1))))
= = min min
max
Σ
0gg
B PamPas Pay y)
E
--lo(Z((5,)) + log
و exp( ووgسيسح
)))
2(ngi ملوا
g=1
(17)
16

---

Further, by defining x ()(())))
1
Trer exp( ) we can rewrite Equation (17) as:
[[aa]
K
6 min max TEll OEAK-1
Σ
ag
E
E
log ((())))))) 10g((((2)))
(18)
By Pag yourl (11ag)
g=1
K
=8 min max tell aEAx-1
ag
( log( (00))))))))
(19)
By
PPx
8=1
= min (T) tell
(20)
where we use
rOL(T):= = max
Σ
Og
E Pa
( - log log (x(r))
oE
It is not hard to show that roL(T) is minimized by n² Indeed, let us consider any other policy
* n.. Then, we have
(TL( =B max
Σ 0g oE
DKL ( (+xxg) -1g((())))
(21)
Dy
Pxg
K agy
| log( ((x()))))))
2 8 max aEAx
E
(22)
P=g
g=1
K
3 max dEAx-1
0gg
E Pg
(1a))))))))) - -log()(o)))))
(23)
g=1
= roo((T)),
(24)
where the inequality follows since DKL (T(11=)[(*((1[1) Equation 20
2 0. This implies π minimises
Note that, we have proved that the optimal policy expression for Equation 12 aligns with the form of the optimal policy for standard (non-robust) KL-regularized reward maximization Equation (
Proposition 3.2. Substituting the closed-form solution of the robust KL-regularized policy maximization problem (Equation (TOp) into the robust reward maximization objective in Equation
leads the group robust DPO loss LGR-DPO in Equation (8)
Proof. In 377. the DPO loss (see Equation
5 is obtained by using the following reward loss:
CR(r Cx(r; (D
[log (ar (r(z,, Vm) r(x 31)))).
(25)
and replacing r(..)) with the relation
T" <y([x)== Z(a) (U)() exp (Ilaw))
(26)
which is the solution to
(r( v) BDDL (
max REll
F (1g Pey (())))))
(27)
However, in our group group-robust formulation, we replace Equation 25 with
K
max Ln(r; (Dy) = max Σ
log (a((( Bw) - r((gg 11) mn))
0kg
E Mw.
gEG
OE
g=1
(28)
17

---

while Equation version, i.e.,
27 remains the same. Alternatively, we can replace Equation 27 with its robust
max min well acAK-1
ag Σ g=1
( 12,)) ro(0z) w)] RD
T ( a (
(29)
however, we have already shown, in Lemma B.1 that this does not change the obtained policy-reward relation from Equation (26). Hence, this proves Proposition 3.2
B.3 Analysis of the Gradient
We elucidate the steps to obtain the expression for the loss gradient in Equation the gradient as follows:
11
We can simplify
a' V Vol(TTgr-1; (Zg. Vio U1))
n2
q V log (8 (Igg Blu W))
ng
( 0g (Blempt (Fg Wwr m)
x [8th (Igr thus 3t)]
ng a (Bh (Tgg
you 21)
(0)
Ba 1 0(31rxge-1 (E Ne 2i)) x 11 m (Egg Bwus U1)
Thg
3ag o( Blompe (Iggr yo w)) x Wii (Irgr Vo :w::)))
(38)
ng
Bag o
(m))
σ (3( log(
Mai Moe-1 (wela)) - log(
Tyd (Maell)) ( )) Uhinge (Figs Mers 313]
Teal(
Rgg
0
(rgr-1 (Fgg 11) r1 (Tig )w))
V log Tgp (Y (zn) Vo log TTg- (y)
fg
Here, follows from the derivative of log(-) function and denoting the derivative of sigmoid function
(.)) as (-). (ii) and (iii) follow from the fact that for any 8 E IR, (3) = (ss) = ((-s).
Finally, (iv) and (v) follow using the definition of h (I, Mwr Mi) = log( x(pulz) log( _ (z)
and substituting ra(#g. v) B log( No(u|zo) ;e(()l.))
,3 log (xrg)) from Equation 44
B.4 Trading off worst-case vs. average performance
In the robust objective of Equation (8) we intend to minimize the worst-case loss. This often might adversely impact the average loss leading to bad average performance across the different groups. To mitigate this, we propose the following robust trade-off direct preference optimization objective for specified policy and set of group weights Hi....,,KK
K
(TR := - x)
Σ
Mg (T + x max (TPO(TT Dg)
gm
Mg (
E, -. log (a (Figs Mw U1)) )[)
=(1 x)
g=1
(30)
E M. log (a( Bw ui)
X max DEG
Here, note that the input weights My can be equal for all groups leading to a trade-off between the average and worst-case performance w.r.t. parameter X. Following Equation (8). this can be
18

---

equivalently recast into:
CGRX( -11- xx) Σ
B B(ep log (a(slix (Fgg Mour mi)) Nw: mi)])
Mg
gm
(31)
K
D( log (olith Me u1 m))
max
0g
o
9=1
K
Σ 9=1
((1 - X14 + xag) (x.. log (I Yw wi)l)
max aEAR-1
(32)
where AK-1 represents the (K -)-dimensional simplex of probabilities. Hence, minimizing the loss in Equation (3T) implies that one can implicitly find an optimal reward function using human group preference data that is robust without losing average performance, and obtain policy that works effectively terms of both average and worst-case performance:
min (T).
(33)
As Equation (8 involves maximizing with respect to a and minimizing with respect æ, it forms two-player game similar to the one considered in Section 3.1, where the policy and act as opponents with inversely related payoffs. However, the contributions of the maximizing player (a) on the final outcome are limited depending on parameters and j.
C IPO Simplification
Within the context of finite data setting, it is common to record two responses (y, y') for each prompt x. Based on these observations, there might be a tendency to inaccurately deduce that the preference distribution fulfills p(y > v/(x)
Consequently, and assuming the Bradley-Tery model of preferences, the reward function has to satisfy rw,, -r(y'22 .x) + Co. Further, given that
DPO policy π is directly dependent on the reward functioon (Equation (4)), it holds that <(y|=)
=(v(aa) 0.
In such a case, the policy overfits irrespective of the KI. regularization factor. To circumvent this, consider reward-fred frre RLHF problem. Specifically, they propose preference optimization in the policy objective instead of reward optimization as follows:
max E
(P(( > s')l=)] - 3KL ( (/]] (=m<(()))))))
(34)
Here, V : [0 + R is non-decreasing function, o refers to the distribution of prompts and, μ refers to the competing behavior policy. Choosing v as the identity function, the optimal policy T of Equation (34) has a similar expression Equation (4) with the reward function r(.) substituted by the following preference function p(.),
π (y x)
ply > m|z)= = 3 log m
+ 3 log Z(x).
(35)
x))
[x)) from Section it follows from Equation (35) that x* adheres to the following identity that equates the true preference with the policy's preference of y over y':
Recalling the definition of h y, v') = log( m(ylar) - log( * <(y'lz) [x)
me((vl))
h (z, y. v. = --1 u|x) ply' > plz)).
(36)
where p④y > μ|z) := Ex-nt (s) [ply > g/z)]] The equation motivates us to find a policy æ that minimizes the squared differences of two sides in Equation (36).
((T) = Ey.r'' ~ hr (z. v. v²) (ply m/z) p(y' > mlxii)
(37)
For an empirical dataset where one observes if you is more preferable to yr the policy minimization objective in Equation (37) becomes the IPO objective:
(T, D)
E
(1, Dwr vn) 23
(38)
(2) DrD
19

---

We can simplify Equation (38 for the IPO Loss using Log-linear policy class similar to Equation 9
as follows
LT, D)
E (3:
[(10(22, Me) 6xx,01),0)
1
)
(39)
23
Contrary to DPO, the IPO Loss results in problem formulation equivalent to linear regression for the linear bandit setting. This leads us obtain a closed form analytical solution for
1 (x²x) 28
Iro
(40)
where 1 {1)8. Here, X E R In particular, each row of X is the difference of the preferred and least preferred feature vectors 60x yw) ((z, y1) for (æ,, yw y1) E X x x x y
((11, V1 (181, Vi
( 1/2) 2112, U2)
X =
LØ(In, Vn) o(In, v/5)
When re-written in this form it is trivial see that the stability of the IPO Loss depends upon the rank of the matrix X.
We begin by conducting experiments with IPO considering the existence of closed form solution for this particular setting. In particular, given set of preference data D {D D2 from either groups with varying ratio, one aims find an optimal that minimizes the group IPO loss,
Group IPO:
2
Laro(E,
I (2.thr All D-
[((6(1, Vnn 0) M). 0)
(41)
+
E (n.Me.Bt)>Dg
yhe 1) Φ(x, 91. 1),0) 23 ) [((0(2, yee
1
(42)
Concisely, one also write this as,
1 2,3
LIPO(T, D)
((o g) 6(x, 91, .),
(43)
(x.
We assume that the feature vectors for each group is known, but the reward parameter 0 is unknown. So, given the preference data with group information and the group dependent feature vectors, one aims learn optimal that balances both groups by minimizing the above loss. The solution this will still be the closed form solution expressed Equation (40) with feature matrix X consisting group dependent features.
Group Robust IPO: It is straightforward to see that in such a setting, the group with higher number of preference data will have unfair advantage as they contribute more to the loss. Hence, a robust approach needs to be considered as follows:
roLIPO ( D) = max
E Σg Og (N.
a ((0(1, yw: g) - o(x, Vr, g). 0)
23 2 )
(44)
One can use, Algorithm D to find the optimal 0 for Equation 44 But considering the weighted regression form of the loss, last step, inadvertently leads to a weighted regression whose solution can be obtained in closed form.
use the following simplified algorithm Algorithm 2 Note that the
D Additional Experiments and Details
In this section, we discuss additional experimental and training details. We use the following hyperparameters for the synthetic experiments. The importance sampling methods use the same hyperparameters as the corresponding vanilla ones. Further, we note that there is no learning rates for IPO and GR-IPO use the closed-form solution detailed in Section 4. for updates.
20

---

Algorithm 2 Policy Optimization for Robust IPO in Linear Bandits
1: Initialize: Step size T/a for group weights a, initial weights g(0) of the policy and weights over each group (0)
2: Input: Dataset DD with size N = [D), group size N₂ for 11, K
3: for 1,..., do
4: Calculate group loss lg for each group on Dg
5:
a + ag exp (nG4g)
Update weights for group
6:
a + a' Eg' d'g I/ Renormalize
7: 0r + arg ming (a...y..
ag ((0(z, Vw. (e. o(z, yr, v1. 0) -
8: end for
9: Return: Output the robust policy T00')
Training Type
Learning Rate
Step Size
DPO
0.9
IPO
0.1
GR-DPO
0.9
1
0.5
GR-IPO
0.1
0.01
Table I: Hyperparameters for synthetic experiments.
D.1 Synthetic Experiments Experiments setup
Here we provide full definition of our synthetic experimental setup discussed in Section
We adapt the standard (non-group based) experimental setup proposed by [26 for the group pref- erences setting by incorporating group information into the reward function r X' x y g - TR. Here, X' represents two-dimensional state space [0, [0, 1]. denotes discrete action space (0. 2, 1,,,....,,, n}, defined by the group-dependent feature vector (xx, V, 9) and parameter vector 0g. given as:
and 9 signifies a discrete group space {0. 1,2,.2.... K}. The reward function,
(0x y. V..)..)),.. 0g)
(45)
Further, we consider n = 7, K = 2 and denote x == x1) (0o(x, (.), 01 y, g), o2 (z. y, g). (I v.g)) and parameters 0g Vgg E G are defined as follows:
x1) E X. The feature vector Φ(x, y. g) ==
cos(x [122 T) if 122 = g otherwise
6.(x.99))
0o (1,3,1,3), 0, (3. 1. 3.
sin(( T)
By design, this parameterization ensures a coordinate-fiiper relationship between groups, effectively mirroring one group the other. Also, study two other feature parameterizations and include their experimental results in Appendix D.3
D.2 Synthetic Experiments Preferences data generation
In our synthetic experiments, we considered three scenarios: (i) Groups are imbalanced in size, (ii) preferences distribution, and (iii) both 0.2 0.8, respectively. Instead, to generate imbalance in terms of preferences we proceed as follows. In scenario (i), after randomly selecting state x within group g and action y/, second action y/2 is randomly chosen (groups have the same distribution (iii), for one group 3/2 is the action most distant from Yı. while for the other, is the closest (groups have different distributions over responses). In both methods, we calculate the rewards r(z, 31, g) and r(x, y2, g), designating the action with the higher reward as preferred.
above. The size imbalance generated by using ratios
responses). Instead, in scenarios (ii) and
Here, we provide the resulting plots for scenarios (i) and (ii).
Trading off worst-case Average performance
We perform ablation studies for various values of x trading-off between the average and worst-case performance. We observe in Figure 6 that the max validation loss decreases while moving from
21

---

DPO Max Validation Group Loss
IPO Max Valiciation Group Loss
Converged Max Reward Frror
0.10
26
Value
0.8
DPO SR-DPP
Value
IPO GR-IPO
Valne
0.05
20
0
2500
Iterations
5000
7600
10000
Iterations
0.00
Methods
Figure 4: A (GR and GR-IPO) leads to a lower worst-case validation loss and reward error compared to importance sampling and vanilla methods. Results refer to the scenario in which groups have different sizes but same responses' distribution. Note that the gap between Algorithm| [ and importance sampling is smaller than in Figure This is expected considering that the primary difference between groups arises from data imbalance, which is handled by importance sampling.
DPO Max Validation Group Loss
IPO Max Validation Group Loss
Convergod Max Reward Ervr
0.60
20
0.10
Volue
0.35
DPO
Vilue
IPO
Value
GR-DPO
GR-IPO
11.06
0.50
18
0:45
2500
7500
50 Iterations
Methods
Iverations
Figure 5: Algorithm (RR-DPO and GR-IPO) leads to lower worst-case validation loss and reward error compared to the non-robust vanilla methods. Results refer to the scenario in which groups have same sizes but different responses' distribution. Unlike the setups of Figure f and Figure importance sampling has no effect here (it coincides with vanilla DPO/IPO since groups have the same sizes).
Max Validation Group Loss
Average Validation Loss
20
GRJPO 0.0 (5-DOO)
16
GR-IPO -. (5-DO)
Value
Value
GR-1PO
GIL-PO
GB-PP
19
14
18
0
20
100
0
20
100
Iterations
Iterations
Figure 6: Ablation study for the trade-off parameter x in the synthetic experimental setup. Results refer to the scenario in which groups have different sizes and different responses' distribution. Note that increasing x improves worst group performance at the expense of average performance.
= 0 X I, where x = 0 corresponds importance sampling with group weights P...... Mg mapping to importance sampling weights (see Equation (30) and x = corresponds to GR-IPO. Further, we plot the average validation loss which increases while moving from X X = demonstrating the trade-off between average and worst-case performance. Note that, GR-IPO aptly increases the average loss (as expected) in order to reduce the worst group loss.
D.3 Synthetic Experiments Additional feature parametrizations
We present further experiments synthetic preference data, using different configurations of the d(x, y.g) and 6, vectors characterising the group-specific reward distributions r(x. y. (0(2, y, 9), 0g)-
With the same action space n 7 and group space K 2, we consider same and flipped configura- tions of Ø(z y. g) (0o, Φ1, 02, Φ3), as follows:
22

---

Same: Here, the feature vectors are the same irrespective of group and reward vectors are coordinate- flipped.
n+1 + +1 cos(z [1/2) T)
1%22
61 9.
== (1,3, 01 13 1,
sin( T) otherwise
Flipped: Here, the feature vectors include alternating sin, cos terms with swapped order for g = {0, w.ith alternating coefficients. And the reward vectors are coordinate-fiiped as before.
-1-1 [9h2-1]
h+I +1)
c0s((1 T)
if 1%2 = 9
(x. =
-11 [96--]
+
sin ( x
otherwise
n+1
The experimental results shown in Figures and 12 for the flipped experiment.
and for the same experiment, and Figures TO
DPO Max Validstion Group Loss
IPO Max Validation Group Loss
Converged Max Reward Error
0.8
0.10
Volue
Value
Value
0.6
IS-DPO
IS-IPO GR-IPQ IIO
0.06
EPO
0.1
Iterations
5000
7500
Iterations
Methods
Figure 7: In this experiment, we consider same feature vectors on even groups imbalanced in size. We observe similar performance as in Figure f where we consider the same setting with swapped feature vectors for groups. GR-DPO slightly improves over importance sampling DPO, however, GR-IPO exactly matches the performance of importance sampling. And this is expected considering that the groups have the same level of difficulty and differ only in terms of data size.
DPO Max Validation Group Loss
IPO Max Validation Group Lass
Converged Max Reward Error
19
0:075
Value
0.5
DPO
Value
IPC
Value
0.0%0
0.025
17
0.4
0.000
100
GB 1910
GR DRO
Iterations
Iteracions
Methods
Figure 8: In this experiment, we consider same feature vectors on wreven groups balanced in size. Here, GR-DPO/GR-IPO outperforms corresponding importance sampling methods in terms of worst- case validation loss, but, GR-IPO tends overfit. This is reflected in the reward errors, where GR-IPO performs worse than IPO. oowever,overall. GR-DPO outperforms all other methods in terms of worst-case reward errors.
23

---

DPO Max Validation Group Loss
IPO Max Valiciation Group Loss
Converged Max Reward Error
0.6
0.075
Valuo
DPO DPC GR-DPO
Value
IPQ
Valne
0.5
IPO GR-IPO
0.000
0,025
0.4
0
2000 Iterations
6000
7600
10000
Iterations
106
0.000
Methods
Figure 9: In this experiment, we consider same feature vectors on uneven groups imbalanced in size. We observe similar performance in Figure & where consider the same features for groups but with balanced data. Here, DPO is overfitting and GR-DPO outperforms both DPO and importance sampling DPO. IPO is stable, considering it has closed form solution. And in terms of worst-case validation loss, GR-IPO tends to overfit, even though it outperforms IPO and importance sampling IPO. The overfitting is evident in reward errors, where GR-IPO performs worse than IPO and importance sampling IPO. However,overall, GR-DPO outperforms all other methods in terms of worst-case reward errors.
DPO Max Validation Group Loss
IPO Max Validation Group Loss
Converged Max Rewand Error
0.6
Value
0.5
DPO GR-DPO
Value
IPO
Value
0.04
16
GR-IPO
0.4
0.00
14
0.3
0
2500
Iterations
501000
75000
10000
25
Iterations
50
75
100
0.00
Methods
Figure 10: In this experiment, we consider slipped feature vectors on even groups imbalanced in size. We observe very similar performance in Figure where GR-DPO/GR-IPO slightly improves importance sampling DPO/IPO.
DPO Validation Group Loss
IPO Max Validation Group Loss
Converged Max Reward Error
0.166
Value
Value
18
Value
0.04
GR-DPO
GR-IPO
0.02
0.7
0
2500
Iterations
50000
7500
10000
25
Iterations
100
0.00
130-00
Methods
CR DPD
Figure 11: In this experiment, consider flipped feature vectors wneen groups balanced in size. Here, GR-DPO/GR-IPO outperforms corresponding vanilla methods terms of worst-case validation loss and GR-DPO outperforms all other methods in terms of worst-case reward errors.
DPO Max Validation Group Loss
IPO Max
Validaaioo Group Loss
Converged Max Reward Error
0.10
Value
0.6
Value
201
Valne
DSO
IS.IPO
0.4
CR-DPO
18
GR-IPO
0.06
0.3
2500
5000
7500
10000
GlaIP Methods
Iterations
Iterations
Figure 12: In this experiment, we consider flpped feature vectors on uneven groups imbalanced in size. Here, we observe similar performance to that of Figure Tigure[wherr where GR-DPO/GR-IPE outperforms corresponding vanilla and importance sampling methods terms of worst-case validation loss. And, GR-DPO outperforms all other methods in terms of worst-case reward errors.
24

---

D.4 Global Opinion Data Experiments Setup and Additional Plots
We use the following prompt type for both SFT and DPO training: "Opinion of people in (country- name) on: (question). Please select the best response: A) Choice-1, B) Choice-2, ...". The target response is just the letter 'A', 'B', etc. The training data size from each country are as follows: Nigeria-572, Egypt- 570, India-376, China-309, Japan-712. Further, the data is split as 80% for training, 10% for validation and 10% for testing. We run the SFT training with learning rate 10-1 for one epoch over the training data pre-trained Gemma-2B model. We use this SFT trained model as the reference model for training IPO and GR-IPO. For the IPO training, the optimal hyperparameters the same / but the optimal learning rate and the exponential step size were 6 10-5 and 5 10 For both IPO/GR-IP -IIO training we use Adam
were learning rate 3 10-5 and B 0.01. For the GR-IPO training,
27 optimizer with adaptive learning rates decreasing by a factor of 10 if there is no improvement in terms of loss (average group loss for IPO/ worst group loss for GR-IPO) validation set. Further, for GR-IPO, the exponential step size is decreased by factor of 2 whenever learning rate is reduced. This adaptive learning rates and exponential step sizes ensures stability in training and leads convergence. We then evaluate both the methods based on the worst group loss and accuracy. Here, the accuracy refer to the percentage of prompt, winning response and losing response correctly ordered by the leamed preference function (Bquation (35)).
for 4k iterations
All experiments were run on single node of A SXM4 machine with 40GB GPU memory, 30 CPU cores, 200GB RAM, and 525GB SSD memory. Further, for each seed, the execution time of the experiments approximately 3-5 hours until convergence. We also provide the exact hyperparameters used in the table below.
Training Type
Learning Rate
G
Step Size
Optimizer
SFT
10
RmsProp
IPO
3 10-5
0.01
AdamW
GR-IPO
6 x 10-5
0.01
x
10
AdamW
Table 2: Hyperparameters for SFT, IPO, and GR-IPO training.
Max Group Loss Test)
Min Rewards Accuracics (Tost)
Log-Prob Avctracics (End)
2500
0.80
110
GR-IPO
0.25
Valuue
2000
GR-IPO
0.50
IPO
Groep
Ierations
Iserations
Group
Figure 13: Global opinion data: Evolution of worst-case loss, reward accuracy during IPO and GR-IPO training. Notably, both IPO and GR-IPO improve their accuracy. However, GR-IPO achieves better worst-case alignment performance. In the right, we plot the end training log-prob, accuracies for GR-IPO.
25

---

E
Convergence Proofs for Different Sampling Strategies
In this section, we study the convergence rates for different sampling strategies including the proposed Algorithm
E.1
Convergence Proof for Algorithm
1
In this section, prove the convergence of the proposed Algorithm terms of the error of the average iterate Tp.:;),
ET = LGR (TT(()))))
min (T))
(46)
as stated in Proposition 4.1
Proposition 4.1. Suppose that the loss U(; (Tg. v') is non-negative, convex, By -ipschhhi con- timuous, and bounded by Bı for all (Tgs V. y') E X 2 y x y and ||0||2 I Be for all 6 E θ with convex θ c Rd. The error of the average iterate of Algorithm i.e., Tgarr = 7 Ei= 0", satisfies
min (an( 1/7 AEB
Proof. We build our proof based upon the online mirror descent algorithm's 's regret bound in 32 Eq. 3.1]. 32 s theorem considers the following saddle-point stochastic optimization problem,
K
{ {00,a)= = Σ ag (1F
min max 6EB at aEAK-1
(47)
Denote (0) (0,EE) In our problem, we consider,
= (5gg, v. 1/) ~ D,
(48)
which is equivalent to and we define (g (0, (Tg. (Fg.
N
E : U.
Dg.
(49)
N No
Lo=0" [(0; (Fgz s')). Then, the expectation
K
Ng
Eg (0,EE(
Ep
N
g'=1
Ng
ED, (0. =6
N
No
N 1(0: (Igg v. v/ g = g
ED,
N
[Ng
= ED, [1(;; (I y, 1f M) g
= (0)
Mirror Descent mapping. We begin by mapping Algorithm to the general mirror descent framework of 32)
Further, we denote the gradients of the objective @(0 a)) w.r.t. A and a as
0o(0. a) = (0.0(0, - a)).
Here, the negative sign for the gradient w.r.t. o indicates that we perform maximization w.r.t. a. Let (00, a) := Ege 0gg (0.EE) Then the stochastic subgradients for s(0, a) for given & is as follows:
04 a. (0.4 a, E), (0,
26

---

Simplifying the gradients we obtain the following for Ç = (I V. v.8'):
Oo Σ o== Og' [Fg. (0,E))
@ @ a,))
0a Lg'=1 ag' [F (0,
R
do g1=1 ag Ny 1 la= (0: (Eg. V, v'))
0a g'==1 ag' N2' - (0: (Igr v.
N
Na. Vol(0; (I v/)
N₂
(0, No
N [0; (I y, 1/
Based on the above gradient expressions, the general mirror descent update rule:
map the updates of 6 and a in Algorithm
to
r = (8'-1, a a1 = Picu Pr aare",a',,00 a²,E)).
(50)
Here, use the compact notation Ç (f) Further, y denotes the step-size at time t of the algo- rithm and (-) is the ppox-mappee mapping corresponding to the mirror descent algorithm defined as follows:
Pe(v)=
arg min ('E(0xāR-))
(C
(51)
Here, ((.,)) is the prox-function associated with the distance-gentn generatine function (Bregmann Di- we define the following combined distance-generatle
vergence) (.). ). For our setup in Algorithm
|
function over Ç E (0 Ax-1):
(0)
(a)
w(C) 2D 0
(52)
2D2
Wa A
where wo 11011 (1) Eg= ai In D = (max wo( - minøee wa (0)) 1/2 and Dw.. = (maxa Wa (a) mino Wa (a)) 1/2 Also, the corresponding prox- functions individually for 0 and a would be (0,
01 and (a, ΣØ a² In of a. It can be shown that the above prox-fure frx-functions Vo and Va satisfy the following inequalities (322);
max (0 0) S D2 De BEE
Va a) S D -...A
(53)
a
Due to the above definitions, the corresponding prox-mapping P(-) corresponds to gradient descent w.r.t. and exponentiated gradient ascent w.r.t. a, as defined in Algorithm Moreover, due to the definition of w in Equation (52). the combined prox function (---)) over would satisfy
max CE@XAR-1
(( $1
(54)
For further details regarding prox-function, kindly refer to its usage in (3]
Bounding the error. According to the introduced notation, the approximation error eq can be defined and bounded as:
ET max 6(QLT a) min max $(0,a) a) aEAK-1
dee aEAx-1
max 0(0 ,a) min @08 1:7)
aEAR-1
#EB
To bound ( invoke the result from for max-min problems.
32
Equation 3.23] for mirrrr-dsceet convergence
R2M
In( M2
$2/
10
(55)
T
27

---

Further, Ma and Ma correspond to the following upper bounds to the maximum of the expected norms of (6, and (8.E)
N 1 (o= Vi(0: (Fg, //)l
N₂ N2
max B| F 1Sask
max E 1 K Ng
= max
VI(0; (rg Us y MI??
1sask N No
Vi(O; ( v.y v/ ))1
max
1595K K N
N
B& max
yEs N₂
N
B
mingeg Ng
Similarly,
N
2
N
N2
max [F,
=E max 1595k
1000) (cg,
5
|1(0; (Ig, v/))
1595K
Ng
N
N3
9
N
[[((; (Iig v'
N
K B
max gEG Ng
N
K KB
mingeg N₂
.M3
Here, we have used |/Va((0; (5 x))ll < By and [I((; (x,, 8))1l S Bi. Here, dØ, Aa correspond to the strong convexity parameters of the distance generating functions wo( and Wa (a) respectively. For our given functions, (0) = Y 1011 and w. (a) Σø= a; an;; both Ào = Aa 1 (33)
KK
D²
Let, R3
BB (maxo |l0lle - ming 101)), obtaining the overall error bound.
Xo
10 N mingeg Ng
Bg B2 + KBBlk K
( s 2
(56)
T
Lemma E.1. For the log-linear policy class parameterized with respect to O, the DPO loss function [(Te: = log (a (Bh ())) (see Equation 55) is convex and Lipschitz continuous in O. Consequently; Proposition applies to this case.
Proof. We want to show, 1(0; (x, ywe 31)) is convex in d and B Li-Liscchitz in 0 for the log-linear policy class defined as follows:
exp 6" ((2.9) T x) Evey exp 87 d(x.y)
(57)
Here belongs a convex θ satisfying ||0|| < Bg and o(x, y) denotes the feature vector such that [|0( Vw)) oFzg 3I < B&- In conjunction with a uniform reference policy neet the loss function I(0: (Igg Vw 91) for the log-linear policy class is as follows:
1(0; (Zig Bw, u1)) log (a (0((((( Nww - Hl ()),,)))))) 0))
(58)
28

---

Algorithm 3 Mirror Descent for Group Robust Preference Optimization (GRPO)
1: Initialize: Step size T/o for group weights a, step size f/ for policy т with weights 0, initial weights (00) of the policy and weights over each group ( Projection operator Pe
2: Input: Dataset D with size N = (D), group size Ng for 2 ... ,K), loss (
3: for ===,, 1......, do
4: a' a'+a(i-i)
5:
g~ Uninform(11..... ,))
6:
(Inr 31) D.
Og - 0g exp (ooaall(Trrt
i (Tg (( ( Mee w:)) // Update weights for group a e'/Eg' o'g I/ Renormalize
7:
8:
800 + Pe (((- no [ Vol(Tou--: (Zg: Me. B1)) // Use a to update
9:
end for 11: Return: Output the robust policy T(((2))
Next, we compute the derivative of I(0; (z, Bwus 91)) w.r.t. 0 from Equation (58) to check the Lipschitz continuity.
Vol(O: (Eg. Gw U1))
o (3((5, yo (( (1 (B((((g))))))) (F 01)) (6(2g. Meo (x 31))
a ( Mw)) 6( 11), 0;)
(1 (B(db(zg. Vio) g 31 m)) 01) (Ø(zg) the) - (Fg 31)
= =3(((0( yw) (5 31 - 0)) (0(ggg Uar) ((5gg 21))
Then, the gradient norm can be bounded as follows:
[ (Igg Vw 31 I)l S [10( Uw) p(zg vtll
< Bg.
Hence, by the definition of Lipschitz functions for continuously differentiable functions, we have that 1(0; (z, Yw. 91)) is Lipschitz continuous with Lipschitz constant Bg. Further, we bound the loss function using the bounds for 0 and o(..)).
(Ō(Ig, yw)) 6(zg. 31), 0) 2 -110(1g: (w) - (()lle))l
(0(1g) y 6(zg. m).0) 2 Bg
(( (w) - (b (() 0))) Z log(o((-)) BB Be))
log(ar(((<I))) Fw) ((Ig.
(lo((o(-BBBa)
log(1 + exp Ba))
< log(exx (B, exp (Ba
< log((2 + Be) = B
Hence, the loss function is bounded by Bi for bounded 0 and ø.
E.2 Alternate Sampling Strategy
In this section, we study the convergence of an alternate algorithm in Algorithm 3 wherein groups sampled uniformly rather than categorical distribution proportional to the group sizes, in terms of the error cT as defined in Equation (46).
Proposition E.2. Suppose that the loss 1(:; (Ig. y. y')) is non-negative, convex, By Lipschitz ) E X @ G x y x y and [|01|2 < Bg for all 0 E θ with convex O C Rdd. Then, the average iterate of Algorithm 3 achieves an error at the rate
continuous, and bounded by B, for all (zg: V.V')
B B2
KB K
( $2)22/111
(59)
T
29

---

Proof. We follow a similar proof structure as in the proof of Proposition 4. We recall the saddle- point stochastic optimization problem from [32]
stated earlier in Equation
min max BEB asAR
{ (0(0a))
Σ ag B(F
(60)
In lieu of the alternate sampling strategy in Algorithm 3 where groups sampled uniformly, consider,
{: : (... 3/)
Dg
(61)
and we define Fg (0, (Zg, v. 3/)) = K1 | 1 ((0; (Ig. V, v'))), Then, the expectation
K
1 K
Ee (e = Σ
DD (Fg(0, g'l
9'=1
ED, ( (0,E) E)gg =9]
K
1
ED [K] (s 1')'')) 96)
K
ED, [((0; (Eg. U. 1)g =g'l
(0)
Mirror Descent mapping. We map Algorithm 3 to the general mirror descent framework of [2] as done in the proof of Proposition 4.1 We begin by recalculating the gradients of Φ(0, a) w.r.t. 8 and for this alternate definition of (),
0. Σ g'=1 Og I (0, E)] 0n 2g'=1 ag F (0, [)
@ =
K
Oe ΣK= 0'=1 ag' [ -n= 100: (zgg. v, v.,')))
dn Eg= =1 Ogg' [K] (0=0 (0: (Igg V, v'))]
(0. K1(0; (I (...)))........)).. .0)
Kag (0; (1
Based the above gradient expressions, we can similarly map the updates of O' and a' in Algorithm 3 to the general mirror descent update rule with corresponding prox- fun-ctnntoon as done for Proposition 4.1 We omit further details regarding the mirror descent related definitions as they follow from the proof of Proposition 4.1 We directly proceed to bounding the error ((1:T) using Equation 3.23],
M3
(
= 10
(62)
T
We recalculate Mo and M.. for this alternate definition of Fg (0,E) as they correspond to the upper bounds to the maximum of the expected norms of Fg (0, {) and (oFg(0,5)
max E (F S
BIJK1 | (g=99) VI(0; (xg, v, v/ 1))2 S
max
max K2 Vi(0; (Fg o/ I)
ISASK K
K Vi(B; (I V.v.v.)))) v/
max
1S
- B&K K
B
M3
30

---

Similarly,
K
1 K
E max 15g5
(0,5()))
=
| [g=g]) (0;; (Ig, v, 12
Σ
K
15g5K
9=1
K
Σ
K K||[(0; (T V: 2/)2
9=1
B7K²
B K2
Ma
Here, we have used Vol(O: (zg. y))ll s By and |[[[; (zg. S Bi.
Using the alternate Mg and M, corresponding to this sampling rule, we obtain the overall error bound follows:
BB + K
< s 2/ 10K 9/10K
(63)
31

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: All the claims regarding the proposed methodology in terms of theory and experiments have been concretely illustrated in Sections and Theory) and Section 5
(Experiments).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are detailed along with conclusions in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

All assumptions are mentioned in the theorem statements and the corresponding proofs are detailed in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All experimental details (models, datasets, configurations, training strategies, compute resources) are discussed in the experimental sections of the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
33

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide links to Github repositories that contain all the codes required to reproduce the experiments with detailed instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//bdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameters, optimizers, data splits, etc., are detailed in the experi- mental section in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are shown based on different seeds of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources required and used to run the experiments are mentioned in the experimental sections in the main text and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics / httprr/psscpppullii////eeeeeiite htt.ai/eeeehesSSiielllll.le.ee
cc/public
Answer: [Yes]
Justification: We are using open-sourced data and language model. We do not foresee any potential harm caused by our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader positive societal impacts of our work in Section We do not foresee any negative societal impacts of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We using publicly available model and data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the data and models used have been appropriately cited with URLs which includes the licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and algorithm implementation has been properly documented and links to the Github repositories have been provided.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct any such experiments for this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "fih8XjRWa3", "reviewer": "Reviewer_V78J", "summary": "This paper addresses the problem of improving fairness in preference optimization. It proposes a new loss function and algorithm. Experiments conducted show that the proposed algorithm indeed achieves better fairness. Additionally, the paper provides a theoretical analysis indicating the convergence of the optimization.", "questions": "1. The paper mentioned that the algorithms can also be used for multi-objective preference optimization. Were any related experiments conducted?", "limitations": "See the 'Weakness' part", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of improving the fairness is good. This is a problem that deserves more attention.\n\n2. The derivation of the loss and the algorithm, though simple, is clear.\n\n3. The results provided for the algorithm appear to be sound.", "weaknesses": "1. The author mentioned that the idea can also be applied to multi-objective preference optimization. In my opinion, a comparison with works in multi-objective preference optimization [1] and group optimization [2] is needed.\n\n\n\n2. It is unclear whether this algorithm can be applied to scenarios beyond multiple choices.\n\n3. It is unclear how Proposition 4.1 relates to common metrics like regret.\n\n\n[1]. Zhao, Siyan, John Dang, and Aditya Grover. \"Group preference optimization: Few-shot alignment of large language models.\"\n\n[2]. Wang, Haoxiang, et al. \"Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive opinion about our work recognizing the importance of the problem, clear and sound exposition of our algorithm and theoretical results. \n\n**Regarding comparison with works in multi-objective preference optimization [1] and group optimization [2] and related experiments:** \n\nThe primary focus of this paper is to design a robust fine-tuning approach to align a language model with diverse group preferences and validate it both theoretically and numerically. Indeed, our method applies to robust multi-objective preference optimization. It is definitely interesting to conduct experiments for the robust multi-objective setting. However, these are beyond the scope of this project and will be considered for future work.\n\nCompared to [1], the distinctive feature of our method is that we are not modeling multi-reward objectives but consider a reward-free setting. Specifically, we consider the robust alignment problem optimizing for the worst group performance. Further, they align with user preferences assuming that each user/group has varied importance over the distinct metrics in their multi-objective reward model. The output policy is trained to output a response based on both the prompt and the importance/weights over the individual metrics. Whereas, our methodology directly models each group’s preferences through a group-dependent latent reward model where the group dependency is injected through the prompt.\n\nCompared to [2], as explained in Lines 68-71 and also in the appendix (Lines 520-523), we note that they consider alignment to multiple groups’ preferences through an in-context learning approach that is different from our LLM fine-tuning methodology. In particular, they consider a distributional objective that is different from the robust one we consider. \nHence, we believe that [1,2] are not directly comparable to ours and we will include this explanation and comparison regarding [1] in our related work section.\n\n**Regarding application of our algorithm to scenarios beyond multiple choices:**\n\nYes, it can be used for scenarios beyond multiple choices. The reason why we use the global opinion data with a multiple choice structure in our experiments is to clearly demonstrate the efficacy of our approach in aligning LLMs to diverse group preferences and such datasets are typically used in similar pluralistic alignment studies (e.g., [2,3]).\n\n**Regarding relation of Proposition 4.1 with regret:**\n\nProposition 4.1 is indeed related to regret. It bounds the expected difference in loss between the average iterate policy and the optimal policy after T iterations and demonstrates a sublinear dependency on T. This allows us to provide convergence guarantees for our algorithm in terms of the average iterate. Such a regret formulation is common in minimax problems as discussed in Nemirovski et al. [30]. \n\n[1] Wang, Haoxiang, et al. \"Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards.\"  \n[2] Zhao, Siyan, John Dang, and Aditya Grover. \"Group preference optimization: Few-shot alignment of large language models.\"  \n[3] Sorensen, Taylor, et al. \"A roadmap to pluralistic alignment.\" arXiv preprint arXiv:2402.05070 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "95rXIwf4nT", "reviewer": "Reviewer_qNjA", "summary": "This paper introduces GRPO, a method to optimize policy preferences across different groups in a robust way. GRPO looks at the worst group alignment loss by taking the maximum loss across all groups, ensuring the policy performs well even when there are group-specific differences or overlaps in prompts.", "questions": "please see weakness.", "limitations": "1. This methods requires knowing the number and nature of groups in advance to perform optimization. \n2. When new group is introduced, will have to train the entire optimization again, which is expensive.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper writes well and easy to follow.\n2. The paper provides a thorough theoretical analysis of GRPO.\n3. The proposed framework accommodates both distinct and overlapping group scenarios.\n4. GRPO can trade off worst-case for average performance with a hyperparameter.", "weaknesses": "1. GRPO's experiment on real world datasets is limited, on only one group (5 countries) of the GlobalQA dataset.\n2. How is the training time as compared to other baselines? The GRPO framework's training process involves a min-max optimization, which can be potentially computationally intensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive opinion about our work recognizing our thorough theoretical analysis and broad applicability of our GRPO approach. \n\n**Regarding the groups in GlobalQA dataset experiment:**\n\nOur proposed GRPO method can be applied to any set of finite groups. To demonstrate the effectiveness of our method, we focus on five groups that are diverse in terms of both dataset size and ethnicity (see Lines 295-296).\n\n**Regarding training time comparison:**\n\nThe training time for both IPO and GR-IPO are *almost the same* and are approximately around 4-5 hours for the GlobalOpinionQA data run on pre-trained Gemma-2B model. We have detailed the time and the configuration of the GPU processors used for our experiments in Appendix D.4 (Lines 685-688).\n\nExplanation: We do not need to solve the inner maximization in our minimax problem (see Eq. [6]) at each iteration. Instead, we update the weights over the groups based on the loss of the current sample as per Line 6 of our algorithm. Thanks to Danskin’s theorem, computing the gradient over the policy parameter $\\theta$ only requires computing the partial gradient w.r.t. $\\theta$ for a fixed $\\alpha$, which approximates the maximizer $\\alpha^*(\\theta)$. As $\\alpha$ is also updated iteratively, the computation time is comparable to the amount of computation time in the standard case. Thus, it does not lead to any significant computational overhead. \n\n**Regarding the requirement of knowing the number of groups in advance:**\n\nWe focus on settings with known groups, which are common in pluralistic alignment datasets and tasks (see [1,2]). When groups are unknown, one can still apply several approaches that we hypothesize below.  \n\nTo identify unknown groups in data, one can: a) use clustering: Apply algorithms like k-means or EM to detect potential groups; b) apply representation learning: Use techniques like variational autoencoders (VAEs) to discover hidden structures; c) analyze features: Look for features with high variance or correlations to hypothesize groupings; d) consult experts: Leverage domain expertise to identify meaningful groups. These methods can help uncover group structures in the data.\n\n**Regarding requirement of re-training when new group is introduced:** \n\nWe agree with the reviewer that introducing a new group requires additional training. However, note that one does not need to restart from scratch. The current policy parameter serves as a warm starting initialization point. Moreover, such additional training is necessary to ensure equitable alignment for both new and existing groups. We do not consider this an explicit limitation of our proposed algorithm, but rather an interesting future extension of our approach to continual learning settings.\n\n[1]. Zhao, Siyan, John Dang, and Aditya Grover. \"Group preference optimization: Few-shot alignment of large language models.\"  \n[2] Sorensen, Taylor, et al. \"A roadmap to pluralistic alignment.\" arXiv preprint arXiv:2402.05070 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vjSqBrBmy1", "reviewer": "Reviewer_kmNK", "summary": "The work tackles the important problem of robust RLHF for diverse groups. Traditionally, RLHF assumes that a single model can fit the diverse feedback from multiple groups of users. In this paper, the authors introduce a method to learn a robust policy that maximizes for worst-case group performance. To achieve this, the method adaptively weights the loss for each group based on the size and cumulative training loss incurred by the feedback samples for that group. As a result, LLMs trained on diverse group data demonstrate reduced loss imbalance and improved accuracies across all the groups. The authors also present a convergence analysis of the proposed method assuming a log-linear policy class.", "questions": "1. In the current experiments, is only the final layer of the LLM trained? I would be curious to know the result if the method used full finetuning of the model.\n2. As GRPO assumes that each prompt has access to group information through a prompt, how much does the prompt affect policy? If the prompts are good enough, would it just create non-overlapping distributions for all groups? As one of the problems in non-robust baselines is that they converge to unwanted biases or majority groups for shared prompts, if the group identification prompt is finetuned will it alleviate this issue altogether?\n3. Does GRPO finetune only the final layer of the Gemma-2B model in the results presented in the paper?\n4. Does GRPO ensure that there is a non-decreasing change in performance across all groups as compared to the non-robust baselines?", "limitations": "1. The paper introduces GRPO, an optimization method to weight each group in the RLHF update step proportional to the size and loss of the group. This ensures a balanced performance of the model across all the groups. In the results, we see an improved performance over all the individual groups, which intuitively violates the no-free lunch theorem. I would be curious to see an analysis of the method that gives insights as to what allows the model or the objective to achieve consistently higher performance.\n2. Here, the method assumes access the groups in the dataset. However, in practical settings the group information is unavailable and the model has to cluster or implicitly model the group information from the ungrouped dataset.\n3. The evaluation is limited only to the accuracy of the reward model over the preference dataset. So, currently, it provides weaker evidence of the translation of this robustness during the generation phase. It would be nice if the authors could include experiments showing if GRPO enables LLMs to robustly generate better-aligned responses to prompts from all the groups.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a critical problem of robust RLHF for diverse groups. The intuition of the overall method is well understood, the framework and the parameters are clearly mentioned, and the results are shown over the standard datasets and compared to multiple baselines.\n2. The authors present a thorough literature review and background. Past work has mainly focused on making RLHF and LLMs robust to noisy or out of distribution data. Meanwhile, this work focuses on a group robust formulation of training LLMs using state-of-the-art methods (mainly DPO).\n3. The method introduced “GRPO” is useful is scenarios beyond diverse groups. As the authors mention, it is a general formulation that can enforce robustness to diverse tasks, domains, or objectives occurring in the feedback dataset.\n4. To achieve robustness, GRPO presents a robust optimization approach to minimize worst-case loss amongst the diverse groups. Further, the paper introduces a less aggressive object by trading off worst and average case performance. I would be curious to see an ablation study showing the effectiveness of this tradeoff.\n5. The paper supplements the approach with some strong theoretical proofs on the convergence properties of the method under a log-linear policy. The authors also present a close form solution for the RLHF update step, replacing DPO with IPO.\n6. The results show that GRPO improved performance across all groups, and the weight update behaves as expected by assigning higher weights to groups with higher cumulative loss i.e. the gradients from the worst performing group are scaled the most.", "weaknesses": "1. This paper proposes a method for group robust optimization for LLMs. However, the metrics evaluated are only max validation loss and reward error over the groups. GRPO uses a reward free approach to update the LLM, but the evaluations are restricted to the performance of the reward model over the feedback dataset. It would be nice to see the performance of the finetuned vs base model (such as win rate) in generating responses that align with the individual groups.\n2. The authors provide detailed training setup, but I would suggest that they also include information about the evaluation. \n3. GRPO performs well over all the groups, however, the performance of the importance sampling baseline is very close. It would be helpful if the authors could provide additional ablations and discussions to show the effectiveness of optimizing against the worst-case loss over the groups vs using only the IS approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive opinion about our work recognizing the importance of our problem setting,  broad applicability of our GRPO approach, and strong theoretical analysis of our proposed algorithm’s convergence properties. \n\n**Regarding ablation for trade-off parameter between worst-case and average performance:**\n\nDue to time constraints, we provide the results of our ablation study for the synthetic experiments in Figure R1 of the attached PDF. Note that as observed in Figure R1 of the attached PDF, the max validation loss decreases while moving from $\\chi=0$ to $\\chi=1$, where $\\chi=0$ corresponds to importance sampling with group weights $\\mu_1,\\cdots,\\mu_g$ mapping to importance sampling weights (see Eq. [30] in Appendix B.4) and $\\chi=1$ corresponds to GR-IPO. Further, we plot the average validation loss which increases while moving from $\\chi=0$ to $\\chi=1$, demonstrating the trade-off between average and worst-case performance. Note that, GR-IPO aptly increases the average loss (as expected) in order to reduce the worst group loss.\n\n**Regarding details about evaluation:**\n\nOur primary evaluation metrics are the worst group loss and accuracy. The loss refers to the IPO loss for each group and the accuracy refers to the percentage of winning response and losing response pairs correctly ordered by the learned preference function [Eq. 35 in Appendix]. We have defined this in the main text in Lines 302-304 and also in Appendix D.4 (Lines 683-684) along with details about data splits for training, test, and validation (Lines 672-673). We will revise and emphasize the definitions in a more visible manner.\n\n**Regarding alignment performance of finetuned vs base model and alternate evaluation metrics:**\n\nWe would like to refer the reviewer to Figure-3 (Bottom left), where we indeed measure the model’s alignment through worst group log-prob. accuracies. The log-prob. accuracy measures the accuracy of the model to assign higher probability to the chosen response compared to the rejected response (see Lines 302-304). We note that the GR-IPO fine-tuned model is better aligned with the worst group compared to the IPO fine-tuned model.\n\nMoreover, in our experiments, the responses/choices are included in the prompt and, our goal is to measure whether the chosen choice (for e.g., A) is preferred over the rejected choice (for e.g., C). Hence we focus on log-prob. accuracies metric. Further, as per the reviewer’s request, we plot and compare the alignment performance of GR-IPO, IPO, and SFT fine-tuned models with the pre-trained model across groups in Figure-R2 of the attached PDF. Note that the pre-trained model performs significantly worse in comparison to other methods and attains its worst performance in Group 4. And, GR-IPO outperforms both IPO and SFT in terms of worst group performance (Group 5). This is consistent with our methodology that aims to improve the alignment of the LLM and reduce the bias across groups w.r.t. SFT fine-tuned model rather than the pre-trained model.\n\n**Regarding proximity between the performances of importance sampling and GRPO:**\n\nWe kindly request the reviewer to clarify the particular figure to which they are referring to. We note that in the two scenarios corresponding to Figure 2 and Figure 5 (Appendix D.2), the groups have different responses’ distributions and there is a clear and significant gap between our proposed method and the importance sampling approach.\nHowever, we agree that in Figure-4 of Appendix D.2, the gap between GR-IPO/GR-DPO and the corresponding importance sampling methods is indeed small. This is because Figure 4 corresponds to the scenario where both groups have the same responses’ distribution but are imbalanced in size. In this scenario, such a small performance gap is expected considering the difference between groups arises solely from data imbalance, which is handled by importance sampling. \n\n**Regarding fine-tuning more layers:**\n\nWe kindly refer the reviewer to our response to reviewer SYLq regarding the same question\n\n**Regarding prompt-tuning techniques to alleviate group biases:**\n\nWe agree that the prompt does affect the policy response and policy optimization, as studied in various previous works, including [2]. In our experiments, the prompts are appended with group information (detailed in Lines 669-670 of Appendix D.4), creating non-overlapping distributions for all groups. However, we disagree with the reviewer that tuning prompts alone will alleviate the issue of biased performance across groups.  Even with tuned group identification prompts, all groups will still use the same prompt template with only the group information varying. Hence, there is no guarantee that the IPO-based fine-tuning will lead to reduced bias across groups, as the losses might still be distributed unevenly as we observed in the performances of both SFT and IPO fine-tuned models (see Figure-3). Therefore, a group robust fine-tuning strategy like GRPO is still necessary to reduce bias across groups.\n\n**Regarding improved performance across groups compared to non-robust baselines:**\n\nWe kindly refer the reviewer to our response to reviewer SYLq regarding the same question\n\n**Regarding access to group information:**\n\nWe kindly refer the reviewer to our response to reviewer qNjA regarding the same question. We believe that this is not a limitation of our work and can be an interesting direction for future research.  \n\n[1] Lester, Brian, Rami Al-Rfou, and Noah Constant. \"The power of scale for parameter-efficient prompt tuning.\" arXiv preprint arXiv:2104.08691 (2021).  \n[2] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DjwsxxFMvu", "reviewer": "Reviewer_7Bpj", "summary": "This paper addresses the limitation of traditional reinforcement learning with human feedback (RLHF) approaches that indiscriminately optimize a single preference model, disregarding the unique characteristics and needs of diverse labeler groups. The authors propose a Group Robust Preference Optimization (GRPO) method to align LLMs with individual groups' preferences.", "questions": "Please refer to the Weaknesses section for questions regarding the paper.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's motivation is clear.", "weaknesses": "1. How does the proposed method handle scenarios where preference data lacks clear group information? In practice, much preference data does not come with explicit group classifications.\n\n2. The paper should address whether the proposed method can learn an invariant preference standard from data that may contain various group information, such as different preference evaluation criteria and preferences annotated at different iterations.\n\n3. The authors need to provide and analyze the alignment results of the model, not just the reward model analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive outlook on our work and for highlighting the clear motivation of our problem setting to address the limitations in traditional RLHF techniques.\n\n**Regarding access to group information:**\n\nWe focus on settings with known groups, which are common in pluralistic alignment datasets and tasks (see [1,2]). When groups are unknown, one can still apply several approaches that we hypothesize below.\n\nTo identify unknown groups in data, one can: a) use clustering: Apply algorithms like k-means or EM to detect potential groups; b) apply representation learning: Use techniques like variational autoencoders (VAEs) to discover hidden structures; c) analyze features: Look for features with high variance or correlations to hypothesize groupings; d) consult experts: Leverage domain expertise to identify meaningful groups. These methods can help uncover group structures in the data.\nWe believe that this is not a limitation of our work and can be an interesting direction for future research. \n\n**Regarding learning an invariant preference standard from data that may contain various group information:**\n\nWe kindly ask the reviewer to clarify what an invariant preference standard means in this context so that we can provide an appropriate answer to this question.\n\n**Regarding the analysis of alignment results of the model:**\n\nWe would like to refer the reviewer to Figure-3 (Bottom left), where we indeed analyze the model’s alignment through worst group log-prob. accuracies. The log-prob. accuracy measures the accuracy of the model to assign higher probability to the chosen response compared to the rejected response (see Lines 302-304). We note that the GR-IPO fine-tuned model is better aligned with the worst group compared to the IPO fine-tuned model.\n\nFurther, we plot and compare the alignment performance of GR-IPO, IPO, and SFT fine-tuned models with the pre-trained model across groups in Figure R2 of the attached PDF. Note that the pre-trained model performs significantly worse in comparison to other methods and attains its worst performance in Group 4. Further, GR-IPO outperforms both IPO and SFT in terms of worst group performance (Group 5). This is consistent with our methodology that aims to improve the alignment of the LLM and reduce the bias across groups.\n\nWe believe that we have answered all the questions raised by the reviewer. In light of the reviewer’s opinions about the strengths of our work and our detailed rebuttal to the reviewer’s questions, we kindly ask the reviewer to reconsider their score. We are happy to answer and clarify any further questions the reviewer raises.\n\n[1]. Zhao, Siyan, John Dang, and Aditya Grover. \"Group preference optimization: Few-shot alignment of large language models.\"  \n[2] Sorensen, Taylor, et al. \"A roadmap to pluralistic alignment.\" arXiv preprint arXiv:2402.05070 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTv0m9OGRj", "reviewer": "Reviewer_SYLq", "summary": "This paper proposes a novel preference optimization technique, GRPO, which utilizes the group distributional robust optimization. Specifically, the method aims to maximize the worst-case group performance for improving the robustness of LLM models. This paper provides several theoretical aspects of GRPO when it comes to convergence guarantee and closed-form solution when considering the log-linear policy class. Empirically, the GRPO objective function is minimized by the mirror gradient descent. Several theoretical results on synthetic and real-world datasets demonstrate that their method works well.", "questions": "1. In the experiments in Section 5.2, what are the zero-shot performances of Gemma-2B for each group? It would be better to report the degree of bias in the pre-trained model together, because it can enable comparing the bias degrees between the pre-trained and fine-tuned models.\n\n2. Would you provide the results when fine-tuning more layers or the entire neural networks using your method? I think that this can strengthen this paper's contribution by showing the performance of the method in various settings.", "limitations": "They addressed the limitations of this paper in the manuscript.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This is the first study to adopt the group DRO technique in the RLHF setting. \n\n* Several theoretical results are provided to enhance understanding of the proposed method. \n\n* The proposed method is general so that it can be applied to various RLHF optimization techniques.", "weaknesses": "* This paper lacks novelty in its contribution. The group DRO technique is already a popular method used in a variety of applications [1, 2].  While the authors offer some theoretical results and an algorithm for their method, these may also lack novelty. In specific, Proposition 3.1 on the Nash equilibrium is a well-known result in game theory and Proposition 3.2 seems straightforward because $\\pi^*$ is dependent only on an input value, not on the group to which the input belongs. Furthermore, there is no significant difference in the algorithm and proof of convergence guarantee compared to existing works [37, 30 in the manuscript], except for slight modifications.\n\n* Although this paper is novel in applying the group DRO technique to RLHF preference optimization for the first time, stronger motivations and empirical results are needed to highlight the necessity of the proposed method. But, I also think that to strengthen this contribution, For example, as mentioned in lines 141-153, it would be beneficial to show that existing Large Language Models (LLMs) suffer from group robustness issues in various applications involving helpful/harmful instances or domain-specific categories when fine-tuned by RLHF optimization techniques.\n\n* The interpretation of the results in Section 5.2 is insufficient. In Figure 3, GR-IPO improved the performance of all groups, including Group 1. This result contrasts with findings reported in the Group DRO paper [37], where a trade-off between majority and minority group performance is generally observed, with majority group performance typically decreasing to compensate for minority group performance. Therefore, a more thorough analysis or performance comparison is needed to explain this result.\n\n[1] Re-weighting based group fairness regularization via classwise robust optimization, 2023.\n[2] Distributionally Robust Multilingual Machine Translation, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the novel application of our technique to the RLHF setting, our insightful theoretical analysis, and the broad applicability of our proposed algorithm. \n\nNext, we provide answers to **all** the questions posed by the reviewer. \n\n**Regarding theoretical novelty:**\n\nWe appreciate the reviewer's evaluation of the theoretical results in our paper, though some key aspects of our contributions may have been overlooked.\n\nWe agree that the group DRO is widely used, as noted in Lines 56-60. However, to the best of our knowledge, we are the first to apply it for LLM alignment. Specifically, we motivate and formulate the first group robust alignment of LLMs and derive a novel group robust MLE objective (Eq. 6). Additionally, we show that a naive application of group robustness to the LLM policy maximization objective does not offer robustness benefits (Lemma B.2 in Appendix B).\n\nRegarding our analysis, while Proposition 3.1 is well-known in game theory, it guarantees the existence of Nash equilibria for the special case of log-linear policy classes and supports our algorithmic design. Compared to [37,30], we adopt an alternate sampling strategy (see Line-5 of Algorithm-1), which samples groups proportionally to their sizes and assigns uniform weights to all the data points. From a practical perspective, this facilitates stable multi-epoch batch training. However, this might cause samples from smaller groups to be sparsely included in a given batch, thus requiring a compensatory scaling factor of $N/N_g$ in the update of Lines 6 and 8 in Algorithm-1. As a result, our algorithm requires a different proof technique compared to [37,30] in order to attain the convergence guarantees as detailed in Appendix C. \n\nFurthermore, our development of the GR-IPO objective for the log-linear policy class yields a closed-form weighted regression update for the policy parameters rather than a gradient update (Section 4.1 and Appendix C). To the best of our knowledge, this is a novel contribution towards efficient fine-tuning through preferential data. We are happy to further elaborate on these differences in our paper.\n\n**Regarding other potential use cases:**  In Lines 141 - 153, we outline additional potential applications of our approach, to engage readers and inspire future research. Our primary focus is on applying our approach to **pluralistic alignment tasks**, such as ensuring equitable alignment across diverse demographic preferences, as explored in related works (e.g., [1,2]).  While extending our method to **multi-objective** applications is interesting, it falls outside the intended scope of this work (and requires additional computational resources), which we leave for future research.\n\nRegarding helpful and harmful instances, we refer the reviewer to [3], which discuss the trade-offs between optimizing for helpfulness and minimizing harmfulness (Section 5.1) in LLM fine-tuning. Unlike their approach, which uses an additional hyperparameter to control the trade-offs, our approach is “parameter-free”, ensuring equitable performance and proving more effective for multi-objective tasks. \n\n**Regarding improved performance across groups compared to non-robust baselines:**\n\nWe thank the reviewer for their detailed examination of our results. We agree with the reviewer that GRPO improves the worst group's performance, often reducing the average or best group's performance. In contrast, we can observe improved performance across all groups (see Figure-3 top right of our paper). This is only due to the **erroneous inclusion** of an IPO run with zero training steps while plotting the IPO group losses. The corrected plots are now in the attached PDF (Figure R3), where the new plot shows Group-5 loss of IPO and max group loss of IPO (Figure-3 top left) indeed align, unlike in the previous version. The performance figures for GR-IPO were already accurate and unchanged between the old and new plots. With this revised plot, we note that our method, GR-IPO, improves Group 5's performance, albeit at the expense of Group 1 which is consistent with the reviewer’s observations. \n\n**Regarding zero-shot performances of pre-trained Gemma-2B model:**\n\nWe have included the zero-shot performance of the Gemma-2B model in Figure R4 of the attached PDF as requested. Significant group bias is observed in the pre-trained model's performance. However, we would like to emphasize that it is more relevant to view the performances of SFT fine-tuned model as our robust methodology builds upon it. We visualize the SFT performance (on the same data) in Figure 3 (Bottom right in the paper) and note that the SFT model’s degree of group bias is aligned to those of weights and group losses in Figure 3 (Bottom middle and top right). For further comparisons, we kindly refer to Lines (312-319) and our response to Reviewer kmNK on the alignment performance of the fine-tuned vs. base model.\n\n**Regarding fine-tuning more layers:**\n\nThanks for the question. We believe there has been a misunderstanding here. Although our theoretical framework considers only the last layer fine-tuning, in our experiments, we apply the LoRA strategy to fine-tune **all layers of** the model as detailed in the code (also available). To avoid further confusion, we will clearly state this comprehensive fine-tuning approach in our experiments and give details in the Appendix.\n\nWe have addressed all the questions raised by the reviewer. In light of the reviewer’s opinions about the strengths of our work and our detailed rebuttal to the reviewer’s questions, we kindly ask the reviewer to reconsider their score. \n\n[1] Zhao, Siyan et al. \"Group preference optimization: Few-shot alignment of large language models.\"  \n[2] Sorensen, Taylor, et al. \"A roadmap to pluralistic alignment.\"   \n[3] Bai, Yuntao, et al. \"Training a helpful and harmless assistant with reinforcement learning from human feedback.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback and for recognizing the strengths of our work. We appreciate the constructive comments raised by the reviewers and we believe we have addressed all of them in detail further strengthening the validity of our work. 

In summary, the reviewers recognize the following **strengths** of our work\
(i) The criticality of our problem setting in the current LLM alignment paradigm\
(ii) The broad applicability of our approach in the real-world\
(iii) Strong theoretical backing of our methodology and algorithm\
(iv) Consistent experimental setup showcasing the efficacy of our approach

Further, the following shared comments were mentioned by reviewers:

**(i) Regarding the requirement of knowing the number of groups in advance:** \
We agree with the reviewers that our method requires knowing the groups in advance and note that we focus on settings with known groups, which are common in pluralistic alignment datasets and tasks (see [1,2]). Hence, we do not see this as an explicit limitation of our approach. We also provide details on other approaches that can be employed to learn the groups, when they are unknown, in our responses to reviewers qNjA and 7Bpj.
 
**(ii) Regarding alignment performance of finetuned vs base model:** \
As per the reviewers’ requests, we plot and compare the alignment performance of GR-IPO, IPO, and SFT fine-tuned models with the pre-trained model across groups in Figure R2 of the attached PDF. Note that the pre-trained model performs significantly worse in comparison to other methods and, our approach GR-IPO outperforms both IPO and SFT in terms of worst group performance. 
We also provide further details about the plot in our responses to reviewers kmNK and 7Bpj.

**(iii) Regarding fine-tuning more layers:** \
Although our theoretical framework considers only the last layer fine-tuning, in our experiments, we apply the LoRA strategy to fine-tune all layers of the model as detailed in the code (also available). We will clearly state this comprehensive fine-tuning approach in our experiments and give details in the Appendix.

Further, as per the requests of reviewers,\
(i)  We have included the zero-shot performance of the Gemma-2B model in Figure R4 of the attached PDF  
(ii) We provide the results of our ablation study for the synthetic experiments in Figure R1 of the attached PDF 

We have also responded to each reviewer’s questions and comments individually. We believe we have addressed all of them thoroughly and are happy to answer any further questions that are raised by the reviewers. 

[1]. Zhao, Siyan, John Dang, and Aditya Grover. "Group preference optimization: Few-shot alignment of large language models." \
[2] Sorensen, Taylor, et al. "A roadmap to pluralistic alignment." arXiv preprint arXiv:2402.05070 (2024).

Meta-review: This paper brings attention to the existence of diverse preferences across different groups, proposing a robust RLHF method. Defining the optimization problem with the weighted loss for each group, the authors have discovered that the problem can be reduced to a minimax optimization problem. The minimax optimization problem is a convex-concave game, and by combining MWU and gradient descent updates, it can be guaranteed from classical results that the average iterate will converge to the equilibrium. When the policy is log-linear, essentially one can compute the best response of one player each round analytically (as a classical regression), resulting in a simplified algorithm. Through synthetic experiments and real-world data experiments, the authors demonstrate that their method performs best for the group that would likely have the worst loss, suggesting the significance of the approach in the real world. There was a concern from a reviewer about other applications and motivations, and questions about improved performance seen across all groups (in fact, the authors included an incorrect plot in the initial manuscript). In their rebuttal, the authors presented the correct experimental results, which were confirmed to be consistent. Overall, while the technical/experimental contributions are not particularly significant, the contribution of bringing the concept of group robustness to LLM alignment and conducting appropriate analysis is substantial, which is generally recognized by the reviewers.
Decision: Accept (poster)