Title: Group Robust Preference Optimization in Reward-free RLHF
Venue: NeurIPS 2024
Accepted: yes
Keywords: RLHF

DPO

Robust Alignment
Abstract: Adapting large language models (LLMs) for specific tasks usually involves fine-tuning through reinforcement learning with human feedback (RLHF) on preference data. While these data often come from diverse labelers' groups (e.g., different demographics, ethnicities, company teams, etc.), traditional RLHF approaches adopt a "one-size-fits-all" approach, i.e., they indiscriminately assume and optimize a single preference model, thus not being robust to unique characteristics and needs of the various groups. To address this limitation, we propose a novel Group Robust Preference Optimization (GRPO) method to align LLMs to individual groups' preferences robustly. Our approach builds upon reward-free direct preference optimization methods, but unlike previous approaches, it seeks a robust policy which maximizes the worst-case group performance.  To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We theoretically study the feasibility of GRPO and analyze its convergence for the log-linear policy class. By fine-tuning LLMs with GRPO using diverse group-based global opinion data, we significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies compared to non-robust baselines.

Group Robust Preference Optimization in Reward-free RLHF

Abstract

Adapting large language models (LLMs) for specific tasks usually involves finetuning through reinforcement learning with human feedback (RLHF) on preference data. While these data often come from diverse labelers' groups (e.g., different demographics, ethnicities, company teams, etc.), traditional RLHF approaches adopt a "one-size-fits-all" approach, i.e., they indiscriminately assume and optimize a single preference model, thus not being robust to unique characteristics and needs of the various groups. To address this limitation, we propose a novel Group Robust Preference Optimization (GRPO) method to align LLMs to individual groups' preferences robustly. Our approach builds upon reward-free direct preference optimization methods, but unlike previous approaches, it seeks a robust policy which maximizes the worst-case group performance. To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We theoretically study the feasibility of GRPO and analyze its convergence for the log-linear policy class. By fine-tuning LLMs with GRPO using diverse group-based global opinion data, we significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies compared to non-robust baselines.

1 Introduction

As the usage of large language models (LLMs) has grown in recent years, the question of their alignment has come to the forefront. Their remarkable capability to address a wide range of tasks (Radford et al. [36]) stems from pre-training on a self-supervised objective over internet-scale text. This vast internet-scale content, however, carries a higher risk of biases, inaccuracies, and controversial content than smaller, curated datasets. Thus, ensuring that the model's responses and behaviors correspond to human intentions and values is crucial.

Typical approaches to alignment [11, 34, 37] involve gathering preference feedback from human labelers to train models that reflect their desires. Such approaches often treat individual preferences as samples from a broader preference distribution. However, this perspective often oversimplifies the complex reality that human societies consist of numerous distinct groups (e.g., different demographics, ethnicities, company teams, etc.), each with their own set of preferences that can significantly

Alignment

performance

G2

G2

R1

R2

R1

R2

Supervised

Fine-Tuning

(SFT)

Pref.

optimization

DPO / IPO

Prompt:

Should social media

platforms enforce stricter

regulations?

R1:

Yes, social media platforms should enforce

stricter regulations to combat misinformation.

R2:

No, they should not enforce stricter regulations

because it risks infringing on free speech.

Alignment

performance

Pre-trained

LLM

GRPO

(group-robust)

G2

G2

Adaptive group weights

(loss influenced)

Pre-trained

LLM

Supervised

Fine-Tuning

(SFT)

Gradient updates

scaled by group

weights

diverge. Consequently, prevalent alignment strategies tend to adopt a "one-size-fits-all" model and disproportionately favor the preferences of the majority group, often at the expense of minority groups and their preferences, as illustrated in Figure 1.

To improve alignment performance for even the most disadvantaged groups, we propose to robustly solve the problem of diverse group preferences by (i) including group information in the context of the LLM and (ii) optimizing against the worst-case alignment performance across all groups. We develop policies that guarantee equitable performance across all groups, ensuring that no group is disproportionately disadvantaged due to inherent biases or imbalances in the training data.

Related work. The established process for alignment of LLMs using Reinforcement Learning from Human Feedback (RLHF) is set out in [45, 60] and [34]. The RLHF fine-tuning process consists of learning a reward model from human comparisons between responses to a given prompt, using the Bradley-Terry model [5]. Then, one performs policy optimization using Proximal Policy Optimization [40] to learn a policy that maximizes the learned reward function. For a comprehensive overview and perspective of the RLHF topic, we refer the reader to [24, 7, 23].

Due to the challenges of tuning PPO and the vulnerability of reward models ([50, 37, 17, 49]), alternative approaches to PPO-based RLHF have been proposed, including rejection sampling fine-tuning [14, 18, 49, 30] and conditional supervised fine-tuning [20, 54, 9]. In particular, Rafailov et al. [37] introduce Direct Preference Optimization (DPO), which optimizes policies directly based on human preferences, avoiding the need for a separate reward model. This approach simplifies training and reduces reward overfitting. Other studies, such as [1, 58, 47, 43, 16], propose novel reward-free RLHF methods, with some bypassing preference datasets altogether ([16, 6]). We utilize a reward-free framework similar to [37, 1], however, unlike previous works that assume a single preference distribution, we consider multiple preference distributions from diverse groups. Further, we aim to robustly fine-tune the LLM to ensure minimal disparity in performance across all groups. Other studies addressing robustness in preference optimization include [21] and [26]. However, these works primarily focus on different aspects of robustness, such as robustness to noise and resilience against out-of-preference data.

Robust language modeling techniques have been studied by [33, 52] to optimize performance of language models over a wide-range of topics. They consider robust pre-training of language models based on the group Distributionally Robust Optimization (DRO) approach. A concrete theoretical study of the group DRO approach was performed in [39] and applied to vision problems. These are designed by extending previous minimax algorithm for solving DRO from [31]. In the RLHF setup, [3] consider weighting of loss from different topics (harmless vs helpful) for robust reward learning. Also, Chakraborty et al. [8] consider robust policy optimization by learning multiple reward functions corresponding to sub-populations and learning a robust policy w.r.t. the learned

rewards. Differing from these works, we embed group robustness directly into the reward-free tuning paradigm. We provide an concrete algorithm that adaptively weighs the loss for different groups and optimizes for a policy that minimizes the weighted loss. Further, our algorithm employs a novel gradient estimator tailored to the group robust DPO problem.

In the non-robust setup, [57] also explore group-based preference learning with LLMs by including group information in prompts with a transformer module that is trained optimally to choose an example sequence of prompts, LLM responses, and group preferences for in-context learning. [49] consider alignment with user preferences assuming that each user has varied importance over the distinct metrics in their multi-objective reward model. In contrast, we are not modeling multi-reward objectives but consider a reward-free setting. And, our methodology directly models each group's preferences through a group-dependent latent reward model where the group dependency is injected through the prompt. Further, unlike the non-robust problem setups of [57, 49], we consider the robust alignment problem optimizing for the worst group performance. We detail other related works extensively in Appendix A.

Main Contributions. The following are the main contributions of this work: (i) We present GRPO, the group robust formulation of Direct Preference Optimization (DPO) [37], wherein we augment the context of the LLM with the group information, and pose the problem as a robust optimization problem to minimize the worst-case loss amongst the diverse groups. We also show that a naive application of group robustness to the LLM policy maximization objective does not offer robustness benefits. To the best of our knowledge, this is the first study that focuses on group robustness in RLHF preference optimization; (ii) We analyze the theoretical aspects of GRPO by examining the convergence and the feasibility of finding optimal solutions within the log-linear policy class. (iii) We present a tailored algorithm to tackle this robust optimization challenge, providing convergence guarantees for certain loss functions; (iv) We show the versatility of our approach by demonstrating how our algorithm can be utilized with other reward-free preference optimization methods such as Identity Preference Optimization (IPO) [1]. In particular, for the GR-IPO objective optimized over the log-linear policy class, we derive a closed-form weighted regression update for the policy parameters rather than a gradient update. To the best of our knowledge, this is a novel contribution towards efficient fine-tuning through preferential data; (v) Our empirical evaluations across synthetic datasets, real-world data, and publicly available LLMs show that the proposed GRPO significantly improves performance for the worst-performing groups, reduces loss imbalances across groups, and increases probability accuracies compared to non-robust baselines.

2 Background

We address the challenge of fine tuning a large language model (LLM) to align with user preferences. This process usually follows the Reinforcement Learning from Human Feedback (RLHF) protocol, using either an explicit reward model ([3, 34, 45, 60]) or an implicit reward model ([37]). RLHF typically comprises three key phases: (i) supervised fine-tuning of an initial (pre-trained) largelanguage model, (ii) reward learning, and (iii) RL fine-tuning.

In the supervised fine-tuning phase (SFT), the goal is to fine-tune a pre-trained LLM on a specific highquality dataset suited for the downstream task of interest. It results in a probabilistic model expressing the probability of the response y given a prompt x as π ref ( y | x ) . Subsequently, in the reward learning phase , the goal is to learn a reward model from a dataset of prompts x and responses y w , y l , with y l ≺ y w | x meaning that human labellers preferred y w over y l . It is typically assumed that preferences follow some choice models with an unknown reward (utility) r ∗ ( x, y ) function. A popular model is the Bradley-Terry model [5] that assumes the preference distribution p admits the following form :

Based on the above model, a maximum likelihood estimate of the reward function is obtained as:

where σ ( · ) is the sigmoid function and D represents the dataset consisting of { ( x, y w , y l ) } .

Then, in the RL fine-tuning phase the objective is to train a policy π that maximizes the learned reward function. Simultaneously, the policy should stay closely aligned with the reference, π ref , as

quantified by the KL divergence, leading to the following KL-regularized optimization problem:

Direct Preference Optimization (DPO). The recent approach proposed by [37] exploits the closed-form solution of the problem in Equation (3) and sidesteps the explicit modelling of rewards to directly optimize the policy. Specifically, under the Bradley-Terry preference model, the reward function can be expressed directly in terms of the optimal policy π ∗ as follows:

for a partition function Z ( x ) = ∑ y π ref ( y | x ) exp( 1 β r ( x, y )) . Via a change of variable, finding the optimal reward function in Equation (2) is equivalent to finding the optimal policy π ∗ utilizing the given set of preference data D . With a slight abuse of notation, we use π to denote π ∗ . Denote h π ( x, y w , y l ) := log( π ( y w | x ) π ref ( y w | x ) ) -log( π ( y l | x ) π ref ( y l | x ) ) . Then, Equation (2) translates into the DPO loss:

With a parameterized policy model π θ , minimizing the DPO loss involves calculating the gradient over θ using backpropagation and the log-probabilities of each completion, y w and y l , given the prompt x for both the policy π θ and the reference policy π ref .

3 Group Robust Preference Optimization

In this section, we discuss Group Robust Preference Optimization (GRPO), i.e., instead of learning a reward function that maximizes the likelihood, we aim to derive (implicitly) a robust reward function and subsequently learn a robust DPO policy.

Group Preferences. Suppose that preferences come from an underlying latent reward r ∗ ( x, y, g ) , with g ∈ G = { 1 , 2 , . . . , K } indexing the groups. When group information is available (e.g., as a text), we can represent the reward as r ∗ ( x g , y ) , where x g = x ⊕ g denotes merging 1 of the prompt with group information (e.g., string concatenation). We continue to apply a Bradley-Terry model as described in Equation (1), substituting x with x g . Moreover, we assume access to a collective dataset D = ⋃ K g =1 D g where D g = { ( x ( i ) g , y ( i ) w , y ( i ) l ) } N g i =1 with the available group information. Additionally, our dataset accommodates the exposure of different groups to identical prompts, meaning that the same x can appear across various groups g in our dataset, and these groups may favor different responses y .

Given such D , although one may obtain a common reward model using Equation (2), it could result in poor generalization for particular groups, especially with significant group-wise disparities in the data (see Figure 1). Such disparities might stem from imbalanced data across groups or difficulties associated with learning different groups.

GRPO Objective. Consequently, we propose to measure the alignment of the reward model on the worst-case group loss:

Incorporating the reward expression from (Equation (4)) into (Equation (6)), we establish the group robust preference optimization (GRPO) objective for a specified policy π :

Leveraging the equivalent formulation of maximizing over discrete set, the GRPO problem becomes

where ∆ K -1 represents the ( K -1) -dimensional simplex of probabilities. 2 The inner maximization becomes a linear programming over simplex such that α represents the weights of groups. In addition, it forms a two-player zero-sum game (see Section 3.1), where the policy π and α act as opponents with inversely related payoffs. The DPO loss (logistic log loss) in Equation (7) can be replaced with alternatives like hinge or squared loss (see [47]). We label this objective GR-DPO when using DPO loss, and explore GRPO with squared loss in Section 4.1.

Applications. In this study, we do not assume any specific distribution for groups D g . The collection of prompts per group, P x g , may have varying degrees of overlap. The GRPO framework accommodates both distinct and overlapping prompt scenarios across different groups.

Apart from human groups, GRPO can be useful in scenarios where groups D g represent distinct tasks or topics within preference datasets, like helpful/harmful, truthful/unanswerable instances, or domainspecific categories (e.g., math, physics, chemistry). Typically, these prompt distributions {P x g } N g =1 are disjoint, and GRPO seeks to optimize performance even across the most challenging categories.

GRPO is also applicable in scenarios where groups reflect diverse user preferences for a shared set of prompts, with the goal of achieving equitable performance across user groups. This contrasts with non-robust DPO, which aims to optimize preferences on average and might overlook minority groups.

Lastly, we acknowledge that the max-min objective of Equation (8) might be overly conservative, potentially degrading average performance. We explore a more balanced approach between worst-case and standard preference optimization objective in Appendix B.4.

3.1 Further Discussion and Insights

This section provides two insights regarding the GR-DPO loss in Equation (8).

Log-linear policy class. The zero-sum game perspective allows us to explore the presence of a Nash equilibrium, serving as a benchmark for convergence during the policy optimization process. Given that the domain of α is a simplex ∆ K (in Equation (8)), we further define a parameterized policy class Π θ for the policy π θ . We assume that the parameterized policy π θ is of the form π θ ( y | x ) = exp f θ ( x,y ) ∑ y ∈Y exp f θ ( x,y ) , where f θ is a linear function or a neural network, and θ belongs to a convex set Θ .

In LLM fine-tuning, sometimes practitioners concentrate on modifying solely the final layer. It corresponds to a linear function, f θ ( x, y ) = θ T ϕ ( x, y ) , with ϕ ( x, y ) denoting the embedding derived from the language model removing its last layer, and θ as the parameters of the last layer. When applying this linear parameterization, in conjunction with a uniform reference policy π ref, the robust objective outlined in Equation (8) is as follows (details in Appendix B.1):

The objective defined in Equation (9) is concave with respect to α and convex with respect to θ . This structure allows the invocation of the minimax theorem for convex-concave functions ([42]) to assert the existence of a Nash equilibrium.

Proposition 3.1. Under log-linear parameterization of the policy class, there exists a Nash equilibrium for the group robust direct preference optimization problem in Equation (9) .

Robust policy optimization. The earlier derivation for the GR-DPO objective L GR -DPO ( π ) relies on incorporating robustness in the reward modeling step (in Equation (7)) while using the solution to the non-robust KL-regularized reward maximization objective in Equation (4).

Interestingly, we can obtain the identical expression for L GR -DPO ( π ) if incorporating robustness in the KL-regularized reward maximization objective and using the reward function learnt in a non-robust way. Consider the robust KL-regularized reward maximization

The following proposition characterizes such an invariant property.

Algorithm 1 Mirror Descent for Group Robust Preference Optimization (GRPO)

1: Initialize: Step size η α for group weights α , step size η θ for policy π with weights θ , initial weights θ (0) of the policy and weights over each group α (0) , Projection operator P Θ 2: Input: Dataset D with size N = |D| , group size N g for g = { 1 , 2 , · · · , K } , loss l ( π θ ; · ) 3: for t = 1 , . . . , T do 4: α ′ ← α ( t -1) 5: g ∼ Categorical( N 1 /N, · · · , N K /N ) , ( x g , y w , y l ) ∼ D g 6: α ′ g ← α ′ g exp η α ( N · l ( π θ ( t -1) ;( x g ,y w ,y l )) N g ) // Update weights for group g 7: α ( t ) ← α ′ / ∑ g ′ α ′ g ′ // Renormalize α 8: θ ( t ) ← P Θ ( θ ( t -1) -η θ ( Nα ( t ) g ∇ θ l ( π θ ( t -1) ;( x g ,y w ,y l )) N g )) // Use α to update θ 9: end for 10: Return: Output the robust policy π ( θ ( T ) )

Proposition 3.2. Substituting the closed-form solution of the robust KL-regularized policy maximization problem (Equation (10) ) into the robust reward maximization objective in Equation (6) leads to the same group robust DPO loss L GR -DPO in Equation (8) .

The analysis leverages the fact that the optimal policy of Equation (10) is identical to the solution of the non-robust KL-regularized reward maximization in Equation (4) and is derived in Appendix B.2.

4 Algorithm

In this section, we discuss the policy optimization algorithm for solving the group robust DPO problem in Equation (8). In particular, we aim to design an algorithm that performs updates in the parameterized space Θ ⊂ R d , i.e., updating θ of the parameterized policy π θ . Leveraging the perspective of the 2-player zero-sum game, we propose an alternating updating algorithm wherein one updates α and θ alternatively. We summarize the overall approach in Algorithm 1, which we discuss and analyze next.

We employ the DPO loss l ( π θ ; · ) = log ( σ ( βh π θ ( · ))) (Equation (5)) in Algorithm 1, however, our algorithm can support other preference optimization losses (see Section 4.1). The algorithm performs a gradient descent type update on θ and a deterministic mirror ascent on α using a Bregman divergence with the distance generating function as the KL divergence. Since the α lies in a simplex and the objective is linear, the update of α becomes multiplicative weights update with renormalization to a simplex via softmax (see Nemirovski et al. [32] for details). Further, the weights α are determined by the cumulative losses l ( π θ ; · ) accrued by each group, ensuring that groups with higher cumulative losses get higher weights. The size of the group N g appears as the empirical distribution D g involves N g . We call it alternating update as the updated α t is used in the update from θ t -1 to θ t . In particular, the gradient descent type update on θ is weighted by α in order to orient the update towards groups with higher losses. The projection operator P Θ ensures the updated θ t lies within Θ .

What does the weighted DPO update do? In Line 9 in Algorithm 1, the algorithm performs parameter updates based on the weighted gradients. By using the DPO loss, i.e., l ( π θ ; · ) = log ( σ ( βh π θ ( · ) ) (see Equation (5)), we obtain the following gradient update expression ignoring the N/N g constant

The final term plays the critical role of enhancing the likelihood of the preferred response while simultaneously diminishing the likelihood of the rejected response. This adjustment is proportional to the disparity in rewards between the two responses. Moreover, the inclusion of α g is pivotal for ensuring group robustness. This coefficient scales the gradient w.r.t. θ based on the cumulative loss previously received by all samples within a specific group. Such a mechanism ensures that the model's focus is increasingly directed towards groups that have historically suffered higher losses. Additionally, the scaling factor N g guarantees that groups with a smaller volume of data do not face a disadvantage.We defer further details in obtaining the gradient update expression to Appendix B.3.

We demonstrate the global convergence with the following proposition.

Proposition 4.1. Suppose that the loss l ( · ; ( x g , y, y ′ )) is non-negative, convex, B ∇ -Lipschitz continuous, and bounded by B l for all ( x g , y, y ′ ) ∈ X ⊕ G × Y × Y and ∥ θ ∥ 2 ≤ B Θ for all θ ∈ Θ with convex Θ ⊂ R d . The error of the average iterate of Algorithm 1, i.e., π ¯ θ (1: T ) = 1 T ∑ T t =1 θ t , satisfies

We defer the proof of this proposition to Appendix E. The analysis follows from an adaptation of the analysis in Nemirovski et al. [32] for the proposed sampling strategy in Algorithm 1 3 . We note that when fine-tuning only the final layer of a LLM, the output policy exists within the log-linear policy class (see Section 3.1), and the corresponding loss function satisfies the assumptions in Proposition 4.1 (see Lemma E.1).

4.1 Group Robust Identity Preference Optimization

The standard regularized reward maximization objective (Equation (3)) in DPO [37], tends to overlook the KL-regularization and learn deterministic policies. This learned policy assigns preference probability one to winning responses in the data which is often not realistic (see [1][Section 4.2] and Appendix C). Recently, Azar et al. [1] show that the standard regularized reward maximization objective (Equation (3)) in DPO [37] tends to overlook the KL-regularization and learn deterministic policies (see [1, Section 4.2] and Appendix C). They thus propose an alternative approach called Identity Preference Optimization (IPO) that is more likely to learn a randomized policy which assigns appropriate probability to the preferred response and prevents overfitting. Following a similar derivation as we did for group robust DPO with details given in Appendix C, we develop the corresponding group robust IPO (GR-IPO):

For the log-linear policy class (introduced in Section 3.1), the objective function simplifies to

To solve the GR-IPO above, it suffices to use Algorithm 1 with slight modifications, see Algorithm 2 in Appendix C. In particular, the update of θ is replaced by a weighted regression update:

For fixed α , we show (in Appendix C) that such an update admits a closed-form solution:

where g ( i ) is the group of each sample i , N g ( i ) is the number of samples in group g ( i ) and 1 is a column vector of ones of dimension N . Here S is a matrix S := [ ( ϕ ( x (1) g , y (1) w ) -ϕ ( x (1) g , y (1) l )) T , . . . , ( ϕ ( x ( N ) g , y ( N ) w ) -ϕ ( x ( N ) g , y ( N ) l )) T ] . Each row of S represents the difference in feature mappings ϕ of the preferred and less preferred response for each prompt. The group robust IPO (GR-IPO) algorithm is presented in Appendix C, and its empirical results are shown in Section 5.

5 Experiments

In this section, we study the empirical performance of our proposed Algorithm 1 on synthetic and real-world datasets 4 . First, we simulate multi-group data disparities by varying the size and

0

2500

5000

7500

10000

Iterations

0

.

5

0

.

6

Value

DPO Max Validation Group Loss

DPO

IS-DPO

GR-DPO

0

25

50

75

100

Iterations

18

20

Value

IPO Max Validation Group Loss

IPO

IS-IPO

GR-IPO

IPO

IS-IPO

GR-IPO

DPO

IS-DPO

GR-DPO

Methods

0

.

00

0

.

05

0

.

10

Value

preference distributions of two synthetic groups. In the real-world setup, we study the alignment of an LLM to the real preferences of people from various countries. We examine whether GRPO aligns the LLM in a more equitable manner to reduce discrepancies in alignment among various groups. Finally, we demonstrate that performance is improved by explicitly addressing the grouped nature of the data during alignment.

5.1 Synthetic Experiments

We evaluate the performance of Algorithm 1 using synthetically generated group preference data for the loss function l ( π θ ; · ) - either DPO loss or IPO loss and denote them as GR-DPO and GR-IPO, respectively. We compare them against vanilla DPO and IPO ([37, 1]), and the importance-sampling (IS) variants of DPO and IPO (where the loss of each datapoint is inversely weighted by its group data size).

Experimental Setup. Our experiments are designed to analyze settings where there exist multiple groups with distinct characteristics. We adapt the standard (non-group based) experimental setup proposed by [26] for the group preferences setting by incorporating group information into the reward function r : X × Y × G → R . Here, X represents a two-dimensional state space [0 , 1] × [0 , 1] , Y denotes a discrete action space { 0 , 1 , 2 , 3 , . . . , n } , and G signifies a discrete group space { 0 , 1 , 2 , . . . , K } . The reward function, defined by the group-dependent feature vector ϕ ( x, y, g ) and parameter vector θ g , is given as r ( x, y, g ) := ⟨ ϕ ( x, y, g ) , θ g ⟩ , while the feature vectors ϕ ( x, y, g ) have a coordinate-flipped relationship and are defined in Appendix D.1.

We consider the following scenarios: (i) Groups are imbalanced in terms of size but have the same distribution over responses, (ii) Groups are balanced in terms of size but have different response distributions, and (iii) Groups are imbalanced in terms of size and also have different response distributions. Note that having different response distributions leads to a difference in the difficulty of learning, since groups with responses distant from each other (in terms of rewards or preference probabilities) are typically more distinguishable and easier to learn. We discuss in Appendix D.2 how we generate these three scenarios.

Implementation. Leveraging the linearity in the reward model, we utilize a log-linear policy class parameterized by θ : π θ ( y | x ) = exp ⟨ ϕ ( x,y,g ) ,θ ⟩ ∑ y ′ ∈Y exp ⟨ ϕ ( x,y ′ ,g ) ,θ ⟩ . We run Algorithm 1 for both DPO and IPO loss relative to the policy class detailed above with a dataset of 300 action pairs with preferences.

Evaluation Metrics. We use the following criteria to assess the performance of the algorithms:

Max Validation Loss. For each group g , with preference data denoted as ( x i , y i w , y i l , g ) N g i =1 , where N g is the number of data points in the group, we compute the DPO/IPO validation loss separately for each group and identify the maximum loss among them in each run.

Max Reward Error. This metric compares the true reward of the optimal action determined by θ ∗ g with that of the action deemed optimal by estimate ˆ θ for each group, and identifies the maximum error across all groups in each run. In particular, for data in the form ( x i , g ) N g i =1 , which includes only states and groups, we calculate reward errors for every group g as follows: E ( x,g ) ∼ ( x i ,g ) Ng i =1 [max y ⟨ ϕ ( x, y, g ) , θ ∗ g ⟩ - ⟨ ϕ ( x, arg max y ⟨ ϕ ( x, y, g ) , ˆ θ ⟩ , g ) , θ ∗ g ⟩ ] .

Results. We present the average performance of Algorithm 1 (error bars over 20 seeds) alongside baseline methods in Figure 2 for scenario (iii) , while scenarios (i) and (ii) are Figures 4 and 5

in Appendix D.2 due to space constraints. Our findings indicate that the robust methods consistently surpass both vanilla and importance-sampling approaches. Notably, the robust methods demonstrate significant superiority in uneven group scenarios, where the importance-sampling technique falls short as it exclusively deals with data imbalance.

5.2 Global Opinion Experiments

For the real-data experiments, we consider the survey dataset GlobalOpinionQA ([15]) and the publicly available Gemma-2B model [48]. 5 The data contains multiple choice questions answered by participants from various countries, amounting to 2,554 questions covering various topics, including politics, media, technology, religion, race, and ethnicity. For each question, the dataset provides a probability vector over the choices, signifying the percentage of people from a particular country choosing each option. Note that this probability vector would be different for different countries. Hence, the goal is to align the LLM to the probability vector corresponding to each country in a robust manner.

We consider the following five countries in the dataset: Nigeria, Egypt, India, China and Japan, with data sizes 572, 570, 376, 309, and 712, respectively. We construct our training set as follows: For the SFT training, we choose the best option (the choice with the highest probability) as the target. For both IPO and GR-IPO training, we consider the best option as the winning response and another randomly chosen option as the losing response. We outline the exact prompt we use in Appendix D.

We run the SFT training for one epoch over the training data on the pre-trained Gemma-2B model. For both IPO/GR-IPO training we use the AdamW [27] optimizer with adaptive learning rates. For SFT/IPO/GR-IPO training, we apply the LoRA strategy to fine-tune all layers of the model. We then evaluate both the methods based on the worst group loss and accuracy. Here, the loss refers to the IPO loss for each group and the accuracy refers to the percentage of winning response and losing response pairs correctly ordered by the learned preference function (Equation (35)). We defer further training and hyperparameter details to Appendix D.

Results. We present the average performance of GR-IPO over five seeds alongside IPO in Figure 3 (top plots). Our findings indicate that GR-IPO outperforms IPO in terms of maximum group loss and minimum group reward accuracies. Moreover, GR-IPO effectively reduces the imbalance in loss values among different groups. Additionally, we observe an improvement in log-probability accuracies (which measure if the probability assigned by the fine-tuned model is higher for the winning response compared to the losing response) for both IPO and GR-IPO, with GR-IPO demonstrating better alignment for the worst-performing group compared to IPO.

Insights. We further note that the worst-performing groups are Groups-2,5, as shown in Figure 3. GR-IPO improves the loss for these groups by assigning more weight to them, as illustrated in Figure 3 (bottom middle plot). Additionally, we plot the initial log-probability accuracies for different groups in Figure 3 (bottom right plot), assessing how accurately the SFT model classifies the winning versus losing response for different groups. It is evident that Groups-2,5 are already underperforming. Given that SFT training converges within one epoch without any discrepancies between groups, this indicates that the base LLM inherently struggles with classifying responses for Groups-2,5. However, by employing GR-IPO, we have mitigated the imbalance in the performance of the fine-tuned model.

6 Conclusions

Weformalize the problem of robustly aligning an LLM to preference distributions from diverse groups. To tackle the same, we introduced GRPO, a group robust formulation of reward-free RLHF, aiming to minimize worst-case loss among groups. We explored the theoretical aspects of GRPO and demonstrated its improved robust alignment performance through various experiments. We believe our approach will be highly valuable for future tailored LLM fine-tuning, specifically aimed at aligning with the needs of diverse teams and user groups. In a broader context, it holds promise for mitigating biases and discrepancies across various societal groups encountered in the task-specific adaptation of LLMs.

Limitations. When the dataset is balanced among groups and difficulty levels are comparable, our GRPO approach does not offer a significant advantage over standard reward-free RLHF algorithms. In addition, minimax methods often improve the worst group's performance at the cost of reducing the average or best group's performance. Our proposed GRPO formulation (see Equation (8)),

GR-IPO

IPO

Methods

1600

1800

2000

Value

Max Group Loss (Test)

GR-IPO

IPO

Methods

0

.

70

0

.

75

Value

Min Reward Accuracies (Test)

Group 1

Group 2

Group 3

Group 4

Group 5

Groups

1000

1500

2000

Value

Group Loss (Test)

GR-IPO

IPO

0

10000

20000

30000

Iterations

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

Value

Min Log-Prob Accuracies (Test)

GR-IPO

IPO

0

10000

20000

30000

Iterations

0

.

1

0

.

2

0

.

3

0

.

4

Value

Group Weights (Train)

GR-IPO Group 1

GR-IPO Group 2

GR-IPO Group 3

GR-IPO Group 4

GR-IPO Group 5

Group 1

Group 2

Group 3

Group 4

Group 5

Groups

0

.

5

0

.

6

0

.

7

0

.

8

Value

Log-Prob Accuracies (Start)

stemming from a minimax framework, will comply with the same property. Hence, in scenarios where optimizing worst-case performance is less critical, we define a trade-off parameter to balance between the worst-case performance and the average performance. This modified objective and the necessary algorithmic changes are elaborated in Appendix B.4. The appropriate tuning of the trade-off parameter for the specific application remains a subject for future investigation. Further, we focus on settings with known groups, which are common in pluralistic alignment datasets and tasks (see [57, 44]). When groups are unknown, one can still undertake several approaches such as clustering, representation learning, feature analysis, expert consultations, etc., to help uncover group structures in the data.

7 Acknowledgments

PGS was gratefully supported by ELSA (European Lighthouse on Secure and Safe AI) funded by the European Union under grant agreement No. 101070617. YH was supported as a part of NCCR Automation, a National Centre of Competence (or Excellence) in Research, funded by the Swiss National Science Foundation (grant number 51NF40_225155). IB was supported by the EPSRC New Investigator Award EP/X03917X/1; the Engineering and Physical Sciences Research Council EP/S021566/1; and Google Research Scholar award. SSR was supported by Department of Electronic and Electrical Engineering and the Institute of Communications and Connected Systems at UCL. The authors would like to thank William Bankes, Seongho Son, Matthieu Zimmer, Afroditi Papadaki, Eduardo Pignatelli, and Nagham Osman for the useful discussion.

References

Mohammad Gheshlaghi Azar et al. 'A general theoretical paradigm to understand learning from human preferences'. In: International Conference on Artificial Intelligence and Statistics . 2024.

Yuntao Bai et al. 'Constitutional ai: Harmlessness from ai feedback'. In: arXiv preprint arXiv:2212.08073 . 2022.

Yuntao Bai et al. 'Training a helpful and harmless assistant with reinforcement learning from human feedback'. In: arXiv preprint arXiv:2204.05862 . 2022.

William Bankes et al. 'REDUCR: Robust Data Downsampling Using Class Priority Reweighting'. In: Neural Information Processing Systems . 2024.

Ralph Allan Bradley and Milton E Terry. 'Rank analysis of incomplete block designs: I. The method of paired comparisons'. In: Biometrika . JSTOR, 1952.

Tianchi Cai et al. 'ULMA: Unified Language Model Alignment with Demonstration and Point-wise Human Preference'. In: arXiv preprint arXiv:2312.02554 . 2023.

Stephen Casper et al. 'Open problems and fundamental limitations of reinforcement learning from human feedback'. In: Transactions on Machine Learning Research . 2023.

Souradip Chakraborty et al. 'MaxMin-RLHF: Alignment with Diverse Human Preferences'. In: International Conference on Machine Learning . 2024.

Lili Chen et al. 'Decision transformer: Reinforcement learning via sequence modeling'. In: Neural Information Processing Systems . 2021.

Zixiang Chen et al. 'Self-play fine-tuning converts weak language models to strong language models'. In: International Conference on Machine Learning . 2024.

Paul F Christiano et al. 'Deep reinforcement learning from human preferences'. In: Neural Information Processing Systems . 2017.

Josef Dai et al. 'Safe rlhf: Safe reinforcement learning from human feedback'. In: International Conference on Learning Representations . 2024.

Nirjhar Das et al. 'Provably Sample Efficient RLHF via Active Preference Optimization'. In: arXiv preprint arXiv:2402.10500 . 2024.

Hanze Dong et al. 'Raft: Reward ranked finetuning for generative foundation model alignment'. In: Transactions on Machine Learning Research . 2023.

Esin Durmus et al. 'Towards measuring the representation of subjective global opinions in language models'. In: Conference on Language Modeling . 2024.

Kawin Ethayarajh et al. 'Kto: Model alignment as prospect theoretic optimization'. In: International Conference on Machine Learning . 2024.

Leo Gao, John Schulman, and Jacob Hilton. 'Scaling laws for reward model overoptimization'. In: International Conference on Machine Learning . 2023.

Caglar Gulcehre et al. 'Reinforced self-training (rest) for language modeling'. In: arXiv preprint arXiv:2308.08998 . 2023.

Jiwoo Hong, Noah Lee, and James Thorne. 'Reference-free monolithic preference optimization with odds ratio'. In: arXiv preprint arXiv:2403.07691 . 2024.

Jian Hu et al. 'Aligning language models with offline reinforcement learning from human feedback'. In: arXiv preprint arXiv:2308.12050 . 2023.

Shawn Im and Yixuan Li. 'Understanding the Learning Dynamics of Alignment with Human Feedback'. In: International Conference on Machine Learning . 2024.

Kaixuan Ji, Jiafan He, and Quanquan Gu. 'Reinforcement Learning from Human Feedback with Active Queries'. In: arXiv preprint arXiv:2402.09401 . 2024.

Timo Kaufmann et al. 'A survey of reinforcement learning from human feedback'. In: arXiv preprint arXiv:2312.14925 . 2023.

Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. 'Entangled preferences: The history and risks of reinforcement learning and human feedback'. In: arXiv preprint arXiv:2310.13595 . 2023.

Harrison Lee et al. 'Rlaif: Scaling reinforcement learning from human feedback with ai feedback'. In: arXiv preprint arXiv:2309.00267 . 2023.

Ziniu Li, Tian Xu, and Yang Yu. 'Policy Optimization in RLHF: The Impact of Out-ofpreference Data'. In: arXiv preprint arXiv:2312.10584 . 2023.

Ilya Loshchilov and Frank Hutter. 'Decoupled weight decay regularization'. In: International Conference on Learning Representations . 2019.

Viraj Mehta et al. 'Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration'. In: 2023.

Rémi Munos et al. 'Nash learning from human feedback'. In: International Conference on Machine Learning . 2024.

Reiichiro Nakano et al. 'Webgpt: Browser-assisted question-answering with human feedback'. In: arXiv preprint arXiv:2112.09332 . 2021.

Hongseok Namkoong and John C Duchi. 'Stochastic gradient methods for distributionally robust optimization with f-divergences'. In: Neural Information Processing Systems . 2016.

Arkadi Nemirovski et al. 'Robust stochastic approximation approach to stochastic programming'. In: Society for Industrial and Applied Mathematics . 2009.

Yonatan Oren et al. 'Distributionally robust language modeling'. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . 2019.

Long Ouyang et al. 'Training language models to follow instructions with human feedback'. In: Neural Information Processing Systems . 2022.

Jing-Cheng Pang et al. 'Language model self-improvement by reinforcement learning contemplation'. In: International Conference on Learning Representations . 2023.

Alec Radford et al. 'Language models are unsupervised multitask learners'. In: OpenAI blog . Vol. 1. 8. 2019, p. 9.

Rafael Rafailov et al. 'Direct preference optimization: Your language model is secretly a reward model'. In: Neural Information Processing Systems . 2024.

Corby Rosset et al. 'Direct nash optimization: Teaching language models to self-improve with general preferences'. In: arXiv preprint arXiv:2404.03715 . 2024.

Shiori Sagawa et al. 'Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization'. In: International Conference on Learning Representations . 2019.

John Schulman et al. 'Proximal policy optimization algorithms'. In: arXiv preprint arXiv:1707.06347 . 2017.

Avi Singh et al. 'Beyond human data: Scaling self-training for problem-solving with language models'. In: Transactions on Machine Learning Research . 2023.

Maurice Sion. 'On general minimax theorems.' In: 1958.

Feifan Song et al. 'Preference ranking optimization for human alignment'. In: AAAI Conference on Artificial Intelligence . 2024.

Taylor Sorensen et al. 'A roadmap to pluralistic alignment'. In: International Conference on Machine Learning . 2024.

Nisan Stiennon et al. 'Learning to summarize with human feedback'. In: Neural Information Processing Systems . 2020.

Gokul Swamy et al. 'A minimaximalist approach to reinforcement learning from human feedback'. In: International Conference on Machine Learning . 2024.

Yunhao Tang et al. 'Generalized Preference Optimization: A Unified Approach to Offline Alignment'. In: International Conference on Machine Learning . 2024.

Gemma Team et al. 'Gemma: Open models based on gemini research and technology'. In: arXiv preprint arXiv:2403.08295 . 2024.

Haoxiang Wang et al. 'Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards'. In: Association for Computational Linguistics . 2024.

Tianhao Wu et al. 'Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment'. In: arXiv preprint arXiv:2310.00212 . 2023.

Yue Wu et al. 'Self-Play Preference Optimization for Language Model Alignment'. In: arXiv preprint arXiv:2405.00675 . 2024.

Sang Michael Xie et al. 'DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining'. In: Neural Information Processing Systems . 2023.

Wei Xiong et al. 'Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint'. In: International Conference on Machine Learning . 2023.

Rui Yang et al. 'Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment'. In: International Conference on Machine Learning . 2024.

Chenlu Ye et al. 'A theoretical analysis of nash learning from human feedback under general kl-regularized preference'. In: arXiv preprint arXiv:2402.07314 . 2024.

Weizhe Yuan et al. 'Self-rewarding language models'. In: International Conference on Machine Learning . 2024.

Siyan Zhao, John Dang, and Aditya Grover. 'Group Preference Optimization: Few-Shot Alignment of Large Language Models'. In: International Conference on Learning Representations . 2024.

Yao Zhao et al. 'Slic-hf: Sequence likelihood calibration with human feedback'. In: arXiv preprint arXiv:2305.10425 . 2023.

Han Zhong et al. 'DPO Meets PPO: Reinforced Token Optimization for RLHF'. In: arXiv preprint arXiv:2404.18922 . 2024.

Daniel M Ziegler et al. 'Fine-tuning language models from human preferences'. In: arXiv preprint arXiv:1909.08593 . 2019.

Appendix

The supplementary section constitutes the following: An extended related work section in Appendix A, proofs for the theoretical insights and the weighted DPO gradient expression in Appendix B, an overview of IPO and the simplification for robust IPO in Appendix C, detailed experimental setup in Appendix D, and convergence proof for Algorithm 1 and other sampling strategies in Appendix E.

A Additional Related Work

We detail a more extensive related work exposition in this section.

The Reinforcement Learning from Human Feedback (RLHF) is a concrete way to adapt a LLM towards specific tasks or objectives using human feedback. This framework for fine-tuning LLMs is inspired from [11] that uses human feedback to improve reinforcement learning policies. For LLMs, it was first established in [60] and further developed in [45, 34]. It involves learning a reward function using the pairwise human feedback data that provides comparisons between responses for a given prompt. This reward learning is performed through the Bradley-Terry model [5]. This learned reward model is then used as an objective for optimizing the LLM policy using Proximal Policy Optimization [40].

However, obtaining high-quality human feedback data is expensive and several works such as [35, 2, 25] study the usage of AI feedback to reduce reliance on human input. Further, studies like [13, 28, 22] analyze the usage of active learning strategies to minimize the amount of human feedback data required to perform RLHF. For a comprehensive overview and perspective of the RLHF topic, with detailed discussions on various approaches and gaps, we refer the reader to [24, 7, 23].

Due to the inherent issues of tuning the hyperparameters of a PPO ([50, 37]) and susceptible nature of reward models ([17, 49]), alternative approaches to the PPO-based RLHF have been proposed. Foremost of them is the rejection sampling fine-tuning ([14, 18, 49]) which is inspired by the best-of-n inference technique ([30]). The technique involves sampling n responses per prompt and fine-tuning the model based on the highest scoring ones in terms of a learned reward function. Further works such as [20, 54] avoid reward learning and propose conditional Supervised Fine-Tuning (SFT) inspired by the reward-conditioned RL ([9]).

Another strategy proposed as an alternative to PPO-based RLPHF is DPO ( Rafailov et al. [37]), wherein one optimizes the policy directly based on human preferences, bypassing the need for learning a separate reward model. This method simplifies the training process and potentially reduces overfitting and model misalignment issues. Theoretical advancements in understanding the dynamics of DPO have been provided by [21, 53], who analyzed the convergence and stability of DPO policy learning. Some other works such as [1, 58, 47, 43, 16] also study a similar reward-free RLHF setup. In particular, [1] analyze the potential overfitting issues in DPO. Further, [16, 6] propose strategies that bypass the need for preference datasets. [38, 55, 29] adopt a game-theory perspective to learn Nash equilibrium policies. Typically, the reward-free RLHF approaches utilize a reference policy trained via supervised fine-tuning, which is then further optimized. However, [19] propose Monolithic Preference Optimization without reference model, bypassing supervised fine-tuning entirely. [59] address the RLHF problem from an MDP framework, introducing a DPO-type algorithm integrated with Proximal Policy Optimization (PPO). Additionally, works such as [51, 46, 10, 41, 56] investigate self-play preference optimization, where new data generated in each round by the current policy is used to train an improved policy. Our work utilizes a reward-free framework similar to [37, 1] but differs from all the above works in the following sense. Unlike previous works that assume a single preference distribution, we consider fine-tuning a LLM given data from multiple preference distributions across diverse groups. Further, we aim to fine-tune the LLM in a robust manner such that there is minimal imbalance in performance across all groups.

Robust language modeling technique in the context of language model was first studied in [33] to optimize performance over a wide-range of topics. It was further improved by [52] wherein they use smaller model to learn the weights/importance that needs to be assigned to different topics and train a larger model based on these weights. Both works aim to minimize the worst group loss based on the group-DRO approach and focus on the pre-training aspects of language models. Concrete theoretical study of this group-DRO approach was performed in [39] which studies the minimax formulation of this group-DRO problem based on [31]. In the context of online batch selection, [4] propose a robust

data downsampling method that performs class-aware priority reweighting to reduce training costs while preserving worst-class performance.

In the RLHF setup, recent research has increasingly focused on enhancing the robustness of policy learning to address certain inherent gaps in the RLHF framework as detailed in [7] such as reward hacking and model misspecification. In particular, [3] formulated a weighted group loss for reward model learning w.r.t. a parameter λ that determines the importance assigned to a group and apply it to the anthropic Harmless vs Helpful data ( 6 ). In a similar manner,[12] design a weighted objective for policy optimization w.r.t. parameter λ to trade-off the importance between task reward maximization and safety cost minimzation. Another work [8] consider alignment to different sub-populations by learning multiple reward functions corresponding to each sub-population and learning a robust policy (max-min) w.r.t. the reward functions. Distinct from previous works, our approach circumvents reward model learning and integrates group robustness directly into the reward-free tuning framework. This allows us to directly learn a robust policy without the need for learning reward models. To learn this robust policy, we present a concrete algorithm that adaptively assigns weights to the losses of different groups and optimizes the policy to minimize this weighted loss. Additionally, our algorithm features a novel gradient estimator specifically designed for the group robust DPO problem.

Other studies addressing robustness in preference optimization, such as [21] and [26], focus on different facets of robustness, including robustness to noise and resilience against out-of-preference data. In the non-robust group preference alignment setup, [57] explores learning group preferences with LLMs by incorporating group information into prompts. However, their approach does not involve fine-tuning a LLM but training a separate transformer module that optimally selects an example sequence of prompts, LLM responses, and group preferences for in-context learning using a LLM. Also, Wang et al. [49] consider alignment with user preferences assuming that each user/group has varied importance over the distinct metrics in their multi-objective reward model. The output policy is trained to output a response based on both the prompt and the importance/weights over the individual metrics. In contrast, the distinctive feature of our method is that we are not modeling multi-reward objectives but consider a reward-free setting. Specifically, we consider the robust alignment problem optimizing for the worst group performance. And, our methodology directly models each group's preferences through a group-dependent latent reward model where the group dependency is injected through the prompt.

B Theoretical Insights

In this section, we detail the proofs for the theoretical insights elucidated in Section 3 and the weighted DPO gradient expression discussed in Section 4.

B.1 Robust Objective for the Log-linear Policy Class

Here, we describe obtaining the robust objective for the log-linear policy class as in Equation (9). Starting from the robust objective for a general policy class in Equation (8), L GR can be specialized as follows:

Here, (i) follows from the definition of h π ( x g , y w , y l ) = log( π ( y w | x g ) π ref ( y w | x g ) ) -log( π ( y l | x g ) π ref ( y l | x g ) ) , (ii) follows from the fact that for a uniform reference policy π ref ( y w | x g ) = π ref ( y l | x g ) , (iii) follows from substituting π θ ( y | x g ) = exp f θ ( x g ,y ) ∑ y ∈Y exp f θ ( x g ,y ) , and (iv) follows from substituting f θ ( x g , y ) = θ T ϕ ( x g , y ) .

B.2 Robust KL-Regularized Policy Maximization

In this section, we consider the robust version of the classical KL-regularized reward maximization objective (Equation (3)) detailed in Equation (10) for a reward function r ( x g , y ) , reference policy π ref, and a general class of policies Π .

Lemma B.1. The optimal policy π ∗ for the robust KL-regularized reward maximization objective (Equation (12) ) for a reward function r ( x g , y ) , reference policy π ref is

where Z ( x g ) = ∑ y π ref ( y | x g ) exp( 1 β r ( x g , y )) is a partition function.

Proof. We recast Equation (12) based on Equation (8) and perform the following analysis similar to [37][Appendix A.1]:

Further, by defining π ∗ ( y | x g ) = 1 Z ( x g ) π ref ( y | x g ) exp( r ( x g ,y ) β ) , we can rewrite Equation (17) as:

where we use

̸

It is not hard to show that roL ( π ) is minimized by π ∗ . Indeed, let us consider any other policy π = π ∗ . Then, we have

where the inequality follows since D KL ( π ( ·| x ) || π ∗ ( ·| x ) ) ≥ 0 . This implies π ∗ minimises Equation (20).

Note that, we have proved that the optimal policy expression for Equation (12) aligns with the form of the optimal policy for standard (non-robust) KL-regularized reward maximization Equation (4).

Proposition 3.2. Substituting the closed-form solution of the robust KL-regularized policy maximization problem (Equation (10) ) into the robust reward maximization objective in Equation (6) leads to the same group robust DPO loss L GR -DPO in Equation (8) .

Proof. In [37], the DPO loss (see Equation (5)) is obtained by using the following reward loss:

and replacing r ( · , · ) with the relation

which is the solution to

However, in our group-robust formulation, we replace Equation (25) with

while Equation (27) remains the same. Alternatively, we can replace Equation (27) with its robust version, i.e.,

however, we have already shown, in Lemma B.1, that this does not change the obtained policy-reward relation from Equation (26). Hence, this proves Proposition 3.2.

B.3 Analysis of the Gradient

We elucidate the steps to obtain the expression for the loss gradient in Equation (11). We can simplify the gradient as follows:

Here, (i) follows from the derivative of log( · ) function and denoting the derivative of sigmoid function σ ( · ) as σ ′ ( · ) . (ii) and (iii) follow from the fact that for any s ∈ R , σ ′ ( s ) = 1 -σ ( s ) = σ ( -s ) . Finally, (iv) and (v) follow using the definition of h π ( x, y w , y l ) = log( π ( y w | x ) π ref ( y w | x ) ) -log( π ( y l | x ) π ref ( y l | x ) ) , and substituting r θ ( x g , y ) = β log( π θ ( y | x g ) π ref ( y | x g ) ) -β log Z ( x g ) from Equation (4).

B.4 Trading off worst-case vs. average performance

In the robust objective of Equation (8), we intend to minimize the worst-case loss. This often might adversely impact the average loss leading to bad average performance across the different groups. To mitigate this, we propose the following robust trade-off direct preference optimization objective for a specified policy π and set of group weights µ 1 , . . . , µ K :

Here, note that the input weights µ g can be equal for all groups leading to a trade-off between the average and worst-case performance w.r.t. parameter χ . Following Equation (8), this can be

equivalently recast into:

where ∆ K -1 represents the ( K -1) -dimensional simplex of probabilities. Hence, minimizing the loss in Equation (31) implies that one can implicitly find an optimal reward function using human group preference data that is robust without losing average performance, and obtain a policy that works effectively in terms of both average and worst-case performance:

As Equation (8) involves maximizing with respect to α and minimizing with respect to π , it forms a two-player game similar to the one considered in Section 3.1, where the policy π and α act as opponents with inversely related payoffs. However, the contributions of the maximizing player ( α ) on the final outcome are limited depending on parameters χ and µ .

C IPO Simplification

Within the context of a finite data setting, it is common to record two responses ( y, y ′ ) for each prompt x . Based on these observations, there might be a tendency to inaccurately deduce that the preference distribution fulfills p ( y ≻ y ′ | x ) = 1 . Consequently, and assuming the Bradley-Terry model of preferences, the reward function r has to satisfy r ( y, x ) -r ( y ′ , x ) →∞ . Further, given that DPO policy π is directly dependent on the reward function (Equation (4)), it holds that π ( y ′ | x ) π ( y | x ) = 0 . In such a case, the policy overfits irrespective of the KL regularization factor. To circumvent this, [1] consider reward-free RLHF problem. Specifically, they propose preference optimization in the policy objective instead of reward optimization as follows:

Here, Ψ : [0 , 1] → R is a non-decreasing function, ρ refers to the distribution of prompts and, µ refers to the competing behavior policy. Choosing Ψ as the identity function, the optimal policy π ∗ of Equation (34) has a similar expression to Equation (4) with the reward function r ( · ) substituted by the following preference function p ( · ) ,

Recalling the definition of h π ( x, y, y ′ ) = log( π ( y | x ) π ref ( y | x ) ) -log( π ( y ′ | x ) π ref ( y ′ | x ) ) from Section 2, it follows from Equation (35) that π ∗ adheres to the following identity that equates the true preference with the policy's preference of y over y ′ :

where p ( y ≻ µ | x ) := E ¯ y ∼ µ ( ·| x ) [ p ( y ≻ ¯ y | x ) ] . The equation motivates us to find a policy π that minimizes the squared differences of two sides in Equation (36),

For an empirical dataset where one observes if y w is more preferable to y l , the policy minimization objective in Equation (37) becomes the IPO objective:

We can simplify Equation (38) for the IPO Loss using Log-linear policy class similar to Equation (9) as follows

Contrary to DPO, the IPO Loss results in a problem formulation equivalent to linear regression for the linear bandit setting. This leads us to obtain a closed form analytical solution for θ

where 1 = { 1 } N . Here, X ∈ R d × N . In particular, each row of X is the difference of the preferred and least preferred feature vectors ϕ ( x, y w ) -ϕ ( x, y l ) for ( x, y w , y l ) ∈ X × Y × Y

When re-written in this form it is trivial to see that the stability of the IPO Loss depends upon the rank of the matrix X .

Group IPO: We begin by conducting experiments with IPO considering the existence of closed form solution for this particular setting. In particular, given a set of preference data D = {D 1 , D 2 } from either groups with varying ratio, one aims to find an optimal θ that minimizes the group IPO loss,

Concisely, one can also write this as,

We assume that the feature vectors for each group is known, but the reward parameter θ is unknown. So, given the preference data with group information and the group dependent feature vectors, one aims to learn an optimal θ that balances both groups by minimizing the above loss. The solution to this will still be the closed form solution expressed in Equation (40) with feature matrix X consisting group dependent features.

Group Robust IPO: It is straightforward to see that in such a setting, the group with higher number of preference data will have an unfair advantage as they contribute more to the loss. Hence, a robust approach needs to be considered as follows:

One can use, Algorithm 1 to find the optimal θ for Equation (44). But considering the weighted regression form of the loss, one can use the following simplified algorithm Algorithm 2. Note that the last step, inadvertently leads to a weighted regression whose solution can be obtained in closed form.

D Additional Experiments and Details

In this section, we discuss additional experimental and training details. We use the following hyperparameters for the synthetic experiments. The importance sampling methods use the same hyperparameters as the corresponding vanilla ones. Further, we note that there is no learning rates for IPO and GR-IPO as we use the closed-form solution detailed in Section 4.1 for updates.

Algorithm 2 Policy Optimization for Robust IPO in Linear Bandits

1: Initialize: Step size η α for group weights α , initial weights θ (0) of the policy and weights over each group α (0)

2: Input: Dataset D with size N = |D| , group size N g for g = { 1 , 2 , · · · , K }

3: for t = 1 , . . . , T do

4: Calculate group loss l g for each group g on D g

5: α ′ ← α t -1 ; α ′ g ← α ′ g exp( η G l g ) // Update weights for group g

6: α ( t ) ← α ′ / ∑ g ′ α ′ g ′ // Renormalize α

7: θ t ← arg min θ E ( x,y w ,y l ,g ) ∼D [ α ( t ) g ( ⟨ ϕ ( x, y w , g ) -ϕ ( x, y l , g ) , θ ⟩ -1 2 β ) 2 ]

8: end for

9: Return: Output the robust policy π ( θ t )

D.1 Synthetic Experiments - Experiments setup

Here we provide a full definition of our synthetic experimental setup discussed in Section 5.

We adapt the standard (non-group based) experimental setup proposed by [26] for the group preferences setting by incorporating group information into the reward function r : X × Y × G → R . Here, X represents a two-dimensional state space [0 , 1] × [0 , 1] , Y denotes a discrete action space { 0 , 1 , 2 , 3 , . . . , n } , and G signifies a discrete group space { 0 , 1 , 2 , . . . , K } . The reward function, defined by the group-dependent feature vector ϕ ( x, y, g ) and parameter vector θ g , is given as:

Further, we consider n = 7 , K = 2 and denote x := ( x 0 , x 1 ) ∈ X . The feature vector ϕ ( x, y, g ) := ( ϕ 0 ( x, y, g ) , ϕ 1 ( x, y, g ) , ϕ 2 ( x, y, g ) , ϕ 3 ( x, y, g )) and parameters θ g ∀ g ∈ G are defined as follows:

By design, this parameterization ensures a coordinate-flipped relationship between groups, effectively mirroring one group to the other. Also, we study two other feature parameterizations and include their experimental results in Appendix D.3.

D.2 Synthetic Experiments - Preferences data generation

In our synthetic experiments, we considered three scenarios: (i) Groups are imbalanced in size, (ii) preferences distribution, and (iii) both of the above. The size imbalance is generated by using ratios 0 . 2 : 0 . 8 , respectively. Instead, to generate imbalance in terms of preferences we proceed as follows. In scenario (i) , after randomly selecting a state x within group g and an action y 1 , a second action y 2 is randomly chosen (groups have the same distribution over responses). Instead, in scenarios (ii) and (iii) , for one group y 2 is the action most distant from y 1 , while for the other, it is the closest (groups have different distributions over responses). In both methods, we calculate the rewards r ( x, y 1 , g ) and r ( x, y 2 , g ) , designating the action with the higher reward as preferred.

Here, we provide the resulting plots for scenarios (i) and (ii) .

Trading off worst-case vs. Average performance

We perform ablation studies for various values of χ trading-off between the average and worst-case performance. We observe in Figure 6, that the max validation loss decreases while moving from

0

2500

5000

7500

10000

Iterations

0

.

6

0

.

8

Value

DPO Max Validation Group Loss

DPO

IS-DPO

GR-DPO

0

25

50

75

100

Iterations

20

25

Value

IPO Max Validation Group Loss

IPO

IS-IPO

GR-IPO

IPO

IS-IPO

GR-IPO

DPO

IS-DPO

GR-DPO

Methods

0

.

00

0

.

05

0

.

10

Value

0

2500

5000

7500

10000

Iterations

0

.

45

0

.

50

0

.

55

0

.

60

Value

DPO Max Validation Group Loss

DPO

GR-DPO

0

25

50

75

100

Iterations

17

18

19

20

Value

IPO Max Validation Group Loss

IPO

GR-IPO

IPO

GR-IPO

DPO

GR-DPO

Methods

0

.

00

0

.

05

0

.

10

Value

0

20

40

60

80

100

Iterations

18

19

20

Value

Max Validation Group Loss

GR-IPO

χ

= 0

.

0 (IS-IPO)

GR-IPO

χ

= 0

.

2

GR-IPO

χ

= 0

.

4

GR-IPO

χ

= 0

.

6

GR-IPO

χ

= 0

.

8

GR-IPO

χ

= 1

.

0

0

20

40

60

80

100

Iterations

14

16

Value

Average Validation Loss

GR-IPO

χ

= 0

.

0 (IS-IPO)

GR-IPO

χ

= 0

.

2

GR-IPO

χ

= 0

.

4

GR-IPO

χ

= 0

.

6

GR-IPO

χ

= 0

.

8

GR-IPO

χ

= 1

.

0

χ = 0 to χ = 1 , where χ = 0 corresponds to importance sampling with group weights µ 1 , · · · , µ g mapping to importance sampling weights (see Equation (30)) and χ = 1 corresponds to GR-IPO. Further, we plot the average validation loss which increases while moving from χ = 0 to χ = 1 , demonstrating the trade-off between average and worst-case performance. Note that, GR-IPO aptly increases the average loss (as expected) in order to reduce the worst group loss.

D.3 Synthetic Experiments - Additional feature parametrizations

We present further experiments on synthetic preference data, using different configurations of the ϕ ( x, y, g ) and θ g vectors characterising the group-specific reward distributions r ( x, y, g ) = ⟨ ϕ ( x, y, g ) , θ g ⟩ .

With the same action space n = 7 and group space K = 2 , we consider same and flipped configurations of ϕ ( x, y, g ) = ( ϕ 0 , ϕ 1 , ϕ 2 , ϕ 3 ) , as follows:

Same: Here, the feature vectors are the same irrespective of group g and reward vectors are coordinateflipped.

Flipped: Here, the feature vectors include alternating sin , cos terms with swapped order for g = { 0 , 1 } ,with alternating coefficients. And the reward vectors are coordinate-flipped as before.

The experimental results are shown in Figures 7, 8 and 9 for the same experiment, and Figures 10, 11 and 12 for the flipped experiment.

0

2500

5000

7500

10000

Iterations

0

.

4

0

.

6

0

.

8

Value

DPO Max Validation Group Loss

DPO

IS-DPO

GR-DPO

0

25

50

75

100

Iterations

16

18

Value

IPO Max Validation Group Loss

IPO

IS-IPO

GR-IPO

IPO

IS-IPO

GR-IPO

DPO

IS-DPO

GR-DPO

Methods

0

.

00

0

.

05

0

.

10

Value

0

2500

5000

7500

10000

0

.

4

0

.

5

0

.

6

Value

DPO Max Validation Group Loss

DPO

GR-DPO

0

25

50

75

100

17

18

19

Value

IPO Max Validation Group Loss

IPO

GR-IPO

IPO

GR-IPO

DPO

GR-DPO

Methods

0

.

000

0

.

025

0

.

050

0

.

075

Value

Iterations

Iterations

0

2500

5000

7500

10000

Iterations

0

.

4

0

.

5

0

.

6

Value

DPO Max Validation Group Loss

DPO

IS-DPO

GR-DPO

0

25

50

75

100

Iterations

18

20

Value

IPO Max Validation Group Loss

IPO

IS-IPO

GR-IPO

IPO

IS-IPO

GR-IPO

DPO

IS-DPO

GR-DPO

Methods

0

.

000

0

.

025

0

.

050

0

.

075

Value

0

2500

5000

7500

10000

Iterations

0

.

3

0

.

4

0

.

5

0

.

6

Value

DPO Max Validation Group Loss

DPO

IS-DPO

GR-DPO

0

25

50

75

100

Iterations

14

16

18

Value

IPO Max Validation Group Loss

IPO

IS-IPO

GR-IPO

IPO

IS-IPO

GR-IPO

DPO

IS-DPO

GR-DPO

Methods

0

.

00

0

.

02

0

.

04

0

.

06

Value

0

2500

5000

7500

10000

Iterations

0

.

3

0

.

4

0

.

5

0

.

6

Value

DPO Max Validation Group Loss

DPO

GR-DPO

0

25

50

75

100

Iterations

16

18

Value

IPO Max Validation Group Loss

IPO

GR-IPO

IPO

GR-IPO

DPO

GR-DPO

Methods

0

.

00

0

.

02

0

.

04

0

.

06

Value

0

2500

5000

7500

10000

Iterations

0

.

3

0

.

4

0

.

5

0

.

6

Value

DPO Max Validation Group Loss

DPO

IS-DPO

GR-DPO

0

25

50

75

100

Iterations

16

18

20

22

Value

IPO Max Validation Group Loss

IPO

IS-IPO

GR-IPO

IPO

IS-IPO

GR-IPO

DPO

IS-DPO

GR-DPO

Methods

0

.

00

0

.

05

0

.

10

Value

D.4 Global Opinion Data Experiments - Setup and Additional Plots

We use the following prompt type for both SFT and DPO training: "Opinion of people in (countryname) on: (question). Please select the best response: A) Choice-1, B) Choice-2, ...". The target response is just the letter 'A', 'B', etc. The training data size from each country are as follows: Nigeria-572, Egypt- 570, India-376, China-309, Japan-712. Further, the data is split as 80% for training, 10% for validation and 10% for testing. We run the SFT training with learning rate 10 -4 for one epoch over the training data on pre-trained Gemma-2B model. We use this SFT trained model as the reference model for training IPO and GR-IPO. For the IPO training, the optimal hyperparameters were learning rate 3 ∗ 10 -5 , and β = 0 . 01 . For the GR-IPO training, we use the same β but the optimal learning rate and the exponential step size were 6 ∗ 10 -5 and 5 ∗ 10 -7 . For both IPO/GR-IPO training we use AdamW [27] optimizer with adaptive learning rates decreasing by a factor of 10 if there is no improvement in terms of loss (average group loss for IPO/ worst group loss for GR-IPO) for 4 k iterations on a validation set. Further, for GR-IPO, the exponential step size is decreased by a factor of 2 whenever learning rate is reduced. This adaptive learning rates and exponential step sizes ensures stability in training and leads to convergence. We then evaluate both the methods based on the worst group loss and accuracy. Here, the accuracy refer to the percentage of prompt, winning response and losing response correctly ordered by the learned preference function (Equation (35)).

All experiments were run on a single node of A100 SXM4 machine with 40GB GPU memory, 30 CPU cores, 200GB RAM, and 525GB SSD memory. Further, for each seed, the execution time of the experiments is approximately 3-5 hours until convergence. We also provide the exact hyperparameters used in the table below.

0

10000

20000

30000

Iterations

1500

1750

2000

2250

2500

Value

Max Group Loss (Test)

GR-IPO

IPO

0

10000

20000

30000

Iterations

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

Value

Min Rewards Accuracies (Test)

GR-IPO

IPO

Group 1

Group 2

Group 3

Group 4

Group 5

Groups

0

.

80

0

.

85

0

.

90

0

.

95

1

.

00

Value

Log-Prob Accuracies (End)

E Convergence Proofs for Different Sampling Strategies

In this section, we study the convergence rates for different sampling strategies including the proposed Algorithm 1.

E.1 Convergence Proof for Algorithm 1

In this section, we prove the convergence of the proposed Algorithm 1 in terms of the error of the average iterate π ¯ θ (1: T ) ,

as stated in Proposition 4.1.

Proposition 4.1. Suppose that the loss l ( · ; ( x g , y, y ′ )) is non-negative, convex, B ∇ -Lipschitz continuous, and bounded by B l for all ( x g , y, y ′ ) ∈ X ⊕ G × Y × Y and ∥ θ ∥ 2 ≤ B Θ for all θ ∈ Θ with convex Θ ⊂ R d . The error of the average iterate of Algorithm 1, i.e., π ¯ θ (1: T ) = 1 T ∑ T t =1 θ t , satisfies

Proof. We build our proof based upon the online mirror descent algorithm's regret bound in [32, Eq. 3.1]. [32]'s theorem considers the following saddle-point stochastic optimization problem,

Denote f g ( θ ) = E [ F g ( θ, ξ )] . In our problem, we consider,

and we define F g ′ ( θ, ( x g , y, y ′ )) := N N g 1 [ g = g ′ ] l ( θ ; ( x g , y, y ′ )) . Then, the expectation

Mirror Descent mapping. We begin by mapping Algorithm 1 to the general mirror descent framework of [32].

Further, we denote the gradients of the objective ϕ ( θ, α ) w.r.t. θ and α as

Here, the negative sign for the gradient w.r.t. α indicates that we perform maximization w.r.t. α . Let Φ( θ, α, ξ ) := ∑ K g =1 α g [ F g ( θ, ξ )] . Then the stochastic subgradients for ϕ ( θ, α ) for a given ξ is as follows:

which is equivalent to

Simplifying the gradients we obtain the following for ξ = ( x g , y, y ′ ) :

Based on the above gradient expressions, we can map the updates of θ t and α t in Algorithm 1, to the general mirror descent update rule:

Here, we use the compact notation ζ = ( θ, α ) . Further, γ t denotes the step-size at time t of the algorithm and P ζ ( · ) is the prox-mapping corresponding to the mirror descent algorithm defined as follows:

Here, V ( · , · ) is the prox-function associated with the distance-generating function (Bregmann Divergence) ω ( · ) . For our setup in Algorithm 1, we define the following combined distance-generating function over ζ ∈ (Θ × ∆ K -1 ) :

where ω θ ( θ ) = 1 2 ∥ θ ∥ 2 , ω α ( α ) = ∑ K g =1 α i ln α i , D ω θ , Θ := (max θ ∈ Θ ω θ ( θ ) -min θ ∈ Θ ω θ ( θ )) 1 / 2 , and D ω α , ∆ K -1 := ( max α ∈ ∆ K -1 ω α ( α ) -min α ∈ ∆ K -1 ω α ( α ) ) 1 / 2 . Also, the corresponding proxfunctions individually for θ and α would be V θ ( θ, θ ′ ) = 1 2 ∥ θ -θ ′ ∥ 2 2 and V α ( α, α ′ ) = ∑ K g =1 α ′ i ln α ′ i α i . It can be shown that the above prox-functions V θ and V α satisfy the following inequalities ([32]):

Due to the above definitions, the corresponding prox-mapping P ( · ) corresponds to gradient descent w.r.t. θ and exponentiated gradient ascent w.r.t. α , as defined in Algorithm 1. Moreover, due to the definition of ω in Equation (52), the combined prox function V ( · , · ) over ζ would satisfy

For further details regarding prox-function, kindly refer to its usage in [32].

Bounding the error. According to the introduced notation, the approximation error ϵ T can be defined and bounded as:

To bound ϵ ϕ ( ˜ ζ 1: T ) , we invoke the result from [32, Equation 3.23] for mirror-descent convergence for max-min problems.

Further, M θ and M α correspond to the following upper bounds to the maximum of the expected norms of F g ( θ, ξ ) and ∇ θ F g ( θ, ξ ) ,

Similarly,

Here, we have used ∥∇ θ l ( θ ; ( x g , y )) ∥ ≤ B ∇ and ∥ l ( θ ; ( x g , y )) ∥ ≤ B l . Here, λ θ , λ α correspond to the strong convexity parameters of the distance generating functions ω θ ( θ ) and ω α ( α ) respectively. For our given functions, ω θ ( θ ) = 1 2 ∥ θ ∥ 2 and ω α ( α ) = ∑ K g =1 α i ln α i , both λ θ = λ α = 1 ([32]).

Let, R 2 θ = D 2 ω θ λ θ = B 2 Θ = (max θ ∥ θ ∥ θ -min θ ∥ θ ∥ 2 θ ) , obtaining the overall error bound.

Lemma E.1. For the log-linear policy class parameterized with respect to θ , the DPO loss function l ( π θ ; · ) = log ( σ ( βh π θ ( · ))) (see Equation (5) ) is convex and Lipschitz continuous in θ . Consequently, Proposition 4.1 applies to this case.

Proof. We want to show, l ( θ ; ( x, y w , y l )) is convex in θ and B ϕ -Lipschitz in θ for the log-linear policy class defined as follows:

Here θ belongs to a convex set Θ satisfying ∥ θ ∥ ≤ B Θ and ϕ ( x, y ) denotes the feature vector such that ∥ ϕ ( x g , y w ) -ϕ ( x g , y l ) ∥ ≤ B ϕ . In conjunction with a uniform reference policy π ref, the loss function l ( θ ; ( x g , y w , y l )) for the log-linear policy class is as follows:

Algorithm 3 Mirror Descent for Group Robust Preference Optimization (GRPO)

1: Initialize: Step size η α for group weights α , step size η θ for policy π with weights θ , initial weights θ (0) of the policy and weights over each group α (0) , Projection operator P Θ

2: Input: Dataset D with size N = |D| , group size N g for g = { 1 , 2 , · · · , K } , loss l ( π θ ; · )

3: for t = 1 , . . . , T do

4: α ′ ← α ( t -1)

5: g ∼ Uniform(1 , · · · , K )

6: ( x g , y w , y l ) ∼ D g

7: α ′ g ← α ′ g exp η α ( l ( π θ ( t -1) ; ( x g , y w , y l ))) // Update weights for group g

8: α ( t ) ← α ′ / ∑ g ′ α ′ g ′ // Renormalize α

9: θ ( t ) ← P Θ ( θ ( t -1) -η θ ( α ( t ) g ∇ θ l ( π θ ( t -1) ; ( x g , y w , y l )) )) // Use α to update θ

10: end for

11: Return: Output the robust policy π ( θ ( T ) )

Next, we compute the derivative of l ( θ ; ( x, y w , y l )) w.r.t. θ from Equation (58) to check the Lipschitz continuity.

Then, the gradient norm can be bounded as follows:

Hence, by the definition of Lipschitz functions for continuously differentiable functions, we have that l ( θ ; ( x, y w , y l )) is Lipschitz continuous with Lipschitz constant B ϕ . Further, we bound the loss function using the bounds for θ and ϕ ( · , · ) .

Hence, the loss function is bounded by B l for bounded θ and ϕ .

E.2 Alternate Sampling Strategy

In this section, we study the convergence of an alternate algorithm in Algorithm 3, wherein groups are sampled uniformly rather than a categorical distribution proportional to the group sizes, in terms of the error ϵ T as defined in Equation (46).

Proposition E.2. Suppose that the loss l ( · ; ( x g , y, y ′ )) is non-negative, convex, B ∇ -Lipschitz continuous, and bounded by B l for all ( x g , y, y ′ ) ∈ X ⊕ G × Y × Y and ∥ θ ∥ 2 ≤ B Θ for all θ ∈ Θ with convex Θ ⊂ R d . Then, the average iterate of Algorithm 3 achieves an error at the rate

Proof. We follow a similar proof structure as in the proof of Proposition 4.1. We recall the saddlepoint stochastic optimization problem from [32] stated earlier in Equation (47),

In lieu of the alternate sampling strategy in Algorithm 3 where groups are sampled uniformly, we consider,

and we define F g ′ ( θ, ( x g , y, y ′ )) := K 1 [ g = g ′ ] l ( θ ; ( x g , y, y ′ )) . Then, the expectation

Mirror Descent mapping. We map Algorithm 3 to the general mirror descent framework of [32] as done in the proof of Proposition 4.1. We begin by recalculating the gradients of ϕ ( θ, α ) w.r.t. θ and α for this alternate definition of F g ( θ, ξ ) ,

Based on the above gradient expressions, we can similarly map the updates of θ t and α t in Algorithm 3, to the general mirror descent update rule with corresponding prox-function as done for Proposition 4.1. We omit further details regarding the mirror descent related definitions as they follow from the proof of Proposition 4.1. We directly proceed to bounding the error ϵ ϕ ( ˜ ζ 1: T ) using [32, Equation 3.23],

We recalculate M θ and M α for this alternate definition of F g ( θ, ξ ) as they correspond to the upper bounds to the maximum of the expected norms of F g ( θ, ξ ) and ∇ θ F g ( θ, ξ ) ,

Similarly,

Here, we have used ∥∇ θ l ( θ ; ( x g , y )) ∥ ≤ B ∇ and ∥ l ( θ ; ( x g , y )) ∥ ≤ B l .

Using the alternate M θ and M α corresponding to this sampling rule, we obtain the overall error bound as follows:

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the claims regarding the proposed methodology in terms of theory and experiments have been concretely illustrated in Sections 3 and 4(Theory) and Section 5 (Experiments).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are detailed along with conclusions in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are mentioned in the theorem statements and the corresponding proofs are detailed in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental details (models, datasets, configurations, training strategies, compute resources) are discussed in the experimental sections of the main text and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide links to Github repositories that contain all the codes required to reproduce the experiments with detailed instructions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All hyperparameters, optimizers, data splits, etc., are detailed in the experimental section in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are shown based on different seeds of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources required and used to run the experiments are mentioned in the experimental sections in the main text and appendix

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We are using open-sourced data and language model. We do not foresee any potential harm caused by our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader positive societal impacts of our work in Section 6. We do not foresee any negative societal impacts of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are using publicly available model and data.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the data and models used have been appropriately cited with URLs which includes the licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code and algorithm implementation has been properly documented and links to the Github repositories have been provided.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not conduct any such experiments for this work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel preference optimization technique, GRPO, which utilizes the group distributional robust optimization. Specifically, the method aims to maximize the worst-case group performance for improving the robustness of LLM models. This paper provides several theoretical aspects of GRPO when it comes to convergence guarantee and closed-form solution when considering the log-linear policy class. Empirically, the GRPO objective function is minimized by the mirror gradient descent. Several theoretical results on synthetic and real-world datasets demonstrate that their method works well.\n\nSTRENGTHS:\n* This is the first study to adopt the group DRO technique in the RLHF setting. \n\n* Several theoretical results are provided to enhance understanding of the proposed method. \n\n* The proposed method is general so that it can be applied to various RLHF optimization techniques.\n\nWEAKNESSES:\n* This paper lacks novelty in its contribution. The group DRO technique is already a popular method used in a variety of applications [1, 2].  While the authors offer some theoretical results and an algorithm for their method, these may also lack novelty. In specific, Proposition 3.1 on the Nash equilibrium is a well-known result in game theory and Proposition 3.2 seems straightforward because $\\pi^*$ is dependent only on an input value, not on the group to which the input belongs. Furthermore, there is no significant difference in the algorithm and proof of convergence guarantee compared to existing works [37, 30 in the manuscript], except for slight modifications.\n\n* Although this paper is novel in applying the group DRO technique to RLHF preference optimization for the first time, stronger motivations and empirical results are needed to highlight the necessity of the proposed method. But, I also think that to strengthen this contribution, For example, as mentioned in lines 141-153, it would be beneficial to show that existing Large Language Models (LLMs) suffer from group robustness issues in various applications involving helpful/harmful instances or domain-specific categories when fine-tuned by RLHF optimization techniques.\n\n* The interpretation of the results in Section 5.2 is insufficient. In Figure 3, GR-IPO improved the performance of all groups, including Group 1. This result contrasts with findings reported in the Group DRO paper [37], where a trade-off between majority and minority group performance is generally observed, with majority group performance typically decreasing to compensate for minority group performance. Therefore, a more thorough analysis or performance comparison is needed to explain this result.\n\n[1] Re-weighting based group fairness regularization via classwise robust optimization, 2023.\n[2] Distributionally Robust Multilingual Machine Translation, 2021.\n\nQUESTIONS:\n1. In the experiments in Section 5.2, what are the zero-shot performances of Gemma-2B for each group? It would be better to report the degree of bias in the pre-trained model together, because it can enable comparing the bias degrees between the pre-trained and fine-tuned models.\n\n2. Would you provide the results when fine-tuning more layers or the entire neural networks using your method? I think that this can strengthen this paper's contribution by showing the performance of the method in various settings.\n\nLIMITATIONS:\nThey addressed the limitations of this paper in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper addresses the limitation of traditional reinforcement learning with human feedback (RLHF) approaches that indiscriminately optimize a single preference model, disregarding the unique characteristics and needs of diverse labeler groups. The authors propose a Group Robust Preference Optimization (GRPO) method to align LLMs with individual groups' preferences.

STRENGTHS:
The paper's motivation is clear.

WEAKNESSES:
1. How does the proposed method handle scenarios where preference data lacks clear group information? In practice, much preference data does not come with explicit group classifications.

2. The paper should address whether the proposed method can learn an invariant preference standard from data that may contain various group information, such as different preference evaluation criteria and preferences annotated at different iterations.

3. The authors need to provide and analyze the alignment results of the model, not just the reward model analysis.

QUESTIONS:
Please refer to the Weaknesses section for questions regarding the paper.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work tackles the important problem of robust RLHF for diverse groups. Traditionally, RLHF assumes that a single model can fit the diverse feedback from multiple groups of users. In this paper, the authors introduce a method to learn a robust policy that maximizes for worst-case group performance. To achieve this, the method adaptively weights the loss for each group based on the size and cumulative training loss incurred by the feedback samples for that group. As a result, LLMs trained on diverse group data demonstrate reduced loss imbalance and improved accuracies across all the groups. The authors also present a convergence analysis of the proposed method assuming a log-linear policy class.\n\nSTRENGTHS:\n1. The paper tackles a critical problem of robust RLHF for diverse groups. The intuition of the overall method is well understood, the framework and the parameters are clearly mentioned, and the results are shown over the standard datasets and compared to multiple baselines.\n2. The authors present a thorough literature review and background. Past work has mainly focused on making RLHF and LLMs robust to noisy or out of distribution data. Meanwhile, this work focuses on a group robust formulation of training LLMs using state-of-the-art methods (mainly DPO).\n3. The method introduced “GRPO” is useful is scenarios beyond diverse groups. As the authors mention, it is a general formulation that can enforce robustness to diverse tasks, domains, or objectives occurring in the feedback dataset.\n4. To achieve robustness, GRPO presents a robust optimization approach to minimize worst-case loss amongst the diverse groups. Further, the paper introduces a less aggressive object by trading off worst and average case performance. I would be curious to see an ablation study showing the effectiveness of this tradeoff.\n5. The paper supplements the approach with some strong theoretical proofs on the convergence properties of the method under a log-linear policy. The authors also present a close form solution for the RLHF update step, replacing DPO with IPO.\n6. The results show that GRPO improved performance across all groups, and the weight update behaves as expected by assigning higher weights to groups with higher cumulative loss i.e. the gradients from the worst performing group are scaled the most.\n\nWEAKNESSES:\n1. This paper proposes a method for group robust optimization for LLMs. However, the metrics evaluated are only max validation loss and reward error over the groups. GRPO uses a reward free approach to update the LLM, but the evaluations are restricted to the performance of the reward model over the feedback dataset. It would be nice to see the performance of the finetuned vs base model (such as win rate) in generating responses that align with the individual groups.\n2. The authors provide detailed training setup, but I would suggest that they also include information about the evaluation. \n3. GRPO performs well over all the groups, however, the performance of the importance sampling baseline is very close. It would be helpful if the authors could provide additional ablations and discussions to show the effectiveness of optimizing against the worst-case loss over the groups vs using only the IS approach.\n\nQUESTIONS:\n1. In the current experiments, is only the final layer of the LLM trained? I would be curious to know the result if the method used full finetuning of the model.\n2. As GRPO assumes that each prompt has access to group information through a prompt, how much does the prompt affect policy? If the prompts are good enough, would it just create non-overlapping distributions for all groups? As one of the problems in non-robust baselines is that they converge to unwanted biases or majority groups for shared prompts, if the group identification prompt is finetuned will it alleviate this issue altogether?\n3. Does GRPO finetune only the final layer of the Gemma-2B model in the results presented in the paper?\n4. Does GRPO ensure that there is a non-decreasing change in performance across all groups as compared to the non-robust baselines?\n\nLIMITATIONS:\n1. The paper introduces GRPO, an optimization method to weight each group in the RLHF update step proportional to the size and loss of the group. This ensures a balanced performance of the model across all the groups. In the results, we see an improved performance over all the individual groups, which intuitively violates the no-free lunch theorem. I would be curious to see an analysis of the method that gives insights as to what allows the model or the objective to achieve consistently higher performance.\n2. Here, the method assumes access the groups in the dataset. However, in practical settings the group information is unavailable and the model has to cluster or implicitly model the group information from the ungrouped dataset.\n3. The evaluation is limited only to the accuracy of the reward model over the preference dataset. So, currently, it provides weaker evidence of the translation of this robustness during the generation phase. It would be nice if the authors could include experiments showing if GRPO enables LLMs to robustly generate better-aligned responses to prompts from all the groups.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper introduces GRPO, a method to optimize policy preferences across different groups in a robust way. GRPO looks at the worst group alignment loss by taking the maximum loss across all groups, ensuring the policy performs well even when there are group-specific differences or overlaps in prompts.

STRENGTHS:
1. This paper writes well and easy to follow.
2. The paper provides a thorough theoretical analysis of GRPO.
3. The proposed framework accommodates both distinct and overlapping group scenarios.
4. GRPO can trade off worst-case for average performance with a hyperparameter.

WEAKNESSES:
1. GRPO's experiment on real world datasets is limited, on only one group (5 countries) of the GlobalQA dataset.
2. How is the training time as compared to other baselines? The GRPO framework's training process involves a min-max optimization, which can be potentially computationally intensive.

QUESTIONS:
please see weakness.

LIMITATIONS:
1. This methods requires knowing the number and nature of groups in advance to perform optimization. 
2. When new group is introduced, will have to train the entire optimization again, which is expensive.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

comments: SUMMARY:
This paper addresses the problem of improving fairness in preference optimization. It proposes a new loss function and algorithm. Experiments conducted show that the proposed algorithm indeed achieves better fairness. Additionally, the paper provides a theoretical analysis indicating the convergence of the optimization.

STRENGTHS:
1. The idea of improving the fairness is good. This is a problem that deserves more attention.

2. The derivation of the loss and the algorithm, though simple, is clear.

3. The results provided for the algorithm appear to be sound.

WEAKNESSES:
1. The author mentioned that the idea can also be applied to multi-objective preference optimization. In my opinion, a comparison with works in multi-objective preference optimization [1] and group optimization [2] is needed.



2. It is unclear whether this algorithm can be applied to scenarios beyond multiple choices.

3. It is unclear how Proposition 4.1 relates to common metrics like regret.


[1]. Zhao, Siyan, John Dang, and Aditya Grover. "Group preference optimization: Few-shot alignment of large language models."

[2]. Wang, Haoxiang, et al. "Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards."

QUESTIONS:
1. The paper mentioned that the algorithms can also be used for multi-objective preference optimization. Were any related experiments conducted?

LIMITATIONS:
See the 'Weakness' part
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17