Group Robust Preference Optimization in Reward-free RLHF

Authors: shyam  sundhar
yifan  hu
eth  zurich
viraj  mehta
pier giuseppe sessa
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 18:03:42

Abstract: Adapting large language models (LLMs) for specific tasks usually involves fine-tuning through reinforcement learning with human feedback (RLHF) on preference data. While these data often come from diverse labelers' groups (e.g., different demographics, ethnicities, company teams, etc.), traditional RLHF approaches adopt a "one-size-fits-all" approach, i.e., they indiscriminately assume and optimize a single preference model, thus not being robust to unique characteristics and needs of the various groups. To address this limitation, we propose a novel Group Robust Preference Optimization (GRPO) method to align LLMs to individual groups' preferences robustly. Our approach builds upon reward-free direct preference optimization methods, but unlike previous approaches, it seeks a robust policy which maximizes the worst-case group performance.  To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We theoretically study the feasibility of GRPO and analyze its convergence for the log-linear policy class. By fine-tuning LLMs with GRPO using diverse group-based global opinion data, we significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies compared to non-robust baselines.

Main paper: 
 introduction: 
 As the usage of large language models (LLMs) has grown in recent years, the question of their alignment has come to the forefront. Their remarkable capability to address a wide range of tasks (Radford et al. [36] ) stems from pre-training on a self-supervised objective over internet-scale text. This vast internet-scale content, however, carries a higher risk of biases, inaccuracies, and controversial content than smaller, curated datasets. Thus, ensuring that the model's responses and behaviors correspond to human intentions and values is crucial.Typical approaches to alignment [11, 34, 37] involve gathering preference feedback from human labelers to train models that reflect their desires. Such approaches often treat individual preferences as samples from a broader preference distribution. However, this perspective often oversimplifies the complex reality that human societies consist of numerous distinct groups (e.g., different demographics, ethnicities, company teams, etc.), each with their own set of preferences that can significantly Figure 1 : Current reward-free preference optimization methods typically optimize based on average human feedback. This often aligns predominantly with the preferences of the majority group (G1, R1 > R2) at the expense of minority groups (G2, R2 > R1). In contrast, our GRPO algorithm introduces adaptive weighting for different user groups and prioritizes optimizing for the worst-case group performance, leading to better alignment for the most disadvantaged groups. diverge. Consequently, prevalent alignment strategies tend to adopt a "one-size-fits-all" model and disproportionately favor the preferences of the majority group, often at the expense of minority groups and their preferences, as illustrated in Figure 1 .To improve alignment performance for even the most disadvantaged groups, we propose to robustly solve the problem of diverse group preferences by (i) including group information in the context of the LLM and (ii) optimizing against the worst-case alignment performance across all groups. We develop policies that guarantee equitable performance across all groups, ensuring that no group is disproportionately disadvantaged due to inherent biases or imbalances in the training data.Related work. The established process for alignment of LLMs using Reinforcement Learning from Human Feedback (RLHF) is set out in [45, 60] and [34] . The RLHF fine-tuning process consists of learning a reward model from human comparisons between responses to a given prompt, using the Bradley-Terry model [5] . Then, one performs policy optimization using Proximal Policy Optimization [40] to learn a policy that maximizes the learned reward function. For a comprehensive overview and perspective of the RLHF topic, we refer the reader to [24, 7, 23] .Due to the challenges of tuning PPO and the vulnerability of reward models ( [50, 37, 17, 49] ), alternative approaches to PPO-based RLHF have been proposed, including rejection sampling fine-tuning [14, 18, 49, 30] and conditional supervised fine-tuning [20, 54, 9] . In particular, Rafailov et al. [37] introduce Direct Preference Optimization (DPO), which optimizes policies directly based on human preferences, avoiding the need for a separate reward model. This approach simplifies training and reduces reward overfitting. Other studies, such as [1, 58, 47, 43, 16] , propose novel reward-free RLHF methods, with some bypassing preference datasets altogether ( [16, 6] ). We utilize a reward-free framework similar to [37, 1] , however, unlike previous works that assume a single preference distribution, we consider multiple preference distributions from diverse groups. Further, we aim to robustly fine-tune the LLM to ensure minimal disparity in performance across all groups. Other studies addressing robustness in preference optimization include [21] and [26] . However, these works primarily focus on different aspects of robustness, such as robustness to noise and resilience against out-of-preference data.Robust language modeling techniques have been studied by [33, 52] to optimize performance of language models over a wide-range of topics. They consider robust pre-training of language models based on the group Distributionally Robust Optimization (DRO) approach. A concrete theoretical study of the group DRO approach was performed in [39] and applied to vision problems. These are designed by extending previous minimax algorithm for solving DRO from [31] . In the RLHF setup, [3] consider weighting of loss from different topics (harmless vs helpful) for robust reward learning. Also, Chakraborty et al. [8] consider robust policy optimization by learning multiple reward functions corresponding to sub-populations and learning a robust policy w.r.t. the learned rewards. Differing from these works, we embed group robustness directly into the reward-free tuning paradigm. We provide an concrete algorithm that adaptively weighs the loss for different groups and optimizes for a policy that minimizes the weighted loss. Further, our algorithm employs a novel gradient estimator tailored to the group robust DPO problem.In the non-robust setup, [57] also explore group-based preference learning with LLMs by including group information in prompts with a transformer module that is trained optimally to choose an example sequence of prompts, LLM responses, and group preferences for in-context learning. [49] consider alignment with user preferences assuming that each user has varied importance over the distinct metrics in their multi-objective reward model. In contrast, we are not modeling multi-reward objectives but consider a reward-free setting. And, our methodology directly models each group's preferences through a group-dependent latent reward model where the group dependency is injected through the prompt. Further, unlike the non-robust problem setups of [57, 49] , we consider the robust alignment problem optimizing for the worst group performance. We detail other related works extensively in Appendix A.Main Contributions. The following are the main contributions of this work: (i) We present GRPO, the group robust formulation of Direct Preference Optimization (DPO) [37] , wherein we augment the context of the LLM with the group information, and pose the problem as a robust optimization problem to minimize the worst-case loss amongst the diverse groups. We also show that a naive application of group robustness to the LLM policy maximization objective does not offer robustness benefits. To the best of our knowledge, this is the first study that focuses on group robustness in RLHF preference optimization; (ii) We analyze the theoretical aspects of GRPO by examining the convergence and the feasibility of finding optimal solutions within the log-linear policy class. (iii) We present a tailored algorithm to tackle this robust optimization challenge, providing convergence guarantees for certain loss functions; (iv) We show the versatility of our approach by demonstrating how our algorithm can be utilized with other reward-free preference optimization methods such as Identity Preference Optimization (IPO) [1] . In particular, for the GR-IPO objective optimized over the log-linear policy class, we derive a closed-form weighted regression update for the policy parameters rather than a gradient update. To the best of our knowledge, this is a novel contribution towards efficient fine-tuning through preferential data; (v) Our empirical evaluations across synthetic datasets, real-world data, and publicly available LLMs show that the proposed GRPO significantly improves performance for the worst-performing groups, reduces loss imbalances across groups, and increases probability accuracies compared to non-robust baselines. 
background: 
 We address the challenge of fine tuning a large language model (LLM) to align with user preferences. This process usually follows the Reinforcement Learning from Human Feedback (RLHF) protocol, using either an explicit reward model ([3, 34, 45, 60] ) or an implicit reward model ( [37] ). RLHF typically comprises three key phases: (i) supervised fine-tuning of an initial (pre-trained) largelanguage model, (ii) reward learning, and (iii) RL fine-tuning.In the supervised fine-tuning phase (SFT), the goal is to fine-tune a pre-trained LLM on a specific highquality dataset suited for the downstream task of interest. It results in a probabilistic model expressing the probability of the response y given a prompt x as π ref (y|x). Subsequently, in the reward learning phase, the goal is to learn a reward model from a dataset of prompts x and responses y w , y l , with y l ≺ y w | x meaning that human labellers preferred y w over y l . It is typically assumed that preferences follow some choice models with an unknown reward (utility) r * (x, y) function. A popular model is the Bradley-Terry model [5] that assumes the preference distribution p admits the following form :p(y 1 ≺ y 2 | x) = exp(r * (x, y 2 )) exp(r * (x, y 1 )) + exp(r * (x, y 2 )).Based on the above model, a maximum likelihood estimate of the reward function is obtained as:EQUATIONwhere σ(•) is the sigmoid function and D represents the dataset consisting of {(x, y w , y l )}.Then, in the RL fine-tuning phase the objective is to train a policy π that maximizes the learned reward function. Simultaneously, the policy should stay closely aligned with the reference, π ref , as quantified by the KL divergence, leading to the following KL-regularized optimization problem:EQUATIONDirect Preference Optimization (DPO). The recent approach proposed by [37] exploits the closed-form solution of the problem in Equation ( 3) and sidesteps the explicit modelling of rewards to directly optimize the policy. Specifically, under the Bradley-Terry preference model, the reward function can be expressed directly in terms of the optimal policy π * as follows:EQUATIONfor a partition functionZ(x) = y π ref (y|x) exp( 1 β r(x, y)). Via a change of variable, finding the optimal reward function in Equation ( 2) is equivalent to finding the optimal policy π * utilizing the given set of preference data D. With a slight abuse of notation, we use π to denote π * . Denote h π (x, y w , y l ) := log( π(yw|x) πref(yw|x) ) -log( π(y l |x) πref(y l |x) ). Then, Equation (2) translates into the DPO loss:EQUATIONWith a parameterized policy model π θ , minimizing the DPO loss involves calculating the gradient over θ using backpropagation and the log-probabilities of each completion, y w and y l , given the prompt x for both the policy π θ and the reference policy π ref .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Limitations are detailed along with conclusions in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
group robust preference optimization: 
 In this section, we discuss Group Robust Preference Optimization (GRPO), i.e., instead of learning a reward function that maximizes the likelihood, we aim to derive (implicitly) a robust reward function and subsequently learn a robust DPO policy.Group Preferences. Suppose that preferences come from an underlying latent reward r * (x, y, g), with g ∈ G = {1, 2, . . . , K} indexing the groups. When group information is available (e.g., as a text), we can represent the reward as r * (x g , y), where x g = x ⊕ g denotes merging 1 of the prompt with group information (e.g., string concatenation). We continue to apply a Bradley-Terry model as described in Equation (1), substituting x with x g . Moreover, we assume access to a collective datasetEQUATIONw , y(i) l )} Ng i=1with the available group information. Additionally, our dataset accommodates the exposure of different groups to identical prompts, meaning that the same x can appear across various groups g in our dataset, and these groups may favor different responses y.Given such D, although one may obtain a common reward model using Equation (2), it could result in poor generalization for particular groups, especially with significant group-wise disparities in the data (see Figure 1 ). Such disparities might stem from imbalanced data across groups or difficulties associated with learning different groups.GRPO Objective. Consequently, we propose to measure the alignment of the reward model on the worst-case group loss:EQUATIONIncorporating the reward expression from (Equation (4)) into (Equation ( 6)), we establish the group robust preference optimization (GRPO) objective for a specified policy π:EQUATIONLeveraging the equivalent formulation of maximizing over discrete set, the GRPO problem becomesEQUATIONwhere ∆ K-1 represents the (K -1)-dimensional simplex of probabilities. 2 The inner maximization becomes a linear programming over simplex such that α represents the weights of groups. In addition, it forms a two-player zero-sum game (see Section 3.1), where the policy π and α act as opponents with inversely related payoffs. The DPO loss (logistic log loss) in Equation ( 7) can be replaced with alternatives like hinge or squared loss (see [47] ). We label this objective GR-DPO when using DPO loss, and explore GRPO with squared loss in Section 4.1.Applications. In this study, we do not assume any specific distribution for groups D g . The collection of prompts per group, P xg , may have varying degrees of overlap. The GRPO framework accommodates both distinct and overlapping prompt scenarios across different groups.Apart from human groups, GRPO can be useful in scenarios where groups D g represent distinct tasks or topics within preference datasets, like helpful/harmful, truthful/unanswerable instances, or domainspecific categories (e.g., math, physics, chemistry). Typically, these prompt distributions {P xg } N g=1 are disjoint, and GRPO seeks to optimize performance even across the most challenging categories.GRPO is also applicable in scenarios where groups reflect diverse user preferences for a shared set of prompts, with the goal of achieving equitable performance across user groups. This contrasts with non-robust DPO, which aims to optimize preferences on average and might overlook minority groups.Lastly, we acknowledge that the max-min objective of Equation ( 8) might be overly conservative, potentially degrading average performance. We explore a more balanced approach between worst-case and standard preference optimization objective in Appendix B.4.This section provides two insights regarding the GR-DPO loss in Equation (8) .Log-linear policy class. The zero-sum game perspective allows us to explore the presence of a Nash equilibrium, serving as a benchmark for convergence during the policy optimization process. Given that the domain of α is a simplex ∆ K (in Equation ( 8)), we further define a parameterized policy class Π θ for the policy π θ . We assume that the parameterized policy π θ is of the form π θ (y| x) = exp f θ (x,y) y∈Y exp f θ (x,y), where f θ is a linear function or a neural network, and θ belongs to a convex set Θ. In LLM fine-tuning, sometimes practitioners concentrate on modifying solely the final layer. It corresponds to a linear function, f θ (x, y) = θ T ϕ(x, y), with ϕ(x, y) denoting the embedding derived from the language model removing its last layer, and θ as the parameters of the last layer. When applying this linear parameterization, in conjunction with a uniform reference policy π ref , the robust objective outlined in Equation ( 8) is as follows (details in Appendix B.1):EQUATIONThe objective defined in Equation ( 9) is concave with respect to α and convex with respect to θ. This structure allows the invocation of the minimax theorem for convex-concave functions ( [42] ) to assert the existence of a Nash equilibrium. Proposition 3.1. Under log-linear parameterization of the policy class, there exists a Nash equilibrium for the group robust direct preference optimization problem in Equation (9) .Robust policy optimization. The earlier derivation for the GR-DPO objective L GR-DPO (π) relies on incorporating robustness in the reward modeling step (in Equation ( 7)) while using the solution to the non-robust KL-regularized reward maximization objective in Equation (4).Interestingly, we can obtain the identical expression for L GR-DPO (π) if incorporating robustness in the KL-regularized reward maximization objective and using the reward function learnt in a non-robust way. Consider the robust KL-regularized reward maximizationEQUATIONThe following proposition characterizes such an invariant property. 2 From a distributionally-robust viewpoint, this is equivalent to defining the uncertainty set as K g=1 αgDg : α ∈ ∆K , and minimizing the worst-case expected loss across the uncertainty set.Algorithm 1 Mirror Descent for Group Robust Preference Optimization (GRPO) 1: Initialize: Step size η α for group weights α, step size η θ for policy π with weights θ, initial weights θ (0) of the policy and weights over each group α (0) , Projection operator P Θ 2: Input: Dataset D with size N = |D|, group size N g for g = {1, 2, • • • , K}, loss l(π θ ; •) 3: for t = 1, . . . , T do 4: α ′ ← α (t-1) 5: g ∼ Categorical(N 1 /N, • • • , N K /N ), (x g , y w , y l ) ∼ D g 6: α ′ g ← α ′ g exp η α N •l(π θ (t-α (t) ← α ′ / g ′ α ′ g ′ // Renormalize α 8: θ (t) ← P Θ θ (t-1) -η θ N α (t) g ∇ θ l(π θ (t-1) ;(xg,yw,y l )) Ng// Use α to update θ 9: end for 10: Return: Output the robust policy π(θ (T ) ) Proposition 3.2. Substituting the closed-form solution of the robust KL-regularized policy maximization problem (Equation (10)) into the robust reward maximization objective in Equation (6) leads to the same group robust DPO loss L GR-DPO in Equation (8) .The analysis leverages the fact that the optimal policy of Equation ( 10) is identical to the solution of the non-robust KL-regularized reward maximization in Equation ( 4) and is derived in Appendix B.2.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer We are using open-sourced data and language model. We do not foresee any potential harm caused by our work. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the broader positive societal impacts of our work in Section 6. We do not foresee any negative societal impacts of our work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
algorithm: 
 In this section, we discuss the policy optimization algorithm for solving the group robust DPO problem in Equation (8) . In particular, we aim to design an algorithm that performs updates in the parameterized space Θ ⊂ R d , i.e., updating θ of the parameterized policy π θ . Leveraging the perspective of the 2-player zero-sum game, we propose an alternating updating algorithm wherein one updates α and θ alternatively. We summarize the overall approach in Algorithm 1, which we discuss and analyze next.We employ the DPO loss l(π θ ; •) = log (σ(βh π θ (•))) (Equation ( 5)) in Algorithm 1, however, our algorithm can support other preference optimization losses (see Section 4.1). The algorithm performs a gradient descent type update on θ and a deterministic mirror ascent on α using a Bregman divergence with the distance generating function as the KL divergence. Since the α lies in a simplex and the objective is linear, the update of α becomes multiplicative weights update with renormalization to a simplex via softmax (see Nemirovski et al. [32] for details). Further, the weights α are determined by the cumulative losses l(π θ ; •) accrued by each group, ensuring that groups with higher cumulative losses get higher weights. The size of the group N g appears as the empirical distribution D g involves N g . We call it alternating update as the updated α t is used in the update from θ t-1 to θ t . In particular, the gradient descent type update on θ is weighted by α in order to orient the update towards groups with higher losses. The projection operator P Θ ensures the updated θ t lies within Θ.What does the weighted DPO update do? In Line 9 in Algorithm 1, the algorithm performs parameter updates based on the weighted gradients. By using the DPO loss, i.e., l(π θ ; •) = log σ(βh π θ (•) (see Equation ( 5)), we obtain the following gradient update expression ignoring the N/N g constantα (t) g ∇ θ l(π θ (t-1) ; (x g , y w , y l )) = α (t) g ∇ θ log σ(βh π θ (t-1) (x g , y w , y l )) (11) = α (t) g σ r θ (t-1) (x g , y l ) -r θ (t-1) (x g , y w ) × [∇ θ log π θ (t-1) (y w |x g ) -∇ θ log π θ (t-1) (y l |x g )].The final term plays the critical role of enhancing the likelihood of the preferred response while simultaneously diminishing the likelihood of the rejected response. This adjustment is proportional to the disparity in rewards between the two responses. Moreover, the inclusion of α g is pivotal for ensuring group robustness. This coefficient scales the gradient w.r.t. θ based on the cumulative loss previously received by all samples within a specific group. Such a mechanism ensures that the model's focus is increasingly directed towards groups that have historically suffered higher losses. Additionally, the scaling factor N g guarantees that groups with a smaller volume of data do not face a disadvantage.We defer further details in obtaining the gradient update expression to Appendix B.3.We demonstrate the global convergence with the following proposition. Proposition 4.1. Suppose that the loss l(•; (x g , y, y ′ )) is non-negative, convex, B ∇ -Lipschitz continuous, and bounded by B l for all (x g , y, y ′ ) ∈ X ⊕ G × Y × Y and ∥θ∥ 2 ≤ B Θ for all θ ∈ Θ with convex Θ ⊂ R d . The error of the average iterate of Algorithm 1, i.e., πθ(1:T) = 1 T T t=1 θ t , satisfies E[L GR (πθ(1:T) )] -min θ∈Θ L GR (π θ ) = O T -1/2 .We defer the proof of this proposition to Appendix E. The analysis follows from an adaptation of the analysis in Nemirovski et al. [32] for the proposed sampling strategy in Algorithm 1 3 . We note that when fine-tuning only the final layer of a LLM, the output policy exists within the log-linear policy class (see Section 3.1), and the corresponding loss function satisfies the assumptions in Proposition 4.1 (see Lemma E.1).The standard regularized reward maximization objective (Equation ( 3)) in DPO [37] , tends to overlook the KL-regularization and learn deterministic policies. This learned policy assigns preference probability one to winning responses in the data which is often not realistic (see [1] [Section 4.2] and Appendix C). Recently, Azar et al. [1] show that the standard regularized reward maximization objective (Equation ( 3)) in DPO [37] tends to overlook the KL-regularization and learn deterministic policies (see [1, Section 4.2] and Appendix C). They thus propose an alternative approach called Identity Preference Optimization (IPO) that is more likely to learn a randomized policy which assigns appropriate probability to the preferred response and prevents overfitting. Following a similar derivation as we did for group robust DPO with details given in Appendix C, we develop the corresponding group robust IPO (GR-IPO):min π L GR (π) := max g∈G L IPO (π, D g ) = max α∈∆ K-1 K g=1 α g E (xg,yw,y l )∼Dg h π (x g , y w , y l ) -1 2β 2 .For the log-linear policy class (introduced in Section 3.1), the objective function simplifies tomin θ∈Θ max α∈∆ K-1 K g=1α g E (xg,yw,y l )∼Dg ⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ -1 2β 2 .To solve the GR-IPO above, it suffices to use Algorithm 1 with slight modifications, see Algorithm 2 in Appendix C. In particular, the update of θ is replaced by a weighted regression update:θ ← arg min θ∈Θ (xg,yw,y l )∼D α g N g ⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ - 1 2β 2 .For fixed α, we show (in Appendix C) that such an update admits a closed-form solution:θ = 1 2β (S T W S) -1 S T W 1 with W := Diag α g (1) N g (1) , . . . , α g (N ) N g (N ),where g (i) is the group of each sample i, N g (i) is the number of samples in group g (i) and 1 is a column vector of ones of dimension N . Here S is a matrix S := (ϕ(xg , yEQUATIONg , yl )) T , . . . , (ϕ(xEQUATION)) T . Each row of S represents the difference in feature mappings ϕ of the preferred and less preferred response for each prompt. The group robust IPO (GR-IPO) algorithm is presented in Appendix C, and its empirical results are shown in Section 5. 
experiments: 
 In this section, we study the empirical performance of our proposed Algorithm 1 on synthetic and real-world datasets 4 . First, we simulate multi-group data disparities by varying the size and preference distributions of two synthetic groups. In the real-world setup, we study the alignment of an LLM to the real preferences of people from various countries. We examine whether GRPO aligns the LLM in a more equitable manner to reduce discrepancies in alignment among various groups. Finally, we demonstrate that performance is improved by explicitly addressing the grouped nature of the data during alignment.We evaluate the performance of Algorithm 1 using synthetically generated group preference data for the loss function l(π θ;• ) -either DPO loss or IPO loss and denote them as GR-DPO and GR-IPO, respectively. We compare them against vanilla DPO and IPO ( [37, 1] ), and the importance-sampling (IS) variants of DPO and IPO (where the loss of each datapoint is inversely weighted by its group data size).Experimental Setup. Our experiments are designed to analyze settings where there exist multiple groups with distinct characteristics. We adapt the standard (non-group based) experimental setup proposed by [26] for the group preferences setting by incorporating group information into the reward function r :X × Y × G → R. Here, X represents a two-dimensional state space [0, 1] × [0, 1],Y denotes a discrete action space {0, 1, 2, 3, . . . , n}, and G signifies a discrete group space {0, 1, 2, . . . , K}. The reward function, defined by the group-dependent feature vector ϕ(x, y, g) and parameter vector θ g , is given as r(x, y, g) := ⟨ϕ(x, y, g), θ g ⟩, while the feature vectors ϕ(x, y, g) have a coordinate-flipped relationship and are defined in Appendix D.1.We consider the following scenarios: (i) Groups are imbalanced in terms of size but have the same distribution over responses, (ii) Groups are balanced in terms of size but have different response distributions, and (iii) Groups are imbalanced in terms of size and also have different response distributions. Note that having different response distributions leads to a difference in the difficulty of learning, since groups with responses distant from each other (in terms of rewards or preference probabilities) are typically more distinguishable and easier to learn. We discuss in Appendix D.2 how we generate these three scenarios.Implementation. Leveraging the linearity in the reward model, we utilize a log-linear policy class parameterized by θ: π θ (y|x) = exp ⟨ϕ(x,y,g),θ⟩y ′ ∈Y exp ⟨ϕ(x,y ′ ,g),θ⟩ . We run Algorithm 1 for both DPO and IPO loss relative to the policy class detailed above with a dataset of 300 action pairs with preferences.Evaluation Metrics. We use the following criteria to assess the performance of the algorithms: Max Validation Loss. For each group g, with preference data denoted as(x i , y i w , y i l , g) Ng i=1, where N g is the number of data points in the group, we compute the DPO/IPO validation loss separately for each group and identify the maximum loss among them in each run.Max Reward Error. This metric compares the true reward of the optimal action determined by θ * g with that of the action deemed optimal by estimate θ for each group, and identifies the maximum error across all groups in each run. In particular, for data in the form (x i , g) Ng i=1 , which includes only states and groups, we calculate reward errors for every group g as follows:E (x,g)∼(x i ,g) Ng i=1 [max y ⟨ϕ(x, y, g), θ * g ⟩ -⟨ϕ(x, arg max y ⟨ϕ(x, y, g), θ⟩, g), θ * g ⟩].Results. We present the average performance of Algorithm 1 (error bars over 20 seeds) alongside baseline methods in Figure 2 for scenario (iii), while scenarios (i) and (ii) are Figures 4 and 5 in Appendix D.2 due to space constraints. Our findings indicate that the robust methods consistently surpass both vanilla and importance-sampling approaches. Notably, the robust methods demonstrate significant superiority in uneven group scenarios, where the importance-sampling technique falls short as it exclusively deals with data imbalance.For the real-data experiments, we consider the survey dataset GlobalOpinionQA ( [15] ) and the publicly available Gemma-2B model [48] . 5 The data contains multiple choice questions answered by participants from various countries, amounting to 2,554 questions covering various topics, including politics, media, technology, religion, race, and ethnicity. For each question, the dataset provides a probability vector over the choices, signifying the percentage of people from a particular country choosing each option. Note that this probability vector would be different for different countries. Hence, the goal is to align the LLM to the probability vector corresponding to each country in a robust manner.We consider the following five countries in the dataset: Nigeria, Egypt, India, China and Japan, with data sizes 572, 570, 376, 309, and 712, respectively. We construct our training set as follows: For the SFT training, we choose the best option (the choice with the highest probability) as the target.For both IPO and GR-IPO training, we consider the best option as the winning response and another randomly chosen option as the losing response. We outline the exact prompt we use in Appendix D.We run the SFT training for one epoch over the training data on the pre-trained Gemma-2B model. For both IPO/GR-IPO training we use the AdamW [27] optimizer with adaptive learning rates. For SFT/IPO/GR-IPO training, we apply the LoRA strategy to fine-tune all layers of the model. We then evaluate both the methods based on the worst group loss and accuracy. Here, the loss refers to the IPO loss for each group and the accuracy refers to the percentage of winning response and losing response pairs correctly ordered by the learned preference function (Equation ( 35)). We defer further training and hyperparameter details to Appendix D.Results. We present the average performance of GR-IPO over five seeds alongside IPO in Figure 3 (top plots). Our findings indicate that GR-IPO outperforms IPO in terms of maximum group loss and minimum group reward accuracies. Moreover, GR-IPO effectively reduces the imbalance in loss values among different groups. Additionally, we observe an improvement in log-probability accuracies (which measure if the probability assigned by the fine-tuned model is higher for the winning response compared to the losing response) for both IPO and GR-IPO, with GR-IPO demonstrating better alignment for the worst-performing group compared to IPO.Insights. We further note that the worst-performing groups are Groups-2,5, as shown in Figure 3 . GR-IPO improves the loss for these groups by assigning more weight to them, as illustrated in Figure 3 (bottom middle plot). Additionally, we plot the initial log-probability accuracies for different groups in Figure 3 (bottom right plot), assessing how accurately the SFT model classifies the winning versus losing response for different groups. It is evident that Groups-2,5 are already underperforming.Given that SFT training converges within one epoch without any discrepancies between groups, this indicates that the base LLM inherently struggles with classifying responses for Groups-2,5. However, by employing GR-IPO, we have mitigated the imbalance in the performance of the fine-tuned model. 
conclusions: 
 We formalize the problem of robustly aligning an LLM to preference distributions from diverse groups.To tackle the same, we introduced GRPO, a group robust formulation of reward-free RLHF, aiming to minimize worst-case loss among groups. We explored the theoretical aspects of GRPO and demonstrated its improved robust alignment performance through various experiments. We believe our approach will be highly valuable for future tailored LLM fine-tuning, specifically aimed at aligning with the needs of diverse teams and user groups. In a broader context, it holds promise for mitigating biases and discrepancies across various societal groups encountered in the task-specific adaptation of LLMs.Limitations. When the dataset is balanced among groups and difficulty levels are comparable, our GRPO approach does not offer a significant advantage over standard reward-free RLHF algorithms. In addition, minimax methods often improve the worst group's performance at the cost of reducing the average or best group's performance. Our proposed GRPO formulation (see Equation ( 8)), stemming from a minimax framework, will comply with the same property. Hence, in scenarios where optimizing worst-case performance is less critical, we define a trade-off parameter to balance between the worst-case performance and the average performance. This modified objective and the necessary algorithmic changes are elaborated in Appendix B.4. The appropriate tuning of the trade-off parameter for the specific application remains a subject for future investigation. Further, we focus on settings with known groups, which are common in pluralistic alignment datasets and tasks (see [57, 44] ). When groups are unknown, one can still undertake several approaches such as clustering, representation learning, feature analysis, expert consultations, etc., to help uncover group structures in the data. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We are using publicly available model and data.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the data and models used have been appropriately cited with URLs which includes the licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Our code and algorithm implementation has been properly documented and links to the Github repositories have been provided. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We detail a more extensive related work exposition in this section.The Reinforcement Learning from Human Feedback (RLHF) is a concrete way to adapt a LLM towards specific tasks or objectives using human feedback. This framework for fine-tuning LLMs is inspired from [11] that uses human feedback to improve reinforcement learning policies. For LLMs, it was first established in [60] and further developed in [45, 34] . It involves learning a reward function using the pairwise human feedback data that provides comparisons between responses for a given prompt. This reward learning is performed through the Bradley-Terry model [5] . This learned reward model is then used as an objective for optimizing the LLM policy using Proximal Policy Optimization [40] .However, obtaining high-quality human feedback data is expensive and several works such as [35, 2, 25] study the usage of AI feedback to reduce reliance on human input. Further, studies like [13, 28, 22] analyze the usage of active learning strategies to minimize the amount of human feedback data required to perform RLHF. For a comprehensive overview and perspective of the RLHF topic, with detailed discussions on various approaches and gaps, we refer the reader to [24, 7, 23] .Due to the inherent issues of tuning the hyperparameters of a PPO ( [50, 37] ) and susceptible nature of reward models ( [17, 49] ), alternative approaches to the PPO-based RLHF have been proposed. Foremost of them is the rejection sampling fine-tuning ( [14, 18, 49] ) which is inspired by the best-of-n inference technique ( [30] ). The technique involves sampling n responses per prompt and fine-tuning the model based on the highest scoring ones in terms of a learned reward function. Further works such as [20, 54] avoid reward learning and propose conditional Supervised Fine-Tuning (SFT) inspired by the reward-conditioned RL ( [9] ).Another strategy proposed as an alternative to PPO-based RLPHF is DPO ( Rafailov et al. [37] ), wherein one optimizes the policy directly based on human preferences, bypassing the need for learning a separate reward model. This method simplifies the training process and potentially reduces overfitting and model misalignment issues. Theoretical advancements in understanding the dynamics of DPO have been provided by [21, 53] , who analyzed the convergence and stability of DPO policy learning. Some other works such as [1, 58, 47, 43, 16] also study a similar reward-free RLHF setup. In particular, [1] analyze the potential overfitting issues in DPO. Further, [16, 6] propose strategies that bypass the need for preference datasets. [38, 55, 29] adopt a game-theory perspective to learn Nash equilibrium policies. Typically, the reward-free RLHF approaches utilize a reference policy trained via supervised fine-tuning, which is then further optimized. However, [19] propose Monolithic Preference Optimization without reference model, bypassing supervised fine-tuning entirely. [59] address the RLHF problem from an MDP framework, introducing a DPO-type algorithm integrated with Proximal Policy Optimization (PPO). Additionally, works such as [51, 46, 10, 41, 56] investigate self-play preference optimization, where new data generated in each round by the current policy is used to train an improved policy. Our work utilizes a reward-free framework similar to [37, 1] but differs from all the above works in the following sense. Unlike previous works that assume a single preference distribution, we consider fine-tuning a LLM given data from multiple preference distributions across diverse groups. Further, we aim to fine-tune the LLM in a robust manner such that there is minimal imbalance in performance across all groups.Robust language modeling technique in the context of language model was first studied in [33] to optimize performance over a wide-range of topics. It was further improved by [52] wherein they use smaller model to learn the weights/importance that needs to be assigned to different topics and train a larger model based on these weights. Both works aim to minimize the worst group loss based on the group-DRO approach and focus on the pre-training aspects of language models. Concrete theoretical study of this group-DRO approach was performed in [39] which studies the minimax formulation of this group-DRO problem based on [31] . In the context of online batch selection, [4] propose a robust data downsampling method that performs class-aware priority reweighting to reduce training costs while preserving worst-class performance.In the RLHF setup, recent research has increasingly focused on enhancing the robustness of policy learning to address certain inherent gaps in the RLHF framework as detailed in [7] such as reward hacking and model misspecification. In particular, [3] formulated a weighted group loss for reward model learning w.r.t. a parameter λ that determines the importance assigned to a group and apply it to the anthropic Harmless vs Helpful data ( 6). In a similar manner, [12] design a weighted objective for policy optimization w.r.t. parameter λ to trade-off the importance between task reward maximization and safety cost minimzation. Another work [8] consider alignment to different sub-populations by learning multiple reward functions corresponding to each sub-population and learning a robust policy (max-min) w.r.t. the reward functions. Distinct from previous works, our approach circumvents reward model learning and integrates group robustness directly into the reward-free tuning framework. This allows us to directly learn a robust policy without the need for learning reward models. To learn this robust policy, we present a concrete algorithm that adaptively assigns weights to the losses of different groups and optimizes the policy to minimize this weighted loss. Additionally, our algorithm features a novel gradient estimator specifically designed for the group robust DPO problem.Other studies addressing robustness in preference optimization, such as [21] and [26] , focus on different facets of robustness, including robustness to noise and resilience against out-of-preference data. In the non-robust group preference alignment setup, [57] explores learning group preferences with LLMs by incorporating group information into prompts. However, their approach does not involve fine-tuning a LLM but training a separate transformer module that optimally selects an example sequence of prompts, LLM responses, and group preferences for in-context learning using a LLM. Also, Wang et al. [49] consider alignment with user preferences assuming that each user/group has varied importance over the distinct metrics in their multi-objective reward model. The output policy is trained to output a response based on both the prompt and the importance/weights over the individual metrics. In contrast, the distinctive feature of our method is that we are not modeling multi-reward objectives but consider a reward-free setting. Specifically, we consider the robust alignment problem optimizing for the worst group performance. And, our methodology directly models each group's preferences through a group-dependent latent reward model where the group dependency is injected through the prompt.In this section, we detail the proofs for the theoretical insights elucidated in Section 3 and the weighted DPO gradient expression discussed in Section 4.Here, we describe obtaining the robust objective for the log-linear policy class as in Equation ( 9). Starting from the robust objective for a general policy class in Equation ( 8), L GR can be specialized as follows:L GR (π) = max α∈∆ K-1 K g=1 α g -E (xg,yw,y l )∼Dg log σ(βh π (x g , y w , y l ) (i) = max α∈∆ K-1 K g=1 α g -E (xg,yw,y l )∼Dg log σ(β log( π(yw|xg) πref(yw|xg) ) -β log( π(y l |xg) πref(y l |xg) ) (ii) = max α∈∆ K-1 K g=1 α g -E (xg,yw,y l )∼Dg log σ(β log(π(y w |x g )) -β log(π(y l |x g )) (iii) = max α∈∆ K-1 K g=1 α g -E (xg,yw,y l )∼Dg log σ(β log( exp f θ (xg,yw) y∈Y exp f θ (xg,y) )- β log( exp f θ (xg,y l ) y∈Y exp f θ (xg,y) ) (iv) = max α∈∆ K-1 K g=1 α g -E (xg,yw,y l )∼Dg log σ(β log(exp θ T ϕ(x g , y w ))- β log(exp θ T ϕ(x g , y l )) = max α∈∆ K-1 K g=1 α g -E (xg,yw,y l )∼Dg log σ β⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ .Here, (i) follows from the definition of h π (x g , y w , y l ) = log(π(yw|xg) πref(yw|xg) )-log( π(y l |xg)πref(y l |xg) ), (ii) follows from the fact that for a uniform reference policy π ref (y w |x g ) = π ref (y l |x g ), (iii) follows from substituting π θ (y|x g ) = exp f θ (xg,y) y∈Y exp f θ (xg,y) , and (iv) follows from substituting f θ (x g , y) = θ T ϕ(x g , y).In this section, we consider the robust version of the classical KL-regularized reward maximization objective (Equation ( 3)) detailed in Equation ( 10) for a reward function r(x g , y), reference policy π ref , and a general class of policies Π.EQUATIONLemma B.1. The optimal policy π * for the robust KL-regularized reward maximization objective (Equation ( 12)) for a reward function r(x g , y), reference policy π ref isEQUATIONwhere Z(x g ) = y π ref (y|x g ) exp( 1 β r(x g , y)) is a partition function.Proof. We recast Equation ( 12) based on Equation ( 8) and perform the following analysis similar to [37] [Appendix A.1]:EQUATION= maxEQUATION= -β minEQUATIONFurther, by defining π * (y|x g ) = ), we can rewrite Equation ( 17) as:EQUATION=β minEQUATIONEQUATIONwhere we useroL(π) := max α∈∆ K-1 K g=1 α g E xg∼Px g D KL π(•|x g )||π * (•|x g ) -log(Z(x g )) .It is not hard to show that roL(π) is minimized by π * . Indeed, let us consider any other policy π ̸ = π * . Then, we haveEQUATIONEQUATION= β maxEQUATION= roL(π * ),where the inequality follows since D KL π(•|x)||π * (•|x) ≥ 0. This implies π * minimises Equation (20) .Note that, we have proved that the optimal policy expression for Equation (12) aligns with the form of the optimal policy for standard (non-robust) KL-regularized reward maximization Equation (4). Proposition 3.2. Substituting the closed-form solution of the robust KL-regularized policy maximization problem (Equation (10)) into the robust reward maximization objective in Equation (6) leads to the same group robust DPO loss L GR-DPO in Equation (8) .Proof. In [37] , the DPO loss (see Equation ( 5)) is obtained by using the following reward loss:EQUATIONand replacing r(•, •) with the relationEQUATIONwhich is the solution toEQUATIONHowever, in our group-robust formulation, we replace Equation (25) withEQUATIONwhile Equation ( 27) remains the same. Alternatively, we can replace Equation ( 27) with its robust version, i.e.,EQUATIONhowever, we have already shown, in Lemma B.1, that this does not change the obtained policy-reward relation from Equation (26) . Hence, this proves Proposition 3.2.We elucidate the steps to obtain the expression for the loss gradient in Equation (11) . We can simplify the gradient as follows:α t g ∇ θ l(π θ t-1 ; (x g , y w , y l )) n g = α t g ∇ θ log σ(βh π θ t-1 (x g , y w , y l )) n g (i) = α t g n g σ ′ (βh π θ t-1 (x g , y w , y l )) σ(βh π θ t-1 (x g , y w , y l )) × [βh ′ π θ t-1 (x g , y w , y l )] (ii) = βα t g n g 1 -σ(βh π θ t-1 (x g , y w , y l )) × [h ′ π θ t-1 (x g , y w , y l )] (iii) = βα t g n g σ -βh π θ t-1 (x g , y w , y l ) × [h ′ π θ t-1 (x g , y w , y l ) ] (iv) = βα t g n g σ β log( π θ t-1 (y l |x)πref(y l |x) ) -log(π θ t-1 (yw|x) πref(yw|x) ) × [h ′ π θ t-1 (x g , y w , y l )] (v) = α t g n g σ r θ t-1 (x g , y l ) -r θ t-1 (x g , y w ) × [∇ θ log π θ t-1 (y w |x g ) -∇ θ log π θ t-1 (y l |x g )]Here, (i) follows from the derivative of log(•) function and denoting the derivative of sigmoid function σ(•) as σ ′ (•). (ii) and (iii) follow from the fact that for any s ∈ R, σ ′ (s) = 1 -σ(s) = σ(-s).Finally, (iv) and (v) follow using the definition of h π (x, y w , y l ) = log( π(yw|x) πref(yw|x) ) -log( π(y l |x) πref(y l |x) ), and substituting r θ (x g , y) = β log( π θ (y|xg) πref(y|xg) ) -β log Z(x g ) from Equation (4).In the robust objective of Equation ( 8), we intend to minimize the worst-case loss. This often might adversely impact the average loss leading to bad average performance across the different groups. To mitigate this, we propose the following robust trade-off direct preference optimization objective for a specified policy π and set of group weights µ 1 , . . . , µ K :L GR,χ (π) :=(1 -χ) K g=1 µ g L DPO (π, D g ) + χ max g∈G L DPO (π, D g ) =(1 -χ) K g=1 µ g -E (xg,yw,y l )∼Dg log σ βh π (x g , y w , y l ) + χ maxg∈G -E (xg,yw,y l )∼Dg log σ βh π (x g , y w , y l ) .Here, note that the input weights µ g can be equal for all groups leading to a trade-off between the average and worst-case performance w.r.t. parameter χ. Following Equation ( 8), this can be equivalently recast into:EQUATION= maxEQUATIONwhere ∆ K-1 represents the (K -1)-dimensional simplex of probabilities. Hence, minimizing the loss in Equation ( 31) implies that one can implicitly find an optimal reward function using human group preference data that is robust without losing average performance, and obtain a policy that works effectively in terms of both average and worst-case performance:EQUATIONAs Equation ( 8) involves maximizing with respect to α and minimizing with respect to π, it forms a two-player game similar to the one considered in Section 3.1, where the policy π and α act as opponents with inversely related payoffs. However, the contributions of the maximizing player (α) on the final outcome are limited depending on parameters χ and µ.Within the context of a finite data setting, it is common to record two responses (y, y ′ ) for each prompt x. Based on these observations, there might be a tendency to inaccurately deduce that the preference distribution fulfills p(y ≻ y ′ |x) = 1. Consequently, and assuming the Bradley-Terry model of preferences, the reward function r has to satisfy r(y, x) -r(y ′ , x) → ∞. Further, given that DPO policy π is directly dependent on the reward function (Equation ( 4)), it holds that π(y ′ |x) π(y|x) = 0. In such a case, the policy overfits irrespective of the KL regularization factor. To circumvent this, [1] consider reward-free RLHF problem. Specifically, they propose preference optimization in the policy objective instead of reward optimization as follows:EQUATIONHere, Ψ : [0, 1] → R is a non-decreasing function, ρ refers to the distribution of prompts and, µ refers to the competing behavior policy. Choosing Ψ as the identity function, the optimal policy π * of Equation ( 34) has a similar expression to Equation (4) with the reward function r(•) substituted by the following preference function p(•),EQUATIONRecalling the definition of h π (x, y, y ′ ) = log( π(y|x) πref(y|x) ) -log( π(y ′ |x) πref(y ′ |x) ) from Section 2, it follows from Equation ( 35) that π * adheres to the following identity that equates the true preference with the policy's preference of y over y ′ :EQUATION)where p(y ≻ µ|x) := E ȳ∼µ(•|x) p(y ≻ ȳ|x) . The equation motivates us to find a policy π that minimizes the squared differences of two sides in Equation (36) ,EQUATION)For an empirical dataset where one observes if y w is more preferable to y l , the policy minimization objective in Equation ( 37) becomes the IPO objective:EQUATION)We can simplify Equation (38) for the IPO Loss using Log-linear policy class similar to Equation ( 9) as followsEQUATION)Contrary to DPO, the IPO Loss results in a problem formulation equivalent to linear regression for the linear bandit setting. This leads us to obtain a closed form analytical solution for θEQUATIONwhere 1 = {1} N . Here, X ∈ R d×N . In particular, each row of X is the difference of the preferred and least preferred feature vectors ϕ(x, y w ) -ϕ(x, y l ) for (x,y w , y l ) ∈ X × Y × Y X =     ϕ(x 1 , y 1 ) -ϕ(s 1 , y ′ 1 ) ϕ(x 2 , y 2 ) -ϕ(x 2 , y ′ 2 ) . . . ϕ(x n , y n ) -ϕ(x n , y ′ n )     .When re-written in this form it is trivial to see that the stability of the IPO Loss depends upon the rank of the matrix X.Group IPO: We begin by conducting experiments with IPO considering the existence of closed form solution for this particular setting. In particular, given a set of preference data D = {D 1 , D 2 } from either groups with varying ratio, one aims to find an optimal θ that minimizes the group IPO loss,EQUATION+ E (x,yw,y l )∼D2⟨ϕ(x, y w , 1) -ϕ(x, y l , 1), θ⟩ -EQUATIONConcisely, one can also write this as, L IPO (π, D) = E (x,yw,y l ,g)∼D ⟨ϕ(x, y w , g) -ϕ(x, y l , g), θ⟩ -EQUATION)We assume that the feature vectors for each group is known, but the reward parameter θ is unknown. So, given the preference data with group information and the group dependent feature vectors, one aims to learn an optimal θ that balances both groups by minimizing the above loss. The solution to this will still be the closed form solution expressed in Equation ( 40) with feature matrix X consisting group dependent features.Group Robust IPO: It is straightforward to see that in such a setting, the group with higher number of preference data will have an unfair advantage as they contribute more to the loss. Hence, a robust approach needs to be considered as follows:EQUATION)One can use, Algorithm 1 to find the optimal θ for Equation (44) . But considering the weighted regression form of the loss, one can use the following simplified algorithm Algorithm 2. Note that the last step, inadvertently leads to a weighted regression whose solution can be obtained in closed form.In this section, we discuss additional experimental and training details. We use the following hyperparameters for the synthetic experiments. The importance sampling methods use the same hyperparameters as the corresponding vanilla ones. Further, we note that there is no learning rates for IPO and GR-IPO as we use the closed-form solution detailed in Section 4.1 for updates.Algorithm 2 Policy Optimization for Robust IPO in Linear Bandits 1: Initialize:Step size η α for group weights α, initial weights θ (0) of the policy and weights over each group α (0) 2: Input: Dataset D with size N = |D|, group size N g for g = {1, 2, • • • , K} 3: for t = 1, . . . , T do 4:Calculate group loss l g for each group g on D g 5: α ′ ← α t-1 ; α ′ g ← α ′ g exp (η G l g ) //α (t) ← α ′ / g ′ α ′ g ′ // Renormalize α 7:θ t ← arg min θ E(x,y w ,y l ,g)∼D α(t)g ⟨ϕ(x, y w , g) -ϕ(x, y l , g), θ⟩ - Here we provide a full definition of our synthetic experimental setup discussed in Section 5.We adapt the standard (non-group based) experimental setup proposed by [26] for the group preferences setting by incorporating group information into the reward function r :X × Y × G → R.Here, X represents a two-dimensional state space [0, 1] × [0, 1], Y denotes a discrete action space {0, 1, 2, 3, . . . , n}, and G signifies a discrete group space {0, 1, 2, . . . , K}. The reward function, defined by the group-dependent feature vector ϕ(x, y, g) and parameter vector θ g , is given as: r(x, y, g) := ⟨ϕ(x, y, g), θ g ⟩.Further, we consider n = 7, K = 2 and denote x := (x 0 , x 1 ) ∈ X . The feature vector ϕ(x, y, g) := (ϕ 0 (x, y, g), ϕ 1 (x, y, g), ϕ 2 (x, y, g), ϕ 3 (x, y, g)) and parameters θ g ∀g ∈ G are defined as follows:ϕ i (x, y, g) :=    y n+1 + 1 • cos(x ⌊i/2⌋ • π) if i%2 = g 1 y n+1 +1 • sin(x ⌊i/2⌋ • π) otherwise , θ 0 := (1, 3, 1, 3), θ 1 := (3, 1, 3, 1).By design, this parameterization ensures a coordinate-flipped relationship between groups, effectively mirroring one group to the other. Also, we study two other feature parameterizations and include their experimental results in Appendix D.3.In our synthetic experiments, we considered three scenarios: (i) Groups are imbalanced in size, (ii) preferences distribution, and (iii) both of the above. The size imbalance is generated by using ratios 0.2 : 0.8, respectively. Instead, to generate imbalance in terms of preferences we proceed as follows.In scenario (i), after randomly selecting a state x within group g and an action y 1 , a second action y 2 is randomly chosen (groups have the same distribution over responses). Instead, in scenarios (ii) and (iii), for one group y 2 is the action most distant from y 1 , while for the other, it is the closest (groups have different distributions over responses). In both methods, we calculate the rewards r(x, y 1 , g) and r(x, y 2 , g), designating the action with the higher reward as preferred.Here, we provide the resulting plots for scenarios (i) and (ii).We perform ablation studies for various values of χ trading-off between the average and worst-case performance. We observe in Figure 6 , that the max validation loss decreases while moving from χ = 0 to χ = 1, where χ = 0 corresponds to importance sampling with group weights µ 1 , • • • , µ g mapping to importance sampling weights (see Equation ( 30)) and χ = 1 corresponds to GR-IPO. Further, we plot the average validation loss which increases while moving from χ = 0 to χ = 1, demonstrating the trade-off between average and worst-case performance. Note that, GR-IPO aptly increases the average loss (as expected) in order to reduce the worst group loss.We present further experiments on synthetic preference data, using different configurations of the ϕ(x, y, g) and θ g vectors characterising the group-specific reward distributions r(x, y, g) = ⟨ϕ(x, y, g), θ g ⟩.With the same action space n = 7 and group space K = 2, we consider same and flipped configurations of ϕ(x, y, g) = (ϕ 0 , ϕ 1 , ϕ 2 , ϕ 3 ), as follows:Same: Here, the feature vectors are the same irrespective of group g and reward vectors are coordinateflipped.ϕ i (x, y, g) :=    y n+1 + 1 • cos(x ⌊i/2⌋ • π) if i%2 = 0 1 y n+1 +1 • sin(x ⌊i/2⌋ • π)otherwise , θ 0 := (1, 3, 1, 3), θ 1 := (3, 1, 3, 1).Flipped: Here, the feature vectors include alternating sin, cos terms with swapped order for g = {0, 1},with alternating coefficients. And the reward vectors are coordinate-flipped as before. We observe similar performance as in Figure 4 where we consider the same setting with swapped feature vectors for groups. GR-DPO slightly improves over importance sampling DPO, however, GR-IPO exactly matches the performance of importance sampling. And this is expected considering that the groups have the same level of difficulty and differ only in terms of data size. Here, GR-DPO/GR-IPO outperforms corresponding importance sampling methods in terms of worstcase validation loss, but, GR-IPO tends to overfit. This is reflected in the reward errors, where GR-IPO performs worse than IPO. However,overall, GR-DPO outperforms all other methods in terms of worst-case reward errors. We observe similar performance as in Figure 8 where we consider the same features for groups but with balanced data. Here, DPO is overfitting and GR-DPO outperforms both DPO and importance sampling DPO. IPO is stable, considering it has a closed form solution. And in terms of worst-case validation loss, GR-IPO tends to overfit, even though it outperforms IPO and importance sampling IPO. The overfitting is more evident in reward errors, where GR-IPO performs worse than IPO and importance sampling IPO. However,overall, GR-DPO outperforms all other methods in terms of worst-case reward errors. We observe very similar performance as in Figure 4 where GR-DPO/GR-IPO slightly improves over importance sampling DPO/IPO. Here, we observe a similar performance to that of Figure 2 where GR-DPO/GR-IPO outperforms corresponding vanilla and importance sampling methods in terms of worst-case validation loss. And, GR-DPO outperforms all other methods in terms of worst-case reward errors.ϕ i (x, y, g) :=        y n+1 + 1 -1•1 i%2=1 • cos(x ⌊i/2⌋ • π) if i%2 = g y n+1 + 1 -1•1 i%2=1 • sin(x ⌊i/2⌋ π)We use the following prompt type for both SFT and DPO training: "Opinion of people in (countryname) on: (question). Please select the best response: A) Choice-1, B) Choice-2, ...". The target response is just the letter 'A', 'B', etc. The training data size from each country are as follows: Nigeria-572, Egypt-570, India-376, China-309, Japan-712. Further, the data is split as 80% for training, 10% for validation and 10% for testing. We run the SFT training with learning rate 10 -4 for one epoch over the training data on pre-trained Gemma-2B model. We use this SFT trained model as the reference model for training IPO and GR-IPO. For the IPO training, the optimal hyperparameters were learning rate 3 * 10 -5 , and β = 0.01. For the GR-IPO training, we use the same β but the optimal learning rate and the exponential step size were 6 * 10 -5 and 5 * 10 -7 . For both IPO/GR-IPO training we use AdamW [27] optimizer with adaptive learning rates decreasing by a factor of 10 if there is no improvement in terms of loss (average group loss for IPO/ worst group loss for GR-IPO) for 4k iterations on a validation set. Further, for GR-IPO, the exponential step size is decreased by a factor of 2 whenever learning rate is reduced. This adaptive learning rates and exponential step sizes ensures stability in training and leads to convergence. We then evaluate both the methods based on the worst group loss and accuracy. Here, the accuracy refer to the percentage of prompt, winning response and losing response correctly ordered by the learned preference function (Equation ( 35)).All experiments were run on a single node of A100 SXM4 machine with 40GB GPU memory, 30 CPU cores, 200GB RAM, and 525GB SSD memory. Further, for each seed, the execution time of the experiments is approximately 3-5 hours until convergence. We also provide the exact hyperparameters used in the table below.Training Type Learning Rateβ Step Size Optimizer SFT 10 -4 - - RmsProp IPO 3 × 10 -5 0.01 - AdamW GR-IPO 6 × 10 -5 0.01 5 × 10 -7AdamW Table 2 : Hyperparameters for SFT, IPO, and GR-IPO training. Figure 13 : Global opinion data: Evolution of worst-case loss, reward accuracy during IPO and GR-IPO training. Notably, both IPO and GR-IPO improve their accuracy. However, GR-IPO achieves better worst-case alignment performance. In the right, we plot the end of training log-prob. accuracies for GR-IPO.In this section, we study the convergence rates for different sampling strategies including the proposed Algorithm 1.In this section, we prove the convergence of the proposed Algorithm 1 in terms of the error of the average iterate πθ(1:T) ,EQUATIONas stated in Proposition 4.1.Proposition 4.1. Suppose that the loss l(•; (x g , y, y ′ )) is non-negative, convex, B ∇ -Lipschitz continuous, and bounded by B l for all (x g , y, y′ ) ∈ X ⊕ G × Y × Y and ∥θ∥ 2 ≤ B Θ for all θ ∈ Θ with convex Θ ⊂ R d .The error of the average iterate of Algorithm 1, i.e., πθ(1:T) = 1 T T t=1 θ t , satisfies E[L GR (πθ(1:T) )] -min θ∈Θ L GR (π θ ) = O T -1/2 .Proof. We build our proof based upon the online mirror descent algorithm's regret bound in [32, Eq. 3.1]. [32] 's theorem considers the following saddle-point stochastic optimization problem,EQUATIONDenotef g (θ) = E[F g (θ, ξ)].In our problem, we consider,EQUATIONwhich is equivalent toEQUATIONand we define F g ′ (θ, (x g , y, y ′ )) := N Ng 1 [g=g ′ ] l(θ; (x g , y, y ′ )). Then, the expectationE ξ [F g (θ, ξ)] = K g ′ =1 N g ′ N E D g ′ [F g (θ, ξ)|g = g ′ ] = N g N E Dg [F g (θ, ξ)|g = g ′ ] = N g N E Dg N N g l(θ; (x g , y, y ′ ))|g = g ′ = E Dg [l(θ; (x g , y, y ′ ))|g = g ′ ] = f g (θ)Mirror Descent mapping. We begin by mapping Algorithm 1 to the general mirror descent framework of [32] .Further, we denote the gradients of the objective ϕ(θ, α) w.r.t. θ and α as∂ϕ(θ, α) = (∂ θ ϕ(θ, α), -∂ α ϕ(θ, α)).Here, the negative sign for the gradient w.r.t. α indicates that we perform maximization w.r.t. α. Let Φ(θ, α, ξ):= K g=1 α g [F g (θ, ξ)].Then the stochastic subgradients for ϕ(θ, α) for a given ξ is as follows:∂Φ(θ, α, ξ) = (∂ θ Φ(θ, α, ξ), -∂ α Φ(θ, α, ξ)).Simplifying the gradients we obtain the following for ξ = (x g , y, y ′ ):∂Φ(θ, α, ξ) = ∂ θ K g ′ =1 α g ′ [F g ′ (θ, ξ)] -∂ α K g ′ =1 α g ′ [F g ′ (θ, ξ)] = ∂ θ K g ′ =1 α g ′ [ N N g ′ 1 [g=g ′ ] l(θ; (x g , y, y ′ ))] -∂ α K g ′ =1 α g ′ [ N N g ′ 1 [g=g ′ ] l(θ; (x g , y, y ′ ))] = N αg Ng ∇ θ l(θ; (x g , y, y ′ )) -0, • • • , N Ng l(θ; (x g , y, y ′ )), • • • , 0 .Based on the above gradient expressions, we can map the updates of θ t and α t in Algorithm 1, to the general mirror descent update rule:EQUATIONHere, we use the compact notation ζ = (θ, α). Further, γ t denotes the step-size at time t of the algorithm and P ζ (•) is the prox-mapping corresponding to the mirror descent algorithm defined as follows:EQUATIONHere, V (•, •) is the prox-function associated with the distance-generating function (Bregmann Divergence) ω(•). For our setup in Algorithm 1, we define the following combined distance-generating function over ζ ∈ (Θ × ∆ K-1 ):EQUATIONwhere ω θ (θ) = 1 2 ∥θ∥ 2 , ω α (α) = K g=1 α i ln α i , D ω θ ,Θ := (max θ∈Θ ω θ (θ) -min θ∈Θ ω θ (θ)) 1/2 , and D ωα,∆ K-1 := max α∈∆ K-1 ω α (α) -min α∈∆ K-1 ω α (α) 1/2 . Also, the corresponding proxfunctions individually for θ and α would beV θ (θ, θ ′ ) = 1 2 ∥θ -θ ′ ∥ 2 2 and V α (α, α ′ ) = K g=1 α ′ i ln α ′ i αi .It can be shown that the above prox-functions V θ and V α satisfy the following inequalities ( [32] ):EQUATIONDue to the above definitions, the corresponding prox-mapping P (•) corresponds to gradient descent w.r.t. θ and exponentiated gradient ascent w.r.t. α, as defined in Algorithm 1. Moreover, due to the definition of ω in Equation ( 52), the combined prox function V (•, •) over ζ would satisfyEQUATIONFor further details regarding prox-function, kindly refer to its usage in [32] .Bounding the error. According to the introduced notation, the approximation error ϵ T can be defined and bounded as:ϵ T = max α∈∆ K-1 ϕ( θ1:T , α) -min θ∈Θ max α∈∆ K-1 ϕ(θ, α) ≤ max α∈∆ K-1 ϕ( θ1:T , α) -min θ∈Θ ϕ(θ, α1:T ) = ϵ ϕ ( ζ1:T )To bound ϵ ϕ ( ζ1:T ), we invoke the result from [32, Equation 3.23] for mirror-descent convergence for max-min problems.EQUATIONFurther, M θ and M α correspond to the following upper bounds to the maximum of the expected norms of F g (θ, ξ) and ∇ θ F g (θ, ξ) ,max 1≤g≤K E∥∇F g (θ, ξ)∥ 2 = max 1≤g≤K E N N g 1 [g=g ′ ] ∇l(θ; (x g , y, y ′ )) 2 = max 1≤g≤K N g N N 2 N 2 g ∥∇l(θ; (x g , y, y ′ ))∥ 2 = max 1≤g≤K N N g ∥∇l(θ; (x g , y, y ′ ))∥ 2 ≤ B 2 ∇ max g∈G N N g = B 2 ∇ N min g∈G N g = M 2 θ Similarly, E max 1≤g≤K ∥F g (θ, ξ)∥ 2 = E max 1≤g≤K N N g 1 [g=g ′ ] l(θ; (x g , y, y ′ )) 2 ≤ K g=1 N g N N 2 N 2 g ∥l(θ; (x g , y, y ′ ))∥ 2 = K g=1 N N g ∥l(θ; (x g , y, y ′ ))∥ 2 ≤ KB 2 l max g∈G N N g = KB 2 l N min g∈G N g = M 2 αHere, we have used ∥∇ θ l(θ; (x g , y))∥ ≤ B ∇ and ∥l(θ; (x g , y))∥ ≤ B l . Here, λ θ , λ α correspond to the strong convexity parameters of the distance generating functions ω θ (θ) and ω α (α) respectively. For our given functions, ω θ (θ) = 1 2 ∥θ∥ 2 and ω α (α) =K g=1 α i ln α i , both λ θ = λ α = 1 ([32]). Let, R 2 θ = D 2 ω θ λ θ = B 2 Θ = (max θ ∥θ∥ θ -min θ ∥θ∥ 2 θ ), obtaining the overall error bound.EQUATIONLemma E.1. For the log-linear policy class parameterized with respect to θ, the DPO loss function l(π θ ; •) = log (σ(βh π θ (•))) (see Equation ( 5)) is convex and Lipschitz continuous in θ. Consequently, Proposition 4.1 applies to this case.Proof. We want to show, l(θ; (x, y w , y l )) is convex in θ and B ϕ -Lipschitz in θ for the log-linear policy class defined as follows:EQUATIONHere θ belongs to a convex set Θ satisfying ∥θ∥ ≤ B Θ and ϕ(x, y) denotes the feature vector such that ∥ϕ(x g , y w ) -ϕ(x g , y l )∥ ≤ B ϕ . In conjunction with a uniform reference policy π ref , the loss function l(θ; (x g , y w , y l )) for the log-linear policy class is as follows: l(θ; (x g , y w , y l )) = -log σ β⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ .Algorithm 3 Mirror Descent for Group Robust Preference Optimization (GRPO)1: Initialize:Step size η α for group weights α, step size η θ for policy π with weights θ, initial weights θ (0) of the policy and weights over each group α (0) , Projection operator P Θ 2: Input: Dataset D with size N = |D|, group size N g for g = {1, 2, • • • , K}, loss l(π θ ; •) 3: for t = 1, . . . , T do 4:α ′ ← α (t-1) 5: g ∼ Uniform(1, • • • , K) 6:(x g , y w , y l ) ∼ D g 7:α ′ g ← α ′ g exp η α (l(π θ (t-1) ; (x g , y w , y l ))) // Update weights for group g 8:α (t) ← α ′ / g ′ α ′ g ′ // Renormalize α 9: θ (t) ← P Θ θ (t-1) -η θ α (t)g ∇ θ l(π θ (t-1) ; (x g , y w , y l )) // Use α to update θ 10: end for 11: Return: Output the robust policy π(θ (T ) )Next, we compute the derivative of l(θ; (x, y w , y l )) w.r.t. θ from Equation (58) to check the Lipschitz continuity.∇ θ l(θ; (x g , y w , y l )) = σ β⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ 1 -σ(β⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩) * (ϕ(x g , y w ) -ϕ(x g , y l ) σ β⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ = 1 -σ β⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ * (ϕ(x g , y w ) -ϕ(x g , y l ))= σ β⟨ϕ(x g , y w ) -ϕ(x g , y l ), -θ⟩ * (ϕ(x g , y w ) -ϕ(x g , y l )).Then, the gradient norm can be bounded as follows:∥∇ θ l(θ; (x g , y w , y l ))∥ ≤ ∥ϕ(x g , y w ) -ϕ(x g , y l )∥ ≤ B ϕ .Hence, by the definition of Lipschitz functions for continuously differentiable functions, we have that l(θ; (x, y w , y l )) is Lipschitz continuous with Lipschitz constant B ϕ . Further, we bound the loss function using the bounds for θ and ϕ(•, •).⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ ≥ -∥ϕ(x g , y w ) -ϕ(x g , y l )∥∥θ∥ ⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩ ≥ -B ϕ B Θ log(σ(⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩)) ≥ log(σ(-B ϕ B Θ )) -log(σ(⟨ϕ(x g , y w ) -ϕ(x g , y l ), θ⟩)) ≤ -log(σ(-B ϕ B Θ )) ≤ log(1 + exp (B ϕ B Θ )) ≤ log(exp (B ϕ B Θ ) + exp (B ϕ B Θ )) ≤ log(2) + (B ϕ B Θ ) = B lHence, the loss function is bounded by B l for bounded θ and ϕ.In this section, we study the convergence of an alternate algorithm in Algorithm 3, wherein groups are sampled uniformly rather than a categorical distribution proportional to the group sizes, in terms of the error ϵ T as defined in Equation (46) . Proposition E.2. Suppose that the loss l(•; (x g , y, y ′ )) is non-negative, convex, B ∇ -Lipschitz continuous, and bounded by B l for all (x g , y, y ′ ) ∈ X ⊕ G × Y × Y and ∥θ∥ 2 ≤ B Θ for all θ ∈ Θ with convex Θ ⊂ R d . Then, the average iterate of Algorithm 3 achieves an error at the rateEQUATIONProof. We follow a similar proof structure as in the proof of Proposition 4.1. We recall the saddlepoint stochastic optimization problem from [32] stated earlier in Equation (47) ,EQUATIONIn lieu of the alternate sampling strategy in Algorithm 3 where groups are sampled uniformly, we consider,EQUATIONand we define F g ′ (θ, (x g , y, y ′ )) := K1 [g=g ′ ] l(θ; (x g , y, y ′ )). Then, the expectationE ξ [F g (θ, ξ)] = K g ′ =1 1 K E D g ′ [F g (θ, ξ)|g = g ′ ] = 1 K E Dg [F g (θ, ξ)|g = g ′ ] = 1 K E Dg [Kl(θ; (x g , y, y ′ ))|g = g ′ ] = E Dg [l(θ; (x g , y, y ′ ))|g = g ′ ] = f g (θ)Mirror Descent mapping. We map Algorithm 3 to the general mirror descent framework of [32] as done in the proof of Proposition 4.1. We begin by recalculating the gradients of ϕ(θ, α) w.r.t. θ and α for this alternate definition of F g (θ, ξ),∂Φ(θ, α, ξ) = ∂ θ K g ′ =1 α g ′ [F g ′ (θ, ξ)] -∂ α K g ′ =1 α g ′ [F g ′ (θ, ξ)] = ∂ θ K g ′ =1 α g ′ [K1 [g=g ′ ] l(θ; (x g , y, y ′ ))] -∂ α K g ′ =1 α g ′ [K1 [g=g ′ ] l(θ; (x g , y, y ′ ))] = Kα g ∇ θ l(θ; (x g , y, y ′ )) -0, • • • , Kl(θ; (x g , y, y ′ )), • • • , 0 .Based on the above gradient expressions, we can similarly map the updates of θ t and α t in Algorithm 3, to the general mirror descent update rule with corresponding prox-function as done for Proposition 4.1.We omit further details regarding the mirror descent related definitions as they follow from the proof of Proposition 4.1. We directly proceed to bounding the error ϵ ϕ ( ζ1:T ) using [32, Equation 3.23],EQUATIONWe recalculate M θ and M α for this alternate definition of F g (θ, ξ) as they correspond to the upper bounds to the maximum of the expected norms of F g (θ, ξ) and ∇ θ F g (θ, ξ) , Guidelines:max 1≤g≤K E∥∇F g (θ, ξ)∥ 2 = max 1≤g≤K E∥K1 [g=g ′ ] ∇l(θ; (x g , y, y ′ ))∥ 2 = max 1≤g≤K 1 K K 2 ∥∇l(θ; (x g , y, y ′ ))∥ 2 = max 1≤g≤K K∥∇l(θ; (x g , y, y ′ ))∥ 2 ≤ B 2 ∇ K = B 2 ∇ K = M 2• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Corresponding author: shyam.ramesh.22@ucl.ac.uk.The group information added to the prompt can represent, e.g., an index or a textual description of the group.We study different sampling strategies and their error rates in Appendix E and we present the most numerically stable one in Algorithm 1.Codes for synthetic and real-world experiments can be found at https://github.com/rsshyam/GRPO-bandits and https://github.com/rsshyam/GRPO, respectively.The model and dataset are available on Hugging Face at https://huggingface.co/google/gemma-2b and https://huggingface.co/datasets/Anthropic/llm_global_opinions, respectively.The data is publicly avaiable in Huggingface website https://huggingface.co/datasets/Anthropic/hh-rlhf 


Decision: This paper brings attention to the existence of diverse preferences across different groups, proposing a robust RLHF method. Defining the optimization problem with the weighted loss for each group, the authors have discovered that the problem can be reduced to a minimax optimization problem. The minimax optimization problem is a convex-concave game, and by combining MWU and gradient descent updates, it can be guaranteed from classical results that the average iterate will converge to the equilibrium. When the policy is log-linear, essentially one can compute the best response of one player each round analytically (as a classical regression), resulting in a simplified algorithm. Through synthetic experiments and real-world data experiments, the authors demonstrate that their method performs best for the group that would likely have the worst loss, suggesting the significance of the approach in the real world. There was a concern from a reviewer about other applications and motivations, and questions about improved performance seen across all groups (in fact, the authors included an incorrect plot in the initial manuscript). In their rebuttal, the authors presented the correct experimental results, which were confirmed to be consistent. Overall, while the technical/experimental contributions are not particularly significant, the contribution of bringing the concept of group robustness to LLM alignment and conducting appropriate analysis is substantial, which is generally recognized by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper addresses the problem of improving fairness in preference optimization. It proposes a new loss function and algorithm. Experiments conducted show that the proposed algorithm indeed achieves better fairness. Additionally, the paper provides a theoretical analysis indicating the convergence of the optimization.
strength_weakness: strength_weakness: 1. The author mentioned that the idea can also be applied to multi-objective preference optimization. In my opinion, a comparison with works in multi-objective preference optimization [1] and group optimization [2] is needed.



2. It is unclear whether this algorithm can be applied to scenarios beyond multiple choices.

3. It is unclear how Proposition 4.1 relates to common metrics like regret.


[1]. Zhao, Siyan, John Dang, and Aditya Grover. "Group preference optimization: Few-shot alignment of large language models."

[2]. Wang, Haoxiang, et al. "Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards."
questions: questions: 1. The paper mentioned that the algorithms can also be used for multi-objective preference optimization. Were any related experiments conducted?
limitations: limitations: See the 'Weakness' part
paper_summary: paper_summary: This paper introduces GRPO, a method to optimize policy preferences across different groups in a robust way. GRPO looks at the worst group alignment loss by taking the maximum loss across all groups, ensuring the policy performs well even when there are group-specific differences or overlaps in prompts.
strength_weakness: strength_weakness: 1. GRPO's experiment on real world datasets is limited, on only one group (5 countries) of the GlobalQA dataset.
2. How is the training time as compared to other baselines? The GRPO framework's training process involves a min-max optimization, which can be potentially computationally intensive.
questions: questions: please see weakness.
limitations: limitations: 1. This methods requires knowing the number and nature of groups in advance to perform optimization. 
2. When new group is introduced, will have to train the entire optimization again, which is expensive.
paper_summary: paper_summary: The work tackles the important problem of robust RLHF for diverse groups. Traditionally, RLHF assumes that a single model can fit the diverse feedback from multiple groups of users. In this paper, the authors introduce a method to learn a robust policy that maximizes for worst-case group performance. To achieve this, the method adaptively weights the loss for each group based on the size and cumulative training loss incurred by the feedback samples for that group. As a result, LLMs trained on diverse group data demonstrate reduced loss imbalance and improved accuracies across all the groups. The authors also present a convergence analysis of the proposed method assuming a log-linear policy class.
strength_weakness: strength_weakness: 1. This paper proposes a method for group robust optimization for LLMs. However, the metrics evaluated are only max validation loss and reward error over the groups. GRPO uses a reward free approach to update the LLM, but the evaluations are restricted to the performance of the reward model over the feedback dataset. It would be nice to see the performance of the finetuned vs base model (such as win rate) in generating responses that align with the individual groups.
2. The authors provide detailed training setup, but I would suggest that they also include information about the evaluation. 
3. GRPO performs well over all the groups, however, the performance of the importance sampling baseline is very close. It would be helpful if the authors could provide additional ablations and discussions to show the effectiveness of optimizing against the worst-case loss over the groups vs using only the IS approach.
questions: questions: 1. In the current experiments, is only the final layer of the LLM trained? I would be curious to know the result if the method used full finetuning of the model.
2. As GRPO assumes that each prompt has access to group information through a prompt, how much does the prompt affect policy? If the prompts are good enough, would it just create non-overlapping distributions for all groups? As one of the problems in non-robust baselines is that they converge to unwanted biases or majority groups for shared prompts, if the group identification prompt is finetuned will it alleviate this issue altogether?
3. Does GRPO finetune only the final layer of the Gemma-2B model in the results presented in the paper?
4. Does GRPO ensure that there is a non-decreasing change in performance across all groups as compared to the non-robust baselines?
limitations: limitations: 1. The paper introduces GRPO, an optimization method to weight each group in the RLHF update step proportional to the size and loss of the group. This ensures a balanced performance of the model across all the groups. In the results, we see an improved performance over all the individual groups, which intuitively violates the no-free lunch theorem. I would be curious to see an analysis of the method that gives insights as to what allows the model or the objective to achieve consistently higher performance.
2. Here, the method assumes access the groups in the dataset. However, in practical settings the group information is unavailable and the model has to cluster or implicitly model the group information from the ungrouped dataset.
3. The evaluation is limited only to the accuracy of the reward model over the preference dataset. So, currently, it provides weaker evidence of the translation of this robustness during the generation phase. It would be nice if the authors could include experiments showing if GRPO enables LLMs to robustly generate better-aligned responses to prompts from all the groups.
paper_summary: paper_summary: This paper addresses the limitation of traditional reinforcement learning with human feedback (RLHF) approaches that indiscriminately optimize a single preference model, disregarding the unique characteristics and needs of diverse labeler groups. The authors propose a Group Robust Preference Optimization (GRPO) method to align LLMs with individual groups' preferences.
strength_weakness: strength_weakness: 1. How does the proposed method handle scenarios where preference data lacks clear group information? In practice, much preference data does not come with explicit group classifications.

2. The paper should address whether the proposed method can learn an invariant preference standard from data that may contain various group information, such as different preference evaluation criteria and preferences annotated at different iterations.

3. The authors need to provide and analyze the alignment results of the model, not just the reward model analysis.
questions: questions: Please refer to the Weaknesses section for questions regarding the paper.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a novel preference optimization technique, GRPO, which utilizes the group distributional robust optimization. Specifically, the method aims to maximize the worst-case group performance for improving the robustness of LLM models. This paper provides several theoretical aspects of GRPO when it comes to convergence guarantee and closed-form solution when considering the log-linear policy class. Empirically, the GRPO objective function is minimized by the mirror gradient descent. Several theoretical results on synthetic and real-world datasets demonstrate that their method works well.
strength_weakness: strength_weakness: * This paper lacks novelty in its contribution. The group DRO technique is already a popular method used in a variety of applications [1, 2].  While the authors offer some theoretical results and an algorithm for their method, these may also lack novelty. In specific, Proposition 3.1 on the Nash equilibrium is a well-known result in game theory and Proposition 3.2 seems straightforward because $\pi^*$ is dependent only on an input value, not on the group to which the input belongs. Furthermore, there is no significant difference in the algorithm and proof of convergence guarantee compared to existing works [37, 30 in the manuscript], except for slight modifications.

* Although this paper is novel in applying the group DRO technique to RLHF preference optimization for the first time, stronger motivations and empirical results are needed to highlight the necessity of the proposed method. But, I also think that to strengthen this contribution, For example, as mentioned in lines 141-153, it would be beneficial to show that existing Large Language Models (LLMs) suffer from group robustness issues in various applications involving helpful/harmful instances or domain-specific categories when fine-tuned by RLHF optimization techniques.

* The interpretation of the results in Section 5.2 is insufficient. In Figure 3, GR-IPO improved the performance of all groups, including Group 1. This result contrasts with findings reported in the Group DRO paper [37], where a trade-off between majority and minority group performance is generally observed, with majority group performance typically decreasing to compensate for minority group performance. Therefore, a more thorough analysis or performance comparison is needed to explain this result.

[1] Re-weighting based group fairness regularization via classwise robust optimization, 2023.
[2] Distributionally Robust Multilingual Machine Translation, 2021.
questions: questions: 1. In the experiments in Section 5.2, what are the zero-shot performances of Gemma-2B for each group? It would be better to report the degree of bias in the pre-trained model together, because it can enable comparing the bias degrees between the pre-trained and fine-tuned models.

2. Would you provide the results when fine-tuning more layers or the entire neural networks using your method? I think that this can strengthen this paper's contribution by showing the performance of the method in various settings.
limitations: limitations: They addressed the limitations of this paper in the manuscript.

Comments: title: Gentle Reminder
Response to Ethics Reviewer CLC3
Response to Ethics Reviewer ipoT
comment: We thank all the reviewers again for the time taken to review our work. As the author-reviewer discussion period is nearing its end, we would appreciate it if the reviewers would respond to our rebuttals. If there any further questions, we are more than happy to clarify and discuss further.
We thank the reviewer for acknowledging our detailed rebuttal and raising their score.
I thank the reviewers for a detailed response to my concerns. I have read the rebuttals, and it adequately addresses all my questions. I am increasing the score to accept.
We thank the reviewer for their review and recommendations and for pointing out that our manuscript **contains no ethical issues.**

We provide the answers below: 

- We strongly disagree with the reviewer that our statement is not supported by evidence. As explored in both the synthetic and real-world experiments, due to inherent structures in the data and the pre-trained model’s capabilities, there is a significant bias in the performance of non-robust baselines across groups. Our method improves the worst group performance, thereby mitigating the biases in alignment performance across groups. This is further evidenced in Figure R3 of the attached PDF, where Group 5’s performance improves albeit at the expense of Group 1, hence ensuring a more equitable alignment.  
We also want to emphasize that we were very careful in our phrasing, which is why we used cautious language, such as "hold promise”.

- We note that “distinct” does not imply “non-overlapping”.  We believe that even different overlapping groups can be considered distinct. If the reviewer still finds this confusing, we are happy to reword the sentence.  
Our approach is independent of whether the groups are mutually exclusive. We treat them through distinct identifiers/contexts, rather than overlapping sets, and optimize for the worst-case performance across groups. While some groups may have preference distributions that are more “similar” to others, others may have “fairly” different preference distributions. We observed this in our real-world data experiment in Section 5.2. However, this has no impact on the performance of our method.
We thank the reviewer for their review and recommendations. In our humble opinions, our work is proposed to address the potential ethical issues in current LLM training, i.e., our robust alignment can ensure more equitable LLMs across different demographic groups. 

Regarding your questions, which are raised by other reviewers and addressed, we reiterate the answer.

1) The popular datasets used in pluralistic alignment research typically have a well-defined group notion. Our formal problem statement explicitly mentioned that we consider the case of known groups. Moreover, our approach is independent of whether the groups are mutually exclusive. We treat them as distinct identifiers/contexts (and not as sets that can overlap) and we are optimizing the worst-case.    
When the group information is unknown, one may apply methods to identify unknown groups. a) Clustering: Apply algorithms like k-means or EM to detect potential groups; b) Representation learning: Use techniques like variational autoencoders (VAEs) to discover hidden structures; c) Analyzing features: Look for features with high variance or correlations to hypothesize groupings; d) Consulting experts: Leverage domain expertise to identify meaningful groups. These methods can help uncover group structures in the data. We will include the above paragraph in the manuscript and acknowledge that this is an interesting avenue for future research.  
  


2) We clarify that we do not claim that our method will improve performance across all groups over the non-robust baseline. In general, minimax methods improve the worst group’s performance, often reducing the average or best group’s performance (“no free lunch” property). Our proposed GRPO formulation (Eq. [8]), stemming from a minimax framework, will comply with the same property. We will make it clear in the next version. The observation of improved performance across all groups in one of our plots is discussed in further detail in our response to reviewer kmNK. We kindly refer the ethics reviewer to that.    
Regarding the tradeoff over worst-case vs average-case performance, we discuss in detail in Appendix B.4, wherein we introduce a trade-off parameter $\chi$ to balance the worst-case vs average performance. In response to reviewer kmNK, we ran ablation experiments on this trade-off parameter to compare performances at different levels of importance to worst-case vs average performance and plot them in Figure R1 of the attached PDF. Note that the max validation loss decreases while moving from $\chi=0$  to $\chi=1$, where $\chi=0$ corresponds to importance sampling (see Eq. [30] in Appendix B.4) and $\chi=1$ corresponds to GR-IPO. Further, we plot the average validation loss which increases while moving from $\chi=0$ to $\chi=1$, demonstrating the trade-off between average and worst-case performance. Note that, GR-IPO aptly increases the average loss (as expected) in order to reduce the worst group loss. These discussions will be included in the next version.