Group Preference Optimization: Few-Shot Alignment of Large Language Models
Authors: Siyan Zhao
John Dang
Aditya Grover
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
alignment
group preference alignment
few-shot learning
in-context learning
fine-tuning
Primary area: generative models

Abstract: Many applications of large language models (LLMs), ranging from chatbots to
creative writing, require nuanced subjective judgments that can differ significantly
across different groups. Existing alignment algorithms can be expensive to align
for each group, requiring prohibitive amounts of group-specific preference data
and computation for real-world use cases. We introduce Group Preference Optimization (GPO), an alignment framework that steers language models to preferences of individual groups in a few-shot manner. In GPO, we augment the base
LLM with an independent transformer module trained to predict the preferences
of a group for the LLM generations. For few-shot learning, we parameterize this
module as an in-context autoregressive transformer and train it via meta-learning
on several groups. We empirically validate the efficacy of GPO through rigorous evaluations using LLMs with varied sizes on three human opinion adaptation tasks. These tasks involve adapting to the preferences of US demographic
groups, global countries, and individual users. Our results demonstrate that GPO
not only aligns models more accurately but also requires fewer group-specific
preferences and less training and inference computing resources, outperforming
existing strategies such as in-context steering and fine-tuning methods.

Published as a conference paper at ICLR 2024
GROUP PREFERENCE OPTIMIZATION: FEW-SHOT ALIGNMENT OF LARGE LANGUAGE MODELS
Siyan Zhao, John Dang, Aditya Grover
Department of Computer Science, University of California, Los Angeles daiy,nz, john. dang, adi tyag) ucl edu
ABSTRACT
Many applications of large language models (LLMs), ranging from chatbots to creative writing, require nuanced subjective judgments that can differ significantly across different groups. Existing alignment algorithms can be expensive to align for each group, requiring prohibitive amounts of group-specific preference data and computation for real-world use cases. We introduce Group Preference Opti- mization (GPO), alignment framework that steers language models to prefer- ences of individual groups in a few-shot manner. In GPO, we augment the base LLM with an independent transformer module trained to predict the preferences of group for the LLM generations. For few-shot learning, we parameterize this module as an in-context autoregressive transformer and train via meta-learning several groups We empirically validate the efficacy of GPO through rigor- ous evaluations using L.MSs with varied sizes on three human opinion adapta- tion tasks. These tasks involve adapting to the preferences of US demographic groups, global countries, and individual users. Our results demonstrate that GPO not only aligns models more accurately but also requires fewer group-specific preferences, and less training and inference computing resources, outperforming existing strategies such as in-context steering and fine-tuning methods.
Warning: This paper contains qualitative examples that may be viewed as offen- sive harmful.
INTRODUCTION
Large Language Models (LLMs) increasingly being employed for a wide variety of domains, with use-cases including creative writing, chatbots, and semantic search among others (Touvron)
al. 2023b Taori et al. 2023 Ouyang et al. 22022] Bai al. 2022a Brown et al., 2020) Many of these applications are inherently subjective and require generations that cater to different de-
mographies, cultural and societal norms, or simply individual preferences (Hartvigsen et al. 2022
Zhang et al. 2023 Solaiman & Dennison, 2021 Blodgett et al., 2020, Dunbar et al.) 1997). By virtue of their large-scale training, current language models are exposed to diverse data that allows them to represent multitude of such opinions (Glaese et al., 2022 Durmus et al. [2023 Santurkar jet al. 2023). However, expressing these diverse opinions requires steering the LLM generations to user requirements. This brings forth the key question studied in this work:
How do we efficiently adapt LLMs to align closely with the opinions of specific interest groups?
Broadly, prior work has explored two modes of steering language models, which trade-off training complexity with test-time engineering. On one end, prompt engincering approaches avoid explicit modifications to the parameters of the language model and elicit desired behavior by crafting suitable prompt. Often, the prompt is augmented with a few in-context examples (Brown et al.) 2020 Taori et al., 2023 Chowdhery al. (022). While prompting approaches are attractive as they have no additional training complexity over the base model, prompt engineering can be quite tedious and empirically poor when the desired behaviors
more complex (Zhou et al.) 2022 Reynolds &
McDonell 2021
Qin & Eisner 2021 Lester et al. 2021 For example, Santurkar et al. (022) show
'Our code is available at the project wehsite: https://iyaaahhaaaiiiiiniiiiiig

---

Published as a conference paper at ICLR 2024
Prompt q
How satisfied are you with the way democracy is working in your country?
Group
Context Examples
A. Satisfied
2
B. Somewhat Satisfied
773
Query Prompt
C. Not Salisfied
Response
S.r.
2
Response /
i
52
LLM
B. Somewhat Salisfied
A
B
y
2/
y
Group Preference Datasets
Group
Group b
Vv"
MNA
A B
C
A C
B
Do
D2
GPO
Group
v
y
B
A B C
y
ym++1
Da
(a)
(b)
Figure 1: Overview of GPO. Left: Group aligmment aims to steer pretrained LLMs to preferences
catering to a wide range of groups. For each group g. we represent its preference dataset D3 {[x]. v"),,,.. (2% ux) Here, V signifies the preference of group g for pair of given prompt E. and response r., while x is L.M representation obtained with Temb r.)). Right: Once trained, GPO provides a few-shot framework for aligning any base LLM to test group given small amount of in-context preference data.
that LLMs ver-emphasize opinions from privileged demographies and are challenging to rectify via in-context prompting approaches.
On the other end, various kinds of alignment approaches have been proposed that seek to augment or finetune the language model with an additional reward or scoring model. These approaches can steer the model to achieve complex behaviors such as honesty, helpfulness, and harmlessness (Ouyang et al. 2202; Bai et al., [202a;: Glaese et al. 2022, Bansal et al. [023] Askell et al.) [021]
Song et al. 2023 Bai et al. 2022b) Thoppilan et al. 2022 Wang et al. 2022). but come at the cost of additional complexity in gathering sufficient supervision train reward models and subsequent finetuning. As a result, existing alignment approaches, such as PPO (Schulman et al. 2017].. DPO (Rafailov al. 2023). and Best- Best-Of-N, are not designed to efficiently align LLMs when the number of target groups is large and supervision for each group is limited.
We introduce Group Preference Optimization (GPO), few-shot framework for aligning Large Lan- guage Models to opinions and preferences of desired interest group(s). The key idea in GPO is to view the alignment of an LLM policy as a few-shot adaptation problem within the embedded space of an LLM. Specifically, GPO augments an arbitrary base LLM with an independent few-shot preference module. This module is parameterized via an independent transformer and trained to explicitly perform in-context supervised learning to predict preferences (targets) given joint embed- dings (inputs) of prompts and corresponding LLM responses. The use of embeddings guarantees that the preference module can effectively process in-context examples where each example is itself a potentially long sequence of prompt and generated response. In-context learning further provides the ability to efficiently adapt to new, unseen groups at test-time with only handful of exam- for an illustration. Finally, we incorporate various architectural design choices
ples. See Figure [ to guarantee permutation-speciff inductive biases, building on recent work in in-context learning over datasets (Nguyen & Grover 2022). Once learned, the learned module can serve as a drop-in replacement for reward or preference function for policy optimization and re-ranking algorithms.
In our experiments, we validate the effectiveness of GPO for aligning language models to the opin- ions of 22 diverse US demographic groups in the OpinionQA dataset (Santurkar et al. 2023) and
2

---

Published as a conference paper at ICLR 2024
14 global countries in the GlobalOpinionQA dataset (Durmus al. 2023). We consider 2 base language models of different sizes: Alpaca 7B (Taori et al.) 2023), an instruction- version of the LLaMA (Tourron et al.] 2023a) 7B model, and the recent Llama2 13B chat (Touvron et al. 2023b), which has been fine-tuned on a large dataset of human preferences for helpfulness and safety. Empirically, we test GPO against variety of prompting and finetuning baselines. On erage, GPO surpasses the cop-performing baselines by 7. % when adapting to 22 US demographic groups in OpinionQA, and by 8.4% when aligning with 14 global countries in GlobalOpinionQA. Furthermore, GPO performs most effectively in adapting to individual preferences compared to other baselines.
2 GROUP PREFERENCE OPTIMIZATION
2.1 PROBLEM SETUP
A large language model (LLM) expresses a probability distribution over natural language, denoted as π. To accomplish any task, such question answering or summarization, user crafts a suitable query and prompts the LLM to generate a response r obtained via sampling from the conditional distribution o(- q). Rather than decoding responses from single distribution x g). our goal in this work is to align the language model to the preferences of desired target group g" E G. Here, we adopt fairly general definition of a group to refer to any collection of agents (e.g., g., demographic groups, individual personas), and we use G to denote the space of all possible groups. For training, we assume that we are given access to preference datasets for a finite set of training groups Grain In practical applications, the number of groups can be large (e.g., different demographies and cultures) while the amount of preference data for each group is generally small.
2.2 RELATED WORK
Existing approaches for steering LLMs challenging to apply for group alignment, especially when the underlying groups are complex and per-group supervision is scarce. Below, we summarize key approaches and their trade-offs, which will also serve baselines in our experiments. We provide additional discussion of related work in Appendix
These approaches modify the input prompt q + g' guide the LLM to- wards a group-aligned distribution (Jiang et al. 2023] (Waang et al. [023] Deshpande et al. (023). Techniques include meta-data utilization, where group-specifee meta-data, are appended to the input prompt. Further, the engineered prompts can be improved via in-context few-shot prompting, in which the prompt concatenated with examples of desired behavior. Given the flexibility of lan- guage, even preference dataset D, could be converted into in-context examples for improving the prompt. Prompt engineering approaches are computationally efficient as they involve no training. but designing the prompt itself can be a tedious task that relies heuristics (Zhou et al., 2022
Prompt Engineering:
Lester et al. 2021 Qin & Eisner 2021), which are not guaranteed to transfer well across differ- ent I.LMs. Finally, it has been shown prompt engineering has limited gains in aligning LLMs to complex groups on challenging survey datasets (Santurkar et al. 2023 Durmus et al.) (023).
Algorithms that fine-tune the base Large Language Model (LLM) or augment it with additional models have successfully aligned LLMs to complex behaviors like honesty, harmfulness, and helpfulness. Broadly, there are two main classes of methods. The first involves supervised learning, using dataset of responses from a target group for fine-tuning, as demonstrated in (Ouyang et al., 2022); Ziegler et al., 2019). This method is straightforward but often suffers from limited generalization and requires extensive group-specific data. The second class uses explicit human-derived preference data to train reward models for response filtering, re- ranking (e.g., Best-of-N, importance weighting (Grover et al. 2019)), or reinforcement learning optimization (c.g., PPO (Ouyang et al. 2022 Schulman et al. 2017)), which may pose challenges in hyperparameter tuning and stability (Sun al. 2023 Santacroce et al. 2023). Newer methods focus on direct optimization of preferences to enhance stability in RL techniques (Rafailov et al.) 2023 Song et al. 2023). These approaches typically require access to large preference datasets. Our method, GPO, is designed to explicitly align with various interest groups under limited supervision constraints, positioning it within the explicit alignment framework.
Gradient-based Alignment:
3

---

Published as a conference paper at ICLR 2024
Predicted preferences
Biutt
Group Preference Optimization (GPO)
Yo
Consext points
Publed inputs
Illustration of the GPO architecture for a sequence of n points, with m context points and n target points. The context (I1:m, V1:m) serves few-shot conditioning for GPO. GPO processes the full sequence using a transformer and predicts the preference scores 9m+1mn
Figure 2:
2.3 PROPOSED METHOD
We desire an alignment approach that generalizes a wide variety of groups, even when constrained by the amount of per-group supervision. Accordingly, we view group alignment as few-shot learning problem and cast it in the framework of in-context meta-learning. For each training group g E Gurin. we represent its preference dataset as D, {a1 vi) vf),... ..... (x% the preference group to a pair of input prompt query of and LLM response r" and x. denotes the LLM representation of the concatenation of the prompt query and LLM response 2 Temb (g) r9)- Here, Femb can be the language model embedding function or an identity function that maintains the input's raw textual format. Note that while the inputs 2 can be shared across different groups (e.g., universal surveys), the preferences are different for each group. At test-time, our goal will be to steer the default LLM distribution a new distribution, say Tg*, given a preference dataset D,- for the target query group g" For brevity of presentation, we consider the preference to be real-valued scalars. Our framework extends other kinds of responses and preferences, such as short-ans short-answer questions (e.g., MCQs) and relative pairwise responses, as discussed in Appendix H
y where yf denotes
Given the above setup, we design GPO perform group aligmment by learning few-shot preference model that augments the base LLM, as shown in Algorithm I Once learned, we can use it to update the I.LM via any standard preference optimization or reweighting algorithm (e.g., PPO, Best-of-N). Specifically, we parameterize GPO via transformer and train it to perform in-context learning on the training preference datasets. Given a training group E Guains we randomly split its preference dataset D, into set of context points and preference dataset for group g. Thereafter, GPO is trained to predict the target preferences y Ian given the context points (zf... vim)) and target inputs x +m Mathematically. we can express the objective as:
target points, where [D, is the size of the
(1) where the training group g Giran and context size m are sampled uniformly. represents the parameters of our model. Figure Segureesshwss an illustration. For decoding, we make the conditional independence assumption, where we assume that the target preferences are independent of cach other given the context samples and the target inputs:
L()) = E. [log Pe (P
A vi.m))
L( = E g.m
Σ logpo(y? xf... vi..)
(2)
In our preliminary experiments, we also investigated alternatives which model the dependencies. We did not find any noticeable improvements and hence use Eq. 2 for the rest of the paper.
Following Nguyen & Grover (2022), we can modify the transformer architecture in GPO to explic- itly account for permutation invariance conditioning over in-context examples. In particular, we discard the positional encodings commonly found in standard transformer architectures. However, this loses the pairwise relations between (æ,, y:). To solve this, we concatenate each pair (z,, y:) into a single token to inform the transformer of their pairwise relation. For the target inputs, we pad the 2," 's with a dummy token (c.g., 0). Finally, we employ a masking strategy where the context pairs can self-attend each other, whereas the padded targets only attend the context points and
4

---

Published as a conference paper at ICLR 2024
not to other target points to follow the conditional independence assumption in Eg GPO satisfies the properties of context invariance (Property 1. and target equivalence (Property
Note that even though GPO uses in-context learning, it is distinct from in-context prompting base LLM. The latter does not update the parameters of the base LLM and requires examples of desired text generations. On the other hand, GPO learns few-shot model which augments the base LLM and only requires preferences of users for the LLM generations. That said, both these schemes are complementary each other as we can use any engineered prompt (e.g., with in-context examples) as drop-in replacement for the default prompt used in the inputs I.
Scaling long dataset contexts. One challenge with GPO is that the effective sequence length for the transformer can grow significantly if we use raw representations of prompts and responses within each input z. This can degrade performance and efficiency significantly. To overcome this challenge, propose to embedded representations of text within can contain sufficient information for solving tasks (Bhatia et al., 2023). particular, we first concatenate the prompt and response and compute their joint embedding Temb (o/" r²) using the base LLM. We explored different techniques for extracting the joint embeddings from the base LLM, as detailed in the ablation study in Appendix D and found it best use the average embedding of all the tokens in the input.
LLM representations
Algorithm Group Preference Optimization (GPO)
1: Input: LLM embeddding function Temb: Preference datasets Dg Vg E Gimin-
2: Initialize GPO transformer with parameters 0.
3: For all g E Guain- cache embedded pairs (=- v/) in Demb where xaf Temb (g: :
4: repeat
5: Sample training group g E Gurain
6: Sample context size ~ Uniform(1, 1] where n = 1D
7: Split Damb randomly into m context (af.m v..m) and (n - m) target (1 v1n
pairs.
8:
Predict target
preferences vn+im using context (fi.m vi.)) and padded targets
(1 L:n. 0).
9:
Update minimize in-context loss function L(f) in Eq.
10:
until convergence
11:
Output: GPO transformer with learned parameters
3
EXPERIMENTS
While GPO is general-purpose purpose and can be applied broadly to many language model use cases, our work is focused on benchmarks which reflect diverse landscape of human preferences. Quantitatively evaluating the diverse opinions through open-ended questions (e.g., creative writing) inherently complex, and often demands expensive human labels. In contrast, closed-ended sponses (c.g., multiple-choice questions) offer a standardized means of capturing diverse opinions, thus reducing ambiguity and noise evaluation. Survey datasets have been used in prior work (San-) turkar et al. 2023 Durmus et al.) 2023) to demonstrate the weaknesses of current LLMs in catering diverse populations, and hence can effectively used to benchmark progress group alignment.
Datasets.
We benchmark group alignment on 2 recent survey datasets: (1) OpinionQA (Santurkar et al.. 20233,
which spans 22 US demographic groups (e.g. income, political ideology, race, and sex) across 500 multiple-choice questions and (2) GlobalOpinionQQ (Durmus et al.) 2023). which contains multiple- choice questions answered by participants from 14 countries, amounting to 2,554 questions which cover various topics including politics, media, technology. religion, race, and ethnicity. Survey questions are shared across different groups, so we use x; (and not x²)) for brevity henceforth. Detailed dataset descriptions
be found in Appendix
Next, we construct group g preference dataset Dg from the survey data. Let Q be the set of all survey questions and G be the groups participating the survey. Consider survey question E Q, with T unique answer options. Each option can be interpreted as a response r, yielding a set of T viewpoints (x:){_ (Femb r=)) The preference score y for the viewpoint æ; is obtained by
5

---

Published as a conference paper at ICLR 2024
aggregating the survey responses given to (q, r.) from group g. These scores are normalized within each question to form the group preference distribution vector Pg = [v ..... y for question q. such that Σx uf 1. Repeating this process for all n questions in Q yields D During training and testing, all viewpoints from the same question belong to either the context or target set. Finally, apply softmax layer predictions for each question, yielding normalized preference scores for each survey question in the target set.
Evaluation Metric. To rigorously assess the degree of alignment between two opinion distribu- tions, P1 and P2. we calculate the Alignment Score, denoted as A(P1 P2: Q) over a set of questions Q. This metric employs a similarity function Sim:
A(Pi, Q2:
Sim (1))), P2(g))
Q
For the OpinionQA dataset (Santurkar et al. 2023) with its ordinal answers, we employ the one- dimensional Wasserstein Distance as our similarity metric. Conversely, for the GlobalOpinionQA dataset, which often presents non-ordinal answer structures, we use the Pensen-Shanmee Distance as suggested by the original paper. Further details are available in Appendix B
Base Large Language Models. We use two different-sized LMs as our base models for baselines and GPO. The first, Alpaca- Taori et al. (2023), is instruction tuned variant of the Llama-7B (Touvron et al. 2023a), crafted using 52K instruction-response pairs. The second, Llama2-1 113 chat 1M human preferences for helpfulness and safety. For baseline methods requiring updates of model weights, we use low-rank adaptation (LoRA) (Hu et al. 202T)
version, is finetuned
Baselines. We compare our method against extensive baseline approaches as introduced below. For detailed description of the baselines, refer to the Appendix [F] (1) Uniform Distribution as- sumes equal preference scores for all options; (2) LM Base following Santurkar et al. (2023); Dur- mus et al. (202)), gets the LM's default opinion distribution Pa (g) by extracting and normalizing prediction scores for answer choices; (3) LM Steered uses prompting strategies to convey group information to the LM (examples in Appendix K): (4) Few-shot Prompt appends few examples showing a group's 's preferences for m context questions to the prompt, where m is constrained by the LM's context window size and Cy includes the context samples { Vo 1 (see Figure 8 in the Appendix); (5) SFT per group fine-tunes the LM separately for each group with maximum likelihood loss on augmented training examples created by sampling responses according to the preference distribution Py(g): (6) Reward Model trains per- gorugroup reward model by adding a linear MLP head on a base LLM and training it on m context samples (I,, yf | with MSE loss predict preference scores; and (7) In-Context Finetune investigates few-shot in-context alignment ability by partitioning the group set into meta-train/test sets, splitting training questions into context/query, supplementing each query with context Gg of m ground truth preferences, and fine-tuning the LM with maximum likelihood loss where responses are sampled according to (4)).
3.1 RESULTS AND DISCUSSION
Adapting to US demographies in OpinionQA.. We conducted experiments with three distinct meta-train and meta-test splits, allocating 40%, 60%, and 80% of the 22 US demographics groups respectively as the meta-train groups Gtrain This group split was consistent with the In-context Finetune baseline and GPO, For other baselines that operate on a per-group basis, we calculated the alignment score for the meta-test groups and present results averaged over three random seeds.
Our results are presented in Figure B. Alpaca-7b base model exhibit alignment scores that are similar the alignment score of uniform distribution. This does not necessarily imply an absence of biases, as averaging across groups could obscure biases towards certain demographics. Prior work has found LMs may disproportionately over-represent some groups and under-represent others (Snnturkar et al. 2023). However, Llama2-1 1-b---ha---hhal base model exhibits a lower alignment score as compared to the uniform distribution. This might be attributed to its fine-tuning for safety, causing the model to lean towards the least harmful option, which can be seen from the qualitative examples in Appendix D] When we incorporate group information into the LMs, we deploy various prompting strategies- QA, BIO, and PORTRAY - convey this information (see Appendix K examples).

---

Published as a conference paper at ICLR 2024
Alpaca-7b- OpinionQA Dataset
Llama2-13b OpinienQA Dataset
0.8
Miracrab 6th gruss
bidh mos in ahi grean Mata-crah on groupe
0.90
Mignmeett Scare
3.90
Aliggment
2.79
0.20
3.6
0.40
Unatarm
LM-base
Few st prompr
Ld ststered
Reward model
per group context
SFT
finetune
GPO
Unaform
LM-baso
Fese shot
prompt
Lts Lasteeed
Reward model
SPT
group
fineuue
GPO
Alpaca-7b
Globa MlobalOpinione OpAA Dataser
Llama2-13b
GlobalOpinionOA Dataset
Micacoile
Mata-roah
Mecalise 66% prues
Matetroc on pruun groups
miignmant: Score
Aligh
5.
Unitem
LM-aase
Fow shet
prempt
(t-steered
Reward model
per group
SPY
GPO
Uniform
LM-
Feweshoot
L Restoved
Reward model
per Incontes
SPT
finetune GPO
Figure 3: Alignment score comparisons the OpinionQA dataset and GlobalOpinionQA dataset with Alpaca-7b and Llama2-13be 13b-chat as base models. Results have been averaged across three group split setups and three random seeds, with standard deviations provided.
We report results for the strategy that yields the best alignment as LM-steered. Given explicit group information, Alpaca-7b-ste ae-uca--- displays slightly lower relative gains as compared to Llama2-138 --b-- steered. Next, when provided with few-shot group preference context samples, which serve as an implicit method of conveying group information, the LM's alignment performance significantly declines compared to the base language model's performance. We hypothesize this decline might be due to the prompting format being outside the distribution of the language model's training corpus.
For methods involving gradient updates, we maintain a consistent number of context samples across all baselines, which is also the same number of context examples used in Few-shot prompt. Specifi- cally, we use 15 samples for Alpaca-7b 7b and 20 for Llama2- experiments. With gradient updates, SFT per-group brings improvement compared to other gradient-free free steering methods. However, training a Reward Model to predict alignment scores from context samples, and subsequently use it predict preference scores for query examples underperforms SFT methods. This outcome may suggest risk of overfitting when working with a limited sample size.
GPO achieves notably higher alignment scores on this dataset compared to the baselines for both the Alpaca and Llama2 base models. GPO uses the same number of context samples for adaptation and the test groups are unseen during training. We observed performance increases when a larger number of meta- meta-training groups were used. GPO's closest baseline, the In-context Finetune method ranks second, where the LMs are trained infer from few-shot context samples. On average over the two base models and the three group split settings, GPO achieves a 7.1 % increase over the In-context Finetune.
Figure Aegure-qqalitann qqulltaaivvll illustrates the predicted alignment scores from different methods in response to an OpinionQA example concerning climate change concerns across six demographic groups The first row depicts the ground truth group opinion distribution. Given just 15 context samples, GPO successfully adapts to match the opinion distributions of different groups. For instance, increases preference for option A when adapted the group Hindus, while the steered LMs do not exhibit correct distribution changes. For example, Llama2-130 1ib-a------ttereed appears to be biased towards specific option, overrepresenting it rather than accurately reflecting the distribution of the
7

---

Published as a conference paper at ICLR 2024
targeted group. On the contrary, in demographics with more balanced distribution like College graduate/some postgrad, GPO maintains this balance more consistently. This demonstrates that GPO does not merely adapt to the overall dataset group preferences, but can align to specific groups using limited context.
Q: Thinking about the future of our country, how worried are you, if at all, about climate change?
Crlkge grobelsbore pollgadd
Less then 1igh scheet
Jown
A. Very worried
SN
Fairly worried
C. Not too worried
D. Not at all worried
Culige yrokatooiiee poctgal
GPC Lest than ceplstess
GPO HEVEI
BVO joich
CIO Eeval
CIO NAW
2EIN
26
US- SSAG
Apaci
Apacc deengp
M.2%
TV
40.7% 2EE4
23.%
Usrre sheend
Ulera2 strendd lays
youchh
278
50.7%
523%
37.6%
Figure 4: Qualitative comparison of GPO alignment with steered LMs, where each pie chart denotes the preference distribution of the group. Here, GPO uses Alpaca-7b's embedding.
Adapting to cross-nation groups in GlobalOpinionQA. The diverse and highly contrasting opinions across nations GlobalOpinionQA presents more complex landscape than the Opin- ionQA dataset. Upon analyzing performance, trends in the GlobalOpinionQA dataset closely fol- lowed those observed in the OpinionQA, as depicted in Figure Notably, the alignment score of the Alpaca-7b base model surpasses that of the uniform distribution while Llama2-13b base model shows lower alignment. For Alpaca-7b LM-base, this could suggest that the base models might exhibit stronger alignment to certain specific countries and this hypothesis is supported by the in- creased standard deviation of the Alpaca-7b 7b M-basse alignment scores, hinting at varied alignment across different countries, phenomenon also reported in the dataset (Durmus al., 2023). Al- ternatively, this could imply that the base models tend to align more with the dataset's general respondents, which naturally would exceed a uniform distribution. With gradient updates, the SFT per-group method here surpasses the alignment performance of steering methods, while the Reward Model underperforms SFT methods. The In-context Finetune method emerges as the third-best and second- best in terms of alignment for Alpaca-7b and Llama- respectively, which showcases enhanced in-context few-shot adaptation post meta-training. However, its training demands are sub- stantially higher; it requires approximately 4.7 times more training time compared with GPO on an NVIDIA RTX A6000 to achieve the depicted performance. Averaged across both base models and the three group split scenarios, GPO posts 8.4% improvement over the second-best baseline.
Scalability with Increasing Context Samples. We evaluate the scalability of different methods with respect the size of the in-context examples. Figure 5 demonstrates that for Nigeria in the GiobalOpinionQA dataset, GPO enhances alignment scores with fewer than 10 preference con- text samples. The performance of Few-shot Prompt improves with more examples but plateaus with greater variance. In comparison, In-context Finetune exhibits superior adaptability post meta- training than Few-shot Prompt, yet its alignment is still suboptimal and the number of group context samples is limited by the context window size of the LM. Both SFT per-group and Reward Model show incremental improvements with added context samples; however, their sample efficiency is modest. In contrast, GPO adeptly adapts to groups in sample-efficient manner.
Adapting to Individual Preferences. Variations in individual opinions can manifest even within the same demographic groups (Hwang al. 2023). We align GPO with individual- preferences.
8

---

Published as a conference paper at ICLR 2024
Context
[Globol (ogioonk Baw.oooono----oooo (Ngeria
1.85
Figure 5: Alignment score of various meth- ods based Llama2-13B with varying group context sample size. Evaluation conducted on survey questions for Nigeria from the Glob- alOpinionQA dataset. The shaded region rep- resents the standard deviation across three dif- ferent seed results.
3.55
2.70
3.00
2.55
G7O
Intottel Teetue Ranurd
3.4
hurotss hrmnst
2.0
56
1rs
of SattedSSmppe
Alignment Accuracy Comparison topir
Individual Algnment Accuracy Comparison
Algime Accuua
63
Peoge
Coptert
Sure
HM Swoee
Phospe
1
201
Figure 6: Individual alignment accuracy comparisons from the OpinionQA dataset. Left: Individual alignment the gun topic survey. Right: Comprehensive comparison across all 15 topics, show- casing the performance of various methods on diverse subjects. Experiments use Alpaca-7b as the base LM. Both GPO and In-context finetune are meta-trained 40% of individuals and evaluated on the remaining 60%. The horizontal red line represents the average accuracy of random model.
From the OpinionQA dataset, encompassing 15 surveys across 15 unique topics, we randomly select 100 participants from each survey, along with their responses 30 topic-related questions. For each individual, 40% questions serve as context samples and 60% for queries. We use Alpaca-7b here the base model. To steer the LM with individual information, combined demographic variables, such as income, religion, and age, as demonstrated Appendix Figure Since each individual only selects one option, calculate alignment accuracy instead by treating the option with the highest predicted preference score as the predicted option. Due to computational constraints, we confined our evaluations of the SFT per-individual and reward model methods to one survey. Since both of them operate a per-individual basis, the training needed for about thousand individuals made broader comparisons of two baselines impractical. In contrast, other baselines, including in-contex, finetune and GPO, were assessed across all 15 survey topics. Across the full breadth 15 topics, GPO consistently exhibited superior performance in adapting individual preferences relative to other baselines, depicted in Figure
create individual context from
4 CONCLUSION
We introduced, GPO, novel method for few-shot aligning LLM outputs to both individual and group preferences given little preference data. GPO trained on a meta-train dataset containing group-wise preference data. During inference, GPO adapts preferences given a few context examples from that group. GPO significantly outperforms prior methods measured by alignment score for group preference alignment while requiring gradient updates to the base LLM. We find that GPO is also more sample efficient, improving alignment score significantly more than baseline methods while using fewer samples, and is effective across multiple popular open-source LLMs of various parameter and pre- trainininng dataset scales.
group, predicting aligned

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
GPO can be used to align models to preferences of diverse interest groups which can provide more positive, useful, and inclusive experience for end users of LLM applications. We acknowledge that aligning LLMs to the preferences of demographic groups can have malicious applications. For example, making LLMs more capable of producing responses that are more tailored to specific users may be misused to convince or show members of group how to perform unethical actions. Additionally, GPO's methodology can be used align a model group's 's preferences even those preferences are harmful. Biased, offensive, and harmful preferences present in the meta-train or meta-test datasets may be reflected in the outputs of GPO. Future work should investigate methods for aligning LLM outputs group preferences without amplifying harmful outputs.
ACKNOWLEDGMENTS
This research is supported by Google Award for Inclusion Research and Adobe Data Science Award. We want to thank Hritik Bansal for insightful discussions.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant laboratory for alignment. arXiv e-prints, pp. arXiv-2112, 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08072 2022b.
Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models. arXiv preprint arXiv:2305. /5812, 2023.
Kush Bhatia, Avanika Narayan, Christopher De Sa, and Christopher Ré. Tart: A plug-and-play transformer module for task- aagnosic reasoning, arXiv preprint trXivv2306.07536 2023.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454 5476, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimban. Toxicity in chatgpt: Analyzing persona -assoneed language models. arXiv preprint arXiv:2304.05335, 2023.
RIM Dunbar, Anna Marriott, and NDC Duncan. Human conversational behavior. Human Nature, 8 (3):231-246, 1997.
Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Mafficld-Dodde Danny Hernandez, Nicholas Joseph, al. Towards measuring the representation of subjective global opinions in language models. arXiv e-prints, pp. arXiv- 2306, 2023.
Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint 2203::22009.4375.
Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric J Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood-free importance weighting. Advances in neural information processing systems, 32, 2019.
Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Milena Lagos, Pippa Norris, Eduard Ponarin, and Bianca Puranen. World values survey: Round seven country-poodee pooleed datafile version 5.0.0. 2022.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Fce Kamar. Toxigen: A large-scale
dataset for adversarial and implicit hate speech detec- tion. In Proceedings of the 60th Annual Meeting of the Association Computational Linguistics (Volume Long Papers), pp. 3309-3326, 2022.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. International Conference on Learning Representations, 2021.

---

Published as a conference paper at ICLR 2024
EunJeong Hwang, Bodhisattwa Prasad Majumder, and Niket Tandon. Aligning language models to user opinions. arXiv e-prints, pp. arXiv- 2023.
Hang Jiang, Xiajie Zhang, Xubo Cao, Jad Kabbara, and Deb Roy. Personallm: Investigat- ing the ability of gpt-3.5 express personality traits and gender differences. arXiv preprint 230.:2300.02547, 2023.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), San Diega, CA, USA, 2015.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3035-003555 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning
via sequence modeling. In International Conference on Machine Learning, pp. 16569-16594. PMLR, 2022.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
PewResearch.
Writing survey questions.
URL httph:
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts.
In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5203-5212, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ernon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305. 1302 2023.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHIl Conference Human Factors in Computing Systems, pp. 1-7, 2021.
Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient rlhf: Reducing the memory usage of ppo. arXiv preprint arXiv:2309.0077 00754, 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint arXiv:2303. 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 17X7:07777003477 2017.
Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Neural Information Processing Systems, 34:5861- 5873, 2021.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. (tti::300.//49922 /7492, 2023.
Simeng Sun, Dhawal Gupta, and Mohit lyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of rlhf. arXiv preprint 0Xiv:2309.00055. 2023.
12

---

Published as a conference paper at ICLR 2024
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An enstructionfollooing llama model. https::/ggihhe
com/ tatstt 1achsllab/ae stt/sstanforr.ooo alpaca 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kuishreshtha, Heng-Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint vrXiv:2211.08233 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09280 2023b.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruce Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. arXiv preprint
2rXii:2301.13848 2023.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh Interna- tional Conference on Learning Representations, 2022.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv 2019.
preprint
13

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
We highlight few limitations and directions for future work below:
Opinion Datasets: use datasets containing opinions of various demographic groups to validate GPO. Survey data is imperfect and may not be fully representative of entire group's 's population. Additionally, all the datasets that we use in this work are in English. When aligning to groups, the language that is used to collect preference data and during alignment may have significant effect on alignment metrics, especially if the inputs and outputs are in a different language than the native language of members of group. Future work should also investigate more challenging few-shot alignment settings, such as adapting to individual creative preferences where there may be much higher variance between group preferences.
Multiple-choice Format:
Like many previous works, we focus on a multiple-choice format due the availability of existing datasets and ease of quantitative evaluations. LLMs are capable of producing much more complicated long-form responses, and it is important that alignment methods can be extended to the general long-form response setting. While the GPO framework extends more broadly to different formats of LLM generations, future work should validate the effectiveness of GPO for longer form responses and additional considerations such as group preference feedback representation and evaluation metrics needed to extend to the long-form setting.
Alignment Objectives: When aligning LLMs, multiple factors beyond group preference alignment are also very important. Aligning to group preferences may result in worse alignment for other factors including as harmlessness and helpfulness especially if the group preference data includes examples that contradicts these values. Moreover, aligning to group preferences may amplify un- desirable behaviors from LLMs including biased or harmful outputs. Future work should study the impact of group alignment other important alignment factors and methods to reduce regressions for these factors when aligning to group preferences.
Initializing GPO with pretrained LM transformer backbone might offer advantages in performance. Specifically, leveraging a pretrained backbone could potentially enhance GPO's capacity encode world knowledge, thereby improving its ability to generalize to OOD) examples. Investigating the performance and generalization benefits of this initialization approach could be a promising direction for future work.
Model Initialization:
B DATASET DETAILS
B.1 OPINIONQA DATASET
This dataset is sourced from Pew American Trends Panel (PewResearch). This dataset's unique structural characteristics: the answer choices in the survey questions are principally ordinal (San-) turkar et al. 2023). For instance, options often extend across spectrum, ranging from categories such as "A great deal," "Fair amount," "Not much," to "Not at all." Traditional divergence metrics, such the Kullback-Leibles (KL) divergence, are ill-suited for this task, they fail to encapsulate the ordinal relationships inherent in the answer choices. In this dataset, the ordinal answer choices mapped a metric space using corresponding positive integers. For example, a typical mapping in our dataset might look like ( [A 1, B: 2, D: : 4). Therefore, 1-D Wasserstein Distance metric used. The alignment score for two opinion distributions P1 and P2 is consequently expressed as:
= 1 Σ QQI
(D) N -1
P2;
(4)
QEQ
Here, N denotes the total number of selectable answer options, excluding the option to refuse. The normalization factor, representing the maximal possible Wasserstein distance in the given metric space. The score bounded within the interval [0, 1), with a score of indicating perfect alignment between the two distributions.
term N 1 functions
We employ the dataset as encompassing 22 demographic groups within the US as outlined in Table Our analysis focuses 500 contentious questions, characterized by frequent disagreements
14

---

Published as a conference paper at ICLR 2024
among the considered subgroups. These questions the same ones used in the steerability analysis presented in the OpinionQA dataset (Santurkar et al. (023).
Attribute
Demographic Group
CREGION
Northeast, South
EDUCATION
College graduate/some postgrad, Less than high school
GENDER
Male, Female
POLIDEOLOGY
Liberal, Conservative, Moderate
INCOME
More than S100K+, Less than $30,000
POLPARTY
Democrat, Republican
RACE
Black, White, Asian, Hispanic
RELIG
Protestant, Jewish, Hindu, Atheist, Muslim
Table 1: Demographic groups considered in our analysis from the OpinionQA dataset.
B.2 GLOBAALOPINOONQA DATASET
The survey questions in this dataset is sourced from the Pew Research Center's Global Attitudes surveys Reeaaesea) and the World Values Survey (Haerpfer et al. 2022). These questions do not generally contain ordinal structures in the options and the ordinal scores not presented in the datasets. Therefore, we choose to use a different metric for evaluating the alignment in this dataset.
(2;1,);;11
Σ -1- (D-()((P())) Pa(a)) gEQ
Q
In this alternate scenario, JD signifies the densen-Shannon Distance following the paper's choice Durmus et al. 2023).
Country
Nigeria
Egypt
India (Current national sample)
China
Japan
Germany
France
Spain
United States
Canada
Brazil
Argentina
Australia
New Zealand
Table 2: List of countries considered in our study, from GlobalOpinionQA dataset.
Out of the 138 countries in the original GlobalOpinion(A dataset, we selected a subsample of 14 countries for our study due computational constraints. We extract all the survey questions that have the target countries' answers. The countries chosen (in Table 2 span several continents to ensure broad representation in our evaluation. For instance, Nigeria and Egypt cover Africa, while India and China represent Asia. European nations are represented by countries such Germany, France, and Spain, and the Americas include the United States, Canada, Brazil, and Argentina. Lastly, the Oceania region is represented by Australia and New Zealand.
C ABLATION ON THE GPO'S TRANSFORMER ARCHITECTURE
We design GPO with inductive biases that satisfy two properties that are important for accurate preference prediction:
15

---

Published as a conference paper at ICLR 2024
Unlike traditional transformers that utilize positional en- codings, GPO omits these encodings to ensure predictions remain unaffected by the sequence or permutation of context preference pairs. However, this loses the pairwise relations between (2 y:). To solve this, we concatenate each pair (I;, into a single token to inform the transformer of their pairwise relation. It adopts an alternative masking strategy that differs from the conventional causal mask. This approach enables context pairs to exclusively interact with each other, thereby maintaining focus on relevant information.
(1) Context Invariance (Property 1.;:
(2) Target Equivalence (Property 2.):
The masking strategy also makes target points only attend the context points, which ensures that the targets are only influenced by the context points, not by other targets. This aligns with the principle of conditional independence as stated in Equation In this setup, we don't require the query preference scores to be generated autoregressively, meaning the prediction doesn't ' depend on previously predicted queries. Therefore, we use masking strategy where context samples self-attend, while query pairs attend only to context samples, not other query pairs.
Property 1. Context Invariance. A model pa exhibits context invariance if, given any permutation function and any E [1 1). it satisfies:
(ym 1im Em +1ms 21:m: V1:m) = Po Po (1m++1n /Tm (=(1)=(()) (+(1)=r(m)
Property Target Equivariance. Model pa demonstrates target equivariance if, for any permuta- tion function and any [1, 1]. the following is true:
po mo Imm / Fg.mm Imn 1:m 1.m) po (mm)))))) (mm Iam 1q.
To illustrate the effectiveness of these biases, compare GPO with a standard autoregressive trans- former that employs a causal mask, akin to the transformers used in GPT-x series (Radford et al.) 2018 2019 Brown al., 2020p. This basic architecture includes autoregressive generation with the causal mask and uses positional encoding, which we previously omitted to ensure context invari- ance. Using an autoregressive generation approach violates the target equivalence property since the prediction of each query point relies on previously generated ones. As depicted in Table GPO's in- herent inductive biases yield superior alignment performance compared a traditional transformer. It's noteworthy that in this comparison, we still concatenate the (x, y) pairs into single tokens for the standard transformer, thus preserving the relationship between the viewpoint x and the group preference score.
Meta train on 40% groups 0.798 + 0.007
Meta train on 60% groups 0.820 + 0.004
Meta train on 80% groups 0.799 + 0.015
GPO Transformer
0.780 + 0.009
0.782 0.004
0.772 + 0.006
Table 3: Comparison of the alignment scores of GPO and standard autoregressive transformer alignment tasks on GlobalOpinionQA datasets with three group splits and runs are averaged over three seeds. Experiments are conducted on OpinionQA with Alpaca-7b as the base model.
D ABLATION ON GETTING EMBEDDINGS FROM THE LLM.
Given that the base LLMs we considered in our experiments were not explicitly trained for text summarizing, we examined three methods to generate the embedding xr of the sentence: 1) Using the embedding of the last token as the sentence embedding. 2) Averaging over the embeddings of all tokens in the sentence. 3) Concatenating the embeddings obtained from the previous two methods. As depicted in the table averaging over the token embeddings of the sentence yielded the most effective results, whereas relying solely on the last token embedding proved less adept at capturing sentence-level information.
E ABLATION ON ADDING GROUP META-CONTEXT FOR GPO
In the primary experiments, viewpoints x are embedded using an LLM. Notably, in our previous experiments, each x; does not contain group meta-data about the group's identity attributes. This
16

---

Published as a conference paper at ICLR 2024
Embedding Method
Alignment Score
Alpaca-7b last token
0.903 + 0.014
Alpaca-7b average tokens
0.946 = 0.007
Alpaca-7b last token + average
0.942 = 0.009
Table 4: Comparison of different embedding methods using Alpaca-7b as the base model on the OpinionQA dataset, with meta train split of 80%. Results are averaged across three seeds.
ablation study explores the potential performance enhancement that could be achieved by integrating meta-data into GPO. Specifically, the context information Cg is embedded into a vector = which of the same dimension as x as embedded by the same LLM. We examined adding Co from the three kinds of contextual prompts we study in K This embedding is then concatenated with each of the (a, y, Ectx) pairs, serving as the one input token for GPO. As illustrated in Table [ incor- porating context embeddings into the structure doesn't bolster GPO's performance across the three group split scenarios, instead it performs worse, We hypothesize this outcome arises because GPO, unlike LLMs, lacks comprehensive world knowledge of diverse group attributes, making it challeng- ing to adapt to the meta-data embeddings of unfamiliar groups. Instead, GPO excels in deducing preference distributions based the available (z, y) context sample pairs.
Meta train on 40% groups
Meta train 60% groups Meta train 80% groups
GPO GPO meta-data
0.920 0.003
0.926 0.013
0.946 0.007
0.900 + 0.003
0.916 0.017
0.926 0.006
Comparison of the alignment scores of GPO with and without meta-data embeddings with three group splits and runs are averaged over three seeds. Experiments are conducted on OpinionQA with Alpaca-7b the base model.
Table 5:
F BASELINES DETAILS
We compare our method against extensive baseline approaches for aligning LLM's predicted opinion distributions with human groups:
This baseline assumes that all answer options are chosen with equal prob- ability, indicating no preference or bias towards any specific option. For given question & E Q with N answer choices, the distribution Po (q) is represented as: (g) N N..... 1
Uniform Distribution:
The opinion distribution, denoted by Pr., is derived from traitrained LM without any group-specific steering or fine-tuning. For given question q Q, the distribution (g) generated by the model is extracted from the output probability distribution across the N available answer choices. We first extract the prediction scores for the next token from the LM, focusing on the top-K K tokens. We then normalize the values to obtain P. (g). For token that is missing from the top-K set, we allocate the smallest prediction score in the top-K K sct. We use K = 200 in our experiments.
LM Base:
LM Steered:
This baseline gauges the model's adaptability to align with a specific group E G when informed of the group information explicitly through the prompt. We use diverse prompting strategies- QA, BIO, and PORTRAY- convey group information, with examples in Appendix The opinion distribution obtained for group g under this steering expressed as P= Cg),
where cg denotes the context for group g.
Few-shot Prompt: Rather than giving the model explicit group information, we input a few ex- amples showing group's preferences for m context questions, constrained by the LM's context window size. Here the includes the context samples {g, Ti ys ym Using this context, the model is prompted generate a response for new, unseen question that aligns with the group's opinions. See Figure 8 in the Appendix for examples.
SFT per group: The LM is fine-tuned separately for each group g using a supervised loss. Let Quain C denote the subset of m context questions used for training. We create training examples (g, r) by sampling from Quain and then sampling responses r with respect to the preference distribution Pg (g). The loss is defined as:
17

---

Published as a conference paper at ICLR 2024
LSFT Pay log pa (rla)
where db represents the LM parameters and Po denotes the probability of producing the re- sponse given the question q. This procedure fine-tunes the LM to maximize the likelihood of the sampled responses that align with the preference distribution the specific group.
Reward Model: We start with the architecture of the base LM and add linear MLP head. The augmented model is trained on m context samples to predict the preference scores for the { } using mean squared error loss. Then, the model is employed to predict the preference scores for the query questions and softmax is applied to ensure that Σ Ug.q, for each query q-
In-Context Finetune: We investigate whether the LM be fine-tuned, akin GPO, to adapt to a distribution of groups using few-shot learning. This would ideally enable improved few- shot in-context adaptation for unseen groups. To this end, we partition the group set G into meta-train set Gunin and a meta-test set Grest During training, each group in Grain serves as training instance. The training questions for each group are split into context samples and query questions. For given query question q, we supplement it with few-shot context Cg. consisting m questions paired with the respective ground truth preference scores. This context mirrors the Few-shot Prompt strategy with example shown in Appendix 8 For supervision, for each query, we sample responses r, aligned with the human preference distribution (g)). The LM undergoes fine-tuning using dataset formed from these context-enhanced samples. The associated loss function is:
LICT -BB -gg log Pe (rlg,
G TRAINING SETTINGS
For all baseline fine-tuning methods, including SFT per group. reward modeling, and in-context fine-tuning that necessitate training the base LM, employ 8-bit integer quantization and utilize a single Nvidia RTX A6000 GPU with 48GB VRAM. Our parameter search for the learning rate encompassed values (3e-4, 2e-5, le-4). We settled on 1e-4 for the Alpaca baselines and 2e-5 for the Llama2-13B-cha baselines. For both SFT and in-context fine-tuning tasks, our effective batch size was comprised of a batch size of and gradient accumulation steps. contrast, reward model training had a batch size of 4 with the same gradient accumulation steps. All baseline methodologies were trained with LoRA (with 1=12, alpha=32, and a dropout rate of 0.05) with a weight decay of 0.01, utilizing bf16 precision and the AdamW optimizer (Loshchilov & Hutter 2018). For all methods, We use the validation alignment score for early stopping-
For GPO, the transformer's feedforward dimension was set to 128, with an embedding depth of 4, 4 heads, and layers. We sampled m uniformly from the range [10, 100] as context samples for every training task. We also used a learning rate of 3e-4, coupled with the Adam Optimizer (Kingma & Ba 2015). More training details can found our codebase.
H EXTENDING GPO BEYOND MULTIPLE-CHOLE QUESTIONS
The GPO framework presented in the main paper experiments can be extended beyond the multiple choice setting. GPO works for any LLM generation setting where there is some scalar which repre- sents feedback over an LLM response. We present GPO formulations for producing group aligned LLM responses in the long-form generation setting with two common forms of sparse feedback: (1) relative (e.g. response or response 2 better) and (2) absolute (e.g. rate the response on a scale of 1-7).
each context example includes responses and GPO is trained with binary classification objective for each example. During inference, the GPO module can be used to perform inference through a modified version best-of-n sampling where n sample responses are sampled from the base LLM and each of the (*) pairs of responses is inputted to GPO as queries. GPO's output can used to calculate a win rate for each of the n responses and the response with the highest win-rate is chosen as the aligned output response.
Relative feedback:
18

---

Published as a conference paper at ICLR 2024
Absolute feedback: each context example includes 1 prompt and GPO is trained to regress the absolute feedback score. During inference, the GPO module can be used as a reward model in best-of-n sampling to produce group aligned response.
Since GPO predicts group preference scalars, GPO can be used as a reward model to fine-tune the base LLM with PPO in settings where performing inference with an additional model is not desirable.
I ADDITIONAL. RELATED WORK
Alignment via Prompting. The conditional nature of LLMs enables them to be conditioned on specific task information or context data and alter their output distribution to respect the conditional information. Various studies have investigated this capability in adapting groups and personas. Deshpande et al. 2023) observed that prompts like "Speak like xxx" could elevate L.M'' toxicity levels contingent upon the persona's characteristics. ]Tiag et al. (2023) used prompts guide GPT- 3.5 to adopt certain personality traits. Beyond explicit persona group traits, an LLM's behavior can also be influenced by presenting it with few-shot examples from its in-context learning ability (Brown et al. 2020). For example, Hwang et al. (2023) uses the previous opinions of an individual adapt the LLM align with the user. The advantages of this strategy include its computational efficiency, eliminating the necessity for gradient updates. However, the few-shot examples are re-
stricted by the model's context size, and designing prompts for effective completion of tasks often requires careful prompt engineering (Lester et al. 202T Qin & Fisner, 202T Zhou et al., [022] Reynolds & McDonell, 202I). Additionally, when steering the LLM to be more representative of demographies group nuanced societal questions from survey datasets, especially on nuanced societal matters, research by Santurkar et al.] (2023): Durmus et al. (2023) shows that this steerability can be constrained, resulting in limited enhancements in model alignment.
Gradient-based Alignment. Another line of alignment work involves adjusting the LM's parame- ters using a preference dataset through fine-tuning. Methods have been proposed to align I.LMs with specific human values, such as helpfulness, harmlessness, and non-toxicity, using RLHF (Glaese et al. 2022; Ouyang et al. [022] Rafailov et al. [2023] Bai et al. (022a)). This necessitates the mod- eling of a reward model from
preference datasets and the use RL policies, such as PPO (Schulman et al. 2017), to maximize accumulated rewards. This PPO phase poses challenges in terms of computational and memory demands, necessitating the training and storing of large- scale policy, value, reward, and reference models, as well as optimizer states and gradients in GPU memory. Moreover, this process could require complex hyperparameter tuning (Sun et al.] 2023] Santacroce al.) 2023). To address this, methods such as Direct Preference Optimization (Rafailov ct al. 2023) and Preference Ranking Optimization (Song et al. (202)) have been proposed to directly learn from pairwise ranking-bass -basin preference datasets without reward modeling. However, these methods typically result in specialized models for every alignment task. Adapting to multifaceted, sometimes conflicting group preferences requires fine-tuning distinet models for each subgroup.
J QUALITATIVE EXAMPLES OF GPO.
This section contains qualitative examples that may be viewed as offensive or harmful. Here demonstrate multiple qualitative examples of GPO's predicted group preference versus the language model's stecred performance. Here we used only 15 context examples for GPO and the steered LM uses group's meta-data context.
Warning:
19

---

Published as a conference paper at ICLR 2024
Q: As you may know, same-sex marriage legal in the U.S. you think this thing bad thing for society?
theeet
Male
Jevis
Protestent
Dencontt
Conservatte
A. Very good thing
I.. Somenhal good thing
Somewwat bad thing
GRO Male
Joukch
GPO PIEHCINE
Aowcssloarsi-eeee Mathors
Lerra--taarr- lorthant
much, all, you following proposals reduce economic Inequality the U.5.? Increasing taxes on the wealthiest Americans
Narchesst
Athesst
Mlsiii
Moderota
Asiin
Wh
great deal
04
9%
ALA
Not too much
GhO-Hoethsaad
CPO-Musfin
Gro-Medercke
GrG-Alrie
Apace-tteersshorhhed
Apaca-teerrd-Muuii
Apaca-strerciiiiee
cleend Northresd
L'ams dims)-tteeesddriee
[llma) Aslen
HLN
20.%
Q: Do you think the number of legal immigrants the U.S. admits should
Muslee
Washerdo
Aaian
Increase lot
Inrrease a little
Staly about same
SLEN
Decrease litie
Decrease
GRD-Temes Clemaie
GIO Jestees
GPPO Muxiim
EPP Reputtican
GROD-Modrret
ERO-Man
Alpace stee Fema
Alpace steerec-josst
Alpaca Steers Mustim Alpaca steered Aepubi icen Alpace steered Moderate
Alyaco www.eeeeedddooo
steeret- Jearshh
Alamatlatenned LianaZ-ster www.seeerddddipppocontteattaa
wwweenattt-ttteed-eee Mockertte
Ulamad stwered-Aaiin
20

---

Published as a conference paper at ICLR 2024
O: In the futune, what kind of an impact do you think major corporations will have in solving the biggest problems facing the country?
A very posiivve impact
A somowhat live
someetat mowet negative
GPO nuatioed
GIO (PWid
GIO HEAd
GIR Pepdicas
UPO Mabrette
SPO Aia
ITTR
Unwsdolonattetllo oot.
Un Appstic
AIL
Q: you were decicling what rederal government should Improve quality lite tuturee generations, what priority would you give to providing high-quality, affordable health care to all Americans?
fumalc
Mal:
Alsbl
Asice
Important
priontty
Shoule not be done
(PS reriee
OPG Jewn
GPO Goin
Apucesllennsfercle
Apecasieenn--iie
Apuceelsenss-tharc
sterod
Uorea Nepibtiio
Q: How setisfied are you with the way democracy is working in our country - very satisfied, somewhat satisfied. not too sadisiied or not at all satisfied?
SoaP
India IC sument national semp
Aastrolo
Canada
Arpentea
Very Somewhut satiaree sutial ud Dontt knowiRetrsed DicRefused
spac-dees--oos Apecz coeendirii Carntrutioe wtw.h//irrddiiiiieptteccc chsss
pace cieeRdA
Unnooeeeeiger Uk uuuu-torddddd Paasni euiind Returrneeeeessinnrr
K CONTEXTUAL PROMPT EXAMPLES
In this paper, we examine three types of contextual prompts, delineated in Santurkar al. (2023). Below, we present examples of the question-answer, biographical, and portrait-based contextual prompts designed for individuals residing in the Northeastern United States.
21

---

Published as a conference paper at ICLR 2024
Question-Answer Prompt:
which part of the United States do you currently live in? Response: Northeast
Biographical Prompt:
Below, please provide a brief description of the region in which you currently reside within the United States, followed by answers to several questions. Description:
currently reside in the Northeast.
Portrait-Based Prompt:
Answer the following question as if you currently reside in the Northeast.
Figure 7: Three types contextual prompts to provide group information.
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Given the answer distributions from a specific demographic group for certain questions in public opinion survey, answer the subsequent new question by selecting ONE the options, as if you are member of this identified demographic group:
### Input:
Question: Question.1
A. Option
B. Option.2
C. Option_3
Answer Distribution:
A: 25%, B: 358, C: 408
Question: Question.
A. Option_1
В. 0ption_2
C. Option_3
Answer Distribution:
A: 358, B: 258, C: 40%
Based on the above list of answered questions from a demographic group, answer the quest by selecting ONE of the options, as if you are member of this demographic group:
Question: Questionm-1
A. Option_1
B. 0ption.2
C. Option_3
## Response:
Figure 8: Few-shot in-context prompt with context questions in Alpaca prompt format.
22

---

Published as a conference paper at ICLR 2024
Below is an instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
Given that you have the following demographics context:
Marital Status: Married,
Religious attendance: Roman Catholic, Region: Northeast, Age: 65+,
Sex: Male,
Education: Some college or no degree, Income: 000-500, 000, Political ideology: Conservative, Race: White, Answer the following question by picking ONE of the given options
### Input: Would you say Germany has done a good or bad job dealing with the coronavirus outbreak?
Options:
A. Very good
B. Somewhat good
C. Somewhat bad
D. Very bad
### Response:
Figure 9: A randomly selected individual contextual prompt examples in Alpaca prompt format.
23


Reviews:
[{"review_id": "piYrATxa6N", "reviewer": "Reviewer_dRt2", "summary": "This paper proposes a framework to tackle the preference prediction problem, that is given a question, predict a distribution over all possible answers provided by the format of multiple choice question. The authors propose to view the question as a semi-supervised prediction problem and use LLM to augment the input data x into a (x,r) pair. The final prediction is done by training a shallow Transformer model over the augmented.", "questions": "It would be great to see the authors compare with the very basic baseline, perform ablation studies, and fix some config settings to make the paper more complete.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a novel idea to augment the data. The final analysis shows that direct tuning on LLM would not also obtain the best qualities on few-shot datasets.", "weaknesses": "While the paper presents a new infra on the problem of preference distribution, the alignment method does not look too much different than a normal semi-supervised framework and there are some caveats in the experiment design and baseline choice to fully justify the acceptance of this submission.\n\n* For the baseline, the authors seem to fail to include one direct method.\n  * A simple Transformer model that purely learns from the (q, y) pair and uses them in a semi-supervised fashion as a sequence of inputs. As is pointed out by the author, the Reward Model baseline is underperforming a lot of the other baselines, it would really make sense to add a comparable-sized baseline as in the author’s proposed method to rule out the possibility that overfitting is the only cause of inferior baselines w/o really relying on LLM. \n* The PeftConfig configuration seems not consistent with the description of the Reward Model baseline in the paper. The authors argue to use a linear MLP head for the Reward Model baseline. However, in the code, the authors used a LoraConfig which should by default fine-tune every layer of LLM, and might be the major reason for the underperformed score of this baseline.\nAlso, it makes more sense to use cross-entropy for the loss function instead of MSE as the final output is a distribution.\n* The terminology of alignment seems a bit too abused and distracting in this setting. In this work, the authors only tried to learn a separate Transformer architecture that operates upon the output of LLM while the LLM itself does not enjoy any new abilities in its parameters based on the modifications.\n* It would also be an interesting point in this work to justify the reason for the output of (r) as input in this work (through ablation studies). My current understanding is that the sampled response would be used as an anchor point for the training of the proposed method, however, it would lead to a natural question: without r, will the model underperform a lot, meaning that LLM would also be the important ingredient? Also, in this case, it seems that the qualities of the response from the model would also seem to be quite important, and it would make more sense to replace it with random strings etc to further make the study more coherent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EQ0IDT2BTA", "reviewer": "Reviewer_DqhU", "summary": "This work proposes a method to alignment LLM to group preferences so that a LLM grounded on several shots of preference examples can better predict the group preference for the unseen questions.", "questions": "1. In the second last paragraph of page 4, it is said that \"In particular, we discard the positional encodings commonly found in standard transformer architectures\". Wouldn't the removal of positional encodings significantly deteriorate the performance of those pre-trained and fine-tuned LLMs?\n2. Still in the second last paragraph of page 4, it is said that \"we employ a masking strategy where the context pairs can self-attend to each other\". Could you elaborate such a masking strategy?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The problem this work proposes to solve is kind of new and unique.\n2. The baselines are quite comprehensive and the proposed method significantly outperforms all of them, which validates the superiority of this method.", "weaknesses": "1. I am not sure about the significance and broadness of the problem this work tries to solve. Alignment of LLMs to group preferences sounds important, however, the evaluation datasets used by this work look quite specific and narrow and I don't think it is of interest to a broad range of research community. \n2. The proposed method look quite trivial and standard, which is a in-context fine-tuning method. There are some changes in the method details but those details are a bit hard to understand, which I will pose some questions on next.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HVeIfxGfRa", "reviewer": "Reviewer_UCgS", "summary": "The study introduces Group Preference Optimization (GPO), an innovative alignment framework designed to tailor large language models (LLMs) to specific group preferences with minimal data. GPO optimizes the model with reduced data, surpassing current methods like in-context steering and fine-tuning. Experiments on OpinionQA exhibit that GPO effectively adjusted the model's preferences on multiple-choice tasks.", "questions": "1.\tAs mentioned, it is ambiguous which part parameters are tunable. Could the authors further clarify the training details?\n\n2.\tCould the authors provide examples of generation tasks (QA) to better showcase the model's preference? Demonstrating the model's performance on responses could provide a clearer insight into its practical applications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThe paper commendably focuses on the concept of \"Group Preference\", introducing an efficient approach for aligning large language models to specific groups. \n\n2.\tEfficient fine-tuning of a few-shot learning scenario, blending both in-context and fine-tuning methods, stands out. This approach offers a practical solution in settings where extensive labeled data might not be available.\n\n3.\tThe empirical evaluations seem robust and thorough. Not only do the results show clear improvements, but the detailed analysis and discussion also provide insightful opinions.", "weaknesses": "1.\tLack of clarity in the presentation of the method. For instance, it remains ambiguous which specific parameters are subject to training. Incorporating a detailed algorithm diagram or flowchart would greatly enhance the comprehensibility. Besides, more training details would be better.\n\n2.\tThe absence of evaluation results concerning the generalization capability of the GPO method. As GPO optimizes the parameters, it's crucial to show whether the model retains performance on general benchmarks after tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)