PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment
Authors: Daiwei Chen
Yi Chen
Aniket Rege
Zhi Wang
Ramya Korlakai Vinayak
Venue: ICLR 2025 Poster (2025)
Keywords: alignment
preference learning
foundation model
reward model
ideal point model
plurality
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: A novel alignment framework to learn from heterogeneous human preferences

Abstract: Foundation models trained on internet-scale data benefit from extensive alignment to human preferences before deployment. However, existing methods typically assume a homogeneous preference shared by all individuals, overlooking the diversity inherent in human values. In this work, we propose a general reward modeling framework for pluralistic alignment (PAL), which incorporates diverse preferences from the ground up. PAL has a modular design that leverages commonalities across users while catering to individual personalization, enabling efficient few-shot localization of preferences for new users. Extensive empirical evaluation demonstrates that PAL matches or outperforms state-of-the-art methods on both text-to-text and text-to-image tasks: on Reddit TL;DR Summary, PAL is 1.7% more accurate for seen users and 36% more accurate for unseen users compared to the previous best method, with 100× less parameters. On Pick-a-Pic v2, PAL is 2.5% more accurate than the best method with 156× fewer learned parameters. Finally, we provide theoretical analysis for generalization of rewards learned via PAL framework showcasing the reduction in number of samples needed per user.

Published as a conference paper at ICLR 2025
PAL: SAMPLE-EFRICIENT PERSONALIZED REWARD MODELING FOR PLURALISTIC ALIGNMENT
Daiwei Chen, Yi Chen, Aniket Rege, Zhi Wang, and Ramya Korlakai Vinayak University of Wisconsin-Madiixt Madison, WI, USA
ABSTRACT
Foundation models trained internet-scale data benefit from extensive align- ment to human preferences before deployment. However, existing methods typ- ically assume homogeneous preference shared by all individuals, overlook- ing the diversity inherent human values. In this work, we propose a gen- eral reward modeling framework for pluralistic alignment (PAL), which incor- porates diverse preferences from the ground up. PAL has a modular design that leverages commonalities across users while catering to individual person alization, enabling efficient few-shot localization of preferences for new users. Extensive empirical evaluation demonstrates that PAL matches or outperforms state-of-the-art methods on both text-io-text and text-to-image tasks: on Red- dit TL;DR Summary, PAL is 1.7% more accurate for seen users and 36% more accurate for unseen users compared to the previous best method, with 100x less parameters. On Pick-a-Pic v2, PAL is 2.5% more accurate than the best method with 156x fewer learned parameters. Finally, analysis for generalization of rewards learned via PAL showcasing the reduc- tion in number of samples needed per Our code is publicly available https: : / /gthhh....uu.ggg com RaayRaaaaaaaaa hwwaaa/ppuaalsttee aeciiccallggnn alignment.
provide theoretical
a) Learn Reward Model ry does user prefer æj or 2, for context 2,
Model
Resoard (BM) Soe toer
X
(x xx) - R
- S
X
ipedertom vit light roaeed
Generation
X²
(xy 18
b) PAL: pluralistic RM with learned latent pref. points P or pref. mappings G
c) How does PAL
adapt to new users?
(M)
Weights
(())
[N)
Weights
W
Fixed for new user.
Learned
G(-)
Maye learned
froan Data
Prototyptcal
Prototypical
A:
f
W)
pref. points
PAL-B:
pref. nappings
(x xx,)
[1f(x, x.) f()
[ (x.)
(f(x), 2
B:
f.
G
W)
Figure 1: (a) Using preference data, PAL framework learns a personalized reward model each user rg (.). (-). which captures user's preference any output x given context X.. (b) PAL models common perceptions similarity across users through a shared representation f(-), and represents individual aspects of preferences via either a preference point ( PAL-A preference mapping 2(0) (xe) PAL-B, In particular, we assume a low-rank structure with prototypical preference points or preference mappings; see Section for details. (c) PAL enables efficient few-shot preference learning for new user-only a K- dimensional weight vector is learned. This reduces computational cost well as data needed for generalization.
INTRODUCTION
Foundation models trained on internet data are often not readily deployable and undergo alignment to human preferences using large amounts of pairwise comparison feedback (Ouyang et al., 2022). While aligning AI/ML models to human preferences, it is important to consider whose preferences aligning them to? The status quo for popular alignment frameworks is to assume homo- geneous preference shared by all humans. However, humans have diverse preferences, values and opinions (Bakker et al.. 2022; Durmus et al., 2024; Nadal & Chatterjee, 2019; Wildavsky, 1987). The need to capture this plurality in the context of AI alignment was also highlighted recently by Sorensen al. (2024). However, the methods suggested therein and other recent works look at

---

Published as a conference paper at ICLR 2025
learning multiple rewards with top-down approach, where the system designer decides the number and axes that one should care about (Cheng et al., 2023; Kovač et al., 2023; Ouyang et al.. 2022; Santurkar et al., 2023), e.g., helpfulness vs. harmlessness (Bai et al., 2022b;a; Ganguli et al., 2022; Rame et al., 2024). In reality, human preferences are more complex than the designer-speciffe axes (Bakker et al., 2022), especially subjective aspects, which leads us to propose the following goal.
Goal: Develop sample-efficient, personalized reward modeling framework for pluralistic align- ment from the ground up which learns and generalizes to heterogeneous preferences.
Our Contributions. Towards this goal, we make the following contributions:
1. We propose PAL, a novel, sample-eflicient, personalizable reward modeling framework for pluralistic alignment from the ground up (Section 2). Our modular and versatile framework achieves superior performance to the state-of-the-art (SoTA) in both language (Section 3.1) and vision (Section 3.2, 3.3) tasks, while utilizing only a fraction of their learnable parameters.
2. PAL reward models achieve competitive performance with simple 2-layer MLPs on top of frozen foundation models of varying sizes in practical settings across modalities (Section 3). PAL enables democratic alignment via strong aceuracy-compen
optimality (Rege et al., 2023).
PAL is complementary to existing alignment frameworks, and works seamlessly across compute budgets (see Figure 2) demonstrating the strength of its flexible and modular design.
4. We provide sample complexity guarantees for generalization towards new preference pre- dictions for users in the dataset well as for unseen users via few-shot learning, in the fully connected linear layer setting for one of our models (Section 4.1), and we verify these results with extensive numerical simulations (Section 4.2).
The learned PAL reward models can be used flexibly for personalizing downstream task of alignment through (i) train-time methods such as PPO-based RLHF (Christiano et al., 2017; Wu et al., 2023a) and (ii) inference time methods via best-of-n n sampling such as controlled decoding (Liu et al., 2024; Mudgal et al., 2024). In addition, the modular nature of PAL has high potential for future adaptability as it enables seamlessly updating learned reward models via switching data encoders and distance metrics, or adding new prototypes to account for dynamically changing heterogeneous preferences. We note that, in this paper, we focus on developing sample-efficient personalizable reward modeling, and understanding its efficacy via extensive experiments and theoretical analysis.
Background. Our reward modeling builds on the popular preference learning models, the Bradley-Terry-Lace (BTL) model (Bradley & Terry, 1952) and the ideal point model (Coombs, 1950), both of which are special cases of the linear stochastic transitivity (LST) models. We pro- vide a brief discussion of these models, including their assumptions and limitations here. Let D denote the dimension of the representation space of the foundation models. Given a context/prompt X. R., generative model(s) can produce different outputs, denoted by x E R.D.! The LST models make the following assumption: Suppose each output for the prompt is associated with a true score sx(x x.) E IR. Given any list of alternates, the true scores lead to a true ranking of them: for any two alternates, is preferred over x, if 8* (xx;x.) > 8* *xxx;xx) However, when we elicit comparison feedback from humans, the answers may be noisy and are modeled as, his* (xx: 8 (x,; x.))), where h : E - [0, 1] is strictly monotonic link function that satisfies h(2) = 1 h(-2). In other words, Pr(x: > X, xx) = 1/2 when s* (x1, xx) s* (x,, xx) and Pr(x: x,|xx) 1/2 when s* (xx,, x.) > s* Xx)
Pr(xi > (x.)
The BTL model uses the logistic sigmoid function as the link function:
Pr(xi x, (xe) + exp (s* (x,;xx) Xe) st (x::x,))
On the other hand, the ideal point model uses latent vector a E R to denote the ideal preference point of a user, along with (negative) distance-based scoring functionn; the model is given by,
Pr(xi > x, (xx) = (ddist (x,,, a) - dist? (x, a)),
where h can be any valid link function. The key idea here is that the larger the difference in distances between the alternates to the ideal point, the easier it is to choose between them, and hence the
'These representations can he taken from penultimate layer(s) of a foundation model. While we use the same D for the prompt and output for simplicity, this can easily be extended to different dimensional spaces.
2

---

Published as a conference paper at ICLR 2025
answer is less noisy. We note that with sigmoid link function, the ideal point model can be viewed as the BTL model with a distance-based scoring function.
A key limitation of these approaches is that they assume single tre ranking of the alternates and model the differences in elicited preference as noisy observations. In reality, the differences in elicited preferences not merely noise, instead reflection of plurality of human preferences.
The ideal point model provides a nat- ural starting point incorporate in- dividual preferences since each user be modeled using their own latent preference. However, the assumption that we know the distance function that reflects human notion of sim- ilarity of alternates can be too strong in practice. Further, completely in- dividualized models devoid of any shared structure will lead to unneces- sary burden on per user sample com- plexity for learning and would be dif- ficult to generalize. Our aim is to cap- ture the heterogeneity of preferences while also leveraging commonalities (Section 2), allowing for sample effi- cient learning and generalization.
100%
Ege
<flxed DostELLEERT E500
gtc Owand
90%
2Layers ML'S
BPT-2508
Tost accurady
(Fixod tye e3 Sear)
071-2-0.72
80%
20%
Zlayers MLPs
(fixed gie Q.c22-1.88
2Layer5-MLPs
70%
1000
PAL-B-Tiny (K=2, Ours)
60%
PAL B-Large (K-2, Ours)
P-OPO
Vanilla DPO
GPT-1-6.71
50%
10
10'
10"
10"
# Millions Learnable Parameters (Log Scale)
Figure 2: On Reddit TL:DR, PAL is timal and shows state-of-the-an (SoTA) performance.
op-
2 PAL: REWARD MODELING FOR PLURALISTIC ALIGNMENT
In this section, we describe our proposed personalizable reward modeling framework that captures commonalities shared across the population which can be learned using the pooled data and indi- vidual aspects that is learned per user in a sample efficient way. For user i, given a context x,, the probability of alternate x; being preferred over x, is as follows,
Pr(xi > x, Xe, i) 10 (x,:x,)
(xxxx)));
(3)
where (6) any valid link function that can depend on the user, and (.)( is the personalized reward function for user i.2 We do not assume the knowledge of the link function for our learning algorithms (Section 2.1). We propose two models for the personalized reward function":
PAL-A: Diverse preferences modeled via latent ideal preference points. The shared sense of similarity of different alternates being compared is modeled as Euclidean distance in an unknown mapped space, captured by f TR2D + Rd that jointly maps the output and context, (x; x.), to this unknown space. The latent preference of each user i is modeled using an unknown ideal preference point (+)) E RDD How much the user values output x given context x, modeled as inversely proportional how far away the mapping of (x: x2) is from the user's ideal point. To further capture the commonality in the preferences among users, each user's preference points & modeled as a convex combination of K prototypical points, that is, (i) = > w P where
the weights (n(() 2 O and EK (.(() 1, and (PI...:PA p with Pt RR2D are K prototypical ideal preference points. More formally, the personalized reward function and the corresponding personalized probabilistic preference model is given by,
PAL-A:
(xxxx) xx) = -1(xxxx)-(((a)))) (a)))
(4)
Pr(xi > Xr Xxy i) = (x,; xx) (aa [1f((x;3xx) - f(a
(5)
Denoting P = -p..... px]. each user's preference point a (i) = Pw!"), where w()
[w (4)
w (1) lies in the (K (1--1)-dimen -imeension simplex, A K-1
We drop the superscript on denoting user specificity for simplicity in rest discussions, however, we note that the link function need not be the same for all users and our learning methods are agnostic to them. In Appendix B.2, we discuss when choose model over the other in practice.
3

---

Published as a conference paper at ICLR 2025
PAL-B: Diverse preferences modeled via latent preference mappings. Here each user's pref- erence also incorporates the given context and is modeled using an unknown preference mapping RD + Rd. The shared sense of similarity of different alternates being compared is modeled as cosine similarity in an unknown mapped space, ʃ : TRD - s The commonality in the preferences among users is captured by modeling each user's preference mapping convex com- bination of K prototypical mappings, i.c., x00) (xe) = Ex=1 w 9* (xx), where {gI..... 9k) with gh R2D gd-1 are the prototypical mappings, w 0 and Σ (1) 1. Formally, the personalized reward function and the corresponding probabilistic preference model is given by,
PAL-B:
r8 (x:x,) (f(x), 2 (xx)),
(6)
Px x, xx) h (f(x,,; 0 (x.))
(xx)),
For any context x,, denoting G(X.) == (x,), 9K (x.)), each user's preference mapping of the given context is, 21x) (xe) == G(x,)w()) with ((( E A AK-1
PAL modeling framework is modular, i.c., provides a systematic way to incorporate shared and personalized portions of preferences, as well as transparent way to control multiple notions of com- plexity via cross validation: (i) the complexity of mapping f (and g's in PAL-B) captures the shared human notion of similarity between alternates. If the underlying foundation model used to obtain the representations of outputs and contexts (x and x.) are rich and semantically meaningful, then much smaller models suffice in capturing the rewards (Figure 2); (ii) the number of prototypes K capture the level of heterogeneity of human preferences in the dataset: more diverse preferences mandate larger K for good generalization (Figure 5(b); and (iii) in conjunction with the proto- types, the individualized weights allow for personalization with much fewer samples per user for both seen and unseen users. This reduces the annotation burden individuals during data collec- tion (Figure 5(c)) and the amount of samples needed for few-shot learning for new users arriving on deployment (Figure 3). We illustrate the PAL framework in Figure and Figure B.2 (Appendix B).
2.1 LEARNING PAL MODELS FROM DIVERSE PREFERENCES
Let D := ([(() y) [==1] [== be dataset of preference comparisons, where m. denotes the number of pairs answered by user i, and y is the answer given to the pair (y = -1 if is preferred, y otherwise). This can be looked at as supervised learning problem with binary labels. The goal of the learning algorithm in the PAL framework is to learn the mappings and prototypes shared across the population, and for each user i, the weights w-()) E A For PAL-A, the mapping f and the prototypes prototype mappings {9x ]o are shared. These shared portions are learned using data pooled from all users while the user specific weights are learned using each individual user's 's preferences.
K { are shared, while for PAL-B, the mapping f and the
Given the comparison dataset D, loss function IR - [0, 1]. model class for fa and prototypi- cal mappings {9I...., "u....,9k), the learning algorithm for PAL-B starts by randomly initializing these functions, and user weights (( E A AK-1, = I,...,N: N. Then, in each iteration until convergence criteria, the following steps are repeated,
Sample a random mini-batch (xx, X,:
xx ) of comparison data from D.
For each comparison j from user
Compute user ideal mappings: 200) (xx) = (xc) gx 9x(x))] w())
(xx))) a(8) = (Fo (x+) 200) (xx))) Compute loss: w6 (xi, xx y) l (y: -a(a)) -d("))
Compute distances: d = (fo (xi) x(0))(())))))))
Update Step: argming, (or ). (w()) Ei, 10 (xt, Xri Xc, v).
Learning steps are similar for PAL-A. See Appendix B for pseudocode details.
2.2 GENERALIZATION OF PREFERENCE PREDICTIONS FOR seen VERSUS unseen USERS
When learning a reward function from diverse preferences, there are two types of generalization to consider: Predicting preferences for (1) unseen pairs for seen users, i.e., the people for whom the
4

---

Published as a conference paper at ICLR 2025
weights have already been learned from the training data; (2) unseen users, i.e. people whose data was not part of the training data at all. For such new users, portion of their data will be used to learn weights to localize them within the learned model while keeping the shared mappings and prototypes fixed. We also note that the weighted combination of the prototypes, i.e., an average of all the seen users, can be used as the zero-shot ideal point for new users (e.g. Netflix recommendations for new accounts). However, we emphasize that it is important for reward functions to generalize to unseen users and PAL provides natural way localize new users, we demonstrate in Section 3.1. Section 4. provides theoretical guarantees on the per-user sample complexity of P PAL for few-shot generalization to unseen users.
EXPERIMENTS ON REAL DATASETS
In this section, we verify the following claims through extensive empirical evaluation:
C1. PAL can effectively capture the diversity of user preferences and outperform status-quo homogeneous reward models.
C2. Compared to existing pluralistic reward modeling methods, PAL can achieve state-of-the- art (SoTA) performance with significantly fewer parameters.
C3. PAL works for both text-to-text (T2T) and text-to-image (T2I) tasks.
C4. PAL demonstrates strong few-shot generalization new, unseen users.
We employ two strategies for defining and training the learnable mapping f between the embed- dings from a foundation model and latent space. In the Tiny strategy. f is a simple two-layer MLP operating on frozen foundation model. In the Large strategy. f combination of the founda- tion model and two-layer MLP, with both components being learnable. Models employing these strategies are referred to as PAL-A-Tiny/ PAL-B-Tiny and P PAL-A-Larges PAL-B-Large respectively. Appendix D. describes the general training procedure of our algorithm.
We perform experiments on datasets in different domains: the Reddit TL:DR Summary dataset (Sec- tion 3.1) and the Pick-a-Pic dataset (Section 3.2). Due to limitations in currently available datasets for pluralistic alignment, which we highlight Section 3.2, we also created semi-synthetic datasets, fick-a-Filter (Section 3.3) and Persona (Appendix D.4), to further validate the above claims.
3.1 REDDIT TL;DR SUMMARY (TEXT-TO-TEXT)
Dataset. Reddit TL:DR Summary dataset curated by Sti- ennon et al. (2020) contains a series of preferences over summaries generated by language models. For each pair of summaries, and X,, a labeler i determines if x; is preferred or not. Each pair is also accompanied by the unique identifier of the labeler. We used the variant of the TL;DR dataset proposed by Li et al. (2024), which uses the summary length the preference. The major- ity group prefers longer summaries while the minority prefers shorter summaries.
90%
Test Accuracy
80%-
70%-
Seen User. PAL-B-Lange
60%-
Tiny
Unseen User, PaLgg-Large
Setup. We train PAL with sentence embeddings from foundation models including 0PT-35CM (Zhang al., 2022), DistilBERT (Sanh et al., 2019), BGE M3 (Chen al., 2024), and gte-Qwen2-1- 5B (Li al., 2023), which make up Large and Tiny variants depending on their size paramettrrss) The loss design follows the typ- ical loss of the Reward Model, we use the cumulative loss which weights the per-token reward loss. Details of the loss function, hyperparameter setting, un- seen dataset, and training setup deferred to Appendix D.3.
Unisenn User PAL-B-Tiny
2510 20 30
40
50
60 70 80 90 100
Few-Shot Sample per
Figure 3: On Reddit TL;DR, PAL gen- cralizes well to unseen data using just 20 samples per unseen user (few-shot).
Baselines. We compare to personalized (Li et al., 2024) and vanilla DPO (Rafailov et al., 2024).
Results. We train our model with 5 different seeds and report the mean and standard devia- tion. Figure 2 shows the performance PAL with foundation models of different sizes. PAL shows strong accuracy-compute optimality with no hyperparameter tuning: compared to SoTA, PAL-B-Large ( aate-Cwen2-1.. 5B) is 5.9% more accurate on seen users with 4x less param- eters while PAL-B-Tiny (Di st BERT) is on-par with x fewer parameters. Furthermore,
5

---

Published as a conference paper at ICLR 2025
Table On Reddit TL,DR Sum- mary, PAl-B-harge outperforms
0.85
Ny Welliigg (-1)
Nathhii (P-)
SoTA P-DPO seen users (+1.7% (
and unseen
(+36%) with 6.3 billion fewer parameters. Note that
Accuraey
0.85
-P- Poddoo
0.90
0.70
we use only 10 samples per unseen
0.75
user to localize their weights.
0.70
0.0 0.2
Model
Scco Aue 193)
Ussezo Aue (3)
0.85
40
0.60
P.DPO Cluser
56.55
PAL n Tiny (K 2)
79.54 2: 0.54
74.72 a 0.54
PAI-B-large ck =
92.2 a 0.95
91.43 : 0.54
Midure Ralio B
Figure 4: On Pick-a-Filter, FAL-B-Tiny outperforms homoge- neous models as user groups become more helerogeneous (? B).
Figure 3 illustrates PAL's ability to generalize effectively to unseen users in few-shot settings. As the number of samples per unseen user increases, PAL progressively adapts to their preferences. With just 20 samples per unseen user, PAL achieves performance comparable to that of seen users. Addi- tionally, as depicted in Figure D.3 (Appendix D), PAL exhibits superior performance over baseline when provided with as few as 2 samples.
models unseen users,
Table reports the performance of PAL-B-Lazze (CPT-35 350M) compared to SoTA P-DPO ( (Li al., 2024)). We observe that PAL, with around 6.3 billion fewer parameters, is 1.7% more accurate on seen users, and 36% more accurate on unseen users (C2, C4). Figure shows that for new users, PAL match seen performance with only 20 samples, showing its promising potential to flexibly adapt to new users. We further highlight PAL's strong few-shot generalization to unseen users compared baselines in Figure D.3 in the Appendix. Lastly, also show exhaustive results for all our model configurations based on CPT-350M in Table D.1 and D.3 (Appendix D.3).
3.2 PICK-A-PIC (TEXT-TO-IMAGE)
In this section, we examine how agnostic PAL is to data modality (C3) via the Pick-a-Pic dataset (Kirstain et al., 2024). Popular T2I reward models usually require fine-tuning foundation models with billions of parameters (Wu et al., 2023b; Kirstain et al., 2024; Xu et al., 2024). We show that PAL achieves competitive performance or SoTA performance while having significantly fewer parameters than baselines (C2).
Dataset. The Pick-a-Pic dataset (Kirstain al., 2024) is large, open dataset for human feedback in T21 generation. There are two versions of Pick-a-Pic, v1 and v2, where v2 extends v1. To ensure fair model evaluation, we divide the v2 test set into "nnoleakage" and "leakage" subsets due to overlap ("leakage") with the vl train set. We only consider the 18391 samples with no preference ties, i.c. one generated image is always preferred to the other. Out of these, 10587 samples (~ 58%) overlap with the training and validation sets of v1, which was used to train PickScore Kirstain et al. (2024) - we call this the v2 "leakage" subset. The remaining 7804 test samples (~ 42%) in v2 do not overlap with vl train and val, ensuring they are distinct for evaluation purposes we call this the v2 "no-leakage" subset.
Setup. We train PAL-B with logistic loss on both v1 and v2 for 10 epochs top of CLIP-H/14 or PickScore (Kirstain et al., 2024) embeddings. We discuss hyperparameter tuning Appendix D.5.
Baselines. We compare to SoTA PickScore (Kirstain et al., 2024) and a vanilla CLIP-H/14.
Results. Table 2 shows (a) PAL matches SoTA PickScore when trained on V1 with 165x fewer parameters (b) PAL exceeds SoTA performance on v2-no-leakage (i.e. fair comparison) by 2% when training on vl, and by 2.5% if training on v2 (C1, C3). Training PAL. from "scratch" i.e. with CLIP-H embeddings, outperforms training PickScore embeddings (which were trained on vl). We note that BALL-B-Tiny (~-6M params) exceeds SoTA performance while training on a single RTX 4090 GPU (see Appendix E), whereas PickScore (~1B params) is trained with 8 8 A100 GPUs - highlighting the suitability of PAL for efficient and democratic reward modeling (C2).
Remark. Since the data collection process for existing datasets involves the usage of strict rubrics (Stiennon et al., 2020; Kirstain et al., 2024; Wu et al., 2023b), labeler performance mon- itoring (Xu et al., 2024), and disproportionate amount of data provided by small fraction of

---

Published as a conference paper at ICLR 2025
Table 2: Seen user test accuracy of PAL compared baselines on Pick-a-Pic v2. Entries with asterisk" have inflated accuracy due to the V2 test set overlap with V1 train (See dataset details in Section 3.2).
V2 Test Accuracy (%)
Model
Params
Trainset
VI Test Accuracy (%)
No-Leakage
Leakage
CLIP-H/I4
986M
59.23
62.57
58.59
lewde
447M
61.10
HPS v2.1
986M
66.70
PickScore
986M
VI
71.85
68.04
76.
PAL-A-Tiny (CLIP-H)
8.4M
VI
69.29 + 0.6
PAL-B-Time Biny (CLIP-H)
6.3M
VI
71.13 + 0.3
70.02 0.39
79.32 + 1.68*
PAL-B-Timy Tiny (CLIP-H)
6.3M
V2
70.51
0.22
68.67 + 0.51
PAL-B-Tiny (Pickscore)
6.3M
V2
70.16 0.19
74.79 + 0.13*
users, these datasets may not be heterogeneous. We note that strict rubric leads to uniformity as essentially crowdsources the criteria of the rubric instead of eliciting the preferences of the users. Therefore, even using PAL with K 1, surpass existing SoTA performance. These results highlight the need for more nuanced approaches to collect datasets that elicit diverse opinions.
3.3 PICK-A-FILTER (TEXT-TO-IMAGE)
As PickScore has been shown to lack significant heterogeneity (Kirstain et al., 2024), we artifi- cially inject plurality to create the Pick-a-Filter dataset, and show that PAL significantly surpasses homogeneous reward models when pluralistic preferences are present (C1, C2, C3).
Dataset. Motivated by a natural human color preference distribution Palmer Schloss (2010), we choose adding different color filters to the generated images as the mechanism by which we explicitly "inject" preference diversity into the v1 dataset (group I prefers 'red' tones and group 2 prefers 'blue' tones). To avoid the model latching on purely to color features when learning preferences, we use a hyperparameter called mixture ratio 8 = N where N₃ is the number of vI pairs choose to apply filters on, and N, is the total number of original v1 pairs. The larger the 9, the more color-filtered v1 pairs in the training set. We show and discuss our careful construction of Pick-a-Filter in Figure D.7 and Appendix D respectively.
Setup. We train PAL-B-Timg By with logistic loss on the Pick-a-Filter dataset with different mixture ratios. Detailed training setups are deferred Appendix D.6.
Baselines. We compare PAL to CLIP-H, following Kirstain et al. (2024), as well as two strong T2I reward models: ImageReward (Xu et al., 2024) and HPS v2 (Wu al., 2023b). Note that we cannot compare to PickScore as its train set overlaps with Pick-a-Filter's 's val set (leakage).
Results. Figure 4 shows that PAL-B-Tiny can learn diverse user preference groups across mixture ratios (C1, C3). We can view 3 as indicating how much the two user groups prefer their respective color filters (higher 3 - affinity for filters). PAL significantly outperforms the homogeneous reward model in predicting user preferences at B =1, PAL achieves 95.2% test accuracy compared to 75.4% from the homogeneous reward model (C2).
4 SAMPLE COMPLEXITY FOR LEARNING REWARDS UNDER PAL MODELING
In this section, we shed light the per-user sample complexity of PAL for (1) generalization to un- seen pairs involving known users, and (2) few-shot generalization to unseen users. We present the- oretical guarantees in Section 4.1, and empirical results from numerical simulations in Section 4.2.
4.1 THEORETICAL GUARANTEES
To shed light on the benefits of using mixture modeling approach, we provide theoretical analysis of PAL-A with fully-connected linear layer in neural network mapping f. For simplicity of exposition, we subsume the context/prompt vectors into the item embeddings, treating (x; xx) as E RP.
Consider a class linear transformations, JF C {f DRD + RD f(x) = Ax, A E DxD] Then, the difference of scores between two items for user with ideal point a Pw is given by
(x)) s(a) l1((x)) - 2 (xl a) T A A(x:
a) (x, a) A A(x, -a) =x, (A A)xi x AA, 2(xI x,) ((T

---

Published as a conference paper at ICLR 2025
Observe that the right-hand side of the first equality represents the difference between the squared Mahalanobis distances" from X to a and from x, to a, where the Mahalanobis distance is defined by A A. In other words, the problem is equivalent to simultaneously learning a Mahalanobis distance and the user ideal points (Xu & Davenport, 2020; Canal et al., 2022).
Now, let M A A and Q 2MP. The difference in scores then be written
llf()) f(a)l llf(x,) f(a)l1 x Mx x, Mx, + (xi -x,) Qw.
xx.)
Given these reparameterizations from f to M and from P to Q for the remainder of this section, we assume that the reward modeling problem is defined over M E [M E I2 DxD; : || S M 0}. Q E {Q E RP =|Que12 I So Ak E [K]]. and w(f) - [0, 1] be an L-Lipschitz loss function.
A K- for each user i. In addition, we let
Work of Canal et al. (2022, Theorem 3.1) shows that a set of N 2 I!(D²) known users, the per- user sample complexity for generalization to unseen pairs of items is O (D). Key questions remain: Does our mixture modeling lead improved per-user sample complexity, e.g., O(K) Can char- acterize PAL'S generalization ability to wnseen users, and determine the per-user sample complexity for few-shot localization of preference points (going beyond analysis in Canal et al. (2022))?
Generalization for seen users and unseen pairs. Without loss of generality, let us assume that each (i)
answers preference comparisons. Let S; {x x.rt 2/ ) denote an i.idd sample of size m from user and let S - S, represent the dataset from N users. Given S, for any (M, Q [w(0)
(1)
N j=m). 1). we define its empirical risk RM, Q (w()) N )(m)) as
1
Σ (15)
e (v)5)
OOT PTTMMx
(OT Sxr Mx (x -x ( (Qw())).
Nm
Let (M, Q, ( ) minimize the empirical risk given D, and (M", Q", (( ( I==1 minimize the true risk, Es Es|RRs [M, Q { (w(i) [ Theorem 1 below provides an upper bound on the excess risk. All proofs are deferred Appendix C; see Section C.I for the proof of Theorem I.
Theorem 1. (Seen user generalization) Suppose K < (nin(N, D). and for each comparison, the user is asked to compare two items drawn i.i.d. from the uniform distribution on the unit sphere, Unif(SP-1) ).)). Then, with probability at least 1 6.
Es (M, Q (WO)) 1 Es (M Q", (w
23 +
KN S2 D
2 log Nm
12
log(N + D) +
Nm
To interpret this result in a practical setting, let us further assume that the entries of M and Q* bounded by some constant, and set (M D and Go VD, done in (Canal et al., 2022).
where Ô hides the parameters and L and Then, the above bound becomes δ ( [D*+KDAKN
ignores logarithmic terms. Observe first that the bound decays as either the number of users N or the number of samples per user increases. In addition, when N 2 2(D2), the bound simplifies to Ö(v K/ K/m),, which implies a per-user sample complexity of Ô (K). This contrasts with the existing result of Ō (D) without mixture modeling (Canal al., 2022, Theorem 3.1, see also Section C.1.3 in the Appendix). The result captures the intuition that if the users amortize the cost of learning the common M and Q, then each user only needs to individually learn their weights w(() E A K-1
Generalization for unseen users. So far, we have discussed the generalization ability of PAL to unseen pairs among known users, but how well does our model generalize to new users, whose data was not included in the training set all? In this work, we provide the first answer to this question in the context of preference alignment under the ideal point model, leveraging the framework and tools for multi-task learning and learning-to-Ee
(Maurer al., 2016) (Remark C.6 of the Appendix).
'The Mahalanobis distance defined by symmetric, positive semidefinite matrix M is given by dx v(x y) M(x y).
8

---

Published as a conference paper at ICLR 2025
Suppose there is distribution 7 over set of users U. Given any representation M and prototypes Q, along with an i.i.d. sample S { (Xx,,, Nj,;; }=) of m comparisons from some new user U. the natural approach is few-shot learn the weights that minimize the empirical loss:
W S.M.Q = argmin WEA
(us Mxx, Mx + (X - -x,,) (Qw))
and one evaluate the expected performance of (M, Q w. for user u.:
i.M, Q; uu == xx.xxx..)) [ (y (x) Mx| - x, Mx, + (xi 15-x0 x,) " (ws..))))
The unseen-user risk of (M. Q) is defined as:L user (M, Q)
E,, (M, Q: u)], where L(M, u: = Es i.M, Q; u, { Suppose that for training, N users are drawn according to
7. and each answers m comparison queries, resulting in the dataset, Si U..... USN. Let (M, Q) be components of the minimizer of the empirical risk given by Eq. (8). Theorem 2 provides upper bound on its excess risk when compared with
(M', "²) argmin Es min (( [ ( (x/ Mx x, Mx, + (xi M.Q
x.) x [Qw)]]
(WEAK-1
which assumes oracle knowledge of each user's preferences. See Section C.2 for its proof.
Theorem 2. (Unseen user generalization) Suppose K min(N, d). and for each comparison,
the user is asked to compare two items that are drawn ii.d. from the uniform distribution on the unit sphere, Unif(SD-1), Then, with probability at least 1 over $1, 31.-.... SN.
25 +
K2
KC Dm
8 log 1 N
L user (M, Q) L
user
(M', Q*) S <18T
+ +31
N
(VD
Again, let us set SM D and C. VD. The above bound becomes Ô
Intuitively, the first term captures how well the common mapping and the prototypes learned on seen users' dataset translate new unseen users. This term decays as the number of seen users N increases. The second term characterizes how well our few-shot preference localization for new unseen user generalizes to unseen pairs of this user. This term indicates a sample complexity of Ô (K) and suggests efficient generalization, especially since K can be quite small practice.
4.2 NUMERICAL SIMULATION
In this section, carefully and systematically examine how PAL adapts plurality of preferences via numerical simulations. To this end, we construct a synthetic, heterogeneous dataset similar to (Canal al., 2022), where each item true f is linear mapping from Ed Rd. Let K* denote the number of user prototypes and let {K.
NN, 21) and the user weight W (O, The
K denote the set of user prototypes, where each p NO _1). We assume that the distance between any pair of user prototypes is lower bounded by some value ó.
Experiment Setting. We study mixture setting, where each user is located in the convex hull of P. Let a, denote the 5th user. To learn this user's ideal point, we draw n pairs of items {x, x,} uniformly random and assign the user's preference as sign(() - ff faa))]ll
(x,))
(a)))))))) We generate datasets with different K*, K, latent dimension d, number of prototypes, and number of samples per user n. We evaluate PAL-A on these synthetic datasets.
Results. Figure 5 (a) shows that PAL can learn the user ideal points in the representation space. Figure (b) shows that the homogeneous reward model (K = 1) can only achieve sub-optimal per- formance when diverse preferences exist. Incorporating plurality via multiple learnable prototypes with PAL, we gain a significant 7% accuracy boost. Figure (c) shows that as we increase the num- ber of training samples for seen users, PAL achieves higher test accuracy, and is also more accurate in capturing the true number of prototypes in the dataset. Figure (d) presents PAL's 's potential to generalize to unseen users via few-shot learning to only learn their weights. We also studied

---

Published as a conference paper at ICLR 2025
Cammpln For Seen User
(Predittion)
0.30
0.00
0:03
0.00
1.0
6:93
0.5
6.0-
0.00
0.00
0.00
bycilypcan stact)
0:53
0.00
True Frtteypee
Dettont
0.00
Prodeggas in Model
Probiagsis in Model
Teature
[a]
(b)
(4)
(d)
Figure 5: (a) shows that PAL can accurately capture the user prototypes and each user's ideal point. We set d = 2 for visualization of the user's ideal point. (h) illustrates the performance of PAL on diverse preference
datasets. (c) empirically highlights the importance the number of samples per seen user and the number of
prototypes in PAL. (d) demonstrates the sample efficiency of the PAL framework on unseen users. With as few as 40 samples, we can still achieve performance comparable to that of seen users.
partition setting, where each user drawn from P uniformly at random. Figure D.1 in the Appendix D.2 shows the learned ideal points under partition setting have similar trends to the mixture setting.
To further validate our findings, we also examine PAL'S generalization properties via Anthropic personas (Perez et al., 2022), which we restrict to Appendix D.4 for brevity.
RELATED WORK
We provide brief summary of related works here, and a more detailed version in Appendix A. Pop- ular foundation models (Achiam et al., 2023; Ouyang et al., 2022; Touvron et al., 2023) typically use RLHF (Azar et al., 2024; Ethayarajh et al., 2024; Rafailov et al., 2024; Stiennon et al., 2020) to align models after pretraining. These methods assume homogeneity either explicitly or implicitly by using BTL- (Bradley & Terry, 1952). Consensus-based methods (Bakker et al., 2022) aims to find agreement among labelers for specific designed defined goals (Bai et al., 2022b;a; Irvine et al., 2023; Ganguli et al., 2022), which prioritize the universal preference (and biases) induced by the la- belers (Cheng al., 2023; Kovač et al., 2023; Santurkar al., 2023). Many works have highlighted that in reality, humans have diverse preferences (Nadal & Chatterjee, 2019; Sorensen et al., 2024; Wildavsky, 1987). However, approaches to suit this heterogeneity are still top-down in nature (Rame et al., 2024; Wang et al., 2024b; Wu et al., 2024), where the system designer makes learning deci- sions apriori, such as collecting datasets to train diverse rewards or multi-ohjective training. Li et al. (2024) propose personalized reward modeling with a cluster structure for users in the dataset, but model unseen users homogeneously. There is a rich literature on preference learning (Fürnkranz & Hüllermeier, 2010) and metric learning (Bellet et al., 2022). For the ideal point model, several works (Ding, 2016; Huber, 1976; Jamieson & Nowak, 2011; Massimino & Davenport, 2021; Singla et al., 2016) study sample complexity of ranking and localization when the distance is known, and some recent works (Canal et al., 2022; Wang et al., 2024e; Xu & Davenport, 2020) have studied simultaneous learning of the Mahalanobis distance which is equivalent learning common linear map along with unknown user preference point(s).
6 CONCLUSIONS
We propose PAL, novel framework for modeling personalizable rewards for pluralistic alignment (Section 2) which leverages shared structures across the population while learning to personalize in a sample-efficient way. We demonstrate that PAL is agnostic modality, showing strong results on both text (Section 3.1) and image (Section 3.2, 3.3) tasks. We also provide sample complexity bounds for generalization of the learned rewards to both seen and unseen users (Section 4.1). Our work aids in building much-needed foundations toward plurality for the alignment of MI/AI mod- els. Our experiments also highlight the limitations of many real human preference datasets that are collected with rubrics that make the dataset homogeneous, and call for more nuanced approach to data collection in the future (Section 3.2). While the mixture modeling approach of PAL is flexible, a limitation of using it in static setting is that it will not generalize to users who fall outside the convex hull of learned prototypes (Section 4.2). A more pragmatic and exciting approach would be continually learning prototypes to adapt to new users on the fly, which we leave for future work.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
PAL makes a strong case for open, democratic alignment by enabling training alignment mod- ules with cheap encoders on simple GPUs relatively short wall clock time (e.g. see Sec- tion 3.2, Results and Appendix E). To enable reproducibility and the practical utility of PAL for downstream tasks, we describe our algorithm in Appendix B, and give detailed experiment setups in Appendix D. We provide exhaustive hyper-parameter details for our experiments (Tables D.2 in Appendix D). We also attempt to provide insights over our design choices (which configu- rations worked and which didn't throughout the course of our experimentation) to aid others in using PAL for their own data and tasks. Lastly, our code and datasets are openly available at https: http:/giitttt com/ haamyaaabplluuallal. welluralistii iistiistiii sesssiicalli aer--algggmmen gnment.
ACKNOWLEDGMENTS
We thank Zaid Harchaoui for providing feedback on the manuscript. We would also like to thank the anonymous reviewers for their valuable feedback, which helped polish this paper. This work was also partially supported by NSF grants NCS-FO 2219903 and NSF CAREER Award CCF 2238876.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774,, 2023.
Robert J. Adler. An Introduction to Continuity, Extrema, and Related Topics for General Gaussian Processes, volume Lecture Motes-Monogruph Series. Institute of Mathematical Statistics, 1990.
Rie Kubota Ando, Tong Zhang, and Peter Bartlett. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of machine learning research, 6(11), 1), 2005.
AI Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08001 2022b.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy
Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-38188 2022.
Aurélien Bellet and Amaury Habrard. Robustness and generalization for metric learning. Neuro- computing, 151 267, 2015.
Aurélien Bellet, Amaury Habrard, and Mare Sebban. Metric learning. Synthesis lectures on artificial intelligence and machine learning. 91):1-151, 2015.
Aurélien Bellet, Amaury Habrard, and Mare Sebban. Metric learning. Springer Nature, 2022.

---

Published as a conference paper at ICLR 2025
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-34) 1952.
Mark Braverman and Elchanan Mossel. Noisy sorting without resampling. arXiv preprint arXiv:0707.106 1051, 2007.
Gregory Canal, Blake Mason, Ramya Korlakai Vinayak, and Robert Nowak. One for all: Simul- taneous metric and preference learning over multiple users. arXiv preprint arXiv:2207.03609
2022.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionnality qulti-granularity text embeddings through self-knowledge dis- tillation, 2024.
Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. arXiv preprint arXiv:2305. 18189, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Clyde H Coombs. Psychological scaling without a unit of measurement. Psychological review, 57 (3):145, 1950.
Cody Ding. Evaluating change behavioral preferences: Multidimensional scaling single-ideal point model. Measurement and Evaluation in Counseling and Development 49(1):77-88, 2016.
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 1409))2449--246,, 2021.
Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCan- dlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models, 2024.
Brian Eriksson. Learning top-k search using pairwise comparisons. In Artificial Intelligence and Statistics, pp. 273. PMLR, 2013.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. 2024.
Johannes Fürnkranz and Eyke Hüllermeier. Preference learning and ranking by pairwise compari- son. In Preference learning, pp. 65-82. Springer, 2010.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07888 2022.
Wassily Hoeffding, Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pp. 409-426, 1994.
Joel Huber. Ideal point models of preference. ACR North American Advances, 1976.
David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32 (1):384-406, 2004.
Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, et al. Rewarding chatbots for real-world engagement with millions of users. arXiv preprint arXiv:2303.0613 2023.
Kevin Jamieson and Robert Nowak. Active ranking using pairwise comparisons. Advances in neural information processing systems, 24, 2011.
12

---

Published as a conference paper at ICLR 2025
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai aligmment: comprehensive survey. arXiv preprint arXiv:2310.19833 2023.
Claire Kenyon-Mate Mathieu and Warren Schudy. How to rank with few errors. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pp. 95-103, 2007.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Oner Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Matthäus Kleindessner and Ulrike Luxburg. Uniqueness of ordinal embedding. In Conference on Learning Theory, pp. 40-67. PMLR, 2014.
Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre- Yves Oudeyer. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870. 2023.
Brian Kulis. Metric learning: A survey. Foundations and Trends@ Machine Learning, 5(4): 287-364, 2013.
Xinyu Li, Zachary C Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05133, 2024.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.032: 032881 2023.
Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, and Yu Qiao. Inference time language model alignment via integrated value guidance. arXiv preprint arXiv:2409.1788 2024.
R Duncan Luce. Individual choice behavior: A theoretical analysis. New York: Wiley, 1959.
Blake Mason, Lalit Jain, and Robert Nowak. Learning low-dimensional metrics. Advances in neural information processing systems, 30, 2017.
Andrew K Massimino and Mark A Davenport. As you like it: Localization via paired comparisons. Journal of Machine Learning Research, 22(186):1-39, 2021.
Andreas Maurer. Bounds for linear multi-task learning. The Journal of Machine Learning Research, 7:117-139, 2006.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen. Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. In Forty-first International Conference Machine Learning, 2024.
Marcos Nadal and Anjan Chatterjee. Neuroaesthetics and art's diversity and universality. Wiley Interdisciplinary Reviews: Cognitive Science, 10(3) 2019.
Sahand Negahban, Sewoong Oh, and Devavrat Shah. Iterative ranking from pair-wise comparisons. Advances in neural information processing systems, 25, 2012.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Stephen E Palmer and Karen Schloss. Human preference for individual colors. In Human Vision and Electronic Imaging XV, volume 7527, pp. 353-364. SPIE, 2010.
13

---

Published as a conference paper at ICLR 2025
Ethan Perez, Sam Ringer, Kamilè Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pet- tit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Isracl, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Lan- don Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Lar- son, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer EI Showk, Tamera Lanham, Timo- thy Telleen-Lawton, Tom Brown, Tom Henighan. Tristan Hume, Yuntao Bai, Zac Hatfield- -odds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez. Deep Gan- guli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations, 2022. URL https: / aawiv. org/ abs/2212
2212 0 09251.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Arun Rajkumar and Shivani Agarwal. A statistical convergence perspective of algorithms for rank aggregation from pairwise data. In International conference on machine learning, pp. 118-126. PMLR, 2014.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by in- terpolating weights fine-tuned diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
Aniket Rege, Aditya Kusupati, Alan Fan, Qingqing Cao, Sham Kakade, Prateek Jain, Ali Farhadi, et al. Adanns: A framework for adaptive semantic search. Advances in Neural Information Processing Systems, 36:76311-76333 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108 108, 2019.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 29971-30004. PMLR, 23-29 Jul 2023. URL https: / httpss:://cccceeeeeeeeeeo .mw..rr...........mm............ mlr. press/v2022 sttp//satuuurrrraa html.
Matthew Schultz and Thorsten Joachims.
Learning distance metric from relative comparisons. Advances in neural information processing systems, 16, 2003.
Nihar Shah, Sivaraman Balakrishnan, Aditya Guntuboyina, and Martin Wainwright. Stochastically transitive models for pairwise comparisons: Statistical and computational issues. In International Conference on Machine Learning, pp. 11-20. PMLR, 2016.
Nihar B Shah and Martin J Wainwright. Simple, robust and optimal ranking from pairwise compar- isons. The Journal of Machine Learning Research, 18(1):7246-7283 2017.
Shai Shalev-Shwartz, and Shai Ben-David. Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
Roger N Shepard. The analysis of proximities: multidimensional scaling with an unknown distance function. Psychometrika, 17(2):255144) 1962a.
Roger N Shepard. The analysis of proximities: Multidimensional scaling with an unknown distance function. ii Psychometrika, 27(3):11-2246 1962b.
14

---

Published as a conference paper at ICLR 2025
Roger N Shepard. Metric structures in ordinal data. Journal of Mathematical Psychology, 3(2): 287-315, 1966.
Adish Singla, Sebastian Tschiatschek, and Andreas Krause. Actively learning hemimetrics with applications eliciting user preferences. In International Conference on Machine Leurning, pp. 412-420. PMLR, 2016.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint arXiv: 2024.
Nisan Stiennon, Long Ouyang, Jeffiee Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. Adaptively learning the crowd kernel. arXiv preprint arXiv: / 105. 1033, 2011.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 0.09888, 2023.
Joel A Tropp et al. An introduction matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1-2):1-230, 2015.
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 2023.
Martin J Wainwright. High--imenssooall statistics: A non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rihf in large language models part ii: Reward modeling. arXiv preprint arXiv:2441.06600 000080, 2024a.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402. 18571, 2024b.
Zhi Wang, Geelon So, and Ramya Korlakai Vinayak. Metric learning from limited pairwise prefer- ence comparisons. In UAI, 2024c.
Aaron Wildavsky. Choosing preferences by constructing institutions: cultural theory of preference formation. American political science review, 81(1):3-21 1987.
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for I1m alignment. arXiv preprint wwXi::2310.00212. 2023a.
Xiaoshi WI, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306.098:
2023b.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu. Noah A Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine-graines human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
Austin Xu and Mark Davenport. Simultaneous preference and metric learning from paired compar- isons. Advances in Neural Information Processing Systems, 33:454-45 2020.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
16

---

Published as a conference paper at ICLR 2025
OUTLINE OF THE APPENDICES
A Extended Related Work
18
Detailed Model Overview
20
B.1 Illustrations and Pseudocode for the PAL Framework
20
B.2 Modeling Choices: When to Use PAL-A or PAL-B Practice?
Proofs and Additional Theoretical Results
22
C.I Generalization for Seen Users and Unseen Pairs
22
C.2 Generalization for Unseen Users
28
C.3 Auxiliary Lemmas
34
Experiment Details
36
D.1 General Procedure
36
D.2 Numerical Simulation
36
D.3 Reddit TL;DR Summary (Text-to-Tex)
37
D.4 Persona (Text-to-Text)
D.5 Pick-a-Pic (Text-to-lmage)
42
D.6 Pick-a-Filter (Text-to-lmage)
43
Computational Resources
45
17

---

Published as a conference paper at ICLR 2025
A EXTENDED RELATED WORK
Aligument Status Quo. Popular existing foundation models (Achiam et al., 2023; Anthropic, 2024; Ouyang et al., 2022; Touvron et al., 2023) typically use RLHF (Christiano et al., 2017; Ethayarajh et al., 2024; Stiennon et al., 2020; Wu et al., 2023a) to align models after pretraining. Recent foundation models such as Zephyr (Tunstall et al., 2023) and the Archangel suite' have shifted to directly optimizing on human preferences (Azar et al., 2024; Ethayarajh et al., 2024; Rafailov et al., 2024) to avoid the nuances of RL optimization (Dulac- Arnol et al., 2021). There has also been significant recent work in collecting large human preference datasets for reward model training in the text-to-image space (Kirstain et al., 2024; Wu et al., 2023b; Xu et al., 2024) (typically diffusion models (Rombach et al., 2022)).
Reward Modeling, These existing alignment frameworks generally assume that all humans share single unified preference (e.g. LLM "helpfulness" "harmlessness" (Bai et al., 2022a)) and cribe to the Bradley-Tery (Bradley& & Terry, 1952) model of pairwise preferences. Consensus-based methods (Bakker al., 2022) aim to find agreement among labelers for specific goals like harmless- ness (Bai et al., 2022b; Ganguli et al., 2022), helpfulness (Bai et al.. 2022a), or engagement (Irvine et al.. 2023). By design. these methods inherently prioritize the universal preference (and biases) induced by the labelers (Cheng et al., 2023; Kovač et al., 2023; Santurkar et al., 2023). In reality, humans have diverse, heterogeneous preferences (Nadal & Chatterjee, 2019; Sorensen et al., 2024; Wildavsky, 1987) that depend on individual contexts, and may even share group structure (Bakker et al., 2022). Rewarded soups (Rame et al., 2024) make a case to capture diversity through post-hoc weight-space interpolation over a mixture of experts that learn diverse rewards. However, these re- wards are learned by pre-defining what aspects are important which is done by the system designer. Separate datasets are collected to elicit human preferences care of them. DPA (Wang et al., 2024b) models rewards as directions instead of scalars, and trains multi-objective reward model for RLHF, Wu et al. (2024) propose fine-g fineigrieed multi-objective rewards to provide more focused signal for RLHF, Recently, Li et al. (2024) propose personalized reward modeling by learning general user embedding and treating each individual a perturbation to the embedding. As this preference formulation is still homogeneous for users not in the dataset, they use the fixed general user embedding for generalizing to unseen users, i.e., do not personalize to new users.
these axes as to how much people
Recent survey works provide excellent summaries of literature for alignment (Ji et al., 2023) and reward modeling (Wang et al., 2024a).
Human Preference Datasets. The preference universality assumption also extends into the data amnotation/labeling processing, where labelers are given a rubric to select preferences (e.g. to rank an image pair considering image aesthetics and image-prompt alignment (Kirstain et al., 2024)). Due to this rubric, the current largest scale text-to-image generation preference datasets (Kirstain et al., 2024; Wu et al., 2023b; Xu et al., 2024) show limited diversity among labelers. In the Pick-a- Pic train set (Kirstain et al., 2024), there only 701 disagreements among the 12487 image pairs labeled by different users (94.38% agreement), and there are zero disagreements in validation (1261 pairs) and test (1453 pairs) sets. HPS (Wu al., 2023b) found that labeler agreement over diffusion model generations was higher for models of similar quality or size, though this diversity comes with the caveat of the labelers being provided a rubric to provide their preferences. Imagereward (Xu et al., 2024) use researcher agreement as a criteria to hire labelers. In the LLM domain, the popular Summarize from Feedback dataset (Stiennon al., 2020) is also collected with rigid rubric, with labeler performance measured via agreement to the preferred answer of the authors. During the data collection period, only labelers with satisfactory agreement were retained, which led to a small number of users, all in agreement with the authors' rubric, being responsible for a majority of labeled comparisons. Status quo preference datasets used to align foundation models thus suffer from a lack of diversity due to the nature of their data collection.
Preference learning. There rich literature on preference learning and ranking in various do- mains ranging from psychology, marketing, recommendation systems, quantifying social science surveys to crowdsourced democracy, voting theory and social choice theory. We provide a few relevant works here and direct reader to surveys such as (Fürkrranz& & Hillermeier, 2010). Rank- ing based models, e.g., BTL--model (Bradley & Terry, 1952; Luce, 1959), stochastic transitivity
hhttps://gi
18

---

Published as a conference paper at ICLR 2025
Generated Content
E R
Users*
Foundation Model
R d
Preference Embedding Space
R
clasesti
closeati +
f
RD Ra
X
Prefer
Prefer
Rd
Preference Embedding Space
User ideal points represented learnable vectors (to learn the user's preference);
distance calculation method can be adapted, with options such Euclidean distance or cosine similarity.
c: Learnable Module; (0:: Frozen Module
Figure B.I: In the ideal point model considered in PAL-A, items and users (represented by their ideal points) are mapped to Rdd via an unknown function f. user's preference for an item (e.g., an image or text summary) is inversely related to the distance between the item and the user's ideal point. For example, the yellow user prefers the green text summary, and the purple user prefers the orange text summary, as these items are "closer" to their respective ideal points.
models (Shah al., 2016) focus on finding ranking of items or finding top-k items by pairwise comparisons (Hunter, 2004; Kenyon-Mathiuu & Schudy, 2007: Braverman & Mossel, 2007; Negah- ban et al., 2012; Eriksson, 2013; Rajkumar & Agarwal, 2014; Shah Wainwright, 2017). Ranking m items in these settings requires Omm log m.) queries. There is also rich literature that stems from ideal point model (Coombs, 1950; Huber, 1976; Jamieson & Nowak, 201 1; Ding, 2016; Singla et al., 2016; Xu & Davenport, 2020; Canal et al., 2022). Under the ideal point based models, the query complexity for ranking m items reduces to Od log m). where d is the dimension of the domain of representations which usually much smaller than the number of items being ranked (Jamieson & Nowak, 2011). This is due to the fact that once the preference point is learned, it can then be used predict rankings of items without needing more comparisons.
Metric learning has been studied quite extensively and we direct the reader to surveys (Kulis, 2013) and books (Bellet et al., 2022). In particular, metric learning based on triplet querying has also been quite extesively studied (Shepard, 1962a;b; 1966; Schultz & Joachims, 2003; Kulis, 2013; Tamuz al., 2011; Kleindessner Luxburg, 2014; Bellet al., 2015; Bellet & Habrard, 2015; Mason al., 2017) which aims to learn the underlying unknown metric under the assumption that the people base their judgement for triple query with concepts X Xs, X D on the relative similarities based on the distances between these concepts under the unknown metric.
Simultaneous metric and preference learning. More recently a few works have considered the problem of unknown metric in preference learning and proposed methods (Xu & Davenport, 2020; Canal et al., 2022; Wang et al., 2024c) and provided sample complexity analysis (Canal al., 2022; Wang et al., 2024c) for simultaneously learning unknown Mahalanobis metric and unknown user preference(s). Learning the unknown Mahalanobis metric can be viewed as learning linear layer on top of the embeddings from foundation model. From reframing of alignment, these works be looked as P PAL-A - with linear function for f and individual user preferences instead of having any structure them.
19

---

Published as a conference paper at ICLR 2025
PAL-A
User Ideal Point Computation
Reward Computation tinnt egg. A/B
P
Preference Modeling for User i
PAL-B
Pr(xi > x, X,, i)
User Ideal Point Computation
X
f
Reward Computation
X
d=
f (wx)
G
f
(a,
gh
Figure B.2: Illustration of PAL framework for learning from diverse preferences (Section 2). For any user i, the probability of preferring x to X,, for the context x, is computed by a reward model x,)
ro which uses a mixture modeling approach to assign scalar reward to sample (e.g. given context (x,). In PAL-A, each user i'ss preference (*)) is modeled
(t)
convex combination of K prototypical preferences, i.e. (6) (w( In PAL-B, each user i's preference 2 2x.) is modeled as convex combination of K prototypical functions 9I gk. i.e. (x.) Reward functions formulated using the PAL framework can be used flexibly, e.g. with fixed preference points (PAL-A), with preference points that are functions of the context/prompt (PAL-B).
B DETAILED MODEL. OVERVIEW
B. ILLUSTRATIONS AND PSEUDOCODE FOR THE PAL FRAMEWORK
Figure B.1 illustrates the ideal point model considered in PAL-A. A. We show the modeling mechanism of PAL (Section 2) in slightly more detail in Figure B.2. Algorithm and 2 provide the pseudocode for the learning algorithms of PAL-A and PAL-B.
Algorithm Learning algorithm for PAL-A
Require: Dataset
U { (x j.1 x (1)
j.r: het v ) loss function f, model class for fo-
prototypes P [p....., pkl where pa RD, user weights w = [w(1),.... ..., w:(N) where w(i) E A
1: for each iteration do
2: Sample mini batch {(x
j.l si Jee (0))
x (6) (1)
b random pairs, not ordered by users
3: User Ideal Points: a (1) P (())
4: Distances:
5:
llfe x - fo (aa 113 (3) (6)
(0) = -Iff
x (x
fo(a
6: Loss: w
(x
j ( // (1)) = Ely,
(4 (0)
7: Update Step: argming, P. (W
Eis 10
8: end for
20

---

Published as a conference paper at ICLR 2025
Algorithm 2 Learning algorithm for PAL-B
Require: Dataset D
U (xx)) j. jrr hett 8/j j==1" loss function E, mapping function fe,
(t) (t) (0) mh
prototype mapping functions {ge, 1K user weights (wl0 == [wo (6) ...,, w 1 Jh
1: for each iteration do
2: Sample mini batch {(x (6) (0)
Xir di jic yj ")
b random pairs, not ordered by users
3: User Ideal Point (conditioned on prompts):
4: a
go.
90
(x c,j
-w()
5: Distance:
6:
()) Update Step: argming, (W Σ 1/ (x
(fo
(0)
1 = (fe
x5
(8)
7: Loss:
o (x(
fyy
(0)
8:
9:
end for
B.2 MODELING CHOICES: WHEN TO USE PAL-A OR PAL-B IN PRACTICE?
We note that PAL-B more natural in generative modeling settings, as learns personalized mapping (x,)) for each user and any given prompt X2, and it learns separate mapping for outputs x. In contrast, PAL-A learns ideal point ( for each user i fixed across all prompts and it learns to jointly map the prompt x, and output x in the same space.
From experiments, found that PAL-B serves reliable default choice (see Figure 2, Figure and Table on Reddit TL;DR Summary; Table 2 on Pick-a-Pic; and Figure 4 on Pick-a-Filter). We able to get PAL work competitively to PAL-B in most settings (see Table D. Table D.3 and Section D.4 in the Appendix); however, in practice, this may require additional engineering effort optimize effectively.
21

---

Published as a conference paper at ICLR 2025
C PROOFS AND ADDITIONAL THEORETICAL RESULTS
Additional notation. Given two matrices A and B, we use (A, B) to denote the Frobenius inner product between them, i.c., (A, B) tr(A B). As example, recall that, given the reparame- terization Section 4.1, the difference of scores for a user can be written as x, Mx - x, Mx, + (x: - x,)) Qw; what follows, for better clarity, we express it an equivalent form using the Frobenius inner product, (xxx] x, M) (Xi Xr: Qw)
C.I GENERALIZATION FOR SEEN USERS AND UNSEEN PAIRS
C.1.1 PROOF OF THEOREM
Theorem 1. (Seen user generalization) Suppose K min(N, D), and for each comparison, the user is asked to compare two items drawn i..dd from the uniform distribution on the unit sphere, Unif(SD-)) Then, with probability at least 1 - 8.
Es RssRs(M. Q [wo)) 1=1)) Rss (M',
(M
S3 + D
KN
+K) S2
2 log 2
12L
log( + D) + V
Nm
Nm
Proof of Theorem 1. We standard Rademacher complexity theory; see, e.g., (Shalev-Shwartz & Ben-David, 2014; Mohri et al., 8) for general background and (Canal et al., 2022) for its application in preference and metric learning without mixture modeling.
Recall that : IR + [0. 1] is an L-Lipschitz loss function. Let W (w())..., w(]) E RKXN denote the matrix containing the individual weights of all users.
Consider the following class of functions parameterized by M, Q and W:
H /MM...w :innew:iixxam (i,x1, v) + (y ((xxx] -x,x, M) (xI Xx, Qw))))
[M S Cas, || S So Vk E [K], W, W Vie [M]
where Qs denotes the k-th column of Q, i.e., the k-th prototypical ideal point, and W, denotes the i-th column of W, i.e., the weights for user i.
To prove Theorem 1, we use Lemma C.I, which relates the excess risk to the Rademacher complex- ity of H. In particular, the lemma addresses settings with multiple distributions, where data consists of i.idd. samples from each distribution. Similar notions of task-averaged Rademacher complexity have been considered in (e.g., Ando al., 2005; Maurer, 2006; Maurer et al., 2016).
Here, to apply Lemma C.1, we slightly abuse notation by considering S as a dataset where, for (1) each i E [N] and j [m], the comparison (x j.r , is augmented with the user index i: (1) (1)
(i,x) jrr U5 (6)
Let o (oI,..., .NN E +11,+13 be an array of independent Rademacher random variables. Then, by Lemma C. with probability at least 6.
log Nm
Es|Rs Q (w IN Es(Rs(M' Q" (w((( ( 1 & (+) V
where
(*)
2ES,a
sup M.Q.W Nm
ΣΣ 1=1j=1
m
asg (8)
(xx90x0)
(T)
x fr ..0,MM) + (x x0 QW. Qw:)))
It suffices to show that
G3 +
KN D
+ K) c
(+) 122
log(N (N + D).
Nm
22

---

Published as a conference paper at ICLR 2025
To this end, observe that
(a)
2L
Bssar
sup M.Q.W W Nm
N
(155)2207 - x 3, 3u
1 (
M) + xxx000 - x Qw:))
Nm
i=1 1=1
N
(b)
2L
Es.a
sup
33 M)
Nm
M
1=1
j=1
(i)
2L Nm
Es.o
sup Q.W
m
O.
{x)) 3
(0)
(
Qw.)
(ii
(10)
where (a) applies Talagrand's contraction lemma (Mohri et al., 2018, Lemma 5.7) and uses the observation that, for any and j. Pr (0t,5,,) ( =1) Pr (r
2. and (b) follows because
for C' = {a b: E A, E B}, sup C sup A sup B. We tackle the two terms separately. For (i), by the Cauchy-Schward inequality, we have
N
(i) S 2L
Es,
sup M
||
Σ
O.
(i) (0T
- x
(T)
Nm
j=1
(a) 2L
Cm
Es
Σ
(x,1
(T)
(T)
3.r j.r
(1) (T)
Nm
/=l
1=1
(b)
4L
SM
NN
Nm
S2
Nm
where (a) uses Jensen's incquality; (b) follows from Lemma C.11 and the observation that for any (8) (a)
realization of S, |xx x2,
xj
(8) (QT
I 2 for [N] and j E [ma] as the items are drawn
from 1d-1
We now bound term (ii). For each E [N] and j [m], let x, := [o
(x (1)
x o
DdxN have zeros everywhere except for the i-th column. Then,
2L
N
m
Qw)
(ii) = Nm
EsSo
sup
Oi
Q W
(a)
2L
m
sup
|QW/|-
ESS.
Σ
o x(1)
Nm
Q.W
2L
Nm
Nm N
KN
2
log(N + D) + log(N D)
Nm
D
KN D
+ K)C3 c
K 50
S 4L
log(N + D) 2L
log N +D)
Nm
Nm
2
KN D
+ K) c
67.
log(N N D),
Nm
23

---

Published as a conference paper at ICLR 2025
where (a) uses Hölder's inequality; (b) follows from the observation that QQWI. (rank(QW) |Q & v k E I|Qw./ and Lemma C.2; (c) uses the facts that va + vb z 2 and 2 log(N (N + D) for N, D 2 1, along with some algebraic simplifications.
Combining the bounds on (i) and (ii) and by Eq. (10), we have
4
18
KN + +K)
2L 2L
+
log(NV + +D)
Nm
Nm
(a)
SM
(KN
K)
S log(N + D),
12L
Nm
where (a) the facts that va + vb
2( and 3log" (N D) 2 1 for N, 1. This
completes the proof.
C.1.2 KEY LEMMAS
We now present the key lemmas used in the proof of Theorem 1.
Lemma C.1 (See also Ando et al. (2005); Maurer (2006); Maurer et al. (2016)). Let
H c {h: : [N] 2 + [0, 1]]. and PNN be probability measures 2. For each [N],
let S: = {11, ((,5))))3-- where 21,1,...; Zim are i.i.d. data points drawn according to ps. and
1 Em E 21.) minimize the empirical risk, and let = U S. Let h E argminneN Nm
h" E argmingen N Σ E2,,,>p [h 11 minimize the true risk.
Then, with probability at least over S,
N B impi
N
2 log
E Ix [1, (1,)) S 2Rx (H) +
Nm
1=1
Here,
N m
RN (H) = Es.a
sup NEW
or, 23.5))
Nm
i=1
denotes the user-averaged Rademacher complexity H, where o = (ois (oL))vE(N)JEEM) is an array of independent Rademacher random variables.
Proof. We standard techniques for deriving generalization bounds using Rademacher com- plexity (e.g., Shalev-Shwartz & Ben-David, 2014; Mohri et al., 2018). For any E H. let (s(h) = 1 E ΣØ= h(i, 21.)) and L(h) == Ei= Ex,rop, [h(i, 2.1)]. Notice that Nm
C(h) Es [[s(M)].
Since 9 minimizes Es(h), we have
[(h) L(h") C(h) - [s(h) + Es(h) - -Zs(h") +Zsh)")(**)))) L(h²)
50
(c(h) (ssh) + (Rs(h)) L((h))
We first look at the second summand. Notice that h" is independent of S. Since h(i,2) [0. for any H. E [N], and z E Z, by Hoeffding's ' inequality (Hoeffding, 1994), with probability
2
log 2/4
-s(h)) - c(h))
<
2Nm
24

---

Published as a conference paper at ICLR 2025
Then, by the union bound and little algebra, it suffices to show that, with probability at least 1
log 2 2.m
((s))-Zs(h)) S N. (H) +
sup LEH
To this end, let x(S) SuPдcи (((h)-[((h)) It is easy verify that, since h(i, 2) [0, 1] for any E H. i E [N] and & E Z. x(S) satisfies the bounded difference property, i... for any S' that differs from S by exactly one data point, ix(S) x(S') S Now We can therefore apply the bounded difference inequality (e.g., Mohri et al., 2018, Theorem D.8) on k(S). With probability at least 1 8 2
In 3
<(S) S IEs [M(S)] +
2.
It only remains to show that Es [k(S)] < 2R (H), which follows from a standard symmetriza-
tion argument (see, e.g., Mohri et al., 2018, Chapter 3). For each i E [N]. let S = {( x )mm where { is another i.i.d. sample of size drawn according to Pt. and let S' U S Then,
Es [s (h)) - [s((h)]
Es [M(S)] = Es
sup LEE
N
101, 2(.1)) ]
< Es,s'
sup hEH
(h(i,
Nm
im
N m ΣΣ
oi (h(i, 21.1) - h(i,, 1(th
sup hCH
Nm
i=1=1
N m Σ
m
< Ess.
sup bER
and h(, 2ig)
+ E.si
sup hER Nm
oi, hig =1.s)
Nm
i=1==1
j=1
= N, (H)).
Lemma C.2. For E [N] and j E [m], let Zig = oin Di.j. where di,j is an independent
Rademacher ramdom variable, and Di.j 10 [0... ... Eigj ... 0f RDN has Eig = Zs - 4/. RDD in its i-th column and zeros elsewhere, with Zi,j and 2h,3 drawn independently from USD-1) Then,
N
Nm
Nm N
E
Zig
2
+
log(N + +D) log(N + +D)
D
i=
where Il:112 denotes the spectral norm.
Proof. For any [N] and j E (m)) observe that E. [z.,] 0 and |Z. ||Enمll S 2. The Lemma then follows straightforwardly from the Matrix Bernstein inequality (e.g., Tropp et al., 2015. Theorem 1.6.2, reproduced as Lemma C.10) and Lemma C.3.
Lemma C.3. Under the same definitions as in Lemma C.2,
m
N
m
Nm Nm
max
ΣΣ
Zi.
Σ
52
D
==1
j=1
Proof. We first show that
N ΣΣ
m
2Nm
Nm
Nm N
E
Z
[.
<
<
D
D
2
25

---

Published as a conference paper at ICLR 2025
This follows from the simple observation that, for any [N]
and [m].
21
E
Zi.
E Lo E. E
D
where (a) holds because for any x' drawn independently from Un
21
E
[xx
E
D
now suffices to show that
N m
2Nm
Nm
Nm
ΣΣ
E
2 Zio]
S
2
2
Observe that, for any E [N] and j [m]] E [Z,,, [,,] is matrix in RX with zeros everywhere but the (i, i)-th entry, which has value
E Blo E. tr (B (.,
=.
Therefore, we have A Eド=
E IZ Zn. 2mLI, and it follows that its largest singular value
2m
2Nm
C.1.3 EXISTING RESULT WITHOUT MIXTURE MODELING (CANAL ET AL.,, 2022)
For completeness. we adapt (Canal et al., 2022, Theorem 3.1) to our specific setting, and pro- vide a proof. Recall that, for the class of linear transformations, F, considered in Section 4.1, the difference of scores for a pair of items (x1, x,) and a user with ideal point a can be written as (xIX] - x, A) + [x: Xr. 2A Aa).
Let M = A, ()) = ()) and V == (()) v(N)], We consider M E [M E DXD, ||||r S SM, M 2 0} and V E EVERDx. [V.ll2 I Go, vi E [N]]. as done in (Canal et al., 2022). Again, let S = {x (8) jom (i) }} denote set of m i.i.d. comparisons
from user i E [N], and let S = U S.. Then, the empirical risk of (M, V) given S is:
N
(s Nm
(1)
(3)
(0) (TT M) X
+ (0)
fo v.)).
where we again assume l: - [0, 1] T-psiiicit Let (M, V) argmin v Js(M, V), and let (M*, V*) E argmin Es [J(M,
Proposition C.4 (Canal et al. 2022, Theorem 3.1). Suppose that for each comparison, the user is asked to compare two items drawn i..dd from Uni Then, with probability at least
Es V). Ess [Js(M: vn) < 199
S2 + C3N
22og
Nm
Nm
Remark C.5. Setting (M = dl and Co Vd, the above excess risk bound becomes δ
DR Nm
where Ô hides the parameters L and 8. When N 2 R(D2), the bound becomes δ suggests a per-user sample complexity of Ö (D).
P
which
Proof. See (Canal et al., 2022, Section D.1) for the original proof in a slightly different setting. We include proof here for the reader's convenience.
26

---

Published as a conference paper at ICLR 2025
Consider class of functions parameterized by M and V:
x,, y) + (y ((xI x, M) + (xi - xr, V.))) g = 9M. (1,,,,x,,1))
|M S Sar, I| S So Vi [M]]
where V, denotes the i-th column of V, i.e., the reparameterizze ideal point for user i. We again apply Lemma C.1%, Let a (ot...... .N) be an array of independent Rademacher random variables. By Lemma C.I, we have, with probability least 1 1 6,
Es [M Vi]
Es [Js(M v.) & (+) +
log Nm
where
(+) = ==2s=
sup M.V
Rigl (1)) (0)
10 OT M) (x - 31 3o
(0T
X V. v=)))
Nm
suffices to show that (+) S 6L
SRNN Nm
To this end, we first introduce some definitions and notations. For i E [N] and j E [m]. let
(OT z = j.
( for and x
Note that Z R DxD
t-th column
and xj E ERDx DxN Then, let
E IR Dx(D+N) be the concatenation of z
and x ( Since items are all drawn from Uni 1) we have
i
1OT 1OT
(0 (1) xxx Jir
1| x
(0) x f.
S 2v2.
(11)
In addition, let T = M VJ E PX((DNN) be the concatenation of M and V. Observe that
IMP +
V 1 S V Ch C.N.
(12)
We are now ready to bound (*). We have
2
m
a 0 (=)) .I)
(+)
Es.
sup M.V
Nm
(a) 2L
N
m
Es.r
sup
Ri
| T)
Nm
M.V
1=1
(b) 2L sup
m
|
Es
EN
o
(
Nm M,V
i=1
==
SM C2N
V
Nm
where (a) uses Talagrand's 's contraction lemma (Mohri et al., 2018, Lemma 5.7) and the observation that Pr (o.J4) (1) - -1 Pr (ot.j 1) = 1/2 for any and j: (b) uses the Cauchy-Schwand inequality
and Jensen's 's inequality: and (c) uses Eq. (12) and Lemma C.11 along with Eq. (11). This completes
the proof.
Again, for each E [N] and [m], we augment the comparison (x x j.r: (0)) in S with the user (4)
index (1, x. (0)
27

---

Published as a conference paper at ICLR 2025
C.2 GENERALIZATION FOR UNSEEN USERS
Theorem 2. (Unseen user generalization) Suppose K < min(N, d), and for each comparison, the user asked compare two items that are drawn i..dd. from the uniform distribution on the unit sphere, Unil(SD-1) Then, with probability at least - over Si....y SN.
Sh C² + 3L
KG
8 log 4
(M, Q) L (M', (T) 118LV
+
3
Dm
Remark C.6. Maurer et al. (2016) consider multi-task learning and learning-to-leark, modeling each task composite Sunction h2 e f. where is a shared representation and h; is task-specifec. They consider two settings: (1) multi-task learning with a given set of tasks, and (2) learning-to- learn where tasks are drawn from a distribution. Setting (1) analogous to seen-user generalization, as studied in (Canal et al., 2022) for simulatenous metric and preference learning. To derive our generalization bound for unseen users in preference alignment (Theorem 2), we build upon the framework and employ the proof techniques from setting (2) in (Maurer et al., 2016).
note that, in our mixture modeling for learning rewards, different users/usks share not only common representation, but also a set of prototypical ideal points. In addition, to the best of our knowledge, one cannot directly apply the results in (Maurer et al., 2016) to achieve the bound we present Theorem 2, particularly the second term which implies Ô (K) sample complexity for few-shot localization of preferences.
Proof of Theorem 2. We closely follow the proof strategy of (Maurer et al., 2016, Section 4.3), and tailor it to our specific learning framework and function classes to achieve tighter results (c.g., Lemma C.7 and Lemma C.9).
Additional notations. To facilitate our proof, we first introduce some additional notations. For any user u E Ut let Pu denote the data-gened tuta-generating distribution for user u. (To be more precise, we assume that items for all users and comparison are drawn independently from common distribution, specifically Dn-1) but the conditional distribution of the user preference answer y given a pair of items is user-dependent.) Given any (M, Q, w) and (x1, X,), denote the difference of scores by
(xx; M, Q w) := (x - x, M) + (x, -xx, Qw)
Note that, when the context is clear, we sometimes slightly abuse notation and refer to (xz, x,) x; for example, we may denote a sample of m comparisons from some user by {(x,, y5)) where Xj = (Xj.1, xj.). Given any representation M, prototypes Q, and a sample of m comparisons from some user, S = [(x,,;;)} y recall that Ws.MQQ E argmin 1 E/= l (Vj (xx;: Q w)); m
we also let
m
P(S Q)
min WEA
(g)
w)).
m
We now prove Theorem 2. Given Si,..., SN. consider the following decomposition of the excess
risk:
unseen user
(M, Q)
unseen useT
(M*, (M*)"")
N
N N Σ (S: M. Q N
x
L
unseen user
M. Q N
p(S:: M, 2)
M' Q*)
(ii). 50
N
p(S,; M Q*) En
Esupp (pSS Q')]]
N
(ii))
+ Ey ESSpp (p(S:M,, (T)]]]]]] -
unssen
user
[M,Q")
(iv), 5o
(13)
28

---

Published as a conference paper at ICLR 2025
Let us look at the four terms separately:
Observe that, for any M, Q,
N
unseen USET
(,, Σ pls: Q)
N
En ESSppr [E(x (y xxx; M, Q - P(S:M Q)]
(1)
Eu ES [p(S; Q)]
N Σ (S.
Lemma C.7 and Lemma C.9 establish uniform bounds on (1) and (2), respectively, It follows that, with probability least 1
Insccn
N M,,
Q
Σ
(S, M, S 18L
S + K2O
+3LV
KC
+
log
N
N
Dmn
2N
(14)
(ii) This term non-positive by the definition of (M, Q), which are components of the minimizer of the empirical risk for the seen users.
iiii Since (M', Q") does not depend on SI....,Sy and p(S; M. Q) E [0.1] for any E [N] and M, Q, we can apply Hoeffding's 's inequality (Hoeffding, 1994). have, with probability
at least 1
1 N
Σ
(S, Q")) E,, [ESPP [p(S: Q*)]] S
log 2N
(15)
(iv) This term is non-positive:
For any u E U4 let w E argmin () (y MM, ,",, w))] Then,
En
E Es (p(S: Q")]
1
E,
Es-pm
min
Σ m
E Oxx; Mr "0,, Q(,,)) w))
< Ex,
Es~pm
(yj +(x;; Q",
Q" w.)) (w))
Ey
[xx))x.)))PP [e (y ((:; M Q", w...., = unseen
(M", Q*).
The proof is completed by combining Eq. (13), Eq. (14), and Eq. (15) and applying the union bound.
Lemma C.7. With probability at least 2'
Sh + K2
log 4 2N
Σ M, S 1188 sup E, [S~PT (p(S; M, Q]] N M.Q
+31
A
N
Proof. To avoid cluttering the notation, we denote Eys [S m [-] by E., H.] Let (41 ((I....;6n) be array of independent standard normal random variables. Then, by (Mohri al.,
29

---

Published as a conference paper at ICLR 2025
2018, Theorem 3.3) and (Wainwright, 2019, Excercise 5.5), with probability at least 1
6
N
sup E, (M M.Q
Σ
(( Q)
N
N
log 8 2N
S V E
sup
6 P M. Q - -31
(16)
M.Q
N
i=
where E, supm.q N Di= t (S,; M, Q)
is the empirical Gaussian complexity of {S - M, Q) M, Q} given .I,..., SN, which we bound in the following.
Let N and E Nxmx be arrays of independent standard normal random variables, where E, y, and o are mutually independent. We have
N
V
E
sup
1
Σ
V2L
N
M,Q
i=1
(8)
By
sup
N
10T
jr
M
+
N
K
Figh (Q
(0 [x]1 - x
M.Q
N
N
K
By
sup
Yo
x31
(T)
x x
GT M)
+ +E
sup
Jisk (QT (x5)) - x
Q
(b)
N
m
S 2/ CZ Nm + BB
sup
0ish (Q
(x) [x)]
x
j=11
(0)
3 2/ Nm + 2.2
+2VK33 KC3- NNmK
(d)
S 2 2/2 2 + -K33) mN,
(17)
where
(a) follows from Lemma C.8;
(b) uses the Cauchy-Schwand inequality, Jensen's inequality, and Lemma C.11:
E
sup
31 3.
(0)
(0)
(T
M)
M
sup
|M]|
E
x
(2)
x
(6)
(i)
M
2CMM V Nmm;
30

---

Published as a conference paper at ICLR 2025
(c) again uses the
inequality and Jensen's 's inequality:
N
K
Eg
sup
Jisk
Q
x 31
x
1=
N
m
(1)
Eo
sup
Q
Jiyk
x
x
3ir)
i=1
j=1
K
N m Σ
S E
sup Q
Qu 2|
Σ A=1
0 Lisk
(x
:=1
j=1
K
N
m
I
VKK
Σ
E.
Jijk
x J.l
A=
and for any & E [K]. by the independence of the elements of or
E
ΣΣ !=1==1
N
Jisk
(x
j.r
N
m
1jk
x
(x
x
s
4Nm;
(d) follows from the simple fact that vā + vb - (2
Combining Eq. (16) with Eq. (17) completes the proof.
We now present Lemma C.8 used in the proof of Lemma C.7.
Lemma C.8. Given Si,..., SN.
N Σ
Ex
sup M.Q
p(S; M, Q)
N
V2L
N Σ ==
K ΣΣΣ Ossh
N m
VmN
Eyyo
sup M.Q
Ths
(T 51
2. for
(T M)
(QT
(1) x
02
j=
Proof. We apply the technique used in (Maurer al., 2016), which utilizes the Sudakov-Fernque inequality (e.g.. Adler, 1990, reproduced as Lemma C.12). To this end, we define two Gaussian processes, indexed by Q):
YM. Σ
6
p(S; M, Q),
where e E RR.. is an array of independent standard normal random variables, and
WM.q mm
V2L
(T)
(OT M
ΣΣΣ>> oisk (a (x
where E Nm and E N are mutually independent arrays of independent standard normal random variables. By the Kadakov--Feriquu inequality (reproduced as Lemma C.12), suffices to show that, for any (M, Q) and (M', Q').
E [YMMQ YM, ) E [(WM, WMr 33
Then, E [suPM.Q YM.q] S E. [suPM. WM.q] which completes the proof.
31

---

Published as a conference paper at ICLR 2025
Recall that M, Q) == min 1 pendent, we have
m -==1(
(1/0 .0()) ; M,
: Q wi). Since e are inde-
E [YYMQ )M)
min 1 Σ (v)) +(x]") M ww) 2
1
U3 1(x)) (1) Q,)
(0)
min w j=1
Now, for any E [
min
Σ
& 11
$(x)' 00 M, Q. Q))
min
-
1
Σ
8 (1/) 0(xx); M : (.))
m ==1
==1
2
(4)
sup
1
Σ
m
y)
(
- 4(xx) M.Q - 1 (V/)) (xx)) (0) MM Q', w/
m
==1
(h)
sup
((x)
M. - (110 -x(xxxx))))) M Q' (www)
2
Q , w))²
sup
M.
m W
L2
(T
-0 M MMM) ((Q -0)- (x(0) -x/x) w))²
sup
m
(d) 2L TN
(x TT
M )r)
2 +
2L
Σ (119-097 (x - x ww) 2
m
sup
m j=1
m
j=1
(0) 2L
2x M 1--M))
2
2L m
((1-- (x x.7)) ) )²
m
m
k=1
(18)
where
(a) follows directly from Lemma C.13;
(b) uses the AM-QM inequality:
(c) follows because é is L-Lipschitz, and (yz - yz²)² = (z - z1)² for y E {+1} and x, , R:
(d) uses the simple fact that (a + b)2 < 2(a² + b²): and
(e) uses the Cauchy-Schwarz inequality and the fact that for any w E AK1 |/w|2 S 1.
We now turn our attention to (WM.q). Since the elements of y and a are independent, it follows that
E
[WM.Q - WMr.q') )?]
27
2L- E
Σ m
Yo x 10 x00000 jr
(T
M M
N m
Σ Ossk (1Q Q7T (x/0 - -0000xx))) x
m
1=1 j=1
1=
21 m
Σ
(xx)
(
x
x
(T) fr
M
-Mr)
+
21 m
K
((Q- Q¹T (xx-0)
- -x x(1))) )²
m
m
k=1 k=1
2 E [YymqQ YM \m )3],
where the inequality follows from Eq. (18). This completes the proof.
32

---

Published as a conference paper at ICLR 2025
Lemma C.9. Suppose for any user, when drawing sample, pairs of items independently drawn from Unif(3D-1) Then,
K Dm
sup Eu Pm [( (y ((x; Q - ((; S <31 M.Q
Proof. Let (oi....,, be an array of independent Rademacher random variables. We have
1
sup E,,
Es
B(z,z) le (1;; Q w...m)))
(D(y Xj; M, Q w.
M,Q
sup E,,
Es
sup
(( [e(v (,
Σ
e (ug (xx; M, Q w))
M,Q
w
2. sup Eu M,Q
E.s.
sup w
jj o. (wj M, Q, w))
1=1
(b)
< 2L sup Eu,S,o M,Q
sup
Σ j=1
Jj ((x,,x3]1
X,,, M) + (X111 xgr, Qw))
w
m Σ 1=1
= =2L sup Eu,S,o M.Q
sup
(x,ك) -xj.r. Qw)
+ Eu,s,o
(x5,1331 - -,, M)
1=1
2L
sup Su Q
sup Σ o,Q (xx,)
Xjr);
m
27
(c)
2L
n
sup
R.
Σ Q ((xxt -()r)
m7
Q
(d) 2L
Eg [o3] tr (QQ
LQQTE,,.s E,,. [(x,, Xx,)((z,11 ()r)7
sup Q
(e) 2L
2
sup Q
tr
(QQT)
D
j=1
(8)
KC2
3L
Dmn
where
(a) introduces the Rademacher random variables and uses the standard symmetrization technique (e.g., Shalev-Shwartz Ben-David, 2014, Lemma 26.2);
(b) uses Talagrand's contraction lemma (e.g., Mohri et al., 2018, Lemma 5.7) along with the fact that the loss function e T.-Lipschitz;
(c) uses the Cauchy- Schwarz inequality, Jensen's inequality and the fact that any vector w e A satisfies [|]]] 22 1.
(d) follows because o, 's are independent Rademacher random variables and uses the linearity of expectation and the trace operator;
33

---

Published as a conference paper at ICLR 2025
(c) uses the observation that for two independent vectors z, z' from Unif(S?-1) DD;
E (2 xz)(z z' ID; and
(f) uses the observation that tr(QQ = =QQ - KGO
L
C.3 AUXILIARY LEMMAS
Lemma C.10 (Matrix Bernstein, Tropp et al., 2015), Let X...., Xm E Rdixds be independent
random matrices such that T. [X:] and |X R for each i [m]. Let
max
E
Σ
X x.
Then,
E
Σ X,
S / 2c log d1 + d2) + R log(d1 + d2).
1=1
Lemma C.11. Let Z = (Z,,...,
Zn) be an array of matrices such that /Zillr S B;, and a =
(o (oi...., .n) be an array of independent random variables such that [os]] = 0 and E [aoll = for
all E [n]. Then,
E
o.Z
B
i=1
Proof. We have
T
12
E
Σ o,
Z
Eø
tr
Σ o.Zi
Σ oi
Σ 0,0 , Zj + oo |
Eg
tr
1r
B [o:0,]
2 Z, + Σ E [ Z
tr (Z, Z.)
i=1
Bi.
Lemma C.12 Baudakov-Fernigee inequality, e.g., Adler, 1990, Theorem 2.9). Let X and Y be two centered, almost surely bounded Gaussian processes indexed by T such that, for all 1, 8 E T,
E [X, xx)
< E
[Yr Y,)
Then,
E
sup X.
< E
sup Y
LITT
KET
Lemma C.13. Let I and g be functions from X to [0, 1] that are parameterized by w E W. Given Fiy... Fy X, have
2 f(z; w) g( w)
X
n
min
Σ w)) min wr n
Σ g( w)
S sup
w
n
n
Proof. Let consider two cases:
34

---

Published as a conference paper at ICLR 2025
1. min Ei= f(;; min J Ei= g( w) 2 0:
Let w argmmn ! Ei=1 g(:;; w). It follows that
0 S min 1 (1; min
1
g w) S
1
Σ
(s(r; g(xx
n
n
11
Therefore,
2
(f(mn - a(ossw))) 2
1
n
min- or 71
min Σ
g [z; w)
5
-
n1
71
i=1
2
S sup
Σ (f(x;; g((x(1
mink 1. Ei (x);w) - min, #/2 Eim ());w) < 0:
Similarly, let w E argmin Ei= f(x;; w). Then,
Σ -(a=; wf)) < min
- g(0;;
(w min Σ g( w) < 0.
follows that
2
n
min 12
fax w)
min w
g w)
$
(f( - -g(x;;
1
S sup
(f(z; w)) - - -
71
35

---

Published as a conference paper at ICLR 2025
D EXPERIMENT DETAILS
D.1 GENERAL PROCEDURE
We initialize our models with random weights for all of our experiments, including the prototypical weight for each user. We apply the softmax function to each prototypical weight to ensure that is probability vector. For the unseen user, we initialize its prototypical weight randomly. However, when we update the model using gradient descent, we fix the learned projectors and the prototypical points and only update the prototypical weight of the unseen users. During the training of the Large variant of PAL, we sample # batchsize samples from the preference datasets and concurrently update the shared foundation model, the mapping functions, and the corresponding user-specific weights for each sample. For the Tiny variant of PAL, we fix the foundation model and keep updating other components.
D.2 NUMERICAL SIMULATION
Experiment Setup. We introduce the dataset simulation procedure in Section 4.2. We use the following hyperparameters to generate the synthetic dataset:
16, K N = 100, n 100,8 = 1. We generate another 50 comparison pairs per user as the held-out dataset. Note that here we don't follow the prompt-guided item generation, i.e. conditioning 2e generates 21 and Tr. Instead, we directly draw the item {zI, Zr from normal distribution for simplicity. In the experimental setup, we apply a toy version of PAL-A, where the distance between the synthetic item and the user's ideal point is measured by [=f f(u)ll2- We use a projection matrix (i.e. one-layer MLP network without bias term and activation function) as the model architecture. We randomly initialize the learnable parameters of prototypical user groups and user weights and use the Adam optimizer. The projector / has learning rate 5e - 4 and weight decay le - 3. The learning rate of the learnable parameters of prototypical user groups and user weights is 5e 3. With the aim of good convergence, we train for 1000 epochs per run. We run multiple trials to explore the influence of each hyperparameter: 1) varying the number of samples of seen users (20, 40, 60, 80, 100, 400, 800, 1000). d = (2, 16). K 5, N = 250, 2) varying the number of samples of new users nnew = {5, 10, 20, 40, 50, 100), d = {2, 16), K 5. n = 50, 3) varying 5, 6), 5, d = {2, 16), 50, N = 50 K. We plot the results of
the number of groups K = {2, 3, this experiment Figure and discuss implications
Section 4.2.
We consider two variants of modeling each user's ideal point through the lens of shared group structure of preferences via prototypical ideal points (henceforth referred to as "prototypes"):
S1. Mixture Model: user ideal point is a convex combination of K prototypes, i.e. lies in the convex hull of all prototypes.
S2. Partition Model: a user ideal point is one of K prototypes.
To visualize how well PAL can adjust to the true number of user groups present in data, via learnable prototypical points to represent each group, we consider a simple setting with d 2, K* = 3, K 21,2,3) 3} and N = 100 and plot the results in Figure D. for both partition and mixture settings. We also plot items in the partition setting in Figure D.2.
Partition Model : With only a single allowed assignment for learnable prototype (Figure D. K T==1 1), the predicted prototype is approximately the centroid of the true prototypes, i.e. the model tries to predict good group assignment on average. Also note that since we have single prototype, all predicted user ideal points lie on the prototype itself and performance is close to random. As we increase the degrees of freedom for learnable prototypes to two (Figure D. K 2), the model can predict one prototype close to true prototype (in red), while the other predicted prototype is approximately an average of the blue and green true prototypes. User ideal points now lie in the convex hull of these two predicted prototypes, i.e. the line joining these points. It is only when we increase K = K*, i.e. we match the "true" number of groups in the data (Figure D. K = 3), the model can correctly predict close to all three true prototypes, and user ideal points are concentrated around the predicted prototypes. These observations extend to Figure D.2, where we additionally plot normally distributed items. Recall that in our modeling design, the distance between the user
36

---

Published as a conference paper at ICLR 2025
Kaut Predecten Prettrpiia kieal halt chrdetion mnstrppca tos mants (ut))
Use FoR (7edicte1)) (Prditiee))
Uer haens Preficer
20
20
20
Hat (ISEN)
Fnttivatia Precition
1.
023
-25
-05
-0.
20
-20
05
15
-20
-13
-20
-0.5
0.2
a. Partition, ==1
b. Partition, K
C. Partition, 33
2.5
hod Pluce
25
oud PuR4
25
15
1.5
20-
Teatue
00
0.0
0.0
-10-
-18
-10
1
PESture
d. Mixture, K =1
c. Mixture, K =
f. Mixture, K =3
Figure D.1: In synthetic dataset experiments (Section 4.2), we model user ideal points in two distinct ways: the partition model and the mixture model. To visualize how PAL performs in these settings, we set = 2, K* 3, K = {1, 2, and N = 100 where each user ideal point and prototype is represented as point in a two-dimensional space. In both scenarios, as the number of prototypes or user groups in our model (K) approaches the true number in the synthetic dataset (K*), the PAL framework effectively learns both the prototypes and the heterogeneous user ideal points.
ideal point and the item reflects the user's preference; hence the closer the predicted user ideal point is to the true ideal point, the higher the performance.
Mixture Model : The results for the mixture model are similar to those of the partition model. With single allowed assignment for learnable prototype (Figure (d.),,, K = 1), the predicted prototype is approximately the centroid of the true prototypes. As we increase the degrees of free- dom to two (Figure (.l(c.), K 2). predicted prototypes close to two true prototypes, but one is neglected. When we increase K = K (Figure D.f K = 3), matching the true number of groups in the data, the mixture model successfully predicts prototypes that lie close to all three true prototypes. This demonstrates that similar to the partition model, the mixture model can also adjust well to the true number of user groups present in the data.
D.3 REDDIT TL:DR SUMMARY (TEXT-TO-TEXT)
Detials of seen dataset. We train PAL reward models on variant of the Reddit TL;DR Summary dataset from Li et al. (2024). In this variant, only the ten workers who gave the most feedback were chosen (Each user contains least 1,000 samples). These ten workers are then divided into a majority and minority group, where the majority prefers the longer response, and the minority prefers the shorter response. More details about how the dataset generated can be found in Section 6.1 of Li et al. (2024). This processed dataset contains 20,969 training samples, 2,330 validation samples, and 4,921 test samples. Each example consists of one user ID, one prompt, two responses, and the user's preference.
Details of unseen dataset. We selected all workers, excluding the ten used in the seen dataset, as candidates for the unseen dataset. From this pool, we filtered users with at least 100 valid comparison pairs (i.c., no missing values), resulting in a total of 31 users. We randomly assigned 70% of these users prefer longer summaries, while the remaining users were designated as preferring shorter
37

---

Published as a conference paper at ICLR 2025
barte Promtypical idaa (huth)
Reing
Prosctytica Perrs 2m)
Proseypical Poinss (Tuth) Peature
faatrel
a. K Test Acc 72.2%
b. 2, Test Ace 83.96%
C. Test Acc 91.26%
Figure D.2: Here we plot all items, the predicted user ideal points, and predicted and true prototypes in two-dimensional feature space. The items are normally distributed with d 2, K 3, N 100, 100. As seen in the figure, when we set the number of prototypes in the model K equal to the true number of user groups K 3, P AL can accurately capture the group structure and predict each user's ideal preference point, well as the prototypes that represent each group (K = 3).
Table D.1: Seen user test accuracy of PAL vs. P-DPO (Li et al. v2) and vanilla DPO the Reddit TL;DR Summary dataset. We run 5 trials with K = 2 and K = 1. Our method consistently performs better than the baseline methods. For the La Large version of PAL, CPT 35 as the foundation model.
Model
Params
Seen User Test Accuracy
PAL-A-Tiny (K = 2)
1.6M
52.91 + 0,55
PAL-B-Tiny (K = 2)
2.4M
79.54 = 0.54
PAL-A-Large (K' 2)
352M
90.00 + 1.89
PAI-B-Large (K = 2)
352M
92.82 + 0.95
P-DPO Individual
6.7B
91.04
P-DPO Cluster (K = 5)
6.7B
91.12
PAL-A-Tiny (K =1)
1.6M
49.99 + 0.17
PAL-B-Ting (K =
1.6M
51.51 1 0.08
PAL-A-Large (K = 352M
61.28 1 2.25
PAL-B-Large (K = 1)
352M
59.96 + 3,45
Vanilla DPO
6.7B
58.91
summaries. Based on these assignments, the users' preferences were re-labeled. Given the varying numbers of few-shot samples in the training set, we partitioned each user's comparison pairs into training, validation, and test sets, resulting in multiple datasets.
We leverage multiple pretrained LLMs, including OPT-350M, DistillBERT, Bge-m3, and gte-Owen2, the base model, combined with two-layer MLPs utilizing GELU activation. In the Tiny variant of PAL, we fix the pretrained foundation model and only train the two-layer MLP, whereas in the Large variant,
also train the foundation model. In PAL reward model, set = 2 and apply different learning rates for various model components: 9.65e-6 for pretrained LLMs (Large), le-4 for the two-layer MLPs, and 5e-3 for user weights. The higher learning rate of user weights can enhance the exploration of each user's weight across user groups. As with typical reward models, we train for only epoch to avoid overfitting. The hyperparameter configurations are detailed in Table D.2. The training process takes roughly hour on 1XRTX4090 GPU.
The loss design follows the typical loss of the Reward Model, we use the cumulative loss which weights the per-token reward loss,
7ii et al. (2024) use GPT-J 6. 78, However, the model card for that model Hugging Face broken.
38

---

Published as a conference paper at ICLR 2025
Table D.2: The training hyperparameter setting of PAL reward modeling on Reddit TL:DR. The corresponding experiment setup is described in Section 3.1.
Hyperparameters
Values
K
Batch size
Projectors
hitp-llayer-eelu-drooouud
Epoch
Learning rate of LLM
9.65e-6
Learning rate of projectors
le-4
Learning rate of user weights
5e-3
Weight decay L.LM
0.0
Weight decay of projectors Weight decay of user weights
0.01
0.0
Loss weighting
cumulative
Dimension of preference embedding
512
End of conversation token
< endoffeext |>
Maximum sequence length
600
E log (r(z, vi r(x, ydd
LAM((E Uw: wc 0)
(L + 1)L/2
where æ is the prompt, (0()) represents LLM backbone prediction at generation timestep i, yo and yn separately represent the winming response and the losing response. Note that in our imple- mentation of PAL-A, we concatenate the prompt and the item on the token level. Therefore, the embedding for item produced by foundation model already contains the information of the prompt. This implementation allows us to use the embedding directly without needing to concate- nate it with the embedding of the prompt. Table D.I reports the performance of our models and the numbers reported in Li et al. (2024). We run our model 5 times and report the mean and stan- dard deviation. We want to note that even though did conduct any hyperparameter tuning, With heterogeneous modeling (K > 1), PAL-B-Large achieves approximately +1.8% higher prediction accuracy compared the state-of-the-ee heterogeneous P-DPO (Li al. (2024)) with clusters. With homogeneous modeling (K 1), PAL ALarge is able to outperform vanilla DPO by +2.4%.
D.3.1 FEW-SHOT GENERALIZATION TO UNSEEN USERS
The procedure for few-shot generalization to unseen users is as follows: We randomly initialize the user weights, done during seen training, and then learn the user weights while keep- ing the LLM components and MLP projectors fixed. Since only the user weights need to be learned, the sample efficiency is significantly higher compared seen user training. Results in- dicate that with just 20 samples per new user, we can achieve performance comparable to that of seen user generalization (Figure 3). In Table D.3, we compare the performance of PAL-A-Ting PAL-A-Large,
K 2 and PAL-B-Tiny. and PAI-B-Large trained on OPT-350M embeddings when 10. 20, 50, 100. Our results show that PAL-A-Large, PAL-B-Tiny, and PAL-B-La rge outperform the baselines substantially. We note that while P-DPO was state-of- the-art on seen users, it's performance drops off dramatically for unseen users (-36.6%), while PAL-B-Tiny (91.63%) exceeds P-DPO's state-of-the-art seen accuracy (91.12%) with only 10 samples on unseen users! This indicates the promising potential PAL for cheap few-shot adaptation to new, unseen users in sample-efficient manner.
We observe that increasing sample complexity N from 10 to 100 is impactful only for PAL-B-Tiny (+3%), while the other configurations gain only +0.4 to +1%. PAL-B vastly out- performs PAL-A for the same size and sample complexity (up to +27%). Lastly, Large models outperform their ni counterparts across sample complexities from +14.2 to +23.5%.
39

---

Published as a conference paper at ICLR 2025
Test Test Accuracy (Unseen User)
90%
80%
P-DPO individual (Li al.)
Vanilla DPO
Vanilla RM
70%-
PAL- B-LLgge (K=2, Ours)
60%
25 20 30
40
50
60
70 80 90 100
# Few-Shot Sample per unseen user
Figure D.3: We evaluate few-shot adaptation capabilities via unseen accuracy on the Reddit TL;DR dataset comparing to state-of-the-art Li et al. (2024) and simple vanilla DPO (Rafailov et al., 2024).
Table D.3: Unseen user generalization of PAL compared to baselines the Reddit TL;DR Summary dataset. Here N refers to the number of samples used to learn the weights for the unseen users, and seen accuracies are those reported Table D.I. We note that with just 10 samples for each new (unseen) user, PAL-E. B- B-Large exceeds state-of-the- performance of P-DPO with K=5 (91.12%), demonstrating the suitability PAL for few-shot adaptation and generalization new users.
Model
Seen Accuracy (%)
Unseen Accuracy (%)
P-DPO individual
91.04
55.34 54.55
P-DPO K=5
91.12
Vanilla DPO
58.91
55.37
PAL-A-Tiny = 2, = 10
50.12 + 1.20
PAL A-L-A- K 2, N 20
50.55 + 0.43
52.91 + 0.55
PAL-A-Timg K = 2, N = 50
50.49 + 0.39
PAL-A-Tiny N 100
50.40 + 0.55
B-L-B- = 2,
10
74.88 + 0.79
PAI-B-Tinn K = 2, N 20
77.37 + 0.37
79.54 + 0.54
PAL-B-Tiny K = 2, = 50
76.19 + 0.49
PAL-B-Tiny K = 2, N 100
77.80 + 0.07
A-agee K
N 10
73.39 + 1.82
AAL-A-Large K = 2, N 20
74.04 + 2.50
90.00 1.89
PAL-A-Large K
50
72.33 + 2.36
PAL-A-Large К = N = 100
74.39 + 1.70
PAL-B-Large K 2, N 10
91.63 + 1.43
PAL-B-Large K 2.N 20
91.72 = 1.40
92.82 0.95
PAL-B-Large K N. 50
92.02 = 1.10
PAL-B-Large K 2, = 100
91.97 + 1.91
D.3.2 CHOICE OF FOUNDATION MODEL.
The choice of foundation model is an important factor that impacts the performance of PAL, es- pecially for PAL-A. A- A-Tiny and PAL-A A A-Large. This because the foundation model directly decides the quality of the embeddings we obtain for the items. Table D.4 illustrates that there is performance gap between using OPT-35 35 0M and Dist 1BERT as the foundation model, especially for Tiny variants where do not train the foundation model. The existence of such a gap is possi- bly due to the fact that embedding than the decoder-based OPT-350M.
an encoder- mased model, which provides a better sentence
40

---

Published as a conference paper at ICLR 2025
Table D.4: Comparison of performance of PAL-A-Tiny and PAL-A --arge with different foun- dation models on the Summary dataset.
Model
Foundation Model
Seen User Test Accuracy
OPT-350M
52.91 = 0.55 72.99 1.21
PAL-A-Tiny (K = 2)
Distil BBERT
OPT-350M
90 + 1.89 91.75 0.41
PAL-A-Large (K
DistilBERT
Table D.5: Personas used across various "true" number of user groups K* in heterogeneous persona dataset.
K
Personas
2 interest in art, interest in literature
3
interest in art, interest in literature, interest in math
4 interest in art, interest in literature, interest in math, interest in music interest in art, interest in literature, interest in math, interest in music, interest in science 6 interest in art, interest in literature, interest in math, interest in music, interest in science, interest in sports
D.4 PERSONA (TEXT-TO-TEX)
Anthropic's ' Persona dataset Perez et al. (2022) consists of a series of personalities (personas), each corresponding with 500 statements that agree with the persona and 500 statements that do not. We denote the set of statements that agrees with a persona as S(p). We construct a semi-synthetic dataset using Anthropic's Persona evaluate PAL.
Dataset. Let = (p::-... pk*) denote the set of personas that exists our semi-synthetic hetero- geneous dataset with K* "true" preference (prototypical) groups i.e. each person (user) has of the K* personalities. For each P5 E we generate N synthetic seen and unseen users. For each seen synthetic user, we generate np queries that ask if the user agrees with given statement from the persona dataset. For each unseen synthetic user, we generate Rppunseen queries. If the statement aligns with the persona Øj of the user, i.e. the statement belongs to S(;)), then the user answers yes, otherwise no. Table D.5 lists the personas used for each K*, and Figure D.4 shows sample question.
Experiment Setup. We evaluate the performance of PAL-A-Tine with hinge loss and model PAL-B-Tiny with logistic loss on the heterogeneous persona dataset various settings. Both model utitlize a 2-layer MLP as the / function. To examine the impact of various hyperparameters. we conduct experiments varying the number true prototypes in the dataset K the number of prototypical groups used in the model K, queries per seen 7p, and latent dimension with fixed number of users per group = 10, 000. Details of the values for each hyperparameter used are listed below:
........... (b) varying n2 (75, 100, 200, 500, 1000} and K = = 5} while fixing K** 4, and d 16, {1..... 5} (c) varying d - (4, ; 16, 32, 64) and K (1
(a) varying K* = {2, ....... 6) and = (l,,... ......,8) 8} while fixing n, = 1000, 16,
while fixing K 4, and n, = 1000, (d) varying R2p.umsenn {1, 10, 20, 50, 100. 200, 500, 1000} and K (1..... 5) while fixing K*
4, np 1000, and d 16.
Both FAL-A-Tiny and PAL-B-Tin Tinn used the same value, except for d. This is because PAL-A utilizes residual connection. Therefore, the latent dimension fixed to 768, the dimension of the input embedding.
Results. We repeat these experiments five times and report the results on PAL-A-Tiny in Fig- D.5 and PAL-B-Tiny in Figure D.6. Both Figure D.6 and Figure D.5 (a, b) illustrate the generalization performance of PAL-A-Tiny and PAL-B-Tiny on the heterogeneous persona dataset. We observe that K - K* the seen accuracy increases to 100% given sufficient number
41

---

Published as a conference paper at ICLR 2025
Is the following statenent sonething you would say? enjoy figuring the solution to challenging nathenatical problen
interest in art
interest 1n matth
Figure D.4: An example of a pairwise comparison query with a prompt from our heterogeneous persona dataset generated using Anthropic's i Personas. For example, a synthetic user assigned the persona interest in art will have ground truth y = -1 by answering no, whereas a synthetic user assigned the persona interest in math will have ground truth y = +1 by answering yes.
é pritoe KC
comparisons 2or soon user (ng)
Koomsorisns por unscon user (fpcrsse))
75
200
1000
130
000
100
500
50
200
1000
100%
100%
100%
Boon 2000000
25%
75%
75%
80%
60%
00%
29%
25%
20%
0%
protecypas
(0)
Figure D.5: Seen accuracy (a.b) and unseen accuracy (c) evaluated on the heterogeneous persona dataset across the number of prototypes K used in PAL-A PAL-A-Ting -inny. The number of true prototypes K* in (b, <) is 4. We vary (a) the number of true prototypes K*, (b) the number of comparisons per seen user n.p (c) the number of comparisons per unseen user ?2p.umsenn Since we used residual connection the MAL-A-TTing we could not vary the size of the latent dimension.
of users and number of comparisons per user. Figure D.5 and Figure D.6 (b) shows that as we get more comparisons per user, we achieve better seen user accuracy, i.c. we can generalize to unseen pairs for users who are seen (provide training samples) in the dataset. Figure D.6 (c) shows that the size of latent dimension d does not affect the seen accuracy dramatically. Figure D.5 (c) and Figure D.6 show the accuracy for unseen users, i.c., users who do not provide training samples. When K ==1, no further learning is needed to generalize to new users. However, when K > 1, we require weights over the K prototypes for the new users to be learned. To learn these new user weights, as discussed in Section 2.2, we fix the K prototypes and the mapping f and use only few test data samples to learn the user weights (C4). We use these learned weights to make predictions on the remaining test data. From Figure D.5 (c) and Figure D.6 (d), we see that for K the number of samples used to learn weights makes no difference since there are no weights to learn over a single prototype. For K 2, we see that as we use more data for learning the new user weights, the performance shows diminishing returns until saturation. We also demonstrate that as the number of prototypes K increases, more comparisons per user are needed to learn the new user weights, since the dimension of the weight vector increases with K.
D.5 PICK-A-PIC (TEXT-TO-IMAGE)
Dataset. The Pick-a-Pic dataset is a large, open dataset designed to capture human preferences in text-to-image generation. It includes over 500,000 examples where users compare two Al-generated images based on text prompt and choose their preferred one. This dataset is used to align models with human preferences.
Experiment Setup. We apply PAL B- Tiny on the Pick-a-Pic dataset. Since the Pick-a-Pic dataset collection process requires strict rubrics, the labels collected from workers may not reflect the worker's diverse preferences. Thus we set K = for the PAL model. We use two-layer MLP networks with ReLU activation and residual connections the mapping functions. To avoid over- fitting we set the dropout rate to 0.5 and weight decay to le 2. We apply different learning rates for various model components: le 4 for the two-layer MLPs and be for user weights.
42

---

Published as a conference paper at ICLR 2025
compontionn user (%()
compootiante
1000
100%
100%
100%
Stan 75%
79%
75%
t1u proketyess OK
pictotyyss (K)
protchyppe 90
protoypos [K]
(6)
(d)
Figure D.6: Seen accuracy (a.b,c) and unseen accuracy (d) evaluated on the heterogeneous persona dataset across the number of prototypes K used in the PAL-B-Tiny. The number of true pro- totypes K in (b, C, d) is 4. We vary (a) the number of true prototypes K", (b) the number of comparisons per seen user Rp. (c) the size of latent dimension d, (d) the number of comparisons per unseen user np, unseene--
Table D.6: Number of samples in each split of the newly constructed Pick-a- a-iiter dataset.
Category
Train
Val
Test
Seen
58831
628
1597
Group
Unseen
9527
79
1886
Total
68358
707
3483
Seen
57200
404
2096
Group
Unseen
9402
52
1812
Total
66602
456
3908
D.6 PICK-A-FILTER (TEXT-TO-IMAGE)
due to the high level of "agreement" among labelers over image preferences on Pick-a- Pic vI (Kirstain al., 2024), we construct a semi-synthetic dataset by applying filters subset of Pick-a-Pic v1, which we call the Pick-a-Filter dataset. To construct the dataset, we consider only samples that have ties, i.e. the labeler decides that one image is decisively preferable the other, given the text prompt. As Pick-a-Pic provides unique and anonymous user IDs for all preference pairs, we consider subset of users who provide samples in both the train and test sets (468 4223 users). We further only consider users who provide more than 50 labels (234 / 468 users) and sort the users by number of samples provided. We split these users into equal groups of 117 each, and we assume without loss of generality that the first group of users (GI) prefers "cold" tones (blue filter) and the second group (G2) prefers "warm" tones (red filter). Lastly, we arbitrarily consider the first 50 users (who provide the most number of samples) in both the train and test sets of Pick-a-Filter. We add this seen vs. unseen distinction to evaluate how well P PAL can adapt unseen (i.c. new) users after training. Currently, a-Filter (Section 3.3) train on vi-train-seen (1 16031 samples) and evaluate on vi-test-seen (3693 samples). We show the number of samples in each of these splits in Table D.6. After constructing splits, we apply the following filtering logic:
Dataset:
"seen" users, i.c. users that provide samples
experiments on Pick-
1. Apply "winning" and "losing" filters to appropriate images depending on label. For GI the winning filter is blue, and for G2 the winning filter is red.
2. Randomly shortlist .%% of samples to add filters. The remaining (1 - .3%% of samples will remain unaltered (default images from Pick-a-Pic vl).
3. Randomly select 50% of above shortssh samples to apply filter to only the winning image, and the remaining 50% to apply a filter to only losing image
We add these sources of randomness make learning preferences Pick-a-Filter less prone to hacking (e.g. the model could trivially learn to predict an image with a filter as the preferred image).
Experiment Setup. We choose 2-layer MLP networks with ReLU activation and residual connec- tion as the prompt mapping function 9% and the output mapping function f. To avoid overfitting, we set the dropout rate 0.5 and weight decay to le 2. We use the Adam optimizer with learning
43

---

Published as a conference paper at ICLR 2025
Original sarple
Step randomly modify 5% (mixture ratio) samples per user
Step manually split users into two groups
images with blue filter and not prefer images with red filter
prefer images
Podafication version
Mods fodatication version
with red filter
Pruptt snteratio
Preatt estomsian bot
not prefer images filter
8
X
X
Figure D.7: The construction diagram for the semi-synthetic Pick-a-Filter dataset. It involves ran- domly selecting approximately 135,000 samples from the Pick-a-Pic v1 dataset and dividing the user IDs into two disjoint groups. We assume one group prefers images with "cold tone" (blue) filters and the other with "warm tone" (red) filters. To incorporate diverse color filter preferences, randomly select 3% of samples per user which to apply filters.
rate 1e - 4. To evaluate the model's performance, we use the checkpoint with the highest accuracy on the validation set.
44

---

Published as a conference paper at ICLR 2025
E COMPUTATIONAL RESOURCES
We conducted most of our experiments using 4xRTX 4090, each with 24 GB of VRAM. For the experiments involving a foundation model that has 1.3B parameters more, we used 2 x A100, each with 80GB of VRAM. A typical run of the experimentw finished within 2 hours.
45


Reviews:
[{"review_id": "ci7N1h6icn", "reviewer": "Reviewer_rMSN", "summary": "While it is well-known that humans have diverse preferences, most foundation model alignment methods assume homogeneous preference across all users. The authors propose a novel reward modeling framework to capture shared as well as personalized preferences to enable pluralistic alignment. The method's ability to combine global and individual preferences allows the method to perform well without requiring an overwhelming number of samples for each persona. The method is also demonstrated to outperform homogeneous reward models when diverse preferences are introduced to the evaluation dataset.", "questions": "1. Are there benchmarks curated with real world data that can highlight the benefits of pluralistic alignment?\n2. In section D.3.2, it is mentioned that the choice of foundation model greatly impacts the performance of the proposed method. It would be great to do an ablation study to investigate whether this effect is observed with other alignment methods as well.\n3. When should one opt for either PAL A or PAL B?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The method addresses the problem of pluralistic alignment, which is currently unaddressed by most alignment methods.\n2. The method has a native flexibility to adjust the shared and personalized portions of the modeled preference.\n3. The method complements existing alignment methods.\n4. The paper rigorously explores the behavior of the method through simulated experiments and the results are visualized clearly, as are the algorithms. The experimental setup is clearly documented.\n5. The method converges quickly to capture the preferences of unseen users", "weaknesses": "It does not seem like there is a benchmark that reflects real use cases that can highlight the benefits of having a heterogeneous model; both Reddit TL;DR and Pick-a-Filter are both semi-synthesized datasets that artificially accentuates the diversity of human preference. This calls into question whether pluralistic alignment is a valuable problem to solve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fh98jpOVmg", "reviewer": "Reviewer_Eo4u", "summary": "This paper tackles the issue of personalized reward modeling. The authors recognize that different users may not generate a consistent preference across different context-sample pairs. They propose two novel modeling method that represent preference with a latent reward function. PAL-A assumes the reward of a sample-context pair is determined by its distance to the user's ideal point in a latent space. It further assumes that the ideal points of a population lies in the convex hull of several supporting points in the latent space, and we can recover an individual's personalized preference through weighted average of supporting points. PAL-B represents preference as unknown preference mapping, and commonalities are similarly modeled as a convex combination of K prototypical mapping.\n\nThe author conducted extensive experiments on both NLP and T2I generation domain, achieving SOTA results", "questions": "See weakness. Additional questions are\n\n1. Why Reddit experiments did not include results of PAL-A? Am I missing anything?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper challenges a commonly overlooked assumption in the alignment literature, that individuals' may have a different preference.  The proposed two approaches are novel and clearly differentiates from existing works (e.g. KTO), which treat such inconsistencies as noises in preference data.  The presentation is clear and easy to follow, with motivations and formulations of the proposed method covered in detail. The experiments showed strong performance of the proposed method. Moreover, the proposed method can be trained on a single consumer grade GPU whereas the baselines are trained on multiple A100s.", "weaknesses": "1.Some experiment details are lossy and not well-documented.  Presentation is a bit unclear. For example, while the authors clearly documented the choice of base model and training data on Pick-v2 results (table 3), such information is not included in Pick-v1 (table 2). **What is the base model for results in table 2? Is it vanilla CLIP-H or PickScore?**\n\nWithout this knowledge, it is hard to evaluate the claim on parameter efficiency. For example, if v1 results are reported via a model that is fine-tuned on Pick-v1 embedding, then it is hard to argue that the model is more parameter efficient since it starts with a fully fine-tuned model. Overall, presentation wise, I think it would make more sense to add table 2 as an extra column in table 3, which would reduce many confusions on the setup.\n\n2.Results on Pick-a-Filter are unconvincing. \n\n2.1  Pickscore baseline is missing. While the authors claim that they cannot compare PickScore as its training set overlaps with Pick-a-Filter’s val set. However, this can be trivially fixed be eliminating the overlapped examples, which the authors already did for table 3. Why not do the same? Alternatively, given the large samples of pick-a-pick v2, it is not hard to construct a custom validation split that does not overlap with the training data of PickScore. \n\n**Please either eliminate the overlapping examples from the Pick-a-Filter validation set and include PickScore as a baseline, or\nconstruct a custom validation split from Pick-a-Pic v2 that doesn't overlap with PickScore's training data, and use this for comparison, or justify why these options are not possible**\n\n\n2.2 The red and blue filter examples seems too rival, and I suspect the obvious color differences will overshadow the \"commonalities\" in preference.  I think the key benefits of the proposed method that it captures both the \"common preferences\" and \"individual variations\". However, for the color filters, but a naive color classifier may also achieve high accuracy in this example. It is unclear if the proposed method offer any benefits. Such comparison is required. \n\nThis is also highlighted in Fig 4, where differences in low beta region is unclear (side note: presentation wise this figure needs improvement. It is hard to tell which line is higher). I think the low beta region might be more representative of the actual discrepancies in human preferences. However, as Pickscore baseline is missing from Figure 4 (See comments in 2.1), it is hard to tell if PAL offer any benefits in this region. I image a proper pickscore comparison would be a flat line that resembles CLIP and HPSv2. The question is whether the line of PickScore would be higher than PAL in low beta region.\n\n**I would highly appreciate it if the authors can provide more discussions on significance of results on Pick-a-Filter, particularly on the non-linear improvement in Figure 4.** It may seem that the model just collapse to a color classifier at high beta region.  **Authors should discuss if PAL is simply collapsing to a color classifier. I suggest authors compare PAL against a naive color classifier. I'm open to other means/discussion on this topic as well.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eD2w65umZm", "reviewer": "Reviewer_g42M", "summary": "The PAL framework aims to capture diverse human preferences from internet-scale data trained foundational models for pluralistic alignment. The modular design of the framework allows it to leverage commonalities among users while providing personalized services to each user, achieving efficient few-shot localization of preferences for new users. Through extensive empirical evaluation, PAL has demonstrated performance that matches or exceeds the state-of-the-art methods in both text-to-text and text-to-image tasks.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The PAL framework enables personalized reward modeling with fewer samples, which is highly beneficial for data collection and model training, especially when data annotation costs are high.\n2. PAL has shown superior performance across multiple tasks, not only surpassing or matching the accuracy of existing state-of-the-art methods but also significantly reducing the number of parameters, demonstrating dual advantages in efficiency and effectiveness.\n3. The paper not only provides empirical evaluations but also theoretical analyses, proving the sample complexity of PAL in generalizing to new users and unseen samples, providing a solid theoretical foundation for the model's reliability and effectiveness.", "weaknesses": "1. Figure 1, as the first graph of the paper, is somewhat confusing, with pasted images and formulas that are unclear. There is a lack of a clear caption explanation, and the introduction does not provide specific details, making it difficult to understand the meaning of the symbols in the graph or the order in which to view them.\n2. The modeling of user preferences in the paper mainly focuses on the preference for summary length, which may not cover a broader range of personalized preferences. The preference distribution of seen and unseen users is limited to the preference for the length of summaries and may not be comprehensive enough.\n3. The datasets used in the paper may lack sufficient diversity, which limits the model's generalization capabilities in a broader range of scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)