Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework
Authors: Kihyun Kim
Jiawei Zhang
Asuman E. Ozdaglar
Pablo A. Parrilo
Venue: ICLR 2026 Poster (2026)
Keywords: AI Alignment
Population-Proportional Alignment
Social Choice Theory
Axiomatic Framework
Rank Aggregation
Pluralistic Alignment
Preference-based Reinforcement Learning
Reinforcement Learning from Human Feedback
Nash Learning from Human Feedback
Large Language Model
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: To address bias and manipulability issues in RLHF and NLHF, we propose a novel preference learning framework grounded in social choice theory that achieves proportional alignment with true population distribution of evaluator preferences.

Abstract: Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation. To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs a policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well as our newly-introduced axioms of population-proportional alignment and population-bounded manipulability. Moreover, we propose a soft-max relaxation method that smoothly trades off population-proportional alignment with the selection of the Condorcet winner (which beats all other options in pairwise comparisons). Finally, we validate the effectiveness and scalability of our approach through experiments on both tabular recommendation tasks and large language model alignment.

Published as a conference paper at ICLR 2026
BEYOND RLHF AND NLHF: AN AXIOMATIC FRAMEWORK
ALIGNMENT UNDER
Kihyun Kim-* Jiawei Zhang Asuman Ozdaglar* Pablo A. Parrilo
MIT LIDS 2 University of Wisconsin-Madison
Ikihyun, jwzhang, asuman, parrilo) @mit edu
ABSTRACT
Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible strategic manipulation. To address this issue, develop novel preference learning framework capable of aligning aggregate opinions and policies propor- tionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well our newly-introduced axioms of population- population-proportiona alignment and ippulation-bounder manipula- bility. Moreover, we propose soft-max relaxation method that smoothly trades off
alignment with the selection of the Condorcet winner (which beats all other options in pairwise comparisons). Finally, we validate the effectiveness and scalability of our approach through experiments on both tabular recommendation tasks and large language model alignment.
INTRODUCTION
Aligning artificial intelligence (AI) systems with complex human preferences is growing priority in fields such as robotics (Kupesik et al. 2017: Bryık et | 2022). recommendation systems (Xue et al.
2023), and large language models (LLMs) Ziegler et al. [2019] Stiennon et al. 22020; Ouyang et al. 2022). A key challenge in this endeavor is how to infer and represent such preferences accurately, particularly when they are only available through incomplete signals like pairwise comparisons. This has prompted reinforcement learning from human feedback (RLHF), which has become a widely used framework for preference learning (Ouyang et al. 2022 Christiano al. 2017). RLHF streamlines the alignment process by first learning a reward model that assigns scalar scores to different alternatives, typically trained using maximum likelihood estimation under the Bradley-Terry (BT) model. In the second stage, policy is optimized through reinforcement learning to maximize the expected rewards, guiding the system toward behaviors aligned with human preferences.
Despite its practical success and simplicity, the standard RLHF framework rests on critical assump- tion that complex human preferences can be captured by a single scalar reward. Recent research highlights that this assumption often breaks down, especially when human feedback reflects incon- sistent or conflicting judgments across evaluators (Chakraborty et al. [0024). In particular, RIHF struggles in scenarios involving intransitive or cyclic preferences, where clear ranking among alternatives can be established, leading to failures in accurately modeling the underlying prefer- ences (Munos et al. 2024 Swamy al.) 2024). To address these limitations, a came-theoretic framework called Nash learning from human feedback (NLHF) has been introduced (Munos et al.
2024 Swamy al. 2024 Ye et al. 2024
al. 2025). NLHF reframes preference
"Equal contribution.
1 "orresponning author.

---

Published as a conference paper at ICLR 2026
learning two-player constant-sum game and identifies equilibrium policies that no competing policy can outperform, regardless of the complexity of the underlying preferences.
Nevertheless, both RLHF and NLHF frameworks remain limited in their ability to address another critical issue: the proportional alignment of evaluator preferences. When preferences are aggregated across multiple evaluator groups with distinct viewpoints, both RLHF and NLHF tend yield policies that do not adequately reflect the full distribution of the evaluator population (Chakraborty
al. 2024). To address these challenges, recent research has turned social choice theory-oriented
approaches, such as maximizing the minimum satisfaction across evaluator groups (Chakraborty
al.
2024 Ramesh et al.) 2024) and optimizing social welfare functions (Zhong et al. [0024] Kim
et al.
2025 Another line of emerging research, pluralistic alignment (Sorensen et al.]
(024)). seeks
to reflect diverse perspectives Al systems through approaches such as mixture-based models (Chen
et al., 2024), helief-conditioned models (Yao et al., 2025), and steerable models (Adams et al. 2025 +
with particular focus on L.Mss. However, these methods generally assume explicit knowledge or clear labels of evaluator groups, which limits their practical applicability since group identities are often implicit or unobservable in the real world. Motivated by this limitation, our research aims to achieve proportional alignment without requiring additional information about the evaluator profile.
Our approach builds upon recent works addressing diverse preference aggregation through an ax- iomatic approach from social choice theory (Mishra 2223 Siththaranjann et al.) 22224) Dai & Fleisig
2024 Conitzer et al., 2024, Ge et al.] 2024 Maura-Rivero et al.) [022)] [hhi et al. [2055] Xiao et al.
2025). Specifically, we propose novel preference learning algorithm that satisfies two foundational axioms, monotonicity (ensuring that improving an alternative's 's ranking cannot decrease its proba- bility) and Pareto efficiency (ensuring that if an alternative preferred by all, favored by the policy), as well as two new axioms we introduce: oopulation-proportion -ouudde manipulability (PBM). The first new axiom, PPA. requires the policy to be at least weakly proportional to evaluator population shares, addressing the insufficient representation of the population distribution under RI.HF and NLH The second axiom, PBM, bounds the incentive for manipulation as an affine function of the true population share, thereby guaranteeing robustness. Recent studies have highlighted that conventional preference learning methods are susceptible to strategic misreporting (Buening et al. 2025). Unlike existing approaches that incorporate explicit mechanism design to ensure strict strategyproofieee (Park et al. 2024] (oomaaiaa et al.) 2024 (un et al., (2024) HHao& & Duan 2025; [Bunning et al.) (2055), ) our method inherently limits manipulative advantage by constraining policy selection based on estimated feasible population distributions.
alignment (PPA) and
Further details related work are provided in Appendix B
1.1 OUR CONTRIBUTION
The first key contribution of this work is demonstrating that the set of feasible population distributions of evaluators can be inferred directly from pairwise comparison data. Leveraging this insight, we develop a novel preference learning framework designed to align policies proportionally with the underlying population distribution. To establish rigorous theoretical basis, we adopt an axiomatic approach, proving that our framework satisfies two fundamental axioms, monotonicity and Pareto efficiency, and two newly introduced axioms, PPA and PBM. In addition, we propose a novel softmax relaxation method to control the trade-off between proportional alignment and the selection of the Condorcet winner. For practical deployment, we present a scalable algorithm with function approximation. allowing framework to scale to high-dimensional settings such LLMs. Finally, the proposed framework is validated through empirical evaluations in both tabular and function approximation settings.
Organization of the paper. In Section 2 we formalize the setting of preference learning and probabilistic social choice, and establish connections between them. In Section 3 motivated by a simple negative example, we introduce two desirable axioms alongside two fundamental axioms. In Section 4 we propose a novel preference learning algorithm that satisfies these axioms and provide theoretical analysis. Finally, Section (eatioonS]prsseee presents empirical evaluations that demonstrate the effectiveness and scalability of our method. For ease of reference, all mathematical notation used in the paper is summarized in Appendix
2

---

Published as a conference paper at ICLR 2026
Profile
Preference Function
P
(1)> = 0.7
0.5
0.7
P =
(1g--11) = 0.3
0.3
0.5
PSCF
Policy
Preference Learning
(c)
(yn) = 1, <1²) =
F(P)
Figure 1: Illustration of the relationships between the profile, preference function, and policy.
PRELIMINARIES
2.1 PROBABILISTIC SOCIAL CHOICE FUNCTION AND PREFERENCE LEARNING
We begin by reviewing key concepts from social choice theory and preference learning to establish foundation for our subsequent analysis. Consider set of M alternatives, denoted by y == (y1, 1/2........................../ yod }. where each y E y may represent response generated by language model or an action in decision-making task. We assume each evaluator has strict and complete ranking over the alternatives, and let S denote the set of possible rankings (i.e., permutations of ))) Each ranking is represented by E S, where ry:) = k indicates that alternative y/ is ranked k-tth under A profile a (S)) is a probability distribution over the set of all rankings, where o, represents the proportion of the population that adheres to ranking
A probabilistic social choice function (PSCF) is a mapping & A(S) + A(Y) that assigns to each profile policy म, which is probability distribution over the alternatives in V. In practice, however, acquiring a complete profile a is often infeasible due to the high cost collecting full rankings over large set of alternatives.
To address this limitation, pairwise preference learning algorithms have been developed, allowing alignment based solely pairwise comparison data. We define preference function P y [0, 1]. where Pyy > y') denotes the probability that alternative is preferred over y'. Given a profile a,, let pa be the preference function induced by the population distribution over rankings, defined as
Pd > 5/)
Σ rES
where (r(s)<(((() is an indicator function equal to if ranking r places alternative in a better (i.c., lower) position than y', and otherwise. This function captures the expected pairwise preference between y and y' under the distribution a.
We define P as the set of all preference functions induced by some profile a E A(S):
P = (P 3o A(S) s.t. P P*).
Any P E P satisfies consistency conditions known as the Block-Marschak inequalities (Block & Marschak 1959), including skew-symmetry: Pyy > s/ + P(y' > y) = Vy, y E y. A preference learning algorithm is mapping F P (Q)) that assigns policy each preference function. Throughout this paper, we say that a preference learning algorithm F implements a PSCF & if, for every profile a ASS), it holds that F(P*) = $(a). The relationships between the profile, preference function, and policy are illustrated in Figure
2.2 TWO STANDARD PREFERENCE LEARNING ALGORITHMS
Next, introduce two prominent preference learning algorithms and discuss their connections to established concepts from probabilistic social choice theory.
Reinforcement learning from human feedback (RLHF). The Bradley-Terry (BT) model, widely used in preference modeling, assigns each alternative y reward r; with preference probabilities
3

---

Published as a conference paper at ICLR 2026
P(t > y,) exp(r:)/ (exp(ra) + exp(ry)). Standard RLHF estimates these rewards by likeli- hood maximization and then trains a policy to maximize expected rewards. Recent work (Siththa ranjan et al. 2024) shows that this procedure is equivalent to the maximal Borda rule from so- cial choice theory, which deterministically chooses the alternative with the highest Borda score B(y) = Eres or(M r(v)). As proved in Appendix [ the ranking from BT-optimized rewards coincides with Borda rankings, so RLHF without regularization (denoted by FRI)) implements the maximal Borda rule (denoted by 4BB)). Direct preference optimization (DPO) generalizes this by adding Kullback-Leibler (KI.) regularization relative to a reference policy (Rafailov et al. 2023).
Nash learning from human feedback (NLHF). As highlighted in recent studies (Munos et al. 2024 Swamy et al., 2024 Maura-Rivero et al. 2025), RLHF has limitations in scenarios involving intransitive or cyclic preferences. An alternative y" is called a Condorcet winner it is preferred by majority over every other alternative, formally stated as P(y" > y) > 0.5 for all y # y². When aggregating preferences across multiple evaluators, scenarios without a Condorcet winner can arise, which is called the Condorcet paradox. In such cases, selecting the alternative with the highest Borda score fails to adequately represent collective preferences, as a deterministic policy cannot capture the lack of consensus or nuanced preferences. To address intransitive preferences, the eame-theoretic approach, known as Nash learning from human feedback (NLHF) (Munos et al., 20244 Swamy et al. 2024; Ye et al. 2024 Maura- Rivero al., 2025), has been adopted to model preference learning as a two-player constant-sum game max,, (A()) min ( (2) (2))))))>~<___ [P( > y2)]. where the equilibrium policy H' cannot be uniformly outperformed. This algorithm, denoted by ENL, implements the well-known PSCF maximal lotteries (ML) (Fishburn 1984), denoted by a.
AXIOMATIC FRAMEWORK FOR POPULATIOX-PROPOTIOVAA ALIGNMENT
3.1 MOTIVATING EXAMPLE WITH BINARY ALTERNATIVES
Despite their practical utility, neither RLHF nor NLHF guarantees alignment proportional to the evaluator's preferences. To illustrate this point, we present a simple scenario involving binary alternatives. Consider two alternatives, y {y1, )2 and profile consisting of two distinet groups of evaluators: group G1 prefers alternative y/1 over y2, while group G2 prefers y/ over Vi- Let wi and w denote the population shares of groups G1 and G2. respectively. Suppose the two alternatives are nearly tied, with (wf, wg)) = (1/2 + 1/2 c) for an arbitrarily small positive scalar c. Then, the corresponding preference function is given by Pd (11 > y2) = 1/2 + c and pa (y2 > vi) 1/2 Despite this minimal margin both algorithms (R and (P") yield a deterministic policy that selects the alternative with slightly greater support, namely y1. because such subtle differences in preferences (or rewards) are lost during the policy optimization.
This binary example highlights two potential limitations of RLHF and NLHF frameworks. First, selecting policies that focus entirely a single alternative may accurately represent preferences across evaluators, raising concerns about bias. Sccond, these methods have high sensitivity to small perturbations in the preference function. Specifically, a slight shift in from negative to positive abruptly flips the policy outcome (T(1), (12)) from (0, 1) to (),0), making such approaches vulnerable small perturbations. These limitations underscore the need for a novel approach that reflects the ratio of (wf, wg) in the resulting policy.
3.2 PROPOSED AXIOMS FOR
PROPORTTNA ALIGNMENT AND ROBUSTNESS
Social choice theory studies the aggregation of individual preferences through an axiomatic approach, which specifies desirable properties (axioms) and characterizes aggregation rules that satisfy them. In particular, two fundamental axioms, monotonicity and Pareto efficiency, are presented in Appendix [ Following this approach, we introduce the axioms that a PSCF & is desired to satisfy and propose preference learning algorithm F that implements such PSCF.
Proposed axioms. Motivated by the earlier example, we next introduce a new axiom designed to ensure alignment with population distribution of preferences. Let G₁ == fr E S r(v)) 1} denote the set of rankings in which alternative ya is ranked first. The population share of group G, is denoted by wg- := Ereg, ar. For notational convenience, we define 0g (SS) as the normalized
4

---

Published as a conference paper at ICLR 2026
Table
Overview of standard PSCFs and axioms
F
Monotonicity
Pareto Efficiency
PPA
PBM
Maximal Borda (MB)
V (RLHF)
X
Maximal lotteries (MI.
V (NLHF)
X
x
x
Random dictatorship (RD)
X
V
Proposed framework
V
sub-profile restricted to rankings in Gk. where our ar/wg for all Gk. and ohrr 0 for all r & Gg. Let P2 denote the group-specific preference function, generated from dg. using the mapping defined in equation I By construction, P7 y) = 1 for all y * yk, since this group unanimously prefers over all other alternatives. The overall preference function then a weighted aggregation of the group-specific preferences: P" =
k= - wCPt.
Under this definition, our first axiom ensures that the policy reflects each group's population share. Note that our proportionality notion focuses solely on the selection probability of each group's 's top choice and does not incorporate lower-ranked preferences.
Definition 3.1 (a-Population on--Ppllaionnproooriionn proportional alignment (a-PPA)). A PSCF satisfies a-population- proportional alignment if n (yk)/uf 2 a()) for all a E A(S) and yk ), where = ((r) and a' ((S) 10.
The function (or) quantifies the strength of alignment: a higher value of a implies stronger alignment with w", with (<))) = indicating perfect proportional alignment. Next, we examine the robustness of & against manipulation through the following axiom.
Definition 3.2 (Single-group manipulated profile). Given a profile o and group index k E [M], a profile o' is called a single-growy group manipulated profile of a if o' can be obtained by modifying only the ranking distribution of the sub-profile Øg. Formally, of is single-group manipulated profile if there exists profile a' such that of + w -k)).
Definition 3.3 ceppulation-bounder manipulability (7-PBM)). A PSCF & satisfies y-population- bounded manipulability if, for any profile o and its single-group manipulated profile o' we have (y)) wn + 72, where 7 (7, 72), n >
and 71 + 72
The --BBM axiom ensures that the maximum influence single group exert through manipulation bounded above by an affine function of its population share. Specifically, group can only achieve deterministic policy selection for its preferred alternative (i.e., (b(o)) 1) only if it constitutes the entire evaluator population (i.e., wy 1). Note that a larger " provides stronger robustness guarantee. Particularly, TI implies that the manipulated policy value is limited exactly to the group's true population share. We also note that the focus of the 7-PBM axiom differs from that of classical strategyproofnnss: it does not constrain an individual participant's 's incentive to misreport, but instead limits the extent to which any group can become over-represented.
3.3 LIMITATIONS OF STANDARD PSCFs: AXIOM VIOLATIONS AND NON- IMPLEMENTABILITY
We next show that the standard PSCFs either fail to satisfy the proposed axioms not imple- mentable by a preference learning algorithm. Consider a PSCF that aligns the policy exactly with each group's population distribution, commonly referred to as a random dictatorship (Brandt, 2017).
Definition 3.4 (Random dietatorship). A PSCF RDD is called a random dictatorship if (RRD(O)) w for all oF E ASS).
By definition, RD satisfies both proposed axioms in their strongest forms: a-PPA with a = for all a E A(S), and PBM with 7 0). The following proposition establishes that o Mn and MLL violate even the weakest forms of these axioms, whereas &RD satisfies all four axioms.
Proposition 3.5. & MB and ML, violate the a-PPA axiom for any a and the PBM axiom for any f.. RRD satisfies all four axioms.
5

---

Published as a conference paper at ICLR 2026
The proof is provided in Appendix Unfortunately, LRD is not implementable by any pairwise preference learning algorithm, since distinct profiles σ1 and 02 may induce identical preference functions p P but different population distributions w * w (see Appendix [F] for an example). Because wº cannot be recovered solely from PS, no mapping from preference functions to policies implement @ Our goal, therefore, is construct preference learning algorithm F that implements PSCF Φ satisfying all four axioms. Table summarrzes the standard PSCFs, their implementability, and satisfaction of the four axioms; see Brandd al. (2022) for additional details.
ALGORITHMIC FRAMEWORK AND THEORETICAL GUARANTEES
4.1 POPULATION DISTRIBUTION RECOVERY FROM PAIRWISE PREFERENCES
In this section, we introduce a preference algorithm F, which implements a PSCF satisfying all four axioms presented the previous section. The framework first estimates the feasible set of underlying population distributions w from given pairwise preferences P. and subsequently constructs policy π closely aligned with the inferred feasible set. We begin with the definition of a feasible population distribution and the characterization of the set of all feasible population distributions.
Definition 4.1. A population distribution w is considered feasible given P, if there exists a profile ASS) such that w w" and P = P Proposition 4.2. The set of all feasible population distributions given P can be expressed as
M
W(P) = {w A()) I(. PM) E P.M s.t. Σ W.P.
Pa(s > y)= Vy E/ (ws), Vie [M]
See Appendix G for the proof. In words, a population distribution w is feasible if and only if there exist group-specific preference functions (Pi,...,PP) Par) such that P is their weighted aggregation, and each P. reflects group of evaluators who unanimously prefer y over all other alternatives.
The exact characterization of the set W(P) is challenging due to the constraints imposed by the set P. We therefore propose tractable polyhedral outer approximation of the set W(P), with the number of constraints growing only linearly with the dimension M.
Definition 4.3. For each [M], define u, == mingevMiut P(y: v). Theorem 4.4. The set of feasible population distributions satisfies
W(P) S W(P) = {u A()) wy S ei [M]].
(4)
maxy P(y mt 1 - P(y' > yt), where y' is the alternative most preferred over yı- Thus, uy represents the remaining population share after excluding those who prefer y/ yi. Thus, w; cannot exceed this value, the w; proportion of evaluators would always report yı as their preferred option. The tightness of the outer approximation is further discussed in Appendix I, and the relation between Theorem 4.4 and Tatli et al. (2024) is examined in Appendix[] Since w" is not identifiable from pairwise comparison data, perfect proportional alignment (i.e., (((a) for all a E A(S)) fundamentally unattainable. Moreover, even achieving uniform guarantee (a) > 2/M for all is impossible for any preference learning algorithm (see Appendix K). This motivates designing algorithms that achieve a-PPA with
The proof is given in Appendix H To provide intuition, note that i,
the largest possible
4.2 PROPOSED ALGORITHMIC FRAMEWORK WITH AXIOMATIC GUARANTEES
Given a polyhedron (P)), our goal is to select a policy и that guarantees the proportional alignment to all w E W(P) To this end, we propose to assign probabilities to alternatives in proportion to the derived upper bounds u,
In the literature, the class of implementable PSCFs is often referred the C2 class Fishburn 1977

---

Published as a conference paper at ICLR 2026
Definition 4.5. The preference learning algorithm F* maps preference function P to the policy π (v,)
U, vi e [M].
(5)
21=1 4j
Let t denote the PSCF implemented by F".
This construction adopts a conservative strategy for handling uncertainty in w by assigning probabil- ities proportional to the most conservative estimate of each wy" By doing so, the algorithm minimizes the worst-case misalignment caused by the inevitable information loss from pairwise comparisons. Formally, solves max)) min we(() ||>////...
We first establish the foundational axiomatic guarantees the proposed framework.
Theorem 4.6 (Monotonicity & Pareto efficiency). The proposed PSCF o* satisfies the monotonicity and the Pareto efficiency.
The proofs are provided in Appendix L Next, we show that 4 satisfies the a-PPA axiom. The following lemma establishes that the ratio between the resulting policy and the true population share is lower bounded by the inverse of the total sum of the upper bounds u4-
Lemma 4.7. For any profile a E ASS), the policy x 4 (o) satisfies
(ys) w
M
Uj
Vi E [M].
(6)
The next lemma shows that this lower bound depends on the number of non-dominated alternatives: Definition 4.8 (6-dominated alternative). For any 5 E [o. 1]. an alternative y E is said to be 5-dominated in profile a if there exists an alternative y' E EY'
Y {y} such that P (y' > v) 8. Lemma 4.9. Let w and w denote the largest and second-larged elements of w", respectively. Consider any 8 10. and let Ng be the number of alternatives that are 6-dominated in profile a. Then the lower bound in equation lies within the range [a(a), 1]. where
()(1-))(11) a(o)= == [[Ng 1) - w0 + (1-w 2) + (M - Ng (1) -
(7)
See Appendix M foo for the proofs. Combining both Lemmas, obtain the following a-PPA guarantee:
Theorem 4.10 (a-PPA). The PSCF x" satisfies the a-PPA axiom with a defined in equation
Lemma 4.7 suggests that the actual alignment performance improves as ΣM Uj approaches 1. This typically occurs when the number of non-l-dominated alternatives small. Notably, when there are only two non-l-dominated alternatives, substituting No = 2 and w + n = into equation y (<))) = 1, implying the perfect PPA in such cases. Moreover, when there exists single dominating group, meaning (w..1 =² approaches (1, 0), then a(r) also approaches 1. M
Importantly, because Σ Σ uy can be computed directly from a given preference function P, the alignment accuracy of the resulting policy can be evaluated at test time. In Appendix N we present additional bound analysis under a random-ranking model with a practically relevant level of variance. Specifically, we report the average lower bounds from Lemma 4.7 and Lemma 4.9 showing that both yield meaningful guarantees.
Finally, we present the population-boundee bounded manipulability of the proposed method. Theorem 4.11 (r-PBM), Let = 45 (0(t)) denote a policy resulting from single-group manipulation by group Gg. Then, the following inequality holds:
n' ('(c)) <
2A + 1 - u Σ (wg. + +
Ukk
(8)
Thus, the PSCF 4* satisfies 7-PBM with (7, (7) = 11/2,1/2).
The proof is provided in Appendix O Note that (71, )22== (1/2, 1/2) represents worst-case bound. The actual manipulability for each group is more tightly bounded by u + 1 wg). For instance, if ug < 1/2 and w 1/2, then x' (3%) < 1/2. This indicates that non-majority group cannot elevate their preferred alternative to majority status through manipulation. In addition, the above result can be interpreted weaker form of strategyproofness (see Appendix P for details).
+1-wg)
7

---

Published as a conference paper at ICLR 2026
0.80
0.6
0.06
0.75
Win Rate
0..04
0.70
PPA
PBM
0.2
0.02
0.65
0.60
0.0
0.00
10
2
10
10
10
10"
10
10' B (log scale)
10"
B (log scale)
Proposed (Win Rate)
Proposed (PPA)
Proposed (PBM)
RLHF (Win Rate)
RLHF & NLHF (PPA)
RLHF (PBM)
NLHF (Win Rate)
NLHF (PBM)
Figure 2: Tabular experiment results (Section 5.1 for F8, FRL and FNL, Left: win rate (left axis, blue) and PPA level (right axis, orange). Right: PBM level (policy gain through manipulation).
4.3 BALANCING PPA AND CONDORCET CONSISTENCY
While F* and x' are deliberately designed to satisfy PPA, one may still wish to incorporate majority- based principles such as Condorcet consistency. However, is impossible for any method to simultaneously satisfy both a-PPA and Condorcet consistency.
Definition 4.12 (Condorcet consistency). A PSCF + satisfies Condorcet consistency if, for any profile with Condorcet winner y", ((a)(v))=1
Proposition 4.13. No PSCF simultaneously satisfy a-PPA and Condorcet consistency.
See Appendix for the proof. To balance axioms, we propose softmax-relaxee algorithm F (and its corresponding PSCF ) by modifying F* as follows:
We exp(3u;)
x (y:) Σラ= uy exp(Bu,)
E E [M].
The parameter > 0 controls how sharply the policy concentrates alternatives with higher u, values. When 3 = O, the algorithm reduces to the original F". As 3 > oc, the policy becomes deterministic and converges xyy") 1, where y" arg [[ Wy. This limiting t is the well-known minimax Condorcet method (Kramer 1975). which satisfies Condorcet consistency (see Appendix for the proof).
Proposition 4.14. Th satisfies Condorcet consistency.
The softmax relaxation offers smooth trade-off between a-PPA and Condorcet consistency, con- trolled by the parameter 3. We analyze the theoretical behavior of intermediate / values Ap- pendix(S) SS] and empirically demonstrate the effects of varying 3 in Section Additionally, Appendix discusses the connection to pairwise majority consistency (PMC) (Ge et al.) 2024), which imposes stronger consistency requirement, ensuring the entire policy ranking aligns with majority preferences.
5 EXPERIMENTS
TABULAR EXPERIMENT: MOVIE RECOMMENDATION
Datasets and experimental setup. To validate theoretical findings, evaluate the framework on a movie recommendation task using 1,297 evaluator rankings over 20 movies from the Moviel Lens 1M dataset (Harper & Konstan, 2015). In each episode, sample 10 pairwise comparisons i.i.d. from the true preference function p and train 8 alongside two baselines, ERLL and ENL
We report averages and standard deviations over 50 episodes on three metrics: (i) win rate against uniform policy, Egg, (a)))),)) [Pa (y1 ž- y2)]. where U is the uniform distribution over Y (ii) PPA level, (a)r = min.e[M] 7 (v:)/w?, and (iii) PBM, the average policy gain from a single group's strategic manipulation.
8

---

Published as a conference paper at ICLR 2026
Table 2: Win rate and PPA level a(o) across datasets and algorithms
Dataset
Category
Metric
B 3=0
B == 10-1
3 10-2
3 10
DPO
Win rate
0.6157
0.6880
0.6961
0.8429
0.8566
Synthetic
Color
PPA (a)
0.0883
0.0235
0.0183
0.0003
0.0000
Win rate
0.7613
0.7610
0.7634
0.7636
0.7697
Expertise
PPA (ai)
0.1428
0.1418
0.1392
0.1273
0.1321
Alpaca-GPT4
Win rate
0.8398
0.8432
0.8425
0.8530
0.8478
Style
PPA (a:)
0.5012
0.4197
0.3637
0.3635
0.3786
Results and discussion. As shown in the left panel of Figure 2 RL.F and NLHFF achieve high win rates of 0.7784 and 0.7712, respectively, but both yield PPA level of 0. For our proposed algorithm F1,, we observe the expected trade-off:: as 3 increases, the win rate rises from 0,5987 to 0.7784, while the PPA level decreases from 0.4869 to O. These results confirm our theoretical prediction of each algorithm's 's behavior, Additionally, the average value of u4 across i was 0.1892, suggesting that the set (P)) in equation provides a meaningfully tight estimate of w" in our method.
Regarding PBM, the average gain was calculated as 0.0611 for RLHF, 0.0124 for NLHF, and 8.896 x 10-1 when B 10 Overall, pd outperforms the baselines when 3 S 10", indicating that our proposed algorithm significantly reduces susceptibility to manipulation and supports its robustness guarantce.
5.2 LARGE-SCALE EXPERIMENT: INSTRUCTION-TUNED LLMS
Datasets and experimental setup. We next evaluate the algorithm in high-dimensional settings with function approximation by fine-tuning the Qwen2.5-3BBInsttuu model (Yang et al. 2024). For synthetic dataset, construct 10 questions asking evaluators which color they prefer, with 10 candidate colors as possible responses. The true rankings of 1,000 evaluators are generated from randomly sampled rewards, and 10* pairwise comparisons
drawn i.i.d. from Pa We next the algorithm on the A Apaca-GPP4 dataset (Peng et al. 2023), which contains 52k prompts. Following prior work (Jang et al.) 2023] Chakraborty al. 2024), we consider group categories (expertise and style) and sample one pairwise comparison per prompt using GPT-4.1 (Achiam et al. (023).
Further details on data generation and hyperparameters are provided Appendix U
For both datasets, we evaluate two metrics: (i) the win rate against a reference policy (the pretrained model), Err, w))_(a)0,,5500 [P' )P' > //2 =)], and (ii) the PPA level ((a), comparing the results with DPO as the baseline. To estimate the output policy (i.e., the group distribution of generated responses), we used response logits directly for the synthetic dataset, and group classifications from the annotation model (GPT-4.1) 1) for the AppacaGGPT4 dataset. The specific training algorithm is described in Appendix and the full experimental code is included in the supplemental material.
Results and Discussion. Table 2 presents the win rate and PPA level (x(r) across datasets and algorithms. On the synthetic dataset, observe clear trade-off between win rate and PPA, confirming that 3 effectively controls this balance and validating the algorithm's 's effectiveness in high-dimensional settings. For the Alpaca-GPT4 dataset, the trade-offis is present but less pronounced, largely because group distributions are inferred via an annotation model (GPT-4.1), which introduces noise and obscures the effect of 8. In contrast, the synthetic dataset allows direct computation from response logits, enabling more precise estimates. These results suggest that a small synthetic dataset can be used evaluate model's PPA level and tune 33 to reach a desired target.
We highlight several practical considerations for deployment. First, our two-phase function approxi- mation approach (learning u and т). has computational cost comparable to RLHF and higher than DPO, suggesting the need for direct policy-optimization methods. Second, accurately evaluating PPA levels in LLMs remains an open challenge, extending beyond the two approaches we consider, namely logit-based estimation and group classification. As this paper primarily introduces the theo- retical framework with supporting experiments, our findings should be viewed as initial evidence of scalability, with further algorithmic and evaluation advances expected strengthen these results.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION AND FUTURE DIRECTIONS
This paper introduces novel preference-leaminng framework that aligns policies proportionally with population distributions inferred from pairwise comparison data. We believe this framework offers new perspective on alignment algorithms by shifting the focus beyond the conventional emphasis on win rate. Furthermore, our work strengthens the connection between preference learning and social choice theory by implementing new class of probabilistic social choice functions, extending beyond standard rules such as maximal Borda and maximal lotteries. Future research will aim to extend the framework to incorporate lower-ranked preferences and to develop more efficient algorithms for high-dimensional environments.
ACKNOWLEDGMENTS
This work was supported by the Office of Naval Research grant N000142512296, Jiawei Zhang was supported by the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin--Madionn with funding from the Wisconsin Alumni Research Foundation. The authors acknowledge the MIT Office of Research Computing and Data for providing high performance computing resources that contributed to the results reported in this paper.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXhv:2333.08774 2023.
Jadie Adams, Brian Hu, Emily Veenhuis, David Joy, Bharadwaj Ravichandran, Aaron Bray, Anthony Hoogs, and Arslan Basharat. Steerable pluralism: Pluralistic alignment via few-shot comparative regression. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pp. 15-25, 2025.
Stéphane Airiau, Haris Aziz, Ioannis Caragiannis, Justin Kruger, Jérôme Lang, and Dominik Peters. Portioning using ordinal preferences: Fairness and efficiency. Artificial Intelligence, 314:103809, 2023.
Haris Aziz and Barton E Lee. Proportionally representative participatory budgeting with ordinal preferences. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp.
111-5118, 2021.
Haris Aziz and Nisarg Shah. Participatory budgeting: Models and approaches. In Pathways Between Social Science and Computational Social Science: Theories, Methods, and Interpretations, pp- 215-236. Springer, 2020.
Haris Aziz, Markus Brill, Vincent Conitzer, Edith Elkind, Rupert Freeman, and Toby Walsh. Justified representation in approval-basse based committee voting. Social Choice and Welfare, 48(2):461-485, 2017.
Haris Aziz, Barton Lee, and Nimrod Talmon. Proportionally representative participatory budgeting: Axioms and algorithms. In Proceedings of the 17th International Conference Autonomous Agents and Multiagent Systems (AAMAS 2018), pp. 23-31. Association for Computing Machinery, 2018.
Erdem BIyik, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning and optimization. The International Journal of Robotics Research, 6365)66656 684, 2024.
Henry David Block and Jacob Marschak. Random orderings and stochastic theories of response.
Cowles Foundation Discussion Papers, 1959.
Florian Brandl, Felix Brandt, and Christian Stricker. An analytical and experimental comparison of maximal lottery schemes. Social Choice and Welfare, 58(1):5-38, 2022.
10

---

Published as a conference paper at ICLR 2026
Felix Brandt. Rolling the dice: Recent results in probabilistic social choice. Trends in computational social choice, pp. 3- 3 3-26, 2017.
Thomas Kleine Buening, Jiarui Gan, Debmalya Mandal, and Marta Kwiatkowska. Strategyproof reinforcement learning from human feedback. arXiv preprint arXiv:2503.0956. 2025.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. Maxmin-rlhf: Aligmment with diverse human preferences. In Interna- fional Conference on Machine Learning, pp. 61 16-6135. PMLR, 2024.
Daiwei Chen, Yi Chen, Aniket Rege, and Ramya Korlakai Vinayak. Pal: Pluralistic alignment framework for learning from heterogeneous preferences. arXiv preprint arXiv: 2404244460044 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Mosse, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. Position: Social choice should guide ai alignment in dealing with diverse human feedback. International Conference Machine Learning, pp. 9346-9360. PMLR, 2024.
Jessica Dai and Eve Fleisig. Mapping social choice theory to rlhf, arXiv preprint arXiv:2404.13336 2024.
Michael Dummett. Voting procedures. Oxford University Press UK, 1984.
Soroush Ebadian, Anson Kahng, Dominik Peters, and Nisarg Shah. Optimized distortion and proportional fairness in voting. ACM Transactions on Economics and Computation, 12(1):1-39, 2024.
Peter C Fishburn. Condorcet social choice functions. SIAM Journal on applied Mathematics, 33(3): 469 489, 1977.
Peter C Fishburn. Probabilistic social choice based on simple voting comparisons. The Review of Economic Studies, 51(())683-6)) 692,, 1984.
Luise Ge, Daniel Halpern, Evi Micha, Ariel Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. Axioms for ai alignment from human feedback. Advances in Neural Information Processing Systems, 37, 2024.
Allan Gibbard. Manipulation of voting schemes: general result. Econometrica: journal of the Econometric Society, pp. 587-601, 1973.
Daniel Halpern, Evi Micha, Ariel D Procaccia, and Itai Shapira. Pairwise calibrated rewards for pluralistic alignment. arXiv preprint 256.022506.06298, 2025.
Shugang Hao and Lingjie Duan. Online learning from strategic human feedback in Ilm fine-tuning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2025.
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiss, 5(4):1-19, 2015.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11 11564, 2023.
Cheol Woo Kim, Jai Moondra, Shresth Verma, Madeleine Pollack. Lingkai Kong, Milind Tambe, and Swati Gupta. Navigating the social welfare frontier. Portfolios for multi- objectiv reinforcement learning. In International Conference Machine Learning, pp. 10663.. PMLR, 2025.
11

---

Published as a conference paper at ICLR 2026
Kihyun Kim, Jiawei Zhang, Asuman E Ozdaglar, and Pablo Parrilo. A unified linear programming framework for offline reward learning from human demonstrations and feedback. International Conference Machine Learning, pp. 24694-24712. PMLR, 2024.
Gerald H Kramer. dynamical model of political equilibrium. Cowles Foundation Discussion Papers, 629, 1975.
Andras Kupesik, David Hsu, and Wee Sun Lee. Learning dynamic robot-to-human object handover from human feedback. In Robotics Research: Volume 1, pp. 161-176. Springer, 2017.
Roberto-Rand Rabeert-Raffel Maura-Rivero, Mare Lanctot, Francesco Visin, and Kate Larson. Jackpot! alignment as maximal lottery. arXiv preprini arXiv:2501.19266. 2025.
Abhilash Mishra. Ai alignment and social choice: Fundamental limitations and policy implications. arXiv preprini arXiv:2310.1600 2023.
Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, al. Nash learning from human feedback. In International Conference on Machine Learning, pp- pp- 36743- 36768. PMLR, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Chanwoo Park. Mingyang Liu, Dingwen Kong, Kaiqing Zhang, and Asuman Ozdaglar. Rlhf from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254, 2024.
Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Dominik Peters, Grzegorz Pierezyúski, and Piotr Skowron. Proportional participatory budgeting with additive utilities. Advances in Neural Information Processing Systems, 34:12726-12777 2021.
Ariel Procaccia and Jeffrey S Rosenschein. The distortion of cardinal preferences in voting. In International Workshop on Cooperative Information Agents, pp. 317-331. Springer, 2006.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, (6.53728-53744 2023.
Shyam Sundhar Ramesh, Yifan Hu, lason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization reward-free rlhf. 2024.
Advances in Neural Information Processing Systems,
Zhekun Shi, Kaizhao Liu, Qi Long, Weijie Su, and Jiancong Xiao. Fundamental limits of game-theoretic IIm alignment: Smith consistency and preference matching. arXiv preprint arXiv:2505.20627, 2025.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in rlhf. In The Twelfth International Conference Learning Representations, 2024.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: A roadmap to pluralistic aligmment. In International Conference on Machine Learning, pp. 46280-46302. PMLR, 2024.
Ermis Soumalias, Michael J Curry, and Sven Seuken. Truthful aggregation of Ilms with an application to online advertising. arXiv preprint arXiv:2405.05905 2024.
12

---

Published as a conference paper at ICLR 2026
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 13:3008-30214 2020.
Haoran Sun, Yurong Chen, Siwei Wang, Wei Chen, and Xiaotie Deng. Mechanism design for Ilm fine-tuning with multiple reward models. arXiv preprini arXiv:2405. 2024.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. In International Conference on Machine Learning, pp. 47345-47377 PMLR, 2024.
Gokcan Tatli, Yi Chen, and Ramya Korlakai Vinayak. Learning populations of preferences via pairwise comparison queries. International Conference on Artificial Intelligence and Statistics, pp. 1720-1728. PMLR, 2024.
Jiancong Xiao, Zhekun Shi, Kaizhao Liu, Qi Long, and Weijie J Su. Theoretical tensions in rlhf: Reconciling empirical success with inconsistencies in social choice theory. arXiv preprint arXiv:2506.12335 2025.
Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, and Bo An. Prefrec: Recommender systems with human preferences for reinforcing long-term user engagement. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2874-2888 2884, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu. Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.151/5 15111 2024.
Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, and Junjie Hu. No preference left behind: Group distributional preference optimization. In The Thirteenth International Conference on Learning Representations, 2025.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Hanze Dong, Nan Jiang, and Tong Zhang, Online iterative reinforcement learning from human feedback with general preference model. Advances Neural Information Processing Systems, 77781773-88188 2024.
Huiying Zhong, Zhun Deng, Weijie J Su, Zhiwei Steven Wu, and Linjun Zhang. Provable multi-party reinforcement learning with diverse human feedback. arXiv preprint arXiv:2403.05000 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint frXiv:1909008593 2019.
13

---

Published as a conference paper at ICLR 2026
A
NOTATION
We summarize the mathematical notation used in the paper.
Symbol
Description
Rankings, profiles, groups, and preferences
[M]
The set of integers {1, ,1. ,....,,,,,,
V (y). "yl....;MM.
The set of M alternatives.
A()))
Probability simplex over finite set
S
The set of all rankings (permutations) over y A ranking, where r3:) = k means is ranked k-th. A profile, i.c., population distribution over rankings. Proportion evaluators who adopt ranking r.
r PES
E A(S)
O, E 1]
Gk
Group k set of rankings where yk is ranked first. Population share of evaluators whose top choice yk- Sub-profile of group G3 (evaluators who rank yk first). policy, i.e., probability distribution over alternatives. Preference function, Ply 3²) is the probability y is preferred y/. Preference function induced by profile
"g E 10 Ok E A(S)
T A())
PeT P
P EP
PR. P
Group-specifer preference function for G
P
Set of all preference functions induced by some profile in A(S).
Preference learning algorithms, PSCFs, and axioms
F:P. - A())
Preference learning algorithm, mapping a preference function to policy.
::A(S) + A())
Probabilistic social choice function (PSCF), mapping profile to policy.
FRI, MR
RLHF algorithm and its PSCF (maximal Borda rule).
FNI, ML
NLHF algorithm and its PSCF (maximal lotteries).
B(y)
Borda score: B(y) Eres o, (M r(y)).
a(o) E R
Strength of amppulation-roporrionn
alignment (PPA) guarantee.
(2) E TR2 %
Parameters characterizing population-boundde manipulability (PBM). Single-group manipulated profile of (group k perturbs only its sub- profile).
u [0,1]
u, == mings/y. P(y > y). upper bound on feasible population share for V4-
W(P)
Set of feasible population distributions consistent with preference func- tion P.
W(P)
Polyhedral outer approximation of W(P) (via w, S us constraints). Dominance threshold (used in â-domination definition).
E [0, 1]
Ng
Number of alternatives not 5-dominated under profile
wf. w
Largest and second-largest elements of w
Condorcet winner satisfying P(y* y) > 2 for all $ v" Proposed (baseline) algorithm/PSCF using u, with π T(y) xx ug. Softmax-relaxed algorithm/PSCF with concentration parameter / 2 0.
F*, x
Offline learning algorithm with function approximation
X
Context (e.g., prompt or state) and context space. Offline dataset of pairwise comparisons (y" preferred to y²).
D {I v." N p(x), TT(y
Context (prompt) and query data distribution.
Selector model used to form Function classes for μ. and m.
Fus Fr P,p,
Empirical estimate of P, pl, and u.
Softmax policy constructed from ū and final estimated policy
14

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RELATED WORK
In this section, we discuss recent work that aims to proportionally represent the diversity of human preferences.
Limitations of the BT model. Recent studies have highlighted limitations of the standard RLHF approach under BT model assumption, which fails to capture the multifaceted and sometimes con- flicting nature human preferences. For example, Kim et al. (2024) demonstrated that the standard MLE algorithms under the BT model can become unstable, particularly in the presence of evaluators exhibiting greedy behavior. They proposed to address this limitation by estimating a set of feasible reward functions without relying specific modeling assumptions. Additionally, (Siththaramjan et al.) (2024) established theoretical equivalence between RLHF and the Borda voting rule, showing that the optimized rankings from standard methods frequently violate majority preferences. To address this issue, they introduced a distributional approach incorporating hidden context variables to address diverse evaluator preferences. Furthermore, Ge et al. (2024) analyzed reward optimization methods under parameterizations, revealing their inherent violation of fundamental axioms such as Pareto efficiency. They proposed novel algorithm explicitly designed to satisfy these axioms.
Approaches from social choice theory. Parallel research efforts have explored unbiased aggrega- tion of heterogeneous human preferences, grounded in social choice theory. Chakraborty et al. (2024) formally proved the impossibility of equitably aligning single-reware models across diverse evaluator groups, and proposed learning reward mixtures using the EM algorithm followed by maximizing the minimum utility across subpopulations. Additionally, Zhong et al. (2024) conducted a rigorous analysis of multi- group reward learning under various social welfare criteria, such as Nash, utilitarian, and Leximin functions, and provided theoretical alignment guarantees. Park et al. (2024) proposed a probabilistic opinion pooling function that directly aggregates multiple probabilistic models into single policy, as well as personalized algorithms that output individualized policies after estimating confidence sets. [Shi et al. (2025) analyze the theoretical limits of NLHF, showing that exact prefer- ence matching is generally impossible, highlighting intrinsic limitations of this paradigm. Concurrent work by Xiao et al. (2025) is closely related to our work. They investigate the tension between RLHF's empinical success and its incompatibility with social choice axioms (PMC and Condorcet consistency), showing that RLHF can satisfy them under a practical assumption about preference labeling. Moreover, they propose a new axiom, group preference matching, which requires the policy to reproduce group-level preference distributions in proportion to their population weights. However, they do not provide an algorithmic framework that satisfies this axiom.
Proportional representation in voting systems. The concept of proportional representation has been extensively studied through an axiomatic lens within voting systems. A foundational axiom in multi-winner voting systems is proportionality for solid coalitions (PSC), which dictates that any solid coalition (a group of voters who agree on their preferred set of winners) must be guaranteed a number of elected candidates proportional to its population size (Dummett 1984). Building on this, work in approval-base voting introduced justified representation (JR) and its stronger variant, extended justified representation (EJR), which ensure that every cohesive group (voters who approve the same set of candidates) receives proportional representation (Aziz et al.) (017). Proportional representation has also been widely applied to participatory budgeting (PB) (Aziz & Shah 2020), focusing on axiomatic methods for distributing funds among public projects under a budget constraint. More recently, Ebadian et al.] (2024) analyze distortion bounds of voting rules (Procaceia & Rosenschein 2006) under a proportional fairness objective. However, the literature defining these proportional representation notions typically assumes either approval-based multi-winner elections (Aziz et al. 2018) or access to full preference information such ordinal rankings Aziz & Lee| (2021); Peters) ct al. (202T) : Airiau et al. 2203)); [Ebadian et al.] (2024). This stands in sharp contrast to our approach, which operates under the minimal assumption of pairwise comparison data and seeks a probabilistic choice distribution over candidates, rather than a fixed
Pluralistic alignment. Emerging research on pluralistic alignment seeks to reflect diverse per- spectives in AI systems, with particular focus LLMs. Sorensen et al. (2024) outlined three complementary frameworks for pluralistic aligmment: Overton pluralism, which captures the range of reasonable responses; steerable pluralism, which allows models to adapt to particular attributes; and
15

---

Published as a conference paper at ICLR 2026
distributional pluralism, which aligns model outputs with population-level distributions. Chen et al. (2024) introduced a framework that modeled heterogencous human preferences from the ground up using the ideal point model and mixture modeling. Yao et al. (2025) proposed group distribu- tional preference optimization (GDPO), method that aligns models with the group preferences by estimating the underlying belief distribution and conditioning responses those beliefs, ensuring representation of both majority and minority views. Adams et al. (2025) developed a steerable pluralistic alignment algorithm, enabling models to adapt to individual preference profiles through few-shot comparative regression across fine-gre fine-grained attributes. While these approaches show promise, they generally rely on explicit group identification, restricting their applicability scenarios where group labels are unavailable or difficult to determine. In contrast, our work does not require explicit knowledge of evaluator groups. Instead, we infer population distributions directly from pairwise comparison data and align policies accordingly. In parallel with our work, the problem of recovering a distribution over reward functions from aggregate pairwise comparisons is formalized under the pairwise calibration framework (Halpern et al. (025). Our approach can be interpreted as selecting structured solution within this inherently ill-conditioned problem, further constrained to satisfy the proportional alignment and robustness properties that target.
C EQUIVALENCE OF BT-MLE REWARDS RANKING AND BORDA RANKING
Proposition C.I. Let r* E R.M be a maximizer of the likelihood function
L(r) :- Σ (" uj) log
+P > 11) log
E
)
(10)
e 6
eth
Then, the ordering of alternatives induced by r" is identical to the ordering induced by the Borda score B of a. Formally, for any i,j [M].
r > r
B(y:) > B(y,).
(11)
Proof. The gradient of L(r) with respect to r, is given by:
aL(r)
Σ
(y: mi) - signoid((- - -ry)]
(12)
3r;
jtt
where sigmoid(z) (x: 1/(1 + e=). At the optimal solution r", the first-order condition requires that
Σ j
[P" (yx uj) sigmoid(r} r3]] 0.
(13)
Now, consider two distinct alternatives and k, and suppose that r rī. Since the sigmoid function monotonically increasing, for any j # i, k. we have sigmoid(r; r5) sigmoid(rī - rr)) and also sigmoid rit) sigmoid(rt - r.). From the first-order conditions optimality, we have:
pa (ya u5) Σ sigmoid(r; rj) and Σ 14
pa PTy > 4/) Σ sigmoid rj). (14)
Since r > rf. it follows that
Σ sigmoid(r; -r3) > Σ
sigmoid(rī - r3))
(15)
jfz
17k
Therefore, we have
Σ pa (yt 91) Σ
po (yk 43).
(16)
Jt1
j
By definition, Pa (y: > vj) Ereso, o, (((())<((()))- Substituting this into the inequality above,
we get
ΣΦ, 0,
Σ (r(w))rr()))
or,
(17)
rES
1#k
16

---

Published as a conference paper at ICLR 2026
Recall that the Borda score is defined as B(y) Eres or (M r(y)). we can rewrite the inner sums in the inequality as:
Σ
(M- (rtw)rr(n)) (M -1 1 - 1) = M -(()),
(18)
j+1
and similarly,
Σ (rO)))))))))))) =M- - -r(ss. jzk
(19)
Thus, the inequality becomes
Eor (M(i)) Σ Jo. (M r(y))),
(20)
rES
rES
which is equivalent B(y)) B(yk).
For the converse, assume (y;) > B(y))- Following similar steps in reverse, this implies
Σ
Pd (yt > yU)
P
(yk
y3),
(21)
J
j
which leads
Σ sigmoid - -r3) > ftt
Σ sigmoid rr))-
(22)
jØk
This inequality can only hold if r rk Therefore, we have shown that r3 > rj = B(wi) Byy)). completing the proof.
D FUNDAMENTAL AXIOMS: MONOTONICITY AND PARETO EFFICIENCY
In this section, we present the definition of two fundamental axioms in social choice theory: mono- tonicity and Pareto efficiency. For detailed discussions of these axioms, we refer readers to Brandt (2017); Ge al. (2024).
Definition D.1 (Monotonicity). A PSCF t satisfies monotonicity if, for any alternative E y, improving its ranking in a profile without changing other relative rankings cannot decrease its probability in the resulting policy. Formally, if profile o' is obtained from by improving the ranking of y some r E S with a, > 0, then (v) Z (v))
Definition D.2 (Pareto efficiency). A PSCF & satisfies Pareto efficiency if, whenever an alternative is ranked above y' in every ranking r with nonzero population share, the resulting policy assigns least as much probability y as to y'. Formally, if r(y) r(y') for all with o, > 0, then s ((()) (v) 2 (v')
E PROOF OF PROPOSITION 3.5
We first demonstrate that 4MB and MLL violate a-PPA. Consider a preference profile a with the following characteristics: (i) The population share of each group nearly identical, with G₁ having population share w/ that is greater than the average, and G2 having a population share wz that is e less than the average. (ii) Within each group Gt, there indifference between any two alternatives other than yk- That is, Pg > yj} 1/2 for all i.j * k. Given this profile, we will show that for any > O, both RLHF and NLHF yield a deterministic policy that selects y/-
The population distribution and pairwise preference function satisfy
Wh
M +
M'M" M and P (ys > 93)
Vi.j # k.
(23)
M
2'
Then, the aggregated pairwise preferences Pd are computed as follows:
* PP1 > 32) = +
. P y) 1 for any y # y, 92
17

---

Published as a conference paper at ICLR 2026
Pd (y2 y) Po > y') for any y. v/ F 11:11
for any + y1, U2
Under this profile, for any 0. both BBB and TMLL result in policy where m(n) 1, and т (y4) = 0 for any i + 1. This implies that Tty) [ = for any i * 1, which violates a-PPA for any a > 0.
Next, we show that LLLL violates 7-PBM using the profile described earlier with M = 3. The aggregated preference function pa can be represented by the following matrix:
E
P
1
(24)
2
f
1tc
Now, suppose that group G3 manipulates their sub-profile from (3(1 > 12) = 1 to P3 (1 >
y2) = O. Then, the resulting manipulated aggregated preference function pa is calculated as:
1+t
2
po'
(25)
1ffe
7
2
2
M.M yields stochastic policy that depends on the value of e. For example, if 1/12, the resulting
policy is ı = [a. 2 However, as approaches 0, the resulting policy converges to [0, 0, 1]. This shows that n' r' + while wg 1/3, thus demonstrating that there exists no " > 0 for which MT. satisfies 7-PBM.
To show that MB violates PBBM, consider the case with M = 3 where the profile consists of the following three groups of evaluators:
{111 1/2 ys) x 0.30, (92 /1 Vs) 0.45, (ya 31 > y2) 0.25),
(26)
where (i) > 1/2 > ya) represents a ranking and "x0.30" indicates that = 0.30. Then,
the Borda scores are calculated as [1.3, 1.20, 0.5). Thus, (M) n, where (
Next, suppose the second group strategically misreports their preference from (32 > > ya) to (y2 1/3 > y1). Then, the Borda scores are calculated as B' = [0.85, 1.2, 0.95]. The resulting policy is then n' (12) 1, with the population share of the second group being wg = 0.45. This example demonstrates that there exists no 71 for which MB satisfies -BBB.
Next, we show that RD satisfies all four axioms. RDD satisfies monotonicity because improving ranking of y cannot decrease the number evaluators whose top choice y. In addition, &R satisfies Pareto efficiency because if r(u,) < r(ıx) for all E S with o, > 0, then we have wgg = 0 and (o) = 0. Additionally, F.D satisfies a-PPA with a(a) for all a by its definition, and also satisfy 7-BMM with (71, 72) (1, 0) because each group G& cannot increase w by manipulation.
F PROOF OF THE NON-IMPLEMENNTABIIN OF LRD
Suppose that RRD can be implemented by preference learning algorithm FRD, Let M = 3, and consider two preference profiles, 01 and d2, defined follows:
{ > 1/2 > y3) x 1 /3, (U2 > > y3) 1/3, (ys > > 92) x 1/3), σ2 {11 1/2 93) 2 2/3, (ys U2 > 31) 1/3).
(27)
Both of these profiles induce the same aggregated preference function pa P P where
pa
(28)
Therefore, the preference learning algorithm FRD would produce the same policy for both and 02. However, according to the definition RRD we have &RD (o, = [1/3, 1/3, 1/3] and RRD 2/3, O, 1/3), which are different policies. This implies that FRD does not implement RDD which contradicts our initial assumption. Therefore, & RD is not implementable by preference learning algorithm.
18

---

Published as a conference paper at ICLR 2026
G PROOF OF PROPOSITION 4.2
First, consider any feasible population share w given a preference function P. By Definition 4.1 there exists a profile such that w w" and P = po. Then, the group-specifie preference functions (Pī,.... P(r) that constitute pa, satisfy the condition in equation which implies that w E W(P).
Next, consider any w E W(P). By the definition of W(P). there exist (Pi...., PM) E PM such that P- = Σx= k=1 PkP where each Ps: satisfies P& > y) there exists a profile oy that induces Px. such that Pk P Now, if we consider an aggregated profile o == Ek=1 Wy.0% by combining these group profiles with the corresponding weights, then the preference function of will be p Ek= Therefore, w is feasible population distribution given P.
for all * yk- Since Ps: E P.
k=1 wyPPt =Ex Ek=l PtP P and also w"
H PROOF OF THEOREM 4.4
Consider any w E W(P). Then, there exists (P1...., .M)) E PM such that P = Ex= Wy Pk. Fix an index i E [M]. For any y y {u we have
P(y > y) = wy Pi(ui + y) +
Σ lipt
wk PA > v) 2 wi
(29)
since P3( > y) Taking the minimum over y y {y,) yields
min (
P(u > y) 2 Wt:
(30)
which implies w; I us Vi [M] and W(P). Therefore, W(P) S W(P).
I ADDITIONAL REMARKS ON THE TIGHTNESS OF THE OUTER APPROXIMATION
The gap between the true feasible set W(P) and its outer approximation W(P) arises from our profile assumption, namely that each evaluator has strict and complete ranking. To illustrate this point, we show that W(P) provides a tight approximation (i.e., W(P) = W(P)) under an extended profile setting. Consider extended profile setting in which each group G2 is allowed to provide pairwise comparison data according to its own preference function Px. subject only to the skew- symmetry constraint P& (3: > yj) + PP y:) 1 for all y/s Dj Y, and the unanimity constraint Pa(8a > y) = 1 for all y E y. In this case, the set P is defined as
= (P P(y v/) P(y > v) = V..y e).
(31)
We show that W(P) S W(P) also holds under this setting. Consider any w E W(P). By assumption, satisfies w. S u mingey (x.) P(u+ y) for all i E [M]. Define each element of (PI..... PM)
as
Pa v,) = P(y: > My) wi
(32)
1 - Wi
for any i,j * k, and let Pa( > y) = 1, P&(y > yx) = 0 for all y E y. Then Pa(y: > yj) e [o, 1] holds because P( 3j)) E [w;, 1 wy] by assumption. The skew-symmetry condition PA(Us > yÿ) + PA(U; > y/) = is also satisfied. Thus, Px E P and P, can be induced by some profile. Finally, the constraint P- = ΣNI wyPr also holds. Therefore, w W(P), implying W(P) C W(P), and hence W(P) W(P).
J CONNECTION OF THEOREM 4.4 AND TATLI ET AL. (2024)
Tatli et al. (2024) studies the recovery of population preference distributions under spatial model. In their framework, each alternative is represented by a feature vector in Euclidean space, and each voter's preferences determined by distances to these vectors (i.e., voters prefer alternatives that
19

---

Published as a conference paper at ICLR 2026
closer in the Euclidean norm). Theorem 4.4 can also be derived in this setting. Specifically, consider sufficiently high-dimensional feature space partitioned into M! regions by (22 hyperplanes, where each hyperplane is the perpendicular bisector the line segment connecting a pair alternative vectors. Then, (Tatli al.) 2024 Proposition 2) shows that it is impossible to recover the full profile a from aggregated pairwise comparison data P Moreover, by summing the inequality in (Tatli et al. 2024 Proposition 4) over the regions corresponding to voters who most prefer each alternative y;. obtain the bound to; S Ui.
K IMPOSSIBILITY OF A UNIFORM GUARANTEE a(o) 2/M
Proposition K.1. No PSCF can be implemented by a preference-learning algorithm while guaran- teeing a-PPA with a constant a(o) > 2/M for all a E A(S))
Proof. Consider any setting in which the pairwise comparison data are completely uninformative. Specifically, suppose that for every pair of alternatives Yt, Vj. the observed probability satisfies P(u: > yバ) = 0.5. Under such maximally ambiguous data, no ceserennce-eaarinn algorithm can distinguish among alternatives, and any algorithm that aims maximize the worst-case a(o) must output the uniform distribution over the M alternatives.
However, the true distribution of evaluators' top choices may in fact be highly non-uniform while remaining perfectly consistent with the uninformative pairwise data. For example, consider profile situation where is ranked first by half of the evaluators and ranked last by the other half. In this case, the corresponding proportionality guarantee is (or) = 2/M. Therefore, achieving a uniform lower bound (o) > 2/M for all a E (SS) is impossible.
which w = [1/2, 1/(2M - 2), 1/(2M 2),.. ...... 1/(2M 2)]. corresponding
L PROOF OF THEOREM 4.6
We first prove monotonicity. Improving the ranking of y: for some evaluator can only increase Pd (yi > y) for any * Vi. and decrease Pd (y > yt). This implies that u, cannot decrease, while Uy for * i cannot increase. Therefore, (1y: u:/ Ma Uz) cannot decrease, establishing monotonicity.
Next, we prove Pareto efficiency. Suppose y is ranked above in every input ranking, i.e., r(y:) r(y,) for all S with or > 0. Then, we have Pa (yı > 3ソ) Eres o, (rox)))))))))) = and also P" (リı yi) 0. Thus, we get uj minger (7 y) = 0. Thus, the resulting policy satisfies 4 (v;) = u, (Eki k= ux) = O. Therefore, 4' (x:) 2 4- (v;) establishing that 4' satisfies Pareto efficiency.
M PROOF OF LEMMA 4.7 AND LEMMA 4.9
Lemma 4.7 follows directly from the fact that w s un for all i E [M], which gives
x (w:)
(33)
w
w 1j=1" 1
Σj=1
Next, we show Lemma 4.9 Let S [M] be the set of indexes for 5-dominated alternatives, where [ = M Ng. Then, for any i I, have
U:
min ME()
pa (y. 41) S Pd (y: v < 8,
(34)
where yy denotes an alternative that 5-dominates y;- Additionally, let & E arg [a wf- Then, for any i + k we have
U4 min (())
Pr > ;j) S Pd yys > yk) $ wh-
(35)
20

---

Published as a conference paper at ICLR 2026
Similarly, let l E arg
[M w/, then have up 1 wf. Combining these results,
M
Σ 1=1
Σ uy +
Σ u1 S wil-ww) wf) + (Ng 1)(1 - w + (M N1 -)). (36)
4. uk +
iEI
i=1 w 1. Combining both inequalities and In addition, since ui 2 ws, we have Ei=1 ui 2 Σ
plugging (wg, wf) = (w :0:22 in, we get the result Lemma 4.9 as follows:
M
(AVg 1)(1- (1 - wo (M N1) 5) 4
1
U4
(37)
N ADDITIONAL BOUND ANALYSIS UNDER A RANDOM- RANKING MODEL
In this appendix, provide additional analysis of the lower bounds in Lemma 4.7 and Lemma 4.9
under a random-ranking model. Our objective is to evaluate whether these guarantees remain meaningful under realistic levels of heterogeneity in the underlying ranking distributions.
Setup. We construct a base ranking by sampling latent rewards for each of the M alternatives independently from (O 1). The rankings of N 1000 evaluators then generated by adding independent Gaussian noise from ((0, 1) to these rewards and sorting the alternatives according to the perturhed values. This procedure induces a ranking distribution with substantial dispersion. For instance, when M = 20, the largest population share maXke 20) W averages approximately 0.25, indicating that the distribution is far from concentrated on single ranking.
Results. Table 3 reports the average values of the two lower bounds, (E u,) and a(o). over 10 independent runs for different values of M, with 8 = 0.7 fixed. In all cases, both bounds are substantially stronger than the naive baseline of 1/M, corresponding to the uniform policy. These findings suggest that the theoretical guarantees remain nontrivial in empirically relevant regimes.
Table Average lower bounds under the randongranking model.
M
20
50
100
E uz) a(o)
0,5553
0.3360
0.2085
0.1427
0.2539
0.1254
0.0570
0,0305
PROOF OF THEOREM 4.
Let o' be the profile manipulated by group Gk, and let x' o0'o')) be the resulting policy. G3 aims to maximize
x' (un) u/k E u
(38)
where u' represents the value of u after the manipulation. To maximize n' (j(e), G₁. will attempt to maximize u/ and minimize Σnfe ut, Since increasing the ranking of ys in their profile increases (or at least does not decrease) the value of u', without increasing the value of Eitk u/, the optimal strategy for G3 is to truthfully report yı as its top choice. In this strategy, have uig uy and the sum Lifk u/ has the following lower bound:
Σ
Σ
3
= wo.
(39)
Substituting this lower bound into equation
38
we obtain
<'(6)) <
uk + wg <
Ub
[wg + 1).
(40)
2
where the final inequality holds if (wg. - Uy. + () - 1) $ 0. which follows from the fact that ug, ug [0, 1).
21

---

Published as a conference paper at ICLR 2026
P WEAK STRATEGYPROOFNES GUARANTEE
In social choice theory, a mechanism is considered strategyproof if participants cannot benefit (i.e., increase their utility) by misreporting their true preferences (Gibbard 1973 regardless of what other participants report. In our preference learning framework, we assume each group G3 's utility is the probability assigned its top choice, represented by (B))). A preference learning algorithm is strategyproof if no participant can improve its outcome by misreporting preferences. However, as noted by [uening et al. (205)), strict strategyproofness typically too restrictive and is not satisfied by the conventional preference learning algorithms (with ex-post efficiency). Our method does not satisfy strict strategyproofness like other methods, but satisfies a weaker form that provides a bounded guarantee the maximum potential gain from strategic misreporting in equilibrium.
Let o' denote the profile resulting from strategical misreporting by all groups, and let x' 4- (a') be the resulting policy. Each group G3 aims to maximize x' (1%)) which involves maximizing u'g and minimizing Eitt u/;
Since improving the ranking of ys in their reported preferences increases (or at worst, does not decrease) the value of t/g without increasing Lifa uy the optimal strategy for G3. is to truthfully report yk as their top choice. Hence, all groups truthfully report their top choice regardless of other groups' strategies, meaning P (yn > y) for all y + yk. where Pj denotes the reported preference function of Gk.
In this equilibrium, following steps analogous to the proof of Theorem 4.11 we have:
u
(yne S
(ug + 1) vk
[M].
(41)
U/2 + +1+1
Note that ((wg) is not a tight bound. Further exploration into tighter bounds and detailed analysis of each group's 's strategic behavior is left for future research.
Q PROOF OF PROPOSITION 4.13
Suppose M = 2 and PT y2) E (0.5, 1). so yı is the Condorcet winner. If PSCF & satisfies Condorcet consistency, it must return the deterministic policy x (z)) = 1. However, this violates the a-PPA axiom because (yy2) while wz > which implies that ()2)) cannot be lower bounded by any a(o) > 0.
R PROOF OF PROPOSITION 4.14
Suppose yi is Condorcet winner. Then Pa P" > y3) > 0.5 for all * i, which implies that w; 0.5. For any other j A i, we have új < po (yj > (:) < 0.5. Therefore, y: has the highest u, i..., i E arg maX Uy, and t00 returns (Bx) = 1, satisfying Condorcet consistency.
S FINITE BEHAVIOR OF $
The following proposition quantifies how large the parameter /3 needs to be to ensure that Condorcet winner receives a sufficiently high probability under the softmax policy.
Proposition S.1 (Condorcet consistency at finite 3). Let yi be a Condorcet winner with u, > 0.5.
Then, the softmax policy satisfies (y;) 2 a if
1
(M 1)ac (1)e)
8 2 u, 0.5
log
(42)
2(1 ac)
Proof. Since yi is Condorcet winner, we have Uj 1o (y3 > 9i)
p Pr 13) 0.5 for
any i. From the given condition
1
(M 1)ac
3
log
(43)
u, 0.5
2(1 a.)
22

---

Published as a conference paper at ICLR 2026
can establish the following lower bound:
exp (Bu,) 2 - (M - exp (0.58)) 2 ac Σ j
ug exp (Suz).
(44)
1 ae
Thus, the softmax policy satisfies
u4 exp (Bus) exp (Bu,) Ejzt = Ui exp (3uj)
T(yi =
2 ac.
(45)
In addition, it can be shown that the a-PPA guarantee deteriorates as + oo, since the lower bound in Lemma 4.7 becomes
(V:)
M
Σ
u exp (B(u, - u.)) ,
(46)
w
which converges to zero as B + oc, unless maxje[M[] Uz-
T CONNECTION TO PAIRWISE MAJORITY CONSISTENCY (PMC)
We discuss the connection pairwise majority consistency (PMC) (Ge et al. 2024), which imposes stronger consistency requirement, ensuring the entire policy ranking aligns with majority preferences.
Definition T.1 (Pairwise majority consistent ranking (PMC ranking)). ranking r is called PMC ranking of a profile a if for all Vt M) E y, a majority of evaluators prefer alternative to alternative yj in a if and only if y: is ranked higher than yj r Formally, p P > 41) 1/2
and only if r (y:) < r (Uz).
Definition T.2 (Pairwise majority consistency (PMC)). A PSCF t satisfies PMC if, for any profile a E (S) that has PMC ranking r", ((r) has the same ranking with r", i.e. (1)) 2 (1))
r (y:) < (z;).
It can be shown that any & with finite 3 > 0 violates PMC, and only the limiting PSCF 4 satisfies PMC.
Proposition T.3. Any Th with finite B 2 0 violates PMC. Th0 satisfies PMC.
Proof. First, we show that & violates PMC for any 8 2 0. suffices to demonstrate that there exists profile with PMC ranking rd for which & (o)()) < (0())())) while r (3:) r" (w,))
Consider the following profile a with M = 3: {1v) U/2 > ys) x 0.3, (y2 Va 31 ) x.1,, (ya V1 > 22) x 0.3, (ya U2 > v1) x 0.3}.
(47)
which yields the following preference function:
50.5 0.6 0.3
P7 0.4
0.5
0.4
(48)
0.7
0.6
0.5
Then, the PMC ranking satisfies r (gs)
(11) < r" (22). PT > 31). Pr) (vs >
92), Pa (y) > 82) 0.5. However, we have 01 0.3 and U2 0.4. Since u1 < u2 it fol-
lows that (o)(()) < (T) regardless of 8, contradicting r (vn) < r" (12). Therefore, t violates PMC regardless of
Next, we show that t satisfies PMC. Consider profile a with its PMC ranking r", and let y" e y be the alternative ranked first in rd (i.e., r" (yy) 1). Then, y² must be Condorcet winner, as pa (v" > y) > 1/2 for all y * y²" Thus, == Φ0 (a) is deterministic policy with x(v") = 1.
Consequently, we have Tzy") ) >> (ly for all y + y and trivially * = ((,) 0 for any ya, y # y², satisfying the condition for PMC.
O
の approximately satisfies PMC as 3 + if we allow some slack in the rankings (e.g., ( > ( e for some small e > 0) in the definition of PMC. However, exploring this approximate consistency is beyond the scope of this paper and is left for future research.
23

---

Published as a conference paper at ICLR 2026
U ADDITIONAL DETAILS OF EXPERIMENTS
U.1 SYNTHETIC DATASET
Dataset generation. For the synthetic dataset, we used 10 prompts and 10 responses for the color- preference alignment task, shown in T To construct the ground-truth profile d, we sampled the true (center) rewards independently from the normal distribution A (00 1) for each response. We then added i.i.d. random noise from (0,, 1) to each true reward to generate 1,000 independent rankings, Finally, we drew 10 pairwise comparison samples i.i.d. from the true preference function pa to train each algorithm.
Table 4: Prompts and responses in synthetic dataset
Prompt (x:)
Response (v)
Which color do you find the most appealing?
Red
Which color best represents your personality?
Blue
When decorating your room, what color do you prefer?
Green
What is your favorite color?
Yellow
Which color do you like the most?
Purple
If you had to choose just one color, which would it be?
Orange
Among all colors, what's your top pick?
Pink
If you could only wear one color forever, what would you choose?
Brown
What color makes you feel happiest? Which color do you prefer most?
Black
White
Evaluation methods. We evaluate the fine-tuned policy using two metrics: (i) win rate against reference policy (the pretrained model), E ) [P (y1 > Y/2 x)]. and (ii) the PPA level a(o). To estimate the fine-tuned policies over responses, and the softmax policy (with temperature 1). We then calculate the win rate and PPA level directly from their definitions using the estimated policy for each prompt. The results are averaged over all prompts.
compute the logits of each response
U.2 ALPACA-GPT4 DATASET
Dataset generation. We considered two groups of evaluators, defined across two categories: expertise and style. For the expertise category, evaluators were grouped into two levels: 'elementary school student' and 'PhD student'. For the style category. evaluators were grouped into 'friendly' and 'unfriendly'. The true population distribution instruction prompts from the Alpaca-GPT4 dataset (Peng et al. 2023). group-specific responses were generated using G with the prompts listed in Table 5 Then, the pairwise comparison samples are drawn i...d. from p
w." (0.8,0.2) For each of the 52k
Table 5: Prompts used for generating responses from each group
Category
Prompt
(1) Generate response that can be easily understood by an elementary school
Expertise
student.
(2) Generate response that only PhD Student in that specific field could understand.
(1) Generate response that is friendly, witty, funny, and humorous, like close friend.
Style
(2) Generate response that answers in an unfriendly manner.
Evaluation methods. To estimate the fine-tuned policies over responses, we sample a response from the policy and use the annotation model (GPT-4.1) classify its group. Table 6 shows the
24

---

Published as a conference paper at ICLR 2026
prompts used to classify the group of generated responses. Based on these classifications, evaluate the policy's 's win rate and the PPA level from their definitions.
Table 6: Prompts used for classification
Category
Prompt
Expertise
Does the expertise level of this response align more closely with the elementary level or the PhD student level? Please answer with only one of these exact options: "elementary' or 'PhD'.
Style
Is this response friendly or unfriendly? Please answer with only one of these exact options: 'friendly' 'unfriendly'.
U.3 HYPERPARAMETER SETTING
The Qwen2.5 --ntttte- model (Yang et al. 2024) was fine-tuned using each algorithm, where both the reference policy Tref and the data sampling policy Ta were set to the same pretrained model. All algorithms were trained on the same offline dataset for the same number of iterations. NLHF was not included in the comparison, as the algorithm does not support offline learning. Specific training hyperparameters are provided in Table Each training run utilized one H100 GPU, requiring approximately 0.5-1 hour per epoch with about 20-40GB of memory usage using LoRA.
Table 7: Training hyperparameters
Hyperparameter
Synthetic
Alpaca-Gre GPPT
Training & Reference Model
Bwen2.553B-IIntruee
Qwen2.5-3B-Inss IBIIrrcttt le-5
Learning Rate
lc-4
Batch Size
4
Epochs
3
Optimizer
AdamW
Adam
Gradient Clipping
1.0
1.0
Learning Rate Scheduler
Linear
Linear
Warmup Steps
100
100
KL Coefficient
0.1
0.01
LORA Rank
32
LoRA
32
32
V SCALABLE OFFLINE ALGORITHM WITH FUNCTION APPROXIMATION
V.I OFFLINE PAIRWISE COMPARISON DATASET
In practical applications of preference learning, the preference function often depends on additional context or state, denoted by X. For instance, LLMs, x represents the input prompt or conver- sational history that provides the specific context for generating preferred response. Accordingly. we define the context-dependent preference function as P(- > -): y x X + [0, 1]. which is offline dataset of pairwise comparisons D {I,, y," v/) N where y," is preferred over y under context r;. Each query is assumed to be drawn i..dd from joint distribution of p(x: and Ta(y x)), and labeled according to the preference function P. Our goal is to use this offline dataset to learn a policy क X + A()) following the framework introduced in the previous sections.
unknown and must be estimated from empirical data. We consider
25

---

Published as a conference paper at ICLR 2026
V.2 TWO-PHASE OFFLINE PREFERENCE LEARNING ALGORITHM
We approximate a softmax policy proposed in Section 4.3
u(y x) exp(3u(y x:)
( x) ΣvEr "(y ær) exp(Bu(y x))
where u(y x) min zey
P(y z x). (49)
Specifically, we use two-phase algorithm that first estimates and then estimates based on u.
Phase 1: Estimating u. To estimate we first train the selector model ze using the following loss function, the offline dataset D, and the parameterized function class Fj::
1 N
N
ply Ie y!")
j arg min PEFP
(50)
Then, the estimated ú can be obtained from ú 2:) = ey P(y + x))(2> x, where P denotes the empirical estimate of the preference function. The derivation of the loss function and the relationship between μ and u are provided in Appendix W
Phase 2: Estimating m. Let it be the normalized softmax policy constructed with follow- ing equation 44 In the second phase, the policy model is trained by minimizing the distance to t over a function class Fx:
i E arg arg min Ex-p L (z Ix), a) x))
(51)
REF,
Here, L denotes divergence or distance metric between two policies. In our experiments, we employ the KL divergence for La Specifically, substituting the KL divergence into L from equation 51 the loss function becomes
P(- > 2 x0)a(2 ( 1
Expp
DKL
x+)
Σ zEY
(52)
T x) log
x T x)
((x.)))(ppraa)
x))
EvEy Ply > 2 x) x,
Using the offline dataset and function approximation, we obtain
(1)
ar)
xyy" xr)
it arg min LE (n
log
nEFa
Ta (
xx)
[ " xr, v¹ exp 1 +
e " ar, (,,"
(T |x)
{ x))
log
aa(y' xr)
( 1 x. y² cxp (
2 Aly" xx, [x,
(53)
V.3 ADDITIONAL TECUNIQQES FOR LLM FINE-TUNING
Regularization via reference policy. Fine-tuning large language models (LLMs) requires main- taining alignment with a reference policy, typically the pretrained model. To prevent excessive drift, we incorporate KL-divergence regularization terms into the training objectives for both the selector model and the policy model Specificalll, we add the following regularization terms to the loss functions Phase and Phase 2:
(p( x, y) Tror( (- (x,,)) m)) Bre DKI.
Bp Be. DKL.
((( x)) Trof( x))
(54)
Training with single model. To reduce computational cost, we propose to train both μ and マ using single model. This is enabled by encoding structural differences through specialized input formats.
Specifically, .( - a) selects preferred responses given a prompt, while μ(( xx y) selects responses given a prompt and candidate response. By distinguishing these cases with separator tokens, achieve performance comparable to training separate models, while improving memory usage and training efficiency.
26

---

Published as a conference paper at ICLR 2026
W DERIVATION OF THE LOSS FUNCTION IN PHASE
Step 1: LP reformulation.
Recall the definition u(y x) == min P(y x 2 a). Each
uy 2) can rewritten as
uly [x)== - 2EY
P(y 12)121p' (z xr x,,)).
(55)
where a selector distribution (- v) E A()) places all its mass on the minimizer of P(y > z)) Such μ² and the corresponding pointwise minimum can be obtained via the following linear programming (LP):
w(y x)
min ((
P zEY
2))((2)))))))))))))))1 xr, x,,))
(56)
Step 2:
Aggregation of pointwise LPs. Assume the data-generating distribution p(-) and тT( x)) have full support. Multiplying and dividing equation 56 by Ta(z x)) and then taking the expectation over all } gives
Ply > 2 x ((( x. y)
u(y x)
min E Bga E((
(57)
Ta(z
Next, we aggregate these pointwise LPs by multiplying each pointwise objective by ((z)Ta( x)) and summing over (I, y) X' V. We also add the symmetrical term with swapped y and 2, which does not change the optimal solution:
P(y > z x)n(z x.
P( > x)p(y I, z)
E arg min ((a.v.s))(pmmmaa) -xX))~AA>)
(58)
Ta(2 x))
Ta(y x>)
Step Empirical counterpart. Given an offline preference dataset D, we approximate the expec- tation in equation 58 usiing its empirical counterpart and restrict the function class to a parameterized family F,,:
N plyl
Jis y!"))
f E arg min PEF,
(59)
N
aa(s/ x;)
Given the estimate ğ. can estimate using equation 56 with the estimated preference function P:
N(x..s) N(N. 1/2,
if Nf v,2) (x, > 0,
-1>>11>)
(60)
otherwise,
where N(x,y,2) Hi E [N] (0,,8." vt) = (2, 2)}
27


Reviews:
[{"review_id": "fm1e3HFsIu", "reviewer": "Reviewer_JSie", "summary": "This paper studies preference optimization algorithms under social choice theory.\nSpecifically, any profile induces a preference function, and preference optimization algorithms learn a policy from the preference function, which together form a PSCF.\nPSCF is desired to satisfy four axioms, *i.e.*, monotonicity, Pareto efficiency, PPA, and PBM.\nThe paper shows that popular preference learning algorithms, *e.g.*, RLHF and NLHF, do not satisfy PPA and PBM.\nWhile a typical PSCF, *i.e.*, random dictatorship, satisfies all axioms, it cannot be implemented by any preference algorithms from preference function.\nTo bridge the gap, this paper proposes a preference learning algorithm that satisfies all axioms.\nThis paper also proposes a softmax relaxation of the proposed algorithm to achieve trade-off between two conflicted axioms, *i.e.*, PPA and Condorcet consistency.\nThe proposed algorithm is evaluated empirically under the tabular and function approximation settings, validating the theoretical analyses.", "questions": "I only have an undergraduate-level understanding of social choice theory and am not sure whether the proposed algorithm is trivial within the field.\nMaybe the authors could add literature review to help the audiences from ML community to better understand the background of the problem being addressed.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* This paper studies a fundamental and important problem, reveals the shortcomings of existing algorithms, and proposes desired method through solid theoretical analyses, which is insightful and meaningful.\n* This paper has clear structure and writing, making it easy to understand.\nI enjoyed reading the paper.", "weaknesses": "I found no obvious flaw of this paper.\nMy only concern is that this paper mostly studies social choice theory in the context of preference learning, and therefore may not be interest of most audiences in the ML community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z1HEa766SY", "reviewer": "Reviewer_GtAR", "summary": "The paper proposes “population-proportional alignment” where a model’s output distribution is meant to mirror how many people put each option at the top of their ranking, i.e., options that are the first choice for larger groups should get more probability mass. Because the true share of “ranked first by voters” isn’t directly observable from pairwise comparisons because the authors assume annotator identities are not available, the authors estimate very simple per-option bounds from the weakest head-to-head results and then allocate probability in proportion to those bounds. They pair this with a loose “manipulability” control that limits over-representation by any one group, and they add a softmax variant that can shift mass toward majority winners. The experiments illustrate a tunable trade-off between win rate and this top-choice proportionality notion, with weaker effects when the group shares have to be estimated. Experiments on MovieLens and small LLM fine-tunes show the intended trade-off between plurality and top-choice proportionality (and lower PBM in the simulator), but the effect is weak on Alpaca when group shares are estimated by a classifier; the two-phase training budget is comparable to RLHF and higher than DPO.", "questions": "I don't have any questions.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The main strength is the attempt to do dsitributional/proportional alignment based on preference structure instead of based on pre-assigned group labels.", "weaknesses": "This is a weak paper as is. The main “contributions” rely on a narrow, first-choice notion of proportionality and a misnamed “strategyproofness” that is really an over-representation cap. Core claims are overstated, proofs are straightforward given the simplified setup, and the paper overlooks major, directly relevant social-choice literature.\n\n1. If “alignment” means matching users’ top choices, supervised fine-tuning with annotators providing preferred options (either in free form text or by choosing their top option) already does this. That setup is well known and not novel. It's perfectly reasonable to assume that we could get annotator ids and seems like an unnecessary constraint in this paper. (The necessary/ important constraint is indeed not having access to group labels)\n\n2. Your proportionality tracks only top ranks. Focusing on first choices wastes richer preference information that could support consensus-oriented outcomes. A rule like plurality (which returns the winner with the most top votes) is the rule within social choice that I would say is at a similar level of sophistication. However, it is well known that it cannot capture compromises or broad consensus (e.g., the “everyone’s #2” candidate that increases average welfare). With such a weak target, the results do not speak to meaningful proportional representation.\n\n3. The paper largely ignores decades of work on proportional representation and budgeting (e.g., JR/EJR, proportionality for solid coalitions (PSC) and fairness in budgeting/ portioning) etc that directly addresses proportionality using full rankings or other preference formats. \n\nSome references:\nPortioning using ordinal preferences: Fairness and efficiency by Airiau et al.\nProportional Participatory Budgeting with Additive Utilities by Peters et al.\nJustified representation in approval-based committee\nvoting by Aziz et Al.\nVoting procedures (book by Dummett)\n\n4. Because proportionality is defined at the top-choice level and feasibility bounds are simple, the proofs are immediate. The most involved argument repeats a known result (Proposition C.1 that the rewards output when optimizing the BT objective are in the same order as the Borda ranking of candidates). There is no technical depth. \n\n5. Consider e.g. Lemma 4.7 and 4.9. The latter has a lower bound that is sometimes 1, but often seems to be of the order of constant /m. Interpretations of how good this simple bound is in a larger class of examples is missing.\n\n6. Claiming RLHF “fails proportionality” in Example 3.1 is misleading. n the two-candidate case, the reward/DPO model can encode the correct split (e.g., 1/2+ε), it just that during expected reward maximization this information gets lost.\n\n7. The manipulability condition simply bounds over-representation. It is not strategyproofness in the social-choice sense (unilateral profitable deviations). In particular, with γ=1/2, an arbitrarily small minority can still drive up to half of the probability mass. It's way too weak to be meaningful. The discussion around lines 246–248 is unclear and mixes group and unilateral deviations.\n\n8. Section 4.3 is conceptually confused. If the output is a distribution over candidates, Condorcet consistency and proportionality are fundamentally at odds (and as a technical claim this is trivial); the paper’s own motivation - distributional alignment/Overton pluralism implies we should favor proportional mixtures, not a winner-takes-all outcome. Dragging Condorcet back in (e.g., via large-β limits) contradicts that goal and reads like a reversion to single-winner logic. Arguably, a Condorcet winner matters mainly in settings with a ground-truth label, which is not the setup here. If you insist on accommodating Condorcet, do it only as a secondary priority within the most proportional distributions: when a Condorcet option exists, give it the highest probability within the proportional mix, without collapsing the mixture. As written, the section is misleading and undermines the paper’s stated objectives.\n\n9. I looked at the experiments in the main body, but did not look into the appendix in detail. The experiments confirm the theory/approach prediction in some sense, but I am not convinced by the theory/approach and its novelty in the first place.\n\n\nMinor comment: In the experiments you seem to use \"RLHF\" and \"DPO\" interchangably? At least in the first experiment you say that you compare to RLHF not to DPO, but then Table 2 contains a comparison to DPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UbON8N36MN", "reviewer": "Reviewer_pwXS", "summary": "The authors attempt to expand beyond RLHF and NLHF in the area of AI alignment. Existing strategies often prioritize majority views at the expense of the minority; this not only excludes these views but can be brittle when majorities are slim. The authors use axioms of social choice to propose a method that better represents the proportion of disagreement in minority-view groups, using the top preference. They also introduce 2 axioms of their own. \n\nThey later propose a way of trading off between dueling objectives and validate their proposals empirically.", "questions": "1. Can you elaborate on the movielens experiments? What did the full \"path\" of beta values look like there?\n\n2. What guidance would you give practictioners on the choice of $\\beta$?\n\n3. Do you have comparisons between RLHF/NLHF on the experimental sections? If not could you attempt to provide some?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The authors tackle a key problem in alignment; under representation of minority class preferences.\n\n* I did not rigorously check the math, but I was able to follow the equations and to my eye they seemed correct.\n\n* The tradeoffs of the method are clear, the authors also include a parameter to tradeoff between PPA (a metric they introduce) and Condorcet consistency (more on that in weaknesses)\n\n* Ultimately this is a well-written, well-justified paper tackling a key problem in alignment. The social choice framing fits very well.", "weaknesses": "* I felt that the experiments section had 3 major deficits:\n\n** The movielens results seems somewhat underdeveloped. In many ways this seems to be a fairly ideal case, with potentially large disagreement. The results provided seem to back this, but they are provided briefly in sentence form, and not included with tables, more specific intermediate values, making it hard to get a sense for how robust or reliable those values are.\n\n** The baselines seem relatively reasonable, but given the framing of the paper as \"Beyond RLHF And NLHF\", I am puzzled by the lack of comparison between these methods. It seems fairly paramount to understand how the method performs relative to those.\n\n** The use of $\\beta$ is underdiscussed prior to the experiments, in my view. While it is true that the prupose of $\\beta$ is discussed, the authors show a wide variety of beta values. It is unclear to me how one would pick a specific value ina  principled manner.\n\n* I felt 4.3 could have used a better exploration of use cases for $\\beta$. I applaud the authors for including such a lever, but it seems to me that how one would decide on usage of $\\beta$ is underdeveloped.\n\n* I feel also that the discussion of manipulation in 5.1 is underdeveloped. While the authors do have bounds on manipulation by the minority class, a more detailed set of empirical results would be appreciated. I feel this would also tie into a discussion on when to use $\\beta$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GXhfcxh7wA", "reviewer": "Reviewer_JTvB", "summary": "The paper targets distributional pluralism without group labels. It introduces PPA (probability mass at least proportional to each group's top choice share) and PBM (single-group manipulation is bounded by its true share) and gives a simple rule by considering the worst pairwise win it has against any other option and then set each option's probability to that worst win score, rescaled. There's a temperature knob shifting probabilities into the strongest options. The authors offer some experimental validation of this approach.", "questions": "- Is there a principled way to choose or adapt beta across contexts and over time (under distribution shift), especially when the two desiderata are hard to measure reliably and given the trade-offs noted in the paper?\n\n- Can the authors validate on more standard RLHF-style settings (preferably with human or auditor-verified cohort labels, etc.), given that the current Alpaca example relies on classifier-inferred groups and model-generated outputs and seems too noisy to support firm conclusions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "It clearly exposits what RLHF/NLHF does not guarentee. It provides theoretical analysis to back the proposed rule, and the rule itself is simple with a tunable $\\beta$ term, and bridges social choice theory and preference learning.", "weaknesses": "There are a few issues I have with the paper.\n- First of all, even though the author motivates with examples, the proposed method exhibits a clear trade-off as shown in the experiment: as beta increases, there's less pluralism but the performance increases. The paper shows the trade-off but offers little operational guidance for picking beta in real systems where “the population” is dynamic and contested. This pushes heavy governance choices onto practitioners.\n\n- Perhaps most concerning to me is a lack of evidence that this method works. The only LLM study fine-tunes Qwen2.5-3B-Instruct; NLHF isn’t included, and group proportions in Alpaca are inferred by GPT-4.1 classifiers (no human labels). The authors themselves say differences across beta are \"less distinct\" due to annotator noise and that measuring PPA for LLMs are still an open challenge. This cast doubt on practical employability. The clearest empirical story is the tabular MovieLens task (1,297 rankings, 20 movies), where PPA/PBM can be computed exactly and the trade-off is clean; but this is not an LLM domain, so generalization to instruction tuning is unclear.\n\n- PBM is weaker than strategy-proofness and difficult to measure in practice. Because it relies on unknown group labels, its guarantees aren’t verifiable. It limits—but does not eliminate—misreporting incentives; when manipulation is cheap, the remaining incentives can still be significant.\n\n- As the authors pointed out in sec 4.3, the proposed rule is not condorcet consistent - even with a majority winner, it deliberately allocates mass to minority options, which comes at odds with desirable notions in many application which wants to pick the majority winner.\n\n- The method only observes head-to-head outcomes and not true group sizes. It falls back on conservative upper bounds but when many matchups are close, the resulting policies become nearly flat even if one group is actually much larger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eTIrsvXcSb", "reviewer": "Reviewer_YHTf", "summary": "This work proposes an approach to better represent human preferences from a diverse population, ie reflecting the mix of views not just the majority, in a trained policy. It comes with guarantees on its proportionality to the actual population (population‑proportional alignment, or PPA) and its manipulation by a group (population‑bounded manipulability, PBM). And with a parameter knob to trade off proportionality against picking the Condorcet winner (the one option that beats all the others, pairwise) Their experiments show that they are able to get a good PPA with a win rate on par with RLHF, offering no proportionality.", "questions": "- out of curiosity, how would the approach be adapted to handle more than pairwise comparisons, e.g. respect the ranking of multiple options by a diverse population ?\n- the influence of a single group is bounded, but isn't there a risk of collusion amongst groups?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- RLHF/NLHF tend to pick a single preference even when the \"camps\" are close; their approach better handles mix views from a diverse population of evaluators and align a policy according to its distribution ;\n- it offers axiomatic guarantees, first monotonicity (if an option pairwise standing improves its probability doesn't go down) and Pareto efficiency (if everyone prefers $y$ to $y’$, then $y$ gets at least as much weight as $y’$), and their proposed PPA (every option gets at least a fraction of its true population share) and PBM (the influence of a group is bounded by a function of its share) ;\n- the $\\beta$ parameter adds flexibility between converging to Condorcet (higher $\\beta$) or aiming for proportionality (lower $\\beta$) ;\n- they confirm their predicted results (with RHLF, NHLF, and their algorithm), observing the  $\\beta$-induced trade-off.", "weaknesses": "- this is not an easy read: the topic has inherent complexity but the intuition for the algorithm is not easy to grasp from the math-heavy notation —it also means that, under time constraint, I cannot guarantee that sections 3 and 4 are fully sound ;\n- despite this machinery, the true share of the population remains elusive, and a conservative strategy has to be picked ;\n- it would have been nice to better illustrate the theory with use cases clearly showing how the $\\beta$ parameter can be adjusted in situation ;\n- the second, large-scale, experiment, shows a muted trade‑off on Alpaca‑GPT4, so the central PPA claim is hard to appreciate —the suggestion that \"this is because the group distributions of outputs are estimated using an annotation model\" is not satisfying regarding that claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)