Enforcing Axioms for AI Alignment under Loss-Based Rules
Authors: Alexandros Hollender
Sonja Kraiczy
Venue: ICLR 2026 Poster (2026)
Keywords: Social Choice
AI Alignment
Reinforcement Learning from Human Feedback
Constitutional AI
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Recent alignment methods for large language models, most notably reinforcement learning from human feedback (RLHF), often train an auxiliary reward model to minimize a loss function on binary preference data over model responses. We study a theoretical setting inspired by principle-guided methods such as Constitutional AI, in which a small set of principles (e.g., helpfulness, toxicity) act as “voters” that guide binary comparisons---such as preferring the less toxic response. We model these principles as linear directions in an embedding space of responses, a simplifying assumption motivated by the Linear Representation Hypothesis---concepts are linear directions in representation-space---a useful first-order approximation in practice.
In this \emph{linear social choice model}, Ge et al. (2024) showed that an optimal linear reward model can violate Pareto optimality (PO): From the principles-as-voters lens, this means a response A can be less helpful and more toxic than B, yet still receive a higher reward. We analyze axiomatic violations in the linear social choice setting and probe the robustness of negative results under realistic assumptions. We show that added expressivity does not resolve the issue: polynomial reward models can still fail PO. We then offer a pragmatic alternative showing that when the data uniformly covers the embedding space, broad classes of loss-based rules in the limit exactly recover the axiomatic guarantees. This yields a recipe for constitutional-style alignment with provable guarantees: enforce balanced coverage \emph{via dataset design} to restore axiomatic guarantees without abandoning standard training pipelines.

Published as a conference paper at ICLR 2026
ENFORCING AXIOMS FOR AI ALIGNMENT UNDER LOSS-BASED RULES
Alexandros Hollender* University of Oxford
Sonja Kraiczy" University of Oxford, PIBBSSt Sonja. Kaajja.kraczyccs.. ax.ac.a.acco....................
al alexandros.holle www.:hhhlleeeeeeccoeoeooo ac.uk
ABSTRACT
Recent alignment methods for large language models, most notably rein- forcement learning from human feedback (RLHF), often train an auxil- iary reward model to minimize loss function on binary preference data over model responses. We study a thcoretical sctting inspired by principle- guided methods such as Constitutional AI, in which small sot of principles (e.g,, helpfulness, loxicity) act "voters" that guide binary comparisons such as preferring the less toxie response. We model these principles as linear directions in embedding space of responses, simplifying assump- tion motivated by the Linear Representation Hypothesis concepts are lin- directions in representation-spaace practice. In this linear social choice model, Ge et al. (2024) showed that an optimal lincar reward model can violate Parcto optimality (PO): From the principles-as-voter lens, this means a response A can be less helpful and more toxic than B, yet still receive higher reward. We analyze axiomatic violations in the linear social choice setting and probe the robustness of negative results under realistic assumptions. We show that added expres- sivity does not resolve the issue: polynomial reward models can still fail PO. We then offer a pragmatic alternative showing that when the data uniformly covers the embedding space, broad classes of loss- rules in the limit exactly recover the axiomatic guarantees. This yields a recipe for constitutionallstyle alignment with provable guarantees: enforce bal- anced coverage vía dataset design to restore axiomatic guarantecs without abandoning standard training pipelines.
useful first-order approximation in
INTRODUCTION
Many recent alignment methods follow a common pipcline: collect pairwise (binary) pref- erences over model responses, fit a reward (or preference) model to these comparisons, and then optimize the base model to minimize a loss correlated with aligned behavior. In the clas- setting of Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017), human annotators select the preferred response between candi- date responses. Rather than eliciting unguided preferences, Anthropic introduced principle- oriented foodback (Bai et al., 2022a) in which annotators judge responses against explicit principles-hellfulneee honesty, and harmlessness (HHH); this principle-guided supervision later informed Constitutional AI, which formalizes written set of principles to guide bi- nary comparisons of model responses (Bai et al., 2022b). We use "constilutionallstylee as an umbrella term for such principle-guided supervision.
sical human-in-the-lowe
Adopting social-choice perspective on constituttonnllsttlee alignment, we treat principles as voters that evaluate pairs of model responses by judging which better adheres to the relevant principle. From this perspective, a minimal requirement "axiom" for any ag- gregation method is Parcto Optimality (PO) (also known as Unanimity (Arrow, 1951)):
Authors are listed in alphabetical order. PIBBSS (Principles of Intelligence).

---

Published as a conference paper at ICLR 2026
e.g., in the HHH framework, if response A is more helpful, more honest, and more harmless than B, then should receive higher reward and thus be more likely to be generated by the model. We study setting in which principles (such as HHH) are modeled as linear directions in a representation space. This is a simplifying assumption motivated by the Linear Representation Hypothesis (LRHH) (Park et al.), which posits that some high-level concepts are well-approxinated by linear features in learned embeddings and which has been operationally useful in mechanistic interpretability.
Formally, most of our results lie within the lineur social-choice framework of Ge et al. (2024): utilities over a fixed representation are linear, pairwise comparisons yield binary labels, and optimizing those labels learns a linear reward model. Although the linear social- choice framework was introduced as a model for RLHF, we contend that both the linearity assumption and axioms such as PO are more realistic and salient in constitutional-style alignment with small number of principles.
Within the linear social choice model, the authors show that, perhaps surprisingly, an opti- mally trained linear reward can violate Pareto optimality (PO): everyone prefers response A to response B, yet B receives higher reward. The concrete counterexample involves only two (weighted) principles, thus covering the HHH scenario; we discuss it as a warm-up in Section 3. Finding this result counterintuitive, we ask:
How robust are axiomatic violations under loss-based training?
1.1 OUR CONTRIBUTION
We revisit the axiomatic violations through a lens which is compatible with training pipelines. As part of our warm-up in Section 3, we provide new intuition on why the PO violation occurs in the first place, by providing a simplified minimal example that clarifies what goes wrong. While Ge et al. (2024) show that a social choice combinatorial approach can recover axiomatic guarantees, our paper focuses on approaches to obtain guarantees that align with modern ML pipelines namely, loss-based rules. Our main contributions are three-fold.
1. Beyond linear rewards. A natural hypothesis is that linear rewards with frozen embedding are simply too restrictive; perhaps keeping voters linear but using a more expressive reward model, such as bounded-degree polynomials, restores PO. We show that, again surprisingly, this is not the case: even with linear voters and expressive reward models, violations of PO (and of another natural axiom called Pairwise Majority Consistency) persist; see Section 4.
2. Generalization. In Section 5, we adopt an axiomatic framework on the embedding space- rather than fixed train set-and define generalized version of Pareto Optimality (PO). This recognizes the key feature of the linear model directional differences between candidates matter, not their embeddings. With finite data, we cannot expect to satisfy these axioms exactly; This aligns with the core role of reward models: their value is generalization, the key question is how well any desirable property transfers to unseen data.
3. Data-centric perspective. While social-choice axiomatic analyses typically rea- son from a worst-case perspective, real-world training dynamics are far more sensi- tive to how we curate and sample (preference) data. Recognizing the limitation of the worst case approach, we ask: can we choose binary comparison queries so that axiomatic guarantees are provably recovered? In Scction 5, we show that a random sampling scheme already suffices and in the limit we achieve perfect PO. This sug- gests practical levers (data inspection, reweighting or careful datasel design) when combinatorial social-choice algorithms are hard to deploy in real systems.
Results marked with have their proofs deferred to the appendix.
With many raters, agreement a binary comparison becomes increasingly unlikely.
2

---

Published as a conference paper at ICLR 2026
1.2 RELATED WORK
The field of social choice (Arrow et al., 2010; Brandt et al., 2016) offers long tradition of axiomatic guarantees that provide lens through which to compare different aggregation rules. This perspective is directly relevant to preference-based alignment: Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) and Nash Learning from Human Feedback (NLHF) (Munos et al., 2024) have natural analogues in social choice. The optimal solution to the Bradley-Terry loss in RLHF is known to induce ranking that is equivalent to the Borda ranking from social choice theory (Anderson et al., 2009; Siththa- ranjan et al., 2024). Formally, NLHF coincides with the von Neumann winner defined in Dudík et al. (2015), which in turn matches an old rule from social choice, Fishburn's 's maximal lotteries (Fishburn, 1984).² Building on this connection, several position papers have recently argued for social choice as a useful lens on RLHF and alignment more broadly (Conitzer et al., 2024; Dai & Fleisig).
Among the first wave of technical results that has emerged since, most relevant to us is the work by Ge et al. (2024) who propose the linear social-choice model where voters are linear directions over fixed embedding. Building on prior work by (Noothigattu et al., 2020) which explores the axiomatic properties of reward functions defined as MLE estimators of underlying random utility models, they analyze the ranking over fixed candidate set in- duced by an optimal linear reward with respect to axioms including Pareto optimality (PO) and Pairwise Majority Consistency (PMC). Because these axioms can still be violated in that setting, they adapt a combinatorial social-choice rule to the linear social-choice model to obtain the guarantees. The closest analogue in social choice to linear preferences over a fixed embedding arises in the literature on restricted preference domains (over rankings) (Elkind et al., 2022).
Procaccia et al. (2025) analyze clone robustness and show that an appropriate reweight- ing of the Bradley- Terry loss can be made satisfy their axiom. Most existing work at the intersection of alignment and social choice adopts the classic social-choice setting with unstructured alternatives, in contrast to the metric setting we study here. Recent work in this vein includes representative social choice proposed by Qiu (2024), the extension of the distortion framework of Procaccia & Rosenschein (2006) to preference distributions by Gölz et al. (2025), an analysis of RLHF when cach comparison is labeled by single annotator by Xiao et al. (2025), and a proposal towards proportional alignment by Kim et al. (2025).
PRELIMINARIES
Social choice model. The set of alternatives is the d-dimensional space Rd. reward function r Rd + I induces a (weak) ordering *, over the set of all alternatives Bd by ranking alternatives according to their rewards, namely a <rr + r(a) S r(b) for all a, a E Rdd.
There are n voters, and each voter has a (weak) ordering % over the set of alternatives Rd. Unlike in the usual social choice setting, we do not observe the full orderings, but instcad we are only offered a partial view of cach voter's preferences. For cach voter i, we observe list of pairwise (strict) comparisons. Formally, for each i we are given a nonempty finite set P. C (Rdy² that satisfies b) E P; = a - b. A natural special case is when there set C' of m candidates in IR and the sets P. contain all' pairwise comparisons between candidates in C, i..., P.- = {(a.b) b) E C² a <, 6}.
In this context, a voting rule takes as input sets of pairwise comparisons Pl...., PR, one for each voter, and outputs reward function whose induced ordering attempts to aggregate the voter preferences.
Both concepts are Nash equilibria of games and coincide because the corresponding payoff matrices are related by a positive affine trausformation; see (Wang et al., 2023; Mavra-Rivero et al., 2025) for the explicit reference to maximal lotteries.
We assume that the set of candidates O is such that no two candidates are tied for any of the voters, i.e, for any a, E C either a -; X4 a holds.
3

---

Published as a conference paper at ICLR 2026
Loss-based voting rules. We restrict our attention to voting rules that output a reward function minimizing a total loss function. Given a particular loss function + R, we define the total loss incurred by reward function
((r) = Σ Σ ((r(a) r(()
kE(n, (a.b)EP:
where Pi....,PP Pn are the sets pairwise comparisons of the n voters, as described above. The voting rule defined by loss function l outputs a reward function minimizing the total loss L.
We obtain the Soadley-Terry loss (Bradley & Terry, 1952; Zermelo, 1929) CBT used in RLHF by choosing the loss function & to be the cross-entropy loss, i.c., ( = log(1 + e*).
We will mention the precise assumptions on the loss function in the thcorem statements later, but for now let us just think of & as strictly increasing function. Then the idea is that some voter has ranked a over their set P,, a reward function should be penalized for ranking b over (i.e., for assigning a higher reward to b than a).
Linear social choicc. We will mostly focus on the linear social choice model, where the reward functions are lincar, i.c, of the form rz(z)- (0,z). In particular, in this context, a loss-based voting rule will minimize the total loss L over the set of all linear reward functions. Furthermore, the voters are also assumed to have orderings that are induced by linear reward functions.
Axioms. In this paper we will focus on the following two natural axioms.
Definition (Pareto Optimality (PO)). A loss-based based voting rule satisfies Pareto optimality (PO), if it outputs a reward function r that ranks any comparison (a, E n,P, correctly, Le., r(a) <b)).
In other words, if there exists a comparison between two candidates on which all voters agree, then PO requires the voting rule also rank these candidates in the same way as the voters.
Definition (Pairwise Majority Consistency (PMC)). An ordering over all alternatives appearing in the instance is PMC ordering, if it satisfies: a < b if and only if a strict majority of voters rank a below b, i.e., {i (a, b) E P.H. n/2. A loss-based voting rule satisfies PMC, if whenever a PMC ordering < exists, it outputs a reward function that induces <.
Note that if a PMC ordering exists, then it is necessarily unique.
3 REVISITING THE PARETO OPTIMALITY VIOLATION
The counterexample to PO provided by Ge et al. (2024) is constructed in R2. It uses two (weighted) voters with direction vectors U1 = (1, 1) and U22 = 0) (their magnitudes are irrelevant) and six candidates in R², arranged as two triples. The key idea is place one triple (a at (2, 1), 6 at (1, and e at (0,0)) so that the two voters disagree on the induced ordering of its three candidates; with suitable weights, however, the optimal dircction is (1,
Each candidate is then duplicated and the copies are perturbed within an e-neighborhood of the originals. This replicates every pairwise comparison fourfold and adds an almost constant term to the loss from comparing each candidate with its own copy. By a generalized continuity argument, for sufficiently small the optimal linear reward for the perturbed instance remains close to the original one, since as - 0 the new loss is (in the limit) lincar transformation of the old loss. Finally, approach e = (0, 0) from a direction in the left half-space that makes both in and U2 strictly prefer e' to c, yielding the desired PO violation.
forces one of these copics, say c', to
This construction can be tightened to use only four candidates by duplicating a single point, as we do in the proof of Theorem 4.1. With three non-collinear candidates and any number
4

---

Published as a conference paper at ICLR 2026
of voters, PO is always satisfiable: optimal rewards (note that these are only unique up to an additive constant) of the uncoustrained objective over these three points can always be realized by some 0 RR Disagreement between voters (e.g., 2 prefers a > b > while is prefers the reverse) acts like length constraint in the Bradley-Terry loss. For a pair (a, b), the loss contains both
log(1 e (0.00-0)) and log(1 + e (..b-a)))
(possibly with different weights). Letting ||||| + oo drives exactly one of these two terms to +00 while the other tends to O; hence the loss penalizes unbounded |0]] (The same phenomenon holds more generally for losses bounded below but diverging to - oc as their argument + co.)
We next study a minimal example with one voter and three candidates where the optimal linear reward subject to a norm constraint violates PO. This instance provides insight into why PO can fail even without such explicit constraints on instances with multiple voters.
Consider a single voter e = 1) with & > and three candidates a = (1,0), = (0,0),
c = (-5,6) where 0 < < 1. Constrain 0 to unit length, 0 = (01, +V - 02) with 0 [-1,1] The pairwise Bradley -errly terms are
(), log(1 + e -0))aaii) = log(1 + )
(a,e): log(1 + e (0,a-c)) log(1 + E - (11-5) + 6V1-0)),
(b,c): log(1 + e (0(00.)h)) = log(1 1 + Als + vi-(f) S log(1 + 2
since m(x.)e(- = v1 0;) V2. Thus the (b,c) term is O while the (a, term strietly decreases as 0j increases, and for small the (a,c) term also decreases with 01- Consequently, for sufficiently small any minimizer has 0, close to and in the upper-right quadrant near the r-axis, yielding (0, (0,c) while (v, >> (V.c). Thus, the presence of other voters can impose a length constraint, and such length constraints lead to PO violations even for single voter. Intuitively, the norm constraint can be interpreted as a finite "budget"; ; directions (such as a a- b) that are more common or are longer dominate the loss because "misclassification" in such directions contributes larger terms to the loss, and so these terms are prioritized given a length constraint.
4 POLYNOMIAL REWARD FUNCTIONS
In this section, we study an extension of the linear social choice model, where allow more general reward functions. Namely, we consider polynomial rewards and show that PO and PMC fail even in this case.
Theorem 4.1. Any loss-based voting rule with loss function & that is strictly conver, lower bounded, and differentiable with (() > 0, fails to salisfy PO and PMC even with polynomial reward functions of bounded degree. Furthermore, this already holds in two dimensions and with three voters that all lie in the positive quadrant.
In particular, this theorem applies to the Bradley Terry loss, which thus fails to satisly PO and PMC even with polynomial reward functions bounded degree.
4.1 PROOF OF THEOREM 4.1
The instance consists of m + 1= in 8² will be specified later. Furthermore, there are two weighted voters:
:= d(d 1) + 2 candidates co, C1, c,,..., Cm. whose positions
Voter tı (1, has fraction (1/2,1) of the votes and ranks the candidates in the order
C Co C x C3 < < Cm-
Voter v2 (0, 1) has fraction 1 of the votes and ranks the candidates in the order
Cm x Ci x Co-
5

---

Published as a conference paper at ICLR 2026
Note that the voters disagree on all comparisons, except the comparison between cp and ci where they agree. The proof works for any a E (1/2,1), but we can set, e.g.; a = 2/3 to obtain setting with three (unweighted) voters.
Total loss function. For now, consider allowing arbitrary reward values To, Ph.... I'm E R for the candidates co, Cly..., Cm- The total loss of the reward vector on this instance can be written as
C(r) T1. (m) ==)= a ((ri ((11-1o) To)
Σ
E(ro -Ty)+
e(ri (r)-r;)) Igicism
E
j=2
(1)
+ +(1
Σ
E(r ri)
oSi<j$m
Note that can assume without loss of generality that, say, 1ı 0, since the total loss does not change if we subtract vi from all rewards.
Optimal rewards in the degenerate instance. In our instance, we will position can- didate op very close to C₁. Thus, the instance will be closely related to a "degenerate" instance, where cy and 61 lic the same position. In that degenerate instance, the optimal (arbitrary) rewards are given by the following optimization problem:
min [(r)
s.t. r
(2)
Claim 1 (+). The optimization problem (2) has unique solution 0- < < ... r
Positioning of the candidates. We use these optimal rewards 0 = rī < < < ... rm to define the positions of the candidates. First, we let с₁ (0,0) and Co = (6,8) for some sufficiently small 8 > 0 to be specified later. Next, for each j E [d], we let Lj := {(x, y) E R² y -2x + j}, i.c., Lj is the line going through point (0,j) with slope -2. Then, for each = 3....
line Lj such that - - y = ri, where j = [íi 1)/(d 1)]. As a result, for any i 2 2, candidate & is positioned at (z,z) (rf + -r - j],, where [(i 1)/(d+
we position candidate e, at the unique point (x,
We give a more detailed description of the positioning of candidates & for i 2 2. We let 22 be the unique point (z, y) line L1 such that -I - y r.. Similarly, we let c3 be the unique point (z, y) on line L1 such that -I - =3. We continue positioning points on L1 in this manner, until d + 1 points have been positioned. Then, words, Cd+a is the unique point (I on line L2 such that -x = r243. We proceed in this manner, placing + points on each line, and then moving on the next line. Note that the number of candidates to be positioned (without c and 61, since these have been fixed above) exactly m 1 d(d + 1), so we are able to position exactly + candidates on each of the d lines Li...., ......,ee La- See Figure 1 for an illustration. By construction the following holds.
switch to L2. In other
Claim 2. The positions of the condidates co. co...... Cm are consistent with the rankings of voters and U2. Furthermore, the polynomial p² (x, = x - satisfies pa (c(:) = rī for all i [m] and ranks the condidates in the order co C1 - 02 x Cm.
Degree-d polynomial reward functions. We consider the class of reward functions that are polynomials of degree most. d. As before, without loss of generality, we can restrict our attention to polynomials that satisfy p(cr) = 0, i.e., the constant term is zero (since e₁ lies at the origin). The optimal such polynomial reward functions are given by the following optimization problem:
min
s.b. p(z, y) polynomial of degree at most d 0), = 0
(3)

---

Published as a conference paper at ICLR 2026
y
40
L1
C2
L2
Co
CB
C7
Figure 1: Illustration of the positioning of the candidates for degree d= The red arrow labeled p" indicates the direction of increase of the linear polynomial p² - y-
As shown in Claim 2, the polynomial p' (x, y) - - -y ranks ei above co when used as reward function, ie., p" (c1) pc()... On the other hand, both voters U1 and va rank co over C1. In the rest of this proof, our goal will be to show that the optimal polynomial degree-d reward function for our instance (i.e., any solution to (3)) is close p", and thus also ranks C1 above co. This will immediately show that Pareto optimality does not hold for our instance.
Claim In the degenerate instance where d == 0, and thus co = =1 = (0, 0), the polymomial p² v) = = is the unique optimal solution of (3).
Proof. By Claim the polynomial p" achieves the optimal rewards r" from (2), and thus p" is an optimal solution of (3) in the degenerate instance where 5 = In order to show that p" is the unique optimal solution, it suffices to show that other degree-d polynomial achieves the optimal rewards r", which are the unique optimal solution of (2) by Claim 1. This follows from the fact that the zero polynomial is the only degree-d polynomial that simultaneously vanishes at all points G a fact which we prove next.
Consider polynomial p of degree at most such that p(c;) that = 0. First, we apply a rotation around the origin to the (x, plane such that the lines Ly are now of the form Lj {(x, y) E R2 y = sy} for some 0 < $1 $2 < Note that p is still a polynomial of degree at most d in this new basis.
for all [m]. We will show
sd.
Now since p vanishes d+I 1 distinct points on Lj, it follows that p restricted to y = sy is the zero polynomial, i.e., p) Ply=s, 0. As a result, 4 p must contain factor (y (y-8,) for each j E [d], Since has degree at most d, it follows that p be written as p(z, v) C- [[ But now p(0,0) 0) ri 0, together with 8j * for all j, implies that C and thus p= O.
s,).
We will use Berge's ' maximum theorem to argue that for sufficiently small 8 > 0, any optimal solution to (3) must be close to p".
Theorem 4.2 (Berge's Maximum Thcorem (Berge, 1997); simplified version). Let A S R and B C Rm such that B is nonemply and compact. Let f: A x B R be comtinuous.
'More formally, divide polynomial p by polynomial y in (R[) We oblain p(r, y) (y $1)((2, v) + r(x), where we note that only depends since it must have degree strictly less than one in y. Now pl g-==, 0 implies that O. Then, we can continue by dividing q by y 82 in the same manner obtain the full factorization.
7

---

Published as a conference paper at ICLR 2026
Define the set-valued function f*: A = B by f*) arg m f(a, b). Then f" is upper-hemicontinund with nonempty and compact values.
Claim 4. There erists a sufficiently small 6 > 0 such that any optimal solution of (3) salisfies p(c1) p(ca)-
Before proving this claim, let us see why this implies Theorem 4.1. For d as given by the claim, any optimal solution ranks e₁ over co- On the other hand, both voters v and v2 rank co over C₁2 so Pareto optimality is not satisfied. Furthermore, voter ts always (trivially) agrees with strict majority of the voters; in particular, the ordering of voter is PMC for the instance. However, any optimal solution ranks e1 over cu, even though majority (in fact, everyone) agrees to the opposite. Thus, the example also fails PMC.
Proof. In order to use Berge's maximum theorem, we need to make sure that the domain over which we are optimizing is compact. Let = (8.)),,00++5S4 81, E R, 80.0 = 0} represent all polynomials of degree at most d with zero constant coefficicnt, i.c. for cach
S, the corresponding polynomial is given by pz(z.y) El - s.g Let S denote the subset of where all coefficients have magnitude at most 2, i.e. S = {s E S [ss S 2). Let 8* E S be such that Ps- = p", Le., $1.0 $0.1 = -1 and all other coefficients are zero. Note that S' is compact and 8 lies in the interior of S
Now define f [0. 1] x S' R. (6.8) + (e)(oo))) (e..ec)).....P(((mm))) Clearly, f is continuous, since the loss function l is continuous, and S" is nonempty and compact. Thus, by Berge's maximum theorem (Theorem 4.2), the set-valued function f": [0, = 5 arg min f6,, s) is upper- heppooremmioniiiuuus..
Recall that by Claim 3, when 8 = 0, the polynomial p" (x. y) = - is the unique minimizer of C, i.., (0) {s*}, where s* E S is defined above such that p' = Per As shown in Claim 2, the polynomial p* = Ps- satisfies (c1) > p" p'(co)- By continuity, it follows that there exists small enough
(0. 1), such that we also have ps(c)) > ps(co) for all s E S we know that for sufficiently
with ||s - s" looo S e. Now, since f" is upper-hemicontinunn small 5 (0, 1J, we have ||s s- I for all s E f*()). In particular, since & < (0) lies in the interior of S". By convexity of the function s + L(ps (c)())), Pa (c(c))..... Pp follows that f8) is the set of minimizers over all of S (not just S'). So for sufficiently small 8 > O, any minimizer p of (3) must satisfy p(c)) > p(co).
RECOVERING PARETO OPTIMALITY WITH UNIFORM DATA
In this section, we return to the setting of linear social choice. We consider an idcalized setting where (i) the datasct contains comparisons in all possible directions of space, (ii) each of these comparisons has unit length, and (fii) each direction appears uniformly. More formally, this corresponds to setting where the total loss can be written
(())
((-(0,z)) do
(4)
ZES4 1(0,))20
where ..,..., Cn E (0) are the voters (inducing ranking according to the corresponding reward function ru, (x) (vs, z)).
We can define a version of PO over this "complete" datasot.
Definition 3. We say that 0 E Rd is Pareto optimal (PO) over Rd with respect to voters Ul...., Un E IRd \ {0} if whenever a direction x E gd-1 satisfies (vi, x) > for all i, we also have (0,z) 0.
Thcorem 5.1. In the idealized setting, any loss-based voting rule with loss function that is conver, nondecreasing, lower bounded, and differentiable with (()) > O, satisfies PO, as long as there are al. least. two disláncl volers.
Thhe theorem clearly also applies to minimization by replacing f by ff-
8

---

Published as a conference paper at ICLR 2026
On the other hand, show that the analogous definition of Pairwise Majority Consis- tency (PMC) is not satisfied even in this setting.
Definition 4. We say that O E R' is pairwise majority consistent (PMC) over R with respect to voters U1..... Un R {0} if whenever a direction z E S satisfies (v;, for a strict majority of all i, we also have (0.x) > 0. A voting rule satisfies PMC if it outputs a PMC vector whenever one exists.
Thcorem 5.2 )+) In the idealized setting, any loss-based voting rule with a loss function & that is strictly convez, nondecreasing, lower bounded, and differentiable with (0) fails PMC.
Although PMC fails in this uniform data setting, it is not at all obvious whether PMC is even desirable here. For example, if a p fraction votes come from direction on and a (1 -p) fraction from direction 22, then in the uniform data setting PMC becomes discontinuous requirement: the rule must output direction ta whenever p > 0.5 and direction 12 whenever p 0.5. In many applications, one may instead prefer to interpolate between the two. See the work by Lederer et al. (2024) for a recent discussion of similar considcrations in rank aggregation (from which PMC inherited).
5.1 PROOF OF THEOREM 5.1
Let Lú denote the loss with respect to voter i, i.e.
c,(0) = [ JEESt-1. {(- (0(( dx
/ E( (==sg((((,)))) x)) (0, x)) dx, =ESe-1
where the sign function defined as
sgn(t) 1 if if 0, sgn(t) 0 if tf 0, sgn(t) -1 if <.
Then we can write L(0) = Ei
Claim 5
150,0' 0', E TRd satisfy [10|12 = 110/12 and (0', vz) > (0, v.), then Lx(0') < ()). If there are at least two distinct voters, then C ollains its minimum.
Claim 6
Claim 7 If there erists E Sa-1 such that (vi, > O for all voters i E [n], then 0 = 0 is not a minimum of C.
Finally, we use these three claims to prove the following. Claim 8, Any minimizer 0r of L satisfies PO over Rd with respect to the voters o....., Un-
Proof. First of all, note that if there does not exist x E $ such that (r1,2) 0 for all i i [n], then PO is trivially satisfied. Thus, from now on assume that the set D == for e Sd-1; : (U,, > for all E [n]] is not empty. In particular, by Claim 7, this implies any minimizer H" of L is not the zero vector.
Consider any 0 + such that there exists x D such that (0, S 0. We will show that there exists 8' with L(8') < L(0) and thus 0 is not optimal. Since the set D is open, we can assume without loss of generality that in fact. (0,z) < 0. Furthermore, for simplicity we will assume that. |10]|2 = 1 = ||,||/2- If that is not the case, then one can simply scale so that it has the same length as and the same proof idea applies.
Let 0r (1 + 8)0 er. We will show that for some carefully selected e, > 0, 0' satisfies (0', 00) > (0, vi) and ||0'||2 = ||0||2 = As a result, using Claim 5, it will follow that (i))") < (.) for all E [n], and thus L(0') L(Ø), as desired.
The condition (0', v8) > (8, v)) holds as long as 8(0, 8) + =(U,, x) > 0, which holds as long as E (n,0)) holds for all E [n]. Let M = 2 + max (0. -1/(0, x)), max; (v,,,))/ (v/ })}. Then letting 8 = M5 ensures that the aforementioned condition is satisfied.

---

Published as a conference paper at ICLR 2026
It remains to pick 8 > 0 such that ||0 1. We can write
||0' 12 = (8',0" 0²) = ((1 + 6)8 Máz, (1+ 810 + Mox)
= [2+00)2(11-)000000
= +1+2 + +20006000.))+000(() M²) + 62 2M (0,z)))
M2
Note that 1+ 1+2M M 2 1-11-M 2M M > 0 and 2+ 2+ < 0, by the choice of M
and since (0,z) E [-,,0) As a result, we have that for sufficiently large d > 0, ||01|| 1, and for sufficiently small 8 >> 0, |00'|2 < 1. By the intermediate value theorem, if follows that there exists > such that | 1, and this completes the proof.
O
Theorem 5.1 follows by combining Clain 6, which guarantees that minimizer exists, with Claim 8, which guarantees that any minimizer satisfies PO.
DISCUSSION AND FUTURE DIRECTIONS
We contribute to the recent effort to import ideas from social choice-cspeciee the ax- iomatic approach- into the study Al alignment, pluralistic aligument and multi-objective alignment. At the same time, classical social choice is largely framed for discrete settings (e.g,, complete rankings over a fixed option set), which do not directly reflect modern MI. pipelines. Tf social choice is to inform alignment, the axioms and tools must be adapted to the knobs that actually exist in these pipelines.
In this spirit, we analyze why a seemingly minimal axiom such Pareto Optimality (PO) can fail. Seeking models that better reflect training constraints, we focus on loss-based rules and pursue theoretically tractable proxies for practice: (1) enlarging the reward class, (2) requiring guarantees that hold out-of-sample, and (3) making the data-generating distribu- tion explicit. The first point acknowledges that, while many analyses assume unconstrained rewards, such optima are unlikely to be realized in practice. For (1), we establish a negative result for bounded-degree polynomial rewards and conjecture that fixed-width, fixed-depth MLPs over frozen embedding space fail for similar reasons.
As a step toward generalization, we show that structured settings (e.g., linear social choice), axioms specified over fixed alternative set admit natural analogues over the en- tire embedding space. In particular, between embeddings and b can inform different queries such as + b b-5 (noisy per- turbations) or a + r vs. + (shared direction, translated), providing leverage beyond the observed preferences. For (3); we prove that PO is achievable under data distríbutions in which both inter-embeddding distances and directions are suitably uniform/ batrrm/baalnnedd balanced. Tak- ing our simplified model as given, one can then examine the distribution of embedding differences e.g,, via PCA to assess how close it is to this regime.
the linear social choice model, pairwise judgments
While one can balance the lengths of comparison directions by renormalizing the loss, achiev- ing more uniform directional coverage may require reweighting comparisons, and in the worst case, collecting additional data. A natural next step is to derive explicit sampling bounds in the uniform setting. Beyond random sampling, promising direction is to formalize reweighting and data-query strategics can we gather data to attain PO in a more targeted (and more efficient) manner?
While proposing new axioms is not our focus, future research should consider how other axioms extend to embedding spaces and can be satisfied through data selection. Overall, our results point to a different way of addressing axiomatic violations within loss- based frameworks.
ACKNOWLEDGMENTS
Sonja Kraiczy was supported by the PIBBSS Fellowship (Principles Intelligence) and the Cooperative AT Foundation.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lowell Bruce Anderson, Helena Dandurova, James E. Falk, and Lana Yeganova. Rela- tionships between borda voting and zermelo ranking. Social Choice and Welfare, 32(3):
355-365, 2009. doi: 10.1007 10.0077/<00355-000003077
Kenneth J. Arrow. Social Choice and Individual Values. Wiley, 1951.
Kenneth J Arrow, Amartya Sen, and Kotaro Suzumura. Handbook of social choice and welfare, volume 2. Elsevier, 2010.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. orNiv preprint arxi::2204. 058662, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Consti- tutional Al: Harmlessness from AI feedback. arXiv preprint 2XXiv2212.08001 2022b.
Claude Berge. Topological Spaces: including a treatment of multi-valued functions, vector spaces, and converily. Courier Corporation, 1997.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs. i the mothod of paired comparisons. Biometrika,
1952. doi: 10.1093/bion /bimett///
39.3-4.324.
Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D Procaccia. Handbook of computational social choice, Cambridge University Press, 2016.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. Position: social choice should guide AI alignment in dealing with diverse human feodback. In Proceedings of the 41st International Conference on Machine Learning, pp. 9346-9360, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xic, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled AI feedback. In Forty- first International Conference on Machine Learning.
Jessica Dai and Eve Fleisig. Mapping social choice theory to RLHF.. In ICLR 2024 Workshop on Retiable and Responsible Foundation Models.
Miroslav Dudík, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Proceedings of the 28th Conference Learning Theory, pp. 563 563-587, 2015.
Edith Elkind, Martin Lackner, and Dominik Peters. Preference restrictions in computational social choice: A survey. arXiv preprinl ar w:Xx:22050090922 2022.
Peter C Fishburn. Probabilistic social choice based simple voting comparisons. The Review of Economic Studies, 51 (1):6683--66 692, 1984.
Luise Ge, Daniel Halpern, Evi Micha, Ariel D. Procaccia, Itai Shapira, Yevgeniy Vorob- eychik, and Junlin Wu. Axioms for AI Alignment from Human Feedback. Advances in Neural Information Processing Systems, 37:80439-80465, 2024.
Amelia Glaese, Nat McAloese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Im- proving alignment of dialogue agents via targeted human judgements. arXiv preprint ar arx:::2009 14875, 2022.
11

---

Published as a conference paper at ICLR 2026
Paul Gölz, Nika Haghtalab, and Kunhe Yang. Distortion of AI alignment: Does preference optimization optimize for preferences? arXiv preprint arXiv:2505.2311 2025.
Kihyun Kim, Jiawei Zhang, Asuman Ozdaglar, and Pablo A Parrilo. Population-
proportional preference learning from human leedback: An axiomatic approach. arXiv preprint arXiv:2506. 05619, 2025.
Patrick Lederer, Dominik Peters, and Tomasz Was. The squared Kemeny rule for averaging rankings. In Proceedings of the 25th ACM Conference on Economics and Computation (EC), pp. 755, 2024. doi: 1441145/377065538733221.
Saberto-Raffae Maura- Maura-Rivero, Marc Lanctot, Francesco Visin, and Kate Larson. Jackpot! aligument as a maximal lottery, 2025. URL https://arxiv: http:///axxi..oog/b//////..// org/abs//50.. 19266
Remi Munos, Michal Valko, Daniele Calandricllo, Mohammad Gheshlaghi Azar, Mark Row- land, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. Nash learning from human leedback. In International Conference on Machine Learning, pp. 36743 36768. PMLR, 2024.
Ritesh Noothigattu, Dominik Peters, and Ariel D Procaccia. Axioms for learning from pairwise comparisons. Advances in Neural Information Processing Systems, 33:17745- 17754, 2020.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference Machine Learning.
Ariel D Procaccia and Jeffrey Rosenschein. The distortion of cardinal preferences in voting. In International Workshop Cooperalive Information Agenls, pp. 317 331. Springer, 2006.
Ariel Procaccia, Benjamin Schiffer, and Shirley Zhang. Clone-robust AI alignment. arXiv preprint arXiv:2501.092 2025.
Tianyi Qiu. Representative social choice: From learning theory to AT alignment. arXiv preprint 20. 2024.
R. Tyrrell Rockafellar. Conver analysis. Princcton Mathematical Series. Princeton Univer- sity Press, 1970.
Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell Distributional prefer- ence learning: Understanding and accounting for hidden context in RLHF. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Informalion Processing Systems, 36:2511-225 15611 2023.
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF more difficult than standard RL? theoretical perspective. In Advonces in Neural Information Processing Systems 36 (NeurIPS 2023), 2023.
Jiancong Xiao, Zhekun Shi, Kaizhao Liu, Qi Long, and Weijie J Su. Thcoretical tensions in RLHF: Reconciling empirical succoss with inconsistencies in social choice theory. arXiv preprint urXiv:2506. 12350, 2025.
Erast Zermelo. Die Berechnung der Trrnbii-Erge als ein Maximumproblem der "ahhsshhiinicchkeitsrecnmm Mathemalische Zeilschrift, 29:436 460, 1929. doi: 10.1007/ BF01180541.
12

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
As an alternative to unguided preference elicitation ("which response do you prefer?"), sev- eral authors (Bai et al., 2022a; Glaese et al., 2022) proposed principle-guided supervision: annotators judge responses with respect to explicit critcria such as helpfulncss, honesty, and harmlessness (HHH). In current practice, RLHF often clicits binary preferences from users as a general signal without specifying how to evaluate "better," " and human comparisons are noisy commonly modeled with the Bradley Terry framework. Asking raters to compare responses with respect to a named principle (e.g., "which is more harmful?"" provides clearer target and greater control, reducing variance from idiosyncratic tastes (e.g., prefer- ring flattering style). Taking this perspective, we can view the principles as the voters. As models become more capable, human raters can be replaced or augmented with model-based judges, yielding Reinforcement Learning from Al Feedback (RLAIF)) This makes aligmment methods more scalable. E.g., it is cheap to score the same pairwise comparison against mul- tiple principles, which may legitimately disagree (a response can be more helpful yet more harmful). Constitutional AI (CAI) (Bai et al., 2022b) is a concrete framework built around RLAIF: it introduces explicit "constitution" "M curated set of principles which judges which of two responses better adheres to a given principle. Notably, when CAT was intro- duced, Anthropic reported Pareto improvement. on the helpfulness harmlessness harneeseessharnnesssnss frontier: increased harmlessness at fixed helpfulness relative to RLHF baselines. Many other works implement constitutional-style aligmment, often under different labels, by conditioning bi- nary comparisons on explicit criteria (rubrics attributes), using verifier- or sudgo-guided foodback, or training separate objectives that are later combined (Cui et al.; Sun et al,, 2023; Glaese et al., 2022).
B MISSING PROOFS FROM SECTION 4
B.1 PROOF OF CLAIM 1
Proof. Recall that r1 0. For cach j E {2...., m²,, we have that both the terms (1(y). and a (((rr) appear in L. Since l is convex and (0) > 0, it follows that limg-4400 {xx) +00. Furthermore, recall that É is lower bounded over R. As a result, it follows that [(r) + +0 when |ryl +00.
Since the loss function l is strictly convex, the function &i (r2,..., Tm.) + L(,, 0, r2..... mm) is also strictly convex. Thus, together with the previous paragraph, it follows that 6 attains its unique minimum
It remains to argue that the optimal rewards are ordered 0 = rf < < <.. <r. To prove
this, we first introduce some additional notation. Define h R R, a a-(z)) +1 a)- f(-x). Note that h is convex and differentiable, and thus its derivative N is nondecreasing. Furthermore, it satisfies
(0(0)= a ((()) (1 a) (() (2a 1) ((()) >
since a > 1/2 and (0). We can rewrite the total loss function as
m
m
m
C(1, 0,,,,,,r........................ mm) = ((0) +
M(- Σ j=it1
j=2
and thus the partial derivatives for all i 2 as
ac (r) = -2h'(-re) Σ h' h'(rx r1))
{-1
m Σ kmi+1
h'(re - -rk)-
Or,
Assume towards a contradiction that r 2 rit for some i E f2...., m 1}. We will handle the remaining case i = 1 separately at the end. Since r* is the optimal solution and L is
Here use the fact that for (r2,.... # (r/....,r'k) there exists j E (2. (2...., m) such that ry # r3y and thus & contains term for which E((ry + r2)/2) < l22))/2 ((r))/2
13

---

Published as a conference paper at ICLR 2026
differentiable, we have aC/Or, (r*) = 0 for all 2 2. However, we can write
OC (r")
0c
- 0r: (r*) = -()((-))- (-(-((())))) 1/ (r: - r(t1) -1((r - r(+1))
drit1
i-1
(h - -1 - (1 -)+1)) +
Σ (11 - -r1) - -1' -()) k=
S 222((r) - r(41) <0
where in the first inequality we used the fact that r7 2 ri+1 and h' is nondecreasing. In the
second inequality we used W((r - r2+1) 2 W(()) > 0. Since the partial derivatives are zero, this is contradiction. So, it must be that rf < r/41 for all 2 2.
It remains to show that rf < ră. Le., rž > 0. Assume towards a contradiction that r S O. We can write
ac
122
m -
m-
m
- (r")
W' ((-r)))
since every term h!(r;- - r3) appears once with a positive sign and once with a negative sign. Thus, can write
m ac
OL
m
m
(r*) + Ör2
(r")= = -2 Σ
M (((r))-A((--)))) 4-1(-15) + Σ
Or:
i=2
k=3
--hh +2 > (N/(1 - r() 1 h1' (-rri)
<-th'(-ri) <0
where in the first incquality we used the fact that r S 0 and h' is nondecreasing. In the second inequality we used h((-r)) 2 (0) > 0. Since all the partial derivatives are zero, this is contradiction. So we also have r r---
C MISSING PROOFS FROM SECTION 5
C.1 PROOF OF CLAIM 5
Proof. First of all, note that since ||0||2 = 10'1|2 and (0', v.) > (0, vy), it must be that 0 0 and 0' # O. Without loss of generality, since the statement of the claim does not change if we rotate the space, we can assume that v2 = e₁ is the unit length vector with entry in the first dimension and 0 otherwise.
Next, we argue that, without loss of gencrality, we can assume that 62 2 0 and 0, = for all j 2 3, and similarly for 8' Let R' Id-1 + Rd-1 denote the rotation around the origin in (d -)-dimensional space that maps (02,... ..) to (a, Oe.......o .),, where 2 0. Define R Rdd -> Rd by R(x) = (x1, R' (12..... .aa)). Note that [[(0]]]22 = ||0| and (R(0), u)) = (8, vz), since U, 61. We will show that (R(()) (.). Note that R is bijection and we have
(R(0), = 0121 + (R' (02...... (82....,0a)). 0,),, (f = 01σ1 + = (0, (x))
R' Ra'i (z2...., x.))
where R1-1 is the inverse rotation to R. We can thus write
(,((())) - f( - 20 ((-(R(0), dr =
[
E(-(0, R (~-(x))) da
JUESU 1,120
{(- x)) dx
JNESA-I,
(,(
14

---

Published as a conference paper at ICLR 2026
since R- is smooth bijection that preserves distances. Thus, can assume that 02 2 and 8 = for all j 2 3, and similarly for 8'.
Let T be the rotation that maps 0 to 0', We can write
(()) = /
LEESH 12 2,120
{1-(0, 2)) dx
JESS
-288-111-110011 (x)))) dz
8(- (T)) x)) dx
((-(0', dx f
1288--111--10 20 ((-(0', dr.
(1) 20
We can also decompose
[(0') /
((- x)) dx
JuESd-1,2133 21120
115)--211-11))22000 ((-(f'xx)) dx /
(xE8)-1,171(00) f(-(8', (f'''())))))) x\) dx.
Thus, we obtain
[(0) - C(0') /
/ -15---------llloloa E(-(6', (1-(6'xz)) x\) dæ
LT ((-(0', x)) dr
(T)) 50,,, ((10'xx)) - ((-(0'(, x²) de
JZES4 1/T (x)) <0, 1110' x)) - 11-(000,, xx))
(0',''))))) de.
Note that the domain fir E 54-1, (x)] 0, > 0) has positive measure, since (0', (4) > (0, vz). Furthermore, we have {(t) > {(-t) for all 0.7 As result, we and Iı 0, then we will
can show that (0',x) x) > 0 for all x 8d-1 with [x)] obtain [(0) - C(0') > O, as desired.
Recall that assunne without loss of generality that 02, 02 2 0 and 0, 03 0 for all j 22 3. Furthermore, by assumption we have 01 > 01. Consider any x E gd-1 with (0', S 0 and I1 > 0. Assume towards a contradiction that [T (x)]1 < 0. The rotation T maps B' to A which both lie in the upper hemisphere, and 6, > 01. Thus, the rotation is counter-clokwwiee and of angle at most π. Since and 02 = 0, ie., is the negative æ axis direction. Indeed, since Ii 0, if rotating x in the counterclockwise direction by some angle < yields point y with V/1 < 0, then rotating it by any angle/ that the rotation T-1 that maps 0' to the negative axis direction also maps any point with (0', x) < 0 to a point y = T (x) with yn 2 0, a contradiction.
0, we can assume without loss of generality that 0, < 0
(a, т] will also yield a point y with y1 < 0. Now it is easy to
C.2 PROOF OF CLAIM 6
Proof. We show that for any M R, there exists a > such that if ||0||2 t then conclude that L attains its minimum. Observe that since at least two voters are distinct, say iT and U2, there exists x < such that for any 8 ddd-1 there exists E {1,2] such that (0, v)) I Here assume that [ll llz = | = without loss generality.
C()) > M. Then, since C(0) is finite,
Let M be arbitrary. Let 0 E Sd-1 be arbitrary. Without loss of generality, say that voter in = én is such that (0, v1) & k. Since the loss function is lower bounded, there exists
TThis follows from the fact that nondecreasing and & (())) 0.
15

---

Published as a conference paper at ICLR 2026
K such that
[(t0) K Ci(t0) K
((-(t0,x)) dx
JuESU 1,120
[
2K
{( ((-(t0,x))
dx
12ESC-1,0100.00 2=1,0==_0020000015555
where [(x,d) d) < is such that the set for E Su-1 x1 positive measure bounded away from zero by at least g d) > 0. As result, we obtain that
(0,2) [Nx, d)} has strictly
[(t0) 2K + g(K, d) 6(-t ((, M for sufficiently large t > 0, since l is strictly increasing for positive inputs.
C.3 PROOF OF CLAIM 7
Proof. Let a E 5d-1 be such that (Uz, 2) > 0 for all voters i E [n]. We can write
L,(l [ Jress ((0,))20
f(- 2, dr.
Taking the derivative with respect to La L 0 we obtain
ã c((- [
d
/
(2, dz - (())
(2, dr < 0
JacSs-1,(m,a)20
Jesu 1,1
which follows from (((0) > and
J
dx > 0 (z,x) [ (maa))20
which follows from (u, 2) > 0 using similar ideas to those in the proof of Claim 5.
C.4 PROOF OF THEOREM 5.2
Proof. Consider 2 and let = (), = (1,0). For E [1/4, 1/2], we consider the instance where a fraction p the voters are in direction U1, and the remaining fraction
(1 - p) are in direction v2. Thus, the total loss function that is minimized over 0 E R2 can equivalently be wrillen as
(0,) do +2(1111-- p) o
C(0,p) = 2p
(() dx.
By Claim 6, L(-P) attains its minimum for each E [1/4,1/2]. Furthermore, since é is strictly convex, is L(:P)) and thus the minimizer of L(.p) unique.
For any p <1 1/2, 12 is the strict majority voter and so 0 = 02 is the only pairwise majority consistent direction. In order to show that PMC is not satisfied, it thus suffices to prove that for some < 1/2, the minimizer of the total loss is different from direction U2.
First, for p 1/2 we can write
I
C(0,
JEEST
DEST xx) dx (((- x)) ((-(0,xx)) dx,
((-(0,z)) dr
JEEST .220 2220
JES .120
TThis follows from the fact that is convex and (((0) > TThis follows from standard results in convex analysis (Rockafellar, 1970) together with the observation that for any b, the set of points on 8 such that (a b,z) = has measure zero (on 8.).
16

---

Published as a conference paper at ICLR 2026
where R denotes the 90" clockwise rotation, i..., R(Ø) (82. -01)- Let #' (02, 01) and note = (8', (Ø1. -22))- Since the domain of integration is symmetric with respect to the transformation (I1, + (I1, -22), obtain
that (R(0), = 8- - 02
((0,1/2) = -f JJEST 22120
(((-(0', x)) + ( x)) dz.
It follows that 1(0, _('', 1/2), i.e,, the loss invariant under permuting 0₁ and O2. By convexity,
1 ((0000/2211))) - x
((0',1/2)) C(0, 1/2),
80 the unique minimizer of C(., 1/2) satisfies 01 = 0z. From Claim 7, we know # 0. By Theorem 5.1, PO holds for this instance, 0" lies in the positive cone of and V2, and in particular 82 > 0.
Now let M 3| and ( {0 E R ||||| M.. The map L: B(M) 11/4 R continuous, so by Berge's theorem (Theorem 4.2) the function"
L: [1/4,1/2| B(M),
(p) arg min C(0.p) (EB(M)
is continuous. Thus, for sufficiently small 6' E (0, 1/4).
(1/(1/2 - 8) - 00 < 03 < |00))
80
[1//1 - 5)11 1 |- (1/2 6) 0*l < 2| < M.
In particular, L (1/2-88) -8) does not lie the boundary of (MM), and therefore also minimizes C(1/2 - 6,8) over all / E R
Finally, any (t, 0) with > 0 has distance at least 0% from 0. L112 6) is not scalar multiple of the majority voter U2 = (1, 0), and therefore PMC is not satisfied when p 1/2 12-68 8.
10For each e 11/4,1/2) the minimizer unique.
17


Reviews:
[{"review_id": "qRLgDhqDeu", "reviewer": "Reviewer_KC58", "summary": "This paper investigates when loss-based training rules, such as RLHF, satisfy fundamental social-choice axioms, focusing on Pareto Optimality (PO). The authors work in the linear social-choice model of Ge et al. (2024). They extend the analysis to from linear to polynomial reward functions and prove that PO violations persist for any polynomial, establishing a general impossibility. Finally, they take a data-centric perspective: when pairwise-comparison data uniformly cover the embedding space, they prove that PO is satisfied. This yields a practical takeaway, axiomatic guarantees can be restored not by changing loss functions but by ensuring balanced sampling or reweighting of preference data.", "questions": "see weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The main strength of this paper lies in its two key theoretical results. \n\nTheorem 4.1 extends the findings of Ge et al. (2024) from linear to polynomial reward functions, showing that Pareto Optimality (PO) remains violated even when polynomial reward models are used. \n\nTheorem 5.1 demonstrates that under idealized conditions, specifically, unit-length representations and uniformly distributed directions, a linear reward model can in fact satisfy PO.", "weaknesses": "The main limitation of this paper lies in its significance comparing to previous results and its practical implications. First, several closely related works are not discussed [1,2,3,4]. As noted in [1,2], the nonparametric global solution of the reward-training loss does satisfy Pareto Optimality (PO). In practice, reward models are typically large language models (LLMs) implemented with transformer architectures and thus possess strong function-approximation capabilities. Consequently, practical reward models are much closer to expressive models than to the simplified linear or polynomial formulations considered in this paper. Hence, the extension from linear to polynomial reward functions has limited practical significance.\n\nSecond, Theorem 5.1 attempts to provide a positive result by showing that PO can be satisfied under certain conditions. However, the assumption of a uniform distribution is highly idealized and far from practical scenarios. Moreover, existing work do have some more practical positive results. They already indicate that when the reward model has sufficient approximation capacity, PO can in fact be satisfied.\n\nLine 106: The statement “NLHF coincides with maximal lotteries” is missing an appropriate citation. The current reference (Fishburn, 1984) pertains to the definition of maximal lotteries, not to the connection between NLHF and maximal lotteries. Reference [3] establishes this equivalence and should be cited here. In addition, [4] applies these axioms to propose a new alignment approach. \n\nLastly, the English writing and representation should be improved. The current manuscript does not fully adhere to the ICLR style guidelines, and the use of fonts and formal structure is inconsistent with the official template. \n\n[1] Ritesh Noothigattu, Dominik Peters, and Ariel D Procaccia. Axioms for learning from pairwise comparisons. Advances in Neural Information Processing Systems, 33:17745–17754, 2020.\n\n[2] Jiancong Xiao, Zhekun Shi, Kaizhao Liu, Qi Long, Weijie J Su. Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory, 2025.\n\n[3] Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, and Kate Larson. Jackpot! Alignment as a maximal lottery. arXiv preprint arXiv:2501.19266, 2025.\n\n[4] K Kim, J Zhang, A Ozdaglar, PA Parrilo Population-Proportional Preference Learning from Human Feedback: An Axiomatic Approach, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEs8hKp6Ui", "reviewer": "Reviewer_U5VA", "summary": "The paper take the impossibility result from Ge et al. (2024) regarding Pareto optimality of reward model training, and (i) generalize it to impossibility for polynomial hypothesis classes (instead of linear ones); (ii) show that the impossibility dissolves when we have preference samples that covers all directions of the embedding spaces uniformly and that the between-response distance in each sample is constant. By doing so, it argues that (i) the impossibility is real and not solved by simply adding representational capacity, yet (ii) can be resolved in practice via data distribution interventions.", "questions": "- Potential typo: In the proof of Thm 4.1, on line 298, should it be $j=\\\\lceil\\\\frac{i}d\\\\rceil$ instead of $j=(i-1)\\\\bmod d$?\n- What do you think are direct implications for data collection practices in preference alignment / constitutional AI? Are there ways you can quickly test them?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Significance: Ge et al. (2024) is a seminal work on an important topic (pluralistic alignment). The work addresses the most important shortcoming of Ge et al., which is its practical implications for today's model training pipelines. It shows what the data distribution needs to be like for the impossibility result to vanish, and the resulting suggestions (uniformly distributed embedding directions & constant embedding distances for preference pairs) seem highly actionable.\n- Soundness: I followed the proofs in the body and spot no errors (except possible typo that I mentioned in the questions section). The \"claims\"/lemmas involved in the proofs seem generally easy to give proof sketches to, so I did not verify their proofs in the appendix.\n- Presentation: The authors went with the simplest structure (first background, then present results in logical succession), and it worked well. The message is also clear.", "weaknesses": "- It is a pity, although definitely not a fatal flaw, that the paper gives no empirical validation on language models despite its aim at guiding practice. It would be helpful to see, e.g., comparisons of more uniform vs less uniform sampling methods for preference data collection, in terms of impact on downstream alignment performance against multiple constitutional principles.\n\n**Minor points (don't affect score):**\n- Presentation: Illlustrations would make following the proofs easier. For the proof of Thm 4.1, an illustration of the parallel lines + candidate points + voter vectors will be useful (same goes for Sec 3). For the proof of Thm 5.1, an illustration of a half-plane (i.e. $\\\\{x:\\\\langle x,\\\\theta\\\\rangle>0\\\\}$) gradually rotating towards the intersection $D$ of the voter half-planes will be useful.\n- One unmentioned related work is [1] which found circularity in the hypothesis class's privilege graph as a condition for the impossibility. This may give a high-level reason why the impossibility continues to hold for the polynomial hypothesis class.\n\n[1] Representative Social Choice: From Learning Theory to AI Alignment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gjndxepPVT", "reviewer": "Reviewer_ZJsy", "summary": "The paper analyzes axiomatic guarantees (Pareto Optimality) for preference-based alignment under a loss-minimization view. It (i) shows PO violations persist even when moving beyond linear rewards to bounded-degree polynomial rewards; (ii) gives intuition via minimal counterexamples; and (iii) proves that in an idealized setting with uniform directional coverage of pairwise data, broad classes of loss-based rules do satisfy PO—suggesting a data-centric recipe (balanced coverage) for constitutional-style alignment without changing training pipelines.", "questions": "1.  Coverage diagnostics. How should practitioners measure directional coverage in real datasets (e.g., a concrete statistic or test), and what thresholds approximate the theorem’s regime?\n2.  Finite-sample guarantees. Can you provide sample-complexity or robustness bounds showing how PO violation probability decays as coverage improves?\n3.  Beyond PO. Do analogous results (negative or positive) hold for other alignment-relevant axioms such as Pairwise Majority Consistency under similar coverage assumptions?\n4.  Practical recipe. Can you outline a reweighting/sampling procedure (pseudocode) that moves an arbitrary dataset toward the desired coverage while preserving data efficiency?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  Crisp negative/positive results. Clear impossibility for polynomial rewards and a clean, sufficient condition (uniform coverage) restoring PO.\n2.  Training-pipeline relevance. Positions guarantees within standard loss-based training, aligning with RLHF/RLAIF practice rather than bespoke voting rules.\n3.  Intuition + formality. Minimal counterexamples illuminate why PO fails; proofs formalize the phenomenon.\n4.  Actionable takeaway. Suggests data-centric interventions (balancing directions/lengths) as a path to axioms in practice", "weaknesses": "1.  Idealized coverage assumption. The uniform directional coverage condition is strong; guidance on how much imbalance is tolerable is missing (no finite-sample or robustness bounds).\n2.  Empirical gap. No experiments (even synthetic) to quantify coverage diagnostics (e.g., spectrum of pairwise direction differences) or to validate the recipe on real preference data.\n3.  Scope of axioms. Focuses on PO; discussion of other axioms (e.g., Pairwise Majority Consistency) is limited to related work, with no analogous guarantees.\n4.  Practical measurement. The paper hints at PCA/geometry checks but lacks a concrete coverage diagnostic or reweighting algorithm practitioners can run.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PgnmpUIMw2", "reviewer": "Reviewer_6CiF", "summary": "This paper studies axiomatic violations in AI alignment, specifically how loss-based reward models can violate Pareto optimality even when all principles agree on preferences. The authors show polynomial rewards don't resolve this issue, but prove that uniform data coverage can restore axiomatic guarantees in constitutional-style alignment methods.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides rigorous mathematical analysis of axiomatic violations in AI alignment, extending the linear social choice framework with formal proofs and theorems.\n2. The paper proves that even polynomial reward functions (beyond linear) still fail Pareto optimality, strengthening the robustness of the violation result.\n3. The paper provides intuitive examples and simplified cases that clearly illustrate why Pareto optimality violations occur", "weaknesses": "Only examines Pareto optimality while other important social choice axioms (e.g., Pairwise Majority Consistency) receive limited attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)