Title: Axioms for AI Alignment from Human Feedback
Venue: NeurIPS 2024
Accepted: yes
Keywords: Social choice

reinforcement learning from human feedback

AI alignment
Abstract: In the context of reinforcement learning from human feedback (RLHF), the reward function is generally derived from maximum likelihood estimation of a random utility model based on pairwise comparisons made by humans. The problem of learning a reward function is one of preference aggregation that, we argue, largely falls within the scope of social choice theory. From this perspective, we can evaluate different aggregation methods via established axioms, examining whether these methods meet or fail well-known standards. We demonstrate that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms. In response, we develop novel rules for learning reward functions with strong axiomatic guarantees. A key innovation from the standpoint of social choice is that our problem has a *linear* structure, which greatly restricts the space of feasible rules and leads to a new paradigm that we call *linear social choice*.

Axioms for AI Alignment from Human Feedback

Abstract

In the context of reinforcement learning from human feedback (RLHF), the reward function is generally derived from maximum likelihood estimation of a random utility model based on pairwise comparisons made by humans. The problem of learning a reward function is one of preference aggregation that, we argue, largely falls within the scope of social choice theory. From this perspective, we can evaluate different aggregation methods via established axioms, examining whether these methods meet or fail well-known standards. We demonstrate that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms. In response, we develop novel rules for learning reward functions with strong axiomatic guarantees. A key innovation from the standpoint of social choice is that our problem has a linear structure, which greatly restricts the space of feasible rules and leads to a new paradigm that we call linear social choice .

1 Introduction

The alignment of AI models with human values is widely recognized as a crucial task. A prominent method for this task, reinforcement learning with human feedback (RLHF), has been used in different applications, such as robotics [4, 17] and recommendations [28, 1]. Recently, RLHF has attracted significant attention as a tool for fine-tuning large language models (LLMs) [22, 32, 26]. A typical implementation of RLHF involves learning a reward model using a pre-trained LLM, which is then utilized to fine-tune an existing LLM. During the learning step, human feedback is provided in the form of ordinal comparisons, and a reward function is learned from these. The most common learning method assumes an underlying random utility model such as the Bradley-Terry-Luce (BTL) model [5, 22, 8] and computes a reward function that corresponds to a maximum likelihood estimator for the observed comparisons.

Is this the 'right' way of aggregating individual preferences towards a socially desirable reward function? To answer this question, we draw on social choice theory , a field that studies collective decision making through a mathematical lens [6]. The maximum likelihood estimation approach is in line with a well-established body of work that assumes that different human participants have preferences stemming from noisy estimation of a common ground truth, and the goal is to learn this ground truth as accurately as possible [29]. But this is not the case when it comes to questions of AI

alignment, where individuals can have legitimate differences of opinion rooted in different values or priorities.

We argue that when preferences are truly heterogeneous, the axiomatic approach -which rose to prominence in social choice with the work of Arrow [2] - may be more suitable. This approach analyzes the desirability of aggregation methods by their satisfaction of certain axioms that capture notions of consensus, fairness, and economic efficiency. Specifically, we are interested in the axiomatic properties of aggregation methods that take ordinal preferences as input and output a reward function. We address the following two research questions: What axioms are satisfied by aggregation methods used by existing RLHF algorithms? And are there alternative aggregation methods that offer stronger axiomatic guarantees?

1.1 Our Approach

In social choice theory, axioms are typically defined for rules that map rankings over candidates to a single winner (social choice functions) or a ranking of the candidates (social welfare functions). By contrast, we are interested in rules that assign a reward to each candidate. This gap is easy to bridge, though: we simply consider a ranking of the candidates by in descending reward order.

A much more significant gap is that in classical social choice, all relevant candidates appear in the input preferences, whereas in our setting (where candidates correspond, e.g., to prompts and their responses), we are only given preferences over a relatively small set of candidates identified by their (known) features , and we need to generalize from this information. In practice, this entails using a restricted-commonly, parametric-class of reward models which map candidate features to real-valued rewards, and which we fit to existing data.

Specifically, we assume that a linear reward function defined by a parameter vector determines the reward of each candidate by computing the inner product of the parameter vector and the feature vector of the candidate; these modeling choices are consistent with prior and concurrent work [31, 30, 15] and aim to capture the practice of RLHF. 1 Each human participant (henceforth referred to as a voter ) is associated with a parameter vector, which is unknown to us and is used to specify ordinal preferences over the candidates. Our task is to design linear rank aggregation rules , which aggregate rankings induced by these individual linear functions 2 into a collective ranking that is also induced by a linear function; this is a new paradigm in social choice, for which we coin the term linear social choice .

To evaluate linear rank aggregation rules, we adapt fundamental axioms from social choice theory [6]. The first is Pareto optimality (PO) , which requires that if a candidate a is ranked above candidate b in every input ranking, then the resulting ranking should rank a above b . This is seen as a basic requirement and is satisfied by every standard voting method in the classical setting.

The second axiom is pairwise majority consistency (PMC) : If there exists a reward function that generates a ranking where, for each pair of candidates, a majority of voters agree with the ranking, then the resulting ranking should match that ranking. This axiom is an extension of Condorcet consistency to rankings, and is satisfied by some, but not all, standard voting methods in the classical setting.

1.2 Our Results

We start by examining, in Section 3.1, a family of loss-based rules that finds a ranking induced by a parameter vector that optimizes a measure of loss; this measure increases for every disagreement with a voter on a pair of alternatives, where the larger the difference in rewards, the larger the penalty. Crucially, by plugging in binary cross-entropy loss we can recover the BTL model. Our first main result is that whenever the loss function is weakly convex and nondecreasing, or strictly convex conditions satisfied by binary cross-entropy loss, as well as, e.g., exponential and hinge loss - the

corresponding rule fails both PMC and PO. This result suggests that the prevailing practice of RLHF is flawed from an axiomatic viewpoint.

In Section 3.2, we take a first step towards addressing this shortcoming. We modify the loss-based formulation to focus on majority preferences rather than individual preferences. This modification defines a family of rules that are PMC, but we show that all of them fail PO by establishing an even stronger impossibility result: In stark contrast to the classical setting, any linear rank aggregation rule that depends only on majority preferences must fail PO.

In order to achieve both PO and PMC, we design (in Section 4) a linear rank aggregation rule that we call Leximax Copeland subject to PO . Not only does it satisfy our two main axioms, it also satisfies two additional ones, majority consistency and winner monotonicity .

To summarize, while widely applied rules fail to meet basic axioms, there are alternative methods that are desirable from this viewpoint. Our approach, therefore, provides a discriminative lens through which to evaluate RLHF methods and AI alignment methods more broadly.

1.3 Related Work

During the eight months in which we have actively worked on this project (from September 2023 until May 2024) - and especially in the first few months of 2024 - a slew of independent, concurrent papers seeking to build bridges between social choice and RLHF have become publicly available [10, 12, 19, 30, 23, 7, 15, 27, 25]; this surge of interest points, in our view, to the importance of the agenda.

Three of those papers are position papers that conceptually support our work in that they discuss the possibility of applying an axiomatic approach to RLHF [10, 12, 19], although they do not provide any technical results. By contrast, existing technical papers on RLHF do not take an axiomatic approach. Of the concurrent technical papers, the one that is most closely related to ours is that of Siththaranjan et al. [25]. They show, among other results, that the ranking induced by the reward function that the MLE estimator of the Bradley-Terry-Luce Model returns follows the famous Borda count rule when unrestricted reward functions are allowed. In the classical setting, Borda count has strong axiomatic guarantees, including PO (but not PMC). However, it cannot be realized as a linear rank aggregation rule, and it is arguably impractical for RLHF.

Our work builds on an earlier study by Noothigattu et al. [21], which explores the axiomatic properties of reward functions defined as MLE estimators of underlying random utility models. The key difference is that their approach allows for general reward functions, not just linear ones, and they do not consider features at all. Unlike our findings, they show that the BTL model satisfies Pareto Optimality under these conditions. Additionally, they find that pairwise majority consistency is violated even without assuming linearity. However, their results strongly depend on varying the number of comparisons across different pairs of candidates. By contrast, our findings demonstrate that pairwise majority consistency is violated even when the number of comparisons is equal across all pairs of candidates.

2 The Linear Social Choice Model

Let C be a set of m distinct prompt/responses, referred to as candidates , and let V = { 1 , . . . , n } be a set of n human participants, known as voters . We denote by R d the d -dimensional real space in which both candidate feature vectors and chosen parameter vectors lie.

Each candidate c ∈ C is associated with a distinct feature vector x c ∈ R d . A parameter vector θ ∈ R d induces a linear reward function r θ : C ↦→ R defined by taking the dot product with feature vectors r θ ( c ) = ⟨ θ, x c ⟩ . We will primarily be interested in how these parameterized functions rank the candidates by reward. Let R a ≻ b = { θ | r θ ( a ) ≥ r θ ( b ) } be the region where the reward of a is at least as large as that of b . Note that R a ≻ b and R b ≻ a split R d into two half spaces, separated by the hyperplane orthogonal to x a -x b . Parameter vectors θ on the hyperplane have r θ ( a ) = r θ ( b ) , while rankings in the interior of either half-space strictly rank one over the other.

For a ranking σ over the candidates, we say that θ induces σ , denoted θ ▷ σ , if a ≻ σ b implies r θ ( a ) ≥ r θ ( b ) . Let R σ = { θ | θ ▷ σ } be the set of vectors θ that induce it. Note that this can be written as the intersection of corresponding half spaces R σ = ⋂ a,b : a ≻ σ b R a ≻ b . Further, the

collection of { R σ } essentially form a partition of R d , covering the space and intersecting only at their boundaries.

̸

We call a θ non-degenerate if it is fully on one side of each of the separating hyperplanes, i.e., r θ ( a ) = r θ ( b ) for all a, b ∈ C . Non-degenerate parameter vectors lie in the interior of some R σ , and thus induce exactly one ranking. We call σ feasible if R σ has a nonempty interior, i.e., is induced by some nondegenerate θ . 3

Each voter i ∈ V submits a ranking over the candidate σ i . We assume that the feature space is rich enough that voter preferences can be captured via non-degenerate parameter vectors. In other words, we assume that each σ i is feasible. We refer to the vector of voter rankings π = ( σ i ) i ∈ V as a profile . Further, for two candidates a, b , we write n a ≻ b ( π ) := |{ i ∈ V | a ≻ σ i b }| for the number of voters that prefer a to b , and w a ≻ b ( π ) = n a ≻ b ( π ) /n for the proportion of such voters. When the profile π is clear from context, we may shorten these to n a ≻ b and w a ≻ b , respectively.

We define a parameter aggregation rule as a function that takes as input a profile π and outputs a parameter vector θ ∗ . Our goal is to design parameter aggregation rules such that r θ ∗ satisfies desirable properties with respect to the voter preferences. However, as the properties we care about will only be with respect to how r θ ∗ ranks the candidates, it will be more convenient to work with what we call linear rank aggregation rules that take as input a profile π and output a feasible ranking σ . There is a natural way to interpret a parameter aggregation rule as a linear rank aggregation rule, namely, output any feasible ranking induced by θ ∗ . The exact properties of the parameter aggregation rule could in principle be sensitive to the tie-breaking of non-degenerate outputs, however, all of our results will be robust to such tie-breaking. 4

We pay special attention to a prominent family of rules from social choice theory referred to as C 1 rules [14], whose outputs depend only on majority relationships, i.e., they only need to know for each pair of candidates ( a, b ) whether the majority prefers a or b .

In our study, we examine several axioms borrowed from social choice theory to evaluate the reasonableness (fairness) of our aggregation mechanisms. These axioms include:

Definition 2.1 (Pareto Optimality) . A linear rank aggregation rule f satisfies Pareto optimality if, whenever every voter prefers candidate a over candidate b on π , i.e., w a ≻ b ( π ) = 1 , then candidate a is ranked higher than candidate b in the output ranking, i.e., a ≻ f ( π ) b .

Definition 2.2 (Pairwise Majority Consistency (PMC)) . A ranking σ is called a PMC ranking for profile π if for all a, b ∈ C , a ≻ σ b if and only if a majority of voters rank a ≻ σ i b , i.e., w a ≻ b > 1 / 2 . A linear rank aggregation rule satisfies PMC if, when a PMC ranking σ exists for the input profile π and σ is feasible, then f ( π ) = σ .

Note that a PMC ranking for each π need not exist, but when one does, it is unique. The words ' σ is feasible' allude to the possibility that no non-degenerate parameter vector θ induces the unique PMC ranking. Indeed, we have such an example; see Appendix B for details.

Our research question, then, is whether these axioms can be simultaneously satisfied by linear rank aggregation rules. Our approach seeks to provide a concrete illustration of how theoretical insights from social choice can inform practical algorithm design in RLHF.

3 Loss-Based Rules

3.1 Standard Loss Formulation

We begin our study of linear social choice by considering a quite broad yet natural class of rules that capture how RLHF is currently being done. Their core idea is the following: when considering parameter vector θ , for each voter i that ranks a pair of candidates a ≻ i b , we should incur some loss for giving b a higher reward than a . To formalize this, let ℓ : R → R be a loss function , which we assume is nonnegative. We can then choose a parameter vector minimizing

̸

Note that the BTL model fits within this framework using ℓ ( x ) = ln(1 + e x ) , i.e., binary crossentropy loss . 5 One caveat to this approach, however, is that an optimal θ need not be well-defined: it is possible that no minimum is attained. Fortunately, since we only care about rankings induced by optimal parameter vectors, we can conveniently remedy this by saying the output is any ranking that is induced by parameter vectors that are arbitrarily close to optimal. More formally, we say that a linear rank aggregation rule f minimizes ℓ if for all σ = f ( π ) ,

Even if no minimum is attained, there is always a choice of feasible ranking σ such that Equation (1) is satisfied.

With this definition in hand, we proceed to our first main result, which spells rather bad news for this class of rules: Any loss-based aggregation rule using a nondecreasing and convex loss function (of which BTL is one, and hinge loss is another) will fail our two core axioms, PMC and PO. This paints a negative picture for current RLHF methods with respect to their social choice guarantees. Note that we will exclude the discussion of loss functions with a global minimum at zero, like ReLU, because the loss minimizer will be zero, making all rankings vacuous consequently. And we have focused on convex loss functions due to their practical optimization ease.

Theorem 3.1. If a linear rank aggregation rule f optimizes a loss function ℓ that satisfies inf x ℓ ( x ) < ℓ (0) and is either nondecreasing and weakly convex, or strictly convex (and possibly nonmonotone), then f fails PMC and PO.

̸

Proof. Fix a loss function ℓ satisfying the theorem conditions. Note that since ℓ is convex, we may also assume it is continuous [24, Corollary 10.1.1]. Furthermore, since inf x ℓ ( x ) < ℓ (0) , we know that there exists x = 0 such that ℓ ( x ) < ℓ (0) . The case where x > 0 is relatively simple (as such loss functions lead to unnatural behavior), and we handle it at the end of the proof. For now, we assume that there exists x < 0 such that ℓ ( x ) < ℓ (0) . Note that this also implies that for all y ≥ 0 , ℓ is lower bounded by the affine linear function connecting ( x, ℓ ( x )) and (0 , ℓ (0)) , and thus, lim x →∞ ℓ ( x ) = ∞ .

We begin with a small instance of just three candidates C core = { a, b, c } to gain some traction on how ℓ behaves. We will later extend this instance with additional candidates to demonstrate a profile where PO and PMC fail. The candidates will have feature vectors x a := (2 , 1) , x b := (1 , 1) , and x c := (0 , 0) , respectively. Furthermore, a p -fraction of voters (for p to be chosen later) will rank a ≻ b ≻ c , while the remaining (1 -p ) -fraction will have inverted preferences, ranking c ≻ b ≻ a . 6

Let

̸

̸

̸

with w x ≻ y ∈ { 1 -p, p } be the loss function on this instance (scaling n x ≻ y down to w x ≻ y leads to an equivalent formulation). Let g ( x ) = p · ℓ ( -x ) + (1 -p ) ℓ ( x ) . Note that we can rewrite L com as

Note that r θ ( c ) = 0 for all θ , so we can simplify this to

We will consider an unconstrained version of this problem where we are free to choose rewards r a , r b ∈ R arbitrarily, and later show by which vectors θ these optimal values can be induced. That is, we will first find r a , r b ∈ R minimizing

Let OPT core = { θ | L core ( θ ) = inf θ ′ L core ( θ ′ ) } and OPT unconstr { ( r a , r b ) | L unconstr ( r a , r b ) = inf r ′ a ,r ′ b L unconstr ( r ′ a , r ′ b ) } be the set of minimizers for these two loss functions. In Appendix A, we establish the following results about these optimal sets.

Lemma 3.2. There exists a rational p ∈ (1 / 2 , 1] and values A 1 < A 2 with A 2 > 0 such that OPT unconstr is nonempty and for all ( r a , r b ) ∈ OPT unconstr , r a > A 2 and r b ≤ A 1 .

Lemma 3.3. Suppose Lemma 3.2 holds for values p, A 1 and A 2 , then, for this same choice of p , OPT core is nonempty and there exist A 3 and A 4 with A 3 > 0 such that for all ( θ 1 , θ 2 ) ∈ OPT core , θ 1 > A 3 and θ 2 < A 4 .

We will now explicitly construct a family of instances with candidate feature vectors parameterized by a value ε ∈ R such that for sufficiently small ε > 0 , the output of f fails the two axioms. Fix p, A 3 and A 4 from Lemma 3.3, and choose δ with 0 < δ < 1 such that δA 4 -A 3 < 0 ( δ < A 3 /A 4 works if A 4 > 0 , and otherwise, any 0 < δ < 1 will do).

Each instance will have six candidates, which we will think of as two groups of three, C = C core ∪ C copies . The first group C core = { a, b, c } will be the same as the three-candidate instance from above, while the second group C copies = { a ′ , b ′ , c ′ } will be new. The candidates a, b, c will still be located at x a := (2 , 1) , x b := (1 , 1) , and x c := (0 , 0) , respectively. The candidates a ′ , b ′ , c ′ will be located near their undecorated counterparts at x a ′ := x a + ( -ε, 0) , x b ′ := x b + ( -ε, 0) and x c ′ := x c +( -ε, δ · ε ) .

Next, we describe the voter preferences. A p -fraction of voters will have the ranking a ≻ a ′ ≻ b ≻ b ′ ≻ c ′ ≻ c , and the remaining (1 -p ) -fraction of voters will have ranking c ′ ≻ c ≻ b ′ ≻ b ≻ a ′ ≻ a . As long as 0 < ε < 1 (which will be the case for our final chosen ε ), these are both feasible rankings. The former is induced by the nondegenerate feature vector (1 , 1) 7 and the latter by ( -1 , 0) . 8

For each ε ∈ R , let

̸

with w x ≻ y ∈ { 0 , 1 -p, p, 1 } be the loss function we are optimizing using candidate locations parameterized by ε .

We will show that for sufficiently small ε > 0 , inf θ ∈ R c ′ ≻ c L ε ( θ ) > inf θ L ε ( θ ) . This means that f must output a ranking with c ≻ c ′ . Observe that this is a PO violation because all voters agree that c ′ ≻ c . Furthermore, this is a PMC violation because a majority of voters have the ranking a ≻ a ′ ≻ b ≻ b ′ ≻ c ′ ≻ c , yet this is not the output.

Let OPT ( ε ) be the set of vectors optimizing L ε . The rest of the proof will follow from the following two lemmas, whose proofs are in Appendix A.

Lemma 3.4. OPT (0) ⊆ R c ′ ≻ c .

Lemma 3.5. Suppose OPT (0) ⊆ R c ′ ≻ c , then, for sufficiently small ε > 0 , inf θ : θ ∈ R c ′ ≻ c L ε ( θ ) > inf θ L ε ( θ ) .

Finally, we handle the case that exists x > 0 such that ℓ ( x ) < ℓ (0) . Note that by convexity, this implies that for all y < 0 , ℓ ( y ) > ℓ (0) > ℓ ( x ) , so inf y ≤ 0 ℓ ( y ) < inf y ℓ ( y ) . Now, consider an instance with two candidates { a, b } located at x a = (1 , 0) and x b = (0 , 1) , and a single voter ranking a ≻ b (feasible via the parameter vector (1 , 0) ). It is possible to achieve a loss of ℓ ( x ) , e.g., by outputting the parameter vector (0 , x ) . On the other hand, any θ inducing the ranking a ≻ b will be lower bounded by ℓ (0) > ℓ ( x ) from above. Hence, f must output b ≻ a , which is both a PO and PMC violation.

3.2 Majority-Based Loss Formulation

Despite the negative results for loss-function-based rules, we may hope for a remedy using slightly different information. Specifically, we consider a similar loss-based function that rather than getting penalized for disagreeing with each voter only gets penalized if it disagrees with a majority of voters. That is, we choose θ minimizing

̸

Defining a parameter aggregation function based on this loss suffers from the same caveat as before, that in some cases no optimal θ exists. Nevertheless, we can apply an analogous fix for a ranking variant. We say that a linear rank aggregation rule f minimizes ℓ in the majority formulation if for all σ = f ( π ) ,

We first show (in Appendix A.5) that this does indeed help achieve PMC with essentially all loss functions.

Theorem 3.6. Fix a nondecreasing loss function ℓ with ℓ (0) > inf x ℓ ( x ) . If a linear rank aggregation rule f minimizes ℓ in the majority formulation, then f satisfies PMC.

Note that if the ℓ (0) > inf x ℓ ( x ) condition is not satisfied, i.e., ℓ (0) = inf x ℓ ( x ) , then all linear rank aggregation rules f minimize ℓ in the majority formulation, so satisfying this is a vacuous condition. Indeed, the parameter vector 0 of all 0 s achieves optimal loss of ℓ (0) for each pair and is consistent with every ranking σ . Therefore, the condition ℓ (0) > inf x ℓ ( x ) is as innocuous as possible to rule out these edge cases.

However, despite this good news for PMC, we show that this does not help in achieving PO. In fact, our negative result extends to every C 1 linear rank aggregation rule. Note that if f minimizing ℓ in the majority formulation breaks ties consistently (i.e., if multiple feasible rankings are optimal, then it consistently chooses the same one), then it is C1. We then have the following result, whose proof is relegated to Appendix A.6.

Theorem 3.7. All C 1 linear rank aggregation rules fail PO.

This result is quite unfortunate, because if there were a rule that is both C 1 and PO, we would automatically achieve PMC: Whenever there is a feasible PMC ranking, a C 1 rule cannot distinguish between this profile and a profile where all voters submit this ranking, hence, under the PO criterion, it must output it. Furthermore, whenever there is a PMC ranking, outputting it is necessarily PO, as for every pair, a majority of voters agree with the PMC ranking. Interestingly, in the proof, we construct a profile which has a PMC ranking, yet it is not feasible, and no matter how a C 1 linear rank aggregation rule breaks ties, there is an underlying profile in which this output violates PO.

4 Social Choice-Based Rule

In light of the above negative results, in this section, we ask whether there are linear rank aggregation rules that concurrently satisfy our two core axioms, PO and PMC. We answer this question affirmatively by presenting a new method based on a prominent rule from voting theory.

The Copeland rule assigns a Copeland score to each alternative equal to the number of other alternatives it beats in a pairwise competition, i.e., the score for a is |{ b | w a ≻ b > 1 / 2 }| . It then ranks the candidates in descending order according to their Copeland scores (breaking ties arbitrarily). It

is known that Copeland satisfies PO, PMC, and additional axiomatic properties. However, in linear social choice, since not every ranking is feasible, we cannot always output the Copeland ranking.

We, therefore, define a new linear rank aggregation rule, which we call leximax Copeland . This rule chooses a feasible ranking as follows. It ranks first the candidate with the highest Copeland score that can be feasibly ranked first under some parameter vector θ . Subject to this first position, it ranks second the candidate with the highest Copeland score which can be feasibly ranked second, and continues this process for subsequent positions.

Copeland's rule is a C 1 rule because it only requires the majority relationships between the candidates. Analogously, leximax Copeland is also a C 1 linear rank aggregation rule. Therefore, by Theorem 3.7, it does not satisfy the PO criterion. To address this issue, we define a variant called leximax Copeland subject to PO (LCPO) , which incorporates the PO criterion. Under LCPO, for every pair of alternatives where one dominates the other, the rule restricts rankings to place the dominating alternative above the dominated one.

The rule remains well-defined since the set of feasible rankings when enforcing the PO criterion is non-empty, as whenever a dominates b , all the rankings in the input profile rank a above b . Note that if the Copeland ranking is feasible, then this rule outputs that ranking, since unrestricted Copeland satisfies PO.

In addition to PO and PMC, we wish to show that LCPO satisfies two additional properties, which we define presently.

Definition 4.1 (majority consistency) . A linear rank aggregation rule satisfies majority consistency if when a candidate a is ranked first by a majority of voters in the input profile, a is ranked first in the output ranking.

Majority consistency ensures that the collective decision reflects the preference of the majority when there is a clear favorite. This principle aligns with PMC, but specifically focuses on the majority's favorite alternative. However, as we discussed above, a PMC ranking does not necessarily exist, and even when it exists, it is not necessarily feasible. By contrast, when a majority winner exists, this candidate is necessarily ranked first by a majority of voters in the input profile, who themselves (by assumption) submit feasible rankings. Therefore, we need not handle the case where it is impossible to rank the majority winner first.

Definition 4.2 (winner monotonicity) . A linear rank aggregation rule satisfies winner monotonicity if, when a candidate a is ranked first in the output ranking, elevating a in any voter's preference does not cause a to lose their top position in the updated aggregate ranking.

Winner monotonicity ensures that improving a leading candidate's position among individual voters will not result in that candidate's demotion.

We now state and prove the main result of this section.

Theorem 4.3. LCPO satisfies PO, PMC, majority consistency and winner monotonicity.

Proof. LCPO trivially satisfies PO since it always outputs a ranking that respects the PO criterion. Moreover, since Copeland satisfies PMC, and whenever Copeland's ranking is in the domain, leximax Copeland subject to PO returns this ranking, it clearly satisfies PMC.

Note that if an alternative a is ranked first by at least half of the voters, then a has the highest Copeland score, meaning that leximax Copeland subject to PO will rank this candidate first if this is possible. We see that this is indeed possible, by noticing that there is at least one feasible ranking in the input profile where a is ranked first, and any such input ranking is feasible (by assumption) and satisfies the PO requirement. Therefore, majority consistency is satisfied.

It remains to show that LCPO satisfies winner monotonicity. Suppose that on input profile π , the rule outputs a ranking σ where candidate a is ranked first. Now, consider a profile π ′ which is similar to π with the only exception being a ranking in which a is placed in a higher position. Let S be the set of agents that ranked above a in Copeland's ranking under π and let S ′ be the set of agents that ranked above a in Copeland's ranking under π ′ . Note that S ′ ⊆ S , since when moving from π to π ′ , only the Copeland score of a can increase, and therefore it is not possible for a candidate b to beat a under π ′ but not under π .

Now, suppose that R and R ′ are the set of rankings that satisfy the PO criterion with respect to π and π ′ , respectively. We show that R ′ ⊆ R . First, note that since a is ranked first under π , no alternative dominates a in π , as otherwise the PO criterion would be violated. Therefore, we get that no other alternative dominates a in π ′ as well. Moreover, note that if b dominates c in π , then this remains true in π ′ as well. On the other hand, it is possible that a dominates an alternative b in π ′ but not in π . From all of the above, we conclude that R ′ ⊆ R .

Since a is ranked first in σ , we get that for every candidate b ∈ S , there is no ranking in R in which b is ranked first, since otherwise, LCPO would output such a ranking. This also means that for every candidate b in S ′ , there is no ranking in R ′ in which b is ranked first, since R ′ ⊆ R and S ′ ⊆ S . Moreover, note that every ranking in R in which a is ranked first is also in R ′ since it satisfies all the PO restrictions of π ′ . Therefore, under π ′ , LCPO outputs a ranking in which a is ranked first.

Leximax Copeland subject to PO can be implemented in polynomial time by solving O ( | C | 2 ) relatively small linear programs. Specifically, given an input profile, we sequentially choose the candidate that is ranked in position r +1 as follows. We denote by σ r the partial ranking, where the first r positions have been fixed. For each candidate c that has not been ranked yet, we want to check if there is a parameter vector that adheres to the partial ranking σ r , respects the Pareto optimality criterion and ranks c at position r + 1 . Since all these constraints can be expressed as pairwise comparisons, we can use a linear program such as the one described in Footnote 4 to check if such a feasible ranking exists. Among the candidates meeting this criterion, we select the one with the highest Copeland score for position r .

5 Discussion

We conclude with a discussion of several extensions and limitations of our approach and results.

First of all, we wish to emphasize that our results are theoretical. While they highlight some shortcomings of the current practice of RLHF, our goal was not to 'outperform' existing RLHF methods. Rather, we see our model as giving a framework for understanding and comparing rules and methods -it is a (useful, we believe) lens through which researchers and engineers can examine their AI alignment methods.

Second, as written, our model has voters give their complete rankings, while in practice, this would be infeasible. In the real world, we are likely to elicit only relatively few pairwise comparisons per person. For our negative results, this assumption only makes them stronger: the BTL model fails both PO and PMC even with access to complete voter rankings. By contrast, for the positive results, specifically implementing leximax Copeland subject to PO, this ostensibly seems like a serious limitation. However, the complete rankings are not necessary for computing this rule, rather, all we need to know are PO dominance relationships and majority directions. We can therefore apply the rule whenever we can approximate this information, for example, through sampling. An alternative approach is to infer a complete ranking of each voter by fitting a parameter vector based on their pairwise responses; this process of learning a complete ranking and then running voting rules has been used before in a variety of settings [20, 18].

Third, our work initiates the study of the axiomatic method in our linear social choice model. However, we leave open many questions about which axioms are compatible and finding rules that achieve them. It should be clear by now that the primary challenge in linear social choice is that not every ranking over the candidates can be output. This means that essentially all known aggregation rules cannot directly be used without at least some modification. A natural direction to tackle is to try to find methods of converting known voting rules into linear aggregation ones while maintaining some of their axiomatic properties. To this end, we conclude with some preliminary results, and somewhat surprising findings within this space.

Some rules which optimize over rankings can be naturally transformed. For example, consider the Kemeny rule, which returns the ranking with the smallest pairwise disagreement over all votes. This can easily be transformed to the linear setting by simply outputting the optimal feasible ranking. In fact, in Appendix C.2, we show that this rule carries over the property of separability , 9 a social choice axiom that is violated by Copeland (in the classical setting) and leximax Copeland subject to

PO (in our setting). We show this in Appendix C.1. However, quite strikingly, although separability remains, this transformation makes Kemeny no longer PO (Appendix C.2).

Finally, note that the 'leximax' portion of leximax Copeland can be seen as a general purpose tool for mapping traditional rules to linear aggregation rules. In Appendix C.3, we explore leximax plurality (run leximax on the ranking of candidates by plurality scores), and show that it satisfies majority consistency, winner monotonicity, and separability. Additionally, the 'subject to PO' can be seen as another 'tool' for enforcing the Pareto optimality criterion when a rule does not independently satisfy it. However, enforcing PO can again cause somewhat surprising results. For example, in Appendix C.2, we show that linear Kemeny subject to PO, while now trivially satisfying PO, again violates separability. These observations indicate the challenges inherent in linear social choice, and we hope these open questions inspire fruitful follow-up research.

Acknowledgments

This research was partially supported by the National Science Foundation under grants IIS-2147187, IIS-2229881, CCF-2007080, IIS-1905558, and IIS-2214141; and by the Office of Naval Research under grants N00014-20-1-2488 and N00014-24-1-2704.

References

Nir Ailon and Mehryar Mohri. Preference-based learning to rank. Machine Learning , 80: 189-211, 2010.

Kenneth Arrow. Social Choice and Individual Values . Wiley, 1951.

Claude Berge. Topological Spaces . Oliver and Boyd, 1963.

Erdem Bıyık, Nicolas Huynh, Mykel J Kochenderfer, and Dorsa Sadigh. Active preferencebased Gaussian process regression for reward learning. arXiv preprint arXiv:2005.02575 , 2020.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Felix Brandt, Vincent Conitzer, Ulle Endriss, Jerˆ ome Lang, and Ariel D. Procaccia, editors. Handbook of Computational Social Choice . Cambridge University Press, 2016.

Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. MaxMin-RLHF: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925 , 2024.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems , 30, 2017.

F.H. Clarke. Optimization and Nonsmooth Analysis . Wiley New York, 1983.

Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Moss´ e, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. Social choice for AI alignment: Dealing with diverse human feedback. arXiv preprint arXiv:2404.10271 , 2024.

Thomas M Cover. The number of linearly inducible orderings of points in d -space. SIAM Journal on Applied Mathematics , 15(2):434-439, 1967.

Jessica Dai and Eve Fleisig. Mapping social choice theory to RLHF. arXiv preprint arXiv:2404.13038 , 2024.

H. Edelsbrunner. Algorithms in Combinatorial Geometry , volume 10 of EATCS Monographs on Theoretical Computer Science . Springer, 1987.

Peter C Fishburn. Condorcet social choice functions. SIAM Journal on applied Mathematics , 33(3):469-489, 1977.

Luise Ge, Brendan Juba, and Yevgeniy Vorobeychik. Learning Linear Utility Functions From Pairwise Comparison Queries. arXiv preprint arXiv:2405.02612 , 2024.

Henry W Gould. A note on the number of linearly inducible orderings of points in d -space. SIAM Journal on Applied Mathematics , 26(3):528-530, 1974.

Andras Kupcsik, David Hsu, and Wee Sun Lee. Learning dynamic robot-to-human object handover from human feedback. Robotics Research: Volume 1 , pages 161-176, 2018.

Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, et al. Webuildai: Participatory framework for algorithmic governance. In Proceedings 22nd ACM Conference on ComputerSupported Cooperative Work and Social Computing, , pages 1-35, 2019.

Abhilash Mishra. AI alignment and social choice: Fundamental limitations and policy implications. arXiv preprint arXiv:2310.16048 , 2023.

Ritesh Noothigattu, Snehalkumar Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel Procaccia. A voting-based system for ethical decision making. In Proceedings of the 32th AAAI Conference on Artificial Intelligence , pages 1587-1594, 2018.

Ritesh Noothigattu, Dominik Peters, and Ariel D Procaccia. Axioms for learning from pairwise comparisons. Advances in Neural Information Processing Systems , 33:17745-17754, 2020.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Chanwoo Park, Mingyang Liu, Kaiqing Zhang, and Asuman Ozdaglar. Principled RLHF from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254 , 2024.

R Tyrrell Rockafellar. Convex Analysis . Princeton University Press, 1970.

Anand Siththaranjan, Cassidy Laidlaw, and Dylan Hadfield-Menell. Distributional preference learning: Understanding and accounting for hidden context in RLHF. arXiv preprint arXiv:2312.08358 , 2023.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056 , 2024.

Paolo Viappiani and Craig Boutilier. Optimal Bayesian recommendation sets and myopically optimal choice query sets. Advances in Neural Information Processing Systems , 23, 2010.

HPeyton Young. Condorcet's theory of voting. The American Political Science Review , 82(4): 1231-1244, 1988.

Huiying Zhong, Zhun Deng, Weijie J Su, Zhiwei Steven Wu, and Linjun Zhang. Provable multi-party reinforcement learning with diverse human feedback. arXiv preprint arXiv:2403.05006 , 2024.

Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k -wise comparisons. In Proceedings of the 40th International Conference on Machine Learning , pages 43037-43067, 2023.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential for positive societal impacts. Negative societal impacts are implausible.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Deferred Proofs

A.1 Proof of Lemma 3.2

Proof. Webegin with some observations on g . First, we have that since ℓ is nonnegative, g must also be nonnegative. This along with the fact that lim x →∞ ℓ ( x ) = ∞ , we have that both lim x →∞ g ( x ) = ∞ and lim x →-∞ g ( x ) = ∞ . Together, these imply that L unconstr attains a minimum. Indeed, L unconstr (0 , 0) = 3 g (0) , and there is some bound B such that for all x > B and x < -B , g ( x ) > 3 g (0) . We can therefore restrict the optimization problem to r a , r b ∈ [ -B,B ] without changing the solutions. Since L unconstr is continuous and [ -B,B ] 2 is compact, a minimum is attained.

Next, note that g is convex because compositions of convex functions with monotonic functions and convex combinations of convex functions are convex [24]. From this, we claim that if there is an optimal solution ( r a , r b ) , then ( r a , r a / 2) is also an optimal solution. Indeed, fix such an ( r a , r b ) ,

where the inequality comes from convexity. This implies that ( r a , r a / 2) is also optimal.

By above, we have that if ( r b , r a ) is optimal, it must be the case that r a minimizes

Observe that h is again convex by monotonic composition and convex combinations.

Next, we will make use of the following facts about convex functions. Although they need not be differentiable, right- and left-hand derivatives always exist. For a function k these are defined as

Further, if k is convex, we have that [24]:

(i) k ′ + and k ′ -are nondecreasing,

(ii) k ′ -( x ) ≤ k ′ + ( x ) for every x,

(iii) x minimizes k if and only if k ′ -( x ) ≤ 0 ≤ k ′ + ( x ) ,

(iv) k ′ + and k ′ -are right and left continuous, respectively .

They also follow standard linearity and chain rule properties, which allow for simpler computation. For example:

(all of these hold for left-hand derivatives by swapping the positions of + and -) [9].

Next, we claim that we can find a valid p (rational with 1 / 2 < p < 1 ) and w > 0 such that g ′ + ( w ) > 0 , while h ′ + ( w ) < 0 . To that end, expanding the first derivative, we have

As long as ℓ ′ -( -x ) + ℓ ′ + ( x ) > 0 , this is strictly more than 0 for p satisfying

For h ,

As long as ℓ ′ -( -x/ 2) + ℓ ′ -( -x ) + ℓ ′ + ( x/ 2) + ℓ ′ + ( x ) > 0 , then this is strictly less than 0 for

Now, choose w > 0 such that ℓ ′ -( -w/ 2) > ℓ ′ -( -w ) ≥ 0 . This is possible by using the following procedure. We know ℓ ′ -(0) > 0 (as otherwise ℓ (0) < ℓ ( x ) for all x < 0 , contradicting our assumption on ℓ ). We will split into cases depending on whether ℓ is nondecreasing or strictly-convex (at least one must be true by the theorem assumptions).

First, suppose ℓ is non-decreasing. This implies that ℓ ′ -( x ) ≥ 0 for all x . We know that there is a point x < 0 such that ℓ ′ -( x ) < ℓ ′ -(0) (as otherwise ℓ would eventually become negative). Let d = ℓ ′ -( x ) . Take w such that

Note that ℓ ′ -( -w ) = d because ℓ ′ -is left continuous (from (iv) above). Since -w < 0 , so -w 2 > -w . This implies that ℓ ′ -( -w 2 ) > d , as otherwise -w would not be the supremum of such points. In addition, we have that d ≥ 0 because ℓ is nondecreasing.

Next, suppose ℓ is strictly convex. then ℓ ′ -is strictly increasing. Further, since it is left continuous and ℓ ′ 0 (0) > 0 , there is a γ > 0 such that for all x ∈ [ -γ, 0] , ℓ ′ -( x ) ≥ 0 . Therefore, choosing w = γ will do: ℓ ′ -( -γ ) ≥ 0 by choice of γ , and -γ/ 2 > -γ , so ℓ ′ -( -γ/ 2) > ℓ ′ -( -γ ) since ℓ ′ -is strictly increasing.

For this choice of w , we first claim that the preconditions of denominators being positive hold for (2) and (3). Indeed, let us write z 1 , z 2 , z 3 , z 4 for ℓ ′ -( -2 w ) , ℓ ′ -( -w ) , ℓ ′ + ( w ) , ℓ ′ + (2 w ) . We know that

by properties of convexity, and since ℓ ′ -( -w/ 2) > ℓ ′ -( -w ) ≥ 0 , we have that 0 ≤ z 1 < z 2 . The denominators are of the form z 1 + z 4 and z 1 + z 2 + z 3 + z 4 , which are now both necessarily positive. In addition, we also claim that a rational p with 1 / 2 < p < 1 satisfying both inequalities (2) and (3) will exist. Note that inequality (2) can now be represented as z 4 z 1 + z 4 , and we have that

because 0 ≤ z 1 < z 4 . Additionally, inequality (3) can be represented as

which is at least 1 / 2 because z 3 + z 4 > z 1 + z 2 . Finally, we have

Hence, there exists some rational p in this interval, which is necessarily between 1 / 2 and 1 , as needed.

To summarize, we have found a valid p and value w > 0 such that h ′ + ( w ) < 0 while g ′ + ( w ) > 0 . Because h ′ + is right continuous (from (iv) above), there is some γ > 0 such that h ′ + ( w + γ ) < 0 as well. We will now show for all ( r a , r b ) ∈ OPT unconstr , r a > w + γ and r b ≤ w . Indeed, note that r a must minimize h , so h ′ + ( r a ) ≥ 0 (from (iii) above), which implies r a > w + c . For r b , suppose which implies that

for a contradiction r b > w as well. Note that g ′ + ( w ) > 0 means g is increasing to the right of w . Let d = min( r a , r b ) -w > 0 . Consider ( r ′ a , r ′ b ) = ( r a -d, r b -d ) . We then have

where the equality holds because r ′ a -r ′ b = r a -r b and the inequality because g is increasing to the right of w . Therefore, we reach a contradiction, because then ( r a , r b ) would not be optimal. Thus, we have found values satisfying the lemma statement with A 1 = w and A 2 = w + γ .

A.2 Proof of Lemma 3.3

Proof. Fix p inducing a nonempty OPT unconstr satisfying Lemma 3.2 with values A 1 and A 2 . We first claim that for any ( r a , r b ) (regardless of optimality), it is possible to find a θ such that r θ ( a ) = r a and r θ ( b ) = r b . Indeed, note that

Since M = ( 2 1 1 1 ) is invertible with inverse

Thus, OPT com is nonempty, and is simply the image of OPT unconstr under M -1 . Now, fix ( r a , r b ) ∈ OPT unconstr , and let θ = M -1 ( r a r b ) . By assumption, we have r a > A 2 and r b ≤ A 1 . This implies that θ 1 = r a -r b > A 2 -A 1 , while θ 2 = 2 r b -r a < 2 A 1 -A 2 . Thus, setting A 3 = A 2 -A 1 and A 4 = 2 A 1 -A 2 satisfy the desired properties.

A.3 Proof of Lemma 3.4

Proof. We first claim that OPT (0) = OPT core . This will follow from showing

Indeed, in L 0 , the copied candidates are in exactly the same location as their counterparts. Hence, each term in L core appears 4 times, one for each combination of original and copy. In addition to these, there are the 6 terms for each ordered pair of ( a, a ′ ) , ( b, b ′ ) , and ( c, c ′ ) . Note that, each r θ ( x ) = r θ ( x ′ ) for each x ∈ { a, b, c } regardless of θ since they are in the same location. Therefore, the ℓ portion is always ℓ (0) , and the corresponding w x ≻ x ′ and w x ′ ≻ x terms add up to one for each pair. Hence, the total sum of these terms is 3 · ℓ (0) . Since L 0 is equivalent to L core up to positive scaling and translation, they have the same optima.

Finally, fix a θ ∈ OPT (0) . Since θ ∈ OPT core , by Lemma 3.3, θ 1 > A 3 and θ 2 < A 4 . Thus,

by choice of δ . Hence, θ ∈ R c ′ ≻ c

A.4 Proof of Lemma 3.5

Proof. We first show that when optimizing each L ε , it is sufficient to consider only θ coming from a bounded region. Indeed, observe that L ε ( 0 ) = ( 6 2 ) ℓ (0) for all ε . Since lim x →∞ ℓ ( x ) = ∞ , we for any r a , r b , we can simply set

̸

can find some B > 0 such that for all x > B , ℓ ( x ) > ( 6 2 ) ℓ (0) 1 -p = L ε ( 0 ) 1 -p . For a pair of candidates x = y ∈ C com , in the two terms concerning these candidates, we have

Applying this to { a, b } and { b, c } ,

This implies that we may restrict our attention to θ in the region

Indeed, for θ / ∈ R bounded , either | θ 1 | > B or | θ 1 + θ 2 | > B . In either case, we have L ε ( θ ) ≥ (1 -p ) ℓ ( B ) > L ε ( 0 ) .

Note that L ε ( θ ) is continuous not only in θ , but also in ε . Additionally, R bounded is closed and bounded, and hence, compact. Therefore, by Berge's Maximum Theorem, OPT ( ε ) is nonempty and upper semi-continuous in ε [3]. As per the definition of upper semi-continuous, since OPT (0) ⊆ R c ′ ≻ c , an open set, for sufficiently small ε > 0 , OPT ( ε ) ⊆ R c ′ ≻ c . Finally, note that R c ′ ≻ c ∩ R bounded is compact, so a minimum is attained, and this minimum must therefore be strictly larger than the values attained by members of OPT ( ε ) .

A.5 Proof of Theorem 3.6

Proof. Without loss of generality, we may assume that inf x ℓ ( x ) = 0 , as otherwise we could translate ℓ without affecting the optimization problem. Fix a profile π with feasible PMC ranking σ , and let θ PMC be a non-degenerate parameter vector that induces σ .

First, we show that inf θ L maj ( θ ; π, ℓ ) = 0 . Indeed, note that c · θ PMC ∈ R π for all c > 0 . Further, note that for any a, b with w a ≻ b ( π ) > 1 / 2 , r θ PMC ( b ) -r θ PMC ( a ) < 0 . Therefore, by making c large, the nonzero terms in L maj will have an input to ℓ negative and becoming arbitrarily large in magnitude. Since ℓ is nondecreasing, these approach the infemum of 0 .

̸

Next, for any σ ′ = σ , inf θ L maj ( θ ; π, ℓ ) ≥ ℓ (0) . Indeed, there must be some pair of candidates a, b with a ≻ σ b and b ≻ σ ′ a . For any θ ∈ R σ ′ , r θ ( b ) ≥ r θ ( a ) , so ℓ ( r θ ( b ) -r θ ( a )) ≥ ℓ (0) , and this lower bounds the loss function.

A.6 Proof of Theorem 3.7

Proof. We construct an explicit instance and pairwise majority relationships such that no matter what feasible ranking a rule picks, there is an underlying profile where that output was a PO violation.

We will have 9 candidates; 8 will be labeled c + i and c -i for i = 1 , 2 , 3 , 4 , and one labeled c ∗ . They will have feature vectors in R 4 . Each c ± i will be located at x c ± i = ± e i where e i is the i 'th standard basis vector, i.e., c + 2 is at (0 , 1 , 0 , 0) and c -4 is at (0 , 0 , 0 , -1) . Finally, c ∗ will be located at (1 / 5 , 1 / 5 , 1 / 5 , 1 / 5) .

There will be 5 voters. Their pairwise majority graph will be as follows. Candidate c ∗ will pairwise beat all others. In addition, each c + i will pairwise beat each c -j . Among the c + i candidates, there will be a cycle c + 1 ≻ c + 2 ≻ c + 3 ≻ c + 4 ≻ c + 1 and between the remaining two pairs c + 1 ≻ c + 3 and c + 2 ≻ c + 4 . The c -i candidates will be the exact reverse of this, i.e., a cycle c -4 ≻ c -3 ≻ c -2 ≻ c -1 ≻ c -4 , along with c -3 ≻ c -1 and c -4 ≻ c -2 . A pictorial representation can be found in Figure 1.

A C1 rule must pick a θ solely based on the pairwise majority graph. We will show that regardless of what θ it outputs, this will lead to a PO violation.

c

∗

c

+

1

c

+

2

c

+

3

c

+

4

c

-

1

c

-

2

c

-

3

c

-

4

To that end, the first fact we will show is that no θ can rank c ∗ first. Indeed, for any θ , r θ ( c ∗ ) = 1 5 ∑ i θ i ≤ 1 5 ∑ i | θ i | < max i | θ i | . On the other hand, for i maximizing | θ i | , at least one of c ± i will achieve this reward, strictly larger than r θ ( c ∗ ) . Hence, regardless of the output θ , some candidate must be ranked above c ∗ . We will show that this leads to a PO violation.

To construct profiles consistent with the pairwise majority graph, voters will always have rankings of the following form:

for some { i, j, k, ℓ } = { 1 , 2 , 3 , 4 } . In other words, they will rank a single + candidate above c ∗ and the rest in some order, followed by all -candidates in the reverse order. This is always achievable with the voter vector that puts values 1 , 3 ε, 2 ε, 1 ε in entires i, j, k , and ℓ , respectively, for some small ε > 0 .

̸

Fix an output θ with induced ranking σ . There must be at least one candidate c = c ∗ ranked above c ∗ . We now split into cases depending on which candidate this is. For each choice, we will construct a profile consistent with the pairwise majority graph where the candidate above c ∗ is Pareto dominated by c ∗ . We will describe each voter's ranking only by an ordering over the + candidates, assuming they otherwise take the form described in (4). Note that c ∗ is always ranked second, so if a candidate is never ranked first, they are Pareto dominated by c ∗ . The profiles for each candidate c + i can be found in the following table. One can check that all pairwise relationships are satisfied, and the corresponding c + i is never ranked first.

Finally, if a -candidate is ranked above c ∗ , then any of the following profiles work, as all -candidates are Pareto dominated by c ∗ with rankings shown in (4).

B PMCInfeasibility Example

Consider the case with d = 3 and seven candidates: one special candidate a ∗ located at (1 / 4 , 1 / 4 , 1 / 4) , and six others c ± i located at standard basis vectors e ± i . We have three voters with parameter vectors (1 , 2 ε, ε ) , (2 ε, 1 , ε ) , and (2 ε, ε, 1) , where ε < 1 / 5 is a small positive number. These voters have the following induced rankings:

We argue the ranking a ∗ ≻ c + 1 ≻ c + 2 ≻ c + 3 ≻ c -3 ≻ c -2 ≻ c -1 is a PMC ranking but no linear reward function can position a ∗ at the top of this ranking.

For any reward vector θ = ( θ 1 , θ 2 , θ 3 ) ∈ R 3 , the reward for a ∗ is :

Given the placement of c ± i at the standard basis vectors, each c ± i achieves a reward equivalent to one of the absolute values of the components of θ , thus surpassing r θ a ∗ since

C Additional Axiomatic Properties of Social Choice-Based Rules

We begin by stating another prominent axiom from social choice theory.

Definition C.1 (Separability) . A ranking aggregation rule satisfies ranking separability (or separability for short) if, when two profiles yield identical output rankings, when combined into a single profile, this should also produce the same output ranking.

Ranking separability preserves consistency in aggregation outputs and ensures stable decisions across similar preference distributions.

C.1 Copeland Violates Separability

Theorem C.2. Both Copeland (in traditional social choice) and LCPO (in linear social choice) fail separability.

In the first profile, the Copeland scores are 5 , 4 , 3 , 3 , 3 , 2 , 1 for candidates a, b, c, d, e, f, g , respectively. Similarly, in the second profile, they are 6 , 5 , 3 , 3 , 3 , 1 , 0 . So under any consistent tie-breaking rule, both of these profiles would output the ranking a ≻ b ≻ c ≻ d ≻ e ≻ f ≻ g .

However, if we combine these two profiles, then the score of a is 5 while the score of b is 6 , and thus, b will be ranked above a , violating separability.

To see that this also holds for LCPO, note that when every ranking is feasible, LCPO coincides with Copeland. We can simply have 7 candidates in R 7 all located at unit vectors, and voters with inputs from the above profiles.

C.2 Linear Kemeny Rule

Next, we consider a different rule from social choice theory, the Kemeny rule, which can be transformed to the linear setting while maintaining the separability can be achieved along with PMC. Given an input profile π , the Kemeny rule returns a ranking σ ∗ that minimizes the total number of pairwise disagreements with voters rankings, i.e.

where S m contains all possible permutations of the m candidates. This expression can be equivalently written as

̸

Here, we define the linear Kemeny rule , which outputs a parameter vector θ ∗ that induces a ranking that minimizes the total number of pairwise disagreements with voters' rankings, i.e.,

̸

Note that this rule conforms to the standard loss formulation, where the loss function is binary: it is 0 if two rankings agree with respect to the relative ranking of a pair of candidates and 1 otherwise. Since binary loss is not convex, it does not fit in the impossibility result of Theorem 3.1.

Note that Kemeny is generally NP-hard to compute; however, even for linear Kemeny, there is at least an exponential time aglorithm by brute-force computing the score of every ranking, and determining whether or not it is feasible.

Theorem C.3. Linear Kemeny satisfies PMC and separability.

Proof. PMC holds because the linear Kemeny score minimizes disagreements even among nontransitive rankings, making the PMC ranking the optimal choice whenever it is feasible. Separability is evident as the Kemeny score of a ranking over two datasets is simply the sum of the scores in each dataset. If the same ranking minimizes the score in both datasets independently, it will also minimize the score in their combination.

Theorem C.4. Linear Kemeny does not satisfy PO or majority consistency.

Proof. Consider the scenario with 20 candidates whose feature vectors are represented in the table below:

Each vector is constructed such that candidates are prioritized based on the magnitude of their first non-zero entry, leading to a natural ordering within grouped subsets: { 1 , 2 } ≻ { 3 , 4 , 5 } ≻ { 6 , 7 , 8 } ≻ { 9 , 10 , 11 } ≻ { 12 , 13 , 14 } ≻ { 15 , 16 , 17 } ≻ { 18 , 19 , 20 } . We will have six voters, with rankings induced by the parameter vectors described below.

Each voter ranks candidates within the group in increasing order except for a single reversed group. For instance, v 1 ranks candidates as 1 ≻ 2 ≻ 5 ≻ 4 ≻ 3 and so forth. Under this setup, no parameter vector θ can create a ranking that satisfies all voters' preferences due to the cyclic nature and individual group preferences.

We can check that linear Kemeny rule outputs a ranking in which candidate 2 is ranked above 1. From this analysis, we also conclude that the rules does not satisfy majority consistency either.

A possibly easy fix to the problem that linear Kemeny does not satisfy PO would be to enforce the PO criterion, as we did for the LCPO, i.e., to restrict to parameter vectors that respect PO. However, we show that if we do that, then linear Kemeny subject to PO does not satisfy separability anymore.

Theorem C.5. Linear Kemeny subject to PO violates separability and majority consistency.

Proof. First, consider a set of candidates and a profile similar to the one that is given in the proof of Theorem C.4. When we restrict to reward functions that 1 above 2 , then we can check that Linear Kemeny subject to PO outputs one of the rankings that are in the input profile. Without loss of generality, assume that it outputs the ranking of v 1 .

Second, consider the same set of candidates and three voters, with rankings induced by the following parameter vectors:

In this case, Linear Kemeny subject to PO outputs the ranking of v ′ 1 which is the same with this of voter v 1 .

When the two profiles are combined, then we do not anymore restrict on rankings in which 1 is above 2 , since 1 does not Pareto dominates 2 anymore. Then, we can check that linear Kemeny subject to PO outputs a different ranking than before in which 2 is ranked above 1 . From this example, we see that linear Kemenery subject to PO still violates majority consistency.

C.3 Leximax Plurality

Plurality is probably the most ubiquitous voting rule in the world. Its ranking variant ranks the candidates in decreasing order with respect to their plurality scores . The plurality score of a candidate is equal to the number of her appearances in the first position. This rule is known to satisfy several axioms but in linear social choice cannot be directly applied, as not all rankings are feasibly.

Similarly to leximax Copeland, we define Leximax Plurality as follows. It ranks first the candidate with the highest plurality score that can be ranked first under some parameter vector. Subject to this first position, it ranks second the candidate with the highest plurality score that can feasibly be ranked second, and so on, until all the positions are filled.

Theorem C.6. Leximax Plurality satisfies majority consistency, winner monotonicity and separability.

Proof. Note that leximax Plurality always returns a ranking in which the candidate with the highest plurality score is ranked first, since there exists at least one feasible ranking in which this candidate is ranked first. From this observation, we immediately see that leximax Plurality satisfies majority consistency.

Now, suppose that on input profile π , the rule outputs a ranking such that candidate a is ranked first. This means that a has the highest plurality score. Now, consider a profile π ′ which is similar to π with the only exception being a ranking in which a is ranked in a higher position. It is clear that a continues to have the highest plurality score and therefore leximax Plurality will output a ranking in which a is ranked first. Therefore, winner monotonicity is satisfied. .

It remains to prove that the rule satisfies separability. Suppose that under two different profiles π 1 and π 2 , the rule outputs σ , and under the aggregated profile π 3 , it outputs σ ′ . We will show that σ = σ ′ . One main observation is that if a candidate a has a higher plurality score than a candidate b under both π 1 and π 2 , then a has a higher plurality score than b under π 3 as well. We will show the desired property by induction on the positions of the ranking σ . Start from the first position in which say candidate a is ranked first. From above, we know that a has the highest plurality score under both π 1 and π 2 , which remains true in π 3 , and therefore a is ranked first in σ ′ . Now, assume that up to position t -1 , σ and σ ′ are similar and denote with a ′ the candidate that is ranked at position t of σ . We denote by S 1 , S 2 and S 3 the set of candidates that are not ranked among the first t positions in σ and have higher plurality score than a ′ under π 1 , π 2 , and π 3 respectively. Since, a ′ is ranked at the t -th postion in σ , we get that, subject to the fixed first t -1 position, no candidate in S 1 ∪ S 2 can be ranked at the t -th position. The theorem follows by noticing that S 3 ⊆ S 1 ∪ S 2 , which follows from the main observation above.

Peer reviews:
comments: SUMMARY:
Authors argue that for RLHF, the preferences are pairwise and we need to train a model that respects the preferences in aggregate which is in scope of social choice theory. Then they evaluate different aggregation methods on if they respect well established axioms. They showed that the popular TB model does not respect some axioms and came up with novel rules for reward learning.

STRENGTHS:
- principled approach to model preferences from population, rooted in social choice theory
- relevant topic for this conference
- initated the filed to approach RLHF reward modeling to follow axiomatic guidance through social choice theory

WEAKNESSES:
- only theoretical contributions, we do not know how the theory translates to in practice, even in toy settings

QUESTIONS:
- could you cite the places where you obtained the axioms of social choice theory?
- since you did not include where these axioms are from, are you missing any investigations of other axioms? If so, why did you choose the ones you investigated and why leaved the others out?

LIMITATIONS:
written in the discussion section
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecent months have seen a flood of concurrent papers studying the relationship between RLHF, preference aggregation, and social choice theory. This paper joins these lines of work and studies how to aggregate diverse human preferences (in the context of RLHF) that are modeled as a random utility model (e..g., BTL). The authors adopt a social choice perspective and show that the BTL model (and similar ones) fail to satisfy basic axioms known from social choice theory. The paper then also proposes a leximax Copeland (subject to PO) rule that satisfies desirable properties such as pareto optimality and majority consistency.\n\nSTRENGTHS:\n- The paper is very well-written. \n- The studied problem is relevant and timely. \n- The authors do a good job explaining concepts from social choice theory so that the paper is also easy to read for researchers who don't have a background in computational social choice.   \n- While unsurprising, the main theoretical result that linear aggregation (using a linearly parameterized reward function) is insufficient to achieve basic axioms of \"fair\" and reasonable preference aggregation is useful and interesting.\n\nWEAKNESSES:\n- The assumption that for each voter the complete ranking is available is obviously unrealistic, which limits the applicability of the proposed leximax Copeland rule.  \n- I believe that this paper's practical relevance is quite limited. It is unclear how leximax Copeland could be implemented in practice. The sampling strategy over voters and candidates (prompt/responses) is far from obvious and not discussed in the paper. Any additional comments about how such a sampling strategy would look like would be appreciated. \n- In contrast to the various related works on RLHF + social choice, which provide algorithms/solutions to augment the traditional RLHF framework, this work does not provide a tractable approach to address the problem of aggregating preferences in a reasonable way.\n\nQUESTIONS:\n- What about non-linear reward function classes?\n\nLIMITATIONS:\nThe authors address the limitations of the work adequately in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a axiomatic social choice framework for the problem of doing RLHF on group preferences. It shows that classical RLHF (and indeed a wider class of similar methods) violates the Pareto Optimality and Pairwise Majority Consistency axioms, and shows (via explicit construction) that there are mechanisms that satisfy both axioms along with two other axioms. (To simplify the model, the space of preferences is assumed to be implementable via linear classification.)\n\nSTRENGTHS:\n**High importance and novelty**\n- LLMs, and also AI systems in general, are expected to have large societal impact now and in the future. In light of this expectation, how to make sure model policies fairly represent the welfare of all stakeholders is important. This question, to my knowledge, has not received sufficient analysis except in this work and some of its concurrent works.\n\n**Flawless construction of the theoretical framework**\n- The theoretical setup seems flawless, covering exactly all the fundamental components of the problem, and in a very elegant manner.\n\nWEAKNESSES:\nI have some suggestions for improvement/future work, but I don't feel like they count as \"weaknesses\" *per se*, since those are very high standards which I don't feel like a normal conference paper would be held up to. I am instead putting those suggestions in the limitations section.\n\nAlso, please note that I did not check the proofs.\n\nQUESTIONS:\n- Do you think the PO and PMC axioms (possibly along with majority consistency, winner monotonicity etc.) are in any sense the \"gold standards\"? Could there be other similarly reasonable axioms that are contradictory with PO/PMC? I'm asking this because I have the intuition (which could be wrong) that an Arrow-like impossibility result would also haunt the linear social choice setting; elaboration in the limitations section.\n\nLIMITATIONS:\nI highly appreciate the detailed discussion of limitations and future directions in the Discussion section of the paper, and I agree with most of the points there. I have the following two additional remarks, which are meant not as critiques but as suggestions for exploration.\n\n1. **(Non)existence of a gold standard in linear social choice**\n    - Let's look at Theorem 3.1 first. If the RLHF method decide to overturn a perfect consensus on one pair with a tiny margin, in order to much more significantly reduce loss on a divisive pair (which is possible due to strict convexity), this actually seems desirable, despite violating PO.\n        - One could counter that whether the margin is small or large does not matter when what we care about the outcome is only the ordering. However, the exact margins (<> the exact reward values) do matter a lot in practice, where they indirectly (in RLHF) or directly (in DPO) determines what probabilities to assign to each response (as opposed to merely a ranking relation between the responses).\n    - In general, I suspect there is some arrow-like impossibility result in this space, where all the desirable properties just cannot be met at the same time. It's unclear which among the conflicting properties (somewhere among them are PO and PMC, and somewhere else is \"prioritize preference violations with larger margins\", with many others) are the most important.\n    - This intuition seems to be confirmed by the LCPO construction, which introduces rather arbitrary requirements (namely hard-coding the PO rule into the LCPO algorithm) in order to satisfy PO. \n2. **Human evaluations and human subject experiments**\n    - I suspect that human evaluation (e.g. letting human subjects judge the fairness of preference aggregation outcomes) could be an equally, if not more, important criteria for preference aggregation mechanisms than formal axioms are, given that (1) it's unclear which formal principles are more important than others and (2) these principles could be in conflict with each other (as in Arrow's theorem).\n    - This could be used to evaluate both aggregation mechanisms (does the outcomes align with human judgment?) and axioms (is human judgment in line with this axiom in general? If not, why, and who is right?). \n    - Here, the line between computer science and cognitive science seems to be dissolving.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an axiomatic approach to study preference aggregation for AI alignment. Inspired by works in social choice theory, the authors investigate a paradigm that they call linear social choice where preferences are representable by a linear model. In this context, they notably prove that if the linear rank aggregation rule minimizes some natural loss function (like the one induced by the Bradley-Terry model), then it cannot satisfy Pareto optimality. To circumvent this issue, the authors propose a variation of Copeland rule that outputs a ranking representable by a linear model.\n\nSTRENGTHS:\nThe paper investigates an important research question with a theoretically-founded approach.\n\nThe obtained results seem to be novel as far as I know. The failure of PO by minimizing a loss points to a strong limitation of the current approaches using human feedback.\n\nThe paper is well and clearly written. I notably appreciate the footnotes, comments, and connections with other works that the authors make.\n\nWEAKNESSES:\nI feel that the linear social choice paradigm may be too restrictive. I believe that the authors focus on linear models, because this is indeed a natural machine learning model. However, I don't think it is necessary to assume that the rankings of voters (i.e., humans in AI alignment) have to be linearly representable. In my opinion, the paper would be stronger if the results were presented without this latter assumption.\n\nThere are some gaps between the setting in machine learning (e.g., RLHF) and social choice theory. As the authors mention, in the latter, one usually observes pairwise comparisons instead of rankings. In addition, I think in the latter, one may not be interested to recover the full ranking, but only part of it, e.g., in RLHF, as the agent is trained, the reward approximator only needs to be good in the region of state-action pairs visited by a good policy. Could the authors comment on how this would impact their results?\n\nThe proposed social choice-based rule seems to me to be a bit artificial and only created to enforce Pareto optimality. It is not clear if such rule could easily be implemented in RLHF for instance and obtained via learning. For instance, in RLHF, this rule would require to consider all trajectories, which is impractical for any complex problem.\n\nQUESTIONS:\n1) Could the authors comment on how only trying to recover part of the full ranking would impact their results?\n\n2) Could you the authors comment on their proposed rule could be implemented in practice?\n\nLIMITATIONS:\nThe current discussion is adequate.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}