Selective Preference Aggregation
Authors: Shreyas Kadekodi
Hayden McTavish
Berk Ustun
Venue: ICML 2025 poster (2025)
Keywords: Ranking
Disagreement
Preference Aggregation
Social Choice
Pluralistic Alignment
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We propose Selective Preference Aggregation (SPA), which abstains on disputed preference pairs to produce robust, supported rankings that better align with raters.

Abstract: Many applications in machine learning and decision-making rely on procedures to aggregate human preferences. In such tasks, individual express ordinal preferences over a set of items through votes, ratings, or pairwise comparisons. We then summarize their collective preferences as a ranking. Standard methods for preference aggregation are designed to return rankings that arbitrate individual disagreements in ways that are faithful and fair. In this work, we introduce a paradigm for *selective aggregation*, where we can avoid the need to arbitrate dissent by abstaining from comparison. We summarize collective preferences as a *selective ranking* -- i.e., a partial order where we can only compare items where at least $100\cdot(1 - \tau)\%$ of individuals agree. We develop algorithms to build selective rankings that achieve all possible trade-offs between comparability and disagreement, and derive formal guarantees on their safety and stability. We conduct an extensive set of experiments on real-world datasets to benchmark our approach and demonstrate its functionality. Our results show selective aggregation can promote transparency and robustness by revealing disagreement and abstaining from arbitration.

Selective Preference Aggregation
Shreyas Kadekodi Hayden McTavish 2* Berk Ustun
Abstract
Standard methods for preference aggregation represent col- lective preferences as ranking - i.e., a total order over items where we can infer the collective preference between items by comparing their positions. Rankings reflect an approximate summary of collective preferences because it is impossible to define a coherent order when individuals disagree. This impossibility - which is enshrined in founda- tional results such as Condorcet's Paradox [21] and Arrow's Impossibility Theorem [8] has cast preference aggregation as an exercise in arbitration.
Many applications in machine learning and de- cision making rely on procedures to aggregate human preferences. In such tasks, individuals ex- press ordinal preferences over set of items by voting, ratings, or comparing them. We then ag- gregate these data into ranking that reveals their collective preferences. Standard methods for pref- erence aggregation are designed to return rankings that arbitrate conflicting preferences between in- dividuals. In this work, we introduce a paradigm for selective aggregation where we abstain from comparison rather than arbitrate dissent. We sum- marize collective preferences ing i.e. partial order that reflects all collective preferences where at least 100 (1 - 7)% of in- dividuals agree. We develop algorithms to build selective rankings that achieve all possible trade- off's between comparability and disagreement, and derive formal guarantees on their recovery and robustness. We conduct extensive set of ex- periments real-world datasets to benchmark our approach and demonstrate its functionality. Our results show how selective rankings can pro- mote transparency and robustness by revealing disagreement and abstaining from arbitration.
Over the past few decades, we have developed countless algorithms from this perspective [see 4, 73] to reap benefits
from collective intelligence in new use cases:
selective rank-
Support Group Decisions e.g., to fund grant proposals or hire employees [16, 74).
Qualitative Benchmarks e.g., to rank colleges [20], prod- uets [17], or language models [58].
Model Alignment - e.g., to fit or fine-tune models whose predictions align with the preferences of their users [61, 22].
In many of these cases, do not need total order. When we aggregate preferences to fund grant proposals, for example, rankings can lead to worse decisions as we arbitrarily select the top items. When we aggregate prefer- ences to rank colleges, a total order can strongly influence where students apply and how institutions invest [see e.g., 41, 27, 65, 66]. When we aggregate preferences to predict helpfulness [25], total order can lead us to build models that are aligned with the preferences of a slim majority [61],
Introduction
Many of our most important systems rely on procedures where we elicit and aggregate human preferences.In such systems, we ask group of individuals to express their preferences over a set of items through votes, ratings, or pairwise comparisons. We then use these data to order items in a way that represents their collective preferences as a group. Over the past century, we have applied this pattern to reap transformative benefits from collective intelligence in elections [21], online search [26], and model alignment [22].
In this work, we propose to address these challenges through selective aggregation. In this paradigm, we express collec- tive preferences as a tiered ranking - i.c., a partial order where we are only allowed to compare items in different tiers. We view tiers as simple solution to avoid the impossi- bility of arbitration: given pair of items where individuals express conflicting preferences, we can place them in the same tier to abstain from comparison. We capitalize on this structure to develop a new representation for collective pref- erences that can reveal disagreement, and new algorithms that can allow us control it.
"qqual contribution 'UCSD Duke University, Correspondence to: Berk Ustun <berk edu..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Selective Preference Aggregation
User Preferences
Aggregation
Collective Preferences
Standard
Comparisons 4 Max Dissent =
A B
C
D
(Majoritariann)
A B C>D
Comparisons Max Dissent =0
A>BxC=D
Selective (T 0)
A
B
C
D
B> D
Comparisons = Max Dissent =2/
Selective (T
A
B
C D
Fig. 1. Comparison of collective preferences for task where users express their ordinal preferences over a set of n. = 4 items. Standard methods represent the collective preferences of all users ranking. The resulting structure arbitrates conflicting preferences without revealing the existence or severity of disagrecment. In comparison, selective aggregation represents the collective preferences as a seleetive ranking- i.c., partial order with < tiers where can only compare items in different tiers, and guarantees that any such comparison overrules the preferences of at most 100 7% of users. The resulting structure reveals disagreement through its tiers and reveals that 100% users unanimously prefer {-A, B) (C, D).
dissent parameter E [o, 0.5). Here, the selective ranking The selective ranking for shows that we can recover a total order when we are willing to overrule up 40% of users.
Our main contributions include:
ambiguity, subjectivity, or lack expertise [54, 59, 75J. In medicine, for example, conflicting annotations reflect un- certainty regarding ground truth [see e.g., 69, 62, 52, 48]. In content moderation, conflicting annotations reflect differ- ences in opinion [37,32] 32).
1. We introduce paradigm for preference aggregation where we summarize collective preferences as selec- tive ranking - i... a partial order where each comparison aligns with the preferences of at least 100(1 r) % of users.
Our work is related to an extensive stream of research in social choice [43]. This body of work establishes the math- ematical foundations for preference aggregation by defining salient voting rules and characterizing their properties [see 46, 14, for list]. Although much of this effort is driven by the impossibility of reconciling individual preferences [see e.g., 8, 55), few works mention that we could abstain from arbitration by representing collective preferences as a partial order. Abstention is not a viable option in many of the appli- cations that have motivated work in this field. In elections, for example, we cannot aggregate ballots into partial order because we must identify a single winner [47].
2. We develop algorithms to construct all possible selective rankings for a preference aggregation task. Our algo- rithms are fast, easy to implement, and behave in ways that are safe and predictable.
3. We conduct a comprehensive empirical study of prefer- ence aggregation in modern use cases with diverse pref- erence data. Our results show how selective rankings can promote transparency and robustness compared to existing approaches.
4. We demonstrate how selective aggregation can be used to learn from subjective annotations in case study in toxic- ity detection. Our results show how selective aggregation can improve model performance and align predictions with a plurality of users.
On technical front, our work complements stream of research on rank aggregation [13, 26, 34, 3]. Although most work focuses on representing collective preferences as rankings, some focus on coarser representations such as bucket orderings [see e.g. 2, 5, 31, and references therein]. For example, Achab et al. [2] view bucket orderings as a "low-dimenssonal" total order and characterize their poten- tial for recovery. Andrieu et al. [5] use them vehicle to efficiently combine multiple rankings. In general, these differences in motivation lead to differences in algorithm design and interpretation. For example, items that we would consider "equivalent" in a bucket ordering would be "in- comparable" in a tiered ranking.
5. We provide an open-source Python library for selective preference aggregation, available on GitHub and instal- lable via pip install sclectiverank.
Related Work
Our work is motivated by growing set of applications where aggregate conflicting preferences. In machine learning, such issues arise in tasks such data annota- tion [6, 50, 28] and alignment [59, 22, 24] as a result of
2

---

Selective Preference Aggregation
2 Framework
problem over the space of all tiered rankings T::
We consider a standard preference aggregation task where we wish to order n items in way that reflects the collective preferences of p users. We start with a dataset where each instance , represents the pairwise preference of a user k E [p] = (l,....P) .p) between pair of items j, E [m]:
max TET
Comparisons(()
(SPA,)
s.t.
Disagreements(()) S TP
Here, the objective maximizes the number of valid compar- isons in tiered ranking T:
1 if user l strictly prefers to + $
0 if user k is indifferent
+ i _ j
Comparisons
II (T) * 1] (5E(M)
Σ
if user k strictly prefers j to i e j
The constraints restrict the fraction of individual preferences
Pairwise preferences can represent a wide range of ordi- nal preferences, including labels, ratings, and rankings. In practice, we can convert all of these formats to pairwise pref- erences described in Appendix A.2. In doing so, we can avoid restrictive assumptions elicitation. For example, users can state that items are equivalent by setting n = O, or express preferences that are intransitive. In what follows, we assume that datasets contain all pairwise preferences from all users for the sake of clarity. We describe how to relax this assumption in Section 4, and work with datasets with missing preferences in Section 5.
that can be contradicted by any valid comparison in T
Disagreements(T) = max
Σ T (m) = =1 m. +
A.j(( kE(P)
The dissent parameter T limits the fraction of individual preferences that can be violated by any selective comparison. Given a selective ranking S, that places item i in a tier above item j. at most 100 7% of users may have stated i y j.
We restrict E 0.5) guarantee that the selective rank- ing S, aligns with majority of users, and is unique (see Appendix B for proof). In this regime, we can set to trade off coverage for alignment as shown in Fig. 2. Setting r = O returns a selective ranking that reflects unanimity by show- ing all comparisons on which all users agree. Setting 7 just shy of 0.5 reflects a selective ranking that maximizes tiers without overruling a majority of users. The trade-off is anal- ogous the trade-off in selective classification [30, 29, 38): we output a partial order (selective classifier) that sacrifices "comparisons" (coverage) to reduce "disagreement" (error).
Collective Preferences as Partial Orders Standard ap- proaches express collective preferences as a ranking i.e., total order over items where we can compare any pair of items. We consider an alternative approach in which we express collective preferences as a tiered ranking:
Definition 2.1. A tiered ranking T' is a partial ordering of items into m disjoint tiers T := (Ti,..., Tm). Given tiered ranking, we denote the collective preferences as:
if iE TeTijE? Tp forl < I', (T)(T)= = -1 if
iE Túi Tp for l > V,
100%
A
A
L if e for any I
Comparabilit
80%
A
B
60%
B
B
Tiers provide a way to abstain from arbitration. Given a pair of items where users disagree, we can place them in the same tier and "agree to disagree." Given a tiered ranking T, we can only make claims about collective preferences by comparing items in different tiers. In what follows, we say that pairwise comparison between items i, is valid if (T' * L. We refer to valid pairwise comparison selective comparison.
40%
C
C
C
20%
D
0%
0.0 0.1 0.2
0.3 0.4 05
..2)).2 02.0 [0.4.0.5) Disagreement
Disagreement
Fig. 2. All possible selective rankings for the task in Fig. 1 where we aggregate the preferences of = users over n = items {A, B, C, D}. We show the comparability and disagrcement of each solution to SPA, on the left, and their selective rankings the right. Here, the solution for E [0, ! reveals that all users unanimously prefer (A, B} to {C, D}- The solution for E (t. Z. reveals that we can recover single winner if we are willing to make claims that overrule at most I user, while the solution for E (g. reveals we can only recover a total order if we are willing to overrule at most 2 users.
Selective Aggregation Selective ranking S, is a partial order that maximizes the number of comparisons that align with the preferences of at least 100- (1 - 7) of users. Given a dataset of pairwise preferences over n items from users, we can express S, as the optimal solution an optimization
3

---

Selective Preference Aggregation
User Preferences
Weight Matrix
Preference Graphs
A
C
D
> A>B>C>D
A
0
5
5
T =
0
A
B
C
D
1
5
5
T2
A C=
C
0
0
0
5
D
0
0
2
0
T
555
A
B
C
D
B>A+C>D
T3
Fig. 3. Graphical representations used construct selective rankings for the preference aggregation task Fig. 1. Given a dataset with 5 users and 4 items, we compute set of weights Wa,j = k [ 2 0j in Line 1. Given a dissent parameter т, we first build a directed graph (Vr,Ar) over items by drawing ares between aggregate preferences with weight wx, 2 rp. We then condense (V,, Ar) into directed acyelic graph of supervertices (Vr, Ar) by identifying its strongly connected components (shown in yellow). Here, the selective rankings for r = 0 and have 2 and 4 tiers, respectively.
3 Algorithms
Correctness We show that Algorithm recovers the unique optimal solution SPA, in Theorem B.2. The result follows from the fact that the directed graph (Vr, defines a tiered ranking that is both feasible and optimal with respect to SPA,. Specifically, the tiered ranking must obey the disagreement constraint in SPA, because we only draw ares between items i and j that can violate the prefer- ences of rp users in Line The tiered ranking maximizes the objective of SPA, because the ConnectedCooponnets routine in Line 4 partitions vertices in way that maximizes the number of tiers, which subsequently maximizes the se- lective comparisons under the disagreement constraint.
We present an algorithm to construct selective rankings in Algorithm and depict its behavior in Fig. 3.
Algorithm Selective Preference Aggregation
Input: {xf, (r.jicjj(M)))EE
preference dataset
Input: E [0, 0.5)
dissent parameter
I: Wa,j -kejpp I I[ 2 for all i,j [m]
2: VI + [n]
3: A₁ + {(i j) {)] Wi,j > rp}
4: VT + ComnedeedComponetts V1, Ar)
5: Ar + {(T + T') E T' (i +) Ar) 4p 6: 4p. + TopologicalSon(( Ar)
Recovering All Selective Rankings Algorithm is meant to recover selective ranking in settings where we can set the value of r a priori (e.g.. r = 0% to enforce unanimity). In many applications, we may wish to set T after seeing the entire path of selective rankings. In a funding task where we only have the resources to fund 3 proposals, for example, we can choose the smallest value of from the solution path such that the top tier contains < 3 proposals. In cases where a top three does not exist, this can lead us to save resources or increase our budget. In prediction task where labels encode collective preferences, we could aggregate annotations with a selective ranking and treat r as a hyperparameter to control overfitting.
Output: S, + Ti.............. )
r-selective ranking
Algorithm 1 constructs a selective ranking from a dataset of pairwise preferences and a dissent parameter 7 E [0, 0.5). The procedure first builds directed graph over items (V,, Ar). Here, each vertex corresponds to an item, and each arc corresponds to collective preference that we must not contradict in a tiered ranking. Given (Vr, Ay), the pro- cedure then builds a directed graph over tiers (Vr, Ar). In Line 4, it calls the ConmectedCoopooeents routine to identify the strongly connected components of (V, Ar) which be- come the set of supervertices Vr {Ti...., [] where each supervertex contains items in the same tier. In Line 5, it defines ares between tiers drawing an are from T to T' whose respective elements are connected by an are in Ay. Given (Vr, Ar), the procedure determines an ordering among tiers by calling the TopologicalSort routine in Line 6. In this case, the graph will admit a topological sort as it is a directed acyclic graph.
In these situations, we can produce solution path of se- lective rankings- i.e., finite set of selective rankings that covers all possible solutions to SPA, for E [0, {) [c.f. 63]. We observe that a finite solution path must exist as each selective ranking is specified by the ares in Line 3. In practice, we can compute all selective rankings efficiently by: (1) identifying a smaller subset of dissent parameters to consider as per Proposition 3.1; and (2) re-using the graph of strongly connected components across iterations.

---

Selective Preference Aggregation
Proposition 3.1. Given a dataset of pairwise preferences DD let Sw denote a finite set of selective rankings for dissent parameters in the set:
60] when they exist. In practice, the result implies that we can identify such "top items" by constructing and inspecting solution path of selective rankings.
= = < 1
w
ET(Ti 20] j E [n]} U (0) KEIP
In tasks where majority of users prefers an item to all others, the solution path will contain a selective ranking whose top tier consists of a single item. In this case, we can recover the "single winner" and report the threshold value To as a measure of consensus.
Let Sr be a selective ranking for an arbitrary dissent value E [0. 1). Then, Sw contains a selective ranking Sp such that S = S, for some dissent value 7' S T.
In tasks where such majority does not exist, every selective ranking S for T E [0, 0.5) will include at least two items in the top tier. In settings where we aggregate preferences to identify a "single winner," we can point to the solution path as evidence that no such winner exists and use it as a signal that further deliberation is required [see e.g., 53].
We describe this procedure in Algorithm 2. Both Algo- rithms 1 and 2 run in time O(n²p) i.e., they are linear in the number of individual pairwise preferences elicited (see Appendix B.4). As we show in Fig. 4, the resulting approach can lead to an improvement in runtime in practice.
Stability with Respect to Missing Preferences Standard methods can output rankings that change dramatically once we elicit missing preferences [9, 33, 42]. In Proposition 4.2, show that we can build selective ranking that abstains from unstable comparisons by setting missing preferences to m = O.
50
50.25
Naive
40
P aath
Runtime
00.7s
30
20
15.0s
10
7.0s
Proposition 4.2. Given a preference dataset with missing preferences Dinit let:
0.9s 0.8s
3.75
1.85
100
250
500
1000
Dime 2 Dinit be a complete dataset where we elicit miss- ing preferences; and
# Items
Fig. 4. Runtimes to produce all selective rankings for synthetic task with p = 10 users and n items (see Appendix B for details). We show results for naïve approach where call Algorithm for all possible dissent values, and the solution path algorithm in Appendix B. All results reflect timings on a consumer-grade CPU with 2.3 GHz and 16 GB RAM.
D 2 Dinit be complete dataset where we set missing preferences to T... = O.
For any dissent value [0. 2), let Stafe and S denote selective rankings for Diaae and Dunee, respectively. Then for any selective comparison (S.i(S E (-1,1}, we have:
Tii (Spall) (S
4 Theoretical Guarantees
In this section, we present formal guarantees on the stability and recovery of selective rankings.
Proposition 4.2 provides a simple way to ensure stability when working with datasets where we are missing prefer- ences from certain users for certain items. In such cases, we can always build a S that is "robust to missingness" in the sense that it will abstain from comparisons that may be invalidated once we elicit missing preferences.
On the Recovery of Condorcet Winners We often ag- gregate preferences identify items that are collectively preferred to all others. Consider, for example, a task where aggregate votes select the most valuable player in sports league or ratings to fund the most promising grant proposal [10]. In Theorem 1. we show that we can identify these "top" items from a solution path of selective rankings.
Stability with Respect to New Items In Proposition 4.3, we characterize the stability of selective aggregation as we add a new item to our dataset.
Theorem 4.1. Consider a preference aggregation task where a majority of users prefer item in to all other items. There exists a threshold value To E [0, 0.5) such that, for every T > To, every selective ranking S, will place in as the sole item in its top tier.
Proposition 4.3. Consider a task where we start with a dataset of all pairwise preferences from p users over n items, which we then update to include all pairwise preferences for a new n + 1 item. For any T E [0, 1), let S and S1+11 denote selective rankings over n items and n + items, respectively. Then for any two items i, E [m]. we have:
Theorem 4.1 provides a formal recovery guarantee that en- sures we recover a Condorctt winner or Smith set [see e.g.,
Tiej Ti. (Sa+1) E (11,,)), m1, (S, + =
5

---

Selective Preference Aggregation
The result shows that adding new item to selective rank- ing will either maintain each comparison or abstain. That is, adding a new item can only collapse items that were in dif- ferent tiers into a single tier. However, it cannot lead items in the same tier to split. Nor can it lead items in different tiers to invert their ordering.
Sampling: We use MC4 [26], which returns a ranking that orders items in terms of the stationary probabilities of Markov chain where transitions are defined by random walks over user preferences.
Median Rankings: We use Kemeny[40J, which returns a ranking that minimizes collective disagreement. We report results for an exact approach that handles ties and returns a certifiably optimal ranking by solving an integer program using CPLEX v22 [35]. We report results using the BioConsert heuristic [18], which returns ranking that minimizes collective disagreement through local search heuristic.
On Setting the Dissent Parameter We can draw on the re- sult in Proposition 4.2 to set the dissent parameter to ensure that selective rankings admit comparisons that are robust to missing or noisy preferences. By treating missing prefer- ences as abstentions, we can build selective rankings that only support comparisons that will hold once we elicit miss- ing preferences or correct noisy preferences. In a preference aggregation task where we are missing 5% of preferences, setting T 2 0.05 ensures that a selective ranking will only admit comparisons that will hold even if we were to elicit missing preferences. In a task where we elicit noisy prefer- ences, setting 2 0.05 ensures that selective ranking will only admit comparisons that will hold even after correction.
5.2 Results
We summarize the specificity, disagreement, and robustness of rankings from all methods and all datasets in Table 1. In what follows, we discuss these results.
On Transparency Some of the key issues with standard approaches stem from transparency. When we express col- lective preferences, we are forced to arbitrate disagreement yet unable to reveal information about arbitration. Given ranking, we cannot tell how many users we had to over- rule, which items were subject conflicting preferences, or whether the collective preferences reflect genuine agreement or an artifact of forced ranking structure.
5 Experiments
In this section, we present an empirical study of selective aggregation on real-world datasets. Our goal is bench- mark the properties and behavior of selective rankings with respect to existing approaches in terms of transparency, ro- bustness, and versatility. We include additional results in Appendix D, and code to reproduce our results on GitHub.
Our results highlight how selective rankings can address these issues on multiple fronts. As shown in Fig. 5, a selec- tive ranking can reveal the degree of disagreement through its dissent parameter, and identify items where users dis- agree through its structure. Given a selective ranking, we are only allowed to compare items across tiers and are guar- anteed that any comparison will overrule at most T fraction of users. We can immediately tell that at least 7 fraction of users express confficting preferences over items within the same tier (e.g., Duke and Columbia).
5.1 Setup
We work with 5 preference datasets from different domains listed in Table 1. Each dataset encodes user preferences over items votes, ratings, rankings. We convert preferences to pairwise comparisons with ties and build rankings using our approach and baselines. We construct solution paths using Algorithm 2 and report results for three dissent values:
SPAμ: the selective ranking for r == 0. This solution reflects unanimous collective preferences.
In contrast to existing methods, selective aggregation only reveals single winner or total order when collective prefer- ences align with a majority of users. In Table I, we see that preference aggregation tasks may not admit a single winner or a total order. In effect, we recover a selective ranking that identifies a single winner on 4 out of 5 datasets, and a total order on only out of 5. In many cases, the inability to identify a winner or total order is meaningful. On the Law dataset, for example, the most granular selective ranking SPAmij identifies two "top" schools (Stanford and Yale). On the sushi dataset, we recover selective rankings that iden- tify both a single winner and a total order for T = 0.48. In practice, this means any ranking of population preferences over sushi is highly contentious.
SPAmin: the selective ranking for the smallest positive dissent value 7 > 0 with 2+ tiers. This solution reflects the minimum disagreement we must incur to make any claim about collective preferences.
SPAmaj: the selective ranking for the largest T < 0.5. This solution reflects the maximum number of claims we can make about collective preferences without overruling majority of users.
We construct rankings using the following baseline methods:
Voting Rules: We consider Borda [12] and Copeland [23], which are voting rules from social choice that rank items based on position or pairwise wins.
6

---

Selective Preference Aggregation
Selective
Sundard
Datiset
Metrics
SPAs
BP
SR
Borda
Cooeland
MC4
Kameny
Disageement Rate
0.0%
2.0%
6.4%
6.3%
8.3%
7.9%
8.19
Absiention Rute
1000%
42.9%
25.656
p- 100 usets
NBA (49)
or mixiin
d-Advemaral
Top
Sampliag
0.0%
0.0%
0.15
0.0%
0.09
0.0%
19.09
4.8%
19.0%
4,8%
19.06
0.00%
14.3%
4,5%
Disagreemen Rate
0.0%
0.21
0.216
6.8%
6.6%
6.4%
6.7%
euevtvon heme
Abeiention Rato
94 94.9%
42.9%
42.5%
Tiex
Top
0.0% missing
Perple Rock (S1)
Sampling
0.0%
0.0%
0.0%
0.8%
0.8
0.9%
A=Adverariil
0.0%
0.0%
0.09:
268
1.8%
3.1%
1.6%
lavechool
20 Hemm
Disegremmet Rote
Absicntion Rete
Tiers
40.5%
0.05
36.8%
0.99
4.2%
8.1%
4.7%
20
4.2%
20
4.2%
4.1%
20
y Gusers
12
S.Saa (44)
missing
& Adversaril
Sumpling
0.0%
0.0%
0.0%
0.0%
0.0%6
3.7%6
2.0%
1.11
0.5%
2.8%
45 85.
29.5%
Disagreemen: Rate
0.0%
0.0%
0.1%
12.3%
12.2%
12.2%
13.7%/
derankinng ings " - 175 itemss
Abstention Kote
100.0%
9% 9%
95.8%
Tirr
135
168
170
175'
0.0% misiing
Berger [II]
4. AdAverrsriill
Sampling
0.0%
175
0.11
3.1%
1.7%
0.1%
11.19
9.0%*
Disagreemen Rase
0.0%
13.6%
42.6%%
42.00
42.8%
42.8%
42.0%
Abstention Rute
100.0%
64.0%
0.0%S
9 5,000 users
0.0% missing
" 10 iems
Kamishima (39)
Sumpling à Adversarial
Top Items
* Tless
2
0.U%
10
0.0%
10
10
10
2.2%
10
2.2%
11.1%
11.15
Table 1. Comparability, disagreement, and robustness of rankings for all methods on all datasets. We report the following metrics for each ranking: Disagreement Rate, i.e., the fraction of collective preferences that conflict with user preferences; Abstention Rate, i.e., the fraction of collective preferences that abstain from comparison; # Tiers, the number of tiers or ranks, # Top Items, i.e., the number of items the top tier or rank. A A-Sampling, the average fraction of collective preferences that are inverted when we drop 10% of individual preferences; and A-Adversarial, the maximum fraction of collective preferences that are inverted when we flip 10% of individual preferences, respectively.
Selective Aggregation
Standard Aggregation
Robust to missing preferences.
Sensitive to changes in individual preferences.
SPAm.
SPA
Borda
Bordago
Copeland
Copeland
Yale
Yale
Yale
Stanford
Yale
Harvard
Stanford
Stanford
Stanford
Harvard
Stanford
Harvard
Harvard
Hanvard
Vale
Herverd
Stanford
UChicago
UChcago
UChicago
UChicago
UChicago
UChicago
UVA
UVA
Columbia
Columbia
Penn
Ponn
Penn
Penn
Penn
UVA
Duke
Duke
Duke
Duke
Penn
Columbia
Columbi
Columbia
Columbia
UVA
Duke
UVA
NYU
NYU
NYU
NYU
NYU
NYU
Michigan
Michigan
Michigan
Northwestem
Michigan
Michigan
Northwestern
Northwestem
Berkeley
Michigan
Northwastern
Northwestem
Berkeley
Berkeley
Northwestern
Berkeley
Bankeley
Barkeley
UCLA
UCLA
UCLA
UCLA
UCLA
UCLA
Georgstown
Georgelown
Comell
Comal
Comell
Gomell
Comell
Georgetow
Geornetown
Georgetown
Georgetawn
UT
Vanderbill
UT
Vandertill
Venderoit
UT
Venderbitt
Vanderbit
Vanderbitt
USC
USC
USC
USG
USC
USC
BU
SU
BU
GWU
GWU
GWU
GWU
GWU
GWU
Fig. 5. Consensus rankings of U.S. law schools from seleetive preference aggregation and standard voting rules for the lawschool dataset. On the left, we show selective rankings SPA and SPAmi for dissent values of Tiin 1/8 and Timx - As we discuss in Section 4, these values map to guarantees on both disagreement and robustness: given a selective ranking with a dissent value of 7, any comparison can overrule most 7% of individual preferences, and would hold if were drop or change 7% of preferences. On the right, we show rankings from traditional methods to highlight their sensitivity to missing preferences. Here, Borda and Copeland rankings produced using the original dataset which contains pairwise preferences of all users for all schools, and Bordago and Copelandgo are rankings produced using a dataset where we drop 10% of pairwise preferences.
7

---

Selective Preference Aggregation
On Robustness One of the main limitations of represent- ing collective preferences as a ranking is that it may change dramatically a result of changes to individual prefer- ences 19, 33, 42]. This sensitivity is structural: given a ranking over n items, changes to individual preferences can affect any of the (2) pairwise preferences. In contrast, selec- tive rankings limit sensitivity by grouping items into m S n tiers, which can induce robustness by restricting the number of comparisons that are subject change.
6 Learning by Agreeing to Disagree
One of the most common use cases for preference aggrega- tion in machine learning arises when we align models with the collective preferences of their users. -c.g., to predict the toxicity of an online comment or the helpfulness of a chat- bot response. In the simplest case, we would recruit users to annotate set of training examples. We would then aggre- gate their annotations to obtain aggregate training labels we could use to fit or fine-tune a model [45]. We often apply this pattern in tasks where we wish to predict outcomes where individuals express conflicting preferences due to ambiguity [62] or subjectivity [32, 28]. In such cases, stan- dard aggregation methods such as majority vote can lead to models whose predictions reflect the collective preferences of the majority [61, 22]. In what follows, we explore how selective aggregation can mitigate these effects by returning training labels that better account for all annotators' views.
In Table I, we highlight this behavior by reporting the ex- pected number of collective preferences that are inverted when we build a ranking from a dataset with a small number of missing or noisy preferences. Specifically, we report A- Sampling and A-Adversarial which measure the expected rate of inversions from sampling or gaming. Given each dataset and each method, construct these estimates by applying each method to build modified ranking from dataset where we drop or flip 10% of individual preferences. We repeat this process 100 times and measure the number of collective preferences that change between the original ranking and the ranking we obtain for the modified dataset.
Setup We consider a task to build a classifier to detect toxic conversations with a language model. We work with the DICES dataset [7], which contains individual toxicity labels for 350 chatbot conversations from p 123 users. Here, each label is defined as y E (1,-1,0) if user k labels conversation i as (toxi c, ben gn, unsure] respectively. We randomly split users into two groups: a group of y = 5 users whose labels we use to train our model; and a group of plas = 118 users whose labels we use to evaluate the predictions of the model at an individual level once it is deployed. We set the relative size of each group reflect a practical setting where company would collect labels from small subset of users to train model that assigns predictions to a large population.
On the Arbitrariness of Arbitration Our results high- light how standard algorithms for preference aggregation can output conflicting accounts of collective preferences. When users express conflicting preferences, there are often many to arbitrate these differences. As shown in Fig. 5, voting rules such as Borda and Copeland can lead to dif- ferent rankings. In practice, these effects can stem from differences in individual preferences or algorithm design. Borda rewards high rank placement, while Copeland fo- cuses on pairwise wins. The sensitivity of these methods to missing data leads to differences at less salient positions (see e.g., differences in Berkeley, Michigan, and Northwestern).
We aggregate the toxicity labels from each user in the train- ing set to create three sets of aggregate training labels to train our model. In this case, we drop all annotations where user rates conversation as "unsure" iie., where yi,A = O and only aggregate annotations for conversations that are labeled as toxic or non-toxic - i.e., Vi.6 E 11,
Although methods for preference aggregation must arbitrate conflicting preferences, many are not built to arbitrate these differences with explicit guarantees. In Table I, only SPA and Kemeny can pair rankings with formal guarantees on the arbitration process. Kemeny can return a ranking that is guaranteed minimize collective disagreement by solving a combinatorial optimization problem. In our experiments, we are able to recover a certifiably optimal ranking quickly for 4/5 datasets using a commercial solver on a single-core CPU with 128GB RAM. On the csranking dataset, how-
Maj == I II --==11
ExEip) 1I [v/ = -1]
Vi
which denote aggregate labels from majority vote [57].
Borda E [280], which denote aggregate labels from a pair- wise variant of Borda [15]. As rankings are not provided, an item's score is calculated as its total number of pairwise
y
ever, we are forced to use a heuristic approach because the optimization problem contains a prohibitively large number of constraints. In this case, our results for a heuristic [18] highlight computational trade-offs surrounding arbitration: we find that the approach arbitrates differences in a way that is suboptimal (c.f., disagreement rates of 13.7% for Kemeny vs. 12.2% for Copeland) and returns a ranking that is more sensitive to input perturbations (see Appendix D.4 for results on other datasets).
wins, summed across all users.
yA E [15], which denotes aggregate labels from SPA for the large dissent parameter < 0.5.
y, Exp E [4], which denote ratings elicited from an in-house expert. This reflects a baseline where we train a model using annotations from a single human expert.
We process the training labels from each method to ensure that we can use a standard training procedure across similar
8

---

Selective Preference Aggregation
100%
LabelE PredictE mor
eelective
72.9%
39. 2% 39.9%
Proportion of Users
80%
Test Train
Borda
228.5%
4%
60%
Majority
Borda
44.5%
37 8%
SPA
Expert
40%
Majority
43 .%
43.2%
449 5%
20%
Expert
50.5
49.5 5%
49.3%
0%6
0%
20%
40%
60%
20%
40%
60%
80%
Threshold Value
Collective Error Rate
Fig. 7. Cumulative distributions of individual error rates for toxic- ity detection models using different methods for label aggrega- tion, For each model f. we plot the fraction of pm 118 users in the test set where BER()) S E [o, 1].
Fig. 6. Collective error rates for each method on the DICES dataset. LabelError measures the average disagreement between aggregated labels and individual preferences, and PredictError measures the average disagreement between model predictions and individual preferences. We report values for the train sample of J annotators, and a test sample of yd" = 118 held-out users. SPA achieves the lowest collective error the training and test sample.
40% or less under y SPA compared to roughly 20% of users for y Borda and Maj.
Our results highlight benefit from building models using labels that encode collective preferences. In this case, the large values of label error for y imply that many users disagree with the expert. These findings capture the perfor- mance of each approach in a task where we threshold the predictions of each method to optimize the BER. In practice, we observe similar findings at other salient operating points e. requiring collective TPR of 2 90%. In such cases, baselines such majority vote may underperform their labels can only capture binary information.
methods. We use the training labels from each method to fine-tune a BERT-Mini model [68] and denote these models as fSPAA MMai, fBonda, fExpper, We evaluate how each method performs with respect to individuals and users in a specific group in terms of the following measures:
BER (f" = 1/2 FPR +1 (F")
LabelError = (v
k=1
1 PredictError(P = P k=1
Σ BER
7 Concluding Remarks
In many applications where we aggregate human prefer- ences, disagreement "signal, not noise" [6]. In this work, developed foundations aggregate preferences in way that can reveal disagreement and allow to control it. The main limitation of our work stems from algorithm design: the algorithms we have developed in this work are designed to be simple, versatile, and safe. To this end, they behave conservatively tasks where datasets contain large num- ber of missing preferences. Such datasets are common in tasks where costly to elicit preferences or where we must elicit preferences over large collection of items. In these cases, we can still represent collective preferences as selective ranking, but the output may collapse into a single tier. This behavior is intentional: it signals that any claim about the collective preferences could be invalidated once the missing preferences are elicited. At the same time, impractical in large-scale applications that rely on sparse data and elicit only few preferences from each user. Look- ing forward, we can extend our paradigm to such settings by adopting probabilistic assumptions [see e.g., 2] and by developing procedures to streamline preference elicitation.
We evaluate the performance of each in terms of the bal- anced error rate for clarity as the data for each user exhibits class imbalance that changes across users. We include addi- tional details on our setup in Appendix D.5.
Results We summarize our results at a group level in Fig. 6 and an individual level in Fig. 7.
Our results in Fig. 6 highlight how SPA aggregates labels in way that minimizes collective disagreement - -cchheen a label error of 28.2% (c.f. 37.8% with Mai). Moreover, the improved alignment in training labels can propagate into an improved alignment in the predictions of the model. In this case, SSPA has a prediction error of 29.9% on training users and 39.9% on test users (c.f. 38.4 % and 44.5 for fBodd)).
Our results in Fig. 7 show how the prediction error is dis- tributed across the trann 5 annotators in the train set i.c., users whose preferences we would typically observe, as well as the ue = 118 held-out annotators, whose pref- erences we would not typically be able to observe. In this case, roughly 60% of users achieve an individual BER of

---

Selective Preference Aggregation
Acknowledgements
[14] Brandt, F., Conitzer, V., Endriss, U., Lang, J., and Procaccia, A. D. Handbook of computational social choice. Cambridge University Press, 2016.
We thank the following individuals for comments that im- proved this work: Margaret Haffey, Yang Liu, David Parkes, Cynthia Rudin, and Devavrat Shah.
[15] Burka, D., Puppe, C., Szepesváry, I.., and Tasnádi, A. Vot- ing: A machine learning approach. Technical Report 145, Karlsruhe Institute of Technology. Institute of Economics, November 2020. URL https://hdll hwwdah..en/ennini...eeeee net/104119/227115.
Impact Statement
[16] Cachel, K. Fair Consensus Decision Making: Preference Aggregation Techniques for Candidate Group Fairness. PhD thesis, Woreester Polytechnic Institute, 2025.
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal con-
sequences to this work, many of which we have discussed in the manuscript.
[17] Chang, A., Rudin, C., Cavaretta, M.. Thomas, R., and Chou, G. How reverse-engineer quality rankings. Machine learning, 88:369-3009 2012.
References
[18] Johen-Boulakia, S.. Denise, A., and Hamel, S. Using medi- ans to generate consensus rankings for biological data, In In- ternational Conference on Scientific and Statistical Database Management, pp. 73-90, Springer, 2011.
[1] Above the Law. Top law schools 2023, https//abovethelav. com/top-law con/top-law-schooke schoolsh 20230, 2023.
[2] Achab, M., Korba, A., and Clémençon, S. Dimensionality reduction and (bucket) ranking: a mass transportation ap- proach. In Algorithmic Learning Theory, pp. 64-99 93. PMLR, 2019.
[19] College Kickstart LLC.
U.S. News & College
World
Report Posts 2023 www.:/ollegeeiccsttt..oombbbbbbe
Rankings.
u-s-news-worlld-epoon repod posts-oossssssse cole-oleeg--rnnnings
[3] Ailon, N. Learning and optimizing with preferences. In International Conference on Algorithmic Learning Theory, pp. 13-21. Springer, 2013.
2022.
[20]
Collins, H. W., Jenkins. S. M., Strzelecka, N., Gasman, M.. Wang. N., and Nguyen, T. Ranking and rewarding access: An alternative college scorecard. Penn Center for Minority Serving Institutions, 2014.
[4] Alcaraz, J., Landete, M., and Monge, J. F. Rank Aggrega- tion: Models and Algorithms. In The Palgrave Handbook of Operations Research, pp. 153-178. Springer, 2022.
[21] Condorcet, M. d. Essay on the application of analysis to the probability of majority decisions. Paris: Imprimerie Royale, pp. 1785, 1785.
[5] Andrieu, P., Brancotte, B.,, Bulteau, I.., Cohen-Boulakia. S., Demise, A., Pierrot, A., and Vialette, S. Efficient, robust and effective rank aggregation for massive hiological datasets. Future Generation Computer Systems. 124:406-421, 2021.
[22] Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M.. Lambert, N., Mossé, M., Pacuit, E., Rus- sell, S., Schoelkopf, H., et al. Social Choice for Al Align- ment: Dealing with Diverse Human Feedback. arXiv preprint 2404.10271, 2024,
[6] Aroyo, I.. and Welty, C. Crowd truth: Harnessing disagree- ment in crowdsourcing a relation extraction gold standard. WebSci2013. ACM, 20 2013(2013), 2013.
[7] Aroyo, L.,, Taylor, A., Diaz, M., Homan, C., Parrish, A., Serapio-Garcia G., Prabhakaran, V.. and Wang. D. DICES Dataset: Diversity in Conversational AI Evaluation for Safety. Advances in Neural Information Processing Systems, 2024.
[23] Copeland, A. H. A reasonable social welfare function. semi- nar on applications of mathematics to social sciences, univer- sity of michigan. Ann Arbor, 1951.
[24] Dai, J. and Fleisig, E. Mapping social choice theory to rlhf. arXiv preprinu2404. 13038, 2024.
[8] Arrow, K. J. Social Choice and Individual Values. John Wiley & Sons, New York, 2nd edition, 1951. Revised edition published in 1963.
[25] Davani, A. M., Díaz, M., and Prabhakaran, V. Dealing with disagrcements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92-110, 01 2022. ISSN 2307- 387X.
[9] Asudeh, A., Jagadish, H., Miklau, G., and Stoyanovich, J. On obtaining stable rankings. Proceedings of the VLDB Endowment, 12(3), 2018
[10] Azoulay, P. and Li, D. Scientific Grant Funding. Techni- cal Report 26889, National Burcau of Economic Research, March 2020. Revised June 2021.
[26] Dwork, C., Kumarr R., Naor, M., and Sivakumar, D. Rank aggregation revisited, 2001.
[27] Espeland, Wendy Nelson and Sauder, Michael. Engines of anxiety: Academic rankings. reputation, and accountability. Russell Sage Foundation. 2016.
[11] Berger, F. D. Csrankings: Computer science rankings. https: //csrankings.org/, 2025. Accessed: 2025-08-13.
[12] Borda, d. Mémoire sur les élections au scrutin. Histoire de I' l'Académie Royale des Sciences, 1781.
[28) Fleisig, E., Abebe, R., and Klein, D. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6715-6726. 2023.
[13] Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)332--345, 1952. doi: 10.2307/2334022.
10

---

Selective Preference Aggregation
[29] Frane, V., Prusa, D., and Voracek, V. Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24(1 (4(11):1-45 1):1-49, 2023.
[45] Lukasik, M., Chen, L.., Narasimhan, H., Memon, A. K., Jitkrit- tum, W., Yu, F. X., Reddi, S. J., Fu, G., Bateni, M., and Kumar, S. Bipartite ranking from multiple labels: On loss versus label aggregation. arXiv preprint 2504.11288 April 2025.
[30]
Geifman, Y. and Yania R. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
[46] Moulin, H. Axioms of cooperative decision making. Num- ber 15. Cambridge university press. 1991.
[31] Gionis, A., Mannila, H., Puolamäki, K., and Ukkonen, Al- gorithms for discovering bucket orders from data. Proceed- ings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 561-566. Asso- ciation for Computing Machinery, 2006. ISBN 1595933395.
[47] Myerson, R. B. Nash equilibrium and the history of economic
theory. European Economic Review, (3(4-6))671-697 April 1999.
[48] Nagaraj, S., Liu, Y., Calmon, F., and Ustun, B. Regretful decisions under label noise. The Thirteenth International Conference on Learning Representations. 2025.
[32] Gordon, M. L.. Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., and Bernstein, M. Jury learning; Inte- grating dissenting voices into machine learning models, In Proceedings of the 2022 CHI Conference on Hunan Factors in Computing Systems, pp. 1-19, 2022.
[49] National Basketball Association. 2020-21 NBA
Coach of the Year Voter Selections. https:
[33] Halpem, D., Kehne, G., Procaccia, A. D., Tucker-Foltz, J., and Wüthrich, M. Representation with incomplete votes.
06/2020-22 21-00-11---1 21-NBA- Cooch-offtthe Year- 2021.
In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp- 5657-5664, 2023.
[50] Prahhakaran, V. Davani, A. M., and Diaz, M. On releasing annotator-level labels and information in datasets. arXiv preprint 211005569, 2021.
[34] Hochbaum, D. S. and Levin, A. Methodologies and algo- rithms for group-rankings decision. Management Science, 12(9))1394-1408, 2006.
[51] Purple Rock.
free summaries. Survivor season rankings spoiler- survivor-seasee season- season-rankings
www.purpeeeeocpoocaassoooo coms
coemssumariiss,, 2023.
[35] IBM ILOG. CPLEX User's Manual. International Business Machines Corporation, 2024. Version 22.1.
[52] Schackermann, M. Human-Al Interaction in the Presence of Ambiguity: From Deliberation-bar Labeling to Ambiguity- aware AI. PhD thesis, University of Waterloo, 2020.
[36] Jamieson, K. G. and Nowak, R. Active ranking using pair- wise comparisons. Advances in neural information process- ing systems, 24, 2011.
[53] Schackermann, M., Beaton, G., Habib, M., Lim, A., Larson, K., and Law, E. Capturing expert arguments from medi- cal adjudication discussions in a machine-readable format. In Companion Proceedings of The 2019 World Wide Web Conference, pp. 1131-1137, 2019.
[37] Jiang. N.-. and Marneffe, M.-. d. Investigating reasons for disagreement in natural language inference. Transactions of the Association for Computational Linguistics, 10:1357- 1374, 2022.
[54] Schoeffer, J.. De-Arteaga, M., and Elmer, J. Perils of label indeterminacy: A case study on prediction of neurological recovery after cardiac arrest. arXiv preprint 2504.04243 2025.
[38] Joren, H., Marx, C. T., and Ustun, B. Classification with con- ceptual safeguards. In The Twelfth International Conference on Learning Representations, 2023.
[39] Kamishima, T. Nantonac collaborative filtering: recommen- dation based on order responses. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge dis- covery and data mining, pp. 583-588, 2003.
[55] Sen, A. K. A possibility theorem on majority decisions. Econometrica: Journal of the Econometric Society, pp. 491 499, 1966.
[56] Shah, N. B., Balakrishnan, S., Bradley, J., Parckh, A., Ram- chandran, K., and Wainwright, M. When is it better to com- pare than to score? arXiv preprint 1406. 6618, 2014.
[40] Kemeny, J. G. Mathematics without numbers. Daedalus, 88 (4):577-59) 591, 1959.
[41] Larkin, M. How northeastera gamed the college rankings. Boston Magazine. Aug 2014. URL https://wwwbbssoomaaamne
[57] Sheng, V. S.S Provost, F. and Ipeirotis, P. G. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 614-622, 2008
http://wwwwainnncoo////////////e www///www//w/////oooooooo 1/088/26/
mwwthhaster--aaaaeeeeee -wwee-the-collegeeeeeee collccolege-rann rankiigg....
[42] Liang, R., Soloff, J. A., Barber, R. F. and Willett, R. Assumption-free stability for ranking problems, 2025.
[58] Singh, S., Nan, Y., Wang. A., D' D'Souza, D., Kapoor, S., Üstlin, A., Koyejo, S.. Deng, Y., Longpre. S., Smith, N., et al. The I. Leaderboard Illusion. arXiv preprint 2004,2 2025.
[43] List, C. Social Choice Theory. In Zalta, E. N. and Nodelman, U. (cds.), The Stanford Encyclopedia of Philosophy. Winter 2022 edition, 2022.
[59] Siththaranjan, A., Laidlaw, C., and Hadnilld-Menel, D. Dis- tributional Preference I Learning: Understanding and Account- ing for Hidden Context in RLHF, arXiv preprint 23 038288 2023.
[44] L.SData. Law school rankings 2022. https//www.lsd.aww/ www.schoo--rnnnngss----- 2022, 2022.

---

Selective Preference Aggregation
[60] Smith, Aggregation of preferences with variable elec- torate. Econometrica: Journal of the Econometric Society, 1027-1041, 1973.
[61] Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghal- lah, N., Rytting, C. M., Ye, A., Jiang, I., Lu, X., Dziri, N., et al. A roadmap to pluralistic alignment. arXiv preprint 2402.05070, 2024.
[62] Stutz, D., Cemgil, A. T., Roy, A. G., Matejovicova, T., Barsbey, M., Strachan, P., Schaekermann, M., Freyberg, J., Rikhye, R., Freeman, B., et Evaluating systems under uncertain ground truth: a case study in dermatology. arXiv preprint 2307.02 331992299, 2023.
[63] Tang. G. and Fan, N. A survey of solution path algorithms for regression and classification models. Annals of Data Science, 94):744 789, 2022.
[64] TestMax Inc. LSATMax: Comprehensive Isat prep course. https//testmmxxppp.commm come/stt, 2025.
[65] The Economist. Columbia is the latest university caught in rankings scandal, 2024.
[66] The Washington Post. Colleges are dropping out of rankings: Here's why yale says it's had enough, Nov 2022.
[67] Top Universities. Qs world university rankings for law and legal studies 2024. http::/wwww..opuuivsstttses.oo university-subjees raaph/eterraninnsslwwwwweea-e legal- legal- studies, 2024.
[68] Ture, I., Chang, M.-W., Lee, K., and Toutanova, K. Well- read students learn better: On importance pre-training compact models. arXiv preprint 08962, 2019.
[69] Uma, A. N., Fornaciari, T., Hovy, D., Paun, S., Plank, B., and Poesio, M. Learning from Disagreement: A Survey, Journal of Artificial Intelligence Research, 72:1385-14 1470, 2021.
[70] Velocity Test Prep. Top 50 law school rankings & comparisons.
sop-law-schools, 2023.
[71] Wang, J. and Shah, N. B. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. arXiv preprint 05085, 2018
[72] Wang, J. and Shah, N. B. Ranking and rating rankings and ratings. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13707, 2020.
[73] Wang, S. Deng. Q.. Feng. S., Zhang. H., and Liang. A survey on rank aggregation. In Proceedings of the Thirry- Third International Joint Conference on Artificial Intelli- gence (IJCAI-24), pp. 8281-8289, 2024.
[74] WorldQuant, Voting issues: A brief history of preference aggregation, 2025. URL https:///// http:://wwwwwwluuu......cmm worldquant.com/ieen beaing-issussa-hre hfff---tttoooo--r-eeeeeee cessed: 2025-01-31.
aggregation/. Ac-
[75] Wu, Z.. Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P.. Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Supplementary Materials
Selective Preference Aggregation
Supplementary Material for Section 2
14
A.I Notation
14
A.2 Encoding Individual Preferences Pairwise Comparisons
14
Supplementary Material for Section 3
15
B.1 Proof of Correctness
15
B.2 Proof of Uniqueness
15
B.3 Constructing All Possible Selective Rankings
16
B.4 Proofs of Algorithm Runtime
17
Supplementary Material for Section 4
18
C.I On the Top Tier
18
C.2 On Missing Preferences
18
C33 On the Distribution of Dissent
19
C.4 On Stability with Respect to New Items...
19
Supplementary Material for Sections 5 and
D.1 Descriptions of Datasets
21
D.2 List of Metries
21
D.3 Selective Ranking Paths
21
D.4 Expanded Table of Results
25
D.5 Supplementary Material for Section
25
D.6 Model Training
26

---

Selective Preference Aggregation
A Supplementary Material for Section 2
A.1 Notation
We provide list of the notation used throughout the paper in Table 2.
Object
Symbol
Description
Items
E [n]
{1,
;1....,.0 m
The objects being ordered, for which users have expressed preferences. Individuals expressing preferences for given items.
Users
k [p] = (l.
Individual preferences
k 1. E (-1,0,1}
Pairwise preference between items and j for user k. A partial ordering of n items into m fiers The preference between items and j in given ranking. The partial order returned by solving SPA, (D). The admitted dissent between two items and j.
Tiered ranking
T
Collective preference
Ti.j (T) (-1,0,1)
Selective ranking
S,
Dissent parameter
T [0, 2)
Table 2. Notation
A.2
Encoding Individual Preferences as Pairwise Comparisons
Representation
Notation y {0,1}
Conversion
Labels
Th. = [ut > 53 1I [ < w
Ratings
y E [m]
Tij = I [ut > x [ y/
Rankings
rh
[n] - [n]]
mf. = I (i* > r (1) - Ir
Table 3. Data structures capture ordinal preferences over n items. Each representation can converted into a set of (+) pairwise preferences in way that ensures (and assumes) transitivity. Ttem-level representations require fewer queries but may be subject to calibration issues between annotators.
One of the benefits in developing machinery to aggregate preferences is that provide practitioners with flexibility in deciding how to elicit and aggregate the preferences. In practice, such choices involve trade-offs that we discuss briefly below. Specifically, eliciting pairwise preferences from users requires more queries than other approaches [36]. However, may recover a more reliable representation of ordinal preferences than ratings or rankings [i.e., 56]. In tasks where we work with a few items, we can elicit preferences as ratings, rankings, or pairwise comparisons. tasks where we elicit rankings, can convert them into pairwise comparisons without a loss of information. In this case, eliciting pairwise comparisons test implicit assumptions such transitivity. In tasks where we elicit labels and ratings, the conversion is lossy since converting cardinal preferences to ordinal preferences. In practice, this conversion resolve issues related to calibration across users [see e.g, 72, 71J. In theory, it may also resolve disagreement [55].
14

---

Selective Preference Aggregation
B Supplementary Material for Section 3
This appendix provides supplementary material for Section 3, including proofs of the claims in this section and a description of the solution path algorithm.
B.1 Proof of Correctness
Lemma B.1. Consider the graph before running condensation or topological sort, but after pruning edges with weights below Tp. Items can be placed in separate tiers without violating Disagreements(T) S Tp if and only if there no cycle in the graph involving those items.
Proof. We start by connecting the edges in graph to conditions on the items in a tiered ranking and eventually expand that connection to show the one-to-one correspondence between cycles and tiers.
> Tp. This follows trivially from the definition of Wi,j == Ei= [ # 1]. From this, we know that if and only if there exists an are (i,j) that is not pruned before condensation, we cannot have tiered ranking with و -1 1 without violating Disagreements(T) 2 Tp.
First note that for any items j. Way > = Ei 1 | # 1]
If there exists a cycle in this graph, then we know the items in that cycle must be placed in the same tier. To show this, consider some edge j in the cycle. We know item j cannot be in a lower tier than without violating the disagreements property, from the above. So item j must be in the same or higher tier. But item j has an arrow to another item, k, which must be in the same or a higher tier than both j and i, and so on, until the cycle comes back to item i. This corresponds to the constraint that all items must be in the same tier.
If set of items is not in cycle, then these items do not need to be placed in the same tier. If the items are not in cycle, then there exists pair of items (i. such that there is no path from j to i. Thus i can be placed in higher tier than j without violating any disagreement constraints. Thus not all items in this set need to be placed in the same tier.
Thus we have shown that for a graph pruned with a given value of T, items can be placed in separate tiers for tiered ranking based on that same parameter т, if and only there no cycle in the graph involving all of these items.
We draw on this Lemma to prove the main result:
Theorem B.2. Given a preference aggregation task with items and p users, Algorithm I returns the optimal solution to SPA, for any dissent parameter E [0, 4).
Proof of Theorem B.2. Consider that items in our solution are in the same tier if and only if they are part of a cycle in the pruned graph (i.e., if and only if they are in the same strongly connected component). So items in the same tier if and only if they must be in the same tier for the solution to be feasible. No other feasible tiered ranking could have any of these items in separate tiers. So no other tiered ranking could have any more tiers, any more comparisons. To do so would require placing some tame-tierin items in different tiers. Thus, our solution is maximal with respect to the number of tiers, and with respect to the number of comparisons.
B.2 Proof of Uniqueness
Theorem B.3. The optimal solution to SPA, is unique for 7 E 0.5).
Proof of Theorem B.3. Let T denote an optimal solution to SPA,, We will show that the optimality T is fully specified by: (1) the items in each tier and (2) the ordering between tiers. That is, if we were to produce tiered ranking T" that assigns different items to each tier, or that orders tiers in a different way would be suboptimal or infeasible.
Consider a tiered ranking I' that is feasible with respect to SPA, for some T E [0,0.5). Let T' denote a tiered ranking where we swap the order of two tiers in T. We observe that the T" must violate a constraint. To see this, consider any pair of items i.j such that (T) = 1 before the swap, but (T') = 1 after the swap. One such pair must exist for any swapping of tier orders, because all tiers are non-empty. Because we elicited complete preferences, one of the following conditions
15

---

Selective Preference Aggregation
must hold:
Σ [ kE[p!
(I)
Σ kEip
I + + TT
(2)
Assuming that was an optimal solution to SPA,, we observe that the condition in Eq. (1) must be violated be- cause the original optimal solution was valid. Thus, we must have that EkENs) [T), * > Tp. This implies that Disagreements(T") > Tp for this tiered ranking. Thus, swapping the order tiers violates constraints because 7 < 0.5.
Now note that any separation of items from within the same tier is not possible without violating a constraint. This follows from Lemma B. which states that items that are part of a cycle in our graph representation of the problem' must be the same tier for solution be valid. And, as specified in our algorithm, we know our optimal solution has tiers only where there eycles in the graph representation of the problem. So any tiers in the optimal solution cannot be separated.
We can still merge two tiers together without violating constraints, but such an operation reduces the number of comparisons and would no longer be optimal. And after merging two tiers, the only valid separation operation would be simply to undo that merge (since any other partition of the items that merged tier, would correspond to separating items that were within the same tier in the optimal solution). we cannot use merges as part of an operation to reach a valid alternative optimal solution.
So know that for the optimal solution, we cannot separate out any items within the same tier, and we cannot reorder any of the tiers. Merging, meanwhile, sacrifices optimality. Thus, the original optimal solution is unique.
B.3 Constructing All Possible Selective Rankings
We start with a proof for Proposition 3.1.
Proof of Proposition 3.1. Recall that in Algorithm 1, an edge (i, with weight way is excluded if at least Tp users disagree with the preference i. We observe that Wayj Ekjpp Il | 2 0) corresponds the number of users who disagree with the preference j > i. Given dataset, denote the set of dissent values that could lead to different outputs as:
== U 7 Bi
I 20
<
kEjp
This corresponds to the set of unique wsg/p p for all é,j, with the value 0 included as well. To see this, note wi, Exe(o! [ 2 0]. We will now show the following Lemma, which will resolve the original claim.
Lemma B.4, Given any two adjacent elements D W (2). All dissent values in E b) lead to the same selective ranking as the selective ranking for a.
Proof. To show this, note that there exists no edge i + j such that op < wij < bp. If there did exist, then we would have
Wijj b.
This would imply that W would have to include an additional between a and b. But and are adjacent in W. This is a contradiction.
Since there exists edge such that ap way < bp. there exists edge such that the decision to include its are in the graph changes based on what value of dissent we select in (a, b). Recall that we exclude i - j iff Waj 2 Tp
Now that we know that for any two adjacent values b in W U ( all dissent values in [a, b) lead to the same tiered ranking as with dissent value a, we know that for any dissent value 7 E [0, })), the largest value of 7' E W that is < T will
after pruning edges of weight below 7
16

---

Selective Preference Aggregation
lead to the same tiered ranking. Simply substitute 7 in for a, and the smallest value above T in W U (5) for b (such value exists, on both sides, because 0 and 2 both E WU (2), and E (0.2)).
Thus we have shown the required claim.
Algorithm
We present an algorithm to construct a solution path of selective rankings in Algorithm 2.
Algorithm 2 Solution Path Algorithm
Input: D = (estm)kee))))
preference dotaset initialize solation path
1: S =}
Construct Initial Preference Graph
2: wij & -kk(n) I m 2 0] for all i E [n]
preferences claiming t
3: VI + [n]
Verfices represent
4: A + {(i +) 5) Wijj 2 0}
Arcs for observed preferences
Construct Selective Rankings for All Possible Dissent Values
5: W + [wi, for all i, E [n]] way < [[] U {0}}
Set of dissent parameters (see Proposition 3.1)
6: for E W do
7:
A₁ + Ar/((i + j) El wa, 2 Tp}
Add ares with support 2 TP Group items into tiers
8:
VT + ConnectedComponents (T, Ar))
9:
AT {(T' T") 3i T.j j 7" (i+ +) A13
Add edges between items supervertex components on directed edges
10:
(h (1........ .I.....,IIvy)) TopologicalSort Ar))
11:
S. + (Ti.....: (iv
12:
S S SU {S+}
13: end for
Output: S
Selective rankings that cover the cisigrrrionnnnissnneemnnt frantier
Given preference dataset Algorithm 2 returns finite collection of selective rankings S that achieve all possible trade-off's of comparability and dissent. The procedure improves the scalability by restricting the values of the dissent parameter 7 as per Proposition 3.1 in Line 2, and by reducing the overhead of computing graph structures. In this case, we construct the preference graph once in Line 4, and progressively add ares with sufficient support in Line 7.
Algorithm 2 assumes a complete preference dataset i.e.. where have all pairwise preferences from all users. In practice, we can satisfy this assumption by imputing missing preferences as described in Proposition 4.2. Alternatively, we can also add additional step after Line check that the item graph (V,, remains connected.
We benchmarked Algarithm 2 against Algorithm in Fig. 4 on synthetic preference aggregation tasks where we could vary the number of users and items. We fixed the number of users to 10 users. For each user k E [p). we sampled their pairwise preferences as T... ~ Uniform(1,0, -1).
Details on Synthetic Dataset in Fig. 4
B.4 Proofs of Algorithm Runtime
Algorithm Line computes sum while visiting each pairwise preference for each judge, taking O(n'p) time, All subsequent steps linear in the graph size: both CommeteddCopponents and TopologicalSort are linear in input size, and the other steps are just operations each edge. So the total runtime is On+pp)
Algorithm Note that [W]= = [8], because Wij only takes integer values and there are [&] integers between 0 and [&] inclusive of and exclusive of [ so the for loop runs [ times, and everything in the loop runs in time linear in the graph size, so 2n Thus the whole runtime of the loop is On²p). The preprocessing, as before, is On+2p) time. Note that computing W can done On²pp) time: just iterate through all wiy for each the [T] possible distinct values, and add the value W if it occurs least once. Thus the total runtime is the sum of constant number of O(?2p) steps, meaning the total runtime is On²pp).
17

---

Selective Preference Aggregation
C
Supplementary Material for Section 4
This appendix provides proofs and additional results to support the claims in Section 4.
C.1 On the Top Tier
Theorem C.1. Consider a preference aggregation task where most <. 3 of users strictly prefer one item over all other items. Given any T E [0, 2). the tiered ranking from SPA, will include at least two items in its top tier.
Proof. We show the contrapositive: having > (1- users rank an item first guarantees having only one item in the top tier. Without loss of generality, call an item with > (1 - T) users rating a specific item first A. Consider WLOG any other item B. No more than r users claim either of > or B A, because we know > (1 - T) users claim A > B. So for any tiered ranking that places some other item tier, and have one more item. Since the result of our algorithm must have the maximal number of tiers, we cannot have a case where A is in the same tier any other item.
the same tier as A, we could instead place A above all other items in that
Lemma C.2. Consider a preference aggregation task where a majority of users strictly prefer an item in over all items i + io. There exists some threshold dissent E [0, 1) such that for all >> To, every selective runking we obtain by solving SPA, will place is as the sole item in its top tier.
Proof. Let a denote the fraction of users who strictly prefer is over all items. Since a > 1. we observe that at most 1- a < 11-5 users can express a conflicting preference. Given any item i + io, let To = 1 - a denote the fraction who users who believe either of i > io or ~ ig- For any tiered ranking that places in and in the same tier, we could instead place i above all other items that tier, and have one more tier. Since our algorithm returns tiered ranking with the maximal number of tiers, we cannot have case where is in the same tier any other item.
2.
C.2 On Missing Preferences
Proof of Proposition 4.2. If we are missing preferences, our algorithm's behavior is to assume all missing preferences would be in disagreement with any asserted ordering. This exactly corresponds to the actual disagreement if the true values are all asserted equivalenneeindiiffeeeee and upper bound on dissent if the preferences are directional. By doing this, we guarantee that the disagreement property will be satisfied under any possible missingness mechanism, even a worst-case adversarial mechanism. We denote missingness as (i, =? if the preference is missing. This property is trivial to show. Consider that
Disagreements(T)
max "ETT T>T
Σ I [ I + 1]
kE[p]
max (JT,.T
Σ
[ E -1,7-
TyT"
kElp]
Σ
[ E [13] if we we set all missing values n., =? to T = 0
max (JET T>T"
kElpl
Given that overall disagreement when preferences are imputed cannot increase, we have that T..j ( Ti. (SSafee))
More formally: from the disagreements argument above, we know that Dsafe has the same or more disagreements for any preference than does Dirue Every selective comparison in Stafe corresponds to a pair of items in distinct strongly connected components under the constraints from Deafe (see Lemma B. When we relax to only the constraints from Dime, we cannot have more disagreement for any preferences, so those items will remain in distinct strongly connected components. Since they remain in distinct strongly connected components, Lemma B.1 tells us the two items will not be in the same tier.
To show that the two items will have the same ordering in both tiered rankings, note that even under Dinue there must be constraint on one of the two directions of the preference". And that constraint will still hold under Diafe, which is no less constrained than Dimee Thus, StM" cannot have a preference in the opposite direction from Stafe
Given dataset of complete preferences and [0, 1) at least of the following must hold: Exero I | # 1] Tp Ekels) [ * -1] > Tp. This is because for the former claim to true, we'd 'd need at least (1 - -)p preferences to be 1, which then
18

---

Selective Preference Aggregation
C.3 On the Distribution of Dissent
selective ranking only allows comparisons that violate at most rp of preferences in dataset. In practice, these violations may be disproportionately distributed across users or items. For example, we may have task with where the same user disagrees with all comparisons in dataset. Alternativly,, the violations may be equally distributed across users so that there is no coalition of users who agrees with all preferences. In Remark C.3, we bound the number of users who can disagree with selective ranking.
Remark C.3. A T-selective ranking contradicts the preferences of at most 4 Tp users.
The result in Remark C.3 only applies in tasks where the number of users exceeds the number of selective comparisons. In other tasks where the number of selective comparisons exceeds the number of users the statement is vacuous we cannot rule out worst-case where every user disagrees with at least one comparison.
Proof. We observe that selective ranking with a single tier makes no claims. Thus we can restrict our attention cases where the r-selective ranking contains at least two tiers. Given selective ranking with more than 2 tiers, then any user who disagrees with the ranking of items from non-adjacent tiers, also disagrees with the ranking of two items in adjacent tiers. So every user with a conflict must disagree about the ordering of at least one pair of items in adjacent tiers. This bounds the number of users who disagree as T times the number of distinct pairs of items in adjacent tiers. This is because no more than 7 proportion of users can disagree with any one pairing.
The number of distinct, adjacent-tier pairs is of the form E together contain all n items (E,=, [T|m = n). This quantity is maximized when we have [T] = 2 tiers that contain 2 items each (rounding if n is odd). In this case, the maximum value is I (or slightly below if n is odd). The worst case is tight, achieved with two tiers, each with half the items, and an even number of items.
-1 n where tier; contains my items, and all the tiers
C.4 On Stability with Respect to New Items
We start with a simple counterexample to show that selective rankings do not satisfy the "independence of irrelevant alternatives" axiom [8].
Example C.4 (Selective Rankings do not Satisfy IIA), Consider a preference aggregation task where we have pairwise preferences from 2 users for items and] j where both users agree that i j.
User 1: ix j
User 2: > j
In this case, every r-selective ranking would be (T) 1 for any [0, 0.5).
Suppose we elicit preferences for a third item z, and discover that each user asserts that is equivalent to a different item:
User 1: i~2ビj <> i j 2 j inz
User 2: 8 aスi~z - itj j~z i8
In this case, every r-selective ranking would be TT. 0 for all E 10, 1). This violates IIA because the relative comparison Wi.j (T)) changes depending on the preferences involving 2.
Proposition C.5. Consider preference aggregation lask where for given 7 E [0, 1) we construct selective ranking Sn using dataset D of complete pairwise preferences from users over items in the itemset [n]]. Say we elicit pairwise preferences from all users with respect to a new item n t 1 and construct a selective ranking Sa41 for the same over the new itemset in + 1). Given any two items i.j E [nl, we have that
(S = (S,) V
= -).
Proof. It is sufficient to show the following:
forces the latter claim be false because we'v 'e set (1 - T)p > Tp values to be something other than -1.
19

---

Selective Preference Aggregation
When Tey + -1, we never have (5,(((+1)) -1
When Ta.g T (Sn) # 1, never have Ti.j (Sn+1) 1.
Given a dataset of complete pairwise preferences and T E [0, 2), at least one of the following conditions must hold:
Condition I:
I
[ + 1]
TP
KE [p)
Condition II:
Σ kEIP
I -1 Tp
This is because for Condition I to be False, we would need at least (1 7 preferences to be 1, which then forces Condition II to be true because we have set (1 - T)p Tp values to be something other than -1.
[p) [ * #1] > Tp: then we know that (T.((S.) * 1. Otherwise We would violate the disagreement constraint in SPA,, Note that eliciting preferences for new item does not change Exe(pl I ( * 1]. So we still have ΣxE(p) [[ [ # 1] > TP, and we still have (Sn+() # 1. Thus, we have that both (S,( o and T ( (5n+)) + 1. We can apply symmetric argument to show Condition II holds. In this case, we would have that E II [ -1] > ry and see that both (S.) + -1 and (S.((nn1) -1.
Consider WLOG that Condition holds. If E>e(p
This guarantees that the claim of Proposition 4.3 cannot be violated. When ( Ti.j(Sn) + -1 we never have (S,(()+1) = -1, when (S,(Sn)) j 1 we never have (S,+()+)) = 1. Thus we have proven the claim by cases.
too does (S,,() 0. When
20

---

Selective Preference Aggregation
D Supplementary Material for Sections 5 and 6
In what follows, we include additional details and results for the experiments in Section and our demonstration Section 6.
D.1 Descriptions of Datasets
Dataset
Format
Description
abs
7 Coaches
100 Voters
Ballots
2021 NBA Coach of the Year Award, where sports journalists vote for the top coaches
lawechool
Rankings
26 Schools
Runkings
Top U.S. law schools ranked orgumizatiins bused acudemic performance, reputation, and other metrics in 2023.
survivor
Fans
39 Scasons
Rankings
to worst, Benchmark recommendation dataset collected in Japan, where participants provided pairwise preferences over different types of sushi: ehi (shrimp). anago (sea eel). maguro (luna), ika (squid), uni (sea urchin), ikura (salmon roe), tamago (egg), (fatty tuna), ckkaa-maii (tuea roll), and kappa-maki (cucumber roll).
Rankings where fans show Survivor seasons
suchi
5,000 Respondents
10 Sushi Types
Pairwise
csrankinga
Subfields
175 Departments
Rankings
Rankings of computer science departments from esrankings.org based on research output AI, NLP, Computer Vision, Data Mining, and Web Retrieval.
Table 4. Overview datasets. We consider five datasets from salient use cases preference aggregation.
D.2 List of Metrics
In what follows, we provide detailed descriptions of the metrics in Table 1.
Metrie
Formula
Description
AbstentionRate(()
[Tr,((]) = .1]
Given a selective ranking over n items T, the abstention rate represents the fraction pairwise comparisons where we abstain.
DiagreementRate((T7 D)
Σ Σ
jott # +m,(()mm.(()) 1]
Given a ranking over n items T, the disagreement rate repre- sents the fraction of individual preferences in D that disagree with the collective preferences in T..
n(n Dpp Rely (JEin)
#Tiers(St))
18&1
Giienn selective ranking Sr, the numher of tiers. For standard methods, each rank is converted to a tier.
#Topltems(S.)
[Ti
Given S, ,, the number of items in the top tier. For standard methods, each rank is converted to tier.
DiagremeenPorteer((.D))
median
Σ I [ntl o mL)(T)]
The median fraction of preference violations users.
AElel
n(n 11/2
Eisemn)
[T.y # Tú ATu #0
Sampling (T,D)
median Ne().....)
Given the ranking produced on the full dataset T, the median proportion of collective preferences that inverted when drop 10% of preferences. We construct bootstrap esti- mate by applying the method domly drop 10% of all preferences and obtuin Nú, rankings (T' T%...
Σ Σnjejo 1 ITis Tra 0
datasets where
Enseio [eg # Th A ATu #0A T #0
Adversarial (I,D)
MBX
Liseju Tip # #0]
Given the original ranking T. the maximum proportion collective preferences inverted when flip of individual preferences. We construct bootstrap estimate where first upply the method to No datasets where randomly flip 10% all preferences and obtain No rankings (T*, I .N.)
Table 5. Metrics used to evaluate comparability, disagreement, and robustness of rankings in Table 1 and Appendix D.4
D.3 Selective Ranking Paths
We present the solution paths of selective rankings for each dataset in Section 5 in Fig. 8 to Fig. 12.
21

---

Selective Preference Aggregation
100%
Williams
80%
Thibodeau
Comparison Rate
nyder
60%
Rivers
40%
Nash
Malone
20%
McMillan
0.0
0.1
0.2
0.3
0.4
0.5
0%
0.0
0.1
0.2
0.3
0.4
0.5
Max Disagreement
Max Disagreement
Fig. 8. Selective rankings for the nba dataset (n 7 items and 100 users). We show the tradeoff between comparision and disagreement (left) and the unique rankings over the dissent path (right).
100%
Caperan
Cambodie
blaadd
Phutpin
80%
Chine
Gollah
Comparison Rate
Amazon
Regi
60%
P enamm
(I
Samos
Jatemals
40%
Sur
Extrncto
Chargers
Ialand
20%
Vorld
Gabon
Pactfic
1806
0%
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
ax Disagreement
Max Disagreement
Fig. 9. Selective rankings for the survivor dataset (n = 39 items and = 6 users). We show the tradeoff between comparision and disagreement (left) and the unique rankings over the dissent path (right).
22

---

Selective Preference Aggregation
100%
Maguro
Ika
80%
Uni
Comparison Rate
Anago
60%
Ikura
Ebi
40%
Tamago
Toro
20%
Tekka
Kappa
0.46
0.48 Max Disagreement
0.5
0% 0.4.
0.48 Disagreement
0.5
Fig. 10. Selective rankings for the sushi datasct 10 items and p 5000 users). We show the tradcoff between comparision and disagreement (left) and the unique rankings over the dissent path (right). Note that only subset of dissent values are shown for clarity, focusing on largest areas change.
100%
CMU
Stanford
80%
UIUC
Comparison Rate
Cornell
60%
Item
ASU
BU
40%
Augusta
BC
20%
BU
BYU
0%
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Max Disagreement
Max Disagreement
Fig. 11. Selective rankings for the carankings dataset (n = 175 items and p p users). We show the tradeoff between comparision and disagrcement (left) and the unique rankings over the dissent path (right). We show the top 10 items, sorted by tier and alphabetically within each tier.
23

---

Selective Preference Aggregation
100%
Yale
Stanford
harvar
UChicago
80%
UVA
Penn
Companison Rate
Duke
60%
chigan
Berkeley
40%
UCLA
Georgetow
Comel
UT
20%
Vancerbll
USC
BU
GWU
0%
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Max Disagreement
Max Disagreement
Fig. 12. Selective rankings for the lawschool dataset (n 20 items and p users). We show the tradeoff between comparision and disagreement (left) and the unique rankings over the dissent path (right).
24

---

Selective Preference Aggregation
D.4 Expanded Table of Results
We include an expanded version of our results for all methods and all datasets in Appendix D.4. This table covers the same results as in Table 1, but includes the following additional metrics:
1. A Abstentions [Intervention], which measures the proportion of strict collective preferences (e.g., A > B or A < B) that turn into ties or abstentions in the ranking that we obtain after running the method on a modified dataset.
2. A Specifications [Intervention], which measures the proportion of ties or abstentions that turn into ties or abstentions in the ranking that we obtain after running the method on a modified dataset.
We report these values for same interventions we consider in Section 5, namely: Sampling, where we run the method dataset where we randomly omit 10% of individual preferences; and Adversarial, where we run the method dataset where randomly flip 10% of individual preferences. Each value corresponds bootstrap estimates where we perform the same estimate 100 times. For clarity, we list the A Sampling as A Inversions -Sampling, and Adversarial - Inversions.
Selective
Standad
Datest
3%
SPA
Bortla
Copaland
MC4
KemenySeac
Iisagrement
2.0%
8.1%
1.1%
Moliai Disgreenen Use
0.0%
0.0%5
4.85
9.59
9.56
Abstoriot Rae
100,0%
42.9%
26.655
niba
28 6% mis
NBA (o9)
e aerra
100 users
Specitictionn Sampling
A Drvesiooss Sunpling
Invessiees AEveriaiial
Top frre
Disson
Virr
0,0000
0.04
0.0%
0.045
0,2600
0.045
0.046
9.515
0,4900
0.06
0.065
0.065
19.0
0.09
4,8%
1900
0.07
4.8%
19.05
0L0%
4.5%
14.59
4.5%
14.39
DO9
4.5%
A Specttratiee Alvininial
0.05
9.19
0.065
0.0
D05
A. Abstettios Sunpling
0.0%6
0.066
28.66
0.0%
0.0%
0.0%%
0.0
0.6
Aloieatioss Adversatial
0.046
19:06
28.06
0.0%
4.8%
33.59
0.09
0.9
Disgresneet
0.0%
0.2%
0.15
6.5%
6.6%
6.7%
Modias Disggeesent pei Uses
0.0%
0.11
0.15
7.2%
7.1%
6.5%
0.14
7.1%
Aboientin Rae
94.9%
42.5%
Top
0,3338
Puple Rock
0.00 missing
S Specifcation Sampling
A lrvesisus
liveskess Adversarial
0.5%
0.045
0.0%
0.066
0.995
2.0
1.3%
034
0.346
1.5%
0.0%
0.02
Spacificiions Advenanal
0.0%
5.1%
0.99
64%
0.94
0.0%
0.0%
Abacntion Sumpling
0.0%
52.4%
57.5%
0.%
0.1%
$0.0%
0.%%
0.0%
Absieaioas Advenanial
0.0%
$7.5%
$7.5%
0.0%
0.46
89.5%
0.4%
0.4%
Diagreenent Raie
0.0%
0.15
3.15
4.0%
4.%%
6.216
4.1%
6.1%
Medias Disgraenen per Use
0.0%
0.0%
1.65
4.2%
2.45
2.68
2.19
2.15
Abssemio Rae
40.5%
36.55
4.25
lawschool
Top
Tark
15
20
70
30
Diseni
0,000
0,4000
09 nissng LSDau (44]
Inversoess Adveniarial
Irvesions Sunpling
0.0%
0.0%
0.06
0.06
0.06
1.0%
3.7%
2.6%
1.1%
24.55
29.5%
Spectication Sumpling
0.0%
Specit
Sampling
Alolcatioes
59.5%
0.06
1.66
0.09
0.06
Disareenze Raie
0.0%
0.0%
0.15
12.3%
12.2%
12.2%
13.7%
Modias Disgeezeen Use:
0.0%
0.0%
0.15
12.3%
12.6%
12.3%
13.5%
Aboienton Rale
100.0%
98.9%
95.56
esrani lnge = 175 Home
Top liens
a Tiess
178
175
165
170
175
same 0.00 missing Borget (11)
A Invesious Sunpling
Diacai
0,0000
0.0%
0,2000
0.0%
0,4000
0.0%
0.8%
0.8%
0.11%
Inversiuss Advessaiial
0.06
0.06
0.05
3.1%
1.7%
0.11%
11.19
Specilictions Surpling
0.06
0.06
0.06
0.0%
0.1%
$4.6%
Spactiiatioss Advenanaa
0.5%
0.0%
0.0%
0.0
0.1%
0.0%
Absention Sumpling
1.1%
4.5%
0.0
0.0%
0.0%
Absientioss Adveranial
0.2%
4.5%
0.0%
0.1%
0.0%
Modias Disqgeenen
Absomtoon Rae
100.0%
64.45
0.06
a Tlare
10
10
Top fime
0.0% missing Kanisiin (39)
uaes
A Inveniooa Sampling
Inversoess Advenirial
Doami
0,0000
0.0%
0.0%
0.0020
0.06
0.5%
0.496
0.06
0.5%
0.0%
2.2%
2.2%
0.0%
11.15
2.34
11.19
2.2%
11.19
2.24
Specifcation Sampling
0.05
0.0%
0.66
0.05
0.0
000
000
D.5
A Specttrattes Afvenattal
0.05
0.0%
0.6
005
008
DO5
3. Aboration Sampling
0.05
15.55
10005
0.5
008
DO5
Absicnisons Alvemantal
0.0%
0.5%
10005
0.%
0.8
15.6%
000
D.5 Supplementary Material for Section
Selective Aggregation with Binary Annotations A key challenge in applying SPA to the DICES dataset is that it clicits categorical labels for each item individually, rather than comparative ratings. This conversion can create unnecessary
25

---

Selective Preference Aggregation
1.0
0.8
0.6
TPR
0.4
0.2
0.0.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
Borda Expert Majority SPA
Fig. 13. ROC model curves on the training set for all four methods. We highlight the label for cach method closest to tpr> 90% on labels with large dot. ySPA the only method whose chosen operating point keeps the true-positive above 80 % on the model output while controlling FPR.
cquivalence, where pairwise preference is inferred tie (Tf 0). This is not reflection of user's true judgment but an artifact of two limitations: (1) users annotate items individually rather than comparing them, and (2) the annotations are restricted to {0, } instead of granular ratings. For example, user may believe item A is significantly more toxic than item B, but the conversion results in a tie both were labeled "toxic" distinction that is lost this setting.
address this by running a variant selective aggregation where we construct aggregate labels from users who express strict preference between items i j or j > In addition, we assume that users who have not asserted an opinion (because of dataset scope) are "deferring judgment" to those who have.
For each pair of items i, [ri]. we define:
$1, = E
| = 1] denote number of users who strictly prefer item to item
$3 Exelol I m 1 denote the number of users who strictly prefer item item i.
The aggregate preference weight W,y as the proportion of users who strictly prefer to among those who expressed strict preference, scaled to n items. Note that all item pairs had at least preference:
$L
Wa == n
Si.j + 3j.
In this setup, the dissent parameter no longer maintains its standard interpretation because users may not assign preference to each item, and items may be assigned different weights. As result, we produce selective rankings for all possible dissent parameters that lead to connected graph in Algorithm 2. In this case, the maximum dissent value is set to threshold value where Line returns disconnected graph.
D.6
Model Training
All experiments used 5-fold cross-validation on the training split. We fine-tuned a BERT-Mini model; all fine-tuning experiments used 5-fold cross-validation on the training split. We optimized with a learning rate of 2 x 10-5 for up to 25 epochs, employing early stopping. We trained in mini-batches of size 16 and enabled oversampling of minority classes in each batch.
26


Reviews:
[{"review_id": "sWSTMorkC9", "reviewer": "Reviewer_dGCS", "summary": "This paper introduces selective preference aggregation (SPA), a framework that aggregates ordinal preferences into partial orders (tiered rankings) to avoid arbitrating disagreements. The core contributions include a graph-based algorithm, theoretical guarantees (e.g., stability under missing data), and empirical validation across datasets like NBA rankings and toxicity detection. SPA demonstrates improved transparency and robustness compared to traditional methods (e.g., Borda, Kemeny).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your response! We provide tables at https://tinyurl.com/2ybsfs95\n\n> In RLHF settings where the number of items n is typically small (<5) …This would clarify whether SPA’s advantages (e.g., robustness, transparency) persist in the small-scale preference comparisons characteristic of RLHF.\n\nCertainly! SPA's ranking robustness primarily hinges on the number of judges (p) and their agreement (1 - $\\tau$). High consensus is needed for admitted pairwise preferences. More judges clarify patterns and enable rankings at lower dissent.\nLow granularity suggests the need for more judges. Deep disagreement might warrant other methods (like expert input), highlighting SPA's transparency. (See reviewer responses mXsg/FgRQ for details). \nWe provide an example in Table 9. Increasing p (10 -> 30) allows more tiers (3 -> 4) at lower dissent (0.5 -> 0.333). Limited rankings (dissent < 0.5) can signal the need for more judges and identify items with strong agreement.\n\n\n\n\n> SPA assumes uniform weighting of user preferences. However, in real-world applications (e.g., expert-driven labeling), users may have heterogeneous weights (e.g., experts’ preferences matter more). Does SPA can be directly applied in such a scenario? and analyze how it affects tiered rankings and guarantees (e.g., stability under missing data)?\n\nThis is possible. The most trivial way is to include expert preferences as equivalent to multiple judges. We can also use arbitrary nonnegative weights on each judge. The guarantees then become relative to not overruling a certain sum over weighted preferences of judges. \n\nJust set\n\n\n$\\\\textrm{Disagreements}(T) := \\\\max_{i,j \\\\in [n]} \\\\sum_{k=1}^p w_k \\\\mathbb{1}[\\\\pi_{i,j}^{k} \\neq 1, \\\\pi_{i,j}(T) = 1]$\n\nWhere $w_k$ is the weight of a given judge, renormalized so that $\\sum_{k\\in\\intrange{p}} w_k = p$. We can maintain many of the guarantees premised on the disagreement constraint - for example, stability under missing data. Some theory does change, for example, $|\\mathcal{W}|$ is no longer necessarily linear in p. \n\n\n> Key theorems are valid under stated assumptions but require clarification to resolve inconsistencies …may not hold for non-random missingness.\n\nProposition 4.2 does hold even for adversarial, non-random missingness - that’s one of the more fascinating ramifications of our approach, actually. The proposition holds because indifference is a conservative assumption to make - it counts as the same or more disagreement with any possible comparison than would any other possible value. We never make comparisons that would be invalid given the true values, regardless of the missingness mechanism.\n\nIt may be possible to model missing preferences to extend coverage. See our response to reviewer sHCm (future work). \n\n\n\n> SPA’s linear runtime is untested on large n (>10^4), limiting real-world applicability.\n\nWe appreciate your prior note that there are applications for n<5; please also see our response to FgRQ and sHCm, which address the scale of experiments and use cases. We would like to note that in many domains (e.g., fine-tuning), a limited amount of examples (50-100) is enough to improve performance. From Meta: “A general trend we’ve seen is that quality is more important than quantity …  documentation suggests even a 50- to 100-example dataset can potentially make a difference.”\n\n* https://ai.meta.com/blog/how-to-fine-tune-llms-peft-dataset-curation/ \n\n\n> SPA’s 0% inversion rate under Δ-Gaming (Table 1) supports robustness claims, but the absence of p-values or confidence intervals undermines statistical significance.\n> Robustness claims (e.g., Δ-Gaming) lack significance tests, making it hard to validate improvements.\n\nWe would like to clarify the nature of the Δ-Gaming (Δ-Adversarial) metric reported in Table 1. SPA exhibited zero inversions across 100 simulations, each with 10% adversarial preference flips. This 0% reflects the maximum observed inversion rate under the worst-case condition, demonstrating strong stability against manipulation. As such, CIs/p-values on this reported maximum are not directly applicable. We refer you to Appendix C for guarantees on stability and robustness.\n\n\n> The proposed methods are theoretically sound, but evaluation criteria and baseline selections are outdated and contextually limited.\n\nWe wish to clarify our intent with the choice of metrics. Our chosen evaluation criteria (e.g., Disagreement Rate, Abstention Rate, Δ-Adversarial robustness) directly quantify the core characteristics of SPA – its ability to manage the trade-off between coverage and dissent, and the resulting robustness gained through abstention. If the reviewer feels specific metrics would enhance the evaluation, we are open to revising/adding these where possible. \n\n\nWe hope our responses clarify the points raised and are happy to address any remaining questions.", "claims_and_evidence": "Most claims are supported by evidence, but critical gaps in statistical rigor and baseline comparisons weaken their persuasiveness.\n\n1. SPA’s 0% inversion rate under Δ-Gaming (Table 1) supports robustness claims, but the absence of p-values or confidence intervals undermines statistical significance.\n\n2. SPA’s 18.4% label error in toxicity detection (Figure 5) is compelling, but the expert baseline’s high error (43%) limits its interpretability.\n\n3. RLHF Applicability: The claim that SPA generalizes to RLHF with small n (\\(<5\\)) lacks empirical support.", "methods_and_evaluation_criteria": "The proposed methods are theoretically sound, but evaluation criteria and baseline selections are outdated and contextually limited.", "theoretical_claims": "Key theorems are valid under stated assumptions but require clarification to resolve inconsistencies.\n\n- Proposition 4.2: Stability under missing data relies on imputing preferences as indifference ($\\pi$=0), which may not hold for non-random missingness.", "experimental_designs_or_analyses": "Experimental designs are generally rigorous but lack critical controls and scalability tests.\n\n1. SPA’s linear runtime is untested on large n (>10^4), limiting real-world applicability.\n2. Robustness claims (e.g., Δ-Gaming) lack significance tests, making it hard to validate improvements.", "relation_to_broader_scientific_literature": "The paper situates itself within social choice theory and machine learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper’s originality and practical impact are strong, but structural and clarity issues hinder accessibility.\n\n**Weaknesses**\n1. Related work is embedded in the Introduction, lacking a dedicated section. This prevents readers from gaining a systematic understanding of the field’s context.\n2. Non-essential theorems (e.g., Section 4) and complex proofs in appendices impede readability.", "other_comments_or_suggestions": "**Typos**\n- Missing notation of \"$\\Delta$-Gaming\" in Table 1. It should be \"$\\Delta$-Adversarial\", based on the caption.\n- L349, \"We randomly sample or flip\" should be \"We randomly drop or flip \"", "questions_for_authors": "1. In RLHF settings where the number of items n is typically small (<5), SPA’s tiered rankings may collapse to trivial solutions (e.g., all items in a single tier due to insufficient data). To validate SPA’s utility in AI alignment tasks, could you provide ablation studies or theoretical analysis demonstrating its behavior for n < 5? This would clarify whether SPA’s advantages (e.g., robustness, transparency) persist in the small-scale preference comparisons characteristic of RLHF.\n2. SPA assumes uniform weighting of user preferences. However, in real-world applications (e.g., expert-driven labeling), users may have heterogeneous weights (e.g., experts’ preferences matter more). Does SPA can be directly applied in such a scenario? and analyze how it affects tiered rankings and guarantees (e.g., stability under missing data)? This would enhance SPA’s applicability to scenarios where users have varying levels of credibility or expertise.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NyynUslAAY", "reviewer": "Reviewer_FgRQ", "summary": "The paper introduces a new framework for ranking via preference aggregation while allowing for disagreement of the voters. Unlike many traditional methods that enforce a total order, the approach aims to construct a partial ranking, only comparing items where a sufficient majority agrees. The paper proposes an algorithm graph based algorithm to construct ordering relationships of the items. They provide a correctness analysis and asymtotical runtime of it. The authors also apply their method on small datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response\nThank you for your time and feedback! We include supplementary tables here: https://tinyurl.com/2ybsfs95\n\n> The authors claim that the algorithm is fast and scalabile however this is not convincing …\n> … 60-100 seconds for 500 items seems rather slow.\n\nAs you’ve noted, our empirical runtimes were slower than our asymptotic analysis suggests. That is due to some inefficiencies that we have since resolved in our implementation. We provide a table of updated runtimes (in seconds) in Table 6 and have updated the figure in our manuscript. (Note that the naive method remains slow because it does not leverage the path approach’s tricks.)\n\nWe’re happy that you agree our proven asymptotic runtime of O(n^2*p) is reasonable. Indeed, many competing methods can be quite computationally intensive - for instance, Kemeny-Young is NP-Hard and must be solved with integer programming in many contexts. Our algorithm’s asymptotic scaling - which is linear in the number of input pairwise preferences - gives a strong characterization of our algorithm’s scalability, independent of any quirks in implementation or hardware.\n\n\n> Often, there are (of the order of) n*log n pairwise comparisons available…, for missing pairwise comparisons for bigger datasets.\n\nWe appreciate your point regarding the lack of missing pairwise comparisons. We provide Table 7 highlighting the number of comparisons at $\\tau$_max with 5% of all pairwise comparisons dropped. We also refer you to values of Δ-sampling, which serve as a median change with 10% of samples dropped, equivalent to 10% missing comparisons.\n> SPA_0 and SPA_min very often do not produce actual rankings …\n\nTo clarify, we do feel these values (even when creating minimal tiers) highlight key information. At SPA_0, any tiers > 1 indicate unanimous preference among users. The dissent rate at which we see SPA_min can be informative as well - in certain datasets such as Sushi, this value is high (> 0.4), which reveals high levels of underlying disagreement. This information can also be found essentially for free, since Algorithm 2 gives these tiered rankings as part of the process of finding more granular rankings like SPA_max.\n> The datasets are way too small …\n\nWe appreciate the reviewer's suggestion. Existing larger datasets are generally sparse; we discuss future adaptations with reviewer sHCm that would enable SPA. We note that the number of potential pairwise preferences processed in existing scenarios is already substantial - over 7.5 million for DICES (Section 6) – demonstrating SPA's ability to handle a considerable volume of preference data. SPA’s design and theoretical guarantees (Section 4) ensure predictable behavior at any scale. Several applications remain at small values of n; reviewer dGCS points out potential applications of SPA in RLHF at n < 5, and we note applications that exist at the current scale. If the reviewer wants further clarity on larger scale data, we are willing to conduct experiments on synthetic data.\n\n\n> The distinction with ranking with ties is not very clear … this distinction theorically.\n\nWe use 'incomparable' when no strict pairwise preference exists between items. Items in the same tier are thus 'incomparable', which can arise from various situations like cycles, evenly split preferences, or judges explicitly marking items as equivalent (see Fig 3). Therefore, 'incomparable' (in the same tier) does not necessarily mean 'equivalent'. For example, if ⅔ of judges think A > B, ⅔ think B > C, and ⅔ think C > A, that does not necessarily mean A = B = C. Perhaps 0 judges think any of those items are equivalent. Users can state equivalence, and our ranking considers it. However, the distinction between abstention/disagreement and asserted equality is a key benefit within our tiers.\n>  Also the empirical analysis should benefit from a direct comparisons with ranking with ties methods.\n\nRegarding comparison to tie-handling methods: standard baselines like Borda or MC4 do produce ties when scores are equivalent, but by design these instances are unlikely. Our mechanism —abstaining based on exceeding a pairwise dissent threshold — differs from methods inducing ties via score equivalence. \n> what is exactly Comparisons(T )? Also T_mathcal is undefined, I guess it is the family of tiered rankings\n\nWe define $\\Comparisons{\\tierset} := \\sum_{i,j \\in \\intrange{n}} \\indic{\\aggprf{i}{j}{\\tierset} \\neq \\dnc{}}$ in Section 2, although we acknowledge issues with formatting in our submission that we have now fixed. Please let us know if you would like further clarification on our definition of Tiered Rankings (T_mathcal) in Definition 2.1.\n> What the colors in figure 2 represent?\n\nThe colors denote items in the same tier.\n> Perhaps the Bradley-Terry model….\n\nWe have added a reference, and include an additional experiment in our response to reviewer mXsg.\n\nWe hope these clarifications address your concerns! We are happy to resolve any further questions.", "claims_and_evidence": "- The authors claim that the algorithm is fast and scalabile however this is not convincing, expecially from an experimental perspective.\nIndeed:\n\n- The datasets are way too small. The higher number of items is 175. Since the runtime seems to be quadratic (in the items), I don't see reasons why not to apply it on thousands of items (also because other methods can handle such dimension) \n\n- Often, there are (of the order of) n*log n pairwise comparisons available (this is because it is enough to rank the n items with such amount of comparisons), but the case analyzed in the paper with % of missing pairwise comparisons different from 0, is only the one with 7 items. It would be interesting to see the behaviour of the algorithm in different cases, i.e., for missing pairwise comparisons for bigger datasets.\n\n- SPA_0 and SPA_min very often do not produce actual rankings. Often the number of Tiers is 1 which would mean that all items are incomparable, and thus not very informative\n\n- in figure 4 there is not reported the unit measure of the runtime. I think it is seconds, but 60-100 seconds for 500 items seems rather slow.\n\nFrom a theoretical perspective, the runtime of O(n^2 * p) is reasonable.", "methods_and_evaluation_criteria": "See the issues above on the datasets and experiments.", "theoretical_claims": "The theoretical analysis seems rigorous and sound. I checked the algorithm 1 and its runtime.", "experimental_designs_or_analyses": "Besides the issues previously mentioned in Claims And Evidence, the rest is valid and clear.", "relation_to_broader_scientific_literature": "The distinction with ranking with ties is not very clear. The reason why the method would deem items as incomparable and not equivalent is not very well explained. In particular, the items deemed incomparable might actually be evaluated and deemed equal, by some individuals/evaluators. I think that there is the need to better clarify this distinction theorically.\nAlso the empirical analysis should benefit from a direct comparisons with ranking with ties methods.", "essential_references_not_discussed": "Perhaps the Bradley-Terry model, which is commonly used to rank from pairwise comparisons colud be mentioned:\nBradley, Ralph Allan; Terry, Milton E. (1952). \"Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons\". Biometrika. 39 (3/4): 324–345.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "Typos and similar: \n- line 48R, the sentence seems incomplete\n- appendix A2, issues with the references and an extra ,", "questions_for_authors": "- In the problem statement $SPA_\\tau$, what is exactly Comparisons(T )? Also $\\mathbb{T}$ is undefined, I guess it is the family of tiered rankings.\n\n- What the colors in figure 2 represent? \n\n- What is the unit measure of the runtimes in Figure 4? Seconds?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AnQ9GRnpCd", "reviewer": "Reviewer_sHCm", "summary": "The paper introduces a new preference aggregation solution, called Selective Preference Aggregation (SPA). Its essential feature is to return a partial order of items based on beyond-majority principles. More precisely, for any $\\tau \\in [0, 0.5)$, SPA constructs a total order over the finest partition of the items such a derived order $i \\succ j$ cannot occur if more than a fraction $1-\\tau$ of users disagree with $i \\succ j$. In a sense, the partial order is \"safe\", i.e. it only makes ordering decision if a large, potentially overwhelming, majority agree with the decision. The paper proves desirable property of SPA and evaluates it empirically on several datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback! We appreciate your feedback and your detailed suggestions for improvement, including further datasets to improve our work. We provide tables at https://tinyurl.com/2ybsfs95\n\n\n> However, the experiment in Section 6 is less compelling … The reported prediction error should then correspond to the evaluation set.\nIn our revised setup, we use only a subset of users and items (80/20 train/test split) to create rankings for the given items. We switch to using a pre-trained model (bert-mini) as a starting point, and replace pairwise majority with the Copeland method for consistency with our experiments. We then note the total per-user error, as well as how well a model trained on these rankings generalizes to new users in Table 10.\n\n\nWe also provide a table of model generalization to new items (with train users), new users (on train items), and new users on new items. We binarize to pick the threshold with the Maximum TPR s.t FPR is capped to 10% (Table 11). We hope to spend further time exploring other pre-trained setups and architectures to ensure higher performance for test set items and users, and to demonstrate SPA’s performance advantages with other choices of threshold.\n\n\n> Also, I am not sure I fully understood how SPA was adapted to Section 6.\n\nWe made an error in our description of the adaptations, now corrected. DICES uses annotators who rate an item toxic/non-toxic, but the level of toxicity beyond that is not clarified. In our new setup, in order to avoid excessive (and incorrect) levels of “ties”, we have only included preferences where there is distinction (toxic vs non-toxic). We then scale the weights of each preference pair to the same total weight, to make each item-pair equally important.\n\n\n>How is \"distinctions among conversations\" measured?\n\n“Distinctions between conversations” is used to specify the greatest number of tiers (greatest comparability). We have added text to our manuscript to make that distinction clear. There are several tiers created - the chosen threshold determines which are grouped under toxic/non-toxic.\n\n> The definition of label error (line 420, left column) introduces a variable t … Could the authors clarify?\n\n$t$ in this instance represents the majority threshold, at which point a majority of users rate a conversation toxic. We have revised the text to make this clear as well.\n\n\n> Finally, I am confused about the use of the word \"consensus\" throughout the paper. In plain English, \"consensus\" seems synonymous with \"quasi-unanimity\", which suggests using . But this is not how it seems to be used, e.g. lines 323, right column, or line 420, left column.\n\nWe agree that the term was not well-defined and should be clarified. In this setting, we use “consensus” to refer to the true majority vote of annotators.\n\n> Additionally, one interesting feature of pol.is … especially when the communities are not predefined, and have instead to be learned)?\n\n\nThis is a fascinating direction for potential future work! We take the question to mean that we want to find cases where multiple distinct communities agree (please correct us if needed!). One could imagine applying this approach hierarchically in such a setting - finding a selective preference aggregation for each community individually, then treating each of those tiered rankings as an individual “judge” and finding a preference aggregation across communities.  If communities need to be learned, one could imagine assigning each individual a weight based on an estimated probability of belonging to each community (see our response to reviewer dGCS for further details on adding weights to our approach). SPA could also be used as part of the identification process; identifying groups of users who agree with each other in areas where overall disagreement is high (within a tier, for instance) can be used to help determine these communities.\n\n\n>  I would also be interested in the authors' thoughts on how SPA could be adapted for highly sparse comparisons, as is the case in RLHF and recommendation AIs (see e.g. [3]).\n\nIn future work, we plan to explore different ways of adapting SPA for use in larger/sparser datasets.  The most straightforward is to build on existing work using LLMs to make judgments (RLAIF). We could use multiple LLMs to make judgements to avoid amplifying model biases; one could also use lightweight models and “recruit” larger models only when preferences conflict to save on computational cost.\n\nSPA could also be used with modeling. SPA could be used to limit uncertain responses from existing models, or could be used to help create models that better model users. Comparisons within tiers could be used to train a base model for accuracy, and identified groups (see previous response) could be used to train personalized models with high accuracy for each group.\n\n\nWe hope our responses clarify your points! Thank you for your feedback and suggestions.", "claims_and_evidence": "The paper shows that:\n- The values of all SPAs for $\\tau \\in [0, 0.5)$ can be computed efficiently, using graph algorithms.\n- For $\\tau \\to 0.5$, SPA isolates an existing Condorcet winner.\n- SPA is \"safe\": under reasonable conditions, adding missing preferences cannot modify previously orders.\n- Adding a new item to SPA (with comparisons to other items) will not invert orders (though it may cancel some).\n\nWhile I have not carefully analyzed the provided proofs, I am fully convinced that the theorems hold.\n\nThe paper is also evaluated on 5 datasets, with findings consistent with the intuition, i.e. SPA makes fewer mistakes by abstaining to return a total order, though this can lead to a very coarse partition (and thus a very sparse comparison graph). The results seem reasonable.\n\nFinally, the paper shows how SPA can be used for preference learning in a machine learning context (and thus, in principle, with generalizations to non-evaluated items). I am less convinced by the value of this experimental setup (see below).", "methods_and_evaluation_criteria": "The experiments in Section 5 make sens to me, and I am satisfied with their design, their presentation and the results.\n\nHowever, the experiment in Section 6 is less compelling. In particular, the value of doing machine learning is to allow for generalization to non-evaluated items. Yet, as far as I understand, the trained models are evaluated on the training set (line 422, left column). I would suggest that they instead separate the DiCES dataset into two subsets, one for training and the other for evaluation. The reported prediction error should then correspond to the evaluation set.\n\nAlso, I am not sure I fully understood how SPA was adapted to Section 6.", "theoretical_claims": "While I have not carefully analyzed the provided proofs, I have overviewed the Appendix.\nI am fully convinced that the theorems hold, as they leverage classical graph theoretical constructions.", "experimental_designs_or_analyses": "As I said above, I believe that Section 6 would gain by separating the dataset into a training and an evaluation set, to evaluate the out-of-training-set predictions of the SPA-based trained model.", "relation_to_broader_scientific_literature": "The literature review seems satisfactory to me. \nPerhaps the authors could add references to more datasets that fit their algorithms (see suggestions below).", "essential_references_not_discussed": "The paper does seem to be missing any key reference.", "other_strengths_and_weaknesses": "I really appreciated the paper's motivation section, especially with respect to the limitations of the principle of majority and the need to go beyond this, not only for legitimacy reasons, but also because of security. This yields an originality that is of great strength to the paper.\n\nThe paper is also very well written, with an exception for Section 6 which I found confusing at times (see below).\n\nI think that, especially in the current age of irreconcilable judgments on AIs' preferred behaviors, especially in the context of social media moderation and content amplification, the paper is of very high significance.", "other_comments_or_suggestions": "I did not understand how SPA was precisely defined in Section 6 (line 383, right column). The paper writes \"SPA for the largest value of dissent, which leads to the clearest distinctions among conversations\". How is \"distinctions among conversations\" measured? Moreover SPA creates a partition. Are the authors guaranteeing a partition into two subsets (toxic and non-toxic)? What if there are more subsets? I would appreciate clarifications on this definition. \n\nAdditionally (and perhaps relatedly), the definition of label error (line 420, left column) introduces a variable $t$ which is never defined (except using the word \"consensus\" which I do not seem to understand, see below), and whose value in the experiments does not seem to be given. Could the authors clarify?\n\nLess importantly, I would urge the authors to look into the data of https://pol.is [1]. There has been recent efforts to make such data openly available, and they really match the DiCES dataset structure. But there are many more such data. Most importantly, I believe that SPA would be a great addition to the pol.is website.\n\nThe authors could also be interested to look into the Tournesol dataset [2], though the comparisons in this dataset are a lot more sparse, which raises additional challenges.\n\nFinally, I am confused about the use of the word \"consensus\" throughout the paper. In plain English, \"consensus\" seems synonymous with \"quasi-unanimity\", which suggests using $\\tau \\to 0$. But this is not how it seems to be used, e.g. lines 323, right column, or line 420, left column. \n\n[1] Polis: Scaling Deliberation by Mapping High-Dimensional Opinion Spaces. Christopher Small, Michael Bjorkegren, Timo Erkkilä, Lynette Shaw, Colin Megill (2021). Departament de Filosofia, Sociologia i Comunicació Audiovisual i Publicitat.\n\n[2] The Tournesol dataset: Which videos should be more largely recommended? Lê-Nguyên Hoang, Romain Beylerian, Julien Fageot, Louis Faucon, Aidan Jungo, Adrien Matissart, Nathaël Noguès (2024). https://openreview.net/forum?id=5WFzk0H27p", "questions_for_authors": "Could the authors more rigorously define the adaptation of SPA in Section 6?\n\nI would also be interested in the authors' thoughts on how SPA could be adapted for highly sparse comparisons, as is the case in RLHF and recommendation AIs (see e.g. [3]).\n\nAdditionally, one interesting feature of pol.is is to leverage community detection to find agreement across (unbalanced) communities. How could SPA be adapted to such settings (especially when the communities are not predefined, and have instead to be learned)?\n\n(note that combining the two problems, namely sparsity and cross-community agreements, makes the problem even more challenging, as the sparsity may be adversarial, i.e. some community overrates some items rather than others)\n\n[3] Plackett-luce regression mixture model for heterogeneous rankings. M Tkachenko, HW Lauw (2016). https://dl.acm.org/doi/abs/10.1145/2983323.2983763", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "srcTSdiuCV", "reviewer": "Reviewer_mXsg", "summary": "This paper proposes aggregating ordinal preferences by producing selective rankings. The proposed selective aggregation framework explicitly reveals and controls dissent. The authors develop efficient graph-based algorithms (Algorithm 1 and Algorithm 2) with theoretical guarantees on correctness, uniqueness, and runtime. Experiments across diverse real‐world datasets demonstrate that selective rankings are more robust, transparent, and fair compared to standard voting rules and ranking algorithms.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Response\nThank you for your feedback! We address them and include tables at https://tinyurl.com/2ybsfs95\n\n\n> Is the focus on the algorithmic framework for selective aggregation, or on its application to a specific problem domain?\n\nThe primary contribution is the proposed algorithmic framework. We will revise the text to state this. The explored problem domains serve as illustrations of the framework's capabilities in different contexts.\n\n\n> The approach of simply abstaining from comparisons does not address situations where a complete ranking is essential …\n\nWe appreciate this concern. Many relevant real-world tasks (e.g. RLHF, content moderation, search ranking) do not require total orders. When necessary, the dissent parameter can be raised to 0.5, and further until graph disconnect (with loss of guarantees). If there is still no total order, there is no majority preference between certain items or a cycle is formed.\nIf a total ordering is truly necessary, other methods can be applied within tiers to produce local complete orderings. This process distinguishes comparisons supported by the algorithm's guarantees (between tiers) from those less well-founded. Online, the approach highlights where more information could resolve disagreement. SPA serves as a robust first step: identify consensus comparisons first, then apply targeted methods (experts, more users) where disagreement persists. We’re happy to add this to the text.\n\n> The paper glosses over the trade-offs involved in choosing the dissent parameter.\n\nWe have several strong points to consolidate a new subsection in section 3 (Algorithms), which are already in the paper:\nThe path algorithm (Appendix) allows us to avoid preselecting a tau value. Importantly, computing all possible rankings for any $\\tau$ < 0.5 incurs no additional asymptotic cost.\n\n\nTheoretical guarantees can guide its selection. The discussion starting at line 281 gives an example of how we might select tau based on assumptions about noise or missing preferences.\n\n\n> There is little discussion of alternative robust ranking methods, and key references (e.g., for baselines like ORPO or CPO) are not adequately introduced when they first appear.\n\nWe're happy to add discussion of ORPO/CPO/other methods. ORPO focuses on alignment during the training process. SPA is less constrained - it may be possible to use it in tandem; SPA could filter for high-quality responses with ORPO used for fine-tuning.\n\nCPO more closely resembles SPA, but its purpose is to contrast “near-perfect but flawed translations”. While comparing top tiers in SPA could have a similar purpose, our method captures preferences across all items. As a result, we feel rank aggregation methods are more appropriate baselines.\n\n\n> The selective rankings are claimed to be more robust and transparent … yet the overall performance differences are marginal and not convincingly argued.\n\nWe appreciate this and will highlight SPA's benefits. SPA provides practical benefits through transparency and robustness. It reveals underlying consensus strength (e.g., Sushi's weak majority via path, App D.3). Existing methods obscure this. As noted, SPA offers stronger robustness guarantees (see dGCS). SPA exhibits no inversions while others vary significantly - this stability is crucial in domains like RLHF, preventing reward signals from flipping due to small sample changes. SPA consistently yields lower disagreement rates (Table 1) compared to baselines (often 0-6% for SPA vs. 4-12%+ for baselines).\n\n\n> The overall novelty is limited; the idea of abstaining from forced arbitration is not revolutionary and may offer little benefit in many practical applications. \n> The main contribution, the idea of using selective rankings to “agree to disagree” is not clearly motivated ….\n\nBeyond abstaining, SPA reveals preference strength, crucial for trustworthy AI. For instance, methodologies aiming to learn from demonstrations [Brown et al. 2019] or generative reward models [Mahan et al. 2024] use steps with simple majority rules.  Our rankings are always supported by a majority of users, clarify where preferences conflict, remain stable under noise, and can be adjusted. Our response to sHCm further details future use cases in ML workflows like RLAIF and personalized models.\n\n* Mahan, et al. \"Generative reward models.\" arXiv preprint arXiv:2410.12832 (2024).\n* Brown, et al. \"Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations.\" ICML. PMLR, 2019.\n\n> More extensive and realistic experiments are needed ….\n\nWe appreciate the feedback. Table 8 (in our response) shows performance in scenarios with and without consistent user preferences. Only SPA reveals user contradictions. See also our responses to dGCS (ablation) and fgRQ (dropped comparisons).\n\nWe hope that our response resolves any misunderstandings! Thank you again for your time, and we look forward to resolving any remaining concerns that you may have.", "claims_and_evidence": "- The paper asserts that standard aggregation methods hide disagreement by forcing a total order, while selective aggregation naturally reveals dissent. Although the paper provides theoretical definitions and proofs, the empirical evaluation is questionable. The experiments appear contrived and lack a convincing demonstration that selective rankings offer a meaningful advantage in real-world scenarios. Moreover, the claim of “extensive” evaluation is overstated given the limited scope and sometimes arbitrary selection of datasets.\n- The algorithms (Algorithm 1 and Algorithm 2) are presented as efficient and unique (optimal) solutions for the selective aggregation problem.\n- The selective rankings are claimed to be more robust and transparent compared to traditional methods (Borda, Copeland, Kemeny, MC4). The robustness metrics show some promise, yet the overall performance differences are marginal and not convincingly argued to translate into practical benefits. Furthermore, the paper glosses over the trade-offs involved in choosing the dissent parameter $\\tau$.", "methods_and_evaluation_criteria": "- The main contribution, the idea of using selective rankings to “agree to disagree” is not clearly motivated by practical needs. In many applications, a total order is still necessary, and the paper does not adequately address how its partial order can be converted or used in those contexts.\n- The algorithms are described in mathematical detail.\n- The evaluation is carried out on several datasets converted into pairwise preferences.", "theoretical_claims": "The paper’s theoretical results (e.g., Theorems on uniqueness and robustness) are mathematically rigorous under ideal assumptions.", "experimental_designs_or_analyses": "- The experiments do not convincingly show that selective aggregation improves decision quality; instead, they often merely illustrate that the method can “abstain” from making comparisons—a trivial consequence of the design.\n- The paper fails to explore how sensitive the results are to the choice of $\\tau$.", "relation_to_broader_scientific_literature": "- The paper is grounded in classic social choice theory and traditional ranking algorithms.\n- The paper also relates to contemporary research in machine learning where transparency and robustness of aggregated annotations (e.g., in RLHF) are increasingly important.\n- There is little discussion of alternative robust ranking methods, and key references (e.g., for baselines like ORPO or CPO) are not adequately introduced when they first appear.\n- The approach of simply abstaining from comparisons does not address situations where a complete ranking is essential, and the paper lacks discussion on how its method compares to other state-of-the-art aggregation techniques under realistic conditions.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "### Strengths\n- The paper is mathematically detailed, providing rigorous proofs and a comprehensive supplementary section.\n- It introduces the idea of selective aggregation, which is novel in its explicit treatment of dissent.\n\n### Weaknesses\n- The overall novelty is limited; the idea of abstaining from forced arbitration is not revolutionary and may offer little benefit in many practical applications.\n- The method’s sensitivity to the dissent parameter $\\tau$ is underexplored, and the paper does not offer practical guidance on setting this parameter.\n- Presentation issues (such as unclear figures and poor integration of citations) further detract from the work.", "other_comments_or_suggestions": "- The paper would benefit from a clearer statement of its primary contribution. Is the focus on the algorithmic framework for selective aggregation, or on its application to a specific problem domain?\n- More extensive and realistic experiments are needed to justify claims of improved transparency and robustness.\n- A discussion on how to convert a selective (partial) order into a total order when required would strengthen the practical relevance of the approach.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews. The discussion lead to the feeling that the issue is whether one finds the paper well-motivated/significant or not. As even the negative reviewers wrote that the paper is methematically detailed, I am willing to believe that if the paper raises controversies, it might actually be a valuable addition to the confernece, that may be setting some paths for future research.
Decision: Accept (poster)