Designing Rules to Pick a Rule: Aggregation by Consistency
Authors: Ratip Emin Berker
Ben Armstrong
Vincent Conitzer
Nihar B Shah
Venue: ICLR 2026 Poster (2026)
Keywords: rank aggregation
rule picking rules
consistency
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We give a novel framework for picking aggregation rules most appropriate for a given dataset, and design our own consistency-based rule picking rule.

Abstract: Rank aggregation has critical applications for developing AI agents, as well as for evaluating them. However, different methods can give rise to significantly different aggregate rankings, impacting these applications. Indeed, work in social choice and statistics has produced many rank aggregation methods, each with its desirable properties, but also with its limitations. Given this trade-off, how do we decide which aggregation rule to use, _i.e._, what is a good _rule picking rule (RPR)_? In this paper, we design a data-driven RPR that identifies the best method for each dataset without assuming a generative model. The principle behind our RPR is to maximize consistency if the data collection process was repeated. We show that our method satisfies several consistency-related axioms failed by a wide class of natural RPRs. While we prove that the computational problem of maximizing consistency is hard, we provide a sampling-based implementation that is efficient in practice. We run this implementation on known statistical models to experimentally demonstrate its desirable properties, as well as on real-world data where our method provides important insights into how to improve consistency.

Published as a conference paper at ICLR 2026
DESIGNING RULES TO PICK A RULE: AGGREGATION BY CONSISTENCY
Ratip Emin Berker', Ben Armstrong", Vincent Conitzer & Nihar B. Shah CCarnegie Mellon University, Tulane University, SUniversity of Oxford (rberker, conitzer, nihars}8cs cmu. edu, researchebennmmttonn...
ABSTRACT
Rank aggregation has critical applications for developing AI agents, as well as for evaluating them. However, different methods can give rise significantly dif- ferent aggregate rankings, impacting these applications. Indeed, work in social choice and statisties has produced many rank aggregation methods, each with its desirable properties, but also with its limitations. Given this trade-off, how do we decide which aggregation rule to use, ie., what is good rule picking rule (RPR)? In this paper, design data-driven RPR that identifies the best method for each dataset without assuming a generative model. The principle behind our RPR is maximize consistency if the data collection process repeated. We show that our method satisfies several consistency related axioms failed by a wide class of natural RPRs. While prove that the computational problem of maximizing consistency is hard, we provide a sampling-basee in practice. We run this implementation on known statistical models experi- mentally demonstrate its desirable properties, as well as on real-world data where our method provides important insights into how to improve consistency.
implementation that is efficient
INTRODUCTION
Suppose you have a collection of items, and evaluators who individually rank them. Finding the best method for aggregating such data an age-old problem, with critical use cases in artificial intelligence and machine learning. For example, the recent surge of large language models has been driven in part by reinforcement learning from human feedback (RLHF) (Christiano et al. 2017 Ziegler et al.) 2020), where evaluators provide ordinal preference data over model outputs, which must then be aggregated into a reward model. In other cases, human evaluators can be replaced with high-level principles, each of which ranks outputs based how compliant they are with that principle, as is done in constitutional A/ (Bai et al. 2022). Further, rank aggregation can be useful not just for developing capable Al models, but also for evaluating their performance. For example, interpreting incomparable benchmarks as separate evaluators, Lanctot et al. (2025a) aggregate these rankings compare the overall performance of competing models. Last but not least, peer review is an essential tool for evaluating ML research, where asking reviewers to rank their assignments later to be aggregated for final decisions can help mitigate several shortcomings of other evaluation formats; cf., for example, an experiment by Liu et al. (2022) on ICLR 2017 conference data.
In each of these use cases, different methods can lead to vastly different aggregations, thereby af- fecting the outcome. Failure to pick "good" methods can result in outcomes inconsistent with the rankings of the evaluators or benchmarks, as has been observed in RLHF (Ge et al. 2024 Xu al.) 2024) and agent evaluation (Lanctot et al. 2025b). But what makes aggregation method good?
One common approach to this question is the axiomatic approach from the social choice litera- ture (Ploti, 1976): first select certain criteria (axioms) that the aggregation should satisfy, then de- sign rules that meet these axioms. However, celebrated impossibility results (Arrow 1963 Gibbard 1973 Satterthwaite 1975) prove some fundamental axioms are incompatible, eliminating all hope for one "ideal" 1 rule fulfilling every desideratum. On the other hand, even if the chosen axioms are satisfiable once, there may be many rules that do so, making the selection among them arbitrary.
Another approach for picking the aggregation method, which we refer to as the statistical approach, treat the rankings noisy estimates of objective ground truth. By assuming noise model

---

Published as a conference paper at ICLR 2026
(e.g.. Plackett 19755) Luce 1959) or Mallows (957))
(e.g.,
for the data generation, the aggregate ranking can be chosen as the one maximizing the likelihood of the data under this model. A key challenge is the accuracy of the assumed model, which is commonly addressed by considering multiple models and choosing one via cross validation (Zucchini 2000). However, irrespective of model selection, assuming ground truth may he fundamentally flawed, especially in settings with legitimate differ- ences of opinion, such as AI alignment (Ge et al. 2024). Further, many natural voting rules with desirable properties are not the maximum likelihood estimator (MLE) for any noise model (Conitzer & Sandholm 2005 Conitzer et al. 2009) and are thereby precluded by the statistical approach.
Given this vast array of (partly incompatible) tools from statistics and social choice, and no clear way priori of selecting from them, a natural question emerges: In a given setting, how do we pick which rule to use? other words, what makes a good rule picking rule (RPRj?
Our work addresses this question. Unlike previous literature that largely focuses on picking an aggregate ranking. we want to explicitly pick a rule. Such an approach has several benefits: First, employing an RPR naturally leads to better interpretability by providing a formal justification of why other rules (under which the winners could be different) were not adopted. Second, different (perfectly reasonable) rules may be appropriate for different settings with different requirements, and RPRs offer a principled way of deciding which rule is the most appropriate for given setting. Lastly, as we will see, our novel framework allows designing natural RPRs that can choose from any set of rules, making it easy to continually incorporate novel rules into the aggregation process.
Our contributions picking rules (RPR) (Section()) Our framework allows designing principled ways of adopting a rule appropriate for the data, without committing
as follows: (1) We introduce novel framework for formally defining rule
set of axioms or a generative model priori.
(2) Inspired by prior work emphasizing the link between consistency and quality in related settings, introduce our own RPR, Aggregation by Consistency ( (AbC), with the explicit goal of maximizing the consistency in the output if the data collection process was repeated (Section 4p.
(3) We define several natural axioms for RPRs, and prove AbC' satisfies them, including those failed by a wide class of RPRs. For two axioms that AbC fails, we prove impossibility results (Section 5
(4) We prove that the computational problem of checking if "complete" consistency can be achieved for given input (i.e., picking a rule that produces the exact same output) is Nomcleee (Section 6
Nevertheless, we provide an implementation of AbC that is efficient in practice and performs well in experiments on known distributions (Section 7). Our implementation includes a learning component that optimizes consistency over infinitely many positional scoring rules. We show that AbC' can be applied to both score- and lank-based evaluations across large variety of empirical settings, at times improving significantly upon the consistency given by rules used in practice.
Omitted proofs are in Appendix B An implementation of AbC' was awarded as one of the four win- ners in the 2nd Computational Social Choice Competition at the 33rd International Joint Conference on Artificial Intelligence, which scored rules via hidden welfare function, demonstrating AbC' per- forms well in general settings. Taken together, our work lays a robust theoretical and computational foundation for principled rule picking, paving the way for future work in this novel framework.
2 OVERVIEW OF THE PROPOSED APPROACH: WHY CONSISTENCY?
Before introducing our formal framework, we discuss the basic idea of our approach. Suppose we have two sets of evaluators (e.g, benchmarks or human voters) who independently rank the same items (e.g., AI models/outputs or academic papers). A popular measure of the "quality" of an aggregation method is the consistency between its output on two independent sets of rankings. As such, we seek to design an RPR that picks the rule that maximizes this consistency[] This intuition is inspired by a number of related settings where past work emphasizes the importance of consistency:
(1) Peer review is classical setting in which evaluations need to be aggregated, and much work has been done to optimize this process; cf. Shah (2022) for an overview. Many experiments split review- ers into two panels evaluating the same data (Jecmen et al. 2022), where interpanel disagreement
We will restrict our RPR to rules that satisfy basic axiomatic properties such neutrality (all items being ranked treated equally), which avoids the pathological case of constant function with maximal consistency.
2

---

Published as a conference paper at ICLR 2026
is interpreted shortcoming (Obrecht et al. 2007 Fogelholm et al. 2012 Pier et al. 2017 Bast 2020). For example, in the NeurlPS 2014 and 2021 conferences, the two panels disagreed on over
half of accepted papers, taken sign of arbitrariness in the review process (Lawrence & Cortes)
2014 Cortes & Lawrence] 2021 Beygelzimer et al., 2023). Interpanel consistency is also used to
compare distributed peer review with expert panel (Patat et al., 2019; Kerzendorf' et al.) 22220)
Overall, it is clear that the peer review community considers consistency an indicator for quality.
(2) Clustering: One can view the task of rule picking as learning rule from rankings. Unlike earlier work with a ground truth rule (Procaccia et al., 2009), our setting is wnsupervised and thus closely related to clustering. Indeed, Ailon et al. (2005) show clustering and rank aggregation can be approached with near-identical algorithms.. Similar to rule picking, an important challenge in clus- tering is model selection (e.g., number of clusters). Prior work shows picking the model maximizing stability- ie.. obtains similar results on several datasets from an underlying model-yields higher aceuracy (von Luxburg 2010). Again, consistency and quality of the output are closely related.
(3) iamimum-vvriinne unbiased estimator (MVUE): Among all unbiased estimators (expected value equals the true value of the parameter being estimated), the one with the smallest mean squared error has the lowest variance. Indeed, MVUEs are commonly studied in statistics (Chapman &
Robbins 1951 Rao 1949) and related bounds can be used for rank aggregation (Hajek et al.) [014]] Khetan & Oh 2016 Consider applying unbiased estimator to two i..d. datasets. As variable's variance equals half the expected squared difference between its two i.i.d. copies, the estimator with the smallest expected difference between the two datasets is again the one with the min. variance, i.e., the MVUE. This motivates the most consistent rule in our setting, where unbiasedness is interpreted as basic constraints for any acceptable rule, which we achieve by restricting our RPR pick from neutral & anonymous rules (those that treat all items and evaluators the same, respectively).
There is nascent interest in applying tools from social choice to Al alignment processes, such as reinforcement learning from human feedback (RLHF) (Conitzer et al. 2024 Dai & Fleisig 2024; Mishra 2023). with particular emphasis on consistency. Much like peer review RLHF inevitably relies on a limited set of evaluators, despite aiming for broad societal alignment. As such, aggregation methods that are robust to repetitions of the process can decrease the arbitrariness of the final AI model due to the choice of evaluators. As noted by Conitzer et al. (2024), the focus of social choice on producing consistent aggregations makes it an appropriate tool for this setting.
(4) AI Alignment:
While these four settings (peer review, clustering, MVUEs, and RLHF) motivate rule picking rule that maximizes the consistency between two independent evaluations, it is not clear how one would implement that. After all, in practice, we often have only one copy of the process. To explain how we circumvent this, Algorithm introduues our RPR, named Aggregation by Consistency (AbC).
Algorithm 1:
Aggregation by Consistency (AbC),, informal (see Section for formal definition).
Input: A set of evaluations over items, a set of acceptable ("candidate") "" rules
Output: A chosen rule, to be used for aggregating the evaluations
1. Split the evaluators uniformly at random into two groups, considering each group as copy of the process (in line with the peer review experiments above);
2. For each candidate rule, compute the rule's outputs separately on the two groups, and measure the disagreement between these two outputs; return the rule with min. disagreement
Importantly, Algorithm I is agnostic to the types of input/output of the rules it is picking among; only requires measure of disagreement for their outputs (Step 2). is therefore widely applicable in settings with various evaluation formats-es rankings, ratings/scores, approval sets sets-and desired outputs-e.g.. an aggregate ranking, a single winner, reward function. In this paper, we focus on rules that output an aggregate ranking, providing a concrete analysis and experiments.
As we will show, AbC has many advantages. First, it satisfies important axioms for RPRs, even those failed by many natural RPRs. Second, ADC does not assume a ground truth; however, a generative model indeed approximates the data, as we show experimentally, its MLE yields high consistency across random splits. ABC' then chooses this MLE, obtaining the benefits of the statistical approach. Third, many social choice axioms can be imposed on AbC by restricting the candidate rules to those satisfying them. obtaining the benefits of the axiomatic approach. Lastly, any AbC implementation is easy to continually extend by implementing new rules and adding them to candidate rules.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES AND RULE PICKING RULES
We consider a set of voters . = [r]] and a set of alternatives A with |A|| m. For our axiomatic analysis (Section (5), it will be helpful to assume each voter ranks all alternatives, although our method easily extends to other evaluations formats, including partial rankings (Section 4 and ratings (Section 7. A weak ranking is a complete and transitive binary relationship on A. A strict ranking is weak ranking that is also asymmetric. Intuitively, a weak ranking is strict ranking where ties are allowed. We denote the set of all strict (resp. weak) rankings of A by ((A) (resp. R(A)). For a (strict/weak) ranking we write >- b if ranks strictly above b, and b Zn otherwise. Each voter i E N has a strict ranking o; E C(A). A profile o E ((A)" contains the rankings of all voters.
Candidate rules A social welfare function (SWF) is a mapping f that, given profile outputs a single weak ranking To aggregate voters' rankings, we are interested in picking an SWF to use from set of acceptable "candidate" rules (e.g., those that are the MLE of noise model, or voting rules that satisfy certain axioms). From a machine-learning perspective, the candidate rules can be viewed as hypothesis class. Importantly, our framework does not place any restrictions the candidate rule set. We next introduce one such class of SWFs we will sometimes pay attention to.
Definition 1. A (monotonic) pasitional scoring rule is an SWF f. associated to a vector s = (84) with say Mo (a, il is the number of voters ranking a in the 7th position, and t Σi s,Mo [a, a]. Then, f, ranks a Sa(0) b iff 1 [a] 1a [b]. We denote the set of all positional scoring rules by F's.
$1 2 ... 2 5m O. Given profile o, for each alternative a E A and E [m],
Positional scoring rules are easy to represent/compute, and include well-known SWFs, e.g., plurality (fp. for p (1, (1,0.....0)) 0)), veto (fu, for (1...., 1, 0)), and Borda count (fu. for b; =={)
Rule Picking Rules We now introduce novel framework for picking rules.
Definition 2. A rule picking rule (RPR) is function Z that given set of SWFs F (called candidate rules) and a profile a, outputs subset of rules ((F, .)) S F.
Just many natural SWFs (e.g., positional scoring rules) can lead to ties among alternatives, will see, many natural RPRs may tie some SWFs for certain profiles, leading to (F, a- > 1. In such cases, tie-breaking order over F can be used pick the single rule be adopted.
RPRs offer a principled way to pick an SWF for aggregating the rankings in a.. Ideally, an RPR will capture the idea that different (perfectly reasonable) SWFs may be appropriate for different profiles, e.g., because the profile resembles specific distribution, or certain positions in the rankings are more informative than others. Section 5 formalizes these cases by defining natural axioms for RPRs.
AGGREGATION BY CONSISTENCY (AbC)
Having introduced our framework of RPRs, we now formally present our method. As discussed in Section [ we want our RPR to maximize the consistency between two independent sets of rankings. Our approach can be implemented with any measure of distance comparing the outputs of a rule. For concreteness in our analysis and experiments, we turn to one such well-known measure.
Definition 3. For weak rankings r2, E R(AA) and alternatives a, b E A. let & be indicator variable that i and r2 strictly disagree about how and b should be ordered. Similarly, let To.. in- dicate that a and b are tied by at least one of ri or r2. More formally, De.b ri.r2 = I[(a >r, b and b >ry a) or (b >r, and a >ry 6)] and r1:72 = I[(a Zry b and b Zr. a) or (a Zr, b and 6 Ery a)]. The Kendall tau distance with ties between r, & 72 is (T(),,)2) E(o.D)EAR (Da + 1T
KT is more traditionally defined for strict rankings, without the Fi.r term. We explicitly add the term for ties in order to penalize indecisiveness (e.g., a rule that always returns all alternatives tied). Weighing ties by 2 is inspired by Kendall's Tau-b correlation coefficient (Kendall 1945). To study the impact of ties, we rerun some of our experiments with different weights for lies in Appendix C.8
We are now ready to introduce our consistency based RPR, which we formally define in Box [
OOtputting weak rankings allows returning a single ranking without violating neutrality or anonymity.
4

---

Published as a conference paper at ICLR 2026
Box I: Aggregation by Consistency (AbC)
Given a profile a, consider the following random process: Initialize two sets N1 = N₂- = 0. For each voter i N, pick j E {1,2] uniformly at random and Nj + .N (i). Let σ(1) HeN, (i.e., the restriction of profile a to voters in N,) for each j E [1.2]. Then, given a set of candidate rules F, Agreement by Consistency (AbC) is an RPR defined as
AFC o = arg min E KT (f(o")),((())
fEF
where the expectation is over splitting o into (11) and (2) by process described above.
In words, AbC' returns the SWFs among that minimize, expectation 3 the disagreement when applied separately to two sides of a random split of a. The next example illustrates this.
Example 4. Fix some integer k 2 2 and consider the profile a with alternatives A = {a, c} and n = 3k voters. The voters comprise three groups of size k each, with the following preferences: Group 2 ranks a C b.
Group 3 ranks b > c > a. Consider the set of candidate rules F = {fps for (i.e., plurality and veto). Say (!!) and ( are the random variables resulting from splitting a via the random process Box First, consider fp: say r = =fp(o(1) and r = (f(ar())). Since alternative is not ranked top by any voter, both rp and r will rank a > c (resp. b > c) unless all of the 2k (resp. k) voters ranking a (resp. b) as their top alternative end up on the same side of split, in which case we gel tie. Thus,
Group I ranks a > >cc C.
E
D
E
0:
E
22k-1 ; E
Thic p
1
p
2
Using the linearity of expectation, this implies the expectation in for
for fo is E
KT (r) r
E Da.b th + TTo. 2 + Dare. +Tu. /2- Dbe + Tb.. 311 //.rg// / 2
/2,13/2+
rhir x2.0//2
2 1.32.
(2)
Now, consider fo: say r1 = (2()(1)) and r3 fo(o(2)). Since the number of voters that rank and b bottom are tied, To and re will either disagree about a and b, have them tied. Thus, we cannot have Da T =0 The same is true for any other pair of alternatives. This implies
E. [KT (r.. 1.] = E. Dab
r
+
Tm.. r/.r2/2 22 Da.
r + r2 +
bio + TThe ri.
r2/2//2 2 1.5. (3)
Comparing (2) and (3). we see that AbC(F, o) = (fr.. Choosing plurality over veto indeed makes sense for o,, as the former clearly gives more information on how alternatives compare each other.
We now discuss three extensions of ABC, which we later implement (Section 7 As computing the expected disagreement in (I) for each SWF can be difficult for more complicated profiles and rules than in Example 4 in practice AbC' can be approximated for finite [L'l via Monte Carlo sampling, i.e., by splitting voters via the random process in Box 1 computing (((((((())) [( (()) for each ff E F, and repeating for desired number of splits, eventually returning the SWF with the minimum average disagreement. Even for infinite |F). appimizaatio//eaamingg algorithms can be used to find the minimizer of (T). For example, if F = Fs (Def. []. one can run constrained optimization with the scoring vector (8x (e))) as variables. Indeed, our ABC implementation in Section 7 Juses Monte Carlo sampling and includes (among other rules) an optimization over all positional scoring rules.
more general setting is that of partial rankings, where each voter i E N ranks a subset A, C A. This setup is more appropriate for certain settings where we would like to utilize AbC, such as peer review and RLHF. To extend ADC to partial rankings, we must take into account that each alternative is no longer ranked by every voter. Thus, some alternatives may have more evaluations one side of a random split, while others' evaluations will be more evenly split. Idcally, we should penalize disagreements over the former alternatives less harshly, information about them. Indeed, in the extreme case where all evaluations of an alternative are on the same side, is unreasonable to expect any SWF to rank this alternative consistently across the split. Thus in Section 7 we replace KT in i with the weighted KT function (Kumar & Vassilvitskii 2010 setting the weight of an alternative according to how evenly it is represented across a split.
side of the split may lack sufficient
In Box 1 there is 2 (n-1) probability that N, 0 for some i E {1, [1,2]. In this case, we take (()( to he the empty ranking (all alternatives tied) for all f E F. This effectively gives zero weight all such splits.
5

---

Published as a conference paper at ICLR 2026
5 AXIOMATIC ANALYSIS OF AbC
We next investigate the behavior and properties of AbC by introducing natural axioms for RPRs.
5.1 CONSISTENCY AXIOMS
As RPRs intend to identify rules that are appropriate to a given profile, it is natural to seek RPRs that behave consistently with changes to the profile. Below, we define axioms that formalize this.
Reversal symmetry: One basic change to a profile is to flip every voter's ranking. In this case, we would like the SWTs picked by RPR to flip too. Formally, for positional scoring rule fa E Fs. say 81), e.g. the reverse of plurality is veto, whereas Borda count is its own reverse. For F C fg. we write rev(F)= {rev(f):
rev(fa) is the scoring rule Jar associated with s' = (1 8m:
E F). For any ranking r E ((A) U A)). say rev(r) is the reversed ranking (e.g., a > b > becomes > b > a). Last, say rev (rev(a;))ien for any profile We now define reversal symmetry, inspired by the homonymous property for rank aggregation methods by |Saari (1994).
Definition An RPR Z satisfies reversal symmetry if for any subset of positional scoring rules F C F's such that rev(P) FF and for all profiles o,, we have rev o)) Z(F,
To see why this is a natural property, suppose the "signal" in the votes is concentrated at the top: for each voter, only the choice of the top alternative is statistically informative. In that case, a reasonable RPR should choose plurality. But if we flip all the votes, the signal is concentrated at the bottom, and reasonable RPR should choose veto. As shown in Theorem Saaremm[eblll below, AbC satisfies this axiom.
Shuffling consistency: As with the example above, in some profiles certain parts of the rankings may be more informative than others. For example, we uniformly "shuffle" an interval of positions in every voter's ' ranking, exactly where in this interval an alternative ends up reveals no information on how compares to other alternatives in the interval. Thus, for such a "shuffled profile," we would want our RPR to pick rules that treat these positions equivalently. We now formally define shuffling.
Definition 6. Given a profile a and a subset S S {1, 2, m), the k-shuffling of a with respect to (denoted x² (ar ((rrr))) S)) is a profile obtained as follows: For each voter E N. (1) Create k t-m! copies of voter i's ranking o; and separate them into |S|! groups of equal size; (2) Assign each group to a unique permutation of S. Modify the votes in each group so that the candidates in the positions in S are permuted according to the assigned permutation; (3) Add all copies to the final profile.
For instance, if a consists of only a > b > c, then (a, (1, 23) has six rankings, three a > b > and three b
c. Our next axiom looks at the extreme case of shuffling all but single position.
Definition 7. Say (i.e., fo E F)) and a profile a such that fo((r) has no ties. Then, an RPR Z satisfies plurality-shufftin consistency (PSC) if for any such F and a,, there is k E Z_ so that Z(F, n (a,, [2, ml)) = {fp} for all k' 2 k, i.e., Z picks only fn for the shuffling with respect to [2, m] = (,,
given finite set of positional scoring rules F C Fs containing plurality
Intuitively, for a (sufficiently large) profile where the top positions of the votes gives an unambiguous ranking, but the remaining m - positions are effectively indistinguishable, an RPR satisfying PSC identifies the rule that treats these m positions equally (plurality) as the only appropriate rule. Such profile can approximate settings where the voters exclusively know/state their top alternative, e.g. in certain types of RLHF queries (Xu al., 2024). PSC natural property that respects the symmetry in the profile and identifies where the "signal" in the votes is concentrated. Despite this, a large class of RPRs, which we introduce next, all fail PSC, as we will show in Theorem [
Definition 8. An RPR is welfare-maximiziin Z(F,o) = arg max fer u(a, f()) for all o and F. where we write u(a,r) r) = >i= u(a..r).
there exists u L(A) R(A) + such that
Welfaare-maximizigg RPRs capture a common approach: interpreting the optimal voting rule as one that maximizes social welfare with respect to some utility function (Caragiannis & Procaceia 2011 Gershkov et al. 2017). As Definition & puts no restrictions on the function u, lefifar-maximizing RPRs constitute wide class. Nevertheless,
welfare-maximizzig will show all welfare- maximizing RPRs fail PSC.
On the other hand, this is not the case for AbC. To see this, take a random split of the shuffled profile x (ar, (2, ml). Given the balanced nature of all positions except the first, any rule that treats dis-

---

Published as a conference paper at ICLR 2026
tinction between alternatives in these positions as signal will inevitably observe the reversed signal on the other side of the split, leading to a higher disagreement. We formalize this in Theorem I
Union consistency: Next, we study the behavior of RPRs when we combine sets of voters, inspired by an analogous axiom for voting rules, simply named consistency (Young 1975). Informally, it dictates that whenever an alternative is the winner of voting rule applied to the rankings of two distinct sets of voters, the same alternative should still win when we bring those sets of voters together. We now define our version of the axiom, named union consistency, specifically for RPRs.
Definition 9. For two profiles aa and a over the same alternatives but with two disjoint sets of voters N₂ and No, say da + Th is the profile with all voters N₂ L No. An RPR Z satisfies union consistency (UC) ( o.)) n (F, σb) o 0 implies ( oa + ab) = Z o.) n o.).
Unlike previous axioms, AbC does not satisfy UC, but (as we show below) neither does any RPR satisfying reversal symmetry and PSC, giving our first impossibility result for RPRs. Further, we argue that union consistency is less significant for an RPR than its counterpart for voting rules: just because a rule is appropriate for two different sets of rankings does not necessarily mean it is appropriate for their union, especially if they are differently distributed. Indeed, in the extreme case of single voter, any sensible aggregation rule (i.e. that returns the ranking itself) is appropriate. We now present Theorem | which states our results regarding the axioms in this subsection.
Theorem 1. (I) AbC satisfies reversal symmetry. (II) Any relfare-maximizing RPR fails PSC; AbC satisfies it. (III) No (anonymous) RPR can satisfy all three of reversal symmetry. PSC, and UC.
Proof sketch. The proof can be found in Appendix B. (l) follows from the symmetries of the KT function. (II) is proven by showing plurality cannot achieve better welfare than any other positional scoring rule in shuffled profile. For (III), we show that the difference between the total scores assigned two alternatives by any fixed rule on the shuffled profile the weighted sum of a several (shifted) binomials. A careful application of tight lower and upper tail bounds for binomials (Ash 1990) then shows that the disagreement of plurality strictly lower than that of any other rule.
5.2 PRESERVED AXIOMS
While using an RPR has advantages beyond just outputting an aggregate ranking (see Section it is true that an RPR Z, when paired with set of candidate rules F, induces an SWF: FE(r)
det
for where ( or) = =f)). In words. f3 is the SWF that first maps a profile to an SWF using 2, and then applies that SWF to the profile. For 1& to be well defined, we must have IZ(F,
1.
As we are interested in the properties of / for this subsection alone, we restrict our setting to for these profiles.
profiles for which IABC(F, aه = 1, and to RPRs Z for which (F)
Clearly, the axiomatic properties of f3 will depend on our choice of the candidate rules F. By selecting F that all satisfy a certain property, can we ensure that so will fE. More formally, we say an RPR Z preserves a property P if whenever example, recall that an SWF is neutral (resp. anonymous) if permuting the alternatives A (resp. the voters N) in a profile results in the output ranking being permuted in the same way (resp. the output ranking not changing). It is immediate from Box T that AbC preserves these two properties. On the other hand, this is not true for all RPRs, e.g., for any a E A, say Zμ maps each o to the SWF f E F that ranks highest in ((r). Then, JŁ, is clearly Theorem 2 below, certain fundamental social choice axioms are preserved by all RPRs.
true for all f E F, then P is true for fE. For
neutral, even if all f E F are. Still, as show in
Not all axioms are easily preserved. An SWF satisfies monotonicity if promoting an alterative in a voter's ranking while keeping all else constant does not hurt that alternative in the output ranking (cf. Appendix B.2 for formal definition). It turns out AbC does not preserve monotonicity. This is because promoting an alternative may cause AbC' to now pick different rule ranking that alternative further below, even if it does not hurt that alternative under any specific rule. Still, much like union consistency, we show that preserving monotonicity is incompatible with an axiom that AbC' satisfies.
Theorem 2. (1) AbC preserves anonymity & neutrality. (II) Any RPR preserves the Smith criterion, Condorcet consistency, majority winner, pairwise majority consistency, and unanimity. (III) No (anonymous) RPR can satisfy reversal symmetry and preserve monotonicity.
7

---

Published as a conference paper at ICLR 2026
Ma Mallows
Plackett Luce
KT Distance To Crouna Truth
0.3
0.3
Best Posit onal Scores
Borda
0.2
Kemeny
0.2
PL MLE
Murality
Plurality veto
0,1
Wiimmed Borda
0.1
Approval
Veto
Rest 04,
-0.04, m
0.04
0.08
0.12 0.34
0.08 0.12
KT Distance Between Spllts
Figure 1: Log-log plots of KT between ground truth and the ranking generated by SWFs the complete profile (i.e., error) vs. KT between rankings generated by SWFs on splits the data (i.e., inconsistency)
partial rankings drawn from the Mallows & Plackett-Luce models. Each point shows average distance over 10 splits for a single profile. The plots display clear positive relationship. In both cases, the MLE of the model outperforms other SWFs in both axes. As AbC picks the rule with the min. distance between splits, it would pick the model's MLE in both cases.
For readers unfamiliar with the axioms in (II), provide the formal definitions in Appendix B.2 along with the proof of the full theorem. (II) demonstrates strength of not just AbC, but of our RPR framework, showing how one can still reap the benefits of the axiomatic approach by restricting the candidate rule set to certain SWFs. For proving (III), we construct an explicit profile in which promoting an alternative results the reverse of the profile, and show that the demands of reversal symmetry and monotonicty cannot be simultaneously satisfied on this profile.
Overall, we have shown that AbC satisfies many natural axioms (reversal symmetry, plurality- shuffling consistency, and preservation of the axioms in Theorem 2) as for axioms it fails (union consistency and the preservation of monotonicity), we have given impossibility results showing each of them is incompatible with axioms that A&C' satisfies. In Appendix C.9 we apply the data-driven approach Caiata et al. (2025) to show experimentally that violations of monotonicity and union consistency typically across well-known distributions, well real-life data.
COMPUTATIONAL PROBLEM: PERFPOS
AbC' picks the SWF(s) from F that minimizes the expected disagreement over random split (Box [T. In this section, we show that when F = Fs (positional scoring rules), the algorithmic problem of minimizing disagreement over even given split is hard. Say we are given a ranking 0, C(A) for each voter i E N, well an even split of voters N N₁ U N22 with |N1] [N2]- Then, PERFPOS (Perfect Positional) asks: Is there a positional scoring rule fa E F's that achieves perfect consistency over this split, i.e., obtains (T) (((((()) = 0?
Theorem 3. PERFPOS is NP-complete.
Proof sketch. The proof, which is a reduction from 3SAT, can be found in Appendix B.3 Starting from 3CNF formula Φi explicitly construct two half-profiles (corresponding the sides of split) with pairs of alternatives for each clause and each binary variable in ø. The positions of the pairs associated with the variables force the difference between successive components of the score vector to be either sufficiently small or sufficiently large, corresponding to the associated variable being assigned True False. The pairs of alternatives associated with the clauses then ensures that perfect consistency can be reached if and only if this assignment of binary variables satisfies Φ.
This shows that when F Fs. the problem of computing the minimal possible disagreement is not only hard, but also hard to approximate to any multiplicative factor (since any algorithm with such guarantee must return solution with disagreement 0 when possible). While technically distinct, our result is aligned with other hardness results for optimizing over positional scoring rules, e.g., for picking the rule most consistent with underlying true ranking (Caragiannis et al. 2019)
8

---

Published as a conference paper at ICLR 2026
Table 1: Mean and St. Dev. of KT distance over 1000 splits from several metries on review scores provided by Kerzendorf et al. (2020). While the max. score of items are often prioritized in peer review (Nierstrasz )000), AbC shows that mean functions lead to more consistent outcomes here.
Arithmetic Mean
Min
Max
Median
Geometric Mean
0.364 0.001
0.444 = 0.001
0.409
0.001
0.371 0.001
0.369 + 0.001
Distance
0.15
Distance
0.5-
0:35
0.41
0.47
0.3
0.73
0.2
0.13
0.00
F1 031
ra 89)
Fl (20-222
Bost contonal
Bonta
Teo Approvol
Riumlity Jeto
Plunabity
Heto
Election
Hurality
Two
Huralty
(a) Distance between splits of rankings in empirical election data from cities using Instant Runoff Voting (IRV). Details data this plot are found in Table 2 The consistency ordering of the rules differs from elec- tion to election, and IRV may perform worst. As AbC picks the rule with min. disagreement for cach elec- tion, it is able to reliably select consistent ranking.
(b) Distance between splits for SWFs aggregating rankings of drivers in Fl races, where each race plays the role of voter. Rules labeled Fl correspond to scor- ing vectors used practice and are evaluated for only the years in which that rule was in use. AbC' can be used to evaluate the impact of rule changes; in this case, both changes in F1 improved consistency.
Figure 2: Split distance with several rules on data from political elections and Formula One races.
EXPERIMENTS
Despite our complexity result, AbC' can be efficiently implemented using Monte Carlo sampling, even when all positional scoring rules are included in candidate rules. We now give experimen- tal results evaluating AbC and discuss their implications. See Appendix C for additional details regarding the experiments, including specific datasets, distributions, and parameter settings.
Ground fruth distance vs. disagreement: While AbC does not assume a ground truth, a gen- erative model is reasonable approximation of the data, it performs desirably: Figure ] clear positive relationship between distance to ground truth and disagreement between splits for sev- eral SWFs under partial rankings drawn from two well-known distributions, confirming intuition from MVUEs (Section 2). The SWFs we test include an optimization for minimizing disagreement over all positional scoring rules ("Best Positional Scores". discussed further below). Importantly, for both Mallows and Plackett-Luce, the model's MLE (Kemeny and PL. MLE, respectively) outper- forms other SWTs in both axes. As each MLE has the lowest disagreement for its model, AbC' would pick it when given data from this model, hence obtaining the benefits of the statistical approach.
Score data: ABC can also be applied when evaluators submit ratings/scores rather than rankings; we simply split the ratings received by each alternative via the random process in Box [[] Running AbC' then provides insights into functions from ratings to aggregate rankings. Table shows the mean disagreement of score aggregation functions on the astronomy peer review dataset provided by Kerzendorf's et al. 22220) While past work suggests items with a high maximum score are often prioritized in peer review (Nerstrrazz 2000), AbC shows mean functions provide better consistency.
Peer review of proposals: Peer review data can also take the form of rankings. In Appendix C.2, we run AbC' on anonymized rankings provided by evaluators of projects proposed for the Atacama Large Millimeter Array (ALMA) (Meyer al., 2022), which is the largest ground-based radio tele- scope on carth. Our results reveal that modifications previously considered by ALMA-e,, e.g., adding outlier rejection to Borda count- significantly hurt the consistency of their aggregation rule.
Aggregating contests: In some settings such evaluating Al agents (Lanctot et al. 2025a), contes- tants compete in several events (each of which acts as a voter), and how to best aggregate the results into overall ranking is unclear. Indeed, Formula One (F1) has changed the positional scoring rule used for aggregating races several times (Bochmer et al.) (002).. With AbC, we measure the impact of such changes on consistency (Fig. 2b); analogous experiment on Olympics is in Appendix C.4

---

Published as a conference paper at ICLR 2026
Political elections: Figure 2a considers running AbC on several political elections, each with thou- sands of voters. Each election corresponds to a single value on the x-axis; for each election we show the KT distance of each SWF (e.g. veto, indicated by the yellow square, gives a distance of 0.14 on election 3). In each election, the real-world winner was determined using Instant Runoff Voting (IRV). However, in some of these elections (e.g., 1,16,18) IRV is the rule that gives the worst dis- agreement. Importantly, in 21 out of the 25 elections, there is a rule that achieves zero disagreement; however, this rule is different between elections, confirming our intuition that different rules are appropriate for different settings (Section 1. As a result, AbC reliably selects a consistent ranking,
Recommender systems: In order to demonstrate the wide applicability of ABC across di- verse set of domains, Appendix [C66 reports the result of running our method on the MovieLens dataset (Harper & Konstan 2015), which is used widely to evaluate recommender systems. As shown there, the most consistent rankings balance high (perceived) quality with wide appeal.
Other distance functions: While we have defined AbC using the Kendall- distance function for our axiomatic analysis in Section [ the AbC framework can also be used in conjunction with other distance functions, as mentioned in Section 4 The choice of the distance function can depend on the specific setting and the goals of the implementer. For example, if disagreements on top of the output ranking should be prioritized over disagreements in lower positions (e.g., we are aggregating search results from different algorithms for meta-search engine (Dwork et al.] 2001)), then Normalized Discounted Cumulative Gain can be used to measure disagreement. If, on the other hand, the desired output is not aggregate ranking but instead subset of alternatives of a fixed size (e.g., picking k < m proposals to fund, and the ranking among the winners is not important), then Jaccard dissimilarity can be used as a distance measure between the chosen subsets. In Appendix C.7, we rerun the experiments for the ALMA and FI data using these two alternative distances to show they produce qualitatively similar results. We believe studying the axiomatic properties of AbC for different output formats and different distance functions is an important avenue of future research.
Optimizing positional scoring rules: While decid-
Mean Distance
0.21
Anneeling
ing whether a positional scoring rule leads to zero dis- agreement is No-complet,, we can, in practice, learn
SGD
0.18
positional score vectors which approximately mini- mize the KT distance for sampled splits. |Figure 3 shows the KT distance of the best vector found by two optimization methods Formula One race data: Stochastic Gradient Descent (SGD) and Simulated Annealing. While both methods find score vectors which improve upon their initial state, we use Sim- ulated Annealing in the experiments above it typ- ically finds a score vector with lower disagreement and requires much less compute time than SGD.
0.15
0
100
200
300
400
500
Optimization Step
Figure 3: SGD and Simulated Annealing performance during optimization of KT dis- tance for Formula One race data. Anneal- ing outperforms SGD in terms of finding a rule with low disagreement high consis- tency.
Overall, our method provides a principled way in which users can evaluate the impact of modifications to existing methods by running AbC' on their own data. In some cases, the answer is that it is an improvement: both changes to F1 rules signif- icantly reduced disagreement (Figure 2b). In others, the proposed modification hurts consistency: Meyer et al.] (2022) consider adding outlier rejection to Borda (removing the min. & max. scores of each alternative) for peer review, but our experiment suggests this method (Trimmed Borda) in- creases disagreement on their dataset (Appendix
8 CONCLUSION AND FUTURE WORK
Treating voters' rankings as fixed, we addressed the problem of picking the most appropriate rule. is of interest to understand RPRs, and specifically AbC, more generally under strategic behavior of voters. Such behavior makes it harder to study the impact of a proposed rule change on consistency by running AbC' on prior data. Another direction of future work is to investigate RPRs under other axioms, either translated from axioms for rules or novel axioms for rule picking rules. Overall, RPRs (and AbC') open the door principled, data-driven rule selection for diverse real-world applications.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank John Carpenter and Andrea Corvillon from the Atacama Large Millineterssbmillinete Array (ALMA), as well as Kate Larson and Ariel Procaccia for helpful discussions. We also thank Sinan Karabóctoolu and Bogaçhan Arslan for implementing our method for the 2nd Computational Social Choice Competition at the 33rd International Joint Conference on Artificial Intelligence (IJ- CAI 2024). R.E.B. and V.C. thank the Cooperative AI Foundation, Macroscopic Ventures (for- merly Polaris Ventures / the Center for Emerging Risk Research), and Jaan Tallinn's donor-advised fund at Founders Pledge for financial support. R.E.B. is also supported by the Cooperative Al PhD Fellowship. The work of N.S. was supported by NSF CAREER award 1942124 and ONR N000142512346.1r
REFERENCES
Nir Ailon, Moses Charikar, and Alantha Newman. Aggregating inconsistent information: Ranking and clustering. In Proceedings of the Anmual Symposium on Theory of Computing (STOC), 2005.
John Aldrich, Jason Reifler, and Michael C. Munger. Sophisticated and myopic? citizen preferences for electoral college reform. Public Choice, 158(3/4):511-558 2014. 2014.
Ben Armstrong. opt imal -omam-otiinngggg voting
Optimal
voting, 2025.
URL http:://pypl.orrg/ppojeee
Kenneth Arrow. Social choice and individual values. New Haven: Cowles Foundation, 2nd edition, 1963. 1st edition 1951.
R. B. Ash. Information Theory. Dover books on advanced mathematics. Dover Publications, 1990.
Hossein Azari Soufiani, David C Parkes, and Lirong Xia. A statistical decision-theoretic framework for social choice. Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS), 2014.
Ali Baharev, Hermann Schichl, Amold Neumaier, and Tobias Achterberg. An exact method for the minimum feedback are set problem. Journal of Experimental Algorithmics (JEA), 26:1-28, 2021.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Nocmi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly. Tom Henighan, Tristan Hume, Samuel R. Bowman, Zae Hatfield-Dode Doddds Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv 22/2.08021 222208073, 2022.
Hannah Bast. How objective is peer review?, 2020. bl wllooacccoooooooeeeeee vesettivvviiieeeeee reneerr-evveww
URL https: htt.///ccc...cm acm org/ Online; accessed 20-Jamuary-2025.
Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan. Has the Ma- chine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment, 2023.
André Blais, Jean-François Laslier, François Poinas, and Karine Van Der Straeten. Citizens' prefer- ences about voting rules: self-interest. ideology, and sincerity. Public Choice, 164(3/4):423 442, 2015.
Niclas Bochmer, Robert Bredereck, Piotr Faliszewski, Rolf Niedermeier, and Stanistaw Szufa. Putting compass the map of elections. arXiv preprint arXiv:2105.07811 2021.
11

---

Published as a conference paper at ICLR 2026
Niclas Boehmer, Robert Bredereck, Piotr Faliszewski, and Rolf Niedermeier. A quantitative and qualitative analysis of the robustness of (real-world) election winners. In Proceedings of the 2nd ACM Conference Equity and Access in Algorithms, Mechanisms, and Optimization, pp. 1-10, 2022.
Arthur Boixel and Ulle Endriss. Automated justification of collective decisions via constraint solv- ing. In Autonooouu Agents and Multi-Agent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-34) 345, 1952.
Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D. Procaccia (eds.). Handbook of Computational Social Choice. Cambridge University Press, 2016.
Joshua Caiata, Ben Armstrong, and Kate Larson. What voting rules actually do: data-driven analysis of multi-winner voting. arXiv preprint arXiv:2508.0644 06454, 2025.
Olivier Cailloux and Ulle Endriss. Arguing about voting rules. In Autonomous Agents and Multi-
Agent Systems, 2016.
Ioannis Caragiannis and Ariel D. Procaccia. Voting almost maximizes social welfare despite limited communication. Artificial Intelligence, 175(9)11655-161,, 2011.
Ioannis Caragiannis, Xenophon Chatzigeorgiou, George A. Krimpas, and Alexandros A. Voudouris. Optimizing positional scoring rules for rank aggregation. Artificial Intelligence, 267:58-77, 2019.
Douglas G Chapman and Herbert Robbins. Minimum variance estimation without regularity sumptions. The Annals of Mathematical Statistics, pp. 581-586, 1951.
Paul Christiano. Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2017.
Vincent Conitzer. The maximum likelihood approach to voting social networks. In Proceedings of the Conference on Communication, Control, and Computing (Allerton), 2013.
Vincent Conitzer and Tuomas Sandholm. Common voting rules as maximum likelihood estimators. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAT), 2005.
Vincent Conitzer, Matthew Rognlie, and Lirong Xia. Preference functions that score rankings and maximum likelihood estimation. In Proceedings of the International Joint Conference on Artifi- cial Intelligence (IJCAI), 2009.
Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, and William S. Zwicker. Social choice should guide Al alignment in dealing with diverse human feedback. In Proceedings of the International Conference Machine Learning (ICML), 2024.
Corinna Cortes and Neil D. Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment, 2021.
Jessica Dai and Eve Fleisig. Mapping social choice theory to RLHF. arXiv preprint arXiv:2404. 13038, 2024.
Marie Jean Antoine Nicolas de Caritat (Marquis de Condorcet). Essai sur l'application de I' l'analyse la probabilité des décisions rendues à la pluralité des voix. Paris: L'Imprimerie Royale, 1785.
Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th World Wide Web Conference, pp. 613-622, 2001.
Edith Elkind, Piotr Faliszewski, and Arkadii Slinko. On distance rationalizability of some voting rules. In Proceedings of the Conference on Theoretical Aspects of Rationality and Knowledge (TARK), 2009.
12

---

Published as a conference paper at ICLR 2026
Edith Elkind, Piotr Faliszewski, and Arkadii Slinko. Good rationalizations of voting rules. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2010.
Mikael Fogelholm, Saara Leppinen, Anssi Auvinen, Jani Raitanen, Anu Nuutinen, and Kalervo Väänänen. Panel discussion does not improve reliability of peer review for medical research grant proposals. Jourmal of Clinical Epidemiology, 65(1):47-52, 2012.
Luise Ge, Daniel Halper, Evi Micha, Ariel D. Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. Axioms for Al alignment from human feedback, 2024.
Alex Gershkov, Benny Moldovanu, and Xianwen Shi. Optimal voting rules. The Review of Eco- nomic Studies, 84(2):688-717 2017.
Allan Gibbard. Manipulation voting schemes. Econometrica, 41:587-602, 1973.
Ayça E. Giritligil Kara and Murat R. Sertel. Does majoritarian approval matter in selecting social choice rule? exploratory panel study. Social Choice and Welfare, 25(1):43-73, 2005.
Randi Griffin.
120
years of olympic history:
Athletes and results. Online
dataset, 2018.
URL https:// kaggle wtt.co/attttttt/eeeeeeeeeee
120 120-years-o0 of o hypicchistoryy---oeooooooo aestory-athhettee athletes and results/
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2024. URL https:://// wwww. gurobi com
Bruce Hajek, Sewoong Oh, and Jiaming Xu. Minimax-optimal inference from partial rankings. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS), volume 27, 2014.
Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm Transactions on Interactive Intelligent Systems (TISS, 5(4):1-19, 2015.
Reinhard Heckel, Nihar B Shah, Kannan Ramchandran, and Martin J Wainwright. Active rank- ing from pairwise comparisons and when parametric assumptions do not help. The Annals of Statistics, 47663:309- 2019.
Paul Jaccard. Érude comparative de distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547-579, 1901.
Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions Information Systems (TOIS), 20(4):422-446) 2002.
Steven Jecmen, Hanrui Zhang, Ryan Liu, Fei Fang, Vincent Conitzer, and Nihar B. Shah. Near- optimal reviewer splitting in two-phase paper reviewing and conference experiment design. In Autonomous Agents and Multi-A Aeet Systems, 2022.
John Kemeny. Mathematics without numbers. Duedalus, volume 88, pp. 571-591. The MIT Press, 1959.
M. G. Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239 251, 1945.
Wolfgang Kerzendorf, Ferdinando Patat, Dominic Bordelon, Glenn van de Ven, and Tyler A. Pritchard. Distributed peer review enhanced with natural language processing and machine learn- ing. Nature Astronomy, 4(7):711-717, 2020.
Ashish Khetan and Sewoong Oh. Data-driven rank breaking for efficient rank aggregation. Journal of Machine Learning Research, 1-119)) 2016.
Ravi Kumar and Sergei Vassilvitskii. Generalized distances between rankings. Proceedings of the International Conference on World Wide Web (wwW), 2010.
Mare Lanctot, Kate Larson, Yoram Bachrach, Luke Marris, Zun Li, Avishkar Bhoopchand, Thomas Anthony, Brian Tanner, and Anna Koop. Evaluating agents using social choice theory. arXiv preprint arXiv:23/2.0312// 2025a.
13

---

Published as a conference paper at ICLR 2026
Mare Lanctot, Kate Larson, Michael Kaisers, Quentin Berthet, Ian Gemp, Manfred Diaz, Roberto- Rafael Maura-Rivert Yoram Bachrach, Anna Koop, and Doina Precup. Soft condorcet optimiza- tion for ranking of general agents. In Autonomous Agents and Agent--geet Systems, 2025b.
Jean-François Laslier. And the Loser Is... Plurality Voling, pp- 327-351. Springer Berlin Heidel- berg, 2012.
Neil Lawrence and Corinna Cortes. inverseprobablittt cww. /ttpa//aaaaaaaaaaoooo0/////////2 cessed 24.July-2024.
The NIPS Experiment, 2014.
URL http:// Online: ac-
Yusha Liu, Yichong Xu, Nihar B. Shah, and Aarti Singh. Integrating rankings into quantized scores in peer review. In ML Evaluation Standards Workshop at ICLR 2022, 2022.
R. Duncan Luce. Individual Choice Behavior: A Theoretical analysis. Wiley, 1959.
C. L. Mallows. Non-null ranking models. I. Biometrika, 44(1/2):114-330) 1957.
Nicholas Mattei and Toby Walsh. Preflib: A library for preferences. In Algorithmic Decision Theory, volume 8176 of Lecture Notes in Computer Science, pp. 259-270. Springer, 2013.
Tommi Meskanen and Hannu Nurmi. Closeness Counts in Social Choice, pp. 289-306. Springer Berlin Heidelberg, 2008.
Jennifer Donovan Meyer, Andrea Corvillón, John M Carpenter, Adele I. Plunkett, Robert Kurowski, Alex Chalevin, Jakob Bruenker, D-C Kim, and Enrique Macías. Analysis of the ALMA cycle distributed peer review process. arXiv preprint arXiv:2204.05390 2022.
Abhilash Mishra. Al alignment and social choice: Fundamental limitations and policy implications.
arXiv preprint arXiv:2310. 2023.
Oscar Nierstrasz. Identify the champion. Pattern Languages of Program Design 4, 2000.
Michael Obrecht, Karl Tibelius, and Guy D' Aloisio. Examining the value added by committee discussion in the review of applications for research awards. Research Evaluation, 16(2):79-91, 06 2007.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Patat, W. Kerzendorf, D. Bordelon, G. Van de Ven, and T. Pritchard. The Distributed Peer Review Experiment, The Messenger, 177:3-13, September 2019.
Elizabeth L. Pier, Joshua Raclaw, Anna Kaatz, Markus Brauer, Molly Carnes, Mitchell Nathan, and Cecilia E. Ford. 'your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review. Research Evaluation, 26 1: 1-14, 2017.
R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics). 24(2):193-202, 1975.
Charles R. Plott. Axiomatic social choice theory: An overview and interpretation. American Journal of Political Science, 20(3):511-596, 1976.
Ariel D. Procaccia, Aviv Zohar, Yoni Peleg, and Jeffrey S. Rosenschein. The learnability of voting rules. Artificial Intelligence, 173(12):1133-1144 2009.
C Radhakrishna Rao. Sufficient statistics and minimum variance estimates. In Mathematical Pro- ceedings of the Cambridge Philosophical Society, volume 45, pp. 213-218, 1949.
D.G. Saari. Geometry of Voting. Springer Berlin Heidelherg, 1994.
14

---

Published as a conference paper at ICLR 2026
Mark Satterthwaite. Strategy-proofine and Arrow's conditions: existence and correspondence the- orems for voting procedures and social welfare functions. Journal of Economic Theory, 10: 187-217, 1975.
Marie Christin Schmidtlein and Ulle Endriss. Voting by axioms. In Autonomous Agents and Multi- Agent Systems, 2023.
Murat R. Sertel and Ayça E. G. Kara. Selecting a Social Choice Rule - An Exploratory Panel Study, pp. 19-52. Springer Berlin Heidelberg, 2003.
Nihar B. Shah. An overview of challenges, experiments, and computational solutions in peer
review. https:/ / www. cmu wttp://maaaassspppppeeee.t http://pppp.ttttt/urvveeeeeeeeee /ww.veyyeeereeeeeeeeeeeeeeeeeeee pdf (Abridged version published in the Communications of the ACM), 2022.
Nihar B Shah and Martin J Wainwright. Simple, robust and optimal ranking from pairwise compar- isons. Journal of machine learning research, 18(199):1-38, 2018.
Nihar Shah, Sivaraman Balakrishnan, Adityanand Guntuboyina, and Martin Wainwright. Stochastically transitive models for pairwise comparisons: Statistical and computational issues. IEEE Transactions on Information Theory, 63(2):934-959, 2017.
Louis Thurstone. The method of paired comparisons for social values. The Journal of Abnormal and Social Psychology, 21(4):384, 1927.
T. Nicolaus Tideman and Florenz Plassmann. Which voting rule is most likely to choose the "best"" candidate? Public Choice, 158(344):311-357, 2014.
Ilia Tsetlin, Michel Regenwetter, and Bernard Grofman. The impartial culture maximizes the prob- ability of majority cycles. Social Choice and Welfare, 21(3)3877398,, 2003.
Ulrike von Luxburg. Clustering stability: An overview. Found. Trends Mach. Learn., 2(3):235-274,
March 2010.
Matthias Weber. Choosing the rules: Preferences voting systems for assemblies of representa- tives. Journal of Economic Behavior & Organization, 174:420-434, 2020.
Lirong Xia. Bayesian estimators as voting rules. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2016.
Lirong Xia and Vincent Conitzer. A maximum likelihood approach towards aggregating partial orders. In Proceedings of the International Joint Conference Artificial Intelligence (IJCAI), 2011.
Lirong Xia, Vincent Conitzer, and Jérôme Lang. Aggregating preferences in multi-issue domains by using maximum likelihood estimators. In Autonomous Agents and Multi-Agent Systems, 2010.
Wanqiao Xu, Shi Dong, Xiuyuan Lu, Grace Lam, Zheng Wen, and Benjamin Van Roy. RLHF and IIA: Perverse incentives. arXiv preprint arXiv:2312.01057, 2024.
H. P. Young. Social choice scoring functions. SIAM Jourmal Applied Mathematics, 28(4):824- 838, 1975.
Peyton Young. Optimal voting nules. Journal of Economic Perspectives, 9(1):51-64, 1995.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arxiv: 08593, 2020.
Walter Zucchini. An introduction to model selection. Journal of Mathematical Psychology, 44(1): 41-61, 2000.
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
In this section, we further elaborate prior work conceptually related to our work.
Rule selection The question of comparing different voting rules has been widely studied in the social choice literature. For an overview of the axiomatic and computational properties of various voting rules, see Brandt al. (0016). There has been earlier empirical work on collecting vot- ers' preferences over voting rules themselves (Sertel & Kara 2003 Giritligil Kara & Sertel, [005] Aldrich et al. 2014). However, it has been shown that (unsurprisinglly
many real-life cases voters simply prefer the rules that benefit their preferred candidate (Blais et al.) 2015 Weberr 2020). In general, it is clear that even social choice theorists themselves cannot reach a consensus on what is the best voting rule (Laslier 2012). There has also been interest in using tools from automated reasoning for instantiating and generating formal justifications for using a voting rule (Cailloux & Endriss 2016) or imposing axiom (Boixel & Endriss 2020). Similar tools were used for bypass- ing voting rules altogether by picking a set of axioms that impose a certain outcome (Schmidtlein & Endriss 2023).
MLE approach The idea of treating votes as noisy estimates of ground truth goes back to Con- dorcet [1785]. who designed a noise model where every voter has a fixed probability of ranking each pair of alternatives correctly, and solved for two and three alternatives. Young (1995) later extended this model to an arbitrary number of alternatives, showing that its MLE is equivalent to an earlier rule introduced by Kemeny (1959). Conitzer & Sandholm (2005) later studied the ques- tion of which voting rules are MLEs for some noise model where votes are sampled independently, in particular showing that any rule that violates consistency cannot be an MLE. Combined with the result that a continuous and neutral social choice function (rules outputing set of winners) is consistent and only if it is a scoring rule (Young 1975). this shows that no other (neutral and continuous) social choice function can be the MLE any noise model. [Conitzer al. (2009) did a similar analysis for social preference functions (rules outputting a set of rankings) providing an exact characterization of MLE rules as simple ranking scoring functions. Similar analyses followed for voting in multi-issue domains (Xia et al. 2010), with partial rankings (Xia & Conitzer 2011), and on social networks (where voters no longer vote (agpenddetlyyCoonizzee
(2013). In the oppo- site direction, A Soufiani et al. (2014) study the MLE of two known noise models to identify whether they satisfy certain axiomatic properties, and Xia (206) extend their results. [Tideman & Plassmann| (2014), the other hand, construct spatial noise model specifically to approximate data from actual elections, and evaluate the performance of common voting rules under this model. Lastly, a related but distinct approach to classify voting rules is distance-ration ratiioraaiiiiaaiaaaiiiyy (DR) Meskanen & Nurmi 2008 Elkind et al., 2009). which requires a rule to map each election to the result of the closest consensus according some metric. [Elindd et al, (2010) combine the MLE and DR approaches to better understand and compare rules, and in this framework declare Kemeny to be the best rule, as it fits both frameworks with the same underlying function. It is worth noting, however, that DR only applies to social choice functions.
Statistical model dependence in estimation from pairwise comparisons Another line of lit- crature on model (in)dependence focuses on estimation from pairwise comparisons. Commonly used statistical models in this setting include the Bradley-Teryy-Lee -eery-Luce (BTL) (Bradley & Terry 1952 Plackett 1975) and Thurstone models (Thurstone] 1927j. which all fall under the class of parameter-based models (also known random utility models). However, recent studies (Shah) ct al. 2017] [Shah & Wainwright, 2018 (Heckel et al., 2019) have demonstrated that estimators de- rived from more general "permutation--asee" models offer two notable advantages. First, when data are generated from these broader permutation-baset models, these estimators exhibit significantly better performance. Second, even when data originate from parameter- baseed models such as BTL or Thurstone, the guarantees provided by these general estimators are within logarithmic factors of those achieved by estimators specifically tailored to parameter- based models.
B OMITTED PROOFS
In this section, we give the proofs that were omitted in the main body of the paper.
16

---

Published as a conference paper at ICLR 2026
B.1 PROOF OF THEOREM !
We first recall our main result from Section 5.1 regarding the consistency axioms for RPRs.
Theorem 1. (I) AbC satisfies reversal symmetry. (II) Any welfare-maximizzng RPR fails PSC; AbC satisfies it. (III) No (anonymous) RPR can satisfy all three of reversal symmetry, PSC, and UC.
We prove each claim in the theorem as a seperate proposition.
Proposition 10, ABC' satisfies reversal symmetry.
Proof. Given any positional scoring rule fa F and profile a, say for is the reverse rule of fs. i.e., s' $2....... sm) alternative A by for when run rev(ar) (see Definition !
(1 Sm, 8m-1,..., 81). The total scores assigned to any
(ev( [a]
Σ
M
sm+1-))Mole la, m 1-100
Σ [a [a].
implying that fa(or) rev(fa (rev(r)) rev(rev(fa) (rev(()))) Additionally, we note that the Kendall-Tau distance (Definition 3) symmetric with respect reversals, i.e. (T(r1, (rev((r)) Thus,
(TTeevter((SSS(ee(("")))) (oee:///o (f (ef()))(
This implies that fa minimizes equation Box over random splits of a, then rev(f.) mini- mizes it over random splits of rev(ar, and therefore AbC or) (ev(AbC((F revr)))))
Proposition 11. Any welfare-maxxmizznn RPR 7. fails plurality-shufling consistency.
Proof. Take any Sa E Fs. We will show that for any k E Z+, we have ʃa (T (a [2, - (p(or
where fp is plurality. Recall from [ we have 81 = 1 and 5m 0 WLOG. By Definitions and 6 the total score assigned given alternative A by input x (ar, [2, m)) (say t (a)
is
[a] = MM (a, 1] ls mls1 +
m Σ
Ma Ma la
Σ
me! (m 1)
1=2
j=2
k.m! (m 11)
=Mo (a, - m! +
Σ 8j
(n 1--Mola la, 11)
n + Mo 1] k.kkmmm ml! (1
5m-1 8)
k-m!
8j
(m
= C + D Ma [a, 1).
where and does not depend on a. Since = 81 2 82 ... sm 0. we have 5m-1 83 <
m 1 and therefore D This implies that for any a, E A we have
> (x b + 4x [a] > t
M, a. 1] Mb
+ (T) b.
Since fo arbitrarily chosen, this implies all rules in Fs return the same output n² (a, (m 1m-11 1)).
Hence, for any ceffarr-mmaimiziinn RPR and any F C Fs. we have Z(F, ( [m 1)) = F, showing that the rule fails Definition
17

---

Published as a conference paper at ICLR 2026
Since all fs Fs produce the same output on (a [m - 1]). one might wonder whether it matters which rule an RPR returns. However, in practice an RPR might be employed to "lock in" a rule for to it in future profiles coming from the same source without re-running the RPR at each round, potentially increasing interpretabillty. Hence, with a plurality-shufflng consistent rule, would be able to keep using Jp in future profiles that approximately (but not exactly) shuffled, as future repetitions of the same process are likely to once again have their signal concentrated in the top position. Moreover, identifying that no fa I's provides any information beyond fp has a computational advantage: while the output of fp can be computed in On + m) time, an arbitrary fo E Fs might require ((m)) time. Proposition 12. AbC satisfies blurality-suuffling consistency.
future use, allowing
Proof. Fix some finite C Fs. If Fl we are done. Otherwise, given any profile o (that satisfies the condition in Definition 7 ie., (p( contains no ties) with internativesdddefinn
voters m m 2 3 T (a [2, ml) for all k. Let M be the matrix such that for each alternative a E and index E [m]. .M [a, i] indicates the number of voters in o that rank a in the 7th position, and let Mg be the analogous matrix for dk.
For any f E F and any two candidates a, b A, say D.' is the indicator random variable that the (1)
two output rankings resulting from applying / to the sides of random split of a into o and
(where each voter is independently and uniformly placed in one of the two sets, as defined in disagree about and b's position; i.e.,
k
Box
a.b D..tk = I[(a > m(a and > ((a
) a) or (b > (( a and a > ( 6)]-
Similarly, say T is the indicator random variable that f ties a and b on one of the two profiles;
i.e.
1i.r2 = I[(a = !) and b 5(= a) or (a )(a) b and b {(a a)].
Then, using linearity of expectation and the fundamental bridge, we get
T
Pr
E (TT(f =E
Σ a.DEA
D a.b
T 2 a,b
1
Σ Pr Di.k ab
Pr.
= +
2
a.eEA
(4)
The rest of the proof will follow from the next lemma.
Lemma 13. For any f E F. (fr there exists kj zz such that
1. for all k. k we have Pr D
a,b = 1Pr
= >
Pr Dfaie = +
a.b
1 Pr Pr[TT
2 a.b = for all a, such that M(a, 1] > 0 and M b, 1] >
2. If there exists c E A such that M(c, 1] = 0, then Pr[ = 1] + Pr T 2
Pr Dirak =1 + Pr Thykk =1 for all a e and Z-
Since fo(o) contains no ties, there can be at most one c E A with M (c, 1] = 0. Since m 2 3, [
the lemma implies that Eabea Pr D =1 z
Pr
T a.b =1 Ea,es Pr D = + a.b
Pr TTank = for all 2 ky.. By definition of AbC, this implies that for all k: 2 ky we have a.b f AbC(F,T*(e (o, (])), since (4) is strictly greater for f than for fp. Since |F] is finite, picking A max/erM(fa) k will ensure AbC(F, R (a, |m 1]) {fp} for all k > k, desired. We will now prove the lemma.
'If m = 2, there is single (monotonic) positional scoring rule satisfying Definition vector 0), which is also equivalent to fz.
the with score
18

---

Published as a conference paper at ICLR 2026
Proof of Lemma 13 For any f F, say {s hie(m] is the scoring vector associated with f. Fix any candidates a, A. Say Maa 1] Pa and M (b, 1] Ps. with Pa. > pb WLOG By Definition we have M& [a, 1] = kp and [1 1] = kpom! whereas My(a, k(n mm! and My d
m-1
k(n-pu)m! for all < f m. Similarly, let MM[((a, a): (b.j)] be the number of voters mn-1 in og that rank in the ith position and b in the jth position (with + j). Then
Cp
if
Ma((a, (b,5))
CipCi
j
kO2
otherwise
where Ci
m-1 and C2
(n Pe Palm! (7n-1)(m-2) Given random split of ar into of and ox say X
is the random variable indicating the total score of minus the total score of 6 in 0% for each i (1, 2). Naturally, regardless of the split, x-f* + x-_ must add up the total score of a minus the total score of b in σj according to so
(i) according to
m2
x X
of M&(a. la,
Σ
o M b,
k(n Pa)m!
k(n Po)m!
kp kpsm! +
m
X kk(pp pb)m
m -
where 2 = (m - 1) Σ $ > 0, as s/n = 0. Then, using symmetry of the two sides and the
fact that Pa > pb, get
1 Pr x f X-4* =0 Pr x ( [Nk(pp Po)m!
Pr
D 0.b
xxf.t
m ---1
N (Pa Pb Pb)m!
[x(** < 00
+ Pr x
Pr
m
= 2r x//. < 0]
x o]. Now, if and b are ranked ith and jth in a ranking, respectively, the score assigned to a minus that assigned to by f for that ranking is of ss. Say Bin(z) is the fair binomial distribution where is the number of experiments and the probability of success for each experiment is 1/2. Then,
A similar analysis gives Pr T =1 2P
X f.k
(s/ - Bin (M&l(a, i): (b. j)l)
(Xj[(m]
Σ
(sf
s/) (Bin (Ma i); (b.j)) Bin (Mal(a,)); O1))
= Σ
(of sf) (Bin (Mal(a, f); (b.j)) + Bin (.. (b. OD - Mal(a, (I,)))
Σ
(of sf) (Bin (M,l(a, f); (b.5) (a,((. Me((a,)); i))
K</E/m]
Σ (of - 3)) (Bin (2kC2) kC2) +
(1 of (Bin (k(p, + P1) kip (5)
K]E[2.m] In particular, for plurality, equation ggives us:
x Lnik
(Bin (k(p. + Pb)C1) C1) Bin (k(pa + po)mm) kppm!
(6)
1=2
We cannot have Pa Pe by assumption fo( contains ties.
19

---

Published as a conference paper at ICLR 2026
Now, fixing some f E FV {fp}, using equation
we write X =Y= Ei=2(1 - sf)Z,, where Σ (of sf) (Bin (2kC2) - kC2). and
~
7 (Bin (k(pa + Cb) - kp) for all i E [2, ml,
each sampled independently. It easy to check that B[y] = 0 and that Y is a symmetric distribution. We will treat two cases separately.
Case 1: pb = 0. this case, we cannot have x-fxi* < 0, as seen by 66. Therefore,
Pr Pr[D For f. on the other hand, we have
a. = | + Pr r = 0 + Pr x b..k
0 = 2 g-ppmm
(7)
Pr x .* o] 2 Pr|Y = 0: Z: = 0 for all i E m|] = Pr|Y = =0
Pr|Y = 0 2- kpum!
(2hr. )
and Pr [xxx** <0] 2 Pr|Y < 0: 2z 0 for all i E [2, ml]
1 Pr[Y 0) 2 kpam
2
Pr D 1 Pr b =1] _
Pr X th 0 + Pr
X fh 0 2 kpam! mi!
Since E A and k E Z+ were arbitrarily chosen, combining 7 and ( gives us statement 2 from the lemma.
Case 2: ps > 0. Say (NN, r)) is Pr(A S rJ for A ~ Bin (N), given that 0 <r < 1/2 and rN is an integer. By Ash 1990. Lemma 4.7.2), we know that
2 2-N(2-(())
888((11 S S2-
where h(r) = -r logogr- - (1 r) log(1 r) is the binary entropy function. We will use equa- tion 9 to upper bound the error probability for fp and lower bound the error probability for f. For convenience, define
r)
pb
hp def h(rp),
No def (pa + C1)
Pe + pb
First we start with fp. By
and
we have
Pr Pr[DAtu**
1]
Pr
Souk a.b
= 1
= Pr x <
+ Pr Pr|xXfort X
fonk = 0]
2
$ 2r x
0 = 20(k((p + pp)mml, rp)
< 2 (m- 13(1-1,) = U(k),
where we will use (k: as an upper bound that depends on k. Now consider f. Since ʃ + Jp sfn 0, must have s/ 0. Fix some O such that Pa-Ps is sufficient). Recalling
and since 1 sf 2 $ 2
< min Pa-Ps
(if s 1, then ensuring < Mpotpa)
PPJ] (p. (p.+pa)(10f)
that X f.k = Y + 21 - ) and that s/h = O, we have
Pr x so]
2 Pr(y <0 Pr/Ze S skNN Pr 7m S -ekNN(1 - x/)]
II
Pr =0, 1=3
Pr[x/** f < 0]
2 Pr|y < Pr < <khNol Pr Zm < -ekNp(1 - s//]
Pr[z, S
3=3
Pr|y > 0J Pr[Ze < skN Pr 2m < -ckNN (1--s]))) sf)]
Pr[Z, S 0]:
20

---

Published as a conference paper at ICLR 2026
therefore,
Pr D a.b
= + Pr
1
T/- a.b
= = Pr X * 0
+ Pr | =0]
= Pr X fkk <0] + Pr
X i =0
mh
2 Pr|Zz s skNDp] Pr Zm S -skNp(1 - -b4
II 3
Pr[Zz
12-1
2 Pr|Z2 S |<kN,l1 11 Pr Zm S -TEkNp(11 - s6o1|
II A=3
Pr|Z; 0
2 pk p. 12 (A)) @(kNN, Tm (k)) (o(K:N xp)) m
1e and I'm (k) = Tp [ERN,(1-s() AN,
where r2(k) = Tp +
As k - 0c, we have (k(
AN
up + = E (0, ()) and Im (k) 7 Tp - a(1 - -5/) E (0, }): hence, we can choose k large enough such
that 0- < ri < 1/2 for both i E (2, m). Further, by construction, kN, rr(k) is an integer for all
def h((r)(k)) for i E m2. By equation we then have 1 and i e {2, mJ. Define he (b)
Pr (D = =1] 1] + Pr
Pr a. = obNN,, r2k))) (K&NN, I' (k)) (O(N:, p,.)) m-))
2 KN (1-hh(1)))
2
2 2-NN1(1-h,)
r (k) (k) V 88NN (m) - rm)())
P Tp))
2 NN, (R) (m-3))pp)
V(8kN,)'m 1- (r (k) (2))) rm (k) I' (k:) (rpp))(p)p))))))))))) rp)) (1-rp))m--- L(k).
where we will use T.(k) as a lower bound that depends on k. Now, we would like to show that there exists knd such that L(k) (kk for all & 2 kgp. We have
L(k)
P())-
(10)
U(k)
(8KN (k) T2 (1) (k)) ),
where P(k) = (m - 1)(1 -1- h1) - (m- - 1 112()) hm (1:) - (m - 3)4₂) = hr (k) + Am (A) 2h p- We would like show lime 100 U(K)
L(K)
co. giving us our desired relationship. We recall that
(k)) + Tp + and rk) + Tp - s(1 - s66) as k + 3a. so the denominator of equation 10 scales
as ( " > for large k. As for the nominator, we have lim P(k) = h(r, - =) + h((, - =(1 -
{))) 2h(rp) = (e). We have H(0)= 0 and (0) h h((r,) (1 s5) s/ (rp)
since of > 0 and
-1) log(2 - = 0. 1t' 1'(rp) = log rp - 1 + log(1 rp) +1= log (1 rp
as Tp Potps 3. We can therefore set > 0 small enough such that H(s) > 0. Then the nominator of equation 10 scales as ((2)N,H(()) for large k. dominating the denominator. Hence,
lim L(k) U(k) oo, desired. Therefore, can choose a large enough Ag such that for all
k > kabb have
Pr [DL a.b =
1] + Pr
a. pr T = L(k) U(k) 2 Pr Pr D =1 1] + 1 Pr 1]]
a.b
a.b
Since there are finitely many candidates A, picking ky max(a, kg is sufficient for proving statement from the lemma.
21

---

Published as a conference paper at ICLR 2026
Proposition 14. No (anonymous) RPR can satisfy all three of reversal symmetry, plurality-shufflin
consistency (PSC), and union consistency (UC).
Proof. Assume an RPR Z satisfies PSC and reversal symmetry. We will show that Z necessarily fails UC. Fix A = fa, b, c) and = ppp (.ppJell Je J Say da consists of two voters ranking > b > and one voter ranking b > a > c. Similarly, say σ consists of two voters ranking e > b > and one voter ranking b > > We have fp( _ satisfy the preconditions of Definition By PSC, there exist kas Ko such that for all kg 2 ka and Kg 2 ki we have (F, x (a [2, ml)) = ((F, x²% (as. [2, ml)) = {fp). Pick & = max Ri. Ko) and let a = (a. [2, ml) + x ( ml). It is straightforward to check that a = rev(a) up permutation of voters, with each
> b > and fp(os) = <> b > so both profiles
L(A) appearing 6k times. By anonymity and reversal symmetry, this implies ((F,o) = rev( ( o)), which is only possible oF = (Sr, Sph. proving UC violated.
B.2 PROOF OF THEOREM 2
Theorem 2. (I) AbC preserves anonymity & neutrality. (II) Any RPR preserves the Smith criterion, Condorcet consistency, majority winner, pairwise majority consistency, and unanimity. (III) No (anonymous) RPR can satisfy reversal symmetry and preserve monotonicity.
We prove each claim separate proposition.
Proposition 15, AbC preserves anonymity & neutrality.
Proof. The proof follows definitionally from the anonymity and neutrality of the Kendall-Tau dis- tance (Definition 3). and therefore of the AbC function (Box
Next, we formally define the social choice axioms given in part (2) of Theorem
2 Given profile o
and a, b A. we say a pairwise defeats if ii N a >a, b} > lii E : a}}. Then, say an SWF f E F satisfies...
the Smith Criterion (SC) if the alternative(s) ranked highest in f(or) belongs to the Smith set of a i.e., the smallest set S C A such that every a E S pairwise defeats every E A
Condorcet Consistency (CC) if it satisfies the SC for all profiles that have Smith set containing a single alternative (called the Condorcet winner).
Majority Winner (MW) if is satisfies CC whenever the Condorcet winner is the top- ranked candidate of a majority (> m) of voters.
...Paiiiiis Majority Consistency (PMC) if whenever there exists E R(A) such that a >> b if and only a pairwise defeats b in σ,, then f(or = r. (Ge et al. 2024)
....iimmmi if whenever a, for all E N, then f(a)= =r.
Proposition 16. Any RPR Z preserves the Smith criterion, Condorcet consistency, majority winner, pairwise majority consistency, and unanimity.
Proof. Like Proposition 16 the proof follows from the definitions of the axioms. Each of them are of the form "if a satisfies conditions X, then (ar must satisfy conditions Y." Since for each we have 1 (o) f(or) for some f E F, restricting F to rules that satisfy this axiom will ensure f (a) will satisfy whenever o satisfies X.
Next, we formally define monotonicity.
Definition 17. Given weak ranking r E R(A) and an alternative a E A, say rank,(a) + Hb E A : b >y aH} is the rank of a in r. We say an SWF f satisfies monotonicity if for any profile a and alternative a E A, if o is the same a except some voters now rank a higher, then rank (a) 2 ran))
Before proving the general impossibility result in part (3) of Theorem 2 we will show that AbC does not preserve monotonicity.
22

---

Published as a conference paper at ICLR 2026
{fp. fur (plurality and veto). Both of these rules are monotonic, all positional scoring rules are. Fix 2 and consider the following profile a:
Example 18. Say F
Group voters rank a > - c> d.
Group 2: k people voted d > a b.
Group 3: k people voted
a b.
Group 4: k voters rank > a c² d.
Group 5: 3k voters rank c > > b x d.
Group 3k voters rank d > b c
The veto scores (the number of voters that rank them bottom) of a.b, are 0, 2k, 3k, 5k, respectively. Pr[fo(o(")) fo(o(2))= (a > c d) taken over the random process in Box ! Hence, the expectation in Equation (7) of Box Z approaches 0 for fo as grows. However, the plurality scores (the number of voters that rank them top) of and b are tied, implying the the probability that (p((r()) and (p(a((2) will disagree about a and converges to I as + oc. This implies the expectation in (7) approuches at least 1 for Sp as l grows. Therefore, there exists kı such that AbC(F, o)) {fo for all k 2 kl, and therefore f (o) = (a > > c > d). Now say the voters in group 1 and 2 promote by single spot, to get the profile a':
By an analogous argument to that in Example f
as k - co, where the probabilities are
Group 1: k voters rank x cx d.
Group people voted > > b
Group 3: k people voted d > > a b.
Group 4: k voters rank >
d.
Group voters rank e > a >
Group 6: 3k voters rank d > bbbaa a>c.
Importantl, we have o' = rev up to permuting Groups 1 with 2, 3 with 4, and with 6. By anonymity and reversal symmetry, this implies AbC a') {fpp. so (o') (d > > b > a). Thus, by promoting b, we have gotten ra (a (b) = < 3 = rank r (or) (b), which violates monotonicity.
We now show how the same example can be used for other (anonymous) RPRs satisfying reversal symmetry, in order to prove part (3) of Theorem 2
Proposition 19, No (anonymous) RPR can satisfy reversal symmetry and preserve monotonicity.
Proof. Say RPR Z satisfies reversal symmetry and is anonymous. Take a, a' and from Exam- Pl We consider two cases.
Case 1: ((F, o = {.f.. Then, by the same reasoning as in the proof of Example 18 we must have o(F,o') {fp} (as that example only uses anonymity and reversal symmetry to show this),
and monotonicity is violated by JE.
Case 2: Z(F, σ) {fp). In that case, ( (d c> a b), iee,, and b are tied in the bot- tom. By reversal symmetry, However, one can go from o' to a by simply promoting a by a spot in the rankings of groups 1 and (o) (a), promoting results in increasing 2. This implies that rank (ar) (a) = = rank (0)
have Z(F, a') {fu} and therefore f (o') (a > d).
its rank, therefore f is not monotonic.
In either case, we see that fo violates monotonicity. Since fr and f. are both monotonic, this proves that Z does not preserve monotonicity.
23

---

Published as a conference paper at ICLR 2026
B.3 PROOF OF THEOREM!
3
We recall the complexity result from Section 6
Theorem 3. PERFPOS is NP-complete.
We first introduce a generalization of the computational problem PERFPOS, which we will call PPRRFPoS: For each voter E N, alternatives A; C A, with |A,] = k. We are also given a split of voters N = N₁ LI N₂ with N1 = |N2]- For each j {1,2), in Nj that rank in their 1th position. Then, PEREPOS asks: is there positional scoring rule fa E Ps that achieves zero disagreement over this split, i.e., is there a vector 8 (8))se(K) with
given strict ranking a; E L(A.) over subset of
A, and E [R), Ma,a,P i] indicates the number of voters
1 81 82 2... ... 2 Sk 0 such that for all a, E A
(Ti Ti (T) [a] T2(6) > 0,
where = = Myla, d|s, for any a E A and j E {1.2].
Clearly &PERFPOS contains PERFPOS (for h = m). However, as we show next, it is not harder: Given an instance of kPERFPOS (with imput profile σ). complete the ranking of each voter E N to strict ranking over all alternatives by placing the remaining - k alternatives in A \\ A, at the bottom of os giving rise to a new (complete) profile a'. Then define o" ='(a',, [k. ml ), i.e, the l-shuffling of o' with respect to positions [k, m] {k, + 1, ..... m) (Definition 6). Then, we claim the answer to the original &PEREPOS instance with profile a is a yes if and only the answer to the PERFPOS instance using o" (and the same split
is yes. Indeed, if there exists a positional scoring rule fa with 81 2 $2 2 2 8% 0 that achieves zero disagreement with
Ss if achieves zero disagreement with o otherwise
o, then fo" with s" (sc))E{m) defined as s!
o". On the other hand, given a positional scoring rule far with 1 = o 2 s 2.. 2 s" = 0 that achieves zero disagreement with o", define x +o t*. and define vector s = (8)) as
1+1
= (s /1 - x) for all i パ - 1] and 8& = 0 (We have r < 1 since s = 0). We will show
f. achieves zero disagreement with a. Given any a E A and j E {1, 2], the total score assigned by
f to a on input "() (restriction o" voters in N₂) is
N - E Ma
Ty(4]
K.M la, o3s"
+
k!
k: +
kmn
kM x||
Σ
M [s! x))
by Definition the other hand, is
The score assigned by fa on input g(6) (restriction of to voters in Ny), on
[a = M, Σ M,la, 8 [a k.a
1- kd(1 x)
Since the total score of every alternative is just shifted by constant and then rescaled by another
constant, for any a, E A we have [a] > [b] and only if T, [a] > T2 As f&" achieves zero disagreement with o", this proves fa achieves zero disagreement with o. Thus, in the proof of Proposition 20 below, we reduce 3SAT to &PERFPOS, which proves the NP-hardness for PERFPOS too (membership follows from the fact that positional scoring rules are easy to compute).
Proposition 20, &PERFPOS is Narhard.
Proof. We will be reducing from 3SAT. Say Ø is 3CNF formula with clauses C1, Cz...., C2 and binary variables æ1, .2, 2t. For each 3 E {1,2], 2}. we will construct and instance of PERFPOS by first specifying M2 i] for alternative a E A and position E [K], and then explicitly designing rankings that is consistent with that Mg. We start with setting & + 2 and 7(k+2)
24

---

Published as a conference paper at ICLR 2026
For each i [t], add two candidates a, and b,, with
(I ++))(-1]]
if
M1 [
++
j = i otherwise
0
(k + 3) 1)
if j and +
2+2
if j = if j = k otherwise
Mi [i
1
0
and
1
i
M2
1/s
i
otherwise
(1/E+ 1)(i -I)
j 1 and i + if i if j otherwise
1/s
M2
For each i E [f], say the clause C, in ø consist of variables {εybev for V22 S [t] with 1 s |M 3. For this clause C₁ add two candidates Cj, dj. with
3))+3)- 22-
if if =
t(k + 3) 2i if otherwise
Mi [[
Mi =
0
otherwise
For constructing M2[cr. and M2[d.. j]. say 2 E {0, 1, is the number of negated literals in C.. To build M2 and M2[d;, for each! j [A:]. start from the following:
(11 +1 6(k + 3) 1) if
Myca.
1
if j = otherwise
0
1+ +0
if
Mg[d;,]] =
2z
j = 1: otherwise
0
Now for each E V,, do the following:
If žy appears non-negated in C, then add 2(k++2) M2 and 2(k+2) to M3 11. If x, appears negated C₁ then add 2(k: + 2) to M2 c.. + 1] and 2(k: + 2) to Mf [d4,
Finally, set M2[c, j] and M2[d; to the resulting M2 .l and M2(d, .| for each j E [k}]
bbjj C {cihetg, and {d bej.- We now construct profile that corresponds the above Mg. First, add & more candidates É {eihe(k)) Now for each {1, 2), each f A LJ BUCUD D and each E [k), add Mo[f. i] voters the set Ng that rank in the tth position, and ranks e, in the 7th position for all [k] {i). Say have added ny and ng voters to Ni and N2 so far respectively and that 3 = arg max (1.2) ng (If it's a tic, pick 8 arbitrarily). Add (14k + 28) mg and (14k + 29) n2-93 voters to Nar and Na- respectively, all of whom rank e, in the tth position for all E [t). This also ensures that [ = [Ngl. For each | E [1,2] say o (8) is the final vector rankings of Ng. as specified.
Say = faj)eej)
By construction, for each 3 E {1,2), f E AU U C U D, and E [k]. o (8) indeed has Ma[f, i] voters that rank in their tth position. Further, we have total of |A LI PABE LI
25

---

Published as a conference paper at ICLR 2026
C D U E| 2(t + + + 3t 28 + candidates (which polynomial in t, t, and |N1] = |N2l = (14k + 29) ma. ESEAUBLOUD Ei=1 Ms[ I (14k: + 29)2(t + t)k- max JEABBUUAUC Ma[S, i] (which also polynomial in f since all entries of Mn i] are).
We now claim that & is satisfiable if and only if there exists positional scoring rule that gives full agreement between (1)) and (²))
(=) Assume there is a positional scoring rule fo with s (st)aek that gives full agreement for M1 and Mz. Say [a = Σi= Ma (a, i|s; for all a E A and B E {1,2}. In partieular, we must have agreement between a; and 62 for each i [t]. We have
Ti [a.] (1 + (4: + 3 1))ss (k- 2)81+1,
Ti [b] ((k: + 3) 1))s1 (k + 2)s; + 8K. T2[b:] = ((1/e+ + 1) 1))s1 + si/E + sk-
[2: = (1+ + (1/E + (i - 1))s1 + 8141/E,
Since 81 = and 8m = 0, perfect agreement implies that we must have
[Ti - Ti (6)6)))(())))) (T2 [b,b,) = (1 - (I: + 2){ s1+1))(1 - (s; - s s1+1)/E) > O.
This implies must either have s; 81+1 < 8; 8141 > 1
k+2" Set the binary variable x; to
False if 5; - s++1 < E and to True if s; 81+1 > k+2 We now argue that the resulting {aiheet satisfies Φ) i.e., satisfies all of its clauses. Fix any i E [e]. We will show that C2 is satisfied. By assumption of full agreement,
(T - (Td T2ld:) >0
Since [c T [1 (1(k + 3) + 21)81 (t(k + 3) + 1)81 8% 1 >>>>> 0, this implies
that (T2[c]] - 72(d.]) > O. Say {ay hev. are the variables that appear in C, (for Vi C [t], and that
Z = (j E V zzy is negated in C.) with [Z| z. This implies we have
7]c4] = (2x + 111/ + 1) 6(k + 3i - 1))ss + +(k
Σ
-
Σ
8g+1
AgEVIZZ
gEZ
T2[4] = (1 + (1/6 + 1) 33 1))81 #22zk + 2(h + 2)
Σ
8gt + Σ
8g
(
SEZ
Tylc]] T2[d,] 22 1 2+k
Σ (sy - syt1) gEVi
Σ (Sg (9+1)
9E%
The only way for C, to be not satisfied is if xy is assigned to True (i.e. 8g - 9g-1 K4p for all
g Z and Iy is assigned to False (i.e.s 8h+1 < E) for all h E V, \ Z. This would imply, however [2cc--TT2[d < 22 2 (EgevizE EGEZ K+3 = 22 1+ 2([) -2 <
-1+ 7 < 0, which gives a contradiction.
As assuming C, not satisfied gives contradiction to the assumption that gives agreement be- tween (1) and (2) for e; and d,, C2 must be satisfied. Since this is true for all i [f]. this implies that & is satisfiable.
(=) Assume & is satisfiable for truth assignments (z (x7)e For any E [t]. define
if is True otherwise.
5,
k+1
Define the positional scoring rule s (8i)se[a) as 8,
2
i
1 (E 6,)
i > i> 1. Since Sk-1 sk-1=1- 1 [Σ = 1
k-2 = Sk. mono- k+1
i k tonicity satisfied, and s is a valid scoring rule. We will show that s gives perfect agreement between (1) and d²)) As always, say Ts(f]- = Σk_ Malf,j]s; for all E (1,2} and f e A. Since 81 2 82 2 ... 8k = 0 and since 8; for each i [k: 1). the total scores for each
0
26

---

Published as a conference paper at ICLR 2026
e [ and E [4] are
Tiaa (1+
(AI)
Ta(a.] = [1--0-1|||||00000000000000001100
(A2)
= (4
1)
(B1)
[2 =
1)(1
(B2)
T [// =
[()])))+))))))))))))))))))))))>) 11)
(CI)
Tale] = (2|Z)) + 111/ + +)+ +) - 1))s1 + + 22)+2) 22)
Σ
$g + Σ
8g41
gEV,\Zz
9EZ,
= (11/E+1)++ 66k + +3) 2 Tace)> > 01/6 + +666(k++++ + +3)(1 - 1)
(C2)
[idd;1 =
2j 1)s; + +(++)) 3) + 2 > T[d t(k + +3) + 2 1)
(D1)
Txdd]] = (L + +0/= 6(4 + 6 +3 1))s + 2 s/ + +((
Σ
sg+1
Σ
$9
/
gEZ,
+ +(1//++1) + 6(k + 3]j Txd, > t(L/ 1) + 6(k + +3 1)
(D2)
where Vi C [t] and Zj S Vi indicate the indices of the variables that appear in and that appear negated in the clause Cy. respectively. We now show that gives perfect agreement between o (1) Ti [T] [2(h)) > 0 for all distinct pairs of g, AUBuCUDLE.
i.e., (Ti and o ,i,,,,e, We will proceed by a case by case analysis:
Case 1: g E AU h E C D. By eqs. Al and B1 J. we have [g < t(k 3),
since i <. t. By eqs. CI and (DI. we have Ti h] >
Elk + 3). as j 2 1. This implies
[g]
Ti [... Similarly, by eqs. (22 and B2 we have T2[9] I 1(1/E + 1), and by
eqs. C2 and D2 we have [y(h] > 81/e + 1), as j 2 1. This implies Talg] < [2]
Hence,
[Li T [Ta) - [h) > O, as desired.
Case 2: g E b.). h E {ay, b,} for some t 2 i > أ>حد 1. By eqs. Al and B we
have Ti [g] > 3) 1) > (k 3)j and [h S (k+ This implies [g > [1].
Similarly, by eqs. (A2) and (2 we have T2ggl > (1/8 1)(i 1) (1/E + 1)j and
T2 < (1/E+1)5 This implies Tzlg T2] Hence, (Ti(g) (M)) - T2(h)) > 0,
as desired.
Case 3: g = a; h = by for some E [t]. In this case [i [h] = 1k-++)+) (s--1))))))
1- (k+ + 2)0; and Talg] [2[4] = 1- (1/=)(s: s1+1) 1- 0./e. if 3 (i.e. x
is False), (Ti (T(lo) - Ta(h)) (1 (1 0. If k+1 (i.e. x7 is
True), [Ti - (T2o)
= (1- -1))) -41 T(++2) = (E+) 66-
0.
A+1
. Case 4: g E d.). h E (cj,d,) for some 1.2 j > 1. By eqs. [C and DI). we
have Ti [9] > [(k++3)++ + 2 1) ((k+ + 2j and Ti [h] < ((k+3) (2) This implies
[g [i]h]. Similarly, by eqs. C2) and D2) we have T2 > t(1/E + 1) + 6(k +
3i 1) 2 t(1/= ([1/=++)==0 +1++++000 3)j and T S t(1/s + +1 + 6(k + +3). This implics
Tg > T2[h]. Hence, (Tilg) Ti (Talg) - [2] > 0. as desired.
Case 5: g = C₁, h d; for some i E [e]. In this case Ti - Ti [h]
1 and
Talgi 1b]4]
(sg s-+1)
Σ NEZ.
(sh sh+1)
gEVi
2|/z]| - 1 -
Σ
og
Σ 8n
DEV
hEZ,
27

---

Published as a conference paper at ICLR 2026
where V; [t] and Z; S V; indicate the indices of the variables that appear in and that appear negated in the clause C₁, respectively. Since {a; het is a satisfying assignment by assumption, we must either have bg/ k+1 (i.e. F.g. is True) for some g' E V, Z or 6g 2 (i.e. zh is False) for some h' E Z₁ If the former is true (ōg = 1 for some
k+1
g' E Vi Z,):
Tels T2MM- -1
Σ
8g
Σ on
k+1
9EVi
bEZ:
1
([V.)
1Z.
1)E
1Zd
2 2/12/| - 1 + 2))
+
3 k+1 1
2
1: + 1
1- Z
2 2 1 + 2(k: + 2)
h+ 1
2
2
++3 1
-2
+
kk
20
7
k
++1 1
7
k+1
Similarly, the latter true (ōn- = for some 1/' E Z,):
Tb - 12 = 2/2 - -
Σ 5g
E
Σ
2
DEVV
NEZ.(N)
- (-2
8
[2:1
-
2|Z
k+3 3 1 k+1
2 2| - 1 + +2)
h:
k+1
1: +
7
- + k + 20
k ++1
which gives the same inequality. Consider two cases: if 2 2 (i.e., hi = =1+ 1 2 4), we get
-
+ k
20
-6 +
24
20
2
[2 - [2(h] 2
7 7
0,
k+1
++1
k+1
since Then,
|2,1 3. If
(i.e., 3), then |2,1 S 1. since t is the number of variables in Φ.
2 -2| + 6k +
20
2 +
18
+
20 7
- -
7
7
7 > 0. 1+1
++1
k+1
In both cases, have T2[g] - [h] > 0 and hence (Tilg) (T2lg) T2(h)) 0, as desired.
Case 6: EE ferh.h h & E. Say 9 = e, for some 1 S <
and ng =
ErEAuR ICI ID Ej= Molf.j] for 8 E 21. Due to the last set of voters that we added while constructing o (1) and o (2) (those that rank only elements of F),, we have
Malen,d] 2 14(k= + 2) max ny 2 14(k+ +2) 7E(1,2)
Mg[h,
j=1
k
2 14(k + 2)
Σ
Ma jlss 14(k + [)T
j=1
Moreover, since 8; 2 Sk-1 1 -
3
K.T we have Talal Molg, > T2(h)) > as
Maleidl [M] for each 3 [1, This gives (Ti[g) k+1 desired.
Case 7: g ck. By construction Make, j > 0 = j== k, and hence Tg6g] = T [ex] Ma k]s; = for each 3 E (1,2]. If E AUBUCUD, we have T [h] >
for both |3 E [1,2} by eqs. (A1) to (D2p h e; for some i E [K - 1), on the other hand, Ma[h, i] >> 0 by the last set of voters added to (()) and o (those that only rank the elements of E), so once again we have [a[h] = Ms[h, ils; > 0. Hence, we have (Ti [o] - T, (T) [g] T2(M)) O, as desired.
28

---

Published as a conference paper at ICLR 2026
Case 8: g= Ct. h = Cj for some i < < k. Fix some 8 (1,2). By construction, o contains two types of rankings: (a) those that contain one elements in AUBUCUD and all remaining elements are those in E, of which there Rg = ΣSEAUBUCUD Σ Ma many and (b) those that rank only and all elements in E, of which there are say ye many. with y/ 2 m 14(k + 2)m,
14(k + 2)ng. Hence, for any j' [k: - 1]. we have ng + yB > M Meyy Ji] ya. where the first inequality is strict since [f
for some
I E A for all j' E [k- 1] (see, for example, the definitions for M3 [ay: +1 and Mayby,33 3'] for each j' E [t], so there least some voters the group of n. that do not rank ey, This implies
Tola - Te(h) = Tole Tole,] = Mole,, i|s; Moles,
> yas; (ng + yo)s; ys(s; 8,) ngs,
2 us(s: S:+1) ng ygo; ng
E
2 14(k + 2)ma
Rt ng ns = 0
2
Hence, we have (g) Ti [k]) (Ta T3 [h]) 0, desired.
We have shown that (Ti Ti [1)l -22(6]) > 0 for all g, h E A, proving that s indeed gives full agreement between o(!) and o (2).
C EXPERIMENT DETAILS
This appendix provides additional details on experiments discussed in Section 7
Several components of our experiments are held constant across all settings. Below we describe these components and provide some general information about our experimental setting. In subse- quent subsections we provide additional detail and discussion around each of our experiments and definitions of each voting rule used throughout this paper.
Number of Splits: With two exceptions, all experiments report the average disagreement resulting from each rule over 10 random splits of voters. The exceptions are: When eval- uating axioms (Appendix C.9 Figure IT) we report the average over 50 splits, and when evaluating score data (Section Table we report the average over 1000 splits. These values were decided based on computational and time constraints.
Splitring Procedure: When creating two groups of voters we assign each voter to one group or the other uniformly at random, as detailed in Box [ In the case that no voters are assigned to one group, we assign that group single weak ranking in which all alternatives are tied (this effectively gives weight of zero to all such splits).
Weighting Partial Rankings: In settings where voters submit partial rankings, we weigh each alternative based the number of times it appears each side of split, as discussed in Section 4 This serves to correct for mismatches in the amount of information known about each alternative (i.e. the number of times each alternative is ranked) between the two sides of the split. Consider a split of a fixed profile or into (1) (2)) via the process in Box [ For any alternative a A, let m. denote the minimum number of times is included in rankings in (1)) or o (2), and let to refer to the total number of times is included in ranking across the entire profile σ. Then the weight of w. me 7
This ensures that the largest weights are assigned to alternatives that are evenly split across the split (m, = 22) whereas any alternative that does not appear on one side of the split (m, = 0) gets zero weight, as it is unreasonable to expect this alternative to be placed consistently across the split by any SWF. All our experiments use a constant factor of 2. Weights are used as a scaling factor in computing the Kendall-Tau distance (as in Kumar & Vassilvitskii 2010); when alternatives and b are in opposite positions across two rankings then they contribute WA W, to the weighted Kendall-Tar distance (as opposed to the unweighted setting where any misordered pair adds 1 to the distance).
29

---

Published as a conference paper at ICLR 2026
0.14
0.14
01
6.105
0.307
6,300
6.115
0,302
0.13
9.10
5/092
0.093
0.059
0.099
0.010
0.88
Postt Roythondd
faroe
Pl
Farrery
Reo Aquitree
Poomeat
moomee Bonda
Best Pasthional
Scorey
denta
Pt. ALEE
rameny
Ten Agortval
Phonaity
Purealty
Ploraits
Drmmnee
(a) Distance between splits of partial rankings for each SWF on reviews of ALMA Cycle 10 project proposals.
(b) Distance between splits of partial rankings for each SWF on reviews AL.MA Cycle project proposals.
Figure 4: Split distance and standard error of several rules on partial rankings over project proposals for the Atacama Large Millimeter Array (ALMA).
Computational Power: All experiments are performed locally on a 2022 M2 Macbook Air with 16 GB of memory. Each individual experiment shown in the paper took between 0.5 and hours to complete.
C.1 GROUND TRUTH AND DISAGREEMENT
In first experiment, evaluate the relationship between the rankings produced by SWFs and the ground truth ranking of a preference distribution.
Both Mallows and Plackett-Luce (PL) preference models naturally correspond to "ideal" ranking. These distributions also have known maximum likelihood estimators: Mallow's 's MLE is the Kemeny function while estimate the MLE of PL preferences using the choix Python library
In our experiment, for each distribution we generate 50 elections with 100 voters and 100 alterna- tives. We set Φ 4.4 for Mallow's preferences and a; (0.((m-i) for Plackett-Luce. Each voter provides a ranking over 10 alternatives, sampled from the relevant distribution, such that each alter- native is ranked by 10 voters. This type of partial ranking aligns with a paper reviewing framework where conference organizers may wish ensure that each paper is reviewed by a certain number of reviewers while each reviewer receives a certain number of assignments.
For each generated election, we do the following for several SWFs: Assign each voter into of two groups, chosen uniformly at random for each voter. Each of these groups is then used as a complete profile to generate weak ranking according the SWF, We measure the KT distance between these two rankings. We report the mean over these 10 splits as the "split distance." We also calculate the KT distance from the ranking generated by the SWF applied to all voters the ground truth ranking (the "ground truth distance").
We plot each pair of distances for each SWF In Figure the MLE of each noise model tends to minimize each distance. In general, there is very strong relationship between the two distances. As the split distance increases, so too does the ground truth distance.
In this experiment the SWFs we evaluated were Kemeny, Plackett-Luce MLE (as implemented by the choix library), Borda Min-Max, and Optimized Positional Scores (see (subsection C.10) in addition to positional scoring rules with the following vectors:
Plurality 01,0,0,
Plurality + Veto (1, 0.5, 0.5...., 0.5,
Veto (1, 1, ...,, 1,0)
Two-Approval (1, 1... 0)
Borda m-
m
30

---

Published as a conference paper at ICLR 2026
0.5
'91-'02
0.46
'03-'09
'10-'18
0.47
0.4
0.37
0.38
0.39
0.41
0.30
Distance
0.35
0.3
0.240.240.24
0.2
0.21
0.17
0:21
0.20
0.15
0.13
0.11
0.1
FI (191 02)
F1 ('03-09)
F1 (110-18)
Best Positional Scores
Borda
Two Approval
Plurality Veto
Plurality
Veto
F1 (191 022)
F1 ( (09)
FI ('10-18)
Best Pos/tional Scoress
Borda
Two Approval
Plurality Veto
Plurality
Vetto
F1 ('91-02)
FI ('03 (09)
FI ('10-18)
Best Positional
Bonda
Two Approval
Plurality Veto Plurality
Veto
Scores
Figure 5: Distance between splits for SWFs aggregating rankings of drivers in F1 races. Each rule
is evaluated on each period races. Newer Fl rules provide lower distance on all race periods.
C.2 ATACAMA LARGE MILLIIETTER ARRAY PROJECT RANKING DATA
While in Table I we consider score data generated in a peer review process, peer review data can also take the form of rankings. Here we consider anonymized rankings provided by evaluators of projects proposed for the Atacama Large Millimeter Array (ALMA) (Meyer et al. 2022). which is the largest ground-basse bbsed radio telescope on earth.
We conduct our experiments on two sets of proposals, from Cycle 10 and Cycle 11. In this review process, each proposer is asked to review. Cycle 10 contains 1635 proposals and 1635 review- ers, while Cycle 11 contains 1729 proposals and 1729 reviewers. In each cycle, every reviewer is assigned proposals to review, and every proposal is assigned to 10 reviewers. Each reviewer provides strict ranking of the 10 proposals assigned to them.
We run AbC on the rankings provided by the reviewers. The mean Kendall-Tau distances over 10 splits for several SWFs displayed in Figure We see pattern consistent with the results of Fig- ure 1 PL MLE, Borda, and optimized positional scores result in very similar distances, suggesting natural lower bound on the Kendall-Tau distance based the data and the noise induced by gen- erating splits. Kemeny rule on the other hand, performs notably worse than this lower bound. Our experiment also finds that better consistency is achieved by positional scoring rules which provide more information about the profile (i.e., Two Approval and Plurality + Veto provide two "bits" of in- formation, compared to one bit from Plurality and Veto, whereas Borda and the optimized positional scores provide significantly more information).
Importantly, our method can be used to evaluate proposed changes to current practices. For example, Meyer et al. (2022) consider removing the minimum & maximum score of each alternative before computing the Borda score, termed 'Trimmed Borda', but our experiment suggests this method increases disagreement their dataset compared to Borda count, significantly more than it did on data generated from known noise models (Figure
C.3 FORMULA ONE DATA
The Fl portion of Figure is generated using data about races found on Preflib Mattei & Walsh (203);; specifically, we use the complete form of dataset ID 00053. This contains one profile for each Formula One season with a preference order corresponding to each individual race the sea- son. Each preference order lists the drivers that competed in every race in that season, ordered by the position which they finished that race.
'htts://choix.. www. wuw..i//n//aats//ndee.. html
Note that, due computational limits, in this setting we use the hest ranking found by the Kemeny function (Appendix C. within time limit of 15 minutes per split.
31

---

Published as a conference paper at ICLR 2026
0364
Third Coordinate
1.0
Optimieed
Distance
0.8
Mostl Count
Loximax
0.30
E214
0.8
Puriliy
Borda
Two Approvst
x
0.25
0.2
Foaddonall
Medal Couse
Loinee
Bou Aupooaal
Sconet
0.0
0.4 0.6 0.8
Second Coordinate
(a) Average distance and standard error between splits for SWFs over rankings induced by Olympic medals.
(b) Distribution Best Positional Score vectors. Each X marker corresponds to the optimized score vector for single Olympic games. Score vectors consist of three values (corresponding to Gold, Silver, Bronze winners) and normalized so that their first coordi- nate is 1. The vector for Leximax is (1, 1.100" 1,0
Optimization of positional scores is only occasionally able to improve upon the ranking generated by giving point to each country for each medal they win,
regardless of medal type.
Figure 6: KT distance of various scoring rules (left) and analysis of our optimization results (right) for Olympic Medal data.
We show in Figure 5 the KT distance for all rules divided by racing period. While the rules with highest distance (Iwo-Approval, Plurality Veto, Plurality, Veto) stay quite consistent, all other rules provider lower KT distances on more recent race periods. Notably, the Fl rules themselves become much more consistent over time.
C.4 OLYMPIC MEDAL DATA
To evaluate rules Olympic medal wins we use Kaggle dataset providing details of all Olympic results (Summer and Winter) between 1 1896 and 2016 Griffinn 2018). From this we extract the winning countries of each medal for each event. We convert each event into a partial ranking by assigning to first place all countries receiving gold medals, to second place all countries receiving silver medals, and to third place all countries receiving bronze medals. Countries that did not win medals or did not compete in an event are not included in ranking.
In the large majority of cases this results in a partial order consisting of exactly three countries, each in different rank. In rare cases a position might be empty or have multiple winners (e.g. in 1992 Canada and USA won Gold in women's solo synchronized swimming, no Silver was awarded, and country might occur multiple times (c.g. in 2008 Jamaica won Gold
Bronze was won by Japan), and two Silver medals in the women's 's 100 metre; no Bronze was awarded). In these exceptional cases we do nothing different and award that country points for each of the positions that it occupies.
This results in one "election" for each year in which the Olympics occurred where preference or- ders correspond partial rankings induced by medal wins. To these rankings we apply the AbC framework as we have in all other experiments: We generate splits by randomly placing the profile induced by each event into one split the other, then find the distance between splits and report the average over many sets of splits. Here we add two new rules:
Medal Count:
Each medal is treated equivalently. Each country receives point each time it appears a preference order, regardless of rank.
Leximax: Rank countries by the number of Gold medals they receive, breaking ties by counting Silver medals, breaking remaining ties by counting Bronze medals. In practice we implement this as the score vector (1000000, 1000, 1).
Results of this analysis are displayed in Figure 6a We that simply counting the total number of medals each country receives results in much lower disagreement than standard voting rules, only very rarely do optimized scores improve upon the Medal Count rule.
Additionally, Figure 6b shows the resulting scoring vectors from our optimization ("Best Positional Scores"). Each positional scoring vector (with the points awarded for receiving Gold, Silver, Bronze respectively) is normalized to have a maximum value of I in the first coordinate (the number of points for each gold medal). The second and third coordinates are placed the figure to map the
32

---

Published as a conference paper at ICLR 2026
resulting score vector values. While optimized scoring vectors vary they consistently award non- zero number points for all medals, and the second coordinate is often approximately equal to either the first coordinate (i.e., vector equal to (1, y)) the third coordinate (i.e., (1, y)). The Medal Count rule is the special case where both of these are true and all medals are worth an equal number of points.
C.5 CITY ELECTION DATA
In Figure 2a we show results of several rules on Preflib data from several real-world political elec- tions. Hach of these elections used Instant Runoff Voting (IRV) compute the empirical winner. Given a profile, IRV iteratively eliminates the alternative with the least plurality score (i.e., with the least number of voters ranking them top) by removing them from the profile, until single alterna- tive remains. As an SWF, we interpret IRV as the rule outputing the alternatives in the reverse of this elimination order (the alternative eliminated first is ranked bottom, and so on). We provide details on each election shown in Table 2 the GPL-3.0 license.
The data from Prefibb can be freely distributed and used under
Initially we collected all elections included in Preflib Elections 5, 16, 17 ,18, 19, 20, 21, and 22. However, Figure 2a we have filtered out two types election from our starting data data order to make the plot more readable:
Any elections where all rules had split distance of O. While this random event, we found that this consistently included set of 10 elections.
Any election with more than 100 candidates. This excluded two specific elections that occurred in 2009 in the city of Minneapolis with 379 and 477 alternatives. These two elections have exceptionally high split distances and were removed as outliers due to their split distance and unusual number of candidates.
Table lists rules in increasing order of KT distance and shows how often each ordering occurred when AbC was run on the 25 elections. As seen there, only two out of the 25 elections result in the same ranking of the rules in terms of their consistency when run on the election. This confirms our intuition from Section that different rules are appropriate for different settings.
C.6 MOVIELENS DATA
The MovieLens dataset (Harper& & Konstan 2015) used widely to evaluate recommender systems. The data consists of 32,000,200 ratings (scores from 0.5 to 5 in increments of 0.5) of 87,585 movies by 200,948 users. We are able to apply ABC to demonstrate how it might serve to generate ordered recommendation balancing quality and consistency. We take subsample of the dataset to reach a tractable amount of data for use by our optimization method. Specifically, we filter the data to include only rankings from the 100 most prolific users, and filter that include only the 100 most rated (across the entire dataset) movies. This results in 9401 ratings across the 100 movies. We convert the scores from each user into an ordinal ranking in order evaluate the SWFs we have used throughout (although one could also use AbC to pick among functions for aggregating cardinal evaluations, as demonstrated Table for peer review data).
Distance between splits on this data is shown in Figure 7 As is common in several of our experi- ments, Borda count outperform other non-optimized score vectors. In Table 4 we show the top and bottom movies in the ranking produced by the Best Positional Scores rule. The most highly ranked movies (of the 100 movies receiving the most rankings) consist of extremely well-known "classics" while the movies ranked at the bottom of the list are also well-known but may have less wide appeal (Ace Ventura and The Mask are primarily targeted at children) or were quite contentious at time of release (Star Wars: Episode I).
C.7 ALTERNATIVE DISTANCE FUNCTIONS
As mentioned in Section the ABC' framework can also be used in conjunction with distance functions other than the (weighted) Kendall-Tau distance. This can be especially for settings where the goal is not necessarily to rank all items but, instead, to select a set of winners or satisfy other criteria.
33

---

Published as a conference paper at ICLR 2026
Plot Index
Preflib Election
Flection Name
Voters
Candidates
ID
5
City (2009 Burlington
Mayoral
8980
Election)
1
16
City (Aspen City Council 2009)
2477
11
2
16
City (Aspen Mayor 2009)
2527
3
17
City (2010 Berkeley City Council
4173
4
District 7)
4
18
City (2009 Minneapolis Board of Estimate and Taxation Election
32086
7
No Write In)
18
City (2009 Minneapolis Park and Recreation Commissioner At-Large
36655
9
Election No Write In)
6
19
City (2010 Oakland Mayor)
119256
11
7
19
City (2010 Oakland City Council
20981
8
District
8
19
City (2012 Oakland City Council
22079
7
District
9
19
City (M2012 Oakland City Council
28660
8
District 1)
10
20
City (2008 Pierce County Assessor
262312
7
Treasurer)
11
21
City (2011 San Francisco Mayor)
194530
25
12
21
City (2011 San Francisco Sheriff) City (2012 San Francisco Board of Supervisors- District 5)
183192
5
13
21
35183
9
14
21
City (San Francisco Board of Su-
31437
10
pervisors District 7)
15
21
City (2010 San Francisco Board of Supervisors District 2)
24109
16
21
City (2008 San Francisco Board of Supervisors- District 9)
26634
17
City (2008 San Francisco Board of Supervisors District 3)
27310
18
21
City (2008 San Francisco Board of
28777
Supervisors District I)
19
21
City (2010 San Francisco Board of
18001
22
Supervisors District 10)
20
21
City (2008 San Francisco Board of
24717
10
Supervisors District 11)
21
21
City (2010 San Francisco Board of Supervisors District
21188
15
22
22
City (2010 San Leandro Mayor)
22407
7
23
22
City (2012 San Leandro City Coun-
23236
cil District 4)
24
22
City (2012 San Leandro City Coun-
25355
cil District 2)
Table 2: Details on the election represented at each index Figure 2a
C.7.1 JACCARD DISTANCE
Project proposals submitted to the Atacama Large Millimeter Array (ALMA) (Meyer et al., 2022) are ranked by several reviewers then either selected, or not selected.. Here, the goal is not necessarily generate aggregate ranking over all proposals, but pick & proposals to be funded, for some
34

---

Published as a conference paper at ICLR 2026
Count
Rule Order
Two Approval > Veto > Plurality > IRV > Borda > Plurality + Veto
IRV > Plurality + Veto Borda Veto
Plurality > Two Approval
Plurality Veto > IRV > Veto > Plurality > Borda > Two Approval
IRV > Two Approval > Plurality Veto > Plurality Veto Borda
Borda > Plurality Veto > Veto > Plurality Two Approval > IRV
Plurality + Veto > Plurality IRV > Borda > Veto > Two Approval
Veto Plurality Veto Two Approval Plurality > IRV Borda
IRV > Two Approval > Plurality Veto > Plurality > Borda > Veto
Two Approval > Borda Plurality Plurality Veto > IRV Veto
IRV > Plurality + Veto > Plurality > Veto > Borda > Two Approval
IRV Two Approval Borda Plurality Veto Veto Plurality
Borda > Veto > Two Approval > Plurality + Veto > IRV > Plurality
Plurality Veto Two Approval > Veto > Borda IRV Plurality
Two Approval > Borda > IRV Plurality + Veto > Plurality > Veto
Two Approval > Veto > Plurality Veto > IRV > Borda > Plurality
Borda > Two Approval > Veto IRV Plurality > Plurality + Veto
Plurality > Plurality + Veto Two Approval > IRV Borda Veto
1
Plurality > IRV Plurality Veto > Two Approval Borda Veto
Borda > Veto > IRV > Plurality > Plurality Veto > Two Approval
1
Two Approval Plurality + Veto > Veto > IRV > Borda > Plurality
Two Approval > Veto > Plurality + Veto > Borda > Plurality > IRV
Borda Veto Two Approval > IRV Plurality Veto Plurality
Borda > Veto > Plurality > Two Approval > Plurality + Veto > IRV
Plurality Plurality Veto Borda Veto Two Approval > IRV
Table 3: The frequency of each rule ordering resulting from running AbC across 25 political elec- tions, Rules are listed in increasing order of KT distance.
0.5
Distance
0.3
0,248
0.2
2.155
0:156
0.1
0.0
Best Positional Scores
Borda
Two Approval
Pturaliyy
Purrlity Veta
Veto
Figure 7: KT distance between splits for SWFs aggregating ratings of movies.
35

---

Published as a conference paper at ICLR 2026
Rank
Movie
The Godfather (1972)
2
Pulp Fiction (1994)
3
Indiana Jones and the Raiders of the Lost Ark (1981)
Schindler's List (1993)
5
Star Wars: Episode V The Empire Strikes Back (1980)
m
4
Pretty Woman (1990)
17 Mrs. Doubtfire (1993)
2
Star Wars: Episode The Phantom Menace (1999)
m The Mask (1994)
Ace Ventura: Pet Detective (1994)
Table 4: Top and bottom 5 movies appearing in the ranking produced by the Best Positional Score vector.
3.0
0.992
0.936
0.831
0.24
0.051
0.59
0.813
9:514
0,834
0.829
0.8
0.8
Bust Pouslann
Bera
Pt ALE
Hemwey
Agpiova
Thmmmed Borda
Brcrt Hasttonal
Borsl
ALLE
Femmrr
Pionaity 1000
Aqunoval
Phonalty
Vuloo
Brommned Borta
Phoaeity
(a) Jaceard dissimilarity between sets of winners for splits of partial rankings of ALMA Cycle 10 project proposals.
(b) Jaccard dissimilarity between sets of winners for splits of partial rankings of ALMA Cycle 11 project proposals.
Figure 8: Jaccard dissimilarity and standard eπror of several rules on sets of winners using partial rankings over project proposals for the Atacama Large Millimeter Array (ALMA). Winners of a rule are the first 240 proposals in the ranking output by that rule.
k < m. It may thus be more useful to study the top k proposals picked by each rule. Indeed, we explore this possibility by using the Jaccard dissimilarity (Jaccard 1901) measure the similarity of winning proposals on ALMA data.
Jaccard index measures the overlap of two sets; it is defined as the ratio of the size their intersection over the size of their union. Jaccard dissimilarity is then I minus the Jaccard index. For each voting rule that we evaluate, we select the 240 highest ranked proposals as the winning proposals (chosen to match the number of winning proposals chosen in ALMA Cycle 10).
In Figure we see the Jaceard dissimilarity for several rules on proposal rankings from Cycle and 11 of ALMA. The resulting disagreements have similar relative order to the Kendall-Tauu distance of the rules applied to the same data (Figure in particular, we once again see that Trimmed Borda hurts consistency in comparison to Borda count.
C.7.2 NORMALIZED DISCOUNTED CUMULATIVE GAIN
Normalized Discounted Cumulative Gain (NDCG) metric of ranking quality typically applied to search engine or information retrieval settings. but also used in recommender systems (Järvelin & Kekäläinen 2002). NDCG evaluates ranking based on the "relevance" of an item being ranked in a particular position, based on some ground truth ordering. As we do not have ground truth against which we can evaluate a ranking, we measure the distance between a split of data by alternately treating the ranking resulting from each side of the split of the data as the optimal ranking (i.e.. we measure NDCG when one side the split is the ground truth ranking and the other side is the ranking being evaluated. We then measure NDCG again when the roles of the two sides switched. Finally, we compute the average between these two values). For m total items being ranked we say that the relevance of the item being ranked in position is That is, for rankings
36

---

Published as a conference paper at ICLR 2026
0.08
0.063
Distance
0.06
0.047
0.036
0.035
0.036
0.04
0.030
0.024
0.017
0.02
0.00
F1 ('91-02)
F1 (03300)
F1 ('10-18)
Best Positional Scores
Two Approval
Plurality
Borda
Plurality Veto
Veto
Figure 9: NDCG distance between splits for SWFs aggregating rankings of drivers in Fl races. F1 vectors are evaluated for only the years in which that rule was in use.
(21, generated by a rule for split preference data we say that the distance between these rankings is equal to 1 - 2 (NDCG r2) N (1))),,)))), where N NDCG Topt) denotes the NDCG of when measured against the optimal ordering Topt-
We calculate the NDCG score for variety of voting rules Formula One race data and present the results in Figure 9 While prior trends hold in that optimization of score vectors provide significant improvement
most existing scoring rules (excepting the F1 (2010-2018) vector when applied only to races during those years), there is variation among the best performing non-optimized rules. While Borda scores have resulted in low distance on most other settings we find that Two Approval and Plurality are more effective at reducing NDCG distance, as both rules naturally focus on the ranking of alternatives that have been ranked top by most voters (and therefore will likely end near the top of the output ranking).
C.8 DIFFERENT TIE WEIGIITS
As mentioned in Section we explicitly add a term in our definition for KT distance for ties in order to penalize indecisiveness (e.g., rule that always returns all alternatives tied). In all exper- iments above, ties in a ranking add to the KT distance 1 that of a strict disagreement, which is inspired by Kendall's Tau-b correlation coefficient (Kendall 1945). Figure TO reports the results of rerunning our experiment on Fl data (Figure 2b) while using different weights for the ties. The qualitative results identical for the tie weights {0.5, 0.75, 1} terms of the relative order- ing of the Fl vectors, as well as in terms of the relative ordering of the named positional scoring vectors. For smaller tie weights (in {0, 0.5)) is worth noting that the distances associated with Two-Approval, Plurality, Veto, and Plurality+ Veto are significantly lower than they are for higher tie weights, whereas the distances associated with Borda count stay relatively constant across different tie weights. This makes intuitive sense, as the former four rules are more likely result in final rankings with ties since each voter gives the same score to every alternative except one (for Plurality and Veto) two (for Two-Approval and Plurality+ Veto). In particular, all alternatives that have not won first place (resp. last place) in any race will tied by Plurality (resp. by Veto) for every possible split of the profile. It is worth pointing that different weights for the ties will result in different "Best Positional Scores" since our optimization minimizes the loss function that includes this weighted term for ties.
C.9 AXIOM VIOLATIONS
To further explore the theoretical results of Section Se analyze experimentally how often certain axioms are violated across several distinct preference distributions. Results of this experiment are found in Figure 11 We measure the axiom violation rate, as defined by Caiata et al. (2025); an experimental measure of how often axioms are violated in practice on given preference distributions and voting rules. For this experiment we sample, for each distribution, 500 profiles with 100 voters that each provide a full ranking over m E 5, 10, 15, 20 alternatives from the Impartial Culture, Urn (with sampled from Gamma distribution with shape k 0.8 and scale 0 Boehmer
37

---

Published as a conference paper at ICLR 2026
Weight
Weight
0.264
Distance
0.2
Distance
0.185
0.208
0.200
0.115
098
0.2
0.062
0071
0.1
0.008
0.008
0.199
0.0
(1101000))
rat: ow)
Ft rxp 18)
Best Proetimal Storiss
Ploradty
Nrm Appemeal
Purally Vero
Burta
0.0
(1199-022)
F3 (9333-00
(10-10
Partttonall
Scorgs
Twi Appnmell
Piomaity
Bovda
Pinnklyr Veto
Veto
(a) Tie weight
Tie weight 0,25
Formule One
Weight
Formula One Weight 0.75
8.6
0,470
0.8
Distance
0.353
0.387
0.407
Distance
0.542
0.25
0.2
0.183
0.157
0.207
0.149
0.162
0.2
0.0
Ft (31-3-02
P1 (or0-ag
Fl (10 18)
Rotitional
Brooaa
Approval
Hefo
Ruralit
Welo
0.0
PX P11-9-03
Ft P03
F1 P10-18)
Postional
Bordi
Tort Apprnvval
Phinedyy
Into
Piiality
Pluraity:
(c) Tie weight 0.5
Tie weight 0.75
Formuls One The Weigh
1.00
0.802
6.872
Distance
0.75
0.30
0.243
0.25
FI 192--222
FI ra3
F1 (10-10
Best Poational
Scores
Borde
Peo Approval
Vite
Piirlity
Weto
Parality
(c) Tie weight
Figure 10: KT distance between splits for SWFs aggregating rankings of drivers in F1 races, when ties weighted by different coefficients (Definition 3). F1 rules evaluated for only the years in which that rule was in use.
Importial
Single
PrefLib
0.15
-w- Usion Consisency
Reversal Symmetry
-
Monalonally
Violttion
0.03
Number of Alternatives
Figure 11: Violation rate of axioms across several preference distributions and real-world election data (rightmost). Axiom violations generally decrease as the number of alternatives increases.
et al. (2021)), Mallows (Øb 0.4), Plackett-Luce (a; = =.5mmm-))) and Single-Peaked preference distributions Brandt et al. (2016). Additionally, we consider all elections with complete preferences from PrefLib with up to 1000 voters and alternatives for m < 20 (a total of 1392 elections) Mattei & Walsh (201)). This real-world election data is compiled from a wide variety sources with varying underlying preference distributions.
On each election we take 50 random splits and calculate the fraction of splits on which each axiom violated by Rule Picking Rule using several positional scoring rules.
As we evaluate the Reversal Symmetry axiom we are constrained to using positional scoring rules in our Rule Picking Rule. For all axioms, RPR uses the following rules: Plurality, Plurality
38

---

Published as a conference paper at ICLR 2026
Veto, Veto, Two-Approval, and Borda count. The scoring vector associated with each of these rules given Appendix C.1
We provide example to illustrate how measure axiom violations.
Example. Consider the monotonicity axiom. In a profile o and a modification a where some voters increase the rank they give to alternative Monotonicity is violated if the rank of under the RPR lower in aa than in a.
We test for violation of monotonicity in profile by selecting the alternative ranked first in by the RPR, selecting some uniform random fraction of voters in (0.2, 0.8) to increase their ranking of If the RPR 0, assigns different rank report the fraction of instances tested in which the axiom violated.
say that the axiom has been violated. We
As shown in Proposition 10 (which easily extends to implementation using Monte Carlo sam- pling), reversal symmetry is never violated and only checked as a test. Moreover, under all but one distribution, violations of union consistency and monotonicity are rare (< 0.05 of the time for 2 10 alternatives) and generally decrease as the number of alternatives increases
C.10 ADDITIONAL DEFINITIONS OF VOTING RULES
Through our experiments we use several rules which are not fully described. In this section describe each rule found throughout our paper.
C.10.1 KEMENY
Kemeny's rule is defined as the ranking which minimizes the sum Kendall-Tau distances between the output ranking and each voter's 's individual ranking Kemeny| (1959). We implement the Kemeny method code provided by Bahaarev et al. (202I) and the Gurobi optimization library (Baharev et al.,
2021
Gurobi Optimization, LLC 2024).
C.10.2 TRIMMED BORDA
Calculate the number of voters ranking each alternative at each rank. For each alternative a,, remove of the highest rankings which a; has received and remove of the lowest rankings which a; has received. After all removals are complete, calculate Borda scores as normal (Meyer et al. 2022
1.3..3 BEST POSITIONAL SCORES (SIMULATED ANNEALING)
This method uses the opt imal vot ing package generate a positional scoring vector which min- imizes KT distance between splits on a given profile (Armstrong 2025). This package uses sim- ulated annealing
generate novel positional scoring rules which optimize target function. In each of our experiments, we run optimization multiple times; starting once from the initial score vector of each other positional scoring rule used the experiment. For instance, if we were com- paring with Borda's 's rule and Plurality we run optimization twice, starting from (m 1, m 2, 0) (Borda), and (1, O,..,..., 0) (Plurality). In all cases, we run annealing for 500 steps. At each step, optimal -voting updates index in the state vector (i.e., the positional score vector) by some amount sampled uniformly at random from (0.05, 1). We restrict updates to those that result in the state vector being weakly monotonically decreasing (the value at each index is not higher than the previous value). Using the updated state vector as a positional scoring vector we calculate the mean KT split distance over each split of voters. We use the same set of splits in annealing as do when evaluating each other voting rule. The new state is accepted with a probability related to the magnitude of difference between the current and previous mean split distances, and the number of steps that have already occurred. A small magnitude of difference is more likely to be accepted earlier than later. States with lower mean split distance are always accepted.
'The one exception to this trend is monotonicity in the Impartial Culture (IC) distribution. which is not centered around ground truth. One possible explanation for this is IC maximizing the probability of majority cycles (Teetlin al. 2003), under which monotonicity violations more likely occur.
39

---

Published as a conference paper at ICLR 2026
C.10.4 STOCHASTIC GRADIENT DESCENT ON POSITIONAL SCORES
In Figure 3 use Stochastic Gradient Descent (SGD) find a positional score vector which min- imizes KT distance between sampled splits on a given profile. This method is included largely for comparison with the above approach which uses Simulated Annealing optimize score vectors for the same metric. While both methods are effective at finding optimized positional scoring vec- tors, we find Simulated Annealing achieves results of slightly better quality with significantly less compute time.
In this method we apply the standard SGD approach, summing the KT distance over all pairs of profile splits generate the loss of profile. Recall that [/] is the total score of alternative under score vector for profile p (Definition I.
We approximate the KT distance between two rankings 12 using the sigmoid function. Note that here a refers to the sigmoid function rather than a profile and y some constant scaling used to scale the sigmoid function:
KTSGD(TI,12)
Σ E (( (1) tr. ()))(, Sr, (i)))
jEAjEAj
As q - oo, the sigmoid approaches the step functions, and KTSGG approaches KT (Definition 3 Rankings ri and 12 are calculated from the score vector for each pair of splits, and KTsan calculated for each pair rankings. This then normalized by the sum over each product of weights w,Wy, which is the maximum value of KTSGD achievable for that given split. The total loss is the normalized sum of KTSGD over all splits in a profile.
We use the SGD Optimizer provided by Pytorch (Paszke et al. 2019) to perform gradient calcu- lations. We initialize the scaling factor = 1000000 and the learning rate to 0.1. Learning rate is divided by 2 if the loss from step to the next increases by more than 0.1 and has minimum value 10 6
C.10.5 LEXIMAX
This rule is used exclusively in evaluating Olympic data. For a given profile, alternatives are ranked according to the number of gold medals they have received. To break ties, tied alternatives ordered according to the number silver medals they have received. tied alternatives are ordered according to the number of bronze medals they have received. Note that we can instantiate Leximax as a positional scoring rule where each position much larger than the subsequent position. In Olympic experiments we use the (pre-normalization) vector (1000000, 1000, 1). As there are never more than 1000 opportunities for a single country to receive a medal of one type this is equivalent Leximax.
break subsequent ties,
1.10.6 MEDAL COUNT
This rule is used exclusively in evaluating Olympic data. For given profile, alternatives ranked purely based on the number of medals they received with no regard for the type of medals.
Our Olympies data uses as input the partial rankings containing only countries winning medals in an event. this case, the Medal Count rule is equivalent to the positional scoring rule with the vector (1, 1, 1).
40


Reviews:
[{"review_id": "kNXKQHIxbf", "reviewer": "Reviewer_6kf7", "summary": "The paper tackles the problem of selecting the best rank aggregation method in the setting where different aggregation rules can lead to very different outcomes. To address this problem, the authors introduce the concept of a Rule Picking Rule (RPR), a meta-framework for deciding which aggregation method to use for a given dataset without assuming any specific generative model.  Instead of relying on fixed axioms or assumed noise models, the authors  provide a data-driven approach for picking aggregation rules. They propose a new RPR, called \"AbC\", which selects the aggregation method in such a way to maximize consistency, that is to say, producing stable results if the data collection were repeated. The authors show that AbC satisfies certain consistency-related axioms that some natural RPRs fail to meet.  For two axioms it cannot satisfy, the authors prove impossibility results, showing no rule can satisfy them simultaneously. Additionally they show that the problem of finding a perfectly consistent rule is NP-complete. Despite this hardness result, the authors have implemented a sampling-based algorithm that seems to perform reasonably well, in the sense that it outperforms some commonly used aggregation methods. The implementation also provide interpretable insights into when and why specific rules work best. It is stated that an implementation of AbC was awarded in a recent competition\nat a top-tier AI conference.", "questions": "It is stated that an implementation of AbC was awarded in a recent competition at a top-tier AI conference.\n\n1) What place/score did the mentioned implementation get? 1st place, 2nd place, 3rd place? \n2) How many competitors were there in the competition? \n\nWithout more details it is difficult to judge the strength of the implementation with respect to competing approaches.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is that the proposed RPR has been implemented and evaluated in practice, and obtained one of the top scores in a relevant AI competition.", "weaknesses": "The theoretical part is dense and most proofs are relegated to the appendix, which is quite long (22 pages). Therefore, verifying the correctness of the claims is a bit tricky. I did not verify the correctness of the proofs in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vPoHxc06Cg", "reviewer": "Reviewer_aygk", "summary": "The paper introduces Rule Picking Rules (RPRs) : procedures that choose which aggregation rule to use for a given dataset; and proposes Aggregation by Consistency (AbC), which selects the rule whose outcomes are most stable across random evaluator splits. \n\nIt develops an axiomatic basis (e.g., reversal symmetry, shuffling consistency), proves that achieving perfect consistency over positional scoring rules is NP-complete, and provides a computationally practical Monte-Carlo implementation.\n\nAbC (i) often rediscovers the model-MLE when the generative model matches the data, (ii) yields actionable guidance on rating aggregations, and (iii) extends to top-k selection via set-based distances.", "questions": "Did AbC ever select a rule whose behavior appears to be intermediate between known positional scoring rules (e.g., between Kemeny and Borda)? Did it ever select a rule not a combination of existing rules?\n\nWhen optimizing over positional weights to minimize split-disagreement, do the resulting weights cluster around classical forms (e.g., linear, exponential), or is there a continuum?\n\nAre there theoretical conditions under which AbC provably recovers Kemeny or PL-MLE — beyond the empirical confirmation on Mallows/PL data?\n\nIs there any identifiable set of axioms that exactly describe the rules that minimize disagreement under random resampling? I.e can you simulate an uniqueness axiom?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and answers an important problem, especially motivated by the Neurips paper acceptance experiment.\n\nThe axiom satisfactions depict that the proposed method meets sanity checks. \n\nThe proofs in the appendix seem thorough.\n\nThe monte carlo estimation makes the implementation practical.", "weaknesses": "The paper focuses on Kendall's Tau as the rank comparision metric. Metrics like Cumulative Gain (CG), Discounted CG (DCG), and Normalized DCG are popular for comparing top heavy ranks. A discussion involving those metrics and how the algorithms fares to them would be good to have.\n\nThe empirical section is somewhat narrowly scoped. Although they show correlation between disagreement and distance to ground truth in the synthetic models, no end-to-end evaluation links “most consistent rule” to “best downstream decision quality.” For instance, in peer review: does the AbC-selected rule produce higher eventual citation impact or expert consensus? \n\nMoreover, the real-world data are both from astronomy. It would be helpful to experiment on a diverse set of domains (e.g., recommendation rankings, RLHF preference data, crowd judgments - all readily available) to show generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BPhoJZppFU", "reviewer": "Reviewer_iZ34", "summary": "This paper studies a Rules to Pick a Rule framework: given a ranking profile, what is the ``best'' social choice function to make an aggregation? This paper proposes a surprisingly simple yet effective method: split the profile into two halves u.a.r, run SCF candidates on both halves, and pick the SCF where two outcomes agrees the most (i.e. smallest Kendall-Tau distance between two aggregated rankings). The basic idea is that a voting rule should be consistent among ranking data drawn from the same distribution. Theoretically, this paper considers a series of axioms where their consistence RPR satisfies or those nobody can do better. Empirically, the show the Monte-Carlo approximation of their RPR presents pretty good outcome on datasets.", "questions": "1. What is the specific reason to motivate and interpret as a \"rule to pick a rule\"? (Weakness 1)\n2. How would you answer to this incosistency and justify your axiomatic analysis?  (Weakness 2)\n3. Are there any references to the given consistency axioms? Specifically, why \"reverse symmetry\" sounds so important? At first glance it does not look like a fundamental axiom like anonymity or neutrality. \n4. Is your AbC rule not welfare maximization? If so, do you have any comments on this? \n\nI am open to discussion and happy to raise my score if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is quite impressive to me. In the introduction, the paper exhibits a very strong connection with previous studies and place itself well among them. They also clearly justified how their key idea -- consistency -- is natural and intuitive in many interdiscinpline research topics. I don't check the proofs, but the theorems sounds reasonable at a high level. This paper give a reasonably significant conceptual contribution in rank aggregation with a suprisingly simple method, which I think is the most valuable part of this paper. Besides that, it has a thorough picture, from theory, to computation, and to emprical studies. The topic is also very related to ICLR.", "weaknesses": "I am not 100% percent convinced by the idea of \"rules to pick a rule\". For me, it sounds perfectly ok to interpret it as a another type of rank aggregation rules (as the paper said, the rule RPR induces), which brings more explainability on \"we always align with (one of) the SCF with highest consistency\".  Specifically, given the method is profile-based, the rule elected is used solely for this data. What is the specific reason to motivate and interpret as a \"rule to pick a rule\"?\n\nAnother concern is that, I feel a little bit inconsistent on the motivation of this paper and the approach it adopts. Consider two area of works. (1) Data driven rank aggregation like preference learning and RLHF. In these scenarios, rankings are all datas, and what rules to pick wholely relies on your goal (like maximizing consistency or minimizing other loss functions). (2) Traditional social choice. In this area, \"rules to pick a rule\" based on voting profiles is very problematic as a strong signal of manipulation. This paper adopts the scenarios in (1) yet adopt the methodology in (2). Given that the overall goal of this paper is to justify \"consistency is a gold standard on picking SCF\", the axiomatic approach, which sounds more like \"consistency brings natural behaviors under certain hypothetical scenarios\", is not a strong support to the paper. How would you answer to this incosistency and justify your axiomatic analysis?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PewdcBAQIF", "reviewer": "Reviewer_FWni", "summary": "In all types of domains, individuals disagree. This disagreement is fundamental. As a result, rank aggregation rules must make tradeoffs, picking between different axioms that contradict each other to provide the best output. However, each methods tradeoffs mean that they are acceptable in different situations, and the correct choice of method will vary across domain and task. As a result, the authors here propose AbC (Aggregation by Consistency), a rule-picking rule, which provides a principled way to pick between existing methods. The authors propose a method of splitting data and comparing the consistency of given methods across those splits, allowing for the choice of a method well-suited for any given problem. The method splits voters (the authors explain how this can be used with incomplete preferences), and runs each chosen candidate rule on the split. They measure the Kendall-Tau of each method on each split, and pick the rule with the minimum expected disagreement. \n\nIn order to show the utility of their method, the authors explain what axioms their method satisfies, provide a few examples, and conclude with a set of experiments where they demonstrate AbC on a synthetic dataset, peer review dataset, and a F1 dataset.", "questions": "One thing I wanted to ask the authors was about the use of 1/2 in place of a tie. Did you try any other values there, or measure any outcomes? In many settings such as RLHF, preferences are sparse. It does not seem unreasonable that one would want to avoid penalizing ties, or perhaps even the opposite.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "*The problem the authors address is a fairly foundational one. Any given ranking rule must make choices on what to prioritize. These choices cannot generalize perfectly. A principled way of making that choice is a strong contribution and one that has a widespread impacts in many domains(as the authors note)\n\n*While I have not checked the author's math in detail, it appears to be well-formulated and correct. The appendix contains a great deal of rigorous proofs to back up the author's assertions.\n\n*The authors do a strong job not only in demonstrating the strengths and axioms their rules fulfill, but explain why they cannot fulfill the others. I did not find myself questioning any of those choices.\n\n*The authors are explicit about which axioms they satisfy( reversal symmetry, plurality-shuffling consistency, unconditional anonymity, and neutrality).\n\n*As an overall comment, I was satisfied with the writing of the paper, which was well-written and seemed to me very considered throughout.", "weaknesses": "The piece that stood out to me were the figures in the paper. They did not add much clarity to me, and in many cases they added to my confusion. Understanding the content of one given figure involves inspecting the appendix. What's good and what isn't doesn't jump out, and the number of methods felt like clutter and made it difficult for me to really get a sense of how each method performed. The labels are small and hard to read, 2A in particular is hard to get a sense of.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)