The Surprising Effectiveness of SP Voting with Partial Preferences
Authors: Hadi Hosseini
Debmalya Mandal
Amrit Puhan
Venue: NeurIPS 2024 poster (2024)
Keywords: Surprisingly Popular Algorithm
Preference Aggregation
Partial Rankings
Primary area: machine_learning_for_social_sciences
TL;DR: We extend surprisingly popular algorithm to partial preferences, and evaluate our approach through crowdsourcing.

Abstract: We consider the problem of recovering the ground truth ordering (ranking, top-$k$, or others) over a large number of alternatives. 
The wisdom of crowd is a heuristic approach based on Condorcet's Jury theorem to address this problem through collective opinions.
This approach fails to recover the ground truth when the majority of the crowd is misinformed. The \emph{surprisingly popular} (SP) algorithm~\citep{prelec2017solution} is an alternative approach that is able to recover the ground truth even when experts are in minority. The SP algorithm requires the voters to predict other voters' report in the form of a full probability distribution over all rankings of alternatives. However, when the number of alternatives, $m$, is large, eliciting the prediction report or even the vote over $m$ alternatives might be too costly. 
In this paper, we design a scalable alternative of the SP algorithm which only requires eliciting partial preferences from the voters, and propose new variants of the SP algorithm. In particular, we propose two versions---\emph{Aggregated-SP} and \emph{Partial-SP}---that ask voters to report vote and prediction on a subset of size $k$ ($\ll m$) in terms of top alternative, partial rank, or an approval set. Through a large-scale crowdsourcing experiment on MTurk, we show that both of our approaches outperform conventional preference aggregation algorithms for the recovery of ground truth rankings, when measured in terms of Kendall-Tau distance and Spearman's $\rho$. We further analyze the collected data and demonstrate that voters' behavior in the experiment, including the minority of the experts, and the SP phenomenon, can be correctly simulated by a  concentric mixtures of Mallows model. Finally, we provide theoretical bounds on the sample complexity of SP algorithms with partial rankings to demonstrate the theoretical guarantees of the proposed methods.

The Surprising Effectiveness of SP Voting with Partial Preferences
Hadi Hosseini
Debmalya Mandal
College of Information Sciences and Technology Penn State University, USA hadi@psu. edu
Department of Computer Science
University of Warwick, UK
Debmalya. Mandal wwwaalewwwwcccccmmmmm ac. uk
Amrit Puhan
College of Information Sciences and Technology Penn State University, USA avp6267@psu. edu
Abstract
We consider the problem of recovering the ground truth ordering (ranking, top-k, or others) over a large number of alternatives. The wisdom of crowd a heuristic approach based Condorcet's Jury theorem to address this problem through collective opinions. This approach fails to recover the ground truth when the majority of the crowd misinformed. The surprisingly popular (SP) algorithm [36] is an alternative approach that is able to recover the ground truth even when experts in minority. The SP algorithm requires the voters predict other voters' report in the form of full probability distribution over all rankings of alternatives. However, when the number of alternatives, m, is large. eliciting the prediction report or even the vote over m alternatives might be too costly. In this paper, we design a scalable alternative of the SP algorithm which only requires eliciting partial preferences from the voters, and propose new variants of the SP algorithm. In particular, we propose two versions Aggreggteddrg and Partial-SP- -hat ask voters to report vote and prediction on subset of size k (& m) terms of top alternative, partial rank, or an approval set. Through a large-scale crowdsourcing experiment on MTTurk, we show that both of our approaches outperform conventional preference aggregation algorithms for the recovery of ground truth rankings, when measured in terms of Kendall-Tie Tau distance and Spearman's 's p. We further analyze the collected data and demonstrate that voters' behavior in the experiment, including the minority of the experts, and the SP phenomenon, can be correctly simulated by a concentric mixtures Mallows model. Finally, provide theoretical bounds the sample complexity of SP algorithms with partial rankings to demonstrate the theoretical guarantees of the proposed methods.
Introduction
The wisdom of the crowds is systematic approach to statistically combine the opinions of a diverse group of (non-expert) individuals to achieve final collective truth. It dates back Sir Francis Galton's 's observation- on Aristotle's 's hypothesis-that the point estimation of a continuous value using the noisy individual opinions can recover its true value with high accuracy [22]. In the modern era, the wisdom of the crowd has been the foundation of legal, political, and social systems with the premise that one can recover the truth by collecting the opinion of large number of diverse individuals (e.g. trial by jury, election polling, and Q&A platforms such as Reddit/Quora).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The formal arguments of this crenomenon--ootie in social choice theory- is provided by Con- dorcer's 's Jury Theorem [20]. which states that under the condition of independent opinions and each individual having more than 50% chance of selecting the correct answer, the probability of the majority decision being correct increases as the size of the crowd increases. However, this approach may fail when the majority of the crowd are misinformed (less than 50% chance of selecting the correct answer) [19] or are systematically biased [46]. In other words, when the experts are in minority, simply aggregating individuals' opinions (regardless of the aggregation method) cannot recover the truth.
To overcome this challenge, Prelec et al. [36] proposed a simple, yet effective, method called the Surprisingly Popular (SP) algorithm, which is able to uncover the ground truth even when the majority opinion is wrong. The approach works by asking each individual about their opinion (the vote) along with an additional mela-question to predict the majority opinion of other individuals (the prediction). The surprisingly popular algorithm then selects an answer whose actual frequency in the votes is greater than its average predicted frequency, and will provably recover the correct answer with probability 1, as the number of individuals grows in the limit, even when experts are in minority.
While the SP algorithm is effective in estimating a continuous value (e.g. the value of a painting) or a binary vote (e.g. "Is São Paulo the capital of Brazil?"), it cannot be directly applied to recover true ordinal rankings over a set of m alternatives due to the large number of votes (m!), and more importantly, eliciting predictions over a complete rankings. Hosseini et al. [25] extended this approach to rankings by proposing an algorithm, called Surprisingly Popular Voting, that can accurately recover the ground-truth ranking over multiple alternatives by eliciting a complete ranking as vote and only a single majority prediction (as opposed to full probability distributions over m! rankings). Despite its success in finding the ground-truth ranking over a small number of alternatives, it remains unclear how to adapt it to settings with large number of alternatives where only partial preferences (e.g, pairwise comparisons or partial ranks) can be elicited. Thus, it raises the following questions:
How can we design scalable algorithums based on the surprisingly popular method that recovers the ground truth only by eliciting partial preferences from voters? ? What elicitation formats and aggregation algorithms are more effective in recover- ing the full ranking over all alternatives?
Our Contributions. We focus on developing methods, based on the surprisingly popular approach, that only elicit partial vote and prediction information to find the full ranking. Given set of m alternatives, we ask individuals to provide their rank-ordered vote and predictions on a subset of size k (< m) of alternatives. Informally, we ask them to identify the most preferred alternative among the choices (Top), select the 1 < most preferred alternatives with no order (Approval (t)), or provide rank-ordered list of all k alternatives (Rank). The precise formulation is provided in Section 2.
Given that the SP algorithm [36] and its extension to rankings [25] do not generalize to partial preferences with large number of alternatives, we design two novel aggregation methods, namely Partial-SP and Aggregated algorithms. On high level, these algorithms use carefully crafted method to select subsets of size k < m for vote and prediction elicitation, and apply the SP method either independently on each subset (Partial-SP) or on the aggregated (potentially partial) votes and ranks (Aggregated-S))
We conduct a human-subject study with 432 participants recruited from Amazon's 's Mechanical Turk (MTurk) to empirically evaluate the performance of our SP algorithms using metrics such as the Kendall- Tau distance from the full ground truth ranking and Spearman's 's rank correlation coefficient. We consider several classical vote aggregation methods (c.g. Borda, Copeland, Maximin, Schulze) as benchmarks-roott in the computational social choice theory- that operate solely on votes (and not prediction information). Our results show that the SP voting algorithms perform significantly better than the classical methods when the vote and prediction information only contain partial rankings. We also observe that SP voting algorithms are effective even when restricted to approval votes.
Moreover, we demonstrate that voters' behavior in the experiment, including the minority of the experts can be correctly simulated by a concentric mixtures of Mallows model [33, 10]. Finally, we provide theoretical bounds on the sample complexity of the SP algorithms with partial preferences to
'This paper also explored various elicitation techniques combining vote and prediction questions based on only top choice and complete rankings.

---

further demonstrate the theoretical guarantees of the proposed methods. We show that the sample complexity only depends on the size of the subset & which significantly smaller than m.
1.1 Related Work
Our work is related to information elicitation, and partial aggregation which we discuss briefly.
Information Elicitation. Various information elicitation schemes [35, 36, 48, 17] attempt to in- centivize voters to reveal useful information, often through the investment of efforts. Our work is primarily related to the surprisingly popular algorithm [36] which is a novel second-order informa- tion based elicitation scheme. This framework has since been used to incentivize truthful behaviour in agents [35, 42, 43], mitigate biases in academic peer review [32], elicit expert knowledge [27], and aggregate information [9]. Our study builds upon this literature, specifically addressing the challenges in rank recovery. Originally, the SP algorithm by Prelec al. [36] required data on all m!! potential requirement that becomes impractical as m increases, Hosseini et al.
rankings for m alternatives, [25] addressed this by developing a Surprisingly Popular Voting algorithm that leverages pairwise preference data across (*) alternatives. This approach doesn't scale when m is large, and our contri- bution lies in advancing this methodology by proposing a scalable generalization of the Surprisingly Popular Voting method with partial preferences.
Partial Aggregation. In situations where it is difficult not necessary to clicit complete rankings from voters, partial preferences are used, Partial vote aggregation has different solution concepts [6]. Partial preferences can be used to conclude which alternatives
necessary and possible winners based on the preference profiles [26, 14, 47, 2, 3, 49]. The primary goal of partial aggregation methods is to either minimize the amount of information communicated by the voters [12, 45, 40]
or to reduce the number of queries that each voter needs to answer [37, 14]. Our work attempts to reduce such communication from the voters by eliciting partial preferences.
More broadly, our work is also related to information aggregation, and probabilistic rank-order models which we discuss further in Appendix A.
2 Model
In this section, we formally define the model for Surprisingly Popular Voting in the context of partial preferences. Let A (a1, a2..-.mmmm denote the set of possible alternatives. The set A) represents all possible complete rankings over the alternatives, Let C(A) represent a complete ranking of the m possible alternatives. We denote the ground truth ranking by n'' E C(A): which is assumed to be drawn from a prior P--) over L(A). Voter observes a ranking π. that is assumed to be a noisy version of the ground truth ranking x*. We will write Pr, (TT (=) to denote the probability that the voter i observes her ranking Ti given the ground truth n².
Given voter i's ranking π. and the prior P(-), voter i can compute the posterior distribution over the ground truth using the Bayes rule.
Pry( TT) Ex'ecca) Pr. P
Using the posterior over the ground truth, voter can also compute distribution over the rankings observed by another voter.
Pro(TT -.) = Pr,(TT) |T)- Pr, (T' m))
(2)
The surprisingly popular algorithm [36] asks voters to report their votes, and posterior over others' votes. For each ranking n', then computes the frequency f(=') = E, 1[m = T'], and posterior
g(iT| x') normalized votes.
Ren,=n? Ext,en (T xz), and finally picks the ranking with highest prediction
f E argmax, V(x) f(T) Eren
(( (()))
As observed by Hosseini al. [25], eliciting full posterior and even the vote might be prohibitive the number of alternatives m is huge. In this work, we are concerned about eliciting partial rankings
This is the direct application of SP algorithm [36] by considering m! possible ground truths.
3

---

over subsets of size & < m. Let us fix subset T C A of size k. Then the probability of partial ranking o; is given as
Pry(o, | TT) = ExSpo, Pre(T x=))
Here the notation oj to indicate that the ranking when restricted to the set T is dG.
We can also naturally extend definition I to define the posterior distribution over partial preferences given a partial preference dg. In order to do so, let us first define the posterior over full ground truth T² given 03 as,
(r (o.) Pr.(a((""P((""
where one can use definition (4) to compute Pr, x*). probability over the partial ground truth as follows.
Now we can write down the posterior
Pro(ō o.) Enmod (T o,)
(5)
Finally, we can write the posterior over another partial ranking o' over the subset T.
Pr.(o' o.) E. (r, Pr.)Pr('''
(6)
In this work, our aim is to propose several versions of surprising popular algorithm that work with partial preferences. As shown in definition 3, it requires eliciting information regarding voters partial preferences, and posterior over others' partial preferences (as defined in eq. (6)). Next, we diseuss various ways of eliciting such information from the voters.
2.1 Elicitation Formats
Given subset of size & alternatives, voter i's prediction Pr.(- (,,)) is distribution over k! rankings. In practice, this renders clicitation of full prediction information difficult, if not impossible, due its cognitive overload. Thus, we focus on simple, and more explainable, elicitation methods that rely on ordinal information either by identifying the most preferred alternative (Top), selecting complete ranking of the partial set (Rank). The
the most preferred f alternatives (Approval (t) formal definitions can be found in Appendix B.
Given these elicitation methods, study different combinations of formats for votes and predictions where the first component indicates the vote format and the second component denotes the prediction format. These give rise nine formats: Top-None, Top-Top, Top-Approval (!), Hop-Rank, Approval (1) Raank, Approval (/) -Aprroval (t), Rank-None, Rank-Top, and Rank-Rank. For Approval (t), the approval of size & (1,2,3) is selected. Note that Approval (1) Top.
3 Aggregation Algorithms for Partial Preferences
The surprisingly popular method (as we discussed in section 1) cannot be applied directly to find a full ranking using only partial preferences. Thus, we develop two vote aggregation algorithms that only rely partial ordinal preferences both for votes and predictions. On the high level, the two algorithms differ on how and when they implement the SP method, whether independently on each subset (Partial-SP) Here we provide a high-level description for each of the algorithms; additional details and exact pseudo-codes are relegated Appendix D.
the aggregated (potentially partial) votes and ranks (Aggregated-SP)
The key clement of this algorithm is utilizing SP voting on the partial rankings obtained at each step. It takes set of potentially overlapping subsets of alternatives and voting rule input and proceeds as follows: For each subset S, of alternatives, collect votes and predictions from voters on this subset according to one of the elicitation formats detailed in Section 1. Compute the ground truth partial ranking on the subset S, using the SP algorithm. Aggregate all partial rankings using a voting rule (e.g. Condorcet) to find a full ranking over all alternatives (breaking ties at random).
Partial-SP.
Aggr This variation utilizes SP voting on the final rankings over votes and predictions. takes set of potentially overlapping subsets of alternatives and a voting rule as input and proceeds as follows: For each subset S, of alternatives, collect votes and predictions from voters on this subset according to one of the elicitation formats detailed in Section 2. Aggregate all votes (partial

---

rankings) using a voting rule (e.g., Condorcet) to find the aggregated vote over all alternatives, breaking ties random. Predictions are not aggregated to preserve conditional prediction information crucial for SP voting. Apply SP algorithm pairwise across all alternatives where for each pair (a. b). the vote information is derived from the scores of a and based on the aggregation rule used, and the prediction information is used to find the conditional probabilities, P(a(b) and P(bla). Breaking ties at random throughout this process results in a full ranking over all alternatives.
Subset Selection. The algorithms described in this section rely on partial rankings on the subset of alternatives. Given m alternatives, we carefully select subsets of size with an aleeravee pairwise distance of between elements from the ground-trun ranking. Formally, a subset S, of size k is generated as follows:
Sj = {ai+j, 01+j+a) :1 (1+)+(A-1)s)
where j 2 and j (k: - 1)s < m, ensuring elements are within the range of m alternatives. We get total of m (k: 1)s subsets. Note that we use overlapping subsets so as to introduce transitivity among different subsets enabling us to compare alternatives across different subsets. This leads to an improvement in the accuracy of our algorithms as we discuss in Section 5.
4 Experimental Design
This section describes the experimental design of the Amazon Mechanical Turk (M'Turk) study to assess the comparative efficacy of Partial-SP and Aggregated-S against other voting rules for partial preferences. Participants in this study were asked to answer a series of questions, wherein they were required to express their preferences by voting on a range of alternatives. In addition to casting their own votes, participants were asked to predict the collective preference of others for the same set of alternatives. Data was collected from 432 respondents. Fach participant was given a 20-minute window to complete series of 18 questions (see details below). Datasets. The survey encompassed three distinct domains: (i) The geography dataset contains 36 countries with their population estimates, according to the United Nations, (ii) The movies dataset contains 36 movies with their lifetime box-office gross earnings, and (iii) The paintings dataset contains 36 paintings with their latest auction prices."
Questions. We explored 36 alternatives per do- main, aiming to gather partial preferences from voters. Each question featured subset of alter- natives, with the size of each subset maintained uniformly throughout the experiment.
Thtorial (Elecratio Fumnst
Cuuoctione (Eliettarinn Format 1)
Reccire [Eartation it
Porticipant
Preriew:& Conpesll
Beriew (Electation
Queatian (Elicierkon Rorastt 3)
Sabuit
Quis
Tusocial
Fonnsk 2)
Each participant was presented with a subset of alternatives, selected based on an inter- alternative gap of 6 positions within the ground-truth ranking. This strategy was designed to balance the cognitive load against the quality of the responses. We tested subset sizes of alternative gaps of 3 to 8, finding that larger sizes and wider gaps generally enhanced ground-truth recovery. However, larger subset sizes increase cogmitive load for participants, and wider gaps reduce overlap between subsets when limited to 36 alternatives. For each combination of 12 subsets, 9 elicitation formats, and 3 domains, each question received 16 responses.
Figure I: Workflow of a participant
and inter-
The survey was structured for each participant to answer two questions from each of the three domains and two elicitation formats, totaling 12 questions per participant. Figure shows the workflow for each participant; each participant was assigned 18 questions to answer. Refer to Appendix E for details on the tutorials, participant qualifications for the MTurk study, and review questions regarding the perceived difficulty and expressiveness of the study.
Elicitation Formats. We various elicitation formats (as described in section 2.1). For example, consider a question that requires participants to rank five movies: a) Rogue One: A Star Wars Story, b) Titanic, e) Toy Story 3, d) The Dark Knight Rises, and e) Jumanji: Welcome to the Jungle- based on their lifetime gross earnings. Under the Approval (3) -Rank elicitation format, the structure of the vote and prediction questions would be framed follows:
"This study received IRB approval from the ethics hoard, which is available upon request. 'The dataset can be found here http://ghiiuioommmmm////////reee
Popular- Voting-
5

---

Part A (vote): "Which among the following movies are the top three in terms of highest grossing income of all time?"
Part B (prediction): "Considering that other participants will also respond to Part A, in what order do you predict the following movies will be ranked, from the most common response (top) to the least common (bottom)?"
Refer Appendix for further details about formulations of all nine elicitation formats, the consent form, the tutorial for each domain, screenshots, and other details.
5 Results and Analysis
In this section, we present the results of this study averaged across all three domains. We measure the accuracy of the proposed SP algorithms (Partial-SP and Aggregated -PP) in predicting the full ground-truth ranking, in comparison with common vote aggregation methods (e.g. Borda, Copeland, Maximin, Schulze). The details of these aggregation methods is provided in Appendix C.
Additionally, we compare the elicitation formats (described Section 2.1) with respect to cognitive effort (measured by response time and difficulty) and expressiveness (measured directly by survey questions). They are provided in Appendix G.2
5.1 Accuracy Metrics
To capture the error in predicting the full ground-truth ranking, we use three different metries: (i) the Kendall-Tau correlation, which measures the distance between ordinal rankings, (ii) Spearman' p correlation, which measures the statistical dependence between ordinal rankings, (iii) Pairwise hit rate, which measures the fraction of pairs at distance d that are correctly ranked with respect to the ground-truth ranking, and (iv) Top-t hit rate, which measures the fraction of alternatives that are predicted correctly (in no order) in most preferred compared to the ground-truth ranking. The formal definitions be found in Appendix G.I.
For example, consider the ground-truie truth ranking a > c> d. The predicted ranking b > a > d > e has pairwise hit rate of 1/3 at distance 1, at distances 2 and 3. Its Top-1 hit rate is 0, Top-2 is 1, Top-3 is 2/3, and Top-4 is
5.2 Predicting the Ground Truth Ranking
Figure 2 illustrates the performance of SP algorithms measured by Kendall-Ta -aa and Spearman's ss correlations. We fix Copeland as the aggregation rule used in both variations of SP voting and compare the accuracy with applying Copeland on votes alone (without the use of prediction information).
Statistical correlations and elicitation. SP voting produces rankings with significantly higher correlation with the ground truth ranking, and this effect improves as the information provided as votes and prediction becomes more expressive. In particular, Rank-Ram Rank and Approval (3) -Rank outperform all other elicitation formats. We note that Aggregated-SP seem to be more reliant on the vote information, compared to the predictions, as it can seen in Top-Rank vs. Rank-Top. In contrast, Partial-SP does not exhibit any significant favor for vote vs. prediction information as both Top-Rank and Rank-Top Top improve by additional information. However, the difference between them is not statistically significant.
Interestingly, eliciting unordered information for both noisy votes and predictions (e.g. Approval (2) -Aprroval (2)) seem to be sufficient in recovering the ground truth-raising the question of whether pairwise comparisons are necessary in designing SP algorithms.
Hit rates. With respect to pairwise hit rate and the Top-t hit rate, the noisy prediction information significantly improves the performance of the Partial-SP algorithm (with lower variance) as shown in Figure 3. The results for Aggregated-SD are qualitatively similar and are presented in Appendix G.5. The slight dip in pairwise hit rate, can be explained by the survey's design choice of inter-alternative distance of 6, leading to fewer comparisons being available for these pairs.
Partial-SP vs. Aggregated-SP While both variants of the SP algorithm significantly outperform common voting rules by utilizing (noisy) prediction information, the Partial-SP algorithm significantly outperforms the Aggregated-SP algorithm (see Figure 2 and Figure 4). This could be explained

---

K
Capeland
PartialSP
090M
Aggregated SP
Speermeny bulte Cerreattoo
1.0
Kendal Corrention
0.8
0.6-
0.6
0.4
0.2
0.0
Top
Top-App(
Bp-Rank
Rani Top
App -Aop()
Appt patti--o Rank
Rank Rank
Top-Pop
Tap-Aspi
Top-Rank
Rank-
A Anstaj-Rank
Rank-Rank
Appiz
Figure 2: Comparing the predicted and ground-trun -ruth rankings for different elicitation formats using Kendall-Tau and Spearman's 's p correlations (higher is better). All results use Copeland as their aggregation rule.
1.0
0.9
0.8-
0.8
0.7
0.5
Mean Cornect Hits Partial-SP
Mean Cornect Hits Partial-PP
0.4
0.0
10
20
20
Top-t
(a) Pairwise hit rate at distance d
(b) Top-t t hit rate
Figure 3: Comparing the Partial-SP algorithm with Copeland (no prediction information) measured
by pairwise and Top-t hit rates. The elicitation format is Approval (2) -Approval (2).
by the importance "correcting" noisy votes on the subsets of alternatives because the prediction information of the Partial-SP algorithm helps identify experts early predicting partial rankings of these alternatives.
For Partial-SP, the plots indicate no statistical significance between Approval (2) -Approvallaccc (2), Approval (3) -Rank, and Bank-Rank elicitation formats, suggesting they perform as well as Rank Rank. An interesting ramification here is demonstrating that approval sets not only per- form well in predicting the ground truth, but also pose less cognitive burden on voters compared to those elicitation formats that ask for rankings (see appendix G.2).
Domain impacts. Performance of Partial-SP and Aggregated-SP robust across domains. They outperform common voting rules with the sole exception the Schulze method, which matches the performance of Aggregated-SP (see Figure 4). The difference performance is notably high for Paintings domain, where specialized expertise is required to predict painting prices. Here, Partial-SP significantly outperforms common aggregation rules, showcasing its effectiveness in leveraging expert knowledge and correcting misinformation. For further details see Appendix G.3,
Simulated Model of Voter Behavior
In this section, we investigate whether there is any underlying probabilistic model that can explain the voters' behaviours when measured in terms of the vote and predictions. successful, such a model will also enable us to theoretically analyze the sample complexity of SP algorithms (as we present Section 7). particular, we posit that concentric mixtures of Mallows model can explain the users' reports (both vote and prediction) in the dataset. The concentric mixtures of Mallows model is a type of mixture models where there one ground truth, but different groups users have different dispersion parameters, and hence different distribution over observed preferences.

---

Common Aggregation Rule
Intal
Partial
B000 Aggregated SP
1.0
Spearman's Saamanns--hoo 9-hho Carrelaioo
10
Kondall Thu Comelation
0.8
0.6
0.6
0.4
0.2
0.2
0.0
BordslG)
Copeland(()
MaximinG)
SculzeeG)
BordalP)
CopetandiP)
Moxinin(P)
SchutzeiP)
BerdaiG)
Copeland(G)
MaximiniG
SchulretG)
BordaiP)
copeland(P)
Moximin
Schuize(P)
Figure 4: Comparing the predicted and ground-truth rankings for different aggregation rules using Kendall-Tai and Spearman's 's correlations (higher is better). The elicitation format is Rank Rank; each comparison uses the same aggregation rule in the SP algorithm.
Concentric mixtures of Mallows model. We assume that each voter is likely to be an expert with probability p( p(< 1) and non eoppeet with probability 1 Given ground truth ranking x", an expert voter observes a ranking that distributed according to a Mallows model with center '² and dispersion parameter Φx. On the other hand, non- non-expert voter observes ranking that is again distributed according to a Mallows model with center n², but with a larger dispersion parameter ONE In particular, the ranking observed by voter is distributed as
Pr,(T; (TT m²* p (r,(ms m",0)) (1 p) Pr,(m; n² (NNE)
where Pr, (T) n², (T,0) .)) is the standard Mallows model with dispersion @ i.. Pr, (TT x², () Φ ((r,a")
Zto.m]
The term (6, m) is the normalization constant, and is defined as Z(ó, m) = EE, gd(a,x") With slight abuse of notation, we will write (()) as Z(o, (m) since the number of alternatives in the ground truth is assumed to be fixed.
Note that, Equation (8) defines distribution size we can naturally extend this definition to define a distribution over partial preferences e.g. Pr.(o, x=") (eq. (4)), and posterior partial preferences of other voters e.g. Pro o;) (eq. (6)).
complete preferences, but given subset of
Original
Synthetic
@ Expert Votes
Expert Predictions
0 Non-Expert Votes
Non-Expert Predictions
0.5
0.5
Densily
033
0.2
00
0.2
0.4
0.5 0.0
1.0
3.0
022
0.4
06 2.8 10 0.0 0.2 0.4
0.5
0.
1.0
0.
0.2
84
0.6
03
15
Kendall-Tau distance from the ground truth ranking
Figure 5: Comparison of inferred parameters of the Concentric mixtures of Mallows model for real data with all domains combined and synthetic data. The experts vote closer to and predict farther from the ground-truth. The non-experts vote and predict far from the ground truth. The proportion of experts in both datasets was found to be less than 20%.
We fit the mixture model eq. (8) on the real datasets and estimate the following prrameters-proportion proportion of experts (p), dispersion parameters of experts" votes (δE-votes) and predictions (dg-predictiona) as well as the dispersion parameters of non-experts' votes (ΦN E-vates) and predictions (ó E-predictions) The parameters were inferred using Bayesian inference [24]. We also generated synthetic data using the concentrie mixtures of Mallows model, and again used Bayesian inference to estimate the parameters. The details of estimation and data generation are provided in the appendix F. Figure shows the posterior distributions for dispersion parameters when datasets of all the three domains are combined. We see that the synthetic data generation process accurately replicates real data characteristies, and highlights that the concentric mixtures Mallows accurately model voters' behaviours MTurk. Furthermore, Figure 6 also plots the same posterior distributions but only for

---

Original
Synthelic
Expert Votes
Expert Predictions
Non-Expert Votes
Non-Expert Predictions
0.5
Density
02
60
0.2
0.4
0.6
06
1.0
000
02
0.4
06 5
0.
0.2
6.4
64
0R
1.0
000
6.2
04
0.6
08
14
Kendall-Tar Taau distance from the ground truth ranking
Figure 6: Comparison of inferred parameters of the Concentric mixtures of Mallows model for real data of Movie domain and synthetic data. The quality of model fit improves if the focus is on one single domain.
the Movies domain. Now we see almost perfect fit between the synthetic data and the original data. Further similarity in results between real and simulated data is described in Appendix G.6.
7 Analysis of Sample Complexity
In this section, we use concentric mixture of Mallows models, and provide upper bound on the sample complexity of the surprisingly popular voting method with partial preferences. We start with simple problem. Given a subset T of size k, suppose our goal recover the true partial ranking over the alternatives in T, then how many samples does SP algorithm require?
We will analyze the following simplified version of the SP algorithm: Voter reports vote o; over the subset T, which is used to build an estimate of f(rr) for all E II,. For each the posterior report by the voter partial ranking drawn from the distribution g(- o). These reports are used to build
Maa'(a) an estimate ā(o') о)) for all a', Select partial ranking ā E argmax, V(a) = - B(rlo")
It is impossible to recover the partial ground truth ranking if the fraction of the experts p can be very small, or the dispersion parameter of the non-experts &NE can be very large. In order to ensure recovery of the true partial ranking we will make the following assumption.
Assumption The dispersion parameters OE ONE, and the fraction of experts p satisfy the following inequality,
2 2
Z(ØNE) Z(oE)
k(k
k(k-1)/2
I-p
where k(o,k) = = d'll,oo))
The next theorem states that sample complexity under the above assumption.
Theorem 1. Suppose Assumption I holds, and the total number of samples kl
10k log(2k/5)
where μ = p 2ko
Zles) of 1(k-1)/2 + p1- 2(oxn.m-k)
M(k-1)/2 Then the surprisingly
ONE
Z(exE)
popular algorithm recovers true ranking over the subset T of size l with probability at least 8.
k2 1//411 / and implies
Suppose óg &NN 1. Then Assumption requires 1 2.20 Ω
2
(NN
&E
that if ΦNE is very large compared to dE (i.e. noisy non-experts) then we need larger value of (i.e. more experts).
We provide the full proof of the theorem in Appendix I. The main ingredient of the proof is Lemma which shows that under Assumption there is a strict separation between the true prediction- normalized score of the partial ranking and any other ranking. In fact, show that V(a*) 2 (T(r) for any with d(T, o²) 2 1. Given this result, to show that ((a) is close to V(a) for all when the number of samples is large, and V(a*) will be larger than ((T) for any 7 # o". Therefore, picking the ranking with the largest empirical prediction-normalizee score returns the correct ranking.
apply standard concentration inequality
Note that the sample complexity grows proportional to only because we compute prediction- normalized votes over all k! partial rankings. If we interested in recovering top t-alternatives then

---

it will grow proportional to (*). Moreover, the subset size k is assumed to be very small compared to the number of alternatives m, and Theorem shows the benefit of applying SP algorithm to partial preferences. We can immediately apply Theorem 1 to collection of subsets 5 through a union bound, and extend our analysis to the Partial-SP algorithm. Let us assume that in the second stage of Partial-SP, we apply a I-consistent voting rule f that recovers top-t alternatives as long as each partial ranking in correct.
Corollary 1. Under the same setting Theorem suppose the number of samples from each subset in S is 2 // 1 log(2]5]k//), Then the Partial-SP algorithm with t-consistent voting rule, recovers the top alternatives of the ground truth x² with probubility at least 6.
"
Finally note that, the total sample complexity of ([[S)-AIvis is needed only because we adopt naive version of the SP algorithm for proving theoretical guarantees. For the experiments, we adopt a pairwise version of the SP algorithm which applies SP-voting to each pair within a subset. We believe that under further assumptions, the total sample complexity can be reduced to Ö([S) k²) with such pairwise variant of the partial-SP algorithm, and we leave this analysis for the future.
8 Discussion and Future Work
We conclude by discussing some limitations and future directions. When dealing with partial preferences, even when majority have the correct information, effective preference elicitation or finding necessary winner in most vote aggregation rules are often computationally intractable [11. 39, 18]. These challenges, together with the minority of experts, further highlight the efficacy of the SP approach in balancing information elicitation and accuracy, by employing additional prediction information. Future research can explore the setting of SP beyond the dichotomy (e.g- informed, but not expert voters) or when malicious voters are present (e-g. in detecting misinformation). Theoretically, the sample complexity can be explored beyond Mallows model under other probabilistic models to enhance our understanding of this approach, particularly in notable applications such as political polling or collective moderation of online content.
Acknowledgments and Disclosure of Funding
Hadi Hosseini acknowledges support from National Science Foundation (NSF) IIS grants #2144413 and #2107173. We thank the anonymous reviewers for their constructive feedback.
References
[1] Yoram Bachrach, Nadja Betzler, and Piotr Faliszewski. Probabilistic possible winner determi- nation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 24, pages 697-702, 2010.
[2] Dorothea Baumeister and Jörg Rothe. Taking the final step to a full dichotomy of the possible winner problem in pure scoring rules. Information Processing Letters. 196, 2012.
[3] Nadja Betzler and Britta Dorn. Towards dichotomy for the possible winner problem in elections based on scoring rules. Journal of Computer and System Sciences, 76(8):812-836, 2010.
[4] Niclas Bochmer, Robert Bredereck. and Dominik Peters. Rank aggregation using scoring rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5515-5523, 2023.
[5] JC de Borda. M'emoire sur les' elections au scrutin. Histoire de I' l'Acad' emie Royale des Sciences, 1781.
[6] Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D Procaccia. Handbook of computational social choice. Cambridge University Press, 2016.
[7] Clément I. Canonne. A short note on learning discrete distributions, arXiv preprint 212.2,000211445,2222
10

---

[8] Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus A Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of statistical software, 76, 2017.
[9] Yi-Chun Chen, Manuel Mheller-Frannk and Mallesh Pai. The wisdom of the crowd and higher- order beliefs. In Proceedings of the 24th ACM Conference on Economics and Computation, pages 450-450, 2023.
[10] Fabien Collas and Ekhine Irurozki. Concentric mixtures of mallows models for top-k rankings: sampling and identifiability. In International Conference on Machine Learning, pages 2079- 2088. PMLR, 2021.
[11] Vincent Conitzer and Tuomas Sandholm. Vote elicitation: Complexity and strategy strategy-proofness In Proceedings of the AAAI Conference on Artificial Intelligence, volume 18, pages 392-397, 2002.
[12] Vincent Conitzer and Tuomas Sandholm. Communication complexity of common voting rules. In Proceedings of the 6th ACM conference Electronic commerce, pages 78-87, 2005.
[13] Vincent Conitzer and Tuomas Sandholm. Common voting rules maximum likelihood estima- tors. In Proceedings of the Twenty-Först Conference on Uncertainty in Artificial Intelligence, UAI05, page 145-152 Arlington, Virginia, USA, 2005. AUAI Press. ISBN 0974903914.
[14] Vincent Conitzer, Tuomas Sandholm, and Jérôme Lang. When are elections with few candidates hard to manipulate? Journal of the ACM (JACM), 54(3):14-ss, 2007.
[15] Vincent Conitzer, Matthew Rognlie, and Lirong Xia. Preference functions that score rankings and maximum likelihood estimation. Proceedings of the Twenty-First International Joint Conference Artificial Intelligence, 2009.
[16] Arthur H Copeland. A reasonable social welfare function. Technical report, mimeo, 1951. University of Michigan, 1951.
[17] Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd international conference on World Wide Web, pages 319-330, 2013.
[18] Andrew Davenport and Jayant Kalagnanam. A computational study of the kemeny rule for preference aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 4, pages 697-702, 2004.
[19] Patrick M De Bocr and Abraham Bernstein. Efficiently identifying well-performing crowd process for a given problem. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pages 1688-1699, 2017.
[20] Nicolas De Condorcet. Essai l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix. Cambridge University Press, 2014.
[21] Cynthia Dwork, Ravi Kumar, Moni Naor, and Dandapani Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web, pages 613-622, 2001.
[22] FRANCIS GALTON, Vox populi. Nature, 75(1949):450-451 1907.
[23] Noam Hazon, Yonatan Aumann, Sarit Kraus, and Michael Wooldridge. On the evaluation of election outcomes under uncertainty. Artificial Intelligence, 189:1-18, 2012.
[24] Matthew D. Hoffman and Andrew Gelman. The no-u-turn sampler: Adaptively setting path lengths hamiltonian monte carlo. Journal of Machine Learning Research, 15(47):1593-1623 2014.
[25] Hadi Hosseini, Debmalya Mandal, Nisarg Shah, and Kevin Shi. Surprisingly popular voting recovers rankings, surprisingly! Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 245-251, 2021.

---

[26] Kathrin Konczak and Jérôme Lang. Voting procedures with incomplete preferences. In Proc. IJCAI-05 Multidiscipinnay Workshop on Advances Preference Handling, volume 20. Citeseer, 2005.
[27] Yuqing Kong and Grant Schoenebeck. Eliciting expertise without verification. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 195-212, 2018
[28] Yuqing Kong, Yunqi Li, Yubo Zhang, Zhihuan Huang, and Jinzhao Wu. Eliciting thinking hierarchy without a prior. Advances in Neural Information Processing Systems, 35:13329-13341, 2022.
[29] Tyler Lu and Craig Boutilier. Learning mallows models with pairwise preferences. In Pro- ceedings of the 28th international conference machine learning (ieml-11), pages 145-152, 2011.
[30] Tyler Lu and Craig Boutilier. Robust approximation and incremental elicitation in voting protocols. In Proceedings of the Twenty-Secone International Joint Conference on Artificial Intelligence, volume 1, pages 287-293, 2011.
[31] Tyler Lu and Craig Boutilier. Vote elicitation with probabilistic preference models: Empirical estimation and cost tradeoffs. In Algorithmic Decision Theory: Second International Conference, ADT Piscataway, NJ, USA, October 26-28, 2011. Proceedings 2, pages 135-149. Springer, 2011.
[32] Yuxuan Lu and Yuqing Kong. Calibrating "cheap signals" in peer review without a prior. Advances in Neural Information Processing Systems, 36, 2024.
[33] Colin L Mallows. Non-null ranking models. i. Biometrika, 44(1/2):114 14--130, 1957.
[34] John Marden. Analyzing and modeling rank data. CRC Press, 1996.
[35] Drazen Prelec. A bayesian truth serum for subjective data. science, 306(5695):462 466, 2004.
[36] Dražen Prelec, Sebastian Scung, and John McCoy, solution to the single-question crowd wisdom problem. Nature, 571(685))532-555 2017.
[37] Ariel D Procaccia. A note the query complexity of the condorcet winner problem. Information Processing Letters, 108(6))390-333, 2008
[38] GWM Rauterberg. A method of quantitative measurement of cognitive complexity. In Human- computer interaction: tasks and organisation: proceedings of the 6th European conference on cognitive ergonomics, ECCE'92 92, pages 295-307. CUD, 1992.
[39] Jörg Rothe, Holger Spakowski, and Jörg Vogel. Exact complexity of the winner problem for young elections. Theory of Computing Systems, 36:375-386, 2003.
[40] Shin Sato. Informational requirements social choice rules. Mathematical Social Sciences, 57 12):188-198, 2009.
[41] Frans Schalekamp and Anke van Zuylen. Rank aggregation: Together we're strong. In 2009 Proceedings of the Eleventh Workshop on Algorithm Engineering and Experiments (ALENEX). pages 38-51. SIAM, 2009.
[42] Grant Schoenebeck and Biaoshuai Tao. Wisdom the crowd voting: Truthful aggregation of voter information and preferences. Advances in Neural Information Processing Systems, 34: 1872-1883, 2021.
[43] Grant Schoenebeck and Fang- Yi Yu. Two strongly truthful mechanisms for three heterogencous agents answering one question. ACM Transactions on Economics and Computation, 10(4):1-26. 2023.
[44] Markus Schulze. new monotonic, clone-independent, reversal symmetric, and condorcet- consistent single-winner election method. Social choice and Welfare, 36:267-303, 2011.
12

---

[45] Travis C. Service and Julie A Adams. Communication complexity of approximating voting rules. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), pages 593-602, 2012.
[46] Joseph P Simmons, Leif D Nelson, Jeff Galak, and Shane Frederick. Intuitive biases in choice versus estimation: Implications for the wisdom of crowds. Journal of Consumer Research, 38 (1):1-15, 2011.
[47] Toby Walsh. Uncertainty in preference elicitation and aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 7, pages 3-8, 2007.
[48] Jens Witkowski and David Parkes. A robust bayesian truth serum for small populations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 1492-1498, 2012.
[49] Lirong Xia and Vincent Conitzer. Determining possible and necessary winners given partial orders. Journal of Artificial Intelligence Research, 11:25-67, 2011.
[50] Lirong Xia, Vincent Conitzer, and Jérôme Lang, Aggregating preferences in multi-issue domains by using maximum likelihood estimators. In Proceedings of the 9th International Conference on Autonomous Agent and Multiagent Systems: volume 1-Volume 1, pages 399-408, 2010.
[51] H Peyton Young. Extending condorcet's rule. Journal of Economic Theory, 16(2):335-333,
1977.
13

---

Appendix
Table of Contents
A Additional Related Work
14
B Formalism of Elicitation Formats
C Common Voting Rules
16
C.I. Borda
16
C.2 Copeland
16
C,3 Maximin
17
C.4 Schulze
17
Algorithms
D.1 Extracting Reports from Voters
................................
19
D.2 Partial-SP
19
D.3
Aggregated-St
20
E Additional Details of Experimental Design
21
F Additional Details of Simulation
22
F1 Parameter Inference for the concentric mixtures Mallows model
22
Synthetic Data Generation
G Missing Results and Analysis
23
G.1 Evaluation Metrics..
23
G.2 Response-Time, Difficulty and Expressiveness
23
G.3 Missing Figures for Predicting the Ground Truth Ranking
24
G.4 Missing Figures for Partial-SP....
27
Missing Figures Aggregated-SP
29
G.6 Comparing Performance hetween and Simulated Data
H. Missing Proofs
31
H.l Proof of Theorem (1)
31
H.2 Separation Lemma
32
Screenshots from our MTurr Survey
33
A
Additional Related Work
Information Aggregation. Information Aggregation by eliciting votes from voters is well-studied problem in social choice theory. The aggregation rules proposed by De Condorcet [20], Borda [5], Copeland [16], Young [51], and many others focus on information aggregation by eliciting votes. These rules can be adapted elicit ranked information from voters and then aggregate them into single ranking representing the collective opinion of the crowd. [4]. Information Aggregation has also been examined from statistical perspective, where the aggregated ranking viewed as maximum likelihood estimate of the population's rankings [20, 15, 50, 13]. Within this framework, individual votes are considered outcomes of probabilistic models such as the Thurstonian model, Bradley-Tere Terry model. Mallows' model, or Plackett-Luce model [34].
14

---

Partial Aggregation. In situations where it is difficult or not necessary to elicit complete rankings from voters, partial preferences are used. Partial vote aggregation has different solution concepts [6]. Partial preferences can be used to conclude which alternatives are necessary and possible winners based on the preference profiles [26, 14, 47, 3, 49]. Alternatively, a regret based approach can be used to assess the quality of winning alternative where the optimal alternative is the one that minimizes regret [30]. Apart from these epistemic notions, a lot of work has been done on probabiliitic analysis of winners from partial profiles [1. 23, 29, 31J. To minimize the information elicited from a population, it is crucial to understand the methods used for eliciting partial preferences. It can either be by minimizing the amount of information communicated by each voter in their answer [12, 45, 40] or by reducing the number of queries that each voter needs to answer [37, 14]. In either of these cases, the objective is still to determine the winner accurately. Cognitive complexity also plays an important part in this, as all these notions are highly correlated. Finally, the recovery of complete rankings from partial preferences is another solution concept that is studied [21, 41]. Due to the combinatorial nature of the rankings, winner determination, communication complexity, query complexity, and cognitive complexity are all relevant here. This is where our research work contributes.
Surprisingly Popular Framework. In their seminal work, Prelec [35], Prelec et al. [36] introduced the Surprisingly Popular (SP) algorithm, novel second-order information based method that recovers truthful subjective data in scenarios where objective truth remains unknown. This framework has since been used to incentivize truthful behaviour in agents [35, 42, 43], mitigate biases in academic peer review [32], elicit expert knowledge [27], model thinking hierarchy of people without any prior [28], aggregate information [9] and recover ground-trutt ranking [36, 25J. Our study builds upon this literature, specifically addressing the challenges in rank recovery. Originally, the SP algorithm by Prelec et al. [36] required data on all m! potential rankings for m alternatives a requirement that becomes impractical as increases. Hosseini et al. [25] addressed this by developing Surprisingly Popular Voting algorithm that leverages pairwise preference data across (2) alternatives. However, this approach encountered scalability limitations when dealing with more than four alternatives in partial preference profiles. Our contribution lies in advancing this methodology by proposing scalable generalization of the Surprisingly Popular Voting method for partial preferences, thus broadening its applicability and effectiiveness.
B Formalism of Elicitation Formats
In this section, we formally define the elicitation formats used in our study. Let v; and pæ denote the vote and prediction submitted by voter i. Let T = (a1, of size k that voters will report and L(T) denote the set of all possible rankings of alternatives in T. Let denote ranking of the alternatives in T and (j)) denote the alternative at the jth position in a. The elicitation formats are defined as follows:
ax} denote the subset of alternatives
Top-None: Voter reports the top alternative in her observed noisy ranking, i.e., i (1), and does not provide any inference about other's 's aggregated votes.
Top-Top: Voter i reports the top alternative in her observed noisy ranking, i.c., V/ = (1). and provides the estimate of the most frequent alternative among the other voters, i.e., pi arg maxaer Eo Pro(ajas)
Top-Approvall (3): Voter reports the top alternative in her observed noisy ranking, i.e., v = (1). and provides the estimate of the top three most frequent alternatives, in no specific order, among the other voters., i.e., Pt arg maeTT Ear(abee)((6()).(()..x3))) Pro(a)az)
Top-Rank: Voter reports the top alternative in her observed noisy ranking, i.e., i (1), and provides the estimate of other's rankings i.e, Pa E L(T) such that Eoec(A) (r.(o));;) 2
(=<(()))(()==(())) (o)o,) for all x > y.
Approval AApproal (2): Voter i reports the top two alternatives, in no specific order, in her observed noisy ranking, i.e., e;- (o(1),(()) = {a.b) b) with a, b E T in no particular order and provides the estimate of the top two most frequent alternatives, in no specific order, among the other voters., i.c., Pt arg maX Eo.(a.))( Pro
15

---

Approval (3) Rannk Voter reports the top three alternatives, in no specific order, in her observed noisy ranking, ie., v; (2)) (3)) {a. cf with a, b, E T no particular order, and provides the estimate of other's rankings i.e, E L(T) such that (oEc(A)o(1==x(e)) (ro(g(o:) Z
() (r,(o));) for all y-
Rank-None: Voter i reports her entire observed noisy ranking, i.e., di, and does not provide any inference about other's aggregated votes.
Rank-Top: Voter reports her entire observed noisy ranking, i.c., es = di. and pro- vides the estimate of the most frequent alternative among the other voters, i.c. ps arg maXaeT ( (ro(o));))
Rank-Rank: Voter reports her entire observed noisy ranking, i.e, is = ai and provides the estimate of other's rankings i.e, pt E L(T) such that _=Ec(A)a()=a.)))) Pro(ala.) 2 ()E Pro(olo;) for all x > y.
C Common Voting Rules
Vote aggregation rules social choice functions that used aggregate individual votes make conclusions about the collective opinion of canti-candidate voting system [6]. Given below are the vote aggregation rules used our study. We will only focus on aggregating ranked preferences.
Number of Voters
Preference Profile
44
A B C D B> C > D >
24
18
C D B
14
D> B> A
Table Voter Preferences
C.1 Borda
The Borda rule [5] is voting rule in which voters order candidates by ranked preference, and candidates are awarded points based on their position in each voter's ranking. The winner is the candidate with the highest total score after all votes counted. It can be mathematically represented as:
Borda Score(a) = >- (m- (a))
where o, (a) represents the position which alternative present. The aggregated ranking is derived by sorting the Borda scores of the alternatives in descending order.
Example Apply Borda Rule the preference profile given Table
Table provides the preference profiles of voters. Applying Borda Rule, we find that Borda Scores of A, B. C, D are 132, 192, 174, and 102 respectively. Thus arranging the scores in descending order results in the aggregated ranking of B > C > A A>D. D.
C.2 Copeland
The Copeland rule [16] is voting method used to select single winner from set of candidates based on pairwise comparisons between each pair candidates. the Copeland method, each candidate receives score based on the number of head-to-head contests they win against other candidates, with ties potentially receiving half point for each candidate involved in the tie. can be mathematically represented as:
16

---

if V ((b,a) V(a b) = ((b,
Copeland Score(a)
Σ
0.5
a.DEA.
0
V(,, b) < (()
where Vaa, b) represents all the voters who preferred a over b. The aggregated ranking is derived by sorting the Copeland scores of the alternatives in descending order.
Example Apply Copeland Rule to the preference profile given in Table I
Applying Copeland Rule (with Borda tie-breaking) to the preference profiles given in Table in the following pairwise table:
results
Pairwise Comparisons
Winner
A vs
A vs C
C
A vs D
D
B vs
B
D C
B vs
B
C
Table 2: Results of Pairwise Comparisons
Thus, arranging the alternatives in decreasing order of number of time they become winners results in the aggregated ranking of B > C > D A.
C.3 Maximin
The Maximin rule [51], also known the Simpson -ramerr Kramer method, selects winner from set of candidates by considering the strength candidate's s worst-case pairwise comparison against all other candidates. It identifies the candidate whose least favorable comparison superior those of the others, aiming to find the most robust candidate against the strongest opponent. This can be mathematically expressed as:
Maximin Score(a) = min A,bba
(a, b)
where V(a, b) represents all the voters who preferred a over b. The aggregated ranking is derived by sorting the Maximin scores of the alternatives in descending order.
Example Apply Maximin Rule to the preference profile given in Table
In order to apply Maximin Rule to the preference profiles given Table we first analyze the worst
pairwise defeat and its margin by the following table:
Alternative
Worst Pairwise Defeat
Margin of Worst Pairwise Defeat
B
0
-12
D
86
72
Table 3: Analysis of Worst Pairwise Defeats
The alternative that has higher score of worst pairwise defeat that loses by higher margin is considered worse off. Thus, arranging in ascending order of the scores of any of the column results in the aggregated ranking of
Cx D.
C.4 Schulze
The Schulze rule [44] selects a ranking of set of candidates based on the strength of preferences expressed by voters. The strength of preference is considered to be the number of voters who prefer
17

---

one candidate over another. For every pair of candidates, a directed graph is constructed where the edges represent the strength of preference. The method then calculates the strongest path (defined as the weakest link in the path being as strong as possible) between every pair of candidates. A candidate wins if, for every other candidate, there exists stronger (or equal strength) path to that candidate than from that candidate. It can be mathematically expressed as:
Initially, for all pairs of candidates a, b:
P(a.b) ((
Vf, a(
0
otherwise
Then, for each a, b, E with a + + e, update:
P(a, max(P min(P, c), P(c,
For each candidate a, calculate a comprehensive score that may involve the sum of all positive differences P(a, b) P(b, a) against other candidates b. The aggregated ranking is obtained by sorting these scores descending order.
Example 4. Apply Schulze Rule to the preference profile given in Tuble
In order apply Schulze Rule the preference profile given in Table Graph where the vertices denote the alternatives and the weight of the edges denote the score by which alternative defeats the other. For the preferences in Table 1, we get the following graph:
first generate Directed
68
B
C
56
56
86
68
D
56
Figure 7: Directed Graph with Vertices Arranged as Corners of Square
The Table 4 finds the strongest path for each pair of vertices:
A
B
C
D
A
0
B
B 68
+
86 D
56
+ A
X
B 68
C
B
68
C 86 D
C
C- 86
+ 56 A
X
C- 86 - D
D
X
Table 4: Strongest Path between Vertices
Finally, the Table 4 provides the weakest link between each pair of vertices:
A
B
D
A
X
0
0
0
B
56
X
68
68
56
0
86
D
56
0
0
X
Table 5: Strength of weakest link between vertices
Arranging the alternatives decreasing order of their pairwise wins Table results in the aggregated ranking of B > D> A.
18

---

D Algorithms
Appendix D.I details the approach used to extract information from various elicitation formats. Explanation and pseudocode for Partial-SP and Aggregated-SP is provided in Appendix D.2 and Appendix D.3, respectively.
D.1 Extracting Reports from Voters
Algorithm describes how information extracted from different elicitation formats. The algorithm takes as imput the votes (fe hefm)) and predictions ((p he(m)) of n voters, pair (a, b) of alternatives, and parameters a and B. Using grid search on the datasets from the three domains, it was observed that any > 0.5 and 3 < 0.5 can be used. In our experiments we use a 0.55 and / = 0.1. For votes and predictions expressed as top choices approvals, if is the preferred alternative, o/0.0) (a.b)
is set to 1. with p. adjusted to if the prediction aligns with the vote, or to 3 otherwise; if b is B, depending on prediction alignment. For ranks, Vi (o.b) indicates preference based on rank ordering, and pj (a.b) reflects the confidence in this preference (a or aligned, or if not). The algorithm returns the processed vote and prediction reports.
chosen, D; (a.b) is 0, and p(a.)) is set
ALGORITHM 1: Extract-Reports
Input: Votes (els)))) Predictions (phhet)) pair b). and probabilitics a >> 0.5, and 8 < 0.5. 1, do /a Extract Vote Information
vi is elicited as rank then
Set o)
{
if ia.
0 o.w.
else if n. elicited as top approval then
Set (a.b)
if 14
Ignore (t), (tl.gg)
/* Extract crediction Information
if p. is elicited as rank then
a >r, b and o (0,0)
Set
1 if
and v (0.0)
if Pry and (9.b))
5
o.w:
else if p. elicited as top or approval then
if pt a and af [s.8]
a
Set pi (0.8)
1 - a
if p. b and (0.8)
1 g
if p. and (o.0)
5
0.wL
Set p.
return (o/110), ,p hic1ll)
D.2 Partial-SP
Algorithm describes the proposed Partial-SP aggregation approach. The algorithm takes as input the number of voters (n), number of alternatives (mn), set of all subsets that voters voted on (S), voters' votes the conditional probabilities that would be returned for the predictions, and Vote Aggregation Rule (V). For n voters and m alternatives, depending on the elicitation format, the voters will be providing votes (t,,)) and predictions (P;,)) as Top choices, Approvals(t), Rankings over a subset S, E S. Additionally, we use Borda, Copeland, and Maximin aggregation rule for V.
voters' predictions
heii),ssss).. parameters a and 3 representing
For every pair of alternatives (a, b) within a subset Sj. we extract information about the number of people that voted for > and b > a represented by [(a > b) and s(b > a) respectively, and the conditional probability of their predictions (g(0)0), ((01),, ( (1)()) (1]))) Refer to Appendix D. for a detailed explanation of how information extracted for different elicitation formats. With this, the prediction normalized score Vaa b) and (b > a) is calculated and higher prediction
19

---

normalized score decides the correct ordering for each pair (a, b) within a subset Sy. This results in partial ground-truti ranking representing the correct relative ordering for the alternatives within that subset. Q represents the set of partial ground-truth ranking for all subsets. Finally, Vote-Aggregation rule is applied on Q to find the complete ground-truth ordering of the m alternatives.
ALGORITHM 2: Partial-SP
Input: Number of Voters n, Subsets of alternatives S. Votes (uta heln),es.. Predictions (pi.hett))ess probabilities and 0.3, and Vote-Aggregaion rule
Q
1.... |S.
Apply SP-voting on votea predictions for each aubset for each pair of alternatives b) Sj de
hemi) <- Extract Extract PMg pain(a, () /a Signal (reap. 0) correaponde to a (reap. 6 a).
Pi
Nox
ci (a.))
No-a { (e: =0)
f(a 8) =,; of
/ 1} /(NNo>s + (No.al)
f(b > 0) 11 f(a 1a>0
9(1 1) 1 Lie Ps and P(O 1) =) 1 P(1 11)
o(1 0) Pinno] EieNo Pa and P(O 0) P(I (0)
/* Compute prediction normalzed vote
(1)00))1)
V(a f(ax
(1/00)110)
V(b> f(b a) Σ
(0.0)
/+ Ties are broken unifornly at random
ir (r Pib> a) then GLab
else G+Gubxa
Q QG GT V(Q)
return GT
D.3 Aggregated-SP
Algorithm describes the proposed Aggregated-SD aggregation approach. The algorithm takes as input the number of voters (n), number of alternatives (m), set of all subsets that voters voted on (S), voters' votes (fvt.) he(s))jEs), voters' predictions (fPr. heee). parameters a and 3 representing the conditional probabilities that would be returned for the predictions, and Vote Aggregation Rule (V). For voters and m alternatives, depending on the elicitation format, the voters will be providing votes (vig) and predictions (pi.)) as Top choices, Approvals((), or Rankings over a subset S, E S. Additionally, we use Borda, Copeland, and Maximin aggregation rule for V.
For every subset Sy, we aggregate the votes using V. resulting in the set of partial aggregated subsets represented by Q. Q is dictionary containing the alternative and its corresponding score according to the aggregation rule V. We now apply SP-Algorithm on Q. For every pair of alternatives (a, b) within partial aggregated subset Q,, we extract information about the conditional probability of their predictions (g(0))), (0)1),, g(1 ((][1)). Refer to Appendix D. for detailed explanation of how information is extracted for different elicitation formats. With this, the prediction normalized score (a > b) and (b > a) is calculated where we use the scores of the alternatives represented by ((a) and (b).. A higher prediction normalized score decides the correct ordering for each pair (a, b). Parsing all pairs, results in the complete ground-truth ordering of the alternatives.
20

---

ALGORITHM 3: Aggregated-St SP Aggregation
Input: Number of Voters n, Subsets allernatives S. Votes (v,, heissess Predictions (pi, Jest prohabilities and 0.5, and Vote-Aggreatiin mile
Q
GT
for
1....1 [S]
G
G - V(ucs)
9 QG
1a Apply pairwise SP- vo-voinng on aggregated votes and non aggragggett predictions
for each pair of alternatives (a. in Q; do Extract-Repors(vvg Jes b), B)
((vf- p he(-l
Na 0:
Nos
g(1 1) Mapus
Lic M.y and P(O 1]= P(1 g(1 0) [Nar Eien, and P(O 0) P(1 (0)
/+ Compute prediction normalized vote
11)
V(a Q(a))
Mof (2.0)
P(b Q(1) Σ
/+ Ties are broken unifornly at random
it V b) Pib a) then G7 Gr
else
GT Grubra
return GT
E Additional Details of Experimental Design
This section provides additional details about the MTurk study
Tutorial. Prior to engaging with each set of 6 questions within a specific elicitation format, par- ticipants completed a tutorial designed to evaluate their understanding of the voting process and prediction tasks. To proceed, participants had to accurately apply these beliefs within the voting and prediction framework, ensuring they were adequately prepared.
Reviews. Following the completion of each set of questions, participants were asked evaluate the preceding questions' elicitation format in terms of difficulty (ranging from "Very Easy" to "Very Difficult") and expressiveness (from "Very Little" "Very Significant"). Although question complexity was standardized within each domain, the domains themselves varied considerably in difficulty. To mitigate potential bias from implicit comparisons between two elicitation formats assigned to each participant, the sequence of domains in the first set of questions was mirrored in the subsequent set. This methodological approach ensured consistency and fairness in the evaluation of the clicitation formats, thereby enhancing the reliability of participants' feedback
Response qualifications & payment. To ensure reliable responses, established several qual- ification criteria for participants in our study on MTurk. Participants were required to have: (a) minimum approval rate of 90% for their previous tasks, (b) completed at least 100 tasks on the platform, and (c) specified the region as US and Canada (to ensure language proficiency). To check attentiveness the participants, we included an additional quiz that repeated one of the previous questions. The compensation structure included a base payment of 50 cents for completing the survey, which encompassed tutorials, questions, and evaluations. Additionally, a 50-cent bonus was offered for accurately completing the attentiveness quiz.
21

---

F Additional Details of Simulation
F.1 Parameter Inference for the concentric mixtures of Mallows model
To assess the accuracy of the simulations, the real and simulated data to infer the model parameters. This process allows us to compare the inferred parameters, thereby evaluating how effectively the model captures the underlying patterns in the data. Specifically, we infer the proportion of experts (p), dispersion parameters experts' votes (Øg-votes) and predictions (Ø E-prediccioon) as well as the dispersion parameters of non-experts' votes E-vte)) and predictions - For the real data, the distribution of these parameters can help in understanding the voting behavior of experts and non-experts. For the synthetic data, generated based on known parameters, we can check if the model accurately recovers these parameters.
fit the concentric mixtures of Mallows model both
The parameters are inferred using Bayesian inference, implemented through the No-U-Turn Sampler (NUTS) [24], extension of Hamiltonian Monte Carlo (HMC) available in Stan [8]. Before sampling, we calculate the Kendall- -Tu distance between the ground-truth ranking and the votes (Trotes) and predictions (Fpredictions) of all the voters. We then define the priors for our parameters as follows:
p B 2.5)
DE votes No0.15, 0.075)
PE- .V0.7,0.3)
O E-votss ON E Eperddiciionns
.V(0.7.0.)
.V(07,00.)
We then combine the likelihood of the parameters of our mixture model and perform inference over the following target function:
target+ = log mix (p:
N (T [ni]
0, Φ E-noees) + N (Tpredictions [ri] 0, F-peddittions
N (Tvores [n]
0, ΦN N + N (T
predictions [n] 0, ΦN predictions))
The log-likelinne ligelikli function incorporates the observed Tuotes and Tpredictions using the mixture model to account for the possibility that each voter could be an expert non-expert. We run four chains, each for 4000 iterations with 1000 iterations of warm-up for the NUTS algorithm. The algorithm explores the parameter space and updates the parameter estimates iteratively based on the input data and priors.
F.2 Synthetic Data Generation
The synthetic data was generated by simulating voter behavior using the concentric mixtures of Mallows model to construct preference rankings. This approach allows for the simulation of both expert and non-expert voters, with experts' votes closely aligning with a ground truth ranking and non-experts showing a broader dispersion in their preferences. The subsets to be voted on were generated as described in Section 4.
Voter Classification. To simulate the voting process effectively, voters initially classified into experts and non-experts. Since we need the experts to be the minority, we determine the probability of voter being an expert by sampling the proportion parameter as p (11, 2.5).
Voting Simulation: The voting behavior simulated by the concentric mixtures of Mallows model.
(r.(m; (m: -=))== = Pra(m; ,T, + (1 - p) (r. n (NNE)
where dE ~ N( 15, 0.075) and ΦNE ~ N(0.9, 0.4). Kendall-Tau distance is used the distance metric between the rankings n, and ...
22

---

G Missing Results and Analysis
G.1 Evaluation Metries
To quantitatively assess the alignment between the rankings derived from the voting rule, denoted as o', and the ground truth ranking, Ø", we use metrics described in the following subsections.
G.1.1 Pairwise hit rate
This metric evaluates the accuracy of the voting rule in identifying the correct relative order between pairs of alternatives, focusing on pairs with an increasing distance in their positions in the ground truth ranking:
Pairwise hit rate
Σ (i.))EP
[P|
where P represents the of pairs determined by the difference in their positions in o², and the indicator function.
G.1.2 Top-t hit rate
The Top-t hit rate metric quantitatively assesses the accuracy of a ranking algorithm by measuring the presence of the top-t elements from the ground truth ranking within the top-f elements of the predicted ranking. The formula for calculating the Top-t hit rate for rankings up a given t given by:
TTpp- elements in ground truth O Top-/ elements in predicted ranking
Top-t hit rate
t
G.1.3
Kendall- -au Correlation Coefficient
The Kendall- Tau correlation coefficient is measure of the ordinal association between two rankings:
a(o' n(n 1) i4j
- oj)) > 0)
where n is the number of elements in the ranking.
G.1.4 Spearman's p
The Spearman's p or Spearman correlation coefficient between a' and a* quantifies the rank correla- tion:
6- " - de n(n²2
p(o', oa*)
with d, o' (i) representing the rank difference of each element between o' and o².
Bootstrapping. To ensure the robustness of our estimates, we used bootstrapping to approximate the sampling distribution of various statistics. This method offers insights into the variance and bias of our estimates without relying on the stringent assumptions required by traditional parametric methods. Bootstrapping is particularly advantageous when the distribution of metric values is unknown does not conform to common distributional assumptions. After generating the bootstrapped distribution of metric values, we calculated the 95% confidence interval for each metric. Reporting these intervals alongside the metric values serves two purposes: it quantifies the uncertainty of our estimates, providing a transparent measure of their statistical precision, and it enhances the credibility of our findings by acknowledging the variability and potential error margins associated with our estimates.
G.2 Response-Time Difficulty and Expressiveness
Here, we measure the response time, difficulty, and expressiveness of the elicitation formats we used in our study.
23

---

We measure the average response time spent by the participants on each question for each elicitation format in our survey, This measure gives idea of the cognitive load perceived for each elicitation format [38].
Response Time:
Perceived Difficulty: In the review phase of our MTurk survey, we asked participants to select from "Very Easy', "Easy", 'Neutral', 'Difficult', and "Very Difficult' to subjectively indicate the case of answering questions within each elicitation format. This approach provides measure of the perceived difficulty associated with different elicitation formats.
Perceived Expressiveness: In the review phase of our MTurk survey, prompted partici- pants to select from the options 'Very Little', 'Little', 'Adequate', 'Significant', and "Very Significant" to indicate the amount of information they could convey using each elicitation format.
Cuzatiena
120
Time
60
20
Top-None
Top-Top
Top- Approval
Top-Rank
App(2)-App(2)
App(3)RRnkk
Rank None
Rank Top
Rank, Rank
Figure 8: The figure shows the average time spent by the participants on each tutorial and question for different elicitation formats across all domains. As expected, the participants spent similar or more time on the tutorial than on the questions. Additionally, the only elicitation format that has a statistical significance for the Questions Approval (3) -Rank, where more time is spent by the participants. Thus, for all other elicitation formats, the participants face similar cognitive complexity while responding to the questions.
G.3 Missing Figures for Predicting the Ground- -Tuthh Ranking
24

---

Difficulty
Expressiveness
100
100
82-
80-
Parcentage
Percentage
40-
40
Top Nore
Top-Top
Tap Approval
Top-Rank
Aspe (App(2
ApR Rank
Rank Nore
Rankt Rank
Top- None
Top App-Approval
Rop-Rank
App(2)-Ap11z)
Aop(3)-Rant
Rank None
Rank Top
Rank Ronk
Rank
Top
Figure 9: The figure shows the difficulty (easier better) and expressiveness (higher is better) of different elicitation formats as reported by the participants. A higher percentage of participants found the tasks to be relatively easy, indicating that they could answer questions effortlessly across all elicitation formats. Conversely, they demonstrated similar expressiveness not strongly leaning to either side of the scale.
(a) Top-Top
(b) Top- Toop-Approval. (3)
(c) Top-Rank
(d) Approval -2) Approval (2)
(c) Rank- Top
(f) Approval (3) Rann -R)-Rank
Figure 10: The plots show the Kendall-Tau Correlation between rankings derived from Common Aggregation Rules (blue), Partial-SP(red), and Aggregated-SP(red) for Top-Top, Top-Approval (3), Top-Rank, Approval (2) AApproval (2), Rank-Top, and Approval (3) RRan elicitation formats across Geography(C) Movies(M), and Paintings(P) domains. high Kendall-Tau Correlation implies higher pairwise alignment of alternatives between the ground-truth ranking and the aggregated ranking. The usage of different aggregation rules for Partial-SP and Aggregated-SP has similar impact on the outcome. However, the performance improves with an increase in information elicited as seen by the high correlation and increases statistical difference between the conventional methods and proposed methods. For example, Approval (2) -Aprroval (2) recovers ground-trute ranking more accurately than Top-Top. Note: We Schulze method in Rank-Top as it only works for preference data.
25

---

E.R T 5
Top-Top
(b) Approapp (3)
(c) Top-Rank
Coptneery
(d) Approval (2) -Approval (2)
(e) Rank-Tor
Approval (3) -Rank
Figure 11: The plots show the Spearman's p Correlation between rankings derived from Common Ag- gregation Rules (bluc), Partial-SP(red), and Aggregated-SPee (Pree)))) for Top-Top. Top-Rank, Approval (2) Approval (2), Rank-Top, and Approval (3) RRnk elicitation formats across Geography(G), Movies(M), and Paintings(P) domains. A high Spearman's Correlation implies higher alignment between the ground truth ranking and the aggregated ranking. The usage of Maximin aggregation rule for Partial-SP and Aggregated- SP has a better impact on the outcome as compared to other common aggregation rules. Additionally, the performance improves with an increase in information elicited as seen by the high correlation and increases statistical difference between the conventional methods and proposed methods. For example, Approval (3) -Rank recov- ers ground-truth ranking more accurately than Top-Approval (3). Note: We see Schulze method in Rank hank-Top as it only works for preference data.
(3),
26

---

G.4 Missing Figures for Partial-SP
Oorert Ras
Dorert MiyWen
15
20
(a) Top-Top
(b) Top-Approvee Approval (3)
Top-Rank
92
16
Dbrince
16
Dertence
Peneba Etrisnce
20
(d) Approval (2) AAprroval (2)
(c) Approval (3) Rank
(f) Rank-Topp
(g) Rank Rank
The figures show the effect of different elicitation formats for ground-truth recovery using Copeland and Partial-SP using metric defined in Section G.1.1. The metric assesses the number of pairs that were correctly predicted according to the aggregation rule based on their increasing dis- tance in the ground-truth ranking. Comparable performance between Approval (2) Approval (2), Approval (3) -Rank, and Rank-Rank show that eliciting Approvals on half the size of the subset recovers truth good as eliciting Ranking over the subset.
Figure 12:
27

---

Fortal
Furter (b) Top- Approvapr (3)
Coreet Furtur
(a) Top-Top
(c) Top-Rank Rank
Naai Carnic HE
Near Cared HELFMS
Gareed 3R
96
(d) Approval (2)-Appo (2)
(c) Approval (3) -Rank
(l Rank-Top
96
(g) Rank-aannaan Rank
Figure 13: The figures show the effect of different elicitation formats for ground-truth recovery using Copeland and Partial-SP using metric defined in Section G.1.2. We again observe comparable performance between Approval (2) AApproval (2), Approval (3) -Ran,,, and Rank Rank show that eliciting Approvals on half the size of the subset recovers truth as good as eliciting Ranking over the subset.
28

---

G.5 Missing Figures for Aggregated-SP
Mean Drenet Videe
Moir Conpr Mis
Moan Conee HR' Vile
15
T
Etrranes
(a) Top-Top
Top- Aep--pprovaal (3)
Top-Rank Rank
Moui Danni Wii
Mour Danid MiLSie
16 Dbrince
16
13
Peneba 15
20
(d) Approval(C (2) -Approval (2)
(c) Approval Raan
(f) Rank-Toor
Crrdd
(g) Rank-Rank
Figure 14: The figures show the effect of different elicitation formats for ground-truth recovery using Maximin and Aggregated SP using metric defined in Section G.1.1. Improved performance in Approval (3) RRank, and Rank-Rand are consistent with the observations made in Figure 12 except for Approval (2) Approval (2) where no statistical significance is observed.
G.6 Comparing Performance between Real and Simulated Data
Figure 16 shows the performance of Partial-SP with Copeland Aggregation on Real Data and Simulated Data using metries described in Section G.I.I and G.1.2. both of the metrics, we observe similar trends across various pairwise distances, and top-t metrics.
29

---

20
(a) Top
(b) Top- Aop-Approval(C (3)
(c) Top-Rank Rank
Lo
Lo
Hear Agpepeed
Cred
17
(d) Approval (2) (A)pAppoova1 (2)
(e) Approval (3) -Rank
(f) Rank-Top
Hain Carad HE 2
22
gg) Rank- Rank
Figure 15: The figures show the effect of different elicitation formats for ground-truth recovery
using Maximin and Aggregated-SP using metric defined in Appendix G.1.2. Improved performance, especially after 10, in Approval (2) Approval (2), Approval (3) -Rank, and Rank Rank consistent with the observations made in Figure 13.
Correct Hiss
0.3
Teal Deca SP
Poot Pato Vite
Keal Dusa St
Rool Bria Wite
Fairwise Distance
Top-t
(a) Correct-Hits Vs Pairwise distance.
(b) Correct Hits vs Top-t
Figure 16: Comparison of pairwise and Top-t hit rate for Copeland-Aggregatee Aggregated Partial-SP and Copeland rule for Rank-Ran RRan on Real and Simulated Data. Similar trends are noticed in real and simulated data.
30

---

H
Missing Proofs
H.1 Proof of Theorem (1)
Proof. We first establish lower bound on g(@ o') for any o'.
g(a o') = Σ Pro(ā o') (r, | Ø) 2 min Pra(a σ) Σ Prg o')=min (ra(a ()
a
8
Under the assumption of uniform prior we have,
Pr(x))
Pra(a lō) Σ
(r
())
Pr
1
Pry [))
(m- k)!
1
Σ
Σ
Prş(T' T)
(m - k.)!
T:rDd x n'mm'ba
d(w'm m)
d(n'. .x)
(m - k)!
1
Σ
p
PE
Z(øg)
+ + (1-p)
Z(ØNE)
ONE
(m k)) T:
1
Z(OE, m - k)
d(
+ (1 -p)
Z(ONE, m. k))
Z(
ONE
d(a.5)
7 m - k))
)
Z m m--ai k)
d(a.a)
p.
Z(ΦE)
+ + (1 p))
Z(ØNE)
.PNE
Here Z(o, m - -k)- -i- dd(TT)" Therefore, g(a lo') 2 x for any o,a'.
We will first prove a multiplicative concentration inequality the estimates g(- o) for any a. Now fix any a. By using lemma (1) we obtain that with probability at least 1 ő1 max, Ig(a' ()) - g(o' |o)l = 1o This implies that @(a' ( 2 g(o" [o) log(1/6)) Since n²
g(o' o) 2 μt in order to have g(o' 10) 2 (1 - =)g(o' | σ) is sufficient have 2
1og(1/61)
6.
Moreover, because of the fact that μ < 1, we also have J(o' σ) < (1 + (lta) |a). Finally, we can use union bound over all k! permutations a and substituting 8 = 8/ k!) we obtain that as long as
the number of samples from each permutation a is at least log (2k/6)
we have
e.p
8 2
Pr (Va, d', - s)g(( ()) s g(o' [0) + e)g(d (o)) 2
We now apply a similar concentration inequality for the frequency terms f(-). Since Pr, o+) μ for any we have f((r) 2 k. By argument very similar to the previous paragraph we have that as long as 2 v [og(2/8]
c' we guaranteed that (1 =)f(a) & f(a) (1 + (o) with probability
at least 1 - 8/2.
Now we provide lower bound on the estimate V(o*).
Hoo a+)) (1 (1=e)²
gio' (0))
= (1-8)²
-(0))= f(or))
o") (I+E) f(ot)
V(o*)
g(o 07) (1+s)
We now provide an upper bound on ((r) for any o0.
v(r) = f(7) Σ j0 7)
< (1+
g(o'1 7) {)
f(T) Σ
V(T)
g( o) E)
g(T σl) (1 (1-6) 2)
31

---

We now use lemma (.). i.. V(0*) 2 (T)...
2(1 2(1-E) V(o*) 2 (1 -1 ((o)) 2(1-a) ((T) 2 (1+e)a
(("))
V(T) > ((T)
E1+
(1+s)
as long as E
87-1 0.115. We substitute E = 0.1. Finally, observe that we pick the outcome with 2-+1
highest empirical prediction normalized vote (or) and with probability at least 6, the empirical prediction normalized vote of a. will be the highest, and will be picked the outcome.
Lemma 1 (Theorem 9 of [7]). Let X1,.... Xn be be i.dd drawn from a discrete distribution p= p=(pi...... (pi..... Pt)) and let Pj Ei= 1{x;=0] Then we have
log(1/6) n²
Pr max P6 po1 jE(A]
2 8
H.2 Separation Lemma
Lemma 2. Assume uniform prior and assumption / holds. Then for any ground truth a² over subset of size and any T with d((T,"') 2 we have, V(a*) 2 (T.)..
Proof. Using the definition of g(- 1) we can establish the following lower and upper bounds.
g(a o') = Σ Pr (o, Ø) 2 min Pr,(o Ø)
Σ Prg(ö (o') min Pra(a ())
5
gla o') PPii(0 P') (8) s Σ Pr Ø)
Pry(O oo') 1o')= Σ Pr.(a ()
Now we can establish the following lower and upper bounds on the nommmllizd- vote.
V(a) f(o) Σ g (o)
f(o)
f(o)
Σ g(o') lo) min, Pr, ō)
g o')
min, Pr,(o
(
Σ g(o' glo of) 2
S(a) (r, Eg gld 0) Es Pr, (8)
f(a)
V(o)= f(o)
E.
Now consider partial ranking 7 such that a(T, = 1. Then have,
((a")
Pra(a* o
V(o') 2 (r. [ō)
Σg Po (5)
and
f(T)
Pr,(T lo))
V(T)
(ō) ming Pr.(T] ō)
ming PT (5)
Under the assumption of uniform prior we have,
Pr(m)
Pr, 0
Σ REnDi
Pr.(o") (x)
Pr(ō)
Σ Pr. (T) (T)
(m k
Σ
Σ
Pry(T' (T)
(m k)
RiTTP n'in'po
dl d(n'.)) ,))
Σ
PE
+ + (1-p)
NNE Z(ĀNE)
(m k)
p
Z(ØE)
c(T")
d(ā.a")
ONF
d(a.a')
(m - k:)!
p
((EE)
+1 p)
7(ŌNE)
32

---

Here c(T) is a constant depending only on the subset T. Using the above identity we obtain the following lower bound on ((o*).
[ + (1 p) Z(ŌNE)
V(o') 2
A(t,,0))
Σa
+ (1 p)
UNE Z(uNE]
[(0E)
We can also obtain the following upper bound on V(T).
OF p Z(EE)
+ (1 p) [(NNE) E)
ONE
V(r) <
d(a.T)
d(p,rr)
ming P Z(dE)
+ (1 - p)
Z(6NE)
We now use the relationship p < (1 - p) and OÆ < ONN to improve the bounds. We will also write
[(0, k) = >a )d(..)).
2p
V(o') 2 2(1-p) ((dNe.kk)
Z(dNE)
Z(ØE)
1- p D((NE)) Z(ŌNE,
Z(ØR)
V(T) 5
2
Z(őn)
1 Z(ÓNE) 13/2
OE
k(h 1)/2
p Z(EE)
2p
[lonn)
Therefore, in order to have V((a)= 2 2V (T( we need the following inequality to hold.
P Z(ØE)
1
1 - - ((NNE))) _k/-1)/22
22
gg
--p ((ŌNE) Z(dNE,k)
P Z(dE)
I
Screenshots from our MTurk Survey
Here, we provide screenshots of different phases of our MTTurk survey.
Consent Pardiipetionn
Have Sunn with this HIT!
Figure 17: Preview and Consent Form
33

---

luaegle
Taterial (lank)
Wele seh offeren slll poeertad
prymee Sipeet
Figure 18: Tutorial for Rank-None Elicitation Format
Port (Your Opinion)
Part (Your View Others)
Opinton)
Were fuliiing mesii verve willi
O
O
3.
3.
All the Huker
O
K/.
4.
O
Wenchr Semee
Figure 19: Questions for Top-None and Top-Top Elicitation Format
ffour Opinion)
Part (oour Othhrn))
anarg Calerr io
and
spntec cau
lecing Acloneg"
tent coemmin Borton
respane Brd
lmnge
O
2.
ligers
O
2.
Ages
O
:
Iok phou
#
0
Hidepres
inlep
Figure 20: Questions for Top Approval(3) and Top-Rank Elicitation Format
34

---

Yourr Opinion)
Part (four Otthea)
Part (Ybrr Opinso))
Part (Your View Others)
you andile ost prpuiiiiia"
s-dd uremn
poperddrl
calevo for
Werh worrt Inforter toateet lighrrone oldior
Boor berteee de soviiiy onteral
1.
Hork
Hyrl
Phlapien
Pilycies
:
Piyee
Myes
Figure 21: Questions for Approval(2) Approval(2) and Approval Rank Elicitation Format
Taak
Part (Yorr Opinkon)
Part B (Your View Others)
Perl Tlour Opinion]
Han par filouiing uaui sholld
lmaglr that otlir kiceers
:
1.
South Aitis
2.
Ofra
3.
221
O
Figure 22: Questions for Rank-None and Rank-Top Top Elicitation Format
Taak
Onie
band Mylio peoting
:1
:1
Figure 23: Question for Rank-Ran Elicitation Format and Survey Questions
35

---

Review
Veytay
2.
Uibe
3.
Armqute
Nelna
4.
5.
Figure 24: Difficulty and Expressiveness Questions
36

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
37

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ gewuddee/ooeeuuuiieeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. //ipps.ccpppliiii wwtpcccccppppppccccceeeoooppccce http://gggiiees//deeuuuiiiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
39

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification:
The experiments can run on any computer.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
40

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp:::///euriss.... hccpppbiic/thhhcssieeeeeeeeeeeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
41

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswitheeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/modde part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper includes full text of instructions given to participants, screenshots, and details of compensation for the crowdsourcing experiment conducted during research.
42

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have obtained IRB approval for the crowdsourcing experiment conducted during research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "NaxIIByr60", "reviewer": "Reviewer_1pvg", "summary": "The paper extends the previous work of (Hosseini et al. 25) by overcoming a major weakness: eliciting a full ranking and a prediction about the ranking is too costly. The paper designed a method that elicit partial preferences and recover the full ranking by aggregating partial rankings. They empirically test the method through a large-scale crowdsourcing experiment on MTurk and show their approaches outperform conventional preference aggregation algorithms for the recovery of ground truth rankings. They analyze the collected data and demonstrate that voters' behavior in the experiment, including the minority of the experts, and the SP phenomenon, can be correctly simulated by a concentric mixtures of Mallows model. Finally, they provide theoretical bounds on the sample complexity of the proposed methods.", "questions": "None.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper tackles an intriguing problem and addresses a significant flaw in previous approaches: the impracticality of eliciting full rankings and making predictions about those rankings in real-world scenarios. The authors develop simple methods to mitigate this issue by eliciting and aggregating partial rankings. They test the effectiveness of their approach through real-world experiments and introduce a novel explanation of voters' behavior based on the data. I see the experimental results as the primary contribution of the work, given that previous literature on the topic has been largely theoretical. It is particularly nice that the authors apply their method in practical settings and prove its effectiveness. Additionally, their analysis of sample complexity offers a valuable theoretical contribution.", "weaknesses": "The primary technique follows the methodology of Hosseini et al. (25), with a relatively simple extension. Nonetheless, I do not see the technical contribution as the most important contribution of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your feedback and insightful comments.  If you have additional questions, do let us know and we will be happy to answer them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ff220YyXNT", "reviewer": "Reviewer_TQiC", "summary": "This paper studies the problem to recover the ground truth ordering over a large number of alternatives. The assumption is that the ground truth ranking is drawn from a prior, and each voter observes a noisy version of the ground truth. It was previously shown that the surprisingly popular (SP) algorithm could recover the ground truth even when experts are in minority. In this paper, the authors propose Aggregated-SP and Partial-SP, that ask voters to report vote and predictions on a subset of alternatives in terms of top alternative, partial rank, or an approval set. Experimental results show that the proposed algorithms outperform conventional preference aggregation algorithms for the recovery of ground truth rankings. The authors also provide theoretical bounds on the sample complexity of SP algorithms with partial rankings.", "questions": "* In the introduction, the authors mention that the Surprisingly Popular Voting algorithm can recover the ground-truth ranking. I wonder if this is a theoretical guarantee or empirical / experimental.\n* Have you compared different aggregation algorithms for other elicitation formats besides Rank-Rank as shown in Figure 4?\n* Is it possible to show that recovering the true partial ranking is impossible if Assumption 1 is not true?", "limitations": "The authors discuss limitations and future directions in the paper, including exploring the setting of SP beyond the majority-minority dichotomy, or with malicious voters.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper extends previous work on the SP algorithms to only require the voters to report predictions on a subset of alternatives instead of for all the alternatives. Human generated datasets are used to evaluate the proposed algorithms. The experiments are well designed and clearly described. The results are sufficiently analyzed and discussed. Finally, sample complexity is analyzed under a Mallows model distribution assumption.", "weaknesses": "The theoretical assumptions in Section 6 are somehow constrained, especially Assumption 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and insightful questions. Below we provide answers to the questions.\n\n\n> “In the introduction, the authors mention that the Surprisingly Popular Voting algorithm can recover the ground-truth ranking. I wonder if this is a theoretical guarantee or empirical / experimental.”\n\nTheorem 3 of Prelec et al. [36] guarantees that the Surprisingly Popular Voting (SP-Voting) can recover the ground-truth ranking with a probability of 1 as the number of voters approaches infinity (As written in line 46 in our paper). Note that this guarantee holds only with infinite data, and to the best of our knowledge, theorem 1 in our paper is one of the first results providing finite-sample guarantees for SP-type algorithms.\n\n> “Have you compared different aggregation algorithms for other elicitation formats besides Rank-Rank as shown in Figure 4?”\n\nYes, we have compared different aggregation algorithms for all the elicitation formats used in our study and have shown the results in Figure 9 and Figure 10 of Appendix G.3.\n\n\n> “Is it possible to show that recovering the true partial ranking is impossible if Assumption 1 is not true?”\n\n\n- If Assumption 1 is not true, then either $p$ (fraction of experts) is very small or the dispersion parameter of the non-experts ($\\phi_\\textrm{NE}$) is very large. First, suppose $\\phi_{\\textrm{NE}}$ is large (e.g. arbitrarily close to $1$) then the distribution $Pr_s(\\pi | \\pi^\\star, \\phi_{\\textrm{NE} })$ is almost a uniform distribution and has no information about the true rank $\\pi^\\star$. In such a scenario if $p$ is also very small ($\\approx 0$), then the signal distribution $Pr_s(\\pi_i | \\pi^\\star)$ is almost a uniform distribution, and, the prediction distribution of both the experts and the non-experts are similar. Both will predict approximate uniform distribution, and it is impossible to recover the true ranking. Once either $p$ increases or $\\phi_{\\textrm{NE}}$ decreases, then either the prediction distribution of the experts will change ($p$ bounded away from $0$) or the signal distribution will be concentrated around $\\pi^\\star$ (when $ \\phi_{\\textrm{NE}} \\ll 1$), making identification possible.\n\n- We would like to point out that for applying the SP algorithm to more than two alternatives, Prelec et. al. [36] also required a condition on the underlying data generating process ($P(v_i|s_i) > P(v_i|s_j)$). Our condition is adapted for the specific case of concentric Mallows model, and implies their condition. We believe it might be possible to obtain a better trade-off between the parameters $p$ and $\\phi_{\\textrm{NE}}$ through a more sophisticated analysis, but a condition specifying the trade-off between the two parameters is necessary for identifying the true rank.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ujYquwSrM2", "reviewer": "Reviewer_GckY", "summary": "The paper generalizes the surprisingly popular algorithm to the partial ranking setting. The prior method that considers this generalization only works when every agent provides her signal over the full ranking. The current paper considers how to elicit only partial information from agents and aggregate them using SP. The proposed method can scale with the number of items. Furthermore, the paper proposes a model to capture the responses from human agents, which can be used for theoretical analysis.", "questions": "See weakness", "limitations": "Limitations are fairly discussed", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "* Generalizing existing methods for broader real-world applications is very necessary. The paper, overall, is making non-trivial progress in this direction.\n* The paper proposes a theoretically trackable model to capture agents’ behaviors while reporting ranking information. If calibrated (I think we should be careful to claim calibration here, see weakness), the model can be useful for future research on ananlyzing crowdsourcing of ranking data.\n* The experiments are robust: tested on three datasets, measured using various methods, and compared with multiple baselines.", "weaknesses": "* First of all, the prior paper [25] seems to greatly discount the contribution (and effort) of this work. In particular, they use the same datasets, very similar experimental designs, and the same model. However, the current paper only speaks about the high-level differences between these two papers. In particular, as stated in line 98, [25] seems to elicit pairwise comparison from agents and then aggregate them, which is pretty much the proposed method Partial-SP. The only difference I can see is that Partial-SP allows k partial rankings while [25] only uses 2. I could be wrong about [25], but the authors should have a more detailed comparison. Also, it doesn’t seem the method in [25] is considered as a baseline. Why not?\n* In line 310, “As seen 310 from the posterior distributions for the dispersion parameters in Figure 5 synthetic data generation 311 process accurately replicates real data characteristics”. I’m not sure how “accurate this is as the distributions for non-expert seem to be way off. How confident can we trust the considered model for downstream theoretical analysis?\n* In line with the previous comment, is the same dataset used to fit the model, then compared with the simulated data as shown in Figure 5? Or was the dataset separated into a training set and a test set? If it’s the former, how do we know if Figure 5 is not overfitting the data? A lot of the content is left in the Appendix, but some of it is necessary to understand the results, so it might be better to have it back.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for insightful feedback. Below we provide detailed responses to the questions.\n\n> “First of all, the prior paper [25] seems to greatly discount the contribution (and effort) of this work. In particular, they use the same datasets, very similar experimental designs, and the same model.”\n\n- We would like to highlight that our work is significantly different than [25] and not a natural extension of prior work. [25] is interested in recovering a ground truth over (m=4) alternatives by presenting these same alternatives to all voters and asking for various reports. We, on the other hand, are interested in recovering ground truth over a large number of alternatives ($m\\sim 30$) by asking reports on smaller subsets ($k=5,6$). This introduces two new challenges – design of new elicitation formats for partial preferences (e.g. Approval(t)-Rank as mentioned in line 151), and figuring out how to extend SP-voting for handling partial preferences.\n\n- Although we adopt a randomized experimentation framework similar to [25], there are two differences. First, we need to determine which subsets (of size $k\\ll m$) we should use to elicit partial information. Having too many subsets will blow up the sample complexity, and having too few (or non-overlapping) subsets will render the problem impossible. Besides, as mentioned above, we also consider the design of new elicitation formats suitable for partial reports.\n\n- We don’t use the dataset used by [25], and in fact, the dataset is not even applicable for our setting. This is because, in their experiment, the voters provide reports (vote or prediction) under the instruction that the considered set of 4 alternatives are the only alternatives in the ground truth. Moreover, the subsets in the datasets are non-overlapping and cannot help us recover a ground truth over the $m$ alternatives.\n\n> “In particular, as stated in line 98, [25] seems to elicit pairwise comparison from agents and then aggregate them, which is pretty much the proposed method Partial-SP. The only difference I can see is that Partial-SP allows k partial rankings while [25] only uses 2.”\n\nFor handling partial preferences, we propose two methods – Partial-SP and Aggregated-SP. The first method, Partial-SP goes beyond just eliciting pairwise comparisons and aggregating them. The main idea is to first determine the correct partial rank for each subset (by eliciting any type of information be it approval, rank, etc.) and then aggregating them through a voting rule. As discussed in the introduction, applying the basic version of SP-voting requires eliciting $O(m^2)$ pairwise information which can be large. Hence the generalization of $k > 2$ is important and non-trivial. Besides, partial-SP applies SP-voting at each subset level and then aggregates them through a voting rule, and in that sense, it is significantly different than the prior work [25]. Furthermore, the second algorithm, Aggregated-SP, offers a different perspective on handling partial preferences in the SP voting framework. In lines 161-176 we only provide an outline of the two algorithms because of limited space, but we can definitely move some details from the appendix.\n\n> “Also, it doesn’t seem the method in [25] is considered as a baseline. Why not?”\n\nIt's important to note that SP-voting [25] requires pairwise preference data for all $O(m^2)$ pairs as it builds a tournament over the m alternatives by applying SP-voting on each pair. When m is large (e.g. $m\\sim 30$ for our setting) and it is impossible to collect human preferences (vote and report) for all pairs. Therefore, because of the missing data, SP-voting algorithm cannot be evaluated. Our novelty lies in generalizing SP-voting with partial preferences, eliciting O(mk) amounts of information and yet recovering the ground truth over m alternatives.\n\n\n> “ I’m not sure how “accurate\" this is as the distributions for non-expert seem to be way off. How confident can we trust the considered model for downstream theoretical analysis?”\n\n- Thank you for your observation. Upon re-evaluating our parameter inference approach, we identified that the scipy.stats.kendalltau function we used computes Kendall's tau correlation, and not Kendall's tau distance. This distinction led to discrepancies in the non-expert distributions between real and synthetic data. We have addressed this by implementing the correct Kendall's tau distance metric. Figures 1 and 2 in our NeurIPS Rebuttal plots file show the updated plots with a perfect fit, particularly for the **Movie** domain (figure 1).\n\n- When combining all datasets, we observe significant overlap in individual expert and non-expert distributions, as well as a prominent distinction between expert and non-expert groups in terms of Kendall's tau distance. Although there is some mismatch between the original data and synthetic data for the non-experts, we believe this is because of a limited number of samples, and heterogeneity of different domains. \n\n\n> “In line with the previous comment, is the same dataset used to fit the model, then compared with the simulated data as shown in Figure 5? Or was the dataset separated into a training set and a test set? If it’s the former, how do we know if Figure 5 is not overfitting the data? “\n\nYes, we used the whole dataset to fit the model, sampled from the posterior of the fitted model, and then checked the overlap. The practice of splitting the dataset into test and training sets is common in supervised learning setting. We work in a Bayesian setting where different types of checks (e.g. posterior predictive checks) are used to determine the fitness of a model. Additionally, we have a limited number of samples, and that's another reason we used the whole dataset to fit the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DnFXKsS9Ut", "reviewer": "Reviewer_227u", "summary": "Surprisingly popular voting allows to value expertise by eliciting both judgments and predictions over others' judgments. Its application to ranking is however challenged by the combinatorial size of reporting predictions over others' judgments. This paper provides practical solutions to adapt surprisingly popular voting to sparser elicitations. In particular, the authors apply this to MTurk's inputs, and show that their solutions outperform classical voting solutions. Theoretical guarantees of ground-truth recovery are also provided.", "questions": "Do I understand correctly that, in Partial-SP, we compute for each voter a set of pairwise comparisons $a \\succ b$, each of which unbiases non-expertise with SP? And that the resulting voter-wise sets of unbiased comparisons are aggregated through a voting rule?\n\nCorollary 1 says $n \\geq \\Omega(k!)$ inputs are needed *for all subsets of $S$*. Does this imply that the total number of inputs must be exponential in $S$?\n\nMore generally, I am surprised that, in the proposed Mallows models, $k!$ inputs are needed. I would have guessed that $k^2$ are sufficient, as it suffices to be correct on all comparisons of pairs of alternatives. I even feel that this insight should hold for SP with a non-expert ranking distribution that is not centered on $\\pi^\\star$. Would the authors agree?\n\nFirst, what value $s$ is chosen? More importantly, it feels that two human subjects will either likely have zero or many (around k/2) alternatives to compare in common. Why would this be preferable to random subsets? What about minimizing the number of pairs of subjects that have $< 2$ alternatives in common?\n\nI am quite open to increasing my rating, depending on the authors' response.", "limitations": "One limitation that failed to be highlighted is the dependency of the new algorithms on hyperparameters $\\alpha$ and $\\beta$. Not mentioning this is misleading as it suggests that the new algorithms are less arbitrary than they actually are.\n\nI am disturbed by the conclusion on \"political polling or collective moderation of online content\". While expertise is clearly lacking and not uniformly distributed, it is questionable the extent to which SP (and variants) should be applied to such issues, where there is arguably no ground truth (formally, there is no \"common prior\" of the SP model). I would have appreciated adding caveats, and perhaps a mention of the (exciting) social science / philosophy questions that generalizing SP to such topics would raise.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper provides original solutions to generalize surprisingly popular (SP) voting to sparse elicitation, which is strongly needed in many practical applications. As such, they address an important problem, with effective results.\n\nInterestingly, the paper actually provides two solutions, and empirically show that one clearly outperforms the other (and classical voting). This is a valuable insight for this research question.", "weaknesses": "I found the paper quite hard to read. Key comprehension elements, such as how sparse elicitations are turned into pairwise voting inputs, or the principles of the new algorithms, are poorly explained in the main text. Even Appendix D is hard to parse (what if $a$ and $b$ are both approved? What if none is selected? Do we consider $v_i^{(a,b)}$ undefined?), with many typos (in Algorithm 2 and Algorithm 3, I don't understand what $Q$, $Q_j$, $G$ and $GT$ stand for, and how they differ).\n\nThe simulations given concentric mixtures of Mallows models also yields good insight. However, I am not sure they are convincing, because of the fact that non-expert's rankings are statistically centered around the ground truth as well. If so, this greatly undermines the story-telling of the paper, which is about valuing expertise (through SP).\n\nThe theoretical guarantees seem far from tight.\n\nI am not quite convinced by the subset selection (line 177). \n\nThere are many imprecisions in the text:\n- Line 46: \"will provably recover\" => it should be stressed that this depends on many assumptions. Typically, SP requires Bayesian agents, with a common prior, which update correctly based on their signals, report truthfully their vote and are numerous to avoid statistical fluctuations.\n- Line 307: \"using Bayesian inference\" => As I understand, what was used was a MCMC sampling of the posterior, which only has statistical asymptotical properties. This is a far cry from Bayesian inference.\n- Line 131: \"definition 1\" => Not sure what this refers to. Do you mean \"equation (1)\" ?\n- Line 138: \"definition 3\" => Not sure what this refers to. Do you mean \"equation (3)\" ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. They are very helpful and we will incorporate them in the next version of the paper.\n\n> “Do I understand correctly that, in Partial-SP, we compute for each voter a set of pairwise comparisons $a\\succ b$ , each of which unbiases non-expertise with SP? ...”\n\nThe right way to think about Partial-SP is the following: for each subset (not voter), we compute a set of pairwise comparisons a>b after collecting votes from the voters. Then SP unbiases non-expertise to give us unbiased ranking over the subsets (each of size $k$). Then the resulting partial rankings are aggregated through a voting rule to generate a ranking over the $m$ alternatives.\n\n> “Corollary 1 says $n\\ge \\Omega(k!)$ inputs are needed for all subsets of S. Does this imply that the total number of inputs must be exponential in S?”\n\nCorollary 1 uses theorem 1 for the sample complexity results which derives bounds for a naive version of the SP algorithm. Theorem 1 treats each of the k! permutations as possible ground truths and picks the ranking with largest prediction-normalized score. The sample complexity is approximately $O(G \\sqrt{\\log G})$ where G is the number of possible ground truths. For $G=k!$ we get the sample complexity bound of $\\tilde{O}(k! \\sqrt{k})$. It is straightforward to consider a version that applies SP-voting to each pairwise preference and then aggregates them. For a pairwise preference, $G = 2$ and as long as $n \\ge O(\\sqrt{\\log(k/\\delta)}$ SP-voting can recover the pairwise preference with prob at least $1-\\delta/k^2$. Therefore, by a union bound, the total number of samples required to recover the ground truth over a subset of size k is $\\tilde{O}(k^2)$.\n\n> “More generally, I am surprised that, in the proposed Mallows models, k! inputs are needed...”\n\nAs mentioned above, the bound of $\\tilde{O}(k!)$ is needed only when one applies a naive version of the SP algorithm. The reviewer is right that the total sample complexity can be reduced to $O(|S| \\cdot k^2 \\log (|S| k / \\delta) )$ when one applies SP-voting to each pair within a subset. However, even with the naive version of the SP algorithm the total sample complexity $\\tilde{O}(|S| \\cdot k!)$ can be better than $m^2$ when $|S| = O(m)$ and $k = o(\\log m)$. This is precisely our setting since the subset size $k$ is $5$ or $6$ and the total number of alternatives in the ground truth is $m = 35$, and hence $k! < m^2$. \n\n> “First, what value s is chosen? … Why would this be preferable to random subsets? ...”\n\nWe chose stepsize $s$ to be $6$. Through simulation, we observed that increasing subset size ($k$) and step size ($s$) results get better but step size $6$ was a fair number to balance cognitive load and also create overlaps between subsets. \nThe main idea is to choose subsets that have alternatives in common. Then we can estimate the partial rankings of each subsets and use transitivity of rankings to infer a global ranking over the $m$ alternatives. If we choose subsets uniformly at random then we will need a lot of subsets (and hence a lot of voters) before we have high overlap among different subsets. This is the reason we fixed the choice of the subsets and then determined assignments to voters through an integer programming which guarantees that all the voters report approximately an equal number of comparisons.\n\n> “One limitation that failed to be highlighted is the dependency of the new algorithms on hyperparameters $\\alpha$ and $\\beta$ ...”\n\nWe have highlighted the dependency of the new algorithms on hyperparameters $\\alpha$ and $\\beta$ in the appendix (line 662). Using grid search, we observed that as long as $\\alpha > 0.5$ and $\\beta < 0.5$, the new algorithms recover similar results. This means that the algorithms are robust to the choices of the hyperparameters $\\alpha$ and $\\beta$.\n\n> “I am disturbed by the conclusion on \"political polling or collective moderation of online content\"...”\n\nThank you for raising the issue of extending SP to social science/philosophy questions! We would like to point out that SP is already being used in a couple of applications.\n\n- Polling opinions in political elections e.g. by CBC in Canadian elections.\n- Some conferences have been experimenting with SP type methods for ranking papers by asking reviewers to score a paper and also provide a prediction of what they think scores of others will be (EC and ICML this year).\n\nThe main difficulty is that most of the questions in these domains are inherently subjective with no ground truth. Therefore, it is difficult to determine the expertise of a voter and more research is needed to determine what should be the right framework for differentiating voters. \n\n\n> “The simulations given concentric mixtures of Mallows models also yields good insight. However, I am not sure they are convincing…”\n\n- Yes, the non-expert’s rankings are statistically centered around the ground truth, however, note that their distance from the ground truth ranking is around 0.7 in terms of average Kendall-Tau distance. There was a minor bug in our code and the new plots are attached with the rebuttal, see figures 1 and 2. An alternative would be a mixture model with two centers where experts and non-experts report close to their own centers (respectively $\\pi^\\star_E$, and $\\pi^\\star_{\\textrm{NE}}$). However, we believe this is not the right model for our setting as there is already an objectively correct ranking over all the alternatives. \n\n- Furthermore, to demonstrate that the second type of mixture model is unsuitable for our setting, we conducted a comparative analysis. We selected ground-truth as centers for experts, while for non-experts, we chose random centers within a [0,1] range of Kendall's tau distance from the ground truth. Figure 3 in our NeurIPS Rebuttal plot illustrates the resulting poor fit, highlighting a significant lack of overlap between the non-expert distributions in synthetic data compared to real data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

Many thanks for your feedback and insightful comments. Upon re-evaluating our parameter inference approach, we identified that the scipy.stats.kendalltau function we used computes Kendall's tau correlation, and not Kendall's tau distance. This distinction led to discrepancies in the non-expert distributions between real and synthetic data. We have addressed this by implementing the correct Kendall's tau distance metric. 

- Figures 1 and 2 in our NeurIPS Rebuttal plots file show the updated plots with a perfect fit, particularly for the Movie domain. When combining all datasets, we observe significant overlap in individual expert and non-expert distributions, as well as a more prominent distinction between expert and non-expert groups in terms of Kendall's tau distance. In fact, the votes of non-experts are now centered around 0.7 which is further away from the experts' votes.
 
- Figure 3 compares original data and synthetic data when experts and non-experts have different centers. However, we observed that model fit is poor for non-expert groups.

Meta-review: The reviewers were initially mixed on this paper, but throughout the rebuttal process they were raised significantly. The paper is overall well-motivated and has compelling empirical results. I had some concerns about writing quality, but the authors were responsive throughout the review process and the typos identified by reviewers are all easily fixable. The authors also concerningly had an error in one of their evaluation methods, but they claim to have fixed it. Lastly, they responded to (extremely minor) ethical considerations raised by ethics review for their use of MTurk. 

Personally, I found the paper compelling. The authors' main theoretical result for sample complexity isn't the most technically complex, but the paper's main contribution is on the empirical side. The main weakness identified by reviewers was a very related paper ([25]) that only differed in the size of the partial rankings elicited (pairwise vs. k). However, the authors did a good job explaining why this paper made a meaningful contribution past [25], and they will add it to the final draft.
Decision: Accept (poster)