ICML 2025

Selective Preference Aggregation
Shreyas Kadekodi 1 * Hayden McTavish 2 * Berk Ustun 1
Abstract
Many applications in machine learning and de-
cision making rely on procedures to aggregate
human preferences. In such tasks, individuals ex-
press ordinal preferences over a set of items by
voting, ratings, or comparing them. We then ag-
gregate these data into a ranking that reveals their
collective preferences. Standard methods for pref-
erence aggregation are designed to return rankings
that arbitrate conflicting preferences between in-
dividuals. In this work, we introduce a paradigm
for selective aggregation where we abstain from
comparison rather than arbitrate dissent. We sum-
marize collective preferences as a selective rank-
ing – i.e., a partial order that reflects all collective
preferences where at least 100 · (1 −τ)% of in-
dividuals agree. We develop algorithms to build
selective rankings that achieve all possible trade-
offs between comparability and disagreement, and
derive formal guarantees on their recovery and
robustness. We conduct an extensive set of ex-
periments on real-world datasets to benchmark
our approach and demonstrate its functionality.
Our results show how selective rankings can pro-
mote transparency and robustness by revealing
disagreement and abstaining from arbitration.
1
Introduction
Many of our most important systems rely on procedures
where we elicit and aggregate human preferences.In such
systems, we ask a group of individuals to express their
preferences over a set of items through votes, ratings, or
pairwise comparisons. We then use these data to order items
in a way that represents their collective preferences as a
group. Over the past century, we have applied this pattern to
reap transformative benefits from collective intelligence in
elections [21], online search [26], and model alignment [22].
*Equal contribution 1UCSD 2Duke University. Correspondence
to: Berk Ustun <berk@ucsd.edu>.
Proceedings of the 42 nd International Conf...