ICML 2025

Online Clustering of Dueling Bandits
Zhiyong Wang 1 Jiahang Sun 2 Mingze Kong 3 Jize Xie 4 Qinghua Hu 5 John C.S. Lui 1 Zhongxiang Dai 3
Abstract
The contextual multi-armed bandit (MAB) is a
widely used framework for problems requiring se-
quential decision-making under uncertainty, such
as recommendation systems. In applications in-
volving a large number of users, the performance
of contextual MAB can be significantly improved
by facilitating collaboration among multiple users.
This has been achieved by the clustering of ban-
dits (CB) methods, which adaptively group the
users into different clusters and achieve collabo-
ration by allowing the users in the same cluster to
share data. However, classical CB algorithms typ-
ically rely on numerical reward feedback, which
may not be practical in certain real-world applica-
tions. For instance, in recommendation systems, it
is more realistic and reliable to solicit preference
feedback between pairs of recommended items
rather than absolute rewards. To address this lim-
itation, we introduce the first ”clustering of du-
eling bandit algorithms” to enable collaborative
decision-making based on preference feedback.
We propose two novel algorithms: (1) Cluster-
ing of Linear Dueling Bandits (COLDB) which
models the user reward functions as linear func-
tions of the context vectors, and (2) Clustering of
Neural Dueling Bandits (CONDB) which uses a
neural network to model complex, non-linear user
reward functions. Both algorithms are supported
by rigorous theoretical analyses, demonstrating
that user collaboration leads to improved regret
bounds. Extensive empirical evaluations on syn-
thetic and real-world datasets further validate the
effectiveness of our methods, establishing their
potential in real-world applications involving mul-
tiple users with preference-based feedback.
1The Chinese University of Hong Kong 2Tongji University
3The Chinese University of Hong Kong, Shenzhen 4Hong Kong
University of Science and Technolo...